Pantz.org Technical Reference Site

Below are one off lines that perform different tasks at a command prompt. The commands can use any number of programs usually found on some type of Unix machine. Things like awk or perl.

Warning: These lines were tested on Linux with a bash shell. Your milage may vary with your version of *nix and your shell. Also the different programs (mostly GNU) used here may be different versions on your system and may work differently or use different command line options. If need be check your man pages to adapt the commands below to your distro.

Any awk statements below assume the awk defauts which is spliting fields on spaces unless specified with -F. If using the awk statement and you want to work on a file instead of stdin then put a filename at the end of the awk statement. FYI: Awk refers to lines from files as "Records" and each "Record" has "fields". "Fields" are sections of characters (words) with delimeters between them.

Numbers and Counting

Print the total number of lines that have the name Bill.

awk '/Bill/{n++}; END {print n+0}'

Print line numbers using a tab instead of a space.

awk '{print FNR "\t" $0}'

Use awk to pull the seventh field (ex. url string) of each line of the logfile (logfile should be seperated by spaces). Sort the fields then find the unique fields and count them. Then do a reverse sort on numeric count. Filter out anything but JPEG files and only give me the top 10 of that list. This is for things like counting unique hostnames or urls from a logfile.

awk '{print $7}' logfile | sort | uniq -c | sort -rn | grep "\.jpg" | head

end the passwd file to awk. Print the 5th line using the delimiter ":". Send that to awk again and use built in var "NF" to show how many fields with spaces are in the current record.

cat /etc/passwd | awk -F ":" '{print $5}' | awk -F '{print NF}'

Fields

Print the last field of each line of the file.

awk '{ print $NF }' filename

Count the lines in a file. Just like "wc -l".

awk 'END{print NR}'

Total the number of lines that contain the name Jose

awk '/Jose/{n++}; END {print n+0}' file

Add up the numbers at the eighth position (field) of each line. Print the total.

awk '{ sum += $8 } END { print sum }'

If the fourth field starts with a number then print that fourth field.

awk '$4 ~ /^[0-9]/ { print $4 }'

Text Conversion and Substitution

Find and replace "dog" or "cat" or "bird" with "pet" on every line and print it out.

awk '{gsub(/dog|cat|bird,"pet");print}'

Find and replace "dog" with "cat" in every file with extension txt.

awk '{gsub("dog", "cat", $0); print > FILENAME}' *.txt

From a unix os: convert DOS newlines (CR/LF) (removes the ^M) to Unix format. Works when each line ends with ^M (Ctrl-M).

awk '{sub(/\r$/,"");print}'

From a unix os: Convert Unix newlines (LF) to DOS format.

awk '{sub(/$/,"\r");print}

Delete the leading whitespace (spaces or tabs) from front of each line. Text will end up flush left.

awk '{sub(/^[ \t]+/, ""); print}'

Delete the trailing whitespace (spaces or tabs) from end of each line.

awk '{sub(/[ \t]+$/, "");print}'

Delete leading and trailing whitespace from each line.

awk '{gsub(/^[ \t]+|[ \t]+$/,"");print}'

Delete the trailing whitespace (spaces or tabs) from end of each line.

awk '{sub(/[ \t]+$/, "");print}'

Delete leading and trailing whitespace from each line.

awk '{gsub(/^[ \t]+|[ \t]+$/,"");print}'

Insert 6 blank spaces at beginning of each line.

awk '{sub(/^/, " ");print}'

Substitute "dog" with "cat" on lines that don't contain the word "pet".

awk '!/pet/{gsub(/dog/, "cat")};{print}'

Print the first 2 fields with a space between the output. Split the fields on the colon (field separator).

awk -F ":" '{print $1 " " $2}'

Swap the first 2 fields.

awk '{tmp = $1; $1 = $2; $2 = tmp; print}' file

Remove the second field in each line and then print it.

awk '{ $1 = ""; print }'

Line Operations

Print the first 6 lines of a file.

awk 'NR <= 6' filename

Print the last line of a file

awk 'END{print}'

Print the lines matching the regular expression. (emulates grep).

awk '/regex_here/'

Print the lines that don't match the regular expression. (emulates grep -v).

awk '!/regex_here/'

Print the line before the regular expression match.

awk '/regex_here/{print x};{x=$0}'

Print the line after the regular expression match.

awk '/regex_here/{getline; print}'

Print the lines less than 50 characters.

awk 'length < 50'

Print the lines 20 through 30.

awk 'NR==20,NR==30'

Print the line 50.

awk 'NR==50 {print;exit}'

Print lines between the match starting at "Dog" and ending at "Cat".

awk '/Dog/,/Cat/'

File and Process Operations

Find a program by name from process listing that is not awk and kill it. Or just use pkill.

ps aux | awk '/program_name/ && !/awk/ {print $2}' > kill

Create a 2 meg file (in 1 kilobyte blocks) in /tmp called zero. 1k can be changed to 1M for meg or 1G for gig in my version of dd.

dd if=/dev/zero of=/tmp/zero bs=1k count=2000

From the root dir (/) find all files with the .txt extention and delete them. Using xargs is faster than find's -exec.

find / -type f -name "*.txt" -print | xargs rm

To delete a file who's file name is a pain to define (ex. ^H^H^H) find it's inode number with the command "ls -il". Use the line below to find and delete a file who's (for example) inode number is 128128.

find . -inum 128128 | xargs rm

Mark the end of each line with a dollar sign so you can see where the filename ends. Good for filenames with spaces at the end.

ls -lA | cat -e

To delete files with control characters in them like ^M or ^L use the control-V shell feature. This tells many shells to interpret the next input character as a literal character (instead of as a control character). Like below to delete a file with a space before the ctrl-L " ^L" you would issue the following keystrokes in this order (separated by commas) r,m, ,\, ,ctrl-v,ctrl-l. The \ escapes the space. The command looks like:

rm \ ^L

Synchronize files in a directory between 2 hosts using the program rsync. host1's /disk01 (source) is the remote host and /disk01 (destination) is a local directory. The destination is always made to look like the source even if files need to be deleted in the destination (--delete). The source's data is never touched. The source is always named first and the destination is always named second. Trailing / on the source as means copy the contents of this directory to the destination. Without the trailing / on the source you get the directory name copied with all it's files in it. Below uses ssh as the remote-shell program as the transport. It also turns on the lowest grade encryption to speed up the transfer.

rsync -avx --delete --rsh="ssh -c arcfour -l root" host1.domain.lan:/disk01/ /disk01/

Misc

Take a file with a list of hostnames and login via ssh and get disk usage. SSH keys will need to be setup for auto login. Each command is backgrounded so all commands are executed one after another.

for HOST in $(< ListOfHosts); do ssh $HOST `df -h` & done

Set group id and sticky bits on a dir.

chmod g=swrx,+t DIR/

Del.icio.us! | Digg Me! | Reddit!

Numbers and Counting

Print the total number of lines that have the name Bill.

Print line numbers using a tab instead of a space.

end the passwd file to awk. Print the 5th line using the delimiter ":". Send that to awk again and use built in var "NF" to show how many fields with spaces are in the current record.

Fields

Print the last field of each line of the file.

Count the lines in a file. Just like "wc -l".

Total the number of lines that contain the name Jose

Add up the numbers at the eighth position (field) of each line. Print the total.

If the fourth field starts with a number then print that fourth field.

Text Conversion and Substitution

Find and replace "dog" or "cat" or "bird" with "pet" on every line and print it out.

Find and replace "dog" with "cat" in every file with extension txt.

From a unix os: convert DOS newlines (CR/LF) (removes the ^M) to Unix format. Works when each line ends with ^M (Ctrl-M).

From a unix os: Convert Unix newlines (LF) to DOS format.

Delete the leading whitespace (spaces or tabs) from front of each line. Text will end up flush left.

Delete the trailing whitespace (spaces or tabs) from end of each line.

Delete leading and trailing whitespace from each line.

Delete the trailing whitespace (spaces or tabs) from end of each line.

Delete leading and trailing whitespace from each line.

Insert 6 blank spaces at beginning of each line.

Substitute "dog" with "cat" on lines that don't contain the word "pet".

Print the first 2 fields with a space between the output. Split the fields on the colon (field separator).

Swap the first 2 fields.

Remove the second field in each line and then print it.

Line Operations

Print the first 6 lines of a file.

Print the last line of a file

Print the lines matching the regular expression. (emulates grep).

Print the lines that don't match the regular expression. (emulates grep -v).

Print the line before the regular expression match.

Print the line after the regular expression match.

Print the lines less than 50 characters.

Print the lines 20 through 30.

Print the line 50.

Print lines between the match starting at "Dog" and ending at "Cat".

File and Process Operations

Find a program by name from process listing that is not awk and kill it. Or just use pkill.

Create a 2 meg file (in 1 kilobyte blocks) in /tmp called zero. 1k can be changed to 1M for meg or 1G for gig in my version of dd.

From the root dir (/) find all files with the .txt extention and delete them. Using xargs is faster than find's -exec.

To delete a file who's file name is a pain to define (ex. ^H^H^H) find it's inode number with the command "ls -il". Use the line below to find and delete a file who's (for example) inode number is 128128.

Mark the end of each line with a dollar sign so you can see where the filename ends. Good for filenames with spaces at the end.

Misc

Take a file with a list of hostnames and login via ssh and get disk usage. SSH keys will need to be setup for auto login. Each command is backgrounded so all commands are executed one after another.

Set group id and sticky bits on a dir.

Related stories