Below are one off lines that perform different tasks at a command prompt. The commands can use any number of programs usually found on some type of Unix machine. Things like awk or perl.

Warning: These lines were tested on Linux with a bash shell. Your milage may vary with your version of *nix and your shell. Also the different programs (mostly GNU) used here may be different versions on your system and may work differently or use different command line options. If need be check your man pages to adapt the commands below to your distro.

Any awk statements below assume the awk defauts which is spliting fields on spaces unless specified with -F. If using the awk statement and you want to work on a file instead of stdin then put a filename at the end of the awk statement. FYI: Awk refers to lines from files as "Records" and each "Record" has "fields". "Fields" are sections of characters (words) with delimeters between them.

Numbers and Counting

Print the total number of lines that have the name Bill.

awk '/Bill/{n++}; END {print n+0}'

Print line numbers using a tab instead of a space.

awk '{print FNR "\t" $0}'

Use awk to pull the seventh field (ex. url string) of each line of the logfile (logfile should be seperated by spaces). Sort the fields then find the unique fields and count them. Then do a reverse sort on numeric count. Filter out anything but JPEG files and only give me the top 10 of that list. This is for things like counting unique hostnames or urls from a logfile.

awk '{print $7}' logfile | sort | uniq -c | sort -rn | grep "\.jpg" | head

end the passwd file to awk. Print the 5th line using the delimiter ":". Send that to awk again and use built in var "NF" to show how many fields with spaces are in the current record.

cat /etc/passwd | awk -F ":" '{print $5}' | awk -F '{print NF}'

Fields

Print the last field of each line of the file.

awk '{ print $NF }' filename

Count the lines in a file. Just like "wc -l".

awk 'END{print NR}'

Total the number of lines that contain the name Jose

awk '/Jose/{n++}; END {print n+0}' file

Add up the numbers at the eighth position (field) of each line. Print the total.

awk '{ sum += $8 } END { print sum }'

If the fourth field starts with a number then print that fourth field.

awk '$4 ~ /^[0-9]/ { print $4 }'

Text Conversion and Substitution

Find and replace "dog" or "cat" or "bird" with "pet" on every line and print it out.

awk '{gsub(/dog|cat|bird,"pet");print}'

Find and replace "dog" with "cat" in every file with extension txt.

awk '{gsub("dog", "cat", $0); print > FILENAME}' *.txt

From a unix os: convert DOS newlines (CR/LF) (removes the ^M) to Unix format. Works when each line ends with ^M (Ctrl-M).

awk '{sub(/\r$/,"");print}'

Remove all carriage returns from file and rewrite the edits back to same file. tr uses the octal form for cr.

tr -d "\015" < filename | tee > filename

From a unix os: Convert Unix newlines (LF) to DOS format.

awk '{sub(/$/,"\r");print}

Delete the leading whitespace (spaces or tabs) from front of each line. Text will end up flush left.

awk '{sub(/^[ \t]+/, ""); print}'

Delete the trailing whitespace (spaces or tabs) from end of each line.

awk '{sub(/[ \t]+$/, "");print}'

Delete leading and trailing whitespace from each line.

awk '{gsub(/^[ \t]+|[ \t]+$/,"");print}'

Delete the trailing whitespace (spaces or tabs) from end of each line.

awk '{sub(/[ \t]+$/, "");print}'

Delete leading and trailing whitespace from each line.

awk '{gsub(/^[ \t]+|[ \t]+$/,"");print}'

Insert 6 blank spaces at beginning of each line.

awk '{sub(/^/, "      ");print}'

Substitute "dog" with "cat" on lines that don't contain the word "pet".

awk '!/pet/{gsub(/dog/, "cat")};{print}'

Print the first 2 fields with a space between the output. Split the fields on the colon (field separator).

awk -F ":" '{print $1 " " $2}'

Swap the first 2 fields.

awk '{tmp = $1; $1 = $2; $2 = tmp; print}' file

Remove the second field in each line and then print it.

awk '{ $1 = ""; print }'

Line Operations

Print the first 6 lines of a file.

awk 'NR <= 6' filename

Print the last line of a file

awk 'END{print}'

Print the lines matching the regular expression. (emulates grep).

awk '/regex_here/'

Print the lines that don't match the regular expression. (emulates grep -v).

awk '!/regex_here/'

Print the line before the regular expression match.

awk '/regex_here/{print x};{x=$0}'

Print the line after the regular expression match.

awk '/regex_here/{getline; print}'

Print the lines less than 50 characters.

awk 'length < 50'

Print the lines 20 through 30.

awk 'NR==20,NR==30'

Print the line 50.

awk 'NR==50 {print;exit}'

Print lines between the match starting at "Dog" and ending at "Cat".

awk '/Dog/,/Cat/'

File and Process Operations

Find a program by name from process listing that is not awk and kill it. Or just use pkill.

ps aux | awk '/program_name/ && !/awk/ {print $2}' > kill

Create a 2 meg file (in 1 kilobyte blocks) in /tmp called zero. 1k can be changed to 1M for meg or 1G for gig in my version of dd.

dd if=/dev/zero of=/tmp/zero bs=1k count=2000

From the root dir (/) find all files with the .txt extention and delete them. Using xargs is faster than find's -exec.

find / -type f -name "*.txt" -print | xargs rm

To delete a file who's file name is a pain to define (ex. ^H^H^H) find it's inode number with the command "ls -il". Use the line below to find and delete a file who's (for example) inode number is 128128.

find . -inum 128128 | xargs rm

Mark the end of each line with a dollar sign so you can see where the filename ends. Good for filenames with spaces at the end.

ls -lA | cat -e

To delete files with control characters in them like ^M or ^L use the control-V shell feature. This tells many shells to interpret the next input character as a literal character (instead of as a control character). Like below to delete a file with a space before the ctrl-L " ^L" you would issue the following keystrokes in this order (separated by commas) r,m, ,\, ,ctrl-v,ctrl-l. The \ escapes the space. The command looks like:

rm \ ^L

Synchronize files in a directory between 2 hosts using the program rsync. host1's /disk01 (source) is the remote host and /disk01 (destination) is a local directory. The destination is always made to look like the source even if files need to be deleted in the destination (--delete). The source's data is never touched. The source is always named first and the destination is always named second. Trailing / on the source as means copy the contents of this directory to the destination. Without the trailing / on the source you get the directory name copied with all it's files in it. Below uses ssh as the remote-shell program as the transport. It also turns on the lowest grade encryption to speed up the transfer.

rsync -avx --delete --rsh="ssh -c arcfour -l root" host1.domain.lan:/disk01/ /disk01/

Misc

Take a file with a list of hostnames and login via ssh and get disk usage. SSH keys will need to be setup for auto login. Each command is backgrounded so all commands are executed one after another.

for HOST in $(< ListOfHosts); do ssh $HOST `df -h` & done

Set group id and sticky bits on a dir.

chmod g=swrx,+t DIR/

Use wget grab all of a certain type of file listed on a web page and put them in the current dir. The example will use jpeg's.

wget -r -l1 --no-parent -A "*.jpg" http://www.website.com/pictures/

Del.icio.us! | Digg Me! | Reddit!

Related stories

• Perl notes
• Perl script to output block stats from PF log file
• Perl script to check for OpenBSD package updates
• Setting up a Postfix mail server
• Doing perl search and replace from the commandline

Daniel Robbins the President-CEO of Gentoo Technologies, Inc wrote a nice little article about enhancing your Bash prompt. If you want to know how to get that prompt to say what you want or change it's colors read on.

Daniel Robbins (drobbins@gentoo.org)
President-CEO, Gentoo Technologies, Inc.
September 2000

Why stick with the standard boring shell prompt when you can easily make it colorful and more informative? In this tip, Daniel Robbins will show you how to get your shell prompt just the way you like it, as well as how to dynamically update your X terminal's title bar.

As Linux/UNIX people, we spend a lot of time working in the shell, and in many cases, this is what we have staring back at us:

bash-2.04$

If you happen to be root, you're entitled to the "prestige" version of this beautiful prompt:

bash-2.04#

These prompts are not exactly pretty. It's no wonder that several Linux distributions have upgraded their default prompts that add color and additional information to boot. However, even if you happen to have a modern distribution that comes with a nice, colorful prompt, it may not be perfect. Maybe you'd like to add or change some colors, or add (or remove) information from the prompt itself. It isn't hard to design your own colorized, tricked-out prompt from scratch.

Prompt basics

Under bash, you can set your prompt by changing the value of the PS1 environment variable, as follows:

$ export PS1="> " >

Changes take effect immediately, and can be made permanent by placing the "export" definition in your ~/.bashrc file. PS1 can contain any amount of plain text that you'd like:

$ export PS1="This is my super prompt > " This is my super prompt >

While this is, um, interesting, it's not exactly useful to have a prompt that contains lots of static text. Most custom prompts contain information like the current username, working directory, or hostname. These tidbits of information can help you to navigate in your shell universe. For example, the following prompt will display your username and hostname:

$ export PS1="\u@\H > " drobbins@freebox >

This prompt is especially handy for people who log in to various machines under various, differently-named accounts, since it acts as a reminder of what machine you're actually on and what privileges you currently have.

In the above example, we told bash to insert the username and hostname into the prompt by using special backslash-escaped character sequences that bash replaces with specific values when they appear in the PS1 variable. We used the sequences "\u" (for username) and "\H" (for the first part of the hostname). Here's a complete list of all special sequences that bash recognizes (you can find this list in the bash man page, in the "PROMPTING" section):

Sequence	Description
\a	The ASCII bell character (you can also type \007)
\d	Date in "Wed Sep 06" format
\e	ASCII escape character (you can also type \033)
\h	First part of hostname (such as "mybox")
\H	Full hostname (such as "mybox.mydomain.com")

These prompts are especially handy for people who log in to various machines under various, differently-named accounts, since it acts as a reminder of what machine you're actually on and what privileges you currently have.

In the above example, we told bash to insert the username and hostname into the prompt by using special backslash-escaped character sequences that bash replaces with specific values when they appear in the PS1 variable. We used the sequences "\u" (for username) and "\H" (for the first part of the hostname). Here's a complete list of all special sequences that bash recognizes (you can find this list in the bash man page, in the "PROMPTING" section):

Sequence	Description
\a	The ASCII bell character (you can also type \007)
\d	Date in "Wed Sep 06" format
\e	ASCII escape character (you can also type \033)
\h	First part of hostname (such as "mybox")
\H	Full hostname (such as "mybox.mydomain.com")
\l	The name of the shell's terminal device (such as "ttyp4")
\j	The number of processes you've suspended in this shell by hitting ^Z
\n	Newline
\r	Carriage return
\s	The name of the shell executable (such as "bash")
\t	Time in 24-hour format (such as "23:01:01")
\T	Time in 12-hour format (such as "11:01:01")
\@	Time in 12-hour format with am/pm
\u	Your username
\v	Version of bash (such as 2.04)
\V	Bash version, including patchlevel
\w	Current working directory (such as "/home/drobbins")
\W	The "basename" of the current working directory (such as "drobbins")
\!	Current command's position in the history buffer
\#	Command number (this will count up at each prompt, as long as you type something)
\$	If you are not root, inserts a "$"; if you are root, you get a "#"
\xxx	Inserts an ASCII character based on three-digit number xxx (replace unused digits with zeros, such as "\007")
\\	A backslash
\[	This sequence should appear before a sequence of characters that don't move the cursor (like color escape sequences). This allows bash to calculate word wrapping correctly.
\]	This sequence should appear after a sequence of non-printing characters.

So, there you have all of bash's special backslashed escape sequences. Play around with them for a bit to get a feel for how they work. After you've done a little testing, it's time to add some color.

Colorization

Adding color is quite easy; the first step is to design a prompt without color. Then, all we need to do is add special escape sequences that'll be recognized by the terminal (rather than bash) and cause it to display certain parts of the text in color. Standard Linux terminals and X terminals allow you to set the foreground (text) color and the background color, and also enable "bold" characters if so desired. We get eight colors to choose from.

Colors are selected by adding special sequences to PS1 -- basically sandwiching numeric values between a "\e[" (escape open-bracket) and an "m". If we specify more than one numeric code, we separate each code with a semicolon. Here's an example color code:

"\e[0m"

When we specify a zero as a numeric code, it tells the terminal to reset foreground, background, and boldness settings to their default values. You'll want to use this code at the end of your prompt, so that the text that you type in is not colorized. Now, let's take a look at the color codes. Check out this screenshot:

Color chart

To use this chart, find the color you'd like to use, and find the corresponding foreground (30-37) and background (40-47) numbers. For example, if you like green on a normal black background, the numbers are 32 and 40. Then, take your prompt definition and add the appropriate color codes. This:

export PS1="\w> "

becomes:

export PS1="\e[32;40m\w> "

So far, so good, but it's not perfect yet. After bash prints the working directory, we need to set the color back to normal with a "\e[0m" sequence:

export PS1="\e[32;40m\w> \e[0m"

This definition will give you a nice, green prompt, but we still need to add a few finishing touches. We don't need to include the background color setting of 40, since that sets the background to black which is the default color anyway. Also, the green color is quite dim; we can fix this by adding a "1" color code, which enables brighter, bold text. In addition to this change, we need to surround all non-printing characters with special bash escape sequences, "\[" and "\]". These sequences will tell bash that the enclosed characters don't take up any space on the line, which will allow word-wrapping to continue to work properly. Without them, you'll end up with a nice-looking prompt that will mess up the screen if you happen to type in a command that approaches the extreme right of the terminal. Here's our final prompt:

export PS1="\[\e[32;1m\]\w> \[\e[0m\]"

Don't be afraid to use several colors in the same prompt, like so:

export PS1="\[\e[36;1m\]\u@\[\e[32;1m\]\H> \[\e[0m\]"

Xterm fun

I've shown you how to add information and color to your prompt, but you can do even more. It's possible to add special codes to your prompt that will cause the title bar of your X terminal (such as rxvt or aterm) to be dynamically updated. All you need to do is add the following sequence to your PS1 prompt:

"\e]2;titlebar\a"

Simply replace the substring "titlebar" with the text that you'd like to have appear in your xterm's title bar, and you're all set! You don't need to use static text; you can also insert bash escape sequences into your titlebar. Check out this example, which places the username, hostname, and current working directory in the titlebar, as well as defining a short, bright green prompt:

export PS1="\[\e]2;\u@\H \w\a\e[32;1m\]>\[\e[0m\] "

This is the particular prompt that I'm using in the colortable screenshot, above. I love this prompt, because it puts all the information in the title bar rather than in the terminal where it limits how much can fit on a line. By the way, make sure you surround your titlebar sequence with "\[" and "\]", since as far as the terminal is concerned, this sequence is non-printing. The problem with putting lots of information in the title bar is that you will not be able to see info if you are using a non-graphical terminal, such as the system console. To fix this, you may want to add something like this to your .bashrc:

if [ "$TERM" = "linux" ]
then
	#we're on the system console or maybe telnetting in
	export PS1="\[\e[32;1m\]\u@\H > \[\e[0m\]"
else
	#we're not on the console, assume an xterm
	export PS1="\[\e]2;\u@\H \w\a\e[32;1m\]>\[\e[0m\] " 
fi

This bash conditional statement will dynamically set your prompt based on your current terminal settings. For consistency, you'll want to configure your ~/.bash_profile so that it sources your ~/.bashrc on startup. Make sure the following line is in your ~/.bash_profile:

source ~/.bashrc

This way, you'll get the same prompt setting whether you start a login or non-login shell.

Well, there you have it. Now, have some fun and whip up some nifty colorized prompts!

Resources

rxvt is a great little xterm that happens to have a good amount of documentation related to escape sequences tucked in the "doc" directory included in the source tarball.

aterm is another terminal program, based on rxvt. It supports several nice visual features, like transparency and tinting.

bashish is a theme engine for all different kinds of terminals. Check out some great screenshots of bashish in action!

Del.icio.us! | Digg Me! | Reddit!

Related stories

• Shell oneliners