pantz.org banner
Shell oneliners
Posted on 07-25-2007 00:13:00 UTC | Updated on 01-03-2010 22:02:22 UTC
Section: /software/shell/ | Permanent Link

Below are one off lines that perform different tasks at a command prompt. The commands can use any number of programs usually found on some type of Unix machine. Things like awk or perl.

Warning: These lines were tested on Linux with a bash shell using the GNU versions of each piece of software. Your mileage may vary with your version of *nix and your shell. The different programs (mostly GNU) used here may be different versions or not the GNU version on your system. This means they may work differently or use (or not have) different command line options. If need be check your man pages to adapt the commands below to your OS.

Any awk statements below assume the awk defaults which is spiting fields on spaces unless specified with -F. If using the awk statement and you want to work on a file instead of stdin then put a filename at the end of the awk statement. FYI: Awk refers to lines from files as "Records" and each "Record" has "fields". "Fields" are sections of characters (words) with delimiters between them.

Numbers and Counting

Print the total number of lines that have the name Bill.

awk '/Bill/{n++}; END {print n+0}' filename

Print line numbers using a tab instead of a space.

awk '{print FNR "\t" $0}' filename

Use awk to pull the seventh field (ex. url string) of each line of the logfile (logfile should be separated by spaces). Sort the fields then find the unique fields and count them. Then do a reverse sort on numeric count. Filter out anything but JPEG files and only give me the top 10 of that list. This is for things like counting unique hostnames or urls from a logfile.

awk '{print $7}' logfile | sort | uniq -c | sort -rn | grep "\.jpg" | head

Fields

Print the last field of each line of the file.

awk '{ print $NF }' filename

Count the lines in a file. Just like "wc -l".

awk 'END{print NR}' filename

Total the number of lines that contain the name Jose

awk '/Jose/{n++}; END {print n+0}' filename

Add up the numbers at the eighth position (field) of each line. Print the total.

awk '{ sum += $8 } END { print sum }' filename

If the fourth field starts with a number then print that fourth field.

awk '$4 ~ /^[0-9]/ { print $4 }' filename

Text Conversion and Substitution

Find and replace "dog" or "cat" or "bird" with "pet" on every line and print it out.

awk '{gsub(/dog|cat|bird,"pet");print}' filename

Find and replace "dog" with "cat" in every file with extension txt.

awk '{gsub("dog", "cat", $0); print > FILENAME}' *.txt

Find every line that begins with cat. In that line replace furry with nothing. Change the file called filename inline (-i).

sed -i '/^cat/{s/furry//}' filename

Find cat by itself on it's own line even if there are spaces or tabs before it or after it. Replace it with dog. Then print the line.

awk '{sub(/^[ \t]*cat .*$/,"dog");print}' filename

Find any line starting with the defined shell variable SHELLVAR (notice ' ' around it so it's evaluated). When the line is found substitute in foo or boo or coo with bar. Edit the file inline.

sed -i '/^'${SHELLVAR}'/s/\(foo\|boo\|coo\)/bar/' filename

From a unix os: convert DOS newlines (CR/LF) (removes the ^M) to Unix format. Works when each line ends with ^M (Ctrl-M).

awk '{sub(/\r$/,"");print}' filename

Remove all carriage returns from file and rewrite the edits back to same file. tr uses the octal form for cr.

tr -d "\015" < filename | tee > filename

From a unix os: Convert Unix newlines (LF) to DOS format.

awk '{sub(/$/,"\r");print} filename

Delete the leading whitespace (spaces or tabs) from front of each line. Text will end up flush left.

awk '{sub(/^[ \t]+/, ""); print}' filename

Delete the trailing whitespace (spaces or tabs) from end of each line.

awk '{sub(/[ \t]+$/, "");print}' filename

Delete leading and trailing whitespace from each line.

awk '{gsub(/^[ \t]+|[ \t]+$/,"");print}' filename

Delete the trailing whitespace (spaces or tabs) from end of each line.

awk '{sub(/[ \t]+$/, "");print}' filename

Insert 6 blank spaces at beginning of each line.

awk '{sub(/^/, "      ");print}' filename

Substitute "dog" with "cat" on lines that don't contain the word "pet".

awk '!/pet/{gsub(/dog/, "cat")};{print}' filename

Print the first 2 fields with a space between the output. Split the fields on the colon (field separator).

awk -F: '{print $1 " " $2}' filename

Swap the first 2 fields.

awk '{tmp = $1; $1 = $2; $2 = tmp; print}' filename

Remove the second field in each line and then print it.

awk '{ $1 = ""; print }' filename

Line Operations

Print the first 6 lines of a file.

awk 'NR <= 6' filename

Print the last line of a file

awk 'END{print}' filename

Print the lines matching the regular expression. (emulates grep).

awk '/regex_here/' filename

Print the lines that don't match the regular expression. (emulates grep -v).

awk '!/regex_here/' filename

Print the line before the regular expression match.

awk '/regex_here/{print x};{x=$0}' filename

Print the line after the regular expression match.

awk '/regex_here/{getline; print}' filename

Print the lines less than 50 characters.

awk 'length < 50' filename

Print the lines 20 through 30.

awk 'NR==20,NR==30' filename

Print the line 50.

awk 'NR==50 {print;exit}' filename

Print lines between the match starting at "Dog" and ending at "Cat".

awk '/Dog/,/Cat/' filename

File and Process Operations

Find a program by name from process listing that is not awk and kill it. Or try the programs pkill or killall.

ps aux | awk '/program_name/ && !/awk/ {print $2}' > kill

Create a 2 meg file (in 1 kilobyte blocks) in /tmp called zero. 1k can be changed to 1M for meg or 1G for gig in my version of dd.

dd if=/dev/zero of=/tmp/zero bs=1k count=2000

From the root dir (/) find all files with the .txt extention and delete them. Using xargs is faster than find's -exec.

find / -type f -name "*.txt" -print | xargs rm

To delete a file who's file name is a pain to define (ex. ^H^H^H) find it's inode number with the command "ls -il". Use the line below to find and delete a file who's (for example) inode number is 128128.

find . -inum 128128 | xargs rm

Mark the end of each line with a dollar sign so you can see where the filename ends. Good for finding file names with spaces at the end.

ls -lA | cat -e

To delete files with control characters in them like ^M or ^L use the control-V shell feature. This tells many shells to interpret the next input character as a literal character (instead of as a control character). Like below to delete a file with a space before the ctrl-L " ^L" you would issue the following keystrokes in this order (separated by commas) r,m, ,\, ,ctrl-v,ctrl-l. The \ escapes the space. The command looks like:

rm \ ^L

Synchronize files in a directory between 2 hosts using the program rsync. host1's /disk01 (source) is the remote host and /disk01 (destination) is a local directory. The destination is always made to look like the source even if files need to be deleted in the destination (--delete). The source's data is never touched. The source is always named first and the destination is always named second. Trailing / on the source as means copy the contents of this directory to the destination. Without the trailing / on the source you get the directory name copied with all it's files in it. Below uses ssh as the remote-shell program as the transport. It also turns on the lowest grade encryption to speed up the transfer.

rsync -av --delete --rsh="ssh -c arcfour -l root" host1.domain.lan:/disk01/ /disk01/

Misc

Take a file with a list of hostnames and login via ssh and get disk usage. SSH keys will need to be setup for auto login. Each command is back grounded so all commands are executed one after another.

for HOST in $(< ListOfHosts); do ssh $HOST `df -h` & done

Set group id and sticky bits on a dir.

chmod g=swrx,+t DIR/

Use wget grab all of a certain type of file listed on a web page and put them in the current dir. The example will use jpeg's.

wget -r -l1 --no-parent -A "*.jpg" http://www.website.com/pictures/

Del.icio.us! | Digg Me! | Reddit!

Related stories


RSS Feed RSS feed logo
About


3com
3ware
alsa
alsactl
alsamixer
amd
android
apache
areca
arm
ati
auditd
awk
badblocks
bash
bind
bios
bonnie
cable
carp
cat5
cdrom
cellphone
centos
chart
chrome
cifs
cisco
cloudera
comcast
commands
comodo
compiz-fusion
corsair
cpufreq
cpufrequtils
cpuspeed
cron
crontab
crossover
cu
cups
cvs
database
dbus
dd
dd_rescue
ddclient
debian
decimal
dhclient
dhcp
diagnostic
diskexplorer
disks
dkim
dns
dos
dovecot
drac
dsniff
dvdauthor
e-mail
echo
editor
emerald
ethernet
expect
ext3
ext4
fat32
fedora
fetchmail
fiber
filesystems
firefox
firewall
flac
flexlm
floppy
flowtools
fonts
format
freebsd
ftp
gdm
gmail
gnome
greasemonkey
greylisting
growisofs
grub
hacking
hadoop
harddrive
hba
hex
hfsc
html
html5
http
https
idl
ie
ilo
intel
ios
iperf
ipmi
iptables
ipv6
irix
javascript
kde
kernel
kickstart
kmail
kprinter
krecord
kubuntu
kvm
lame
ldap
linux
logfile
lp
lpq
lpr
maradns
matlab
memory
mencoder
mhdd
mkinitrd
mkisofs
moinmoin
motherboard
mouse
movemail
mplayer
multitail
mutt
myodbc
mysql
mythtv
nagios
nameserver
netflix
netflow
nginx
nic
ntfs
ntp
nvidia
odbc
openbsd
openntpd
openoffice
openssh
openssl
opteron
parted
partimage
patch
perl
pf
pfflowd
pfsync
photorec
php
pop3
pop3s
ports
postfix
power
procmail
proftpd
proxy
pulseaudio
putty
pxe
python
qemu
r-studio
raid
recovery
redhat
router
rpc
rsync
ruby
saltstack
samba
schedule
screen
scsi
seagate
seatools
sed
sendmail
sgi
shell
siw
smtp
snort
solaris
soundcard
sox
spam
spamd
spf
sql
sqlite
squid
srs
ssh
ssh.com
ssl
su
subnet
subversion
sudo
sun
supermicro
switches
symbols
syslinux
syslog
systemrescuecd
t1
tcpip
tcpwrappers
telnet
terminal
testdisk
tftp
thttpd
thunderbird
timezone
ting
tls
tools
tr
trac
tuning
tunnel
ubuntu
unbound
vi
vpn
wget
wiki
windows
windowsxp
wireless
wpa_supplicant
x
xauth
xfree86
xfs
xinearama
xmms
youtube
zdump
zeromq
zic
zlib