pantz.org banner
Auditd crashes when overloaded
Posted on 10-06-2008 17:46:21 UTC | Updated on 10-06-2008 18:41:17 UTC
Section: /software/auditd/ | Permanent Link

On CentOS 5.2 I found out that the audit daemon will crash when put under heavy load with it's default settings. I was using the CIS Benchmarks to help lock down some machines and made some of the changes they suggest for auditd. After making the changes I ran one of the scripts to find files that did not belong to any user or group currently on the system. Here is that line.

for PART in $(grep -v '^#' /etc/fstab | awk '( $3 ~ "ext[23]" ) { print $2 }' );
 do
  find $PART -xdev -nouser -o -nogroup -print ;
 done

About halfway through the find the auditd would die with these messages in the syslog:

auditd[6135]: Cannot allocate audit reply
auditd[6135]: Cannot allocate audit reply, exiting
auditd[6135]: Error receiving audit netlink packet (No buffer space available)
auditd[6135]: Error setting audit daemon pid (No buffer space available)

The netlink error was because of the messages going to a syslog server getting cut off. That still did not explain why it died in the first place. After much googling of the error I found nothing. The only time I saw this error was from the source code of the original program. You know it's bad when that's the only spot the error shows up in.

After reading the man pages a few times for auditctl and bumping up the buffers (-b) to a massive number nothing would keep it from crashing on the find. Finally I saw the -r setting which is for rate limiting the messages. The -r setting in audit.rules file will set the limit in messages/sec. If the rate is non-zero and is exceeded, the failure flag is consulted by the kernel for action. The failure action could be do nothing, print a message, or just panic. Default panic action is to print a message. The default setting for rate limit is 0 which means no limit. With no limit the find must have been killing auditd. Setting the limit to 21000 (-r 21000) in the /etc/audit/audit.rules fixed it.

Just a warning. I believe this will keep some things from logging when it hits it's rate limit. So if your environment must have all audit logs then this might not be for you. The rate of 21000 is what I found to be high enough on my system so it would not crash auditd. You should experiment with bumping the limit higher if you can.

Del.icio.us! | Digg Me! | Reddit!

Related stories


RSS Feed RSS feed logo
About


3com
3ware
alsa
alsactl
alsamixer
amd
android
apache
areca
arm
ati
auditd
awk
badblocks
bash
bind
bios
bonnie
cable
carp
cat5
cdrom
cellphone
centos
chart
chrome
cifs
cisco
cloudera
comcast
commands
comodo
compiz-fusion
corsair
cpufreq
cpufrequtils
cpuspeed
cron
crontab
crossover
cu
cups
cvs
database
dbus
dd
dd_rescue
ddclient
debian
decimal
dhclient
dhcp
diagnostic
diskexplorer
disks
dkim
dns
dos
dovecot
drac
dsniff
dvdauthor
e-mail
echo
editor
emerald
ethernet
expect
ext3
ext4
fat32
fedora
fetchmail
fiber
filesystems
firefox
firewall
flac
flexlm
floppy
flowtools
fonts
format
freebsd
ftp
gdm
gmail
gnome
greasemonkey
greylisting
growisofs
grub
hacking
hadoop
harddrive
hba
hex
hfsc
html
html5
http
https
idl
ie
ilo
intel
ios
iperf
ipmi
iptables
ipv6
irix
javascript
kde
kernel
kickstart
kmail
kprinter
krecord
kubuntu
kvm
lame
ldap
linux
logfile
lp
lpq
lpr
maradns
matlab
memory
mencoder
mhdd
mkinitrd
mkisofs
moinmoin
motherboard
mouse
movemail
mplayer
multitail
mutt
myodbc
mysql
mythtv
nagios
nameserver
netflix
netflow
nginx
nic
ntfs
ntp
nvidia
odbc
openbsd
openntpd
openoffice
openssh
openssl
openvpn
opteron
parted
partimage
patch
perl
pf
pfflowd
pfsync
photorec
php
pop3
pop3s
ports
postfix
power
procmail
proftpd
proxy
pulseaudio
putty
pxe
python
qemu
r-studio
raid
recovery
redhat
router
rpc
rsync
ruby
saltstack
samba
schedule
screen
scsi
seagate
seatools
sed
sendmail
sgi
shell
siw
smtp
snort
solaris
soundcard
sox
spam
spamd
spf
sql
sqlite
squid
srs
ssh
ssh.com
ssl
su
subnet
subversion
sudo
sun
supermicro
switches
symbols
syslinux
syslog
systemrescuecd
t1
tcpip
tcpwrappers
telnet
terminal
testdisk
tftp
thttpd
thunderbird
timezone
ting
tls
tools
tr
trac
tuning
tunnel
ubuntu
unbound
vi
vpn
wget
wiki
windows
windowsxp
wireless
wpa_supplicant
x
xauth
xfree86
xfs
xinearama
xmms
youtube
zdump
zeromq
zic
zlib