pantz.org banner
IPMI conflicts with RPC
Posted on 09-18-2008 02:46:52 UTC | Updated on 09-18-2008 03:38:13 UTC
Section: /hardware/ipmi/ | Permanent Link

I have a friend that had a really weird problem with a program that used RPC. Every once and a while the program that used RPC would open a port and tell the client to contact on the port it just opened. The client would try to do that and fail. Problem was this only happened randomly. Random problems are a pain to track down. The problem was eventually tracked down. The following is a description of what it was and the fix for it.

RPC uses a range of ports on Linux to allow clients to contact the server. By default these are 650-1023. RPC will pick a port at random from the range of ports. That is as long as port in the range is not already being used. Their servers have built in Baseband Management Controllers (BMC). BMC communicates with a BMC management utility (BMU) on a remote client using IPMI protocols. The BMC for the servers are in band and use the network controller on the motherboard to send and recieve data. The ports the BMC uses are 623 and 664.The thing is the controllers don't run any software in Linux to do this. So how do they work you ask? Well, they intercept the traffic on those ports before it gets into the OS's network stack. That means any data you send to those ports on those machines the OS will never see. You won't see it on any network sniffer on that machine at all. So any machine that RPC told to connect to it on 664 was failing. But it was random because that port would only come up from the pool of ports randomly.

Now that you know what the problem is how do you solve it? You could just run a dummy process out of xinetd to listen on port 664 (and 623 if your RPC uses ports that low) and be done with it.

You could also try to bump up your port range to start at 665. That can be done on some 2.6 linux kernels in proc.

echo 665 > /proc/sys/sunrpc/min_resvport

Restart your daemons that use RPC. If that works then you can put line sunrpc.min_resvport=665 in /etc/sysctl.conf so it sets on reboot. I have heard that some of the latest Linux kernels are going to up the port range to start after 665 to avoid this problem in the future.

RPC is most notably used by NFS. This can cause NFS mounts to fail randomly or just any other program that uses RPC.

Del.icio.us! | Digg Me! | Reddit!

Related stories


RSS Feed RSS feed logo
About


3com
3ware
alsa
alsactl
alsamixer
amd
android
apache
areca
arm
ati
auditd
awk
badblocks
bash
bind
bios
bonnie
cable
carp
cat5
cdrom
cellphone
centos
chart
cifs
cisco
cloudera
comcast
commands
comodo
compiz-fusion
corsair
cpufreq
cpufrequtils
cpuspeed
cron
crontab
crossover
cu
cups
cvs
database
dbus
dd
dd_rescue
ddclient
debian
decimal
dhclient
dhcp
diagnostic
diskexplorer
disks
dns
dos
dovecot
drac
dsniff
dvdauthor
e-mail
echo
editor
emerald
ethernet
expect
ext3
ext4
fat32
fedora
fetchmail
fiber
filesystems
firefox
firewall
flac
flexlm
floppy
flowtools
fonts
format
freebsd
ftp
gdm
gnome
greasemonkey
greylisting
growisofs
grub
hacking
hadoop
harddrive
hba
hex
hfsc
html
html5
http
idl
ie
ilo
intel
ios
iperf
ipmi
iptables
ipv6
irix
javascript
kde
kernel
kickstart
kmail
kprinter
krecord
kubuntu
kvm
lame
ldap
linux
logfile
lp
lpq
lpr
maradns
matlab
memory
mencoder
mhdd
mkinitrd
mkisofs
moinmoin
motherboard
mouse
movemail
mplayer
multitail
mutt
myodbc
mysql
mythtv
nagios
nameserver
netflow
nginx
nic
ntfs
ntp
nvidia
odbc
openbsd
openntpd
openoffice
openssh
openssl
opteron
parted
partimage
patch
perl
pf
pfflowd
pfsync
photorec
php
pop3
pop3s
ports
postfix
power
procmail
proftpd
proxy
pulseaudio
putty
pxe
python
qemu
r-studio
raid
recovery
redhat
router
rpc
rsync
samba
schedule
scsi
seagate
seatools
sed
sendmail
sgi
shell
siw
smtp
snort
solaris
soundcard
sox
spam
spamd
sql
sqlite
squid
ssh
ssh.com
ssl
su
subnet
subversion
sudo
sun
supermicro
switches
symbols
syslinux
syslog
systemrescuecd
t1
tcpip
tcpwrappers
telnet
terminal
testdisk
tftp
thttpd
thunderbird
timezone
ting
tools
tr
trac
tuning
tunnel
ubuntu
vi
wget
wiki
windows
windowsxp
wireless
wpa_supplicant
x
xauth
xfree86
xfs
xinearama
xmms
youtube
zdump
zic
zlib