pantz.org banner
Setting up spamd
Posted on 12-02-2005 03:33:00 UTC | Updated on 12-02-2005 03:33:00 UTC
Section: /software/spamd/ | Permanent Link

Spamd is a fake sendmail-like daemon created by the OpenBSD developers which rejects false mail (spam). It will attempt to waste the time and resources of the spam sender. Spamd is designed to work in conjunction with OpenBSD's pf, and should function on any system that has access to pf. The way it will be implimented below is through a feature called greylisting. It will use black, white, and grey lists to determine weather or not to pass the mail onto the actual mail server.

The first mail delivery attempt (by a server that has never attempted delivery before) connects to port 25 is redirected to port 8025 on localhost. This is done with pf redirect rules. If the delivering server is not currently in the whitelist pf table it goes directly to spamd. The delivering mail server is denied delivery by spamd and told to come back at a later time. The mail servers visit is logged in spamd database and a timer starts. During this time the marked server is not allowed to deliver it's mail. It can come back as many times as it wants in this time period but will always be denied delivery. The connecting server is in what is called a "grey state" and is put on a greylist. This means it is not blacklisted (never allowed) nor whitelisted (always allowed).

Once our specified deny time period ends the delivering mail server will hopefully attempt another delivery (most legit servers will). When it does it will be denied delivery once again. The difference this time is it's status in the spamd database will be changed from grey to white. It's ip address is put in the pf whitelist table and any delivery from this server is redirected straight to the mail server (port 25) on any future delivery attempts. The future delivery attempts for a whitelisted server is done via the whitelist table in pf. Once spamd puts the server's ip in this table the best match in the firewall rules is the whitelist table match. Any servers in this table are redirected to the mail server.

The pf whitelist table is mantained by the spamd deamon. It will insert and remove entries based on it's database. Spamd removes whitelist entries from the spamd database if no mail delivery activity has been seen from the whitelisted address by spamlogd within our specified number of hours from the initial time an address is whitelisted.

After being white listed the delivering server will try another delivery attempt. This time it will be redirected right to mail server.

Spamd is installed on OpenBSD by default. The following are configuration options we are going to use to run spamd. The "-5" option is for black-listed entries, it returns an error code 550 to the spammer. The "-S12" option will stutter greylisted connections for the specified amount of seconds, after which the connection is not stuttered. The next option is "-G25:4:864". The first 25 stands for pass time in mins. Spamd will not accept mail from the initial connection attempt time up to this set time (25). At this point the server is grey listed. The second number 4 is called grey expiration time (hrs). This is the maximum amount of time the server to try to make another delivery after the 25 min pass time period. So, the server can attempt a delivery between 25 mins and 4 hrs after the initial connection attempt and it will be whitelisted. But remember when it connects during this time period it will be denied and whitelisted at the same time. It will have to come back one last time to delivery it's mail. It can come back any time after it is whitelisted. The last number is called the whitelist experation time. It is the time you have in hours to send a message to the server after you have been whitelisted. If you don't send a message to the server from the time your whitelisted up to this time then you will be removed from the whitelist and have to go through the whole filtering process over again. "-b" tells spamd what ip to bind to. "-v" turns on verbose logging. "-nOpenMailServer" is the SMTP version banner that is reported upon initial connection. Change that to whatever you want. The spamd line is put in the /etc/rc.conf file. An example of the revelant lines from this file are below.

The /etc/rc.conf below has pf being turned on and points to it's config file. It will also turn on spamd (in greylisting mode) and sendmail (or if you installed and configured postfix then that). The rest of the lines are not needed to get spamd working but are there for reference.

# These are the settings needed for spamd and postfix and or sendmail to work.
pf=YES
pf_rules=/etc/pf.conf
sendmail_flags="-bd -q30m"
spamd_grey=YES
spamd_flags="-5 -S20 -G25:4:864 -b127.0.0.1 -v -nOpenMailServer"
syslogd_flags="-a /var/spool/postfix/dev/log"

# These settings are not needed to get spamd and postfix working they are just what I use.
# The rest of the options in the rc.conf are either left blank or turned off.
dhcpd_flags="fxp1"
ntpd_flags="-s"
sshd_flags="-4"

These are the lines you will need in your pf.conf file to get spamd working. Only the lines that involve spamd are below. You will have to fit these into your ruleset in the correct order (Macros, tables, options, traffic normalization, queuing, traslation, and packet filtering). The comments explain what the rules do. I have left in the rules for my MTA Postfix as an example of what you would need to do when using spamd and Postfix together. Postfix and spamd listen on localhost and connections for both are redirected to localhost from the outside. After putting the rules in restart PF (pfctl -f /etc/pf.conf).

# use a macro for the interface name, so it can be changed easily
ext_if = "fxp0"

# Spamd whitelist table
table <spamd-white> persist

# Outside servers whitelist. Uncomment if you use a generated whitelist like in script at bottom of page.
# table <whitelist> persist file "/etc/whitelist.txt"

# Outside servers whitelist redirect to postfix. Uncomment if used/needed. Line above goes with this.
# rdr on $ext_if proto tcp from  <whitelist> to ($ext_if) port smtp -> lo0 port smtp

# Spamd and Postfix redirect on external interface. 
# If in whitelist table go to Postfix. If not in whitelist table go to spamd.
rdr on $ext_if proto tcp from  <spamd-white> to ($ext_if) port smtp -> lo0 port smtp
rdr on $ext_if proto tcp from !<spamd-white> to ($ext_if) port smtp -> lo0 port spamd

# Postfix Mail Server (internal interface) redirect for internal hosts.
rdr on $int_if proto tcp from any to  $int_if  port smtp -> lo0 port smtp

# Allow traffic for spamd from outside. Synproxy the connection.
pass in quick on $ext_if inet proto tcp from any to lo0 port spamd flags S/SAFPRU synproxy state

# Allow traffic for postfix mail server from anyone on the outside and the inside.
pass in log on $ext_if inet proto tcp from any to lo0 port smtp flags S/SAFPRU synproxy state
pass in     on $int_if inet proto tcp from any to lo0 port smtp flags S/SAFPRU keep     state

Setup and save a very simple /etc/spamd.conf file for now. We will do blacklist checks with the MTA.

all:\
        ::

You should be able to just reboot your machine to have all of these services start. I would suggest doing it to be sure they all start when the machine is rebooted. After rebooting you should see spamd (spamd,tcpdump) running. Hopefully you will see your MTA running also (postfix or sendmail or whatever). Tcpdump is running for spamd to watch incoming connections from mail servers. It uses it to keep the spam database (spamdb) up to date.

After being setup you can try to send an e-mail to your machine. Here's to hoping you setup all your dns records correctly. If you did and the e-mail gets to your machine spamd should respond with the a general message of "Come back later". Now you have an entry in your spam database (spamdb). To check this just execute the program: spamdb. You will see output like the following below.

GREY|10.20.30.40|||1152284462|1152298862|1152298862|1|0

Then after comming back after our pass time but less then our greyexp time you will see a whitelist entry like the following

WHITE|10.20.30.40|||1152284462|1152286088|1155396513|2|0

For GREY or WHITE entries, the format is:

type|source ip|from|to|first|pass|expire|block|pass

The fields are as follows:

type       WHITE if whitelisted or GREY if greylisted.
source ip  IP address the connection originated from.
from       envelope-from address for GREY (empty for WHITE entries).
to         envelope-to address for GREY (empty for WHITE entries).
first      time the entry was first seen.
pass       time the entry passed from being GREY to being WHITE.
expire     time the entry will expire and be removed from the database.
block      number of times a corresponding connection received a
           temporary failure from spamd. Usually the blocked delivery
           attempts up to the passtime mins time. 
pass       number of times a corresponding connection has been seen
           to pass to the real MTA by spamlogd. Successful e-mails
           passed to postfix.

To see the time entries in a human readable format just use the date command with "-r" on the system. Times are in seconds since the Unix Epoch. For example:

date -r 1152286088

Will give the output:

Fri Jul  7 11:28:08 EDT 2006

Once the delivering server is whitelisted it can deliver the mail unhindered. The server will make the delivery and MTA will handle the mail transfer. That should be it. Spamd should be up and rolling.

One thing about Spamd. It can be a problem with webmail services like gmail and hotmail. The problem goes like this. When someone sends and e-mail from this service the first delivery is sent out by lets say server A. It will be greylisted like it is supposed to. Then the webmail system waits and tries delivery again but this time from server B. Sever B is then greylisted. It the webmail system waits for some time again like it is supposed to and tries to deliver the mail again this time from server C. Server C is then greylisted. It can go on and on like this never having the same server get back to our server in the right amount of time before the greylisting is removed. Then the process would have to start over again. This happens because large e-mail services use pools of servers to deliver mail. You can never expect to get the mail delivered from the same server. People who run greylisting servers try to get around this by whitelisting as many webmail servers then can find. You can comb through past e-mails from people that sent mail from these services and try to guess the naming scheme. There is a nice script that checks the SPF records and a mantained whilelist file. This will help in trying to whitelist valid servers. The results of this file need to be put in a whitelist table for PF. For anyone intrested the script looks like:

!/bin/sh
# Script from http://home.xnet.com/~ansible/openbsd_spamd_conf.html

FILE=/etc/whitelist.txt

rm -f $FILE
touch $FILE

for domain in \
        aol.com \
        apple.com \
        amazon.com \
        gmx.net \
        _spf.google.com \
        spf-a.hotmail.com \
        spf-b.hotmail.com \
        spf-c.hotmail.com \
        spf-d.hotmail.com \
        _spf-a.microsoft.com \
        _spf-b.microsoft.com \
        _spf-c.microsoft.com \
        mynethost.com

do
echo \#$domain >> $FILE;
dig $domain TXT +short | tr "\ " "\n" | grep ^ip4: | cut -d: -f2 >> $FILE;
done

echo '# http://cvs.puremagic.com/viewcvs/*checkout*/greylisting/schema/whitelist_ip.txt' >> $FILE;
ftp -o -
http://cvs.puremagic.com/viewcvs/*checkout*/greylisting/schema/whitelist_ip.txt \
    | awk '/^[^#]/ {print $0}' >> $FILE;

Whatever you do it's a pain but it may be a small price to pay to for the reduction in spam.

Del.icio.us! | Digg Me! | Reddit!

Related stories

Using spamd as a mail pit
Posted on 11-26-2003 03:33:00 UTC | Updated on 11-26-2003 03:33:00 UTC
Section: /software/spamd/ | Permanent Link

Spamd is a fake sendmail-like daemon which rejects false mail. If the pf packet filter is configured to redirect port 25 (SMTP) to this daemon, it will attempt to waste the time and resources of the spam sender. spamd is designed to be very efficient so that it does not slow down the receiving machine. Spam is never accepted, but always rejected with either a 450 or 550 error message. The normal way that spam has been dealt with in the past is to either accept and drop, or outright block. When configured to use 450 responses, spamd takes neither of these actions: it rejects the mail back to the senders' queue. Spamd is best started from rc.

In this config I use spamd on the machine as a straight tarpit. I have no MTA running to do mail transfer. If you want to know how to setup spamd with an MTA like postfix see my spamd configuration page on setting this up. In this configuration if you touch the server on port 25 you get pitted. Don't touch my 25! If your touching a port that has not been advertised to be open your usually up to no good. To lure in spammers if you don't have an static ip signup at dyndns.org for a free dynamic host name that changes with your ip. Run the ddclient software in OpenBSD packages that watches for an ip address to change and updates dyndns if it does. Then post fake e-mail addresses around the net using the dyndns or static hostnames in the e-mail address. Post on places like test newsgroups and free webpages where spammers like to scrape up e-mail addresses. But to be sure your passive about this on webpages do the text in the href mailto: tag but don't put it in the text area so the text will not show up on the webpage. This way you know the mailto: link was scraped from the page by a bot not a human. This way you can put it on webpages and it won't show up as text and will not effect the layout of the page. Some fun example e-mail addresses to post with your new host name (dyndns has a big list of host names) are things like: jail@thepit.dyndns.org or spam@timewaster.dyndns.org

1. Open /etc/spamd.conf and make it look like the one below. Save it. This will pit all connecitons.

# $OpenBSD: spamd.conf,v 1.9 2004/01/21 08:07:39 deraadt Exp $

all:\
        ::

2. Open /etc/rc.conf find the line spamd_flags= and put in "-b127.0.0.1 -v -s5 -nWelcome_To_The_Pit"

3. Put lines below in /etc/pf.conf to redirect to spamd. Put the rdr line at the top before the pass rule. Then save the file and restart pf (pfctl -f /etc/pf.conf).

#Replace any $ext_if with your external interface name. Ex. fxp0.

# redirect any connection from the outside to localhost port 8025 (spamd).
rdr on $ext_if proto tcp from any to port smtp -> 127.0.0.1 port spamd

# allow the access to spamd port on localhost. 
pass in quick on $ext_if inet proto tcp from any to lo0 port spamd synproxy state flags S/SA

4. Start spamd up with the command "/usr/libexec/spamd -b127.0.0.1 -v -s5 -nWelcome_To_The_Pit"

5. The logs are in /var/log/deamon by default. I've made a Perl script (below) that adds up all the stats and ouputs them to an html file. I have a webserver running on the internal interface and a cronjob set up (50 23 * * * /var/www/cgi-bin/PantzSpamdStats.pl >>/dev/null 2>&1) to run the script every night and dump it to the website directory. Then I can browse to it every day and check out how many spammers are trying to send mail to my fake passive e-mail addresses.

#!/usr/bin/perl

################################
# Start Configuration Settings #
################################

# Set to "all" of you want all files (compressed also) read.
# Set to "one" if you just want the one file named "spamd" read.
$oneorallfiles = "all";

# Path to spamd logfile(s). No "/" after last dir name. Ex. "/var/log".
$spamdpath = "/var/log";

# Spamd log file (daemon by default)
$spamdfile = "daemon";

# Path to output html file
$spamdhtmlfile = "/var/www/htdocs/spamdstats.shtml";

##############################
# End Configuration Settings #
##############################

#####Begin: Assembling date code.#####

($Second, $Minute, $Hour, $Day, $Month, $Year, $WeekDay, $DayOfYear, $IsDaylightSavings) = localtime(time);

$Month += 1;
$Year += 1900;
if ($Month < 10) { $Month = "0" . $Month; }
if ($Hour < 10) { $Hour = "0" . $Hour; }
if ($Minute < 10) { $Minute = "0" . $Minute; }
if ($Second < 10) { $Second = "0" . $Second; }
if ($Day < 10) { $Day = "0" . $Day; }

#####End: Assembling date code.#####

#####Begin: Get one or all spamd filenames.#####

if ($oneorallfiles eq "all") {
  @spamdfilenames = <$spamdpath/$spamdfile*>;
  @spamdfilenames = sort { $b cmp $a } @spamdfilenames;
} else {
  push(@spamdfilenames, "$spamdpath/$spamdfile");
}

#####End: Get one or all spamd filenames.#####

#####Begin: Read spamd file(s) and input data into a hash of arrays for sorting.#####

foreach $spamdlogfile (@spamdfilenames) {
  open(IN, "gzcat -f $spamdlogfile |") or die ("Can't open file. Permissions?. UserID?");
    while( <IN> ) {
      if ((/spamd/) && (/disconnected after/)) {
        ($date,$hostname,$daemonandid,$ipaddress,$seconds) =
        ($_ =~ /(\w+\s{1,2}\d+ \d+:.\d:.\d+) (.*) (.*)\: (\d+\.\d+\.\d+\.\d+)\: disconnected after (\d+)/);
        # Ipaddress is hash key. Update the 1st value in the array with seconds count calling old seconds count and
        # adding it to the new count. Update the 2nd value in the array by 1. Count is per unique ip address hit.
        $spamd_hash{$ipaddress} = [$spamd_hash{$ipaddress}[0] + $seconds,$spamd_hash{$ipaddress}[1] + 1];
      }
    }
  close(IN);
}

#####End: Read spamd file(s) and input data into a hash of arrays for sorting.#####

#####Begin: Output of HTML file.#####

open(SPAMDHTMLSTATS, ">$spamdhtmlfile") or die ("Can't create file");
print SPAMDHTMLSTATS "<html><HEAD><TITLE>Pantz Spamdlog Stats</TITLE></HEAD><div align=\"center\">\n";
print SPAMDHTMLSTATS "<b><font size=\"4\">Pantz Spamdlog Stats</font></b><br>\n";
print SPAMDHTMLSTATS "Script run on: $Month-$Day-$Year $Hour:$Minute:$Second <br><br>\n";
print SPAMDHTMLSTATS "<TABLE BORDER=\"1\"><tr><td align=\"center\"><b>IP Address</b></td>

<td align=\"center\"><b>Hits</b></td><td align=\"center\"><b>Seconds</b></td></tr>\n";

# Loop thru the hash of arrays. Sort on the second value in the arrays. Print it.
for $ips ( sort {  $spamd_hash {$b}[1]  <=>  $spamd_hash{$a}[1] }  keys %spamd_hash ) {
  print SPAMDHTMLSTATS "<tr><td align=\"center\">$ips</td>

  <td align=\"center\">$spamd_hash{$ips}[1]</td>
  <td align=\"center\">$spamd_hash{$ips}[0]</td></tr>\n";
  $totalseconds += $spamd_hash{$ips}[0];
  $totalhitcount += $spamd_hash{$ips}[1];
  $totaluniquehosts++;
}

$avgpitmin = sprintf("%4.1f",($totalseconds/60)/$totalhitcount);
$avgnumhits = sprintf("%4.1f",$totalhitcount/($totaluniquehosts));
$totalpitmin = sprintf("%4.1f",$totalseconds/60);
$totalpithr = sprintf("%4.1f",($totalseconds/60)/60);

print SPAMDHTMLSTATS "<tr><td align=\"center\"><b>Totals:</b></td>

<td align=\"center\"><b>$totalhitcount</b></td><td align=\"center\">
<b>$totalpitmin Mins<br>$totalpithr Hrs</b></td></tr>";

print SPAMDHTMLSTATS "<tr><td align=\"center\"><b>Averages:</b></td>

<td align=\"center\"><b>$avgnumhits</b></td>
<td align=\"center\"><b>$avgpitmin Mins</b></td></tr>";

print SPAMDHTMLSTATS "</html>";

close(SPAMDHTMLSTATS);

Del.icio.us! | Digg Me! | Reddit!

Related stories


RSS Feed RSS feed logo
About


3com
3ware
alsa
alsactl
alsamixer
amd
android
apache
areca
arm
ati
auditd
awk
badblocks
bash
bind
bios
bonnie
cable
carp
cat5
cdrom
cellphone
centos
chart
chrome
cifs
cisco
cloudera
comcast
commands
comodo
compiz-fusion
corsair
cpufreq
cpufrequtils
cpuspeed
cron
crontab
crossover
cu
cups
cvs
database
dbus
dd
dd_rescue
ddclient
debian
decimal
dhclient
dhcp
diagnostic
diskexplorer
disks
dkim
dns
dos
dovecot
drac
dsniff
dvdauthor
e-mail
echo
editor
emerald
ethernet
expect
ext3
ext4
fat32
fedora
fetchmail
fiber
filesystems
firefox
firewall
flac
flexlm
floppy
flowtools
fonts
format
freebsd
ftp
gdm
gmail
gnome
greasemonkey
greylisting
growisofs
grub
hacking
hadoop
harddrive
hba
hex
hfsc
html
html5
http
https
idl
ie
ilo
intel
ios
iperf
ipmi
iptables
ipv6
irix
javascript
kde
kernel
kickstart
kmail
kprinter
krecord
kubuntu
kvm
lame
ldap
linux
logfile
lp
lpq
lpr
maradns
matlab
memory
mencoder
mhdd
mkinitrd
mkisofs
moinmoin
motherboard
mouse
movemail
mplayer
multitail
mutt
myodbc
mysql
mythtv
nagios
nameserver
netflix
netflow
nginx
nic
ntfs
ntp
nvidia
odbc
openbsd
openntpd
openoffice
openssh
openssl
openvpn
opteron
parted
partimage
patch
perl
pf
pfflowd
pfsync
photorec
php
pop3
pop3s
ports
postfix
power
procmail
proftpd
proxy
pulseaudio
putty
pxe
python
qemu
r-studio
raid
recovery
redhat
router
rpc
rsync
ruby
saltstack
samba
schedule
screen
scsi
seagate
seatools
sed
sendmail
sgi
shell
siw
smtp
snort
solaris
soundcard
sox
spam
spamd
spf
sql
sqlite
squid
srs
ssh
ssh.com
ssl
su
subnet
subversion
sudo
sun
supermicro
switches
symbols
syslinux
syslog
systemrescuecd
t1
tcpip
tcpwrappers
telnet
terminal
testdisk
tftp
thttpd
thunderbird
timezone
ting
tls
tools
tr
trac
tuning
tunnel
ubuntu
unbound
vi
vpn
wget
wiki
windows
windowsxp
wireless
wpa_supplicant
x
xauth
xfree86
xfs
xinearama
xmms
youtube
zdump
zeromq
zic
zlib