pantz.org banner
Perl script to output block stats from PF log file
Posted on 10-07-2007 01:13:00 UTC | Updated on 11-01-2008 03:50:30 UTC
Section: /software/pf/ | Permanent Link

My OpenBSD firewalls all have default block policies. Any connection attempt not initiated from the inside will be blocked by the firewall and logged in the PF log files. Usually these hits are people/scripts scanning for open ports. I thought it would be fun to see what ports people are scanning for and what ip's they are from.

To do this I needed a script to parse the log files and add up all blocks. So I made a perl script to do this. I am not a fantastic Perl programmer but the script does what I need it to. There is not much error checking but like I said it works. It is not intended for gigantic log files but it does not do to bad with them. Feel free to make modifications and if your so inclined send them to me. I just might throw them up if I like them. The script runs from a cron job and dumps the stats to an html file on my webservers. I can access it from a browser on the firewall every night to see who's scanning for what. Watch out for the settings like thresholds. If you set them to high you might not get any data back. Same goes for turning on port filtering.

Script Output Example

Pantz PFlog Stats

Log file Data from 10-05-2007 21:00:14 to 10-06-2007 13:24:02

Showing data from 10-06-2007 13:20:00 to 10-06-2007 13:24:00

Script run on: 10-06-2007 17:54:08

Source IP # of blocks in
206.248.132.102 2
221.208.208.95 2
202.97.238.200 1
202.97.238.202 1
82.166.13.106 1
221.208.208.97 1
Destination Port# of blocks in
1026 4
1027 2
7381 2






206.248.132.102
Destination Port # of blocks in
7381 2
202.97.238.200
Destination Port # of blocks in
1026 1
202.97.238.202
Destination Port # of blocks in
1027 1
221.208.208.97
Destination Port # of blocks in
1026 1
82.166.13.106
Destination Port # of blocks in
1026 1
221.208.208.95
Destination Port # of blocks in
1027 1
1026 1






1027
Source IP # of blocks in
202.97.238.202 1
221.208.208.95 1
1026
Source IP # of blocks in
202.97.238.200 1
221.208.208.97 1
82.166.13.106 1
221.208.208.95 1
7381
Source IP # of blocks in
206.248.132.102 2

Below is the code for the perl script that generates the example above. Just configure it and run it. It's pretty well commented on what to do.

#!/usr/bin/perl -s

# This perl script generates some stats (in html format) for OpenBSD's firewall
# PF. It was made with a home firewall in mind. One that has a default block
# in policy. Which means any packets hitting the firewall inbound that were
# not allowed by other rules or not part of another connection don't belong.
# These are the ones we want to see. Most likely these have malicious intent.

# The stats are generated from the binary formated PF logs. OpenBSD's
# implementation of Tcpdump is the only Tcpdump that can read this log. This
# script was tested on an OpenBSD 4.2 machine with the version of PF and
# Tcpdump that comes with 4.2. The script only does TCP and UDP stats
# currently. Also, it only does stats for block in rules and only for
# destination ports and source IP addresses.

# The external programs needed to run this script are: Tcpdump (OpenBSD's
# version), gzcat, and of course Perl. The "host" program is need if you want
# to do hostname lookups. All programs mentioned come with OpenBSD by default.
# You will also need to run this script as root because the permissions on
# the pflog files can only be read by root.

# I have recently tested this script on FreeBSD 6.3 successfully. The
# only thing to watch out for is FreeBSD compresses it's logfiles using 
# bzip2 by default. You will need to change the gzcat line in the script
# to bzcat for it to work. FreeBSD's Tcpdump was modified to work with PF.

# I welcome any additions or added features to the script just send them to
# webmaster at pantz dot org. This code is free to distribute as long as credit
# is given to the website (pantz.org) in the modified code.

use Time::Local;

################################
# Start Configuration Settings #
################################

# You can set any variable below on the command line in the form of: -[variable]=[value]
# Example:  -interface=em0   would set the $interface variable to 'em0'.
# Arrays (begin with @) can't be set from command line.
# Command line settings overides variables set below.

# Interface to be evaluated (usually the external network interface).
$interface = "fxp0" unless $interface;

# Path to output html file.
$pfhtmlfile = "/var/www/pantz.org/stats/pfhtmlstats.html" unless $pfhtmlfile;

# Analyze one file in the dir or all files.
# Set to "one" for one file.
# Set to "all" for all pf files (even compressed ones)
# in the /var/log dir that start with  name "pflog".
$oneorallfiles = "all" unless $oneorallfiles;

# Full path to a single pflog file you want analyzed (compressed or uncompressed).
# Used if var above is set to "one". Example: /home/user/pflog.0.gz
$pflogfile = "/var/log/pflog" unless $pflogfile;

# Set to "exclude" to exclude only certain ports in stats output.
# "include" to include only certian ports in stats output.
# "off" to show all ports in stats output.
$exclude_include_ports = "off" unless $exclude_include_ports;

# Port list (array) for the above setting. For ranges use ".." in between the # range.
# Example of ports 1 thru 10 and 22 and 30 would be: 1..10,22,30
@in_ex_clude_ports = (99..100);

# Threshold values. Will not show or count "# of blocks in" totals from a host
# to a port or ip at or below the value set. Ex. Set to 10. No port or ip hit
# count (per host) that totaled 10 or below is counted.
$src_ip_threshold_value = 0 unless $src_ip_threshold_value;
$src_port_threshold_value = 0 unless $src_port_threshold_value;

# Set to "on" to have hostnames instead of ip's in the output. Set to "off" for just ip addresses.
# If this is turned on the script will take longer to run. Depending on the ammout of unique hosts.
$hostname_lookup = "off" unless $hostname_lookup;

# Filter by date? Settings are: Yes or No.
$date_filter = "no" unless $date_filter;

# Type of filter. Specific date range, current date and time run (now) backwards, or end of the current day backwards.
# Settings are: range, now, or curday.
$filter_type = "now" unless $filter_type;

# If now filter then set Units of time. Settings are: sec, min, hour, or day.
$time_unit = "hour" unless $time_unit;

# Amount of time units. How far back from now or end of current day? Whole number.
$time_amount = "12" unless $time_amount;

# If filtering by date range then specify date range. Form: YearMonthDayHourMinSec.
# Pad all single digits with zero. Ex 5 becomes 05.
# December 6th, 2006 20:05:01 would be: 20061206200501.
$lower_date = 20070205000000 unless $lower_date;
$upper_date = 20070206235959 unless $upper_date;

##############################
# End Configuration Settings #
##############################

#####Begin: Assembling date/time code.#####
$curtimeformat = format_epoch(time);

if ($date_filter eq "yes") {
  if ($filter_type eq "range") {
    ($low_year,$low_month,$low_day,$low_hour,$low_min,$low_sec) = ($lower_date =~ /(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})/);
    ($up_year,$up_month,$up_day,$up_hour,$up_min,$up_sec) = ($upper_date =~ /(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})/);
    # timelocal, it expects values with the same range as those which localtime returns.
    # Namely, months start at 0, and years need 1900 subtracted from them.
    $low_year -= 1900;
    $low_month -= 1;
    $up_year -= 1900;
    $up_month -= 1;
    # DMYHMS to epoch seconds corrected for year and month -1900 and -1 respectively.
    $low_epochsec = timelocal($low_sec, $low_min, $low_hour, $low_day, $low_month, $low_year);
    $up_epochsec = timelocal($up_sec,$up_min,$up_hour,$up_day,$up_month,$up_year);
  } elsif ($filter_type eq "now") {
    $up_epochsec = timelocal(localtime(time));
    calc_low_epoch_sec();
  } elsif ($filter_type eq "curday") {
    ($cur_second, $cur_minute,$cur_hour,$cur_Day,$cur_month,$cur_year) = localtime(time);
    $up_epochsec = timelocal(($cur_second, $cur_minute,$cur_hour,$cur_Day,$cur_month,$cur_year) = (59,59,23,$cur_Day,$cur_month,$cur_year));
    calc_low_epoch_sec();
  } else {
    die ("Date filter was requested but no filter type matched. Check your date filter type.");
  }
}

#####End: Assembling date/time code.#####

#####Begin: Get one or all pflog filenames.#####
if ($oneorallfiles eq "all") {
  @pflogfilenames = </var/log/pflog*>;
  if ($date_filter eq "yes") {
    foreach $pflogfilesstat (@pflogfilenames) {
      ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,$atime,$mtime,$ctime,$blksize,$blocks) = stat $pflogfilesstat;
      # Don't open log files with dates not in our date range. Using last modify time on log file as date.
      if ( $mtime >= $low_epochsec ) {
        push(@newpflogfilenames, "$pflogfilesstat");
      }
    }
    @pflogfilenames = @newpflogfilenames;
  }
  @pflogfilenames = sort { $b cmp $a } @pflogfilenames;
} else {
  push(@pflogfilenames, "$pflogfile");
}
#####End: Get one or all pflog filenames.#####

#####Begin: Reading pflog file(s) and inputing data for sorting and minipulation.#####

foreach $pflogfilename (@pflogfilenames) {
open(IN, "gzcat -f $pflogfilename | tcpdump -nettqr - 2>&1 |") or die ("Can't open file. Permissions?");
  $isfirstline = 0;
  while( <IN> ) {

    ($line_date,$line_points,$line_rulenum,$line_action,$line_interface,$line_src_host,$line_src_port,$line_dst_host,$line_dst_port,$line_remainder)
    = ($_ =~ /(\d+)\.(\d+) rule (\d+).*\(match\)\:?? (\w+ \w+) \w+ (\w+)\: (\d+\.\d+\.\d+\.\d+)\.(\d+) > (\d+\.\d+\.\d+\.\d+)\.(\d+)\:(.*)/);

    # If line had no match with date it is probably no good. Go to next line. When a line finally matches increase by one.
    next unless $line_date;
    $isfirstline++;

    # Get date from first and last line of each file. Put in array.
    if ($isfirstline == "1" || eof(IN)) { push(@date_array, "$line_date"); }

    next unless (/block in/ && /$interface/);
    $exclude_include_port_match = "no";

    # If using date filter go to the next log line if date does not fall within set filters.
    if ($date_filter eq "yes") {
        next unless ($line_date >= $low_epochsec && $line_date <= $up_epochsec)
    }
    if ($line_remainder =~ /tcp/) {
      if ($exclude_include_ports ne "off") {
        foreach $exin_ports (@in_ex_clude_ports) {
          if ($exin_ports eq $line_dst_port) {
            $exclude_include_port_match = "yes";
          }
        }
        if (($exclude_include_port_match eq "yes") && ($exclude_include_ports eq "include")) {
          push_tcp();
          next;
        }
        if (($exclude_include_port_match eq "no") && ($exclude_include_ports eq "exclude")) {
          push_tcp();
          next;
        }
      } else {
        push_tcp();
        next;
      }
    }
    if ($line_remainder =~ /udp/) {
      if ($exclude_include_ports ne "off") {
        foreach $exin_ports (@in_ex_clude_ports) {
          if ($exin_ports eq $line_dst_port) {
            $exclude_include_port_match = "yes";
          }
        }
        if (($exclude_include_port_match eq "yes") && ($exclude_include_ports eq "include")) {
          push_udp();
          next;
        }
        if (($exclude_include_port_match eq "no") && ($exclude_include_ports eq "exclude")) {
          push_udp();
          next;
        }
      } else {
        push_udp();
        next;
      }
    }
    #if ($line_remainder =~ /icmp/) {
    #  push_icmp();
    #}
  }
close(IN);
}

#####End: Reading pflog file(s) and inputing data for sorting and minipulation.#####

#####Begin: Output of HTML file.#####

open(PFHTMLSTATS, ">$pfhtmlfile") or die ("Can't create html file");

print PFHTMLSTATS qq{
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head><title>Pantz PFlog Stats</title>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
</head>
<body style="text-align: center;">
<h2>Pantz PFlog Stats</h2>
};

print PFHTMLSTATS "<p>Log file Data from " . format_epoch($date_array[0]) . " to " . format_epoch($date_array[-1]) . "<br></p>\n";

if ($date_filter eq "yes") {
  print PFHTMLSTATS "<p>Showing data from " . format_epoch($low_epochsec) . " to " . format_epoch($up_epochsec) . "<br></p>\n";
}

print PFHTMLSTATS qq {
<p>Script run on: $curtimeformat <br></p>
<TABLE BORDER="0">
<tr><td valign="top">
<TABLE BORDER="1">
};

# Create host totals. Hash of hash. With threshold evaluation. Lookup host name if set. Sort most to least hits.
for $tcud_src_hstct_key ( keys %tcud_src_hstct ) {
  for $tcud_src_hst_dst_portct_key ( keys %{ $tcud_src_hstct{$tcud_src_hstct_key} } ) {
    # Check if our totals meet the threshold set. If so add it to a total for that ip.
    if ($tcud_src_hstct{$tcud_src_hstct_key}{$tcud_src_hst_dst_portct_key} > $src_ip_threshold_value) {
      $porthitcount += $tcud_src_hstct{$tcud_src_hstct_key}{$tcud_src_hst_dst_portct_key};
    }
  }
  # Lookup hostname if var set. Else set to host's ip.
  if ($hostname_lookup eq "on") {
    hostnamelookup ($tcud_src_hstct_key);
  } else {
    $key_hostname = "$tcud_src_hstct_key";
  }
  # Make hash of arrays. IP, hostname, and hit count. Only if we met thresholds.
  if ($porthitcount != 0) {
    $tcud_src_hst_tot_ct{$tcud_src_hstct_key} = [$key_hostname,$porthitcount];
  }
  $porthitcount = 0;
}

# Print heading line with hostname if var is set. If not just print the ip heading line.
if ($hostname_lookup eq "on") {
  print PFHTMLSTATS "<tr><td><b>Hostname</b></td><td><b>Source IP</b></td>
                     <td><b># of blocks in</b></td></tr>\n";
} else {
  print PFHTMLSTATS "<tr><td><b>Source IP</b>
                     </td><td><b># of blocks in</b></td></tr>\n";
}

# Print hostname, ip, and count if hostname var is set. Otherwise just print ip and count lines.
for $print_tot_src_ip_key1 ( sort {  $tcud_src_hst_tot_ct{$b}[1] <=> $tcud_src_hst_tot_ct{$a}[1] }  keys %tcud_src_hst_tot_ct ) {
  if ($hostname_lookup eq "on") {
    print PFHTMLSTATS "<tr><td>$tcud_src_hst_tot_ct{$print_tot_src_ip_key1}[0]</td>
                       <td><a href=\"#IP:$print_tot_src_ip_key1\">$print_tot_src_ip_key1</a>
                       </td><td>$tcud_src_hst_tot_ct{$print_tot_src_ip_key1}[1]</td></tr>\n";
  } else {
    print PFHTMLSTATS "<tr><td><a href=\"#IP:$print_tot_src_ip_key1\">$print_tot_src_ip_key1</a></td>
                       <td>$tcud_src_hst_tot_ct{$print_tot_src_ip_key1}[1]</td></tr>\n";
  }
}

print PFHTMLSTATS qq{
</table>
</td> <td></td> <td valign="top">
<TABLE BORDER="1">
<tr><td><b>Destination Port</b></td><td><b># of blocks in</b></td></tr>
};

# Create destination port totals. Hash of hash. With threshold evaluation. Sort most to least hits.
for $tcud_dst_portct_key1 ( keys %tcud_dst_portct ) {
  # Check if our totals meet the threshold set. If so add it to a total for that port.
  for $tcud_dst_portct_key2 ( keys %{ $tcud_dst_portct{$tcud_dst_portct_key1} } ) {
    if ($tcud_dst_portct{$tcud_dst_portct_key1}{$tcud_dst_portct_key2} > $src_port_threshold_value) {
      $iphitperportcount += $tcud_dst_portct{$tcud_dst_portct_key1}{$tcud_dst_portct_key2};
    }
  }
  # Create hash of ip and port count. Only if we met thresholds.
  if ($iphitperportcount != 0) {
    $tcud_dst_port_tot_ct{$tcud_dst_portct_key1} = $iphitperportcount;
  }
  $iphitperportcount = 0;
}

# Print the hash from most to least port hits.
foreach $print_tot_dst_port_key (sort { $tcud_dst_port_tot_ct {$b} <=> $tcud_dst_port_tot_ct {$a} } keys %tcud_dst_port_tot_ct) {
  print PFHTMLSTATS "<tr><td><a href=\"#PORT:$print_tot_dst_port_key\">$print_tot_dst_port_key</a></td>
                     <td>$tcud_dst_port_tot_ct{$print_tot_dst_port_key}</td></tr>\n";
}

print PFHTMLSTATS qq {\n</table>\n</td></tr>\n</table>\n\n<p><br><br></p><hr><p><br><br></p>};

$hostportcount = 4;
$hostporttablerowcount = 4;
$was_anything_printed = 0;

print PFHTMLSTATS "\n\n<TABLE BORDER=\"0\">\n\n";

# Start looping thru hash of hash. Print tables if thresholds were met.
for $tcud_src_hstct_key ( keys %tcud_src_hstct ) {
  # Start a new row if our counter is reset.
  if ($hostportcount == $hostporttablerowcount) {
    print PFHTMLSTATS "<tr valign=\"top\">\n\n";
    $hostportcount = 0;
  }
  for $tcud_src_hst_dst_portct_key (sort { $tcud_src_hstct{$tcud_src_hstct_key}{$b} <=> $tcud_src_hstct{$tcud_src_hstct_key}{$a} } keys %{ 
$tcud_src_hstct{$tcud_src_hstct_key} } ) {
    # Check if threshold is met for total host ip hits to a port.
    if ($tcud_src_hstct{$tcud_src_hstct_key}{$tcud_src_hst_dst_portct_key} > $src_ip_threshold_value) {
      # Trip the counter if threshold was met.
      $was_anything_printed++;
      # If this is our first time through and we met the threshold print the table heading.
      if ($was_anything_printed == 1) {
        print PFHTMLSTATS "<td><TABLE BORDER=\"1\">\n";
        print PFHTMLSTATS "<tr><td colspan=\"2\"><a name=\"IP:$tcud_src_hstct_key\"></a>
                           <a href=\"http://ws.arin.net/cgi-bin/whois.pl\?queryinput=$tcud_src_hstct_key\">
                           <b>$tcud_src_hstct_key</b></a></td></tr><tr><td><b>Destination Port</b>
                           </td><td> <b># of blocks in</b></td></tr>\n";
      }
      # Print a table row.
      print PFHTMLSTATS "<tr><td><a href=\"http://isc.sans.org/port_details.php\?port=$tcud_src_hst_dst_portct_key\">
                         $tcud_src_hst_dst_portct_key</a></td><td>
                         $tcud_src_hstct{$tcud_src_hstct_key}{$tcud_src_hst_dst_portct_key}</td>
                         </tr>\n";
    }
  }

  # If something was printed at least once increase column counter. Close our table.
  if ($was_anything_printed > 0) {
    $hostportcount++;
    print PFHTMLSTATS "</table></td><td> </td>\n\n";
  }
  # If host port counter has it its max (set above) close the whole row.
  if ($hostportcount == $hostporttablerowcount) {
    print PFHTMLSTATS "</tr><tr><td> </td></tr>\n\n";
  }
  $was_anything_printed = 0;
}

# If hash ended before row finished then close it.
if ($hostportcount < $hostporttablerowcount) {
  print PFHTMLSTATS "<td> </td></tr>\n";
}


print PFHTMLSTATS "\n</table>\n\n<p><br><br></p><hr><p><br><br></p>\n";

$porthostcount = 4;
$porthosttablerowcount = 4;
$was_anything_printed = 0;

print PFHTMLSTATS "\n\n<TABLE BORDER=\"0\">\n";

for $tcud_dst_portct_key3 ( keys %tcud_dst_portct ) {
  # Start a new row if our counter is reset
  if ($porthostcount == $porthosttablerowcount) {
    print PFHTMLSTATS "<tr valign=\"top\">\n\n";
    $porthostcount = 0;
  }
  for $tcud_dst_portct_key4 (sort { $tcud_dst_portct{$tcud_dst_portct_key3}{$b} <=> $tcud_dst_portct{$tcud_dst_portct_key3}{$a} } keys 
%{$tcud_dst_portct{$tcud_dst_portct_key3} } ) {
    # Check if threshold is met for total host port hits to a ip.
    if ($tcud_dst_portct{$tcud_dst_portct_key3}{$tcud_dst_portct_key4} > $src_port_threshold_value) {
      # Trip the counter if threshold was met.
      $was_anything_printed++;
      # If this is our first time through and we met the threshold print the table heading.
      if ($was_anything_printed == 1) {
        print PFHTMLSTATS "<td><TABLE BORDER=\"1\">\n";
        print PFHTMLSTATS "<tr><td colspan=\"2\"><b><a name=\"PORT:$tcud_dst_portct_key3\"></a>
                           <a href=\"http://isc.sans.org/port_details.php\?port=$tcud_dst_portct_key3\">
                           $tcud_dst_portct_key3</a></b></td></tr><tr><td><b>Source IP</b></td>
                           <td> <b># of blocks in</b></td></tr>\n";
      }
      # Print a table row.
      print PFHTMLSTATS "<tr><td><a href=\"http://ws.arin.net/cgi-bin/whois.pl\?queryinput=$tcud_dst_portct_key4\">
                         $tcud_dst_portct_key4</a></td><td>
                         $tcud_dst_portct{$tcud_dst_portct_key3}{$tcud_dst_portct_key4}</td></tr>\n";
    }
  }
  # If something was printed increase column counter
  if ($was_anything_printed > 0) {
    $porthostcount++;
    print PFHTMLSTATS "</table></td><td> </td>\n\n";
  }
  # If host port counter has it its max (set above) close the whole row.
  if ($porthostcount == $porthosttablerowcount) {
    print PFHTMLSTATS "</tr><tr><td> </td></tr>\n\n";
  }
  $was_anything_printed = 0;
}

# If hash ended before row finished then close it.
if ($porthostcount < $porthosttablerowcount) {
  print PFHTMLSTATS "<td> </td></tr>\n\n";
}

print PFHTMLSTATS "\n</table>\n</body></html>";

close(PFHTMLSTATS);

#####End: Output of HTML file.#####

#####Start: Subroutines.#####
sub push_tcp {
  # Hash of a hash. Value is increasing counter.
  $tcud_src_hstct{$line_src_host}{$line_dst_port}++;
  $tcud_dst_portct{$line_dst_port}{$line_src_host}++;
}
sub push_udp {
  # Hash of a hash. Value is increasing counter.
  $tcud_src_hstct{$line_src_host}{$line_dst_port}++;
  $tcud_dst_portct{$line_dst_port}{$line_src_host}++;
}
sub push_icmp {
  #push(@icmp_src_hst_array, "$line_src_host");
  #push(@icmp_dst_hst_array, "$line_dst_host");
}
sub hostnamelookup {
  # Use the program "host" to do reverse lookups on ip's.
  $key_hostname = $_[0];
  $key_hostname = `host $key_hostname`;
  if ($key_hostname =~ /pointer/) {
    ($key_hostname) = ($key_hostname =~ /pointer (.*)/);
    $key_hostname = substr($key_hostname,0, -1);
  } else {
    $key_hostname = "No PTR Record";
  }
}
sub format_epoch {
  my ($fmt_second, $fmt_minute, $fmt_hour, $fmt_Day, $fmt_month, $fmt_year) = localtime($_[0]);
  $fmt_month += 1;
  $fmt_year += 1900;
  if($fmt_month < 10) { $fmt_month = "0" . $fmt_month; }
  if ($fmt_hour < 10) { $fmt_hour = "0" . $fmt_hour; }
  if ($fmt_minute < 10) { $fmt_minute = "0" . $fmt_minute; }
  if ($fmt_second < 10) { $fmt_second = "0" . $fmt_second; }
  if($fmt_Day < 10) { $fmt_Day = "0" . $fmt_Day; }
  return "$fmt_month-$fmt_Day-$fmt_year $fmt_hour:$fmt_minute:$fmt_second";
}
sub calc_low_epoch_sec {
  if ($time_unit eq "min") { $time_amount *= 60; }
  if ($time_unit eq "hour") { $time_amount *= 3600; }
  if ($time_unit eq "day") { $time_amount *= 86400; }
  return $low_epochsec = $up_epochsec - $time_amount;
}
#####End: Subroutines.#####

Changes

07-01-04 - First change log entry. Added date sort so filenames are read from
           oldest to newest. Done via filenames numbers in order.
         - Fixed date range date so it evaluates all lines not just block in lines.
         - Added interface filter.

11-23-04 - Added port breakdown by IP address's.
         - Added quick navigation to IP address port breakdown.

11-27-04 - Added IP breakdown by port numbers.
         - Added quick navigation to ports IP address breakdown.

12-06-04 - Added IP and Port thresholds for output.

12-01-05 - Fixed pattern matching for PF logs in 3.8.

02-01-06 - Rewrote code to make it simpler and slightly faster.
           Fixed thresholds so all counts reflect correct numbers.
10-25-06 - changed tcpdump switch to dump dates as unix epoch time.
         - changed regex to use new date format.
         - put in "next" lines to skip evaluation of rules we don't want
           or have matched. This increased speed and resource usage.

10-26-06 - Added ability to set config settings from commandline. perl -s.
         - Added basic module to convert DMYHMS to epoch seconds.
         - Added ability to only process lines in a specific date range.

10-06-07 - Fixed html output so it is HTML 4.01 Strict.

11-12-07 - Fixed problem where if first line of a log file was not a matched
           line the date would not be set correctly for the "Log data from" line.

10-31-08 - Fixed include exclude ports that was broken because I forgot to rename
           some old vars to the new name.

Possible To Do's

1. Number of blocks in by days of the week or just date. Do a ascii graphic with symbol.
2. Do something with ICMP.
3. Keep cache of stats from last run to help with large log files.
4. Check log file extention. If it is bzip2 then use bzcat instead of zcat.

Reddit!

Related stories


RSS Feed RSS feed logo

About


3com

3ware

alsa

alsactl

alsamixer

amd

android

apache

areca

arm

ati

auditd

awk

badblocks

bash

bind

bios

bonnie

cable

carp

cat5

cdrom

cellphone

centos

chart

chrome

chromebook

cifs

cisco

cloudera

comcast

commands

comodo

compiz-fusion

corsair

cpufreq

cpufrequtils

cpuspeed

cron

crontab

crossover

cu

cups

cvs

database

dbus

dd

dd_rescue

ddclient

debian

decimal

dhclient

dhcp

diagnostic

diskexplorer

disks

dkim

dns

dos

dovecot

drac

dsniff

dvdauthor

e-mail

echo

editor

emerald

encryption

ethernet

expect

ext3

ext4

fat32

fedora

fetchmail

fiber

filesystems

firefox

firewall

flac

flexlm

floppy

flowtools

fonts

format

freebsd

ftp

gdm

gmail

gnome

google

gpg

greasemonkey

greylisting

growisofs

grub

hacking

hadoop

harddrive

hba

hex

hfsc

html

html5

http

https

hulu

idl

ie

ilo

intel

ios

iperf

ipmi

iptables

ipv6

irix

javascript

kde

kernel

kickstart

kmail

kprinter

krecord

kubuntu

kvm

lame

ldap

linux

logfile

lp

lpq

lpr

maradns

matlab

memory

mencoder

mhdd

mkinitrd

mkisofs

moinmoin

motherboard

mouse

movemail

mplayer

multitail

mutt

myodbc

mysql

mythtv

nagios

nameserver

netflix

netflow

nginx

nic

ntfs

ntp

nvidia

odbc

openbsd

openntpd

openoffice

openssh

openssl

openvpn

opteron

parted

partimage

patch

perl

pf

pfflowd

pfsync

photorec

php

pop3

pop3s

ports

postfix

power

procmail

proftpd

proxy

pulseaudio

putty

pxe

python

qemu

r-studio

raid

recovery

redhat

router

rpc

rsync

ruby

saltstack

samba

schedule

screen

scsi

seagate

seatools

sed

sendmail

sgi

shell

siw

smtp

snort

solaris

soundcard

sox

spam

spamd

spf

spotify

sql

sqlite

squid

srs

ssh

ssh.com

ssl

su

subnet

subversion

sudo

sun

supermicro

switches

symbols

syslinux

syslog

systemd

systemrescuecd

t1

tcpip

tcpwrappers

telnet

terminal

testdisk

tftp

thttpd

thunderbird

timezone

ting

tls

tools

tr

trac

tuning

tunnel

ubuntu

unbound

vi

vpn

wget

wiki

windows

windowsxp

wireless

wpa_supplicant

x

xauth

xfree86

xfs

xinearama

xmms

youtube

zdump

zeromq

zic

zlib