pfSense 2 log visualization with glTail on OSX Lion

pfSense has changed considerably since the first generation of the fantastic FreeBSD based firewall platform and with that, 3rd party utilities such as glTail require updated configs and log parsers to restore the ability to visualize pfSense 2 firewall logs since the logging system in pfSense 2 has been revised considerably.

glTail rendering of logs generated by pfSense 2 on OSX

Necessary changes:
A commit to the glTail repo by pfSense developer JimP for an updated logging parser for pfSense 2 (https://github.com/Fudge/gltail/pull/14) included a comment about an additional and mandatory updated log output command used by pfSense 2 to be added to config.yaml called /usr/local/bin/filterpaser.php

In trying out the commit for the updated pfsense2.rb logging processor, glTail output was broken since the updated pfSense 2 logging parser had been revised considerably with a new block naming convention and as well as dropping blocks that were previously used in the pfSense 1 configuration. This resulted a config.yaml revision to use the new block naming convention and depreciated block names.

Additionally updates were needed to the hostwithport value processing for IPv4 traffic in the pfsense2.rb logging processor to parse out the host and port combinations from the old xxx.xxx.xxx.xxx.zzz format from pfSense 1 to the xxx.xxx.xxx.xxx:zzz format now used by pfSense 2. The committed parser did not yet include this necessary change.

Installation of glTail on OSX Lion:
Provided here are some simple steps to take to get glTail installed on OSX. This as been tested on Lion only.

wget https://github.com/Fudge/gltail/zipball/master
unzip Fudge-gltail-9d2b843.zip 
cd Fudge-gltail-9d2b843
sudo gem install net-ssh ruby-opengl file-tail net-ssh-gateway chipmunk -r
mv config.yaml config.yaml.old

Configuration of glTail for pfSense 2:
The following updated config and logging parser include all the fixes necessary to run glTail against pfSense 2 generated log output from /usr/local/bin/filterpaser.php.

Download and save the following config.yaml to the root of your glTail install and update with the host value with the IP of your pfSense 2 server.

servers:
    # Example 1: Connect directly to a pfSense router
    pfsense1:
        host: 192.168.1.1
        user: root
        password: 
        command: /usr/sbin/clog -f /var/log/filter.log | /usr/local/bin/filterparser.php
        files: /var/log/filter.log
        parser: pfsense2
        color: white
 
    # Example 2: Logs forwarded to a syslog host 
    # pfsense2:
    #     host: 192.168.1.2
    #     user: logview
    #     password: logviewpassword
    #     command: /usr/bin/tail -f -n0
    #     # Adjust this based on where you have syslog direct the output
    #     files: /var/log/hosts/pfsense.log
    #     parser: pfsense
    #     color: 0.2, 1.0, 0.2, 1.0

config:
    dimensions: 1024x700
    min_blob_size: 0.004
    max_blob_size: 0.02
    highlight_color: orange
    bounce: true
    left_column:
        size: 45
        alignment: -0.99
        blocks:
            action:
                order: 1
                size: 5
                color: red
            ipprotocol:
                order: 2
                size: 5
                color: magenta
            int:
                order: 3
                size: 5
            sourcedestination:
                order: 4
                size: 10
                color: pink

    right_column:
        size: 45
        alignment: 0.99
        blocks:
            destinationhost:
                order: 1
                size: 15
            destinationport:
                order: 2
                size: 15
                color: cyan
            sourcehost:
                order: 3
                size: 15
            sourceport:
                order: 4
                size: 15
                color: blue
resolver:
    reverse_ip_lookups: true
    reverse_timeout: 0.5

Download and save pfsense2.rb to lib/gl_tail/parsers

# gl_tail.rb - OpenGL visualization of your server traffic
# Copyright 2007 Erlend Simonsen ([email protected])
#
# Licensed under the GNU General Public License v2 (see LICENSE)
#
 
# Parser for pfSense PF Logs, specifically those from pfSense (2.0)
# Jim Pingle ([email protected])
 
# Available Blocks
#action: block|pass
#rule: Rule number matched
#ipprotocol: carp|icmp|tcp|udp|ah|igmp|esp|gre you get the idea..
#int: This will be the actual interface (fxp0, vlan2, em1, etc) as the 'friendly' name is not put in the logs.
#sourcehost: source host/IP
#sourceport: source port
#destinationhost: destination host/IP
#destinationport: destination port
#sourcedestination:  source host and port > destination host and port
 
# Use with command:  /usr/sbin/clog -f /var/log/filter.log | /usr/local/bin/filterparser.php
 
class PFSense2Parser < Parser require 'date' def getipandport(hostwithport) # Test for IPv6 if (hostwithport.count(':') > 2)
      if (hostwithport.count('.') == 1)
        thisport = hostwithport.split('.')[1].to_s
        thishost = hostwithport.split('.')[0].to_s
      else
        thishost = hostwithport
        thisport = "none"
      end
    else
      # IPv4
      if (hostwithport.count('.') == 3 && hostwithport.count(':') == 1)
        thisport = hostwithport.split(':')[-1,1].to_s
        thishost = hostwithport.split(':')[0,1].to_s
      else
        thishost = hostwithport
        thisport = "none"
      end
    end
 
    if thisport.include?(':')
      thisport = thisport.split(':')[0]
    end
    if thisport.include?(' ')
      thisport = thisport.split(' ')[0]
    end
 
    return [thishost, thisport]
  end
 
  def getport(thisport)
    if thisport == "none"
      return ""
    else
      return ":" + thisport.to_s
    end
  end
 
  def parse( line )
    lmonth, lday, ltod, action, int, ipprotocol, src, dst = line.split(' ')
    ltime = [ lmonth, lday, ltod ].join(' ')
 
    # Assume the server is in the same time zone as the viewing client.
    timewithoffset = ltime.to_s + DateTime.now().zone()
 
    # Alternately, just set it this way to assume UTC/GMT
    #timewithoffset = ltime.to_s
 
    hours,minutes,seconds,frac = Date.day_fraction_to_time(DateTime.now() - DateTime.parse(timewithoffset))
 
    # When connecting directly, there is no way to only view the end of the log. The clog program to view
    # circular logs will dump the entire log to the parser, then will tail it showing new messages.
    # Therefore, we can run a simple time check and only view entries from the last 5 minutes, or the
    # "future". On some systems, I have seen the clock show negative (-1hr 59mins) instead of 0, so we
    # can allow "future" messages just to be safe.
    if ((hours == 0) and (minutes < 5)) or (hours < 0) # Debug # printf("Adding entry from %s hours, %s minutes ago\n", hours.to_s, minutes.to_s) sourcehost, sourceport = getipandport(src) destinationhost, destinationport = getipandport(dst) add_activity(:block => 'action',  :name => action.to_s)
      add_activity(:block => 'int',     :name => int.to_s)
      add_activity(:block => 'ipprotocol',   :name => ipprotocol.to_s)
      add_activity(:block => 'sourcehost', :name => sourcehost.to_s)
      if sourceport != "none"
        add_activity(:block => 'sourceport', :name => sourceport.to_s)
      end
      add_activity(:block => 'destinationhost', :name => destinationhost.to_s, :type => 5)
      if destinationport != "none"
        add_activity(:block => 'destinationport', :name => destinationport.to_s, :type => 5)
      end
      add_activity(:block => 'sourcedestination',  :name => sourcehost.to_s + getport(sourceport) + " > " + destinationhost.to_s + getport(destinationport) + " (" + ipprotocol.to_s + ")")
    else
      # Debug
      # printf("Not adding entry from %s hours, %s minutes ago\n", hours.to_s, minutes.to_s)
    end
  end
end

Executing:
Executing glTail is done from the root of the glTail installation folder location.

./bin/gl_tail config.yaml

Notes:
This has not been tested against IPv6 traffic yet.