PulledPork 0.4.1, I see your sensitive data!

In conjunction with the Snort 2.8.6 release and the new Snort Rules tarball format, pulledpork 0.4.1 is now released!  As noted below, there are a number of changes and fixes.  When updating your pulledpork, please be sure to use the latest master configuration file that is included in the release tarball and read through it thoroughly.

Notable changes include the tarball filename change, preprocessor rules and sensitive data rules.  Note that pulledpork 0.4.0 will still work with 2.8.6 but will not properly make use of the new rules that I just listed and that you will need to change the rules tarball name for VRT releases.  Please also note that if you use pulledpork 0.4.1 and are still using Snort 2.8.5.3 that you need to make some changes in the "ignore" variable section of the pulledpork.conf file.

New Features/changes:

• Flowbit tracking! - This means that all flowbits are not enabled when a specific base ruleset is specified (security etc...) but rather all flowbits are now tracked, allowing for only those that are required to be enabled.

• Adjusted pulledpork.conf to account for new snort rules tarball naming and packing scheme, post Snort 2.8.6 release.

• Added option to specify all rule modification files in the master pulledpork.conf file - feature request 19.

• Added capability to specify base ruleset (see README.RULESETS) in master pulledpork.conf file.

• Handle preprocessor and sensitive-information rulesets

Bug Fixes:

• 18 - non-rule lines containing the string sid:xxxx were being populated into the rule data structure, added an extra check to ensure that this does not occur

• Cleaned up href pointers, syntactical purposes only...

• Modified master config to allow for better readability on smaller console based systems

• Error output was not always returning full error, fixed this

Thanks to the community for continued support and feedback!

Cheers,
JJC

Snort 2.8.6 Release is OUT, WGET it nao! kthx!

That's right, the new Snort 2.8.6 Release is out, get it at snort.org!

Release Notes:

2010-04-22 - Snort 2.8.6

[*] New Additions
   * HTTP Inspect now splits requests into 5 components -
     Method, URI, Header (non-cookie), Cookies, Body.
     Content and PCRE rule options can now search one or more of these buffers.

     HTTP server-specific configurations to normalize the HTTP header and/or
     cookies have been added.

     Support gzip decompression across multiple packets.

   * Added a Sensitive Data preprocessor, which performs detection of
     Personally Identifiable Information (PII).  A new rule option is available
     to define new PII.  See README.sensitive_data and the Snort Manual
     for configuration details.

   * Added a new pattern matcher and related configurations.  The new pattern
     matcher is optimized to use less memory and perform at AC speed.

[*] Improvements
   * Addressed problem to resolve output obfuscation affecting packets
     when Snort is inline.

   * Preprocessors with memcap settings can now be configured in a "disabled"
     state.  This allows you to configure that memcap globally, but only enable
     the preprocessor in targeted configurations.

Pulling Pork with the Drunken Leprechaun (PP 0.4.0)

PulledPork 0.4.0 (Drunken Leprechaun) is officially released and can be downloaded here -> pulledpork-0.4.0.tar.gz

This version constitutes a major rewrite of the rule reading, modification and writing system to improve speed, future module addition, supportability, and of course reliability.  Incidentally, the codename was partially chosen due to a majority of the rewrites being finished on St. Patrick's Day.

One specific change to note is the use of Archive::Tar, this makes PulledPork more system independent.  As such though, you will need to install Archive::Tar if you do not have it currently installed, you can do so using CPAN, please see the PulledPork FAQ for further information.

New Features/changes:

• Enablesid (-e enablesid.conf)
• Moved all .conf files under etc/
• Ability to define sid ranges in any of the sid modification .conf files
• Ability to specify references in any of the sid modification .conf files
• Ability to ignore entire rule categories (i.e. not include them)
• Specify locally stored rules files that need their meta data included in sid-msg.map
• All rulestate modifications, comparisons etc.. are now handled in-memory
• Rewrite of sid-msg.map generation code to allow for all proper character reading and addition to sid-msg.map
• No longer reliant on tar binary, now using Archive::Tar
• Ability to specify your arch for so_rules
• Added significant amounts of debug output when an error is detected
• Rules are now written to only two distinct files
• Cleaned up changelog and added more information to it

Bug Fixes:

• Properly account for whitespace in non-standard rulesets such as ET
• Cleaned up and improved the changelog to display new / deleted sids and rule totals
• Certian conditions caused the md5 check to fail even when valid - This was primarily an ET issue, but did manifest on VRT rulesets also
• Many small fixes that were not tracked well :-P
• Do not overwrite local.rules, but still include in sid-msg.map generation

A little more detail about some of the new key features, note that there are more.. please read through all of the conf files and README thoroughly:

Initially you may not notice a significant performance increase, unless you already have a large count of disable or drop sids specified in your configuration because this is where the major improvement was made.  I can't help how slow your internet connection is and thusly how long it takes you to download the tarball itself ;-).

One key change that you will note is that all rules are now written to only two distinct files.. one for GID:1 rules and one for GID:3 rules.  The logic behind this is simple; if a new rule category comes out (a new or different .rules file within the VRT or ET tarball) then it will automatically be included in your snort.conf as you will have only one or both of the aforementioned GID:1 or GID:3 rules files included .  Please note these changes in the rule_path and sostub_path within the pulledpork.conf file.

Somewhat hand-in-hand with the previous change is the addition of the ignore variable within the pulledpork.conf file.. this specifies what categories/rule files that you want excluded from your configuration.  By default these are deleted, experimental, and local.

If you have a local.rules file or other already locally existing rules files, you can specify them  with the local_rules variable, doing so will tell pulledpork to read these rules and populate their meta data into the sid-msg.map.

Enablesid - This was a widely requested feature, the capability to enable specific sids etc.

Sid modification ranges - This stemmed from one of the enablesid requests (an option to enable ALL sids) and my interpretation of what I thought would be more useful.  This feature gives you the capability to specify a range of sids in any of the sid state modification configuration files in the format of GID:SID-GID:SID.  Please see the individual configuration files for additional information.

Reference modification - This was another community request and allows the user to specify any reference within a rule and perform an operation on that rule (disable, enable, drop...).  The formatting is simple, the user specifies, in one of the sid state modification configuration files, the reference information such as cve|XXX-XXXX,MSXX-XXXX.  Please see the individual configuration files for additional information.

Excerpt from an example configuration file:

# example of enabling ranges and references!

# you should be specific when enabling a range of rules.. don't just put an extremely high number

# this would be at the cost of speed and memory usage.

1:1101,1:800,1:1200-1:2000,cve|1999-0499,bugtraq|22026,MS09-004

Excerpt from new changelog format:

-=Begin Changes Logged for Tue Mar 23 19:15:02 2010 GMT=-

New Rules

        1:16492

        1:16493

        1:16494

        1:16495

        1:16496

        1:16497

        1:16498

        1:16499

        1:16500

Set Policy: security

Rule Totals

        New:-------9

        Deleted:---0

        Enabled:---5378

        Dropped:---0

        Disabled:--3606

        Total:-----8984

-=End Changes Logged for Tue Mar 23 19:15:02 2010 GMT=-

You will want to take the paths out of your old pulledpork.conf and use the new pulledpork.conf, since there are so many new features and variables pulledpork will not function without the updated pulledpork.conf file.  All of the other sid modification conf files remain unchanged, however.

Please be sure that you read the README and all configuration files thoroughly as there are many changes.

JJC

Hogging the Snort Host Attribute Table

Hogger is a new Snort supportive tool written in Perl, by Parker Crook, that allows you to create a Host Attribute Table from an nmap scan. But first, a little primer; A feature within Snort that has received some traction lately is that of the --enable-targetbased configuration option. This allows you to specify a Host Attribute Table that contains critical information about what your network host topology is (i.e. OS, services etc..). Using this information, snort can then properly reassemble fragments, track streams and a number of other things. All of these items are covered in Joel Esler's recent CSO article that can be found at This URL. This is an excellent article that covers what Host Attribute Tables are and how to use them, so please read the article for a better understanding!

Now that you know all about the Host Attribute Table, let's jump into the purpose and use of hogger. As mentioned previously, hogger was written by Parker Crook to create a Host Attribute Table using the resulting output of an nmap scan. Without further adieu, let's walk through the usage of hogger!

Requirements:

• Perl
• XML::Writer (perl module)
• Nmap
• Hogger

Steps:

1. Install XML::Writer
2. Get hogger
3. Install Nmap
4. Run Nmap with correct options
5. Run hogger against Nmap output file
6. Start your snorting!

1: Installing XML::Writer

$perl -MCPAN -e shell

cpan[1]> install XML::Writer

2: Get Hogger

$wget http://hogger.googlecode.com/files/hogger.tar.gz

$tar xvfz hogger.tar.gz

3: Install Nmap

Use whatever tool that your distribution / OS uses to install Nmap, or get the source from nmap.org and build it yourself!

4: Run Nmap

$mkdir ~/hogger/nmap

$cd ~/hogger/nmap

$nmap -sV -T4 -oN scan.nmap 192.168.1.0/24

Starting Nmap 5.21 ( http://nmap.org ) at 2010-02-25 18:46 UTC

..output suppressed...

5: Run hogger (against scan.nmap)

$cd ~/hogger

$./hogger.pl -c nmap/hostmap.csv -n nmap/scan.nmap -x nmap/host_attrib_table.xml

6: Start your snorting - At this point you can take the newly created host_attrib_table.xml file and place the path to it in your snort.conf, assuming your built snort with the correct option:

attribute_table filename /path/to/host_attrib_table.xml

Now that we have all of this running, let's examine some of the options that are currently available in hogger and dissect our hogger run: "./hogger.pl -c nmap/hostmap.csv -n nmap/scan.nmap -x nmap/host_attrib_table.xml".

Hogger help output:

Usage: ./hogger.pl [-r? -help] -n -c -x

Options:

-c Where the human-readable/modifiable csv file containing host information lives.

-n Where the nmap file containing host information lives.

-r Process the csv file and output to xml for snort, but do not read an nmap file.

-x Where you want to create the host_attribute table.xml (Overwrites existing files)

-help/? Print this information

Starting with the -c flag, this is a file that will be created by hogger if it does not exist, and is simply a csv file that you can modify (for those hosts that nmap either misses or is not as accurate as you would like). A few sample entries in the file (hostmap.csv) that we created in the above test run:

192.168.1.1, Linux, 23|tcp|telnet 53|tcp|domain 443|tcp|ssl/http

192.168.1.2, Linux, 23|tcp|telnet 53|tcp|domain 443|tcp|ssl/http

192.168.1.7, FreeBSD, 22|tcp|ssh 53|tcp|domain 80|tcp|http 3000|tcp|http 3128|tcp|http-proxy 3306|tcp|mysql 5000|tcp|http-proxy 8443|tcp|http

Next we see the -n flag, this is the flag that specifies where the nmap output file (that we previously created using the nmap -oN scan.nmap option). This is the file that hogger reads to create entries in the -c .

The -r flag is fairly straightforward and specifies that you ONLY want to read the csv file specified with the -c flag value.

The final flag that we will discuss is the -x flag, this is a required flag and tells hogger where you want the resulting output (the Host Attribute Table) to be placed. Examples from the output, matching those noted in the -c flag information above:

<SNORT_ATTRIBUTES>

<ATTRIBUTE_TABLE>

<HOST IP="192.168.1.1">

<OPERATING_SYSTEM>

<NAME ATTRIBUTE_VALUE="Linux" CONFIDENCE="90"></NAME>

<FRAG_POLICY>Linux</FRAG_POLICY>

<STREAM_POLICY>linux</STREAM_POLICY>

</OPERATING_SYSTEM>

<SERVICES>

<SERVICE>

<PORT ATTRIBUTE_VALUE=" 23" CONFIDENCE="100"></PORT>

<IPPROTO ATTRIBUTE_VALUE="tcp" CONFIDENCE="100"></IPPROTO>

<PROTOCOL ATTRIBUTE_VALUE="telnet 53" CONFIDENCE="95"></PROTOCOL>

</SERVICE>

</SERVICES>

</HOST>

<HOST IP="192.168.1.2">

<OPERATING_SYSTEM>

<NAME ATTRIBUTE_VALUE="Linux" CONFIDENCE="90"></NAME>

<FRAG_POLICY>Linux</FRAG_POLICY>

<STREAM_POLICY>linux</STREAM_POLICY>

</OPERATING_SYSTEM>

<SERVICES>

<SERVICE>

<PORT ATTRIBUTE_VALUE=" 23" CONFIDENCE="100"></PORT>

<IPPROTO ATTRIBUTE_VALUE="tcp" CONFIDENCE="100"></IPPROTO>

<PROTOCOL ATTRIBUTE_VALUE="telnet 53" CONFIDENCE="95"></PROTOCOL>

</SERVICE>

</SERVICES>

</HOST>

<HOST IP="192.168.1.7">

<OPERATING_SYSTEM>

<NAME ATTRIBUTE_VALUE="FreeBSD" CONFIDENCE="90"></NAME>

<FRAG_POLICY>BSD</FRAG_POLICY>

<STREAM_POLICY>bsd</STREAM_POLICY>

</OPERATING_SYSTEM>

<SERVICES>

<SERVICE>

<PORT ATTRIBUTE_VALUE=" 22" CONFIDENCE="100"></PORT>

<IPPROTO ATTRIBUTE_VALUE="tcp" CONFIDENCE="100"></IPPROTO>

<PROTOCOL ATTRIBUTE_VALUE="ssh 53" CONFIDENCE="95"></PROTOCOL>

</SERVICE>

</SERVICES>

</HOST>

Having said all of this, I am not going to go into detail about the flags used during the Nmap scan, suffice it to say that those are the suggested flags and that the -oN is required to produce the output file for hogger to read.

Overall I think that the concept behind hogger is excellent and that it should provide useful aide to all you snort heads out there! This tool gets a thumbs up from me and should be one that you put into your snort bag of tricks and is also one that I am planning on contributing to.

Cheers,

JJC

Writing Snort Rules Correctly (via Joel Esler)

Joel Esler recently published an article entitled "Writing Snort Rules Correctly". I certainly suggest having a read through of this ,as it discusses a number of the finer points (including PCRE) when writing a snort rule using a previously published example rule. Joel dissects the rule, pointing out the good and bad while making note of better methods.

Just a short post, but I thought it worth posting to bring more attention to the aforementioned article by Joel Esler.

JJC