Splunk Enterprise Security and TA-mcafee

When configuring Splunk and Splunk ES to work with McAfee ePO make use of the vendor-supplied TA, aptly named TA-mcafee.  The basic instructions for getting it working found in $SPLUNK_HOME/etc/apps/TA-mcafee/README are a great starting point, but I stumbled into a few issues.  Hopefully this guide will help others avoid spending the amount of time I spent working my way through them.

Important caveats

• Test system ran Splunk 6.1.1 on RHEL 6.5
• I used the Pymssql option in the README
• I have a valid RHEL license (CentOS should work the same, but there are no guarantees)
• This was for a test system, as a result there may be more packages installed than strictly necessary
• I could not get the connection to work using domain credentials for authentication (e.g. MYDOMAIN\username).  I think it has something to do with the backslash and freetds (and yes, I did try escaping the backslash).  I used SQL credentials (sa to be precise) which requires setting your database to SQL Server and Windows Authentication mode (instructions here).

Installing

First steps (from the README)

1. Rename the mcafee_epo.py.pymssql to mcafee_epo.py
2. Enter credentials for accessing the database into the script

Here is where we diverge from the README, as it uses the built-in system Python (which is version 2.6.6 for RHEL 6.5) and this WILL NOT work with required Splunk libraries we have to import. This is the error you will get (as part of a stack trace) when using the built-in version of python:

ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-mcafee/bin/mcafee_epo.sh" ImportError: /opt/splunk/lib/python2.7/site-packages/lxml/etree.so: undefined symbol: PyUnicodeUCS2_DecodeUTF8

To find the error in your logs, use this Splunk search: index=_internal sourcetype=splunkd ERROR mcafee_epo

To check the unicode version of your instance of python, run the following in a python interpreter (just type python in a terminal window):

import sys
if sys.maxunicode > 65535:
 print 'UCS4 build'
else:
 print 'UCS2 build'

Now we want to download and build an alternate version of python from source.  The steps that follow are adapted from the helpful guide here.

VERY IMPORTANT: make sure you do a make altinstall otherwise you will overwrite the system's version of python and this is likely a REALLY bad idea.

yum groupinstall "Development tools"
yum install zlib-devel
yum install bzip2-devel
yum install openssl-devel
yum install ncurses-devel
wget --no-check-certificate https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tar.xz
tar xf Python-2.7.6.tar.xz
cd Python-2.7.6
./configure --prefix=/usr/local --enable-unicode=ucs2
make && make altinstall

Next, edit mcafee_epo.sh to invoke python2.7 (by default our newly built python will be at /usr/local/bin/python2.7).

Then we rejoin the standard process as described in the README for installing freetds and pymssql.

VERY IMPORTANT: Make sure to get the exact versions listed in the README file.  For example, I installed freetds 0.91 on my first attempt and when you do that, your queries to the McAfee ePO MSSQL database return no results!

Once we enable the mcafee_epo.sh scripted input everything should work!  But what if it doesn't?

Troubleshooting

Debug the mcafee_epo.py script

Edit mcafee_epo.py and set debug = True at the top of the file.  This will log to $SPLUNK_HOME/var/log/splunk/mcafee_epo.log and is searchable in Splunk.  To narrow down where your failure is occurring, add some logFile statements to the code.

If it is the connection to the database that is failing,  see the next section.

If it is the query that is failing, check your database and make sure all the tables and columns listed in the SQL query actually exist in the database.

Test your connection to the database

tsql -H '<hostname or IP address>' -p <port> -U '<user>'

You will be prompted for the password and then get a 1> prompt if the connection is successful.  If this fails, make sure you have no backslashes in your user field (see my caveats at the start of this article).  If that checks out, you need to go back and verify your host/IP, port, and password.

SSL certificate chains

I learned something interesting today when an intermediate certificate in our certificate chain expired over the weekend.  Our certificate chain looked like this:

Trusted Certificate Authority (CA) Cert
----> 2nd Level Certificate
--------> 3rd Level Certificate
------------> Machine/Application Certificate

It was the "2nd Level Certificate" that expired for us.  When that happened, our ArcSight console application would no longer allow us to log into the ArcSight ESM.  What was puzzling was that the ArcSight web interface continued to work just fine, as did other web applications who had that certificate in their chains.  We spent about a day working on this and eventually figured out why only the ArcSight Console was failing and that the fix turned out to be relatively simple.

When the browser initiates a secure connection to a server, it requests the application's certificate and walks the certificate chain until it finds a trusted CA certificate.  In this corporate environment, the 2nd level certificate had been revoked and a replacement added as a root certificate.  In addition the "3rd Level Certificate" was also listed as a trusted root certificate.  So as the browser walked up the certificate chain it did the following:

1. Examine "Machine/Application Certificate" -> this certificate is not trusted. Must examine the issuing authority for this certificate, the 3rd level cert. (Side note, adding the application certificate as an accepted root cert is what you do when you use a self-signed certificate.)
2. Examine "3rd Level Certificate" -> this is a trusted root certificate.
3. Certificate is trusted and connection is created.

Note that the expired "2nd Level Certificate" is never examined here.  However, the ArcSight Console application only had the "Trusted CA Cert" at the top level in its cacerts file.  So its flow looked like this:

1. Examine "Machine/Application Certificate" -> this certificate is not trusted.  Must examine the issuing authority for this certificate, the 3rd level cert.
2. Examine "3rd Level Certificate" -> this certificate is not trusted.  Must examine the issuing authority for this certificate, the 2nd level cert.
3. Examine "2nd Level Certificate" -> this certificate is not trusted and is EXPIRED.  Cease walking the certification chain.
4. A trusted connection can't be made.

We added the new "2nd Level Certificate" and the "3rd Level Certificate" to the ArcSight console's cacerts trust store and the problem was solved.  We could have added only the "3rd Level Certificate" and we would have fixed the problem.  However, the ArcSight keytoolgui application will object if you attempt to add a cert to the cacerts store and the certificate for its issuing authority is not present.  (It just throws a warning but will still let you override if you want to, this is how you install a self-signed cert.)  The reason the new "2nd Level Certificate" was installed was to avoid having to override that message when updating the cacerts store.

Are there any security implications to this?  It seems like we're creating a bypass around an expired certificate.

Why does a certificate authority (in this case 2nd level) issue certificates that expire after its own?  That seems like bad practice.