Splunk Enterprise Security and TA-mcafee

When configuring Splunk and Splunk ES to work with McAfee ePO make use of the vendor-supplied TA, aptly named TA-mcafee.  The basic instructions for getting it working found in $SPLUNK_HOME/etc/apps/TA-mcafee/README are a great starting point, but I stumbled into a few issues.  Hopefully this guide will help others avoid spending the amount of time I spent working my way through them.

Important caveats

• Test system ran Splunk 6.1.1 on RHEL 6.5
• I used the Pymssql option in the README
• I have a valid RHEL license (CentOS should work the same, but there are no guarantees)
• This was for a test system, as a result there may be more packages installed than strictly necessary
• I could not get the connection to work using domain credentials for authentication (e.g. MYDOMAIN\username).  I think it has something to do with the backslash and freetds (and yes, I did try escaping the backslash).  I used SQL credentials (sa to be precise) which requires setting your database to SQL Server and Windows Authentication mode (instructions here).

Installing

First steps (from the README)

1. Rename the mcafee_epo.py.pymssql to mcafee_epo.py
2. Enter credentials for accessing the database into the script

Here is where we diverge from the README, as it uses the built-in system Python (which is version 2.6.6 for RHEL 6.5) and this WILL NOT work with required Splunk libraries we have to import. This is the error you will get (as part of a stack trace) when using the built-in version of python:

ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-mcafee/bin/mcafee_epo.sh" ImportError: /opt/splunk/lib/python2.7/site-packages/lxml/etree.so: undefined symbol: PyUnicodeUCS2_DecodeUTF8

To find the error in your logs, use this Splunk search: index=_internal sourcetype=splunkd ERROR mcafee_epo

To check the unicode version of your instance of python, run the following in a python interpreter (just type python in a terminal window):

import sys
if sys.maxunicode > 65535:
 print 'UCS4 build'
else:
 print 'UCS2 build'

Now we want to download and build an alternate version of python from source.  The steps that follow are adapted from the helpful guide here.

VERY IMPORTANT: make sure you do a make altinstall otherwise you will overwrite the system's version of python and this is likely a REALLY bad idea.

yum groupinstall "Development tools"
yum install zlib-devel
yum install bzip2-devel
yum install openssl-devel
yum install ncurses-devel
wget --no-check-certificate https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tar.xz
tar xf Python-2.7.6.tar.xz
cd Python-2.7.6
./configure --prefix=/usr/local --enable-unicode=ucs2
make && make altinstall

Next, edit mcafee_epo.sh to invoke python2.7 (by default our newly built python will be at /usr/local/bin/python2.7).

Then we rejoin the standard process as described in the README for installing freetds and pymssql.

VERY IMPORTANT: Make sure to get the exact versions listed in the README file.  For example, I installed freetds 0.91 on my first attempt and when you do that, your queries to the McAfee ePO MSSQL database return no results!

Once we enable the mcafee_epo.sh scripted input everything should work!  But what if it doesn't?

Troubleshooting

Debug the mcafee_epo.py script

Edit mcafee_epo.py and set debug = True at the top of the file.  This will log to $SPLUNK_HOME/var/log/splunk/mcafee_epo.log and is searchable in Splunk.  To narrow down where your failure is occurring, add some logFile statements to the code.

If it is the connection to the database that is failing,  see the next section.

If it is the query that is failing, check your database and make sure all the tables and columns listed in the SQL query actually exist in the database.

Test your connection to the database

tsql -H '<hostname or IP address>' -p <port> -U '<user>'

You will be prompted for the password and then get a 1> prompt if the connection is successful.  If this fails, make sure you have no backslashes in your user field (see my caveats at the start of this article).  If that checks out, you need to go back and verify your host/IP, port, and password.