David Vassallo's Blog

If at first you don't succeed; call it version 1.0

Category Archives: Open Source

AlienVault ELK Integration

In the last couple of blog posts[1][2] we’ve been exploring how to use the ELK stack as a forensic logging platform. We also had a couple of posts on deploying some AlienVault features [3][4]. In this post we explore a quick and easy way to integrate between the two systems. Apart from the flexible querying that ELK brings, ELK is also extremely easy to scale and replicate data across a distributed cluster of machines. In addition to all this, Kibana allows you to have auto-updating visualizations, such as trend analysis of the events per seconds from a particular data source (which by the way is a much more elegant and simple solution to the problem originally presented in [4])

Source trend analysis (with a very limited dataset unfortunately)

Source trend analysis (with a very limited dataset unfortunately)

AlienVault’s best feature in my opinion is the OSSIM (https://www.alienvault.com/products/ossim), an open source SIEM with a very flexible rule and correlation engine that works very well! Less of a joy to use is the AlienVault logger (https://www.alienvault.com/docs/data-sheets/AlienVault-Logger.pdf), which while it does what it’s advertised to do, is nowhere near as flexible or as polished as ELK is. After all, ELK is built from the ground up to deal with searching and scalability. The objective for me was to get the AlienVault OSSIM logs into ELK, one way or another. We came up with two ways of doing this:

1. Streaming logs from the AlienVault OSSIM servers to ELK in a “live” fashion. This is the preferred method but it does involve installing the NXLog [5] agent on alienvault systems.

2. Loading the OSSIM logs into ELK manually, “on-demand” in a bulk fashion. This is the best option for those deployments (maybe in highly sensitive or contractually-binding environments) where the alienvault sytems cannot be touched directly but logs still need to be shipped to ELK in some way.

– Streaming Logs

This is the preferred method because it “just works”. Simply build and install the NXLog agent on the Alienvault OSSIM server (there’s a guide on how to do this here [1]) and use something similar to the following for nxlog.conf:

We simply monitor the text logs being produced under “/var/ossim/logs/*.log” and that’s it. NXLog is intelligent enough to recursively search through the directory tree under /var/ossim/logs/ and pick up any files ending in “.log”. In this case, they get sent to host on port 5142. Fire up NXLog and that’s it…

– Manual, on-demand logs.

In those environments where we cannot simply install NXLog on alienvault systems, it’s still possible to ship logs semi-manually into ELK. To do this we install NXLog directly onto the ELK server (or any other staging server) and use the following nxlog.conf:

The interesting part of the above config file is line 23 – where we set “ReadFromLast FALSE”. This is necessary so that we can load past log files by simply copy/pasting OSSIM log files into the directory specified (“/elk/historic_data/ossim/*.log” in this case). We then send this data on to port 5142. So shipping any log data we need to ELK is as simple as using SFTP to login to the alienvault servers, locating the OSSIM logs of interest under /var/ossim/logs and copy / pasting those files into the /elk/historic_data/ossim directory on our staging server.

Logstash configuration

In either case, we need to properly configure Logstash to receive the logs. Here’s the configuration file used:

The most important section in the config log above is the part where we use the kv filter to parse the received log files:

         kv {
            value_split => "='"
            field_split => "' "

This uses the “key – value” logstash filter [6] to parse log file automatically without needing to define any complicated grok filters. The KV filter turned out to be incredibly useful because the OSSIM logs differ slightly according to which AlienVault plugin produced the log, but all OSSIM logs thankfully keep the same format of key-value pairs seperated by an equals (=) sign (trust me, going after the grok filters manually can get hairy… this is what it was looking like)

OSSIM logs produced in key value format

OSSIM logs produced in key value format

The KV filter neatly abstracts all this and caters for any changes in the key fields, leaving you with easily queried and searchable logs in ELK:

OSSIM logs in ELK

OSSIM logs in ELK



[1] Building a Logging Forensics Platform using ELK (Elasticsearch, Logstash, Kibana), http://blog.davidvassallo.me/2015/04/21/building-a-logging-forensics-platform-using-elk-elasticsearch-logstash-kibana/

[2] Beyond the basics : Logging Forensics with ELK (Elasticsearch, Logstash, Kibana), http://blog.davidvassallo.me/2015/06/25/beyond-the-basics-logging-forensics-with-elk-elasticsearch-logstash-kibana/

[3] AlienVault: Adding a logger to a distributed deployment, http://blog.davidvassallo.me/2015/06/03/alienvault-adding-a-logger-to-a-distributed-deployment/

[4] AlienVault: Monitoring individual sensor Events Per Second [EPS]http://blog.davidvassallo.me/2015/02/03/alienvault-monitoring-individual-sensor-events-per-second-eps/

[5] NXLog, http://nxlog.org/

[6] ElastichSearch KV filter: https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html

Building a Logging Forensics Platform using ELK (Elasticsearch, Logstash, Kibana)

During a recent project we were required to build a “Logging Forensics Platform”, which is in essence a logging platform that can consume data from a variety of sources such as windows event logs, syslog, flat files and databases. The platform would then be used for queries during forensic investigations and to help follow up on Indicators of Compromise [IoC]. The amount of data generated is quite large, ranging into terabytes of logs and events. This seemed right up elasticsearch’s alley, and the more we use the system, the more adept at this sort of use case it turns out to be. This article presents some configuration scripts and research links that were used to provision the system and some tips and tricks learned during implementation and R&D of the system

Helpful Reading

The ELK stack is proving to be a very popular suite of tools, and good documentation abounds on the internet. The official documentation is extremely helpful and is a must read before starting anything. There are some additional links which are most definitely useful when using ELK for logging:



General Architecture

One of the main advantage of ELK is it’s flexibility. There are multiple ways to achieve the desired result, so the rest of this blog post must be taken in context and adapted to your own environment where appropriate. To give some context to the configuration files which follow, below is the high level architecture which was implemented:

High Level Architecture for ELK forensic logging platform

High Level Architecture for ELK forensic logging platform

There were a couple of design considerations that led to the above architecture:

1. In this particular environment, there were major discussions around reliability vs performance during log transport. As most of you would know, this roughly would translate to a discussion around TCP vs UDP transport. UDP is a lot faster since there’s less overhead, however that same overhead allows TCP to be a lot more reliable and prevent loss of events/logs when there is a disconnection or problem. The resulting architecture uses both, with the reasoning being that most syslog clients are network nodes like routers, switches and firewalls, which are very verbose and hence performance is more of an issue. However, on the actual servers, we opted for TCP to ensure absolutely no events are lost, and the servers do in fact tend to be less verbose so the reliability gains are worth it for high value assets such as domain controllers.

2. The logstash defaults of creating a separate daily index in elasticsearch are actually the most sane settings we found, especially for backup and performance purposes, so we didnt change these

3. One of the nicest features of elasticsearch are the analyzers [1], which allow you to do “fuzzy searches” and return results with a “relevance score” [2]. However, when running queries against log data, this can actually be a drawback since log messages more often than not are very similar to each other, so keeping the analyzers on returned too many results and made forensic analysis needlessly difficult. The analyzers were therefore switched off, as can be seen in the below configuration files. Add to this a slight performance bonus for switching off analyzers and the solution was a very good fit.

NXLog Windows Configuration:

We ran across an issue when installing the NXLog client on windows servers. After restarting the NXLog service we would see an error in the logs along the lines of:

apr_sockaddr_info_get() failed

I never figured out the root cause of this error, even inserting the appropriate hostnames and DNS entries did not help. However using the below configuration and re-installing the client got rid of the error.

One point of interest in the above configuration is on line 30. By default NXLog will monitor the Application, Security, and System event logs. In this configuration sample one can see an example of also monitoring the post-2003 style event log “containers” where windows now stores application specific logs that are useful to monitor.

Also note the use of the to_json module, which converts the messages to JSON format. We will use this later when configuring logstash.

NXLog installation instructions on alienvault:

One of the requirements of the environment where this forensic logging platform was installed, is to integrate with AlienVault Enterprise appliance to be able to send logs from these systems to the elasticsearch nodes. Here are the steps that worked for us:

1. Installation did not work via pre-built binaries. Luckily, building from source is very easy. Download the tar.gz source package from NXLog community site here

2. Before proceeding, install dependencies and pre-requisites:

apt-get update
 apt-get install build-essential libapr1-dev libpcre3-dev libssl-dev libexpat1-dev

3. Extract the TAR.GZ file, and change directory into the extracted tar.gz folder and run the usual configure, make, install:

 make install

NXLog configuration (Linux):

The rest of the configuration for AlienVault servers is the same as a generic Linux host, with the exception that in the below config file we monitor OSSEC logs, which you may need to change depending on what you would like to monitor

Logstash configuration on elasticsearch:

This leaves us with the logstash configuration necessary to receive and parse these events. As noted above, we first need to switch off the elasticsearch analyzers. There are a couple of ways to do this, the easiest way we found was to modify the index template [3] that logstash uses and switch of analyzers from there. This is very simple to do:

– First, change default template to remove analysis of text/string fields:

vim ~/logstash-1.4.2/lib/logstash/outputs/elasticsearch/elasticsearch-template.json

– Change the default “string” mapping to not_analysed (line 14 in the default configuration file in v1.4.2)

analyzed –> not_analyzed

– Point logstash configuration to the new template (see line 121 in the logstash sample configuration below)

– If need be, delete any existing logstash indices / Restart logstash

Also note lines 27-32 in the above config file. This has to do with the fact that we are converting messages into JSON format in the NXLog client. The logstash documentation [4] states that:

For nxlog users, you’ll want to set this to “CP1252”.

In a future article we’ll go into a bit more depth into the above logstash configuration, and how we can use it to parse messages into meaningful data


[1] Elasticsearch Analyzers: http://www.elastic.co/guide/en/elasticsearch/reference/1.4/indices-analyze.html

[2] Elasticsearch relevance: http://www.elastic.co/guide/en/elasticsearch/guide/master/controlling-relevance.html

[3] Elasticsearch Index Templates: http://www.elastic.co/guide/en/elasticsearch/reference/1.x/indices-templates.html

[4] Logstash JSON documentation: http://logstash.net/docs/1.4.2/codecs/json


Get every new post delivered to your Inbox.

Join 229 other followers