One common requirement for users of Elasticsearch is to have automatic alerts sent out whenever some query gets matched, or when some other condition gets satisfied. In fact, Yelp have come up with a python-based solution for this in the form of Elastalert, which at time of writing, is extremely popular with over 5.5K stars … Continue reading Is it Elastalert? No – it’s NiFi!!
The problem Several network devices (especially Cisco) tend to use netflow for auditing network connections. It would be useful to log these connections in a structured data store (Elasticsearch is my data store of choice). Alternative Solutions Using the elasticsearch netflow module: https://www.elastic.co/guide/en/logstash/current/netflow-module.htmlThis works well right out of the box, and supports all netflow versions. … Continue reading Consuming Netflow using NiFi
Introduction In a previous article, we explored how PCA can be used to plot credit card transactions into a 2D space, and we proceeded to visually analyse the results. In this article, we take this process one step further and use hierarchical clustering to automate parts of our analysis, making it even easier for our … Continue reading Analyzing credit card transactions using machine learning techniques – 3
Principal Component Analysis - Introduction and Data Preperation Principal Component Analysis [PCA] is an unsupervised algorithm which reduces dimensionality and is widely used. A good visual explanation can be found here: http://setosa.io/ev/principal-component-analysis/ As mentioned in our previous article, Correspondence Analysis works exclusively on categorical data. In contrast, PCA accepts only numerical data. This means our data … Continue reading Analyzing credit card transactions using machine learning techniques – 2
In this article we'll explore how Principal Component Analysis [PCA]  - a popular data reduction technique - can help a busy security or network administrator. Any such administrator has often been faced with a daunting problem... going through reams of firewall or router connection logs trying to figure out if any of the thousands … Continue reading Data mining firewall logs : Principal Component Analysis
During a recent study module for a MSc I am undertaking we discussed the importance of continuous monitoring of data sources as part of a sound security defensive strategy. This lead me down a very interesting path, eventually culminating in my discovery of an entire subset in security discipline many refer to as "Security visualization". There are … Continue reading Getting started with Neo4J and security data analysis