Apache NiFi: From Syslog to Elasticsearch

Apache Nifi is the result of an project open-sourced by the NSA. It's described as a "data traffic program"... For users familiar with the Elastic ecosystem, think of it as a GUI-enabled mashup of Filebeat, Winlogbeat and Logstash. In essence Nifi allows you to build data processing pipelines of arbitrary complexity and enrich the data or … Continue reading Apache NiFi: From Syslog to Elasticsearch

Advertisements

Lessons Learned: Winlogbeat & Forwarded Events – no event description

Scenario: Shipping Azure Cloud Logs to an Elasticsearch Cluster The Azure Log Service [AZLog ] audits events across your Azure Cloud infrastructure, and sends these to a central log collector. It leverage the Windows Event Forwarding subsystem to do this, meaning that the collector server will be able to view the AZLog alerts via the … Continue reading Lessons Learned: Winlogbeat & Forwarded Events – no event description

Elasticsearch REST API: JEST upsert

I've already written about tips and tricks when using the Elasticsearch Java API. The Elasticsearch REST API has been going from strength to strength, and it seems that going forward the Elasticsearch team will focus more on the REST API than the native JAVA client. At the time of writing however, the official java REST library … Continue reading Elasticsearch REST API: JEST upsert

Nugget Post: Insert an alert / rule name in emails sent from ElastAlert

ElastAlert is a fantastic python module that makes it easy to setup near - realtime alerts and responses to entries of interest. A quick piece of information  which I didnt find clearly documented was how to insert an alert name into a custom subject line sending email alerts.  A typical custom email alert rule file would include … Continue reading Nugget Post: Insert an alert / rule name in emails sent from ElastAlert

Embedding Javascript and HTML into Kibana 5.x

Reading this post: http://www.supermind.org/blog/1213/embed-custom-javascript-and-html-in-a-kibana-4-x-visualization Kelvin makes a really good suggestion.... and very good inspiration. The suggestion works fine but in my case I needed it to go a step further - I needed to embed an iframe into Kibana. The iframe (and indeed any other <script> tag) was being filtered out by Angular's sanitization protection. I … Continue reading Embedding Javascript and HTML into Kibana 5.x

Elasticsearch & Java: Tips for faster re-indexing

Notes based on some feedback: Elasticsearch seem to be pushing the REST client rather than using the native Java client... to future proof your code you may be better off going down this route. Why not just use the Re-Index API? Although it's still considered experimental, this may be a good option if you dont have … Continue reading Elasticsearch & Java: Tips for faster re-indexing

ELK : exporting to CSV

Note: the following requires the "jq" json parser, available from: http://stedolan.github.io/jq/ 1. Run the desired query through the Kibana WebUI 2. Expand the additional options pane by clicking on the arrow underneath the graph as indicated in the below screenshot: 3. Select “Request” and copy the request displayed: 4. Open a linux terminal and use the … Continue reading ELK : exporting to CSV