Apache NiFi: Custom Web Scraper Processor – Powered by Selenium

In this article we explore how to build a custom Apache Nifi processor. Our objective is to build a custom NiFi processor, written in Java, that uses Selenium to scrape an arbitrary piece of information off a web-page. The end result will look like this: https://www.youtube.com/watch?v=alRC8owgjl4&feature=youtu.be This highlights the flexibility of Apache NiFi, showing off … Continue reading Apache NiFi: Custom Web Scraper Processor – Powered by Selenium

Advertisements

Lessons learned: Gephi not starting up

Scenario: When using Ubuntu Trusty (14.04.3LTS) Gephi 0.82 initializes but gets stuck on "Loading Cached Objects" Solution: The default Gephi startup bash script does not correctly initialize the JAVA path (it uses $jdkhome) The solution is simply to add the following to the beginning of the startup script (~/gephi/bin/gephi) export JAVA_HOME=/usr/lib/jvm/default-java jdkhome=$JAVA_HOME The full script is … Continue reading Lessons learned: Gephi not starting up

ELK : exporting to CSV

Note: the following requires the "jq" json parser, available from: http://stedolan.github.io/jq/ 1. Run the desired query through the Kibana WebUI 2. Expand the additional options pane by clicking on the arrow underneath the graph as indicated in the below screenshot: 3. Select “Request” and copy the request displayed: 4. Open a linux terminal and use the … Continue reading ELK : exporting to CSV

AlienVault: Monitoring individual sensor Events Per Second [EPS]

In a distributed AlienVault environment, it is important to be able to monitor individual sensor's output. In our case, the requirements was to: Monitor each sensor's generated events over a configurable interval If the number of generated events of a sensor goes below a configured threshold, then notify the user via email There are several … Continue reading AlienVault: Monitoring individual sensor Events Per Second [EPS]

OSSEC event loss troubleshooting

There is a general consensus that OSSEC will lose events in the event that the main OSSEC server goes offline for whatever reason ( [1] , [2] ) - be it the service is stopped, a network disconnection, or anything in between. However, there doesn't seem to be much information on when exactly even loss can occur, for … Continue reading OSSEC event loss troubleshooting

Plotting the 95th percentile using Centreon

Calculating the 95th percentile of bandwidth used by a client is a common method of billing for ISP and service providers [1]. Hence, it is also of great interest to the client to plot these values as well to keep track of their service provider fees and double check bills. Plotting the 95th percentile on Centreon … Continue reading Plotting the 95th percentile using Centreon

Lessons Learned: Setting up TVHeadend for live TV streaming in Malta

I recently had the opportunity of setting up a system which captures free-to-air channels and streams them over IP for use by TV set-top boxes such as RaspBMC (or plain old XBMC). We decided on using the excellent TVHeadend. The system was made up of two components: A TVHeadend server. This was simply a desktop … Continue reading Lessons Learned: Setting up TVHeadend for live TV streaming in Malta