Kotlin Coroutines – Flow parallel processing

Meant as an alternative to the Kotline Coroutine "Channel", a "Flow" is another way of enabling communication between two co-routines. The difference between the two is essentially that a channel is "hot" (i.e. starts as soon as it is declared), while a flow is "cold" (i.e. only starts when required (or "subscribed to" in reactive … Continue reading Kotlin Coroutines – Flow parallel processing

Encrypting traffic in transit to Apache NiFi

In this article we'll explain how to encrypt traffic going to an HTTP handler in NiFi, which is then forwarded to a backend HTTP server - in other words, an SSL offloading reverse proxy. Encrypting traffic in transit to NiFi involves the following steps: Creating a keystore containing a CA certificateCreating a truststore, which contains … Continue reading Encrypting traffic in transit to Apache NiFi

Is it Elastalert? No – it’s NiFi!!

One common requirement for users of Elasticsearch is to have automatic alerts sent out whenever some query gets matched, or when some other condition gets satisfied. In fact, Yelp have come up with a python-based solution for this in the form of Elastalert, which at time of writing, is extremely popular with over 5.5K stars … Continue reading Is it Elastalert? No – it’s NiFi!!

Consuming Netflow using NiFi

The problem Several network devices (especially Cisco) tend to use netflow for auditing network connections. It would be useful to log these connections in a structured data store (Elasticsearch is my data store of choice). Alternative Solutions Using the elasticsearch netflow module: https://www.elastic.co/guide/en/logstash/current/netflow-module.htmlThis works well right out of the box, and supports all netflow versions. … Continue reading Consuming Netflow using NiFi

Lessons Learned: GoLang GORM – filtering associations

The problem Given the following code: type User struct { gorm.Model Username string Orders []Order } type Order struct { gorm.Model UserID uint Foobar string } The above Golang code defines a "Has Many" association, leading to a schema where a "User" "has many" "Orders", with the "user_id" key acting as a foreign key. How … Continue reading Lessons Learned: GoLang GORM – filtering associations

Python Pickling in the cloud (or how to get python to execute code it hasn’t seen yet)

Problem - What are we trying to solve? Let's assume you have the beginnings of a simple distributed system: You use redis as your event queueYou have a "master" python script that pickles a python class, and sends it to your redis queueYou have a "worker" python script that waits for a new event, unpickles … Continue reading Python Pickling in the cloud (or how to get python to execute code it hasn’t seen yet)

Embedding Bokeh into a ReactJS app using BokehJS

This project highlights how to use BokehJS to (very) easily include Bokeh visualizations in an HTML webpage. The project uses the following tech: Bootstrapped with Create React App. Bokeh is an excellent python vizualization library that is quite popular among data scientists. Bokeh Homepage Flask is a python webserver microframework that allows you to quickly … Continue reading Embedding Bokeh into a ReactJS app using BokehJS