How to create a “heatmap” graph network visualization

What we’re after

@CyberSiftIO we’ve been going through an exercise of adding “confidence levels” to our visualizations. In other words, how confident is the CyberSift engine that an alert really is an anomaly/outlier? The above screenshot shows one of the ways we visualize the output from this exercise. Each blue node is an internal PC/Server, while the other nodes are detected anomalies – ranging from green (low confidence) to red (high confidence). This heatmap-style visualization immediately allows an analyst to focus on those anomalies that really matter. Without the different colors, there may be too many alerts to investigate, but with the heatmap colors and analyst immediately figures out that best start looking at the deep orange alert on the top right corner. In this post we’ll outline how we built the below visualization.

WhatsApp Image 2017-10-16 at 16.54.21.jpeg


The toolset

Here’s the libraries we used:

  • ReactJS as our base framework (optional – in reality any JS framework could be used, we just like how easy and structured ReactJS makes everything)
  • CytoscapeJS as our graph network visualization library
  • D3.js for some helper functions

 


The coding

We’ll assume you have a basic ReactJS app up and running (if not… use create-react-app). The first order of the day is to format our data in a way that CytoscapeJS expects it. In this particular case, this means building an array of objects. Assuming our array of objects is going to be called cystoscape_elementswe first loop through the internal nodes (light blue ones) and push a data object onto this array:

The most important thing to note in the above is that we add an “extra” object attribute named “backgroundColor” and set to “internal“, which we’ll later use for styling these nodes with the appropriate light blue color.


Next, we need to append our external nodes to the cytoscape_elements object. However unlike in our above code, these nodes need to be given a different color depending on their “confidence rating”. Let’s assume that the confidence rating can range from 0 (low outlier score) to 1 (high outlier score). We need to convert this range into a color palette. Fortunately, D3.js allows you to do exactly that, in a simple way:

In the above code, we defined a linear scale with a domain (possible number values) between 0 and 1. In the last line, we map the domain to a custom color range. The first RGB value is the start color which maps to the numerical value of “0”, the middle is the “pivot” value, and the last is the end color which maps to a numerical value of “1”.

Once we have this color scale, we can push external nodes to our array using a similar strategy as above:

Note how  as before, we define a new object attribute called “confidence”, and we subsequently populate this attribute with our previously defined color scale to convert the numerical confidence to an actual RGB color. We’ll use this attribute later to style the node.


Next, we add the “edges” in the graph connecting the nodes together in a pretty straightforward fashion:

Note the use of javascript type coercion (from int to string) as our id, and note the javascript scope of allowing the inner forEach function to increment the id_counter variable.


Last, we put it all together:

In the above code, note the lines:

  • Line 12: CytoscapeJS uses a system of “selectors” which allow you to style different elements of the graph network. In this case, we’re interested in nodes.
  • Line 16: We set the background-color CSS attribute to the value of our “confidence” attribute in the data object (if present) using the notation below (NB: this would be my entire takeaway from this article…)
'data(attribute)'
  • Line 32: We again use selectors to style the internal nodes with a light blue. Recall that internal nodes had a data attribute of “backgroundColor” set to “src“, and we leverage this in the selector:
'node[backgroundColor = "src"]'

The above code in plain English says: select those nodes whose “backgroundColor is set to “src

 


Wrap Up

As you can see, the most important points in this article relate to how to effectively use CytoscapeJS custom data attributes along with CytoscapeJS selectors. Together, they allow you to create very striking visualizations that really communicate a point efficiently to your target audience.

Advertisements

From JQuery to ReactJS

I have previously worked with ReactJS – most notably during my Master’s dissertation, however the main Javascript library I work with when working for clients and companies still remains the venerable JQuery. This is changing as more and more organizations I interact with move to more modern frameworks like Angular and ReactJS.

Where to start if you’re looking to upgrade your skills, or migrate a product from JQuery to ReactJS? Any tips? Here’s what we learned so far:

Be ready to change your mindset – but never give up

It’s not so much a learning curve as it is a roller coaster. Blogger Ben Nadel has an excellent drawing that I guarantee will also be your experience learning any JS framework; be it Angular or React:

feelings_about_angularjs_over_time
Pretty much…

What makes ReactJS so powerful compared to JQuery is the fact that it’s fundamentally different. Sometimes your own prior experience with JQuery will hamper you in learning React because of pre-conceived notions on how things should be done. Power through it – through all the ups and downs, the curve still has an upwards trend. Once you get to grips with the whole idea of unidirectional data flow,  components, props and states, React makes you a lot more productive. That said, there are a few things that helped me transition easier…

Create-react-app is a must

No doubt about it, the worst part of being a newbie is getting used to the new tools, environments, moving parts and components of your new pipeline. Create-react-app is your savior here. Not only does it setup your dev environment in a few simple commands, you can rest assured that it’s doing so according to industry best practices (which eases those nagging questions at the back of your brain: “Am I doing this right?”). One very useful article I found was how to configure create-react-app to proxy through API requests directly to my back-end:

How to get “create-react-app” to work with your API

Unleash the power of reusable components

It takes banging your head with few project requirements to realize just how powerful and useful React Components are. In reality, the whole point of using React is to define a component once:

class David extends React.Component {
      render() {
                   return (<h1>David says hi...</h1>);
      }
}

… and then re-using that component over and over again as a “normal” HTML attribute

 <David />

 Thing is, when clever people start publishing their own components, you can re-use these components and requirements that took days before now take you minutes. These components could be anything, my favorite so far:

cryptoportfolio
Applying Google’s MDL to React JS becomes a breeze…

Redux-Saga vs Redux-Thunk

Talking about Dexie.js brings me to my next point. One of the aspects of Redux that always caught me with my pants down was the inability of reducers to process async functions (it seems logical in hindsight – an async function returns a handle to a future result such as a Promise – not the actual result). This is something you need to keep in mind since some very common requirements like fetching data from an API or indeed using libraries like Dexie require you to use async functions.

In order to tackle this, redux-thunk was born. A nice and easy blog post on how to use it can be found here:

http://blog.nojaf.com/2015/12/06/redux-thunk/

Coming from a JQuery background, I was already used to the idea that events are used to trigger certain actions – just like in redux actions are used to trigger reducers that in turn change your state. The redux-thunk approach of defining an async function within an action ran a bit contrary to my thinking – an “action creator” as the blog above refers to it. It would be easier (to my JQuery mind) to have a function that “listens” for an async action, performs that async action, and then returns another action with the result for your dispatch to process. This thought process resonated more with me (again, coming from an event / subscription model). In other words:

Redux-Saga Workflow

As you can see, this is where Redux-Saga enters the picture. This redux middleware listens for specified actions (usually you’d specify those that need to trigger async actions like API operations or Dexie.js), runs a function to deal with the async actions (tip: definitely try stick to using the yield keyword..) and then dispatches another action with the results that your usual redux reducers can then pick up. I guess it comes down to personal preference, but this paradigm was a lot easier to deal with in my head

Always try code a simple / sample project while learning

In my case most of my learning was done through reading documentation and via code-academy. My side / sample project was to build a web app that tracks my cryptocurrency investments, incorporating features like indexeddb via Dexie.js, API calls via Saga and so on…

https://github.com/dvas0004/react-crytoportfolio

http://dvas0004.github.io/

(Note sparklines only work if you visit the page at least twice from the same browser – the app has no server side components, it runs entirely in the browser so the sparklines represent change in crytocurrency price since your last visit)