Drill Down into Spring Boot Actuator metrics

We’ve only seen this very useful feature documented in the official Spring Actuator API Documentation, so maybe not many are aware that you actually have some control over what the Spring Actuator metrics return to your requesting client.

The Spring Actuator API allows you to expose several useful metrics that you can use to monitor your Spring-based application. You can use these metrics to monitor your application health, the last few HTTP requests processed, system load, and so on. All this is rather well documented both by Spring themselves and other bloggers. However, when consuming the API, you may notice part of the response advertising “availableTags”, for example if we query the http.server.requests metric, we get:

In lines 5-18 we see the measurements array, which contains three statistics about all the requests made to the Spring App until that point in time. In the above example, we see COUNT, TOTAL_TIME, and MAX. However, we can also query Spring Actuator to return those statistics for only a subset of the requests that the Spring App has handled. Note the availableTags array that is defined in lines 19-48 in the above code snippet. This array informs you how to issue a request to Spring Actuator and limit the returned statistics to your chosen subset. For example, using the information we got from the above response, we now craft another GET request, like so:

http://localhost:8080/actuator/metrics/http.server.requests?tag=status:404

The difference is the ?tag=status:404 suffix at the end of the url. This query will now result in statistics for only those requests handled by Spring that have resulted in an HTTP status 404. Building the suffix is easy… Note the objects that make up the availableTags array in our code snippet above. Each object has two attributes, tag and values. The tag attribute is used in the first part of the query parameter value, while any option from the values array is used in the seconds part of the query parameter value. For example, the following object in the availableTags array:

{
     "tag":"uri",
     "values":[
            "NOT_FOUND"
    ],
}

results in the following possible queries:

/actuator/metrics/http.server.requests?tag=uri:NOT_FOUND

While the following object:

{
     "tag":"status",
     "values":[
           "404",
           "200",
           "500",
    ],
}

results in the following possible queries:

/actuator/metrics/http.server.requests?tag=status:404
/actuator/metrics/http.server.requests?tag=status:200
/actuator/metrics/http.server.requests?tag=status:500

Tip: The “values” array of each object gets populated dynamically depending on the different values the Spring App has generated. For example, in our above scenario using http.server.requests, the “values” array will not include a “204” option until your server responds to at least one request with that status. Therefore, monitoring the values array is a quick and easy way of getting the cardinality of a particular metric you are interested in.

Aside: For those of you looking for how to change the response returned by the “HTTP Trace” metric, this has changed between Spring v 1.5.x and Spring v2.0.x, so it’s best to have a look at the actual API documentation available here:

https://docs.spring.io/spring-boot/docs/2.0.x/api/org/springframework/boot/actuate/trace/http/Include.html

As a cheatsheet:

spring_boot_trace_comparison.png

The above constants can be used in the management.trace.http.include property in resources/application.properties, for example:

management.trace.http.include = remote_address, request_headers, response_headers
Advertisements

How to create a “heatmap” graph network visualization

What we’re after

@CyberSiftIO we’ve been going through an exercise of adding “confidence levels” to our visualizations. In other words, how confident is the CyberSift engine that an alert really is an anomaly/outlier? The above screenshot shows one of the ways we visualize the output from this exercise. Each blue node is an internal PC/Server, while the other nodes are detected anomalies – ranging from green (low confidence) to red (high confidence). This heatmap-style visualization immediately allows an analyst to focus on those anomalies that really matter. Without the different colors, there may be too many alerts to investigate, but with the heatmap colors and analyst immediately figures out that best start looking at the deep orange alert on the top right corner. In this post we’ll outline how we built the below visualization.

WhatsApp Image 2017-10-16 at 16.54.21.jpeg


The toolset

Here’s the libraries we used:

  • ReactJS as our base framework (optional – in reality any JS framework could be used, we just like how easy and structured ReactJS makes everything)
  • CytoscapeJS as our graph network visualization library
  • D3.js for some helper functions

 


The coding

We’ll assume you have a basic ReactJS app up and running (if not… use create-react-app). The first order of the day is to format our data in a way that CytoscapeJS expects it. In this particular case, this means building an array of objects. Assuming our array of objects is going to be called cystoscape_elementswe first loop through the internal nodes (light blue ones) and push a data object onto this array:

The most important thing to note in the above is that we add an “extra” object attribute named “backgroundColor” and set to “internal“, which we’ll later use for styling these nodes with the appropriate light blue color.


Next, we need to append our external nodes to the cytoscape_elements object. However unlike in our above code, these nodes need to be given a different color depending on their “confidence rating”. Let’s assume that the confidence rating can range from 0 (low outlier score) to 1 (high outlier score). We need to convert this range into a color palette. Fortunately, D3.js allows you to do exactly that, in a simple way:

In the above code, we defined a linear scale with a domain (possible number values) between 0 and 1. In the last line, we map the domain to a custom color range. The first RGB value is the start color which maps to the numerical value of “0”, the middle is the “pivot” value, and the last is the end color which maps to a numerical value of “1”.

Once we have this color scale, we can push external nodes to our array using a similar strategy as above:

Note how  as before, we define a new object attribute called “confidence”, and we subsequently populate this attribute with our previously defined color scale to convert the numerical confidence to an actual RGB color. We’ll use this attribute later to style the node.


Next, we add the “edges” in the graph connecting the nodes together in a pretty straightforward fashion:

Note the use of javascript type coercion (from int to string) as our id, and note the javascript scope of allowing the inner forEach function to increment the id_counter variable.


Last, we put it all together:

In the above code, note the lines:

  • Line 12: CytoscapeJS uses a system of “selectors” which allow you to style different elements of the graph network. In this case, we’re interested in nodes.
  • Line 16: We set the background-color CSS attribute to the value of our “confidence” attribute in the data object (if present) using the notation below (NB: this would be my entire takeaway from this article…)
'data(attribute)'
  • Line 32: We again use selectors to style the internal nodes with a light blue. Recall that internal nodes had a data attribute of “backgroundColor” set to “src“, and we leverage this in the selector:
'node[backgroundColor = "src"]'

The above code in plain English says: select those nodes whose “backgroundColor is set to “src

 


Wrap Up

As you can see, the most important points in this article relate to how to effectively use CytoscapeJS custom data attributes along with CytoscapeJS selectors. Together, they allow you to create very striking visualizations that really communicate a point efficiently to your target audience.