Drill Down into Spring Boot Actuator metrics

We’ve only seen this very useful feature documented in the official Spring Actuator API Documentation, so maybe not many are aware that you actually have some control over what the Spring Actuator metrics return to your requesting client.

The Spring Actuator API allows you to expose several useful metrics that you can use to monitor your Spring-based application. You can use these metrics to monitor your application health, the last few HTTP requests processed, system load, and so on. All this is rather well documented both by Spring themselves and other bloggers. However, when consuming the API, you may notice part of the response advertising “availableTags”, for example if we query the http.server.requests metric, we get:

In lines 5-18 we see the measurements array, which contains three statistics about all the requests made to the Spring App until that point in time. In the above example, we see COUNT, TOTAL_TIME, and MAX. However, we can also query Spring Actuator to return those statistics for only a subset of the requests that the Spring App has handled. Note the availableTags array that is defined in lines 19-48 in the above code snippet. This array informs you how to issue a request to Spring Actuator and limit the returned statistics to your chosen subset. For example, using the information we got from the above response, we now craft another GET request, like so:

http://localhost:8080/actuator/metrics/http.server.requests?tag=status:404

The difference is the ?tag=status:404 suffix at the end of the url. This query will now result in statistics for only those requests handled by Spring that have resulted in an HTTP status 404. Building the suffix is easy… Note the objects that make up the availableTags array in our code snippet above. Each object has two attributes, tag and values. The tag attribute is used in the first part of the query parameter value, while any option from the values array is used in the seconds part of the query parameter value. For example, the following object in the availableTags array:

{
     "tag":"uri",
     "values":[
            "NOT_FOUND"
    ],
}

results in the following possible queries:

/actuator/metrics/http.server.requests?tag=uri:NOT_FOUND

While the following object:

{
     "tag":"status",
     "values":[
           "404",
           "200",
           "500",
    ],
}

results in the following possible queries:

/actuator/metrics/http.server.requests?tag=status:404
/actuator/metrics/http.server.requests?tag=status:200
/actuator/metrics/http.server.requests?tag=status:500

Tip: The “values” array of each object gets populated dynamically depending on the different values the Spring App has generated. For example, in our above scenario using http.server.requests, the “values” array will not include a “204” option until your server responds to at least one request with that status. Therefore, monitoring the values array is a quick and easy way of getting the cardinality of a particular metric you are interested in.

Aside: For those of you looking for how to change the response returned by the “HTTP Trace” metric, this has changed between Spring v 1.5.x and Spring v2.0.x, so it’s best to have a look at the actual API documentation available here:

https://docs.spring.io/spring-boot/docs/2.0.x/api/org/springframework/boot/actuate/trace/http/Include.html

As a cheatsheet:

spring_boot_trace_comparison.png

The above constants can be used in the management.trace.http.include property in resources/application.properties, for example:

management.trace.http.include = remote_address, request_headers, response_headers
Advertisements

Elasticsearch REST API: JEST upsert

I’ve already written about tips and tricks when using the Elasticsearch Java API. The Elasticsearch REST API has been going from strength to strength, and it seems that going forward the Elasticsearch team will focus more on the REST API than the native JAVA client. At the time of writing however, the official java REST library doesn’t seem to have support for the abstraction of the bulk API, so I followed some advice and looked into the JEST library.

The only snag with the Jest library is that when it comes to bulk operations, the documentation only gives examples of scripted updates. The Elasticsearch update API also allows for updates using partial documents. Jest supports this functionality, but I couldn’t find good documentation for this. Here-under is an example for anyone looking for this:

The important points:

  • You can still use the official java elasticsearch client’s “XContentFactory.jsonBuilder” library to more easily build your JSON objects.
  • The trick is in line 26 above:

jsonBuilder().startObject().startObject(doc”)

This creates a nested object with “doc” as the inner JSON object, as outlined by the elasticsearch documentation:

{
    "doc" : {
        "name" : "new_name"
    }
}

The first “startObject()” creates the outer curly brackets, while the second startObject(“doc”) creates the inner “doc” object.

  • We add content to the JSON object in lines 27-29
  • Just like we had to use two startObject() calls, we need to close the object with two endObject() calls as shown in line 31

The rest of the snippet deals with the actual bulk update. We pass the object we just created into an Update Builder, which gives us a “Bulkable Object” that we can pass on to the jest bulk processor. The snippet is taken from a larger program where it resides in a loop – which explains the if/else clause in lines 37-48; it’s important to “flush” the bulk service every so often. The native java client would to this automatically – so far in Jest you need to account for this yourself