Some weeks ago, I wrote a post detailing how to send JMX metrics from a Spring Boot app to Elasticsearch by developing another Spring Boot app.
Getting to create such an app is not always the right idea but developers are makers - software makers, and developing new apps is quite alluring to them. However, in the overall scheme of things, this means time is not only spent in development, but also for maintenance during the entire lifetime of the app. Before going the development path, one should thoroughly check whether out-of-the-box alternatives exist.
Back to JMX metrics: only the straightforward Logstash jmx plugin was tried before calling it quits because of an incompatibility with Elasticsearch 5. But an alternative exists with the Logstash http_poller plugin.
This Logstash input plugin allows you to call an HTTP API, decode the output of it into event(s), and send them on their merry way. The idea behind this plugins came from a need to read springboot metrics endpoint, instead of configuring jmx to monitor my java application memory/gc/ etc.
Jolokia is already in place, offering HTTP access to JMX.
The last and only step is to configure HTTP poller plugin, which is fairly straighforward.
The URL is composed of the standard actuator URL appended with /jolokia/read/ and the JMX’s ObjectName of the desired object.
Here’s a sample configuration snippet, with URLs configured for:
- Response site for the root page
- HTTP 200 status code counter for the root page
- Operating system metrics
- Garbage collector metrics
With the help of the jconsole, adding more metrics is a no-brainer.
input {
http_poller {
urls => {
"200.root" =>
"http://localhost:8080/manage/jolokia/read/org.springframework.metrics:name=status,type=counter,value=200.root"
"root" =>
"http://localhost:8080/manage/jolokia/read/org.springframework.metrics:name=response,type=gauge,value=root"
"OperatingSystem" => "http://localhost:8080/manage/jolokia/read/java.lang:type=OperatingSystem"
"PS Scavenge" => "http://localhost:8080/manage/jolokia/read/java.lang:type=GarbageCollector,name=PS%20Scavenge"
}
request_timeout => 60
schedule => { every => "10s"}
codec => "json"
}
}
This should output something akin to:
{
"request" => {
"mbean" => "org.springframework.metrics:name=response,type=gauge,value=root",
"type" => "read"
},
"@timestamp" => 2016-01-01T10:19:45.914Z,
"@version" => "1",
"value" => {
"Value" => 4163.0,
"LastUpdated" => "2016-12-30T16:29:28+01:00"
},
"timestamp" => 1483121985,
"status" => 200
}
Beyond the raw output, there are a couple of possible improvements:
valueandrequestfields are inner objects, for no added value. Flattening the structure can go a long way toward making writing queries easier.- Adding tags depending on the type improve categorization.
A possible alternative would be to parse the JMX compound name into dedicated fields with
grokordissect. - The
@timestampfield can be replaced with theLastUpdatedvalue and interpreted as a date.
Filters to the rescue:
filter { (1)
mutate { add_field => { "[mbean][objectName]" => "%{request[mbean]}" }}
mutate { remove_field => "request" }
}
filter { (2)
if [mbean][objectName] == "java.lang:type=OperatingSystem" {
dissect {
mapping => {
"mbean[objectName]" => "%{[mbean][prefix]}:type=%{[mbean][type]}"
}
}
mutate { remove_field => "value[ObjectName]" }
} else if [mbean][objectName] == "java.lang:name=PS Scavenge,type=GarbageCollector" { (3)
dissect {
mapping => {
"mbean[objectName]" => "%{[mbean][prefix]}:name=%{[mbean][name]},type=%{[mbean][type]}"
}
}
mutate { remove_field => "value[ObjectName]" }
} else if [mbean][objectName] =~ "^.*,type=gauge,.*$" or [mbean][objectName] =~ "^.*,type=counter,.*$" { (4)
date { match => [ "%{value[lastUpdated]}", "ISO8601" ] }
mutate { replace => { "value" => "%{value[Value]}" }}
mutate { convert => { "value" => "float" }}
if [mbean][objectName] =~ "^.*,type=gauge,.*$" { (5)
dissect {
mapping => {
"mbean[objectName]" =>
"%{[mbean][prefix]}:name=%{[mbean][name]},type=%{[mbean][type]},value=%{[mbean][page]}"
}
}
} else if [mbean][objectName] =~ "^.*,type=counter,.*$" {
dissect {
mapping => {
"mbean[objectName]" =>
"%{[mbean][prefix]}:name=%{[mbean][name]},type=%{[mbean][type]},value=%{[mbean][status]}.%{[mbean][page]}"
}
}
}
}
}A little explanation might be in order.
| 1 | Move the initial request→mbean field to the mbean→objectName field. |
| 2 | For OS metrics, create mbean nested fields out of the objectName nested field and remove it from the value field. |
| 3 | For GC metrics, create mbean nested fields out of the objectName nested field using a slightly different pattern and remove it from the value field. |
| 4 | For gauge or counter metrics, interpret the value→lastUpdated nested field as a date, move the nested value→Value field to the root and interpret as a float value. |
| 5 | For gauge or counter metrics, create mbean nested fields using a pattern specific for each metric. |
Coupled with the initial configuration, this outputs to the following (for the gauge):
{
"@timestamp" => 2016-01-01T10:54:15.359Z,
"mbean" => {
"prefix" => "org.springframework.metrics",
"name" => "response",
"objectName" => "org.springframework.metrics:name=response,type=gauge,value=root",
"type" => "gauge",
"value" => "root"
},
"@version" => "1",
"value" => 4163.0,
"timestamp" => 1483206855,
"status" => 200,
"tags" => []
}
Given the current complexity of the configuration, remember the next logical step is to decouple the single snippet into multiple files. The order used by Logstash is by lexicographical file name. Name files accordingly:
00-app-input.conf10-global-filter.conf20-specific-filter.conf99-console-output.conf
| gaps in the number scheme allow to add intermediary filters in the future, with no renaming |
All in all, everything required is available through configuration, without any coding. Always check that it’s not the case before reinventing the wheel.