I haven't tried this before, but I set up a local instance very quickly to see what it does when you use that endpoint and I think there may be a clue a little further down in the logs:
[prometheus-8db6a00c-ea7b-3066-9308-3da27177abe8][10312][out]|[2022-04-13 19:40:19.087Z] [ds:prometheus] [info] [OK] Sending event message: {"endpoint":"http://localhost:9090/api/v1/query/","message":"There are no queryMetrics defined in the extension. Nothing to query Prometheus API for.","timestamp":""}
It is not in the documentation online but it is in the downloadable schemas, it looks like you need to define queryMetrics to use and endpoint like that. This makes sense as in that case the query would be appended to the url and not 404.
I have not tried this before but will update if I can try it before someone else who has responds.
"prometheus_query_metric" : {
"type" : "object",
"additionalProperties" : false,
"required" : [ "key" ],
"properties" : {
"key" : {
"type" : "string",
"description" : "generic definition of multiple metrics"
},
"featureSet" : {
"$ref" : "extension.schema.json#/definitions/featureSet"
},
"interval" : {
"$ref" : "extension.schema.json#/definitions/interval"
}
}
}
I am trying the queryMetrics and have slimmed my file down to just this one metrics for testing but I get an error when I try to upload the extension. This is my config and the error:
name: custom:prometheus-confluent
version: 1.0.2
minDynatraceVersion: "1.236"
author:
name: ccloud-exporter
prometheus:
- group: confluent kafka metrics
interval: 1m
dimensions:
- key: confluent
value: kafka
queryMetrics:
- key: custom.prometheus-confluent.confluent_kafka_server_active_connection_count
value: metric:confluent_kafka_server_active_connection_count
type: gauge
Ok I think I have a better understanding of this now. The error I was seeing was just an informational log message and in this case queryMetrics are not required (you may be using an older schema hence the error you saw afterwards as the variable names changed in more recent versions).
Regardless, you may be able to resolve your original error simply be removing 'query' from the url you enter in the configuration. Just .../api/v1/ is enough and Dynatrace will be appending query?query=... when it runs. Putting query in the url seems to mess it up.
This is what I have working now (with the above URL configured):
name: custom:prometheus-server-api-test
minDynatraceVersion: "1.238"
version: 0.0.17
author:
name: "Me"
metrics:
- key: new_prometheus_go_gc_cycles_automatic_gc_cycles_total
metadata:
displayName: "New Prometheus GO GC Cycles"
unit: Count
prometheus:
- group: testing
featureSet: testing
interval:
minutes: 1
metrics:
- key: new_prometheus_go_gc_cycles_automatic_gc_cycles_total
value: metric:go_gc_cycles_automatic_gc_cycles_total
The message about no queryMetrics being defined was what I focused on at first as well, but I found it is merely informational and it should be able to collect the specific metrics that are defined in the metrics section (like in my example).
[prometheus-58f5d752-87b4-39e5-b68e-eded6b0923c0][38696][out]|[2022-04-15 02:04:15.488Z] [ds:prometheus] [info] [OK] Sending event message: {"endpoint":"http://localhost:9090/api/v1/","message":"1 items returned from Prometheus API.","timestamp":""}
[prometheus-58f5d752-87b4-39e5-b68e-eded6b0923c0][38696][out]|[2022-04-15 02:04:15.597Z] [ds:prometheus] [info] [OK] Sending event message: {"endpoint":"http://localhost:9090/api/v1/","message":"There are no queryMetrics defined in the extension. Nothing to query Prometheus API for.","timestamp":""}
The important thing to look for is the "X items returned from Prometheus API" which in my case is the 1 metric I had configured and in yours is 0 which I can only interpret as the query not having any values for that metric.
If you're able to see this working with your metrics:
I'm not sure what else can be looked at apart from double checking everything. My environment is ahead of what you would be using but I can't say if it is a 'version' issue that is already fixed.
The only difference I see between ours is in mine I have a trailing forward slash in the URL - I don't think that should break it and in fact I've seen that it is supposed to be explicitly added but it is something to try.
I had been testing with a remote activation but also tried local now and it also is behaving the same and collecting my single metric.
Featured Posts
