topic Data Explorer: How do I view the most recent Dimension value? in Open Q&A

Data Explorer: How do I view the most recent Dimension value?

ASE — Thu, 24 Oct 2024 08:33:34 GMT

I have created a custom Log Metric using the Occurrence of logs records Metric measurement.

It contains 2 Dimensions that I plan to use to split and sort by.

When viewing my custom Log Metric in Data Explorer, is there a way to display only the most recent member_uuid/eventtime record?

As you can see below, the same member_uuid appears multiple times inside of the logs.

But as I create the Metric Event, all I really care about is the most recent log record entry.

Re: Data Explorer: How do I view the most recent Dimension value?

Fin_Ubels — Thu, 24 Oct 2024 00:11:07 GMT

Hey @ASE

I'm not too sure if there is a way to achieve this without using DQL which you wouldn't have access to if you are on Dynatrace Managed. You can use :lastreal in advanced mode but this takes the last real value of each instance of each dimension and since eventtime is a dimension it is going to evaluate it for each instance of eventtime too.

Also, using eventtime as a dimension could be problematic in the future if there are lots of these logs being ingested as there are limits on dimensions as mentioned in the docs here. I'd recommend keeping track of the dimension limits using the dsfm metric dsfm:server.metrics.custom_dimensions_usage. There are some other useful self monitoring metrics for this too.

Re: Data Explorer: How do I view the most recent Dimension value?

ASE — Thu, 24 Oct 2024 16:51:19 GMT

Thank you for your response, @Fin_Ubels.

We technically don't have to use the eventtime Dimension in our Metric Event, as all we really care about is the presence of the member_uuid.

I'm mainly using it to illustrate to you that the member_uuid record shows up multiple times in the logs (and is sorted by eventime) and all we want to see is the most recent record.

You are correct. Using a DQL query works as expected. We can both sort and remove any duplicate member_uuids.

However, when we create the custom Log Metric, we can't use the full DQL. We are limited with what we can use in the Matcher.

We also don't want to schedule and execute a full DQL query (using a Notebook or Dashboard) to retrieve these log records, as that is what creating a custom Log Metric is for.

The issue is that the log records themselves contain multiple records of the same member_uuid.

Here's a bit more context on what exactly we are trying to alert on.

We are planing to alert when the DQL query below returns any records. Value > 0.

I decided to create 2 separate Log Metrics, as these are technically 2 separate DQL queries that are joined. Each query is for a different entity's logs.

My thought is to use a Metric Expression in our Metric Event.

When Log Metric 1 + Log Metric 2 = 2, send us an alert.

We only care about a member_uuid that appears in BOTH Log Metrics at the same time.

1 + 1 = 2

fetch logs | filter dt.system.bucket == "bucket_name_1" | filter dt.host_group.id == "host_group_id_name" | parse content, "JSON:event" | filter event[message] == "message_name" | filter event[product_type] == "product_type_name" | fieldsAdd member_uuid = event[member_uuid] | join [ fetch logs | filter dt.system.bucket == "bucket_name_2" | filter ecs_cluster == "ecs_cluster_name" | filter matchesValue(requirement, "requirement_name") | dedup member_uuid, sort: { timestamp desc } ], on: { member_uuid } | filter right.requirement_status == "requirement_status_name"