Unexpected behavior in Log Processing

[DanielS]

Hello Community,

I've been facing a hard and intensive migration from another brand (🐕) to Dynatrace Grail. The following is only one example but I have the same erratic behavior with other rules.

In this case I've been able to setup a Processing Rule to add 3 columns to the log and then extract a metric with 3 dimensions. All this has been working without problems but... always a but, it suddenly stopped working. I have an open ticket with support but with almost 20 days and still waiting for a solution to this I turned to the community in case general knowledge finds another solution.

I've tried putting the rule in first place, it has less than 50 arguments, no other rule is processing the same service or matchers. But what was working is no longer working and I have no way to re-enable it. The test rule works fine, but no columns are added.

Any idea, same behavior, questions?

Thanks in advance.

[Tomasz_Wozniak]

Hi Daniel,

It would require a detailed investigation and I think this is not a good place for that.

It's good that you have created a support ticket. I would like to understand why this topic haven't been resolved in such a long time and haven't reached dev-level (or at least my team).

It's hard to say anything for sure without looking at this particular tenant. There are many posibilities, for example the max number of dimension values might have been reached. In such case the metric system would reject those datapoints. You could check customer-facing metric called "Server - Metrics - Rejections" (dsfm:server.metrics.rejections).

Of course that is just a guess. I would rather have a ticket for that, know the environment and perfor standard detailed investigation.

[DanielS]

Thanks, @Tomasz_Wozniak the ticket is still opened and it is been escalated to dev. May be if you want to see it. #342416

[Maheedhar_T]

Hi @DanielS ,
Since this is form logs, I can suggest one approach here. Switch to the latest UI of Dynatrace and then from the Data Explorer click on Open With

Then from the options select Notebooks. (Choose to create a new notebook or use an existing one).
You should see something like this

Which is a "DQL query" of your metric. Now you should be seeing the missing Data.


This missing data is due to the log metrics slowly being moved to Grail. Going forward you'll be seeing all the log metrics data using a DQL Query and not from the metrics Data Explorer page.

This can also be due to metric limit being reached as @Tomasz_Wozniak mentioned. Working with DQL is always a better alternative.

Hope this helps.

Regards,
@Maheedhar_T 

[DanielS · Accepted solution]

Well, we finally got to the solution to this. I'm leaving the answer because it's more than likely that someone else has the same problem.: 

Hello Daniel,

The developers have clarified why the matcher matchesValue(k8s.cluster.name, "AWS EKS PROD FINTECH") was not working in the log processing rule.

The field k8s.cluster.name is resolved after log processing during the enrichment stage, so there isn't such a field in the raw source log record. This is why it can't be used in the matcher during log processing.

For log processing rules, you should use fields that are in the raw data that is ingested. Unfortunately, you can't preview the raw data in Dynatrace, only the final result of the whole ingest processing and enrichment.

Hopefully this clears everything up! Please let me know if you have any questions!