on - edited on by
Michal_Gebacki
Summary
OneAgent 1.313 includes an important change to LogAnalytics logic for timestamp processing of log file content. The new default timezone for automatically recognized timestamp patterns is the local host's timezone instead of UTC. This is a positive change, because we don't have to create timestamp rules to override the timezone of app logs timestamped in the local timezone.
Problem
Some applications, including OneAgent processes, may timestamp logs in timezone UTC instead of the local host's timezone, which may cause issues with metric and event extraction in Dynatrace pipelines. Metrics and events are time sensitive data sets, so any logs that are older than an hour will be ignored for metric extraction and any events older than 15 minutes when they are created may not generate any alerts.
For classic metric extraction that stores metrics in our classic storage, our preset classic dashboard Metric & Dimension Usage + Rejections will show discard reasons, for example, rejections that cause gaps because data is too far in the past.
Troubleshooting steps
We can confirm the time zones or UTC offsets that OneAgent used for timestamp patterns of each log source in OneAgent diagnostics or the oneagent-logmon-detailed.log file in folder loganalytics. This folder exists in /var/log/dynatrace/oneagent on Linux and %PROGRAMDATA%\dynatrace\oneagent\log on Windows.
In the below sample, the Diagnostic statistics for LGI: /opt/app/log shows that a timestamp pattern %Y-%m-%d %H:%M:%S was processed with UTC offset 0 and the time offset between the current time and the processed time is an average, minimum and maximum of 4 hours (14400 / 3600) in the past. Our app logs in timezone EST or UTC-4.*
[2025-03-18 09:19:3240.915 UTC] [/loggroupinstance.cpp] [debug] Diagnostic statistics for LGI: /opt/app/log report period: 15 minutes:
Date time patterns statistics:
Pattern: %Y-%m-%d %H:%M:%S uses: 736, time offset avg: 14430s, min: 14401s, max: 14464s, timezones: [default: 0min], not monotonic timestamps: 0
Lines without matched pattern: 0
In cases when we see the below with a minimum time offset that is not equivalent in hours to the average and maximum, LogAnalytics processed a newly added log source with many older records, so the app timestamp pattern is written in the correctly configured timezone UTC or a UTC offset of 0. LogAnalytics logs the diagnostic statistics of each auto-discovered and custom log source once every 15 minutes, so we may wait at least 15 minutes after adding a new log source before checking the time offset statistics in the most recent Diagnostic statistics log pattern for our log source.
Pattern: %Y-%m-%d %H:%M:%S uses: 736, time offset avg: 14430s, min: 1s, max: 14464s, timezones: [default: 0min], not monotonic timestamps: 0
*LogAnalytics logs refer to auto-discovered and custom log source log paths as LGIs, which is an abbreviation of Log Group Instances.
Resolution
We can resolve issues related to metric gaps or missing alerts on events of extraction on logs by configuring a timestamp rule with the correct timezone. If your app's logged timestamp pattern is recognized by the OneAgent as evidenced by Diagnostic statistics, we don't have to add a timestamp pattern to our rule using format specifiers.
We can add a timestamp rule in classic Settings > Log monitoring > Timestamp/splitting pattern with at least:
- the timezone set the local server time or the timezone of your app logs.
- a condition matcher to limit the scope of the rule to only your affected log source.
We don't need to override the timestamp search limit unless your timestamp patterns are more than 64 bytes from newlines, or we want to ignore your timestamp patterns in content with a timestamp search limit of 0 bytes.
What's next
If a timestamp rule doesn't resolve your issue with missing metrics or alerts on data extracted from logs, please don't hesitate to contact Support with the results of the above steps and OneAgent SupportArchive.
