How to troubleshoot ingestion of missed log data after the Log age limit

jgrant · ‎07 Sep 2026

Summary
Problem
Troubleshooting steps
Resolution
What's next

Summary

Data of Ingest sources can be delayed and lost before reaching our platform for a myriad of reasons: system failures, delayed configuration changes to match application changes, major network disruptions, injudicious revisions to sensitive configurations, binary or invalid content forcing discards by the sender, receiver or OpenPipeline, etc. Some data sources are mission critical, and require storage in our platform so teams can review that data, monitor metrics and receive alerts. When a gap in data occurs, it can be difficult for app teams to untangle the delayed data from truly missing data without going to the source or an active 3rd party vendor's solution, and comparing to our platform.

Problem

Some data can be backfilled but not all data is buffered client-side or backfilled equally. For example, OneAgent log module saves up to 10 Mebibytes of logs before it has to either send or drop that data. Once dropped, that data is not automatically recollected from the source. For generic ingest to our API endpoints, different cloud platforms measure and handle backpressure of logs and other datatypes differently, often with exponential backoff, maximum retry and rate limits. Grail has maximum ages for accepting new records with dated timestamps, which is the Log age ingestion limit in the Logs table.

In Dynatrace, teams can check the Total ingested records per configuration tile in the tenant's Ready-made dashboard OpenPipeline usage overview. Since data can be backfilled up to age limit, you can expect a spike in ingest on recovery to a monitoring gap of a major data source. You can compare the area of a spike to previous days and days of the previous weeks to gain some insight into the impact of data loss, however, this is not a practical approach to understand the precise impact of data delay and loss, especially in detail.

A better approach is to query the affected table or bucket, and aggregate a count over the timeframe, to check how data was backfilled. You can compare timestamps in the message of records, use DPL to extract the timestamp from messages, and calculate and graph a delta to the ingest time, like now(), called during log processing in OpenPipeline, and set to a field like delta. The higher that delta, the more data of that type and source was stored with that delta between creation and ingest times. For example, a log with a message timestamp of 9 AM and a timestamp field or time of ingest of 11 AM was buffered and sent 2 hours late, but stored at the time of ingest, not backfilled to the time of creation. Consequently, a gap exists where teams would expect to find their data, and they may have to shift their query timeframe by the delta to see that data. When we calculate delta, the time of ingest or calling now() in a processing rule, is always expected to be greater than the time of a supported timestamp format extracted from content.

fetch logs, bucket: {"astroshop_log_bucket"}, timeframe: "2026-09-01 08:00-6/2026-09-01 14:00-6"
| parse content, """LD TIMESTAMP("yyyy-MM-dd HH:mm:ss.S", tz="UTC"):timestamp.content"""
| fieldsAdd delta = (timestamp - timestamp.content)/power(10,6) // output is ns, so you convert to ms
| makeTimeSeries {delay_ms=avg(delta)}, by: {host.name, log.source}

Notebook query showing example of how delta calculation on demand works

Some data, like logs, may inevitably be lost. You can restore ingest of those lost logs in a custom bucket and pipeline, especially by keeping the timestamps to the time of ingest when storing logs exceeding the log age limit after a gap, and fetching those logs by the timeframe overlapping ingestion. To reduce querying consumption from the larger data sizes correlated with larger timeframes, you recommend calculating delta in the Processing stage and visualizing the metric as a graph after the Metric extraction stage of OpenPipeline.

Troubleshooting steps

You can ingest any old data that exists on your filesystems or cloud platforms by storing the real timestamp in a non-timestamp key, like timestamp.creation, and then sending that data to Dynatrace. You do not need to follow our troubleshooting steps exactly to resolve your problem. You can adapt some steps to your needs, but the order of our steps should be followed:

Modify at the source
Update OpenPipeline configuration
Modify any source configuration for timestamp extraction
Enable ingest of the modified source.

Instead of new custom buckets and pipelines, you can also add our delta processors to your existing pipelines or via a base pipeline to your pipeline groups, and filter data with higher deltas by the processed and set delta custom record attribute. Query consumption can be expensive for wide timeframes and deep, greedy scans of tables, but you can find the data you're seeking by adhering to DQL best practices with these steps, which are not strictly limited to restoring missing log data with OneAgent log module:

Move or copy the missed old log data sources to a different path so it's not tracked with any existing log monitoring rules. We will bypass our OneAgent log module's 15 minute ingestion limit on newly added log sources with supported timestamp formats in step 6, by setting the Timestamp search limit to 0 for the modified log path. Each file in the modified path should not exceed 10 MiB for complete ingestion by our OneAgent log module. When selecting a different path for old log data source, you should keep in mind our OneAgent log module's pre-defined security rules.
Add a custom bucket in Settings > Storage management.
Add a custom pipeline and add processors.

Processing stage > Add a DQL processor for delta calculation. Check the Conversion Patterns in DPL Time and Date to build the patterns for timestamps in your data messages, like "yyyy-MM-dd HH:mm:ss.S". Please also note that you must set the time zone, like tz="BST", or OpenPipeline will extract as UTC from the message. The provided example includes a generic DPL expression, but you should review your log sources, and add as many delta processors with matching timestamp and DPL expression as needed, checking if isNull(delta) before execution of each one.
```
parse content, """LD TIMESTAMP("yyyy-MM-dd HH:mm:ss.S", tz="CET"):timestamp.content"""
| fieldsAdd delta = (timestamp - timestamp.content)/power(10,6) // output is ns, so we convert to m
```
1. Optional: You can alternatively subtract now() to represent time of processing, whereas the timestamp field is the time when our OneAgent log module extracted the entries, because step 6 configures our log modules to ignore old supported timestamps in content.
  fieldsAdd delta = (now() - timestamp.content)/power(10,6) // output is ns, so we convert to ms
Metric extraction stage: Add a value metric for delta.
1. Add properties like dt.source_entity, host.name, and log.source to this metric.
Storage stage: Add your store processor to save your logs to your custom bucket.
Optional: You can also add copies of processing, event and metric extraction processors from other pipelines but you may want to append the copied metric keys with a text fragment, like .missed so they're separated, and add property deltato your metric and event processors, so teams can calculate the time of creation by subtracting delta from the time of ingest, which is the timestamp used to ingest your missing, old records from their sources. If you added log.source to the delta metric, that dimension should include our modified log path, so you can filter by the modified source to separate extracted timeseries from the rest in the same delta metric key as sources with low delta ingestion.

Optional: If you use OpenPipeline groups to apply processors to multiple member pipelines, add a new pipeline group, and your chosen base pipelines for your member pipeline.
Add a dynamic route matching your new log path as log.source == "/log/path/to/old/data/*.log" and set the target pipeline to your custom pipeline.
Add a timestamp/splitting rule with a Timestamp search limit of 0 bytes and a condition matching our modified log path.
Add our modified log path to a custom log source rule. Wildcards of any character except / and \ are supported in the form of an asterisk *.

If your environment scope log ingest rule [Built-in] Ingest logs from custom log sources is disabled, either enable it or add a log ingest rule for your log path of the custom log source configuration.

Query your metric timeseries avg(your.deltas.metric.key), by: {host.name, log.source} to know the approximate timeframe to fetch your late logs. Pulling timeseries for larger timeframes has a query consumption that can be many orders of magnitude lower than fetch logs for the same timeframe, especially for a user who has access to a very large number of records.
Add fetch parameter bucket like fetch logs, bucket: {"my.custom.bucket"} to significantly lower query consumption.
Avoid using fetch parameter scanLimitGBytes set at especially large values or -1, which incur unnecessary expense when you can instead, both narrow your timeframe and fetch logs only from select buckets.

Resolution

Once you complete these steps, you should find your data for the time of ingest, with any related events and metrics. With extraction and propagation of the custom log attribute delta, your app teams will know when and if any data is missing by the delta log value metric, and when the cause of an alert originated on a monitored entity by the delta event property.

What's next

If you experience blockers completing these steps and finding your missing log data from the modified source in Dynatrace, please don't hesitate to review our other log monitoring resources, and open a case with our support team with the steps completed, results and a link to the monitored host or files of the client-side log forwarding to API configuration, so we can start our investigation.