https://community.dynatrace.com/t5/Troubleshooting/How-to-troubleshoot-ingestion-of-missed-log-data-after-the-Log/ta-p/304040 <P><LI-TOC indent="15" liststyle="disc" maxheadinglevel="2"></LI-TOC></P> <DIV class="lia-message-template-content-zone"> <H2>Summary</H2> <P>Data of <A href="https://docs.dynatrace.com/docs/shortlink/openpipeline-api-ingestion-reference" target="_blank" rel="noopener">Ingest sources</A> can be delayed and lost before reaching our platform for a myriad of reasons: system failures, delayed configuration changes to match application changes, major network disruptions, <SPAN>injudicious&nbsp;</SPAN>revisions to sensitive configurations, binary or invalid content forcing discards by the sender, receiver or OpenPipeline, etc. Some data sources are mission critical, and require storage in our platform so teams can review that data, monitor metrics and receive alerts. When a gap in data occurs, it can be difficult for app teams to untangle the delayed data from truly missing data without going to the source or an active 3rd party vendor's solution, and comparing to our platform.</P> <P>&nbsp;</P> <H2>Problem</H2> <P>Some data can be backfilled but not all data is buffered client-side or backfilled equally. For example, OneAgent log module saves up to 10 Mebibytes of logs before it has to either send or drop that data. Once dropped, that data is not automatically recollected from the source. For generic ingest to our API endpoints, different cloud platforms measure and handle backpressure of logs and other datatypes differently, often with&nbsp;exponential backoff, maximum retry and rate limits. Grail has maximum ages for accepting new records with dated timestamps, which is the <A href="https://docs.dynatrace.com/docs/shortlink/lma-limits#log-ingestion-limits" target="_blank" rel="noopener">Log age</A>&nbsp;ingestion limit in the Logs table.<BR /><BR />In Dynatrace, teams can check the <STRONG>Total ingested records per configuration</STRONG> tile in the tenant's Ready-made dashboard <A href="https://docs.dynatrace.com/docs/shortlink/ready-made-dashboards#dynatrace-openpipeline-openpipeline-usage-overview" target="_blank" rel="noopener"><STRONG>OpenPipeline usage overview</STRONG></A>. Since data can be backfilled up to age limit, you can expect a spike in ingest on recovery to a monitoring gap of a major data source. You can compare the area of a spike to previous days and days of the previous weeks to gain some insight into the impact of data loss, however, this is not a practical approach to understand the precise impact of data delay and loss, especially in detail.</P> <P>&nbsp;</P> <P>A better approach is to query the affected table or bucket, and <A href="https://docs.dynatrace.com/docs/shortlink/dql-use-queries#aggregations" target="_blank" rel="noopener">aggregate a count</A> over the timeframe, to check how data was backfilled. You can compare timestamps in the message of records, use DPL to extract the timestamp from messages, and calculate and graph a delta to the ingest time, like <A href="https://docs.dynatrace.com/docs/shortlink/time-functions#now" target="_blank" rel="noopener">now()</A>, called during log processing in OpenPipeline, and set to a field like delta. The higher that <CODE>delta</CODE>, the more data of that type and source was stored with that <CODE>delta</CODE> between creation and ingest times. For example, a log with a message timestamp of 9 AM and a timestamp field or time of ingest of 11 AM was buffered and sent 2 hours late, but stored at the time of ingest, not backfilled to the time of creation. Consequently, a gap exists where teams would expect to find their data, and they may have to shift their query timeframe by the <CODE>delta</CODE> to see that data. When we calculate&nbsp;<CODE>delta</CODE>, the time of ingest or calling&nbsp;<A href="https://docs.dynatrace.com/docs/shortlink/time-functions#now" target="_blank" rel="noopener">now()</A>&nbsp;in a processing rule, is always expected to be greater than the time of a <A href="https://docs.dynatrace.com/docs/shortlink/lma-supported-timestamp-formats" target="_blank" rel="noopener">supported timestamp format</A>&nbsp;extracted from content.</P> <PRE>fetch logs, bucket: {"astroshop_log_bucket"}, timeframe: "2026-09-01 08:00-6/2026-09-01 14:00-6"<BR />| parse content, """LD TIMESTAMP("yyyy-MM-dd HH:mm:ss.S", tz="UTC"):timestamp.content"""<BR />| fieldsAdd delta = (timestamp - timestamp.content)/power(10,6) // output is ns, so you convert to ms<BR />| makeTimeSeries {delay_ms=avg(delta)}, by: {host.name, log.source}</PRE> <span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Notebook query showing example of how delta calculation on demand works"><img src="https://community.dynatrace.com/t5/image/serverpage/image-id/34211i71D50E4858187FCA/image-size/large?v=v2&amp;px=999" alt="Notebook query showing example of how delta calculation on demand works" title="jgrant_0-1788481573718.png" /></span> <P>&nbsp;</P> <P>Some data, like logs, may inevitably be lost. You can restore ingest of those lost logs in a custom bucket and pipeline, especially by keeping the timestamps to the time of ingest when storing logs exceeding the log age limit after a gap, and fetching those logs by the timeframe overlapping ingestion. To reduce querying consumption from the larger data sizes correlated with larger timeframes, you recommend calculating delta in the Processing stage and visualizing the metric as a graph after the Metric extraction stage of OpenPipeline.</P> <P>&nbsp;</P> <H2>Troubleshooting steps</H2> <P>You can ingest any old data that exists on your filesystems or cloud platforms by storing the real timestamp in a non-timestamp key, like&nbsp;<CODE>timestamp.creation</CODE>, and then sending that data to Dynatrace. You do not need to follow our troubleshooting steps exactly to resolve your problem. You can adapt some steps to your needs, but the order of our steps should be followed:</P> <OL> <LI>Modify at the source</LI> <LI>Update OpenPipeline configuration</LI> <LI>Modify any source configuration for timestamp extraction</LI> <LI>Enable ingest of the modified source.</LI> </OL> <P>&nbsp;</P> <P>Instead of new custom buckets and pipelines, you can also add our delta processors to your existing pipelines or via a base pipeline to your <A href="https://docs.dynatrace.com/docs/shortlink/openpipeline-pipeline-groups" target="_blank" rel="noopener">pipeline groups</A>, and filter data with higher deltas by the processed and set <CODE>delta</CODE> custom record attribute.&nbsp;<A href="https://docs.dynatrace.com/docs/shortlink/dps-log-analytics-query" target="_blank" rel="noopener">Query consumption</A> can be expensive for wide timeframes and deep, greedy scans of tables, but you can find the data you're seeking by adhering to DQL&nbsp;<A href="https://docs.dynatrace.com/docs/shortlink/dql-best-practices" target="_blank" rel="noopener">best practices</A>&nbsp;with these steps, which are not strictly limited to restoring missing log data with OneAgent log module:<EM><BR /></EM></P> <OL> <LI>Move or copy the missed old log data sources to a different path so it's not tracked with any existing <A href="https://docs.dynatrace.com/docs/shortlink/lma-ingestion-oneagent#oneagent-log-configuration-flow" target="_blank" rel="noopener">log monitoring rules</A>. We will bypass our OneAgent log module's 15 minute <A href="https://docs.dynatrace.com/docs/shortlink/lma-ingestion-oneagent#recent-past-logs-ingestion-when-enabling-the-log-module" target="_blank" rel="noopener">ingestion limit</A> on newly added log sources with <A href="https://docs.dynatrace.com/docs/shortlink/lma-supported-timestamp-formats" target="_blank">supported timestamp formats</A> in step 6, by setting the <STRONG>Timestamp search limit</STRONG> to <STRONG>0</STRONG> for the modified log path. Each file in the modified path should not exceed 10 MiB for complete ingestion by our OneAgent log module. When selecting a different path for old log data source, you should keep in mind our OneAgent log module's <A href="https://docs.dynatrace.com/docs/shortlink/lma-security-rules#predefined-security-rules" target="_blank" rel="noopener">pre-defined security rules</A>.</LI> <LI>Add a <A href="https://docs.dynatrace.com/docs/shortlink/lma-bucket-assignment#custom-bucket" target="_blank" rel="noopener">custom bucket</A> in Settings &gt; <A href="https://docs.dynatrace.com/docs/shortlink/grail-data-model" target="_blank" rel="noopener">Storage management</A>.</LI> <LI>Add a <A href="https://docs.dynatrace.com/docs/shortlink/lma-openpipeline#add-custom-pipeline" target="_self">custom pipeline</A> and <A href="https://docs.dynatrace.com/docs/shortlink/openpipeline-configure" target="_blank" rel="noopener">add processors</A>.</LI> <OL> <LI><A href="https://docs.dynatrace.com/docs/shortlink/lma-openpipeline#stages" target="_blank" rel="noopener">Processing stage</A> &gt; Add a DQL processor for <CODE>delta</CODE>&nbsp;calculation. Check the <A href="https://docs.dynatrace.com/docs/shortlink/dpl-time-date#conversion-patterns" target="_blank" rel="noopener">Conversion Patterns</A>&nbsp;in&nbsp;<A href="https://docs.dynatrace.com/docs/shortlink/dpl-time-date" target="_blank" rel="noopener">DPL Time and Date</A>&nbsp;to build the patterns for timestamps in your data messages, like&nbsp;<CODE>"yyyy-MM-dd HH:mm:ss.S"</CODE>. Please also note that you must set the time zone, like&nbsp;<A href="https://www.timeanddate.com/time/zones/" target="_blank" rel="noopener">tz</A>="BST", or OpenPipeline will extract as UTC from the message. The provided example includes a generic DPL expression, but you should review your log sources, and add as many delta processors with matching timestamp and DPL expression as needed, checking if&nbsp;<A href="https://docs.dynatrace.com/docs/shortlink/boolean-functions#isNull" target="_blank" rel="noopener">isNull</A>(delta) before execution of each one.<BR /> <PRE>parse content, """LD TIMESTAMP("yyyy-MM-dd HH:mm:ss.S", tz="CET"):timestamp.content"""<BR />| fieldsAdd delta = (timestamp - timestamp.content)/power(10,6) // output is ns, so we convert to m</PRE> <span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Add DQL processor with definition in member pipeline."><img src="https://community.dynatrace.com/t5/image/serverpage/image-id/34213iE9312813843CA427/image-size/large?v=v2&amp;px=999" alt="Add DQL processor with definition in member pipeline." title="jgrant_0-1788548523952.png" /></span> <OL> <LI><EM>Optional: You can alternatively subtract now() to represent time of processing, whereas the timestamp field is the time when our OneAgent log module extracted the entries, because step 6 configures our log modules to ignore old <A href="https://docs.dynatrace.com/docs/shortlink/lma-supported-timestamp-formats" target="_blank" rel="noopener">supported timestamps</A> in content.<BR /><CODE>fieldsAdd delta = (now() - timestamp.content)/power(10,6) // output is ns, so we convert to ms</CODE><BR /></EM></LI> </OL> </LI> <LI><A href="https://docs.dynatrace.com/docs/shortlink/openpipeline-stage-metric-extraction" target="_blank" rel="noopener">Metric extraction stage</A>: Add a <A href="https://docs.dynatrace.com/docs/shortlink/openpipeline-stage-metric-extraction#value-metric" target="_blank" rel="noopener">value metric</A> for <I><CODE>delta</CODE></I>. <OL> <LI>Add properties like <CODE>dt.source_entity</CODE>, <CODE>host.name</CODE>, and <CODE>log.source</CODE> to this metric.</LI> </OL> </LI> <LI><A href="https://docs.dynatrace.com/docs/shortlink/lma-bucket-assignment#assign" target="_blank" rel="noopener">Storage stage</A>: Add your store processor to save your logs to your custom bucket.</LI> <LI><I>Optional: You can also add copies of processing, <A href="https://docs.dynatrace.com/docs/shortlink/openpipeline-stage-davis" target="_blank" rel="noopener">event</A> and metric extraction processors from other pipelines but you may want to append the copied metric keys with a text fragment, like <CODE>.missed</CODE> so they're separated, and add property <CODE>delta</CODE>to your metric and event processors, so teams can calculate the time of creation by subtracting delta from the time of ingest, which is the timestamp used to ingest your missing, old records from their sources. If you added <CODE>log.source</CODE> to the <CODE>delta</CODE> metric, that dimension should include our modified log path, so you can filter by the modified source to separate extracted timeseries from the rest in the same <CODE>delta</CODE> metric key as sources with low <CODE>delta</CODE>&nbsp;ingestion.</I></LI> </OL> <LI><I>Optional: If you use <A href="https://docs.dynatrace.com/docs/shortlink/openpipeline-pipeline-groups" target="_blank" rel="noopener">OpenPipeline groups</A> to apply processors to multiple member pipelines, add a new pipeline group, and your chosen base pipelines for your member pipeline.</I></LI> <LI>Add a <A href="https://docs.dynatrace.com/docs/shortlink/openpipeline-route" target="_blank" rel="noopener">dynamic route</A> matching your new log path as <CODE>log.source == "/log/path/to/old/data/*.log"</CODE> and set the target pipeline to your custom pipeline.</LI> <LI>Add a <A href="https://docs.dynatrace.com/docs/shortlink/lma-timestamp-configuration" target="_blank" rel="noopener">timestamp/splitting rule</A> with a <STRONG>Timestamp search limit</STRONG> of <STRONG>0</STRONG> bytes and a condition matching our modified log path.</LI> <LI>Add our modified log path to a <A href="https://docs.dynatrace.com/docs/shortlink/lma-custom-log-source#mainclscuipage" target="_blank" rel="noopener">custom log source rule</A>. Wildcards of any character except&nbsp;<CODE>/</CODE> and&nbsp;<CODE>\</CODE> are supported in the form of an asterisk <CODE>*</CODE>.&nbsp;</LI> <OL> <LI>If your <A href="https://docs.dynatrace.com/docs/shortlink/settings#scope-and-hierarchy-of-settings" target="_blank" rel="noopener">environment scope</A> <A href="https://docs.dynatrace.com/docs/shortlink/lma-log-ingest-rules" target="_blank" rel="noopener">log ingest rule</A> <STRONG>[Built-in] Ingest logs from custom log sources</STRONG> is disabled, either enable it or add a log ingest rule for your log path of the custom log source configuration.</LI> </OL> <LI><A href="https://docs.dynatrace.com/docs/shortlink/metrics-on-grail-examples" target="_blank" rel="noopener">Query your metric</A> <CODE>timeseries avg(your.deltas.metric.key), by: {host.name, log.source}</CODE> to know the approximate timeframe to fetch your late logs. Pulling timeseries for larger timeframes has a query consumption that can be many orders of magnitude lower than fetch logs for the same timeframe, especially for a user who has access to a very large number of records.</LI> <LI>Add <A href="https://docs.dynatrace.com/docs/shortlink/data-source-commands#fetch" target="_blank" rel="noopener">fetch parameter</A>&nbsp;bucket like&nbsp;<CODE>fetch logs, bucket: {"my.custom.bucket"}</CODE> to significantly lower query consumption.</LI> <LI>Avoid using fetch parameter <CODE>scanLimitGBytes</CODE> set at especially large values or <CODE>-1</CODE>, which incur unnecessary expense when you can instead, both narrow your timeframe and fetch logs only from select buckets.</LI> </OL> <P>&nbsp;</P> <H2>Resolution</H2> <P>Once you complete these steps, you should find your data for the time of ingest, with any related events and metrics. With extraction and propagation of the custom log attribute delta, your app teams will know when and if any data is missing by the delta log value metric, and when the cause of an alert originated on a monitored entity by the delta event property.<EM><BR /></EM></P> <P>&nbsp;</P> <H2>What's next</H2> <P>If you experience blockers completing these steps and finding your missing log data from the modified source in Dynatrace, please don't hesitate to review our other log monitoring resources, and open a case with our support team with the steps completed, results and a link to the monitored host or files of the client-side log forwarding to API configuration, so we can start our investigation.</P> <P>&nbsp;</P> <H3 id="toc-hId-1600907322" data-heading="More articles can be found on the [Logs Troubleshooting Map](https://community.dynatrace.com/t5/Troubleshooting/Logs-Troubleshooting-Map/ta-p/302260)">More articles can be found on the<SPAN>&nbsp;</SPAN><A class="external-link" href="https://community.dynatrace.com/t5/Troubleshooting/Logs-Troubleshooting-Map/ta-p/302260" target="_blank" rel="noopener" aria-label="https://community.dynatrace.com/t5/Troubleshooting/Logs-Troubleshooting-Map/ta-p/302260" data-tooltip-position="top">Logs Troubleshooting Map</A></H3> <P>We recommend reviewing&nbsp;<A id="link_6" class="page-link lia-link-navigation lia-custom-event" href="https://community.dynatrace.com/t5/Troubleshooting/Troubleshooting-missing-logs-in-Log-module/ta-p/300209" target="_blank" rel="noopener">Troubleshooting missing logs in Log module</A>&nbsp;for any monitoring gaps related to our log modules.</P> </DIV> Mon, 07 Sep 2026 06:23:14 GMT jgrant 2026-09-07T06:23:14Z https://community.dynatrace.com/t5/Troubleshooting/How-to-troubleshoot-ingestion-of-missed-log-data-after-the-Log/ta-p/304040 <P><LI-TOC indent="15" liststyle="disc" maxheadinglevel="2"></LI-TOC></P> <DIV class="lia-message-template-content-zone"> <P>*<EM>Use a table of contents for longer articles.&nbsp;</EM></P> <H2>Summary</H2> <P><EM>*This is a mandatory section.</EM></P> <P><EM>Short description of which part of the Dynatrace platform the article refers to and what kind of problem it will help resolve/ task it will describe.</EM></P> <P>&nbsp;</P> <H2>Problem</H2> <P><EM>*This is a mandatory section.</EM></P> <P><EM>Provide a precise description of the problem/ task to be described. Use anonymized screenshots, and include text for&nbsp;important messages, errors, or information that will help the customer find this article when searching.</EM></P> <P>&nbsp;</P> <H2>Troubleshooting steps</H2> <P><EM>*This section can be omitted as necessary.</EM></P> <P><EM>Explain what troubleshooting steps should be taken to ensure the problem matches this article.</EM></P> <P>&nbsp;</P> <H2>Resolution</H2> <P><EM>*This section can be omitted for articles that guide customers on case creation.</EM></P> <P><EM>Explain the solution or all possible solutions resulting from the troubleshooting steps.<BR /><BR /></EM></P> <H2>What's next</H2> <P><EM>*This is a mandatory section. Customers need a way to respond or follow up if they have questions.&nbsp;</EM></P> <P><EM>Explain what to do if the article didn't help.&nbsp;</EM><EM>Note that there are multiple options available, including:</EM></P> <OL> <LI><EM><STRONG>Opening a chat or support case&nbsp;</STRONG>- be as specific as possible about the information the customer should include in the case.&nbsp;</EM><EM>Use the following formula and fill or change the bullet points:<BR /><BR /></EM> If this article didn't help, <STRONG>create a chat or open a support case</STRONG> <UL> <LI>Mention that you reviewed this article.</LI> <LI>Include the following details: <UL> <LI>Link to XYZ</LI> <LI>Screenshot of XYZ</LI> <LI>Information about XYZ</LI> </UL> </LI> </UL> </LI> <LI><EM><STRONG>Suggesting Product Idea</STRONG> - encourage the customer to suggest/ vote for a Product Idea explaining their business use case.<BR /></EM> <DIV>Please submit a Product Idea with detailed information about your request and use case in the Product Ideas forum. Our Product Management team regularly reviews these suggestions:</DIV> <A href="https://community.dynatrace.com/t5/Dynatrace-product-ideas/idb-p/DynatraceProductIdeas" target="_blank" rel="noopener">https://community.dynatrace.com/t5/Dynatrace-product-ideas/idb-p/DynatraceProductIdeas</A></LI> <LI><STRONG><EM>Explain this will change in the future </EM></STRONG><EM>- explain that this behavior will change in a future release. (No product idea / support case needed)</EM></LI> <LI><STRONG><EM>Alternatives -</EM></STRONG><EM>&nbsp;any other actions or links to other articles that could move the customer forward.</EM></LI> </OL> <H2><BR />Related reading</H2> <P><EM>*Link this article to other relevant configuration or troubleshooting guides, including any applicable troubleshooting maps.<BR />:open_book:&nbsp;&nbsp;</EM><A href="https://community.dynatrace.com/t5/Troubleshooting/" target="_blank" rel="noopener">XXX Troubleshooting Map</A></P> <P>:open_book:&nbsp; Link 1</P> <P>:open_book:&nbsp; Link 2</P> <P>:open_book:&nbsp; Link 3</P> <P>:open_book:&nbsp; Link 4</P> </DIV> Mon, 07 Sep 2026 06:23:14 GMT https://community.dynatrace.com/t5/Troubleshooting/How-to-troubleshoot-ingestion-of-missed-log-data-after-the-Log/ta-p/304040 jgrant 2026-09-07T06:23:14Z