https://community.dynatrace.com/t5/Troubleshooting/Troubleshooting-Log-volume-spiked-suddenly-where-is-it-coming/ta-p/303794 <P><LI-TOC indent="15" liststyle="disc" maxheadinglevel="2"></LI-TOC></P> <DIV class="lia-message-template-content-zone"> <P>&nbsp;</P> <H2>Summary</H2> <P><SPAN>Use Log ingest overview, Top log producers, and OpenPipeline self-monitoring metrics to identify which source caused an unexpected volume increase, inspect its records, correct the ingest rule scope, and set up alerting to catch future spikes.</SPAN></P> <P>&nbsp;</P> <H2>Problem</H2> <P>Use this article when log consumption increased unexpectedly, or when one source appears to be dominating ingested volume without an obvious explanation.</P> <P>This workflow also covers the reverse scenario&nbsp; an unexpected volume <STRONG>decrease</STRONG>&nbsp; since both start with the same investigation path.</P> <P>&nbsp;</P> <H2>Troubleshooting steps</H2> <H3>&nbsp;</H3> <H3>Step 1: Establish the affected timeframe</H3> <P>Before opening any dashboard, record:</P> <UL> <LI>When the change started (and whether it is still ongoing)</LI> <LI>The timezone you will use consistently throughout the investigation</LI> <LI>Whether the change is continuous or limited to a specific window</LI> <LI>Any application deployments, logging configuration changes, or ingest-rule changes during that period</LI> </UL> <P>Without a clean timeframe, volume comparisons will be misleading.</P> <HR /> <H3>Step 2: Identify the top producer</H3> <OL> <LI>Open <STRONG>Dashboards</STRONG> → <STRONG>Log ingest overview</STRONG>.</LI> <LI>Set the timeframe to cover the spike window.</LI> <LI>Review the <STRONG>Top log producers</STRONG> panel.</LI> <LI>Identify which source, host, host group, Kubernetes cluster, or workload is associated with the volume change.</LI> <LI>Use the <STRONG>Actions menu</STRONG> (⋯) on the relevant row to open that source directly in the <STRONG>Logs</STRONG> app.</LI> </OL> <BLOCKQUOTE> <P><STRONG>Note:</STRONG> The <STRONG>Log ingest overview</STRONG> dashboard will be upgraded with SFM event tiles from Dynatrace version 1.342+. Until then, also import the <STRONG>Log module self-monitoring</STRONG> dashboard for the full component-health view — see <A href="https://docs.dynatrace.com/docs/analyze-explore-automate/logs/lma-log-ingestion/lma-log-ingestion-via-oa/lma-log-agent-sfm" target="_blank" rel="noopener noreferrer">Monitor log source health with SFM events</A> for instructions.&nbsp;</P> </BLOCKQUOTE> <span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="noel_david_0-1787899314909.png"><img src="https://community.dynatrace.com/t5/image/serverpage/image-id/34141i66A6D80C445835CB/image-size/large?v=v2&amp;px=999" alt="noel_david_0-1787899314909.png" title="noel_david_0-1787899314909.png" /></span> <P>&nbsp;</P> <HR /> <H3>Step 3: Inspect the records from the top producer</H3> <P>Open the identified source in the <STRONG>Logs</STRONG> app and review the records from the spike window.</P> <P>Determine whether the increase is explained by:</P> <UL> <LI>Debug or trace-level records that should not be in a production ingest rule</LI> <LI>Repetitive application errors generating high-frequency identical records</LI> <LI>A new deployment or application that was not previously covered by an ingest rule</LI> <LI>A change in application logging verbosity or destination</LI> <LI>A recent ingest-rule or matcher change that expanded scope</LI> </UL> <P>Do not infer the cause from volume alone. Confirm it from the actual records.</P> <HR /> <H3>Step 4: Check OpenPipeline self-monitoring metrics</H3> <P>If you want to verify the volume change at the pipeline level rather than just the source level, use the OpenPipeline self-monitoring metrics. These count records at each stage of the pipeline — from ingestion to storage.</P> <TABLE> <THEAD> <TR> <TH class="lia-align-center"><FONT size="2">Metric</FONT></TH> <TH class="lia-align-center"><FONT size="2">What it shows</FONT></TH> </TR> </THEAD> <TBODY> <TR> <TD><CODE>dt.sfm.openpipeline.ingest_sources_in.records</CODE></TD> <TD>Records entering at the ingest source, per data type and source path</TD> </TR> <TR> <TD><CODE>dt.sfm.openpipeline.routing.records</CODE></TD> <TD>Records processed through routing rules, per route and target pipeline</TD> </TR> <TR> <TD><CODE>dt.sfm.openpipeline.pipelines_out.records</CODE></TD> <TD>Records leaving a pipeline after processing, per data type and storage bucket</TD> </TR> <TR> <TD><CODE>dt.sfm.openpipeline.not_stored.records</CODE></TD> <TD>Records that were dropped, not persisted, or invalid — check<SPAN>&nbsp;</SPAN><CODE>pipeline_id</CODE><SPAN>&nbsp;</SPAN>to determine whether it happened at ingest or within a pipeline</TD> </TR> </TBODY> </TABLE> <P>The <STRONG>OpenPipeline usage overview</STRONG> ready-made dashboard provides a preconfigured view of these metrics. Use it to identify where in the pipeline the volume change entered or dropped out.</P> <HR /> <H3>Step 5: Review the ingest rule that covers the top producer</H3> <P>Open:</P> <DIV class="dutmcw0"> <DIV class="dutmcw1"> <DIV class="_1bzvwaf5q-3-9-0 _1bzvwaf6e-3-9-0 _1bzvwaf2-3-9-0 _1bzvwaf79-3-9-0" data-dt-component="Grid"> <DIV class="strato-button-label _10kgnsa9-3-9-0 _10kgnsab-3-9-0">&nbsp;</DIV> </DIV> </DIV> <PRE class="dutmcw2"><CODE>Settings app &gt; Collect and capture &gt; Log monitoring &gt; Configure log module &gt; Sources </CODE></PRE> </DIV> <P>Locate the source and review its active and inherited rules.</P> <P>Check whether:</P> <UL> <LI>A matcher intended for one scope also matches an unintended source or environment</LI> <LI>Current source attributes differ from the attributes that existed when the rule was created</LI> <LI>A recent matcher change expanded coverage beyond the intended scope</LI> <LI>The rule applies at the correct level (environment, host group, host)</LI> </UL> <P>Correct only the specific rule or matcher that caused the unintended coverage. Do not broaden exclusions without confirming the intended policy first.</P> <HR /> <H3>Step 6: Verify the correction</H3> <P>After adjusting the rule:</P> <OL> <LI>Return to <STRONG>Log ingest overview</STRONG>.</LI> <LI>Set the timeframe to the period after the change.</LI> <LI>Review the <STRONG>Top log producers</STRONG> panel.</LI> <LI>Confirm that intended logs remain available and searchable.</LI> <LI>Confirm that the high-volume unintended records are no longer dominating the producer list.</LI> </OL> <HR /> <H3>Step 7: Set up alerting to catch future spikes</H3> <P>Once the current issue is resolved, create a log metric for the affected source and configure an anomaly detector for future volume thresholds.</P> <P>To do this, use <STRONG>Settings &gt; Log monitoring &gt; Log metrics</STRONG>&nbsp;for Managed and for Saas <SPAN>Metrics are created in&nbsp;</SPAN><STRONG>OpenPipeline</STRONG><SPAN>&nbsp;(via the OpenPipeline configuration or&nbsp;</SPAN><STRONG>Settings &gt; Log Monitoring &gt; Log metrics</STRONG><SPAN>&nbsp;</SPAN><BR />to define a metric scoped to the relevant source, then configure an anomaly detection rule or Davis anomaly detector against that metric.</P> <P>You can also use&nbsp;<CODE>dt.log.status_per_entity_count</CODE> is a built-in, pre-aggregated Dynatrace log metric that counts ingested log records per entity and per log status<SPAN> (e.g., </SPAN><CODE>ERROR</CODE><SPAN>, </SPAN><CODE>WARN</CODE><SPAN>, </SPAN><CODE>INFO</CODE><SPAN>). </SPAN></P> <P>For context on how log-based metrics behave, see <A href="https://community.dynatrace.com/t5/Troubleshooting/Dynatrace-Log-Monitoring-Metric-Shows-No-Data/ta-p/290822" target="_blank" rel="noopener noreferrer">Dynatrace Log Monitoring: Metric Shows No Data</A>.</P> <HR /> <H2>For a volume decrease instead</H2> <P>If the scenario is a volume <STRONG>drop</STRONG> rather than a spike, follow the same path to Step 2 to identify the affected scope, then:</P> <OL> <LI>Check <STRONG>Ingest components health</STRONG> on the <STRONG>Log ingest overview</STRONG> dashboard for any unhealthy component.</LI> <LI>If a component is unhealthy → <A href="https://community.dynatrace.com/t5/Troubleshooting/My-logs-went-silent-where-do-I-start/ta-p/303793" target="_blank" rel="noopener noreferrer">My logs went silent&nbsp; where do I start?</A></LI> <LI>Confirm the source still writes new records at the original system.</LI> <LI>Check source coverage in the Sources tab → <A href="https://community.dynatrace.com/t5/tkb/workflowpage/tkb-id/troubleshooting/article-id/1187?prePageCrumb=TkbDashboardPage" target="_blank" rel="noopener noreferrer">What is Dynatrace actually capturing right now?</A></LI> <LI>Check for delay or rate limiting: <UL> <LI><A href="https://community.dynatrace.com/t5/Troubleshooting/Troubleshooting-log-ingest-delay-in-Log-module/ta-p/300190" target="_blank" rel="noopener noreferrer">Troubleshooting Log Ingest Delay in Log Module</A></LI> <LI><A href="https://community.dynatrace.com/t5/Troubleshooting/Dynatrace-does-not-ingest-logs-HTTP-429/ta-p/230512" target="_blank" rel="noopener noreferrer">Dynatrace does not ingest logs: HTTP 429</A></LI> </UL> </LI> </OL> <P>&nbsp;</P> <H2>What's next</H2> <P><EM><STRONG>Opening a support case with below details</STRONG></EM><EM><BR /></EM></P> <UL> <LI>Approximate start time and timezone of the volume change</LI> <LI>Affected source, host group, cluster, workload, or ingestion method</LI> <LI>Screenshot or export from <STRONG>Top log producers</STRONG> showing the spike window</LI> <LI>Screenshot or export from <STRONG>Ingest components health</STRONG> (if a decrease was also observed)</LI> <LI>OpenPipeline self-monitoring metric values for the affected pipeline and timeframe</LI> <LI>Sanitized sample records from the affected source during the spike period</LI> <LI>Applicable active and inherited ingest rules with matchers and scopes</LI> <LI>Recent logging, application, deployment, or rule changes</LI> <LI>Confirmation that the original source still produces records</LI> <LI>Applicable OneAgent, Dynatrace Operator, or collector version</LI> </UL> <H2><BR />Related reading</H2> <UL> <LI><A href="https://docs.dynatrace.com/docs/platform/openpipeline/reference/self-monitoring-metrics" target="_blank" rel="noopener noreferrer">OpenPipeline self-monitoring metrics</A></LI> <LI><A href="https://docs.dynatrace.com/docs/analyze-explore-automate/logs/lma-log-ingestion/lma-log-ingestion-via-oa/lma-log-agent-sfm" target="_blank" rel="noopener noreferrer">Monitor log source health with SFM events</A></LI> <LI><A href="https://community.dynatrace.com/t5/Troubleshooting/My-logs-went-silent-where-do-I-start/ta-p/303793" target="_blank" rel="noopener noreferrer">My logs went silent — where do I start?</A></LI> <LI><A href="https://community.dynatrace.com/t5/tkb/workflowpage/tkb-id/troubleshooting/article-id/1187?prePageCrumb=TkbDashboardPage" target="_blank" rel="noopener noreferrer">What is Dynatrace actually capturing right now?</A></LI> <LI><A href="https://www.dynatrace.com/news/blog/modern-log-management-resilience-and-self-service-upgrades-for-log-ingestion/" target="_blank" rel="noopener noreferrer">Improve log ingest health with smarter log management</A></LI> <LI><A href="https://community.dynatrace.com/t5/Troubleshooting/Logs-Troubleshooting-Map/ta-p/302260" target="_blank" rel="noopener noreferrer">Dynatrace Logs Troubleshooting Map</A></LI> </UL> </DIV> Fri, 04 Sep 2026 06:48:49 GMT noel_david 2026-09-04T06:48:49Z https://community.dynatrace.com/t5/Troubleshooting/Troubleshooting-Log-volume-spiked-suddenly-where-is-it-coming/ta-p/303794 <P><LI-TOC indent="15" liststyle="disc" maxheadinglevel="2"></LI-TOC></P> <DIV class="lia-message-template-content-zone"> <P>*<EM>Use a table of contents for longer articles.&nbsp;</EM></P> <H2>Summary</H2> <P><EM>*This is a mandatory section.</EM></P> <P><EM>Short description of which part of the Dynatrace platform the article refers to and what kind of problem it will help resolve/ task it will describe.</EM></P> <P>&nbsp;</P> <H2>Problem</H2> <P><EM>*This is a mandatory section.</EM></P> <P><EM>Provide a precise description of the problem/ task to be described. Use anonymized screenshots, and include text for&nbsp;important messages, errors, or information that will help the customer find this article when searching.</EM></P> <P>&nbsp;</P> <H2>Troubleshooting steps</H2> <P><EM>*This section can be omitted as necessary.</EM></P> <P><EM>Explain what troubleshooting steps should be taken to ensure the problem matches this article.</EM></P> <P>&nbsp;</P> <H2>Resolution</H2> <P><EM>*This section can be omitted for articles that guide customers on case creation.</EM></P> <P><EM>Explain the solution or all possible solutions resulting from the troubleshooting steps.<BR /><BR /></EM></P> <H2>What's next</H2> <P><EM>*This is a mandatory section. Customers need a way to respond or follow up if they have questions.&nbsp;</EM></P> <P><EM>Explain what to do if the article didn't help.&nbsp;</EM><EM>Note that there are multiple options available, including:</EM></P> <OL> <LI><EM><STRONG>Opening a chat or support case&nbsp;</STRONG>- be as specific as possible about the information the customer should include in the case.&nbsp;</EM><EM>Use the following formula and fill or change the bullet points:<BR /><BR /></EM> If this article didn't help, <STRONG>create a chat or open a support case</STRONG> <UL> <LI>Mention that you reviewed this article.</LI> <LI>Include the following details: <UL> <LI>Link to XYZ</LI> <LI>Screenshot of XYZ</LI> <LI>Information about XYZ</LI> </UL> </LI> </UL> </LI> <LI><EM><STRONG>Suggesting Product Idea</STRONG> - encourage the customer to suggest/ vote for a Product Idea explaining their business use case.<BR /></EM> <DIV>Please submit a Product Idea with detailed information about your request and use case in the Product Ideas forum. Our Product Management team regularly reviews these suggestions:</DIV> <A href="https://community.dynatrace.com/t5/Dynatrace-product-ideas/idb-p/DynatraceProductIdeas" target="_blank" rel="noopener">https://community.dynatrace.com/t5/Dynatrace-product-ideas/idb-p/DynatraceProductIdeas</A></LI> <LI><STRONG><EM>Explain this will change in the future </EM></STRONG><EM>- explain that this behavior will change in a future release. (No product idea / support case needed)</EM></LI> <LI><STRONG><EM>Alternatives -</EM></STRONG><EM>&nbsp;any other actions or links to other articles that could move the customer forward.</EM></LI> </OL> <H2><BR />Related reading</H2> <P><EM>*Link this article to other relevant configuration or troubleshooting guides, including any applicable troubleshooting maps.<BR />:open_book:&nbsp;&nbsp;</EM><A href="https://community.dynatrace.com/t5/Troubleshooting/" target="_blank" rel="noopener">XXX Troubleshooting Map</A></P> <P>:open_book:&nbsp; Link 1</P> <P>:open_book:&nbsp; Link 2</P> <P>:open_book:&nbsp; Link 3</P> <P>:open_book:&nbsp; Link 4</P> </DIV> Fri, 04 Sep 2026 06:48:49 GMT https://community.dynatrace.com/t5/Troubleshooting/Troubleshooting-Log-volume-spiked-suddenly-where-is-it-coming/ta-p/303794 noel_david 2026-09-04T06:48:49Z