https://community.dynatrace.com/t5/Troubleshooting/My-logs-went-silent-where-do-I-start/ta-p/303793 <P><LI-TOC indent="15" liststyle="disc" maxheadinglevel="2"></LI-TOC></P> <DIV class="lia-message-template-content-zone"> <P>&nbsp;</P> <H2>Summary</H2> <P>&nbsp;A structured first-response workflow for when logs that previously worked suddenly stop. Covers component health, SFM events, source verification, ingest-rule coverage, and ActiveGate scope checks.</P> <H2>Problem</H2> <P>Use this article when logs from a source that previously worked have suddenly stopped arriving in Dynatrace.</P> <P>This is a first-response workflow. It does not cover logs that were never visible if the source has never appeared in Dynatrace, start with <A href="https://community.dynatrace.com/t5/tkb/workflowpage/tkb-id/troubleshooting/article-id/1187?prePageCrumb=TkbDashboardPage" target="_blank" rel="noopener noreferrer">Log source not detected or not ingested: Troubleshoot source discovery and ingest-rule coverage</A> instead.</P> <BLOCKQUOTE> <P><STRONG>Availability note:</STRONG> The Log module self-monitoring dashboard and SFM events are generally available at <STRONG>OneAgent version 1.339+</STRONG> and <STRONG>SaaS version 1.340+</STRONG>. For earlier versions, SFM events require opt-in via the Settings API (<CODE>builtin:logmonitoring.log-sfm-settings</CODE>). Log Monitoring Classic customers can also opt in. Verify availability for the affected environment before following this workflow.</P> </BLOCKQUOTE> <P>&nbsp;</P> <H2>Troubleshooting steps</H2> <P><STRONG>Check the source first.</STRONG></P> <OL> <LI>Open the original log file, container output, or upstream system.</LI> <LI>Confirm it contains records written after the issue started.</LI> <LI>Record the exact source timestamp and timezone of a recent record.</LI> <LI>Check whether any of the following changed recently: <UL> <LI>File path, file name, or log destination</LI> <LI>Application logging configuration</LI> <LI>File ownership, permissions, or mount/volume configuration</LI> <LI>Encoding, compression, or rotation behaviour</LI> <LI>Timestamp format or timezone</LI> <LI>Host, container identity, or Kubernetes workload</LI> </UL> </LI> </OL> <P>If the source itself has stopped writing records, the issue is in the application or logging framework investigate that first.</P> <P>&nbsp;</P> <HR /> <H3>Step 1: Open the Log module self-monitoring dashboard</H3> <P>The <STRONG>Log module self-monitoring</STRONG> dashboard is the fastest way to see whether Dynatrace has already detected the problem and raised an event.</P> <BLOCKQUOTE> <P><STRONG>Note:</STRONG> From Dynatrace version 1.342+, SFM event tiles will be built into the <STRONG>Log ingest overview</STRONG> dashboard. Until then, import the dashboard manually&nbsp; see <A href="https://docs.dynatrace.com/docs/analyze-explore-automate/logs/lma-log-ingestion/lma-log-ingestion-via-oa/lma-log-agent-sfm" target="_blank" rel="noopener noreferrer">Monitor log source health with SFM events</A> for the JSON import instructions.</P> </BLOCKQUOTE> <P>On the dashboard, look at:</P> <UL> <LI><STRONG>Currently active issues (24 hours)</STRONG>&nbsp; grouped by severity (<CODE>ERROR</CODE>, <CODE>WARN</CODE>, <CODE>INFO</CODE>) and event type</LI> <LI><STRONG>Active issues – host-based log modules</STRONG>&nbsp; start time, event type, affected file path, host, and process entity</LI> <LI><STRONG>Active issues – Kubernetes log modules</STRONG>&nbsp;affected container or Kubernetes entity</LI> </UL> <P>If an event is listed for the affected source or host, go to <STRONG>Step 2</STRONG> to act on it.</P> <P>If no event is listed, go to <STRONG>Step 4</STRONG>.</P> <HR /> <H3>Step 2: Read the SFM event and match it to the remediation below</H3> <P>SFM events are stored in Grail. To retrieve them directly:</P> <DIV class="dutmcw0"> <DIV class="dutmcw1"> <DIV class="_1bzvwaf5q-3-9-0 _1bzvwaf6e-3-9-0 _1bzvwaf2-3-9-0 _1bzvwaf79-3-9-0" data-dt-component="Grid"> <DIV class="strato-button-label _10kgnsa9-3-9-0 _10kgnsab-3-9-0">&nbsp;</DIV> </DIV> </DIV> <PRE class="dutmcw2"><CODE>fetch dt.system.events </CODE></PRE> </DIV> <P>Filter by the affected entity or log source as needed.</P> <P>Each event includes context: the affected log source path or host ID, and a condition type. Act only on the condition the event actually reports&nbsp; do not assume a common cause without reading it.</P> <H3>SFM event quick-reference</H3> <H4>Source status (<CODE>log_source.status</CODE>&nbsp;INFO)</H4> <P>Generated for every known log source. Use <CODE>log.source.file_status</CODE> and <CODE>log.source.ingest_status</CODE> to decide what to do.</P> <P>&nbsp;</P> <TABLE> <THEAD> <TR> <TH class="lia-align-center"><FONT size="2"><CODE>file_status</CODE><SPAN>&nbsp;</SPAN>value</FONT></TH> <TH class="lia-align-center"><FONT size="2">Action</FONT></TH> </TR> </THEAD> <TBODY> <TR> <TD><CODE>OK</CODE></TD> <TD>File is accessible check<SPAN>&nbsp;</SPAN><CODE>ingest_status</CODE><SPAN>&nbsp;</SPAN>and ingest rules</TD> </TR> <TR> <TD><CODE>Not Exist</CODE><SPAN>&nbsp;</SPAN>(custom source only)</TD> <TD>Check the file path pattern for typos; verify the Log module filesystem access; check<SPAN>&nbsp;</SPAN><CODE>dtuser</CODE><SPAN>&nbsp;</SPAN>(Linux) or OneAgent service account (Windows) permissions</TD> </TR> <TR> <TD><CODE>Binary</CODE></TD> <TD>Check for binary content in matched files; adjust the custom log source pattern or enable<SPAN>&nbsp;</SPAN><STRONG>Allow binary files</STRONG><SPAN>&nbsp;</SPAN>if binary content is intentional</TD> </TR> <TR> <TD><CODE>Sensitive Masking Timeout</CODE></TD> <TD>See<SPAN>&nbsp;</SPAN><CODE>ingest.masking_timeout</CODE><SPAN>&nbsp;</SPAN>below</TD> </TR> </TBODY> </TABLE> <P>If <CODE>ingest_status</CODE> is not as expected, review the active and inherited ingest rules:</P> <UL> <LI>Rules are evaluated top-to-bottom; the first match wins</LI> <LI>The effective rule list concatenates rules from all configuration scopes</LI> <LI>The log source <STRONG>matcher</STRONG> matches the log source <STRONG>name</STRONG>, not the file path (e.g. <CODE>/path/a.2026.log</CODE> does not match a source named <CODE>/path/a.#.log</CODE>)</LI> </UL> <H4>&nbsp;</H4> <H4>Timestamp events</H4> <P>&nbsp;</P> <TABLE> <THEAD> <TR> <TH class="lia-align-center"><FONT size="2">Event type</FONT></TH> <TH class="lia-align-center"><FONT size="2">Severity</FONT></TH> <TH class="lia-align-center"><FONT size="2">Action</FONT></TH> </TR> </THEAD> <TBODY> <TR> <TD><CODE>timestamp.no_pattern</CODE></TD> <TD>WARNING</TD> <TD>Configure a non-standard timestamp format in log monitoring settings, or rely on Log module timestamping; review boundary detection if multiline records are present</TD> </TR> <TR> <TD><CODE>timestamp.multiple_patterns</CODE></TD> <TD>INFO</TD> <TD>No action if expected; configure the timestamp format explicitly if the Log module misidentifies message content as a timestamp</TD> </TR> <TR> <TD><CODE>timestamp.invalid_timezone</CODE></TD> <TD>ERROR</TD> <TD>Correct the timezone configuration in log monitoring settings to match the timezone the source writes in</TD> </TR> </TBODY> </TABLE> <H4>&nbsp;</H4> <H4>Ingest pipeline events</H4> <P>&nbsp;</P> <TABLE> <THEAD> <TR> <TH class="lia-align-center"><FONT size="2">Event type</FONT></TH> <TH class="lia-align-center"><FONT size="2">Severity</FONT></TH> <TH class="lia-align-center"><FONT size="2">Action</FONT></TH> </TR> </THEAD> <TBODY> <TR> <TD><CODE>ingest.access_flag</CODE></TD> <TD>WARNING</TD> <TD>Run<SPAN>&nbsp;</SPAN><CODE>oneagentctl --set-app-log-content-access=true</CODE><SPAN>&nbsp;</SPAN>on the affected host; for a new deployment, enable the flag during installation</TD> </TR> <TR> <TD><CODE>ingest.masking_timeout</CODE></TD> <TD>ERROR</TD> <TD>Review masking rules for the source&nbsp; the most likely cause is an inefficient regular expression; ingestion resumes after correcting the configuration or restarting the Log module</TD> </TR> <TR> <TD><CODE>ingest.log_source_blocked</CODE></TD> <TD>WARNING</TD> <TD>Review OneAgent security rules and adjust them to permit access to the blocked path</TD> </TR> </TBODY> </TABLE> <H4>&nbsp;</H4> <H4>Data loss events</H4> <P>&nbsp;</P> <TABLE> <THEAD> <TR> <TH class="lia-align-center"><FONT size="2">Event type family</FONT></TH> <TH class="lia-align-center"><FONT size="2">Severity</FONT></TH> <TH class="lia-align-center"><FONT size="2">Action</FONT></TH> </TR> </THEAD> <TBODY> <TR> <TD><CODE>data_loss.network</CODE></TD> <TD>ERROR/WARNING</TD> <TD>Review network connectivity between the Log module and its delivery endpoint (Environment ActiveGate or Dynatrace endpoint); check for interruptions during the affected window</TD> </TR> <TR> <TD><CODE>data_loss.*</CODE><SPAN>&nbsp;</SPAN>(general)</TD> <TD>ERROR/WARNING</TD> <TD>Read the full event for the specific condition; do not assume a sub-type without reading it</TD> </TR> </TBODY> </TABLE> <H4>&nbsp;</H4> <H4>Process group events</H4> <P>&nbsp;</P> <TABLE> <THEAD> <TR> <TH class="lia-align-center"><FONT size="2">Event type</FONT></TH> <TH class="lia-align-center"><FONT size="2">Severity</FONT></TH> <TH class="lia-align-center"><FONT size="2">Action</FONT></TH> </TR> </THEAD> <TBODY> <TR> <TD><CODE>pgi.multiple_pgis</CODE></TD> <TD>WARNING</TD> <TD>Determine if intentional (shared sink log file) or unintentional (PGI transition); refine the custom log source definition or process group configuration if unintentional</TD> </TR> <TR> <TD><CODE>pgi.lgi_explosion</CODE></TD> <TD>ERROR</TD> <TD>Create a custom log source rule with a wildcard pattern covering all log files written by the process group instance</TD> </TR> </TBODY> </TABLE> <HR /> <H3>Step 3: Apply the remediation and validate</H3> <P>After applying the fix:</P> <OL> <LI>Write a unique, identifiable test record to the original source.</LI> <LI>Record its source timestamp and timezone.</LI> <LI>Wait for the next Log module polling cycle.</LI> <LI>Check whether the same SFM event is still active on the dashboard.</LI> <LI>Search for the test record in the <STRONG>Logs</STRONG> app, filtered by host, workload, or source.</LI> </OL> <HR /> <H3>Step 4: If no SFM event is active</H3> <P>Check source coverage:</P> <DIV class="dutmcw0"> <DIV class="dutmcw1"> <DIV class="_1bzvwaf5q-3-9-0 _1bzvwaf6e-3-9-0 _1bzvwaf2-3-9-0 _1bzvwaf79-3-9-0" data-dt-component="Grid"> <DIV class="strato-button-label _10kgnsa9-3-9-0 _10kgnsab-3-9-0">&nbsp;</DIV> </DIV> </DIV> <PRE class="dutmcw2"><CODE>Settings app &gt; Collect and capture &gt; Log monitoring &gt; Configure log module &gt; Sources </CODE></PRE> </DIV> <P>Confirm whether the source is:</P> <UL> <LI>Listed under the expected entity</LI> <LI>Covered by an active or inherited ingest rule</LI> <LI>Matched by a rule whose scope and matcher correspond to the current source attributes</LI> </UL> <P>If the source is missing or uncovered, continue with <A href="https://community.dynatrace.com/t5/tkb/workflowpage/tkb-id/troubleshooting/article-id/1187?prePageCrumb=TkbDashboardPage" target="_blank" rel="noopener noreferrer">Log source not detected or not ingested: Troubleshoot source discovery and ingest-rule coverage</A>.</P> <HR /> <H3>Step 5: Check for short, unexplained gaps</H3> <P>The OneAgent Log module has built-in resilience. A dedicated watchdog process monitors the Log module: if it does not respond within <STRONG>15 minutes</STRONG>, the watchdog restarts it. Restart retry delays start at 10 seconds and double on each failure, capping at 1 hour. The delay resets after the module runs successfully for 20 minutes. There is no limit on restart attempts.</P> <P>Short, isolated ingestion gaps without a persistent SFM event can indicate a module restart cycle. If logs resumed without configuration changes and no event persists, a restart is a likely explanation. No action is required unless the gap is recurring or the SFM dashboard shows persistent issues.</P> <HR /> <H3>Step 6: If an Environment ActiveGate is involved</H3> <P>If the affected source uses an Environment ActiveGate and <STRONG>multiple sources on the same ActiveGate</STRONG> are affected simultaneously:</P> <UL> <LI>Check ActiveGate connectivity and health</LI> <LI>Review resource pressure on the ActiveGate during the incident window</LI> <LI>Consult the official sizing guidance: <A href="https://docs.dynatrace.com/docs/ingest-from/setup-on-k8s/guides/deployment-and-configuration/resource-management/ag-resource-limits" target="_blank" rel="noopener noreferrer">ActiveGate resource management and sizing for Kubernetes</A></LI> </UL> <P>If only one source through the ActiveGate is affected, continue with source-level investigation.</P> <P><EM><BR /><BR /></EM></P> <H2>What's next</H2> <P><EM><STRONG>Opening a support case with below details&nbsp;</STRONG></EM></P> <UL> <LI>Environment, region, and deployment type (SaaS / Managed)</LI> <LI>Affected host, host group, Kubernetes cluster, namespace, workload, or source path</LI> <LI>Approximate start time and timezone</LI> <LI>Screenshot or export from the Log module self-monitoring dashboard showing active issues</LI> <LI>Exact SFM event type, severity, and any source or entity context from the event</LI> <LI>Source coverage state from the Sources view</LI> <LI>Applicable active and inherited ingest rules with matchers and scopes</LI> <LI>Sanitized sample record from the affected source</LI> <LI>Confirmation that the original source writes new records</LI> <LI>Unique-record test result (found / not found)</LI> <LI>Recent changes: application, logging, permissions, path, encoding, rotation, deployment, or Dynatrace configuration</LI> <LI>OneAgent, Dynatrace Operator, and Environment ActiveGate versions</LI> </UL> <H2><BR />Related reading</H2> <UL> <LI><A href="https://docs.dynatrace.com/docs/analyze-explore-automate/logs/lma-log-ingestion/lma-log-ingestion-via-oa/lma-log-agent-sfm" target="_blank" rel="noopener noreferrer">Monitor log source health with SFM events</A></LI> <LI><A href="https://docs.dynatrace.com/docs/analyze-explore-automate/log-monitoring/lmc-troubleshooting/log-module-self-monitoring-events-classic" target="_blank" rel="noopener noreferrer">Log module self-monitoring events – Log Monitoring Classic</A></LI> <LI><A href="https://community.dynatrace.com/t5/tkb/workflowpage/tkb-id/troubleshooting/article-id/1187?prePageCrumb=TkbDashboardPage" target="_blank" rel="noopener noreferrer">Log source not detected or not ingested</A></LI> <LI><A href="https://www.dynatrace.com/news/blog/modern-log-management-resilience-and-self-service-upgrades-for-log-ingestion/" target="_blank" rel="noopener noreferrer">Improve log ingest health with smarter log management</A></LI> <LI><A href="https://community.dynatrace.com/t5/Troubleshooting/Logs-Troubleshooting-Map/ta-p/302260" target="_blank" rel="noopener noreferrer">Dynatrace Logs Troubleshooting Map</A></LI> </UL> </DIV> Fri, 04 Sep 2026 06:47:48 GMT noel_david 2026-09-04T06:47:48Z https://community.dynatrace.com/t5/Troubleshooting/My-logs-went-silent-where-do-I-start/ta-p/303793 <P><LI-TOC indent="15" liststyle="disc" maxheadinglevel="2"></LI-TOC></P> <DIV class="lia-message-template-content-zone"> <P>*<EM>Use a table of contents for longer articles.&nbsp;</EM></P> <H2>Summary</H2> <P><EM>*This is a mandatory section.</EM></P> <P><EM>Short description of which part of the Dynatrace platform the article refers to and what kind of problem it will help resolve/ task it will describe.</EM></P> <P>&nbsp;</P> <H2>Problem</H2> <P><EM>*This is a mandatory section.</EM></P> <P><EM>Provide a precise description of the problem/ task to be described. Use anonymized screenshots, and include text for&nbsp;important messages, errors, or information that will help the customer find this article when searching.</EM></P> <P>&nbsp;</P> <H2>Troubleshooting steps</H2> <P><EM>*This section can be omitted as necessary.</EM></P> <P><EM>Explain what troubleshooting steps should be taken to ensure the problem matches this article.</EM></P> <P>&nbsp;</P> <H2>Resolution</H2> <P><EM>*This section can be omitted for articles that guide customers on case creation.</EM></P> <P><EM>Explain the solution or all possible solutions resulting from the troubleshooting steps.<BR /><BR /></EM></P> <H2>What's next</H2> <P><EM>*This is a mandatory section. Customers need a way to respond or follow up if they have questions.&nbsp;</EM></P> <P><EM>Explain what to do if the article didn't help.&nbsp;</EM><EM>Note that there are multiple options available, including:</EM></P> <OL> <LI><EM><STRONG>Opening a chat or support case&nbsp;</STRONG>- be as specific as possible about the information the customer should include in the case.&nbsp;</EM><EM>Use the following formula and fill or change the bullet points:<BR /><BR /></EM> If this article didn't help, <STRONG>create a chat or open a support case</STRONG> <UL> <LI>Mention that you reviewed this article.</LI> <LI>Include the following details: <UL> <LI>Link to XYZ</LI> <LI>Screenshot of XYZ</LI> <LI>Information about XYZ</LI> </UL> </LI> </UL> </LI> <LI><EM><STRONG>Suggesting Product Idea</STRONG> - encourage the customer to suggest/ vote for a Product Idea explaining their business use case.<BR /></EM> <DIV>Please submit a Product Idea with detailed information about your request and use case in the Product Ideas forum. Our Product Management team regularly reviews these suggestions:</DIV> <A href="https://community.dynatrace.com/t5/Dynatrace-product-ideas/idb-p/DynatraceProductIdeas" target="_blank" rel="noopener">https://community.dynatrace.com/t5/Dynatrace-product-ideas/idb-p/DynatraceProductIdeas</A></LI> <LI><STRONG><EM>Explain this will change in the future </EM></STRONG><EM>- explain that this behavior will change in a future release. (No product idea / support case needed)</EM></LI> <LI><STRONG><EM>Alternatives -</EM></STRONG><EM>&nbsp;any other actions or links to other articles that could move the customer forward.</EM></LI> </OL> <H2><BR />Related reading</H2> <P><EM>*Link this article to other relevant configuration or troubleshooting guides, including any applicable troubleshooting maps.<BR />:open_book:&nbsp;&nbsp;</EM><A href="https://community.dynatrace.com/t5/Troubleshooting/" target="_blank" rel="noopener">XXX Troubleshooting Map</A></P> <P>:open_book:&nbsp; Link 1</P> <P>:open_book:&nbsp; Link 2</P> <P>:open_book:&nbsp; Link 3</P> <P>:open_book:&nbsp; Link 4</P> </DIV> Fri, 04 Sep 2026 06:47:48 GMT https://community.dynatrace.com/t5/Troubleshooting/My-logs-went-silent-where-do-I-start/ta-p/303793 noel_david 2026-09-04T06:47:48Z