Enable Log Visibility in Dynatrace

[noel_david]

• Summary
• Troubleshooting
• What's next

 

If your Dynatrace OneAgent logs are not visible, the issue often relates to a missing configuration or insufficient permissions. To resolve this, allow log ingestion, define log sources correctly, and ensure access rights are properly set. These steps improve log monitoring and visibility across your environment.

Summary

Log ingestion is the process of collecting log data from various sources within an infrastructure. These logs can then be analyzed for many purposes within Dynatrace. 

This article provides troubleshooting steps for when logs are not ingested/ visible in Dynatrace.

Important: Log Monitoring is enabled by default but only controls the OneAgent log module capability. To actually ingest and view logs, you must also configure log ingest rules. If you experience issues with log collection, verify that this setting hasn't been disabled.

 

Troubleshooting

For logs ingested via OneAgent

• Check if the OneAgent version running on the host is the supported version/latest version. 
• Check if the Log agent is running on the host. You can do this by going to the host page -> Process analysis section—Filter by technology Dynatrace, and confirm that OneAgent log analytics is running.
• Check if Log Monitoring is enabled
  • From the Dynatrace WebUI
    • To check if Dynatrace Log Monitoring is enabled globally: 
      • Go to "Settings > Monitoring > Monitored technologies".
      • Find "Log Monitoring" in the list of supported technologies, and select "Edit" (pencil icon).
      • Check if the "Monitor Log Monitoring" option is enabled.
    • To check if Dynatrace Log Monitoring is enabled at the host level:
      • Go to "Settings > General ".
      • Find "Log Monitoring" in the list of monitored technologies, and select "Edit" (pencil icon).
      • Check if "Monitor Log Monitoring" on every host option is enabled
  • Using the OneAgent CLI tool
    • Use the --get-app-log-content-access parameter to check whether Log Monitoring is enabled:
      Linux:
      

      ./oneagentctl --get-app-log-content-access​

      Windows:
      

      .\oneagentctl.exe --get-app-log-content-access​

⚠️ Common issue: The app log content access flag may be false on the target host even if the UI shows log monitoring as enabled — this can happen if oneagentctl was run on a different host than intended, or if a migrated configuration file (_migratedloganalytics.conf.json) retains "AppLogContentAccess": false from a previous state. Always verify the flag is true on the correct host.

• Confirm ingest rules are set up correctly. Log Ingest rules are documented here   
• No ingest rule was created to get the log content.
  These are for the automatically detected logs. To see if the logs are detected by Dynatrace, the easiest way to check is to go to the process group page and check the logs tab.
  • If the logs are not detected, then a custom rule is needed. For a custom rule to work correctly, a log ingestion rule is also required.  (There is now a built-in log ingest rule that you can enable that will ingest all logs from custom log sources.)
• Multiple matchers of the same type (not to be confused with multiple values for the same matcher).
  The "AND" operator is used between matchers, so usually you want at most one occurrence of a single attribute type in the scope of a single rule.
• Expecting a partial match in a matcher
  To have a partial match, you need prefix and suffix wildcards.
• Expecting case-insensitive matching (when it is case sensitive, except log source names on Windows).
• Check for any typos, including non-visible characters, or non-ANSI Unicode characters.
• Confirm rule order: More specific rules should be closer to the top as they're executed top to bottom, and the first catch is decisive.
• Scope hierarchy : Rules on more specific levels are evaluated earlier (so they're more important) - host before host group, and host group before tenant 
• Check if any Security rules are violated.
  You can verify this by checking the OneAgent logs. Check this article 
  Sample log with security warning, which can be verified inside the OneAgent logs.
  

  [2024-11-28 10:11:29.463 UTC] [/rework/logprocessing/filelogsource.cpp] [info] LGI: / /logs/appl/conc/out/*.txt doesn't meet security rules. (This message was ignored 14 times​

  You can fix this by creating override rules. 

• Check if OneAgent has access to the file.
  It might especially happen when a file is on an NFS drive on Linux. Then you need to ensure that the user account OneAgent is running on has access to the file, and also enable NFS drive log detection, Settings > Log Monitoring > Advanced log settings.
  
  You can run the below command and check if it's accessible.
  

  sudo -u dtuser ls -l /path/to/logfile
  sudo -u dtuser cat /path/to/logfile

• Confirm the format is supported.
  Supported encodings include UTF-8 and UTF-16.
• Confirm the content was not added after the file was ingested.
  No new content is added after the file has been configured to be sent—OneAgent does not send historical data.
• Confirm that auto-detection limits have not been reached
• Confirm that the Log Monitoring default limits have not been reached.
• For managed environments, check the cluster events to see if there are any warnings, as mentioned here
• Check if you have access to the buckets where the logs are stored. You can run the below query to check the buckets you have access to.

  fetch dt.system.buckets
  | filter dt.system.table=="logs"

• Use Log Module Self-Monitoring (SFM) Events (OneAgent 1.333+ / GA in 1.339+)

  The Log module now includes built-in self-monitoring events (SFM events) that provide log source health status and alert on ingestion issues. SFM events show whether a log source is detected and configured correctly, and provide context (e.g., log source name, host ID) and remediation guidance.

  SFM events are enabled by default. They can be configured via the Settings API using schema ID builtin:logmonitoring.log-sfm-settings.

 

For logs ingested via K8s LogAgent module

• Verify that the  logmonitoring pod is up and running, using the following command
  

  kubectl get pods -n dynatrace​

  You should see a pod named "logmonitoring"  running if it's running correctly.

• Check that you have the prerequisites enabled.
• Check that you have a matching rule created at the K8 cluster level / Global level.
• The requirements for autodiscovery and ingestion of Kubernetes logs are the following:

The containerd, CRI-O, or cri-dockerd container runtime is used.

Logs are written to the container's stdout/stderr stream.

• Kubernetes app-only mode: If running in app-only (not full-stack) mode, ensure the .spec.templates.logMonitoring and .spec.logMonitoring sections are present in the DynaKube custom resource YAML. These are added automatically when the Kubernetes ingestion setup wizard is followed. The Dynatrace Operator is a mandatory prerequisite.

See this article for more detailed steps.

For SysLog ingestion via ActiveGate 

See Syslog Ingestion via ActiveGate Troubleshooting Guide

For logs ingested via Fluent Bit

See Troubleshooting logs ingested via Fluent Bit

For logs ingested via API- Post ingest logs

See Troubleshooting log Ingestion via API - POST ingest logs 

 

What's next

If this article didn't help, please open a support ticket, mention that this article was used, and provide the following:

• Link to the tenant
• Link to the process that is creating the log
• link to log ingest rule and custom log source rule (if needed)
• Copy of the actual log
• Dynatrace Operator logs if it's a Kubernetes deployment 
• OneAgent logs from the host

What to read next: 

Dynatrace does not ingest logs (HTTP 429)

How can I get logs from an Azure subscription in another Azure tenant?

Why my logs are not visible in Dynatrace?

Logs Troubleshooting Map

[Babar_Qayyum]

Hello @noel_david 

Thank you for summarizing the troubleshooting points.

What could be the potential issue of no Auto-discovery of Kubernetes/OpenShift logs?

Regards,

Babar

[noel_david]

Hi @Babar_Qayyum ,

In general, 

•  OneAgent autodiscovers log streams written by pods to stdout/stderror,  files written inside containers are not discovered automatically. but still for those we the log ingest and rule and custom log source rule are applicable.
• If the logs are created by a Cron jobs or short running processes or unimportant process, you can try the declarative process grouping configuration (https://docs.dynatrace.com/docs/platform-modules/infrastructure-monitoring/process-groups/configurat...

[Babar_Qayyum]

Hello @noel_david 

The log ingested rule is applied on the namespaces level, and also these are important processes.

Regards,

Babar

[noel_david]

Hi @Babar_Qayyum ,

Please create a ticket, with all details. 

[Babar_Qayyum]

Hello @noel_david 

I just wanted to update you that log monitoring was not enabled on the global level.

Regards,

Babar

[erh_inetum]

Hi @noel_david ,

Thanks a lot for your great post.

I think we can add another use case in "Improper ingest rules" point:

• The log meets with the autodiscovery requirements but the rule has it a condition a process group without services deployed

In this case the log will not visible in Dynatrace either.

Thanks,

Elena.

[noel_david]

Hi @erh_inetum , Thank you . 
Can you please share an example here, if possible,

[erh_inetum]

Hi @noel_david ,

Here an example: for this kind of rules

we have seen that if the process group doesn´t have deployed services 

the log isn´t captured. But it is in case the process group has deployed services.

Please, let me know in case my explanation is not clear and it try to explain it better again.

Thanks you so much.

Regards,

Elena.

[noel_david]

Hi @erh_inetum ,

I tried to reproduce the issue but was unsuccessful; if you can share a sample, it will be unique. Can you share me if you have one? A support ticket will be perfect.

[erh_inetum]

Hi @noel_david ,

The screenshots that I put in my comment below are the issue we had and how to reproduce it.

We haven't opened a ticket because we configures the rule for an specific host and we got the log.

Let me know if this information os enough.

Thanks so much.

Regards,

Elena

[StrangerThing]

For this point:  

• If the logs are not detected then a custom rule is needed (https://docs.dynatrace.com/docs/shortlink/lma-custom-log-source). For a custom rule to work properly and a log ingestion rule is also needed. 

There is now a built-in log ingest rule that you can enable that will ingest all logs from custom log sources. Just enable that rule and you don't have to worry about making multiple rules.

[noel_david]

Hi @StrangerThing , Thank you.

@erh_inetum , Thank i'll have a  look