on - edited on by
HannahM
- Summary
- Troubleshooting
- For logs ingested via OneAgent
- For logs ingested via K8s LogAgent module
- For SysLog ingestion via ActiveGate
- For logs ingested via Fluent Bit
- What's next
Summary
Log ingestion is the process of collecting log data from various sources within an infrastructure. These logs can then be analyzed for many purposes within Dynatrace.
This article provides troubleshooting steps for when logs are not ingested/ visible in Dynatrace.
Troubleshooting
For logs ingested via OneAgent
- Check if the OneAgent running on the host is the supported version/latest version.
- Check if the Log agent is running on the host. You can do this by going to the host page -> Process analysis section—Filter by technology Dynatrace, and confirm that OneAgent log analytics is running.
- Check if Log Monitoring is enabled
- from the Dynatrace WebUI
- To check if Dynatrace Log Monitoring is enabled globally:
- Go to "Settings > Monitoring > Monitored technologies".
- Find "Log Monitoring" in the list of supported technologies, and select "Edit" (pencil icon).
- Check if the "Monitor Log Monitoring" option is enabled.
- To check if Dynatrace Log Monitoring is enabled at the host level:
- Go to "Settings > General ".
- Find "Log Monitoring" in the list of monitored technologies, and select "Edit" (pencil icon).
- Check if "Monitor Log Monitoring" on every host option is enabled
- To check if Dynatrace Log Monitoring is enabled globally:
- using the OneAgent CLI tool
- Use the
--get-app-log-content-accessparameter to check whether Log Monitoring is enabled:
Linux:./oneagentctl --get-app-log-content-accessWindows:
.\oneagentctl.exe --get-app-log-content-access
- Use the
- from the Dynatrace WebUI
- Confirm ingest rules are set up correctly. Log Ingest rules are documented here
- No ingest rule was created to get the log content.
These are for the automatically detected logs. To see if the logs are detected by Dynatrace, the easiest way to check is to go to the process group page and check the logs tab.- If the logs are not detected, then a custom rule is needed. For a custom rule to work correctly, a log ingestion rule is also required. (There is now a built-in log ingest rule that you can enable that will ingest all logs from custom log sources.)
- Multiple matchers of the same type (not to be confused with multiple values for the same matcher).
The "AND" operator is used between matchers, so usually you want at most one occurrence of a single attribute type in the scope of a single rule. - Expecting a partial match in a matcher
To have a partial match, you need prefix and suffix wildcards. - Expecting case-insensitive matching (when it is case sensitive, except log source names on Windows).
- Check for any typos, including non-visible characters, or non-ANSI Unicode characters.
- Confirm rule order
More specific rules should be closer to the top as they're executed top to bottom, and the first catch is decisive. - Improper scope
Rules on more specific levels are evaluated earlier (so they're more important) - host before host group, and host group before tenant - Check if any Security rules are violated.
You can verify this by checking the OneAgent logs.
Sample log with security warning, which can be verified inside the OneAgent logs.[2024-11-28 10:11:29.463 UTC] [/rework/logprocessing/filelogsource.cpp] [info] LGI: / /logs/appl/conc/out/*.txt doesn't meet security rules. (This message was ignored 14 timesYou can fix this by creating override rules.
-
Check if OneAgent has access to the file.
It might especially happen when a file is on an NFS drive on Linux. Then you need to ensure that the user account OneAgent is running on has access to the file, and also enable NFS drive log detection, Settings > Log Monitoring > Advanced log settings.
You can run the below command and check if it's accessible.sudo -u dtuser ls -l /path/to/logfile sudo -u dtuser cat /path/to/logfile
- Confirm the format is supported.
Supported encodings include UTF-8 and UTF-16. - Confirm the content was not added after the file was ingested.
No new content is added after the file has been configured to be sent—OneAgent does not send historical data. - Confirm that auto-detection limits have not been reached
- Confirm that the Log Monitoring default limits have not been reached.
- For managed environments, check the cluster events to see if there are any warnings, as mentioned here
For logs ingested via K8s LogAgent module
- Verify that the logmonitoring pod is up and running, using the following command
You should see a pod named "logmonitoring" running if it's running correctly.kubectl get pods -n dynatrace
- Check that you have the prerequisites enabled.
- Check that you have a matching rule created at the K8 cluster level / Global level.
- The requirements for autodiscovery and ingestion of Kubernetes logs are the following:
The containerd, CRI-O, or cri-dockerd container runtime is used.
Logs are written to the container's stdout/stderr stream.
For SysLog ingestion via ActiveGate
See Syslog Ingestion via ActiveGate Troubleshooting Guide
For logs ingested via Fluent Bit
See Troubleshooting logs ingested via Fluent Bit
What's next
If this article didn't help, please open a support ticket, mention that this article was used, and provide the following:
Hi @noel_david ,
Here an example: for this kind of rules
we have seen that if the process group doesn´t have deployed services
the log isn´t captured. But it is in case the process group has deployed services.
Please, let me know in case my explanation is not clear and it try to explain it better again.
Thanks you so much.
Regards,
Elena.
