Why my logs are not visible in Dynatrace?

[noel_david]

There can be many reasons why the logs are not visible in Dynatrace, below are the common reasons.

• Check if the One agent running in the host is the latest one.(https://docs.dynatrace.com/docs/shortlink/release-notes)
• Check if the Log agent is running in the host, you can check by going to the host page -> Process analysis section- FIlter by technology Dynatrace and see if the Log agent is running.

                     

• Check if Log Monitoring is enabled in your Dynatrace environment globally (Dynatrace web UI), or check if Log Monitoring is enabled on a host level (OneAgent CLI). To check if Dynatrace Log Monitoring is enabled globally:

  1. Go to "Settings > Monitoring > Monitored technologies".
  2. Find "Log Monitoring" in the list of supported technologies, and select "Edit" (pencil icon).
  3. Check if "Monitor Log Monitoring" on every host option is enabled.

• Log Monitoring is disabled on that host. Check "Settings -> Technology -> Log Monitoring" page on WebUI.

• Log content access disabled (https://docs.dynatrace.com/docs/setup-and-configuration/dynatrace-oneagent/oneagent-configuration-vi...)
• Improper ingest rules (https://docs.dynatrace.com/docs/observe-and-explore/logs/lma-log-ingestion-via-oa, follow also hints visible on configuration pages on WebUI), especially:  

  • No ingest rule was created to get the log content. These are for the automatically detected logs. (https://docs.dynatrace.com/docs/observe-and-explore/logs/lma-log-ingestion-via-oa/lma-autodiscovery)... To see if the logs are detected my Dynatrace, the easier way to check the process group page and check the logs tab.

  • If the logs are not detected then a custom rule is needed (https://docs.dynatrace.com/docs/shortlink/lma-custom-log-source). For a custom rule to work properly and a log ingestion rule is also needed.  (There is now a built-in log ingest rule that you can enable that will ingest all logs from custom log sources. Just enable that rule and you don't have to worry about making multiple rules.)

  • Multiple matchers of the same type (do not confuse with multiple values for the same matcher). Remember that you have "AND" operator between matchers, so usually you want max one occurrence of a single attribute type in the scope of a single rule.

  • Expecting partial match in a matcher (when actually full match is being done, so to have a partial match you need prefix and suffix wildcards).

  • Expecting case insensitive matching (when it is case sensitive, except log source names on Windows).

  • A typo, including non-visible characters, or non-ANSI Unicode characters.

  • Improper rule order - typically you need more specific closer to the top as they are executed top to bottom and the first catch is decisive

  • Improper scope - keep in mind that rules on more specific levels are evaluated earlier (so they are more important) - host before host group, and host group before tenant (https://docs.dynatrace.com/docs/shortlink/lma-log-ingest-rules#configuration-scopes)

• Security rules are violated (https://docs.dynatrace.com/docs/shortlink/lma-custom-log-source#security-rules)

• Log Agent does not have access to the file. It might especially happen when a file is on NFS drive on Linux. Then you need to ensure that a user account Log Agent is running on has access to the file (i.e. read privileges to the file, and read with execute to all the directories on the file path) and also enable NFS drive log detection, Settings > Log Monitoring > Advanced log settings

• Improper format

  • supported encodings include UTF-8 and UTF-16

• No new content after the file has been configured to be sent - Log Agent does not send historical data.

• Auto-detection limits are here: https://docs.dynatrace.com/docs/observe-and-explore/logs/lma-log-ingestion-via-oa/lma-autodiscovery#...

• Log Monitoring default limits are mentioned here (https://docs.dynatrace.com/docs/shortlink/log-monitoring-limits)

• For managed environments, it's also worth checking the cluster events to see if there are any warnings as mentioned here (https://docs.dynatrace.com/docs/shortlink/lmc-troubleshooting)

[Babar_Qayyum]

Hello @noel_david 

Thank you for summarizing the troubleshooting points.

What could be the potential issue of no Auto-discovery of Kubernetes/OpenShift logs?

Regards,

Babar

[noel_david]

Hi @Babar_Qayyum ,

In general, 

•  OneAgent autodiscovers log streams written by pods to stdout/stderror,  files written inside containers are not discovered automatically. but still for those we the log ingest and rule and custom log source rule are applicable.
• If the logs are created by a Cron jobs or short running processes or unimportant process, you can try the declarative process grouping configuration (https://docs.dynatrace.com/docs/platform-modules/infrastructure-monitoring/process-groups/configurat...

[Babar_Qayyum]

Hello @noel_david 

The log ingested rule is applied on the namespaces level, and also these are important processes.

Regards,

Babar

[noel_david]

Hi @Babar_Qayyum ,

Please create a ticket, with all details. 

[Babar_Qayyum]

Hello @noel_david 

I just wanted to update you that log monitoring was not enabled on the global level.

Regards,

Babar

[erh_inetum]

Hi @noel_david ,

Thanks a lot for your great post.

I think we can add another use case in "Improper ingest rules" point:

• The log meets with the autodiscovery requirements but the rule has it a condition a process group without services deployed

In this case the log will not visible in Dynatrace either.

Thanks,

Elena.

[noel_david]

Hi @erh_inetum , Thank you . 
Can you please share an example here, if possible,

[erh_inetum]

Hi @noel_david ,

Here an example: for this kind of rules

we have seen that if the process group doesn´t have deployed services 

the log isn´t captured. But it is in case the process group has deployed services.

Please, let me know in case my explanation is not clear and it try to explain it better again.

Thanks you so much.

Regards,

Elena.

[noel_david]

Hi @erh_inetum ,

I tried to reproduce the issue but was unsuccessful; if you can share a sample, it will be unique. Can you share me if you have one? A support ticket will be perfect.

[erh_inetum]

Hi @noel_david ,

The screenshots that I put in my comment below are the issue we had and how to reproduce it.

We haven't opened a ticket because we configures the rule for an specific host and we got the log.

Let me know if this information os enough.

Thanks so much.

Regards,

Elena

[StrangerThing]

For this point:  

• If the logs are not detected then a custom rule is needed (https://docs.dynatrace.com/docs/shortlink/lma-custom-log-source). For a custom rule to work properly and a log ingestion rule is also needed. 

There is now a built-in log ingest rule that you can enable that will ingest all logs from custom log sources. Just enable that rule and you don't have to worry about making multiple rules.

[noel_david]

Hi @StrangerThing , Thank you.

@erh_inetum , Thank i'll have a  look