Docker container logs are not being ingested into Dyntrace Managed

[irivas1]

Hi,

Not long ago, I migrated from a SaaS environment to a managed environment. The thing is, in the SaaS environment, I had no trouble monitoring a Docker environment, everything worked correctly just by running the OneAgent deployment command as a Docker container.
However, when I tried to install the agent to send data to Dynatrace Managed, the logs aren’t being sent.

The Dynatrace environment consists of three nodes and an ActiveGate environment to which OneAgent connects.

I’ve done the following:

• The following command was run to install the agent as a Docker container:

docker run -d --restart=on-failure:5 --read-only=true --pid=host --net=host --cap-drop ALL --cap-add CHOWN --cap-add DAC_OVERRIDE --cap-add DAC_READ_SEARCH --cap-add FOWNER --cap-add FSETID --cap-add KIL --cap-add NET_ADMIN --cap-add NET_RAW --cap-add SETFCAP --cap-add SETGID --cap-add SETUID --cap-add SYS_ADMIN --cap-add SYS_CHROOT --cap-add SYS_PTRACE --cap-add SYS_RESOURCE --security-opt apparmor:unconfined -v /:/mnt/root -v dynatrace_volume:/mnt/volume_storage_mount -e ONEAGENT_ENABLE_VOLUME_STORAGE=true -e ONEAGENT_INSTALLER_SCRIPT_URL=XXXXX.dynatrace-managed.com/e/7ad2d0d4-de31-497d-9e13-68492b5ef0a3/api/v1/deployment/installer/agent/unix/default/latest?arch=x86 -e ONEAGENT_INSTALLER_DOWNLOAD_TOKEN=dt0c01.XXXXXXX... dynatrace/oneagent

• Under Settings > Monitoring > Monitored Technologies, the log monitoring option is enabled

• Under Settings > Log Monitoring > Advanced Log Settings, the “Detect logs of containerized applications” option is enabled

• The “[Built-in]Ingest all logs” ingest rule is enabled
• I have verified that the DataRoot is /var/lib/docker and that the driver is json-file
• I have verified the mounts:

      docker inspect <oneagent-container> --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}'
     /var/lib/docker/volumes/dynatrace_volume/_data -> /mnt/volume_storage_mount

I'm getting metrics and data from the services, but I can't seem to get the logs.

Does anyone know what might be causing the problem?

Best regards

[AntonPineiro]

Hi,

Maybe log path is something "custom" and you need to include a custom log source.

I would check that.

Best regards

[irivas1]

Hi,
I've created a custom log source with the following path: /var/lib/docker/containers/*/*-json.log. I know there are logs inside the containers at that path:

but I'm getting the same result as with the auto-discovered log sources: “no log data available.”

Best Regards.

[AntonPineiro]

Hi,

Since logs appears there. I would say:

• Confirm if double regular expresion would work in path.
• Confirm ingestion rule is enabled.
• Confirm if log lines are being added in files.

You can check Logs Troubleshooting Map 

Best regards