Troubleshooting Kubernetes logs in Log module

Kacper · ‎02 Jun 2026

Summary

This article helps you troubleshoot Kubernetes-specific log collection issues with Log module. It covers no logs appearing, missing or incorrect Kubernetes metadata, and gaps in container logs. It applies to Log Module deployed via OneAgent (Classic or Cloud-Native Full-Stack) or Dynatrace Operator (Kubernetes Log Module), on both SaaS and Managed environments.

Problem

You see one of the following:

No logs at all from Kubernetes workloads, or no logs from a specific namespace or node.
Logs are visible in the Logs app, but lack Kubernetes metadata such as k8s.namespace.name, k8s.pod.name, k8s.workload.name, or k8s.workload.kind.
Logs are visible in the Logs app, but the Container detail view in the Kubernetes app shows No logs found.
Pod annotation–based ingest rules are ignored.
Time-series log volume charts show periodic gaps.

Kubernetes metadata attached to Container Output logs

For reference, this is the metadata that Log Module attaches automatically to logs ingested as Container Output:

Attribute	Description	Notes
`k8s.container.name`	Container name
`k8s.pod.name`	Pod name	Numbers preserved
`k8s.deployment.name`	Deployment base name	Numbers generalized to `*`
`k8s.pod.uid`	Pod UID
`k8s.namespace.name`	Namespace
`k8s.cluster.name`	Cluster display name
`k8s.workload.name`	Exact workload name	Requires `.spec.logMonitoring` and the Collect all container logs flag
`k8s.workload.kind`	Workload kind	Requires `.spec.logMonitoring` and the Collect all container logs flag
`k8s.pod.annotation`		Requires `.spec.logMonitoring` and the Collect all container logs flag
`k8s.pod.label`		Requires `.spec.logMonitoring` and the Collect all container logs flag
`dt.entity.container_group_instance`	Container entity ID	Full-Stack OneAgent only. Used by the Container screen in the Kubernetes app

Key rule: Logs ingested as Container Output (via the container runtime) automatically receive Kubernetes metadata. Logs ingested from file-system paths via a custom log source pointing to /var/log/pods/... don't.

Troubleshooting steps

Before opening a ticket or investigating a specific scenario, verify all items in this checklist.

#	Check	How to verify
1	A log ingest rule exists for the desired logs	Settings > Collect and capture > Log Monitoring > Log storage configuration. Select the relevant scope at the top left.
2	`.spec.logMonitoring` is present in the correct DynaKube	`kubectl get dynakube -n dynatrace -o jsonpath='{range .items[*]}{.metadata.name}{"\n\t.spec.logMonitoring: "}{.spec.logMonitoring}{"\n\t.spec.metadataEnrichment: "}{.spec.metadataEnrichment}{"\n"}{end}'`
3	`.spec.metadataEnrichment.enabled: true` in the DynaKube	Same command as above
4	ActiveGate has the `kubernetes-monitoring` capability	`kubectl get dynakube -n dynatrace -o jsonpath='{range .items[*]}{.metadata.name}{"\n\t.spec.activeGate.capabilities: "}{.spec.activeGate.capabilities}{"\n"}{end}'`
5	No conflicting ingest rules at a higher scope override your include rule	Check hierarchy and overrides on the Kubernetes cluster scope
6	No `ingest.warning` attribute on existing log records	DQL: `fetch logs \| filter isNotNull(dt.ingest.warnings)`
7	Operator DaemonSet pods are running on every node	`kubectl -n dynatrace get pods -o jsonpath='{range .items[*]}{.metadata.ownerReferences[0].name}{"\t"}{.spec.nodeName}{"\n"}{end}' \| grep -E 'logmonitoring\|oneagent'`

If the checklist passes, match your symptoms to a scenario below.

Resolution

Log storage rule not matching

Symptoms

Container logs don't appear in Dynatrace, even though the cluster is fully monitored.
No ingest warning is present on any record because no record exists.

Cause (brief)

Log ingestion is a two-step process: source detection (automatic) and log acquisition (requires an explicit ingest rule). A source definition alone is not sufficient. It must match a rule with Include in storage set.

Resolution

Go to Settings > Collect and capture > Log Monitoring > Log storage configuration.
Select the Kubernetes cluster scope.
Add a rule with Include in storage that matches your target containers, namespaces, or workloads.
Verify rule order: rules execute top-to-bottom; the first match wins. A broad exclusion above your inclusion silently blocks ingestion.
Check the scope hierarchy: Host scope > Kubernetes cluster scope > Host group scope > Environment scope. A more specific scope takes precedence.

Tip: Add an explicit Exclude all rule at the bottom of the Kubernetes cluster scope to prevent fallthrough to a broad environment-level include rule.

Configuration changes propagate to agents within ~90 seconds.

`PodListFuture not valid` — missing `.spec.logMonitoring`

Symptoms

The OneAgent or Kubernetes Log Module pod logs repeatedly show:

  [containersdetector] [error] Annotation retrieval failed: PodListFuture not valid (This message was ignored N times.)

Logs from specific namespaces or workloads are absent.
k8s.workload.name, k8s.workload.kind, k8s.pod.label and k8s.pod.annotation are missing.

Cause (brief)

Log Module cannot reach the Kubelet API because the DynaKube is missing .spec.logMonitoring, which grants the required RBAC.

Resolution

Add .spec.logMonitoring and .spec.metadataEnrichment to the DynaKube that contains .spec.oneAgent:

yaml

spec:
  logMonitoring: {}
  metadataEnrichment:
    enabled: true

Apply the change:

bash

kubectl apply -f dynakube.yaml

If the error persists after a few minutes, restart the Log Monitoring DaemonSet:

bash

kubectl rollout restart daemonset -n dynatrace <log-monitoring-daemonset-name>

Note for Full-Stack deployments: .spec.logMonitoring does not deploy a separate Kubernetes Log Module DaemonSet when .spec.oneAgent is already defined. It only activates log monitoring inside the existing Full-Stack OneAgent and grants it the RBAC needed to access the local Kubelet API.

The PodListFuture not valid error does not prevent log collection — only metadata enrichment is affected. Records are still ingested, but without k8s.* attributes, they won't appear in the Kubernetes app or workload-filtered views.

`.spec.logMonitoring` in the wrong DynaKube

Symptoms

After adding .spec.logMonitoring (and .spec.templates.logMonitoring), log collection does not improve.
The PodListFuture not valid error persists.

Cause (brief)

.spec.logMonitoring and .spec.templates.logMonitoring must be in the same DynaKube as .spec.oneAgent. If you have separate DynaKube objects for the ActiveGate and the OneAgent, placing the log monitoring configuration in the wrong one is silently ignored.

Resolution

Move .spec.logMonitoring and .spec.templates.logMonitoring to the DynaKube that also contains .spec.oneAgent.

Example of a correct split-DynaKube configuration:

yaml

# DynaKube 1: ActiveGate
spec:
  activeGate:
    capabilities:
      - kubernetes-monitoring
---
# DynaKube 2: OneAgent + Log Monitoring (BOTH must be here)
spec:
  oneAgent:
    applicationMonitoring: {}
  metadataEnrichment:
    enabled: true
  logMonitoring:
    ingestRuleMatchers:
      - attribute: k8s.namespace.name
        values:
          - production
          - staging

Init container permission failure on SELinux / OpenShift

Symptoms

The Log Monitoring DaemonSet fails to start.
The init container (bootstrapper) can't access the required host paths and exits with a permission error.
The DaemonSet ends up in Init:Error or CrashLoopBackOff.

Cause

SELinux blocks the bootstrapper from accessing host paths required for Log Module setup. Most commonly seen on OpenShift, but applies to any environment with SELinux enforcement.

Resolution

Add the feature.dynatrace.com/oneagent-privileged: "true" annotation to the DynaKube manifest:

yaml

apiVersion: dynatrace.com/v1beta3
kind: DynaKube
metadata:
  name: <your-dynakube>
  namespace: dynatrace
  annotations:
    feature.dynatrace.com/oneagent-privileged: "true"
spec:
  ...

Restart the DaemonSet:

bash

kubectl rollout restart daemonset -n dynatrace <log-monitoring-daemonset-name>

Specific namespaces not ingested

Symptoms

Some namespaces produce logs visible in Dynatrace; others do not, despite identical configuration.

Resolution

Work through the common causes in order:

Cause	Fix
No ingest rule for those namespaces	Add a `k8s.namespace.name` matcher to the ingest rule
OneAgent is not running on the nodes hosting affected pods	Confirm the OneAgent DaemonSet has pods on every affected node
Custom log source conflict on `/var/log/pods`	Remove the custom source — see Custom log source blocks Kubernetes enrichment
Higher-priority scope has an exclude override	Check host, host-group, and environment scopes for conflicting rules
The `dynatrace` namespace is missing	The `dynatrace` namespace is excluded by default. Enable Allow OneAgent to monitor Dynatrace logs under Settings → Collect and capture → Log Monitoring → Advanced log settings

Container screen shows "No logs found"

Symptoms

Logs are accessible at the workload, namespace, or pod level in the Kubernetes app.
The Container detail view shows "No logs found."

Cause

Container-level log display requires the dt.entity.container_group_instance attribute. This attribute is only added by the Full-Stack OneAgent. It is not added by the Kubernetes Log module or by App-Only deployments.

Resolution

This is by design for App-Only and Kubernetes Log module deployments. Logs remain queryable via the Logs viewer using k8s.container.name, k8s.pod.name, etc.

If container-level display in the Kubernetes app is required, deploy Full-Stack OneAgent on the affected nodes.

Custom log source blocks Kubernetes enrichment

Symptoms

Kubernetes metadata (pod name, namespace, deployment name) is missing from logs, even though the cluster is fully monitored.
Custom log source rules targeting /var/log/pods/ or /var/log/containers/ are present in the configuration.

Cause

A custom log source matching Kubernetes container log directories bypasses the automatic container log detection pipeline. Log Module treats those files as plain host-level logs and does not attach k8s.* metadata.

Resolution

Remove the custom log source rules that target /var/log/pods/ or /var/log/containers/, then use ingest rules with Kubernetes matchers (such as k8s.namespace.name, k8s.pod.annotation, k8s.workload.name) to control what gets ingested.

For step-by-step resolution, see Troubleshooting missing logs → Custom log source blocks Kubernetes enrichment.

Warning: After removing path-based custom log sources, logs may stop appearing entirely if no matching ingest rule exists. Ensure an ingest rule covers Container Output before removing the custom source.

`k8s.deployment.name` shows wildcards (`*`)

Symptoms

Filtering by k8s.deployment.name fails because numeric parts are replaced with * (for example, pricing-service-v24 becomes pricing-service-v*).

Cause

k8s.deployment.name is the base pod name with numeric suffixes generalized. This is by design, it groups all replica pods under a single entity, consistent with how Dynatrace entity modeling works.

Workarounds

Use k8s.pod.name (numeric suffixes are always preserved):

fetch logs
| filter startsWith(k8s.pod.name, "pricing-service-v24")

Or upgrade to OneAgent ≥ 1.307 and Dynatrace Operator ≥ 1.4.1 and use k8s.workload.name, which contains the exact workload name without generalization:

fetch logs
| filter k8s.workload.name == "pricing-service-v24"

Pod annotation–based rules ignored — `Collect all container logs` toggle

Symptoms

Pod annotation–based log ingestion rules are ignored. Logs from annotated pods are not ingested.

Cause

Whether the Collect all container logs toggle is required depends on the deployment type.

Deployment	Toggle required?	Min version
OneAgent DaemonSet (Classic / Cloud-Native Full-Stack)	Yes	OneAgent 1.309, Operator 1.4.2
Kubernetes Log Module (`spec.logMonitoring` only)	No	OneAgent 1.309, Operator 1.4.2

Resolution (OneAgent DaemonSet only)

Turn on Settings > Collect and capture > Log Monitoring > Log module feature flags > Collect all container logs at the Kubernetes cluster scope.

Log Module at performance limits

Symptoms

Periodic gaps in log volume are visible as drops in time-series charts.
High-throughput pods show intermittently missing logs.

Cause

Log Module processes data in 60-second intervals with configurable throughput limits. When log volume exceeds the default limits, the agent falls behind, and gaps appear.

Resolution

Processing and upload limits are adjustable per environment and are handled individually by Support. Open a support ticket with:

The affected cluster and namespaces.
The approximate log volume from the affected pods.
A screenshot of the volume gap in the time-series chart.

What's next

For boundary issues (multi-line entries split incorrectly, entries merged), see Troubleshooting log record boundary detection.
For broader "missing logs" troubleshooting beyond Kubernetes, see Troubleshooting missing logs.
For ingest delay (logs arrive but late), see Troubleshooting log ingest delay.
For JSON parsing issues (wrong timestamp from JSON field, severity mapping), see Troubleshooting JSON log parsing.
For log rotation issues (gaps at rotation boundaries, missing rotated files), see Troubleshooting log rotation issues with Log Module.