Troubleshooting Kubernetes logs in Log module

Kacper · ‎02 Jun 2026

Summary

This article helps you troubleshoot Kubernetes-specific log collection issues with Log module. It covers no logs appearing, missing or incorrect Kubernetes metadata, and gaps in container logs. It applies to Log Module deployed via OneAgent (Classic or Cloud-Native Full-Stack) or Dynatrace Operator (Kubernetes Log Module), on both SaaS and Managed environments.

Problem

You see one of the following:

No logs at all from Kubernetes workloads, or no logs from a specific namespace or node.
Logs are visible in the Logs viewer, but lack Kubernetes metadata such as k8s.namespace.name, k8s.pod.name, k8s.workload.name, or k8s.workload.kind.
Logs are visible in the Logs viewer, but the Container detail view in the Kubernetes app shows No logs found.
Pod annotation–based ingest rules are ignored.
Time-series log volume charts show periodic gaps.

Kubernetes metadata attached to Container Output logs

For reference, this is the metadata that Log Module attaches automatically to logs ingested as Container Output:

Attribute	Description	Notes
`k8s.container.name`	Container name
`k8s.pod.name`	Pod name	Numbers preserved
`k8s.deployment.name`	Deployment base name	Numbers generalized to `*`
`k8s.pod.uid`	Pod UID
`k8s.namespace.name`	Namespace
`k8s.cluster.name`	Cluster display name
`k8s.workload.name`	Exact workload name	Requires `.spec.logMonitoring` and the Collect all container logs flag
`k8s.workload.kind`	Workload kind	Requires `.spec.logMonitoring` and the Collect all container logs flag
`dt.entity.container_group_instance`	Container entity ID	Full-Stack OneAgent only. Used by the Container screen in the Kubernetes app

Key rule: Logs ingested as Container Output (via the container runtime) automatically receive Kubernetes metadata. Logs ingested from file-system paths via a custom log source pointing to /var/log/pods/... don't.

Troubleshooting steps

Before opening a ticket or investigating a specific scenario, verify all items in this checklist.

#	Check	How to verify
1	A log ingest rule exists for the desired logs	Settings > Collect and capture > Log Monitoring > Log storage configuration. Select the relevant scope at the top left.
2	`.spec.logMonitoring` is present in the correct DynaKube	`kubectl get dynakube -n dynatrace -o jsonpath='{range .items[*]}{.metadata.name}{"\n\t.spec.logMonitoring: "}{.spec.logMonitoring}{"\n\t.spec.metadataEnrichment: "}{.spec.metadataEnrichment}{"\n"}{end}'`
3	`.spec.metadataEnrichment.enabled: true` in the DynaKube	Same command as above
4	ActiveGate has the `kubernetes-monitoring` capability	`kubectl get dynakube -n dynatrace -o jsonpath='{range .items[*]}{.metadata.name}{"\n\t.spec.activeGate.capabilities: "}{.spec.activeGate.capabilities}{"\n"}{end}'`
5	No conflicting ingest rules at a higher scope override your include rule	Check hierarchy and overrides on the Kubernetes cluster scope
6	No `ingest.warning` attribute on existing log records	DQL: `fetch logs \| filter isNotNull(dt.ingest.warnings)`
7	Operator DaemonSet pods are running on every node	`kubectl -n dynatrace get pods -o jsonpath='{range .items[*]}{.metadata.ownerReferences[0].name}{"\t"}{.spec.nodeName}{"\n"}{end}' \| grep -E 'logmonitoring\|oneagent'`

If the checklist passes, match your symptoms to a scenario below.

Resolution

Log storage rule not matching

Symptoms

Container logs don't appear in Dynatrace, even though the cluster is fully monitored.
No ingest warning is present on any record because no record exists.

Cause (brief)

Log ingestion is a two-step process: source detection (automatic) and log acquisition (requires an explicit ingest rule). A source definition alone is not sufficient. It must match a rule with Include in storage set.

Resolution

Go to Settings > Collect and capture > Log Monitoring > Log storage configuration.
Select the Kubernetes cluster scope.
Add a rule with Include in storage that matches your target containers, namespaces, or workloads.
Verify rule order: rules execute top-to-bottom; the first match wins. A broad exclusion above your inclusion silently blocks ingestion.
Check the scope hierarchy: Host scope > Kubernetes cluster scope > Host group scope > Environment scope. A more specific scope takes precedence.

Tip: Add an explicit Exclude all rule at the bottom of the Kubernetes cluster scope to prevent fallthrough to a broad environment-level include rule.

Configuration changes propagate to agents within ~90 seconds.

`PodListFuture not valid` — missing `.spec.logMonitoring`

Symptoms

The OneAgent or Kubernetes Log Module pod logs repeatedly show:

  [containersdetector] [error] Annotation retrieval failed: PodListFuture not valid (This message was ignored N times.)

Logs from specific namespaces or workloads are absent.
k8s.workload.name and k8s.workload.kind are missing.

Cause (brief)

Log Module cannot reach the Kubelet API because the DynaKube is missing .spec.logMonitoring, which grants the required RBAC.

Resolution

Add .spec.logMonitoring and .spec.metadataEnrichment to the DynaKube that contains .spec.oneAgent:

yaml

spec:
  logMonitoring: {}
  metadataEnrichment:
    enabled: true

Apply the change:

bash

kubectl apply -f dynakube.yaml

If the error persists after a few minutes, restart the Log Monitoring DaemonSet:

bash

kubectl rollout restart daemonset -n dynatrace <log-monitoring-daemonset-name>

Note for Full-Stack deployments: .spec.logMonitoring does not deploy a separate Kubernetes Log Module DaemonSet when .spec.oneAgent is already defined. It only activates log monitoring inside the existing Full-Stack OneAgent and grants it the RBAC needed to access the local Kubelet API.

The PodListFuture not valid error does not prevent log collection — only metadata enrichment is affected. Records are still ingested, but without k8s.* attributes, they won't appear in the Kubernetes app or workload-filtered views.

`.spec.logMonitoring` in the wrong DynaKube

Symptoms

After adding .spec.logMonitoring (and .spec.templates.logMonitoring), log collection does not improve.
The PodListFuture not valid error persists.

Cause (brief)

.spec.logMonitoring and .spec.templates.logMonitoring must be in the same DynaKube as .spec.oneAgent. If you have separate DynaKube objects for the ActiveGate and the OneAgent, placing the log monitoring configuration in the wrong one is silently ignored.

Resolution

Move .spec.logMonitoring and .spec.templates.logMonitoring to the DynaKube that also contains .spec.oneAgent.

Example of a correct split-DynaKube configuration:

yaml

# DynaKube 1: ActiveGate
spec:
  activeGate:
    capabilities:
      - kubernetes-monitoring
---
# DynaKube 2: OneAgent + Log Monitoring (BOTH must be here)
spec:
  oneAgent:
    applicationMonitoring: {}
  metadataEnrichment:
    enabled: true
  logMonitoring:
    ingestRuleMatchers:
      - attribute: k8s.namespace.name
        values:
          - production
          - staging

GKE Autopilot — Kubelet API blocked

Symptoms

On GKE Autopilot clusters, the Log Module DaemonSet logs show:

  [containeractivitymonitor] Kubelet API initialization failed
  Checking Kubelet API validity: https://<node-ip>:10250

Most pods are missing from the Logs app; only pods with injected OneAgent are visible.

Cause (brief)

GKE Autopilot blocks direct access to the Kubelet API endpoint for nodes/proxy requests. The default Dynatrace Operator ClusterRole grants nodes/proxy, are insufficient on Autopilot. The Kubelet API requires nodes/pods permission on this platform.

Workaround

Edit the ClusterRole:

bash

kubectl edit ClusterRole/dynatrace-logmonitoring

Change:

yaml

# Before
- apiGroups: [""]
  resources: ["nodes/proxy"]
  verbs: ["get"]

# After
- apiGroups: [""]
  resources: ["nodes/pods"]
  verbs: ["get"]

Restart the Log Monitoring DaemonSet:

bash

kubectl rollout restart daemonset -n dynatrace <log-monitoring-daemonset-name>

Confirm in the logs:

[containeractivitymonitor] [info] Kubelet API initialized. Using endpoint https://<node>:10250

Note: Standard GKE clusters work with both nodes/proxy and nodes/pods. Only GKE Autopilot requires nodes/pods. A permanent fix in the official Dynatrace Operator is tracked separately.

Init container permission failure on SELinux / OpenShift

Symptoms

The Log Monitoring DaemonSet fails to start.
The init container (bootstrapper) can't access the required host paths and exits with a permission error.
The DaemonSet ends up in Init:Error or CrashLoopBackOff.

Cause

SELinux blocks the bootstrapper from accessing host paths required for Log Module setup. Most commonly seen on OpenShift, but applies to any environment with SELinux enforcement.

Resolution

Add the feature.dynatrace.com/oneagent-privileged: "true" annotation to the DynaKube manifest:

yaml

apiVersion: dynatrace.com/v1beta3
kind: DynaKube
metadata:
  name: <your-dynakube>
  namespace: dynatrace
  annotations:
    feature.dynatrace.com/oneagent-privileged: "true"
spec:
  ...

Restart the DaemonSet:

bash

kubectl rollout restart daemonset -n dynatrace <log-monitoring-daemonset-name>

Specific namespaces not ingested

Symptoms

Some namespaces produce logs visible in Dynatrace; others do not, despite identical configuration.

Resolution

Work through the common causes in order:

Cause	Fix
No ingest rule for those namespaces	Add a `k8s.namespace.name` matcher to the ingest rule
OneAgent is not running on the nodes hosting affected pods	Confirm the OneAgent DaemonSet has pods on every affected node
Custom log source conflict on `/var/log/pods`	Remove the custom source — see Custom log source blocks Kubernetes enrichment
Higher-priority scope has an exclude override	Check host, host-group, and environment scopes for conflicting rules
The `dynatrace` namespace is missing	The `dynatrace` namespace is excluded by default. Enable Allow OneAgent to monitor Dynatrace logs under Settings → Collect and capture → Log Monitoring → Advanced log settings

Container screen shows "No logs found"

Symptoms

Logs are accessible at the workload, namespace, or pod level in the Kubernetes app.
The Container detail view shows "No logs found."

Cause

Container-level log display requires the dt.entity.container_group_instance attribute. This attribute is only added by the Full-Stack OneAgent. It is not added by the Kubernetes Log module or by App-Only deployments.

Resolution

This is by design for App-Only and Kubernetes Log module deployments. Logs remain queryable via the Logs viewer using k8s.container.name, k8s.pod.name, etc.

If container-level display in the Kubernetes app is required, deploy Full-Stack OneAgent on the affected nodes.

Custom log source blocks Kubernetes enrichment

Symptoms

Kubernetes metadata (pod name, namespace, deployment name) is missing from logs, even though the cluster is fully monitored.
Custom log source rules targeting /var/log/pods/ or /var/log/containers/ are present in the configuration.

Cause

A custom log source matching Kubernetes container log directories bypasses the automatic container log detection pipeline. Log Module treats those files as plain host-level logs and does not attach k8s.* metadata.

Resolution

Remove the custom log source rules that target /var/log/pods/ or /var/log/containers/, then use ingest rules with Kubernetes matchers (such as k8s.namespace.name, k8s.pod.annotation, k8s.workload.name) to control what gets ingested.

For step-by-step resolution, see Troubleshooting missing logs → Custom log source blocks Kubernetes enrichment.

Warning: After removing path-based custom log sources, logs may stop appearing entirely if no matching ingest rule exists. Ensure an ingest rule covers Container Output before removing the custom source.

`k8s.deployment.name` shows wildcards (`*`)

Symptoms

Filtering by k8s.deployment.name fails because numeric parts are replaced with * (for example, pricing-service-v24 becomes pricing-service-v*).

Cause

k8s.deployment.name is the base pod name with numeric suffixes generalized. This is by design, it groups all replica pods under a single entity, consistent with how Dynatrace entity modeling works.

Workarounds

Use k8s.pod.name (numeric suffixes are always preserved):

fetch logs
| filter startsWith(k8s.pod.name, "pricing-service-v24")

Or upgrade to OneAgent ≥ 1.307 and Dynatrace Operator ≥ 1.4.1 and use k8s.workload.name, which contains the exact workload name without generalization:

fetch logs
| filter k8s.workload.name == "pricing-service-v24"

`k8s.workload.name` / `k8s.workload.kind` not populated

Symptoms

Logs lack k8s.workload.name and k8s.workload.kind. These attributes are required for the Kubernetes app to display workload-level log context.

Resolution

Work through the causes in order:

Cause	Fix
`.spec.logMonitoring` missing from DynaKube	Add `spec.logMonitoring: {}` to the DynaKube manifest
`spec.metadataEnrichment.enabled: false`	Set to `true`
ActiveGate missing `kubernetes-monitoring` capability	Add `- kubernetes-monitoring` to `spec.activeGate.capabilities`
Expiring Kubelet API service account token	Known issue in OneAgent ≤ 1.319; upgrade to 1.321 or later
`spec.logMonitoring` is in a different DynaKube than `spec.oneAgent`	See `.spec.logMonitoring` in the wrong DynaKube
The Collect all container logs switch is off	See Pod annotation–based rules ignored below

To confirm the webhook annotated the pod:

bash

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{":\t"}{.metadata.name}{":\t"}{.metadata.annotations}{"\n"}{end}'

Pod annotation–based rules ignored — `Collect all container logs` toggle

Symptoms

Pod annotation–based log ingestion rules are ignored. Logs from annotated pods are not ingested.

Cause

Whether the Collect all container logs toggle is required depends on the deployment type.

Deployment	Toggle required?	Min version
OneAgent DaemonSet (Classic / Cloud-Native Full-Stack)	Yes	OneAgent 1.309, Operator 1.4.2
Kubernetes Log Module (`spec.logMonitoring` only)	No	OneAgent 1.309, Operator 1.4.2

Resolution (OneAgent DaemonSet only)

Turn on Settings > Collect and capture > Log Monitoring > Log module feature flags > Collect all container logs at the Kubernetes cluster scope.

Log Module at performance limits

Symptoms

Periodic gaps in log volume are visible as drops in time-series charts.
High-throughput pods show intermittently missing logs.

Cause

Log Module processes data in 60-second intervals with configurable throughput limits. When log volume exceeds the default limits, the agent falls behind, and gaps appear.

Resolution

Processing and upload limits are adjustable per environment and are handled individually by Support. Open a support ticket with:

The affected cluster and namespaces.
The approximate log volume from the affected pods.
A screenshot of the volume gap in the time-series chart.

Multi-line entries split or merged

Symptoms

Stack traces, XML payloads, or other multi-line entries appear as multiple separate records (split), or several entries are combined into a single record (merged).
Behavior may be consistent (always splits/merges at the same point) or intermittent (sometimes correct, sometimes not).

Resolution

For consistent split or merge issues, typically caused by timestamp search limits or missing/non-standard timestamp patterns, define a Timestamp/splitting configuration rule that matches your log format.

For step-by-step guidance, see Troubleshooting log record boundary detection.

For intermittent splits caused by delayed continuation lines (continuation written more than ~60 seconds after the first line), the multi-line timeout is configurable per environment. Contact Dynatrace Support — see the workaround in Troubleshooting log record boundary detection → Intermittent splitting of multi-line entries with delayed continuation lines.

Timestamp issues are causing entries to be dropped

Symptoms

Logs are missing for a specific time window.
The Log Module pod logs show:

  Drop statistics: Out of bound time horizon entries, lower: 0+0, upper: 80

Cause

Dynatrace rejects logs with timestamps more than 15 minutes before the start of ingestion (with a 24-hour grace window after) or more than 10 minutes ahead of the current time (default: 10 minutes ahead). Logs with future timestamps are re-timestamped to the processing time rather than dropped.

Common triggers in Kubernetes deployments:

Application logs in local time, agent assumes UTC.
System clock skew between the host and the server.
Timestamp pattern not recognized > observation timestamp assigned > may be out of bounds.

Resolution

Create a Timestamp/splitting configuration rule with an explicit date pattern and timezone for the affected log source.

Accepted timezone values include any IANA timezone identifier (such as Europe/Warsaw, America/Los_Angeles, Etc/UTC), plus the special values Local time zone (agent host timezone) and Default time zone settings (tenant-level default).

For details on additional triggers (such as JSON fields containing misleading date strings) and resolution steps, see Troubleshooting missing logs > Entries dropped — timestamp out of bounds.

What's next

For boundary issues (multi-line entries split incorrectly, entries merged), see Troubleshooting log record boundary detection.
For broader "missing logs" troubleshooting beyond Kubernetes, see Troubleshooting missing logs.
For ingest delay (logs arrive but late), see Troubleshooting log ingest delay.
For JSON parsing issues (wrong timestamp from JSON field, severity mapping), see Troubleshooting JSON log parsing.
For log rotation issues (gaps at rotation boundaries, missing rotated files), see Troubleshooting log rotation issues with Log Module.