topic Re: Inconsistent Logs - Dynatrace Managed in Dynatrace Managed Q&A

Inconsistent Logs from containers on Dynatrace Managed

mattcanty — Mon, 05 Jan 2026 10:56:20 GMT

Versions

Dynatrace Managed 1.324.60.20251007-165512
dynatrace-operator:v1.7.1
oneagent:1.323.46-raw

Problem

We're seeing inconsistent logs from our containers. We get logs fine from the Kubernetes API. But I can only get logs from containers sometimes. I have a test pod called log-writer-pod. It writes a message every 31 seconds with a JSON body containing timestamp and count:

▶ k logs log-writer-pod Defaulted container "log-writer" out of: log-writer, dynatrace-operator (init) Starting log writer pod... {"timestamp":"2025-12-17 11:50:48","level":"INFO","message":"Log entry number 1","pod":"log-writer-pod","counter":1} {"timestamp":"2025-12-17 11:51:19","level":"INFO","message":"Log entry number 2","pod":"log-writer-pod","counter":2} {"timestamp":"2025-12-17 11:51:50","level":"INFO","message":"Log entry number 3","pod":"log-writer-pod","counter":3} {"timestamp":"2025-12-17 11:52:21","level":"INFO","message":"Log entry number 4","pod":"log-writer-pod","counter":4} ... and so on ...

In the Dynatrace UI I see these logs get through only sometimes. Like from yesterday until today we got 2.4k of them, then they stopped at 0816 this morning. Since then, despite restarting oneagent and the log-writer-pod pods I have only managed to get logs from it once again. And then it only ingested about 20 of them whilst the pod itself continued writing logs.

Here is our pod being described just now. Showing injection and no errors.

Name: log-writer-pod Namespace: hosting Priority: 0 Service Account: default Node: <redact>-bw2sp-worker-7vwjs/<redact> Start Time: Wed, 17 Dec 2025 11:50:44 +0000 Labels: app=log-writer environment=dev Annotations: dynakube.dynatrace.com/injected: true k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["<redact>/23"],"mac_address":"<redact>","gateway_ips":["<redact>"],"routes":[{"dest":"<redact>... k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "<redact>" ], "mac": "<redact>", "default": true, "dns": {} }] metadata-enrichment.dynatrace.com/injected: true metadata.dynatrace.com: {} metadata.dynatrace.com/k8s.workload.kind: pod metadata.dynatrace.com/k8s.workload.name: log-writer-pod oneagent.dynatrace.com/inject: true oneagent.dynatrace.com/injected: true openshift.io/scc: anyuid Status: Running IP: <redact> IPs: IP: <redact> Init Containers: dynatrace-operator: Container ID: cri-o://7e6d08f59fdff895843f623dd749017af7f8c991a2b81a63389e202375cc6c5c Image: registry.connect.redhat.com/dynatrace/dynatrace-operator:v1.7.1 Image ID: registry.connect.redhat.com/dynatrace/dynatrace-operator@sha256:c9887132746ba6c387dbdc3d12defe488963a7ddd75ea961ccd413c4f3b547dc Port: <none> Host Port: <none> Args: bootstrap --config-directory=/mnt/config --input-directory=/mnt/input --suppress-error --source=/opt/dynatrace/oneagent --target=/mnt/bin --install-path=/opt/dynatrace/oneagent-paas --fullstack --tenant=<redact> --attribute=k8s.workload.kind=pod --attribute=k8s.workload.name=log-writer-pod --metadata-enrichment --attribute-container={"container_image.repository":"busybox","container_image.tags":"latest","k8s.container.name":"log-writer"} --attribute=k8s.pod.name=$(K8S_PODNAME) --attribute=k8s.pod.uid=$(K8S_PODUID) --attribute=k8s.node.name=$(K8S_NODE_NAME) --attribute=k8s.namespace.name=hosting --attribute=k8s.cluster.uid=<redact> --attribute=k8s.cluster.name=<redact> --attribute=dt.entity.kubernetes_cluster=KUBERNETES_CLUSTER-<redact> State: Terminated Reason: Completed Exit Code: 0 Started: Wed, 17 Dec 2025 11:50:44 +0000 Finished: Wed, 17 Dec 2025 11:50:45 +0000 Ready: True Restart Count: 0 Limits: cpu: 100m memory: 60Mi Requests: cpu: 30m memory: 30Mi Environment: K8S_PODNAME: log-writer-pod (v1:metadata.name) K8S_PODUID: (v1:metadata.uid) K8S_NODE_NAME: (v1:spec.nodeName) Mounts: /mnt/bin from oneagent-bin (ro) /mnt/config from dynatrace-config (rw) /mnt/input from dynatrace-input (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tmlt2 (ro) Containers: log-writer: Container ID: cri-o://d9fce9802290a1e9b9f7b81328f9673dc9367df0da2ad6a2077acdb0857f6ebc Image: busybox:latest Image ID: docker.io/library/busybox@sha256:870e815c3a50dd0f6b40efddb319c72c32c3ee340b5a3e8945904232ccd12f44 Port: <none> Host Port: <none> Command: /bin/sh Args: -c echo "Starting log writer pod..." counter=1 while true; do timestamp=$(date '+%Y-%m-%d %H:%M:%S') echo "{\"timestamp\":\"$timestamp\",\"level\":\"INFO\",\"message\":\"Log entry number $counter\",\"pod\":\"log-writer-pod\",\"counter\":$counter}" counter=$((counter + 1)) sleep 31 done State: Running Started: Wed, 17 Dec 2025 11:50:48 +0000 Ready: True Restart Count: 0 Environment: DT_DEPLOYMENT_METADATA: orchestration_tech=Operator-cloud_native_fullstack;script_version=v1.7.1;orchestrator_id=<redact> LD_PRELOAD: /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so DT_STORAGE: /var/lib/dynatrace/oneagent Mounts: /etc/ld.so.preload from dynatrace-config (rw,path="oneagent/ld.so.preload") /opt/dynatrace/oneagent-paas from oneagent-bin (ro) /var/lib/dynatrace from dynatrace-config (rw,path="log-writer") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tmlt2 (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: kube-api-access-tmlt2: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> oneagent-bin: Type: CSI (a Container Storage Interface (CSI) volume source) Driver: csi.oneagent.dynatrace.com FSType: ReadOnly: true VolumeAttributes: dynakube=<redact> mode=app retryTimeout=10m0s dynatrace-input: Type: Projected (a volume that contains injected data from multiple sources) SecretName: dynatrace-bootstrapper-config SecretOptionalName: 0xc001462a95 SecretName: dynatrace-bootstrapper-certs SecretOptionalName: 0xc001462a96 dynatrace-config: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 48m default-scheduler Successfully assigned hosting/log-writer-pod to <redact>-i-bw2sp-worker-7vwjs Normal AddedInterface 48m multus Add eth0 [<redact>/23] from ovn-kubernetes Normal Pulled 48m kubelet Container image "registry.connect.redhat.com/dynatrace/dynatrace-operator:v1.7.1" already present on machine Normal Created 48m kubelet Created container: dynatrace-operator Normal Started 48m kubelet Started container dynatrace-operator Normal Pulling 48m kubelet Pulling image "busybox:latest" Normal Pulled 48m kubelet Successfully pulled image "busybox:latest" in 2.032s (2.032s including waiting). Image size: 4680654 bytes. Normal Created 48m kubelet Created container: log-writer Normal Started 48m kubelet Started container log-writer

Here I can see that Dynatrace is being injected by the dynatrace-operator container:

▶ k logs log-writer-pod -c dynatrace-operator {"level":"info","ts":"2025-12-17T11:50:45.274Z","logger":"bootstrap","msg":"starting configuration","config-directory":"/mnt/config","input-directory":"/mnt/input"} {"level":"info","ts":"2025-12-17T11:50:45.274Z","logger":"bootstrap","msg":"configuring ld.so.preload","config-directory":"/mnt/config","install-path":"/opt/dynatrace/oneagent-paas"} {"level":"info","ts":"2025-12-17T11:50:45.274Z","logger":"bootstrap","msg":"starting to configure the container","path":"/mnt/config/log-writer"} {"level":"info","ts":"2025-12-17T11:50:45.274Z","logger":"bootstrap","msg":"creating ruxitagentproc.conf","source":"/mnt/bin/agent/conf/ruxitagentproc.conf","destination":"/mnt/config/log-writer/oneagent/agent/config/ruxitagentproc.conf"} {"level":"info","ts":"2025-12-17T11:50:45.275Z","logger":"bootstrap","msg":"configuring container.conf","config-directory":"/mnt/config/log-writer"} {"level":"info","ts":"2025-12-17T11:50:45.275Z","logger":"bootstrap","msg":"fullstack flag detected, configuring accordingly","tenant":"8d6bd164-b1be-44b0-ba2e-563c97cfca43"} {"level":"info","ts":"2025-12-17T11:50:45.275Z","logger":"bootstrap","msg":"input file not present, skipping curl options configuration","path":"/mnt/input/initial-connect-retry"} {"level":"info","ts":"2025-12-17T11:50:45.275Z","logger":"bootstrap","msg":"creating cert file","path":"/mnt/config/log-writer/oneagent/agent/customkeys/custom.pem"} {"level":"info","ts":"2025-12-17T11:50:45.275Z","logger":"bootstrap","msg":"finished oneagent configuration","config-directory":"/mnt/config","input-directory":"/mnt/input"} {"level":"info","ts":"2025-12-17T11:50:45.275Z","logger":"bootstrap","msg":"starting enrichment","config-directory":"/mnt/config","input-directory":"/mnt/input"} {"level":"info","ts":"2025-12-17T11:50:45.275Z","logger":"bootstrap","msg":"starting to enrich the container","path":"/mnt/config/log-writer"} {"level":"info","ts":"2025-12-17T11:50:45.275Z","logger":"bootstrap","msg":"finished enrichment","config-directory":"/mnt/config","input-directory":"/mnt/input"}

I have the following UI settings:

Settings
- Log Monitoring
  - Set up log ingest
    - ✅ Ingest logs from custom log sources
    - ✅ Kubernetes default namespaces logs
    - ✅ Windows system, application, and security logs
    - ✅ Linux system logs
    - ✅ Ingest all detected log sources
  - Log ingest rules
    - ✅Capture Container Output (custom) - Log source is any of: Container Output
    - ✅[Built-in] Ingest all logs
  - Log drop rules
    - ✅Matcher: `content="GET /v1/sys/health"
  - Log module feature flags
    - ✅Collect all container logs requires OneAgent 1.309*
    - ✅ Collect Journald logs
  - Advanced log settings
    - ✅Detect open log files
    - ✅Detect systel logs
    - ✅Detect logs of containerized applications
    - ✅Detect IIS logs
    - ❌Detect logs on network file systems
    - ❌Allow OneAgent to monitor Dynatrace logs - I've just enabled this to see if it tells me anything
    - ✅Detect container time zones
    - Maximum of log group instances per entity limit - count - 200

Our DynaKube, it's probably overly redacted, happy to provide specific items from it if needed.

apiVersion: v1 items: - apiVersion: dynatrace.com/v1beta5 kind: DynaKube metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | <redact> creationTimestamp: "2025-10-19T08:47:10Z" generation: 1 name: <redact> namespace: dynatrace resourceVersion: "50852550" uid: <redact> spec: activeGate: capabilities: - routing - kubernetes-monitoring - dynatrace-api resources: limits: cpu: "1" memory: 1536Mi requests: cpu: 500m memory: 512Mi apiUrl: https://<redact>/e/<redact>/api metadataEnrichment: enabled: true namespaceSelector: {} oneAgent: cloudNativeFullStack: namespaceSelector: {} oneAgentResources: {} tolerations: - effect: NoSchedule key: node-role.kubernetes.io/master operator: Exists - effect: NoSchedule key: node-role.kubernetes.io/control-plane operator: Exists hostGroup: <redact> skipCertCheck: true templates: extensionExecutionController: imageRef: {} resources: {} kspmNodeConfigurationCollector: imageRef: {} nodeAffinity: {} resources: {} otelCollector: imageRef: {} resources: {} status: activeGate: connectionInfoStatus: endpoints: https://<redact>:443/communication,https://<redact>:443/communication tenantTokenHash: "935154033" tenantUUID: <redact> imageID: <redact>/linux/activegate:1.323.29-raw lastProbeTimestamp: "2025-12-17T12:24:19Z" serviceIPs: - <redact> source: tenant-registry version: 1.323.29.20251002-173724 codeModules: lastProbeTimestamp: "2025-12-17T12:24:19Z" source: tenant-registry version: 1.323.46.20251015-192739 conditions: - lastTransitionTime: "2025-10-19T08:55:45Z" message: settings.read optional scope available reason: ScopePresent status: "True" type: ApiTokenSettingsRead - lastTransitionTime: "2025-10-19T08:55:45Z" message: settings.write optional scope available reason: ScopePresent status: "True" type: ApiTokenSettingsWrite - lastTransitionTime: "2025-11-06T17:14:24Z" message: Token ready reason: TokenReady status: "True" type: Tokens - lastTransitionTime: "2025-12-17T12:24:19Z" message: <redact>-activegate-tenant-secret created reason: SecretCreated status: "True" type: ActiveGateConnectionInfo - lastTransitionTime: "2025-12-17T12:24:19Z" message: Kubernetes Cluster MEID is up to date reason: StatusUpdated status: "True" type: MonitoredEntity - lastTransitionTime: "2025-12-17T12:24:19Z" message: <redact>-oneagent-tenant-secret created reason: SecretCreated status: "True" type: OneAgentConnectionInfo - lastTransitionTime: "2025-10-21T02:25:38Z" message: Version verified for component. reason: Verified status: "True" type: ActiveGateVersion - lastTransitionTime: "2025-12-17T12:24:19Z" message: <redact>-pull-secret created reason: SecretCreated status: "True" type: PullSecret - lastTransitionTime: "2025-12-16T09:01:37Z" message: secret created 1 day(s) ago, token:dt0g02.<redact> reason: SecretCreated status: "True" type: ActiveGateAuthTokenSecret - lastTransitionTime: "2025-10-21T05:03:22Z" message: Version verified for component. reason: Verified status: "True" type: CodeModulesVersion - lastTransitionTime: "2025-10-19T09:16:58Z" message: Namespaces mapped and secrets created reason: SecretsCreated status: "True" type: CodeModulesInjection - lastTransitionTime: "2025-12-17T12:24:19Z" message: Metadata-enrichment rules are up-to-date in the status reason: StatusUpdated status: "True" type: MetadataEnrichmentRules - lastTransitionTime: "2025-10-19T09:16:58Z" message: Namespaces mapped and secrets created reason: SecretsCreated status: "True" type: MetadataEnrichment - lastTransitionTime: "2025-12-17T12:24:19Z" message: <redact>-bootstrapper-config created/updated reason: SecretCreatedOrUpdated status: "True" type: BootstrapperConfig - lastTransitionTime: "2025-10-19T09:16:59Z" message: <redact>-bootstrapper-config created/updated reason: SecretCreatedOrUpdated status: "True" type: BootstrapperCertsConfig - lastTransitionTime: "2025-10-21T02:25:39Z" message: Version verified for component. reason: Verified status: "True" type: OneAgentVersion - lastTransitionTime: "2025-10-21T00:39:59Z" message: <redact>-activegate created reason: StatefulSetCreated status: "True" type: ActiveGateStatefulSet - lastTransitionTime: "2025-10-21T00:40:01Z" message: The OneAgent DaemonSet was created successfully. reason: DaemonSetCreated status: "True" type: OneAgentDaemonSet dynatraceApi: lastTokenScopeRequest: "2025-12-17T12:24:19Z" kspm: {} kubeSystemUUID: <redact> kubernetesClusterMEID: KUBERNETES_CLUSTER-<redact> kubernetesClusterName: <redact> metadataEnrichment: rules: - source: app target: dt.cost.product type: LABEL - source: dt.owner target: dt.cost.costcenter type: LABEL oneAgent: connectionInfoStatus: communicationHosts: - host: <redact> port: 443 protocol: https - host: cib-dynaag-02 port: 9999 protocol: https - host: <redact> port: 9999 protocol: https - host: <redact>-activegate.dynatrace port: 443 protocol: https - host: <redact> port: 443 protocol: https - host: cib-ocp-dev-activegate.dynatrace port: 443 protocol: https - host: <redact>-01 port: 9999 protocol: https - host: <redact> port: 9999 protocol: https - host: <redact>-dev-activegate.dynatrace port: 443 protocol: https - host: <redact> port: 443 protocol: https - host: <redact> port: 443 protocol: https - host: <redact> port: 443 protocol: https - host: <redact> port: 443 protocol: https - host: cib-ocp-shared-activegate.dynatrace port: 443 protocol: https - host: <redact> port: 443 protocol: https - host: <redact>-dev-trusted-activegate.dynatrace port: 443 protocol: https endpoints: https://<redact>/communication;https://<redact>/communication;https://<redact>:9999/communication;https://<redact>:9999/communication;https://<redact>:443/communication;https://<redact>-activegate.dynatrace:443/communication;https://<redact>:443/communication;https://<redact>-activegate.dynatrace:443/communication;https://<redact>:9999/communication;https://<redact>:9999/communication;https://<redact>:443/communication;https://<redact>-activegate.dynatrace:443/communication;https://<redact>:443/communication;https://<redact>-activegate.dynatrace:443/communication;https://<redact>:443/communication;https://<redact>:443/communication tenantTokenHash: "935154033" tenantUUID: <redact> healthcheck: Interval: 10000000000 Retries: 3 StartPeriod: 1200000000000 Test: - /usr/bin/watchdog-healthcheck64 Timeout: 30000000000 imageID: <redact>/linux/oneagent:1.323.46-raw instances: <redact>-dev-bw2sp-master-0: ipAddress: 10.100.110.16 podName: <redact>-oneagent-jtlcd <redact>-dev-bw2sp-master-1: ipAddress: 10.100.110.15 podName: <redact>-oneagent-gqh4r <redact>-dev-bw2sp-master-2: ipAddress: 10.100.110.17 podName: <redact>-oneagent-45jkz <redact>-dev-bw2sp-worker-2dh8h: ipAddress: 10.100.110.23 podName: <redact>-oneagent-hxvlr <redact>-dev-bw2sp-worker-52qvp: ipAddress: 10.100.110.31 podName: <redact>-oneagent-z2pkc <redact>-dev-bw2sp-worker-5sg7n: ipAddress: 10.100.110.20 podName: <redact>-oneagent-5f4s2 <redact>-dev-bw2sp-worker-5zjjr: ipAddress: 10.100.110.19 podName: <redact>-oneagent-phd58 <redact>-dev-bw2sp-worker-7vwjs: ipAddress: 10.100.110.32 podName: <redact>-oneagent-kvkqm <redact>-dev-bw2sp-worker-8xhlv: ipAddress: 10.100.110.30 podName: <redact>-oneagent-hg2rk <redact>-dev-bw2sp-worker-9mk4g: ipAddress: 10.100.110.34 podName: <redact>-oneagent-kr5kv <redact>-dev-bw2sp-worker-b2lfg: ipAddress: 10.100.110.24 podName: <redact>-oneagent-bgwd5 <redact>-dev-bw2sp-worker-br875: ipAddress: 10.100.110.35 podName: <redact>-oneagent-b2f5l <redact>-dev-bw2sp-worker-dn95b: ipAddress: 10.100.110.28 podName: <redact>-oneagent-5flpk <redact>-dev-bw2sp-worker-dqp9q: ipAddress: 10.100.110.29 podName: <redact>-oneagent-742t5 <redact>-dev-bw2sp-worker-gmdqm: ipAddress: 10.100.110.25 podName: <redact>-oneagent-wjbwl <redact>-dev-bw2sp-worker-hrvht: ipAddress: 10.100.110.21 podName: <redact>-oneagent-z4xb2 <redact>-dev-bw2sp-worker-sklx7: ipAddress: 10.100.110.22 podName: <redact>-oneagent-gv5hk <redact>-dev-bw2sp-worker-tzqvv: ipAddress: 10.100.110.26 podName: <redact>-oneagent-bv7ql <redact>-dev-bw2sp-worker-xwgqs: ipAddress: 10.100.110.27 podName: <redact>-oneagent-724hr <redact>-dev-bw2sp-worker-zcgfk: ipAddress: 10.100.110.33 podName: <redact>-oneagent-9vcck lastInstanceStatusUpdate: "2025-12-17T12:24:19Z" lastProbeTimestamp: "2025-12-17T12:24:19Z" source: tenant-registry version: 1.323.46.20251015-192739 phase: Running updatedTimestamp: "2025-12-17T12:24:19Z" kind: List metadata: resourceVersion: ""

Our pods

▶ k get pods NAME READY STATUS RESTARTS AGE dynatrace-oneagent-csi-driver-2rnx5 4/4 Running 0 16d dynatrace-oneagent-csi-driver-4swzt 4/4 Running 8 59d dynatrace-oneagent-csi-driver-5x85r 4/4 Running 0 6h20m dynatrace-oneagent-csi-driver-6c89w 4/4 Running 8 59d dynatrace-oneagent-csi-driver-7jckb 4/4 Running 0 16d dynatrace-oneagent-csi-driver-826lq 4/4 Running 0 16d dynatrace-oneagent-csi-driver-82x2h 4/4 Running 8 59d dynatrace-oneagent-csi-driver-85fmz 4/4 Running 8 59d dynatrace-oneagent-csi-driver-bz5gv 4/4 Running 8 59d dynatrace-oneagent-csi-driver-crbfh 4/4 Running 8 59d dynatrace-oneagent-csi-driver-j2xk2 4/4 Running 0 16d dynatrace-oneagent-csi-driver-mspq9 4/4 Running 8 59d dynatrace-oneagent-csi-driver-pxqc8 4/4 Running 0 16d dynatrace-oneagent-csi-driver-qfmck 4/4 Running 8 59d dynatrace-oneagent-csi-driver-s7rlp 4/4 Running 0 16d dynatrace-oneagent-csi-driver-tc2bc 4/4 Running 8 59d dynatrace-oneagent-csi-driver-v4djn 4/4 Running 8 59d dynatrace-oneagent-csi-driver-w6859 4/4 Running 8 59d dynatrace-oneagent-csi-driver-x699g 4/4 Running 0 16d dynatrace-oneagent-csi-driver-z7fv9 4/4 Running 8 59d dynatrace-operator-7cbb87cc9c-ng64z 1/1 Running 0 26h dynatrace-webhook-55c7d8b74b-x9vq9 1/1 Running 0 16d dynatrace-webhook-55c7d8b74b-zlk2q 1/1 Running 0 16d <redact>-activegate-0 1/1 Running 0 27h <redact>-oneagent-45jkz 1/1 Running 0 69m <redact>-oneagent-5f4s2 1/1 Running 0 67m <redact>-oneagent-5flpk 1/1 Running 0 83m <redact>-oneagent-724hr 1/1 Running 0 65m <redact>-oneagent-742t5 1/1 Running 0 55m <redact>-oneagent-9vcck 1/1 Running 0 63m <redact>-oneagent-b2f5l 1/1 Running 0 82m <redact>-oneagent-bgwd5 1/1 Running 0 76m <redact>-oneagent-bv7ql 1/1 Running 0 80m <redact>-oneagent-gqh4r 1/1 Running 0 71m <redact>-oneagent-gv5hk 1/1 Running 0 56m <redact>-oneagent-hg2rk 1/1 Running 0 58m <redact>-oneagent-hxvlr 1/1 Running 0 62m <redact>-oneagent-jtlcd 1/1 Running 0 78m <redact>-oneagent-kr5kv 1/1 Running 0 51m <redact>-oneagent-kvkqm 1/1 Running 0 53m <redact>-oneagent-phd58 1/1 Running 0 72m <redact>-oneagent-wjbwl 1/1 Running 0 85m <redact>-oneagent-z2pkc 1/1 Running 0 60m <redact>-oneagent-z4xb2 1/1 Running 0 74m

Not sure what else to offer at this moment. Very much going round in circles and I'm sure it will be a trivial issue.

I have followed this:
https://community.dynatrace.com/t5/Troubleshooting/Logs-Ingest-on-K8s-with-Dynatrace/ta-p/285827

Re: Inconsistent Logs - Dynatrace Managed

mattcanty — Wed, 17 Dec 2025 15:15:19 GMT

Twice today i've managed to get logs through.

Unchanged pod spec.

Same namespace.

Once 22 messages made it, the next time 43 messages. As stated, it's pushing a log message every 31 seconds. They should be flowing continuously.

Re: Inconsistent Logs - Dynatrace Managed

jgrant — Wed, 17 Dec 2025 17:35:14 GMT

Hi Matt,

If this is Dynatrace Managed and the logs flow inconsistently, do you see any trimmed events in your Cluster Management Console?

Re: Inconsistent Logs - Dynatrace Managed

Julius_Loman — Wed, 17 Dec 2025 21:25:08 GMT

The most common reason, since it's Dynatrace Managed, is the limit of log message for the environment. Default is typically too low, and the also the cluster also has some log processing power which needs to be distributed across the environments.

First, check what you have in the Cluster Management Console for the environment - in this case, I have 0, so no log messages will be processed in this environment

Then in the environment, check two metrics in Data Explorer:
Server - Log And Events Monitoring - Events Rejected Count
Server - Log And Events Monitoring - Events Incoming Count

If you have any events rejected, you are receiving more events than the environment is allowed to process or the cluster itself can process and you need to raise the environment limit and maybe also scale your cluster.

See more https://docs.dynatrace.com/managed/shortlink/log-monitoring-limits#lmc-limit-logs-per-minute-managed and this https://docs.dynatrace.com/managed/shortlink/lmc-troubleshooting

Re: Inconsistent Logs - Dynatrace Managed

mattcanty — Thu, 18 Dec 2025 09:02:09 GMT

I will make enquiries with the service provider.

Re: Inconsistent Logs - Dynatrace Managed

mattcanty — Thu, 18 Dec 2025 09:39:07 GMT

Will Log Drop Rules help with this or should things be shut off at source?

Re: Inconsistent Logs - Dynatrace Managed

Julius_Loman — Thu, 18 Dec 2025 10:06:32 GMT

Nope. As it says

Logs matched by rules below are not stored in the logs backend. They are still processed by log metrics and log events. Learn more by visiting our documentation.

So those are ingested and are processed in the queue. They are not stored. Those matching Log drop rules count toward the environment limit. You need to set the log ingest rules properly not to collect log entries which you don't want to or set the environment limit high enough to process the volume.

Re: Inconsistent Logs - Dynatrace Managed

mattcanty — Thu, 18 Dec 2025 11:13:00 GMT

We want logs though, not events and metrics. I don't quite understand right now why these things are linked. If there were millions of events and a handful of log messages, I'd expect all log messages to be captured.

Thinking out loud. Don't worry about this.

I'm working hard to reduce the amount of events captured, and so far have only managed to increase the number of events by turning things off.

I will report back when there is a tangible result.

Re: Inconsistent Logs - Dynatrace Managed

mattcanty — Thu, 18 Dec 2025 11:44:20 GMT

If Collect all container logs in Kubernetes Settings is enabled but [Built In] Ingest All Logs is disabled.

Are container logs enabled, or disabled.

Re: Inconsistent Logs - Dynatrace Managed

Julius_Loman — Thu, 18 Dec 2025 13:12:49 GMT

First rule matches, so it depends on the order and types of rules.

Re: Inconsistent Logs - Dynatrace Managed

jgrant — Thu, 18 Dec 2025 16:34:01 GMT

To add, Collect all container logs is a log module feature flag and [Built In] Ingest All Logs is a log ingest rule. This log module feature flag enables a new log container detector, but LogAnalytics still needs a matching log ingest rule to send discovered logs to Dynatrace. This can be a relevant Built-in rule, an unconditional rule or any matching rule. To sends logs to storage, the logs need to match any rule of type Include first. To determine if a log source (aka container output on k8s) is allowed, LogAnalytics stops checking the rules after the first match, from host to environment scopes, top-down in the ordered log ingest rules list.

If you enabled Collect all container logs feature flag on a cluster, then you may need to add dynakube parameter `.spec.logMonitoring: {}` to your Dynakube, so LogAnalytics is allowed to ingest logs. Dynatrace Operator will add an unconditional rule to your k8s cluster Settings > Log monitoring > Log ingest rules. If that rule was created and was not deleted, you may disable it, but you will need to enable or add log ingest rules to ingest logs from the namespaces, pods or containers that you want to monitor.