Dynatrace Managed Q&A
cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Looking to upgrade from Dynatrace Managed to SaaS? See how

Inconsistent Logs - Dynatrace Managed

mattcanty
Newcomer_

Versions

Dynatrace Managed 1.324.60.20251007-165512
dynatrace-operator:v1.7.1
oneagent:1.323.46-raw

 

Problem

We're seeing inconsistent logs from our containers. We get logs fine from the Kubernetes API. But I can only get logs from containers sometimes. I have a test pod called log-writer-pod. It writes a message every 31 seconds with a JSON body containing timestamp and count:

▶ k logs log-writer-pod
Defaulted container "log-writer" out of: log-writer, dynatrace-operator (init)
Starting log writer pod...
{"timestamp":"2025-12-17 11:50:48","level":"INFO","message":"Log entry number 1","pod":"log-writer-pod","counter":1}
{"timestamp":"2025-12-17 11:51:19","level":"INFO","message":"Log entry number 2","pod":"log-writer-pod","counter":2}
{"timestamp":"2025-12-17 11:51:50","level":"INFO","message":"Log entry number 3","pod":"log-writer-pod","counter":3}
{"timestamp":"2025-12-17 11:52:21","level":"INFO","message":"Log entry number 4","pod":"log-writer-pod","counter":4}
... and so on ...

In the Dynatrace UI I see these logs get through only sometimes. Like from yesterday until today we got 2.4k of them, then they stopped at 0816 this morning. Since then, despite restarting oneagent and the log-writer-pod pods I have only managed to get logs from it once again. And then it only ingested about 20 of them whilst the pod itself continued writing logs.

Here is our pod being described just now. Showing injection and no errors.

Name:             log-writer-pod
Namespace:        hosting
Priority:         0
Service Account:  default
Node:             <redact>-bw2sp-worker-7vwjs/<redact>
Start Time:       Wed, 17 Dec 2025 11:50:44 +0000
Labels:           app=log-writer
                  environment=dev
Annotations:      dynakube.dynatrace.com/injected: true
                  k8s.ovn.org/pod-networks:
                    {"default":{"ip_addresses":["<redact>/23"],"mac_address":"<redact>","gateway_ips":["<redact>"],"routes":[{"dest":"<redact>...
                  k8s.v1.cni.cncf.io/network-status:
                    [{
                        "name": "ovn-kubernetes",
                        "interface": "eth0",
                        "ips": [
                            "<redact>"
                        ],
                        "mac": "<redact>",
                        "default": true,
                        "dns": {}
                    }]
                  metadata-enrichment.dynatrace.com/injected: true
                  metadata.dynatrace.com: {}
                  metadata.dynatrace.com/k8s.workload.kind: pod
                  metadata.dynatrace.com/k8s.workload.name: log-writer-pod
                  oneagent.dynatrace.com/inject: true
                  oneagent.dynatrace.com/injected: true
                  openshift.io/scc: anyuid
Status:           Running
IP:               <redact>
IPs:
  IP:  <redact>
Init Containers:
  dynatrace-operator:
    Container ID:  cri-o://7e6d08f59fdff895843f623dd749017af7f8c991a2b81a63389e202375cc6c5c
    Image:         registry.connect.redhat.com/dynatrace/dynatrace-operator:v1.7.1
    Image ID:      registry.connect.redhat.com/dynatrace/dynatrace-operator@sha256:c9887132746ba6c387dbdc3d12defe488963a7ddd75ea961ccd413c4f3b547dc
    Port:          <none>
    Host Port:     <none>
    Args:
      bootstrap
      --config-directory=/mnt/config
      --input-directory=/mnt/input
      --suppress-error
      --source=/opt/dynatrace/oneagent
      --target=/mnt/bin
      --install-path=/opt/dynatrace/oneagent-paas
      --fullstack
      --tenant=<redact>
      --attribute=k8s.workload.kind=pod
      --attribute=k8s.workload.name=log-writer-pod
      --metadata-enrichment
      --attribute-container={"container_image.repository":"busybox","container_image.tags":"latest","k8s.container.name":"log-writer"}
      --attribute=k8s.pod.name=$(K8S_PODNAME)
      --attribute=k8s.pod.uid=$(K8S_PODUID)
      --attribute=k8s.node.name=$(K8S_NODE_NAME)
      --attribute=k8s.namespace.name=hosting
      --attribute=k8s.cluster.uid=<redact>
      --attribute=k8s.cluster.name=<redact>
      --attribute=dt.entity.kubernetes_cluster=KUBERNETES_CLUSTER-<redact>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 17 Dec 2025 11:50:44 +0000
      Finished:     Wed, 17 Dec 2025 11:50:45 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  60Mi
    Requests:
      cpu:     30m
      memory:  30Mi
    Environment:
      K8S_PODNAME:    log-writer-pod (v1:metadata.name)
      K8S_PODUID:      (v1:metadata.uid)
      K8S_NODE_NAME:   (v1:spec.nodeName)
    Mounts:
      /mnt/bin from oneagent-bin (ro)
      /mnt/config from dynatrace-config (rw)
      /mnt/input from dynatrace-input (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tmlt2 (ro)
Containers:
  log-writer:
    Container ID:  cri-o://d9fce9802290a1e9b9f7b81328f9673dc9367df0da2ad6a2077acdb0857f6ebc
    Image:         busybox:latest
    Image ID:      docker.io/library/busybox@sha256:870e815c3a50dd0f6b40efddb319c72c32c3ee340b5a3e8945904232ccd12f44
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      echo "Starting log writer pod..."
      counter=1
      while true; do
        timestamp=$(date '+%Y-%m-%d %H:%M:%S')
        echo "{\"timestamp\":\"$timestamp\",\"level\":\"INFO\",\"message\":\"Log entry number $counter\",\"pod\":\"log-writer-pod\",\"counter\":$counter}"
        counter=$((counter + 1))
        sleep 31
      done
      
    State:          Running
      Started:      Wed, 17 Dec 2025 11:50:48 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      DT_DEPLOYMENT_METADATA:  orchestration_tech=Operator-cloud_native_fullstack;script_version=v1.7.1;orchestrator_id=<redact>
      LD_PRELOAD:              /opt/dynatrace/oneagent-paas/agent/lib64/liboneagentproc.so
      DT_STORAGE:              /var/lib/dynatrace/oneagent
    Mounts:
      /etc/ld.so.preload from dynatrace-config (rw,path="oneagent/ld.so.preload")
      /opt/dynatrace/oneagent-paas from oneagent-bin (ro)
      /var/lib/dynatrace from dynatrace-config (rw,path="log-writer")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tmlt2 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  kube-api-access-tmlt2:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
  oneagent-bin:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            csi.oneagent.dynatrace.com
    FSType:            
    ReadOnly:          true
    VolumeAttributes:      dynakube=<redact>
                           mode=app
                           retryTimeout=10m0s
  dynatrace-input:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          dynatrace-bootstrapper-config
    SecretOptionalName:  0xc001462a95
    SecretName:          dynatrace-bootstrapper-certs
    SecretOptionalName:  0xc001462a96
  dynatrace-config:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      
    SizeLimit:   <unset>
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Normal  Scheduled       48m   default-scheduler  Successfully assigned hosting/log-writer-pod to <redact>-i-bw2sp-worker-7vwjs
  Normal  AddedInterface  48m   multus             Add eth0 [<redact>/23] from ovn-kubernetes
  Normal  Pulled          48m   kubelet            Container image "registry.connect.redhat.com/dynatrace/dynatrace-operator:v1.7.1" already present on machine
  Normal  Created         48m   kubelet            Created container: dynatrace-operator
  Normal  Started         48m   kubelet            Started container dynatrace-operator
  Normal  Pulling         48m   kubelet            Pulling image "busybox:latest"
  Normal  Pulled          48m   kubelet            Successfully pulled image "busybox:latest" in 2.032s (2.032s including waiting). Image size: 4680654 bytes.
  Normal  Created         48m   kubelet            Created container: log-writer
  Normal  Started         48m   kubelet            Started container log-writer

Here I can see that Dynatrace is being injected by the dynatrace-operator container:

▶ k logs log-writer-pod -c dynatrace-operator
{"level":"info","ts":"2025-12-17T11:50:45.274Z","logger":"bootstrap","msg":"starting configuration","config-directory":"/mnt/config","input-directory":"/mnt/input"}
{"level":"info","ts":"2025-12-17T11:50:45.274Z","logger":"bootstrap","msg":"configuring ld.so.preload","config-directory":"/mnt/config","install-path":"/opt/dynatrace/oneagent-paas"}
{"level":"info","ts":"2025-12-17T11:50:45.274Z","logger":"bootstrap","msg":"starting to configure the container","path":"/mnt/config/log-writer"}
{"level":"info","ts":"2025-12-17T11:50:45.274Z","logger":"bootstrap","msg":"creating ruxitagentproc.conf","source":"/mnt/bin/agent/conf/ruxitagentproc.conf","destination":"/mnt/config/log-writer/oneagent/agent/config/ruxitagentproc.conf"}
{"level":"info","ts":"2025-12-17T11:50:45.275Z","logger":"bootstrap","msg":"configuring container.conf","config-directory":"/mnt/config/log-writer"}
{"level":"info","ts":"2025-12-17T11:50:45.275Z","logger":"bootstrap","msg":"fullstack flag detected, configuring accordingly","tenant":"8d6bd164-b1be-44b0-ba2e-563c97cfca43"}
{"level":"info","ts":"2025-12-17T11:50:45.275Z","logger":"bootstrap","msg":"input file not present, skipping curl options configuration","path":"/mnt/input/initial-connect-retry"}
{"level":"info","ts":"2025-12-17T11:50:45.275Z","logger":"bootstrap","msg":"creating cert file","path":"/mnt/config/log-writer/oneagent/agent/customkeys/custom.pem"}
{"level":"info","ts":"2025-12-17T11:50:45.275Z","logger":"bootstrap","msg":"finished oneagent configuration","config-directory":"/mnt/config","input-directory":"/mnt/input"}
{"level":"info","ts":"2025-12-17T11:50:45.275Z","logger":"bootstrap","msg":"starting enrichment","config-directory":"/mnt/config","input-directory":"/mnt/input"}
{"level":"info","ts":"2025-12-17T11:50:45.275Z","logger":"bootstrap","msg":"starting to enrich the container","path":"/mnt/config/log-writer"}
{"level":"info","ts":"2025-12-17T11:50:45.275Z","logger":"bootstrap","msg":"finished enrichment","config-directory":"/mnt/config","input-directory":"/mnt/input"}

I have the following UI settings:

  • Settings
    • Log Monitoring
      • Set up log ingest
        •  Ingest logs from custom log sources
        •  Kubernetes default namespaces logs
        •  Windows system, application, and security logs
        •  Linux system logs
        •  Ingest all detected log sources
      • Log ingest rules
        • Capture Container Output (custom) - Log source is any of: Container Output
        • [Built-in] Ingest all logs
      • Log drop rules
        • Matcher: `content="GET /v1/sys/health"
      • Log module feature flags
        • Collect all container logs requires OneAgent 1.309*
        •  Collect Journald logs
      • Advanced log settings
        • Detect open log files
        • Detect systel logs
        • Detect logs of containerized applications
        • Detect IIS logs
        • Detect logs on network file systems
        • Allow OneAgent to monitor Dynatrace logs - I've just enabled this to see if it tells me anything
        • Detect container time zones
        • Maximum of log group instances per entity limit - count - 200

Our DynaKube, it's probably overly redacted, happy to provide specific items from it if needed.

apiVersion: v1
items:
- apiVersion: dynatrace.com/v1beta5
  kind: DynaKube
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        <redact>
    creationTimestamp: "2025-10-19T08:47:10Z"
    generation: 1
    name: <redact>
    namespace: dynatrace
    resourceVersion: "50852550"
    uid: <redact>
  spec:
    activeGate:
      capabilities:
      - routing
      - kubernetes-monitoring
      - dynatrace-api
      resources:
        limits:
          cpu: "1"
          memory: 1536Mi
        requests:
          cpu: 500m
          memory: 512Mi
    apiUrl: https://<redact>/e/<redact>/api
    metadataEnrichment:
      enabled: true
      namespaceSelector: {}
    oneAgent:
      cloudNativeFullStack:
        namespaceSelector: {}
        oneAgentResources: {}
        tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
          operator: Exists
      hostGroup: <redact>
    skipCertCheck: true
    templates:
      extensionExecutionController:
        imageRef: {}
        resources: {}
      kspmNodeConfigurationCollector:
        imageRef: {}
        nodeAffinity: {}
        resources: {}
      otelCollector:
        imageRef: {}
        resources: {}
  status:
    activeGate:
      connectionInfoStatus:
        endpoints: https://<redact>:443/communication,https://<redact>:443/communication
        tenantTokenHash: "935154033"
        tenantUUID: <redact>
      imageID: <redact>/linux/activegate:1.323.29-raw
      lastProbeTimestamp: "2025-12-17T12:24:19Z"
      serviceIPs:
      - <redact>
      source: tenant-registry
      version: 1.323.29.20251002-173724
    codeModules:
      lastProbeTimestamp: "2025-12-17T12:24:19Z"
      source: tenant-registry
      version: 1.323.46.20251015-192739
    conditions:
    - lastTransitionTime: "2025-10-19T08:55:45Z"
      message: settings.read optional scope available
      reason: ScopePresent
      status: "True"
      type: ApiTokenSettingsRead
    - lastTransitionTime: "2025-10-19T08:55:45Z"
      message: settings.write optional scope available
      reason: ScopePresent
      status: "True"
      type: ApiTokenSettingsWrite
    - lastTransitionTime: "2025-11-06T17:14:24Z"
      message: Token ready
      reason: TokenReady
      status: "True"
      type: Tokens
    - lastTransitionTime: "2025-12-17T12:24:19Z"
      message: <redact>-activegate-tenant-secret created
      reason: SecretCreated
      status: "True"
      type: ActiveGateConnectionInfo
    - lastTransitionTime: "2025-12-17T12:24:19Z"
      message: Kubernetes Cluster MEID is up to date
      reason: StatusUpdated
      status: "True"
      type: MonitoredEntity
    - lastTransitionTime: "2025-12-17T12:24:19Z"
      message: <redact>-oneagent-tenant-secret created
      reason: SecretCreated
      status: "True"
      type: OneAgentConnectionInfo
    - lastTransitionTime: "2025-10-21T02:25:38Z"
      message: Version verified for component.
      reason: Verified
      status: "True"
      type: ActiveGateVersion
    - lastTransitionTime: "2025-12-17T12:24:19Z"
      message: <redact>-pull-secret created
      reason: SecretCreated
      status: "True"
      type: PullSecret
    - lastTransitionTime: "2025-12-16T09:01:37Z"
      message: secret created 1 day(s) ago, token:dt0g02.<redact>
      reason: SecretCreated
      status: "True"
      type: ActiveGateAuthTokenSecret
    - lastTransitionTime: "2025-10-21T05:03:22Z"
      message: Version verified for component.
      reason: Verified
      status: "True"
      type: CodeModulesVersion
    - lastTransitionTime: "2025-10-19T09:16:58Z"
      message: Namespaces mapped and secrets created
      reason: SecretsCreated
      status: "True"
      type: CodeModulesInjection
    - lastTransitionTime: "2025-12-17T12:24:19Z"
      message: Metadata-enrichment rules are up-to-date in the status
      reason: StatusUpdated
      status: "True"
      type: MetadataEnrichmentRules
    - lastTransitionTime: "2025-10-19T09:16:58Z"
      message: Namespaces mapped and secrets created
      reason: SecretsCreated
      status: "True"
      type: MetadataEnrichment
    - lastTransitionTime: "2025-12-17T12:24:19Z"
      message: <redact>-bootstrapper-config created/updated
      reason: SecretCreatedOrUpdated
      status: "True"
      type: BootstrapperConfig
    - lastTransitionTime: "2025-10-19T09:16:59Z"
      message: <redact>-bootstrapper-config created/updated
      reason: SecretCreatedOrUpdated
      status: "True"
      type: BootstrapperCertsConfig
    - lastTransitionTime: "2025-10-21T02:25:39Z"
      message: Version verified for component.
      reason: Verified
      status: "True"
      type: OneAgentVersion
    - lastTransitionTime: "2025-10-21T00:39:59Z"
      message: <redact>-activegate created
      reason: StatefulSetCreated
      status: "True"
      type: ActiveGateStatefulSet
    - lastTransitionTime: "2025-10-21T00:40:01Z"
      message: The OneAgent DaemonSet was created successfully.
      reason: DaemonSetCreated
      status: "True"
      type: OneAgentDaemonSet
    dynatraceApi:
      lastTokenScopeRequest: "2025-12-17T12:24:19Z"
    kspm: {}
    kubeSystemUUID: <redact>
    kubernetesClusterMEID: KUBERNETES_CLUSTER-<redact>
    kubernetesClusterName: <redact>
    metadataEnrichment:
      rules:
      - source: app
        target: dt.cost.product
        type: LABEL
      - source: dt.owner
        target: dt.cost.costcenter
        type: LABEL
    oneAgent:
      connectionInfoStatus:
        communicationHosts:
        - host: <redact>
          port: 443
          protocol: https
        - host: cib-dynaag-02
          port: 9999
          protocol: https
        - host: <redact>
          port: 9999
          protocol: https
        - host: <redact>-activegate.dynatrace
          port: 443
          protocol: https
        - host: <redact>
          port: 443
          protocol: https
        - host: cib-ocp-dev-activegate.dynatrace
          port: 443
          protocol: https
        - host: <redact>-01
          port: 9999
          protocol: https
        - host: <redact>
          port: 9999
          protocol: https
        - host: <redact>-dev-activegate.dynatrace
          port: 443
          protocol: https
        - host: <redact>
          port: 443
          protocol: https
        - host: <redact>
          port: 443
          protocol: https
        - host: <redact>
          port: 443
          protocol: https
        - host: <redact>
          port: 443
          protocol: https
        - host: cib-ocp-shared-activegate.dynatrace
          port: 443
          protocol: https
        - host: <redact>
          port: 443
          protocol: https
        - host: <redact>-dev-trusted-activegate.dynatrace
          port: 443
          protocol: https
        endpoints: https://<redact>/communication;https://<redact>/communication;https://<redact>:9999/communication;https://<redact>:9999/communication;https://<redact>:443/communication;https://<redact>-activegate.dynatrace:443/communication;https://<redact>:443/communication;https://<redact>-activegate.dynatrace:443/communication;https://<redact>:9999/communication;https://<redact>:9999/communication;https://<redact>:443/communication;https://<redact>-activegate.dynatrace:443/communication;https://<redact>:443/communication;https://<redact>-activegate.dynatrace:443/communication;https://<redact>:443/communication;https://<redact>:443/communication
        tenantTokenHash: "935154033"
        tenantUUID: <redact>
      healthcheck:
        Interval: 10000000000
        Retries: 3
        StartPeriod: 1200000000000
        Test:
        - /usr/bin/watchdog-healthcheck64
        Timeout: 30000000000
      imageID: <redact>/linux/oneagent:1.323.46-raw
      instances:
        <redact>-dev-bw2sp-master-0:
          ipAddress: 10.100.110.16
          podName: <redact>-oneagent-jtlcd
        <redact>-dev-bw2sp-master-1:
          ipAddress: 10.100.110.15
          podName: <redact>-oneagent-gqh4r
        <redact>-dev-bw2sp-master-2:
          ipAddress: 10.100.110.17
          podName: <redact>-oneagent-45jkz
        <redact>-dev-bw2sp-worker-2dh8h:
          ipAddress: 10.100.110.23
          podName: <redact>-oneagent-hxvlr
        <redact>-dev-bw2sp-worker-52qvp:
          ipAddress: 10.100.110.31
          podName: <redact>-oneagent-z2pkc
        <redact>-dev-bw2sp-worker-5sg7n:
          ipAddress: 10.100.110.20
          podName: <redact>-oneagent-5f4s2
        <redact>-dev-bw2sp-worker-5zjjr:
          ipAddress: 10.100.110.19
          podName: <redact>-oneagent-phd58
        <redact>-dev-bw2sp-worker-7vwjs:
          ipAddress: 10.100.110.32
          podName: <redact>-oneagent-kvkqm
        <redact>-dev-bw2sp-worker-8xhlv:
          ipAddress: 10.100.110.30
          podName: <redact>-oneagent-hg2rk
        <redact>-dev-bw2sp-worker-9mk4g:
          ipAddress: 10.100.110.34
          podName: <redact>-oneagent-kr5kv
        <redact>-dev-bw2sp-worker-b2lfg:
          ipAddress: 10.100.110.24
          podName: <redact>-oneagent-bgwd5
        <redact>-dev-bw2sp-worker-br875:
          ipAddress: 10.100.110.35
          podName: <redact>-oneagent-b2f5l
        <redact>-dev-bw2sp-worker-dn95b:
          ipAddress: 10.100.110.28
          podName: <redact>-oneagent-5flpk
        <redact>-dev-bw2sp-worker-dqp9q:
          ipAddress: 10.100.110.29
          podName: <redact>-oneagent-742t5
        <redact>-dev-bw2sp-worker-gmdqm:
          ipAddress: 10.100.110.25
          podName: <redact>-oneagent-wjbwl
        <redact>-dev-bw2sp-worker-hrvht:
          ipAddress: 10.100.110.21
          podName: <redact>-oneagent-z4xb2
        <redact>-dev-bw2sp-worker-sklx7:
          ipAddress: 10.100.110.22
          podName: <redact>-oneagent-gv5hk
        <redact>-dev-bw2sp-worker-tzqvv:
          ipAddress: 10.100.110.26
          podName: <redact>-oneagent-bv7ql
        <redact>-dev-bw2sp-worker-xwgqs:
          ipAddress: 10.100.110.27
          podName: <redact>-oneagent-724hr
        <redact>-dev-bw2sp-worker-zcgfk:
          ipAddress: 10.100.110.33
          podName: <redact>-oneagent-9vcck
      lastInstanceStatusUpdate: "2025-12-17T12:24:19Z"
      lastProbeTimestamp: "2025-12-17T12:24:19Z"
      source: tenant-registry
      version: 1.323.46.20251015-192739
    phase: Running
    updatedTimestamp: "2025-12-17T12:24:19Z"
kind: List
metadata:
  resourceVersion: ""

 Our pods

▶ k get pods
NAME                                  READY   STATUS    RESTARTS   AGE
dynatrace-oneagent-csi-driver-2rnx5   4/4     Running   0          16d
dynatrace-oneagent-csi-driver-4swzt   4/4     Running   8          59d
dynatrace-oneagent-csi-driver-5x85r   4/4     Running   0          6h20m
dynatrace-oneagent-csi-driver-6c89w   4/4     Running   8          59d
dynatrace-oneagent-csi-driver-7jckb   4/4     Running   0          16d
dynatrace-oneagent-csi-driver-826lq   4/4     Running   0          16d
dynatrace-oneagent-csi-driver-82x2h   4/4     Running   8          59d
dynatrace-oneagent-csi-driver-85fmz   4/4     Running   8          59d
dynatrace-oneagent-csi-driver-bz5gv   4/4     Running   8          59d
dynatrace-oneagent-csi-driver-crbfh   4/4     Running   8          59d
dynatrace-oneagent-csi-driver-j2xk2   4/4     Running   0          16d
dynatrace-oneagent-csi-driver-mspq9   4/4     Running   8          59d
dynatrace-oneagent-csi-driver-pxqc8   4/4     Running   0          16d
dynatrace-oneagent-csi-driver-qfmck   4/4     Running   8          59d
dynatrace-oneagent-csi-driver-s7rlp   4/4     Running   0          16d
dynatrace-oneagent-csi-driver-tc2bc   4/4     Running   8          59d
dynatrace-oneagent-csi-driver-v4djn   4/4     Running   8          59d
dynatrace-oneagent-csi-driver-w6859   4/4     Running   8          59d
dynatrace-oneagent-csi-driver-x699g   4/4     Running   0          16d
dynatrace-oneagent-csi-driver-z7fv9   4/4     Running   8          59d
dynatrace-operator-7cbb87cc9c-ng64z   1/1     Running   0          26h
dynatrace-webhook-55c7d8b74b-x9vq9    1/1     Running   0          16d
dynatrace-webhook-55c7d8b74b-zlk2q    1/1     Running   0          16d
<redact>-activegate-0        1/1     Running   0          27h
<redact>-oneagent-45jkz      1/1     Running   0          69m
<redact>-oneagent-5f4s2      1/1     Running   0          67m
<redact>-oneagent-5flpk      1/1     Running   0          83m
<redact>-oneagent-724hr      1/1     Running   0          65m
<redact>-oneagent-742t5      1/1     Running   0          55m
<redact>-oneagent-9vcck      1/1     Running   0          63m
<redact>-oneagent-b2f5l      1/1     Running   0          82m
<redact>-oneagent-bgwd5      1/1     Running   0          76m
<redact>-oneagent-bv7ql      1/1     Running   0          80m
<redact>-oneagent-gqh4r      1/1     Running   0          71m
<redact>-oneagent-gv5hk      1/1     Running   0          56m
<redact>-oneagent-hg2rk      1/1     Running   0          58m
<redact>-oneagent-hxvlr      1/1     Running   0          62m
<redact>-oneagent-jtlcd      1/1     Running   0          78m
<redact>-oneagent-kr5kv      1/1     Running   0          51m
<redact>-oneagent-kvkqm      1/1     Running   0          53m
<redact>-oneagent-phd58      1/1     Running   0          72m
<redact>-oneagent-wjbwl      1/1     Running   0          85m
<redact>-oneagent-z2pkc      1/1     Running   0          60m
<redact>-oneagent-z4xb2      1/1     Running   0          74m

 

Not sure what else to offer at this moment. Very much going round in circles and I'm sure it will be a trivial issue.

 

I have followed this:
https://community.dynatrace.com/t5/Troubleshooting/Logs-Ingest-on-K8s-with-Dynatrace/ta-p/285827

2 REPLIES 2

mattcanty
Newcomer_

Twice today i've managed to get logs through. 

Unchanged pod spec.

Same namespace.

 

Once 22 messages made it, the next time 43 messages. As stated, it's pushing a log message every 31 seconds. They should be flowing continuously.

Featured Posts