Solved: Re: Kubernetes Activegate error

saisandeep · ‎19 Jun 2023

Hello,

I am facing an issue with activegate pod. I deployed oneagent in kubernetes for monitoring petclinic application and the activegate is in running state always. I've attached error images. can you please help me to solve this problem.

thanks

Romanenkov_Al3x · ‎19 Jun 2023

Hello.

Did you try to use "kubectl delete pod petlinic-activegate-0"?

Can you share response from these commands:

kubectl describe pod petlinic-activegate-0

kubectl exec deploy/dynatrace-operator -n dynatrace -- dynatrace-operator troubleshoot

Regards,

Alex Romanenkov

DT_NGINX_FORCE_UNKNOWN_VERSION_INSTRUMENTATION=1

saisandeep · ‎19 Jun 2023

@Romanenkov_Al3x

even after deleting the pod it's still in running(0/1) state.

sudo kubectl describe pod petlinic-activegate-0 -n dynatrace
Name:             petlinic-activegate-0
Namespace:        dynatrace
Priority:         0
Service Account:  dynatrace-kubernetes-monitoring
Node:             workernode01/192.168.0.207
Start Time:       Mon, 19 Jun 2023 13:29:24 +0530
Labels:           app.kubernetes.io/component=activegate
                  app.kubernetes.io/created-by=petlinic
                  app.kubernetes.io/managed-by=dynatrace-operator
                  app.kubernetes.io/name=activegate
                  app.kubernetes.io/version=1.267.118.20230602-131900
                  controller-revision-hash=petlinic-activegate-5cb68bb95
                  statefulset.kubernetes.io/pod-name=petlinic-activegate-0
Annotations:      internal.operator.dynatrace.com/activegate-configuration-hash: 3794628405
Status:           Running
IP:               10.244.1.119
IPs:
  IP:           10.244.1.119
Controlled By:  StatefulSet/petlinic-activegate
Init Containers:
  certificate-loader:
    Container ID:    containerd://8a345ce4510879ede3dbe4f63e42bfb3383b785731f954aa7af4b1688908042c
    Image:           xxxxx.live.dynatrace.com/linux/activegate:latest
    Image ID:        xxxxxx.live.dynatrace.com/linux/activegate@sha256:a3ceeb36fa5c40c8ab3e73c3de9e16d2035f70dbfd79f5aa94eef3677bec009d
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Command:
      /bin/bash
    Args:
      -c
      /opt/dynatrace/gateway/k8scrt2jks.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 19 Jun 2023 13:29:28 +0530
      Finished:     Mon, 19 Jun 2023 13:29:31 +0530
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  1536Mi
    Requests:
      cpu:        500m
      memory:     512Mi
    Environment:  <none>
    Mounts:
      /var/lib/dynatrace/gateway/ssl from truststore-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bhqq9 (ro)
Containers:
  activegate:
    Container ID:    containerd://eb783166fdfb57ed0ec2a31416d1d784a9efc3fd2edd277689d3bb21df08a092
    Image:           xxxxxx.live.dynatrace.com/linux/activegate:latest
    Image ID:        xxxxxx.live.dynatrace.com/linux/activegate@sha256:a3ceeb36fa5c40c8ab3e73c3de9e16d2035f70dbfd79f5aa94eef3677bec009d
    Port:            9999/TCP
    Host Port:       0/TCP
    SeccompProfile:  RuntimeDefault
    State:           Running
      Started:       Mon, 19 Jun 2023 13:29:34 +0530
    Ready:           False
    Restart Count:   0
    Limits:
      cpu:     1
      memory:  1536Mi
    Requests:
      cpu:      500m
      memory:   512Mi
    Readiness:  http-get https://:https/rest/health delay=90s timeout=1s period=15s #success=1 #failure=3
    Environment:
      DT_CAPABILITIES:            MSGrouter,kubernetes_monitoring,restInterface
      DT_ID_SEED_NAMESPACE:       dynatrace
      DT_ID_SEED_K8S_CLUSTER_ID:  5e381001-fe93-4898-b43c-dcdfd8443512
      DT_DEPLOYMENT_METADATA:     <set to the key 'activegate' of config map 'petlinic-deployment-metadata'>                      Optional: false
      DT_TENANT:                  <set to the key 'tenant-uuid' of config map 'petlinic-activegate-connection-info'>              Optional: false
      DT_SERVER:                  <set to the key 'communication-endpoints' of config map 'petlinic-activegate-connection-info'>  Optional: false
      DT_DNS_ENTRY_POINT:         https://$(PETLINIC_ACTIVEGATE_SERVICE_HOST):$(PETLINIC_ACTIVEGATE_SERVICE_PORT)/communication
    Mounts:
      /opt/dynatrace/gateway/jre/lib/security/cacerts from truststore-volume (ro,path="k8s-local.jks")
      /var/lib/dynatrace/secrets/tokens/auth-token from ag-authtoken-secret (ro,path="auth-token")
      /var/lib/dynatrace/secrets/tokens/tenant-token from connection-info-secret (ro,path="tenant-token")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bhqq9 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  ag-authtoken-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  petlinic-activegate-authtoken-secret
    Optional:    false
  connection-info-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  petlinic-activegate-tenant-secret
    Optional:    false
  truststore-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-bhqq9:
    Type:                     Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:   3607
    ConfigMapName:            kube-root-ca.crt
    ConfigMapOptional:        <nil>
    DownwardAPI:              true
QoS Class:                    Burstable
Node-Selectors:               <none>
Tolerations:                  kubernetes.io/arch=amd64:NoSchedule
                              node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints:  kubernetes.io/hostname:DoNotSchedule when max skew 1 is exceeded for selector app.kubernetes.io/created-by=petlinic,app.kubernetes.io/managed-by=dynatrace-operator,app.kubernetes.io/name=activegate
                              topology.kubernetes.io/zone:ScheduleAnyway when max skew 1 is exceeded for selector app.kubernetes.io/created-by=petlinic,app.kubernetes.io/managed-by=dynatrace-operator,app.kubernetes.io/name=activegate
Events:
  Type     Reason     Age                  From     Message
  ----     ------     ----                 ----     -------
  Warning  Unhealthy  28s (x426 over 88m)  kubelet  Readiness probe failed: HTTP probe failed with statuscode: 503

saisandeep · ‎19 Jun 2023

sandeep@master:~$ sudo kubectl exec deploy/dynatrace-operator -n dynatrace -- dynatrace-operator troubleshoot
{"level":"info","ts":"2023-06-19T09:32:23.278Z","logger":"version","msg":"dynatrace-operator","version":"v0.11.2","gitCommit":"047df7f55d55fc3049e9f04efcdd5a848f06a52b","buildDate":"2023-05-18T14:28:35+00:00","goVersion":"go1.20.3","platform":"linux/amd64"}
[namespace ] 	--- checking if namespace 'dynatrace' exists ...
[namespace ] 	 ✓  using namespace 'dynatrace'
[crd       ] 	--- checking if CRD for Dynakube exists ...
[crd       ] 	 ✓  CRD for Dynakube exists
[oneAgentAPM] 	--- checking if OneAgentAPM object exists ...
[oneAgentAPM] 	 ✓  OneAgentAPM does not exist
[          ] 	no Dynakube specified - checking all Dynakubes in namespace 'dynatrace'
[          ] 	petlinic
[dynakube  ] 	--- checking if 'dynatrace:petlinic' Dynakube is configured correctly
[          ] 	    using 'dynatrace:petlinic' Dynakube
[dynakube  ] 	    secret token 'apiToken' exists
[dynakube  ] 	    checking if syntax of API URL is valid
[dynakube  ] 	    syntax of API URL is valid
[dynakube  ] 	    checking if token scopes are valid
[dynakube  ] 	    token scopes are valid
[dynakube  ] 	    checking if can pull latest agent version
[dynakube  ] 	    API token is valid, can pull latest agent version
[dynakube  ] 	    pull secret 'dynatrace:petlinic-pull-secret' exists
[dynakube  ] 	    secret token '.dockerconfigjson' exists
[dynakube  ] 	 ✓  'dynatrace:petlinic' Dynakube is valid
[imagepull ] 	--- Verifying that OneAgent image xxxxx.live.dynatrace.com/linux/oneagent:latest can be pulled ...
[imagepull ] 	 ✓  OneAgent image xxxxxx.live.dynatrace.com/linux/oneagent:latest can be successfully pulled
[imagepull ] 	--- Verifying that OneAgentCodeModules (custom image) image  can be pulled ...
[imagepull ] 	    No OneAgentCodeModules (custom image) image configured
[imagepull ] 	--- Verifying that ActiveGate image jkr36399.live.dynatrace.com/linux/activegate:latest can be pulled ...
[imagepull ] 	 ✓  ActiveGate image xxxxxx.live.dynatrace.com/linux/activegate:latest can be successfully pulled
[proxy     ] 	--- Analyzing proxy settings ...
[proxy     ] 	 ✓  No proxy settings found.
[          ] 	 ✓  'petlinic' - all checks passed

radek_jasinski · ‎19 Jun 2023

Hi @saisandeep

Is that all the information you have in the log?
I don't see anything indicative of an error here.

Radek

Have a nice day!

saisandeep · ‎21 Jun 2023

Hi @radek_jasinski ,

yes, that is all the information when i execute describe command. the below is the log for that activegate pod. Can you please check this.

Thanks

@radek_jasinski wrote:
Hi @saisandeep
Is that all the information you have in the log?
I don't see anything indicative of an error here.
Radek

sandeep@master:~$ sudo kubectl logs  petlinic-activegate-0 -n dynatrace
Defaulted container "activegate" out of: activegate, certificate-loader (init)
Copy configuration templates to /var/lib/dynatrace/gateway/config
Assigned id = 0x49b1c08e
Setting environment AG token
Applying default config
Enabling selected features: MSGrouter,kubernetes_monitoring,restInterface
Setting dns entry point to https://{environmentid}.live.dynatrace.com/communication;-1], [https://{environmentid}.live.dynatrace.com/communication;-1], [https://{environmentid}.live.dynatrace.com/communication;-1], [https://{environmentid}.live.dynatrace.com/communication;-1], [https://{environmentid}.live.dynatrace.com/communication;-1], [https://{environmentid}.live.dynatrace.com/communication;-1], [https://{environmentid}.live.dynatrace.com/communication;-1], [https://{environmentid}.live.dynatrace.com:443/communication;-1]], 
System: startupTimestamp = 1687354006981, osInfo = Platform: Linux, Version: 5.4.0-150-generic, Architecture: amd64, Processors: 1, jvmInfo = VM: OpenJDK 64-Bit Server VM, Version: 11.0.18, Vendor: Eclipse Adoptium, Memory [maxMemory=759M, initHeap=384M, maxHeap=759M, usedMeta=14M, committedMeta=15M, totalPhysicalMemory=1536M, freePhysicalMemory=1407M], buildVersion = 1.267.43.20230512-112635
EntityType: COLLECTOR, isRunningInContainer = true
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [<com.compuware.apm.debug.beaconforwarder>, BeaconForwarderService] Beacon Forwarder disabled. Skipping initialization of UEM Configuration Poller and synthetic results reporter.
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [HttpClientServiceImpl] No proxy server configured.
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [<collector>, CollectorLifeCycle] Collector Operation State = 'STARTUP_CONNECTING'
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [<collector.services>, ServicesManager] State changed from 'STARTUP_INIT' to 'STARTUP_CONNECTING'
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [<collector.services>, ServicesManager] Initializing services..
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [GenericFileCacheModuleImpl] Generic filecache service initialized.
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [<collector.comm>, LocalSupportArchiveService] LocalSupportArchive service initialized.
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [GenericIngestModule] Generic ingest service initialized.
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [<collector.services>, ServicesManager] Registering message handlers...
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [<collector.modules>, ModulesManager] State changed from 'STARTUP_INIT' to 'STARTUP_CONNECTING'
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [<collector.modules>, ModulesManager] Registering message handlers...
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [<collector.modules>, ModulesManager] Initializing modules..
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [<collector.modules>, ModulesManager] kubernetes_monitoring module is ready.
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [OdinModule] Odin module initialized.
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [<collector.modules>, ModulesManager] odin_collector module is ready.
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [RunningStateController] state: ACTIVE_GATE=false,BEACON_FORWARDER_WAIT=false,BEACON_FORWARDER_READY=false,MODULES=true,operationState='STARTUP_CONNECTING'
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [<collector.core>, CollectorSelfMonitoringPoller] stop Collector Self Monitoring Poller
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [<collector.core>, SelfMonitoringPeriodicSender] stop Collector Self Monitoring periodic sender
2023-06-21 13:27:32 UTC INFO    [<jkr36399>] [<collector.core>, CollectorRuntimeImpl] Collector started successfully.
2023-06-21 13:27:32 UTC DEBUG   [<jkr36399>] [<collector.core>, LogStatisticsWriter] Stats for the last: 0 seconds. Local time: 2023-06-21 13:27:32 GMT. Threads: 82. Active agents: 0. Memory usage in kB: free memory: 319,348; heap: 384,704; max heap: 777,920; direct byte buffers: 8. 
Network statistics: 
 outgoingHttpErrors: 0;  incomingHttpErrors: 0
Queue length: server Id: -999; OneWay: 0; Request: 0. Selfmonitoring: enabled: true; metrics sent: 0. 
2023-06-21 13:27:33 UTC INFO    [<jkr36399>] [HttpClientServiceImpl] No proxy server configured.
2023-06-21 13:27:34 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=ANY_SERVER(id=0x0000000000000000), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=2
2023-06-21 13:27:34 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=ANY_SERVER(id=0x0000000000000000), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=2, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-52-55-57-17-prod-us-east-6-virginia.live.dynatrace.com source=ANY_SERVER 0x0000000000000000
2023-06-21 13:27:34 UTC WARNING [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] Initial collector setup received with error:COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-52-55-57-17-prod-us-east-6-virginia.live.dynatrace.com source=ANY_SERVER 0x0000000000000000 [Suppressing further messages for 1 hour]
2023-06-21 13:27:35 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=ANY_SERVER(id=0x0000000000000000), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=3
2023-06-21 13:27:35 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=ANY_SERVER(id=0x0000000000000000), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=3, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-34-237-88-136-prod-us-east-6-virginia.live.dynatrace.com source=ANY_SERVER 0x0000000000000000
2023-06-21 13:27:36 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=ANY_SERVER(id=0x0000000000000000), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=4
2023-06-21 13:27:36 UTC WARNING [<jkr36399>] [<collector.core>, CommunicationLayer] Collector starts failover mode
2023-06-21 13:27:36 UTC INFO    [<jkr36399>] [<collector>, CollectorLifeCycle] Collector Operation State = 'FAILOVER'
2023-06-21 13:27:36 UTC INFO    [<jkr36399>] [<collector.services>, ServicesManager] State changed from 'STARTUP_CONNECTING' to 'FAILOVER'
2023-06-21 13:27:36 UTC INFO    [<jkr36399>] [<collector.modules>, ModulesManager] State changed from 'STARTUP_CONNECTING' to 'FAILOVER'
2023-06-21 13:27:36 UTC INFO    [<jkr36399>] [<collector.core>, Identification] Stop identification messages thread
2023-06-21 13:27:36 UTC INFO    [<jkr36399>] [<collector.comm>, ClusterRuntimeInfoPoller] stop cluster runtime info poller
2023-06-21 13:27:36 UTC INFO    [<jkr36399>] [<collector.comm>, ClusterTimePoller] stop cluster time poller
2023-06-21 13:27:36 UTC INFO    [<jkr36399>] [<collector.core>, ConfigurationPoller] stop configuration poller
2023-06-21 13:27:36 UTC INFO    [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] start initial collector setup poller in failover mode
2023-06-21 13:27:36 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-999), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=5
2023-06-21 13:27:36 UTC WARNING [<jkr36399>] [<collector.core>, CommunicationLayer] Collector started failover mode
2023-06-21 13:27:36 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-999), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=5, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-52-4-253-12-prod-us-east-6-virginia.live.dynatrace.com source=SERVER -999
2023-06-21 13:27:36 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=ANY_SERVER(id=0x0000000000000000), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=4, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-34-236-11-154-prod-us-east-6-virginia.live.dynatrace.com source=ANY_SERVER 0x0000000000000000
2023-06-21 13:27:37 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-1000), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=6
2023-06-21 13:27:37 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-1000), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=6, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-35-171-243-226-prod-us-east-6-virginia.live.dynatrace.com source=SERVER -1000
2023-06-21 13:27:38 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-1001), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=7
2023-06-21 13:27:38 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-1001), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=7, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-52-55-57-17-prod-us-east-6-virginia.live.dynatrace.com source=SERVER -1001
2023-06-21 13:27:39 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-1002), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=8
2023-06-21 13:27:39 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-1002), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=8, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-34-237-88-136-prod-us-east-6-virginia.live.dynatrace.com source=SERVER -1002
2023-06-21 13:27:40 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-1003), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=9
2023-06-21 13:27:40 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-1003), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=9, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-34-237-88-136-prod-us-east-6-virginia.live.dynatrace.com source=SERVER -1003
2023-06-21 13:27:41 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-1004), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=10
2023-06-21 13:27:41 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-1004), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=10, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: jkr36399.live.dynatrace.com source=SERVER -1004
2023-06-21 13:27:42 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-1005), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=11
2023-06-21 13:28:02 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-1005), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=11, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-52-55-57-17-prod-us-east-6-virginia.live.dynatrace.com: System error source=SERVER -1005
2023-06-21 13:28:03 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-1006), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=12
2023-06-21 13:28:03 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-1006), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=12, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-52-55-57-17-prod-us-east-6-virginia.live.dynatrace.com source=SERVER -1006
2023-06-21 13:28:04 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-1007), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=13
2023-06-21 13:28:24 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-1007), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=13, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-3-220-98-230-prod-us-east-6-virginia.live.dynatrace.com: System error source=SERVER -1007
2023-06-21 13:28:25 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-999), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=14
2023-06-21 13:28:45 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-999), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=14, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-52-4-253-12-prod-us-east-6-virginia.live.dynatrace.com: System error source=SERVER -999
2023-06-21 13:28:46 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-1000), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=15
2023-06-21 13:29:06 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-1000), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=15, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-3-220-98-230-prod-us-east-6-virginia.live.dynatrace.com: System error source=SERVER -1000
2023-06-21 13:29:07 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-1001), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=16
2023-06-21 13:29:07 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-1001), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=16, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-3-220-98-230-prod-us-east-6-virginia.live.dynatrace.com source=SERVER -1001
2023-06-21 13:29:08 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-1002), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=17
2023-06-21 13:29:28 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-1002), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=17, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-34-236-11-154-prod-us-east-6-virginia.live.dynatrace.com: System error source=SERVER -1002
2023-06-21 13:29:29 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-1003), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=18
2023-06-21 13:29:49 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-1003), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=18, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-35-171-243-226-prod-us-east-6-virginia.live.dynatrace.com: System error source=SERVER -1003
2023-06-21 13:29:50 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-1004), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=19
2023-06-21 13:30:10 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-1004), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=19, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-52-4-253-12-prod-us-east-6-virginia.live.dynatrace.com: System error source=SERVER -1004
2023-06-21 13:30:11 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-1005), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=20
2023-06-21 13:30:11 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-1005), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=20, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-3-220-98-230-prod-us-east-6-virginia.live.dynatrace.com source=SERVER -1005
2023-06-21 13:30:12 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-1006), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=21
2023-06-21 13:30:32 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-1006), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=21, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-35-171-243-226-prod-us-east-6-virginia.live.dynatrace.com: System error source=SERVER -1006
2023-06-21 13:30:33 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-1007), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=22
2023-06-21 13:30:53 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-1007), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=22, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-52-55-57-17-prod-us-east-6-virginia.live.dynatrace.com: System error source=SERVER -1007
2023-06-21 13:30:54 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-999), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=23
2023-06-21 13:30:54 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-999), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=23, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-34-237-88-136-prod-us-east-6-virginia.live.dynatrace.com source=SERVER -999
2023-06-21 13:30:55 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-1000), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=24
2023-06-21 13:31:15 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] response: sender=SERVER(id=-1000), target=COLLECTOR(id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=24, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-35-171-243-226-prod-us-east-6-virginia.live.dynatrace.com: System error source=SERVER -1000
2023-06-21 13:31:16 UTC DEBUG   [<jkr36399>] [<collector.core>, InitialCollectorSetupPoller] request: sender=COLLECTOR(id=0x0000000049b1c08e), target=SERVER(id=-1001), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=25

radek_jasinski · ‎21 Jun 2023

Do you have open network traffic? Please verify with Dynatrace recommendations.

You have an error in the logs indicating a lack of communication:

id=0x0000000049b1c08e), content-type=INITIAL_COLLECTOR_SETUP, priority=NORMAL, message-id=2, error COMMUNICATION_ERROR, reason=java.net.UnknownHostException: sg-us-east-1-52-55-57-17-prod-us-east-6-virginia.live.dynatrace.com source=ANY_SERVER 0x0000000000000000

Have a nice day!

saisandeep · ‎21 Jun 2023

@radek_jasinski are you speaking about this recommendations

radek_jasinski · ‎21 Jun 2023

No, based on the log you have issues with the connection between ActiveGate and your Dynatrace cluster. You should to check this part of your system.

Maybe another good way is a use proxy for AG: https://www.dynatrace.com/support/help/setup-and-configuration/dynatrace-activegate/configuration/se...

Have a nice day!

vs185250 · ‎05 Feb 2024

we just opened a request for this similar issue
Activegate in GKE not able to resolve the cluster DNS – Dynatrace

what's the RCA from the Dynatrace side?
it stopped working all of a sudden, but it was working before.

vs185250 · ‎05 Feb 2024

restarting the kube-dns pods worked for us. thanks.