Hi @p_devulapalli - The connectivity issue is isolated to the AWS Resource Groups Tagging endpoint (https://tagging.us-west-2.amazonaws.com/).
The Activate Host is in an AWS environment which is behind a Proxy. So the Tagging endpoint traffic has to go through the Proxy.
The environment variables for HTTPS_PROXY , HTTP_PROXY and NO_PROXY have been set in the Activate Service in systemd ( /etc/systemd/system/dynatracegateway.service) .
Due to the NO_PROXY setting, there is no issue connecting to STS, Monitoring and EC2 endpoints ( as captured in the below log). VPC endpoints have been defined for them.
Unfortunately, it fails for the endpoint - https://tagging.us-west-2.amazonaws.com- with the error: " unable to find valid certification path to requested target"
---------
From Debug.log
2024-09-23 17:53:28 UTC DEBUG [<f73b0e7e190c-2345-44cb-9623-f73b0e7e190c>] [<vtopology.provider>, PartitionAutoDetection] detectedPartition=aws, for credentials: Tools Test [-1061062254360197475]
2024-09-23 17:53:29 UTC DEBUG [<f73b0e7e190c-2345-44cb-9623-f73b0e7e190c>] [<vtopology.provider>, AWSFastCheckCallable] Credentials refresh successful: AWS credentials: AWSCredentialsImpl {identifier: 4DEF18A649159633, accessKey: null, tenantUUID: f73b0e7e190c-2345-44cb-9623-f73b0e7e190c, iamRole: Dynatrace_monitoring_role, accountId: 378629572362, externalId: *****, label: Tools Test, version: 2.0}
2024-09-23 17:53:29 UTC INFO [<f73b0e7e190c-2345-44cb-9623-f73b0e7e190c>] [<vtopology.provider>, AWSFastCheckCallable] STS GetCallerIdentity: {status: OK, statusInfo: , credentials: AWSCredentialsImpl {identifier: 4DEF18A649159633, accessKey: null, tenantUUID: f73b0e7e190c-2345-44cb-9623-f73b0e7e190c, iamRole: Dynatrace_monitoring_role, accountId: 378629572362, externalId: *****,label: Tools Test, version: 2.0}}
2024-09-23 17:53:29 UTC INFO [<f73b0e7e190c-2345-44cb-9623-f73b0e7e190c>] [<vtopology.provider>, AWSFastCheckCallable] AWS FastCheck ec2:DescribeAvailabilityZones permission check completedsuccessfully in region us-west-2 for credentials(AWSCredentialsImpl {identifier: 4DEF18A649159633, accessKey: null, tenantUUID: f73b0e7e190c-2345-44cb-9623-f73b0e7e190c, iamRole: Dynatrace_monitoring_role, accountId: 378629572362, externalId: *****, label: Tools Test, version: 2.0})
2024-09-23 17:53:31 UTC INFO [<f73b0e7e190c-2345-44cb-9623-f73b0e7e190c>] [<vtopology.provider>, AWSErrorLogger] (Credentials: Tools Test) Service AWSResourceGroupsTaggingAPI, endpoint https://tagging.us-west-2.amazonaws.com, returned error: no response, message: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target [Suppressing further identical messages for 3 hours]
2024-09-23 17:53:31 UTC INFO [<f73b0e7e190c-2345-44cb-9623-f73b0e7e190c>] [<vtopology.provider>, AWSFastCheckCallable] AWS FastCheck permissions status in region us-west-2 for credentials(AWSCredentialsImpl {identifier: 4DEF18A649159633, accessKey: null, tenantUUID: f73b0e7e190c-2345-44cb-9623-f73b0e7e190c, iamRole: Dynatrace_monitoring_role, accountId: 378629572362, externalId: *****, label: Tools Test, version: 2.0}):
tag:GetResources ERROR_UNKNOWN. Status: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
cloudwatch:ListMetrics OK.
cloudwatch:GetMetricData OK.
----------------------------------
The application is running as the user: dtuserag
$ ps -ef | grep dtuserag
dtuserag 1338998 1 0 16:09 ? 00:00:00 /opt/dynatrace/gateway/launcher/dynatracegateway -bg -config /var/lib/dynatrace/gateway/config/dynatracegateway.ini
dtuserag 1339002 1338998 2 16:09 ? 00:02:05 /opt/dynatrace/gateway/jre/bin/java -Dcom.compuware.apm.WatchDogTimeout=180 -classpath ./lib/* --add-opens=java.base/java.lang=ALL-UNNAMED -Xms1024M -XX:ErrorFile=/var/log/dynatrace/gateway/hs_err_pid_%p.log -XX:+UseG1GC -XX:+IgnoreUnrecognizedVMOptions -Duser.language=en -Djava.util.logging.manager=com.dynatrace.gen2.foundation.logging.impl.backend.CustomShutdownLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Dorg.xerial.snappy.lib.path=/opt/dynatrace/gateway/lib/native -Dorg.xerial.snappy.lib.name=libsnappyjava.so -Djava.io.tmpdir=/var/lib/dynatrace/gateway/temp -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.security.egd=file:/dev/urandom -DZstdNativePath=/opt/dynatrace/gateway/lib/native/libzstd-jni.so -Xmx10275M -Dcom.compuware.apm.WatchDogPort=50000 com.compuware.apm.collector.core.CollectorImpl -CONFIG_DIR /var/lib/dynatrace/gateway/config
----------------
The Proxy Root cert is stored in trusted.p12 ( created using the info mentioned here )
The application seems to pick all the relevant files to create the runtime cert (runtime.cacerts) by merging cacerts and trusted.p12
Excerpt from the Debug log captured below…
2024-09-23 16:01:44 UTC INFO [<header>] +-----------------------------------------------------------------------------
2024-09-23 16:01:44 UTC INFO [<header>] + Dynatrace LLC
2024-09-23 16:01:44 UTC INFO [<header>] +-----------------------------------------------------------------------------
2024-09-23 16:01:44 UTC INFO [<header>] + Version 1.295.7.20240715-224910
2024-09-23 16:01:44 UTC INFO [<header>] + collector: 0x7cd018cb; tenant: f73b0e7e190c-2345-44cb-9623-f73b0e7e190c;
2024-09-23 16:01:44 UTC INFO [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] Platform: Linux, Version: 5.14.0-284.82.1.el9_2.x86_64, Architecture: amd64, Processors: 4
2024-09-23 16:01:44 UTC INFO [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] VM: OpenJDK 64-Bit Server VM, Version: 17.0.10, Vendor: Eclipse Adoptium, Memory [maxMemory=1
0280M, initHeap=1024M, maxHeap=10280M, usedMeta=7M, committedMeta=7M, totalPhysicalMemory=15809M, freePhysicalMemory=6793M]
2024-09-23 16:01:44 UTC INFO [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] file.encoding: UTF-8, sun.jnu.encoding: UTF-8, user.name: dtuserag
2024-09-23 16:01:44 UTC INFO [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] Input Arguments: -Dcom.compuware.apm.WatchDogTimeout=180 --add-opens=java.base/java.lang=ALL-
UNNAMED -Xms1024M -XX:ErrorFile=/var/log/dynatrace/gateway/hs_err_pid_%p.log -XX:+UseG1GC -XX:+IgnoreUnrecognizedVMOptions -Duser.language=en -Djava.util.logging.manager=com.dynatrace.gen2.
foundation.logging.impl.backend.CustomShutdownLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Dorg.xerial.snappy.lib.path=/opt/dynatrace/gateway/lib/native -Dorg.xerial.snappy.lib.name=libsna
ppyjava.so -Djava.io.tmpdir=/var/lib/dynatrace/gateway/temp -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.security.egd=file:/dev/urandom -DZstdNativePath=/opt/dynatrace/gateway/lib/
native/libzstd-jni.so -Xmx10275M -Dcom.compuware.apm.WatchDogPort=50000
2024-09-23 16:01:44 UTC INFO [<collector>] [<collector.core>, CollectorImpl] No keyfile detected, starting without crypto subsystem.
2024-09-23 16:01:44 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Setting up combined trust store, Java cacerts: TrustStoreSettingsImpl{path=/opt/dynatrace/gateway/jre/lib
/security/cacerts, type=pkcs12}, custom: TrustStoreSettingsImpl{path=/var/lib/dynatrace/gateway/config/../ssl/trusted.p12, type=PKCS12}
2024-09-23 16:01:44 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias concurca
2024-09-23 16:01:44 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias /etc/pki/ca-trust/source/anchors/root-cert-0
2024-09-23 16:01:44 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias /etc/pki/ca-trust/source/anchors/root-cert-1
2024-09-23 16:01:44 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias /etc/pki/ca-trust/source/anchors/root-cert-2
2024-09-23 16:01:44 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias concurcert-0
2024-09-23 16:01:44 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias concurcert-1
2024-09-23 16:01:44 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias concurcert-2
2024-09-23 16:01:45 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Custom certificate configuration created successfully
2024-09-23 16:01:45 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Effective trust store: TrustStoreSettingsImpl{path=/var/lib/dynatrace/gateway/config/../ssl/runtime.cacerts, type=pkcs12}
-------------
I don’t why the AWS Tagging endpoint requests is not using the cert named runtime.cacerts ( in the path /var/lib/dynatrace/gateway/ssl )