AWS Role based Integration - Monitored account error for permissions related to tag and cloudwatch

[dyn98007]

This is in the context of AWS Monitoring setup using IAM Role based access.
ActiveGate version=1.295.27.20240715-230200

An IAM role for a Linux based ActiveGate in the AWS account that hosts the ActiveGate was created, and also a monitoring IAM role for Dynatrace ( named Dynatrace_monitoring_role) in the AWS account to be monitored.

Cloudfromation Templates used for both the IAM Roles can be accessed via https://github.com/dynatrace-oss/cloud-snippets/tree/main/aws/role-based-access

I'm facing an issue during the attempt to create an AWS connection in UI, due to errors from the Monitored AWS account related to tag and cloudwatch permissions.
Ec2 permission didn't face any issue. The policy attached to the Dynatrace_monitoring_role has the relevant tag and cloudwatch permissions,

Excerpt of the Debug log pasted below

----------------------------
2024-09-20 05:03:55 UTC INFO [<f73b0e7e190c-fefc24a4-d245-22rt-7788->] [<vtopology.provider>, AWSClientFactory] trying to get client initial response for credentials: AWSCredentialsImpl {identifier: 803D58634C5E37A4, accessKey: null, secretKey: null, tenantUUID: f73b0e7e190c-fefc24a4-d245-22rt-7788-, iamRole: Dynatrace_monitoring_role, accountId: 237862957236, externalId: *****, label: Tools-monitoring, partition: aws, detectedPartition: null, monitorOnlyTaggedEntities: true, includeTags: [Pair[A: RoleType, B: k8s]], excludeTags: [], excludedRegions: [], logConfigSQSesEnabled: false, logConfigSQSes: [], version: 2.0, legacyServices: [ebs_builtin, lambda_builtin, ELB_builtin, loadbalancer_builtin, s3_builtin, dynamodb_builtin, ec2_builtin, asg_builtin, rds_builtin], services: [], extensionDetails: null} [Suppressing further messages for 15 minutes] [skipped logs: 1]
2024-09-20 05:03:55 UTC DEBUG [<f73b0e7e190c-fefc24a4-d245-22rt-7788->] [<vtopology.provider>, PartitionAutoDetection] detectedPartition=aws, for credentials: Tools-monitoring [-8769628377978171752]
2024-09-20 05:03:55 UTC DEBUG [<f73b0e7e190c-fefc24a4-d245-22rt-7788->] [<vtopology.provider>, AWSFastCheckCallable] Credentials refresh successful: AWS credentials: AWSCredentialsImpl {identifier: 803D58634C5E37A4, accessKey: null, tenantUUID: f73b0e7e190c-fefc24a4-d245-22rt-7788-, iamRole: Dynatrace_monitoring_role, accountId: 237862957236, externalId: *****, label: Tools-monitoring, version: 2.0}
2024-09-20 05:03:55 UTC INFO [<f73b0e7e190c-fefc24a4-d245-22rt-7788->] [<vtopology.provider>, AWSFastCheckCallable] STS GetCallerIdentity: {status: OK, statusInfo: , credentials: AWSCredentialsImpl {identifier: 803D58634C5E37A4, accessKey: null, tenantUUID: f73b0e7e190c-fefc24a4-d245-22rt-7788-, iamRole: Dynatrace_monitoring_role, accountId: 237862957236, externalId: *****,label: Tools-monitoring, version: 2.0}}

2024-09-20 05:03:55 UTC INFO [<f73b0e7e190c-fefc24a4-d245-22rt-7788->] [<vtopology.provider>, AWSFastCheckCallable] AWS FastCheck ec2:DescribeAvailabilityZones permission check completed successfully in region us-west-2 for credentials(AWSCredentialsImpl {identifier: 803D58634C5E37A4, accessKey: null, tenantUUID: f73b0e7e190c-fefc24a4-d245-22rt-7788-, iamRole: Dynatrace_monitoring_role, accountId: 237862957236, externalId: *****, label: Tools-monitoring, version: 2.0})

2024-09-20 05:05:15 UTC INFO [<f73b0e7e190c-fefc24a4-d245-22rt-7788->] [<vtopology.provider>, AWSErrorLogger] (Credentials: Tools-monitoring) Service AWSResourceGroupsTaggingAPI, endpoint https://tagging.us-west-2.amazonaws.com, returned error: no response, message: [Suppressing further identical messages for 3 hours]

2024-09-20 05:05:15 UTC INFO [<f73b0e7e190c-fefc24a4-d245-22rt-7788->] [<vtopology.provider>, AWSFastCheckCallable] AWS FastCheck permissions status in region us-west-2 for credentials(AWSCredentialsImpl {identifier: 803D58634C5E37A4, accessKey: null, tenantUUID: f73b0e7e190c-fefc24a4-d245-22rt-7788-, iamRole: Dynatrace_monitoring_role, accountId: 237862957236, externalId: *****, label: Tools-monitoring, version: 2.0}):
tag:GetResources ERROR_UNKNOWN.
cloudwatch:ListMetrics ERROR_UNKNOWN.
cloudwatch:GetMetricData ERROR_UNKNOWN.

----------------

Not sure what is causing the error for tag and cloudwatch related permissions

[p_devulapalli]

Hi @dyn98007 - are the AWS endpoints reachable from AG?

https://docs.dynatrace.com/docs/shortlink/aws-monitoring-guide#capable-activegate

[dyn98007]

Hi @p_devulapalli  - The connectivity issue is isolated to the AWS Resource Groups Tagging endpoint (https://tagging.us-west-2.amazonaws.com/).

The Activate Host is in an AWS environment which is behind a Proxy. So the Tagging endpoint traffic has to go through the Proxy.

The environment variables for HTTPS_PROXY , HTTP_PROXY and NO_PROXY have been set in the Activate Service in systemd ( /etc/systemd/system/dynatracegateway.service) .
Due to the NO_PROXY setting, there is no issue connecting to STS, Monitoring and EC2 endpoints ( as captured in the below log). VPC endpoints have been defined for them.
Unfortunately, it fails for the endpoint - https://tagging.us-west-2.amazonaws.com- with the error: " unable to find valid certification path to requested target"

---------

From Debug.log

2024-09-23 17:53:28 UTC DEBUG [<f73b0e7e190c-2345-44cb-9623-f73b0e7e190c>] [<vtopology.provider>, PartitionAutoDetection] detectedPartition=aws, for credentials: Tools Test [-1061062254360197475]
2024-09-23 17:53:29 UTC DEBUG [<f73b0e7e190c-2345-44cb-9623-f73b0e7e190c>] [<vtopology.provider>, AWSFastCheckCallable] Credentials refresh successful: AWS credentials: AWSCredentialsImpl {identifier: 4DEF18A649159633, accessKey: null, tenantUUID: f73b0e7e190c-2345-44cb-9623-f73b0e7e190c, iamRole: Dynatrace_monitoring_role, accountId: 378629572362, externalId: *****, label: Tools Test, version: 2.0}

2024-09-23 17:53:29 UTC INFO [<f73b0e7e190c-2345-44cb-9623-f73b0e7e190c>] [<vtopology.provider>, AWSFastCheckCallable] STS GetCallerIdentity: {status: OK, statusInfo: , credentials: AWSCredentialsImpl {identifier: 4DEF18A649159633, accessKey: null, tenantUUID: f73b0e7e190c-2345-44cb-9623-f73b0e7e190c, iamRole: Dynatrace_monitoring_role, accountId: 378629572362, externalId: *****,label: Tools Test, version: 2.0}}

2024-09-23 17:53:29 UTC INFO [<f73b0e7e190c-2345-44cb-9623-f73b0e7e190c>] [<vtopology.provider>, AWSFastCheckCallable] AWS FastCheck ec2:DescribeAvailabilityZones permission check completedsuccessfully in region us-west-2 for credentials(AWSCredentialsImpl {identifier: 4DEF18A649159633, accessKey: null, tenantUUID: f73b0e7e190c-2345-44cb-9623-f73b0e7e190c, iamRole: Dynatrace_monitoring_role, accountId: 378629572362, externalId: *****, label: Tools Test, version: 2.0})

2024-09-23 17:53:31 UTC INFO [<f73b0e7e190c-2345-44cb-9623-f73b0e7e190c>] [<vtopology.provider>, AWSErrorLogger] (Credentials: Tools Test) Service AWSResourceGroupsTaggingAPI, endpoint https://tagging.us-west-2.amazonaws.com, returned error: no response, message: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target [Suppressing further identical messages for 3 hours]

2024-09-23 17:53:31 UTC INFO [<f73b0e7e190c-2345-44cb-9623-f73b0e7e190c>] [<vtopology.provider>, AWSFastCheckCallable] AWS FastCheck permissions status in region us-west-2 for credentials(AWSCredentialsImpl {identifier: 4DEF18A649159633, accessKey: null, tenantUUID: f73b0e7e190c-2345-44cb-9623-f73b0e7e190c, iamRole: Dynatrace_monitoring_role, accountId: 378629572362, externalId: *****, label: Tools Test, version: 2.0}):
tag:GetResources ERROR_UNKNOWN. Status: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

cloudwatch:ListMetrics OK.
cloudwatch:GetMetricData OK.

----------------------------------
The application is running as the user: dtuserag

$ ps -ef | grep dtuserag

dtuserag 1338998 1 0 16:09 ? 00:00:00 /opt/dynatrace/gateway/launcher/dynatracegateway -bg -config /var/lib/dynatrace/gateway/config/dynatracegateway.ini

dtuserag 1339002 1338998 2 16:09 ? 00:02:05 /opt/dynatrace/gateway/jre/bin/java -Dcom.compuware.apm.WatchDogTimeout=180 -classpath ./lib/* --add-opens=java.base/java.lang=ALL-UNNAMED -Xms1024M -XX:ErrorFile=/var/log/dynatrace/gateway/hs_err_pid_%p.log -XX:+UseG1GC -XX:+IgnoreUnrecognizedVMOptions -Duser.language=en -Djava.util.logging.manager=com.dynatrace.gen2.foundation.logging.impl.backend.CustomShutdownLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Dorg.xerial.snappy.lib.path=/opt/dynatrace/gateway/lib/native -Dorg.xerial.snappy.lib.name=libsnappyjava.so -Djava.io.tmpdir=/var/lib/dynatrace/gateway/temp -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.security.egd=file:/dev/urandom -DZstdNativePath=/opt/dynatrace/gateway/lib/native/libzstd-jni.so -Xmx10275M -Dcom.compuware.apm.WatchDogPort=50000 com.compuware.apm.collector.core.CollectorImpl -CONFIG_DIR /var/lib/dynatrace/gateway/config

----------------
The Proxy Root cert is stored in trusted.p12   ( created using the info mentioned here )
The application seems to pick all the relevant files to create the runtime cert (runtime.cacerts) by merging cacerts and trusted.p12

Excerpt from the Debug log captured below...
2024-09-23 16:01:44 UTC INFO [<header>] +-----------------------------------------------------------------------------
2024-09-23 16:01:44 UTC INFO [<header>] + Dynatrace LLC
2024-09-23 16:01:44 UTC INFO [<header>] +-----------------------------------------------------------------------------
2024-09-23 16:01:44 UTC INFO [<header>] + Version 1.295.7.20240715-224910
2024-09-23 16:01:44 UTC INFO [<header>] + collector: 0x7cd018cb; tenant: f73b0e7e190c-2345-44cb-9623-f73b0e7e190c;
2024-09-23 16:01:44 UTC INFO [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] Platform: Linux, Version: 5.14.0-284.82.1.el9_2.x86_64, Architecture: amd64, Processors: 4
2024-09-23 16:01:44 UTC INFO [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] VM: OpenJDK 64-Bit Server VM, Version: 17.0.10, Vendor: Eclipse Adoptium, Memory [maxMemory=1
0280M, initHeap=1024M, maxHeap=10280M, usedMeta=7M, committedMeta=7M, totalPhysicalMemory=15809M, freePhysicalMemory=6793M]
2024-09-23 16:01:44 UTC INFO [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] file.encoding: UTF-8, sun.jnu.encoding: UTF-8, user.name: dtuserag
2024-09-23 16:01:44 UTC INFO [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] Input Arguments: -Dcom.compuware.apm.WatchDogTimeout=180 --add-opens=java.base/java.lang=ALL-
UNNAMED -Xms1024M -XX:ErrorFile=/var/log/dynatrace/gateway/hs_err_pid_%p.log -XX:+UseG1GC -XX:+IgnoreUnrecognizedVMOptions -Duser.language=en -Djava.util.logging.manager=com.dynatrace.gen2.
foundation.logging.impl.backend.CustomShutdownLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Dorg.xerial.snappy.lib.path=/opt/dynatrace/gateway/lib/native -Dorg.xerial.snappy.lib.name=libsna
ppyjava.so -Djava.io.tmpdir=/var/lib/dynatrace/gateway/temp -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.security.egd=file:/dev/urandom -DZstdNativePath=/opt/dynatrace/gateway/lib/
native/libzstd-jni.so -Xmx10275M -Dcom.compuware.apm.WatchDogPort=50000
2024-09-23 16:01:44 UTC INFO [<collector>] [<collector.core>, CollectorImpl] No keyfile detected, starting without crypto subsystem.
2024-09-23 16:01:44 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Setting up combined trust store, Java cacerts: TrustStoreSettingsImpl{path=/opt/dynatrace/gateway/jre/lib
/security/cacerts, type=pkcs12}, custom: TrustStoreSettingsImpl{path=/var/lib/dynatrace/gateway/config/../ssl/trusted.p12, type=PKCS12}
2024-09-23 16:01:44 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias concurca
2024-09-23 16:01:44 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias /etc/pki/ca-trust/source/anchors/root-cert-0
2024-09-23 16:01:44 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias /etc/pki/ca-trust/source/anchors/root-cert-1
2024-09-23 16:01:44 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias /etc/pki/ca-trust/source/anchors/root-cert-2
2024-09-23 16:01:44 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias concurcert-0
2024-09-23 16:01:44 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias concurcert-1
2024-09-23 16:01:44 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias concurcert-2
2024-09-23 16:01:45 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Custom certificate configuration created successfully
2024-09-23 16:01:45 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Effective trust store: TrustStoreSettingsImpl{path=/var/lib/dynatrace/gateway/config/../ssl/runtime.cacerts, type=pkcs12}
-------------

I don't why the AWS Tagging endpoint requests is not using the cert named runtime.cacerts ( in the path /var/lib/dynatrace/gateway/ssl )

[dyn98007]

Also to add.. All the 5 Amazon RootCAs are also present in cacerts  (/opt/dynatrace/gateway/jre/lib/security/cacerts ). This is in addition to the Proxy related Root cert

--------------------------------------------------------

[root@ip-10-xx-x-xx ssm-user]# /opt/dynatrace/gateway/jre/bin/keytool -list -v -keystore /opt/dynatrace/gateway/jre/lib/security/cacerts | grep Amazon
Warning: use -cacerts option to access cacerts keystore
Enter keystore password: changeit
Owner: CN=Amazon Root CA 4, O=Amazon, C=US
Issuer: CN=Amazon Root CA 4, O=Amazon, C=US
Owner: CN=Amazon Root CA 1, O=Amazon, C=US
Issuer: CN=Amazon Root CA 1, O=Amazon, C=US
Owner: CN=Amazon Root CA 1, O=Amazon, C=US
Issuer: CN=Amazon Root CA 1, O=Amazon, C=US
Owner: CN=Amazon Root CA 1, O=Amazon, C=US
Issuer: CN=Amazon Root CA 1, O=Amazon, C=US
Owner: CN=Amazon Root CA 2, O=Amazon, C=US
Issuer: CN=Amazon Root CA 2, O=Amazon, C=US
Owner: CN=Amazon Root CA 3, O=Amazon, C=US
Issuer: CN=Amazon Root CA 3, O=Amazon, C=US
Owner: CN=Amazon Root CA 4, O=Amazon, C=US
Issuer: CN=Amazon Root CA 4, O=Amazon, C=US
[root@ip-10-xx-x-xx ssm-user]# /opt/dynatrace/gateway/jre/bin/keytool -list -v -keystore /opt/dynatrace/gateway/jre/lib/security/cacerts | grep Starfield
Warning: use -cacerts option to access cacerts keystore
Enter keystore password: changeit
Owner: CN=Starfield Root Certificate Authority - G2, O="Starfield Technologies, Inc.", L=Scottsdale, ST=Arizona, C=US
Issuer: CN=Starfield Root Certificate Authority - G2, O="Starfield Technologies, Inc.", L=Scottsdale, ST=Arizona, C=US
Owner: CN=Starfield Services Root Certificate Authority - G2, O="Starfield Technologies, Inc.", L=Scottsdale, ST=Arizona, C=US
Issuer: CN=Starfield Services Root Certificate Authority - G2, O="Starfield Technologies, Inc.", L=Scottsdale, ST=Arizona, C=US
Owner: OU=Starfield Class 2 Certification Authority, O="Starfield Technologies, Inc.", C=US
Issuer: OU=Starfield Class 2 Certification Authority, O="Starfield Technologies, Inc.", C=US
[OU=Starfield Class 2 Certification Authority, O="Starfield Technologies, Inc.", C=US]

[dyn98007 · Accepted solution]

Update:

The issue was resolved by making sure the intermediary certs for the Root cert was getting loaded correctly in the creation of Trusted Root cert (.p12). 
Screenshot also attached... it shows that AWSResourceGroupsTaggingAPI endpoint is being accessed successfully