topic Re: Environment Active Gate - AWS monitoring Failed to load configured trustedstore file, aborting custom certificate configuratio in Cloud platforms

Environment Active Gate - AWS monitoring Failed to load configured trustedstore file, aborting custom certificate configuration

dyn98007 — Thu, 19 Sep 2024 04:48:11 GMT

Related to the AWS account connectivity error ….. com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I’m facing an issue where the trustedstore file is not getting loaded correctly from the path - /var/lib/dynatrace/gateway/ssl/trusted.p12.

This is resulting in a java.io.FileNotFoundException. Details below

The trustedstore file file has mentioned in the custom.properties

[collector]

trustedstore = trusted.p12

trustedstore-password = changeit

trustedstore-type = PKCS12

The error mentioned in dynatracegateway_Debug.0.0.log file captured below

2024-09-19 00:07:00 UTC INFO [<header>] +-----------------------------------------------------------------------------

2024-09-19 00:07:00 UTC INFO [<header>] + Version 1.295.7.20240715-224910

2024-09-19 00:07:00 UTC INFO [<header>] + collector: 0x7cd018cb; tenant: fefc24a4-d245-44cb-9623-f73b0e7e190c;

2024-09-19 00:07:00 UTC INFO [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] Platform: Linux, Version: 5.14.0-284.82.1.el9_2.x86_64, Architecture: amd64, Processors: 4

2024-09-19 00:07:00 UTC INFO [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] VM: OpenJDK 64-Bit Server VM, Version: 17.0.10, Vendor: Eclipse Adoptium, Memory [maxMemory=10280M, initHeap=1024M, maxHeap=10280M, usedMeta=7M, committedMeta=7M, totalPhysicalMemory=15809M, freePhysicalMemory=10068M]

2024-09-19 00:07:00 UTC INFO [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] file.encoding: UTF-8, sun.jnu.encoding: UTF-8, user.name: dtuserag

2024-09-19 00:07:00 UTC INFO [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] Input Arguments: -Dcom.compuware.apm.WatchDogTimeout=180 --add-opens=java.base/java.lang=ALL-UNNAMED -Xms1024M -XX:ErrorFile=/var/log/dynatrace/gateway/hs_err_pid_%p.log -XX:+UseG1GC -XX:+IgnoreUnrecognizedVMOptions -Duser.language=en -Djava.util.logging.manager=com.dynatrace.gen2.foundation.logging.impl.backend.CustomShutdownLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Dorg.xerial.snappy.lib.path=/opt/dynatrace/gateway/lib/native -Dorg.xerial.snappy.lib.name=libsnappyjava.so -Djava.io.tmpdir=/var/lib/dynatrace/gateway/temp -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.security.egd=file:/dev/urandom -DZstdNativePath=/opt/dynatrace/gateway/lib/native/libzstd-jni.so -Xmx10275M -Dcom.compuware.apm.WatchDogPort=50000

2024-09-19 00:07:00 UTC INFO [<collector>] [<collector.core>, CollectorImpl] No keyfile detected, starting without crypto subsystem.

2024-09-19 00:07:00 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Setting up combined trust store, Java cacerts: TrustStoreSettingsImpl{path=/opt/dynatrace/gateway/jre/lib/security/cacerts, type=pkcs12}, custom: TrustStoreSettingsImpl{path=/var/lib/dynatrace/gateway/config/../ssl/concurtrusted.p12, type=PKCS12}

2024-09-19 00:07:00 UTC WARNING [<collector>] [<collector.comm>, TrustStoreCreator] Failed to load configured trustedstore file (full path: /var/lib/dynatrace/gateway/config/../ssl/trusted.p12), aborting custom certificate configuration

java.io.FileNotFoundException: /var/lib/dynatrace/gateway/config/../ssl/concurtrusted.p12 (Permission denied)

at java.base/java.io.FileInputStream.open0(Native Method)

at java.base/java.io.FileInputStream.open(Unknown Source)

at java.base/java.io.FileInputStream.<init>(Unknown Source)

at com.dynatrace.apm.collector.shared.communication.api.truststore.TrustStoreCreator.createKeyStoreFromTrustStore(TrustStoreCreator.java:146)

at com.dynatrace.apm.collector.shared.communication.api.truststore.TrustStoreCreator.handleCustomTS(TrustStoreCreator.java:96)

at com.dynatrace.apm.collector.shared.communication.api.truststore.TrustStoreCreator.setupCerts(TrustStoreCreator.java:66)

at com.compuware.apm.collector.core.CollectorImpl.setupTrustStore(CollectorImpl.java:346)

at com.compuware.apm.collector.core.CollectorImpl.<init>(CollectorImpl.java:314)

at com.compuware.apm.collector.core.CollectorImpl.main(CollectorImpl.java:803)

2024-09-19 00:07:01 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Effective trust store: TrustStoreSettingsImpl{path=/opt/dynatrace/gateway/jre/lib/security/cacerts, type=pkcs12}

----------------------------------------------

Due to the failure to load the configured trustedstore file..below error message show up in the dynatracegateway_Debug.0.0.log.. The error comes up when an attempt is made to connect to an AWS account in the Dynatrace UI. Screenshot attached.

Please advise

Error : Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}

Details below:

2024-09-19 03:03:00 UTC INFO [<d245-44cb-9623-f73b0e7e190c >] [<vtopology.provider>, AWSClientFactory] trying to get client initial response for credentials: AWSCredentialsImpl {identifier: BEEB1840447456B1, accessKey: null, secretKey: null, tenantUUID: -d245-44cb-9623-f73b0e7e190c, iamRole: Dynatrace_monitoring_role, accountId: 1233445, externalId: *****, label: Integration Observe RoleBased, partition: aws, detectedPartition: null, monitorOnlyTaggedEntities: false, includeTags: [], excludeTags: [], excludedRegions: [], logConfigSQSesEnabled: false, logConfigSQSes: [], version: 2.0, legacyServices: [ebs_builtin, lambda_builtin, ELB_builtin, loadbalancer_builtin, s3_builtin, dynamodb_builtin, ec2_builtin, asg_builtin, rds_builtin], services: [], extensionDetails: null} [Suppressing further messages for 15 minutes] [skipped logs: 1]

2024-09-19 03:03:00 UTC DEBUG [<d245-44cb-9623-f73b0e7e190c >] [<vtopology.provider>, PartitionAutoDetection] detectedPartition=aws, for credentials: Integration Observe RoleBased [7025090340361789595]

2024-09-19 03:03:01 UTC INFO [] [, RoleCredentialsProvider] Cannot obtain CLIENT short term credentials for arniam::420785284875:role/Dynatrace_monitoring_role ; AWSCredentialsImpl {identifier: BEEB1840447456B1, accessKey: null, secretKey: null, tenantUUID: d245-44cb-9623-f73b0e7e190c iamRole: Dynatrace_monitoring_role, accountId: 420785284875, externalId: *****, label: Integration Observe RoleBased, partition: aws, detectedPartition: aws, monitorOnlyTaggedEntities: false, includeTags: [], excludeTags: [], excludedRegions: [], logConfigSQSesEnabled: false, logConfigSQSes: [], version: 2.0, legacyServices: [ebs_builtin, lambda_builtin, ELB_builtin, loadbalancer_builtin, s3_builtin, dynamodb_builtin, ec2_builtin, asg_builtin, rds_builtin], services: [], extensionDetails: null} [Suppressing further identical messages for 10 minutes]

com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219)

2024-09-19 03:03:01 UTC WARNING [<d245-44cb-9623-f73b0e7e190c >] [<vtopology.provider>, AWSFastCheckCallable] Credentials refresh failed: {status: ERROR_BAD_CREDENTIALS, statusInfo: Service failed to assume role provided in credentials. An unknown error related to role has occurred, credentials: AWSCredentialsImpl {identifier: BEEB1840447456B1, accessKey: null, tenantUUID: d245-44cb-9623-f73b0e7e190c,iamRole: Dynatrace_monitoring_role, accountId: 1233445, externalId: *****, label: Integration Observe RoleBased, version: 2.0}, exception: com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}

Re: Failed to load configured trustedstore file, aborting custom certificate configuration

p_devulapalli — Thu, 19 Sep 2024 04:43:40 GMT

Hi @dyn98007

Looks like a permission issue , does the user have permissions to the directory?

https://docs.dynatrace.com/docs/shortlink/activegate-configuration-trusted-root-certs#check-permissions

Re: Failed to load configured trustedstore file, aborting custom certificate configuration

dyn98007 — Thu, 19 Sep 2024 05:40:05 GMT

Thank you @p_devulapalli for the pointer to check-permissions.

The ownership of the .p12 file was changed to the dtuserag, and the dyntracegateway service was restarted. It fixed the previous error related to java.io.FileNotFoundException ..( as seen below) ... .

2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Setting up combined trust store, Java cacerts: TrustStoreSettingsImpl{path=/opt/dynatrace/gateway/jre/lib/security/cacerts, type=pkcs12}, custom: TrustStoreSettingsImpl{path=/var/lib/dynatrace/gateway/config/../ssl/trusted.p12, type=PKCS12}
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias trustedca
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias /etc/pki/ca-trust/source/anchors/root-cert-0
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias /etc/pki/ca-trust/source/anchors/root-cert-1
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias /etc/pki/ca-trust/source/anchors/root-cert-2
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Custom certificate configuration created successfully
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Effective trust store: TrustStoreSettingsImpl{path=/var/lib/dynatrace/gateway/config/../ssl/runtime.cacerts, type=pkcs12}

---------------------------

But unfortunately the "Invalid credentials" message still show up in the UI while trying to add an AWS account , and with the below captured error info in the dynatracegateway_Debug.0.0.log

-------------------

2024-09-19 04:57:23 UTC WARNING [<f44cb-9623-f73b0e7e190c>] [<vtopology.provider>, AWSFastCheckCallable] Credentials refresh failed: {status: ERROR_BAD_CREDENTIALS, statusInfo: Service failed to assume role provided in credentials. An unknown error related to role has occurred, credentials: AWSCredentialsImpl {identifier: 9B5D617D1A7B8D61, accessKey: null, tenantUUID: f44cb-9623-f73b0e7e190c,iamRole: Dynatrace_monitoring_role, accountId: 484875, externalId: *****, label: Integration Observe RoleBased, version: 2.0}, exception: com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}
2024-09-19 04:57:33 UTC DEBUG [<f44cb-9623-f73b0e7e190c>] [<vtopology.provider>, PartitionAutoDetection] detectedPartition=aws, for credentials: Integration Observe RoleBased [-4866113436641686085]
2024-09-19 04:57:34 UTC INFO [] [, RoleCredentialsProvider] Cannot obtain CLIENT short term credentials for arniam::484875:role/Dynatrace_monitoring_role ; AWSCredentialsImpl {identifier: ACCC65D26456AF7B, accessKey: null, secretKey: null, tenantUUID: f44cb-9623-f73b0e7e190c, iamRole: Dynatrace_monitoring_role, accountId: 484875, externalId: *****, label: Integration Observe RoleBased, partition: aws, detectedPartition: aws, monitorOnlyTaggedEntities: false, includeTags: [], excludeTags: [], excludedRegions: [], logConfigSQSesEnabled: false, logConfigSQSes: [], version: 2.0, legacyServices: [ebs_builtin, lambda_builtin, ELB_builtin, loadbalancer_builtin, s3_builtin, dynamodb_builtin, ec2_builtin, asg_builtin, rds_builtin], services: [], extensionDetails: null} [Suppressing further identical messages for 10 minutes]
com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219)

2024-09-19 04:57:34 UTC WARNING [<f44cb-9623-f73b0e7e190c>] [<vtopology.provider>, AWSFastCheckCallable] Credentials refresh failed: {status: ERROR_BAD_CREDENTIALS, statusInfo: Service failed to assume role provided in credentials. An unknown error related to role has occurred, credentials: AWSCredentialsImpl {identifier: ACCC65D26456AF7B, accessKey: null, tenantUUID: f44cb-9623-f73b0e7e190c,iamRole: Dynatrace_monitoring_role, accountId: 484875, externalId: *****, label: Integration Observe RoleBased, version: 2.0}, exception: com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}
----------------

Re: Environment Active Gate - AWS monitoring Failed to load configured trustedstore file, aborting custom certificate configuratio

Peter_Youssef — Thu, 19 Sep 2024 05:32:52 GMT

Hello @dyn98007

validate your Activegate configurations against Dynatrace guidance references:

Regards,

Re: Failed to load configured trustedstore file, aborting custom certificate configuration

p_devulapalli — Thu, 19 Sep 2024 05:50:41 GMT

@dyn98007 - It could be a issue with the cert that you are using, please make sure its the right one

Re: Failed to load configured trustedstore file, aborting custom certificate configuration

dyn98007 — Thu, 19 Sep 2024 06:57:38 GMT

Hi @p_devulapalli - I did reconfirm that the right one is being used. Thanks!

Re: Environment Active Gate - AWS monitoring Failed to load configured trustedstore file, aborting custom certificate configuratio

dyn98007 — Fri, 20 Sep 2024 02:12:53 GMT

Update : Cert Issue was Solved:

Hi @p_devulapalli , @Peter_Youssef :

The cert issue was finally resolved by having the proxy specifically for Dynatrace Cluster, while allowing outgoing monitoring traffic to connect directly to AWS . Update done for custom.properties by defining proxy settings in the [http.client.internal] section ( instead of [http.client] )

Re: Environment Active Gate - AWS monitoring Failed to load configured trustedstore file, aborting custom certificate configuratio

p_devulapalli — Fri, 20 Sep 2024 02:40:52 GMT

@dyn98007 Good to know 😊

Re: Environment Active Gate - AWS monitoring Failed to load configured trustedstore file, aborting custom certificate configuratio

Peter_Youssef — Fri, 20 Sep 2024 05:34:51 GMT

Thanks @dyn98007 for the updates