cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Environment Active Gate - AWS monitoring Failed to load configured trustedstore file, aborting custom certificate configuration

dyn98007
Helper

Related to the AWS account connectivity error ….. com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

 

I’m facing an issue where the  trustedstore file is not getting loaded correctly from the path - /var/lib/dynatrace/gateway/ssl/trusted.p12.

This is resulting in a java.io.FileNotFoundException. Details below

The trustedstore file  file has mentioned in the custom.properties

[collector]

trustedstore = trusted.p12

trustedstore-password = changeit

trustedstore-type = PKCS12

 

 

The error mentioned in dynatracegateway_Debug.0.0.log file captured below

 

2024-09-19 00:07:00 UTC INFO    [<header>] +-----------------------------------------------------------------------------

2024-09-19 00:07:00 UTC INFO    [<header>] + Version 1.295.7.20240715-224910

2024-09-19 00:07:00 UTC INFO    [<header>] + collector: 0x7cd018cb; tenant: fefc24a4-d245-44cb-9623-f73b0e7e190c;

2024-09-19 00:07:00 UTC INFO    [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] Platform: Linux, Version: 5.14.0-284.82.1.el9_2.x86_64, Architecture: amd64, Processors: 4

2024-09-19 00:07:00 UTC INFO    [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] VM: OpenJDK 64-Bit Server VM, Version: 17.0.10, Vendor: Eclipse Adoptium, Memory [maxMemory=10280M, initHeap=1024M, maxHeap=10280M, usedMeta=7M, committedMeta=7M, totalPhysicalMemory=15809M, freePhysicalMemory=10068M]

2024-09-19 00:07:00 UTC INFO    [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] file.encoding: UTF-8, sun.jnu.encoding: UTF-8, user.name: dtuserag

2024-09-19 00:07:00 UTC INFO    [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] Input Arguments: -Dcom.compuware.apm.WatchDogTimeout=180 --add-opens=java.base/java.lang=ALL-UNNAMED -Xms1024M -XX:ErrorFile=/var/log/dynatrace/gateway/hs_err_pid_%p.log -XX:+UseG1GC -XX:+IgnoreUnrecognizedVMOptions -Duser.language=en -Djava.util.logging.manager=com.dynatrace.gen2.foundation.logging.impl.backend.CustomShutdownLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Dorg.xerial.snappy.lib.path=/opt/dynatrace/gateway/lib/native -Dorg.xerial.snappy.lib.name=libsnappyjava.so -Djava.io.tmpdir=/var/lib/dynatrace/gateway/temp -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.security.egd=file:/dev/urandom -DZstdNativePath=/opt/dynatrace/gateway/lib/native/libzstd-jni.so -Xmx10275M -Dcom.compuware.apm.WatchDogPort=50000

2024-09-19 00:07:00 UTC INFO    [<collector>] [<collector.core>, CollectorImpl] No keyfile detected, starting without crypto subsystem.

2024-09-19 00:07:00 UTC INFO    [<collector>] [<collector.comm>, TrustStoreCreator] Setting up combined trust store, Java cacerts: TrustStoreSettingsImpl{path=/opt/dynatrace/gateway/jre/lib/security/cacerts, type=pkcs12}, custom: TrustStoreSettingsImpl{path=/var/lib/dynatrace/gateway/config/../ssl/concurtrusted.p12, type=PKCS12}

2024-09-19 00:07:00 UTC WARNING [<collector>] [<collector.comm>, TrustStoreCreator] Failed to load configured trustedstore file (full path: /var/lib/dynatrace/gateway/config/../ssl/trusted.p12), aborting custom certificate configuration

java.io.FileNotFoundException: /var/lib/dynatrace/gateway/config/../ssl/concurtrusted.p12 (Permission denied)

        at java.base/java.io.FileInputStream.open0(Native Method)

        at java.base/java.io.FileInputStream.open(Unknown Source)

        at java.base/java.io.FileInputStream.<init>(Unknown Source)

        at java.base/java.io.FileInputStream.<init>(Unknown Source)

        at com.dynatrace.apm.collector.shared.communication.api.truststore.TrustStoreCreator.createKeyStoreFromTrustStore(TrustStoreCreator.java:146)

        at com.dynatrace.apm.collector.shared.communication.api.truststore.TrustStoreCreator.handleCustomTS(TrustStoreCreator.java:96)

        at com.dynatrace.apm.collector.shared.communication.api.truststore.TrustStoreCreator.setupCerts(TrustStoreCreator.java:66)

        at com.compuware.apm.collector.core.CollectorImpl.setupTrustStore(CollectorImpl.java:346)

        at com.compuware.apm.collector.core.CollectorImpl.<init>(CollectorImpl.java:314)

        at com.compuware.apm.collector.core.CollectorImpl.main(CollectorImpl.java:803)

 

2024-09-19 00:07:01 UTC INFO    [<collector>] [<collector.comm>, TrustStoreCreator] Effective trust store: TrustStoreSettingsImpl{path=/opt/dynatrace/gateway/jre/lib/security/cacerts, type=pkcs12}

----------------------------------------------

 

Due to the failure to load the configured trustedstore file..below error message show up in the dynatracegateway_Debug.0.0.log..  The error comes up when an attempt is made to connect to an AWS account in the Dynatrace UI. Screenshot attached.

Please advise

 

 

Error : Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target} 

 

Details below:

 

2024-09-19 03:03:00 UTC INFO    [<d245-44cb-9623-f73b0e7e190c >] [<vtopology.provider>, AWSClientFactory] trying to get client initial response for credentials: AWSCredentialsImpl {identifier: BEEB1840447456B1, accessKey: null, secretKey: null, tenantUUID: -d245-44cb-9623-f73b0e7e190c, iamRole: Dynatrace_monitoring_role, accountId: 1233445, externalId: *****, label: Integration Observe RoleBased, partition: aws, detectedPartition: null, monitorOnlyTaggedEntities: false, includeTags: [], excludeTags: [], excludedRegions: [], logConfigSQSesEnabled: false, logConfigSQSes: [], version: 2.0, legacyServices: [ebs_builtin, lambda_builtin, ELB_builtin, loadbalancer_builtin, s3_builtin, dynamodb_builtin, ec2_builtin, asg_builtin, rds_builtin], services: [], extensionDetails: null} [Suppressing further messages for 15 minutes] [skipped logs: 1]

2024-09-19 03:03:00 UTC DEBUG   [<d245-44cb-9623-f73b0e7e190c >] [<vtopology.provider>, PartitionAutoDetection] detectedPartition=aws, for credentials: Integration Observe RoleBased [7025090340361789595]

2024-09-19 03:03:01 UTC INFO    [] [, RoleCredentialsProvider] Cannot obtain CLIENT short term credentials for arn:aws:iam::420785284875:role/Dynatrace_monitoring_role ; AWSCredentialsImpl {identifier: BEEB1840447456B1, accessKey: null, secretKey: null, tenantUUID: d245-44cb-9623-f73b0e7e190c iamRole: Dynatrace_monitoring_role, accountId: 420785284875, externalId: *****, label: Integration Observe RoleBased, partition: aws, detectedPartition: aws, monitorOnlyTaggedEntities: false, includeTags: [], excludeTags: [], excludedRegions: [], logConfigSQSesEnabled: false, logConfigSQSes: [], version: 2.0, legacyServices: [ebs_builtin, lambda_builtin, ELB_builtin, loadbalancer_builtin, s3_builtin, dynamodb_builtin, ec2_builtin, asg_builtin, rds_builtin], services: [], extensionDetails: null} [Suppressing further identical messages for 10 minutes]

com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219)

 

2024-09-19 03:03:01 UTC WARNING [<d245-44cb-9623-f73b0e7e190c >] [<vtopology.provider>, AWSFastCheckCallable] Credentials refresh failed: {status: ERROR_BAD_CREDENTIALS, statusInfo: Service failed to assume role provided in credentials. An unknown error related to role has occurred, credentials: AWSCredentialsImpl {identifier: BEEB1840447456B1, accessKey: null, tenantUUID: d245-44cb-9623-f73b0e7e190c,iamRole: Dynatrace_monitoring_role, accountId: 1233445, externalId: *****, label: Integration Observe RoleBased, version: 2.0}, exception: com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}

8 REPLIES 8

p_devulapalli
Champion

Hi @dyn98007 

Looks like a permission issue , does the user have permissions to the directory?

https://docs.dynatrace.com/docs/shortlink/activegate-configuration-trusted-root-certs#check-permissi...

 

Phani Devulapalli

Thank you @p_devulapalli  for the pointer to check-permissions.

The ownership of the .p12 file was changed to the dtuserag, and the dyntracegateway service was restarted. It fixed the previous error related to java.io.FileNotFoundException ..( as seen below) ... .

 

2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Setting up combined trust store, Java cacerts: TrustStoreSettingsImpl{path=/opt/dynatrace/gateway/jre/lib/security/cacerts, type=pkcs12}, custom: TrustStoreSettingsImpl{path=/var/lib/dynatrace/gateway/config/../ssl/trusted.p12, type=PKCS12}
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias trustedca
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias /etc/pki/ca-trust/source/anchors/root-cert-0
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias /etc/pki/ca-trust/source/anchors/root-cert-1
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias /etc/pki/ca-trust/source/anchors/root-cert-2
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Custom certificate configuration created successfully
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Effective trust store: TrustStoreSettingsImpl{path=/var/lib/dynatrace/gateway/config/../ssl/runtime.cacerts, type=pkcs12}

---------------------------

But unfortunately the "Invalid credentials" message still show up in the UI while trying to add an AWS account , and with the below captured error info in the dynatracegateway_Debug.0.0.log

-------------------

2024-09-19 04:57:23 UTC WARNING [<f44cb-9623-f73b0e7e190c>] [<vtopology.provider>, AWSFastCheckCallable] Credentials refresh failed: {status: ERROR_BAD_CREDENTIALS, statusInfo: Service failed to assume role provided in credentials. An unknown error related to role has occurred, credentials: AWSCredentialsImpl {identifier: 9B5D617D1A7B8D61, accessKey: null, tenantUUID: f44cb-9623-f73b0e7e190c,iamRole: Dynatrace_monitoring_role, accountId: 484875, externalId: *****, label: Integration Observe RoleBased, version: 2.0}, exception: com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}
2024-09-19 04:57:33 UTC DEBUG [<f44cb-9623-f73b0e7e190c>] [<vtopology.provider>, PartitionAutoDetection] detectedPartition=aws, for credentials: Integration Observe RoleBased [-4866113436641686085]
2024-09-19 04:57:34 UTC INFO [] [, RoleCredentialsProvider] Cannot obtain CLIENT short term credentials for arn:aws:iam::484875:role/Dynatrace_monitoring_role ; AWSCredentialsImpl {identifier: ACCC65D26456AF7B, accessKey: null, secretKey: null, tenantUUID: f44cb-9623-f73b0e7e190c, iamRole: Dynatrace_monitoring_role, accountId: 484875, externalId: *****, label: Integration Observe RoleBased, partition: aws, detectedPartition: aws, monitorOnlyTaggedEntities: false, includeTags: [], excludeTags: [], excludedRegions: [], logConfigSQSesEnabled: false, logConfigSQSes: [], version: 2.0, legacyServices: [ebs_builtin, lambda_builtin, ELB_builtin, loadbalancer_builtin, s3_builtin, dynamodb_builtin, ec2_builtin, asg_builtin, rds_builtin], services: [], extensionDetails: null} [Suppressing further identical messages for 10 minutes]
com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219)

2024-09-19 04:57:34 UTC WARNING [<f44cb-9623-f73b0e7e190c>] [<vtopology.provider>, AWSFastCheckCallable] Credentials refresh failed: {status: ERROR_BAD_CREDENTIALS, statusInfo: Service failed to assume role provided in credentials. An unknown error related to role has occurred, credentials: AWSCredentialsImpl {identifier: ACCC65D26456AF7B, accessKey: null, tenantUUID: f44cb-9623-f73b0e7e190c,iamRole: Dynatrace_monitoring_role, accountId: 484875, externalId: *****, label: Integration Observe RoleBased, version: 2.0}, exception: com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}
----------------

@dyn98007 - It could be a issue with the cert that you are using, please make sure its the right one 

Phani Devulapalli

Hi @p_devulapalli  - I did reconfirm that the right one is being used. Thanks!

Peter_Youssef
Champion

Hello @dyn98007 

validate your Activegate configurations against Dynatrace guidance references:

  1. trusted-certificates-and-custom-certificates 
  2. activegate-ciphers 

Regards,

dyn98007
Helper

Update : Cert Issue was Solved:

Hi @p_devulapalli , @Peter_Youssef  :

The cert issue was finally resolved by having the proxy specifically for Dynatrace Cluster, while allowing outgoing monitoring traffic to connect directly to AWS . Update done for custom.properties by defining proxy settings in the [http.client.internal] section ( instead of [http.client] )

 

@dyn98007  Good to know 😊

Phani Devulapalli

Thanks @dyn98007 for the updates

Featured Posts