Related to the AWS account connectivity error ….. com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
I’m facing an issue where the trustedstore file is not getting loaded correctly from the path - /var/lib/dynatrace/gateway/ssl/trusted.p12.
This is resulting in a java.io.FileNotFoundException. Details below
The trustedstore file file has mentioned in the custom.properties
[collector]
trustedstore = trusted.p12
trustedstore-password = changeit
trustedstore-type = PKCS12
The error mentioned in dynatracegateway_Debug.0.0.log file captured below
2024-09-19 00:07:00 UTC INFO [<header>] +-----------------------------------------------------------------------------
2024-09-19 00:07:00 UTC INFO [<header>] + Version 1.295.7.20240715-224910
2024-09-19 00:07:00 UTC INFO [<header>] + collector: 0x7cd018cb; tenant: fefc24a4-d245-44cb-9623-f73b0e7e190c;
2024-09-19 00:07:00 UTC INFO [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] Platform: Linux, Version: 5.14.0-284.82.1.el9_2.x86_64, Architecture: amd64, Processors: 4
2024-09-19 00:07:00 UTC INFO [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] VM: OpenJDK 64-Bit Server VM, Version: 17.0.10, Vendor: Eclipse Adoptium, Memory [maxMemory=10280M, initHeap=1024M, maxHeap=10280M, usedMeta=7M, committedMeta=7M, totalPhysicalMemory=15809M, freePhysicalMemory=10068M]
2024-09-19 00:07:00 UTC INFO [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] file.encoding: UTF-8, sun.jnu.encoding: UTF-8, user.name: dtuserag
2024-09-19 00:07:00 UTC INFO [<collector>] [<com.compuware.apm.logging>, LoggingServiceImpl] Input Arguments: -Dcom.compuware.apm.WatchDogTimeout=180 --add-opens=java.base/java.lang=ALL-UNNAMED -Xms1024M -XX:ErrorFile=/var/log/dynatrace/gateway/hs_err_pid_%p.log -XX:+UseG1GC -XX:+IgnoreUnrecognizedVMOptions -Duser.language=en -Djava.util.logging.manager=com.dynatrace.gen2.foundation.logging.impl.backend.CustomShutdownLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Dorg.xerial.snappy.lib.path=/opt/dynatrace/gateway/lib/native -Dorg.xerial.snappy.lib.name=libsnappyjava.so -Djava.io.tmpdir=/var/lib/dynatrace/gateway/temp -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.security.egd=file:/dev/urandom -DZstdNativePath=/opt/dynatrace/gateway/lib/native/libzstd-jni.so -Xmx10275M -Dcom.compuware.apm.WatchDogPort=50000
2024-09-19 00:07:00 UTC INFO [<collector>] [<collector.core>, CollectorImpl] No keyfile detected, starting without crypto subsystem.
2024-09-19 00:07:00 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Setting up combined trust store, Java cacerts: TrustStoreSettingsImpl{path=/opt/dynatrace/gateway/jre/lib/security/cacerts, type=pkcs12}, custom: TrustStoreSettingsImpl{path=/var/lib/dynatrace/gateway/config/../ssl/concurtrusted.p12, type=PKCS12}
2024-09-19 00:07:00 UTC WARNING [<collector>] [<collector.comm>, TrustStoreCreator] Failed to load configured trustedstore file (full path: /var/lib/dynatrace/gateway/config/../ssl/trusted.p12), aborting custom certificate configuration
java.io.FileNotFoundException: /var/lib/dynatrace/gateway/config/../ssl/concurtrusted.p12 (Permission denied)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(Unknown Source)
at java.base/java.io.FileInputStream.<init>(Unknown Source)
at java.base/java.io.FileInputStream.<init>(Unknown Source)
at com.dynatrace.apm.collector.shared.communication.api.truststore.TrustStoreCreator.createKeyStoreFromTrustStore(TrustStoreCreator.java:146)
at com.dynatrace.apm.collector.shared.communication.api.truststore.TrustStoreCreator.handleCustomTS(TrustStoreCreator.java:96)
at com.dynatrace.apm.collector.shared.communication.api.truststore.TrustStoreCreator.setupCerts(TrustStoreCreator.java:66)
at com.compuware.apm.collector.core.CollectorImpl.setupTrustStore(CollectorImpl.java:346)
at com.compuware.apm.collector.core.CollectorImpl.<init>(CollectorImpl.java:314)
at com.compuware.apm.collector.core.CollectorImpl.main(CollectorImpl.java:803)
2024-09-19 00:07:01 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Effective trust store: TrustStoreSettingsImpl{path=/opt/dynatrace/gateway/jre/lib/security/cacerts, type=pkcs12}
----------------------------------------------
Due to the failure to load the configured trustedstore file..below error message show up in the dynatracegateway_Debug.0.0.log.. The error comes up when an attempt is made to connect to an AWS account in the Dynatrace UI. Screenshot attached.
Please advise
Error : Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}
Details below:
2024-09-19 03:03:00 UTC INFO [<d245-44cb-9623-f73b0e7e190c >] [<vtopology.provider>, AWSClientFactory] trying to get client initial response for credentials: AWSCredentialsImpl {identifier: BEEB1840447456B1, accessKey: null, secretKey: null, tenantUUID: -d245-44cb-9623-f73b0e7e190c, iamRole: Dynatrace_monitoring_role, accountId: 1233445, externalId: *****, label: Integration Observe RoleBased, partition: aws, detectedPartition: null, monitorOnlyTaggedEntities: false, includeTags: [], excludeTags: [], excludedRegions: [], logConfigSQSesEnabled: false, logConfigSQSes: [], version: 2.0, legacyServices: [ebs_builtin, lambda_builtin, ELB_builtin, loadbalancer_builtin, s3_builtin, dynamodb_builtin, ec2_builtin, asg_builtin, rds_builtin], services: [], extensionDetails: null} [Suppressing further messages for 15 minutes] [skipped logs: 1]
2024-09-19 03:03:00 UTC DEBUG [<d245-44cb-9623-f73b0e7e190c >] [<vtopology.provider>, PartitionAutoDetection] detectedPartition=aws, for credentials: Integration Observe RoleBased [7025090340361789595]
2024-09-19 03:03:01 UTC INFO [] [, RoleCredentialsProvider] Cannot obtain CLIENT short term credentials for arn
iam::420785284875:role/Dynatrace_monitoring_role ; AWSCredentialsImpl {identifier: BEEB1840447456B1, accessKey: null, secretKey: null, tenantUUID: d245-44cb-9623-f73b0e7e190c iamRole: Dynatrace_monitoring_role, accountId: 420785284875, externalId: *****, label: Integration Observe RoleBased, partition: aws, detectedPartition: aws, monitorOnlyTaggedEntities: false, includeTags: [], excludeTags: [], excludedRegions: [], logConfigSQSesEnabled: false, logConfigSQSes: [], version: 2.0, legacyServices: [ebs_builtin, lambda_builtin, ELB_builtin, loadbalancer_builtin, s3_builtin, dynamodb_builtin, ec2_builtin, asg_builtin, rds_builtin], services: [], extensionDetails: null} [Suppressing further identical messages for 10 minutes]
com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219)
2024-09-19 03:03:01 UTC WARNING [<d245-44cb-9623-f73b0e7e190c >] [<vtopology.provider>, AWSFastCheckCallable] Credentials refresh failed: {status: ERROR_BAD_CREDENTIALS, statusInfo: Service failed to assume role provided in credentials. An unknown error related to role has occurred, credentials: AWSCredentialsImpl {identifier: BEEB1840447456B1, accessKey: null, tenantUUID: d245-44cb-9623-f73b0e7e190c,iamRole: Dynatrace_monitoring_role, accountId: 1233445, externalId: *****, label: Integration Observe RoleBased, version: 2.0}, exception: com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}
Thank you @p_devulapalli for the pointer to check-permissions.
The ownership of the .p12 file was changed to the dtuserag, and the dyntracegateway service was restarted. It fixed the previous error related to java.io.FileNotFoundException ..( as seen below) ... .
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Setting up combined trust store, Java cacerts: TrustStoreSettingsImpl{path=/opt/dynatrace/gateway/jre/lib/security/cacerts, type=pkcs12}, custom: TrustStoreSettingsImpl{path=/var/lib/dynatrace/gateway/config/../ssl/trusted.p12, type=PKCS12}
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias trustedca
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias /etc/pki/ca-trust/source/anchors/root-cert-0
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias /etc/pki/ca-trust/source/anchors/root-cert-1
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] overwriting existing certificate with alias /etc/pki/ca-trust/source/anchors/root-cert-2
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Custom certificate configuration created successfully
2024-09-19 04:53:15 UTC INFO [<collector>] [<collector.comm>, TrustStoreCreator] Effective trust store: TrustStoreSettingsImpl{path=/var/lib/dynatrace/gateway/config/../ssl/runtime.cacerts, type=pkcs12}
---------------------------
But unfortunately the "Invalid credentials" message still show up in the UI while trying to add an AWS account , and with the below captured error info in the dynatracegateway_Debug.0.0.log
-------------------
2024-09-19 04:57:23 UTC WARNING [<f44cb-9623-f73b0e7e190c>] [<vtopology.provider>, AWSFastCheckCallable] Credentials refresh failed: {status: ERROR_BAD_CREDENTIALS, statusInfo: Service failed to assume role provided in credentials. An unknown error related to role has occurred, credentials: AWSCredentialsImpl {identifier: 9B5D617D1A7B8D61, accessKey: null, tenantUUID: f44cb-9623-f73b0e7e190c,iamRole: Dynatrace_monitoring_role, accountId: 484875, externalId: *****, label: Integration Observe RoleBased, version: 2.0}, exception: com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}
2024-09-19 04:57:33 UTC DEBUG [<f44cb-9623-f73b0e7e190c>] [<vtopology.provider>, PartitionAutoDetection] detectedPartition=aws, for credentials: Integration Observe RoleBased [-4866113436641686085]
2024-09-19 04:57:34 UTC INFO [] [, RoleCredentialsProvider] Cannot obtain CLIENT short term credentials for arniam::484875:role/Dynatrace_monitoring_role ; AWSCredentialsImpl {identifier: ACCC65D26456AF7B, accessKey: null, secretKey: null, tenantUUID: f44cb-9623-f73b0e7e190c, iamRole: Dynatrace_monitoring_role, accountId: 484875, externalId: *****, label: Integration Observe RoleBased, partition: aws, detectedPartition: aws, monitorOnlyTaggedEntities: false, includeTags: [], excludeTags: [], excludedRegions: [], logConfigSQSesEnabled: false, logConfigSQSes: [], version: 2.0, legacyServices: [ebs_builtin, lambda_builtin, ELB_builtin, loadbalancer_builtin, s3_builtin, dynamodb_builtin, ec2_builtin, asg_builtin, rds_builtin], services: [], extensionDetails: null} [Suppressing further identical messages for 10 minutes]
com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219)
2024-09-19 04:57:34 UTC WARNING [<f44cb-9623-f73b0e7e190c>] [<vtopology.provider>, AWSFastCheckCallable] Credentials refresh failed: {status: ERROR_BAD_CREDENTIALS, statusInfo: Service failed to assume role provided in credentials. An unknown error related to role has occurred, credentials: AWSCredentialsImpl {identifier: ACCC65D26456AF7B, accessKey: null, tenantUUID: f44cb-9623-f73b0e7e190c,iamRole: Dynatrace_monitoring_role, accountId: 484875, externalId: *****, label: Integration Observe RoleBased, version: 2.0}, exception: com.amazonaws.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target}
----------------
Featured Posts
