[Dynatrace Managed] OneAgent and ActiveGate download fails - cannot verify certificate (expired)

[Radoslaw_Szulgo]

Hey Community!

I've recently been in touch with our users having problems to download OneAgent and ActiveGate from their Dynatrace Managed cluster. The `wget` command fails as following:

wget  -O Dynatrace-OneAgent-Linux-1.217.187.sh https://xxx123.dynatrace-managed.com/e/xxx/api/v1/deployment/installer/agent/unix/default/latest?arch=x86&flavor=default --header="Authorization: Api-Token <token>"
--2021-10-07 19:56:58--  https://xxx123.dynatrace-managed.com/e/xxx/api/v1/deployment/installer/agent/unix/default/latest?arch=x86&flavor=default
Resolving xxx123.dynatrace-managed.com (xxx123.dynatrace-managed.com)... 135.xxx.xxx.xxx, 135.xxx.xxx.xxx, 135.xxx.xxx.xxx
Connecting to xxx123.dynatrace-managed.com (xxx123.dynatrace-managed.com)|135.xxx.xxx.xxx|:443... connected.
ERROR: cannot verify xxx123.dynatrace-managed.com's certificate, issued by '/C=US/O=Let's Encrypt/CN=R3':
  Issued certificate has expired.

"Issues arose not so much because of clients running obsolete versions of operating systems, but because the servers of several organizations failed to serve updated certificate chains to clients due to configuration problems." says portswigger.net. 

The reason is expired Let's Encrypt Root certificate on the host where you want to monitor OneAgent. Here's an official Let's Encrypt statement:

As planned, the DST Root CA X3 has expired and we’re now using our own ISRG Root X1 for trust. We used a cross-sign with DST Root CA X3 to gain broad trust for our certificates when we were just starting out. Now our own root is widely trusted.

(source: https://letsencrypt.org/2021/10/01/cert-chaining-help.html)

Read more to find out details. 

Am I affected?

Instances and containers running the following operating systems might not be able to connect to Dynatrace Managed clusters using Dynatrace-provided certificates:

• CentOS and RHEL 7 or lower
• Amazon Linux and Amazon Linux 2
• Ubuntu 16.04 or lower
• Debian 8 or lower

With OpenSSL 1.0.2, the untrusted chain is always preferred. This means that the expired certificate is seen and the entire chain is distrusted as expired. Servers with the affected version of OpenSSL and the DST Root CA X3 certificate in their root store can't connect to Dynatrace Managed clusters.

[Radoslaw_Szulgo]

Very interesting post-mortem analysis of the issue: https://scotthelme.co.uk/lets-encrypt-root-expiration-post-mortem/

[Radoslaw_Szulgo · Accepted solution]

What should you do?

For most of you, nothing at all! We’ve set up our certificate issuance so your environments will do the right thing in most cases, favoring broad compatibility. However, if you are running legacy operating systems, you’ll need to make sure of two things:

(1) your host or a container must trust ISRG Root X1 (not just DST Root CA X3), and

(2) you must use OpenSSL version 1.1.0 or later. In OpenSSL 1.0.x, a quirk in certificate verification means that even clients that trust ISRG Root X1 will fail when presented with the Android-compatible certificate chain we are recommending by default.

How to trust and update certificates?

How to do this depends on the operating system of the TLS client. For example:

Ubuntu 16.04: Update OpenSSL to version 1.0.2g-1ubuntu4.20 or newer.

Amazon Linux and Amazon Linux 2: Amazon Linux instances can be relaunched to apply the updated ca-certificates package automatically. Should existing instances need to be updated, customers can update ca-certificates by running the following:

sudo yum update ca-certificates

Note: For customers using an AMI with a locked repository GUID, like Elastic Beanstalk, you can install updated ca-certificates package using following commands:

Amazon Linux 1

sudo yum install https://cdn.amazonlinux.com/patch/ca-certificates-update-2021-09-30/ca-certificates-2018.2.22-65.1.24.amzn1.noarch.rpm

Amazon Linux 2

sudo yum install https://cdn.amazonlinux.com/patch/ca-certificates-update-2021-09-30/ca-certificates-2021.2.50-72.amzn2.0.1.noarch.rpm

Red Hat and CentOS 7: Update ca-certificates package to 2021.2.50-72.el7_9 or later.

Or follow a quick fix as in here: https://mender.io/blog/quick-fix-for-lets-encrypt-root-certificate-expiry

Public news:

• Let’s Encrypt root cert update catches out many big-name tech firms