Dynatrace Managed installer fails SSL: CERTIFICATE_VERIFY_FAILED

[mirzet_kadic]

Hi,

When I try to install Dynatrace Managed:

dynatrace-managed-1.191.0.20200316-134212.sh --tmp-dir "/var/opt/dynatrace-tmp" \
--install \
--install-silent \
--system-user "dynatrace:dynatrace" \
--license "XXXXXXXXXX" \
--initial-environment "main" \
--initial-first-name "Max" \
--initial-last-name "Musermann" \
--initial-email "max.mustermann@example.com" \
--initial-pass "Demo123!" \
--binaries-dir "/opt/dynatrace-managed" \
--network-proxy "http://server-proxy.example.ch:8080/ " \
--datastore-dir "/var/opt/dynatrace-managed"

I'm getting the error;

Network proxy used for this installation in silent mode: http://server-proxy.example.ch:8080 
Testing connection to Dynatrace Mission Control ... failed

/var/log/dynatrace/install.log says:

2020-03-25 13:41:34 UTC Network proxy used for this installation in silent mode: http://server-proxy.example.ch:8080 
2020-03-25 13:41:34 UTC Testing connection to: https://mcsvc-dev.dynatracelabs.com:443 , via proxy: http://server-proxy.example.ch:8080  ...
2020-03-25 13:41:35 UTC Testing connection ... failed, with error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
2020-03-25 13:41:35 UTC Testing connection to: https://opcsvc-dev.ruxitlabs.com:443 , via proxy: http://server-proxy.example.ch:8080  ...
2020-03-25 13:41:36 UTC Testing connection ... failed, with error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
2020-03-25 13:41:36 UTC Cannot connect to Dynatrace Mission Control via network proxy http://server-proxy.example.ch:8080 
2020-03-25 13:41:36 UTC Checking connection to Mission Control .. failed. Please check network and security settings

The proxy intercepts the SSL Traffic (MITM) using a certificate which is signed by a internal CA.

Can you help?

Thanks!

[Radoslaw_Szulgo · Accepted solution]

See my instructions below. If you are installing the first node - follow steps 1-3. Then try with the installer. Let me know how it goes.

How to find the root cause of the issue?

See installer log to inspect issues in connectivity to Mission Control service. Installer log can be found at <DYNATRACE_BINARIES_DIR>/log/installer/*-install-of-managed-installer.log (for instance: 20190927-091357-success-install-of-managed-installer.log). Here’s a typical issue occurrence:

2019-09-27 07:18:58 UTC Testing connection tåo Dynatrace Mission Control https://mcsvc.dynatrace.com:443 ... failed, with error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

Why installer cannot connect to Mission Control but the Server is running fine?

Installer checks Mission Control connectivity to configure the cluster to use either opcsvc.ruxit.com domain or mcsvc.dynatrace.com domain for all the communication endpoints. Installer use core OS certificate repository to validate the chain of SSL certificates exposed by Mission Control. Server use it’s keystore to check the chain.

If a proxy server or a firewall service is used between the cluster that intercepts the traffic, the verification of the chain also includes that proxy/firewall certificates to guarantee secure connection.

How to check if it works or not?

The simplest test is to use `curl` command. Example:

$ curl -x http://proxyuser:proxypass@outbound-proxy.dynatracelabs.com:8080 -I https://mcsvc.dynatrace.com

How to solve it?

In that situation it is required to update OS trusted root certificates. You need to prepare:

• SSH access to the cluster node’s host
• root permissions

Use the following steps to add Proxy/Firewall public certificate and chain to the system.

1. Update OS trusted root certificates. Use hints below how to do it in a given Linux distribution:

Linux (Ubuntu)

Function	Method
Add	Copy your CA to dir /usr/local/share/ca-certificates/ Use command: sudo cp foo.crt /usr/local/share/ca-certificates/foo.crt Update the CA store: sudo update-ca-certificates

Linux (Red Hat / CentOs 6)

Function	Method
Add	Install the ca-certificates package: yum install ca-certificates Enable the dynamic CA configuration feature: update-ca-trust force-enable Add it as a new file to /etc/pki/ca-trust/source/anchors/: cp foo.crt /etc/pki/ca-trust/source/anchors/ Use command: update-ca-trust

2. Set system variable to configure OS trusted root certificates path:

export REQUESTS_CA_BUNDLE=<TRUSTED_CERT_REPO_DIR>

for instance for Red Hat:
export REQUESTS_CA_BUNDLE=/etc/pki/ca-trust/source/anchors/


3. For root user add that variable to bash profile, e.g. :

echo 'export REQUESTS_CA_BUNDLE=/etc/pki/ca-trust/source/anchors/' >> ~/.bash_profile

4. Run cluster node reconfiguration (note, it restarts all the services)

<DYNATRACE_BINARIES_DIR>/installer/reconfigure.sh

5. Check install.log and server config if Mission Control endpoint was updated successfully. Server config can be checked at:

<DYNATRACE_BINARIES_DIR>/server/conf/config.properties

And should contain:
 
 #opc management url

opcUrl=https://mcsvc.dynatrace.com/

6. Repeat for all cluster nodes.

[Radoslaw_Szulgo]

And that also might be needed to ensure later on Server has the connection as well:

Add an SSL certificate to Dynatrace Managed cluster TrustStore

[mirzet_kadic]

Thank you @Radoslaw S.

The hint with export REQUESTS_CA_BUNDLE solved the ssl problem but now i'm getting a new error:

2020-03-26 08:58:27 UTC Checking connection to Mission Control ..

2020-03-26 08:58:27 UTC Network proxy used for this installation in silent mode: http://server-proxy.example.ch:8080

2020-03-26 08:58:27 UTC Testing connection to: https://mcsvc-dev.dynatracelabs.com:443, via proxy: http://server-proxy.example.ch:8080 ...

2020-03-26 08:58:27 UTC Testing connection ... failed, with error: [Errno 21] Is a directory

2020-03-26 08:58:27 UTC Testing connection to: https://opcsvc-dev.ruxitlabs.com:443, via proxy: http://server-proxy.example.ch:8080 ...

2020-03-26 08:58:27 UTC Testing connection ... failed, with error: [Errno 21] Is a directory

2020-03-26 08:58:27 UTC Cannot connect to Dynatrace Mission Control via network proxy http://server-proxy.example.ch:8080

2020-03-26 08:58:27 UTC Checking connection to Mission Control .. failed. Please check network and security settings

the curl test is sucessfull:

[root@dyndemo dynatrace-managed-installer]# curl -x http://server-proxy.example.ch:8080 -I https://mcsvc.dynatrace.com

HTTP/1.1 200 Connection established

Proxy-Agent: Fortinet-Proxy/1.0

HTTP/1.1 200 OK

Server: nginx

Date: Thu, 26 Mar 2020 09:09:35 GMT

Content-Type: text/html; charset=utf-8

Content-Length: 78

Last-Modified: Mon, 23 Mar 2020 11:23:00 GMT

Connection: keep-alive

ETag: "5e789c14-4e"

Keep-Alive: timeout=55, max=16384

Accept-Ranges: bytes

[Radoslaw_Szulgo]

my bad... REQUESTS_CA_BUNDLE should be set to the cert file instead of the directory. I'll update the procedure in my comment as well.

Let me know if that helped!

[mirzet_kadic]

Tank you @Radoslaw S.

That solved the problem but the next one popped up:

2020-03-26 13:12:48 UTC REST GET ... url: https://mcsvc-dev.dynatracelabs.com:443/rest/public/v2.0/installation, user: installer ...

2020-03-26 13:12:49 UTC REST GET ... failed, with error 3 - invalid user name or password: {"code":401,"message":"It appears that you don't have permission to visit the page."}

2020-03-26 13:12:49 UTC Failed to get installation info from Mission Control - it can be caused by incorrect network proxy settings or by invalid license key

Installation failed, with status: system verified, connected to Mission Control after 1 minute 50 seconds.

Exit code: 3

Errors:

Failed to get installation info from Mission Control - it can be caused by incorrect network proxy settings or by invalid license key

2020-03-26 13:12:49 UTC Exit code is 3

===== End of installation, started at 2020-03-26 14:10:58, lasting 1m 50s =====

with curl I don't have any issues:

curl -x https://server-proxy.xaas.swissic.ch:8080 -I https://mcsvc-dev.dynatracelabs.com 

the license is valid until Nov. 25, 2021

[Radoslaw_Szulgo]

Installer was downloaded incorrectly. Additionally, license is already in use.

[mirzet_kadic]

Thank you very much @Radoslaw S.

This issue is solved

[AbdelMoh_Salouk]

Hi @mirzet_kadic 

How the issue has been solved, is it by download installer again or something else.

Regards

Abdelmohsen