EdgeConnect Kubernetes Setup: caCertRef Issues

calogajac · ‎19 Dec 2024

I'm currently trying to set up an EdgeConnect resource in my AWS EKS cluster using the following instructions:
- Initial Kubernetes RBAC Setup

- Deploying EdgeConnect Via Dynatrace Operator

My current Operator is on the latest version, and I can confirm the needed CRDs are available in my cluster. When deploying as the instructions describe, my Operator spits out the following log message when calling my Dynatrace instance's API:

tls: failed to verify certificate: x509: certificate signed by unknown authority

At this point, I assumed according to the listed parameters here that I would need to put the proper cacert into a Kubernetes configmap in my cluster that the EdgeConnect yaml would reference. Still, the same message occurs in my Operator's logs.

At this point, I've not only tried my organization's root cert by itself in the configmap, but I've also tried using a number of cacerts used by my organization all together in the configmap at the "certs" directory as described. It's important to note in this configmap that all the certs I've used can be found at the "certs" directory, following each other in pem file format like so...

-----BEGIN CERTIFICATE-----
{CERT1 INFO HERE}
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
{CERT2 HERE}
-----END CERTIFICATE-----

And so on...

Regardless of what I try, it seems that I still get the same message.

For anyone who has attempted to do this setup, what have I gotten wrong here? Do each of these certs need a separate key/value pair in the configmap under the "certs" directory, or would pem format under "certs" be desired here?

I also noticed in Workflows where I would run the Kubernetes automation I'm trying to do, an EdgeConnect connection has to be built, which includes the token of the service account created in the RBAC setup. Have I misunderstood these instructions and that service account token decoded as a cacert is what needs to be put into this configmap?

I'm at a bit of a loss here, so any help is appreciated. Thanks in advance!

Peter_Youssef · ‎20 Dec 2024

Hello @calogajac

Thanks for illustration, PFA comment:

Ensure the CA certificates are in PEM format and concatenated together in a single file.
Verify the ConfigMap is correctly mounted into the EdgeConnect operator container.
Double-check the correct file path to the CA certificates used by the operator.
Confirm that service account tokens are not confused with CA certificates.
Use curl or openssl to verify the certificate trust manually within the cluster.
If necessary, update the operator’s deployment YAML to explicitly reference the CA certificate file.

Hoping it helps.

BR,

Peter

calogajac · ‎20 Dec 2024

@Peter_Youssef Thanks for the response and thanks for confirming some of my setup. Something that I wanted to note that I wasn't specific about to your second bullet point is that the EdgeConnect has not ever gotten to a point where a pod is deployed/provisioned with this configuration. When deploying this setup as described in the documentation, the EdgeConnect custom resource that gets put into my cluster goes into an Error state, which corresponds to the logs I get off of my Dynatrace Operator about there being an issue with calling out to the EdgeConnect API endpoint. As a result, the configmap isn't getting mounted as there is no pod being deployed.

This may be a bit of a silly question, but when they refer to "directories" in a configmap within this documentation, are they simply talking about using a key called "certs" within the data section of the configmap? Perhaps I have my configmap set up wrong and it cannot get to the certs because of the setup of my configmap?

Peter_Youssef · ‎20 Dec 2024

Hi @calogajac

We can also check the below factors:

The ConfigMap Reference in the YAML:

In the EdgeConnect deployment YAML, ensure the operator is correctly pointing to the ConfigMap that contains these certificates.
The operator should be set up to mount the certificates from the ConfigMap into the container. As an example:

yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: edgeconnect-ca-bundle
data:
certs: |
-----BEGIN CERTIFICATE-----
{CERT1 INFO HERE}
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
{CERT2 INFO HERE}
-----END CERTIFICATE-----

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: dynatrace-operator
spec:
template:
spec:
volumes:
- name: ca-bundle
configMap:
name: edgeconnect-ca-bundle
containers:
- name: dynatrace-operator
volumeMounts:
- mountPath: /etc/ssl/certs
name: ca-bundle
assuming the operator is configured to use the CA bundle at /etc/ssl/certs.

Remarks:

Through the Correct Path for the Certificates:

Double-check the path where the EdgeConnect operator is looking for CA certificates.
In some cases, the operator may not be automatically configured to use /etc/ssl/certs (the default path for CA certificates in many Linux-based systems).
Review the Dynatrace Operator documentation.

CA Bundle Format:

Ensure that the CA certificates are in the correct format.
Kubernetes typically expects PEM-encoded certificates, so kindly make sure that they are in the format as an example:

-----BEGIN CERTIFICATE

----- and -----

END CERTIFICATE-----.

This is the correct format, but sometimes issues arise if a certificate is in a different encoding.

Hoping it helps addressing the highlighted issue.

BR,

Peter

Peter_Youssef · ‎20 Dec 2024

Hi @calogajac

Regarding the certificate conversion, feel free to utilize the attached resource:

https://pleasantpasswords.com/info/pleasant-password-server/b-server-configuration/3-installing-a-3r...

BR,

Peter

Peter_Youssef · ‎20 Dec 2024

Hello @calogajac
As far as I've understood from your inputs, kindly find the below clarification:

Service Account Token:

Used for authenticating the operator to the Kubernetes API server, not for SSL/TLS verification.
It's not used as a CA certificate.
Typically stored in a Kubernetes secret and used for authentication by the operator when performing tasks like creating Dynatrace resources.

On the other hand

CA Certificate:

Used to validate the TLS certificates from your Dynatrace instance.
That's what should be placed in the ConfigMap, not the service account token.

Kindly Verify TLS with openssl or curl:

Before retrying the operator, manually verify if the Dynatrace instance's certificate is trusted by the Kubernetes nodes.
You can action it from within a pod running in the same Kubernetes cluster to ensure the root CA is correctly trusted:

Via bash:

curl https://your-dynatrace-instance.com --cacert /path/to/ca.crt
This can help you troubleshoot if the CA certificate is properly recognized in the cluster.

Tip:

Update the Dynatrace Operator Deployment (If Required).
If the operator is still failing, try adding an additional --tls-ca-file argument to the Dynatrace operator deployment YAML, pointing to the location where you have mounted the CA certificates.

As an attempt: (assuming CA certificates are in /etc/ssl/certs)

yaml
spec:
containers:
- name: dynatrace-operator
args:
- --tls-ca-file=/etc/ssl/certs/ca.crt

Last factor to consider checking:

Stored Logs, hopefully it will point to extended level of details about the reported error.

Hoping it helps.

KR,

Peter

Peter_Youssef · ‎20 Dec 2024

Regarding the certificate:

Make sure the Certificate configurations are correct:

Single PEM Format for the CA Bundle:
While putting multiple certificates into the certs directory of the ConfigMap, they should be concatenated into a single file.
it means you should not split them into separate key/value pairs in the ConfigMap, but rather ensure all certificates are in a single PEM file format, like so:
-----BEGIN CERTIFICATE-----
{CERT1 INFO HERE} -----
END CERTIFICATE----- -----
BEGIN CERTIFICATE-----
{CERT2 INFO HERE} -----
END CERTIFICATE-----

Peter_Youssef · ‎20 Dec 2024

Hello @calogajac

PFA references:

Hoping it assist addressing the highlighted concern.

BR,

Peter