Configuration-less Advanced SSL Certificate Check Plugin

[r_weber]

There are multiple variants how to validate SSL certificates and alert on expiry. I've taken a look at all of them and missed a lack of automation. I therefor created another one that hopefully overcomes some of the limitations and is easier to use in large environments.

As we are not having this feature out of the box for a long time this might be useful.

Summarizing the various attempts and threads from:

SSL Certification expiration checks out of the box - Details? (@Larry R.)
Does Dynatrace monitor SSL certificate validation (@Akshay S.)
Monitor SSL certificate expiry and generate alert (@Dario C.)
(also the contributors @Július L.'s OneAgent extension and @Leon Van Z.'s ActiveGate extension)

What is different about this plugin?

1. It doesn't need any configuration for hosts/sites that are checked. The endpoints are determined dynamically from already configured (and tagged) synthetic monitors. Add/remove monitors and they will be checked automatically without any additional configuration needs.
2. Error events (about expiring certificates) are posted/attached to the synthetic monitor, where one would expect it, not to custom devices
3. Check intervals can be adjusted in long timeframes - no one needs to check certificate validity every minute or even hour.
4. It is an active gate remote plugin so it can communicate with the Dynatrace API via the active gate.
5. It doesn't consume any licenses for custom metrics!

Where to find it?

You can find the plugin in my personal github repository.

[pieter_luttig]

After updating to v1.30 the plugin has stopped working. I get the below error. 

(I am running AG v1.247.210 on REHL 7.)

Error(/usr/lib64/libc.so.6: version `GLIBC_2.28' not found (required by /opt/dynatrace/remotepluginmodule/plugin_deployment/custom.remote.python.certcheck/cryptography/hazmat/bindings/_rust.abi3.so))

Is v1.30 supported on RHEL 7? This library(GLIBC_2.28) seems to be only available in RHEL 8. The latest version for RHEL 7 is GLIBC_2.17. 

[r_weber · Accepted solution]

Hi @pieter_luttig ,

isn't RHEL7 EOL already?

I'm building the plugin on Debian Buster and Stretch images with Python 3.8 support. Since openssl has native dependencies (glibc) you might need to rebuild the plugin on a RHEL7 machine.
Unfortunately I can not pre-build it for all possible platforms

[pieter_luttig]

Hi @r_weber ,

No RHEL7 is only EOL June 2024.

Sure understood, will rebuild on RHEL7 thanks.

[r_weber]

@pieter_luttig you can now find a specific build for older gclib versions on the 1.4 release page, please try if this works for you.

[pieter_luttig]

@r_weber just tried it out now, and I am getting the below. 

Error(/usr/lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/dynatrace/remotepluginmodule/plugin_deployment/custom.remote.python.certcheck/cryptography/hazmat/bindings/_openssl.abi3.so))

[dave_vos]

Just did the same, with the same result. Error(/lib64/libc.so.6: version `GLIBC_2.25' not found (required by /apps/tapm/remotepluginmodule/plugin_deployment/custom.remote.python.certcheck/cryptography/hazmat/bindings/_openssl.abi3.so))

[r_weber]

@dave_vos @pieter_luttig 
Can you please take a look at this issue on github and post the output from your environments.
I checked on my test environment (CentOS 7.9) with GLIBC 2.17 - all fine and no dependency to 2.25.
Did you make sure you certainly downloaded the right plugin version, completely removed the old one and replaced it?

[r_weber]

What minor version of RHEL are you? 7.4? This has been built with CentOS 7.9 (should match RHEL7.9) the latest update in the 7.x release.

[dave_vos]

We are on RHEL 7.9.

NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"

[pieter_luttig]

We are on RHEL 7.9.

NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"

[r_weber · Accepted solution]

@pieter_luttig @dave_vos 

I specifically built on an old RHEL 7.9 now and added the binary to the 1.40 release on github. Dave already confirmed it working.
Hope that also works for you pieter.

[r_weber · Accepted solution]

New Version 1.40 Released

I have pushed a new version of the plugin which is mainly addressing performance for large installations with hundred's of checked endpoints. I introduced concurrent execution which should speed up the total execution of individual check runs by a factor 7-10.
(Thanks @ct_27 for the feedback!)

You can find the release here.

[ct_27]

WAWHOO!! can't wait to try it.  The version we're running now does 450 SSL checks in 21 seconds and 899 in 42 seconds. Quite linear.   Can't wait to see how many 1.4 can do.

Running RedHat 8.6 on a 4 core 64-bit and 4GB of memory.

[ct_27]

Just wanted to follow-up. Everything is running well.  We have a total of 1,030 SSL checks running across 4 configurations (SSL5, SSL15, SSL29, SSL29_2), which are spread out over 3 1 ActiveGates (Threading at 20 got us down from 3 to 1).  Everything is completing within the 1 minute timeframe.  Problems are now remaining open (not closing after 10 minutes) thus the cross-configuration bug is fixed.

Next we're switching over to using the SSLCheckExpire tag to give teams more flexibility. 

[r_weber · Accepted solution]

New Version 1.50 Released

I have pushed a new version of the plugin which adds some functionality around TLS version support and reporting. It will now also work with older TLS versions (instead of not accepting TLS < 1.2) and include information about the TLS protocol and used ciphers in problem notifications. Additionally it will create custom info events on a synthetic monitor if it detects outdated TLS versions (no weak cipher checks though).

This should give responsible people that chance to identify weak protocols and ciphers additionally to expiring certs.

[Inder]

Hello r_weber, Much appreciate your efforts. 

as you mentioned on git hub "(no plugin builds for older glibc linux platforms yet)". Any idea when it would be out. Keen to test for rhel 7.9 with older glibc for the client.

[olegus]

Hi, I'm new to DT extensions and feel stupid at the moment as I cant figure how to install this plugin to ActiveGate. I'm trying this page to upload it but it gives me errors-

Any suggestion?

[olegus]

Ok, looks like that place (Upload your custom Extension 2) is not the right place to install this plugin.

What I did so far, following this - https://docs.dynatrace.com/docs/extend-dynatrace/extensions/development/extension-how-tos/deploy-an-...

1. copy custom.remote.python.certcheck  folder to /opt/dynatrace/remotepluginmodule/plugin_deployment/ on ActiveGate host

2. Upload zipped extension on this page -

and on the next page Add ActiveGate extension

I had to use Add new technology button first as Upload extension was greyed out for me.

After that I configure the plugin to get all HTTP checks with product:test tag.

Nothing happened yet as all my certificates are up to date.

How can I be sure that plugin works and certificates are indeed monitored? Can I see a list of URLs that plugin imported?

[dave_vos]

You could temporary increase "minimum certificate validaty in days" to a high number (say 365 days). That way you can test if a problem would be created if a certificate is bound to expire within 365 days.

[olegus]

I see... But is there a way to see a list of URls that this plugin checks?

Also, does this plugin accepts tags in a form key:value ?

[r_weber]

The plugin determines the list of URLs from your tagged synthetic monitors itself.
So you create a synthetic monitor, tag it accordingly e.g with "sslcheck" and then create a plugin config that works with this tag.

[olegus]

Thanks for the reply!

we already have >1k monitors split by different groups/products by a tag in a form of key:value

For testing purpose I configure the plugin with this tag:

and set expiration days to 500.. But nothing happened so I wonder how can I troubleshoot ?

How to confirm that tag filter works as expected and cert check request was sent?

[r_weber]

Either use just the key of the tag (not with value), or tag your monitors with another autotag rule based on your existing tags, and then check the Activegate’s plugin execution logs for any info/errors.

[olegus]

Where can I find execution logs? On AG host?

Also about your suggestion to use simple tag - does it mean that key:value not supported?

[olegus]

Almost got it.

For other people who have same questions:

- logs can be found here - /var/lib/dynatrace/remotepluginmodule/log/remoteplugin/custom.remote.python.certcheck

- key:value tags are supported

One thing that I can't make to work is metrics. I set Certificate validity days to 500 and have my monitors failed but metrics are empty - either on Metrics tab of a plugin page or in Data Explorer. Any hint here?

[r_weber]

Did you check the "report metrics" in the plugin configuration to on?
And did you make sure that the API token you configured for the plugin has the metric.ingest (v2) scope?

[olegus]

Yes and Yes

Let me try it again, may be I gave it not enough time to collect metrics before closing the problem?

[dave_vos]

Hi Olegus,

I made small manual in the past. I translated it to English so you use it. Hope it helps.

Dave Vos

[olegus]

Thanks, Dave, nice small manual!

Couple of questions if you dont mind:

-Did you try key:value tags with this plugin?

-What's a Monaco script? 🙂

[dave_vos]

https://engineering.dynatrace.com/open-source/standards/monaco/

A colleague of mine made a script in Monaco which was able to create synthetic HTTP monitors in bulk using an excel sheet with column's environment|name|url|enabled|location as input.

I did not use key:value tags with this plugin. I simple used this tagging rule: HTTP monitors (Synthetic monitor) on HTTP monitors where HTTP monitor name contains 'Certificate'

Note: In my opinion it is better to use specific HTTP monitors for this purpose as you can disable them using no DEM units.