29 Jan 2021 09:14 AM - last edited on 18 May 2023 02:01 PM by Michal_Gebacki
For those who might encounter this as well when writing their AG Remote Plugins a little hint.
While building another plugin with dependencies to another library I was confronted with an error that the included urllib3 library of the remoteplugin environment couldn't load RooT CA certificates. The error in the plugin's log looked something like this:
ERROR [Python][15052339058944823227][Hostunit Consumption][140518812161792][ThreadPoolExecutor-0_2] - [set_full_status] (0)
Reason: SSLError
[Errno 2] No such file or directory
Traceback (most recent call last):
File "/opt/dynatrace/remotepluginmodule/agent/plugin/engine.zip/site-packages/urllib3/util/ssl_.py", line 319, in ssl_wrap_socket
context.load_verify_locations(ca_certs, ca_cert_dir)
FileNotFoundError: [Errno 2] No such file or directory
After digging into it I found that the invoked external SSL connection required the root certificates that are typically shipped with urllib3 or certifi package. (both are included in the remoteplugin runtime environment). However the virtual (?) path would try to find the CA certificate package at this path:
/opt/dynatrace/remotepluginmodule/agent/plugin/engine.zip/site-packages/certifi/cacert.pem
This is not a real path but points to the content of the site.zip archive. Python certifi gets this path by calling certifi.where() and urllib then fails to get the file, leading to this error.
To resolve this issue I had to explicitly point my plugin to an "real" cacert.pem file in the AG's filesystem that is not within a zip file. I manually added ca cacert.pem file (which could be enahanced with corporate CA certs as well to my plugin directory:
/opt/dynatrace/remotepluginmodule/plugin_deployment/custom.remote.python.missingmetrics/cacert.pem
Just posting this here because I was pretty surprised to see this. Maybe it's a bug in the plugin runtime as well that can be fixed permanently as well...
29 Jan 2021 09:21 AM
This is also how we've resolved this with extensions our services team has created for customers. I never dug into it as deep as you did with the root cause though. Thanks for sharing!
17 Feb 2021 11:45 PM
@Reinhard W. thanks for posting this!