Issue with Citrix NetScaler ADC extension and self-signed certs

LordNykkon — Mon, 19 Aug 2024 11:56:04 GMT

I'm unable to monitor our NetScaler ADCs with the official extension due to an issue with Python and the self-signed SSL certs generated by the ADCs.

The dt.system.events for the extension show the following messages when enabling a configuration:

The ActiveGate logs are as follows (obfuscated server IP for security):

[404422b1-6988-3fc2-9b8e-cd514f1ccb9b][4173992537127303033][7572][out]2024-05-28 11:56:08,237 [INFO] dynatrace_extension.extension (ThreadPoolExecutor-0_0): query method started for Netscaler nitro API. [404422b1-6988-3fc2-9b8e-cd514f1ccb9b][4173992537127303033][7572][out]2024-05-28 11:56:08,237 [INFO] dynatrace_extension.extension (ThreadPoolExecutor-24_0): Polling Netscaler endpoint: "https://*********" [404422b1-6988-3fc2-9b8e-cd514f1ccb9b][4173992537127303033][7572][out]2024-05-28 11:56:08,237 [INFO] dynatrace_extension.extension (ThreadPoolExecutor-24_0): Login: "https://*********/nitro/v1/config/login" [404422b1-6988-3fc2-9b8e-cd514f1ccb9b][4173992537127303033][7572][out]2024-05-28 11:56:08,253 [INFO] dynatrace_extension.extension (ThreadPoolExecutor-0_0): No events were returned [404422b1-6988-3fc2-9b8e-cd514f1ccb9b][4173992537127303033][7572][err]2024-05-28 11:56:08,253 [ERROR] dynatrace_extension.extension (ThreadPoolExecutor-0_0): NetscalerException('Error connecting to Netscaler: "https://*********" : "(\'Connection aborted.\', ConnectionResetError(10054, \'An existing connection was forcibly closed by the remote host\', None, 10054, None))"') [404422b1-6988-3fc2-9b8e-cd514f1ccb9b][4173992537127303033][7572][err]2024-05-28 11:56:08,253 [ERROR] api (ThreadPoolExecutor-0_0): Error running callback Method=netscaler_query: NetscalerException('There were errors when connecting to Netscaler endpoints, check the ActiveGate logs for details') [404422b1-6988-3fc2-9b8e-cd514f1ccb9b][4173992537127303033][7572][err]Traceback (most recent call last): [404422b1-6988-3fc2-9b8e-cd514f1ccb9b][4173992537127303033][7572][err] File "C:\ProgramData\dynatrace\remotepluginmodule\agent\runtime\extensions\python_venvs\com.dynatrace.extension.netscaler_2.0.2\lib\site-packages\dynatrace_extension\callback.py", line 63, in __call__ [404422b1-6988-3fc2-9b8e-cd514f1ccb9b][4173992537127303033][7572][err] result = self.callback(*self.callback_args) [404422b1-6988-3fc2-9b8e-cd514f1ccb9b][4173992537127303033][7572][err] File "C:\ProgramData\dynatrace\remotepluginmodule\agent\runtime\extensions\python_venvs\com.dynatrace.extension.netscaler_2.0.2\lib\site-packages\netscaler\__main__.py", line 77, in netscaler_query [404422b1-6988-3fc2-9b8e-cd514f1ccb9b][4173992537127303033][7572][err] raise NetscalerException( [404422b1-6988-3fc2-9b8e-cd514f1ccb9b][4173992537127303033][7572][err]netscaler.netscaler_api.NetscalerException: There were errors when connecting to Netscaler endpoints, check the ActiveGate logs for details [404422b1-6988-3fc2-9b8e-cd514f1ccb9b][4173992537127303033][7572][out]2024-05-28 11:56:08,612 [INFO] api (ThreadPoolExecutor-1_3): Sent 1 metric lines to EEC: [MintResponse(lines_ok=1, lines_invalid=0, error=None, warnings=None)]

I ran into this problem a few months ago when I was tasked with scripting some reports using the Nitro API. I wrote a client that's similar to yours and would get the following error message when trying to use a standard requests.Session() with the verify=False flag:

After HOURS of researching and troubleshooting, I discovered that its due to a mismatched hostname during the SSL handshake. I found a workaround by sub-classing requests.adapters.HTTPAdapter, setting the check_hostname flag to False, then mounting the adapter to the requests.Session object I passed into the Netscaler class as shown below:

import requests from requests.adapters import HTTPAdapter import ssl import logging from netscaler.nitro_api import Netscaler logging.basicConfig(level=logging.DEBUG) # 1. Subclass HTTAdapter and configure the SSL context class SSLAdapter(HTTPAdapter): def init_poolmanager(self, *args, **kwargs): ssl_context = ssl.create_default_context() ssl_context.set_ciphers('DEFAULT@SECLEVEL=1') ssl_context.check_hostname = False ssl_context.minimum_version = ssl.TLSVersion.TLSv1_2 kwargs["ssl_context"] = ssl_context return super().init_poolmanager(*args, **kwargs) # 2. Create a Requests Session and mount the SSLAdapter this_session = requests.Session() this_session.verify = False this_session.mount('https://', SSLAdapter()) # 3. Create the Netscaler client and pass in the Session along with the IP nitro = Netscaler(session=this_session, ip="<server_IP>") # 4. Login to acquire authentication cookie for the Session logged_in = nitro.login(u_name="NITRO_USERNAME", pw="NITRO_PASSWORD") print(logged_in)

The extension's code (...\site_packages\netscaler\netscaler_api.py) already has logic for handling when Verify SSL Certificates is disabled in the UI configuration:

Could you add the class for the SSLAdapter above and have it be mounted to the session (s) if config.verify=False?

Example:

class SSLAdapter(HTTPAdapter): def init_poolmanager(self, *args, **kwargs): ssl_context = ssl.create_default_context() ssl_context.set_ciphers('DEFAULT@SECLEVEL=1') ssl_context.check_hostname = False ssl_context.minimum_version = ssl.TLSVersion.TLSv1_2 ssl_context.options |= 0x4 kwargs["ssl_context"] = ssl_context return super().init_poolmanager(*args, **kwargs) class NetScaler(): def __init__(self, config: NetscalerConfig): s = requests.session() self.auth = {'username': config.user, 'password': config.password} s.verify = config.verify if not s.verify: s.mount('https://', SSLAdapter()) self.session = s

This would allow users with self-signed certs to connect to the Citrix NetScaler ADCs and get them into Dynatrace!

Re: Issue with Citrix NetScaler ADC extension and self-signed certs (with proposed solution).

jegron — Thu, 30 May 2024 10:05:52 GMT

Hi @LordNykkon ! Very nice investigation. Could you open a support case ?

Re: Issue with Citrix NetScaler ADC extension and self-signed certs (with proposed solution).

LordNykkon — Fri, 31 May 2024 16:02:14 GMT

I brought it up to our account reps and I think they are already escalating it internally. Thank you for the reply though.

Re: Issue with Citrix NetScaler ADC extension and self-signed certs

brayden_neale — Fri, 20 Sep 2024 12:07:01 GMT

Hi @LordNykkon

Thanks for the detailed explanation and solution provided!

I have implemented and tested this in a new version of the Dynatrace NetScaler extension which will be released to the Hub shortly.

This has saved me a lot of time and effort with future support tickets. Before this support guidance was to require customers to enable the netscaler default profile and actually bind ciphers to the internal_frontend_service.

Re: Issue with Citrix NetScaler ADC extension and self-signed certs

LordNykkon — Tue, 24 Sep 2024 16:50:10 GMT

I just updated to the latest version (2.2.1) and can confirm the patch works. All our NetScalers are pulling info now. Thank you!

topic Re: Issue with Citrix NetScaler ADC extension and self-signed certs in Extensions

Issue with Citrix NetScaler ADC extension and self-signed certs

Re: Issue with Citrix NetScaler ADC extension and self-signed certs (with proposed solution).

Re: Issue with Citrix NetScaler ADC extension and self-signed certs (with proposed solution).

Re: Issue with Citrix NetScaler ADC extension and self-signed certs

Re: Issue with Citrix NetScaler ADC extension and self-signed certs