Cluster Activegate Test Connection Fail

[mohit_gupta]

Dear All,

I am setting up a Cluster ActiveGate for Mobile RUM and Synthetic monitoring while Testing connection to URL I am getting below issue.

Is SSL certificate is a mandatory thing for Mobile RUM?

[franz_soengen]

Hi Mohit,

you are getting these errors because your Cluster ActiveGate is not reachable from the Internet. You need to set a publicly available URL that your Mobile Users can reach.

Regarding your second question: all communication is encrypted so you'll need a working SSL configuration.

best regards
Franz

[mohit_gupta]

Hi Franz,

Thanks for your quick reply

Here am not targeting users who are coming from the internet instead am focusing on the on-premises (Local users) for which I think a valid SSL certificate with the domain name will be sufficient.

[Julius_Loman · Accepted solution]

As @Franz S. says, this test is performed from the internet. So if your cluster ActiveGate isn't reachable from the internet, this test will fail.
If you are targeting mobile apps on a private network, it's probably ok. For mobile apps, you definitely need to have the gateway reachable from mobile devices (can be on private ip addresses) and also the certificate, which is issued for the FQDN of your gateway and is trusted by your mobile devices. The default certificate is selfsigned and will not work.

[mohit_gupta]

Hi Julius,

Thanks for the answer.

[kalle_lahtinen]

Hi,

In order for Dynatrace's public synthetic monitoring nodes to send data to a Cluster ActiveGate, do we need port 443, 9999, or both to be open towards the internet?

It doesn't say here: https://www.dynatrace.com/support/help/setup-and-configuration/dynatrace-activegate/configuration/wh...

[kalle_lahtinen]

The below doc contains somewhat conflicting info, as the picture shows TCP 9999 but the text says "[Cluster ActiveGate] external communication is only supported in a secure manner using HTTPS (port 443)". So I'm still not sure which one 🙂

https://www.dynatrace.com/support/help/setup-and-configuration/dynatrace-managed/installation/manage...

[mohit_gupta]

Hi Kallel,

Until mobile beacon for RUM and Synthetic monitoring servers is able to send data to cluster active gate every configuration is fine.

You will need only one port to be opened and accessible from the internet. It depends on your networking team and firewall team what they permit as per the policies. Dynatrace active gate only listens to port 9999.

[kalle_lahtinen]

Thanks for the response, I believe you're correct. The reference in the documentation to port 443 should probably be replaced with 9999...

[rmeli]

We are trying to do the exact same thing. Can someone explain how this whole process flow happens and what IP addresses it is generated from. We have to whitelist specific IP addresses.

[kalle_lahtinen]

I believe it's currently described like this:

Source: Internet

Port: TCP/9999

Destination: Cluster ActiveGate

So basically you'd need to allow all incoming connections from the internet for TCP/9999. I haven't seen any specific IPs mentioned (like there is for Mission Control), the requirement is to allow the whole internet in.

[rmeli]

We can't allow the whole internet access for the Test environment we are currently working in. We can when we get to our Production environment.

[skrystosik]

For test purposes if you want to collect data from agentless rum monitoring or mobile app, devices has to be in network that has access to activegate, so you can use vpn or just use corporate wifi. You can as well use F5 before ActiveGate to not expose it individually.

Sebastian

[rmeli]

Thanks for the answers. I am trying the mobile app now. I am also on the corporate network and I can ping the F5 that sits in front of the cluster activegates. Is there something I can do to test that part of the connection? This is all new to me.

[Babar_Qayyum]

Dear All,

We are going through the same situation. In Scenario 3: Integration with existing IT landscape is mentioned that port # 443 will be used for external contents so we did the same. Even though with the Cluster ActiveGate URL test connection is failing.

https://www.dynatrace.com/support/help/setup-and-configuration/dynatrace-managed/installation/manage...

Any hint in this regard?

Regards,

Babar

[Julius_Loman]

Default is the 9999/tcp. So you either have to reconfigure the Cluster ActiveGate to use the 443/tcp which I think may not directly work since binding to privileged ports (<1024) requires root/administrative rights for the user. Gateway does not run as root.

443 is mentions as it typically is the port firewalls are passing for standard SSL communications. Thus you will need a load balancer before the Cluster ActiveGate that will listen on the 443 and pass it to the Cluster ActiveGate.

[mohit_gupta]

Hi Babar,

Julius is correct, I have implemented the same using Load Balancer. For your better understanding please find the below architecture diagram of what I have implemented for one of our clients.

[Babar_Qayyum]

Hello @Julius L. and @Mohit G.

Thank you for your reply.

We have the same setup. I meant traffic is terminating on the LB using port 443 and then natting with LB VIP and forwarding traffic to the Cluster ActiveGates.

Do we need to open the Firewall for TCP port 9999 between LB and Cluster AciveGates?

This is the different thing I found in @Mohit G. diagram.

Regards,

Babar

[Julius_Loman]

If there is a firewall between the LB and Cluster ActiveGate that is blocking the ActiveGate port (9999) you definitely have to open it.

[mohit_gupta]

Yes,

If there is a firewall b/w LB and Active gate you need to open port 9999.

Regards,

MG

[Babar_Qayyum]

Hello @Mohit G. and @Julius L.

I checked with the security and they said there is no firewall between LB and Cluster ActiveGates.

What else could be the reason for this issue?

Do we need a proxy on the Cluster ActiveGates for the Internet?

Public IP address having the following result but with the domain name, all test are failed.

Which area should be focused on this situation?

Regards,

Babar

[Julius_Loman]

As it shows SSL certificate problem and I see you have an IP address written in the URL.

The Cluster ActiveGate or the F5 (not sure which one does SSL termination in your case) has a certificate. Please check the certificate as likely it is not valid for the URL. The URL should be a valid FQDN and certificate returned by the ActiveGate or the F5 must have a match for the FQDN.

[mohit_gupta]

Hi Babar,

Definitely it will not work. As per the screenshot, you are providing IP and Port (Probably the public IP) it will only work for intranet communications, not for the internet. When you click on the test connection what Dynatrace does it tries to access that URL from mission control or their Datacenters and it checks the SSL certificate for a secure connection which is a must for Dynatrace to communicate from an external context. Procure an SSL certificate with a Domain name and install it in LB then provide the URL with a domain name in this field it will surely work for you.

Instead of IP address provide a valid doamin name.

Regards,

MG

[Babar_Qayyum]

Hello @Julius L. and @Mohit G.

We have a valid SSL certificate which is terminating on the F5 LB.

When I use the URL, test connection to URL fails for all the options but with IP address 2 options are passed as shared in my first screenshot.

Following is the configuration. Can you please verify?

https://domain.com---> Public IP DNS: 000.000.000.115 (Port 443) ---> NATTED IP F5: 000.000.000.110 (Port 443) ---> Cluster ActiveGates Servers: 000.000.000.128 , 000.000.000.129 (port 9999).

Regards,

Babar

[mohit_gupta]

Hi Babar,

The configuration seems to be correct, are you specifying the port 443 when you enter the domain name in cluster active gate URL? if not just try once and check.

It will look like https://domain.com:443

Regards,

MG

[Babar_Qayyum]

Hello @Mohit G.

Following is the result with URL.

Do we need Proxy/Internet configured on the Cluster ActiveGates?

Regards,

Babar

[Julius_Loman]

You don't need any proxy. ActiveGate is only listening for requests in those cases.

You have a mismatch of the URL and the certificate issued in the first screenshot. In the second screenshot, I guess your load balancer (F5) is not balancing requests for the domain and they are unable to reach the activegate - check the F5 rules in this case.

Just a simple check : in your browser. go to the ActiveGate URL you have specified with the path /mbeacon - so something like https://dynatrace.domain.com/mbeacon according to your screenshot.

And check the output. It must not give you any certificate warnings and it should give you the output:

missing querystring

[Babar_Qayyum]

Hello @Julius L.

I am getting a reply with the following message after executing the URL with mbeacon.

ERR_RESPONSE_HEADERS_TRUNCATED

Regards,

Babar

[Julius_Loman]

This is I think the issue at the F5 balancer. Please ask your F5 administrators to check rules.

You have written the F5 does the SSL termination. Don't forget there is also SSL connection from the F5 to the activegate. Maybe the F5 is now configured to do http connection instead of https. Also I don't know if you have the default self-signed certificate on the activegate. If so, please check if your F5 accepts the cert.

Anyway, you have to debug the issues at the F5.

[Babar_Qayyum]

Hello @Julius L.

Yes. We have a default self-signed Cluster ActiveGate SSL certificate which looks like following:

Current SSL certificate

• Issuer: Dynatrace
• Subject: Dynatrace
• Expires: Jul 01, 2029

Regards,

Babar

[Julius_Loman]

Maybe your F5 is not acceping connections to sites with self signed certificates.

Anyway - you need to debug your issue on your F5.

[Babar_Qayyum]

Hello @Julius L.

Do we need SSL communication between F5 and Cluster ActiveGate?

Regards,

Babar

[Julius_Loman]

No, you don't need SSL, but in default configuration Cluster ActiveGate is SSL only. If you want or need non-SSL configuration you need to reconfigure the gateway to open non-SSL port (in custom.properties).

I've recently encountered a case at a customer when F5 was configured to perform a HTTP call to ActiveGate HTTPS port. Normally I would also expect the F5 will not accept selfsigned certificates.

[Babar_Qayyum]

Hello @Julius L.

Do you recommend to change the custom.properties to accept the HTTP communication or we should reconfigure the F5 for the HTTPS communication?

Regards,

Babar

[Julius_Loman]

It depends on your policies. If you are strictly HTTPS, you should stick with HTTPS, but then your gateway should present a valid certificate.

Previously you had IP address in your screenshots. That will never work with HTTPS since certificates are valid for hostnames. (They can be issued for IP addresses too, but it is an antipattern and I've seen this like once in my life). So - never use URLs with IP addresses when doing SSL connections and you are honoring SSL certificates. It will never work unless you really know what you are doing.


So in your case:

https://domain.com---> Public IP DNS: 000.000.000.115 (Port 443) ---> NATTED IP F5: 000.000.000.110 (Port 443) ---> Cluster ActiveGates Servers: 000.000.000.128 , 000.000.000.129 (port 9999).

The F5 must present a certificate (signed by a publicly known CA) for your URL configured in dynatrace - let's say it is https://dynatrace.domain.com.
So if SSL request arrives at the F5, F5 must present this certificate.
Then, since F5 is doing the termination here. It must connect to the activegate. Since we do not know your configuration, I guess it will connect to something like https://clusteractivegate.domain.local:9999 It must not be an IP address, because then the certificate check will fail.

At the Cluster ActiveGate you are presenting an self-signed certificate. Any SSL client will normally not accept such connection because the party certificate is self-signed. That might be your case.

So please validate now what destination (URL, not IP) is used at the F5 for the your public URL.

[Babar_Qayyum]

Hello @Julius L.

While I am discussing this with F5 administrator. Please let me know why the below entry changes automatically from HTTPS to HTTPS after restarting the Cluster ActiveGate service?

dnsEntryPoint = https://10.000.000.000:9999/communication

Regards,

Babar

[Julius_Loman]

The dnsEntryPoint is I think only used for oneagents and should not contain the path. I don't think you need setting the dnsEntryPoint in your case at all.

[Babar_Qayyum]

Hello @Julius L.

Basically, I wanted to change the communication between F5 and Cluster ActiveGate from HTTPS to HTTP.

Where it will be changed?

If you want to start ActiveGate in a secured way using HTTPS, you have to set the port-ssl property in custom.properties, while if you want to start ActiveGate using HTTP, you have to set the port property in custom.properties. Note that the secure way is the default and recommended one. However, you might want to choose this option for performance reasons, if you have, for example, a load balancer installed in front of the Cluster ActiveGate that terminates incoming SSL connections from outside your premises (see the third deployment scenario).

Regards,

Babar

[Julius_Loman]

In the custom.properties in the gateway configuration files directory:

[com.compuware.apm.webserver]
port-ssl = 9998
port = 9999

[Babar_Qayyum]

Hello @Julius L.

I will have to copy the following entries as it is in the custom.properties file?

[com.compuware.apm.webserver]
port-ssl = 9998
port = 9999

Regards,

Babar

[Reef]

Hello @Babar_Qayyum, 

As I am browsing for a solution here, I came across your situation and its as mine currently so if you could recommend the changes that have been applied to fix this. 

for now, I only changed the custom.properties file since its something can be done from my end and I got the following URL results : 

/mbeacon

missing type parameter

---

I want it to be for both Mobile RUM & Synthetic Monitoring, so any idea on what might be missing here ?  

Load balancer configuration: 

Frontend HTTPS

Backend is 9999

Regards, 

[Babar_Qayyum]

Hello @Reef 

Did you get a chance to look at the below deployment models?

https://www.dynatrace.com/support/help/shortlink/managed-deployment-scenarios

Regards,

Babar

[Reef]

Thanks for the reply @Babar_Qayyum, 

and yes, scenario 3 is what has been deployed. 

Regards, 

[Babar_Qayyum]

Hello @Reef 

Thanks for the information. If you have configured it accordingly then try a Synthetic Monitoring test. I can share my personal experience that you should not care about the below test results. 

Regards,

Babar

[Reef]

Hi @Babar_Qayyum, 

Got it, so test results doesn't matter on this but unfortunately its still not working. 

So why I felt the similarity with the situations, its because after having the following

response ERR_RESPONSE_HEADERS_TRUNCATED from my domain name I've added the ports to custom.properties file and restarted the service, somehow its seems worked for a bit as the follow :  

but I couldn't figure why it was only for a bit, as currently no data is available after that time and Synthetic seems still not working.

Regards, 

[Babar_Qayyum]

Hello @Reef 

Is the fallow of traffic LB 443 > Cluster AG 9999?

Did the network/LB team configure the above configuration accordingly?

Also, please have a look at the below link:

https://www.dynatrace.com/support/help/shortlink/rum-firewall

Regards,

Babar

[Babar_Qayyum]

Hello @Julius L.

At last, I got the following:

missing querystring

Below is the current status. How to resolve internal users of web applications issue?

Regards,

Babar

[Julius_Loman]

My guess is that the URL provided is not reachable from the cluster node and it might be OK in your case.
I'm not sure how Dynatrace validates this option.
Can you try if you can reach the /mbeacon from your internal network?

This might be also a DNS issue since in enterprises domain resolves different records internally than externally. Just validate from your PC in the internal network if you can reach the URL. Might be just the case the hostname is even not propagated to internal users.

[Babar_Qayyum]

Hello @Julius L.

The internal DNS already has done for F5 and Cluster ActiveGates even though internal users of web applications (for agentless real user monitoring) is not resolving.

Regards,

Babar

[Julius_Loman]

Is the https://dynatrace.domain.com/mbeacon reachable from a browser in the internal network?Can you do

curl -v https://dynatrace.domain.com/mbeacon

from your cluster node? If your dynatrace cluster is using proxy, you must specify the proxy with -x curl argument.

I believe the verification is done from the dynatrace managed cluster nodes and it fails for you. Maybe just the external URL is not reachable from your cluster node.

[Babar_Qayyum]

Hello @Julius L.

I requested to open the port from my PC to the URL to check the /mbeacon result.

Below is the curl command output from one of the Dynartrace Cluster Node.

Regards,

Babar

[Julius_Loman]

This looks like your proxy is talking to the non-SSL port now with https.

[Babar_Qayyum]

Hello @Julius L.

I configured the non-ssl port for the Cluster ActiveGates to avoid the certificate issue.

Regards,

Babar

[Julius_Loman]

After the change you need to reconfigure the F5 to make a http call instead of https.

[Babar_Qayyum]

Hello @Julius L.

We already have done these changes and Synthetic monitoring started successfully.

Now the only pending thing is the following:

internal users of web applications (for agentless real user monitoring) is not resolving.

Regards,

Babar

[Babar_Qayyum]

Hello @Julius L.

What could be the reason for the intermittent test connection to URL Passed/Failed?

Regards,

Babar

[Julius_Loman]

As I have written before - you changed the activegate port and now the F5 is talking http to your https endpoint.

[Babar_Qayyum]

Hello @Julius L.

Apologies. I could not explain my point well. F5 is reconfigured to talk to the Cluster ActiveGates on Port # 9998 instead of 9999. In the same way Cluster ActiveGates, also configured for the non-ssl traffic to accept.

We started collecting Synthetic data with up and down status e.g. in the below screenshot you can see the grayed-out area for all the locations and I am unable to understand the meanings of No data legend, therefore, I also opened a different question for the same subject to discuss for better understanding.

The strange thing is that availability is showing 100% for all the locations even with almost half of the grayed-out area.

Regards,

Babar

[rmeli]

The diagram is exactly the configuration we have. How are people going about testing whether it works or not? What does the “Test connection to URL” do exactly so we can trace where communications are failing?

[mohit_gupta]

Hi Roger,

Follow below troubleshooting steps.

https://www.dynatrace.com/support/help/offline-doc/how-do-i-set-up-mobile-apps-for-real-user-monitor...

Regards,

MG