Firewall Constraints and allowlisting for Synthetic Monitoring

[HannahM]

• Self Service Summary
• Issues
• Synthetic monitors fail to reach application
• Synthetic Monitor is executing from a public location but failing to connect to application
• Synthetic Monitor is executing from a private location but failing to connect to application
• Troubleshoot using Curl
• Troubleshoot using Powershell
• Troubleshoot using a Browser on the Synthetic ActiveGate
• Synthetic Monitor data is not returned to Dynatrace
• For SaaS tenants
• For Managed tenants
• Solutions
• Allow/ Allowlist by User Agent string
• Allow/ Allowlist location IPs
• Add / update Proxy configuration
• Log Dynatrace Synthetic service on as a domain user

Self Service Summary

When using Synthetic Monitoring, it is sometimes necessary to allow/ allowlist traffic for Browser and HTTP monitors to execute successfully and return data to Dynatrace. For RUM, this is described here. Synthetic uses the same technology to return data to Dynatrace, so the RUM rules should also be used.

In this article, we will only describe the extra contraints for Synthetic.

Issue Solution Tasks Alternative(s)

Synthetic monitors fail to reach application	Allow/ allowlist connection	Confirm what is being blocked. Change configuration	Raise a support ticket or contact us via chat.
Synthetic Monitor data is not returned to Dynatrace	Allow/ allowlist connection/ traffic	Confirm what is being blocked. Change configuration	Raise a support ticket or contact us via chat.

 

Issues

Synthetic monitors fail to reach application

 

Synthetic Monitor is executing from a public location but failing to connect to application

There are many types of failures that can show in a Synthetic Monitor for this, they could be request timeouts, authentication failure, connection refused etc. The key thing is that the monitor is permanently failing with the same outage.

What can you do?
Confirm with the following with the relevant Application team:

1. Should the application should be available outside your network?
2. if so, are there any limitations?

We recommend allowing our traffic by either allowing access by User Agent string or IP address. 

Synthetic Monitor is executing from a private location but failing to connect to application

As well as the checks for public locations, here are some helpful tests you can try from your private location. 

 

Troubleshoot using Curl

Use this command to confirm if the connection issue is related to Dynatrace or the machine itself. 

curl -vki <url you are testing>

or if you are using a proxy:

curl -vki -U proxyUser:proxyPassword -x proxy:proxyPort <url you are testing> 

if the site is redirected to another page, you will need to add -L to follow redirects also. For example, 

curl -vki -L <url you are testing>

What can you do if the result of the command fails
If the curl commands, then the issue lies in the connection between the machine and the application you are testing. Here's what you can do to fix it:

1. Update your proxy or allow the IP access.

If the result of the command is successful

1. Check if it's the Dynatrace User-Agent string 
   1. For HTTP Monitors, add -H "User-Agent: DynatraceSynthetic/1.267.13.20230518-162314" 

      curl -vki <url you are testing> -H "User-Agent: DynatraceSynthetic/1.267.13.20230518-162314"​

   2. For Browser Monitors, add -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 RuxitSynthetic/1.0 v0 t0 cfeatureHash=7efgijmoqtvx caes=1 ccux=1 sia=1 smf=1" 

      curl -vki <url you are testing> -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 RuxitSynthetic/1.0 v0 t0 cfeatureHash=7efgijmoqtvx caes=1 ccux=1 sia=1 smf=1"

What can you do if the result of the command fails? you will need to allow/ allowlist our User-Agent string on your Application/ Firewall. 

Troubleshoot using Powershell

If you are working on Windows, you may only have the option of Powsershell instead of curl. You can use Invoke-Webrequest. 

without proxy:

Invoke-WebRequest -Uri "<url you are testing>"

with proxy:

Invoke-WebRequest -Proxy "<proxy-url>:<proxy-port>" -ProxyUseDefaultCredentials -Uri "<url you are testing>"

with redirects following and printing raw response:

Invoke-WebRequest -MaximumRedirection 10 -Uri "<url you are testing>" | Select-Object -Expand RawContent

What can you do if the result of the command fails
If the curl commands, then the issue lies in the connection between the machine and the application you are testing. Here's what you can do to fix it:

1. Update your proxy or allow the IP access.

If the result of the command is successful

1. check if it's the Dynatrace User-Agent string by adding that header to the Invoke-WebRequest command
   • For HTTP Monitors, add -H "User-Agent: DynatraceSynthetic/1.267.13.20230518-162314"

     Invoke-WebRequest -Uri "<url you are testing>" -Headers @{"User-Agent"="DynatraceSynthetic/1.267.13.20230518-162314"}​

   • For Browser Monitors, add -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 RuxitSynthetic/1.0 v0 t0 cfeatureHash=7efgijmoqtvx caes=1 ccux=1 sia=1 smf=1" 

     Invoke-WebRequest -Uri "<url you are testing>" -Headers @{"User-Agent"="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 RuxitSynthetic/1.0 v0 t0 cfeatureHash=7efgijmoqtvx caes=1 ccux=1 sia=1 smf=1"}

What can you do if the result of the command fails? you will need to allow/ allowlist our User-Agent string on your Application/ Firewall. 

Troubleshoot using a Browser on the Synthetic ActiveGate

If your Synthetic ActiveGate has a GUI, you can test Browser Monitors from there. This is usually the case for Windows ActiveGates so I will just focus on those. 

1. Open a Chrome Browser on your Synthetic ActiveGate and navigate to the url you are testing.
   1. If you're not able to reach it, the problem is that the machine itself cannot reach the url.
      Allow/ allowlist the IP or update your proxy settings
2. If you have access to Dynatrace from the ActiveGate, playback the monitor in the Browser.
   1. If it failed, then it is possibly the User-Agent string, you can use curl or Powershell as mentioned above to confirm this. 
3. Open the Browser the ActiveGate uses to execute the monitors. The default path is C:\Program Files\dynatrace\synthetic\Chrome-bin\chrome.exe.
   1. If this fails, compare the difference in version to the Chrome Browser installed on the machine. 
4. Sometimes, the service user that starts Chrome does not have enough permissions to reach the site. Please do the following to check if that is the case: 
   1. On the Synthetic ActiveGate, download & unzip PsExec https://docs.microsoft.com/en-us/sysinternals/downloads/psexec
   2. Open a command prompt as administrator (Start -> cmd.exe -> Run as administrator) and navigate to the folder you extracted PSExec
   3. To start Chromium impersonating the "Local Service" user, PsExec executed with the following command: 

      PsExec.exe -i -u "NT Authority\LocalService" "C:\Program Files\dynatrace\synthetic\Chrome-bin\chrome.exe"​

      (If you installed the synthetic location in a non-standard path you need to adapt the path to chrome.exe)

5. Then Navigate to the url you are testing.
   1. If this fails, you may wish to try logging the Dynatrace Synthetic service on as a domain user.

Synthetic Monitor data is not returned to Dynatrace

For SaaS tenants

1. confirm that there are no maintenance windows at the time of the data gap, this can be confirmed in the Browser monitor details page or the HTTP Monitors details page.
2. if it's a on a private location, check for WebSocket errors in the logs. 

if you can't find a cause, open a chat or create a ticket. 

For Managed tenants

1. confirm that there are no maintenance windows at the time of the data gap, this can be confirmed in the Browser monitor details page or the HTTP Monitors details page.

2. Confirm that Cluster ActiveGate url is publicly available. It can be limited to access from public locations IP addresses, if needed.

   1. In the Cluster Management Console, under Settings/Public endpoints, click Test connection to URL button.
      This tests 4 types of public connectivity to the Cluster ActiveGate; Synthetic, internal and external users for RUM, and external OneAgent comms.
      If Synthetic fails then the Cluster ActiveGate is not publicly available and you will need to update you Firewall or Load balancer settings to allow access. You can ignore this if you have limited access to specific IPs.  

3. Confirm that the Synthetic endpoints are all open
   beacon/[uuid]
   Synthetic health checks /beacon/synthetic/[uuid]
   HTTP Monitors /beacon/synthetic/httpresults[uuid]
   screenshots /beacon/synthetic/screenshot/[uuid]
   Browser Monitors /beacon/synthetic/browser/[uuid]
   Multiprotocol monitors /beacon/synthetic/multi-protocol/[uuid]
   Credentials /beacon/synthetic/credentials/[uuid]

4. Run a healthcheck. If you are not limiting access to specific IPs, this can be run from anywhere. 

   https://<Cluster AG Public Url>:<port>/beacon/synthetic/<environment UUID>?healthcheck&tenantId=<environment UUID>

   and the response should be json like this

   {"ts":1680601930205,"activeGateId":nnnn,"activeGateVersion":"1.261.166.20230308-123141","activeGateWorkingMode":"MANAGED","tenantAvailable":false,"syntheticLicenseAvailable":false,"rumLicenseAvailable":false,"rumConfigRevision":-1,"imgAvailable":false,"hmResultsAvailable":false,"failureScreenshotInterval":3600000,"clusterVersion":"","clusterUUID":"","jsAgentVersion":"","protoVersion":1,"httpMonitorsResultsPath":"","screenshotsPath":"","mpmResultsAvailable":false,"credentialsPath":""}

   if you can't find a cause, open a chat or create a ticket. 

Solutions

Once you've done the checks and determined the root cause of the problem, here are some possible solutions that you can apply

Allow/ Allowlist by User Agent string

Synthetic Monitors can be recognised by the User Agent string they add in the request headers. This can be used in Firewalls etc to identify Synthetic traffic and allow it. 

For Browser monitors, we add 

RuxitSynthetic/1.0

and some extra parameters after that. You can find the full list here.

For HTTP Monitors, we add 

DynatraceSynthetic/{version}

The version changes with each release, so you may wish to just use DynatraceSynthetic/. You can find the full list here.

 

Allow/ Allowlist location IPs

IPs can be used to identify Synthetic traffic and allow it to pass through Firewalls etc. 

You can find the IPs used by our public locations either using the Frequency and Locations page in the WebUI or the Synthetic locations API - GET all locations API call. You can find the further details here.

We do not list private location IPs but you can find them in the WebUI in Deployment Status> ActiveGates. You can then filter for either just the location you are interested in or all Synthetic ActiveGates and open each ActiveGate details tile to check the IP addresses.

 

Add / update Proxy configuration

If a proxy is needed to connect to your application or you need to bypass it, then you may need to add some further configuration to your proxy to allow access from the machine/ location.

If the proxy already has the necessary change made we provide information on setting up the ActiveGate to use different proxy connection scenarios here.

In addition to this, if you need to only enable the proxy settings for Monitors (both HTTP and Browser) or S3 (screenshot storage), you can add the following flags to ActiveGate Synthetic user.properties file
default Linux path: /var/lib/dynatrace/synthetic/config/user.properties
default Windows path: C:\ProgramData\dynatrace\synthetic\config\user.properties

com.vuc.proxy.s3.enabled=false
com.vuc.proxy.monitor.enabled=true

The above snippet would enable Monitors to use the proxy settings specified in custom.properties but s3 would bypass the proxy and use a direct connection. 

There is no separate setting for Browser and HTTP Monitors at this point, however, Browser Monitors can use PAC files, so this can be used to achieve a different proxy for HTTP Monitors and Browser Monitors. 

Log Dynatrace Synthetic service on as a domain user

For applications that need to be opened in a Browser started by a domain user, you can update the user that the Dynatrace Synthetic service logs on as.

IMPORTANT: This is unsupported by Dynatrace and will need to be re-configured every time the ActiveGate is updated. We recommend auto-update is disabled for these ActiveGates and a maintenance window created to cover the period of the update and re-configuring the log on user. 

1. On the Synthetic Activegate, in Services, right-click on Dynatrace Synthetic service> click properties> select Login On tab.
2. Change  'Local Service' to a domain user and enter password and Save.
3. Restart the Service.

To revert to use the Local Service user

1. On the Synthetic Activegate, in Services, right-click on Dynatrace Synthetic service> click properties> select Login On tab.
2. Change the domain user to 'Local Service' with no password and Save.
3. Restart the Service.

If all else fails and the error persistes, open a support ticket

[ChadTurner]

great write up, thank you for sharing this! 

[DanielS]

Terrific Guide!!!! Thks for sharing this.