Synthetic screenshots: Information and troubleshooting

[Dyna_Patrick]

• Synthetic Screenshots
• Screenshot Storage
• S3
• Cluster
• Proxy Scenarios
• Troubleshooting
• Private Locations
• Upload Errors
• Frequently Asked Questions
• Helpful Links
• What's Next?

Synthetic Screenshots

• Reference (expected/ success) screenshots are taken when the monitor is created or edited, and subsequently, every 24 hours from a random monitoring location. So, if a monitor is executed on several locations, the success screenshot shown for all locations will be the same screenshot, from the one random location.
• Screenshot retention is based on execution retention. They are considered part of the execution data for the execution when they were taken. The default retention time for screenshots is 35 days, however, it can be less in some Managed Environments depending on their storage settings.
• If your Synthetic monitor has never been successful or has not been successful in the last 35 days, you will not have any "expected" screenshots. 
• Screenshots are captured at the end of each script event, even those without timings.
• Success (expected) screenshots are only taken a successful first execution, never on a retry. So if your monitor is always successful and has no screenshots, it is likely that all executions are Execution type: re-run
• You can verify the origin of a screenshot by following these steps:
  1. Navigate to the Synthetic dashboard and click on "Analyze executions."
  2. Select the specific execution you're interested in from the list.
  3. Above the screenshot, you will find a label displaying the date and location, indicating, "Screenshot from <date> in <location>."
  4. Click on this link, and you will be instantly directed to the execution where the screenshot was taken or attempted to be taken. Example:

Screenshot Storage

• SaaS:
  Screenshots are stored in Amazon S3.
• Managed:
  Clusters installed after version 1.216 store screenshots on the cluster, in Cassandra.
  For clusters installed before this version, screenshots are stored in Amazon S3. To modify the storage setup to cluster-based, create a support ticket.
  If screenshots are stored on the cluster, the storage settings cannot be updated to store on s3.  

S3

For screenshots stored in Amazon S3, the following domains in the firewall configuration must be allowed:

• ruxit-synth-screencap.s3-accelerate.amazonaws.com for screenshot storage. This should be allowed on the relevant ActiveGates.
• ruxit-synth-screencap.s3.amazonaws.com for screenshot access and viewing. This should be allowed for any machines that are using the WebUI to view screenshots.

Cluster

For screenshots stored in Cassandra, on a Managed Cluster,  make sure the endpoint below is allowed to receive data on your Cluster ActiveGate (only required if you are using Public locations)

•  /beacon/synthetic/screenshot/[uuid] - Where uuid is the Environment ID.
• you may also wish to allow public location IP addresses on any network devices that may prevent access to the Cluster ActiveGate endpoint or modify the data in any way. 

Proxy Scenarios

For general information and scenarios, see Setting up a proxy for private synthetic 

1. Proxy needed for S3 and tested resources
   Add the proxy connection details to the [synthetic] section of the ActiveGate custom.properties file (which are located in the ActiveGate configuration directory)
   Example:
   

   [synthetic]
   proxy-server=<proxy between AG and S3>
   proxy-port=8080
   proxy-user=username
   proxy-password=password​

2. Proxy not needed for S3 and tested resources
   If the proxy entered at a higher level, http.client or http.client.internal, is not needed for either s3 or tested resources, it will need to be disabled for Synthetic
   

   [synthetic]
   proxy-off=true​

3. Proxy only needed for either S3 or tested resources
   If the proxy should be enabled for S3 but not for Monitors, or vice versa, then you can selectively enable proxy for Monitors or S3. 
   Set up the proxy settings in custom.properties as usual. Then adding following flags to VUC user.properties (default path /var/lib/dynatrace/synthetic/config/user.properties
   1. to enable the proxy for monitors and disable it for s3

      com.vuc.proxy.s3.enabled=false
      com.vuc.proxy.monitor.enabled=true

      
      
   2. to disable the proxy for monitors and enable it for s3
      

      com.vuc.proxy.s3.enabled=true
      com.vuc.proxy.monitor.enabled=false​

Restart vuc.service/ Dynatrace Synthetic service after any changes to the proxy settings.

 

Troubleshooting

• Confirm that the executions are not always re-runs. Go to the Synthetic, click on “Analyze executions”, and in the execution time, select different timings and confirm the “Execution type” is standard (and not re-run) for some of the executions.
  
• For SaaS, or Managed that stores screenshots on S3, ensure that the ActiveGate can access the Amazon S3 service.
• For Managed, that stores screenshots in Cassandra on the cluster, ensure that the Cluster ActiveGate url is publicly available, allowing Public location IPs in any Firewalls etc.

 

Private Locations

How to identify issues in the logs:

• In the browser_#.log file (/var/tmp/dynatrace/synthetic or %PROGRAMDATA%\dynatrace\synthetic):

Look for any errors related to "SCREENSHOT" or "SCREENCAP." These errors are related to taking and storing the screenshots locally on the Virtual User Controller (VUC).

• In the vuc-browser.log file. (/var/log/dynatrace/synthetic/ or <LOG>/synthetic/log or %PROGRAMDATA%\dynatrace\synthetic\log):

Check for "[SCREENCAP]" errors. These errors occur when uploading the images from the VUC to either S3 or the cluster.

Upload Errors

Status Code: 403

These messages can be seen in the vuc-browser.log file.

ERROR AwsS3CloudFileManager: [SCREENCAP] Exception caught while uploading file /var/tmp/dynatrace/synthetic/screenshots/ AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden…)

ERROR AwsS3CloudFileManager: [SCREENCAP] Exception caught while uploading file /var/tmp/dynatrace/synthetic/screenshots/image.jpeg AmazonS3Exception: URLBlockedServers (Service: Amazon S3; Status Code: 403; Error Code: 403 URLBlockedServers)

Confirm what configuration is needed for the private location to reach s3 and update proxy configuration. See Proxy scenarios section.

Unable to find valid certification path to requested target

These messages can be seen in the vuc-browser.log file.

ERROR AwsS3CloudFileManager: [SCREENCAP] Exception caught while uploading file /var/tmp/dynatrace/synthetic/screenshots/ SunCertPathBuilderException: unable to find valid certification path to requested target.

This usually means that there is a certificate issue when ActiveGate is trying to connect to S3.

• Add the certificate to truststore. Here you can find the documentation with more information.

407 Proxy Authentication Required

These messages can be seen in the vuc-browser.log file.

ERROR AwsS3CloudFileManager: [SCREENCAP] Exception caught while uploading file C:\\ProgramData\\dynatrace\\synthetic\\temp\\synthetic\\screenshots\\image.jpeg to S3 bucket... Proxy Authentication Required (Service: Amazon S3; Status Code: 407; Error Code: 407 Proxy Authentication Required; Request ID: null; S3 Extended Request ID: null; Proxy: <yourproxy>)

Proxy credentials are needed.

• Add the proxy username and password details to the section of the custom.properties file, in /var/lib/dynatrace/gateway/config or %PROGRAMDATA%\dynatrace\gateway\config, which contains the proxy connection information, most likely [synthetic].  
  

  proxy-user=username
  proxy-password=password​

  After saving it, the proxy password will be encrypted. You can then restart the Dynatrace Gateway and Dynatrace Synthetic services on Windows or vuc and dynatracegateway services on Linux.

Frequently Asked Questions

1. My site has different languages/ look and feel in different locations but all the success/ expected screenshots are showing the same images.
   1. Each monitor only has 1 set of success/ expected screenshots. These are taken once every 24 hours from a random location. These success screenshots are then shown for all executions. Separate monitors would be needed if the success images from each location are needed.
      Failed executions show the images from the execution that failed. 
2. I changed my monitor and now some screenshots are missing
   1. New screenshots are taken after an update, if some events are no longer used then there will not be a screenshot taken for them and they will appear blank. 
3. I updated my application and want to compare screenshots from before and after but they just show the new pages
   1. Each monitor only has 1 set of success/ expected screenshots. These are taken once every 24 hours from a random location. These success screenshots are then shown for all executions, including executions from previous days.
      To compare before and after screenshots, either create a duplicate monitor and execute it for a period on the original application before disabling it. Then compare screenshots after the new application is launched. Or create a monitor with a last steps that checks something that doesn't exist on the page, to force a failure. Run this monitor before and after the change to compare screenshots. 

Helpful Links

Set up a proxy for private synthetic monitoring

Synthetic details for browser monitors 

Useful Synthetic ActiveGate logs

What's Next?

If the screenshots are still missing, or unexplained, after following the troubleshooting, please create a Support ticket and provide the following:

1. Link to any monitors/ monitor executions that are missing screenshots
2. Support Archive for the ActiveGate machine, if access to the ActiveGate is not available to Support
3. Explanation of troubleshooting steps taken and the results

[ChadTurner]

Thank you for sharing this @Dyna_Patrick 

[radek_jasinski]

Thank you😁