Synthetic screenshots: Information and troubleshooting

[Dyna_Patrick]

• Synthetic Screenshots
• Screenshot Storage
• S3
• Cluster
• Proxy Scenarios
• Troubleshooting
• Private Locations
• Upload Errors
• Frequently Asked Questions
• Helpful Links
• What's Next?

Synthetic Screenshots

• Reference (expected/ success) screenshots are taken when the monitor is created or updated, and subsequently, every 24 hours from a random monitoring location. So, if a monitor is executed on several locations, the success screenshot shown for all locations will be the same screenshot, from one random location.
• Screenshot retention is based on execution retention. When they're taken, screenshots are considered part of the execution data. The default retention time for screenshots is 35 days; however, it can be less in some Managed Environments depending on their storage settings.
• If your Synthetic monitor has never been successful or has not been successful in the last 35 days, you'll not have any "expected" screenshots. 
• Screenshots are captured at the end of each script event, even those without timings.
• Success (expected) screenshots are only taken on a successful first execution, never on a retry. So if your monitor is always successful and has no screenshots, it is likely that all executions are Execution type: re-run
• You can verify the origin of a screenshot by following these steps:
  1. Navigate to the Synthetic dashboard and select "Analyze executions."
  2. Select the specific execution you're interested in from the list.
  3. Above the screenshot, you'll find a label displaying the date and location, indicating, "Screenshot from <date> in <location>."
  4. Select this link, and you'll be directed to the execution where the screenshot was taken or attempted to be taken. Example:

Screenshot Storage

• SaaS:
  Screenshots are stored in Amazon S3.
• Managed:
  Clusters installed after version 1.216 store screenshots on the cluster, in Cassandra.
  For clusters installed before this version, screenshots are stored in Amazon S3. To modify the storage setup to cluster-based, create a support ticket.
  If screenshots are stored on the cluster, the storage settings can't be updated to store on S3.  

S3

For screenshots stored in Amazon S3, the following domains in the firewall configuration must be allowed:

• ruxit-synth-screencap.s3-accelerate.amazonaws.com for screenshot storage. This should be allowed on the relevant ActiveGates.
• ruxit-synth-screencap.s3.amazonaws.com for screenshot access and viewing. This should be allowed for any machines using the WebUI to view screenshots.

Cluster

For screenshots stored in Cassandra, on a Managed Cluster,  make sure the endpoint below is allowed to receive data on your Cluster ActiveGate (only required if you're using Public locations)

•  /beacon/synthetic/screenshot/[uuid] - Where uuid is the Environment ID.
• You may also wish to allow public location IP addresses on any network devices that may prevent access to the Cluster ActiveGate endpoint or modify the data in any way. 

Proxy Scenarios

For general information and scenarios, see Setting up a proxy for private synthetic 

1. Proxy needed for S3 and tested resources
   Add the proxy connection details to the [synthetic] section of the ActiveGate custom.properties file (which is located in the ActiveGate configuration directory)
   Example:
   

   [synthetic]
   proxy-server=<proxy between the AG and S3>
   proxy-port=8080
   proxy-user=username
   proxy-password=password​

2. Proxy not needed for S3 and tested resources
   If the proxy entered at a higher level, http.client or http.client.internal, is not needed for either S3 or tested resources, it will need to be disabled for Synthetic.
   

   [synthetic]
   proxy-off=true​

3. Proxy only needed for either S3 or tested resources
   If the proxy should be enabled for S3 but not Monitors, or vice versa, then you can selectively enable proxy for Monitors or S3. 
   Set up the proxy settings in custom.properties as usual. Then, add the following settings to VUC user.properties (default path /var/lib/dynatrace/synthetic/config/user.properties
   1. To enable the proxy for monitors and disable it for S3

      com.vuc.proxy.s3.enabled=false
      com.vuc.proxy.monitor.enabled=true

      
      
   2. To disable the proxy for monitors and enable it for S3
      

      com.vuc.proxy.s3.enabled=true
      com.vuc.proxy.monitor.enabled=false​

Restart the vuc.service/ Dynatrace Synthetic service after any changes to the proxy settings.

 

Troubleshooting

• Confirm that the executions are not always re-runs. Go to the browser monitor, select “Analyze executions”, and in the execution time, select different timings and confirm the “Execution type” is standard (and not re-run) for some of the executions.
  
• For SaaS, or Managed environments that store screenshots on S3, ensure that the ActiveGate can access the Amazon S3 service.
• For Managed environments that store screenshots on the cluster, in Cassandra, ensure that the Cluster ActiveGate URL is publicly available, allowing Public location IPs in any Firewalls, etc.

 

Private Locations

How to identify issues in the logs:

• In the browser_#.log file (/var/tmp/dynatrace/synthetic or %PROGRAMDATA%\dynatrace\synthetic):

Look for any errors related to "SCREENSHOT" or "SCREENCAP". These errors are related to taking and storing the screenshots locally on the Virtual User Controller (VUC).

• In the vuc-browser.log file. (/var/log/dynatrace/synthetic/ or <LOG>/synthetic/log or %PROGRAMDATA%\dynatrace\synthetic\log):

Check for "[SCREENCAP]" errors. These errors occur when uploading the images from the VUC to either S3 or the cluster.

Upload Errors

Status Code: 403

These messages can be seen in the vuc-browser.log file.

AwsS3CloudFileManager: [SCREENCAP] Exception caught while uploading file /var/tmp/dynatrace/synthetic/screenshots/ AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden…)

AwsS3CloudFileManager: [SCREENCAP] Exception caught while uploading file /var/tmp/dynatrace/synthetic/screenshots/image.jpeg AmazonS3Exception: URLBlockedServers (Service: Amazon S3; Status Code: 403; Error Code: 403 URLBlockedServers)

Confirm what configuration is needed for the private location to reach S3 and update the proxy configuration. See the Proxy scenarios section.

Unable to find valid certification path to requested target

These messages can be seen in the vuc-browser.log file.

AwsS3CloudFileManager: [SCREENCAP] Exception caught while uploading file /var/tmp/dynatrace/synthetic/screenshots/ SunCertPathBuilderException: unable to find valid certification path to requested target.

AwsS3CloudFileManager: [SCREENCAP] Client-side exception caught while uploading file ... to S3 bucket ruxit-synth-screencap:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

This usually means that there is a certificate issue when the ActiveGate is trying to connect to S3. 

• Add the certificate to the truststore. More information in the documentation.
  
  
• For Windows, you can update the Synthetic Module to use the Windows truststore instead
  

  Append the following lines to C:\ProgramData\dynatrace\synthetic\config\user.properties:

  javax.net.ssl.trustStore=NUL
  javax.net.ssl.trustStoreType=Windows-ROOT

  and then restart the Dynatrace Synthetic service.

407 Proxy Authentication Required

These messages can be seen in the vuc-browser.log file.

ERROR AwsS3CloudFileManager: [SCREENCAP] Exception caught while uploading file C:\\ProgramData\\dynatrace\\synthetic\\temp\\synthetic\\screenshots\\image.jpeg to S3 bucket... Proxy Authentication Required (Service: Amazon S3; Status Code: 407; Error Code: 407 Proxy Authentication Required; Request ID: null; S3 Extended Request ID: null; Proxy: <yourproxy>)

Proxy credentials are needed.

• Add the proxy username and password details to the section of the custom.properties file, in /var/lib/dynatrace/gateway/config or %PROGRAMDATA%\dynatrace\gateway\config, which contains the proxy connection information, most likely [synthetic].  
  

  proxy-user=username
  proxy-password=password​

  After saving it, the proxy password will be encrypted. You can then restart the Dynatrace Gateway and Dynatrace Synthetic services on Windows or vuc and dynatracegateway services on Linux.

• If the proxy credentials are already added, check that the password has not expired or been changed. Then update the proxy-password. 
  

  • Replace the line containing the current encrypted password

    proxy-password-encr =

    with 

    proxy-password = yourPassword

    where yourPassword is the correct password

  • Save the file & restart the services
    

Frequently Asked Questions

1. My site has different languages/ looks and feels in different locations, but all the success/ expected screenshots are showing the same images.
   1. Each monitor only has 1 set of success/ expected screenshots. These are taken once every 24 hours from a random location. These success screenshots are then shown for all executions. Separate monitors would be needed if the success images from each location are needed.
      Failed executions show the images from the execution that failed. 
2. I changed my monitor and now some screenshots are missing
   1. New screenshots are taken after an update, if some events are no longer used then there won't be a screenshot taken for them and they will appear blank. 
3. I updated my application and want to compare screenshots from before and after but they just show the new pages
   1. Each monitor only has 1 set of success/ expected screenshots. These are taken once every 24 hours from a random location. These success screenshots are then shown for all executions, including executions from previous days.
      To compare before and after screenshots, either create a duplicate monitor and execute it for a period on the original application before disabling it. Then compare screenshots after the new application is launched. Or create a monitor with a last steps that checks something that doesn't exist on the page, to force a failure. Run this monitor before and after the change to compare screenshots. 
4. How can I force new 'Expected'/ success screenshots to be taken
   1. New screenshots are taken after an update to the monitor. If you're not updating the monitor, you can run an on-demand execution and select 'Take screenshot on successful execution'. 
      

Helpful Links

Set up a proxy for private synthetic monitoring

Synthetic details for browser monitors 

Useful Synthetic ActiveGate logs

What's Next?

If the screenshots are still missing or unexplained after following the troubleshooting, please create a Support ticket and provide the following:

1. Link to any monitors/ monitor executions that are missing screenshots
2. Support Archive for the ActiveGate machine, if access to the ActiveGate is not available to Support
3. Explanation of troubleshooting steps taken and the results

[ChadTurner]

Thank you for sharing this @Dyna_Patrick 

[radek_jasinski]

Thank you😁