Why did the Synthetic Module on a Synthetic ActiveGate fail to update? Showing Failed or Skipped

HannahM · ‎11 Mar 2024

Self Service Summary
Get the Synthetic installer logs:
Check the logs
Update times out
Chromium autoupdate not completing
After resolving the cause of the failure:
What if this doesn't resolve the issue

Self Service Summary

If the Activegate itself didn't update, check the dynatracegateway-install_XXX.log. This article just covers failures to update the Synthetic Module or Chromium.

You can confirm the versions of the ActiveGate, Synthetic Module and Chromium in the Deployment Status> ActiveGates page. The Last update column will report skipped or failed if there was a problem.

Get the Synthetic installer logs:

The Synthetic module installer writes to a separate file. This will either be

/var/log/dynatrace/synthetic/dynatracesynthetic-install_<datetimeOfInstall>.log
or
C:\ProgramData\dynatrace\autoupdater\log\Synthetic_<version>.log

For Linux, if the file doesn't exist, check in /tmp/ for any dynatracesynthetic-install_<datetimeOfInstall>.log files

And, also, the autoupdater log

/var/log/dynatrace/autoupdater/dynatrace_ag_autoupdater_nnn.log
or
C:\ProgramData\dynatrace\autoupdater\log\dynatrace_ag_autoupdater_nnn.log

If the ActiveGate was updated within the last 7 days, you can use the ActiveGate diagnostics feature to download it, otherwise you will need to check on the machine itself.

Check the logs

Most of the time the message in the installer log is quite verbose and helpful. There are some scenarios listed below also:

Update times out

This can be seen in the dynatrace_ag_autoupdater_nnn.log:

2023-02-06 08:51:49 UTC 1.257.216.20230119-103521 (120107) Installing package /var/lib/dynatrace/packages/Dynatrace-Synthetic-Linux-x86-1.257.23.20221220-170840.sh
2023-02-06 09:06:49 UTC 1.257.216.20230119-103521 (120107) Installation process has been killed as timeout expired (1)
2023-02-06 09:06:49 UTC 1.257.216.20230119-103521 (120107) Installation has failed.

Check the last lines in the Synthetic installer log

Unmounting cache…
1. This behaviour is usually due to microsoft defender service, mdatp, interfering. Confirm if there are any antivirus applications running on this server. To complete the update:
  1. Stop microsoft defender/ other antivirus
  2. delete any .sh files from /var/lib/dynatrace/packages
  3. Retry the update - either wait for the autoupdate to retry (usually hourly) or manually click update.
2. This behaviour can also be seen when unmounting the cache takes an extended time to complete
  1. Before trying to update Synthetic again, try unmounting the cache manually:
    sudo umount /var/tmp/dynatrace/synthetic/cache
  2. If that hangs for a longer time, then it can be done as a lazy unmount:
    sudo umount -l /var/tmp/dynatrace/synthetic/cache
    To be safe after doing the lazy unmount we recommend rebooting the machine.
  3. delete any .sh files from /var/lib/dynatrace/packages
  4. Retry the update - either wait for the autoupdate to retry (usually hourly) or manually click update.
Command: yum versionlock chromium chromium-common

Check if there is a reason the command is taking a long time and resolve the issue. Then manually run it before retrying the update.
1. Run the following commands:
  yum versionlock clear
  
  yum versionlock chromium chromium-common
  
  yum versionlock | grep chromium
2. If chromium-common is locked, you can run the update again after deleting any .sh files from /var/lib/dynatrace/packages.
  
  If it's not, check the output from yum versionlock | grep chromium

Chromium autoupdate not completing

Chromium autoupdate is only relevant for Linux installations as Windows ActiveGates have Chromium packaged with the installer. These are the most common scenarios:

Search the Synthetic installer log for
Detected ActiveGate configuration (joined)
Below that is the section that states which options are enabled. For Chromium autoupdate to complete both synthetic.synthetic_autoupgrade_chromium and synthetic_autoinstall must be true.
1. If synthetic.synthetic_autoupgrade_chromium is false or has no value, update this in the location (this cannot be updated in the custom.properties).
  For Private locations, this can be done in the WebUI
  For Cluster locations, this can be done using the Cluster API
2. If synthetic_autoinstall is false or has no value, add synthetic_autoinstall = true to the [synthetic] section of the /var/lib/dynatrace/gateway/config/custom.properties file and restart the vuc and gateway services.
Search the installer log for Downloading Chromium package

This line and the lines after it will show if/ why the download failed. Like this example where the connection was refused
```
01/08 11:22:24 Downloading Chromium packages from https://synthetic-packages.s3.amazonaws.com/Chromium/rpm/chromium-99.0.4844.84-1.el8.tgz...
01/08 11:22:24 Executing command:  curl --silent --show-error --continue-at - --connect-timeout 5 --retry 5 --retry-delay 10 --retry-max-time 80 --output /tmp/chromium/chromium.tgz https://synthetic-packages.s3.amazonaws.com/Chromium/rpm/chromium-99.0.4844.84-1.el8.tgz
curl: (7) Failed to connect to synthetic-packages.s3.amazonaws.com port 443: Connection refused
01/08 11:22:24 ERROR: Failed to download Chromium. Please check if https://synthetic-packages.s3.amazonaws.com/Chromium/rpm/chromium-99.0.4844.84-1.el8.tgz address is accessible.
```
For Chromium autoupdate to be successful, one of the following options must be chosen:
1. Allow direct connection to https://synthetic-packages.s3.amazonaws.com/Chromium/rpm/ from the ActiveGate
2. Use a mirror server on your network to serve the packages. See documentation for configuration details
3. Use a /.curlrc file to force curls to always use your proxy. https://reqbin.com/req/c-ddxflki5/curl-proxy-server
  The Synthetic Installer does not have access to the any proxy information provided in the custom.properties file.
  
  Alternatively, Disable Chromium autoupdate on the location and manually update Chromium.

Search for ERROR
The explanation is usually very helpful. For example:

29/06 05:39:55 ERROR: chromium-common package is not in supported version, min version: 96. Installed version is 90.0.4430.93 (90)
29/06 05:39:55 ERROR: Package libwayland-server is NOT installed!
29/06 05:39:56 ERROR: Package mesa-libgbm is NOT installed!
29/06 05:39:56 
29/06 05:39:56        For a typical OS installation with a working Internet connection, use
29/06 05:39:56        the following commands to install required package(s):
29/06 05:39:56 
29/06 05:39:56            sudo yum install -y libwayland-server mesa-libgbm
29/06 05:39:56 
29/06 05:39:56            sudo rpm -e chromium chromium-common
29/06 05:39:56 
29/06 05:39:56            curl --remote-name https://synthetic-packages.s3.amazonaws.com/Chromium/rpm/chromium-112.0.5615.165-1.el8.tgz
29/06 05:39:56            tar xzf chromium-112.0.5615.165-1.el8.tgz
29/06 05:39:56            sudo yum install -y chromium-112.0.5615.165-1.el8.x86_64.rpm chromium-common-112.0.5615.165-1.el8.x86_64.rpm
29/06 05:39:56 
29/06 05:39:56 ERROR: Install missing or reinstall invalid packages and restart installation.

Here you can see that there we 2 packages that needed to be installed. These are Chromium dependencies. The commands to resolve it are provided.

Private Synthetic locations: ActiveGate installation/update failed because of missing dependencies
Alternatively, you can just try updating all the Chromium and Engine dependencies using the commands relevant to your OS found here

After resolving the cause of the failure:

delete any .sh files from /var/lib/dynatrace/packages
Retry the update - either wait for the autoupdate to retry (usually hourly) or manually click update.

Note:

The ActiveGate status will remain at Update Pending until the .sh files from the failed update are removed from the packages folder. After that the update option will be available

What if this doesn't resolve the issue

If you do not care about the root cause or just want to get the ActiveGate up and running again, you can uninstall and reinstall the ActiveGate. You can choose the private Synthetic Location to add this to during installation now, see step 6, so no Synthetic monitors will need to be updated to use a new Location. or
create a chat or support ticket and provide the support archive or logs you have gathered during your troubleshooting and provide the outcome of the troubleshooting steps.