Why did the Synthetic Module on a Synthetic ActiveGate fail to update? Showing Failed or Skipped

HannahM · ‎11 Mar 2024

Self Service Summary

If the Activegate itself didn't update, check the dynatracegateway-install_XXX.log. This article just covers failures to update the Synthetic Module or Chromium.

You can confirm the versions of the ActiveGate, Synthetic Module and Chromium in the Deployment Status> ActiveGates page. The Last update column will report skipped or failed if there was a problem.

Get the Synthetic installer logs:

The Synthetic module installer writes to a separate file. This will either be

/var/log/dynatrace/synthetic/dynatracesynthetic-install_<datetimeOfInstall>.log
or
C:\ProgramData\dynatrace\autoupdater\log\Synthetic_<version>.log

For Linux, if the file doesn't exist, check in /tmp/ for any dynatracesynthetic-install_<datetimeOfInstall>.log files

And, also, the autoupdater log

/var/log/dynatrace/autoupdater/dynatrace_ag_autoupdater_nnn.log
or
C:\ProgramData\dynatrace\autoupdater\log\dynatrace_ag_autoupdater_nnn.log

If the ActiveGate was updated within the last 7 days, you can use the ActiveGate diagnostics feature to download it, otherwise you will need to check on the machine itself.

Check the logs

Most of the time the message in the installer log is quite verbose and helpful. There are some scenarios listed below also:

Update times out:

This can be seen in the dynatrace_ag_autoupdater_nnn.log:

2023-02-06 08:51:49 UTC 1.257.216.20230119-103521 (120107) Installing package /var/lib/dynatrace/packages/Dynatrace-Synthetic-Linux-x86-1.257.23.20221220-170840.sh
2023-02-06 09:06:49 UTC 1.257.216.20230119-103521 (120107) Installation process has been killed as timeout expired (1)
2023-02-06 09:06:49 UTC 1.257.216.20230119-103521 (120107) Installation has failed.

Check the last lines in the Synthetic installer log

Unmounting cache…
This behaviour is usually due to microsoft defender service, mdatp, interfering. Confirm if there are any antivirus applications running on this server. To complete the update:
1. Stop microsoft defender/ other antivirus
2. delete any .sh files from /var/lib/dynatrace/packages
3. Retry the update - either wait for the autoupdate to retry (usually hourly) or manually click update.
Command: yum versionlock chromium chromium-common
1. Run the following commands:
  yum versionlock clear
  
  yum versionlock chromium chromium-common
  
  yum versionlock | grep chromium
2. If chromium-common is locked, you can run the update again after deleting any .sh files from /var/lib/dynatrace/packages.
  
  If it's not, check the output from yum versionlock | grep chromium

Chromium autoupdate not completing

Chromium autoupdate is only relevant for Linux installations as Windows ActiveGates have Chromium packaged with the installer. These are the most common scenarios:

Search the Synthetic installer log for
Detected ActiveGate configuration (joined)
Below that is the section that states which options are enabled. For Chromium autoupdate to complete both synthetic.synthetic_autoupgrade_chromium and synthetic_autoinstall must be true.
1. If synthetic.synthetic_autoupgrade_chromium is false or has no value, update this in the location (this cannot be updated in the custom.properties. This can be done in the WebUI for private locations and the Cluster API for Cluster locations.)
2. If synthetic_autoinstall is false or has no value, add synthetic_autoinstall = true to the [synthetic] section of the /var/lib/dynatrace/gateway/config/custom.properties file and restart the vuc and gateway services.
Search the installer log for Downloading Chromium package

This line and the lines after it will show if/ why the download failed. Like this example where the connection was refused
```
01/08 11:22:24 Downloading Chromium packages from https://synthetic-packages.s3.amazonaws.com/Chromium/rpm/chromium-99.0.4844.84-1.el8.tgz...
01/08 11:22:24 Executing command:  curl --silent --show-error --continue-at - --connect-timeout 5 --retry 5 --retry-delay 10 --retry-max-time 80 --output /tmp/chromium/chromium.tgz https://synthetic-packages.s3.amazonaws.com/Chromium/rpm/chromium-99.0.4844.84-1.el8.tgz
curl: (7) Failed to connect to synthetic-packages.s3.amazonaws.com port 443: Connection refused
01/08 11:22:24 ERROR: Failed to download Chromium. Please check if https://synthetic-packages.s3.amazonaws.com/Chromium/rpm/chromium-99.0.4844.84-1.el8.tgz address is accessible.
```
For Chromium autoupdate to be successful, one of the following options must be chosen:
1. Allow direct connection to https://synthetic-packages.s3.amazonaws.com/Chromium/rpm/ from the ActiveGate
2. Use a mirror server on your network to serve the packages. See documentation for configuration details
3. Use a /.curlrc file to force curls to always use your proxy. https://reqbin.com/req/c-ddxflki5/curl-proxy-server
  The Synthetic Installer does not have access to the any proxy information provided in the custom.properties file.
  
  Alternatively, Disable Chromium autoupdate and manually update Chromium.

Search for ERROR
The explanation is usually very helpful. For example:

29/06 05:39:55 ERROR: chromium-common package is not in supported version, min version: 96. Installed version is 90.0.4430.93 (90)
29/06 05:39:55 ERROR: Package libwayland-server is NOT installed!
29/06 05:39:56 ERROR: Package mesa-libgbm is NOT installed!
29/06 05:39:56 
29/06 05:39:56        For a typical OS installation with a working Internet connection, use
29/06 05:39:56        the following commands to install required package(s):
29/06 05:39:56 
29/06 05:39:56            sudo yum install -y libwayland-server mesa-libgbm
29/06 05:39:56 
29/06 05:39:56            sudo rpm -e chromium chromium-common
29/06 05:39:56 
29/06 05:39:56            curl --remote-name https://synthetic-packages.s3.amazonaws.com/Chromium/rpm/chromium-112.0.5615.165-1.el8.tgz
29/06 05:39:56            tar xzf chromium-112.0.5615.165-1.el8.tgz
29/06 05:39:56            sudo yum install -y chromium-112.0.5615.165-1.el8.x86_64.rpm chromium-common-112.0.5615.165-1.el8.x86_64.rpm
29/06 05:39:56 
29/06 05:39:56 ERROR: Install missing or reinstall invalid packages and restart installation.

Here you can see that there we 2 packages that needed to be installed. These are Chromium dependencies. The commands to resolve it are provided.
Alternatively, you can just try updating all the Chromium and Engine dependencies using the commands relevant to your OS found here

After resolving the cause of the failure:

delete any .sh files from /var/lib/dynatrace/packages
Retry the update - either wait for the autoupdate to retry (usually hourly) or manually click update.

What if this doesn't resolve the issue

If you do not care about the root cause or just want to get the ActiveGate up and running again, you can uninstall and reinstall the ActiveGate. You can choose the private Synthetic Location to add this to during installation now, see step 6, so no Synthetic monitors will need to be updated to use a new Location. or
create a chat or support ticket and provide the support archive or logs you have gathered during your troubleshooting and provide the outcome of the troubleshooting steps.