Why did the Synthetic Module on a Synthetic ActiveGate fail to update? Showing Failed or Skipped

Platform Troubleshooting
,
1

Summary

This article covers issues with updating the Synthetic Module or Chromium/ Chrome for Testing. Many of these scenarios are also useful for troubleshooting Synthetic Module installation issues.

Confirm the issue

You can check the versions of the ActiveGate, Synthetic Module, and Chromium/ Chrome for Testing on the ‘Deployment Status > ActiveGates’ page. The ‘Last Update’ column will report ‘Skipped’ or ‘Failed’ if an issue occurred.

HannahM_1-1709917229828.png
HannahM_1-1709917229828.png1822×884 50.8 KB

Get the Synthetic installer logs

The Synthetic module installer writes to a separate file. This will either be:

/var/log/dynatrace/synthetic/dynatracesynthetic-install_<datetimeOfInstall>.log
or
C:\ProgramData\dynatrace\autoupdater\log\Synthetic_<version>.log

For Linux, if the file doesn’t exist, check in /tmp/ for any dynatracesynthetic-install_<datetimeOfInstall>.log files

And, also, the autoupdater log

/var/log/dynatrace/autoupdater/dynatrace_ag_autoupdater_nnn.log
or
C:\ProgramData\dynatrace\autoupdater\log\dynatrace_ag_autoupdater_nnn.log

If the ActiveGate was updated within the last 7 days, you can use the ActiveGate diagnostics feature to download it; otherwise, you’ll need to check on the machine itself.

Check the logs​

Most of the time, the message in the installer log is very detailed and helpful. Some scenarios are listed below:

Update times out​

This can be seen in the dynatrace_ag_autoupdater_nnn.log:

2023-02-06 08:51:49 UTC 1.257.216.20230119-103521 (120107) Installing package /var/lib/dynatrace/packages/Dynatrace-Synthetic-Linux-x86-1.257.23.20221220-170840.sh
2023-02-06 09:06:49 UTC 1.257.216.20230119-103521 (120107) Installation process has been killed as the timeout expired (1)
2023-02-06 09:06:49 UTC 1.257.216.20230119-103521 (120107) Installation has failed.

Check the last lines in the Synthetic installer log

  1. Unmounting cache…

    1. This behavior is usually due to security software, such as Microsoft Defender Service, mdatp, interfering. Verify if any antivirus applications are running on this server. To complete the update:
      1. Stop Microsoft Defender/ other antivirus.
      2. Delete any .sh files from /var/lib/dynatrace/packages.
      3. Retry the update - either wait for the autoupdate to retry (usually hourly) or manually select ‘Update’.
        Note: If this resolves the issue, you’ll need to update the Antivirus exclusions to prevent this behavior from happening with every update.
    2. It can also be seen when unmounting the cache takes an extended time to complete
      1. Before trying to update Synthetic again,  try unmounting the cache manually:
         sudo umount /var/tmp/dynatrace/synthetic/cache
      2. If that hangs for a longer time, then it can be done as a lazy unmount:
        sudo umount -l /var/tmp/dynatrace/synthetic/cache
         To be safe after performing the lazy unmount, we recommend rebooting the machine.
      3. Delete any .sh files from /var/lib/dynatrace/packages
      4. Retry the update - either wait for the autoupdate to retry (usually hourly) or manually select ‘Update’.

  2. Command: yum versionlock chromium chromium-common

    The yum versionlock command typically completes in under 10 seconds.

    1. Try running versionlock manually by running the following commands:
      yum versionlock clear

      yum versionlock chromium chromium-common

      yum versionlock | grep chromium
      If this takes more than 30 seconds, check the machine’s health and identify what might be causing its commands to execute more slowly.

      1. If chromium-common is locked, you can rerun the update after deleting any .sh files from /var/lib/dynatrace/packages.

        If it’s not, check the output from yum versionlock | grep chromium

  3. Checking if user dtuserag has read access to:
    There are multiple checks like this in the installer logs, as the installer verifies access to each folder. If the checks are always taking a long time, for instance, more than a minute, it is likely that something like an Antivirus is interfering. Retry the installation after disabling the antivirus from scanning those directories.

Chromium\ Chrome for Testing autoupdate isn’t completing​

Chromium\ Chrome for Testing autoupdate is only relevant for Linux installations, as Windows ActiveGates have Chromium packaged with the installer. These are the most common scenarios:

  1. Search the Synthetic installer log for
    Detected ActiveGate configuration (joined)
    Below that is the section that states which options are enabled. For Chromium\ Chrome for Testing autoupdate to complete, both synthetic.synthetic_autoupgrade_chromium and synthetic_autoinstall must be true.

    1. If synthetic.synthetic_autoupgrade_chromium is false or has no value, update this in the location.
      For Private locations, this can be done in the WebUI
      For Cluster locations, this can be done using the Cluster API
      Note: don’t update this directly in custom.properties as the value is not persisted.
    2. If synthetic_autoinstall is false or has no value, add  synthetic_autoinstall = true to the [synthetic] section of the /var/lib/dynatrace/gateway/config/custom.properties file and restart the vuc and gateway services.

  2. Search the Synthetic installer log for Downloading Chromium package

    This line and the lines that follow will indicate if/ why the download failed. Like this example, where the connection was refused

    01/08 11:22:24 Downloading Chromium packages from https://synthetic-packages.s3.amazonaws.com/Chromium/rpm/chromium-99.0.4844.84-1.el8.tgz...
01/08 11:22:24 Executing command:  curl --silent --show-error --continue-at - --connect-timeout 5 --retry 5 --retry-delay 10 --retry-max-time 80 --output /tmp/chromium/chromium.tgz https://synthetic-packages.s3.amazonaws.com/Chromium/rpm/chromium-99.0.4844.84-1.el8.tgz
curl: (7) Failed to connect to synthetic-packages.s3.amazonaws.com port 443: Connection refused
01/08 11:22:24 ERROR: Failed to download Chromium. Please check if https://synthetic-packages.s3.amazonaws.com/Chromium/rpm/chromium-99.0.4844.84-1.el8.tgz address is accessible.

    The log content is very similar for Chrome for Testing

    06/08 10:43:52 Downloading Chrome package from https://synthetic-packages.s3.amazonaws.com/Chrome/chrome-for-testing-linux64/chrome-for-testing-linux64-138.0.7204.94.zip...
06/08 10:43:52 Executing command: curl --silent --show-error --continue-at - --connect-timeout 5 --retry 5 --retry-delay 10 --retry-max-time 80 --output /tmp/chrome/chrome.zip https://synthetic-packages.s3.amazonaws.com/Chrome/chrome-for-testing-linux64/chrome-for-testing-linux64-138.0.7204.94.zip
curl: (28) Connection timeout after 5000 ms
curl: (28) Connection timeout after 5001 ms
curl: (28) Connection timeout after 5000 ms
curl: (28) Connection timeout after 5001 ms
curl: (28) Connection timeout after 5000 ms
curl: (28) Connection timeout after 5000 ms
06/08 10:45:12 ERROR: Failed to download Chrome. Please check if https://synthetic-packages.s3.amazonaws.com/Chrome/chrome-for-testing-linux64/chrome-for-testing-linux64-138.0.7204.94.zip address is accessible.

    For Chromium autoupdate to be successful, one of the following options must be chosen:

    1. Allow a direct connection to https://synthetic-packages.s3.amazonaws.com
      For Chromium, allow direct connection to https://synthetic-packages.s3.amazonaws.com/Chromium/rpm/ from the ActiveGate.

      For Chrome for Testing, allow direct connection to https://synthetic-packages.s3.amazonaws.com/Chrome/chrome-for-testing-linux64/ from the ActiveGate.

    2. Use a mirror server on your network to serve the packages. See documentation for configuration details.

    3. Use a /.curlrc file to force curl to always use your proxy. See Create a private Synthetic location in Classic — Dynatrace Docs

      Note: The Synthetic Installer does not have access to any proxy information provided in the custom.properties file.

      Alternatively, Disable Chromium autoupdate on the location and manually update Chromium.

  3. Search for ERROR
    The explanation is usually beneficial. For example:

    29/06 05:39:55 ERROR: chromium-common package is not in supported version, min version: 96. Installed version is 90.0.4430.93 (90)
29/06 05:39:55 ERROR: Package libwayland-server is NOT installed!
29/06 05:39:56 ERROR: Package mesa-libgbm is NOT installed!
29/06 05:39:56 
29/06 05:39:56        For a typical OS installation with a working internet connection, use
29/06 05:39:56        the following commands to install required package(s):
29/06 05:39:56 
29/06 05:39:56            sudo yum install -y libwayland-server mesa-libgbm
29/06 05:39:56 
29/06 05:39:56            sudo rpm -e chromium chromium-common
29/06 05:39:56 
29/06 05:39:56            curl --remote-name https://synthetic-packages.s3.amazonaws.com/Chromium/rpm/chromium-112.0.5615.165-1.el8.tgz
29/06 05:39:56            tar xzf chromium-112.0.5615.165-1.el8.tgz
29/06 05:39:56            sudo yum install -y chromium-112.0.5615.165-1.el8.x86_64.rpm chromium-common-112.0.5615.165-1.el8.x86_64.rpm
29/06 05:39:56 
29/06 05:39:56 ERROR: Install missing or reinstall invalid packages and restart installation.

    Here, you can see that two packages needed to be installed. These are Chromium dependencies. The commands to resolve this issue are provided in Private Synthetic locations: ActiveGate installation/update failed due to missing dependencies.
    Alternatively, you can try updating all Chromium and Engine dependencies using the commands relevant to your OS, which can be found here.

General Errors

  1. Search the Synthetic installer log for ERROR
    The explanation is usually beneficial.
    For example:

    18/11 14:46:15 Verifying installed software compatibility...
18/11 14:46:15 ERROR: chromium package is not in supported version, min version: 126. Installed version is 125.0.6422.76 (125)
18/11 14:46:15 ERROR: chromium-common package is not in supported version, min version: 126. Installed version is 125.0.6422.76 (125)
18/11 14:46:15
18/11 14:46:15        For a typical OS installation with a working internet connection, use
18/11 14:46:15        the following commands to install required package(s):
18/11 14:46:15
18/11 14:46:15            sudo rpm -e chromium chromium-common
18/11 14:46:15
18/11 14:46:15            curl --remote-name https://synthetic-packages.s3.amazonaws.com/Chromium/rpm/chromium-140.0.7339.185-1.el9.tgz
18/11 14:46:15            tar xzf chromium-140.0.7339.185-1.el9.tgz
18/11 14:46:15            sudo yum install -y chromium-140.0.7339.185-1.el9.x86_64.rpm chromium-common-140.0.7339.185-1.el9.x86_64.rpm
18/11 14:46:15
18/11 14:46:15 ERROR: Install missing or reinstall invalid packages and restart installation.
18/11 14:46:15 Aborting installation...
18/11 14:46:15 Installation failed.

    This error indicates that the Chromium version installed on the machine is lower than the minimum required for the ActiveGate version, and provides the necessary commands to install the required version.

After resolving the cause of the failure​

  1. Delete any leftover installer files
    1. On Linux, delete any .sh files from /var/lib/dynatrace/packages
    2. On Windows, delete all files, except module. status, from C:\ProgramData\dynatrace\packages
  2. Retry the update - either wait for the autoupdate to retry (usually hourly) or manually select update.

Note:

The ActiveGate status will remain at Update Pending until the .sh files from the failed update are removed from the packages folder. After that, the update option will be available.

What’s Next?

  1. If you’re not interested in the root cause and just want to get the ActiveGate up and running again, you can uninstall and reinstall the ActiveGate. You can now choose the private Synthetic Location to add this to during installation, see step 6, so no Synthetic monitors will need to be updated to use a new Location.
    or
  2. Create a chat or support ticket and provide
    1. the support archive or logs you have gathered during your troubleshooting.
    2. troubleshooting steps you have already completed and the related outcomes.

Related reading 

:open_book: Synthetic Troubleshooting Map

:open_book:  Create a private synthetic location

:open_book:  Why is my Synthetic ActiveGate shutting down?

:open_book:  Why does Chromium show as failed or skipped in Deployment Status when it is up to date?

:open_book:  Private Synthetic locations: ActiveGate installation/update failed because of missing dependencies

12 Likes
2

Thank you @HannahM ,

My synthetic AG failed to update and the root cause was a timeout on the curl command.
The following recommendation solved the issue : Use a /.curlrc file to force curls to always use your proxy. https://reqbin.com/req/c-ddxflki5/curl-proxy-server

Thank you :ok_hand:

1 Like