- Summary
- Troubleshooting
- Chromium
- ActiveGate and Synthetic Module mismatch
- Log Messages
- Windows Machines
- What's Next
Summary
This article contains troubleshooting steps for when a Synthetic-enabled ActiveGate is running, but the Synthetic Engine Status is showing as shutting down, as seen in Deployment Status> ActiveGates
Troubleshooting
Chromium
Old version of Chromium
A common cause of this behavior is an old version of Chromium. Dynatrace supports Chromium versions, which are no more than two versions behind the latest Dynatrace-supported version for a specific ActiveGate release. Chromium can be manually updated using the instructions here.
Chromium version shows as 80.0.0000.00
If Chromium version shows as 80.0.0000.00 then the Synthetic Module in unable to recognise the version.
-
For Ubuntu machines using snap, follow these steps to uninstall and reinstall Chromium (replace the package with the expected version from here)
sudo snap remove chromium curl --remote-name https://synthetic-packages.s3.amazonaws.com/Chromium/snap/chromium-127.0.6533.88-2923.tgz tar xzf chromium-127.0.6533.88-2923.tgz sudo snap ack chromium.assert sudo chown -R root:root /tmp/snap-private-tmp sudo snap install --devmode chromium.snap sudo chown -R dtuserag:dtuserag /tmp/snap-private-tmp sudo systemctl restart vuc.service (or better restart whole machine)
-
If SELinux is enabled, reinstall Chromium with the permissive setting.
-
For Linux machines, check the vuc_browser.log for the following error
/opt/dynatrace/synthetic/browser: symbol lookup error:-
undefined symbol: mpg123_param2
if the full error contains/opt/dynatrace/synthetic/browser: symbol lookup error: /lib64/libopenmpt.so.0: undefined symbol: mpg123_param2
run the following command to get the latest update of library "mpg123-libs" (version 1.32.0 or newer is required)
sudo dnf install mpg123-libs
- undefined symbol: drmSyncobjEventfd
if the full error contains/opt/dynatrace/synthetic/browser: symbol lookup error: /opt/dynatrace/synthetic/browser: undefined symbol: drmSyncobjEventfdinstall libdrm-2.4.123-2.el9.x86_64 or newer
sudo dnf install libdrm-2.4.123-2.el9.x86_64
-
ActiveGate and Synthetic Module mismatch
Check if the Synthetic and ActiveGate major versions match. In the image above they do not.
If the major versions are different, check this article for why the Synthetic module failed to update.
Log Messages
Download the logs from the ActiveGate using the ActiveGate diagnostics feature, otherwise you will need to check on the machine itself. /var/log/dynatrace/synthetic or %PROGRAMDATA%\dynatrace\synthetic\log
Check the vuc-status.log for DiagnosticResultsHolder and check if any modules failed to be operational. There are 4 to check multi-protocol, http, httpScript and browser.
Where the module start up is successful, you will see isOperational=true, e.g.
2024-09-18T01:10:53,190Z [DiagnosticResultsHolder]
browser = `DiagnosticResultsChangedEvent[moduleName=BROWSER, diagnosticCasesResult=DiagnosticCasesResult[isOperational=true, cases={BROWSER_MAIN=DiagnosticCaseDetail[isAvailable=true], BROWSER_FALLBACK=DiagnosticCaseDetail[isAvailable=true]}]]`
And where the module startup is not successful, you will see isOperational=false, e.g.
2024-09-23T07:40:25,237Z [DiagnosticResultsHolder]
browser = `DiagnosticResultsChangedEvent[moduleName=BROWSER, diagnosticCasesResult=DiagnosticCasesResult[isOperational=false, cases={BROWSER_MAIN=DiagnosticCaseDetail[isAvailable=false]}]]`
Depending on which modules were unable to startup, check the relevant logs for ERRORs and WARNINGS
-
for HTTP Monitors, vuc-http.log, vuc-http.debug.log and additional log files with stdout/stderr output in executors subdirectory
-
for Browser Monitors, vuc-browser.log & vuc-browser.debug.log
- Running as root without --no-sandbox is not supported
the issue is likely related to the user running the Synthetic Engine. This should be run by dtuserag and not root. Reinstalling the ActiveGate will correct this behaviour. - Failed to create directory
the issue is likely related to the location of the TEMP directory. For Ubuntu with snap, version 20+, the temp directory path must begin with /var/tmp, for example, TEMP=/var/tmp/syn. Also, Dynatrace requires write access to /var/tmp for the installation of Chromium snap packages. See note in docs - No usable sandbox! If you are running on Ubuntu 23.10+ or another Linux distro that has disabled unprivileged user namespaces with AppArmor
Although this message mentions Ubuntu and AppArmor, it can be seen on any Linux distro with or without AppArmor. This behavior is often seen when system hardening measures have been applied and are too restrictive.- Verify config_user_ns and max_user_namespaces kernel settings
- If AppArmor is used, confirm that AppArmor settings allow the use of unprivileged namespaces
- Running as root without --no-sandbox is not supported
-
for NAM monitors, vuc-multi.log, vuc-multi.debug.log
Windows Machines
-
check the OS is supported.
For instance, we do not support Windows 2012 (R2) and from 1.263 found that this causes the VUC to show this shutting down status. This is due to Google stopping support for Chrome on Windows 2012 (R2) with Chrome 109. https://support.google.com/chrome/a/answer/7100626?hl=en - Check if there have been any recent Windows updates, like this one that prevented Chrome from starting up.
- Check if the Activegate machine has any anti-virus/ anti-malware installed. If it does,
- reboot the machine.
- Update the exclusion rules to exclude Dynatrace and Chromium processes.
As outlined in our documentation, Antivirus and anti-malware software can adversely affect Dynatrace Synthetic monitoring capabilities. The antivirus or anti-malware software might block the Chromium browser or Dynatrace processes responsible for executing synthetic monitors, cause Synthetic-enabled ActiveGate installation failures, interfere with network communication, and impact the reliability of measurements. We recommend working with the antivirus or anti-malware software vendor to find the correct settings.
The minimum list of directories and processes to be excluded from AV/ anti-malware checks can be found here.
Note: Since Chromium 129, AV settings that previously allowed Chromium to be installed and run have been found to be insufficient and the full list from here has needed to be excluded.
For the Chrome process, you may need to be very specific and exclude the instance the ActiveGate is using, C:\Program Files\dynatrace\synthetic\Chrome-bin\chrome.exe is the default path.
What's Next
If none of the previous steps resolved the issue, then open a chat and provide the Support Archive from the ActiveGate the troubleshooting steps you have already completed.
You can find further troubleshooting tips for Synthetic in the Synthetic Troubleshooting Map
