Problem:

I have installed a PLM peer and created a PLM test but no data reported in the test. How to troubleshoot?

Scenario:

I can see the peer listed on the View Network Machine Profiles page (under Measurement Locations > Private Last Mile) but my PLM test has no data.

Product:

Dynatrace Synthetic Classic > Private Last Mile (PLM)

Steps to Troubleshoot:

  1. For newly created PLM tests, please note it may take up to 3 hours to start to run the test and report data. (Check here for how to start your PLM tests sooner.)
    In the meantime, you may still want to run through the following steps to make sure everything is working correctly.

  2. Double check the configuration setting for the Peer Population. Make sure that your desired PLM peer is included in the Peer Population. (Check here for how to configure a Peer Population.)
    1. To do so:
      1. Run an instant test for the test from the Peer Population. 
      2. Run an instant test for the test from the Peer directly.
    2.  Look for the following on the Instant Test results page: 
      1. "Last Check in Time" for the Peer Population 
      2. "Last Check in Time" for the Peer 
      3. Whether the instant tests can be executed and data reported back, within 3 minutes.
    3.  Results Analysis:
      1. If you see "Last Check in Time" for the Peer Population is "never" whereas the "Last Check in Time" for the peer is up to current, it implies that the Peer is not included in the Peer Population. Please edit and update the Peer Population, delete the current instant test and run a new instant test to test again. (Check here for how to configure a Peer Population.) 

        If the peers are checking in fine and those peers are correctly included in the peer population ( peer population criteria are correct), there are two other possibilities.

        1. The peer population has wrong status for the assigned tests. Peer Population needs to have status as Active under MBGPopulation table of PLM DB. If the status of the peer population is wrong, no tests will run from it.

        2. The peer(s) are not correctly synchronized into the PLM job queue servers: plmjq02 or plmjq03/04. (Server name: GLMDC2jq02.gomez.com or GLMDC2jq03.gomez.com).

        >>Double check the PopulationDefinitions table in PLM DB to see if the peer(s) are included in the population's definition. Example of sql query is as below:

        select * from PopulationDefinitions (nolock)

        where Value in ( 'peer's usys_id')--//find population by peer id

        >> Connect to jq02 server and check if you can find the peer(s) under the JobQueueLookup table.

        Use USYS_PROCESS
        select count(machineid) from dbo.JobQueueLookup jql(nolock)
        where jql.MachineId in (5243465)
        --where jql.population_id in (58378,59784,59785)

        If you have confirmed the issue, this needs to be fixed by Dev and DBAs. Open Support ticket.

      2. If you see the "Last Check in Time" is the same for both the Peer Population and the peer, it means the Peer Population configuration is correct.
      3. If the "Last Check in Time" is up to current (within the last 5 minutes), the peer is checking fine.
      4. If the "Last Check in Time" is beyond last 5 minutes ago, but the two instant tests can be executed and has success data reported back. Everything looks good, just wait for the regular test to run and report data.
      5. If the "Last Check in Time" is beyond last 5 minutes ago, the two instant tests can be executed but has failure reported back. Logon the Peer Machine and playback the script locally.
      6. If the "Last Check in Time" is beyond last 5 minutes ago, and the two instant tests are displaying status as "Pending Machine" or "In Progress" for more than 3 minutes, go to next step.

  3. Logon the Peer Machine. Make sure the peer is functioning properly by checking:
    1. Make sure the peer is running:
      1. On the Peer machine, open Task Manager.
      2. Under Processes, double check if GomezPEERSvc.exe is there.
      3. Under Services, double check if "gomezpeer" service is in status of "Running".
    2. Make sure the related Browser Agents are enabled:
      1. On the Peer machine, click Show Private Last Mile UI.
      2. Open Options > Preferences
      3. Under Misc Options, make sure the "Enable *** Native Browser Agent" is selected. For example, if your test is a FF Agent test, make sure the "Enable Gozilla Native Browser Agent" is selected.
        Save the change and restart the Peer and "peer service" after the option is newly selected. 
    3. If all menus under Options are greyed out, make sure the user credentials used for peer service user is up to current.
    4. If your test is an IE Agent PLM test, make sure that IE Account user is configured. (Check here for how to configure the IE Account user)
    5. Run Diagnosis to check general health of the peer:
      1. On the Private Last Mile UI, go to Help > Diagnosis. 
        You want all indicators are in green states.
      2. The key indicators:
        1. exist agent file or not:
          It indicates whether the desired Browser Agent file is available to use or not. The desired result should be: **-true. (** is the browser agent you have enabled. i.e.IE-true)
          1. If you see nothing listed here, then the Browser Agents were not successfully enabled. Please do the steps under 3.b again.
          2. If you do not see the desired browser agent listed here, then the desired browser agent is not enabled. Please do the steps under 3.b again.
          3. If you see the desired browser agent is listed as false, i.e. FF-false. It means the FF native browser agent is enabled but the related files are not available. 
            To fix it, please do a Full re-install (Full re-install includes uninstall.exe, delete all PLM folders/directories then a fresh new re-install.) 
        2. max consequent fatal:
          It indicates whether any fatal error has been reported by the desired Browser Agent. The desired result should be: **-0. (** is the browser agent you have enabled. i.e.IE-0)
          1. If you see the error number is more than 0, please restart the peer and peer service. (Check here for how to restart peer and peer service.)
            Then run the Diagnosis and check again.
            If the above step does not fix it, do a Full re-install. (Full re-install includes uninstall.exe, delete all PLM folders/directories then a fresh new re-install.)
          2. If the Full uninstall does not fix it, try open the browser agent exe file directly.
            1. If you can open the exe file, it looks like a normal browser window, try use it to open the testing page you use for your PLM test. The desired behavior is it can open the page fine.
            2. If you cannot open the exe file, double check your firewall settings and/or local anti-virus software. Make sure those processes are allowed and/or whitelisted.
            3. If those have been done and still cannot fix it, please enable TRACE debug log and let it collect data for a couple of hours. Then open a Support Ticket and attach the debug log. (Check here for details about how to enable TRACE debug log.)
        3. connect to version server:
          1. It indicates the peer machine’s connectivity to the version server at http://lastmile.gomez.com.
          2. Peer machine needs to have continuous connectivity to the server to work properly. If the state is X, please contact your IT team to whitelist the domain.
        4. connect to appserver:
          1. It indicates the peer machine’s connectivity to the two appservers at https://glm-ssl-s.gomez.com and https://glm-ssl-a.gomez.com.
          2. Peer machine needs to have continuous connectivity to the servers to work properly. If the state is X, please contact your IT team to whitelist the domains.