Information:

Environment

  • Private Last Mile
Symptoms

If you are experiencing issues with Private Last Mile, the following troubleshooting measures may be helpful.

Solution

 

Troubleshooting the Private Last Mile version 1.0 (NOTE: version 1.0 covers the PLM software with an installed VMWare machine for hosting the IE Agent; the current PLM software available out dates this by no longer installing a VMWare machine; this means version 1.0 content may not apply to the new PLM software for troubleshooting; look for version 2.0 for troubleshooting the new PLM software)


Client Application = Private Last Mile PEER located at the hosting machine level
Application Server = Private Last Mile Application Servers located in the 'cloud'

In addition to the below - there is additional information that can be found in the attached 'Private Last Mile Admin Guide'. Some of the below information may be repetitive between them.
  • When enabling the 'IE7 Native Browser Agent', application files that are needed for this Agent are downloaded from Private Last Mile Application Servers. As such, a communication channel between the machine hosting the client application and the Application Servers is required.

 

  • When enabling the 'IE7 Native Browser Agent', neither can the machine hosting the client application be running on a Virtual Machine or can the physical machine be running a previously installed Virtual Machine. 'IE7 Native Browser Agent' operates in Virtual Machine; as such, two instances of the Virtual Machines is unsupported for the client application.

 

  • Check that client application processes are up and running:While the windows local 'Task Manager' can accomplish much of the same, Microsoft's Process Explorer is more powerful and useful - it can be downloaded a no cost here
    • When the client application services are stopped, you should expect only the 'GomezPEER.exe' process to be running.
    • When the client application services are started (assuming you have both 'IE7 Native Browser Agent' and 'Gozilla Native Browser Agent' enabled), you should expect to see the following parent and child processes consistently up and running:
      • Parent - GomezPEERSvc.exe; Child - java.exe
      • Parent - GomezNBA.exe; Child - vmplayer.exe; Sub-Child - vmware-vmx.exe
(The above is an image of the services taken from Process Explorer.)
  • An interesting observation that you may notice is that at time additional child process are spawned from the java.exe process with the name gozilla - this is a instance of the Gozilla Agent on the machine executing a test.

  • Proxy setting for a client application depend almost entirely on the customer proxy server and how it is configured. As such, it is imperative to coordinate with the IT/Network team in control of the proxy server. Having said that, there may remain troubleshooting steps that can be explored based off the environment configuration for the machine hosting the PEER client. The following is not a solution, only an avenue for possible investigation and workaround effort. The 'Gomez Peer' service (located your Windows services menu) is by default configured with Log On As = Local System. When configured with 'Local System', the service inherits the security context of the service control manager, which are the local account credentials the active machine user has logged-in as. As such, those credentials are used to open the connections to the proxy server and would need to be configured to allow passage through the proxy server. For example, say the proxy server was setup to utilize corporate network credential (say pulled from Active Directory) then adjusting the log-on properties to 'Log On As = Network Services' - configuring the service to account credentials within the network domain, which have rights to pass through the proxy server - and adjusting the Private Peer proxy settings to 'Use a proxy server for your LAN' (proxy authentication should likely not be necessary since it the log-on account has been pre-configured).

  • The Private Peer supports operation behind a proxy server and supports these type of proxy servers: Microsoft Internet Security and Acceleration (ISA) Server and Squid. Although, each proxy server supports several Authentication Methods and some may have not been fully verified with the client applications proxy module. Under Squid Digest Authentication has been fully verified.

  • The client application is required to have connectivity to the following three hostnames/IPs over ports 80 and 443:
    • http://lastmile.gomez.com/ - (63.251.134.196:80) should redirect to http://www.compuware.com/application-performance-management/ if successful.
    • https://glm-ssl-s.gomez.com/ - (63.251.134.192:443) should only return an SSL handshake, and will not return HTML that can be rendered by a browser (watch for the server certificate warning).
    • https://glm-ssl-a.gomez.com/ - (63.251.134.193:443) should only return an SSL handshake, and will not return HTML that can be rendered by a browser (watch for the server certificate warning).

  • Bandwidth Testing and Security Concerns - Under normal operation our client application performs bandwidth testing in order to categorize current machine bandwidth and allow for proper categorizing of itself for populations. In order to accomplish this client applications request object data from third party bandwidth testing locations. This being said, bandwidth testing can be manually stopped in the client application through the Private Last Mile UI under the Preferences menu. From there adjusting the 'Bandwidth test mode' to 'Fixed' will technically disable the bandwidth testing.

  • When the 'IE7 Native Browser Agent' is enabled and the respective GomezNBA.exe service and associated child process are running, the IE Agent may still not successfully execute. This is often due to a issue with the TCP communications between the "VMware Network Adapter VMnet8" to the Virtual Image hosting the 'IE7 Native Browser Agent' "the and the physical machine. To check whether this is the problem you can runnetstat -na at a command prompt. What you should expect to find is:

where 192.168.10.X in this EXAMPLE is the IP address of "VMware Network Adapter VMnet8":
..basically TCP connections in a "TIME_WAIT" state between the IP of "Ethernet adapter VMware Network Adapter VMnet8" with a random port # TO the IP of the hostname "gmz-lm" with port 7879.
So, if you don't find similar TCP connections it can possibly resolved by adjusting the local hosts file, as such:
Navigate to the machines hosts file (C:\WINDOWS\system32\drivers\etc\hosts); edit and add the below line to the hosts file; finally, save it under the same formate:
XXX.XXX.XXX.128 gmz-lm
where XXX.XXX.XXX are the first three octets of the IP for "Ethernet adapter VMware Network Adapter VMnet8
Once the hosts file has been edited and saved; stop and restart all client application services. Once all services have been successfully started check that you can ping gmz-lm and see the TCP connections.


Log data to gather for troubleshooting:

What log data to gather for best troubleshooting - note: all the following data may not be necessary based off the issue (case-by-case-basis); furthermore, steps below are done to capture log data from a clean slate, in order to establish a solid baseline of the problem:
  • In the Private Last Mile UI, set Kernel TRACE Mode and GUI TRACE Mode (under Options > Preferences);
  • Stop the Private Peer Service through the Private Last Mile UI (under Options > Services);
  • Close the Private Peer UI;
  • Launch the Task Manager and kill the 'GomezPEER.exe' process if it is still running;
  • Navigate to the Private Peer log folder and rename the following files:
    • ppeer-debug.log > ppeer-debug.log.orig
    • ppeer-debug-ui.log > ppeer-debug-ui.log.orig
    • GomezPEERSvc.log > GomezPEERSvc.log.orig
  • Launch the Private Peer UI and Start the client services.
    • Confirm the services are started (if IE Native Browser Agent is enabled, wait ~1min until the VMware machine hosting the WinXP/IE7 environment is up and running).
  • Check the TCP connections:
    1. Run the 'netstat -na' command and capture the data into a file. You will want to see TCP connections in a "TIME_WAIT" between the IP of "Ethernet adapter VMware Network Adapter VMnet8" and random port # TO the IP of the hostname "gmz-lm" and port 7879.
  • Stop Private Peer services and kill the 'GomezPEER.exe' process; if process is still running.
  • Send the following new log files:
    • ppeer-debug.log
    • ppeer-debug-ui.log
    • GomezPEERSvc.log
    • (optional if exists) GomezNBA.log