Dynatrace AppMon 6.5 Documentation

Other Doc Versions & KB

Skip to end of metadata
Go to start of metadata

The following page describes various techniques for troubleshooting and for collecting the necessary data for Customer Support.

Component Set Up Verification

The following sections describe how to identify set up issues.

Verify that the CICS Agent group and resources installed correctly.

Note

Icon

The group name may be different, as well as the two-character suffix representing the CICS release of the Agent (for example, CTS52 uses 69).

From the CICS Region, look for these messages to validate the CICS resources have been defined:

Verify that the CICS Agent PLT program was invoked.

The CICS Agent no longer logs its messages to the JESMSGLG DD.

Its messages are now logged to the zRemote Agent log, accessible from within the AppMon Client, in the same manner as all the other Agent log files. Look for log entries similar to the following:

Verify that the zDC has started correctly.

Verify that the zDC has started the correct version, initialized successfully and has started/attached to the z/OS zLocal (formerly USS – Unix System Services) Agent (messages in zDC SYSPRINT DD).

If the following error message appears in the z/OS zRemote Agent log file:

Then the Collector/Server log file should be searched for the following connection error:

or the following message:

Make the appropriate adjustments to the affected components.

If the following message appears in the zDC job log (your subsystem name may be different than MEPC):

Follow the steps outlined in the Emergency ZDC Termination section.

Insufficient Access Authority message when starting the zDC.

If the following messages (or something similar) appear in the zDC joblog:

And the associated messages appear in the z/OS zLocal Agent logfile:

These can be safely ignored. After downloading the core Agent initially, the bootstrap Agent is trying to set a flag for the downloaded library that is needed in certain circumstances. It doesn't try to set this flag during the second run because it doesn't re-download the library if it's already there.

Verify that the z/OS zLocal Agent is running and connected to the zDC and to the AppMon Collector/Server (in zDC SYSPRINT).

Icon

Note that Agent status information and Agent log files for connected Agents are available through the AppMon Client, but this process can help to diagnose Agent connection failures.

The standard AppMon Agent logfile will contain the standard AppMon Agent messages relating to startup, logging, versioning, and connectivity to the Server/Collector.

The SYSPRINT of the zDC will contain the stdout for the z/OS zLocal Agent, and this will contain the location of the z/OS zLocal Agent log file.

If the SYSPRINT does not contain this information, the zDC can be instructed to display it using the following command:

Once that log file is located, it is accessible by the standard techniques. Use OMVS, ssh/telnet, or view it in the AppMon Client like a regular Agent log file.

This log file should be searched for the following items to verify that all the channels are connected properly.

The corresponding zRemote Agent log file should be searched for the following (the values inside the bracket should reflect your live data).

If the following appears in the zRemote Agent log file:

This indicates that the name specified in the zRemote Agent startup does not match a name defined in any System Profile defined to the AppMon Server. You must alter the name parameter to match your agent mapping, or define a new agent mapping that matches the name parameter. 

Verify the connectivity loop by issuing a Test Ping

  1. Log in to CICS.
  2. Execute the transaction 'DTAX' with a parameter of 'PING'.
  3. Verify that the PING traveled from CICS to zDC to z/OS zRemote Agent and back again from z/OS zRemote Agent to zDC.

Output of the DTAX Transaction:

Log message in the Agent log file:

Verify that the z/OS zRemote Agenthas received the CICS Sensor configuration.

Log messages are in the zRemote Agent log file. You will need to enable the 'config' log level.

Values for SENSOR_CICS:

00 = not placed
01 = placed and disabled
03 = placed and enabled

The default settings:

Sample of customized settings:

z/OS zLocal Agent Could Not Be Started

Check the job log of the zDC for any of the following error messages.

Icon

Note: These will not appear in the z/OS zLocal Agent log file, since the Agent could not be started.

Invalid permission on the z/OS zLocal Agent folder or libraries.

Correct the permission on the z/OS zLocal Agent folder/binaries, so they are at least readable and executable.

TCP/IP access denied.

If your security system, such as Top Secret or ACF2, denies access to TCP/IP by default, add a rule to permit the zDC user ID to access TCP/IP.

Invalid location specified for the z/OS zLocal Agent.

Correct the DTAGTCMD parameter in the SYSIN DD of the zDC to point to the correct z/OS zLocal Agent binary location.

System Under Diagnosis (CICS Region) Could Not Connect to the zDC

Check the job log of the affected CICS region(s) for the following message, where xxxx = the subsystem ID of the zDC that the CICS region is trying to connect to, or blank if it is trying to connect to the default subsystem that is configured with the DEFAULT(YES) parameter. It is advisable to simply search for the error message code.

Verify that the zDC with that subsystem ID is started. If so, then try to issue a DTAX Enable command to re-enable the connections.

Expected Queues and/or Transactions (Including SOAP and CTG Transactions) are Not Being Monitored

Verify that the CICS Sensor is configured correctly, and that the monitored CICS region's configuration matches that of the CICS Sensor in the system profile.

Edit the CICS Sensor Properties for the System Profile under diagnosis.

Issue a DTAX CONF command to display the following output:

Disabled IMS Agent 

If an ABEND occurs in the IMS Agent code, the recovery process will, if possible, produce ABEND diagnostics and then disable the IMS Agent. The IMS system will continue to function. When this occurs a series of WTO messages will be written to the system log for the IMS Control Region and/or IMS dependent region. A sample normal message set follows: 

Different or additional messages may be issued if abnormal conditions are encountered by the recovery process, such as when dynamic storage cannot be obtained, retry is not permitted, or no SDWA was passed. All of the messages related to the ABEND recovery process are documented in the zOS Agent Messages section.  

A Software (SFT) Error Record further describing the ABEND is usually written to the zOS system SYS1.LOGREC data set. You should run the zOS EREP utility program to print the Software (SFT) Error Record associated with the ABEND.

Optionally, an SVC dump may be taken during recovery, depending on the ABEND recovery option specified or defaulted to when the IMS Agent was injected. The default action is to not capture an SVC dump when ABEND recovery is driven. This option can be specified as a parameter when the IMS Agent is initially injected, or specified in conjunction with the Modify or Enable function parameters to toggle dump capture during recovery on or off for a previously injected IMS Agent. See IMS Agent Injection and Configuration 

When the IMS Agent becomes disabled as a result of the ABEND recovery process, it will remain disabled until explcity re-enabled using the IMS Agent injection program. See IMS Agent Injection and Configuration

When the IMS Agent becomes disabled as a result of the ABEND recovery process, you should contact  support for a resolution.

Migration Troubleshooting

In general: Do not connect newer z/DCs to an older AppMon Server or newer Agents to older z/DCs. Newer AppMon Servers with older other components should normally be fine – no matter if Agents are bootstrapped or not (except a "wild" Agent mix across CICS regions).
The bootstrapping / auto-updating difference:
Agents that have been started by the bootstrap part will be updated on application restart.