Information:

Environment

Affected Version: all


Symptoms

Solution

 

Background
When the AppMon Agent (either Java or .NET) is installed on a monitored JVM or CLR runtime environment, there is a small amount of overhead which is incurred on the monitored system. The quantity of overhead can vary depending on how you configure the AppMon agent. This document provides a list of mechanisms to reduce this overhead in the unlikely event that it becomes more than acceptable.

It is important to remember that there are different types of overhead such as Response Time, CPU, Memory. Before embarking on an overhead reduction task, it is best to determine which type of overhead is of concern, and to continue to compare subsequent overhead measurements using the same technique and overhead measurement process.

Techniques on Reducing Overhead

These techniques are listed in the suggested order of implementation. First use technique #1. If that does not provide sufficient overhead reduction, then try technique #2, and so on.

Each technique will provide different levels of overhead reduction, depending on the nature of the system being diagnosed. It is suggested to try different techniques to determine which provides the best benefit.

1) Remove unnecessary instrumentation points This can be accomplished by using Exclusion rules and /or deleting (or deactivating) method sensor Rules which have been created for the agent in question.

2) Use delegation suppression to remove repetitive calls When a sequence of method calls within the same class have no significant performance contribution, these calls should be removed by using delegation suppression on the associated Method Sensor Rule.

3) Remove unnecesssary Servlet Parameter value capturing It is not advisable to capture any parameters that are not explicity needed. This means avoiding the usage of wildcards and explicitly listing only those parameters which are actually needed for Business Transactions or other activities.

4) Do not enable Bind Value Capturing in the JDBC and ADO.NET sensor packs Bind Value Capturing is expensive and should only be used when necessary during the investigation of a specific problem.

5) Reduce SQL Command capture Length The default length of captured SQL Strings is 4096 bytes. It is possible that the same visibility into SQL performance can be acquired by capturing only 2k or 1k of the SQL Strings. Additional benefit can be attained by Inactivating the ADO.NET or JDBC sensor, or setting the SQL capture string length to 0.

6) Do not select 'Include object allocation in PurePaths" In the memory Sensor Properties screen, it is optional to include the object allocations in the PurePath. While this is a very powerful feature, if a specific memory problem is not being investigated, it is suggested to not enable this feature.

7) Do not specify Memory Sensor Rules if not necessary Unless you are investigating specific Memory issues, it is best to unplace or delete any Memory Sensor Rules. This also applies to the Maps and Collection sensors, which are known to cause high overhead in certain applications.

8) Disable Synchronization Diagnosis This feature can be left disabled unless you explicitly suspect synchronization issues, or wish to perform regression testing to verify no synchronization issues.

9) Reduce the PurePath percentage In the Sensor Configuration page of the System Profile Properties screen, the PurePath Percentage can be specified. This provides the ability to only capture a statistical subset of all PurePaths.

10) Do not capture CPU time If CPU time is of no value, it can be disabled in the Sensor Configuration page of the System Profile Properties screen

11) Do not capture Messaging content The contents of a JMS message can be captured and included in the details view of a JMS call. Unless you are specifically investigating an issue that may relate to the contents of a JMS call, this switch should not be enabled. It can be disabled in the Properties page of the JMS Knowledge Sensor Configuration screen.

12) Disable .NET Performance Counters Disabling the collection of these performance counters can have a benefit overall in .NET environments. However this does cause the loss of some metrics. Disabling the querying of Performance Counters can be done by performing one of the following actions:

In the Registry
o Add a key named "disableperfcounters" to the Agent Settings in Registry ("HKEY_LOCAL_MACHINE\SOFTWARE\dynaTrace\Diagnostics\Agent\whitelist\<num>")
o Add a string value containing 'true'
As an Environment Variable
o set the Environment Variable "DT_DISABLEPERFCOUNTERS" to 'true'

When the .NET Performance Counters are disabled, the following counters will no longer be available:

  • NET Common Language Runtime
  • .NET Networking
  • ADO.NET
  • ASP.NET

13) Memory diagnostics with .NET Ensure that Simple/Extended memory dumps are set to AUTOMATIC for .NET environments. Forcing .NET monitored systems to ON, incurs extra overhead, even if memory dumps are not collected during runtime. This setting can be found in the Agent Mapping tab for a given Agent Group, under the Advanced tab for each Agent Mapping alias.

14) Linux kernel version Are you running on a Linux Kernel v2.1.6 or prior? There is a known issue in these kernels which can cause high overhead when collecting CPU time data. Either disable CPU time capturing, or upgrade your kernel.

15) Heavy sensors The Hibernate sensor, Struts sensor, and Spring sensor can cause undesired overhead. Try unplacing some of these sensors to see if they are contributors.

16) Exception sensor Disable stack trace capturing in the Exception sensor, unless this is explicitly needed. Enable exception aggregation. Check for frequently called Exceptions in your PurePaths or in the Exceptions dashlet, right-click them to add an exclude rule for such frequent exceptions.

17) doFilter chaining If there is a lot of doFilter chains, these can be eliminated by unplacing this rule within the Servlet sensor. Use the Method dashlet to determine if large quantities of doFilters are invoked and examine whether they're being reported for static objects such as html/css/js/jpg/jig/png pages.

18) Embedded Repository Ensure you are not using the embedded repository. If performance is a concern, ensure you use an external repository, or simply disable the repository.

19) IBM JREs If monitoring a JVM which is running on an IBM JRE, be sure that the Hot Sensor Placement setting is set to Automatic. This ensures that AppMon is allowed to determine the optimal setting for Hot Sensor Placement from an overhead perspective. Setting this value to "Always On" can cause loss of JIT compilation in IBM JREs.

By trying different combinations of these techniques, overhead reduction goals should easily be attainable.