To troubleshoot your application, use DNA and the techniques described here, starting with application performance triage.
Application triage is the process of identifying performance bottlenecks quickly and accurately, with the goal of understanding the cause and therefore the owner of the problem. The key is to provide proof of the root cause so that triage does not become fingerpointing.
There are three steps to performing application performance triage:
Measure: start with the failing metric of end-user response time.
Analyze: determine how the application interacts with the client, network and server.
Prioritize: define the relevant performance constraints, predict potential improvements, and assign to the appropriate technical resource.
For our purposes, we will focus the triage process on the first-tier of a distributed n-tier application and outline an analysis methodology. (A first-tier application is, for example, a Web server, and subsequent tiers might include an application server or DB server.)
Determine the appropriate capture approach
DNA insight into end-to-end network delays and their effect on application performance can be derived from pure measurement, from active sampling, or from user-defined values. You can also choose to ignore network delays and focus on application behavior.
The approach you take is based on your ability to capture trace files, the type of problem you are analyzing, and the characteristics of the network path between the client and server. There are four methods:
The most robust method to derive network delay information is to measure it. DNA uses simultaneous capture files from both the client and server locations and merges these into a single task for analysis. Because the packet send and received times are measured at each end, the results are extremely accurate.
Active Latency Detection
DNA provides a capability to actively measure network latency. This feature samples network latency from a single capture agent (typically at the client location) as the transaction executes, and determines network delays from the results. Because this method only requires a single agent, it is valuable for troubleshooting problems where installing a second agent (typically at the server) is difficult.
Single Trace Adjust
The Single Trace Adjustment feature uses a constant value for network latency to determine network delays. Like the Active Latency Detection method, it provides end-to-end analysis from a single trace file. Because the delay value is constant, it is best used under normal network conditions, where link utilization (and therefore queuing delay) remains relatively constant.
Unadjusted Single Trace
For local environments (especially switched segments) where delays between the client and server are very small, no merge or adjustment may be necessary. The analysis provided by DNA will not derive delays due to bandwidth or latency, but rather focus on processing and sending delays.
Set up and perform the capture
Use this general procedure to prepare for capturing a trace.
If you will be merging trace files, you will use DNA capture agents installed on (or near) each node. You can use other trace file sources as well. If you will use only one capture file, this can be from either the client or server location. Configure the capture filters at each location so that only pertinent traffic is captured in the traces. These will typically be defined to capture traffic only between the client and server’s IP addresses.
Start the capture and have the user reproduce the problem. This will typically be one click or Enter keystroke on the client machine, avoiding any client “think time” for typing between screens. You want to capture the network traffic that occurs when the system is performing work for the user. When the transaction completes, stop the capture.
As appropriate, merge or adjust the trace files in DNA. The resulting task then includes detailed end-to-end network delay information.
For problems that are intermittent and difficult to reproduce, consider using DNA's unattended capture mode.
Validate and clean up the captures
Use this procedure to help you determine what traffic is important to a transaction and delete extraneous traffic that might slow your analysis. You should duplicate the original task before you begin your editing.
Using the Conversation map view, delete spurious conversations: connections whose traffic is not part of the transaction under test. These may include broadcast traffic, keep-alive packets on a connection from a previous transaction or application, SMB traffic from a background mail application, etc. You may want to split the Conversation map with the Thread analysis or Packet trace view to help you identify which conversations to delete.
Using the Thread analysis view, delete threads which are not part of the transaction under test. These may include threads which have begun prior to the start of the capture, threads that have not yet completed at the end of the capture, or threads within a trace that are not important to the transaction such as SMB file server threads to a SQL Server. You may want to split the Thread analysis with the Packet trace view to help you identify which threads to delete.
Using the Bounce diagram, delete start and end idle times. These may result from edits you make in the validation steps above, or from your settings in the Tools ► Options ► Capture/import tab. Do not perform this step if you want to include client processing times that occur prior to the transmission of the first packet, or after the reception of the last packet.
Using the Conversation map, verify that the client node has been correctly identified. DNA selects as the client the first node to transmit a packet in the task. If you have deleted traffic, you may need to manually specify the client node. You may also want to rename nodes here to make them more identifiable in the reports.
Apply triage troubleshooting methodology
Once you have validated that the task is an accurate representation of the traffic and timing of the problem transaction, you are ready to begin the analysis.
Start by opening the Performance overview window for the task you are analyzing and attempt to determine the source of the bottleneck.
Use the Response time predictor to estimate how much performance would improve should you choose to address the constraint. Note that you will sometimes have multiple significant bottlenecks in a single task. For instance, you may have a task that has both a high server processing delay and numerous network retransmission errors. In these cases, you may follow multiple flows.
Make baseline comparisons
Bottleneck analysis, beginning with the most significant contributor to response time, is the default approach to troubleshooting persistent performance problems, but frequently troubleshooting is driven by a change in performance, and a comparison between acceptable performance and poor performance may yield quicker and more focused results.
DNA provides a Comparison view that intelligently observes and reports on the key differences between two tasks, one of which you define as the baseline or reference task. The Comparison view will help concentrate your analysis efforts on the most significant change in transaction performance, not simply the most significant contributor.
If you have an intermittent problem, it may be quite simple to capture a baseline to use in for comparison purposes. If the problem is isolated to one location, you might use a trace from a different location as a baseline. More pro-actively, you might want to capture a few key transactions of an important application for future comparison use. Be sure to use and document repeatable data.
The Comparison view presents its observations and supporting data in a tabular format, interpreting the key noted differences (such as the response time contribution from network congestion or a notable difference in thread execution). Armed with the interpretation you can then use the appropriate in thread DNA views for supporting detail.
Use power tools
Power Tools enable you to run programs that are not part of the as-delivered DNA product using the Power Tools button on the toolbar.
Typically, these programs are provided by Dynatrace to meet certain customer-specific purposes such as troubleshooting a particular problem, but you can add any program to the menu so that the program is available as a tool within the DNA application.
Select Tools ► Power tools (or click on the toolbar) to list power tools available to you from within DNA.
Select Tools ► Options and click the Power tools tab (or click on the toolbar and select Customize) to add and remove tools listed on the Tools menu and to configure Power tools settings.