Server Timeline Dashlet

Description

The Server Timeline dashlet can be used to perform detailed and visual analysis of transactions captured by any Dynatrace agent (Java, .NET, Webserver, Native ADK).

The following screenshot shows the dashlet with some sample data from EasyTravel along with annotations that explain the various parts of the dashlet UI:

Overview

This is an experimental release. Use at your own risk. Do not use in production.
Please let us know what you think in the forum.

Name	Dynatrace Server Timeline Dashlet
Product Version	Dynatrace 5.6+
Author	Dynatrace LLC
Plugin Version	Matching the product version
Download	AppMon 2018 October (7.2) AppMon 2018 April (7.1) AppMon 2017 May (7.0) Dynatrace AppMon 6.5 Dynatrace 6.3
License	Dynatrace Experimental Software License
Support	Experimental - Not Supported

Table of content

Requirements

This dashlet performs a lot of analysis directly in the client and therefore typically requires

More heap memory in the client: 768M is recommended for analyzing up to 300.000 records, 1024M is recommended for analyzing up to 1 million records.
More bandwidth from the Dynatrace client to the Dynatrace server: A full refresh of the dashlet with 300.000 records transfers about 10 MB of data from the server to the client.

Presented data

The following sample illustrates the data presented by the Server Timeline dashlet. Assume that we analyze a single PurePath that was created for a HTTP request handled by a Servlet. The servlet internally calls 2 backend services via RMI in order to fulfill the request:

| Method                     | Argument     | Exec Total | API      | Agent    |
+----------------------------+--------------+------------+----------+----------+
| doGet                      | /booking.jsp |       1000 | Servlet  | Frontend |
|   checkCreditCard          | -            |        400 | Remoting | Frontend |
|     Synchronous Invocation | -            |          - | -        | -        |
|       checkCreditCard      | -            |        300 | Remoting | Backend  |
|   processBooking           | -            |        500 | Remoting | Frontend |
|     Synchronous Invocation | -            |          - | -        | -        |
|       processBooking       | -            |        400 | Remoting | Backend  |
|         executeQuery       | SELECT * ... |        200 | JDBC     | Backend  |

This results in 4 server calls. 1 for the entry point of the transaction in the frontend, 2 for the backend service calls and 1 special record for the slow database query.

Property	HTTP request	RMI call to checkCreditCard	RMI call to processBooking	Database query
`Name`	/booking.jsp	checkCreditCard	processBooking	SELECT * FROM Account
`AgentName`	Frontend	Backend	Backend	Database
`EntryPoint`	true	false	false	false
`WebRequest`	true	false	false	false
`Kind`	Web Request	Remoting	Remoting	Database
`Http.Request.Method`	GET	-	-	-
`Duration`	1000	300	400	200
`SelfTime`	300	300	400	-
`RemotingCount`	2	0	0	-
`RemotingTime`	900	0	0	-
`DatabaseTime`	0	0	200	-
`TotalDatabaseTime`	200	0	200	-

The timeline will visualize the start time and the duration of each of those server calls with a box.
The tree table below shows all server calls that match a specific filter expression (e.g. all transaction entry points) and group them together hierarchically using certain properties (e.g. by agent name and URL).
The histogram, scatter plot or bar chart on the bottom will help to visualize certain metrics for a group of calls (e.g. all HTTP requests with a specific URL).

The following table shows a few different filter expression and the matching server calls:

Filter Expression	Matching server calls
`EntryPoint`	/booking.jsp
`Agent = "Backend"`	checkCreditCard, processBooking
`Kind = "Database"`	SELECT * FROM Account
`Duration >= 500`	/booking.jsp, processBooking
`DatabaseTime > 100`	processBooking

Organizing the server calls hierarchically into groups is important to make sense of the data, especially if thousands of calls have to be analyzed.

Without any splitting criteria the dashlet will show a flat list of records, similar to the upper table in the PurePath dashlet.

| Name                    | Splitting  | Count | Duration | SelfTime | Database Time |
+-------------------------+------------+-------+----------+----------+---------------+
| /booking.jsp            | -          |     1 |     1000 |      300 |             0 |
| checkCreditCard         | -          |     1 |      300 |      300 |             0 |
| processBooking          | -          |     1 |      400 |      400 |           200 |
| SELECT * FROM Account   | -          |     1 |      200 |        - |             - |

Splitting by Agent results in:

| Name                    | Splitting  | Count | Duration | SelfTime | Database Time |
+-------------------------+------------+-------+----------+----------+---------------+
| Frontend                | Agent      |     1 |     1000 |      300 |             0 |
|   /booking.jsp          | -          |     1 |     1000 |      300 |             0 |
| Backend                 | Agent      |     2 |      700 |      700 |           200 |
|   checkCreditCard       | -          |     1 |      300 |      300 |             0 |
|   processBooking        | -          |     1 |      400 |      400 |           200 |
| Database                | Agent      |     1 |      200 |        - |             - |
|   SELECT * FROM Account | -          |     1 |      200 |        - |             - |

Splitting by EntryPoint results in:

| Name                    | Splitting  | Count | Duration | SelfTime | Database Time |
+-------------------------+------------+-------+----------+----------+---------------+
| true                    | EntryPoint |     1 |     1000 |      300 |             0 |
|   /booking.jsp          | -          |     1 |     1000 |      300 |             0 |
| false                   | EntryPoint |     3 |     1100 |      700 |           200 |
|   checkCreditCard       | -          |     1 |      400 |      300 |             0 |
|   processBooking        | -          |     1 |      500 |      400 |           200 |
|   SELECT * FROM Account | -          |     1 |      200 |        - |             - |

Splitting first by EntryPoint and then by Kind results in:

| Name                      | Splitting  | Count | Duration | SelfTime | Database Time |
+---------------------------+------------+-------+----------+----------+---------------+
| true                      | EntryPoint |     1 |     1000 |      300 |             0 |
|   Web Request             | Kind       |     1 |     1000 |      300 |             0 |
|     /booking.jsp          | -          |     1 |     1000 |      300 |             0 |
| false                     | EntryPoint |     3 |     1100 |      700 |           200 |
|   Remoting                | Kind       |     2 |      900 |      700 |             0 |
|     checkCreditCard       | -          |     1 |      400 |      300 |             0 |
|     processBooking        | -          |     1 |      500 |      400 |           200 |
|   Database                | Kind       |     1 |      200 |        - |             0 |
|     SELECT * FROM Account | -          |     1 |      200 |        - |             - |

Dashlet functionality

The dashlet consists of 3 major areas.

The graph on top visualizes the start time and duration of server calls in a timeline.
The tree in the center can be used to filter and group together server calls in order to analyze the resulting aggregated metrics.
The charts on the bottom can be used to visualize certain metrics for groups of server calls selected in the table.

Dashlet display options

The dashlet display options allow to adapt the basic layout and workflow of the dashlet:

Show timeline
Show table
Show charts
Apply filter expression to timeline: If this option is enabled, the timeline will only show server calls that match the filter expression usually only used for filtering the table.
Apply table selection to timeline: If this option in enabled, the timeline will highlight server calls that are selected in the table below. If this option is disabled, it is possible to filter the table by selection server calls in the timeline.

Timeline

The timeline visualizes the start time and duration of server calls in a chart the uses time as its X-axis.

The toolbar on top can be used to:

Preconfigure the timeline, table, and charts for some common analysis workflows (Run analysis)
Change the zoom level of the chart. However in most cases zooming with the mouse will be simpler and more accurate.
Change the way how similar server calls are grouped together in the timeline (Group by)
Change the way how colors are used to visualize specific metrics in the timeline (e.g. color calls that use a lot of CPU red and others green)

The context menu additionally allows to configure the following options:

Maximum number of rows: This setting allows to limit the maximum number of concurrent server calls that are shown for a single group.
Measures: These settings are used to configure the heat charts that are visible above server calls that are grouped by agent name and host.
- Host CPU utilization:
- Process CPU utilization:
- Host memory utilization:
- Process memory utilization: This requires a subscription of the Java Virtual Machine - Memory Utilization measure in the system profile.

Run analysis

Name	Effect
Analyze all transaction entry points	Timeline: Group by agent name and host. Table filter: `EntryPoint AND NOT Processing` Table splitting: `AgentGroup, AgentName, Kind, Name`
Analyze HTTP requests	Timeline: Group by HTTP host. Table filter: `EntryPoint AND NOT Processing AND WebRequest` Table splitting: `Application, Http.Request.Host, Http.Request.PathTree, Http.Request.URI`
Analyze HTTP errors	Timeline: Group by HTTP host, color by HTTP response code. Table filter: `EntryPoint AND NOT Processing AND WebRequest AND Http.Response.Status >= 400` Table splitting: `Http.Response.Status, Http.Request.Host, Http.Request.PathTree, Http.Request.URI`
Analyze agents	Timeline: Group by agent name and host, color by consumed CPU time. Table filter: `NOT Processing` Table splitting: `AgentGroup, AgentHost, AgentName, Kind, Name`
Analyze database usage per transaction	Timeline: Group by agent name and host, color by amount of time spent for database calls. Table filter: `EntryPoint AND NOT Processing AND TotalDatabaseCount > 0` Table splitting: `AgentGroup, AgentName, Kind, Name`
Analyze garbage collection impact on transaction	Timeline: Group by agent name and host, color by amount of time spent for GC. Table filter: `EntryPoint AND NOT Processing AND TotalGarbageCollection > 0` Table splitting: `AgentGroup, AgentName, Kind, Name`
Analyze synchronization impact on transaction	Timeline: Group by agent name and host, color by amount of time spent for lock synchronization. Table filter: `EntryPoint AND NOT Processing AND TotalSyncAndWaitTime > 0` Table splitting: `AgentGroup, AgentName, Kind, Name`
Analyze geographical client locations of HTTP requests	Timeline: Group by country of HTTP client. Table filter: `EntryPoint AND NOT Processing AND WebRequest` Table splitting: `Http.Client.LocationTree, Http.Request.Host, Http.Request.URI`
Analyze user agents of HTTP requests	Timeline: Group by browser used by HTTP client. Table filter: `EntryPoint AND NOT Processing AND WebRequest` Table splitting: `Http.Client.BrowserTree, Http.Request.Host, Http.Request.URI`
Analyze transferred bytes for HTTP requests	Timeline: Group by HTTP host. Table filter: `EntryPoint AND NOT Processing AND WebRequest AND (Http.Request.Size >= 0 \|\| Http.Response.Size >= 0)` Table splitting: `Application, Http.Request.Host, Http.Request.PathTree, Http.Request.URI`
Analyze HTTP requests taking longer than 1s	Timeline: Group by HTTP host. Table filter: `EntryPoint AND NOT Processing AND WebRequest AND Duration > 1` Table splitting: `Application, Http.Request.Host, Http.Request.URI`
Analyze HTTP requests consuming more than 1s of CPU time	Timeline: Group by HTTP host. Table filter: `EntryPoint AND NOT Processing AND WebRequest AND TotalCPU > 1` Table splitting: `Application, Http.Request.Host, Http.Request.URI`
Analyze HTTP requests making more than 100 database calls	Timeline: Group by HTTP host. Table filter: `EntryPoint AND NOT Processing AND WebRequest AND TotalDatabaseCount > 100` Table splitting: `Application, Http.Request.Host, Http.Request.URI`
Analyze HTTP requests from robots	Timeline: Group by browser. Table filter: `EntryPoint AND NOT Processing AND WebRequest AND Http.Client.BrowserType = "Robot"` Table splitting: `Http.Client.BrowserTree, Http.Request.Host, Http.Request.URI`

Group by

This setting effects the way how server calls are grouped together vertically into named groups. The following splittings are available in the timeline:

Agent ID and Thread
Agent ID
Agent name and host (default)
Agent name
Agent host
Agent group
Application
Kind
HTTP request method
HTTP host
HTTP response code
HTTP response content type
Browser type
Browser
Continent
Country

Color by

This setting effects the color used for painting server calls in the timeline. The following coloring criteria are available in the timeline:

None (default): All server calls are colored in a neutral blue
Agent name and host: Use different colors for server calls on different agents
Application
Kind
HTTP request method
HTTP host
HTTP response code
HTTP response content type
Browser type
Browser
Continent
Country
CPU %: Percent of time consuming CPU time while executing server call
Self time %: Percent of time spent in this server call without time spent in child calls
Garbage Collection %: Percent of time spent in garbage collection
Synchronization %: Percent of time spent for synchronizing with other threads (e.g. using locks)
Database + Web Service + Remoting %: Percent of time spent for waiting for either database, web service or remoting calls
Database %: Percent of time spent for waiting for database calls
Web Service %: Percent of time spent for waiting for web service calls
Remoting %: Percent of time spent for waiting for remoting calls
HTTP Requests %: Percent of time spent for waiting for HTTP requests

Tree with filtering and grouping capabilities

The tree in the center of the dashlet allows flexible filtering and grouping of server calls. Both are based on text-based expressions that are similar to conditions in C-based programming languages.

The toolbar on top can be used to:

Change the current filter or splitting expression
Choose between a couple of predefined filter or splitting expressions
Save the current filter or splitting expression so that is can easily be reused later.

The context menu additionally provides the following actions:

Focus on selected records: Show and analyze only the selected server calls in the timeline and tree. This can be used to reduce data similar to applying a filter expression. E.g. a typical workflow would be to split by URI and then focus on the requests consuming the most amount of CPU time.
Focus on selected PurePaths: Show and analyze only server calls of PurePaths contained in the current selection in the timeline and tree. This is similar to a drill-down without leaving the current dashlet.
Previous focus: Restore the focus after it was changed by one of the first 2 actions.
Reset focus: Reset the focus so that all server calls (matching the filter expression) are again showed in the timeline and tree.

Available values

All values are case insensitive.

Note, that the text controls provide completion of value names by pressing Ctrl + Space.

Property	Type	Description	Example
`Agent`	String	Agent identifier using the format <agent-name>@<agent-host> [<agent-id>]	dotNetFrontend_easyTravel@dynaday [703ad952]
`AgentName`	String	The name of the agent	dotNetFrontend_easyTravel
`AgentGroup`	String	The name of the agent group	Customer Web Frontend (Java)
`AgentNameAndHost`	String	Agent name and host using the format <agent-name>@<agent-host>	dotNetFrontend_easyTravel@dynaday
`AgentHost`	String	The host of the agent	dstadler
`ThreadName`	String	The name of the current thread	http-bio-8080"-exec-7
`EntryPoint`	Boolean	Whether this call is the start of a transaction	-
`ThrewException`	Boolean	Whether this call threw an exception	-
`CompareSource`	Boolean	Whether this call comes from the comparison source session	-
`Processing`	Boolean	Whether this call was still in progress during analysis of the dashlet	-
`Error`	Boolean	Whether this call resulted in some kind of error as specified by the "Error Detection" section in the system profile	-
`Kind`	String	Classification of call in "Method Call", "Web Request", Web Service", "Remoting", "Messaging", "Database", and "Client-side call"	-
`All`	Splitting	Special splitting criteria that always yields the value "All". Useful for getting overall metrics without splitting	-
`Name`	String	A short string representation of the call, e.g. URI for web requests, method name for method calls, etc.	/index.htm, SomeClass.someMethod()
`Application`	String	The name of the application, as specified by the "Applications" section in the system profile	easyTravel portal
HTTP requests
`WebRequest`	Boolean	Whether this call was processing a web request, same as Kind == "Web Request"	-
`Http.Request.Method`	String	The HTTP method used for this request	GET, POST
`Http.Request.Host`	String	The value of the HTTP "Host" request header.	www.compuware.com
`Http.Request.URI`	String	The URI path of the HTTP request	/index.htm
`Http.Request.Query`	String	The query string of the HTTP request	param=value
`Http.Request.Referrer`	String	The value of the HTTP "Referer" request header	http://localhost:8080/legal-orange.jsf
`Http.Request.Header[name]`	String	The value of some specific HTTP request header. This requires capturing of this header in the Servlet, ASP or web request sensor.	E.g. for RequestHeader["Connection"] the value "keep-alive"
`Http.Request.Cookie[name]`	String	The value of some specific cookie	E.g. for Cookie["jsessionid"] the value "19123012397232"
`Http.Request.Path`	Splitting	Special splitting value, that splits the HTTP URI path hierarchically on every folder identified by "/"
`Http.Request.Size`	Integer (bytes)	The amount of data transfered from the HTTP client to the server. Requires capturing of request/response sizes in Servlet, ASP or web request sensor.
`Http.Response.Status`	String	The HTTP response code for the HTTP request	"200 - OK", "304 - Not modified", etc.
`Http.Response.ContentType`	String	The value of the HTTP "Content-Type" response header	"text/html", "image/png", etc.
`Http.Response.ContentEncoding`	String	The value of the HTTP "Content-Encoding" response header	"gzip"
`Http.Response.Header[name]`	String	The value of some specific HTTP response header. This requires capturing of this header in the Servlet, ASP or web request sensor.	E.g. for ResponseHeader["Cache-Control"] the value "max-age=0, no-cache, no-store, must-revalidate"
`Http.Response.Size`	Integer (bytes)	The amount of date transfered from the HTTP server to the client. Requires capturing of request/response sizes in Servlet, ASP or web request sensor.
`Http.Client.IP`	String	The IP address of the HTTP client requesting a resource	192.168.1.50
`Http.Client.Host`	String	The host name of the HTTP client requesting a resource.	machine.host.local
`Http.Client.Continent`	String	The continent where the HTTP client is located	North America
`Http.Client.Country`	String	The country where the HTTP client is located	United States
`Http.Client.Region`	String	The region where the HTTP client is located	California
`Http.Client.City`	String	The city where the HTTP client is located	Palo Alto
`Http.Client.Location`	String	The physical location of the HTTP client requesting a resource	North America - United States - California - Palo Alto
`Http.Client.LocationTree`	Splitting	Special splitting value, that splits first by continent, then by country, region and finally city.
`Http.Client.BrowserType`	String	The category of browser making the HTTP request	"Browser", "Mobile Browser", or "Robot"
`Http.Client.Browser`	String	The browser family making the HTTP request	"IE", "Firefox", "Chrome", "Safari", etc.
`Http.Client.BrowserVersion`	String	The specific browser (with version number) making the HTTP request	"IE 8.0", "Firefox 6.0", etc.
`Http.Client.BrowserTree`	Splitting	Special splitting value, that splits first by browser type, followed by browser family and finally browser version.
`Http.Client.OperatingSystem`	String	The operating system used by the HTTP client.	"Windows XP", "Windows 7", "iOS", etc.
`Http.Client.UserAgent`	String	The value of the HTTP "UserAgent" request header	Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.15) Gecko/20110303 Firefox/5.0
Metrics for a specific server call
`Duration`	Float (seconds)	The total execution time of a call.	-
`CPU`	Float (seconds)	The amount of CPU time consumed by a call	-
`SelfTime`	Float (seconds)	The amount of time spent on a specific subpath, excluding children	-
`GarbageCollection`	Float (seconds)	The amount of time spent for garbage collection on a specific subpath	-
`SyncAndWaitTime`	Float (seconds)	The amount of time spent for synchronization on a specific subpath	-
`DatabaseCount`	Integer	The number of database calls made on a specific subpath	-
`DatabaseTime`	Float (seconds)	The total amount of time spent for database calls on a specific subpath	-
`WebServiceCount`	Integer	The number of webservice calls made on a specific subpath	-
`WebServiceTime`	Float (seconds)	The total amount of time spent for web service calls on a specific subpath	-
`RemotingCount`	Integer	The number of remoting calls made on a specific subpath	-
`RemotingTime`	Float (seconds)	The total amount of time spent for remoting calls on a specific subpath	-
`MessagingCount`	Integer	The number of messages sent on a specific subpath	-
`MessagingTime`	Float (seconds)	The total amount of time spent for sending messages on a specific subpath	-
`ExceptionCount`	Integer	The total number of exceptions thrown on a specific subpath	-
Metrics for a specific server call and its children
`TotalCPU`	Float (seconds)	The total amount of CPU time consumed by subpath and its children	-
`TotalGarbageCollection`	Float (seconds)	The amount of time spent for garbage collection on a specific subpath and its children	-
`TotalSyncAndWaitTime`	Float (seconds)	The amount of time spent for synchronization on a specific subpath and its children	-
`TotalDatabaseCount`	Integer	The number of database calls made on a specific subpath and its children	-
`TotalDatabaseTime`	Float (seconds)	The total amount of time spent for database calls on a specific subpath and its children	-
`TotalWebServiceCount`	Integer	The number of webservice calls made on a specific subpath and its children	-
`TotalWebServiceTime`	Float (seconds)	The total amount of time spent for web service calls on a specific subpath and its children	-
`TotalRemotingCount`	Integer	The number of remoting calls made on a specific subpath and its children	-
`TotalRemotingTime`	Float (seconds)	The total amount of time spent for remoting calls on a specific subpath and its children	-
`TotalMessagingCount`	Integer	The number of messages sent on a specific subpath and its children	-
`TotalMessagingTime`	Float (seconds)	The total amount of time spent for sending messages on a specific subpath and its children	-
`TotalExceptionCount`	Integer	The total number of exceptions thrown on a specific subpath and its children	-

Syntax of filter expressions

The syntax of filter expressions is similar to expressions in other programming languages.

Expression	Description
`false`	Boolean constant `false`
`true`	Boolean constant `true`
`42`	Arbitrary integer constants
`3.14`	Arbitrary floating point constants
`"abc"` `'abc'`	Arbitrary text constants
`/a*(bb)+/`	Arbitrary regular expressions (Java syntax)
`A \|\| B` `A \| B` `A or B` `A OR B`	Either `A` or `B` must be true
`A && B` `A & B` `A and B` `A AND B`	Both `A` and `B` must be true
`A != B` `A <> B`	`A` is not equal to `B`
`A = B` `A == B`	`A` is equal to `B`
`A ~ "test"`	`A` contains string "test"
`A < B`	`A` is smaller than `B`
`A > B`	`A` is greater than `B`
`A <= B`	`A` is smaller than or equal to `B`
`A >= B`	`A` is greater than or equal to `B`
`A ~= /hello\|world/` `A =~ /hello\|world/`	`A` matches a regular expression
`A !~= /hello\|world/` `A !=~ /hello\|world/` `A <>~ /hello\|world/`	`A` does not match a regular expression
`A like "%.gif"` `A LIKE "%.gif"`	`A` matches a simple expression with placeholder `%` (similar to SQL)
`A unlike "%.gif"` `A UNLIKE "%.gif"`	`A` does not match a simple expression with placeholder `%` (similar to SQL)
`A + B`	the sum of `A` and `B`
`A - B`	`B` subtracted from `A`
`A * B`	the product of `A` and `B`
`A / B`	`A` divided by `B`
`NOT A`	`A` must not be true
`A * (B + C)`	Parenthesis can be used to influence operator precedence

The following sample shows a filter expression that shows all server calls, that are entry points into a PurePath, are triggered by HTTP requests, triggered at least 1 call to a database and took longer than 1 second to execute:

EntryPoint AND WebRequest AND TotalDatabaseCount > 0 AND Duration > 1

Syntax of splitting expressions

The syntax of splitting expressions is very simple. It is just a comma (,) separated list of values that can be converted to strings.

For example, the following expression splits first by HTTP host and then by URI path:

Http.Request.Host, Http.Request.Uri

Charts

Dashlet properties

The dashlet properties allow to adapt the analysis of PurePaths for this dashlet:

Result limit: This setting limits the maximum number of server-side calls that can be analyzed by the dashlet and shown in the timeline.
Percent of analyzed PurePaths: This setting can be used to enable sampling of PurePaths for analysis in order to analyze larger timeframes before reaching the result limit
Pre-filter expression: Filter server calls before sending the results from the Dynatrace server to the Dynatrace client. This can be useful when pure client-side filtering is not possible due to reaching the result limit too soon.
Show client-side calls and minimum duration: If this is enabled, the timeline also shows outgoing client-side calls that do not have a corresponding server-side call, e.g. for calls to third-party APIs. Only calls that take longer than a certain threshold are shown.
Show database calls and minimum duration: If this is enabled, the timeline also shows database calls the take longer than a certain threshold.

Additional Screenshots

The following screenshot shows how the timeline could be used in full screen mode in a dashboard for monitoring.