📚 Unlock the full potential of Digital Experience Monitoring with the latest Dynatrace, Grail and OneAgent

[paul_kapeller]

Digital Experience Monitoring (DEM) has become front and center within the observability space. That’s why customers have consistently increased demands to gain better visibility and analytics possibilities of their end users’ experiences. At the same time, the landscape of web and mobile technologies is rapidly evolving and Google’s core web vitals and new mobile vitals have shaped the area significantly.

As a leader in observability, Dynatrace is going to launch a significative upgrade to its current solution for modern frontend monitoring. DEM powered by Grail will allow users to gain even deeper insights, further optimize user experiences across devices and drive better business outcomes.

With this post, we’d like to inform you in advance about upcoming benefits and expected impacts, focusing on RUM and our new data model. We’re informing about new apps and additional DEM enhancements in separate posts.

I’m sure you’re excited about what’s to come and at the same time, you can already prepare your organizations for the changes and actions required and ask any questions that come to mind when reading this. We will try our best to respond to them in a timely manner. The post including the FAQ section will be updated based on your feedback and questions.

Grail and the latest Dynatrace

We have seen how logs, business events, metrics and traces have benefited from being on Grail in the past 18 months. Starting in October (current plan for early adoption), your frontend applications will also be able to reap similar benefits. Existing as well as new use cases will be enriched with more data granularity, advanced querying capabilities and extended retention times on the new Dynatrace platform.

New RUM data model and Grail

The core of the improvement is a new, future-proof data model based on RUM and session events; which will subsequently be captured via a new OneAgent module, sent via an enhanced beacon protocol, ingested via OpenPipeline and stored into our new Platform and Grail. An event-based capturing approach allows us to unlock significant improvements in monitoring coverage as well as data analytics and AI. Events will enhance and elevate the value customers can gain compared to user actions today.

Key benefits

DEM powered by Grail offers a range of features designed to enhance your data analysis capabilities. One of the key features is extended retention of RUM data. This allows you to extend your individual retention time up to 10 years, providing a long-term view of your data trends.

In addition, we auto-capture all requests made by your frontend applications, including their W3C timings. These requests can then be monitored, analyzed and alerted on individually with the latest Dynatrace, offering a granular view of your frontend applications’ performance.

To help you better understand your end-users’ behavior, our new agent module can also auto-capture the main user interactions. This feature covers a variety of interaction types like clicks, scrolls, mouse-overs etc. provides better understanding for each interaction and navigation, enabling advanced user behavior analytics.

We’ve also made improvements to our data semantics and auto-enrichments. For instance, we will provide element context on user interactions and view-context on mobile performance data. This enhancement allows for a more straightforward but also in-depth analysis of your data.

If you’re running into property limits, our new event-based capturing and Grail will solve this. You can enrich +100s of properties on events and sessions to capture your individual context. With DQL, custom apps and custom anomaly detection, all this data is at your fingertips. This feature allows for a more personalized analysis of your data.

Finally, if you’re unable to use cookies or don’t have a user session, there’s no need to worry. We are planning to also allow you to collect and analyze RUM data in session-less mode. This ensures that you can still gain valuable insights from your data, regardless of your ability to use cookies or establish user sessions.

Activation

Eligible customers (SaaS, access to Grail, DPS) need to actively decide to opt into event-based capturing & ingest to Grail on a per-frontend-application basis via new settings.

Technical impact of the activation

There’s no impact on the current capturing based on user actions on classic, therefore classic features, data ingest, metrics, dashboards etc. are not affected by its activation.

Web: Once the setting is enabled, additional configuration and capture code will be added to the RUM JavaScript. This will enable sending events to Grail in addition to existing data capture. It requires a minimum version of the RUM JavaScript (minimum version to be determined). Despite the added capture code, the RUM JavaScript will still be smaller with better performance as we also invested in significant code cleanups and optimization. We are planning to provide evidence for this as well.

Mobile: OneAgent for mobile also requires a minimum version (to be determined) but as usual must be shipped together with the app first and downloaded by users before data is stored on Grail.

We are expecting an increased data volume sent from your end users’ browsers & mobile apps to Dynatrace as additional events and properties are captured and are directly ingested into Grail where you can consume it within the latest Dynatrace.

Consumption impact of the activation

There will be no additional charges for activating the initial features delivered in October 2024 (as part of a private preview program). So RUM data ingest to Grail and also querying the data inside the upcoming new DEM apps on Grail will not be charged. However, querying events in Notebooks and Dashboards are subject to events powered by Grail billing and extending data retention beyond the default 35 days on Grail is subject to events powered by Grail billing.

Availability

DEM powered by Grail only applies to customers on SaaS with access to Grail. A DPS contract (GA) is required.

Our preview program has kicked off early November where we hand-picked the customers. This will be followed by a gradual ramp-up of the preview in the following months. We are expecting the next preview wave to kick-off in CQ1/2025.

Any customer interested can either inquire through their account team or can comment here and we will follow-up with the customer individually. Please beware that there are selection criteria and therefore some candidates won't be able to participate in the preview program.

This preview will include capturing user interactions, navigations, requests, web vitals and errors for web applications and requests, an initial set of mobile vitals, errors and crashes with view-context on mobile applications. It will also include previewing some of our DEM core apps offered with the latest Dynatrace.

Supporting material /contact

Please look out for updates on this post / comments from Dynatrace and future blog posts planned (e.g. DEM metrics FAQs). Go ahead and make use of our in-product chat as well as approach your assigned Product Specialist, Customer Success Manager, Customer Success Engineer or Business Insights Analyst if you have individual questions. We will of course update our public documentation once the features get rolled-out.

FAQs

• What will happen to user actions?
  • In classic apps (e.g. Dashboards Classic) and the current DEM features, user actions stay untouched – so zero impact on your metrics, analysis screens and dashboards.
  • On Grail, user actions will no longer exist in this form. The data currently found within user actions will be stored in separate events, so it will be way easier to analyze the information we capture. E.g. user interactions (e.g. clicks, scrolls etc.) are stored as user interaction events and web requests (e.g. document, xhr,…) as web request events. So any event can be analyzed and monitored with high granularity and full context. As an example, a web request includes properties like the page, full-URL, geolocation, response code, start and end-time, duration and more.  
  • To understand overall timings (e.g. of a page load) we are offering web vitals, and users can view resource loads and their timings in Notebooks or in an updated waterfall chart (to be delivered later this year).
  • Of course, we are listening to our customers' needs and use cases and will adapt our plans accordingly.
• How does the data model look like?
  • We will collect RUM events in the user.events table.
    • with details about requests, navigations, errors and user interactions including additional auto-captured properties. Some properties are captured by default and any RUM event will include them. Other properties are captured in addition depending on the type of data collected. E.g. a web request holds the http.response.status_code . On top of this, customers can enrich the events with their custom properties as well or send in custom RUM events.
  • We will generate user session aggregates stored as separate events in the user.sessions table.
    • With details about users, the session durations and more. The properties of a user sessions are based on RUM events captured.
  • For more information take a look at the semantic dictionary in our public documentation, once we rolled out the first version.
• Can I turn off ingest into classic and completely migrate to Grail?
  • No, currently you can’t turn off classic data ingest fur RUM . However, this will be possible in the mid-term.
• How do I know that this new Grail agent module is active on my environment?
  • You can access the global RUM settings or any frontend application specific settings. If you have permission to read the specific setting you can check if it’s enabled or not. The information can be retrieved via the default settings API as well. Also you can access Grail – e.g. via Notebooks and query RUM events. Later, our apps will also give you an overview of your monitoring coverage with the latest Dynatrace.
• What permission is required to turn on this new module?
  • Users who want to activate these modules and ingest to Grail need the permission to change the settings within the frontend applications (global or local. These are typically included in the monitoring admin permissions. From a technical standpoint, it is going to be a Settings 2.0 schema.
  • In order to access RUM data on Grail, users require the appropriate access permissions to the latest Dynatrace, relevant tables and buckets.
• Due to the change of capturing, will there be metrics not available with RUM on Grail?
  • We will continue offering built-in metrics based on current usage of metrics and customer feedback. And we will start rolling these out in parallel and adding them over time. On top of this, customers can create any on-demand timeseries based on the events and their fields. In addition, OpenPipeline will allow them to create custom metrics on Grail.
  • Some metrics are currently not planned to be offered on the new Platform, either due to the data model changes or because they are substituted by another one. A user action duration for instance will not exist as a measurement. We will measure the page performance mainly with Google web vitals and W3C timings. However, specific request durations can be measured on the event and custom metrics created for them. Also, as mentioned above, data and metrics on classic stay untouched, so you can continue using these on e.g. Dashboards Classic.
  • In addition, we are planning to introduce new built-in metrics or improve existing ones.
  • There's a dedicated blog post about FAQs on DEM metric - take a look for further details

[jegron]

Great, thanks for all this details. It helps communicate to customers what's coming.

[mark_forrester]

Wahoo, been waiting for this for what feels like a long time.

The DPS requirement is this DPS LA or DPS GA? We have DPS LA and currently blocked from using the new Kubernetes apps because of this.

[paul_kapeller]

In order to get access, DPS GA is required.

[dimitris_pistio]

Will it be possible to export live session data? The options to export data will continue to be a) API and b) the session export (streaming). What will be the limitations on those 2 export types?

[paul_kapeller]

Events are accessible through DQL immediately after they are stored. Hence, you can also make use of the Grail Query API to retrieve this data after its ingested. The above-mentioned RUM events are accessible immediately. The session aggregate event is stored after the session is finished and can then be fetched/exported thereafter. I'm sure, many of the use cases can be covered with the RUM event querying already.

[Dant3]

Hi, in the new datamodel, is there a plan to capture position of mouse? To allow to create ClickMaps/Heatmaps with that info?

[michalmajewski]

Yes, it will be possible. We have already started an internal project to evaluate the possibilities for this topic. It looks like the first step will be clickmaps visualizing most clickable elements. Heatmaps based on mouse movements will come later.

[Dant3]

I started to see in API and in a few widgets the "DTAQL" is there any open preview to check?

[paul_kapeller]

@Dant3 can you share a link as to where you discovered the API / the widgets mentioned? 

In general, our current plan for RUM is to kick-off a private preview program with hand-picked customers in October, followed by a gradual ramp-up of the preview in the following months. Any customer interested can either inquire through their account team or can comment here and we will follow-up with the customer individually. Please beware that there are selection criteria and therefore some candidates won't be able to participate in the preview program.

[Dant3]

When you migrate Dashboards with USQL showup talking about DTAQL in gen3 dash app, when we search for it we only found this on gen2 dash api. 

Thanks, will reach AE/SE to pass details of customer. 

[paul_kapeller]

Ok this is a different scenario. In your case, you can use the API to query user session data (with USQL). This API can be used within e.g. a Dashboard or Notebooks app. But essentially, it would query the data from user session storage and not from Grail, where we will store the data to in the next months. And where you can then directly use DQL (e.g. fetch user.events) to retrieve DEM data - with all its event fields and granularity and DQL functions available.

[Nick-Montana]

Hi,

"Finally, if you’re unable to use cookies or don’t have a user session, there’s no need to worry. We are planning to also allow you to collect and analyze RUM data in session-less mode. This ensures that you can still gain valuable insights from your data, regardless of your ability to use cookies or establish user sessions."

Interested in knowing more about this. Will OpenKit be receiving an update reflecting these changes? We have an OpenKit Java abstraction that bootstraps many of our organization's Spring services with the tool. We currently instruct users to create/end sessions for data ingest even though they aren't conventional "user sessions". Example is sendBizEvent() method is only obtained after the creation of a Session object

[paul_kapeller]

Hi,

great to hear this resonates with you. I mentioned this future possibility as there are more and more customers struggling to capture user sessions as they e.g. cannot use cookies. In addition, scenarios like the one you mentioned - basically ingesting events with a certain context but w/o a session, should be doable in future as it is also in line with our vision and value proposition of OpenPipeline. With OneAgent or potentially also with frontend Otel collectors.

However, as this is a completely new "space" for us, there's still a lot of research and conceptual work required, before we can actually offer this in our product. Right now, our focus is to cover our core DEM use cases on Grail. Then we will take a look at this field in more detail.

[Nick-Montana]

I should mention that it isn't necessarily a problem! Creating/ending sessions gives the bizevents a unique identifier. I imagine customers struggle with this because its not immediately apparent why the session-based data ingest behaves the way it does. It took me a while to grasp all the ins-and-outs of how OpenKit works

As I mentioned previously, our architecture is powered through SpringBoot and we have many business-critical services that are now using that sendBizEvent() functionality to send data into GraiI. I bet there are a lot of customers who may be in a similar situation of needing to send bizevents from the client-side application.

I feel like the OpenKit documentation could improve by tailoring the information to this audience. Perhaps another page in the public docs that explains the reasoning behind session-based data ingest (e.i session-id and useful session duration metrics) for business-critical backend services

[IanM]

Hi Paul, is there any update on when RUM metric availability in Grail will be GA? I see an October-24 timeframe is mentioned for some availability as an EA.  KR Ian

[paul_kapeller]

We can't share a GA date yet for RUM on Grail. However, next quarter the preview is planned to be broadened, hence a lot of customer can opt into RUM on Grail and the available metrics on Grail. Which use cases would you like to unlock with it?

[sivart_89]

@paul_kapeller we would definitely like to enroll in this, are we able to get this enabled?

[Dripto]

Hi @sivart_89  - thank you for your interest. If you could let your Dynatrace team (particularly the customer success manager) know, they will submit your request with some additional details on your behalf.

[9sagarelluru]

Thanks, Paul, for sharing more details. I know you guys are working on enhancing the features for RUM as part of 3rd gen. At the same, would be possible to create a custom metric that raises an alert when for a particular user action with frustrated score like FRUSTRATED, TOLERATED, SATISFIED. Right now, unfortunately it seems you do not support useraction name as a dimension. I got to that this feature is not there and requesting you to consider this which will be a big enhancement if you bring up this to show the list of useractions impacted with particular apdex score on alert as other tools showing up this. Kindly consider. 

[mark_forrester]

If you are able to query the data and get that result in Grail then you could use the Davis anomaly detector to do just that (until it is possibly added in any future product), I use it for BizEvents on Grail where we see an abnormal amount of failed order pushes information being received.

[paul_kapeller]

I can confirm that this is the way to go on the new platform:

1. Create a custom anomaly detection (query via DQL + maketimeseries command if it's not a built-in or custom timeseries)

2. Create a workflow listening to the anomaly events to notify or create a ticket

In terms of data points, any measurements available on Grail can be used for that. For RUM, e.g. page load timings of specific pages or views (based on page/view summary events), request durations of specific requests, specific error types etc. Currently, there is no categorization of performance in terms of Adpex available.

For classic, we also offer key user actions to possible cover your use case already today @9sagarelluru .