‎20 Jun 2024 01:17 PM - last edited on ‎20 Nov 2024 09:29 AM by GosiaMurawska
Digital Experience Monitoring (DEM) has become front and center within the observability space. That’s why customers have consistently increased demands to gain better visibility and analytics possibilities of their end users’ experiences. At the same time, the landscape of web and mobile technologies is rapidly evolving and Google’s core web vitals and new mobile vitals have shaped the area significantly.
As a leader in observability, Dynatrace is going to launch a significative upgrade to its current solution for modern frontend monitoring. DEM powered by Grail will allow users to gain even deeper insights, further optimize user experiences across devices and drive better business outcomes.
With this post, we’d like to inform you in advance about upcoming benefits and expected impacts, focusing on RUM and our new data model. We’re informing about new apps and additional DEM enhancements in separate posts.
I’m sure you’re excited about what’s to come and at the same time, you can already prepare your organizations for the changes and actions required and ask any questions that come to mind when reading this. We will try our best to respond to them in a timely manner. The post including the FAQ section will be updated based on your feedback and questions.
Grail and the latest Dynatrace
We have seen how logs, business events, metrics and traces have benefited from being on Grail in the past 18 months. Starting in October (current plan for early adoption), your frontend applications will also be able to reap similar benefits. Existing as well as new use cases will be enriched with more data granularity, advanced querying capabilities and extended retention times on the new Dynatrace platform.
New RUM data model and Grail
The core of the improvement is a new, future-proof data model based on RUM and session events; which will subsequently be captured via a new OneAgent module, sent via an enhanced beacon protocol, ingested via OpenPipeline and stored into our new Platform and Grail. An event-based capturing approach allows us to unlock significant improvements in monitoring coverage as well as data analytics and AI. Events will enhance and elevate the value customers can gain compared to user actions today.
DEM powered by Grail offers a range of features designed to enhance your data analysis capabilities. One of the key features is extended retention of RUM data. This allows you to extend your individual retention time up to 10 years, providing a long-term view of your data trends.
In addition, we auto-capture all requests made by your frontend applications, including their W3C timings. These requests can then be monitored, analyzed and alerted on individually with the latest Dynatrace, offering a granular view of your frontend applications’ performance.
To help you better understand your end-users’ behavior, our new agent module can also auto-capture the main user interactions. This feature covers a variety of interaction types like clicks, scrolls, mouse-overs etc. provides better understanding for each interaction and navigation, enabling advanced user behavior analytics.
We’ve also made improvements to our data semantics and auto-enrichments. For instance, we will provide element context on user interactions and view-context on mobile performance data. This enhancement allows for a more straightforward but also in-depth analysis of your data.
If you’re running into property limits, our new event-based capturing and Grail will solve this. You can enrich +100s of properties on events and sessions to capture your individual context. With DQL, custom apps and custom anomaly detection, all this data is at your fingertips. This feature allows for a more personalized analysis of your data.
Finally, if you’re unable to use cookies or don’t have a user session, there’s no need to worry. We are planning to also allow you to collect and analyze RUM data in session-less mode. This ensures that you can still gain valuable insights from your data, regardless of your ability to use cookies or establish user sessions.
Eligible customers (SaaS, access to Grail, DPS) need to actively decide to opt into event-based capturing & ingest to Grail on a per-frontend-application basis via new settings.
Technical impact of the activation
There’s no impact on the current capturing based on user actions on classic, therefore classic features, data ingest, metrics, dashboards etc. are not affected by its activation.
Web: Once the setting is enabled, additional configuration and capture code will be added to the RUM JavaScript. This will enable sending events to Grail in addition to existing data capture. It requires a minimum version of the RUM JavaScript (minimum version to be determined). Despite the added capture code, the RUM JavaScript will still be smaller with better performance as we also invested in significant code cleanups and optimization. We are planning to provide evidence for this as well.
Mobile: OneAgent for mobile also requires a minimum version (to be determined) but as usual must be shipped together with the app first and downloaded by users before data is stored on Grail.
We are expecting an increased data volume sent from your end users’ browsers & mobile apps to Dynatrace as additional events and properties are captured and are directly ingested into Grail where you can consume it within the latest Dynatrace.
Consumption impact of the activation
There will be no additional charges for activating the initial features delivered in October 2024 (as part of a private preview program). So RUM data ingest to Grail and also querying the data inside the upcoming new DEM apps on Grail will not be charged. However, querying events in Notebooks and Dashboards are subject to events powered by Grail billing and extending data retention beyond the default 35 days on Grail is subject to events powered by Grail billing.
DEM powered by Grail only applies to customers on SaaS with access to Grail. A DPS contract (GA) is required.
Our preview program has kicked off early November where we hand-picked the customers. This will be followed by a gradual ramp-up of the preview in the following months. We are expecting the next preview wave to kick-off in CQ1/2025.
Any customer interested can either inquire through their account team or can comment here and we will follow-up with the customer individually. Please beware that there are selection criteria and therefore some candidates won't be able to participate in the preview program.
This preview will include capturing user interactions, navigations, requests, web vitals and errors for web applications and requests, an initial set of mobile vitals, errors and crashes with view-context on mobile applications. It will also include previewing some of our DEM core apps offered with the latest Dynatrace.
Supporting material /contact
Please look out for updates on this post / comments from Dynatrace and future blog posts planned (e.g. DEM metrics FAQs). Go ahead and make use of our in-product chat as well as approach your assigned Product Specialist, Customer Success Manager, Customer Success Engineer or Business Insights Analyst if you have individual questions. We will of course update our public documentation once the features get rolled-out.
FAQs
‎20 Jun 2024 04:34 PM - edited ‎20 Jun 2024 04:35 PM
Wahoo, been waiting for this for what feels like a long time.
The DPS requirement is this DPS LA or DPS GA? We have DPS LA and currently blocked from using the new Kubernetes apps because of this.
‎21 Jun 2024 09:58 AM - edited ‎21 Jun 2024 11:04 AM
In order to get access, DPS GA is required.
‎21 Jun 2024 06:36 AM
Will it be possible to export live session data? The options to export data will continue to be a) API and b) the session export (streaming). What will be the limitations on those 2 export types?
‎21 Jun 2024 11:26 AM
Events are accessible through DQL immediately after they are stored. Hence, you can also make use of the Grail Query API to retrieve this data after its ingested. The above-mentioned RUM events are accessible immediately. The session aggregate event is stored after the session is finished and can then be fetched/exported thereafter. I'm sure, many of the use cases can be covered with the RUM event querying already.
‎25 Jun 2024 06:55 PM
Hi, in the new datamodel, is there a plan to capture position of mouse? To allow to create ClickMaps/Heatmaps with that info?
‎27 Jun 2024 12:28 PM
Yes, it will be possible. We have already started an internal project to evaluate the possibilities for this topic. It looks like the first step will be clickmaps visualizing most clickable elements. Heatmaps based on mouse movements will come later.
‎14 Aug 2024 03:20 PM - edited ‎14 Aug 2024 03:20 PM
I started to see in API and in a few widgets the "DTAQL" is there any open preview to check?
‎19 Aug 2024 07:43 AM
@Dant3 can you share a link as to where you discovered the API / the widgets mentioned?
In general, our current plan for RUM is to kick-off a private preview program with hand-picked customers in October, followed by a gradual ramp-up of the preview in the following months. Any customer interested can either inquire through their account team or can comment here and we will follow-up with the customer individually. Please beware that there are selection criteria and therefore some candidates won't be able to participate in the preview program.
‎19 Aug 2024 01:24 PM
When you migrate Dashboards with USQL showup talking about DTAQL in gen3 dash app, when we search for it we only found this on gen2 dash api.
Thanks, will reach AE/SE to pass details of customer.
‎23 Aug 2024 11:02 AM - edited ‎23 Aug 2024 11:02 AM
Ok this is a different scenario. In your case, you can use the API to query user session data (with USQL). This API can be used within e.g. a Dashboard or Notebooks app. But essentially, it would query the data from user session storage and not from Grail, where we will store the data to in the next months. And where you can then directly use DQL (e.g. fetch user.events) to retrieve DEM data - with all its event fields and granularity and DQL functions available.
‎16 Oct 2024 12:54 PM
Hi,
"Finally, if you’re unable to use cookies or don’t have a user session, there’s no need to worry. We are planning to also allow you to collect and analyze RUM data in session-less mode. This ensures that you can still gain valuable insights from your data, regardless of your ability to use cookies or establish user sessions."
Interested in knowing more about this. Will OpenKit be receiving an update reflecting these changes? We have an OpenKit Java abstraction that bootstraps many of our organization's Spring services with the tool. We currently instruct users to create/end sessions for data ingest even though they aren't conventional "user sessions". Example is sendBizEvent() method is only obtained after the creation of a Session object
‎16 Oct 2024 01:18 PM
Hi,
great to hear this resonates with you. I mentioned this future possibility as there are more and more customers struggling to capture user sessions as they e.g. cannot use cookies. In addition, scenarios like the one you mentioned - basically ingesting events with a certain context but w/o a session, should be doable in future as it is also in line with our vision and value proposition of OpenPipeline. With OneAgent or potentially also with frontend Otel collectors.
However, as this is a completely new "space" for us, there's still a lot of research and conceptual work required, before we can actually offer this in our product. Right now, our focus is to cover our core DEM use cases on Grail. Then we will take a look at this field in more detail.
‎16 Oct 2024 02:04 PM
I should mention that it isn't necessarily a problem! Creating/ending sessions gives the bizevents a unique identifier. I imagine customers struggle with this because its not immediately apparent why the session-based data ingest behaves the way it does. It took me a while to grasp all the ins-and-outs of how OpenKit works
As I mentioned previously, our architecture is powered through SpringBoot and we have many business-critical services that are now using that sendBizEvent() functionality to send data into GraiI. I bet there are a lot of customers who may be in a similar situation of needing to send bizevents from the client-side application.
I feel like the OpenKit documentation could improve by tailoring the information to this audience. Perhaps another page in the public docs that explains the reasoning behind session-based data ingest (e.i session-id and useful session duration metrics) for business-critical backend services
‎21 Nov 2024 11:44 AM - edited ‎21 Nov 2024 11:44 AM
We can't share a GA date yet for RUM on Grail. However, next quarter the preview is planned to be broadened, hence a lot of customer can opt into RUM on Grail and the available metrics on Grail. Which use cases would you like to unlock with it?
‎21 Nov 2024 04:16 PM
@paul_kapeller we would definitely like to enroll in this, are we able to get this enabled?
‎21 Nov 2024 04:29 PM
Hi @sivart_89 - thank you for your interest. If you could let your Dynatrace team (particularly the customer success manager) know, they will submit your request with some additional details on your behalf.
‎25 Nov 2024 05:11 PM
Thanks, Paul, for sharing more details. I know you guys are working on enhancing the features for RUM as part of 3rd gen. At the same, would be possible to create a custom metric that raises an alert when for a particular user action with frustrated score like FRUSTRATED, TOLERATED, SATISFIED. Right now, unfortunately it seems you do not support useraction name as a dimension. I got to that this feature is not there and requesting you to consider this which will be a big enhancement if you bring up this to show the list of useractions impacted with particular apdex score on alert as other tools showing up this. Kindly consider.
‎25 Nov 2024 05:57 PM
If you are able to query the data and get that result in Grail then you could use the Davis anomaly detector to do just that (until it is possibly added in any future product), I use it for BizEvents on Grail where we see an abnormal amount of failed order pushes information being received.
‎26 Nov 2024 09:02 AM
I can confirm that this is the way to go on the new platform:
1. Create a custom anomaly detection (query via DQL + maketimeseries command if it's not a built-in or custom timeseries)
2. Create a workflow listening to the anomaly events to notify or create a ticket
In terms of data points, any measurements available on Grail can be used for that. For RUM, e.g. page load timings of specific pages or views (based on page/view summary events), request durations of specific requests, specific error types etc. Currently, there is no categorization of performance in terms of Adpex available.
For classic, we also offer key user actions to possible cover your use case already today @9sagarelluru .