cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Grail Aggregation Function Percentiles

GregOReilly
Helper

Hello!

I have found that there are no PERCENTILES as aggregation functions in GRAIL. We have min, max, sum, avg, count and countIf.

GregOReilly_0-1708019330159.png

Any plans to add percentile? Some of my customers only work with percentiles and the previous dashboards could support a percentile. 

10 REPLIES 10

JoseRomero
Dynatrace Advisor
Dynatrace Advisor

Hello,

I understand that, as of now, direct support for retrieving the 90th percentile timeseries data might not be available. However, I remember plans are underway to introduce this functionality. In the meantime, you can use JavaScript along with APIs to query metrics, specifically targeting the 90th percentile. Below, I've shared my optimized JavaScript code for efficiently calculating the 90th percentile response time.

import { metricsClient } from "@dynatrace-sdk/client-classic-environment-v2";

/**
 * Fetches metric data based on a provided query and time range.
 * @param {string} query - The metric query selector.
 * @param {string} from - The start time for the query in ISO format.
 * @param {string} to - The end time for the query in ISO format.
 * @returns The queried metric data.
 */
async function fetchMetrics(query: string, from: string, to: string) {
  const response = await metricsClient.query({
    acceptType: 'application/json; charset=utf-8', 
    from,
    to,
    metricSelector: query
  });
  return response.result;
}

/**
 * Main function to calculate and return the 90th percentile response time for a specific service.
 * @returns {Promise<{response time percentil 90: number}>} - The 90th percentile response time in seconds.
 */
export default async function() {
  const from = new Date('2023-11-01T00:00:00Z');
  const to = new Date('2024-02-20T00:00:00Z');
  const serviceId = "SERVICE-XXXXXXXXXXXX";

  // Prepare time range for the query.
  const fromRequest = from.toISOString().slice(0, 19);
  const toRequest = to.toISOString().slice(0, 19);

  // Construct query for the 90th percentile of response time.
  const responseTimeQuery = `builtin:service.response.time:filter(and(or(in("dt.entity.service",entitySelector("type(service),entityId(~"${serviceId}~")"))))):splitBy():fold(percentile(90))`;
  const responseMetrics = await fetchMetrics(responseTimeQuery, fromRequest, toRequest);
  
  // Extract the 90th percentile response time in seconds.
  const responseTimePercentile90 = responseMetrics[0].data[0].values[0] / 1000;
  
  return { "response time percentil 90": responseTimePercentile90 };
}

JoseRomero_0-1708473881726.png

 

Percentiles are important for any customer or user, because averages really only skim the surface of what is happening. The problem with this solution(not just becaause its javascript, not everyone knows it) is that it is service-based... I need something that does the same for metrics obtained from logs... to be specific,  timeseries data parsed out from fetched log data. Or are you saying, no need to cut metrics, use this approach using raw log data?

GregOReilly_0-1708505091683.png

if I understand correctly, this code simply does an old school data fetch from dynatrace and (potentially) displays it. It doesnt actually implement a percentile based data-crunch.

 

asharpe
Visitor

I too am in need of `percentile` in `makeTimeseries` command, and after reading https://docs.dynatrace.com/docs/platform/davis-ai/basics/percentiles-for-analyzing-performance I was surprised there weren't more aggregation functions available!  https://docs.dynatrace.com/docs/platform/grail/dynatrace-query-language/functions/aggregation-functi...

Can we vote somewhere to request this?  Is it a technical limitation there aren't more aggregation commands available in `makeTimeseries`?

It would also be good to label this with "aggregation", "makeTimeseries", etc.

Edit: in October 2023 Fabrice said "we're going to have percentile very soon", WRT to the grail timeseries command - https://community.dynatrace.com/t5/Videos/A-Practical-Guide-to-DQL-for-Metrics/td-p/225776

asharpe
Visitor

I'm using this approximation, though the accuracy is sensitive to the interval in the bin() call and that calculated by the makeTimeseries() call.  You also seem to need the field as a number, not a string.

 

fetch logs
| summarize _90 = percentile(field, 90), by: { timestamp = bin(timestamp, 1m) }
| makeTimeseries avg(_90)

 

I'm not sure how to get the time span from the dashboard into a query so if you use this in a dash, you'll need to remember that the accuracy might be quite poor depending on the intervals.

GregOReilly
Helper

Today we have noticed that the documentation page was updated (this is cool, but annoying as we dont know what updated). In this case we can see percentiles have appeared:

GregOReilly_0-1710520212269.png

 

 

However, we tried this in our env....we hit a bug

GregOReilly_1-1710520245347.png

 

This is promising!

I feel like it's equal chance a bug in the docs or a bug in the code.  The docs first say that percentile() is available as an aggregation function, but the immediately following bullet list doesn't include it.  The next portion says that percentile is a parameter, but doesn't say to which function (it's not listed in any of the function synopsis).

This leads me to think the percentile parameter is only available on the percentile function, but I haven't tried yet.

Thanks for the heads up, I'll give it a go shortly!

selva
Visitor

Yesterday i used the percentiles and its working. Please check now.

GregOReilly
Helper

Great news, just tried and its working for me now too! 

GregOReilly_0-1713257802019.png

 

nickvogt
Dynatrace Enthusiast
Dynatrace Enthusiast

For any late-comers to the thread, you can read up on timeseries percentile in the Dynatrace Docs.

elodie_lam
Dynatrace Organizer
Dynatrace Organizer

I tried the example in the Dynatrace Docs: https://docs.dynatrace.com/docs/shortlink/aggregation-functions#percentile. Percentile 100 shouldn't be 3201 ?

elodie_lam_0-1729511660312.png

 

Featured Posts