topic Re: API rate limiting for automatied SRG validation? in Automations

API rate limiting for automatied SRG validation?

r_weber — Fri, 05 Apr 2024 10:12:15 GMT

Has anyone encountered this?

I have four (similar) site reliability guardians, each guardian has 18 objectives, all based on DQL queries analyzing traces.
All four guardians are executed periodically by a single workflow (in parallel):

The workflow often fails because it is hitting an API rate limit (apparently too many DQL queries are made in a short time).

This is a problem because the Site Reliability Guardian becomes unreliable 🙂
I thought of creating one workflow per guardian, but that is associated with additional cost (every workflow costs!) and it would duplicate the actions that I'd like to take after the SRG validation steps.

So maybe that API rate limit should be increased for workflows?

Re: API rate limiting for automatied SRG validation?

ChadTurner — Wed, 26 Feb 2025 20:47:34 GMT

@r_weber - have you reached out to the Account Rep in terms of seeing about getting your limit increased? If so, where they able to raise it for you?

Re: API rate limiting for automatied SRG validation?

christian_kreuz — Fri, 21 Mar 2025 07:34:57 GMT

When having a lot of Guardians and evaluations, or generally a lot of load on Grail, SRG used to run into this problem.

With SRG 1.17.0, the team has "Increased Stability and Reliability in Heavy-Load Environments". Please give it a try!