This is my step-by-step guide to smarter anomaly detection - its a cross-posting from my LinkedIn Article.
I have access to several observability production environments, and I regularly dig into the data looking for patterns worth sharing. This week I found one that's a perfect case study — not because something was on fire, but because everything looked like it was on fire when it actually wasn't.
What follows is my step-by-step approach to analyzing a failure rate alert that turned out to be pure noise. My goal is twofold: to give you a practical framework for making better alerting decisions, and to show you exactly where common alerting setups break down.
I'd love to hear where you'd push back — because good alerting is as much about debate as it is about dashboards.
Is a sudden spike in failure rate really an anomaly?
I often see SREs define alerts on service request failure rate (= number of failed requests / total number of requests). While this is not wrong in general the question to ask is: Does the failure rate on its own really tell you if you currently have an abnormal behavior or not?
In my exploration I discovered the following failure rate graph showing the last 12 hours of one of their ingress controllers. And it seems that the team has defined a custom alert rule that also triggered the creation of an Incident:
A Failure Rate spike like this clearly looks suspicious. But does the 12 hour timeframe really tell the full story?
Have we seen this behavior in the past?
Having historical data is always beneficial as it allows us to see whether this is really a sudden spike or a recurring pattern. I typically look into various timeframes such as 7 days (do we have a weekly pattern), 30 days (do we have individual days during a month where we see this) as well as 365 days (do we have specific days or months during the year where this happens).
In my exploration I analyzed the last 7 days and its easy to see that this is a daily recurring pattern:
Analyzing 7 days, 30 days, 365 days help us see whether there is some recurrence going on here!
What type of errors are we looking at?
Now that we know its not a single spike but a recurring event the question is: What type of errors we have here that happen once a day? Here its important to understand that not every error is the same. For HTTP requests we can for instance differentiate between your HTTP 5xx (typically some backend server error), HTTP 4xx (typically something to do with client access or authorization issues) or HTTP 3xx (arguable - those are not real errors but tell us about redirects).
In my exploration I found that the 5xx errors were rather flat and I just saw some increase of the 4xx (client authorization issues). Those however didn't really spike - looks more like a bump in the road:
Its always important to understand what results into a failure. For web requests its good practice to analyze your HTTP Status Codes
What impacts the Failure Rate Spike if we only see bumps in errors?
Failure Rate gets calculated by looking at Failed Requests / Total Requests. So - that number gets impacted during different type of load behavior and also gets impacted by how you quality what an error is (e.g: should an HTTP 403 - Access Forbidden be considered an error or its just a user mistake).
In my exploration I looked at the throughput which shows that we have a very obvious traffic pattern which shows higher traffic during the day, low during the night and overall only about half the traffic over the weekend. And if we put all those pieces together we can see that the failure spike is not an anomaly - and - its also not a problem on its own as it gets impacted by traffic and the way we define what an error is:
Its important to analyze the relationship between metrics to understand the full context to make good conclusions!
How to improve simply failure rate spike alerting?
There are two things that can help improve alerting in this scenario.
1: Clearly define what is an error that needs your attention (HTTP 5xx) and what is just noise (some of the HTTP 4xx)
Getting an overview of all HTTP Status Codes per Endpoint is a good way to start to see which error codes are really important or not. This allows us to fine tune the failure rate calculation.
We can do a detailed analysis of every HTTP Status code and see which ones should really be considered an error / failure
Also keep in mind that besides looking at HTTP Status codes you might also want to look into Exceptions and consider requests with critical exceptions as failures even though they may not result in an erroneous HTTP Status Code.
Exceptions are good indicators for critical errors – but – they are not always propagated back to the end user as an error.
Another interesting finding was that all those 503 requests actually came from the same source - which was the AWS Route53 Health Check Service. We can easily identify this by grouping distributed traces by HTTP Status and User Agent.
Closer analysis reveals that >90% of those HTTP 503 were caused by the Route53 Health Check Service
We should also follow up here and exclude those from the regular alerting definition and also follow up with AWS to see why those checks are made and how they can end up making that many calls that lead to errors. In the User Agent String there is actually a URL encoded that allows someone to report unexpected Health Checks!
2: Apply seasonal baselines across longer timeframes
Seasonal baselining with confidence corridor for our failure rate metric. It wouldn't alert based on our historical data
Using the baseline and a better failure definition will lead to fewer alerts which will make the life of human an AI SRE responders much simpler! You can learn more about baselining here in the Dynatrace doc.
What other findings did we have?
I will not leave you with my additional things I found.
What's the root cause of those HTTP 500s?
First I wanted to know what is causing those HTTP 500 from the "real users" (excluding the AWS Route Health Checks). Looking at some of the traces I could spot problems with backend GraphQL queries that timed out. The traces show the actual API Call to fetch data from a backend service that closes the TCP connection after 10s and therefore results in the HTTP 500 to the end user! What is really strange is that the GraphQL query on the server-side continues after about 30s. This is clearly an issue to look into!
Distributed Traces show us the real root cause of some of those HTTP 500s - like GraphQL Query Timeouts
What about all those HTTP 4xx?
Turns out that most of those have to do that the end user was providing wrong information about documents they wanted to see or upload. All distributed traces showed internal 404 Not Found errors in the document store API.
What about performance?
In my years as Performance Engineer I typically also look into performance related anomalies. And here we could clearly see "a winner". Same service, same timeframe: response time anomaly where P99 (99th Percentile) spiked with 30s:
Response Time typically gets analyzed by looking at the various percentiles (P50, P90, P99)
Analyzing those transactions turned into seeing that some of the downstream services slowed down significantly, leaving the caller with lots of blocked threads which ultimately got aborted when they ran into the 30s timeout.
Dependencies that slow down or fail will have a cascading effect on callers such as blocking threads which will also impact other requests that need those threads
I am sure there would be much more I could find - but - I hope this gives you some insights into how I analyze data and to what conclusions I come
Lessons learned: This only scales through automation!
Coming back to the opening of this blog post. Ingesting observability data is just one step. The question is then: on what do I really need to get alerted on and how do I then identify the root cause.
With Dynatrace we are fortunate that all those steps that I just showed you work well when you explore your data. But - this doesn't scale! Hence Dynatrace has always automatically detected anomalies by applying various different means (custom, auto-adaptive or seasonal baselining) and is then doing automated root cause analysis by looking at all the connected data based on our topology and dependency model.
I leave you with the following Dynatrace problem ticket that you can also explore yourself on the Dynatrace Playground. (don't have access to playground - just sign up here). It really shows how Dynatrace detect abnormal behavior, provides root cause insights and even provides suggestions about which CI/CD deployment or Pull Request most likely caused the issue. And of course - these insights can also be used by your human and AI SREs to speed up mitigation of issues 🙂
Dynatrace Intelligence automates the detection of anomalies, root cause and can also automate remediation
If you have read that far it would be great to get your reaction. Like it, repost it or leave a comment if you have a different opinion on this!
