One of my most successful stories was also one of the most intriguing ones we’ve faced so far, as it very much a complete mistery at first… allow me to elaborate! 
This Client was a huge online retailer, so we were very actively trying to deliver the most valuable insights on user experience, usability, engagement, etc. Obviously, we were also covering the entire lenght of the technology stack, and were making sure everything was running smoothly, but at the point when this story took place, we’ve got that part well covered. The project’s focus then was really to make sure the website was seeing lots of costumers: happy and paying costumers!
:money_mouth_face:
One afternoon, panic settles in: “we’re seeing a huge drop in user actions”! When I say a huge drop, let’s say roughly one third of multi-thousands of actions per minute, gone! Out of thin air. 😱 🚨
At first we were very alarmed, we even checked if everything was alright on the implementation side, if all RUM was being collected, if there was not a change in sample rate… and nothing: everything A-ok! 
Everything on the backend side was also 100% green: databases, application servers, network appliances, firewalls, load balancers, everything!
I mean, we were already covering every inch of their stack, what else could it be causing this issue? 🤔
Even worse: as this was a public website, we tested it out on our own devices and, again, everything was working perfectly! 
Then, only a couple of minutes after realizing there was an issue, a great insight from our dear friend Davis: the unexpected low load in user actions was not affecting every customer… all actions that disappeared were from a specific ISP. 📉
Around that same time, the first synthetic monitor failed: surprise surprise, it was one running on a server connected to the Internet via that same ISP! 🤖
That couldn’t be a coincidence, right??
That’s when we contacted someone who was a client of that specific ISP and also, no access! He couldn’t reach the website. 🚧
So, we figured we only had one thing to do: let’s talk with the ISP, and see what is going on.
At first, we were struggling to get the message to the right teams, and all we got were questions like “are you sure the website is online?” or “are you sure the users are connected to the internet?”. 😐
Obviously, we had to give the domain names and IP addresses of the website (that the users were supposed to access… but couldn’t).
After a few interactions, a manager from that ISP was included in the email thread and wrote the most astonishing email I have received in my inbox:
“The IP addresses you provided were blocked by court order. All complaints regarding that matter must be addressed to <MAJOR STREAMING ENTITY>, who required the block”. 😵
We were shocked! Why did that entity required a block on our client’s IP address?
Well, to be honest, this surprise only lasted for a brief moment… because, right after that, a click in our heads: they’re in the cloud!!! ☁️
The IP addresses we provided were not reserved to our clients, and were being used by many of that cloud provider’s clients.
After a few more interactions, we realized that some other client of that cloud provider was using one of his cloud resources to stream some of that major streaming entity’s content, without permission… and so, the entity required the court to take action, and it did: the ISP should cut access to the infringing IP! What the court did not realize, nor the ISP, was that that IP address was being used by other legit entities (like our client), who were caught in the backfire…
Fortunately, after a few more interactions, the ISP removed the block and went with a different strategy to abide by the court rule, without harming any innocent websites 
I can not imagine how much time would this all take to solve without Dynatrace!
- In a few minutes, we pinpointed the source of the issue to a specific ISP, backed by two independent analisys: RUM and Synthetics!
- This allowed the costumer to achieve a minimum amount of downtime (only for a portion of their clients)! This is where the success really is on this story!

- It was a great example of the added value of Dynatrace’s data in context! We had all these inputs, but it was Davis’ tips and all data on the same platform that allowed us to conclude with confidence that our client had did nothing wrong: we were covering it all, we had the data, everything was green!

- We were able to help the customer find the affected users (as the RUM sessions were being tagged), so that they can help those costumers - after the issue was resolved - in buying the stuff they needed, engaging with the website experience and gaining back the confidence in the service that could have been impacted!
