- last edited on by
Michal_Gebacki
This is off-topic, but a quick overview of what I have "observed" today in my country, Portugal, which along with Spain, had a major electricity blackout today. Major means no electricity, and in the capital Lisbon, it only got restored more than 9 hours after!
If you want a major chaos engineering experiment, we've had it today. Around 10:33 UTC, all electricity from the power grid was suddenly gone. The cause seems to not yet be known, but the Spanish Prime Minister has said that during 5 seconds, 60% of the electricity in the grid "disappeared" (link). A lot will be known about this in the following days.
In the first few minutes after 11:33 local time I confirmed it was national. Pretty fast too, I had confirmation Spain was also affected. So I knew immediately a Black Start was going to be needed. These things take a lot of time, but fortunately an important drill had been done some years ago (link). So it was time for all in our Company to go home, as we don't have backup generators.
Things worked good at first. Mobile connectivity was good, and with proper energy management on my laptop, and hotspot from my phone, I made it till the end of the afternoon. But then mobile connectivity started failing, and that went down too! So what can you do when this also happens? Wardriving some 20 years ago was a thing, and so I had to figure out where. Mobile cell batteries were not gone, as phone calls were possible, but operators must have started preferring voice over net. It turns out that my second idea was a good one: those old Telco Central Office facilities with good energy generation had perfect connectivity. Along the way to Lisbon's second most important one, I also figured out that your car might be the best generator at hand for charging your phone! I'm also the proud owner of a car DC vehicle power inverter, but didn't have to use it this time. Prepping is important when you have to deal with such situations, but I'll leave it for my personal blog.
Well, after this introduction, I'm going to share some cool graphs/ideas of what I have seen. This is very personal, and I believe a lot of experiences will surface in the next few days, in both Portugal and Spain, with the people that were in the IT trenches.
- Globally, despite some problems here and there, mainly with transportation, services were mainly up&running. For all the clients with Dynatrace that I follow, things were quite smooth.
- Some issues did arise after several hours. Proactively, and with projections that this would take all night, some companies started stopping non critical servers. This seems like a good strategy to me, as there was no great way of knowing if this would be lasting even longer than the night. And workers had to go home..
- Users were hit harder. In today's world, where a lot work remotely, chances of using fixed network technology was almost certainly gone. Almost all RUM graphs I have seen are like this (yy axis excluded for data protection):
- Traffic went down quite fast at 11:33, but not all. So servers/applications are good, but users are gone. And at the end of the afternoon, (what I suppose are the difficulties I also sensed) much higher times started appearing, with almost all users gone.
- With Dynatrace, you can easily compare this with a previous timeframe
- And yes, Davis was quick on the diagnosis. And lot's of "Unexpected low traffic", although not as much as I initially felt would happen. This seems to have happened because Easter Monday, last week, is typically a day off for a lot of Portuguese.
- Carbon Impact also shows what happened from another perspective. The energy consumption also went down. I'll have to dig deeper on this, as I believe (in at least this case) that the reason was not driven by servers being down, but less activity, or at least I believe.
There's a lot more I have seen that I cannot openly talk about, but certainly Observability was very important today. As this day finishes, lot's of additional ideas come to mind. Maybe, I'll come back with more.
Last, but not least, a great Thank You for all involved in the electricity sector. People have no idea what it takes to do a Black Start, as it takes decades of experience to do it right. I was in the middle of the city, and when the lights came back, people were clapping and celebrating. Sometimes, you only truly value something, when it's taken away, even if only temporarily...
I experienced it first hand. It was on the client side, as the few that still had Internet access (like me) were in probably one of the following situations:
- Low battery power, so it takes the browser more time to process
- Mobile operators started having difficulties, and even rate-limiting bandwidth
This can be easily seen in the waterfalls. Notice the long First Byte and download times. The quick objects are 1ms, served through cache (importance of caching!). Synthetics also showed sites downloading fast, all through the event 🙂
24 hours after, I just can't get enough of what me & ChatGPT have been discussing about what would be a good analogy of what a Black Start is for the Energy teams, seen from an IT perspective. This is what we came up with 😂:
Featured Posts
