Observe All the Things

[AdrianaVillela]

When we speak about observability, we often associate it with things like applications and infrastructure. But what if I told you that there is an entire world out there beyond our tech bubble that could benefit from the power of observability? Would you believe me? 

To explore this further, let’s consider this: our daily lives are made up of workflows. A workflow is a series of related tasks strung together to accomplish a certain goal. For example: 

• Driving your kid(s) to school 

• Commuting to and from work 

• Getting ready for work in the morning 

• Cooking an evening meal 

These are examples of personal, or individual workflows that are unique to you. 

There are also multi-person workflow situations involving handoff of tasks. For example: 

• Applying for a job 

• Onboarding a new employee at a company 

• Responding to natural disasters caused by climate change (e.g. forest fires, floods) 

• Going to the emergency room (ER) 

As an observability geek, these are the types of workflows that most interest me. Why? Because observability can help us gain deeper insights into our workflows. By understanding what’s truly happening, we can pinpoint and eliminate bottlenecks, fine-tune processes, and enhance overall efficiency. That is, it enables us to take meaningful action based on what we’ve learned. 

Show me one person who hasn’t complained about ER wait times or who hasn’t complained about getting ghosted when applying for a job. Yeah. Exactly. We need this. 

So, this begs the question…how can we extend observability beyond our little tech bubble? Luckily for us, we already have existing tooling at our disposal. We have: 

• The OpenTelemetry (OTel) ecosystem. It gives us a standard for generating, collecting, and exporting telemetry data 

• Observability backends. These help us interpret and the OTel data so that we can make improvements 

To understand this better, let’s look at an example. 

Going to the emergency room 

In Canada, where I live, we are fortunate to have government-provided healthcare; however, we’re certainly not immune to things like long ER wait times. Imagine if we could use observability to improve ER workflows? 

Using OpenTelemetry, we could use traces to capture the end-to-end ER workflow. Spans would represent each step in the workflow: 

• intake/registration 

• triage 

• sitting in the waiting room 

• being brought in to an exam area to be seen by a doctor 

• waiting in the exam to be seen by a doctor 

• speaking to the doctor 

• getting tests 

• waiting for tests 

• speaking to the doctor about test results 

• being discharged 

We can add attributes to further enrich our spans. For example: 

• name of triage nurse 

• name of examining physician 

• medications prescribed 

• tests and/or imaging ordered 

We can add logs to capture more detailed information, and correlate them to our spans: 

• triage nurse’s notes 

• doctor’s exam notes 

• discharge notes 

• patient feedback 

We can capture metrics. Span duration can be expressed as a metric. This can help us identify bottlenecks in our workflow. Metrics can also provide additional quantitative data about our workflow. For example: 

• Average time spent in the waiting room by severity of case 

• Average time spent in the waiting room by time of day, day of the week, time of the year (e.g. holidays vs. non-holidays) 

• Average nurse and physician shift time 

• Number of patients per case type (e.g. heart attacks, strokes, car accidents, animal bites) 

• Number of patients by age group 

We can then send the OpenTelemetry data to an observability backend. This allows us to analyze and interpret data, and identify areas of improvement, so that decision makers can implement these improvements. 

Final Thoughts 

As we saw today, we already have standards and tools for gathering telemetry, so why not use them? OpenTelemetry can help us instrument our workflows, and existing observability backends can help us make sense of the data, allowing us to gain better insights into these workflows, enabling us to improve them. 

Before we part ways, I will leave you with a quote from Hazel Weakly: 

“In a world where every team is expected to bring value to the company, in a world where we need to understand systems of ever increasing complexity, why would we POSSIBLY think that we can get away with siloing away our knowledge and understanding our observability systems inside our own companies?” 

Appendix 

This blog post is based on a talk that I gave at Observability Day EU 2025. You can watch the recording here.