Take the Echoes Lost in Orbit Challenge: Hyperspace Operations [Expert]

[Michal_Gebacki]

Level: Expert

👉 This Challenge, with all of its three levels, created Katharina Sick (@ksick) from the OffOn Community!

This Challenge Level is best suited for:

Platform and DevOps engineers who already know how the delivery pipeline fits together and want to see how a distributed trace ties it into one story. You should be comfortable with Kubernetes, YAML, and reading a tool's logs and UI. Prior exposure to OpenTelemetry, trace context propagation, and a trace viewer like Jaeger helps, but the focus here is using a trace to reason across service boundaries, not any one tool.

Mission objective Automated rollout progression to HotROD version 1.76.0 driven by observability signals. OpenTelemetry Collector configured with an OTLPOpenTelemetry Protocol receiver for HotROD traces, a Spanmetrics connector converting traces to metrics, trace export to Jaeger, and metrics export to Prometheus. Canary analysis with three PromQL queries: traffic detection (at least 0.05 req/s to prevent idle canaries), error rate below 5%, and 95th-percentile latency below 1000ms

Key Learnings Configure OpenTelemetry Collector pipelines Spanmetrics connector (traces to metrics) Detect idle canaries with traffic validation and distributed tracing with Jaeger.

This Challenge’s Level story:

After fixing the Zephyrian communications, word of your progressive release mastery spread across the galaxy. The Bytari, a highly advanced species from the Andromeda sector, were impressed.

They want to apply progressive delivery to their mission-critical service: HotROD (Hyperspace Operations & Transport, Rapid Orbital Dispatch), an interstellar ride-sharing service handling dispatch requests across thousands of star systems. Every millisecond of latency matters, and any error could strand travelers between dimensions.

A previous engineer started instrumenting HotROD with OpenTelemetry and configured Argo Rollouts for automated validation, but left the setup incomplete. The observability pipeline is broken. The Bytari don't use staging/production environments; they believe in single-environment progressive delivery validated purely by trace-derived metrics and automated health checks.

Your mission: fix the observability pipeline and canary validation. Make HotROD deployment-ready with proper distributed tracing.

This Challenge Level’s architecture:

	The observability pipeline is broken and HotROD's canary can't validate. Wire an OpenTelemetry Collector with spanmetrics to convert distributed traces into Prometheus metrics, then write PromQL queries that catch idle canaries, high error rates, and latency spikes.

Walkthrough

1. Get started

Open in GitHub Codespaces.

The devcontainer is pre-configured and starts automatically. When you push from Codespaces, GitHub forks the repository to your account automatically.

Prefer working locally? Clone the repo and open it in any editor that supports the Dev Containers specification (VS CodeVisual Studio Code, JetBrains, and others). The devcontainer config will be detected automatically.

2. Wait for Infrastructure

Wait ~5-10 minutes for infrastructure to deploy. Port forwarding starts automatically after infrastructure is ready, keeping a terminal busy. Open a new terminal to run commands.

3. Explore the UIs

Open the Ports tab and navigate to each service:

• Port 30100: Argo CD (readonly / a-super-secure-password). Shows sync status. Use to refresh applications after pushing commits.
• Port 30101: Argo Rollouts. Shows canary deployment progress and analysis status.
• Port 30102: Prometheus. Explore available metrics and test PromQL queries. CLI tools work equally well if you prefer the terminal.
• Port 30103: Jaeger. Shows distributed traces from HotROD to verify that tracing is working end-to-end.

4. Fix the Manifests

Fix the manifests in adventures/01-echoes-lost-in-orbit/expert/manifests/. Use the Argo Rollouts dashboard, Prometheus UI, and Jaeger UI to debug and validate your changes.

5. Deploy the Changes

Commit and push to trigger the deployment:

git add manifests/
git commit -m "Fix configuration"
git push

If pushing to a branch other than main, also update the ApplicationSet in appset.yaml to point to your branch.

Refresh Argo CD apps:

argocd app get hotrod --refresh
argocd app get otel --refresh

If you changed HotROD, retry the rollout:

kubectl argo rollouts retry rollout hotrod -n hotrod

If you changed the OTelOpenTelemetry Collector config, restart it:

kubectl rollout restart daemonset/collector -n otel

6. Watch the Rollout

Watch rollout progress. The rollout should progress automatically based on analysis metrics:

kubectl argo rollouts get rollout hotrod -n hotrod --watch

7. Run the Smoke Test

Run the smoke test to verify your solution:

make verify

How to complete your challenge?

	When you push from Codespaces, GitHub forks the repository to your account automatically. If you are working locally, fork the repository on GitHub before pushing.
	Verify your solution: make verify
	If it passes, it generates a Certificate of Completion you can paste into the discussion.
	Share your solutions in this thread in answer below and mention your achievement on your LinkedIn account!

Toolbox kubectl - Kubernetes CLICommand Line Interface for interacting with the cluster kubens - fast way to switch between Kubernetes namespaces k9s - terminal UIUser Interface for managing and inspecting your cluster Argo CD CLI - manage Argo CD applications from the command line Argo Rollouts kubectl plugin - extended kubectl commands for managing rollouts

Helpful documentation OpenTelemetry Collector configuration Span Metrics Connector Argo Rollouts analysis PromQL basics

Are you ready? Take the challenge’s mission! Open in Codespaces Free GitHub account required. Launch in a preconfigured devcontainer.

Don’t forget about benefits! For completing at least one level you get a unique badge! For completing all of the three levels you get even more badges! Deadline: 13 October 2026 at 23:59 CET

Other levels of this challenge

Beginner	Broken Echoes
Intermediate	The Silent Canary