Take the Echoes Lost in Orbit Challenge: The Silent Canary [Intermediate]

[Michal_Gebacki]

Level: Intermediate

👉 This Challenge, with all of its three levels, created Katharina Sick (@ksick) from the OffOn Community!

This Challenge Level is best suited for:

Platform and DevOpsDevelopment and Operations engineers who have met these tools before and want to see how they fit together. You should be comfortable with Kubernetes, YAML, and reading a tool's logs and UIUser Interface. Prior exposure to Backstage, Gitea, and the Argo projects helps, but the focus here is the integration between them, not any one tool.

Mission objective Pod info version 6.9.3 deployed successfully in both staging and production environments Rollouts automatically progress through canary stages based on health metrics Two working PromQL queries in the AnalysisTemplate that validate application health during releases. Eventually - all rollouts complete successfully

Key Learnings Progressive delivery with Argo Rollouts Canary deployments & automated analysis Write PromQL queries for health validation Kube-state-metrics for deployment decisions

This Challenge’s Level story:

After fixing the communication outage in Level 1, the Intergalactic Union welcomed a new species: the Zephyrians.

The communications team attempted to deploy their language files using a progressive delivery system, but the rollout is failing. The Zephyrians are still waiting to communicate with the rest of the galaxy.

A previous engineer configured automated canary deployments with health checks but left the setup incomplete.

Your mission: debug the broken rollout and bring the Zephyrians' voices online.

This Challenge Level’s architecture:

	A canary rollout is stuck and the Zephyrians are still waiting to communicate. Debug the broken progressive delivery system by writing PromQL health checks that let Argo Rollouts automatically validate and advance the deployment.

Walkthrough

1. Get started

Open in GitHub Codespaces. The devcontainer is pre-configured and starts automatically. When you push from Codespaces, GitHub forks the repository to your account automatically.

Prefer working locally? Clone the repo and open it in any editor that supports the Dev Containers specification (VS CodeVisual Studio Code, JetBrains, and others). The devcontainer config will be detected automatically.

2. Wait for Infrastructure

Wait ~5-10 minutes for infrastructure to deploy. After it deploys, the setup script starts port forwarding to the Argo Rollouts dashboard, keeping a terminal busy. Open a new terminal to run commands.

3. Explore the UIs

Open the Ports tab and navigate to each service:

• Port 30100: Argo CD (readonly / a-super-secure-password). Shows sync status. Use to refresh applications after pushing commits.
• Port 30101: Argo Rollouts. Shows canary deployment progress and analysis status.
• Port 30102: Prometheus. Explore available metrics and test PromQL queries. CLI tools work equally well if you prefer the terminal.

4. Fix the Manifests

Review and fix the configuration in manifests/.

This challenge uses Kustomize under the hood: a base set of manifests with environment-specific overlays (staging, prod). Argo CD detects and applies these automatically, so you don't need to run Kustomize commands manually.

5. Deploy Your Changes

Commit and push your changes to trigger the deployment:

git add manifests/
git commit -m "Fix configuration"
git push

If pushing to a branch other than main, also update the ApplicationSet in appset.yaml to point to your branch.

Speed up Argo CD sync:

argocd app get echo-server-staging --refresh
argocd app get echo-server-prod --refresh

6. Trigger the Rollout

After Argo CD syncs, retry the rollouts:

kubectl argo rollouts retry rollout echo-server -n echo-staging
kubectl argo rollouts retry rollout echo-server -n echo-prod

7. Watch the Rollout

Watch canary progress (should advance 33% to 66% to 100%):

kubectl argo rollouts get rollout echo-server -n echo-staging --watch
kubectl argo rollouts get rollout echo-server -n echo-prod --watch

In real-world progressive delivery, staging is updated first, validated, and then changes are promoted to production. This challenge skips that separation so you can focus on the canary rollout mechanics and health checks without managing two promotion steps.

8. Run the Smoke Test

Run the smoke test to verify your solution:

make verify

How to complete your challenge?

	When you push from Codespaces, GitHub forks the repository to your account automatically. If you are working locally, fork the repository on GitHub before pushing.
	Verify your solution: make verify
	If it passes, it generates a Certificate of Completion you can paste into the discussion.
	Share your solutions in this thread in answer below and mention your achievement on your LinkedIn account!

Toolbox kubectl - Kubernetes CLICommand Line Interface for interacting with the cluster kubens - fast way to switch between Kubernetes namespaces k9s - terminal UIUser Interface for managing and inspecting your cluster Argo CD CLI - manage Argo CD applications from the command line Argo Rollouts kubectl plugin - extended kubectl commands for managing rollouts

Helpful documentation Argo Rollouts documentation Analysis and progressive delivery PromQL basics kube-state-metrics exposed metrics

Are you ready? Take the challenge’s mission! Open in Codespaces Free GitHub account required. Launch in a preconfigured devcontainer.

Don’t forget about benefits! For completing at least one level you get a unique badge! For completing all of the three levels you get even more badges! Deadline: 13 October 2026 at 23:59 CET

Other levels of this challenge

Beginner	Broken Echoes
Expert	Hyperspace Operations [Coming Soon]