OTel Weaver: a high-level overview of OTel's schema and policy enforcement

[AdrianaVillela]

Today, I'll give you a high-level overview of OpenTelemetry (OTel) Weaver, so that you can better understand what it does, and how it can help your organization as it moves ahead in its OpenTelemetry journey.

What is Weaver, anyway?

At its core, Weaver is a tool that allows you to create, document, and manage OpenTelemetry schemas and policies. But why should you care?

By defining and using a schema for your OpenTelemetry instrumentation, you ensure that everyone in your organization is speaking the same language. You establish:

• Naming conventions
• Spans, metrics, and logs (events) used by your application, and their attributes
• Units of measure (metrics only)
• Example values/format
• Description of your telemetry
• Stability level (in case something is still in the works, or is deprecated)
• Policies

Policies are written in Rego, used by OpenPolicy Agent (OPA) and can be used to define things such as:

• Enforcing a prefix matching the company name for every custom metric, attribute, etc.
• Requiring fields that are considered optional by the semantic conventions, such as brief
• Validating the annotations attached to attributes and/or signals

Not only does Weaver let you define these, it can generate documentation (and/or code) for you, validate your schema, and enforce schema adherence. You can configure your CI process so that if someone introduces code that violates your schema and policies, that code is blocked from going into production until it's fixed.

To use Weaver, you can either run the Weaver binary locally (by building from source), or, if you're lazy like me, you can run it ...

Weaver flow

To help you better understand how Weaver works, let's look at what a Weaver workflow might look like.

1. Create schema

First things first — you must create a schema for your telemetry! You do this by creating YAML files to describe your telemetry schema — i.e. files that describe your spans, metrics, and events (logs).

The schema files should go into a special schema folder in your repository. You can call the folder whatever you want. For example: [telemetry/registry](https://github.com/telemetrydrops/otel-in-practice/tree/main/stage-1-monolith/telemetry/registry) or [telemetry-schema](https://github.com/open-telemetry/opentelemetry-demo/tree/main/telemetry-schema).

The names of the YAML files also don't matter, as Weaver will scan them and interpret them accordingly (as long as you provide the location of the schema files). Although the name doesn't matter, you should call them something meaningful. 

As a best practice, you should create separate YAML files for spans, events (logs), metrics. If your application has multiple services, consider breaking each signal type down by service name. You can see a great example here. Common attributes should be kept in a separate file so that they can be referenced, avoiding repetition.

Once you've created your files, you validate them like this:

weaver registry check \

--registry <path_to_registry>

This ensures that your YAML is formatted correctly. For example, do your metrics definitions include units?

Policy validation is optional, but if you'd like to be extra rigorous, you can add it to your validation process:

weaver registry check \

--registry <path_to_registry>

--policy <path_to_policies>

2. Create templates (optional)

Templates are Jinja2 templates that Weaver can use to transform your YAML schema into code. This code can include but is not limited to: markdown, HTML files, document templates for Confluence, and data structures in your application code.

It's important to note that this part isn't done by Weaver. You have to create your own templates. This also means that you have to be familiar with Jinja2. For more information on how to create your own templates, check out this example.

Now, although Weaver itself doesn't come with its own templates, some OTel projects have created their own:

• Java templates
• Go templates
• Markdown templates for the OTel Demo

This means that you can technically skip this step if there's an existing template that works for you.

3. Generate files from templates

With template files in hand, you can use Weaver to generate code from those templates (see step 2). That is, Weaver looks at your schema definitions, applies them to your templates, and then magically turns that into code.

This is done by running:

weaver registry generate \

--registry <path_to_registry> \

--template <path_to_templates> \

<output_path>

The template path can be either a local path or a remote path (e.g. GitHub repository). Check out a full example here.

4. Validate your code in CI

Here, you can use Weaver's live-check during test execution in your CI process, ensuring that the telemetry emitted by your application (e.g. via an OTel Collector) adheres to the schema and policies you wish to enforce.

The live-check produces a report telling you where you're non-compliant. It exits with code 1 if there are any violation findings in the report. It also produces various stats in the report, such as coverage, which can be used to check if your tests encompass the full registry.

Weaver's live-check is incredibly flexible. It supports multiple:

• input sources (e.g. file path, stdin, OTLP)
• input formats (e.g. text, JSON)
• output formats (e.g. YAML, JSON, JSONL)

It's great to have all that flexibility, but so many options can also be a bit overwhelming!

One way to run it is like this:

weaver registry live-check \

--registry <path_to_registry> \

--policy <path_to_policies> \

--input-source otlp \

--format yaml

--output <path_to_report_output> \

--otlp-grpc-address 0.0.0.0 \

--otlp-grpc-port 4318

Which runs a live-check on telemetry emitted by a Collector at address 0.0.0.0:4318, checks it against the registry (--registry flag) and policies (--policy flag) provided, and outputs a YAML report to the specified path.

5. Refactor!

If your CI process fails, you have to go back to the drawing board and refactor your code to ensure that it's compliant. UGH! But wait — refactoring doesn't have to be a nightmare, because Weaver has an MCP server (weaver registry mcp) which, paired with your AI coding agent, can be used to help you refactor the code that failed the CI validation.