OpenTelemetry & Apache Arrow – a winning combination

[AdrianaVillela]

The OpenTelemetry (OTel) ecosystem has grown and matured a LOT since its inception in 2019. Traces, logs, and metrics have reached general availability (GA). Profiling was added as a new OTel signal. The OpenTelemetry Demo has expanded. The OTel Collector has expanded, with new components being added regularly. We've seen the addition of new components to the OTel ecosystem to help make it more ergonomic, including OpAMP, the OTel Operator, OTel Weaver, and OTel Arrow.

More organizations are adopting OpenTelemetry, and as they start emitting more OTel data, they're starting to see scaling issues. One such issue is the explosion of telemetry data, which translates to increased pressure on resources and increased costs associated with telemetry. Organizations obviously see value in making their systems observable, but they're probably not super excited about spending a small fortune to do it. So what's the solution?

Before we get to that, let's dig into the problem a bit more.

OTLP is great, until it isn't

OpenTelemetry uses the OTLP protobuf data format, which is not optimized for managing high volumes of telemetry data. This is because:

• Telemetry data is hierarchical: Protobuf must recursively encode and decode the data, which is CPU intensive.
• OTLP is row-based: Each record is stored as a single message. This becomes expensive for duplicate (common) values.
• Serialization/deserialization is expensive: The path SDK → Collector → Backend requires encoding/decoding. Again, this is CPU intensive, memory intensive, and causes latency.

This is where OpenTelemetry Protocol with Apache Arrow (OTAP or OTel Arrow) comes to the rescue.

OTel Arrow is based on the Apache Arrow framework, and serves as both a data format (how data is structured) and a transport protocol (how data moves between systems).

It converts OpenTelemetry OTLP data into Apache Arrow's columnar format, providing a more efficient way to represent in-memory batches of OTel signals. OTel Arrow helps thanks to its:

• Columnar format: This is optimized for analytics and data storage (optimized for compression). It also reduces data duplication. Repeated data is stored once in a dictionary. "Duplicated" data is replaced by a pointer to the "source of truth" dictionary item.
• Zero-copy data sharing: When data moves between components, you need to serialize it, copy it to a new buffer, deserialize it, and allocate new memory storage. Zero copy means that everything stays in the same memory buffer. This saves on CPU, lowers memory pressure, and reduces latency.

OTel Arrow is optimized for:

• High telemetry volume
• Multi-cloud or multi-region deployments
• Systems with bandwidth constraints

OTel Arrow at work

Awesome, but how is OTel Arrow implemented?

Phase 1

OTel Arrow is currently implemented in the OTel Collector via the OTel Arrow Exporter, which emits data to a backend that accepts OTAP.

BUT…Since most telemetry backends don't accept OTAP out of the box, you'll likely need to add in an OTel Collector Gateway to ingest OTAP data via the Arrow OTLP Receiver and export OTLP to your OTLP-compatible observability backend.

✨NOTE: If you'd like to see some examples in action, check out this article.

If you're looking at the above diagram and are thinking that this feels like a lot of steps and that we're losing some efficiency having to convert from OTAP back to OTLP, you're not wrong. This is where Phase 2 of the project comes in.

Phase 2

To get even more out of OTel Arrow, the project maintainers are working on Phase 2, the OpenTelemetry Arrow Dataflow Engine, which is nearing completion. The dataflow engine is written from the ground up in Rust, providing lower-level control, better performance characteristics, and elimination of garbage collection-related issues. The end result will be more performant, provide better compression, and have lower latency than through the current OTel Collector and OTAP exporter combination.

When completed, the new flow will look like this, with the OTel Arrow Dataflow Engine taking the place of the OTel Collector.

Using OTel Arrow currently results in a 50% reduction in bandwidth. This number was reached by benchmarking against the same data being of equal batch sizes sent using standard OTLP/gRPC with Zstd compression (fancy compression algorithm). OTel Arrow also boasts a compression factors of 15x-30x compared to uncompressed data. Even greater overall gains are expected for Phase 2.

To learn more about OTel Arrow Phase 2, check out this blog post on opentelemetry.io.