cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

What are the pipelines within OpenPipeline?

GosiaMurawska
Community Team
Community Team

What does it consist of? Can a pipeline be linked to multiple data types?

1 REPLY 1

thomas_billi
Dynatrace Advisor
Dynatrace Advisor

In the context of data ingest, pipelines are where filtering, processing, enrichment, and extraction of incoming data happen. All of this is defined by “Processors.”  

The main benefits of using pipelines are

  • clear split of data streams
  • distribution of processing configuration and testing to teams that are the data owner 

A pipeline is linked to a specific data type. There can be multiple pipelines per datatype. The dynamic routing defines the pipeline that a record is sent one. Each record goes through only one (!) pipeline.

A pipeline contains several steps, called “Stages”: Processing, Metric extraction, Data extraction, Permission, Storage.

a more detailed explanation can be found in the OpenPipeline Documentation.

Featured Posts