In the context of data ingest, pipelines are where filtering, processing, enrichment, and extraction of incoming data happen. All of this is defined by “Processors.”
The main benefits of using pipelines are
- clear split of data streams
- distribution of processing configuration and testing to teams that are the data owner
A pipeline is linked to a specific data type. There can be multiple pipelines per datatype. The dynamic routing defines the pipeline that a record is sent one. Each record goes through only one (!) pipeline.
A pipeline contains several steps, called “Stages”: Processing, Metric extraction, Data extraction, Permission, Storage.
a more detailed explanation can be found in the OpenPipeline Documentation.
