Ability to run a log record through multiple OpenPipeline pipelines?

[jmarbaix]

I am currently setting up new pipelines in OpenPipeline to migrate away from the log processing rules in the settings classics.

What I don't seem to find, is how to run a log record through mulitple pipelines. I want to be able to have a first pipeline which parses some fields out of the log records (for example which application and environment the log record comes from based on the host name). 

What I would like to then do, is have a pipeline for each application which parses the log records further based on the application and environment.

How I see it, the only way to do this is to have only one pipeline, but then we can just as well continue using the old log processing rules...

The documentation also mentions "pre-processing" (https://docs.dynatrace.com/docs/discover-dynatrace/platform/openpipeline/concepts/data-flow#pre-proc...), but no where is explained where/how this should be done. Should this be built in in our software? Is this something that we are able to set up in Dynatrace?

[thomas_billi]

Totally understand what you want to accomplish. It's similar to what we hear from many customers. We do have some ideas for how to solve the problem that you're describing. It would be great if you'd be willing to discuss with us and provide input, please let me know and I'd reach out to you via email to set up a call.

Some background: We think that sending records from one pipeline to another is quickly leading into a situation where it becomes unclear how records actually go through OpenPipeline and how they are processed. Also, pipelines are a complete package of processing, extraction and storage, with a defined order. At the end the record should be ready for storage and not go into processing again.

Regarding "pre-processing":

Pre-processing is processing that happens in Ingest Sources, it's currently only available for the Events configuration. We will extend it to other data types eventually.

[jmarbaix]

Hi Thomas,

I'm certainly open for a call! Don't hesitate to send me an e-mail.

Julien

[thomas_billi]

There's now also a Product Idea for this:
https://community.dynatrace.com/t5/Product-ideas/Enable-Hierarchical-or-Sequential-Processing-in-Ope...