topic Re: Will the new OpenPipeline app allow better support for structured log formats? in Open Q&A

Will the new OpenPipeline app allow better support for structured log formats?

GosiaMurawska — Wed, 16 Oct 2024 08:50:11 GMT

For example multi-line JSON?

Re: Will the new OpenPipeline app allow better support for structured log formats?

thomas_billi — Wed, 16 Oct 2024 08:50:40 GMT

Recently we've released the `depth` parameter for the `fieldsFlatten` command (see docs). This helps a lot with unknown JSON formats as it allows to remove the nesting beyond the first level.

If you are referring to a situation where a multi-line JSON log record that is picked up by OneAgent is split into multiple records then that is something that needs to be handled by OneAgent.

Re: Will the new OpenPipeline app allow better support for structured log formats?

adam_gardner — Mon, 21 Oct 2024 06:06:07 GMT

I posted a fully supported solution for multi line ingestion here. My reply below the solution makes it clear why OpenPipeline is the wrong solution for this (as Thomas suggested above).