topic Re: What are the main benefits of OpenPipeline? in Open Q&A

What are the main benefits of OpenPipeline?

GosiaMurawska — Thu, 24 Oct 2024 12:23:21 GMT

From a user's perspective, why is it worth to use OpenPipeline?

Re: What are the main benefits of OpenPipeline?

thomas_billi — Thu, 24 Oct 2024 12:54:24 GMT

Unified ingest: high scalable stream processing and a unified app to manage all data ingested into Dynatrace

Contextualization, enrichment and transformation:  
- Enrich data and improve its value and quality by adding supplemental attributes (such as geolocation derived from an IP address). 
- Normalize data without losing context. 
- Transform plain text content into well-defined fields, convert raw data to time series, or create business events from log lines. 
- Converge heterogeneous data sources with ease and combine with the powerful schema-on-read from Grail.  
- Prioritize business data for a desired quality of service (QoS). Dynatrace provides unmatched accuracy by treating relevant business data (for example, real-time consumption or revenue dashboarding and analytics) with a higher priority and ensuring that data is not only in context but also not dropped or sampled.

That said: OpenPipeline optimizes data for Dynatrace analytics and AI. 

Re: What are the main benefits of OpenPipeline?

StrangerThing — Thu, 24 Oct 2024 13:25:59 GMT

My favorite part is just the organizational simplicity. You go from having giant, tenant-wide rulesets for processing, metric/event extraction, bucket assignments, etc. to having simple, segregated routes for any way you'd like to split each log/event/bizevent.

Second favorite part is I can build a parser or something of that nature directly in a Notebook in DQL and copy that parse command over to my processing rule. One language to rule them all!

Re: What are the main benefits of OpenPipeline?

PacoPorro — Thu, 24 Oct 2024 14:50:48 GMT

OpenPipeline is an effective tool for organizing logs because it allows you to categorize log entries into different “buckets” based on specific criteria. Each bucket can have its own retention policy, ensuring that logs are stored for the appropriate amount of time based on their importance or relevance.

Re: What are the main benefits of OpenPipeline?

r_weber — Wed, 15 Jan 2025 10:20:45 GMT

Not-so obvious usecases I'm covering with OpenPipeline:

Post-Processing of (built-in) BizEvents

I use pipelines to also post-process/enhance built-in bizevents such as the ones creating by guardian objectives so that they can be used by users more easily in other areas, not only in the guardian app.

E.g. a bizevent with the result of a objective contains a JSON field that has all the details about the objective executiopn (passed, result value,...). With OpenPipeline i enhance this bizevent to extract values, which then makes it easier for user to re-use it.
So instead of a user having to deal with parsing in DQL they can then simple chart all the results of a SRG Objective more easily, without worrying about DPL parsing:

fetch bizevents | filter guardian.id == "..." | filter event.type == "guardian.validation.objective" // parsing is already done in openpipeline //| parse guardian.objective, "JSON:objective" //| fieldsAdd value = objective[value], unit = objective[unit], name = objective[name] | fields timestamp, value, name, unit

Cleaning up default data:

I also use OpenPipeline to cleanup OneAgent captured bizevent that consume storage. E.g. when injecting bizevent data you also get contextual information like if the trace was sampled, the infrastrcuture (k8s cluster, pod, container, host, ...).
This might be useful in some cases but for bizdata where I certainly do not need this I already remove this data with openpipeline to only store the data I really need. This saves storage capacity on DT side and also makes the bizevents more readable when users want to use it in DQL queries.
e.g. I usually add this as the last processing step to bizevents where I really only need the biz data:

fieldsRemove "k8s.*","dt.entity.*", "dt.kubernetes.*", traceparent, span_id, trace_id