Drop records after metric and Davis event creation in pipeline

[suraj_gupta3]

Hello,

I am evaluating a log-processing approach in Dynatrace OpenPipeline to reduce log storage and licensing costs while preserving operational visibility.

The idea is to first analyze incoming logs and generate the required observability signals, such as Metric Events and Davis Events, from the relevant records. Once these events have been successfully created, the original log records would be dropped from the pipeline instead of being retained.

This approach would allow us to continue monitoring critical conditions and triggering alerts through Metric Events and Davis Events, while reducing the volume of logs stored and consumed for licensing purposes.

Proposed Flow:

Incoming Logs
↓
Filter Relevant Records
↓
Generate Metric Events
↓
Generate Davis Events
↓
Validate Event Creation
↓
Drop Original Log Records
↓
Reduce Log Retention & License Consumption

Regards,

Suraj Gupta

[dylan_taelemans]

Hey, I’m not 100% sure, but I don’t think it’s fully possible.
You could create a custom bucket with 1-day storage — it would still have a cost, but I think that would be the closest option to your solution.

[thomas_billi]

Mostly possible. A few things on this before you build it:

The "Validate Event Creation → Drop" gate won't work. OpenPipeline processes each record through its pipeline in one pass — extraction (metrics, Davis events) and the drop/storage decision are separate processors acting on the same record, not a sequential flow where one waits for confirmation from the other. There's no mechanism to hold a log back pending confirmation that its event was created downstream. If you want confidence the extraction is correct, validate it upfront (test the extraction rule against sample data / dry-run in a non-prod pipeline), not at runtime per-record.

Dropping logs loses your ability to investigate. Metric/Davis Events tell you something happened, not why. Once the record's dropped, there's no drill-down, no root-cause query, nothing to check if someone asks what actually happened. Fine for pure threshold alerting, risky if you ever troubleshoot from these events.

Cheaper alternative: route non-critical logs to a lower-retention bucket instead of dropping them outright — you get the cost reduction without losing the raw data.