How to avoid creating duplicate business events when using openpipeline?

strudeau — Mon, 17 Mar 2025 13:46:21 GMT

Hello, we have a system that runs an SQL query at interval and writes the result in a log file. The log file is ingested in dynatrace via openpipeline and bizevents are extracted from the logs. Unfortunately, the SQL query returns often the same information (it is not a tail but rather it reads the content of a table). This results in many duplicates log entries being created as well as multiple duplicate business events. How can we avoid ingesting duplicate business events when using openpipeline? Is there a way we can filter out the entry if it already exists? Each of our log entries contain an ID which uniquely identifies the entry, thus it would be easy to filter it out.

When attempting to do a lookup or fetch command in DQL in openpipeline it seems like this is not permitted.

Your help and recommandation is appreciated.

Re: How to avoid creating duplicate business events when using openpipeline?

yanezza — Sat, 05 Apr 2025 21:16:58 GMT

Hi,
this thread will help you
https://community.dynatrace.com/t5/Extensions/Log-Ingestion-From-a-Database-Table/td-p/231961

TLDR: play with the ingestion system fetching rules or play with filtering option when reading the logs (dedup for example)

topic Re: How to avoid creating duplicate business events when using openpipeline? in Log Analytics

How to avoid creating duplicate business events when using openpipeline?

Re: How to avoid creating duplicate business events when using openpipeline?