Hello, we have a system that runs an SQL query at interval and writes the result in a log file. The log file is ingested in dynatrace via openpipeline and bizevents are extracted from the logs. Unfortunately, the SQL query returns often the same information (it is not a tail but rather it reads the content of a table). This results in many duplicates log entries being created as well as multiple duplicate business events. How can we avoid ingesting duplicate business events when using openpipeline? Is there a way we can filter out the entry if it already exists? Each of our log entries contain an ID which uniquely identifies the entry, thus it would be easy to filter it out.
When attempting to do a lookup or fetch command in DQL in openpipeline it seems like this is not permitted.
Your help and recommandation is appreciated.
