Store query results without duplication with Extension 2.0

[Davidsso]

Imagine building a 2.0 extension to query a website's database, where purchases are stored in a table.

The extension runs every 5 minutes. So I create a query to find the last 5 minutes and store the rows in a metric.

However, sometimes the last values ​​are duplicated, especially if they are close to the 5-minute time limit for running the extension.

Has anyone experienced something similar? Do you know of a way to avoid this?

[Julius_Loman]

What exactly should be the result on Dynatrace side? Number of purchases in the last minute (or other interval)? Purchases ingested as log entries or bizevents? 

With extension 2.0 - for SQL data sources I use the execution interval (5 minutes for example) and the query condition selected the data from the last X minutes (preferably truncated to minute). There is no overlap unless the records are written to the database with delays. But in this approach, you might miss some records - for example when the extension is not running like during an ActiveGate upgrade. Extensions are not designed to be a data pump. 

If you need to ingest every record and have no duplications, you might want to use Python in EF2.0.  You can have a stateful extension and do the query within the Python code where you can store the last queried timestamp and next query will select records starting at this timestamp. But take in mind the extension runs on an activegate group and during activegate unavailability it can run on a different activegate host. Thus the state information has to be stored somewhere.

[Davidsso]

in my case the problem is that the quantity is more than in the database