Incremental ingest of custom sql extension dimensions/metrics?

[moilejter]

Hi!

We've just started looking into custom sql extensions, to run database queries and make the results available within dynatrace.  The simple case seems straightforward enough: run a database query, and the complete result set becomes available within dynatrace.  But then, when the full query is rerun some interval later, there will be a lot of overlap between the two.  Does the ingest process map the "same" rows so that they only appear once?  Or, if the write the query so that it's incremental (get the rows in the last minute, run every minute) - does the ingest process automatically record all the results across all invocations?

The docs I found don't seem to go into this kind of detail...

Thank you!

[sujit_k_singh]

Hi @moilejter 

It depends on Output type
The SQL extension does not deduplicate rows between executions. There is no "same row" detection. What happens to repeated data depends entirely on which output type you're using:

1.  Output Type: Metrics
Each row maps to a metric data point with a timestamp of when the query ran. The metric store is a time series it doesn't care about row identity. Every execution writes new data points at the new timestamp.

Implication for your overlap question:

If you run a full query every minute and the same rows come back, you get new metric data points each minute not duplicates in the traditional sense, because each has a different timestamp.

2. Output Type: Logs or Business Events
This is where overlap becomes a real problem. Each row becomes a discrete record stored in Grail with a timestamp. There is no deduplication.

Full query every minute → every row ingested as a new log/bizEvent record every minute → massive duplication
A row that existed 5 minutes ago and still exists now will appear 5 times in Grail

Thanks,

Sujit