topic Re: PingOne Extension - Suggessions in Extensions

PingOne Extension - Suggessions

observefdfx — Fri, 19 Jun 2026 15:03:17 GMT

We have a use case of ingesting PingOne logs and created workflow and ingesting via direct API call to PingOne. Ingesion is going good but to avoid duplicates, scanning the logs for dups which is causing additional cost for scanning. We thought of PingOne Extension may solve the issue but seems extension will not handle. Looking for suggessions on this.

Re: PingOne Extension - Suggessions

sujit_k_singh — Tue, 23 Jun 2026 00:49:38 GMT

Hi @observefdfx

The PingOne Extension won't help here — it's designed for metrics, not log dedup logic.

The real fix is preventing duplicates at source (cursor tracking), not detecting them post-ingest.
Honesty. I never faced this issue but i would suggest to look at this from the ingestion side rather than scanning after the fact.
PingOne Activity API supports an after cursor for pagination. Store the last processed event ID or timestamp between workflow runs and only pull events after that point. This eliminates overlapping time windows and removes the need for dedup scanning entirely.

Thanks,

Sujit

Re: PingOne Extension - Suggessions

observefdfx — Fri, 26 Jun 2026 15:25:26 GMT

Thanks for the reply. We are already doing storing the last processed event ID or timestamp between workflow runs. We add buffer of one minute from this timestamp because we had cases of missing few seconds messages and adding that buffer leading to dups.

Re: PingOne Extension - Suggessions

t_pawlak — Mon, 29 Jun 2026 08:16:43 GMT

Hi,

In this case I would keep the 1-minute buffer, but avoid doing duplicate detection after the logs are already ingested.

A safer pattern is:

1. Query PingOne with the buffered timestamp, for example last_processed_timestamp - 1 minute.
2. Before sending events to Dynatrace, deduplicate locally in the workflow/integration layer using the PingOne audit event ID.
3. Keep a small state store/cache of recently processed event IDs, for example for the last few minutes or hours.
4. Advance the checkpoint based on the maximum recordedAt / createdAt successfully processed, not simply the workflow execution time.

So the buffer protects you from late-arriving events, but the ID cache prevents re-ingesting the same records.

The PingOne extension will likely not solve this because this is ingestion-state logic, not metric collection logic. PingOne’s Activity/Audit API supports time-based filtering such as recordedAt gt ... / createdAt gt ..., and PingOne APIs also use cursor-style pagination, but the “exactly once” behavior still needs to be handled by the collector/workflow.

If you have the option, I would also recommend keeping this state outside Dynatrace, for example in Redis, PostgreSQL, Azure Table Storage, or another small external storage.

Then you can store two things there:

the last successfully processed timestamp / cursor
a short-lived cache of recently processed PingOne event IDs

With Redis, for example, you could keep the event IDs with a TTL of a few minutes or hours. The workflow would check whether the event ID already exists before sending the log to Dynatrace. If it exists, skip it. If it does not exist, send the event and store the ID in Redis.

This keeps the 1-minute safety buffer, but avoids scanning already ingested logs in Dynatrace.

Re: PingOne Extension - Suggessions

observefdfx — Mon, 29 Jun 2026 14:14:39 GMT

Thanks @t_pawlak for the solution with all details. We need to see and get approvals on using external cache. As an alternative, it looks dedup can be easy in OpenPipeline. I know it incurs log ingestion cost but seems better alternative as it avoids external cache or other caching mechanism. We still need to weigh in and estimate the ingestion cost but seems its better than scanning. Thoughts?