topic Re: PingOne Extension - Suggessions in Extensions

PingOne Extension - Suggessions

observefdfx — Fri, 19 Jun 2026 15:03:17 GMT

We have a use case of ingesting PingOne logs and created workflow and ingesting via direct API call to PingOne. Ingesion is going good but to avoid duplicates, scanning the logs for dups which is causing additional cost for scanning. We thought of PingOne Extension may solve the issue but seems extension will not handle. Looking for suggessions on this.

Re: PingOne Extension - Suggessions

sujit_k_singh — Tue, 23 Jun 2026 00:49:38 GMT

Hi @observefdfx

The PingOne Extension won't help here — it's designed for metrics, not log dedup logic.

The real fix is preventing duplicates at source (cursor tracking), not detecting them post-ingest.
Honesty. I never faced this issue but i would suggest to look at this from the ingestion side rather than scanning after the fact.
PingOne Activity API supports an after cursor for pagination. Store the last processed event ID or timestamp between workflow runs and only pull events after that point. This eliminates overlapping time windows and removes the need for dedup scanning entirely.

Thanks,

Sujit

Re: PingOne Extension - Suggessions

observefdfx — Fri, 26 Jun 2026 15:25:26 GMT

Thanks for the reply. We are already doing storing the last processed event ID or timestamp between workflow runs. We add buffer of one minute from this timestamp because we had cases of missing few seconds messages and adding that buffer leading to dups.

Re: PingOne Extension - Suggessions

t_pawlak — Mon, 29 Jun 2026 08:16:43 GMT

Hi,

In this case I would keep the 1-minute buffer, but avoid doing duplicate detection after the logs are already ingested.

A safer pattern is:

1. Query PingOne with the buffered timestamp, for example last_processed_timestamp - 1 minute.
2. Before sending events to Dynatrace, deduplicate locally in the workflow/integration layer using the PingOne audit event ID.
3. Keep a small state store/cache of recently processed event IDs, for example for the last few minutes or hours.
4. Advance the checkpoint based on the maximum recordedAt / createdAt successfully processed, not simply the workflow execution time.

So the buffer protects you from late-arriving events, but the ID cache prevents re-ingesting the same records.

The PingOne extension will likely not solve this because this is ingestion-state logic, not metric collection logic. PingOne’s Activity/Audit API supports time-based filtering such as recordedAt gt ... / createdAt gt ..., and PingOne APIs also use cursor-style pagination, but the “exactly once” behavior still needs to be handled by the collector/workflow.

If you have the option, I would also recommend keeping this state outside Dynatrace, for example in Redis, PostgreSQL, Azure Table Storage, or another small external storage.

Then you can store two things there:

the last successfully processed timestamp / cursor
a short-lived cache of recently processed PingOne event IDs

With Redis, for example, you could keep the event IDs with a TTL of a few minutes or hours. The workflow would check whether the event ID already exists before sending the log to Dynatrace. If it exists, skip it. If it does not exist, send the event and store the ID in Redis.

This keeps the 1-minute safety buffer, but avoids scanning already ingested logs in Dynatrace.

Re: PingOne Extension - Suggessions

observefdfx — Mon, 29 Jun 2026 14:14:39 GMT

Thanks @t_pawlak for the solution with all details. We need to see and get approvals on using external cache. As an alternative, it looks dedup can be easy in OpenPipeline. I know it incurs log ingestion cost but seems better alternative as it avoids external cache or other caching mechanism. We still need to weigh in and estimate the ingestion cost but seems its better than scanning. Thoughts?

Re: PingOne Extension - Suggessions

36Krazyfists — Mon, 03 Aug 2026 14:03:23 GMT

Let me help you out. The PingOne extension by Dynatrace is great if you have a small PingOne AIC tenant. But if you have a large one that generates a lot of logs, you will struggle. PingOne uses Google Cloud as their backend and that means they have a 60,000 logs per minute rate limit per tenant. That means you will frequently hit rate limit issues if you are querying more than 60,000 log records a minute (which is almost guaranteed, especially if you are querying all log sources, which you really shouldn't as many of the sources aren't necessary for day-to-day monitoring). The extension logic tries to compensate by tracking this and filling in gaps in logs, but this just causes the queries it sends to be for more and more logs, meaning you will constantly hit the rate limit. It's a nightmare.

Instead, the solution is to stop querying for the logs altogether. Fetching logs is probably one of the worst ways to collect logs. Pushing them from your source into your monitoring tool is much better and easier.

Luckily, PingOne AIC offers such a log streaming feature as of earlier this year:
Stream logs to an external monitoring tool | PingOne Advanced Identity Cloud

This will stream logs from the PingOne SaaS tenants to the Dynatrace SaaS OTLP API Logs Ingest endpoint (well, it will once you configure it to do so, of course). This means those logs don't have to traverse your WAN boundaries into your network and you get every single log record streamed in near real time and you will get no duplicates.

Then, you just setup an OpenPipeline to format the logs to your liking and extract Smartscape Nodes/Edges and Metrics and place the logs in buckets so they are fast and easy to query (I recommend creating a Buckets Segment and pointing that Segment at all of your storage buckets. Then people can just use the Buckets Segment and select the PingOne log Buckets that you send your logs to in the PingOne OpenPipeline you setup.).

With this method, the only thing we use the official PingOne extension by Dynatrace for is scraping the Prometheus metrics. We don't query any logs with it because PingOne's log query method stinks.

What we actually did was a bit more involved than just that. Instead, I created what I have termed a "Presentation" type extension. Basically, it's a custom Dynatrace extension that doesn't have a datasource. Most extensions have a datasource, like SNMP, WMI, Prometheus, SQL, or Python (which is the type of extension the official PingOne extension is). Instead, what I figured out you could do is just define an extension without the datasource part. So, instead, the extension has the OpenPipeline definitions bundled within it (which contains the log formatting/transforming and metric extraction and smartscape extraction I mentioned earlier) as well as all of the classic topology extraction, screen definitions (so that the PingOne entities you extract show up in the Infrastructure & Operations app), the new I&O Explorer screen definition JSON files, alert templates, and custom dashboards (classic and latest).

I've actually trained our Cursor AI on how to create these "Presentation" type extensions for me and I use them for all kinds of stuff where the data is already in Dynatrace (like Cloudflare logpush logs that are streamed into Dynatrace or Syslogs that are streamed from devices like DataPower or F5). This allows us to standardize the way we format our logs, extract metrics and entities (classic and new smartscape), create dashboards and screens, and create alerts. It's super-fast and pretty easy once you get it all setup.

These are the entities we are currently extracting from our PingOne logs streaming into our Dynatrace environments:

And most of them have relationships to each other. The Journey entities have a parent_of relationship to the Node entities (Journeys are the TreeName field from the logs while the Node entities are derived from the scripted decision node display name fields from the logs) and then the Journeys have a runs_on relationship to the Tenants. This means we have a relationship of Tenant > Journey > Scripted Decision Nodes

I'm also working with our PingOne team to see if we can create a custom javascript node within PingOne to take the Header Origin field from the initial am-access log entry at the beginning of a user's journey and insert that origin field value into the scripted decision node am-authentication log entries that make up the elements of a user's journey within PingOne AIC so that we can have all of the Journeys' scripted decision node steps have a relationship to the origin address. This will allow full correlation of the telemetry within PingOne as well as allow us to create an entity mapping between the PingOne origin entities and other built-in entity types like Server Request entities and RUM Application entities, giving Dynatrace much better topological awareness of the authentication layer of a monitored web page or API endpoint. This is still a work in progress. I also have an idea for perhaps getting W3C trace propagation to work within PingOne AIC. Not sure if it's possible, but I think it's at least theoretically possible, so we shall see!

And, if you have other Ping products (like PingID or Ping Protect for example), they offer Webhook integrations which give you similar capabilities. We are currently setting that up and doing something similar to what we're doing for the PingOne AIC logs so that we can ingest and utilize those.