Archiving incident-related observability data beyond standard retention periods

[FranciscoMPalos]

In our organization, the SRE team has raised the following requirement:

When an incident occurs, they want to preserve all logs, traces, and metrics collected during the affected day so that they can continue investigating and performing data analysis for an indefinite period of time.

From an Observability perspective, what solutions could we provide to address this need?

We have tried to guide them towards using Dynatrace Notebooks, but their concern is that the underlying data may no longer be available after some time due to retention limits. What they are looking for is the ability to come back weeks or months later and still have access to the same data set in order to continue the investigation and analysis.

Has anyone faced a similar requirement? What would be the recommended approach in Dynatrace for long-term preservation of incident-related observability data?

[sujit_k_singh]

Hi @FranciscoMPalos 

Custom bucket retention — set up a dedicated bucket with longer retention and route incident data into it via a bucket assignment rule. 

Export the incident slice out of Dynatrace entirely — this is what most teams actually end up doing for "indefinite." Run the DQL for that day/incident, dump logs/traces/metrics to CSV/JSON, push to S3 or wherever. We wired a Workflow to auto-trigger this the moment a problem gets flagged as an incident basically takes a snapshot and archives it outside the platform. That way retention policy stops mattering.

Notebooks alone won't save you — like you noticed, the notebook keeps the queries but not the underlying data once it ages out. If you want notebooks to stay useful months later, export the query results alongside them, don't rely on live re-execution.

Realistically: long-retention bucket for active incidents + auto-export workflow on incident creation is the combo that's held up for us.

Thanks,

Sujit

[FranciscoMPalos]

Thanks, Sujit. Your approach makes a lot of sense.

This raises a related question for me:

Instead of focusing solely on extending retention periods, is there a recommended or supported way in Dynatrace to create a forensic snapshot of an incident?

By "forensic snapshot," I mean capturing all relevant observability data associated with an incident at a specific point in time (logs, traces, metrics, events, topology context, and relationships) and preserving it for future investigations, regardless of the underlying Grail retention policies.

Have any of you implemented something similar? If so, how did you preserve both the data and the context needed to continue the investigation months or even years later?

I'm particularly interested in understanding whether this can be achieved natively within Dynatrace or if exporting the data to an external archive remains the recommended approach.

[Fin_Ubels · Accepted solution]

Hey Francisco,

I am not aware of any such native function. The only way I can think of achieving something similar within Dynatrace is creating either some sort of complex workflow that looks at the problem details and then fetches data related to entities captured in the problem, or building a notebook template that effectively does the same thing as the workflow. Both solutions would require a fair bit of work.

I guess the most important thing to understand is, what is the goal of that historic analysis? What are people trying to find, what would be useful? This would help prevent data hoarding and make it more focused and hopefully useful.

Regardless, I think it would be good to open an RFE in the ideas section of this forum as it is a use case I have come across before.

A few problems come to mind though for any such implementation:

• How would you determine what is incident data? For example, lets say there is a response time issue that affects 1 service. Do we keep metrics from the hosts that support that service? Logs from that host too? Full traces that went through that service? Any metrics/logs upstream/downstream of the service?
• How long before/after the problem would you save? For example, lets say that response time problem lasted 10 minutes, would you save 20 minutes of data, 5 minutes either side?
• How would the collection be triggered? Automatic based on rules as well as a way to trigger it manually?

[FranciscoMPalos]

Hi @Fin_Ubels ,

Thank you for your feedback and for highlighting the challenges involved in implementing such a capability.

I agree that the key questions are defining the scope of the incident data, the retention window, and the triggering mechanism. These aspects would need to be carefully designed to avoid unnecessary data collection while still preserving the information required for effective post-incident analysis.

From our perspective, a useful forensic snapshot capability would ideally capture:

• The complete Davis Problem record
• Impacted entities and services
• The entity topology at the time of the incident
• Related logs
• Related traces/spans
• Associated events
• Metrics from the affected entities
• A configurable time window (for example, 30 minutes before and 30 minutes after the incident)

The objective would be to create a point-in-time snapshot that can be preserved for later root cause analysis, audits, post-mortem reviews, or regulatory investigations, without being affected by standard retention policies.

Based on the feedback received in this discussion, we will open an RFE in the Ideas section of the forum describing this use case and the expected scope of such a feature.

Thank you again for sharing your thoughts and experience.

Best regards.

[sujit_k_singh · Accepted solution]

Hi @FranciscoMPalos 

There Is No Native "Forensic Snapshot" Feature.
Dynatrace does not have a built-in forensic snapshot capability that freezes a slice of incident data independently of retention policies. The closest native approximation is a combination of custom Grail buckets + Workflows-based export.

The realistic answer is a two-layer strategy.
First - Extended Retention via Custom Grail Buckets (Native)
Grail supports retention periods from 1 day up to 10 years (3,657 days) for logs, spans, events, bizevents, and user sessions. This is the native path for keeping data queryable inside Dynatrace long-term. (Not adding details assuming you must we be aware of this.)

Second -Workflow-Based Export Archive (Recommended for "Indefinite")
For true indefinite preservation independent of Grail retention, a Workflow triggered on problem creation is the practical solution. Here's the architecture:

Trigger: Problem opened event (dt.davis.problems or Davis problem trigger)

Workflow steps:

1. Capture the incident window — extract problem.start_time, affected entity IDs, impacted services from the problem payload
2. Run DQL queries for that time window across:
                          fetch logs filtered by affected entity IDs
                          fetch spans filtered by affected services
                          timeseries metrics for affected hosts/services
                          fetch events for the problem context
3. Export results via a JavaScript action calling an external API (S3 PutObject)
4. Store a manifest — save a JSON index of what was exported, the DQL queries used, entity topology at incident time, and the problem record itself

Thanks,

Sujit