Tuning Retention Period of Extension Logs

[siavash1996]

With the introduction of Pipeline Groups and Ready-Made OpenPipeline pipelines for extensions, such as SQL Server, am I correct in understanding that we can no longer move logs to a separate bucket with a custom retention period? I just updated our Microsoft SQL Server extension to ver 3.1.2, and I was aware of the breaking changes, but I didn't think it would completely block the customization of bucket assignment.

In my tenant, all of our database extension-related logs are being picked up by a new read-only pipeline named "SQL Server" that has a processor that pushes all logs from SQL extensions to a bucket called "default_database_monitoring".

The bucket it refers to is also read-only. I can't modify its retention period or anything else.

Am I missing something here? Surely there must be a simple way to manipulate these logs with OpenPipeline alongside the new ready-made pipeline.

[p_devulapalli]

@siavash1996 You should still be able to store the logs in a separate bucket with a custom retention period. If my understanding of pipeline group is correct, you should able to create a base pipeline with custom bucket and add it to group. I haven't tried it yet, but looks possible 

[thomas_billi]

---

@siavash1996 wrote:

...am I correct in understanding that we can no longer move logs to a separate bucket with a custom retention period?...

---

These are two separate things that arrived around the same time:

The dedicated "SQL Server" pipeline is a result of the 3rd-gen extension behavior (major version upgrade → dedicated pipeline instead of injecting into the Classic pipeline). That pipeline can be used as a base pipeline in a Pipeline Group, letting you add your own stages in the execution flow.

The storage lock is unrelated to that. All monitoring data from database extensions is assigned to default_database_monitoring via a dedicated processor — this data is treated as database monitoring data, not general-purpose logs. That bucket assignment is intentionally fixed and not overridable today.

If you need custom retention for this data, a feature request to the product team is the path forward.

[siavash1996]

Hi Thomas,

I just want to clarify something. 

Is it possible to override the storage assignment of the SQL extension logs to send them to a different bucket than the default_database_monitoring bucket?

I also don't seem to be able to add the ready-made pipeline as a base pipeline in a pipeline group. I can only add it as a member pipeline. Is this expected behavior?

[lucas_hocker]

Hi @siavash1996,

What is your end goal of moving the internal data for the Databases app? If we allow you to move it to a different bucket, it would no longer show up in the Databases app. The data is not logs per se, rather we use the log signal type to bring the data from the extension to the app. It contains things like execution plans, configurations, etc. The format of the data is subject to change without notice, since it's the internal data for the app.

As documented here: https://docs.dynatrace.com/docs/shortlink/databases-app#permissions. Customers are not charged queries from the Databases app against the data.

Database logs, either collected by OneAgent or ingested via API, are not stored in this bucket.

[siavash1996]

Hi Lucas,

Our specific use case at this time is longer-term storage of long-running queries gathered by the extension from our SQL server's query store. In the past, before the extension upgrade, I could do this pretty easily by adding a separate processor on the Storage step.