Hi everyone,
We are currently redesigning our Log Management architecture in Grail to optimize both query performance and storage costs.
Historically, we kept raw logs for long periods, but we are now moving to a strict maximum retention of 30 days for all log levels (DEBUG, INFO, WARN, ERROR, and FATAL). We are also planning to partition our buckets by cluster rather than keeping a massive single bucket, and we intend to leverage the "Retain with Included Queries" billing model since our retention fits the 10-35 day window.
I would love to hear how you are managing this in your own environments:
Bucket Strategy: Are you partitioning your buckets primarily by cluster, by team, or by application?
Long-Term Data: For business or audit reports that require historical data (1 year+), are you dropping the raw logs and relying entirely on log-based metrics extracted via OpenPipeline?
Access Control: How are you managing IAM policies and Segments so that platform engineers maintain central governance while giving developers autonomy to query their own clusters?
Any insights, challenges, or lessons learned from your implementations would be greatly appreciated!
Thanks in advance.
