How to reduce average daily log volume to license level

[sam_stachura]

Hello all, apologies if there is a real simple fix for this, my googlefu is not expert by any means.

Question: We are logging too much in dynatrace log analytics and need to reduce the avg daily log volume to our license level.
- How do I find and reduce log sources?
- How can I set a daily quota? Monthly/yearly is a little far out...
- How long does the results take to show a difference in logging? (How can I verify my changes?)

Seems simple, went in thru Settings -> Log monitoring -> Sources, switched to process group perspective, then set to "exclude the following logs" checked some boxes, hit save at the top. Later going back to the same place none of my selections are still there (there is one on "hosts perspective" view, but I don't want to do this on a host specific way with lots of hosts).

Thank you

[skrystosik]

Are you sure that your new settings are not overlapping with old one on your time picker?

Sebastian

[sam_stachura]

Not sure what "time picker" is, the clock on the top right? I did not have a problem open / set it to a timeframe as I was making the adjustments.

[skrystosik]

Yes I was talking about this one, but indeed this is only for displaying logs, not for license purposes. Peter is right here 🙂

[peter_zahrer · Accepted solution]

Hi Sam,

regarding your questions:

- How do I find and reduce log sources?

you found the place. Settings -> Log monitoring -> Sources . There you can specify the inclusion and exclusion of logfiles based on their source host or Process Group. The change should be persisted and also visible if you revisit the settings page. Otherwise it would be a bug and thus would have to be investigated.
- How can I set a daily quota? Monthly/yearly is a little far out...

There is no way to set a daily ingestion limit. Currently only monthly.
- How long does the results take to show a difference in logging? (How can I verify my changes?)

After you save the file selection the change takes place immediately. E.g. you exclude a file, the agents will stop sending the file content. I guess you are using our managed deployment model. In this case you will see the change in the license UI after a while (the sync takes place each hour). Due to the fact that the annual average ingest is calculated, the difference might be very little.

apologies for the complex licensing model, we are working on an improvement. See https://www.dynatrace.com/support/help/reference/monitoring-consumption-calculation/#expand-83dynatr...

[sam_stachura]

Ok, based on the link and internal conversations, we likely had "overages allowed" on our managed cluster license pool. We just purchased (double my annual salary of) additional licenses but seem to be missing out on the overage part of log monitoring licensing. So guess we need to chat with sales again on that part. 11gb daily is quite small for our monitoring needs, not sure if I'll be able to reduce to sources and still cover what we want.

[sam_stachura]

Another thing, so the sync window is every ~1 hr for managed env, the changes I made yesterday do not result in any difference in the "Avg. daily log volume" for the specific "Environment". Still shows hundreds of gb a day... but changing the values today seems to be sticking unlike yesterday... bet it was my session / browser acting up and not dynatrace.
If it is fixed you prob won't hear from me again, will post again if the amount logged per day doesn't change after a few hours (just removed logging on tons of stuff).
Thanks for the tips and tricks!

[sam_stachura]

for the record, still above the 3gb quota (somehow after setting this all yesterday), but after talking with dynatrace support they showed where I could get better stats on log file storage:

- Settings > log monitoring > data storage

Currently at:

Log data storage

Estimates are based on current usage.

Estimated yearly averages

Based on current log data storage usage

0.783 GB/Day

The daily cap resets nightly, so will check tomorrow since I still cannot view logs yet 😞

[sam_stachura]

So still cannot view logs today:

- cluster management > license: 11.05 G/11.01 GiB, Avg. daily log volume, No overage allowed
- (same as before I turned off lots of logging sources a few days ago)
- cluster management for the env shows: Log monitoring storage (using) 3.01 GiB, (quota)4 GiB
- settings > log monitoring > data storage: Based on current log data storage usage 0.856 GB/Day

- set the monthly quota to 30gb higher than what is used this month in that env (was lower) a few hrs ago, so should have synced by now.

How long do I have to wait for the logging quota to clear? A month?

[sam_stachura]

Still cannot view logs due to the overage, daily logging volume is still low for that env:

------------
Estimated yearly averages

Based on current log data storage usage

1.386 GB/Day

-----------
So guess the monthly cap is still in effect, wish there was a way to clear that and start fresh for the month...