We recently purchased 30GB of Log analytics for a few logs of a specific team. This was our first time using this. Once it was enabled, the 30GB was very quickly maxed out as you can see below.
After seeing this and then attempting to figure out why that would be the case, I believe I found what did it. Before licensing the 30GB of "Log analytics", we had Dynatrace in all tenants set to "Include all logs" and have had it that way since day one of bringing Dynatrace in. This has been useful as once the OneAgent is installed, it auto detects these logs and shows them right in the interface where they can be queried, setup for specific events, and more.
However, once we purchased the 30GB of "Log analytics", it appears to have taken the above setting into account which obviously maxed it out quickly. I have since changed the settings from "Include all logs" to "Include the following logs" and only selected the log which we purchased the 30GB for recently.
Something just does not make sense
I am very confused on how we went from having "Include all logs" selected and letting us define events around them within Dynatrace without the need for the "Log analytics" to now what seems that every log has a licensing cost in terms of size and storage because we enabled the 30GB for "Log analytics" which was only purchased in order to keep some specific logs a bit longer in terms of storage.
We would like better detailed clarification on exactly what the "Log analytics" licensing provides which to my knowledge is only log storage.
Also we would like to know if the "Log file storage retention" which has options ranging from 5 to 90 days is a global setting or can it be set per log file monitored? From everything I have looked at, it appears to be only a global one size fits all setting.
Another concern that is not helping and only adds to the confusion is some of the terminology
The term "Custom metrics" appears to mean two different things depending on what you are looking at both in Dynatrace and in the online documentation which is very confusing.
We were able to set up rules for detecting something specific within a log which is very convenient and simple to do and there was no additional licensing needed. It appears now that defining a "Detection rule" for logs is being deprecated and will need to be replaced by "Custom metrics", but it is unclear what Dynatrace means by the title "Custom metrics" here because again, this same terminology is being used for two different things.
Is Dynatrace referring to "Custom metrics" as in the licensing pool "Custom metrics" or is it referring to the configuration just in the title as seen on the page Custom metrics for log monitoring, meaning this does not consume "Custom metrics" licensing?
The terminology I feel needs work
There is "Custom metrics" on the account page as in the licensing pool and then there is "Custom metrics" again within the documentation referring to a specific kind of configuration on logs with one possibly having nothing to do with the other.
Would it be correct to say that when licensing "Log analytics" you are actually referring only to the size and retention of log data and nothing more? If true, again as a customer I struggle with the terminology being used. I would expect this to be titled something like "Log size and retention" or something similar vs "Log analytics".
The terminology again is a pain point
If you look at the Calculate Dynatrace monitoring consumption page, it refers to "Log Monitoring". However, if you look at the licensing area of the account interface, it refers to "Log analytics" which I think is really referring to the size and retention of log data and nothing more.
So really my questions boil down to this....
Have you seen these two blog posts from last year? I believe they have answers to most of your questions:
Changes were mostly introduced with 174 and 178. Next big change will be probably 190 when the old Log events will no longer work.
Yeah even going through all that, I feel it is still a huge mess. I can't help but feel like Dynatrace is taking something that we were getting with no additional cost and twisting it into a mold that calls for more licensing needs.
So basically it sounds like without "Log analytics"
With "Log analytics"
If it is true that log monitoring in Dynatrace will now consume two licenses, then they have doomed this approach already in terms of trying to get customs off Splunk.
Splunk charges for the amount of data transfer - PERIOD
Splunk does not charge us also for:
What is wrong with this picture? I must be missing something here...
Hey @Larry R., sorry for late reply. Let me comment and provide some insights - I hope you will find this useful.
Once again thank you for very valuable feedback! Rest assured that we are right now heavily redesigning our Log Monitoring capability to better fit customer needs.
Good morning @Michal D.,
Thank you for the quick responses and details. I have outlined my responses below.
For many different reasons customer might want to send content of log files to central storage. Log Source and configuration allows customer to define content of which files should be streamed to central storage.
But there are also cases where they don't as I point out below. Dyantrace is essentially taking the choice away and forcing customers down the road of additional licensing costs or to discontinue use of log monitoring in Dynatrace and replace with another solution.
Log events and custom metrics: Initially Dynatrace allowed customer to configure event creation based on monitored log files on host level (no need to stream to central storage, no additional cost involved).
This is exactly my point. There are still customers like us where this was sufficient. This is my biggest pain point. Dynatrace has elected to now remove this from customers forcing them to either signup for the additional licensing consumption in more than one licensing pool or abandon Dynatrace completely for log monitoring and use another solution.
Current solution does not allow to create metrics/event out of logs coming from many different sources: logs from cloud services, logs delivered by different log data shippers (fluentD), logs coming from infrastructure components (logs generated by CDN).
I would agree. But there are also cases where what was currently being supplied was sufficient for some customers and that is now being taken away. We are being left with no choice but to either really sign up for all the additional licensing or take another route for logs.
Many customer is willing to move this part of business logic from their infrastructure (be it agents on hosts) to Dynatarace cluster. That's why we are moving away from metrics generation on agent side to server side metrics.
I would challenge that. I think once the cost is fully understood that might not be the case with all customers. The issue I have here is that the choice between metric generation on the agent side is being taken away from customers where nothing more is needed.
In mid term we plan to allow customers to calculate metrics server side without need to store log data in central storage. Please note that additional pricing applies for consuming metrics generated from log messages because - it's different data handler that allows to store information for long time period.
That is great news to hear that we will have the choice to not use central storage IF that is continued as a choice for customers.
Log Monitoring in SaaS: Dynatrace charge for amount of log data ingested and retention time of ingested data. Cost of log data storage in SaaS is on Dynatrace and that's why this factor need to be considered in SaaS pricing framework. We also charge for metrics generation in SaaS.
I can fully understand the need to require additional cost for storage and retention using SaaS, but where I think Dynatrace is taking the wrong direction and in my opinion double dipping a bit into licensing costs is around considering this custom metrics.
Again I must point out that Splunk ONLY charges for ingestion and NOT what you do with the data being ingested. I only mention this because of what was talked about during Perform 2020. In order to create an incentive to move to Dynatrace for what is currently in Splunk, it must be better AND cheaper. Currently, in my opinion Dynatrace has made the potential to move more expensive. That is a show stopper for us.
I do believe a call would be good to have in order to ensure that we have a very firm understanding of where Dynatrace is going with log monitoring, the true cost, options, and more. Please feel free to reach out to me via email so that we can arrange the call.
I agree with you on all points. Especially when Dynatrace is compared to tools like Splunk or Elasticsearch. If you do not need much information from logs, then Dynatrace can be quite costly in for monitoring log files for a few predefined log messages. The costs, however, may highly depend on your use cases.
I've never been a huge fan of monitoring based on logs, especially if you have tracing and code-level instrumentation. I always recommend fetching the information by other methods if that is possible. Sometimes the logs are the only source for some information that may be difficult to obtain by other means.
There is one big advantage of having log monitoring with Dynatrace. It is having the data within Dynatrace and tied to existing entities. It's not another tool for log analytics completely unrelated to Dynatrace.
Agreed, I am not a fan of log monitoring myself. Thankfully we have only had a single request for it in Dynatrace which is what took me down this path when their purchased 30GB quick was consumed. Unless there is a change, I will be recommending to that team that they move that into Splunk and we then disable "Log analytics".
I do agree that having the data within Dynatrace is an advantage. In fact I would say that is really the #1advantage to it all. For our use case and I would imagine many others as well, just to catch specific log entires though, the licensing model is extreme.
You are basically paying for...
The third bullet is the one I really have a problem with. Yes, charge for size and retention which makes perfect sense, but I really do not feel that just because you define a rule to look for something in a log within Dynatrace that you should need to consume custom metrics. While it may seem small, I can see how it could potentially add up very fast.
Then to add to the problem, Dynatrace is effectively taking way the abilities that came at no extra cost and were a good solution for catching log entries here and there.
Also, it would be hard to even approach trying to justify the additional consumption to leadership if we wanted to with the terminology crossing, documentation far from clear, and lack of controls as is for what users can do under settings at a granular level without disabling settings completely.
I think what would be a huge help is if there was just a single matrix a customer can look at that shows a clear, simple, comparison focused on the log monitoring within Dynatrace.
Something like this...
I guess it is like this, @Michal D., please correct me if I'm wrong:
|Use case||Consumes log monitoring license||Consumes Custom metrics license|
|Viewing up logs in UI |
up to 7day old, smaller than 500MB
|Centrally storing and viewing logs in UI with up to 90 days retention time||yes||no|
|Monitoring logs for messages (less than 5GB logs/day average ingest)||no||yes|
|Monitoring logs for messages (above 5GB/logs/day average ingest)||yes||yes|
This I like! Straight forward and easy to understand in terms of a matrix.
Also, this area of Dynatrace needs to be reworked. The first part is exactly what burned us as soon as we enabled "Log analytics" for the 30GB. If Dynatrace is going to continue with this new licensing model around logs, then this should NOT be storing all logs by default. Also this area need to be much more self explanatory.
Looking at this screen alone generates many questions that are not obvious...
Does this mean if I don't have a log or logs selected here that I will not be able to view them?
I think you can still view them, but you are limited to the 500MB/7days retention as they are "pulled" from the host every time you are viewing them.
Is this just intended to control what gets uploaded to Dynatrace and stored or is there more to it?
Yes, it is I believe.
What exactly does it mean to "include" or "exclude" logs on this screen? Whats the benefits or negatives of each choice?
Depends on the combo box set above. You have the "Include following logs". So you will have to specify each log to be collected.
Whats the difference between what I see on this screen and the logs I see listed when looking at hosts and other things?
This should be just a unified view across hosts, settings and logs should match. Does it?
Even after reading 5 times the documentation/blogs it is still not clear in my mind how logs are billed !I think dynatrace need to clarify how it works for an easy point of view.
From my understanding for each subscription we get 100 custom metrcis ? It is the same as the once you speak above ? So we can use them for logs ?
For SaaS Version, we get 5Go of ingestion by year (so 5000/365=13 Mo a day ?) ? Is it correct ? Do we get some storage in the same time ? (As we can buy some storage).
In the documentation it says : "Each Dynatrace environment (SaaS or Managed) comes with 5 GB of log data storage, per year at no cost to you. ".
They don't speak about ingestion at all for the free part but in the blog post it speaks about the 5Go of transfer (so ingestion).
I am lost 😕
I could not agree more
From my understanding for each subscription we get 100 custom metrcis ? It is the same as the once you speak above ? So we can use them for logs ?
Correct and correct. It is the "Custom metrics" licensing pool. From what I have been able to tell and not from looking at any of the documentation is that the formula goes something like this for just the custom metrics part alone of log monitoring...
"Log metric" multiplied by "total logs selected"
For example, if I create a "Log metric" which is really nothing more than a rule saying to catch some error code in a log that lives on 20 servers, that would be 20 "Custom metrics" consumed. Not including the size of the log or retention which both "Log analytics" on top of the "Custom metrics".
That would be for just 1 metric which is why I say this could get costly very fast.
For SaaS Version, we get 5GB of ingestion by year (so 5000/365=13 Mo a day ?) ? Is it correct ? Do we get some storage in the same time ? (As we can buy some storage).
I am still trying to figure this one all out myself.
Spot on @Július L. with both points. Great discussion around this. With Dynatrace being very customer focused, I trust this will generate discussions internally and we will see some changes that benefits both Dynatrace and it's customers. I love the Dynatrace product, but unfortunately I can not recommend our company utilize what is currently titled as "Log analytics" in Dynatrace with the current licensing model and all the confusion around it.