Dynatrace log monitoring needs clarification - It's not clear how it is working today

[larry_roberts]

We recently purchased 30GB of Log analytics for a few logs of a specific team. This was our first time using this. Once it was enabled, the 30GB was very quickly maxed out as you can see below.

After seeing this and then attempting to figure out why that would be the case, I believe I found what did it. Before licensing the 30GB of “Log analytics”, we had Dynatrace in all tenants set to “Include all logs” and have had it that way since day one of bringing Dynatrace in. This has been useful as once the OneAgent is installed, it auto detects these logs and shows them right in the interface where they can be queried, setup for specific events, and more.

However, once we purchased the 30GB of “Log analytics”, it appears to have taken the above setting into account which obviously maxed it out quickly. I have since changed the settings from “Include all logs” to “Include the following logs” and only selected the log which we purchased the 30GB for recently.

Something just does not make sense

I am very confused on how we went from having “Include all logs” selected and letting us define events around them within Dynatrace without the need for the “Log analytics” to now what seems that every log has a licensing cost in terms of size and storage because we enabled the 30GB for “Log analytics” which was only purchased in order to keep some specific logs a bit longer in terms of storage.

We would like better detailed clarification on exactly what the “Log analytics” licensing provides which to my knowledge is only log storage.

Also we would like to know if the “Log file storage retention” which has options ranging from 5 to 90 days is a global setting or can it be set per log file monitored? From everything I have looked at, it appears to be only a global one size fits all setting.

Another concern that is not helping and only adds to the confusion is some of the terminology

The term “Custom metrics” appears to mean two different things depending on what you are looking at both in Dynatrace and in the online documentation which is very confusing.

We were able to set up rules for detecting something specific within a log which is very convenient and simple to do and there was no additional licensing needed. It appears now that defining a “Detection rule” for logs is being deprecated and will need to be replaced by “Custom metrics”, but it is unclear what Dynatrace means by the title “Custom metrics” here because again, this same terminology is being used for two different things.

Is Dynatrace referring to “Custom metrics” as in the licensing pool “Custom metrics” or is it referring to the configuration just in the title as seen on the page Custom metrics for log monitoring, meaning this does not consume “Custom metrics” licensing?

The terminology I feel needs work

There is “Custom metrics” on the account page as in the licensing pool and then there is “Custom metrics” again within the documentation referring to a specific kind of configuration on logs with one possibly having nothing to do with the other.

Would it be correct to say that when licensing “Log analytics” you are actually referring only to the size and retention of log data and nothing more? If true, again as a customer I struggle with the terminology being used. I would expect this to be titled something like “Log size and retention” or something similar vs “Log analytics”.

The terminology again is a pain point

If you look at the Calculate Dynatrace monitoring consumption page, it refers to “Log Monitoring”. However, if you look at the licensing area of the account interface, it refers to “Log analytics” which I think is really referring to the size and retention of log data and nothing more.

So really my questions boil down to this…

• When and why did we go from being able to “Include all logs” and then setup rules for custom log events at no additional charge to now every log selected is counted against this “Log analytics”?

• What exactly is “Log analytics” providing outside of just storage of logs?
• Does this now mean that unless you have selected specific logs in the “Log sources and storage” setting that those logs are now 100% ignored and not taken into consideration for analysis?

Thank you

[Julius_Loman]

Have you seen these two blog posts from last year? I believe they have answers to most of your questions:

What is log monitoring? See why it matters in a hyperscale world

Log Management | Log Analytics

Changes were mostly introduced with 174 and 178. Next big change will be probably 190 when the old Log events will no longer work.

My observations:

• Log metrics are the new “Log events”, they consume custom metrics license but are more powerful
• There is a free tier of few GBs of logs
• To be able to monitor logs, you will have to set up the log storage (applies if you are on Managed)
• Basically now log monitoring consumes two licenses - log analytics (the “shipping” and central storage) and custom metrics (log “events” / merics). Previously it was just Log analytics license if I remember correctly.

[michal_dec]

Hey @Larry R. - Michał Dec here (Log Monitoring Product Manager). Thank you very much for your feedback! Let me digest into your observations and I will provide some comments and plans for improvements shortly.

[larry_roberts]

Yeah even going through all that, I feel it is still a huge mess. I can’t help but feel like Dynatrace is taking something that we were getting with no additional cost and twisting it into a mold that calls for more licensing needs.

So basically it sounds like without “Log analytics”

• Removal of current ability to define detection rules for custom log events which came at no additional cost
• The ability to query across multiple log files will be limited to centralized (monitored) logs

With “Log analytics”

• You pay for log transfer
• You pay for log retention
• You pay for custom metrics if you want to do anything with said logs

If it is true that log monitoring in Dynatrace will now consume two licenses, then they have doomed this approach already in terms of trying to get customs off Splunk.

Splunk charges for the amount of data transfer - PERIOD

Splunk does not charge us also for:

• Retention periods (equates to Dynatrace log transfer / log storage)
• The data within the logs that we build things around (equates to Dynatrace “Custom metrics”)

What is wrong with this picture? I must be missing something here…

[michal_dec]

Hey @Larry R., sorry for late reply. Let me comment and provide some insights - I hope you will find this useful.

1. 30GB license and “Include all logs”: OneAgent out of the box discover which process writes to specific log file. Content of this log files can be analyzed in on-demand mode. This mode have certain limitations: “You can examine a maximum of 500 MB of log data. You can examine the log files for only the past seven days.” (please see our docs). For many different reasons customer might want to send content of log files to central storage. Log Source and configuration allows customer to define content of which files should be streamed to central storage. In this case “Include All logs” was probably configured while buying new license for Log Monitoring - this caused quick consumption of this new license. I fully understand that this might be cumbersome and problematic for customers and we need to make user flow more robust here and this is action point for me. We provide a lot of value with log file auto-discovery but simultaneously we need to make configuration of log sources more generic.
2. Log events and custom metrics: Initially Dynatrace allowed customer to configure event creation based on monitored log files on host level (no need to stream to central storage, no additional cost involved). During preview phase of this functionality we received a lot of feedback from our customers. Current solution does not allow to create metrics/event out of logs coming from many different sources: logs from cloud services, logs delivered by different log data shippers (fluentD), logs coming from infrastructure components (logs generated by CDN). Also there was significant concern related to performance - many customer is willing to move this part of business logic from their infrastructure (be it agents on hosts) to Dynatarace cluster. That’s why we are moving away from metrics generation on agent side to server side metrics. In mid term we plan to allow customers to calculate metrics server side without need to store log data in central storage. Please note that additional pricing applies for consuming metrics generated from log messages because - it’s different data handler that allows to store information for long time period.
3. Terminology - you are right. We recently adjusted naming convention and changed from Log Analytics to Log Monitoring. We will updated terminology in product.
4. Pricing framework - I believe there is some misunderstanding right now. It seems we need to make it more clear. I’m happy to discuss this on call.
   1. Log Monitoring in Managed: Dynatrace charge for log data ingestion and custom metrics calculated out of this log data.
   2. Log Monitoring in SaaS: Dynatrace charge for amount of log data ingested and retention time of ingested data. Cost of log data storage in SaaS is on Dynatrace and that’s why this factor need to be considered in SaaS pricing framework. We also charge for metrics generation in SaaS.

Once again thank you for very valuable feedback! Rest assured that we are right now heavily redesigning our Log Monitoring capability to better fit customer needs.

[larry_roberts]

Good morning @Michal D.,

Thank you for the quick responses and details. I have outlined my responses below.

For many different reasons customer might want to send content of log files to central storage. Log Source and configuration allows customer to define content of which files should be streamed to central storage.

But there are also cases where they don’t as I point out below. Dyantrace is essentially taking the choice away and forcing customers down the road of additional licensing costs or to discontinue use of log monitoring in Dynatrace and replace with another solution.

Log events and custom metrics: Initially Dynatrace allowed customer to configure event creation based on monitored log files on host level (no need to stream to central storage, no additional cost involved).

This is exactly my point. There are still customers like us where this was sufficient. This is my biggest pain point. Dynatrace has elected to now remove this from customers forcing them to either signup for the additional licensing consumption in more than one licensing pool or abandon Dynatrace completely for log monitoring and use another solution.

Current solution does not allow to create metrics/event out of logs coming from many different sources: logs from cloud services, logs delivered by different log data shippers (fluentD), logs coming from infrastructure components (logs generated by CDN).

I would agree. But there are also cases where what was currently being supplied was sufficient for some customers and that is now being taken away. We are being left with no choice but to either really sign up for all the additional licensing or take another route for logs.

Many customer is willing to move this part of business logic from their infrastructure (be it agents on hosts) to Dynatarace cluster. That’s why we are moving away from metrics generation on agent side to server side metrics.

I would challenge that. I think once the cost is fully understood that might not be the case with all customers. The issue I have here is that the choice between metric generation on the agent side is being taken away from customers where nothing more is needed.

In mid term we plan to allow customers to calculate metrics server side without need to store log data in central storage. Please note that additional pricing applies for consuming metrics generated from log messages because - it’s different data handler that allows to store information for long time period.

That is great news to hear that we will have the choice to not use central storage IF that is continued as a choice for customers.

Log Monitoring in SaaS: Dynatrace charge for amount of log data ingested and retention time of ingested data. Cost of log data storage in SaaS is on Dynatrace and that’s why this factor need to be considered in SaaS pricing framework. We also charge for metrics generation in SaaS.

I can fully understand the need to require additional cost for storage and retention using SaaS, but where I think Dynatrace is taking the wrong direction and in my opinion double dipping a bit into licensing costs is around considering this custom metrics.

Again I must point out that Splunk ONLY charges for ingestion and NOT what you do with the data being ingested. I only mention this because of what was talked about during Perform 2020. In order to create an incentive to move to Dynatrace for what is currently in Splunk, it must be better AND cheaper. Currently, in my opinion Dynatrace has made the potential to move more expensive. That is a show stopper for us.

---

I do believe a call would be good to have in order to ensure that we have a very firm understanding of where Dynatrace is going with log monitoring, the true cost, options, and more. Please feel free to reach out to me via email so that we can arrange the call.

Thank you

[larry_roberts]

I think what would be a huge help is if there was just a single matrix a customer can look at that shows a clear, simple, comparison focused on the log monitoring within Dynatrace.

Something like this…

[Julius_Loman]

I agree with you on all points. Especially when Dynatrace is compared to tools like Splunk or Elasticsearch. If you do not need much information from logs, then Dynatrace can be quite costly in for monitoring log files for a few predefined log messages. The costs, however, may highly depend on your use cases.

I’ve never been a huge fan of monitoring based on logs, especially if you have tracing and code-level instrumentation. I always recommend fetching the information by other methods if that is possible. Sometimes the logs are the only source for some information that may be difficult to obtain by other means.

There is one big advantage of having log monitoring with Dynatrace. It is having the data within Dynatrace and tied to existing entities. It’s not another tool for log analytics completely unrelated to Dynatrace.

[Julius_Loman]

I guess it is like this, @Michal D., please correct me if I’m wrong:

Use case	Consumes log monitoring license	Consumes Custom metrics license
Viewing up logs in UI up to 7day old, smaller than 500MB	no	no
Centrally storing and viewing logs in UI with up to 90 days retention time	yes	no
Monitoring logs for messages (less than 5GB logs/day average ingest)	no	yes
Monitoring logs for messages (above 5GB/logs/day average ingest)	yes	yes

[larry_roberts]

Agreed, I am not a fan of log monitoring myself. Thankfully we have only had a single request for it in Dynatrace which is what took me down this path when their purchased 30GB quick was consumed. Unless there is a change, I will be recommending to that team that they move that into Splunk and we then disable “Log analytics”.

I do agree that having the data within Dynatrace is an advantage. In fact I would say that is really the #1advantage to it all. For our use case and I would imagine many others as well, just to catch specific log entires though, the licensing model is extreme.

You are basically paying for…

• The size of the logs (Log analytics licensing)
• The retention of said logs (Log analytics licensing)
• The ability to do anything with those said logs (Custom metrics)

The third bullet is the one I really have a problem with. Yes, charge for size and retention which makes perfect sense, but I really do not feel that just because you define a rule to look for something in a log within Dynatrace that you should need to consume custom metrics. While it may seem small, I can see how it could potentially add up very fast.

Then to add to the problem, Dynatrace is effectively taking way the abilities that came at no extra cost and were a good solution for catching log entries here and there.

Also, it would be hard to even approach trying to justify the additional consumption to leadership if we wanted to with the terminology crossing, documentation far from clear, and lack of controls as is for what users can do under settings at a granular level without disabling settings completely.

[larry_roberts]

This I like! Straight forward and easy to understand in terms of a matrix.

Also, this area of Dynatrace needs to be reworked. The first part is exactly what burned us as soon as we enabled “Log analytics” for the 30GB. If Dynatrace is going to continue with this new licensing model around logs, then this should NOT be storing all logs by default. Also this area need to be much more self explanatory.

Looking at this screen alone generates many questions that are not obvious…

• Does this mean if I don’t have a log or logs selected here that I will not be able to view them?
• Is this just intended to control what gets uploaded to Dynatrace and stored or is there more to it?
• What exactly does it mean to “include” or “exclude” logs on this screen? Whats the benefits or negatives of each choice?
• Whats the difference between what I see on this screen and the logs I see listed when looking at hosts and other things?

Etc… etc…

[alexandre_marl2]

Hello @Július L.

Even after reading 5 times the documentation/blogs it is still not clear in my mind how logs are billed !I think dynatrace need to clarify how it works for an easy point of view.

From my understanding for each subscription we get 100 custom metrcis ? It is the same as the once you speak above ? So we can use them for logs ?

For SaaS Version, we get 5Go of ingestion by year (so 5000/365=13 Mo a day ?) ? Is it correct ? Do we get some storage in the same time ? (As we can buy some storage).

Thanks,

Alexandre

[alexandre_marl2]

In the documentation it says : "Each Dynatrace environment (SaaS or Managed) comes with 5 GB of log data storage, per year at no cost to you. ".

They don’t speak about ingestion at all for the free part but in the blog post it speaks about the 5Go of transfer (so ingestion).

Ref : https://www.dynatrace.com/support/help/reference/monitoring-consumption-calculation/#expand-84free-tier-of-log-monitoring

What is log monitoring? See why it matters in a hyperscale world

I am lost

[Julius_Loman]

Does this mean if I don’t have a log or logs selected here that I will not be able to view them?

I think you can still view them, but you are limited to the 500MB/7days retention as they are “pulled” from the host every time you are viewing them.

Is this just intended to control what gets uploaded to Dynatrace and stored or is there more to it?

Yes, it is I believe.

What exactly does it mean to “include” or “exclude” logs on this screen? Whats the benefits or negatives of each choice?

Depends on the combo box set above. You have the “Include following logs”. So you will have to specify each log to be collected.

Whats the difference between what I see on this screen and the logs I see listed when looking at hosts and other things?

This should be just a unified view across hosts, settings and logs should match. Does it?

[larry_roberts]

I could not agree more

From my understanding for each subscription we get 100 custom metrcis ? It is the same as the once you speak above ? So we can use them for logs ?

Correct and correct. It is the “Custom metrics” licensing pool. From what I have been able to tell and not from looking at any of the documentation is that the formula goes something like this for just the custom metrics part alone of log monitoring…

“Log metric” multiplied by “total logs selected”

For example, if I create a “Log metric” which is really nothing more than a rule saying to catch some error code in a log that lives on 20 servers, that would be 20 “Custom metrics” consumed. Not including the size of the log or retention which both “Log analytics” on top of the “Custom metrics”.

That would be for just 1 metric which is why I say this could get costly very fast.

For SaaS Version, we get 5GB of ingestion by year (so 5000/365=13 Mo a day ?) ? Is it correct ? Do we get some storage in the same time ? (As we can buy some storage).

I am still trying to figure this one all out myself.

[alexandre_marl2]

Thanks @Larry R. for the clarification.

Indeed it is crazy if counts one custom metrics by server ! It will cost a lot !

I think Dynatrace staff need to clarify in a blog post how it works and try to simplify if they want customer use it.

[Julius_Loman]

It depends, but it is definitely more than before. Custom metrics in the price list are not exactly cheap and with dimension included it can really explode.

Having the ability to just define a few log messages in a free tier as before was satisfying for many customers.

[larry_roberts]

Spot on @Július L. with both points. Great discussion around this. With Dynatrace being very customer focused, I trust this will generate discussions internally and we will see some changes that benefits both Dynatrace and it’s customers. I love the Dynatrace product, but unfortunately I can not recommend our company utilize what is currently titled as “Log analytics” in Dynatrace with the current licensing model and all the confusion around it.

[michal_dec]

Hey @Larry R., @Július L., @Alexandre M. - thanks for all this insights and questions. We are now reviewing pricing framework for custom metrics and log monitoring and for sure your feedback will be taken into account.

[larry_roberts]

That is great news @Michal D. I want you and everyone at Dynatrace to know how much it is appreciated that you are a company that truly does focus on listening to its customers. I look forward to hearing about changes to this soon.