topic Re: Log Monitoring v2 - What's your experience? in Log Analytics

Log Monitoring v2 - What's your experience?

ct_27 — Fri, 08 Oct 2021 02:41:30 GMT

Who here has switched to Log Monitoring v2 and do you like it?

Did you lose functionality?

Do you wish you waited longer? (since you can't revert back)

Any advice for those of us thinking/waiting to switch?

I have one major concern holding us up from switching.

On-demand access to log files on monitored host

v1: Yes

v2: No

Our Infrastructure and Application teams heavily use the ability to read the past 500MB or 7 days of Windows system logs and such files by simply accessing them via Dynatrace. This has a been a huge help in troubleshooting issues very quickly (MTTR - I big sales point from Dynatrace). I'm concerned this ability is going to go away. We're a small company with only 1,500 or so servers in Dynatrace but every one of those servers have a Windows System, Windows Event, Windows Security or var/log/messages that is written to frequently. I can't image what the additional charge would be if DT changes their policy on this when we're forced to switch to V2. (similar to when they switched Cloud to DDU, we turned it all off and is no longer seen as a DT value add functionality).

Re: Log Monitoring v2 - What's your experience?

anders_lundin — Wed, 27 Oct 2021 10:57:41 GMT

I am working with an endcustomer that has the exact same concern. They are not going to switch to v2 because they dont want to lose the on-demand log and they are not ready to invest in quite high amounts of DDUs that will be required to store the logdata in Dynatrace instead.

Re: Log Monitoring v2 - What's your experience?

ct_27 — Mon, 01 Nov 2021 12:21:48 GMT

Dynatrace any feedback on this? Feel like we might have gotten setup here, first Logs were switch to be DDU consumers then it was changed that the original expectations of logs would now be charged. We signed only a year ago, quite a drastic change because our DDU expectations and estimates from original signing our now blown out of the water.

A competitor did something similar and now receives comments such as "logging (bills) without limits!" from its customers. Are we heading the same direction?

Re: Log Monitoring v2 - What's your experience?

Andy_Tommo — Mon, 01 Nov 2021 15:38:53 GMT

We have been running v2 for around 6 months and from the experience we have had, I wouldn't recommend going anywhere near v2 until Dynatrace have ironed out all of the teething problems it introduced for us. Although some of the v1 functionality is still missing within v2, the overall experience via the UI is much better than v1. With v2, you also get the ability to use generic log ingestion via the API which includes the ingestion of Azure logs.

Things to watch out for as part of the migration:

Take a copy of your metrics and log events as this will be totally lost during the migration with no way to get it back again. You will also have to configure the metrics from scratch again in v2.
Ensure all of your agents are running 1.227 or higher or you will run into the multiple bugs we have already exposed.
Be prepared to lose all of your v1 ingested logs as v2 uses different storage and non of the v1 logs will be migrated.
Be prepared to lose all on demand log access as log analytics will only be available through ingested logs only. There are no plans to reintroduce this functionality in v2, but I believe they are planning to add log events without the need to ingest.

Although my experience of v2 log analytics hasn't been brilliant so far, I do believe Dynatrace are moving in the right direction in terms of their road map. My hope is that they will eventually be in a position to properly compete against other log analytics tools such as Splunk.

Re: Log Monitoring v2 - What's your experience?

ct_27 — Tue, 25 Jan 2022 17:58:46 GMT

Adding two more items per post (https://community.dynatrace.com/t5/Dynatrace-Open-Q-A/Log-Monitoring-v2-Dynatrace-Managed/td-p/173071)

- I miss the summarize feature - and other things (every log line is now a message - even though the clearly are one log message).
- I hate the fact that I had to upgrade all nodes to 64GB RAM, and this was not mentioned in the documentation.

Re: Log Monitoring v2 - What's your experience?

ct_27 — Tue, 25 Jan 2022 18:03:20 GMT

- Logging v2 missing functionality previously in v1 and may now require you to setup additional tools.

- Logging v2 is splitting multiple line logs into multiple messages making it difficult to analyze

https://community.dynatrace.com/t5/Dynatrace-Open-Q-A/Log-Monitoring-v2-Multi-Line-Log-Entries/m-p/179247

Re: Log Monitoring v2 - What's your experience?

ct_27 — Tue, 25 Jan 2022 18:08:35 GMT

https://community.dynatrace.com/t5/Dynatrace-product-ideas/Bring-back-log-pulling-downloading-in-Log-Monitoring-v2/idi-p/178566

https://community.dynatrace.com/t5/Dynatrace-product-ideas/Log-on-demands-on-log-monitoring-v2/idi-p/176475

Two more customer complaints that V2 removes the ability to download/pull logs from hosts.

Re: Log Monitoring v2 - What's your experience?

ct_27 — Tue, 25 Jan 2022 18:07:28 GMT

https://community.dynatrace.com/t5/Dynatrace-Open-Q-A/Field-Extraction-in-Log-Monitoring-v2/m-p/169775

Field extraction breaks in v2. Impact to graphs that used to work with v1.

Re: Log Monitoring v2 - What's your experience?

ct_27 — Tue, 25 Jan 2022 18:40:32 GMT

agreed with your experience. We're looking to now depart DT as our logging solution. We have other internal options that are now proving to be cheaper, more intuitive, easier to use, and more powerful. Their recent attempts to streamline problems, traces, metrics, and logs is at the moment too little too late.

Re: Log Monitoring v2 - What's your experience?

Enrico_F — Thu, 29 Sep 2022 14:23:14 GMT

We are considering switching to v2 mainly because of the K8s event monitoring feature - which seems to require it (sadly)...

We are currently on Managed and our 3 nodes already have 96GB of RAM and 8 CPU's as well as attached dedicated Elasticsearch storage of 100GB each.

Besides pros/cons that have already been mentioned in this topic there are a couple more things I would like to confirm/understand before we decide to take the plunge or not:

I understand the shared NFS log v1 monitoring storage will no longer be required after the switch since Elasticsearch will be the new (replicated) log storage location, correct?
I understand there is no tool/process to automatically migrate existing log v1 monitoring config (log events, log metrics, log sources) over to v2 which means every aspect of log monitoring will have to be re-configured manually from scratch in v2, correct?
Can I use the rate of ingested log records with v1 (obtained by dividing the DDU log monitoring consumption metric by 0.0005) to estimate the equivalent for log events in v2? In other words: One log "record" in v1 will be equivalent to one log "event" in v2 in the context of any capacity considerations with v2, correct?
With K8s event monitoring enabled I assume only events matching the configured event filters in the K8s monitoring settings will be counted as log events, correct?
Transitioning to v2 is final and there is no going back to v1 after that, correct?

I would be grateful for any input.

Re: Log Monitoring v2 - What's your experience?

Mizső — Thu, 29 Sep 2022 21:51:37 GMT

Hi @Enrico_F,

There are pros and cons. Maybe I can't anwer for your all questions.

It is correct. I realized it two weeks ago, at one of my client. I raied a support ticket about it because the v2 worked without the NFS. :-). They confimed logs are ingested to directly to ElasticSearch.
It is correct. I had to start the configuration from the scratch.
DDU consumption: I have just check it at one customer for the last 7 day period. Because of DDU consumption limited logs are ingested and Kubernetes events (They have the yearly base 200k DDU). Last 7 days the ingested log lines were 1.45 mil and the DDU consumption was 728, so 0.0005 / log lines is correct, regarding Kubernetes event there were 15.9 k and the DDU consumption was 15.9, so the 0.001 / events is correct.
I do not have experence on it. See attached a relevant Kubernetes event configuration in this environment. All of them swithed on for this evironment. I also attached the screenshot from the LOG viewer GUI the list of events with this configuration (90 days). During this check I found an event which was found at he pod level but not at in the LOG viewer GUI (workload spec chage). Maybe because of my configuration.
I think it is also correct.

There are some advanage of the v2, I thnik the speed of filtering is quite fast becasue of the Elastic, I love the view trace frunction when relevant logs are connected to the trace (at distributed traces), SRE and developers find the required logs very quickly.

I hope it helps.

Br, Mizső

Re: Log Monitoring v2 - What's your experience?

Enrico_F — Fri, 30 Sep 2022 10:52:31 GMT

Thanks a lot @Mizső.

Regarding #2: I just found this page in the documentation:

https://www.dynatrace.com/support/help/shortlink/log-monitoring-log-storage#migration-to-the-new-storage-configuration

So it seems that starting with 1.252 any configured log sources under v1 are automatically migrated to v2 as "matchers" under either the host or tenant scopes and therefore won't have to be re-added manually... Would be great if anybody could confirm this.

But still, that means we would have to re-configure log metrics and log events from scratch in order to get the same log monitoring with v2.... Considering the amount of settings involved in our current environment this looks like a show-stopper to me....

Re: Log Monitoring v2 - What's your experience?

Enrico_F — Wed, 05 Oct 2022 07:51:08 GMT

I just received further feedback from support regarding #2: With 1.252+ there will be two log storage configuration options - old and new. While the new storage option (as outlined here) will be available as an "opt-in" the old log sources config will continue to work as the default - so apparently no manual migration will be needed to re-add log sources to monitoring after enabling LMv2.

I suspect the new storage config is designed to tie-in with the brand new Grail™ "data lakehouse" offering...

Re: Log Monitoring v2 - What's your experience?

Mizső — Wed, 05 Oct 2022 09:19:14 GMT

Hi @Enrico_F

Thanks for sharing this info.

Br, Mizső