topic Re: Looking to move from managed single node cluster to 2 nodes...addressing ALR in Dynatrace Managed Q&A

Moving from Dynatrace Managed single node cluster to 2 nodes and addressing ALR

runatyr — Thu, 25 May 2023 07:32:21 GMT

I am looking for assistance and clarification regarding two things

I want to have adaptive load reduction stop occurring

I am trying to figure out what the traffic volume or trigger is causing ALR and then increase the necessary resources on the managed cluster node

I currently have a single managed cluster node and want to increase this to have two managed cluster nodes to create a balance of traffic load distribution and provide a layer of redundancy

My concern is that currently I have a single node located at one Data Center

Is it impactful or less desirable to leverage a network practice known as OTV between the cluster nodes?

This is a network protocol where both nodes reside in the same VLAN,

but the vlan is stretched and routed through IP. Usually to a different physical location.

Some latency occurs

I am curious what latency tolerances are available between the members nodes of a managed Cluster

Are there any commands or statistics the a managed cluster can provide regarding performance between the nodes?

Please advise and thanks!

Re: Moving from Dynatrace Managed single node cluster to 2 nodes and addressing ALR

Radoslaw_Szulgo — Thu, 13 Mar 2025 08:44:41 GMT

You have to ensure near-zero latency (< 10ms) between cluster nodes. Also they have to have the time synchronized with NTP. See also: Managed hardware requirements

You can rely on typical network tools to check metrics/network reliability between nodes. You can also use Cassandra nodetool to check network histograms: https://cassandra.apache.org/doc/latest/troubleshooting/use_nodetool.html

nodetool is in /utils/cassandra-nodetool.sh

Re: Looking to move from managed single node cluster to 2 nodes...addressing ALR

dave_mauney — Thu, 23 Apr 2020 14:43:54 GMT

Two nodes should be avoided. I consider a single node superior to a two node cluster because it avoids the exposure to "split brain" problem that a two node cluster entails.

I would look at scaling vertically on the single node, and then going to a three node cluster if that is not sufficient.

Re: Looking to move from managed single node cluster to 2 nodes...addressing ALR

AntonioSousa — Sun, 26 Apr 2020 15:17:12 GMT

Dynatrace recommends that you use 3 nodes and not 2:

https://www.dynatrace.com/support/help/setup-and-configuration/dynatrace-managed/installation/dynatrace-managed-hardware-and-system-requirements/#expand-551default-directory-paths-and-required-free-disk-space-for-installing-and-upgrading

I too agree that there should be an indication on what factors trigger ALR, as that is affecting one of my clients too.

Re: Looking to move from managed single node cluster to 2 nodes...addressing ALR

kalle_lahtinen — Thu, 13 Mar 2025 08:49:22 GMT

Hi Dave,

Thank you for the valuable input. Would it perhaps make sense to document the fact that we should avoid 2-node installations, for example here:

Managed hardware requirements

And here:

Managed failover mechanism

In my opinion, this is currently not expressed by Dynatrace's documentation.

Re: Looking to move from managed single node cluster to 2 nodes...addressing ALR

dave_mauney — Thu, 11 Jun 2020 14:57:43 GMT

Hi Kallie,

I agree we need to update our docs and have requested this of the documentation team.

Thanks,

dave

Re: Looking to move from managed single node cluster to 2 nodes...addressing ALR

kalle_lahtinen — Tue, 16 Jun 2020 05:52:33 GMT

Thanks Dave! I have one followup question, since you appear to have some expertise in the subject matter. If we do end up with the split brain scenario, are there any feasible methods to recover from it (while keeping the data), or is it just best to restore a previous Managed backup?

Re: Looking to move from managed single node cluster to 2 nodes...addressing ALR

runatyr — Tue, 16 Jun 2020 10:56:08 GMT

Thank you all for the help on this. We will be going to a 3 node solution. The good news is that the servers will be in the same VLAN and OTV will not be used to stretch the VLANS across different physical or logical spacings

Re: Looking to move from managed single node cluster to 2 nodes...addressing ALR

dave_mauney — Tue, 16 Jun 2020 16:12:08 GMT

I believe the recovery mainly involves some commands you have to issue against cassandra. Support can help you recover in the event it happens. @Radoslaw S. might have some instructions, also. I have never recovered from one personally...

Re: Looking to move from managed single node cluster to 2 nodes...addressing ALR

ChadTurner — Tue, 16 Jun 2020 16:57:33 GMT

Ideally a 3 node set up is what you would want.

Re: Looking to move from managed single node cluster to 2 nodes...addressing ALR

kalle_lahtinen — Tue, 16 Jun 2020 20:50:25 GMT

Thanks again Dave, appreciate it! Good stuff to know 🙂

Re: Looking to move from managed single node cluster to 2 nodes...addressing ALR

Radoslaw_Szulgo — Thu, 18 Jun 2020 07:40:25 GMT

You have to run cassandra-nodetool.sh repair after min. 3 hours disconnection between nodes. This is because Cassandra has a mechanism called "Hinted hand-off" which stores on a side all not synchronized chunks of data. It can store max 3 hours of data.

Usually upgrades don't take that long, so 2 node cluster should be safe in that aspect.

Re: Looking to move from managed single node cluster to 2 nodes...addressing ALR

kalle_lahtinen — Thu, 18 Jun 2020 08:02:05 GMT

Sounds good, thanks for the information Radoslaw!