topic Re: Dynatrace Managed - Upgrading nodes in Dynatrace Managed Q&A

Dynatrace Managed - Upgrading nodes

Absh12 — Tue, 28 May 2024 11:59:40 GMT

we're upgrading our cluster and we opted for the replication method where we add one node at a time (Running on Rocky Linux 9) to migrate the data and after we have all of them ready, we will switch off the old nodes.
We would like to understand here if we can proceed with the next node as we added only one and the documentation states the following:

Make sure that all the existing cluster nodes are healthy and reachable before adding another node. If multiple nodes are being migrated, make sure that the new node is fully started before proceeding to the next one.
The newly added node will become visible in Cluster Management Console during the installation as "joining". However, full data synchronization can take a couple of hours. You cannot add other cluster nodes until the one finishes the operation.
The Load value should not differ significantly between the nodes and Status should display UN on all nodes.

We checked the current status by running this `./cassandra-nodetool.shra-nodetool.sh status` on the server and this is what we got:

Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.64.7.71 217.81 GiB 256 73.9%
UN 10.64.7.72 205.98 GiB 256 76.5%
UN 10.64.7.73 210.49 GiB 256 75.7%
UN 10.64.7.172 167.3 GiB 256 73.9%

The new node `10.64.7.172` is up and running but I am a bit hesitant about the Load value since it is quite far from the other 3 existing nodes. This gives me a feeling that data synchronisation/replication is not ready yet.

Would you be able help me find out if it's safe to proceed or should I wait until the load value is close to what we have in the other nodes?

Re: Dynatrace Managed - Upgrading nodes

Esam_Eid — Wed, 12 Jun 2024 11:54:36 GMT

Hi @Absh12 ,

please share your experience with us. As I am planning to use the same approach

regards

Re: Dynatrace Managed - Upgrading nodes

Babar_Qayyum — Wed, 12 Jun 2024 11:31:36 GMT

Hello @Absh12

Provide sufficient time for a complete synchronization. If you try to add one more node, the cluster will not accept it.

Regards,

Babar

Re: Dynatrace Managed - Upgrading nodes

Absh12 — Wed, 12 Jun 2024 12:17:38 GMT

@Esam_Eid Hello,
I would say it would be safe to add the other nodes once the new added node shows UN (Up and Normal).
The OWNS value will never show 100% while adding new nodes as this is related to the cluster's replication factor which by default is 3.
I waited 10 days for retention and today I am removing the old nodes. Will let you know how it will go.

Re: Dynatrace Managed - Upgrading nodes

Esam_Eid — Wed, 12 Jun 2024 12:35:49 GMT

Thank you @Absh12

Re: Dynatrace Managed - Upgrading nodes

rudy_h — Sat, 14 Jun 2025 12:57:42 GMT

Hello everyone,

I have seen nodes upgraded automatically but I can't start the dynatrace cluster server afterward:

Does anyone know how to fix it?

Thanks