cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Looking to upgrade from Dynatrace Managed to SaaS? See how

Dynatrace Managed - Upgrading nodes

Absh12
Visitor
we're upgrading our cluster and we opted for the replication method where we add one node at a time (Running on Rocky Linux 9) to migrate the data and after we have all of them ready, we will switch off the old nodes. 
We would like to understand here if we can proceed with the next node as we added only one and the documentation states the following:
  • Make sure that all the existing cluster nodes are healthy and reachable before adding another node. If multiple nodes are being migrated, make sure that the new node is fully started before proceeding to the next one.
  • The newly added node will become visible in Cluster Management Console during the installation as "joining". However, full data synchronization can take a couple of hours. You cannot add other cluster nodes until the one finishes the operation.
  • The Load value should not differ significantly between the nodes and Status should display UN on all nodes.

We checked the current status by running this `./cassandra-nodetool.shra-nodetool.sh status`  on the server and this is what we got: 

 

Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.64.7.71 217.81 GiB 256 73.9% 
UN 10.64.7.72 205.98 GiB 256 76.5% 
UN 10.64.7.73 210.49 GiB 256 75.7%
UN 10.64.7.172 167.3 GiB 256 73.9% 

The new node `10.64.7.172` is up and running but I am a bit hesitant about the Load value since it is quite far from the other 3 existing nodes. This gives me a feeling that data synchronisation/replication is not ready yet. 

Would you be able help me find out if it's safe to proceed or should I wait until the load value is close to what we have in the other nodes? 
4 REPLIES 4

Esam_Eid
Advisor

Hi @Absh12 ,

please share your experience with us. As I am planning to use the same approach 

 

regards

Babar_Qayyum
DynaMight Guru
DynaMight Guru

Hello @Absh12 

Provide sufficient time for a complete synchronization. If you try to add one more node, the cluster will not accept it.

Regards,

Babar

Absh12
Visitor

@Esam_Eid Hello,
I would say it would be safe to add the other nodes once the new added node shows UN (Up and Normal). 
The OWNS value will never show 100% while adding new nodes as this is related to the cluster's replication factor which by default is 3.
I waited 10 days for retention and today I am removing the old nodes. Will let you know how it will go. 

Esam_Eid
Advisor

Thank you @Absh12 

Featured Posts