Hi Dynatrace Community,
We have a 6-node Dynatrace Managed cluster (3 nodes in DC and 3 nodes in DR). As part of our DC relocation activity, we need to temporarily remove 3 nodes and add them back once the relocation is completed.
Our cluster has a local quorum requirement of 4.
Could you please share the recommended approach and best practices for node removal and re-addition? Specifically, we would appreciate guidance on quorum management, node removal sequence, data retention, cluster stability, and precautions to follow when adding the nodes back.
Any practical experience, official documentation, or expert recommendations would be highly appreciated.
Thanks in advance!
Hello @Sohel_Rashid
With a quorum of 4, taking 3 of 6 nodes offline means the cluster can no longer satisfy quorum in the normal sense. I’d treat this as a planned data center outage (or a DC rebuild), not as routine one-by-one node removal. Please confirm what “local quorum of 4” refers to in your setup. I couldn’t find that figure in the Dynatrace docs, so I’m assuming it describes the cluster-wide requirement.
Also, have a look at the following link.
Regards,
Babar Qayyum
@Babar_Qayyum Thanks for your response.
To clarify, by “local quorum of 4”, I mean that at least 4 out of our 6 cluster nodes must remain available to maintain cluster stability. If the number of available nodes falls below 4, we may face quorum-related issues and potential split-brain conditions.
We have experienced split-brain incidents multiple times in the past, mainly due to network connectivity issues between cluster nodes.
Considering this history, we are concerned about the impact of taking 3 nodes offline for DC relocation. We would appreciate your recommendation on the safest and officially supported approach for this activity, including how to maintain quorum and avoid split-brain issues. We have approximately 1,700 hosts monitored in our Dynatrace Managed environment. If we remove 3 out of the 6 cluster nodes for DC relocation, will the remaining 3 nodes be able to handle the existing monitoring workload, including data ingestion and processing, without impacting cluster stability or monitoring availability?
I would recommend to add one temporary node to your cluster in locality, which remains up. Remove it after relocation is over and 3 relocated nodes are up again.
Hello @Sohel_Rashid
One potential solution is to implement a cluster stretch without a PHA license, provided you are assured of maintaining latency below 10ms between the cluster nodes. You may add three additional cluster nodes from the new data center location and allow time for data replication. After the replication is successfully completed, you can proceed to remove the three nodes from the existing data center one at a time.
Regards,
Babar Qayyum
