Disaster recovery use case for Dynatrace Managed

[Babar_Qayyum]

Dear All,

We have a requirement to use the production Dynatrace Managed Cluster in the DR site for test purposes as well as for the contingency in the later stage. To do this infrastructure team can migrate/ or replicate the same VMs into the DR site, but in the DR site IP addresses of the Dynatrace Managed Cluster nodes will be changed.

Is this possible to do this exercise with minimum efforts?

Did anyone already do this exercise?

Regards,

Babar

[Julius_Loman]

If you are thinking about doing the DR the way you described, you probably have your Dynatrace Managed architecture wrong.
Normally you would have multiple nodes of the Dynatrace Managed cluster distributed across the datacenters. If one site goes down, the rest of the cluster nodes still do the work.

Officially, restore can be done only on application-level - so you need to spin a new clean vm and you can restore Dynatrace Cluster node on this host from the backup. You will, however, loose data that is not backed up. You can restore the cluster node on a host with different IP addresses.

Just changing the IP addresses on the cluster node is AFAIK not directly supported.

[Babar_Qayyum]

Hello @Július L.

We have 3 nodes cluster in the production environment, and a stretched cluster is one of the options under consideration, which you also described in your comments.

If the customer does not agree due to any reason, then what could be the next best option to cover the DR setup with minimum efforts?

Did you verify the following scenario by yourself?

Just changing the IP addresses on the cluster node is AFAIK not directly supported.

Regards,

Babar

[Radoslaw_Szulgo]

Babar,

this can only be achieved right now via "restore" procedure from the backup. Find the manual here:

https://www.dynatrace.com/support/help/setup-and-configuration/dynatrace-managed/operation/back-up-a...

2 things you need to remember about:

1. Use domain name for OneAgent communication address - and update DNS records from normal to DR site IPs.

2. Shutdown the normal cluster before starting DR site.

Let me know if you have any detailed questions.

[Babar_Qayyum]

Hello @Radoslaw S.

Thank you for sharing the link.

I guess my question was not much clear as our objective is different than the restore procedure, which I already started discussing with other colleagues in the same thread.

Regards,

Babar

[ChadTurner]

We had replicated to DR sites on the fly even when we did Mock DR's without issue. But we will be migrating our 3 clustered nodes to have a node in each regional site for DR/Fail over

[Babar_Qayyum]

Hello @Chad T.

Did you change the IP addresses/ or hostnames of the nodes on the DR site?

Regards,

Babar

[Julius_Loman]

If this was on the fly, IPs were probably not changed. My opinion in general - if you have to change IPs in disaster recovery procedures, then it's not disaster recovery, but migration.

[Babar_Qayyum]

Hello @Július L.

It is a customer's general procedure even for the production setup to migrate the applications and change their IPs on the destination.

Therefore, they are also expecting the same in the case of Dynatrace Cluster. Is this possible to change the IPs/hostnames in the DR without any unexpected situation or disrupting the whole monitoring setup?

Regards,

Babar

[Julius_Loman]

No, it's not simple as changing the IPs. See @Radoslaw S. answer - you have to restore it from Dynatrace backup to new nodes. If you change both hostnames and IP addresses of cluster nodes, agents and activegates may not reconnect.

[ChadTurner · Accepted solution]

This is exciting news for managed customers that want a fail-over cluster at regional sites. I recommend checking it out.

https://www.dynatrace.com/support/help/managed-cluster/fault-domain-awareness/premium-high-availabil...

[Radoslaw_Szulgo]

thx for cross-linking! 🙂 Forgot about that question.

[ChadTurner]

You're welcome!

[Babar_Qayyum]

Hello @Chad T.

Thank you for sharing a valuable link.

Regards,

Babar