Restore one sigle node in a 3-node managed cluster

[DamianG]

Hi Community,

I would like to know how to restore a single node in a 3-node managed cluster. Note: I have automatic backup enabled for the cluster.

In the Dynatrace documentation we can find instructions to restore the whole cluster (https://docs.dynatrace.com/managed/managed-cluster/operation/back-up-and-restore-a-cluster#restore-f...), but there are no details about how to restore a single node (or two) nodes of the cluster. In addition, I would like to know if it is possible to restore one node from a snapshot.

Possible scenario --> OS upgrade from RHEL 8.6 to RHEL9.4 in one of the nodes of the cluster. Steps to follow:

1. Take an snapshot of the node.
2. The OS upgrade of the node starts and it fails.
3. Restore from snapshot --> is this possible? if not, what to do?

Many thanks in advance.

[erh_inetum]

Hi Damián,

Restore from snapshot is not supported. Support team told us and here you have another comment about it.

About restoring only one node it not recommended by Dynatrace because restoring less nodes than the original backup you risk losing the cluster configuration

Hope it helps.

Regards,

Elena.

[DamianG]

Hi Elena,

Thank you for your comments.

What I meant in my post is that, in case only one node (of three) fails, I could restore only that node from snapshot because the other two nodes are still up and running without issues.

In addittion, in case the snapshot is not supported to restore one or more nodes, the restore procedure is based on a full restore of the cluster and I only want to restore one node because the other 2 nodes are already running. Does this mean that if one node fails (of three) I have to restore the whole cluster?

Thank you.

[erh_inetum · Accepted solution]

Hi Damián,

> What I meant in my post is that, in case only one node (of three) fails, I could restore only that node from snapshot because the other two nodes are still up and running without issues.

As far as I know it's not supported restoring nodes from snapshot. But I don´t know it would work. I've never tested it to be honest

> In addittion, in case the snapshot is not supported to restore one or more nodes, the restore procedure is based on a full restore of the cluster and I only want to restore one node because the other 2 nodes are already running. Does this mean that if one node fails (of three) I have to restore the whole cluster?

As it's said in the documentation, it seems it's possible restoring less nodes than the original cluster but it's not recommended because you risk losing the cluster configuration.

I hope have clarifying something

Maybe someone can provide a better answer than me.

Regards,

Elena.

[DamianG]

Thank you Elena.

Following this thread. Do we have any restore procedure for ActiveGates? I couldn't find it in the documentation.

Thank you.

[erh_inetum · Accepted solution]

Welcome.

I think it's not possible to restore an AG. There is only a procedure to restore a cluster, not AGs:

https://community.dynatrace.com/t5/Dynatrace-Managed-Q-A/Restore-an-environment-from-backup-on-a-Man...

Try to restart AG processes and if I doesn´t work, try to reinstall the AG.

Hope it helps.

Regards,

Elena.

[DamianG]

Thank you Elena!!

In the other hand, a solution to restore only one node (provided by Dynatrace Support) could be the following, depending on the scenario:

• Replace one node that is working: add a new node, disable OneAgent traffic on the affected node, and wait around 35 days (the default retention period for transaction storage).
  That way, you'll be ready to replace the impacted node and won't lose any data.
• Replace a node that is not working: remove it from the Managed Cluster completely and add a new node.

This is valid to such scenarios to avoid a full cluster restore.

Regards,

Damián

[erh_inetum]

Hi Damián,

Thanks a lot for the information. It's very valuable.

Have a nice day.

Regards,

Elena.