topic Re: Managed Cluster backup to NFS in Dynatrace Managed Q&A

Managed Cluster backup to NFS

Julius_Loman — Wed, 09 Jun 2021 11:28:36 GMT

Backup of a Dynatrace Managed Cluster with more than 1 node requires shared directory mounted at all nodes of the cluster at the same directory (NFS is recommended). Is the backup process resilient to outages of the mounted remote filesystem?

For example - if the NFS server is non-reponsive - can this affect the normal Dynatrace cluster node operation? (of course, backups cannot be done during the outage)
Are there any options for disabling the backup temporary? (for maintenance on NFS server) except for turning the backup on/off in the CMC?

Re: Managed Cluster backup to NFS

Radoslaw_Szulgo — Thu, 21 Nov 2019 07:42:40 GMT

Hi Julius,

backup is triggered every 1 hour for elasticsearch snapshot from the database. In case of NFS unavilability, this operation will time out and not affect normal operation of the server or any other components.

Backup is also triggered every 24 hours for cassandra dump. This is implemented as a separate script launched on each node. And again, if NFS is unavailable script will just fail to execute, or will run very long until times out.

There's no option to pause the backup. Frankly, it's not so needed, as in worst scenario you will miss cassandra daily dump, but the previous backup should still be available.

Re: Managed Cluster backup to NFS

Babar_Qayyum — Thu, 11 Feb 2021 14:10:44 GMT

Hello @Július L. and @Radoslaw S.

I am a little confused regarding the storage path.

Do we need to provide the physical storage path (e.g. 10.10.10.10:/dynatrace-ns/dynatrace-bkt) or the mounting point (e.g. /cluster_configuration_backup) created on all cluster nodes)?

Regards,

Babar

Re: Managed Cluster backup to NFS

malik_roberts — Thu, 11 Feb 2021 14:21:11 GMT

This should be the mounting point that was created across all cluster nodes for example /usr/local/dynatrace.

Re: Managed Cluster backup to NFS

ChadTurner — Thu, 11 Feb 2021 14:36:17 GMT

@Malik R. is correct. We had done the same thing. But Dynatrace might still list it as incorrect but the backup will still perform as expected

Re: Managed Cluster backup to NFS

Babar_Qayyum — Thu, 11 Feb 2021 17:19:42 GMT

Hello @Malik R. and @Chad T.

Thank you for your comments.

It means I am going to use the mount point e.g. /cluster_configuration_backup

Correct?

Regards,

Babar

Re: Managed Cluster backup to NFS

Julius_Loman — Thu, 11 Feb 2021 17:30:55 GMT

Correct. It just needs to be the same mount point across all nodes.

Re: Managed Cluster backup to NFS

ChadTurner — Thu, 11 Feb 2021 17:59:23 GMT

@Babar Q. That's correct. Dynatrace might say that it dosent like it, but it should still work. Or at least thats what it did back when we set it up as managed 2 years ago.

Re: Managed Cluster backup to NFS

malik_roberts — Wed, 19 Mar 2025 12:04:04 GMT

Yes, that is exactly what you will need to do. As @Július L. mentioned it needs to be the same mount point across all nodes. Back up and restore a cluster example dynatrace used "On each target node, mount the backup storage to, for example, /mnt/backup. This path is referred to as <path-to-backup> "

Re: Managed Cluster backup to NFS

Radoslaw_Szulgo — Fri, 12 Feb 2021 08:58:28 GMT

why would Dynatrace list this as incorrect?

Re: Managed Cluster backup to NFS

Babar_Qayyum — Sat, 13 Feb 2021 06:36:58 GMT

Hello @Július L., @Chad T. and @Malik R.,

Thank you for the confirmation and assistance.

Regards,

Babar

Re: Managed Cluster backup to NFS

ChadTurner — Sat, 13 Feb 2021 14:40:25 GMT

Hey @Radoslaw S. this was back at my previous employer where we were a managed instance. For some strange reason Dynatrace would always have yellow under text from the mount point location. I dont quite recall what it stated, but the CMC didnt really like the mount point but never the less back backed up every night right on time. I can always reach back out to my contacts over there to see if the issue is still present, maybe get you a screen capture of it.

Re: Managed Cluster backup to NFS

Babar_Qayyum — Mon, 15 Feb 2021 05:51:13 GMT

Hello @Radoslaw S. @Július L. @Chad T. @Malik R.

According to the documentation, 'The user running Dynatrace services needs read/write permissions for the shared file system.' If I am not wrong, by default the services are running with the user dynatrace. Is my understanding correct?

The user dynatrace has full permisssion on the shared file system, but I am getting the below issue.

Regards,

Babar

Re: Managed Cluster backup to NFS

Radoslaw_Szulgo — Mon, 15 Feb 2021 11:05:24 GMT

Is that set on all nodes?

Re: Managed Cluster backup to NFS

Babar_Qayyum — Mon, 15 Feb 2021 11:42:49 GMT

Hello @Radoslaw S.

Yes. It is configured similarly on all nodes.

Regards,

Babar

Re: Managed Cluster backup to NFS

Radoslaw_Szulgo — Mon, 15 Feb 2021 11:52:49 GMT

pls paste `ls -la` so I could see how it looks like with perms

Re: Managed Cluster backup to NFS

Babar_Qayyum — Mon, 15 Feb 2021 12:05:25 GMT

Hello @Radoslaw S.

Please find the below information.

I am getting the permission deniced with the root user while using the command ls -la

Regards,

Babar

Re: Managed Cluster backup to NFS

Radoslaw_Szulgo — Mon, 15 Feb 2021 12:41:08 GMT

@Babar Q. are you using CIFS by a chance?

Such problems may be caused by the caching in CIFS - that is why we recommend using NFS and not CIFS. The issue happens less often when "peristenthandles" and "actimeo=0" params are used when CIFS is mounted.

Re: Managed Cluster backup to NFS

Babar_Qayyum — Mon, 15 Feb 2021 13:10:59 GMT

Hello @Radoslaw S.

I checked with the customer and they said we mounted as NFS3.

Regards,

Babar

Re: Managed Cluster backup to NFS

Radoslaw_Szulgo — Mon, 15 Feb 2021 13:17:59 GMT

Nothing else comes to my mind. Please open a support ticket, we'll try to troubleshoot this individually.