cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Managed Cluster backup to NFS

Julius_Loman
DynaMight Guru
DynaMight Guru

Backup of a Dynatrace Managed Cluster with more than 1 node requires shared directory mounted at all nodes of the cluster at the same directory (NFS is recommended). Is the backup process resilient to outages of the mounted remote filesystem?

For example - if the NFS server is non-reponsive - can this affect the normal Dynatrace cluster node operation? (of course, backups cannot be done during the outage)
Are there any options for disabling the backup temporary? (for maintenance on NFS server) except for turning the backup on/off in the CMC?

 

TEMPEST a.s., Slovakia, Dynatrace Master Partner
20 REPLIES 20

Radoslaw_Szulgo
Dynatrace Guru
Dynatrace Guru

Hi Julius,

backup is triggered every 1 hour for elasticsearch snapshot from the database. In case of NFS unavilability, this operation will time out and not affect normal operation of the server or any other components.

Backup is also triggered every 24 hours for cassandra dump. This is implemented as a separate script launched on each node. And again, if NFS is unavailable script will just fail to execute, or will run very long until times out.


There's no option to pause the backup. Frankly, it's not so needed, as in worst scenario you will miss cassandra daily dump, but the previous backup should still be available.


Senior Product Manager,
Dynatrace Managed expert

Babar_Qayyum
DynaMight Leader
DynaMight Leader

Hello @Július L. and @Radoslaw S.

I am a little confused regarding the storage path.

Do we need to provide the physical storage path (e.g. 10.10.10.10:/dynatrace-ns/dynatrace-bkt) or the mounting point (e.g. /cluster_configuration_backup) created on all cluster nodes)?

Regards,

Babar

This should be the mounting point that was created across all cluster nodes for example /usr/local/dynatrace.

@Malik R. is correct. We had done the same thing. But Dynatrace might still list it as incorrect but the backup will still perform as expected

-Chad

why would Dynatrace list this as incorrect?

Senior Product Manager,
Dynatrace Managed expert

Hey @Radoslaw S. this was back at my previous employer where we were a managed instance. For some strange reason Dynatrace would always have yellow under text from the mount point location. I dont quite recall what it stated, but the CMC didnt really like the mount point but never the less back backed up every night right on time. I can always reach back out to my contacts over there to see if the issue is still present, maybe get you a screen capture of it.

-Chad

Babar_Qayyum
DynaMight Leader
DynaMight Leader

Hello @Malik R. and @Chad T.

Thank you for your comments.

It means I am going to use the mount point e.g. /cluster_configuration_backup

Correct?

Regards,

Babar

Correct. It just needs to be the same mount point across all nodes.

TEMPEST a.s., Slovakia, Dynatrace Master Partner

@Babar Q. That's correct. Dynatrace might say that it dosent like it, but it should still work. Or at least thats what it did back when we set it up as managed 2 years ago.

-Chad

Yes, that is exactly what you will need to do. As @Július L. mentioned it needs to be the same mount point across all nodes. https://www.dynatrace.com/support/help/setup-and-configuration/dynatrace-managed/operation/back-up-a... example dynatrace used "On each target node, mount the backup storage to, for example, /mnt/backup. This path is referred to as <path-to-backup> "

Babar_Qayyum
DynaMight Leader
DynaMight Leader

Hello @Július L., @Chad T. and @Malik R.,

Thank you for the confirmation and assistance.

Regards,

Babar

Babar_Qayyum
DynaMight Leader
DynaMight Leader

Hello @Radoslaw S. @Július L. @Chad T. @Malik R.

According to the documentation, 'The user running Dynatrace services needs read/write permissions for the shared file system.' If I am not wrong, by default the services are running with the user dynatrace. Is my understanding correct?

The user dynatrace has full permisssion on the shared file system, but I am getting the below issue.


Regards,

Babar

Is that set on all nodes?

Senior Product Manager,
Dynatrace Managed expert

Hello @Radoslaw S.

Yes. It is configured similarly on all nodes.

Regards,

Babar

pls paste `ls -la` so I could see how it looks like with perms

Senior Product Manager,
Dynatrace Managed expert

Hello @Radoslaw S.

Please find the below information.

I am getting the permission deniced with the root user while using the command ls -la

Regards,

Babar

@Babar Q. are you using CIFS by a chance?


Such problems may be caused by the caching in CIFS - that is why we recommend using NFS and not CIFS. The issue happens less often when "peristenthandles" and "actimeo=0" params are used when CIFS is mounted. 
Senior Product Manager,
Dynatrace Managed expert

Hello @Radoslaw S.

I checked with the customer and they said we mounted as NFS3.

Regards,

Babar

Nothing else comes to my mind. Please open a support ticket, we'll try to troubleshoot this individually.

Senior Product Manager,
Dynatrace Managed expert

Hello @Radoslaw S.

The support ticket (SUP-66546) opened.

Regards,

Babar