topic Re: Problem with restore cluster from backup in Dynatrace Managed Q&A

Problem with restore cluster from backup

radek_jasinski — Wed, 19 Mar 2025 12:30:34 GMT

Hi,

I have a problem when trying to restore a cluster from a backup. I have machines that are identical in terms of software and hardware. I'm following the step by step instructions: Back up and restore a cluster

In step 3 I get an error when starting Casandra.

ERROR [main] 2023-07-11 11:13:53,823 UTC CassandraDaemon.java:803 - Exception encountered during startup

java.lang.RuntimeException: Unable to gossip with any peers

at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1603)

at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:628)

at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:888)

at org.apache.cassandra.service.StorageService.initServer(StorageService.java:745)

at org.apache.cassandra.service.StorageService.initServer(StorageService.java:694)

at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:395)

at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:633)

at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:786)

INFO [StorageServiceShutdownHook] 2023-07-11 11:13:53,877 UTC HintsService.java:210 - Paused hints dispatch

WARN [StorageServiceShutdownHook] 2023-07-11 11:13:53,877 UTC Gossiper.java:1728 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown

I'm betting it's a network connection problem, but I need to confirm it.

Has anyone encountered this before?

Radek

Re: Problem with restore cluster from backup

Islam_Zedan — Tue, 11 Jul 2023 12:40:42 GMT

Hi Radek,

How many nodes you have? are they on the same subnet. if no, what is the latency between subnets.

BR,

Islam

Re: Problem with restore cluster from backup

Radoslaw_Szulgo — Tue, 11 Jul 2023 12:47:50 GMT

I can confirm this is a network issue. The gossip - is the protocol used in Cassandra to negotiate the connection details between nodes. Make sure you've ensured proper network connection to the nodes you configured in step 2 - --cluster-nodes

Re: Problem with restore cluster from backup

radek_jasinski — Tue, 11 Jul 2023 13:08:22 GMT

Hi Radek,

Yes, I have verified all the requirements. For now, this environment has only one node because it's a test environment and we are testing the backup procedure before running it on the production environment.

Re: Problem with restore cluster from backup

Radoslaw_Szulgo — Tue, 11 Jul 2023 14:33:01 GMT

Hm.. so if that's only a single node... there should be no networking issue as there's no one to gossip with 😉

Re: Problem with restore cluster from backup

radek_jasinski — Tue, 11 Jul 2023 16:52:43 GMT

The development environment at the client looks like this:

1. The old DT server from which the backup was created
2. The network resource on which this backup is created (sub-mounted to two hosts).
3. The new DT server to which we make a restore from the backup.

Server 1 when starting the new one after the restore is done is disabled. Everything seems to be correct hence I'm surprised by this error. I will have another try tomorrow - let you know! 😊

Re: Problem with restore cluster from backup

radek_jasinski — Tue, 18 Jul 2023 11:58:41 GMT

Ok, the matter turned out to be simpler than I thought 😁
What was missing was specifying the --seed-ip variable when running the restore command.

btw. Casandra error message is highly specific in this case.

Radek