Adding cluster node fails with "Cannot prepare network settings..."

[andre_vdveen]

Hi, we get the following error when trying to add a new node.

Adding new node with ip 10.36.0.222 failed. Reason: Cannot prepare network settings on Dynatrace cluster nodes. Error: Adding IP of this machine ("10.36.0.222") to cluster node "10.36.22.240" failed.

Does anyone have any idea what causes this and how to get around it?

[BertEvo]

Hi Andre,

Is it possible that your node has multiple NIC's?

If so, it happens that the installer is trying to use e.g. the backup or mgt interface.

Maybe try forcing the IP address to correct one? 

bindToNetworkInterface = <IP.address>

Grtz,

Bert

[andre_vdveen]

Hi @BertEvo, no that was one of the first things I checked.

We managed to get past the original error but the node installation failed again with various other errors, each time we manage to get past those after the firewall team allowed open communication between all node IPs and not restricting to any specifc cluster ports that are listed in the docs, but we just faced yet another failed installation - this time, after almost 2.5 hours, it failed stating the API token is invalid!? 

Managed is going to be the death of me, honestly! Wish I could move the client to SaaS already 🤣

[Peter_Youssef]

Hi @andre_vdveen 

Kindly ensure the cluster nodes are in the same data center / subnet and the below ports are properly configured and Multi-node installation requirements are fulfilled:

• https://docs.dynatrace.com/managed/shortlink/managed-network-ports
• https://docs.dynatrace.com/managed/shortlink/managed-requirements#multi-node-installations

Hoping it helps.

BR, 

Peter

[Peter_Youssef]

Hello @andre_vdveen 

PFA troubleshooting path for resolution:

Node:

• Checking the configuration files through the first stable cluster node.
• Ensuring cluster token is valid.
• Through the UI ensuring the first cluster node configurations are valid as per the actual server insights "IP Address, Host Name, cluster token,...etc".
• IP Conflict validation if exist.
• Checking the log entries through "/var/log/dynatrace/" to validate what might block the node from joining the cluster, the log will be the shortest path pinpointing to the actual root cause.
• Checking the cluster initialization logs under "/var/log/dynatrace/cluster/"

Network:

• Checking Network connectivity between the nodes.
• Checking firewall rules configured between the nodes that might block the connectivity.
• Checking DNS resolution via nslookup, If DNS resolution fails, you may need to adjust your DNS configuration or adding entries to "/etc/hosts" to resolve the hostnames locally would be an applicable solution.
• Checking if there are load balancer or proxy issues, if exist it should be configured properly to ensure the stable connectivity and constant synchronization between nodes.
• Restarting the Dynatrace nodes after applying the highlighted options.

Hoping it adds value.

KR, 

Peter