Load balancing with single ActiveGate in Managed cluster

[Julius_Loman]

In a situation, with a 3 node (or more) Managed cluster, does agent traffic get load balanced via an ActiveGate?

For example - an environment with 100 hosts, connected via Environment ActiveGate to a 3 node cluster.
ActiveGate is simultaneously connected to all Cluster nodes according to netstat , but does the agent traffic flowing via a single ActiveGate get balanced across nodes?

It seems the agent traffic gets balanced to all cluster nodes in this case, but a confirmation would be appreciated.

[skrystosik]

I think you should have cluster activegate before 3 nodes which will do the trick.

Sebastian

[Julius_Loman]

That won't help as it is still a single gateway (or multiple Cluster gateways).

Based on my observations this seems to work as expected (agent load gets distributed across nodes in a cluster) and each Environment ActiveGate has two connections to every single Cluster node. We just don't know if is the load balancing happening within the ActiveGate to Cluster communication.

I just need a confirmation if it works this way or do we need to deploy as many environment activegates as the number of cluster nodes for the environment.

[skrystosik]

Interesting case, I don't know the answer but if you will figure it out give me as sign 🙂

[Julius_Loman]

Just to be clear - I'm not asking about Oneagent to ActiveGate(s) load balancing.

[Radoslaw_Szulgo · Accepted solution]

This is random over time. Each OneAgent and AG receives a list of all communication endpoints. So all in all after some time, it's equally balanced.

Some examples:

Setup: 3 Env AG, 1 Cluster AG, 5 Cluster nodes, 100 OneAgents

1. OneAgents in the environment intranet
OneAgents receive 8 (all) endpoints. Each AG has it's own priority. Firstly, Environment AG are picked. So the load is evenly distributed to 3 Env AGs.

2. OneAgents in external internet

OneAgents receive 8 (all) endpoints. Only Cluster AG is reachable. All traffic is directed to Cluster AG. Cluster AG receives only cluster node endpoints (5). Traffic is load balanced evenly by randomly picking up the endpoint and switching every N minutes (I think 5).

3. OneAgents around Cluster.

OneAgent receive 8 (All) endpoints. Only cluster nodes are available. So distribution is done over 5 nodes.

[Julius_Loman]

@Radoslaw S. that's about the agent endpoints. Probably I was not clear, but this is not what I've asked. My question is about load distribution on the cluster node.

Consider the environment with 3 cluster nodes (1 tenant), 100 OneAgents. 1 ActiveGate. Consider high load on those Oneagents (many purepaths/user actions)

All OneAgents connect via Environment ActiveGate. Each cluster node is limited with it's processing power determined by available CPU frequency and memory. (Load reduction on the service data, user actions limits, etc...).

So my question is - is the data from OneAgents balanced between those cluster nodes for processing or not?. Remember agents are still connected via a single ActiveGate (or two ActiveGates for redundancy). In case where I have one ActiveGate - is the data flowing via this ActiveGate somehow distributed across the nodes or will be handled by a single cluster node and then the processing is limited to a single node?

[Radoslaw_Szulgo]

That doesn't matter in that case if it's OneAgent or AG. Both need to communicate to the most prioritized AG. At cluster node you actually have "Embedded" AG.

So to answer your question directly. 1 or 2 Environment AGs will always communicate to Cluster AG or Embedded AG (on cluster nodes). Every interval the "client" will switch the target endpoint so it's load balanced.

Is that clear now?

[Julius_Loman]

Question is about the distribution of load between the cluster nodes. I don't get from your answer if the load on a cluster node is distributed or not when you have a lower number of ActiveGates than the number of Cluster nodes. (Consider all OneAgents communicate via Environment Cluster Nodes for simplicity).

[Radoslaw_Szulgo]

It is balanced. No matter if that is OA or Environment AG it gets list of endpoints that are utilized randomly to achieve balancing.

If you have 1 Env AG and 3 nodes then Env AG will switch from time to time between nodes.

[Julius_Loman]

Yes, this is still OK, but will the other cluster nodes be utilized?

Maybe my question should be "Do the embedded ActiveGates at Cluster nodes balance somehow the load between the server processes at cluster nodes?"

[Radoslaw_Szulgo]

If you take a look at cluster.properties of embedded AG you will see that there are all server endpoints (port 8020). So same things happen. AG picks one at random and swaps from time to time.

If communication endpoint is on port 443 (NGINX) - in that case NGINX always redirects to local AG.

[Julius_Loman]

Well, that would, however, indicate the data processing is not balanced between cluster nodes in this case.

So the Environment ActiveGate connects to one embedded Cluster Node ActiveGate which connects to just one particular node at a time.

Basically this means if you have just one environment ActiveGate, agents connected via this ActiveGate will reach only one Cluster Node server at a time and the data gets processed at this single server process.

So having 3 cluster nodes will not help you to balance the load from the agents and data will be processed just by the single node where the agent traffic actually ends.

Can you confirm this behaviour?

I do understand that with more ActiveGates, etc. this somehow gets balanced because the connectivity itself gets balanced between nodes.

[Radoslaw_Szulgo]

Not really. Let me then go one more step deeper into tech details. Each AG has it's own queue of messages that receives (from AGs or OA). Each message in the queue is then transferred individually to the next endpoint e.g. server at the end. It may send messages for server 1 via AG 1 and messages for server 2 via AG 2.

[Julius_Loman]

Ok, perfect, thank you. I think this answers my original question.

[pahofmann]

This is something I wondered about too, when running into ALR.

If I unterstand it correctly, in the scenario with 100 agents, 1 AG and three cluster nodes, the load (servicecalls/minute each cluster node can handle) is evenly distributed to all three nodes, even if only one AG is present that is randomly switching between all the nodes. Right @Radoslaw S.?

[Radoslaw_Szulgo]

Correct sir

[dave_mauney]

The general recommendation is to deploy at least the number of AGs as you have in your cluster. Otherwise, you may create a bottleneck or even a single point of failure (if the OneAgents cannot communicate directly to the Embedded AGs on the Cluster nodes).

[Radoslaw_Szulgo]

Depends on the load and where are your OA come from and what modules are utilized (VMware, AWS)

[Julius_Loman]

It's not about the communication path availability, but about load balancing method at the cluster level.

Maybe the question should be "Do the embedded ActiveGates at Cluster node balance the load between the server processes at cluster nodes?"