TL;DR: I know the server maintains the agent record for a short while after an agent disconnects so that it can reconnect quickly if the issue was a temporary connectivity issue or a restart, but I am not sure if that applies to the license or not. After an agent goes offline, how long does it take the server to free up that license?
My client has two sets of servers, only one of which is ever active/in use at any one point in time. Consequently, they only bought enough licenses to instrument one of them at a time. However, both sets of servers are running all the time, which means we are running into license exhaustion issues. They want to have a cron job check to see if a set of servers is set to "active", and if it is not, update the OS iptables to cut connectivity between the inactive servers and the collector, removing those connections and thus licenses from Dynatrace without server restarts. However, we need to make sure the cron job waits for Dynatrace to put those licenses back in the pool before updating the iptables to allow the other set of servers to connect.
I am not sure I understood your problem. May I as why you have to servers running? Is that for production and testing environment separation purposes? In that case a good reason would be to have selected collectos for each servers, than you would have control of which agent is consuming each license.
In any case, I believe that if you block the connection from one collector to one of the servers after a while Dynatrace will mark the agents from that collector as disconnected and will free up the licenses. The problem here is that as you will have to wait for a while for the licenses to be freed before starting the other servers this means that during this time you will get no monitoring at all. I am not sure, but I believe it won't take long, but it can take up to 5 minutes (gessing) for this license do be freed.
Ok, so you are not sure how long the timeout is? You just think it is probably below 5 minutes?
I am not really sure why they have two sets of identical servers but only use one set at a time, to be honest. They want them both in the same system profile and agent groups too since they are identical, in order to make the process of switching from one set of servers to another to be transparent to them in Dynatrace. They don't want to have to manually do anything to disconnect the agents, don't want to have to have two system profiles that each have long gaps when their agents are not connected, etc.
I will suggest your idea for two collectors to them though. Having the cron job shut down the collector for the inactive servers might be easier/less impactful than updating iptables.
From your answer it seems that they want a high availability architecture, right? In this case it is possible to be done, having two servers up at the same time, but only one is working. This is actually not officially supported, but some folks here in Brazil did a lab on this and it worked. In that case you would need to request two license files with the argument that they are for a high availability environment. The configuration is not easy, but possible. Maybe just having specific collectors is easier...
Ah, ok. I will try just the doubling the license as Adam mentions below, since right now they do not have the hardware or VM available to spin up a second AppMon server, but I will definitely keep this as a good backup suggestion if Licensing does not double it for us. Thank you very much Igor.
This is something that sales might be able to agree contractually to double your license as long as the client only uses one set at a time. That's often how active / passive setups are dealt with?
The HA suggestion is the
best option in your case.
However to answer the
license question the DT server license usage count is equal to connected
licensed agents. That means if an agent disconnects instantly that license is
free and is available for any new agent trying to acquire license.