On 03/12/2017, after our OS admin restarted the host, one of agents running on the host vanished from the agents overview even if its process is still running on the host.
From its collector log, we see these lines
2017-03-12 08:59:53 INFO [Instrumentor] Agent CustomerService_WAS-penv_svp1_pprod71@chln643:47445 requesting logoff (UNKNOWN)
2017-03-12 08:59:58 INFO [AgentPeerPool] removed agent 'CustomerService_WAS-penv_svp1_pprod71@chln643:47445' from pool - number of agents: 182
2017-03-12 08:59:58 INFO [Instrumentor] Finished shutdown of agent in AgentCenter: CustomerService_WAS-penv_svp1_pprod71@chln643:47445 (ID: 210c3e33)
tFrom he Dynatrace log on the server: we saw tehse
2017-03-12 08:59:52 [dbff88dd] info [native] Reconnecting agent with Server/Collector: lnbrdynp62.highmark.com:9998
2017-03-12 09:00:05 [a09c68dd] warning [native] Could not instrument class because instrumentor channel is not connected. Instrumentation might not be complete.
We are used to seeing the agent sit there in a disconnected state, but this guy is just totally gone.
from these snippets it looks as if it shut down in an orderly way and disconnected from the collector. Then, when it restarted, it apparently couldn't re-establish the connection completely.
- we don't show it as disconnected and there's no incident because it looks like a planned shut down
- we don't show it as connected because it never reached the collector again.
Note that this is mainly a guess, so if we need to debug this further we'd need the full logs. Best regards,