cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Some Agents were disconnected after the AppMon Management Server GC occurred.

khanada
Contributor

Hi


I've been experiencing PurePath skips by the AppMon Management Server GC several times, but this is the first time that the Agent is unable to connect.


There were two target hosts, Web1 and Java1 were installed respectively, and a total of 4 Agents were connected.

Oddly enough, the Agent on one host successfully reconnected, but the Agent on the other host was unable to connect to the management server.


The following log was output to the management server.

2020-05-22 17:06:18 WARNING [CollectorCenter] Tickcount of agent "DT_Server_Frontend@AppMonServer:2609" (8486b6d5) on collector "Dynatrace Self-Monitoring@AppMonServer" has not been updated for 320803ms. This might be caused by a GC run. PurePaths might remain in state "Processing"

2020-05-22 17:06:18 WARNING [CollectorCenter] Tickcount of agent "DT_Server@AppMonServer:3986" (3835ec4a) on collector "Dynatrace Self-Monitoring@AppMonServer" has not been updated for 321002ms. This might be caused by a GC run. PurePaths might remain in state "Processing"

2020-05-22 17:06:18 WARNING [CollectorCenter] Tickcount of agent "Java_Profile@host1:26920" (1b4b200a) on collector "dynaTrace Collector@AppMonServer" has not been updated for 320509ms. This might be caused by a GC run. PurePaths might remain in state "Processing"

2020-05-22 17:06:18 WARNING [CollectorCenter] Tickcount of agent "Web_Profile@host1:6144" (653fca96) on collector "dynaTrace Collector@AppMonServer" has not been updated for 314105ms. This might be caused by a GC run. PurePaths might remain in state "Processing"

2020-05-22 17:06:18 WARNING [CollectorCenter] Tickcount of agent "Web_Profile@host2:2761" (1fdc2351) on collector "dynaTrace Collector@AppMonServer" has not been updated for 314108ms. This might be caused by a GC run. PurePaths might remain in state "Processing"

2020-05-22 17:06:18 WARNING [CollectorCenter] Tickcount of agent "Java_Profile@host2:13489" (0d8c023c) on collector "dynaTrace Collector@AppMonServer" has not been updated for 314103ms. This might be caused by a GC run. PurePaths might remain in state "Processing"

2020-05-22 17:06:18 WARNING [CollectorCenter] Tickcount of agent "Web_Profile[Apache 2.2]@host1:3000" (0d168e07) on collector "dynaTrace Collector@AppMonServer" has not been updated for 314105ms. This might be caused by a GC run. PurePaths might remain in state "Processing"

2020-05-22 17:06:18 WARNING [CollectorCenter] Tickcount of agent "Web_Profile[Apache 2.2]@host2:59689" (2708bbb1) on collector "dynaTrace Collector@AppMonServer" has not been updated for 314108ms. This might be caused by a GC run. PurePaths might remain in state "Processing"

2020-05-22 17:06:18 WARNING [CollectorCenter] Global Agent Time significantly behind current system time. diff: 324,439ms, now: 1590134778694(=Fri May 22 17:06:18 JST 2020), gat: 1590134454254(=Fri May 22 17:00:54 JST 2020)

2020-05-22 17:06:18 WARNING [CollectorCenter] * Old global agent time: 1590134448011(=Fri May 22 17:00:48 JST 2020) -> new GlobalAgentTime: 1590134454254 (=Fri May 22 17:00:54 JST 2020), current virtual agent times (might be slightly different):

2020-05-22 17:06:18 WARNING [CollectorCenter] Virtual Agent Time of affected agents from collector: dtang@AppMonServer, total connected Agents: 0

2020-05-22 17:06:18 WARNING [CollectorCenter] No considered Agent or no virtual-agent-time problem on this collector.

2020-05-22 17:06:18 WARNING [CollectorCenter] Virtual Agent Time of affected agents from collector: Dynatrace Self-Monitoring@AppMonServer, total connected Agents: 2

2020-05-22 17:06:18 WARNING [CollectorCenter] Agent: DT_Server_Frontend@AppMonServer:2609, techtype = Java, diff = 324,325ms: virtualAgentTime: 1590134454369(=Fri May 22 17:00:54 JST 2020), tickCountAge = 321,005ms, tickCount = 77764312417763676

2020-05-22 17:06:18 WARNING [CollectorCenter] Agent: DT_Server@AppMonServer:3986, techtype = Java, diff = 324,440ms: virtualAgentTime: 1590134454254(=Fri May 22 17:00:54 JST 2020), tickCountAge = 321,004ms, tickCount = 77764312302358685

2020-05-22 17:06:18 WARNING [CollectorCenter] Minimum VirtualAgentTime of collector: Dynatrace Self-Monitoring@AppMonServer: 1590134454254(=Fri May 22 17:00:54 JST 2020), diff = 324,440ms

2020-05-22 17:06:18 WARNING [CollectorCenter] Virtual Agent Time of affected agents from collector: dynaTrace Collector@AppMonServer, total connected Agents: 7

2020-05-22 17:06:18 WARNING [CollectorCenter] Agent: Java_Profile@host1:26920, techtype = Java, diff = 323,066ms: virtualAgentTime: 1590134455628(=Fri May 22 17:00:55 JST 2020), tickCountAge = 320,511ms, tickCount = 60619926669041227

2020-05-22 17:06:18 WARNING [CollectorCenter] Agent: Web_Profile@host1:6144, techtype = Web Server, diff = 320,683ms: virtualAgentTime: 1590134458011(=Fri May 22 17:00:58 JST 2020), tickCountAge = 314,106ms, tickCount = 151542622083469029

2020-05-22 17:06:18 WARNING [CollectorCenter] Agent: Web_Profile@host2:2761, techtype = Web Server, diff = 320,359ms: virtualAgentTime: 1590134458335(=Fri May 22 17:00:58 JST 2020), tickCountAge = 314,109ms, tickCount = 153052986969843637

2020-05-22 17:06:18 WARNING [CollectorCenter] Agent: Java_Profile@host2:13489, techtype = Java, diff = 316,692ms: virtualAgentTime: 1590134462002(=Fri May 22 17:01:02 JST 2020), tickCountAge = 314,104ms, tickCount = 61223699698362467

2020-05-22 17:06:18 WARNING [CollectorCenter] Agent: Web_Profile[Apache 2.2]@host1:3000, techtype = Web Server, diff = 320,683ms: virtualAgentTime: 1590134458011(=Fri May 22 17:00:58 JST 2020), tickCountAge = 314,106ms, tickCount = 151542622083469029

2020-05-22 17:06:18 WARNING [CollectorCenter] Agent: Web_Profile[Apache 2.2]@host2:59689, techtype = Web Server, diff = 320,359ms: virtualAgentTime: 1590134458335(=Fri May 22 17:00:58 JST 2020), tickCountAge = 314,109ms, tickCount = 153052986969843637

2020-05-22 17:06:18 WARNING [CollectorCenter] Minimum VirtualAgentTime of collector: dynaTrace Collector@AppMonServer: 1590134455628(=Fri May 22 17:00:55 JST 2020), diff = 323,066ms

2020-05-22 17:07:11 WARNING [AgentProblemReportTimerTask] Forcing agent DT_Server_Frontend@AppMonServer:2609 on collector Dynatrace Self-Monitoring@AppMonServer to reconnect to a collector because the agent did not send ping events for a long time

2020-05-22 17:07:11 WARNING [AgentProblemReportTimerTask] Forcing agent DT_Server@AppMonServer:3986 on collector Dynatrace Self-Monitoring@AppMonServer to reconnect to a collector because the agent did not send ping events for a long time

2020-05-22 17:07:11 WARNING [AgentProblemReportTimerTask] Forcing agent Java_Profile@host1:26920 on collector dynaTrace Collector@AppMonServer to reconnect to a collector because the agent did not send ping events for a long time

2020-05-22 17:07:11 WARNING [AgentProblemReportTimerTask] Forcing agent Web_Profile@host1:6144 on collector dynaTrace Collector@AppMonServer to reconnect to a collector because the agent did not send ping events for a long time

2020-05-22 17:07:11 WARNING [AgentProblemReportTimerTask] Forcing agent Web_Profile@host2:2761 on collector dynaTrace Collector@AppMonServer to reconnect to a collector because the agent did not send ping events for a long time

2020-05-22 17:07:11 WARNING [AgentProblemReportTimerTask] Forcing agent Java_Profile@host2:13489 on collector dynaTrace Collector@AppMonServer to reconnect to a collector because the agent did not send ping events for a long time

2020-05-22 17:07:13 WARNING [AgentProblemReportTimerTask] Forcing agent Web_Profile[Apache 2.2]@host1:3000 on collector dynaTrace Collector@AppMonServer to reconnect to a collector because the agent did not send ping events for a long time

2020-05-22 17:07:13 WARNING [AgentProblemReportTimerTask] Forcing agent Web_Profile[Apache 2.2]@host2:59689 on collector dynaTrace Collector@AppMonServer to reconnect to a collector because the agent did not send ping events for a long time

2020-05-22 17:07:33 WARNING [CachingFqdnResolver] Failed to resolve DNS for IP 'XXX.XXX.XXX.XXX' - had following exception: com.dynatrace.diagnostics.sdk.net.FqdnResolutionException: Cannot resolve FQDN for IP address 'XXX.XXX.XXX.XXX' [log message will be suppressed for 30 minutes]

2020-05-22 17:07:43 WARNING [StringTable] A string could not be found in server side StringTable. Suppressing this warning for another 60000 ms.

2020-05-22 17:07:44 WARNING [RealTimeAnalysisQueue] Skipping a PurePath: /wscs-web/services/GetDeninfoService#Web_Profile[Apache 2.2]@host1:3000(0d168e07).269087965 from realtime analysis and session storage because it is already too old. This PurePath will not be analyzed, as we cannot adapt measures that far in the past. PurePath creation time: Fri May 22 17:07:40 JST 2020, start time: Fri May 22 16:38:26 JST 2020, end time: Fri May 22 16:38:26 JST 2020; PurePath: /wscs-web/services/GetDeninfoService#Web_Profile[Apache 2.2]@host1:3000(0d168e07).269087965, isEndUser: false, isRIA: false, entry point virtual agent time: 1590133114696. This message will be suppressed for the next 60s Suppressing this warning for another 60000 ms.

2020-05-22 17:07:47 WARNING [CachingFqdnResolver] Failed to resolve DNS for IP '127.0.0.1' - had following exception: com.dynatrace.diagnostics.sdk.net.FqdnResolutionException: Cannot resolve FQDN for IP address '127.0.0.1' [log message will be suppressed for 30 minutes]

2020-05-22 17:08:50 WARNING [RealTimeAnalysisQueue] Skipping a PurePath: /#Web_Profile[Apache 2.2]@host1:3000(0d168e07).269088774 from realtime analysis and session storage because it is already too old. This PurePath will not be analyzed, as we cannot adapt measures that far in the past. PurePath creation time: Fri May 22 17:08:28 JST 2020, start time: Fri May 22 16:39:21 JST 2020, end time: Fri May 22 16:39:21 JST 2020; PurePath: /#Web_Profile[Apache 2.2]@host1:3000(0d168e07).269088774, isEndUser: false, isRIA: false, entry point virtual agent time: 1590134918402. This message will be suppressed for the next 60s Suppressing this warning for another 60000 ms.


The following log was output on the Agent that could not connect.

2020-05-22 17:07:11 [77ff8830] info [native] Reconnecting agent with next Server/Collector: XXX.XXX.XXX.XXX:9997

2020-05-22 17:09:52 [6d823830] warning [native] Could not instrument class because instrumentor channel is not connected. Instrumentation might not be complete.

2020-05-25 07:49:16 [b1693830] warning [native] Could not send class unload events because instrumentor channel isn't connected. Events will be skipped.


Do you have any information?


0 REPLIES 0