c.d.s.p.p.EndpointPoller | Endpoint 10.91.27.xx:1436 did not started completely. Health check will be done in next interval

[Levas]

Hello, 

Our custom SQL datasource monitor (used to fetch business data from MSSQL DB) suddenly stopped to work. Something wrong with health check from active gate host.
However I can start session with telnet hostname 1436

in Dynatrace GUI logs I find:

2024-11-13 16:27:07.125WARN

Polling did not started completely, some subgroups are still pending for endpoints: 10.91.27.xx:1436
2024-11-13 16:26:07.587WARN

Failed to configure and schedule polling for endpoint 10.91.27.xx:1436. Retry will be done in the next interval.

in 
/activegate/config/remotepluginmodule/log/extensions/datasources/custom_aph.sqlmonitor.envmaintenancedb/dynatracesourcesql.jar.log 

I find these lines:

[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out][0fe646c3-2f96-30f9-805f-26a64584956f]2024-11-13 | 15:19:17.128 | pool-1-thread-1      |  INFO | c.d.s.c.EecCommunication  | Status:202:OK
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out][0fe646c3-2f96-30f9-805f-26a64584956f]2024-11-13 | 15:19:17.135 | pool-1-thread-2      |  INFO | c.d.s.c.EecCommunication  | TimeDiff:200:OK
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out][0fe646c3-2f96-30f9-805f-26a64584956f]2024-11-13 | 15:19:27.127 | pool-1-thread-1      |  INFO | c.d.s.c.EecCommunication  | Status:202:OK
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out][0fe646c3-2f96-30f9-805f-26a64584956f]2024-11-13 | 15:19:27.127 | pool-1-thread-2      |  INFO | c.d.s.c.EecCommunication  | TimeDiff:200:OK
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out][0fe646c3-2f96-30f9-805f-26a64584956f]2024-11-13 | 15:19:37.127 | pool-1-thread-1      |  INFO | c.d.s.c.EecCommunication  | Status:202:OK
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out][0fe646c3-2f96-30f9-805f-26a64584956f]2024-11-13 | 15:19:37.128 | pool-1-thread-2      |  INFO | c.d.s.c.EecCommunication  | TimeDiff:200:OK
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out][0fe646c3-2f96-30f9-805f-26a64584956f]2024-11-13 | 15:19:47.127 | pool-1-thread-1      |  INFO | c.d.s.c.EecCommunication  | Status:202:OK
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out][0fe646c3-2f96-30f9-805f-26a64584956f]2024-11-13 | 15:19:47.128 | pool-1-thread-2      |  INFO | c.d.s.c.EecCommunication  | TimeDiff:200:OK
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out][0fe646c3-2f96-30f9-805f-26a64584956f]2024-11-13 | 15:19:57.128 | pool-1-thread-1      |  INFO | c.d.s.c.EecCommunication  | TimeDiff:200:OK
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out][0fe646c3-2f96-30f9-805f-26a64584956f]2024-11-13 | 15:19:57.129 | pool-1-thread-2      |  INFO | c.d.s.c.EecCommunication  | Status:202:OK
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out][0fe646c3-2f96-30f9-805f-26a64584956f]2024-11-13 | 15:20:07.123 | pool-1-thread-1      |  INFO | c.d.s.c.EecCommunication  | SfmMetrics:202:OK
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out][0fe646c3-2f96-30f9-805f-26a64584956f]2024-11-13 | 15:20:07.126 | pool-1-thread-2      |  INFO | c.d.s.c.EecCommunication  | Status:202:OK
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out][0fe646c3-2f96-30f9-805f-26a64584956f]2024-11-13 | 15:20:07.130 | pool-1-thread-2      |  INFO | c.d.s.c.EecCommunication  | TimeDiff:200:OK
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out][0fe646c3-2f96-30f9-805f-26a64584956f]2024-11-13 | 15:20:07.135 | pool-1-thread-1      |  INFO | c.d.s.c.EecCommunication  | Events:204:OK
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out][0fe646c3-2f96-30f9-805f-26a64584956f]2024-11-13 | 15:20:07.561 | pool-2-thread-2      |  WARN | c.d.s.p.p.EndpointPoller  | Endpoint 10.91.27.64:1436 did not started completely. Health check will be done in next interval
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out][0fe646c3-2f96-30f9-805f-26a64584956f]2024-11-13 | 15:20:07.583 | pool-2-thread-1      |  WARN | c.d.s.m.u.DeviceResolver  | Unable to resolve hostname when enriching device dimensions. IP will be used as device.name dimension.
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out][0fe646c3-2f96-30f9-805f-26a64584956f]2024-11-13 | 15:20:07.583 | pool-2-thread-1      | ERROR | s.p.c.ContinuousCollector | Failed to configure and schedule polling for endpoint 10.91.27.64:1436. Retry will be done in the next interval.
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out]org.quartz.ObjectAlreadyExistsException: Unable to store Job : 'GroupPollerJob.GroupPoller{endpointUrl='10.91.27.64:1436', sqlObjectName=supplierorders}', because one already exists with this identification.
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out]	at org.quartz.simpl.RAMJobStore.storeJob(RAMJobStore.java:279)
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out]	at org.quartz.simpl.RAMJobStore.storeJobAndTrigger(RAMJobStore.java:251)
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out]	at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:855)
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out]	at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:249)
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out]	at com.dynatrace.sql.polling.pollers.AbstractPoller.schedulePolling(AbstractPoller.java:152)
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out]	at com.dynatrace.sql.polling.pollers.GroupPoller.schedulePolling(GroupPoller.java:50)
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out]	at com.dynatrace.sql.polling.pollers.EndpointPoller.schedulePolling(EndpointPoller.java:80)
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out]	at com.dynatrace.sql.polling.collector.ContinuousCollector.lambda$continuousPolling$1(ContinuousCollector.java:144)
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out]	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out]	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out]	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out]	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out]	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[0fe646c3-2f96-30f9-805f-26a64584956f][-5063718364380454897][1125984][out]	at java.base/java.lang.Thread.run(Unknown Source)

[AntonPineiro]

Hi,

Were you able to find a root cause on this?

Best regards