cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

New collectors failing to connect

capesb
Contributor

we are trying to bring up a pair of new collectors in our production environment. they were setup using dynatrace-collector-6.3.0.1305-x64 which automatically updated to 6.3.15.1305. we are getting the following when the collector is started and need assistance to determine why it keeps timing out and won't connect to the main controller. we have other instances actively connecting to the controller but they are w/in the same data center. These collector are in another data center but we were able to connect up 2 other collectors to our non-production controller a few weeks back from this same data center.

collectorLauncher_4384.log:

2017-05-15 06:53:52 [000000f4] warning [native] Process 6652 not responding, retrieving Java stack trace...
2017-05-15 06:53:52 [000000f4] info [native] Process created successfully, pid: 6276
2017-05-15 06:53:52 [000000f4] warning [native] --- Java stack trace begin --------------------------------------------------------------------------
2017-05-15 06:53:52 [000000f4] warning [java ] 2017-05-15 06:53:52
2017-05-15 06:53:52 [000000f4] warning [java ] Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.66-b17 mixed mode):
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "DestroyJavaVM" #21 prio=5 os_prio=0 tid=0x00000000179de800 nid=0x182c waiting on condition [0x0000000000000000]
2017-05-15 06:53:52 [000000f4] warning [java ] java.lang.Thread.State: RUNNABLE
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] Locked ownable synchronizers:
2017-05-15 06:53:52 [000000f4] warning [java ] - None
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "pool-2-thread-2" #20 prio=5 os_prio=0 tid=0x00000000179e1000 nid=0x17b0 waiting on condition [0x000000001929f000]
2017-05-15 06:53:52 [000000f4] warning [java ] java.lang.Thread.State: WAITING (parking)
2017-05-15 06:53:52 [000000f4] warning [java ] at sun.misc.Unsafe.park(Native Method)
2017-05-15 06:53:52 [000000f4] warning [java ] - parking to wait for <0x00000000895c2188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.lang.Thread.run(Thread.java:745)
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] Locked ownable synchronizers:
2017-05-15 06:53:52 [000000f4] warning [java ] - None
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "pool-2-thread-1" #19 prio=5 os_prio=0 tid=0x000000001703f800 nid=0x1624 waiting on condition [0x0000000018d9e000]
2017-05-15 06:53:52 [000000f4] warning [java ] java.lang.Thread.State: WAITING (parking)
2017-05-15 06:53:52 [000000f4] warning [java ] at sun.misc.Unsafe.park(Native Method)
2017-05-15 06:53:52 [000000f4] warning [java ] - parking to wait for <0x00000000895c2188> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.lang.Thread.run(Thread.java:745)
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] Locked ownable synchronizers:
2017-05-15 06:53:52 [000000f4] warning [java ] - None
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "Worker-JM" #18 prio=5 os_prio=0 tid=0x0000000017086800 nid=0x1570 in Object.wait() [0x0000000018a9f000]
2017-05-15 06:53:52 [000000f4] warning [java ] java.lang.Thread.State: WAITING (on object monitor)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.lang.Object.wait(Native Method)
2017-05-15 06:53:52 [000000f4] warning [java ] - waiting on <0x00000000895c23f0> (a java.util.ArrayList)
2017-05-15 06:53:52 [000000f4] warning [java ] at org.eclipse.core.internal.jobs.InternalWorker.run(InternalWorker.java:58)
2017-05-15 06:53:52 [000000f4] warning [java ] - locked <0x00000000895c23f0> (a java.util.ArrayList)
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] Locked ownable synchronizers:
2017-05-15 06:53:52 [000000f4] warning [java ] - None
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "Start Level Event Dispatcher" #14 daemon prio=5 os_prio=0 tid=0x0000000016ccf800 nid=0x1608 in Object.wait() [0x00000000175ce000]
2017-05-15 06:53:52 [000000f4] warning [java ] java.lang.Thread.State: WAITING (on object monitor)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.lang.Object.wait(Native Method)
2017-05-15 06:53:52 [000000f4] warning [java ] - waiting on <0x00000000895c2408> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.lang.Object.wait(Object.java:502)
2017-05-15 06:53:52 [000000f4] warning [java ] at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:400)
2017-05-15 06:53:52 [000000f4] warning [java ] - locked <0x00000000895c2408> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
2017-05-15 06:53:52 [000000f4] warning [java ] at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:336)
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] Locked ownable synchronizers:
2017-05-15 06:53:52 [000000f4] warning [java ] - None
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "Framework Event Dispatcher" #13 daemon prio=5 os_prio=0 tid=0x0000000016cbc800 nid=0xa74 in Object.wait() [0x00000000174cf000]
2017-05-15 06:53:52 [000000f4] warning [java ] java.lang.Thread.State: WAITING (on object monitor)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.lang.Object.wait(Native Method)
2017-05-15 06:53:52 [000000f4] warning [java ] - waiting on <0x00000000899c9620> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.lang.Object.wait(Object.java:502)
2017-05-15 06:53:52 [000000f4] warning [java ] at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:400)
2017-05-15 06:53:52 [000000f4] warning [java ] - locked <0x00000000899c9620> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
2017-05-15 06:53:52 [000000f4] warning [java ] at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:336)
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] Locked ownable synchronizers:
2017-05-15 06:53:52 [000000f4] warning [java ] - None
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "Framework Active Thread" #10 prio=5 os_prio=0 tid=0x00000000163d4000 nid=0x40c in Object.wait() [0x00000000172cf000]
2017-05-15 06:53:52 [000000f4] warning [java ] java.lang.Thread.State: TIMED_WAITING (on object monitor)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.lang.Object.wait(Native Method)
2017-05-15 06:53:52 [000000f4] warning [java ] at org.eclipse.osgi.framework.internal.core.Framework.run(Framework.java:1817)
2017-05-15 06:53:52 [000000f4] warning [java ] - locked <0x0000000089603728> (a org.eclipse.osgi.framework.internal.core.Framework)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.lang.Thread.run(Thread.java:745)
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] Locked ownable synchronizers:
2017-05-15 06:53:52 [000000f4] warning [java ] - None
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00000000157a0800 nid=0x1850 runnable [0x0000000000000000]
2017-05-15 06:53:52 [000000f4] warning [java ] java.lang.Thread.State: RUNNABLE
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] Locked ownable synchronizers:
2017-05-15 06:53:52 [000000f4] warning [java ] - None
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "C1 CompilerThread1" #7 daemon prio=9 os_prio=2 tid=0x000000001579a000 nid=0x1988 waiting on condition [0x0000000000000000]
2017-05-15 06:53:52 [000000f4] warning [java ] java.lang.Thread.State: RUNNABLE
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] Locked ownable synchronizers:
2017-05-15 06:53:52 [000000f4] warning [java ] - None
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "C2 CompilerThread0" #6 daemon prio=9 os_prio=2 tid=0x00000000157aa000 nid=0x184c waiting on condition [0x0000000000000000]
2017-05-15 06:53:52 [000000f4] warning [java ] java.lang.Thread.State: RUNNABLE
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] Locked ownable synchronizers:
2017-05-15 06:53:52 [000000f4] warning [java ] - None
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x00000000157a8000 nid=0x1670 waiting on condition [0x0000000000000000]
2017-05-15 06:53:52 [000000f4] warning [java ] java.lang.Thread.State: RUNNABLE
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] Locked ownable synchronizers:
2017-05-15 06:53:52 [000000f4] warning [java ] - None
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00000000157a2800 nid=0xb18 waiting on condition [0x0000000000000000]
2017-05-15 06:53:52 [000000f4] warning [java ] java.lang.Thread.State: RUNNABLE
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] Locked ownable synchronizers:
2017-05-15 06:53:52 [000000f4] warning [java ] - None
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "Finalizer" #3 daemon prio=8 os_prio=1 tid=0x000000001575c000 nid=0x181c in Object.wait() [0x0000000015b1f000]
2017-05-15 06:53:52 [000000f4] warning [java ] java.lang.Thread.State: WAITING (on object monitor)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.lang.Object.wait(Native Method)
2017-05-15 06:53:52 [000000f4] warning [java ] - waiting on <0x00000000899caaf0> (a java.lang.ref.ReferenceQueue$Lock)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
2017-05-15 06:53:52 [000000f4] warning [java ] - locked <0x00000000899caaf0> (a java.lang.ref.ReferenceQueue$Lock)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] Locked ownable synchronizers:
2017-05-15 06:53:52 [000000f4] warning [java ] - None
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x0000000015755000 nid=0xef8 in Object.wait() [0x0000000015a1f000]
2017-05-15 06:53:52 [000000f4] warning [java ] java.lang.Thread.State: WAITING (on object monitor)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.lang.Object.wait(Native Method)
2017-05-15 06:53:52 [000000f4] warning [java ] - waiting on <0x00000000899cab30> (a java.lang.ref.Reference$Lock)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.lang.Object.wait(Object.java:502)
2017-05-15 06:53:52 [000000f4] warning [java ] at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
2017-05-15 06:53:52 [000000f4] warning [java ] - locked <0x00000000899cab30> (a java.lang.ref.Reference$Lock)
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] Locked ownable synchronizers:
2017-05-15 06:53:52 [000000f4] warning [java ] - None
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "VM Thread" os_prio=2 tid=0x0000000015750800 nid=0x1ab8 runnable
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "Gang worker#0 (Parallel GC Threads)" os_prio=2 tid=0x0000000000d00800 nid=0x1808 runnable
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "Gang worker#1 (Parallel GC Threads)" os_prio=2 tid=0x0000000000d04800 nid=0x1358 runnable
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "Concurrent Mark-Sweep GC Thread" os_prio=2 tid=0x0000000000d33000 nid=0x1504 runnable
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] "VM Periodic Task Thread" os_prio=2 tid=0x0000000016100800 nid=0x1484 waiting on condition
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [java ] JNI global references: 383
2017-05-15 06:53:52 [000000f4] warning [java ]
2017-05-15 06:53:52 [000000f4] warning [native] --- Java stack trace end ----------------------------------------------------------------------------
2017-05-15 06:53:52 [000000f4] info [native] Process 6276 terminated, exitcode: 0
2017-05-15 06:53:52 [000000f4] severe [native] Process 6652 not responding within time limit - restarting process...
2017-05-15 06:53:52 [000000f4] info [native] Signal 15 sent to process 6652
2017-05-15 06:53:52 [000000e4] info [native] Process 6652 terminated, exitcode: 15
2017-05-15 06:53:52 [000000e4] severe [native] Process was terminated by native watchdog
2017-05-15 06:53:52 [000000e4] info [native] Restarting dynaTrace Collector...
2017-05-15 06:53:52 [000000e4] info [native] Launcher configuration summary for dynaTrace Collector:
2017-05-15 06:53:52 [000000e4] info [native] Launcher: D:\Program Files\dynaTrace\dynaTrace Collector 6.3\dtcollector.exe
2017-05-15 06:53:52 [000000e4] info [native] Java VM: D:\Program Files\dynaTrace\dynaTrace Collector 6.3\jre\bin\java.exe
2017-05-15 06:53:52 [000000e4] info [native] Java version: 1.8.0_66 64-Bit
2017-05-15 06:53:52 [000000e4] info [native] Startup jar: D:\Program Files\dynaTrace\dynaTrace Collector 6.3\collector\plugins\org.eclipse.equinox.launcher_1.2.0.v20110502.jar
2017-05-15 06:53:52 [000000e4] info [native] Working directory: D:\Program Files\dynaTrace\dynaTrace Collector 6.3\collector
2017-05-15 06:53:52 [000000e4] info [native] Configuration file: D:\Program Files\dynaTrace\dynaTrace Collector 6.3\dtcollector.ini
2017-05-15 06:53:52 [000000e4] info [native] Pid file: C:\Windows\SERVIC~2\LOCALS~1\AppData\Local\Temp\dtcollector.pid
2017-05-15 06:53:52 [000000e4] info [native] Native watchdog enabled, port: 50000, timeout: 180 sec (timeout for graceful shutdown: 300 sec)
2017-05-15 06:53:52 [000000e4] info [native] Process created successfully, pid: 6108
2017-05-15 06:53:54 [00000104] info [java ]
2017-05-15 06:56:52 [000000f4] warning [native] Process 6108 not responding within time limit - still waiting for process to connect...
2017-05-15 06:59:14 [00001e00] info [native] Ctrl-Shutdown event (6) received
2017-05-15 06:59:14 [00001e00] info [native] ... last message repeated 100 times ...
2017-05-15 06:59:14 [00001e00] info [native] Ctrl-Shutdown event (6) received
2017-05-15 06:59:14 [00001e00] info [native] ... last message repeated 100 times ...
2017-05-15 06:59:14 [00001e00] info [native] Ctrl-Shutdown event (6) received
2017-05-15 06:59:14 [00001e00] info [native] ... last message repeated 100 times ...
2017-05-15 06:59:14 [00001e00] info [native] Ctrl-Shutdown event (6) received
2017-05-15 06:59:14 [00001e00] info [native] ... last message repeated 100 times ...
2017-05-15 06:59:14 [00001e00] info [native] Ctrl-Shutdown event (6) received
2017-05-15 06:59:14 [00001e00] info [native] ... last message repeated 100 times ...
2017-05-15 06:59:14 [00001e00] info [native] Ctrl-Shutdown event (6) received
2017-05-15 06:59:14 [00001e00] info [native] ... last message repeated 100 times ...
2017-05-15 06:59:14 [00001e00] info [native] Ctrl-Shutdown event (6) received

2 REPLIES 2

pahofmann
Leader

Could you attach or upload the log? It´s really hard to read this way.

capesb
Contributor

we have resolved this issue, there was a permissions issue on both hosts. once this was corrected and the collectors restarted they connected to the controller and are now active

thank you