We have problems with one of our web applications, where a punch of threads are started at every full hour and we do not really know from where they are coming.
I tried to get a thread dump, where I can see the stacktrace for all of the threads, but not those strange ones popping up to every full hour. Has someone a suggestion what it means when there is no Stacktrace for certain threads?
The java Threaddumps will contain the threads for every thread in the JVM. So if you see no data in the thread dump, then that means that thread didn't exist at the time the dump was taken.
You could try running frequent thread dumps, but that's still a hit/miss game.
Another appoach which might work even better is to utilize CPU sampling. This runs continuously for a period of time and gives you visibility into what the various thread activity is during this period from a CPU consumption perspective. So Idle threads wont provide any data, but busy threads will provide data.
I started the CPU Sampling, and indeed got more information.
It seems that those threads are coming from Dynatrace itself:
Are those threads started for the dumping process? This is very misleading when looking for such problems and there are hidden threads by the analysis tool itself.
quick note about the assumption that those threads are started by Dynatrace:
the getOrCreateTraceTag() method or any other methods from the Introspection class do not indicate that the Thread was created by Dynatrace. on the contrary, it usually means that this is a thread from which a PurePath is currently collected.
the threads started by the Dynatrace agent are easily distinguishable by the "dt" prefix in the thread name.
It's unclear to me what these threads are. I suggest you create a support case and have someone take a look at them. These thread dumps are stored in session files, attaching the session file would be helpful to let someone take a closer look.