Rising Windows kernel memory - How to get more information?

[AntonioSousa]

I have a very unusual evolution of memory in a Windows server. Memory usage is going steadily up, and the culprit is the Windows kernel!

Probably some memory leak? I have checked system logs, and some more metrics, but have been unable to find out what might be causing this rise in memory consumption. I'm also checking at the server level... Any ideas on how to investigate this further?

[ChadTurner]

Any luck with looking into this deeper @AntonioSousa 

[AntonioSousa]

No. We have upgraded the Windows OS, but memory keeps climbing... Since it's at the Kernel level, we don't have much clues. Will be back if we discover a solution.

[Davidsandler]

The issue doesn't seem to be with the kernel. Dynatrace indicates that MS SQLSERVER is using 2.6GB of memory, but the actual problem lies in the inconsistent memory reporting by the Windows OS. This inconsistency is visible both in Task Manager and the resource monitor's (working set) as well. We've initiated an internal ticket to thoroughly analyze this matter.

Fortunately, due to the substantial memory usage of this database, we were able to identify the issue with relative ease. This stands in contrast to situations on low-memory servers where identification is more challenging. This inconsistency is impacting the precision of memory analysis in Dynatrace.

[AntonioSousa]

@Davidsandler,

Accounting for memory usage has never been an easy task. Device drivers for instance can do very weird things. And when I discovered "ballooning", than I knew that even the OS didn't have a chance 😉

[Julius_Loman]

A long time ago we had a similar issue, which was triggered by OneAgent, but caused by a bad Fibrechannel driver which had a memory leak when some performance counters were queried. It took quite a while to investigate. Since the driver was out of support, the only workaround was to disable the network monitoring module of OneAgent on such hosts. 

As far as I remember, we used the poolmon command to fetch the kernel allocations and then identified the pool and the driver causing it. 

[AntonioSousa]

@Julius_Loman,

The sysadmin tried poolmon, but was unable to get useful data... We tried several tricks, including installing a new server with the same software. It kept on climbing in an absolute linear form.

But now that I have checked again, the system seams stable for a week now. I'll have to check with him what was done 😉