cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Slow disk problems on a monitored host

matthew_gardner
Dynatrace Helper
Dynatrace Helper

Hello Everyone,

 

I am trying to understand what might be causing some slow disk problems on a monitored host. We can see that latency shoots up in intervals of 5 minutes, and sometimes reaches 300ms+

matthew_gardner_0-1631547734239.png

Just before that, we can see the throughput go up, but not a ton of data is being written:

matthew_gardner_1-1631547819838.png

From the IOPS metrics, we can see after writes/s go up, so does the queue length:

matthew_gardner_2-1631547884476.png

 

 

 

3 REPLIES 3

ChadTurner
DynaMight Legend
DynaMight Legend

The host in question, is it a SQL host? Are there any commits taking place that might cause the latency? 

-Chad

The server uses InterSystems IRIS technology. Which is a database technology, so that could be part of the problem since we see latency go up after some writes.

AntonioSousa
DynaMight Guru
DynaMight Guru

There are a lot of cases where you might see a slow disk problem. Some I have encountered:

  • You have a slow HDD disk, with a slow RPM. This doesn't seem to be your case.
  • You are using logic volumes, or in some way, the disk is virtualized. Another system is using it extensively, so you're getting high latency in your case.
  • You have a filesystem that is clogged. It might be due having a lot of files in a directory, it is heavily fragmented, etc.
  • You have a disk with problems. Check the SMART of your disks.

These are some I remember fast. On the funny side, I haven't experienced the following type of proble, at least that I know of:

https://www.youtube.com/watch?v=tDacjrSCeq4

Antonio Sousa
Brendan Gregg from Sun's Fishworks team makes an interesting discovery about inducing disk latency. For a ca. 2020 retrospective on this 2008 video: https://www.youtube.com/watch?v=_IYzD_NR0W4#t=28m47s