Slow disk problems on a monitored host

[matthew_gardner]

Hello Everyone,

I am trying to understand what might be causing some slow disk problems on a monitored host. We can see that latency shoots up in intervals of 5 minutes, and sometimes reaches 300ms+

Just before that, we can see the throughput go up, but not a ton of data is being written:

From the IOPS metrics, we can see after writes/s go up, so does the queue length:

[ChadTurner]

The host in question, is it a SQL host? Are there any commits taking place that might cause the latency? 

[matthew_gardner]

The server uses InterSystems IRIS technology. Which is a database technology, so that could be part of the problem since we see latency go up after some writes.

[AntonioSousa · Accepted solution]

There are a lot of cases where you might see a slow disk problem. Some I have encountered:

• You have a slow HDD disk, with a slow RPM. This doesn't seem to be your case.
• You are using logic volumes, or in some way, the disk is virtualized. Another system is using it extensively, so you're getting high latency in your case.
• You have a filesystem that is clogged. It might be due having a lot of files in a directory, it is heavily fragmented, etc.
• You have a disk with problems. Check the SMART of your disks.

These are some I remember fast. On the funny side, I haven't experienced the following type of proble, at least that I know of:

https://www.youtube.com/watch?v=tDacjrSCeq4

[AntonioSousa]

Encryption is another issue, especially if done at the OS level...

[Jivan]

Facing Same issue in our environment too and 

we did not find any issue in the disk and linux team also did not get any delay or io wait from the server side.