topic Re: Understanding Slow Disk Problems in Alerting

Slow disk problems on a monitored host

matthew_gardner — Wed, 15 Mar 2023 16:16:09 GMT

Hello Everyone,

I am trying to understand what might be causing some slow disk problems on a monitored host. We can see that latency shoots up in intervals of 5 minutes, and sometimes reaches 300ms+

Just before that, we can see the throughput go up, but not a ton of data is being written:

From the IOPS metrics, we can see after writes/s go up, so does the queue length:

Re: Understanding Slow Disk Problems

ChadTurner — Tue, 14 Sep 2021 12:21:58 GMT

The host in question, is it a SQL host? Are there any commits taking place that might cause the latency?

Re: Understanding Slow Disk Problems

matthew_gardner — Tue, 14 Sep 2021 12:36:37 GMT

The server uses InterSystems IRIS technology. Which is a database technology, so that could be part of the problem since we see latency go up after some writes.

Re: Understanding Slow Disk Problems

AntonioSousa — Tue, 14 Sep 2021 15:22:35 GMT

There are a lot of cases where you might see a slow disk problem. Some I have encountered:

You have a slow HDD disk, with a slow RPM. This doesn't seem to be your case.
You are using logic volumes, or in some way, the disk is virtualized. Another system is using it extensively, so you're getting high latency in your case.
You have a filesystem that is clogged. It might be due having a lot of files in a directory, it is heavily fragmented, etc.
You have a disk with problems. Check the SMART of your disks.

These are some I remember fast. On the funny side, I haven't experienced the following type of proble, at least that I know of:

https://www.youtube.com/watch?v=tDacjrSCeq4

Re: Understanding Slow Disk Problems

AntonioSousa — Wed, 19 Jul 2023 21:36:50 GMT

Encryption is another issue, especially if done at the OS level...

Re: Understanding Slow Disk Problems

Jivan — Mon, 18 Mar 2024 10:05:53 GMT

Facing Same issue in our environment too and

we did not find any issue in the disk and linux team also did not get any delay or io wait from the server side.