Disabling alerting / monitoring for specific disks / filesystems globally

[mariusz_kuczeba]

Since recently our instance is throwing lots alerts about /managed-agents/ and /dev/ filesystems being full. Those events specific to AWS Fargate and should be ignored.

I would preferably disable alerts, eventually disable monitoring for those cases.

How to do that globally?

Excluding disks from global disk options doesn't give us option to add filters (so I would filter AWS Fargate only)

Excluding disks by host one by one doesn't feel like right thing to do

Custom disk-detection rules and Disk Edge seem ok to modify thresholds, but how to exclude a disk or filesystem? Are there any working examples around that I failed to find?

From my reading, I understand that setting up same host group for all AWS Fargate hosts should help here, but it might be easier to exclude them one by one...

Thanks a bunch for any tips on the topic!

[AntonPineiro]

Hi,

Do you have custom disk or disk edge rules about those filesystems?

Best regards

[mariusz_kuczeba]

I had few which I was trying to use to filter out alerting (without luck), but there's currently no enabled rules.

[AntonPineiro]

Hi,

Which use case you have in mind?

You can configure global rules to be matching only some hosts. And you can add properties to include/exclude those events in the alerting profiles.

Best regards

[lubrman]

Hi @mariusz_kuczeba 

You can disable disk monitoring globally, but I wouldn't recommend it, as you would lose all disk-related metrics across all monitored hosts.

A better approach would be to configure a permanent maintenance window based on tags for the specific entity.

[mariusz_kuczeba]

That did sound very promising, but entity selectors on maintenance windows and management zones doesn't seem to support type(DISK)

I'm trying to copilot solution for this a bit, but other than "finding another way that does not work" we don't move much. It currently tries to convince me to accept limitations and disable monitoring of all disks and filesystems for AWS Fargate.

[lubrman]

Within the filter, it should be possible to select the disk and, optionally, the specific disk entity as well. This should allow this type of issue to be muted.

[lubrman]

The disk ID can then be retrieved via the Entities API, along with the corresponding host entity.

[rastislav_danis]

Do you have global diskedge policy, which generates problems from all disks?
If yes, just add exclude filters for specific disk paths into this global policy.

If you have multiple rules you can add specific global policy matching your disk paths above all others to exclude them from further processing. But be warned, that you have to use at least one alert rule enabled (it can be something unreal f.e. Available disk space (MiB) < 0) in your policy to be sure your disks paths will match and no further diskedge processing will follow for them.