It's not that usual that a filesystem is changed to read-only on the fly, and that normally means big problems. But that happened to a server today, on the / filesystem, where OneAgent is, and it stopped sending data. The server was able to continue serving some requests tough.
The question is to confirm if OneAgent is effectively unable to keep working in such conditions? I would imagine that simple metrics like CPU shouldn't be an issue. If someone can confirm that it really stops working, I would submit a Product Idea, as I didn't find one.
Has anyone dealt with this before?
As of my experience, when filesystem becomes read only, OneAgent will not be able to perform activities which it is suppose to under root privileges. However, OneAgent has this capability to switch between root user and default user, this shouldn’t be a problem.
Yes, we've had Linux machines go into ReadOnly Filesystem mode before. Our prior monitoring solution, OpsView, actually had a check for this (https://exchange.nagios.org/directory/Plugins/Operating-Systems/Linux/check_ro_mounts/details).
I've spoken to DT directly about this. They acknowledged my issue and concern but not sure when such a check or solution will be provided.
I'll reach out to my system administrator to recall how the event played out and post back.