OneAgent stops sending data when filesystem becomes read-only?

[AntonioSousa]

It's not that usual that a filesystem is changed to read-only on the fly, and that normally means big problems. But that happened to a server today, on the / filesystem, where OneAgent is, and it stopped sending data. The server was able to continue serving some requests tough.

The question is to confirm if OneAgent is effectively unable to keep working in such conditions? I would imagine that simple metrics like CPU shouldn't be an issue. If someone can confirm that it really stops working, I would submit a Product Idea, as I didn't find one.

Has anyone dealt with this before?

[Sushmita_shet]

Hi @AntonioSousa 

As of my experience, when filesystem becomes read only, OneAgent will not be able to perform activities which it is suppose to under root privileges. However, OneAgent has this capability to switch between root user and default user, this shouldn’t be a problem. 

[ct_27]

Yes, we've had Linux machines go into ReadOnly Filesystem mode before.  Our prior monitoring solution, OpsView, actually had a check for this (https://exchange.nagios.org/directory/Plugins/Operating-Systems/Linux/check_ro_mounts/details).

I've spoken to DT directly about this. They acknowledged my issue and concern but not sure when such a check or solution will be provided.

I'll reach out to my system administrator to recall how the event played out and post back.

[techean]

Its shouldn't be an issue! can you share specific details and report this as a bug, we may be able to troubleshoot. Read only access should be enough for monitoring

[AntonioSousa]

Yes, I imagine it wouldn't stop, at least for basic metrics.

Now, I had synthetic monitoring pointing to a web server on that machine, and the web server continued replying... Of course, don't have any logs at all during the period that was read-only...

[gilles_tabary]

Could this be related ?

We have here VM, with OneAgent in full stack mode, running standalone podman containers. Yet the full-stack does not work when podman containers are started with this option :

podman run ... --read-only=True ...

 We have a (not nice but functional) leagacy (==prior to podman full-stack monitoring capability) hack : 

# in /etc/containers/containers.conf
...
"LD_PRELOAD=/opt/dynatrace/oneagent/agent/lib64/liboneagentproc.so",
...
volumes=[
   "/opt/dynatrace/oneagent/agent:/opt/dynatrace/oneagent/agent:ro",
   "/var/lib/dynatrace/oneagent/agent:/var/lib/dynatrace/oneagent/agent:ro",
   "/<custom install path>/agent:/<custom install path>/agent:ro",
...

I am trying to get rid of this hack.