Challenges with new disk space usage for Linux since OneAgent 1.203

[kalle_lahtinen]

Referring to the fresh blog post:

https://www.dynatrace.com/news/blog/further-improved-handling-and-reliability-of-oneagent-deployment...

"What does this mean for existing installations?

The migration of the logs storage directory will happen automatically upon the update to OneAgent version 1.203; so there is really nothing you need to do."

That's great. But what if I don't want it? Since the old OneAgent disk space requirement for /opt (default) was about 5 GB, we have hundreds of OneAgent installations where we've created separate 5 GB logical volumes and filesystems for the agent, mounted to /opt/dynatrace. That way we were sure that 1. there's enough disk space for the agent 2. the agent won't interfere with the disk needs of other components running on the server. This new approach appears to destroy the previous design.

https://www.dynatrace.com/support/help/technology-support/operating-systems/linux/installation/disk-...

Here's the old table:

	Linux (x86)	Linux (ppcle)	Linux (s390)
Size of installation (with temporary installation files removed)	900 MB	300 MB	130 MB
Recommended free space for runtime data (logs, temporary files, crash reports, memory dumps, etc.)	3 GB	3 GB	3 GB
Additional space required for updates	1.5 GB	530 MB	300 MB
Total	5.4 GB	3.8 GB	3.4GB

And the new one:

	Linux (x86)	Linux (ppcle)	Linux (s390)	Default directory
Size of installation (with temporary installation files removed)	~1 GB	410 MB	145 MB	/opt/dynatrace/oneagent
Temporary files, runtime configuration	200 MB	200 MB	200 MB	/var/lib/dynatrace/oneagent/agent/runtime
Logs	1 GB	1 GB	1 GB	/var/log/dynatrace/oneagent
Crash reports, memory dumps	3 GB	3 GB	3 GB	/var/lib/dynatrace/oneagent/datastorage
Additional space required for updates	1.9 GB	728 MB	226 MB
Total	7.1 GB	5.3 GB	4.6 GB

Well, when comparing the two, this actually seems to go against the other recent post:

https://www.dynatrace.com/news/blog/faster-time-to-value-with-enhanced-handling-of-oneagent-runtime-...

"The Linux version of OneAgent has been reduced in size by over 20%."

-> Size of the Linux agent hasn't gone down by 20 %, it has actually increased? Or then the documentation is incorrect.

So from now on, the 5 GB /opt/dynatrace we've created will remain half empty, am I correct?

To manually manage this, there's an installation parameter to set the DATA_STORAGE path:

https://www.dynatrace.com/support/help/technology-support/operating-systems/linux/installation/custo...

...But there's no (at least documented) way to change that via the command line tool:

https://www.dynatrace.com/support/help/setup-and-configuration/dynatrace-oneagent/oneagent-configura...

If I want to keep using the old OneAgent LVs under /opt/dynatrace, should I then just reinstall all agents with the DATA_STORAGE parameter? That doesn't sound like fun.

I'm starting to wonder if this change was properly evaluated - at least having the possibility to opt-out would have been useful.

[Julius_Loman]

Just to add my experience:
Data storage cannot be changed, you need to reinstall the agent. It is only for memory dumps, crash reports and other "bigger" files. Likely it will be empty most of the time the agent runs.

I think log location cannot be changed at all - it has moved from /opt/dynatrace/oneagent/log to /var/log/dynatrace referencing with a symlink. I would not recommend touching that manually.

[fstekelenburg]

It's linux, so a lot can be changed, without too much effort 😉
Moving symlinks being one of the easiest. Of course with the processes stopped.
The storage configuration can add some level of complexity.

Reinstallation might be the fastest (and safest) route, especially when unix knowledge is scares.

[Julius_Loman]

Changing symlinks itself is easy, but unsupported from Dynatrace. I would bet the symlink for logs would be changed back during OneAgent upgrade. Especially logs can be hard since instrumented processes do log into that directory too.

For some changes, the only supported method is to reinstall oneagent.

[The_AM · Accepted solution]

Hi Kalle,

I'll add my 2 cents here.

-> Size of the Linux agent hasn't gone down by 20 %, it has actually increased? Or then the documentation is incorrect.

The actual compressed installer size is the one that is referred in that blog post as having decreased.

Once unpacked, it does require some space for the installation process. There is a way to calculate that required space as described on the Help page:
https://www.dynatrace.com/support/help/shortlink/oneagent-disk-requirements-linux#how-do-we-calculat...

The OneAgent actually adaptively cleans up log files based on available free space. If you need this to be adjusted, please reach out to Dynatrace ONE for support.

In some cases, you can ignore the space required for crash reports and memory dumps, and therefore you'll find that the 5GB partition you already have allocated will be sufficient. Even if you were to consider the 3GB recommendation for crashes/dump files, this is purely only an average suggestion, as different monitored applications will have different requirements based on their state and configuration.

The more critical requirements will be enforced by the installer at OneAgent deployment time, and at OneAgent update; in which insufficient space will result in agent aborting the install/update.

Andrew M.

[kalle_lahtinen]

Thanks for clearing up the "OneAgent size" reference. Overall it seems that there's no need to do any hasty reinstallations right now just for the DATA_STORAGE parameter, I suppose we'll follow for a while how it goes before deciding if any action is needed.

[Tomasz_Gajger · Accepted solution]

Hello @Kalle L.

Adding to the already great answers by Andrew and Julius, you might want to check out official documentation of OneAgent files aging mechanism, which we've recently published: https://www.dynatrace.com/support/help/shortlink/oneagent-aging-mechanism

As you have accurately noticed, DATA_STORAGE parameter can only be changed during the installation. There are certain technical constraints, which played a significant factor in our decision to not provide it as oneagentctl option and automatically migrate to the new location during the update.

Internally, we're also in the process of discussing the feasibility of introducing a similar parameter for log directory. You might want to follow this product idea: https://community.dynatrace.com/idea/233953/set-a-specific-oneagent-log-folder.html