Folder Size Monitoring

[sivart_89]

I am looking to monitor the size of a folder, is this possible today with dynatrace, via a provided extension or does anyone know if this is coming down? The extension below monitors the size of files, it doesn't seem to work for folders.

Is the direction here to create a custom extension ourselves? If so we will wait for when the python datasource is available. Will this then let us write any code in python to have it ran via the extension? I saw an old post (few years old) where it was noted to use an extension but wanted to confirm this is still the case.

[Julius_Loman · Accepted solution]

Hi @sivart_89 ,
yes, you should use the extension https://www.dynatrace.com/hub/detail/filesystem-monitoring-extension-v2/ and install it directly from your environment. For size of a directory (file size) you can use the size check and use the directory. If you need it recursively, use double asterisk. So like /data/* (just for files in /data) or /data/** (files in /data/ and recursively).

[sivart_89]

Have you been able to get this to work? And I don't mean simply having a datapoint show, is it actually accurate? I have the configuration below. When running the command below I get a result of 40K. Makes sense that there is a value > 0 since we do have the custom.python.generic_command extension deployed there. However, dynatrace plots a datapoint of 0 B.

For the heck of it I also changed the config to use /opt/dynatrace/oneagent/plugin_deployment/ (trailing /), same result.

du -sh /opt/dynatrace/oneagent/plugin_deployment

[Julius_Loman]

The extension only calculates file sizes within directories, not directory sizes (du command ), so the value with du will never match exactly with the output of "du -s". It does not run the du command and it must have access to those directories - just a reminder.

In your case you should add the asterisks (wildchars), the extension assumes the path is a file and you don't have any.  So you should  reconfigure the path to be /opt/dynatrace/oneagent/plugin_deployment/** to sum the size of all files recursively in your directory.

In my case just a simple demo to showcase this is working:

# find /tmp/data -type f -exec ls -l {} \;
-rw-r--r-- 1 root root 1048576 dec 12 18:20 /tmp/data/data1
-rw-r--r-- 1 root root 10485760 dec 12 18:21 /tmp/data/subdir/data3
-rw-r--r-- 1 root root 1048576 dec 12 18:21 /tmp/data/data2

# du -sb /tmp/data
12591104        /tmp/data

I hope this helps. 

Also, be sure not to scan large volumes or limit it to run it in meaningful intervals. 

[sivart_89]

Apologies @Julius_Loman I should have picked up on that with your last post. I had du on my mind instead of recursively looking at the files and adding them up.

So let's say I have the 4 files below showing the size when doing a ls -lh. When looking in data explorer I see a value of .007 MB which doesn't make much sense to me. Doesn't the M represent MB as well? If so why would dynatrace be showing < 1 MB when there is a file that is 6 plus another > 2. For this test all the files are in the same directory with no folders so I am only doing a '<path>\*' type config (instead of **)

[Julius_Loman]

This can be for various reasons. Are you really having the path correct? Does OneAgent access to read the directory and file sizes (if there are subdirs).

Anyway, the extension calculates it this way (for my previous example):

from pathlib import Path
import glob

sum_sizes = 0
path="/tmp/data/**"
for f in glob.glob(path, recursive=True):
    print(f)
    try:
        if Path(f).is_file():
            sum_sizes += Path(f).stat().st_size
    except Exception as e:
        print(f"Could not get size for {f}: {e}")

print(sum_sizes)

[sivart_89]

I see the issue now, unfortunately user error. Thank you for your assistance here! All good now, seems to be accurate.