10 Mar 2021 07:39 AM - edited 10 Mar 2021 07:41 AM
Hi all, We use the dynatrace to monitor our instances in the AWS (CentOS). We spotted that there are a huge amount of core files in the root (/) directory. First we tried to disable the SSM agent, but it helped not so much. Deep analysis showed that the problem could be in the dynatrace oneagent. The audit enabled was showing that the core file is generated when we are trying to reinstall httpd via rpm.
Here is the entry of the auditd watch that is confirming the generation of the core file by the dynatrace oneagent: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- type=PROCTITLE msg=audit(09/03/21 18:34:08.809:257082) : proctitle=/opt/dynatrace/oneagent/agent/rdp -p 2388 -P 2388 -e rpm -s 31 type=PATH msg=audit(09/03/21 18:34:08.809:257082) : item=1 name=//core.2388 inode=738850 dev=ca:01 mode=file,700 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:root_t:s0 objtype=CREATE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 type=PATH msg=audit(09/03/21 18:34:08.809:257082) : item=0 name=// inode=64 dev=ca:01 mode=dir,555 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:root_t:s0 objtype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 type=CWD msg=audit(09/03/21 18:34:08.809:257082) : cwd=/ type=SYSCALL msg=audit(09/03/21 18:34:08.809:257082) : arch=x86_64 syscall=open success=yes exit=5 a0=0x7ffe8e0528e0 a1=O_WRONLY|O_CREAT a2=0777 a3=0x2 items=2 ppid=86 pid=2389 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=rdp exe=/opt/dynatrace/oneagent/agent/lib64/oneagentdumpproc subj=system_u:system_r:kernel_t:s0 key=core_support
10 Mar 2021 07:49 AM
OneAgent analyses process crashes for your applications. This process oneagentdumpproc is responsible for the analysis and it needs to dump this process.
Basically one of your processes is crashing. Do you see any events on the host in Dynatrace about process crashes?
Here is a description how it works under the hood
https://www.dynatrace.com/support/help/how-to-use-dynatrace/diagnostics/crash-analysis/
It should not produce crash dumps in root directory - maybe there is some misconfiguration or agent is unable to store the dump in the DATA_STORAGE directory.
10 Mar 2021 01:57 PM
Thanks for the answer,
I checked that there are no alerts in the Dynatrace dashboard. Meanwhile the gdb tool said that the rpm is crashing as I suppose
Core was generated by `/usr/bin/rpm --dbpath /var/lib/rpm --queryformat %{NAME}|CSTOK|%{EPOCH}|CSTOK|%'.
Program terminated with signal 31, Bad system call.
There are many monitoring tools installed that could potentially call the rpm to check the packages version, etc.