There are times (traditionally within containerized environments) where the agent is properly injected and running but its log file cannot be placed within the normal location on the host, i.e. /opt/dynatrace/oneagent/log.
By using these two commands on a Linux host, or within a Linux based container, one is able to track down where an agent's log file is being stored in case of the file not being in the expected location for either automated or manual retrieval.
- ls -l /proc/<PID of Process in Question>/fd/
- cat /proc/<PID of Process in Question>/fd/<file descriptor number>
Example:
user@host:~$ ps aux | grep tomcat7
techsup+ 12319 0.0 0.0 17016 2396 pts/0 S+ 14:46 0:00 grep --color=auto tomcat7
tomcat7 14752 0.1 1.9 3584088 79556 ? Sl Apr14 16:51 /usr/lib/jvm/default-java/bin/java -Djava.util.logging.config.file=/var/lib/tomcat7/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.endorsed.dirs=/usr/share/tomcat7/endorsed -classpath /usr/share/tomcat7/bin/bootstrap.jar:/usr/share/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/var/lib/tomcat7 -Dcatalina.home=/usr/share/tomcat7 -Djava.io.tmpdir=/tmp/tomcat7-tomcat7-tmp org.apache.catalina.startup.Bootstrap start
user@host:~$ sudo ls -l /proc/14752/fd | grep oneagent
lr-x------ 1 tomcat7 tomcat7 64 Apr 22 14:27 10 -> /opt/dynatrace/oneagent/agent/bin/1.193.0.20200414-084245/any/java/oneagentjava.rmi.jar (deleted)
lr-x------ 1 tomcat7 tomcat7 64 Apr 22 14:27 11 -> /opt/dynatrace/oneagent/agent/bin/1.193.0.20200414-084245/any/java/oneagentjava.sql.jar (deleted)
lrwx------ 1 tomcat7 tomcat7 64 Apr 22 14:27 3 -> /opt/dynatrace/oneagent/log/java/ruxitagent_tomcat__bootstrap_14752.0.log
lrwx------ 1 tomcat7 tomcat7 64 Apr 22 14:27 4 -> /opt/dynatrace/oneagent/log/java/ruxitagent_tomcat__14752.0.log
lr-x------ 1 tomcat7 tomcat7 64 Apr 22 14:27 9 -> /opt/dynatrace/oneagent/agent/bin/1.193.0.20200414-084245/any/java/oneagentjava.jar (deleted)
Locate the file descriptor number for the agent's log file from the output of command 1. Be on the look out for ruxitagent_... as the start of the log file's name at the end of the directory path.
Using command 2 after having both the PID of the injected or companion process as well as the correct file descriptor number for the agent log file will allow you to open up and view the contents of the agent's log; even if the output of command 1 says that the file has been (Deleted).
user@host:~$ sudo cat /proc/14752/fd/4 | more # to read through the ruxitagent_tomcat__14752.0.log file
Once you have the log file open, it can be reviewed to learn what state the OneAgent is running in (IE if its been disabled automatically upon reaching out to the server or if its reaching out to an unexpected environment) and any other sorts of issues it is experiencing beyond being unable to write its log to the expected location.
As outlined above, you would be providing the PID of the process in question, whether it be the PID of the java process or the PID of a Go process etc. within the container or running natively on a host.
- Note: When it comes to the web server agents (Apache, Nginx, IIS) and looking for/into their logs, in this case you would instead use the PID of the companion process for said Web Server if such a companion process is visible.
After having run both commands you have the confirmation evidence that 1) the agent is indeed injected into the app (we would not have an agent log file without injection) and 2) you should have confirmation that its probably running into networking issues that would lead to the agent not being able to reach out to the cluster to report in its metrics. You also have evidence to show that the agent also cannot properly write to the intended log file location so the environment may have other configuration/permission level issues.
