Where does the OneAgent store log files around custom plugins?

[larry_roberts]

I am currently working on a plugin for Tibco EMS running on Linux and trying to do some debugging. Dynatrace is accepting the plugin and everything seems ok from the GUI perspective, however no metrics are collected or displayed and I am trying to figure out why. In doing so I have been trying to locate where Dynatrace would store the logs around a OneAgent plugin. To my knowledge this should do it....

import logging

from ruxit.api.base_plugin import BasePlugin

But I am not finding anything. Any help would be appreciated. Thanks!

[Julius_Loman · Accepted solution]

It's right in the log directory of the agent - /opt/dynatrace/oneagent/log/plugin
Here you can find the logs. It's little messy, but typically you can find that info easily.

For ActiveGate plugins it is /var/lib/dynatrace/remotepluginmodule/log/remoteplugin/ and logs are in separate directories for each plugin.

[larry_roberts]

Thanks @Julius L. I was able to finally locate them. I thought maybe I was looking at the wrong ones, but then I did find in the logs what I was looking for. Much appreciated!

[radme]

hello Julius,

 I deployed plugin for the first time on a host manually under plugin_deployment directory,  and trying to troubleshoot it as it is not reporting to the UI. I dont see this path /opt/dynatrace/oneagent/log/plugin, the only directories i see under /opt/dynatrace/oneagent are 

drwxr-xr-x 1 root root 288 Apr 12 11:06 agent
drwxr-xr-x 1 root root 136 Apr 8 13:10 plugin_deployment


However, i do see some logs under /var/log/dynatrace , but none report on the actual working of the plugin or any debug messages i have inserted . Please advise thanks you!

ls -alt /var/log/dynatrace/oneagent/
total 0
drwxrwxrwt 1 root dtuser 144 Apr 12 11:16 .
drwxrwxr-x 1 root dtuser 1422 Apr 12 11:06 loganalytics
drwxrwxr-x 1 root dtuser 1610 Apr 12 11:06 plugin
drwxrwxrwt 1 root dtuser 490 Apr 12 11:06 process
drwxrwxr-x 1 root dtuser 1244 Apr 12 11:06 os
drwxrwxr-x 1 root dtuser 902 Apr 12 11:06 network
drwxrwxr-x 1 root dtuser 472 Apr 12 11:06 extensions
drwxrwxr-x 1 root dtuser 310 Apr 12 11:06 watchdog
drwxrwxrwt 1 root dtuser 1188 Apr 7 13:29 sdk
drwxrwxr-x 1 root dtuser 178 Apr 7 13:29 installer
drwxrwxrwt 1 root dtuser 0 Apr 7 12:05 dumpproc

[Julius_Loman]

Hello,
the logs are now located in /var/log, previously they were in /opt/dynatrace/oneagent/log (default).  

Logs for OneAgent extensions are in plugin directory, look for the ruxitagent_pluginagent* logs.  Maybe your extension was not loaded due some misconfiguration.

[radme]

Hi Julius,

Yes i did find that log , thanks for confirming it. Unfortunately parsing the entire log for the plugin no errors. Also the module is made in python and i had couple of print statements that show up well on simulator but nothing in these logs for them. This plugin was to capture zombie /defunct count on OS.

Few of the initial logs below:

2021-04-12 15:06:43.727 UTC [2056b8f9] info    [native] Plugin custom.python.zombie_plugin_host compatibility: true reason: no requiredAgentVersion specified, path: /app1/dynatrace/oneagent/plugin_deployment/custom.python.zombie_plugin_host

2021-04-12 15:06:43.728 UTC [2056b8f9] info    [native] Plugin added to newest compatible plugin collection: /app1/dynatrace/oneagent/plugin_deployment/custom.python.zombie_plugin_host

I am not able to get any visibility of what is going at the plugin unlike the simulator where the print statement can be seen. Any suggestion or idea on how to troubleshoot it, really appreciate it !

thanks again

[Julius_Loman · Accepted solution]

Well, first, you should not print output, I believe the stdout won't be captured. For the requiredAgentVersion, you should add it to plugin.json, see docs.

For printing out debug messages, you should use the logger:

self.logger.setLevel(logging.DEBUG)
self.logger.debug("MyMessage")
                                                                                                                                                    Anyway first look at messages such as:
2021-04-12 17:01:19.905 UTC [24eb088f] info    [native] 140252776462080(MainThread) - [do] Plugin loop status: time_taken: 2.0036761360242963, engines_info: ['<PluginEngine, meta_name:custom.python.sslcertcheck_plugin '
 'id:0x7f8f170b6ac8>, executions:0 ',
 '<PluginEngine, meta_name:ruxit.python.postgresql id:0x7f8f170c8128>, '
 'executions:0 ',
 '<PluginEngine, meta_name:ruxit.python.phpfpm id:0x7f8f170c8390>, '
 'executions:0 ',
 '<PluginEngine, meta_name:ruxit.python.mongodb id:0x7f8f170c85f8>, '
 'executions:0 ',
 '<PluginEngine, meta_name:ruxit.python.docker id:0x7f8f170c8860>, '
 'executions:1348 ',
 '<PluginEngine, meta_name:dynatrace.python.ntp id:0x7f8f170c8ac8>, '
 'executions:0 ',
 '<PluginEngine, meta_name:dynatrace.python.heartbeat id:0x7f8f170c8d30>, '
 'executions:1349 '],

This will show you how many times an extension query method has been executed.

[radme]

Hi Julius,

  I dont see even one execution of the custom plugin when i search for "PluginEngine". That is weird considering it says plugin added in the  ruxitagent logs.

All positive messages there. I'd expect it to throw some "error" related to the custom plugin.. Nothing

[Ben_Harvey · Accepted solution]

I found the same issue when developing my first plugin.

Once the plugin has been deployed, the agent will need to be stopped and started before you'll see anything in the ruxitagent.pluginagent log file. Do this by going to:

settings, monitoring overview, hosts.

Once stopped allow a minute or so for the plugin to stop, then allow 5-10 min after restart.

May be helpful for others in future.