Host not appearing in Dynatrace after fresh OneAgent install

[ryan_korteway]

Summary

There are times when you may deploy the OneAgent to a host and see the installation complete successfully, generate some logs for the OneAgent host monitoring process named ruxitagent_host_<PID>.0.log, and yet still not see said host in the Dynatrace UI. 

Problem

If you have a successful install and OneAgent host monitoring logs, ruxitagent_host_<PID>.0.log files, then your first step should be to open the log files and search for a string of several equal signs ===== in a row as this is a distinct sign of OneAgent deactivation for numerous reasons within the logs.

A few examples:

<Timestamp and thread number> info    [native] ============================== Agent inactivated ============================== (no more data will be sent by this agent, except heartbeats and config requests.)

or

<Timestamp and thread number> info    [native] ============================== Agent DEACTIVATED ============================== (it means the agent has been rejected by the server. no more data will be sent by this agent, except heartbeats.)
<Timestamp and thread number> info    [native] Exception handling periodic task heartbeat: received message with error code: REQUEST_REJECTED - sender's message: TokenFilter: Tenant token is invalid

Troubleshooting steps

Firstly it would be best to navigate to the oneagent log file location for your particular Deployment OS and OneAgent style.
Linux: https://docs.dynatrace.com/docs/shortlink/oneagent-disk-requirements-linux#sizes 
Aix: https://docs.dynatrace.com/docs/ingest-from/dynatrace-oneagent/installation-and-operation/aix/instal... 
Windows: https://docs.dynatrace.com/docs/shortlink/oneagent-disk-requirements-windows#oneagent-directories-an... 
Kubernetes: https://docs.dynatrace.com/docs/shortlink/dto-storage-requirements#oneagent-volumes 

As you can see above, the string of ===== is a good marker for signs of the OA being deactivated for some reason upon start up or interacting with the DT Cluster as you can see in the second example.

Sometimes there are more lines that reflect information about why the disabling occurred depending on the nature of the disabling, you may need to scroll slightly above or below the main message line to dig up more information on why the disabling occurred.

Example 1 expanded:

<Timestamp and thread number> warning [native] Unable to send/handle initial cluster time request: received message with error code: REQUEST_REJECTED - sender's message: TokenFilter: Tenant token is invalid (2ms)
.... <approximately 80 lines later> ....
<Timestamp and thread number> info    [native] ============================== Agent DEACTIVATED ============================== (it means the agent has been rejected by the server. no more data will be sent by this agent, except heartbeats.)

Example 2 expanded:

<Timestamp and thread number> info    [native] Successfully sent initial setup request (24ms)
<Timestamp and thread number> info    [native] Got initial setup response (server 0x00000004)
<Timestamp and thread number> info    [native] ConfigFromServer: Updating active state to OFF (server 0x00000004)
<Timestamp and thread number> info    [native] Disablement reason changed to: [HOST_MONITORING_DISABLED]
<Timestamp and thread number> info    [native] ============================== Agent inactivated ============================== (no more data will be sent by this agent, except heartbeats and config requests.)
<Timestamp and thread number> info    [native] Agent configuration updated; agent is now inactive

The potential spacing between the reason and the inactivation message can be large due to possible start up config information printing etc.

Resolution

When it comes to "Invalid Tenant Token" messages from Example 1, you can use the oneagentctl tool to fetch the tenant token from the failing host and compare its value against a currently reporting host.

oneagentctl tool location: https://docs.dynatrace.com/docs/ingest-from/dynatrace-oneagent/oneagent-configuration-via-command-li... 

When it comes to "HOST_MONITORING_DISABLING" messages from Example 2, Navigate to the OneAgent logs (as shared above) and locate the OSI ID file from the ruxitagent_host_<PID>.0.log files and then we can use this OSI ID to create a usable dynatrace link to get to the necessary settings page:

// ruxitagent_host_<PID>.0.log file
.....
<Timestamp and Thread Info> info [native] OSI ID ............. 0xabcdef1234567890

//Take the OSI ID to insert it into a Dynatrace Settings URL:
https://<Environment SAAS ID>.apps.dynatracelabs.com/ui/apps/dynatrace.classic.hosts/ui/settings/HOST-abcdef1234567890/builtin:host.monitoring?gtf=-2000d&gf=all

https://<DT managed cluster URL>/e/<Environment UUID>/ui/apps/dynatrace.classic.hosts/ui/settings/HOST-abcdef1234567890/builtin:host.monitoring?gtf=-2000d&gf=all 

Note: the trimmed 0x from the OSI ID string to the DT URL address. 

Additionally, the above example URL has a very large time frame (last 2,000 days , gtf=-2000d) because the settings page for the host will fail to load if the change was made say 30 days ago and if one is navigating the DT UI with the default timeframe of last 2 hours. 

What's next

The above list of examples is not exhaustive and other blockers for host monitoring do exist so if you are not seeing reasons like either of the above examples or are having issues with crafting your environment link to the host settings page to enable the host monitoring, then please open a Dynatrace Support ticket.

It would be best to open said Support Ticket with a complete OneAgent Support archive as explained on the page https://www.dynatrace.com/support/help/setup-and-configuration/dynatrace-oneagent/oneagent-configura... 

Or with as many OneAgent logs as you can provide for validating the state of the OA install and its current state when it comes to configuration and cluster communication.

Please provide only the first few characters of the --get-tenant-token outputs so that we can validate the token value against what we can see in the Dynatrace Support Backend, sharing the entire token would reduce its security and is not recommended.