Ingesting logs from script: best practices?

AntonioSousa — Fri, 07 Jul 2023 18:17:18 GMT

I have several cases where I need to ingest logs into Dynatrace from scripts that I control, in servers where OneAgent is running.

Now, I try to ingest them directly through the OneAgent log ingest API: https://www.dynatrace.com/support/help/extend-dynatrace/extend-logs/oneagent-log-ingest-api

I do this for several reasons:

Don't need to write them to file, where OneAgent could get them also. Don't use disk space.
I believe it's faster, but have not really measured it.
I can add meta-data to the logs ingested in a more controlled way.

I sometimes have doubts if there are disadvantages in this approach. For instance, it's way easier to dump into a file, and you don't have to code so much. You also have to be careful not committing errors. Are there other point of views, or how are you doing it yourself?

Re: Ingesting logs from script: best practices?

radek_jasinski — Mon, 10 Jul 2023 06:14:46 GMT

Hi Antonio:)

A mechanism that directly reads data from logs and sends relevant fragments to DT in the form of metrics works best. You can make a script that checks if an error has occurred, for example, and then sends the entire line of the entry related to this error to DT after the API.

Writing to files as you write has many disadvantages. You have to keep an eye on file compatibility, define appropriate data overwrite cycles, you use valuable resources, etc.

Regards,

Radek

topic Ingesting logs from script: best practices? in Open Q&A

Ingesting logs from script: best practices?

Re: Ingesting logs from script: best practices?