Doing log enrichment without logs in a disk file

[AntonioSousa]

Log enrichment from log files works particularly well, but a client got me thinking in the last days. The question is quite simple: why do we have to have logs written to a file, to have them read by Dynatrace, and only then matched with Purepaths?

In my case, I'm dealing with an application in .Net, but it could really apply to several other situations. We can use both Serilog and Microsoft Logging Extensions, as in the documentation.

The latter has a well known interface, and OneAgent should be able to hook into it. Not sure how we could tell Microsoft Logging Extensions to sink the logs into something like /dev/null though...

Serilog has more options available, and there seems to even exist a sink for Dynatrace: https://github.com/iojancode/Serilog.Sinks.Dynatrace But I have doubts that this will scale, as it seems to do an API call for each log entry, but am not sure it works that way. Anyone used it?

In any case, looking for a way to do log enrichment, but without having to write to disk files, and getting it to scale. Any additional ideas?

[DanielS]

Well I agree 100% with your client, for me it is clear that using logs should be the last option in every observability model, also it is slow vs other alternatives. I understand that for other systems where it is impossible to pin a webhook or other options it is ok to use it. Linux OS's are very powerful to deal with text files and skim it a little before passing it by to other tools.

Also I try to enforce the use of Dynatrace Openkit (where case applied) and try to reduce the use of logs files in applications or implementing also Dynatrace Openkit logging improving the interaction with Dynatrace in a friendly way. If logs are local to a machine where Oneagent is installed you can ingest them using OA. But if this option is not available you can use https://www.fluentd.org/  also a plugin for Dynatrace is available.

I know that this not cover all the cases but I hope it helps.

[AntonioSousa]

@DanielS,

That first link is interesting, but it does lack my favorite Linux processing command: awk 😎 Also, the notion of pipe is what I'm arguing here, because you can really do a lot of analysis in Linux without ever having to write to a file.

Your reference to OpenKit is very interesting, had not considered it from this perspective, but is a great point! In any case, it does not apply server-side, and especially in log enrichment with server traces (only in RUM).