21 Mar 2024 07:36 PM - last edited on 03 Apr 2024 11:43 AM by Michal_Gebacki
We are attempting to auto ingest Mulesoft logs. There are a significant amount of logs and the number will grow as new APIs are added. The log paths don't fit the default security rules which are necessary for log ingestion to occur. We attempted to use wildcards in our custom log sources rules, however, that just resulted in all files being lumped together under a log source that is the wildcard path itself instead of in individual log sources within that directory. It's also not possible to query this log source when that happens. That really defeats the purpose of using the wildcard. I went looking for a solution and found that we can create a custom security file to include the paths to the logs so that log auto ingestion can occur. https://docs.dynatrace.com/docs/observe-and-explore/log-monitoring/acquire-log-data/log-custom-sourc...
We created the file, placed it in the correct directory on the host, and restarted the oneagent as well as the monitored process. We also created a log ingestion rule for that process group in Dynatrace. Logs still weren't coming through at all.
My question is, aren't the security rules supposed to tell Dynatrace which log paths are acceptable for log ingestion? If so, after creating a log ingestion rule in Dynatrace, why wouldn't the logs begin being auto ingested? The logs meet all of the requirements. The Mulesoft process is an important process. When looking at the process we see the list of logs but also see a message that they aren't monitored.
21 Mar 2024 09:12 PM
Hello @Erica_Kelly I also use Mulesoft but to do the log ingestion you need the OneAgent installed but AFAIK the Mulesoft workers not support agent installation. I have running the extension for Mulesoft to have more insights. You can use the Log ingest API to ingest the logs.
Please, can you explain further in which host do you have the OA? Thanks.
03 Oct 2024 03:56 PM
Hi @Erica_Kelly
The security rules mechanism on OneAgent protects files on a system that are no logs from being ingested, like configs or secrets. We plan to update default settings to allow more log sources to be ingested out-of-the-box with custom log source configurations. However, I'm not sure how the log path looks in your use case, so you might still need to update security rules.
There are two principles: (1) OneAgent security rules can only restrict the log autodetection mechanism (2) OneAgent security rules can loosen or restrict custom log sources that are ingested by OneAgent.
Your approach to defining custom log sources with wildcards is the recommended way. You can use the option to enrich incoming log sources with patterns derived from log source paths or with full log paths. See advanced options under point 6 "Enrich the logs with custom attributes" in official documentation: https://docs.dynatrace.com/docs/shortlink/lma-custom-log-source#mainclscuipage for more details.