Log source value in Dynatrace when using Wildcards

[sivart_89]

Fairly new to log monitoring, we are doing a POC with it. Anyone find it strange you cannot see the EXACT filename unless you actually setup the custom log source configuration rule to have an exact path (aka not using wildcards).

Let's say I have any number of directories (the list is never static) within the folder structure below. Within those directories I have a file by the name of <some_name.log> I could do something like /logs/*/*/*.log to capture all of these files but then when I query within logs and events in dynatrace, I see the log.source value as the custom log source rule matcher I setup, in this example '/logs/*/*/*.log'.

I thought the whole purpose of a log monitoring tool was to be able see the exact file the issue occurred in but instead with dynatrace I would have to still logon to the host and do my own greps to find the file where the message was logged from. I'll also add that yes you can automate the log source rules via api calls (to create the exact matcher) but are we really expected to have potentially hundreds of matchers and then also have to worry about when a new directory is spun up, adding that new matcher?

I could very well be missing something here, looking for feedback here.

Example folder structure below. Note, within each directory can be any number of directories which within them, could have any number of log files.

/logs/sbx
/logs/dev
logs/test
/logs/qa

[ben_wrightson]

At the moment, this is "by design" apparently. I also have a customer struggling with this (they're just using the '#' wildcard in their filenames but also would like to know the concrete filename the entry was pulled from).
I'd be curious whether anyone else has an idea.

[AlvinPeterson]

We have encountered similar issues and disappointment with this.  What completely stumps me though is why the documentation for defining custom log sources under Log File Matching states "Custom log sources can contain wildcards..." when using them may very well break your ability to search the data you just defined for ingestion.  If you use a * anywhere in the middle of the path to the log file the UI will provide a link to your newly defined file as expected.  But here's the challenge, you can't search this new log source.  When you try the UI presents the following message: "Unexpected error: Invalid DQL query: [2,10] MATCHES_VALUE: Wildcard '*" in the middle is not supported".  This can be worked around by playing with the advanced search, but I can't imagine explaining this to everyone at the company that wants to search their data.  I suppose another option/workaround might be fully qualifying the path and filename (removing the wildcard altogether) for every log source that's not automatically discovered but yikes!  

I'm going to assume this is all just something I've not given enough thought to yet because I know Dynatrace is better than this.  🙂

[Joachim_Erdei]

Hi Alvin,
there are two issues present here:
1. If you have custom log sources with wildcards, these wildcards are used to match files, but they are not resolved on UI, i.e. log source name still has wildcards. Shortly speaking, this is good for some use cases, and bad for other ones. Having acknowledged these shortcomings, we came up with an idea to allow for decorating log records with an additional attribute carrying the origin file name. This is planned for CQ4'23. The solution is decided so there is a high chance to deliver it as planned.

2. You cannot use a '*' character in the middle of a log query. This character is interpreted always as a wildcard (which is not the intention here but would work in most cases) and the wildcard is allowed only at the beginning and at the end of matched value. We have a story to provide a possibility to mask the wildcard character to match it literally which would cover your use case, but unfortunately we do not have capacity to implement it in CQ4'23, at least for now. A workaround is possible in most cases. You can use "'prefix*' and '*suffix'" instead of "`prefix*suffix'" condition. I am aware that is not very elegant.

[GerardJ]

Hi @Joachim_Erdei 
I am very interested in this additional attribute with the original file name.
Will it be delivered as planned? More precise information on the delivery date ?

[Joachim_Erdei]

Hi Gerard, this is already in progress. I estimate that it should be ready in OneAgent 1.285, Cluster 1.286. Even if I misjudge the effort here, a slight move in any direction will still land in CQ1'24 - no promise here, but I am quite certain about the date personally.

[sivart_89]

Is this expected to be released soon? We are on cluster version 1.288, hoping to see this improvement soon.

[Joachim_Erdei]

Hi, unfortunately it slipped a little, but it is going to be available in Cluster 1.291

[sivart_89]

No worries, thank you for the update.

[sivart_89]

@Joachim_Erdei did this get pushed again? We are on cluster 1.291 and I've updated to the latest agent version. I don't see any change here and didn't see anything noted in the 1.291 release notes.

[Joachim_Erdei]

It is already done but it will be released in Cluster 1.292 (Agent 1.285 required). Sorry for one more delay, but it was the last one - the functionality is finished and tested and just waiting to be released.

[sivart_89]

No problem, looking forward to this one. Thank you.

[sivart_89]

Hi @Joachim_Erdei. What is the expectation with this, see release notes below. We are using OA version 1.289.135.20240508-180049 and are on cluster version 1.292.40.20240522-160528. What I was hoping to see from this is being able to see the entire log file name. Example, I setup a log ingest and custom log source rule such as /opt/dynatrace-scripts/*/*/*.log (because I don't want to be restricted to specific file names and am essentially trying to capture everything within /opt/dynatrace-scripts recursively). When viewing things within Logs and events I still see the log.source as /opt/dynatrace-scripts/*/*/*.log instead of showing the actual file that contains the content.

Are we not able to see the actual file names here? It seems in order to see the file name we have to spell out the file name in the log ingest rule but how is that efficient? Are we expected to have a ton of log ingest rules just so that we can see the file name? It doesn't do us much good to see the log content but have no clue what file it is coming from, instead just knowing it came from somewhere within /opt/dynatrace-scripts.

Release notes: https://docs.dynatrace.com/docs/whats-new/release-notes/saas/sprint-292#dynatrace-service-broker

[GerardJ]

Hi @sivart_89 
I asked myself the same question and found this information in the documentation : Custom log source - Dynatrace Docs

I haven't tested it yet.

[Joachim_Erdei]

Hi @sivart_89 ,
as @GerardJ pointed, there is an additional config action you need to take, although you obviously do not need to provide any new definitions for specific files (which was already possible before the change). The action you need to take it to edit a custom log source rule with wildcards specifying custom attribute generation. That way each log record reported under this custom log source will have an additional attribute(s) which can contain e.g. full path (specific for given log record, without any wildcards; see the example with "${0}"), or full filename, or even something more customized which can be composed of elements matched to wildcards and any custom string. (You can even utilize the functionality to tag log sources if you wish.)

If you are wondering why just not change the "log.source" attribute - we could not because (1) it is not always what the customers want, (2) it would break backwards compatibility, so even customers wanting it could find their configurations broken after such a change. Hence the idea to allow for defining additional attribute if needed.

In short, if you want to have it working:
- find your custom log source rule and edit it:
- provide the attribute name which you would like to have as path-with-no-wildcards as the key
- provide the attribute value "${0}" (no quotes)
The change is effective for log records sent after configuration change (applying configuration changes may take up to ~1 minute on Log Agents).

If you have more specific case, please see the docs.

If something is still unclear, do not hesitate to ask - we will try to help 🙂

[sivart_89]

Thank @Joachim_Erdei and @GerardJ. I have tested this out and do see the full file path now after adding attribute log.file.path with the key of ${0}. This is a great improvement here, thank you 😊

[sivart_89]

@Joachim_Erdei one last thing I'll say here as I was looking around some more with this. I was racking my brain trying to figure out why I was not seeing the log.file.path attribute in one of our other environments. I found out that we were indeed seeing that attribute, as I was able to see it via the DQL below but I could not see the attribute in the 'Available attributes list' as shown in the screenshot below.

It seems you have to add a filter for that attribute to get it to actually show in the 'Available attributes list'? After I pulled up a log and added the log.file.path attribute as a filter, as shown in the 3rd screenshot then I was able to see it in the list of attributes. Surely this isn't expected behavior right? Why would that attribute not just show without additional filtering here to get it to show in the available list?

fetch logs
| filter isNotNull(log.file.path)
| limit 50

[ben_wrightson · Accepted solution]

Similar product idea: https://community.dynatrace.com/t5/Product-ideas/Log-folders-to-be-visible-populated-when-using-wild...