DQL Tips and Tricks - Parsing Log Data Across a Multi-Line Log

chris_smerek — Fri, 24 Mar 2023 09:39:45 GMT

If you're like me, you're probably brand new to the DQL world and are quickly trying to learn everything you can when it comes to DQL queries.

I recently had a client request a query to pull 2 specific fields out of a log and display them with on the log table within Dynatrace's new Logs and Events (Powered by Grail) tab. After some time, I got a solution that worked exactly how they wanted, and figured others might want the information I discovered.

Here's the setup:

The task was to 'grab text from a multi-line log file and report it out (one being a string called ITN and an IP address called client_ip)'

Here's the DQL query I used:

fetch logs | filter contains(content, "Order modify unsuccessful") and dt.process.name == "tomcat" | parse content, "DATA 'Order modify unsuccessful' SPACE STRING:ITN" | parse content, "DATA 'clientIpAddress' LD ':' DQS:client_ip" | fields timestamp, content, ITN, client_ip

And here's what the output table looks like:

The key points to note on this query:

The 'DATA' content type is used to parse multi-line logs
- This had me stumped for a while as I was using LD (which only works on a single line of log data)
The single quote 'Order modify unsuccessful' sets us up to point to the log data right after this string
- SPACE picks up a white space in between the last string and the next set of data we want to parse
'STRING:ITN' tells the query to look for a string as the next set of log data, and save that string to a variable called 'ITN
The next parse line is very similar to the previous one, except we want to pull out the IP address that's enclosed between double quotes
- 'DQS:client_ip' pulls out any data that exists between a set of double quotes, and saves it to the variable called 'client_ip'
  - Technically I could have used IPADDR here, but DQS made it easier for me to get the exact data
The final line of the query just gives us a clean output table which displays the relevant information, namely the timestamp of the log, log content, and our 2 custom variables

That's it! I've posted a mock log file below this as well if folks want to give it a try on their own time:

2023/03/15 16:05:48.279 [ERROR] http-ono-8080-heyo-117 (com.test.are.we.OrderES) -> {"testTransactionID":"123-456-789-10101-F90000000009","clientIpAddress":"192.168.12.122"}, ErrorException: [2405] Order modify unsuccessful (AB23JK) at com.website.are.bll.order.OrderBLLImpl.modifyOrder(OrderTESTImpl.java:1111) at com.website.are.ws.OrderWS.modifyOrder(OrderWS.java:222) at java.lang.reflect.Method.invoke(Method.java:333) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:444)

Hope this is helpful to anyone just starting out with DQL queries!

Re: DQL Tips and Tricks - Parsing Log Data Across a Multi-Line Log

ChadTurner — Fri, 12 May 2023 19:39:48 GMT

Thank you for these tips and tricks @chris_smerek

topic DQL Tips and Tricks - Parsing Log Data Across a Multi-Line Log in Dynatrace tips

DQL Tips and Tricks - Parsing Log Data Across a Multi-Line Log

Re: DQL Tips and Tricks - Parsing Log Data Across a Multi-Line Log