DQL Tips and Tricks - Parsing Log Data Across a Multi-Line Log

[chris_smerek]

If you're like me, you're probably brand new to the DQL world and are quickly trying to learn everything you can when it comes to DQL queries.

I recently had a client request a query to pull 2 specific fields out of a log and display them with on the log table within Dynatrace's new Logs and Events (Powered by Grail) tab. After some time, I got a solution that worked exactly how they wanted, and figured others might want the information I discovered.

Here's the setup:

• The task was to 'grab text from a multi-line log file and report it out (one being a string called ITN and an IP address called client_ip)'

Here's the DQL query I used:

fetch logs
| filter contains(content, "Order modify unsuccessful") and dt.process.name == "tomcat"
| parse content, "DATA 'Order modify unsuccessful' SPACE STRING:ITN"
| parse content, "DATA 'clientIpAddress' LD ':' DQS:client_ip"
| fields timestamp, content, ITN, client_ip

And here's what the output table looks like:

The key points to note on this query:

• The 'DATA' content type is used to parse multi-line logs
  • This had me stumped for a while as I was using LD (which only works on a single line of log data)
• The single quote 'Order modify unsuccessful' sets us up to point to the log data right after this string
  • SPACE picks up a white space in between the last string and the next set of data we want to parse
• 'STRING:ITN' tells the query to look for a string as the next set of log data, and save that string to a variable called 'ITN
• The next parse line is very similar to the previous one, except we want to pull out the IP address that's enclosed between double quotes
  • 'DQS:client_ip' pulls out any data that exists between a set of double quotes, and saves it to the variable called 'client_ip'
    • Technically I could have used IPADDR here, but DQS made it easier for me to get the exact data
• The final line of the query just gives us a clean output table which displays the relevant information, namely the timestamp of the log, log content, and our 2 custom variables

That's it! I've posted a mock log file below this as well if folks want to give it a try on their own time:

2023/03/15 16:05:48.279 [ERROR] http-ono-8080-heyo-117 (com.test.are.we.OrderES) -> {"testTransactionID":"123-456-789-10101-F90000000009","clientIpAddress":"192.168.12.122"},
ErrorException: [2405] Order modify unsuccessful (AB23JK)
	at com.website.are.bll.order.OrderBLLImpl.modifyOrder(OrderTESTImpl.java:1111)
	at com.website.are.ws.OrderWS.modifyOrder(OrderWS.java:222)
	at java.lang.reflect.Method.invoke(Method.java:333)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:444)

Hope this is helpful to anyone just starting out with DQL queries!

[ChadTurner]

Thank you for these tips and tricks @chris_smerek