Parsing Logs for Concurrent Connections

[idudneymitchell]

I have a query into my logs that looks something like this:

fetch log
| filter matchesValue(dt.entity.process_group_instance, "the-process-im-looking-at")
| parse content, "TIMESTAMP:DATETIME LD ' 443 - ' LD:REQADDR ' ' LD ' - 200 ' LD ' ' LD ' ' LD:DURATION"
| fields timestamp, DATETIME, REQADDR, DURATION=toLong(DURATION), content

The log lines describe a request to an endpoint; each contains a datetime at the beginning, some other info in the middle, and a duration of the request in milliseconds at the end. The application cannot be deep-monitored, due to the technology choices made (enabling deep monitoring brings the whole application down), so we can't get this info the normal way; we can only harvest from the logs. So, if a request took 5 minutes, it'd look something like this:

2025-12-23 17:52:23 10.0.0.23 POST /application/api.json abcd 443 - 12.21.12.21 client-identifier - 200 0 0 300000

There's a need to know how many requests are open at the same time over some time window--let's describe a scenario to explain what we mean:
1. No requests are open at 16:29. A request is opened at 16:30; we'll call it `A`. This request takes 10 minutes to complete.
2. Two requests are opened at 16:35. These requests take 10 minutes to complete. These are called `B` and `C`.
3. `A` closes. Five new requests open between 16:40 and 16:45. These requests all take 10 milliseconds each to complete, and only 3 overlap; the other two open sequentially afterwards. These are `D`, `E`, `F`, `G`, `H`.
4. We make a timeseries with 5-minute buckets, based on info we get from parsing the logs; we want to see "1" for 16:30-16:35, "3" for 16:35-16:40, "5" or "7" for 16:40-16:45, and then "0" for 16:45 onwards.

Ideally, we would see "5" in the 16:40-16:45 bucket; this is because only 5 at most overlapped at any one time (B, C, D, E, F); however, it's also acceptable to do "7", since 7 total were active in that bucket. Is there a way to do this in DQL currently? Is there a simpler option I am missing that works in a much simpler fashion (without deep monitoring)?

I could not get this working in my experiments--I ended up pulling the parsed data down as CSV, and using Python to find the overlaps over a timeseries. This worked accurately, but it would be ideal to do this all in Dynatrace; it would enable things like graphs, alerts, and so on, which would be useful for our case.

[krzysztof_hoja]

Second case is realatively easy. The challenge is to put each transaction against an array representing intervals of interesting period and cheching if each interval ovlaps with rach particula transation. Such array can be generated by using start() funtion in timeseries command with any present metric.

data
record(timestamp=@d+16h+30m, d=10m, r="A"),
record(timestamp=@d+16h+35m, d=10m, r="B"),
record(timestamp=@d+16h+35m, d=10m, r="C"),
record(timestamp=@d+16h+41m+1s, d=10ms, r="D"),
record(timestamp=@d+16h+41m+2s, d=10ms, r="E"),
record(timestamp=@d+16h+41m+3s, d=10ms, r="F"),
record(timestamp=@d+16h+41m+4s, d=10ms, r="G"),
record(timestamp=@d+16h+41m+5s, d=10ms, r="H")

| fieldsAdd fakekey=1
| lookup [
timeseries avg(dt.host.cpu.usage), t=start(), interval:5m
| fieldsAdd fakekey=1
], sourceField:fakekey, lookupField:fakekey, fields:{t, interval}
| fields t = arrayRemoveNulls(iCollectArray(if(
(timestamp>=t[] and timestamp<t[]+interval) or
(timestamp+d>t[] and timestamp+d<=t[]+interval) or
(timestamp<=t[] and timestamp+d>=t[]+interval)
, t[])))

| expand t
| makeTimeseries  count(), time: t, interval: 5m

result looks like this: