📌 TIP#4: Parsing non-standard timestamps embedded in long log lines using DQL

[tracegazer]

The Problem

Sometimes your application logs contain timestamps in non-standard formats buried deep inside long, unstructured log lines. The timestamp isn't at the beginning of the line and doesn't follow ISO 8601 with separators — so toTimestamp() can't parse it directly.

Example: Your logs look like this (simplified):

{hostName: app-server-01.example.com,level: WARN,message: PaymentService#validateAccount input: source=batch account=123456,serverId: node1,userId: svc-user,threadName: default task-1024,contextMap: [{traceid:a1b2c3d4e5f6},{correlationId:f47ac10b-58cc-4372-a567-0e02b2c3d479}],applicationName: payments,timestamp: 20260423T122121.131-0300}

The timestamp field 20260423T122121.131-0300 is:

• Buried at the end of a very long log line
• In a compact format without the usual - and : separators (yyyyMMdd'T'HHmmss.SSS±ZZZZ)
• Not ISO 8601 compliant — toTimestamp() won't parse it directly

The Strategy: indexOf + substring + TIMESTAMP pattern

Instead of trying to parse the entire log line, we use a surgical extraction approach:

1. Locate the timestamp key using indexOf
2. Extract from that position forward using substring
3. Parse the extracted fragment using DPL's TIMESTAMP matcher with a custom format
4. Override the record's timestamp field with the parsed value

The Solution

fetch logs
// ... your filters here ...
| fieldsAdd index = indexOf(content, "timestamp")
| fieldsAdd timestamp_log = substring(content, from:index)
| parse timestamp_log, "LD TIMESTAMP('yyyyMMdd\\'T\\'HHmmss.SSSZ'):parsed_ts LD"
| fieldsAdd timestamp = parsed_ts
| fieldsRemove index, timestamp_log, parsed_ts

Step-by-step breakdown

Step 1 — Find the position

| fieldsAdd index = indexOf(content, "timestamp")

indexOf returns the character position where "timestamp" starts in the content field. This avoids the need to parse the entire log structure.

Step 2 — Cut the string

| fieldsAdd timestamp_log = substring(content, from:index)

This gives us a short fragment like:
timestamp: 20260423T122121.131-0300}

Now we have a manageable string to parse.

Step 3 — Parse with TIMESTAMP pattern

| parse timestamp_log, "LD TIMESTAMP('yyyyMMdd\\'T\\'HHmmss.SSSZ'):parsed_ts LD"

Breaking down the DPL pattern:

• LD — matches timestamp: (any data up to the next matcher)
• TIMESTAMP('yyyyMMdd\\'T\\'HHmmss.SSSZ') — the custom format:
  • yyyyMMdd → 20260423 (year, month, day — no separators)
  • \\'T\\' → literal T (escaped single quotes in DQL string)
  • HHmmss → 122121 (hour, minute, second — no separators)
  • .SSS → .131 (milliseconds)
  • Z → -0300 (timezone offset in RFC 822 format)
• :parsed_ts — export name for the parsed timestamp
• LD — matches any trailing characters (the closing })

Step 4 — Override the timestamp

| fieldsAdd timestamp = parsed_ts

This replaces the record's timestamp field with the correctly parsed value. Dynatrace will now use this timestamp for time-based filtering, sorting, and visualization.

Step 5 — Clean up

| fieldsRemove index, timestamp_log, parsed_ts

Remove the auxiliary fields to keep the result clean.

Self-contained test query

You can validate this approach without any log data using data record():

data record(content = "{hostName: app-server-01.example.com,level: WARN,message: PaymentService#validateAccount input: source=batch account=123456,serverId: node1,userId: svc-user,threadName: default task-1024,applicationName: payments,timestamp: 20260423T122121.131-0300}")
| fieldsAdd index = indexOf(content, "timestamp")
| fieldsAdd timestamp_log = substring(content, from:index)
| parse timestamp_log, "LD TIMESTAMP('yyyyMMdd\\'T\\'HHmmss.SSSZ'):parsed_ts LD"
| fieldsAdd timestamp = parsed_ts
| fields timestamp, type(timestamp)

Expected result:

timestamp type(timestamp)

2026-04-23T12:21:21.131-03:00

timestamp

Note: the original -0300 offset is correctly converted to UTC.

Alternative: If the TIMESTAMP pattern doesn't match the offset

If the Z pattern doesn't parse your specific offset format, you can try replacing it with XX (which explicitly matches offsets without colons like -0300):

| parse timestamp_log, "LD TIMESTAMP('yyyyMMdd\\'T\\'HHmmss.SSSXX'):parsed_ts LD"

Alternative: Manual string reconstruction + toTimestamp

If the DPL TIMESTAMP matcher gives you trouble, you can manually reconstruct an ISO 8601 string and use toTimestamp():

fetch logs
// ... your filters here ...
| fieldsAdd index = indexOf(content, "timestamp")
| fieldsAdd raw_ts = substring(content, from:index+11, to:indexOf(content, "}", from:index))
| fieldsAdd iso_ts = concat(
    substring(raw_ts, from:0, to:4), "-",
    substring(raw_ts, from:4, to:6), "-",
    substring(raw_ts, from:6, to:8), "T",
    substring(raw_ts, from:9, to:11), ":",
    substring(raw_ts, from:11, to:13), ":",
    substring(raw_ts, from:13)
  )
| fieldsAdd timestamp = toTimestamp(iso_ts)
| fieldsRemove index, raw_ts, iso_ts

This transforms 20260423T122121.131-0300 → 2026-04-23T12:21:21.131-0300, which is standard ISO 8601 and toTimestamp() handles it natively.

When to use this approach

This indexOf + substring strategy is useful when:

• The log line is very long and contains many fields
• The timestamp is not at a fixed position in the line
• The log format is semi-structured (key-value pairs without strict JSON)
• You need to override the record's timestamp with the application-level timestamp instead of the ingestion timestamp

References

• DPL Time and Date
• DQL Time Functions
• DQL Extraction and Parsing Commands
• Java DateTimeFormatter patterns