30 Nov 2023 09:35 PM - last edited on 01 Dec 2023 10:49 AM by MaciejNeumann
I'm trying to create a log processing rule to parse each key value pair from the Open Telemetry log record.
I was able to extract first part of the key value pairs with following KVP DQL query. However, there are nested key value pairs and another section with "ScopeValues" that I am unable to extract.
DQL Pattern match: "KVP{'LogRecord.' STRING:key SPACE*? LDATA:value EOL}:attr"
Does anybody know how to extract all key value pairs from the following event:
LogRecord.Timestamp: 2023-11-27T18:07:18.0630011Z
LogRecord.TraceId: a21aaa7581aaa861072f6ae851fe6a3b
LogRecord.SpanId: d073dfe9ea0cea02
LogRecord.TraceFlags: Recorded
LogRecord.CategoryName: Document.Mapper
LogRecord.Severity: Warn
LogRecord.SeverityText: Warning
LogRecord.FormattedMessage: MS-MIP Log: Function Name: std::shared_ptr<mipns::EventContext> | Message: CreateEventContext: Provided Cloud type is invalid
LogRecord.Body: MS-MIP Log: Function Name: {functionName} | Message: {message}
LogRecord.Attributes (Key:Value):
net.host.name: export-api.abc.com
http.method: POST
http.scheme: http
http.target: /export
http.url: http://export-api.abc.com/export
http.flavor: 1.1
http.user_agent: PostmanRuntime/7.29.2
user: aaaaaaaa-a0cd-4ed4-9116-1111f1rrgr3
user_email: jon.doe@abc.com
role: platform-user
http.route: export
LogRecord.ScopeValues (Key:Value):
[Scope.0]:SpanId: d07aaae9eadddd02
[Scope.0]:TraceId: a21aaa7333aaa8610343433e851fe67a4
[Scope.0]:ParentId: 150fe5bdddddd
[Scope.1]:ConnectionId: 0HMEEEEEO73SDB
[Scope.2]:RequestId: 0HMEEEEEO73SDB:00000002
[Scope.2]:RequestPath: /export
[Scope.3]:ActionId: eeeeeeee-7df1-3333-gggg-gggggg
[Scope.3]:ActionName: Export.Api.Controllers.ExportController.ExportFile (Export.Api)
Resource associated with LogRecord:
dt.entity.process_group_instance: PROCESS_GROUP_INSTANCE-WIEIEIEF
dt.entity.host: HOST-A922CFEIEIEIEIE
dt.entity.host_group: HOST_GROUP-18F127EJEJEJEJEJ
dt.host_group.id: abc-app-01_test
dt.security_context: abc-app-01_test
host.name: abcd-3333333-vmss00003T
service.name: Export.Api
telemetry.sdk.name: Open.Telemetry
telemetry.sdk.version: 1.0.0.0
telemetry.sdk.language: dotnet
warn: Document.Wrapper[0]
MS-MIP Log: Function Name: void mipns:(const string&) | Message: Inconsistent label detected.
01 Dec 2023 09:24 PM - edited 01 Dec 2023 09:26 PM
A team member create this pattern expression and it works in Notebook DQL search for pattern extraction, but some expressions are invalid in log "processor definition"
Any ideas on how to escape brackets and colons?
PARSE(content,"DATA KVP{'LogRecord.' LD:key ':'
SPACE*?
LDATA:value
EOL
}:attr
DATA KVP{SPACE LD:key ':'
SPACE*?
LDATA:value
EOL
}:subattr
LD EOL
(DATA KVP{"[Scope." [0-9] "]:" LD:key ':'
SPACE?
LDATA:value
EOL
}:scopes)?
LD EOL
KVP{LD:key ':'
SPACE*?
LDATA: value
EOL
}:resources