cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Open Telemetry log record field extraction with nested key value pairs

mosharref_hossn
Observer

I'm trying to create a log processing rule to parse each key value pair from the Open Telemetry log record. 

I was able to extract first part of the key value pairs with following KVP DQL query. However, there are nested key value pairs and another section with "ScopeValues" that I am unable to extract. 

DQL Pattern match: "KVP{'LogRecord.' STRING:key SPACE*? LDATA:value EOL}:attr"

Does anybody know how to extract all key value pairs from the following event:

 

LogRecord.Timestamp: 2023-11-27T18:07:18.0630011Z
LogRecord.TraceId: a21aaa7581aaa861072f6ae851fe6a3b
LogRecord.SpanId: d073dfe9ea0cea02
LogRecord.TraceFlags: Recorded
LogRecord.CategoryName: Document.Mapper
LogRecord.Severity: Warn
LogRecord.SeverityText: Warning
LogRecord.FormattedMessage: MS-MIP Log: Function Name: std::shared_ptr<mipns::EventContext> | Message: CreateEventContext: Provided Cloud type is invalid
LogRecord.Body: MS-MIP Log: Function Name: {functionName} | Message: {message}
LogRecord.Attributes (Key:Value):
    net.host.name: export-api.abc.com
    http.method: POST
    http.scheme: http
    http.target: /export
    http.url: http://export-api.abc.com/export
    http.flavor: 1.1
    http.user_agent: PostmanRuntime/7.29.2
    user: aaaaaaaa-a0cd-4ed4-9116-1111f1rrgr3
    user_email: jon.doe@abc.com
    role: platform-user
    http.route: export
LogRecord.ScopeValues (Key:Value):
[Scope.0]:SpanId: d07aaae9eadddd02
[Scope.0]:TraceId: a21aaa7333aaa8610343433e851fe67a4
[Scope.0]:ParentId: 150fe5bdddddd
[Scope.1]:ConnectionId: 0HMEEEEEO73SDB
[Scope.2]:RequestId: 0HMEEEEEO73SDB:00000002
[Scope.2]:RequestPath: /export
[Scope.3]:ActionId: eeeeeeee-7df1-3333-gggg-gggggg
[Scope.3]:ActionName: Export.Api.Controllers.ExportController.ExportFile (Export.Api)
Resource associated with LogRecord:
dt.entity.process_group_instance: PROCESS_GROUP_INSTANCE-WIEIEIEF
dt.entity.host: HOST-A922CFEIEIEIEIE
dt.entity.host_group: HOST_GROUP-18F127EJEJEJEJEJ
dt.host_group.id: abc-app-01_test
dt.security_context: abc-app-01_test
host.name: abcd-3333333-vmss00003T
service.name: Export.Api
telemetry.sdk.name: Open.Telemetry
telemetry.sdk.version: 1.0.0.0
telemetry.sdk.language: dotnet
warn: Document.Wrapper[0]
MS-MIP Log: Function Name: void mipns:(const string&) | Message: Inconsistent label detected.

 

1 REPLY 1

mosharref_hossn
Observer

A team member create this pattern expression and it works in Notebook DQL search for pattern extraction, but some expressions are invalid in log "processor definition" 

mosharref_hossn_0-1701465858430.png

Any ideas on how to escape brackets and colons?

PARSE(content,"DATA KVP{'LogRecord.' LD:key ':'
SPACE*?
LDATA:value
EOL
}:attr

DATA KVP{SPACE LD:key ':'
SPACE*?
LDATA:value
EOL
}:subattr

LD EOL

(DATA KVP{"[Scope." [0-9] "]:" LD:key ':'
SPACE?
LDATA:value
EOL
}:scopes)?

LD EOL

KVP{LD:key ':'
SPACE*?
LDATA: value
EOL
}:resources

 

 

Featured Posts