Ingest only JSON portion of this log entry

[chrmckenna]

I have a use case where we are ingesting an entry from the Windows Application Log using the following ingestion rule

The log event ingests fine and looks like this.  You can see embedded JSON within the text.  We are struggling with a processing rule that would throw out everything except the JSON portion as we do not want to save any of that text.

"The description for this event cannot be found.\nAdditional info: \nTIDALSaMaster: {\"Date\": \"20250408\", \"Time\": \"102536\", \"JobName\": \"06 xxxxxxxxx - Consumption to S3\", \"JobStatus\": \"Completed Abnormally\"}\n"

We want to get rid of all of this part and only save the JSON

The description for this event cannot be found.\nAdditional info: \nTIDALSaMaster:

Can anyone help with the proper rule

[HansLougas · Accepted solution]

Hi!

You need a simple processing rule.

Set up a new pipeline in OpenPipeline, add a processor (type DQL) and add a following processor:

parse content, "DATA JSON:json"
| fieldsRemove content
| fieldsRename content = json

The result will have content field with only JSON part. Then make sure to route correct data to that pipeline in the Dynamic routing section.

You may find more helpful resources here

[chrmckenna]

Thanks !!

I have not used the pipeline stuff yet so let me give this a shot.