topic Re: Ingest only JSON portion of this log entry in Log Analytics

Ingest only JSON portion of this log entry

chrmckenna — Mon, 12 May 2025 06:16:21 GMT

I have a use case where we are ingesting an entry from the Windows Application Log using the following ingestion rule

The log event ingests fine and looks like this. You can see embedded JSON within the text. We're struggling with a processing rule that would throw out everything except the JSON portion, as we don't want to save any of that text.

"The description for this event cannot be found.\nAdditional info: \nTIDALSaMaster: {\"Date\": \"20250408\", \"Time\": \"102536\", \"JobName\": \"06 xxxxxxxxx - Consumption to S3\", \"JobStatus\": \"Completed Abnormally\"}\n"

We want to get rid of all of this part and only save the JSON

The description for this event cannot be found.\nAdditional info: \nTIDALSaMaster:

Can anyone help with the proper rule

Re: Ingest only JSON portion of this log entry

HansLougas — Tue, 08 Apr 2025 17:51:02 GMT

Hi!

You need a simple processing rule.

Set up a new pipeline in OpenPipeline, add a processor (type DQL) and add a following processor:

parse content, "DATA JSON:json"
| fieldsRemove content
| fieldsRename content = json

The result will have content field with only JSON part. Then make sure to route correct data to that pipeline in the Dynamic routing section.

You may find more helpful resources here

Re: Ingest only JSON portion of this log entry

chrmckenna — Tue, 08 Apr 2025 17:56:34 GMT

Thanks !!

I have not used the pipeline stuff yet so let me give this a shot.