topic Re: Extract single line CSV data into multiple columns in DQL

Extract single line CSV data into multiple columns

andre_vdveen — Thu, 01 Aug 2024 08:45:53 GMT

Hi, I'm trying to get the data in a CSV log file split into separate columns, but I'm stuck with DQL/DPL.
Any help would be super appreciated!

Also, I assume I'd need a metric for each column if I want to get the values onto a dashboard, right?
Each value separated by a comma, needs to fall into the columns below.

Here's the raw data - the masked text goes with 'Claims' or 'Batch Claims' and is called the 'Component' in the example above:

Re: Extract single line CSV data into multiple columns

andre_vdveen — Thu, 01 Aug 2024 11:12:12 GMT

Figured the 1st part out, with some help:

parse (content, "(LD:dup_timeStamp LD:component FLOAT:AvgTime INT:NoOfClaims INT:WorstTime INT:Time_Outs INT:Exceptions FLOAT:Standin )(fs=',')")

Re: Extract single line CSV data into multiple columns

y_buccellato — Fri, 23 Aug 2024 13:25:54 GMT

I'm in a similar situation.
Kinda surprise that with all the DQL power you cannot auto parse a field which contains multiple value divided by comma ",".
My love for excel still persist 😄

Re: Extract single line CSV data into multiple columns

36Krazyfists — Wed, 10 Sep 2025 20:06:37 GMT

So, you could potentially do this with a sophisticated enough DPL query, however, the main problem is that you have to know the CSV headers ahead of time.

For example, I have tons of pipelines that auto-parse JSON and XML logs data, but that's because the JSON/XML records contain the key names and the values in each record. With CSV, only the first line of a log file would potentially have the column names, and since the log parsing is done one line at a time, it has no way to know about the headers defined in some other record earlier that day or week, so you would have to build a DPL parser that defines the name of each item.