retrieve detected log sources given a host group

[imrj99]

which DQL query lets you retrieve all discovered log sources (ingested or not) for a given host group?

I am able to see these log sources in logsourcestate under dt.entity.process_group_instance, but not able to filter them, it always comes up null...

[angelalanis]

Hi, does this query help you?
You can filter by host group name and you get a list of the unique log sources.

fetch logs,  scanLimitGBytes:10
| fieldsAdd entityName(dt.entity.host)
| fieldsAdd entityName(dt.entity.host_group)
| filter matchesValue(dt.entity.host_group.name, "<HOST-GROUP-NAME>")
| summarize uniqueSources = collectDistinct(log.source)
| expand uniqueSources
| filterOut isNull(uniqueSources)

[imrj99]

unfortunately no....it only retrieves those that there is log data for (ie. already being ingested), but not for all detected log sources, even if not ingested..

using the query below I can find all the detected log sources in "dt.entity.process_group_instance" under "logSourceState", but they each come back as columns, which makes it very hard to filter, not sure why this is

fetch dt.entity.process_group_instance, from:now() - 5m
| fieldsAdd logSourceState = logSourceState
| filter isNotNull(logSourceState)
| limit 200

[angelalanis]

I see what you mean with coming back as columns, I found this workaround, hopefully it helps you.

fetch dt.entity.host
| fields id, logSourceState, hostGroupName
| filter matchesValue(hostGroupName, "extensions")
| filterOut isNull(logSourceState)
| fieldsAdd sLogSourceState = toString(logSourceState)
| fieldsAdd sLogSourceState = substring(sLogSourceState, from:1, to: stringLength(sLogSourceState)-1)
| fieldsAdd sLogSourceState = splitString(sLogSourceState,",")
| expand sLogSourceState
| fieldsAdd sLogSourceState=trim(sLogSourceState)
| parse sLogSourceState, "'\"' LD:file '\":\"storageStatus:' LD:state '\"'"
| fields id, file, state

[imrj99]

it didnt, got no results...... but i think I see where you are going,  I changed the 1st line  to dt.entity.proceess_group_instance and commented out lines 2,3 and I do get all the log sources in a row format! 

I guess from this I need to filter for specific hosts by joining which process group instance runs in which host group?  so close! 

fetch dt.entity.process_group_instance
//| fields id, logSourceState, hostGroupName
//| filter matchesValue(hostGroupName, "HOSTGROUPNAME")
| filterOut isNull(logSourceState)
| fieldsAdd sLogSourceState = toString(logSourceState)
| fieldsAdd sLogSourceState = substring(sLogSourceState, from:1, to: stringLength(sLogSourceState)-1)
| fieldsAdd sLogSourceState = splitString(sLogSourceState,",")
| expand sLogSourceState
| fieldsAdd sLogSourceState=trim(sLogSourceState)
| parse sLogSourceState, "'\"' LD:file '\":\"storageStatus:' LD:state '\"'"
| fields id, file, state

[TracingMySanity · Accepted solution]

Modified your query a bit to get the hostGroup entity attribute using the `belongs_to` relationship from the process_group_instance

fetch dt.entity.process_group_instance
| filterOut isNull(logSourceState)
| fieldsAdd sLogSourceState = toString(logSourceState)
| fieldsAdd sLogSourceState = substring(sLogSourceState, from:1, to: stringLength(sLogSourceState)-1)
| fieldsAdd sLogSourceState = splitString(sLogSourceState,",")
| expand sLogSourceState
| fieldsAdd sLogSourceState=trim(sLogSourceState)
| parse sLogSourceState, "'\"' LD:file '\":\"storageStatus:' LD:state '\"'"
| fieldsAdd host=belongs_to[dt.entity.host]
| fieldsAdd hostGroup=entityAttr(host, type:"dt.entity.host", "hostGroupName")
| fields id, file, state, hostGroup

[imrj99]

awesome!  i just need to be able to filter on a specific host group now, i think i could figure that out (maybe!!) lol

[marusla]

Hello, i have the same message no log for process instance, i try the dql but still is empty, which option i have to enable for see process instance logs, thank you