Expand Tags values

[RPbiaggio]

Guys, I need some help. I have my process group entities with some tags and I need to put two fields in one, one is the value of the tag "[Kubernetes]componentid:" and the other, the value of the tag, "env". I can't seem to put the two together. I've tried a few things, but it didn't work. Can anyone help?

fetch dt.entity.process_group, from:now() - 1h
|fields entity.name, id, primeiraVisualizacao = lifetime[start], getDate = now(), tags
|expand tags
|fieldsAdd idComponent = replaceString(tags,"[Kubernetes]componentid:", "")
|filter (matchesPhrase(tags,"[Kubernetes]componentid"))
//|filter matchesPhrase(tags,"env:")
| sort primeiraVisualizacao desc
| limit 6000

[sinisa_zubic]

Hi @RPbiaggio 

Are you looking for something like this

fetch dt.entity.process_group, from:now() - 1h
|fields entity.name, id, primeiraVisualizacao = lifetime[start], getDate = now(), tags
|expand tags, alias:ts
| parse toString(ts), """(('['LD:tag_context ']' LD:tag_key (!<<'\\' ':') LD:tag_value)|LD:tag_key| (LD:tag_key (!<<'\\' ':') LD:tag_value)|LD:tag_key)"""
| fieldsAdd componentId = if (tag_key == "componentid" and tag_context == "Kubernetes", tag_value)
| fieldsAdd environment = if (tag_key == "env", tag_value)
| summarize { 
  componentId = takeAny(componentId), 
  environment = takeAny(environment), 
  primeiraVisualizacao = takeAny(primeiraVisualizacao)}, by:{id}
| filter isNotNull(componentId) and isNotNull(environment)
| fieldsAdd myNewField = concat(componentId, ":", environment)
| sort primeiraVisualizacao desc
| limit 6000

So basically I am first adding 2 new fields (environmentid and environment) after expanding a parsing the tags array. Then I need to summarize (group) the records by process group id. 

Best,
Sini

[RPbiaggio]

Hi, @sinisa_zubic!!
This is exactly what I need, but for example, my "env" tag still doesn't appear. See this example where I added the tag manually. It does not return information in the DQL result.

Thank you for your help.

[sinisa_zubic]

not sure why you have in line number 11 an "OR" instead of "AND". I have added that one because I am assuming that process groups with tag "componentid" must have tag "env". 

but anyway, there might be some wired characters part of the tag and therefore the query is not working.
can you maybe execute following query and let me know the result? basically it should only show the 2 relevant tags for the process group you have highlighted. And please share the result in "Raw" view in notebooks.

fetch dt.entity.process_group
| filter id == "PROCESS_GROUP-2AD8E1DA4761BD8E"
| expand tags
| filter contains(tags,"environmentid") or contains(tags,"env")

[RPbiaggio]

I put the "OR" just to give me a result, it was blank with both conditions. Both tags must actually exist.

Below is the result, as requested.

fetch dt.entity.process_group | filter id == "PROCESS_GROUP-2AD8E1DA4761BD8E" | expand tags | filter contains(tags,"[Kubernetes]componentid") or contains(tags,"env")

{
  "records": [
    {
      "entity.name": "",
      "id": "PROCESS_GROUP-2AD8E1DA4761BD8E",
      "tags": "[Kubernetes]componentid:97ffed68873361906431ca240cbb3524"
    },
    {
      "entity.name": "",
      "id": "PROCESS_GROUP-2AD8E1DA4761BD8E",
      "tags": "env:DSV"
    },
    {
      "entity.name": "",
      "id": "PROCESS_GROUP-2AD8E1DA4761BD8E",
      "tags": "[Kubernetes]tags.datadoghq.com/env:dsv"
    }
  ],
  "metadata": {
    "grail": {
      "canonicalQuery": "fetch dt.entity.process_group\n| filter id == \"PROCESS_GROUP-2AD8E1DA4761BD8E\"\n| expand tags\n| filter contains(tags, \"[Kubernetes]componentid\") OR contains(tags, \"env\")",
      "timezone": "America/Sao_Paulo",
      "query": "fetch dt.entity.process_group | filter id == \"PROCESS_GROUP-2AD8E1DA4761BD8E\" | expand tags | filter contains(tags,\"[Kubernetes]componentid\") or contains(tags,\"env\")",
      "scannedRecords": 1,
      "dqlVersion": "V1_0",
      "scannedBytes": 4397,
      "analysisTimeframe": {
        "start": "2023-11-20T13:56:26.696Z",
        "end": "2023-11-20T15:56:26.696Z"
      },
      "locale": "",
      "executionTimeMilliseconds": 38,
      "notifications": [],
      "queryId": "0b7f4384-814b-4ec2-b9e9-29878c83f650",
      "sampled": false
    }
  },
  "types": [
    {
      "mappings": {
        "entity.name": {
          "type": "string"
        },
        "id": {
          "type": "string"
        },
        "tags": {
          "type": "string"
        }
      },
      "indexRange": [
        0,
        2
      ]
    }
  ]
}

Thanks.

[sinisa_zubic · Accepted solution]

I think I found the culprit, the DPL pattern was not correct. I have copied it from help, but for the one special case it doesn't work. I will trigger a documentation update.

Can you try this query now

fetch dt.entity.process_group, from:now() - 1h
|fields entity.name, id, primeiraVisualizacao = lifetime[start], getDate = now(), tags
|expand tags, alias:ts
| parse toString(ts), """(('['LD:tag_context ']' LD:tag_key (!<<'\\' ':') LD:tag_value)| (LD:tag_key (!<<'\\' ':') LD:tag_value)|LD:tag_key)"""
| fieldsAdd componentId = if (tag_key == "componentid" and tag_context == "Kubernetes", tag_value)
| fieldsAdd environment = if (tag_key == "env", tag_value)
| summarize { 
  tags = takeAny(tags),
  componentId = takeAny(componentId), 
  environment = takeAny(environment), 
  primeiraVisualizacao = takeAny(primeiraVisualizacao)}, by:{id}
| filter isNotNull(componentId) and isNotNull(environment)
| fieldsAdd myNewField = concat(componentId, ":", environment)
| sort primeiraVisualizacao desc
| limit 6000

[RPbiaggio]

Now it works perfectly. Thanks for your help. Could you tell me what change you made? I appreciate the help.

[sinisa_zubic]

the change I did was in line 4

changed this 

| parse toString(ts), """(('['LD:tag_context ']' LD:tag_key (!<<'\\' ':') LD:tag_value)|LD:tag_key| (LD:tag_key (!<<'\\' ':') LD:tag_value)|LD:tag_key)"""

to this

| parse toString(ts), """(('['LD:tag_context ']' LD:tag_key (!<<'\\' ':') LD:tag_value)| (LD:tag_key (!<<'\\' ':') LD:tag_value)|LD:tag_key)"""

What that line does is it parses the string of tags into context, key and value field. So how the pattern works basically it tries to pars the tag context, tag key, and tag value from the string.

['LD:tag_context ']' LD:tag_key (!<<'\\' ':') LD:tag_value)

In case it doesn't match (there is no context) it should parse the key-value of the tag. and here was the issue. the initial pattern wanted to match & parse the full key, in case there is no context. but actually it should try to match & parse the tag key and tag value. so I just removed following snippet from the middle of line 4

|LD:tag_key|

[RPbiaggio]

When it includes another tag (tags.datadoghq.com/env) that will be discontinued, which is from Datadog, it returned the value. This "env" tag will not have a context like the other, that is, it will not have "Kubernetes".