DQL search command

[m_zol]

Hello,

Is it possible to use the search command in DQL to mask or obfuscate certain information based on a pattern?

For example, I'd like to search logs that contain a credit card number and then mask part of it (e.g., show only the last 4 digits). Is there any built-in function or workaround in DQL that allows this kind of pattern-based masking or transformation?

Any help or examples would be appreciated!

[Mullaneyb]

Why do you have credit card numbers stored 🤣 

Anyway, my take on the "Scenario".

Probably don't need the replace string and can just use substring from whatever position you need.

These should really be filtered on ingestion an ideally not in logs at all. You do you...

data json:"""[
  { "ccnumber": "1234 5678 9012 3456" },
  { "ccnumber": "2345 6789 0123 4567" },
  { "ccnumber": "3456 7890 1234 5678" },
  { "ccnumber": "4567 8901 2345 6789" },
  { "ccnumber": "5678 9012 3456 7890" },
  { "ccnumber": "6789 0123 4567 8901" },
  { "ccnumber": "7890 1234 5678 9012" },
  { "ccnumber": "8901 2345 6789 0123" },
  { "ccnumber": "9012 3456 7890 1234" },
  { "ccnumber": "0123 4567 8901 2345" }
]"""
| fieldsAdd replaceString = replaceString(ccnumber," ","")
| fieldsadd leng = stringLength(trim(replaceString))
| fieldsAdd last4 = substring(replaceString, from: (leng -4 ))
| fieldsAdd concat("**** **** **** ",last4 )

PS: I didn't realise you were in fact referencing  https://docs.dynatrace.com/docs/shortlink/filter-and-search-commands#search . I'm not familiar with this feature at all so unsure if anything I said helps  😅

[Mullaneyb]

there is a DPL example of CC masking at ingestion. Methods of masking sensitive data — Dynatrace Docs

the way search seems to work is check any value so in my example it would search "1234" anywhere in the card and if it hits show back all cc numbers that have "1234" otherwise you could just filter on the last4

//| search "1234" // search all rows/columns
| fieldsAdd replaceString = replaceString(ccnumber," ","")
| fieldsadd leng = stringLength(trim(replaceString))
| fieldsAdd last4 = substring(replaceString, from: (leng -4 ))
| fieldsAdd hidden = concat("**** **** **** ",last4 )
//| filter matchesValue(last4, "1234") // only search last four
| fields hidden

[m_zol]

Thanks for the response!

Just to clarify — the credit card number was just an example. The real concern involves sensitive data in general, like user IDs, account numbers, etc. Of course, the best practice is to avoid logging this kind of information altogether, but unfortunately, that’s not our current reality. We have a large volume of logs where such data may appear, and we're trying to find a way to proactively detect and mask it.

Regarding the examples shared so far: they’re helpful, but still not fully effective for our use case. The main challenge is that we don’t know the exact field names where the sensitive information might appear. That’s why we’re using search to filter based on patterns like "cardNumber", "card number", "card", etc., to catch any record that contains those terms in any field.

What we need next is a way to use DPL to scan for numeric patterns (like potential card numbers or user IDs) within the matched records and then mask or obfuscate them — regardless of where they appear. That’s proving to be a complex task.

Has anyone tried something similar or found an efficient workaround for this kind of scenario?