topic Re: Log processing IP attribute manipulation in DQL

Log processing IP attribute manipulation

AsafAx — Thu, 28 Mar 2024 11:58:58 GMT

Hello all,

I'm currently parsing the "content" of logs and splitting it into different fields, one of which is called "srcaddr".

I want to change the IP addresses so that they show the name of the source, instead of the IP itself.

Sources can have a few IP's within a range, so I'm trying to do a query like:

|FIELDS_ADD(srcaddr: IF_THEN_ELSE(srcaddr ....(range of IP addresses)...., "name of source", srcaddr)

For example

If a "srcaddr" is between xxx.xx2.0.1 and xxx.xx2.15.254, call it "Source 1"
If a "srcaddr" is between xxx.xx2.16.1 and xxx.xx2.30.254, call it "Source 2"
If "srcaddr" is between xxx.xx4.0.1 and xxx.xx4.15.254, call it "Source 3", etc. etc.

I've found this in the documentation- it refers to IPADDR(), but I'm not sure how I'd utilize it properly...

https://docs.dynatrace.com/docs/shortlink/lma-log-processing-functions#ipaddr

How would you think about doing this? When I parse the "content" of the log, is it okay that the IP addresses are in "String" form instead of IP or something else?

Thanks in advance 🙂
Asaf

Re: Log processing IP attribute manipulation

krzysztof_hoja — Thu, 28 Mar 2024 18:30:43 GMT

Let me share my AWS VPC Flow Log parsing rule. It contains braking down content line line into fields and replacing some IP addresses belonging to specific networks (while keeping originals under different names however for temporary/testing purposes):

PARSE(content, " STRING:account_id SPACE STRING:action SPACE STRING:az_id SPACE INT:bytes SPACE IPV4:dstaddr SPACE INT:dstport SPACE STRING:end SPACE STRING:flow_direction SPACE STRING:instance_id SPACE STRING:interface_id SPACE STRING:log_status SPACE INT:packets SPACE STRING:pkt_dst_aws_service SPACE IPADDR:pkt_dstaddr SPACE STRING:pkt_src_aws_service SPACE IPV4:pkt_srcaddr SPACE INT:protocol SPACE STRING:region SPACE IPV4:srcaddr SPACE INT:srcport SPACE STRING:start SPACE STRING:sublocation_id SPACE STRING:sublocation_type SPACE STRING:subnet_id SPACE INT:tcp_flags SPACE STRING:traffic_path SPACE STRING:type SPACE STRING:version SPACE STRING:vpc_id" ) | FIELDS_ADD( log.type:("aws.vpc"), srcaddr_orig:srcaddr, dstaddr_orig:dstaddr) | FIELDS_ADD ( dstaddr : IF( flow_direction=="egress" AND pkt_dst_aws_service=="-" AND NOT ( IP_TRUNC(dstaddr,8)==IPADDR("10.0.0.0") OR IP_TRUNC(dstaddr,12)==IPADDR("172.16.0.0") OR IP_TRUNC(dstaddr,16)==IPADDR("192.168.0.0")), IPADDR("0.0.0.0") , dstaddr), srcaddr : IF( flow_direction=="ingress" AND pkt_src_aws_service=="-" AND NOT ( IP_TRUNC(srcaddr,8)==IPADDR("10.0.0.0") OR IP_TRUNC(srcaddr,12)==IPADDR("172.16.0.0") OR IP_TRUNC(srcaddr,16)==IPADDR("192.168.0.0")), IPADDR("0.0.0.0"), srcaddr) )

Kris

Re: Log processing IP attribute manipulation

AsafAx — Sat, 30 Mar 2024 10:37:57 GMT

Hi Kris, thank you for your reply!

It seems to me that the IP_TRUNK() function with operators (all within a conditional "if" statement) is the right way to do it:

IP_TRUNC(srcaddr, 8 ) >= IPADDR("xx.xx2.0.1") AND IP_TRUNC(srcaddr, 8 ) <= IPADDR("xx.xx2.15.254"))

I have yet to implement it in our environment, but I'm sure that it'll work or get very close to the right result.

Thank you for your help!

Asaf