cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

IP Address Recognition from X-Forward (Cookie)

Hi,



I'm currently working in an environment which has a lot of internet sourced traffic coming in via XML/SOAP, and the x-forward header has been enabled to tag the cookie with the source IPs.



I was under the impression that it would be possible (at least for HTTP) to get the tool to pick this up as the client IP, leaving the client internal IP as the NAT address. In my ignorance I hadn't predicted that this was done using the username method.



We are already currently username matching within the XML/SOAP to gather the actual usernames, so my question is multifold:




  • Is it possible to capture the x-forward from the cookie for XML/SOAP and tag it as the Client IP Address?

  • If we set the client IP address as the username (as the guide suggests), how do we then match the location within the Region/Areas/Sites structure?

  • Has anyone got any experience with working around this to make it fit?




(If possible i'd like to steer clear of utilising the BSM to knife and fork it.



Thanks,

Morgan

35 REPLIES 35

kristof_renders
Dynatrace Pro
Dynatrace Pro

Hi Morgan,



This can either be done global (AMD by AMD) by opening the AMD configuration in RUM Console > Global > Frontend > Web > HTTP. In the bottom you will see a section called Client IP address extraction. If you select Header and then fill in x-forwarded-for in the text field it should be ok.



Alternatively you can do the same for each software service in the HTTP Options tab.



If you define client sites as explained within the guide, this IP will then be used to match those sites.



Hope this helps.



Cheers,

Kristof

Thanks Kristof, that looks like it covers it from a HTTP perspective, but when looking at XML/SOAP over HTTP on individual Software Service configuration, it does not give this option.



How would we perform that in these circumstances?



Thanks,

Morgan

Hi Morgan,



Since SOAP is just HTTP with a SOAP body, it will use the global http settings. If you set the x-forwarded-for in the global settings, the SOAP decode should pick it up.



Let me know if this helps.



Cheers,



Kristof

Hey Kristof,



I am actually on site now and have attempted this.



When configuring the HTTP Global settings, I have set the X-Forwarded-For and it is picking this up for all HTTP software services that it can find it on.



The problem I have is the one you described as working, where I have XML/SOAP over HTTP configured, the X-Forwarded-For is not matching the client IP.



I have tested with changing from XML over HTTP to just plain HTTP and the header is seen in the traffic for that host.



Any ideas?



(This customer is using v12, no SPs).



Thanks,

Morgan

Morgan,



Where are you configuring the client ip extraction on SOAP/XML?



For HTTP, you can simply use the x-forward-for tag, but for SOAP/XML you would have to configure a regular expression to extract the user name and then select the option to try to convert it to an ip address.



Thanks,



Mazen

As Kristof put it, I am using the Global area and have simply set the "X-Forward-For" tag within HTTP, SOAP/XML should be utilising the HTTP settings as it's "over HTTP" from what I can see is working for Kristof?



I'm not sure how username comes in to it in this case?

Morgan,



The global area is segregated based on analysis type and the SOAP/XML section is separate from HTTP. Configuration of the global HTTP section should apply to the HTTP analyzer only. You need to make the same configuration at the SOAP/XML level. In SOAP/XML there is a global option to extract username and then try to convert it to a client ip. There is currently no option to extract client ip directly in SOAP/XML as far as I know.



Thanks,



Mazen

An ok, apologies.



I will try this tomorrow. Do you have an example of a working regex?



I attempted this earlier with the HTTP one and was having some struggles, partly because the reason I wanted to use regex was because of the tag being either "X-Forwarded-For", or "X-XYZ-Forwarded-For", due to two different proxy sources.

Morgan,



If you would like to post a couple of examples here, we can try to help out with the regex. Make sure to remove/hide ip addresses. I just need to see what typically comes before and after the actual ip in the header.



Thanks,



Mazen

All,



My apologies for the confusion I have created. I was for some reason convinced that this was the case. I am glad that now I know this is not the case.



Thanks



Cheers,



Kristof

Hey Mazen,



I've had to modify a bit but the structure is the same, it will either be (just the lines before + after):



RequestGuid: XX23482348294XXxXX

X-Forwarded-For: xxx.xxx.xxx.xxx

Connection: Keep-Alive



OR



RequestGuid: XXX293482948XXXXXX

X-XYZ-Forwarded-For: xxx.xxx.xxx.xxx

Connection: Keep-Alive



The XYZ is something else but I can't divulge the acronym but obviously not relevant to this.



On occasion there is more before / after the RequestGuid/Connection, but the three always maintain the same structure.



Thanks,

Morgan

Morgan,



I think we might need to test a few variations of the regex, but here's a first shot at it. Let me know how it works.



X-\\(?:XYZ-\\)?Forwarded-For:%20\\([0-9.]*\\)






* please replace all (double-slash) entries from the regex string above with only (one slash) entries.




I had to add it as it was taking the text as markup when posting.




Thanks,



Mazen

Morgan,

BTW, you need to adjust the regex above to accommodate the different text which is going to be in place of "XYZ". I didn't know what pattern would be in there to apply a regex for it. If you know it is always going to be 3 alpha characters, then you can replace it with something like "[a-Z]{3}-".

Thanks,

Mazen

Thanks Mazen,

This one is a bit awkward as i'm not with the system so getting the local support guys to enter and test.

They have tested the original string (with the amended XYZ), and it looks like it's extracting the standard (X-Forwarded-For), but not the XYZ string. 

I'll try what you suggest above and amend it to:

X-\\(?:[a-Z]{3}-\\)?Forwarded-For:%20\\([0-9.]*\\)

Will let you know how it goes.. (smile)

So...i've finally found myself in front of the system again.

 

Neither of the above worked unfortunately, the second one came with an "invalid end" error, which I couldn't seem to fix.

I'm in here for the next couple of hours but won't be back in for a while, although my colleague will now have ready access and i'll put him on to this thread.

Does anyone have any further ideas?

 

p.s. I'm attempting to get them just to rename the tag.

Try:

X-.*Forwarded-For: ([0-9.]*)

Morgan,

First of all I'm assuming you're using 12.1.

I'm digging in the topic and it might be true what Kristof says. SOAP global setting contains the following entries:


...
 
false

false

I'm not sure why it's not present in RUM Console (at least mine 12.1.0), but Dev confirms it will work. I will find out how it looks like in upper version and let you know.

Anyway it means that using:


...

true
X-.*Forwarded-For: ([0-9.]*)
false

should work in 12.1. At the current stage of my investigation the change needs to be made manually in applications.xml and RTM process restarted.

12.1 SP1 RUM Console contains these options:

Hey Adam,

Thanks - unfortunately it's 12.0, and i'm not sure when we're due to upgrade, but I think I have these options in 12.0 too so i'll have a look and attempt with that simplified verson.

Cheers,
Morgan 

Right, 12.0 Console displays it only when 12.1 AMD is connected ...

I will find out if any manual change, directly in applications.xml is possible.

Verified, it's possible only in 12.1

Does that mean that I cannot perform this at all in 12.0?

Correct

Correct

pieter_van_heck
Dynatrace Guide
Dynatrace Guide

Hi Morgan,



Option 1 could be to decode the SOAP as HTTP (so not use the SOAP decode), which should give you the HTTP advanced option. But then you might need to use a couple of regex's to extract the right paramaters for the names of you soap requests.



Nevertheless there is also a similar option within the SOAP advanced options.



RUM Console > Global > Middleware Monitoring> SOAP> User Identification.



Regards Pieter.

Hi Pieter,



A question from me around that: I thought this was used to extract the user, not the IP?



There is no place to specify where to get the IP address from in the User Identification section in SOAP settings.



I always thought it would inherit that from the HTTP settings, since that is the underlying protocol.



Please let me know if that is not the case.



Cheers,

Kristof

Thanks guys - and Kristof, I have the same reservation, and fear I suppose.



Unfortunately the customer that this is for is in a datacentre locked away so I can't test this until i'm there on Monday.



Which brings me to my next question... Pieter you may be able to help, do the labs have an example trace file for this scenario that we may be able to use to test?



Thanks,

Morgan

pieter_van_heck
Dynatrace Guide
Dynatrace Guide

Hi Kristof,



As far as I'm aware, the SOAP decode is not built on top of the HTTP decode. So you cannot mix and match both decodes. You either go for SOAP or HTTP.



Regards Pieter.

Thanks for your response Pieter.



Can you please then help me explain the following: the customer has a proxy which forwards both HTTP and SOAP requests and thus the regular client IP is useless as it will always be the proxy. I only set the x-forwarded-for for HTTP and now both Software Services with SOAP and Software Services with HTTP use this field successfully.



I am confused now 🙂

Kristof,

Is there any chance I can see applications.xml from such AMD?

As it would appear this cannot work in 12.0, we'll be attempting to plan an upgrade and i'll report back if we can get a solution in place via that upgrade.

Cheers,
Morgan 

Just thought i'd bring this back, we've now upgraded the system to 12.2 SP1, and will be attempting to get this to work.

Will let you know the outcome.

Implementing the recommendation above; to include X-.*Forwarded-For: ([0-9]*) does almost work.

It brings in the X-Forwarded-For, but does not bring in the X-XYZ-Forwarded-For.

The validation within the Regular Expression Test is pretty much useless to help in this scenario (smile)

What would the next steps be here?

jacob_gannon
Guide

Hey Guys,

I work with Morgan and I'm now looking at trying to implement this.

Is there anyone who can help us move forward?

Thanks,

Jacob

erenedo
Organizer

Hi Kristof,

We are using RUM Console 12.4.12 and we don't see the section "Client IP address extraction". Where can we configure it?

Thanks in advanced.

Best Regards,

Elena.