Define where to get the End-User IP for Real User Monitoring

[r_weber]

Determining the end-user IP for real user monitoring can be challenging. Typically Dynatrace does this by various HTTP headers that can be manually also defined what header to use.

What if this header is unknown? And how could the determination of this header be tweaked. e.g. from where to get the header e.g. on downstream services?

I have this scenario:

A proxy server (actually more than one, varnish, haproxy and nginx) sits in front of an application server. Especially when oneagent (and real user monitoring) is active on the proxy nginx it seems that Dynatrace is ONLY using the ClientIP of this "entry service" to determine the real users client IP. However in my case Dynatrace is not detecting the real client IP on that servis, but it's detecting the real client IP on the downstream AppServer!

As you can see here:

The client IP on Nginx is detected as an internal IP, likely from the HAProxy or Varnish Cache

But the real client IP is passed on to the application server. (Nginx uses X-Real-IP header to pass it on, additionally to the DT auto-detecion I also added a request attribute to capture it)

Now, without knowing the nginx or varnish/HAProxy configuration - if any headers have been added to pass on the client IP (I've configured all usual subjects of IP headers, X-Forwarded-For, ...) - the Real User Monitoring IP is detected as a local one (10.41.0.0) and thus no location info is available.

When I remove the oneagent from nginx, the location detection is based on the appserver IP and works perfectly fine, however I loose visibility into nginx.
Since Nginx obviously HAS the client IP information somewhere (it couln't pass it on to the appserver otherwise), how could this be detected and used for location detection?
The complicated way would be to configure every proxy in the chain to add/pass-on the header, but this is not possible unless you have access to all.

Could there be a way to use the IP from downstream services? Or tell DT in the Applciation configuration to still do RUM injection but if it can't determine the Client IP try on the downstream service?

Any ideas?

[AntonioSousa]

You should check the following page, and include the headers that are being used in your case:

https://www.dynatrace.com/support/help/how-to-use-dynatrace/real-user-monitoring/setup-and-configura...

[AntonioSousa]

Just to confirm that in a default installation, X-Real-IP is not there:

[r_weber]

Hi Antonio,

as I wrote above, I’ve configured all the usual well-known headers in this setting.

Nginx adds the X-Real-IP header only to the proxied service. For location determination DT needs the IP on the nginx side (passed on from nginx’ client) in another - unknown - header.

The question was about either capture ANY header (* to find out, like one was able to do in Appmon) or tell DT to use the header from another downstream service.

[AntonioSousa]

Sorry, I had not read it carefully.

So, you've got a request that goes through Varnish, then haproxy, then nginx, and finally into your SAP Hybris application server.

So, if you see it in SAP, it has to be passed before.

Now, if this is right, I believe it will be passed in one of the HTTP header variables, but given the order in the rules, it will probably not be used. It would be useful to grab one example, but it seems you haven't managed to get one. Never tweaked the order of this configuration, but you could probably get it a try in a controlled environment, or afterhours.

BTW, I believe having Dynatrace show us all those fields would give a great RFE.

[r_weber]

Hi @Antonio S.,

the PROXY protocol has been built specifically to NOT have to mess with headers (e.g. when its proxying SSL traffic). The client Ip is not passed in any header before nginx decyphers it and adds it to the proxied request. But then it's too late for DT to pick it up.

You can read about the proxy protocol here: https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

I've enough examples and know it doesn't work because the nginx agent only either uses the TCP client IP or a header. The header doesn't exist yet and TCP adress is the one from the HA proxy. The real client IP is only in the PROXY protocol payload. Which nginx understands but Dynatrace doesn't...

Confirmed by DT Support.

The RFE is about getting that additional check for client IP:

1. when nginx is using the PROXY protocol (from HAProxy, please see link). Nginx determines the real client from the PROXY protocol (and not a header). This is exactly what the PROXY protocol was built for:
   
   IMO, Dynatrace shouldn't need to inspect the PROXY protocol itself, as nginx is already doing this. When nginx does so, the client IP is available via the nginx variable $proxy_protocol_addr.
   If this is not available, the agent can try to get it from headers.
2. get it from a http header (if available). Note that this will only work if such header exists (e.g. if the requests has been passed through proxies that add this header, http clients/browsers do not add it)
3. if no header is present and no PROXY protocol carried client IP is available use the TCP/IP address.

If this would be done by the agent then we would be fine.

[AntonioSousa]

Hi @Reinhard W.

When you were referring to the PROXY protocol, I was thinking that you were using the CONNECT method as defined in RFC2817:

https://tools.ietf.org/html/rfc2817

[r_weber]

I think I found out why Real user Monitoring breaks and why nginx isn't able to determine the right client IP address in this case (FYI @Antonio S.).

This is a limitation in Dynatrace and I will raise an RFE for it as its a rather common usecase.

The situation is caused by HAProxy using the PROXY protocol to communicate with nginx. This is required as nginx is doing the SSL termination and the whole HTTP traffic (including any headers) is transferred as-is, encrypted. Dynatrace determines the Client-IP as the HAProxy's IP as it doesn't support PROXY protocol to get the real client IP. I think it could get that from nginx as nginx also will have this available internally (it adds it to the decrypted http headers once forwarding to the downstream server).

[AntonioSousa]

Is it HAProxy or Varnish that is doing the CONNECT request? In any case, it seems this might be related to how Dynatrace interprets CONNECT, or eventually some internal stuff regarding how Nginx deals with CONNECTs.

[zietho · Accepted solution]

Dear @Reinhard W.

we looked into this and we acknowledge the fact that we can't interpret the proxy protocol yet.

Our research concludes that we could add client ip detection for the given proxy protocol for:

• Apache
• Nginx

No support for IIS yet.

Please create an RFE for this extension if your need still exists and let's check whether other customers are struggling with this.

regards

Thomas

[r_weber]

Hi Thomas,

I have already created the RFE here. I think it already received some upvotes.
IIS itself doesn't support the PROXY protocol, that is true, but I guess Nginx and Apache are more relevant anyway 🙂

Reinhard