How to send StatsD from within a docker container to OneAgent UDP localhost:18125 port ?

[gilles_tabary]

Hello.

Between OneAgent versions 1.207.242 and 1.209.169 the StatsD listening port changed from UDP *:18125 to UDP localhost:18125. This broke our Traefick sending StatsD from within it's Docker container (running on Linux) to ip:18125 (cannot send to localhost:18125 nor to 127.0.0.0:1825 : this would loop back to the docker container as opposed to Docker host where the OneAgent run).

Potential solution : set the Docker container to --network="host" : not really an option : brakes completely the wanted isolation.

Are there any one using StatsD the same way ? I mean : from several different Containers, sending from within a Container to host OneAgent StasD listening port UDP localhost:18125 ? Any known solutions ?

Regards.

[ChadTurner]

OH NO! I would recommend opening up a support ticket as updates should not break current set ups. 

[gilles_tabary]

Hello. Support feedback : Between OneAgent versions 1.207.242 and 1.209.169 the StatsD listening port changed from UDP *:18125 to UDP localhost:18125 is a design fix, it works as wanted. Regards.

[Julius_Loman · Accepted solution]

Ports opened globally for services without any authorization such as statsd ingest are not a good idea.  At the moment you cannot modify the listening interface, but you can add an iptables rule and a sysctl config to forward the packets to loopback address.

[gilles_tabary]

Hi.  Been there done that :
sudo iptables -t nat -A OUTPUT -d <this host ip> -p udp -m udp --dport 18125 -j DNAT --to-destination 127.0.0.1:18125

I.e. intercept outgoing packets sent to <this ip>, and redirect them to "localhost". It technically works.

Modification refused by Sys Admins & Network admins. Back to square one. 🙂

[gilles_tabary]

Hello again.

Other (non) solution : send StatsD to ActiveGate 1.227+ which can offer remote StatsD port : Leads to loss of metric host context informations added previously by OneAgent ("...no extra enrichment is added. If you want to add context to your metrics, you'll need to add dimensions of your choice to your StatsD metrics....).

AFAIU the change in OneAgent StatsD port listening only to udp:localhost:18125 was to avoid potentially receiving metrics from other hosts, leading to mangled host context. Fair enough.

Yet, sending to ActiveGate leads to loss of information. Back to square one. 🙂

Regards.

[gilles_tabary]

Other (non) solution : install OneAgent within container : it create a strong dependency between our application and OneAgent life cycle management. Refused here. Back to square one. 😐

[RonBogdanoff]

Did you ever find a workable solution other than what was suggest with iptables?  We have run into same issue.  There should be a oneagent config parameter so user (customer) can change listener interface at their discretion and not be force to localhost.  We may need to add a proxy (e.g. nginx) to our dynatrace pod simply to have listener interface be the host ip so other containers that are not running in hostNetwork = true (95% of the) to be able to get metrics to the local agent! 

[gilles_tabary]

Podman containers, not good a solution : 

• not to be forgotten : by 30/05/2022 Podman is *not* supported by Dynatrace ! Big unknown headache ahead.
• start the cont with --network=slirp4netns:allow_host_loopback=true
  (this adds an entry to container /etc/hosts)
• from within container, one can then send StatsD metrics to host.containers.internal:18125 (host.containers.internal is a reserved podman keyword)
• one can see the drawback : loss of some isolation

Docker Containers, not good either and did not work for me:

• starting with Docker 20.10
• start container with --add-host=docker_container_host:host-gateway
  (this adds an entry to container /etc/hosts)
• from within container, one should be able to send StatsD Metrics to docker_container_host:18125
• did not work for me : looks like it could requires some more iptable fiddling
• if it worked, one can also see the drawback : loss of some isolation

This type of strategy (from within a container, pointing StatsD to an equivalent of "localhost", as opposed to host ip) could apparently also be done using explicit container internal reserved IP (gateway), but this is not a way to go AFAIU : may have unexplored side effects, and brake due to dynamic behaviour. Also, fundamentally, it downgrade container isolation. Hmmm. "Tout ça pour ça". 

On our Docker, because we aim at moving away from Docker, we got the support to activate the not recommended flag "debugExtensionDSstatsdacceptremotestatsdtraffic". It works with OneAgent v1.221+. But this is kind on unmanageable : no API nor any UI available for me (the admin) to configure it myself. BTW,  AFAIK also no API available to activate OneAgent StatsD : no automation possible at installation time.

To my understanding the iptable solution looked the cleanest : respect isolation, not opening of unwanted  avenues.