08 Sep 2021 01:24 AM - last edited on 18 Nov 2021 03:23 AM by MaciejNeumann
Between OneAgent versions 1.207.242 and 1.209.169 the StatsD listening port changed from UDP *:18125 to UDP localhost:18125. This broke our Traefick sending StatsD from within it's Docker container (running on Linux) to ip:18125 (cannot send to localhost:18125 nor to 127.0.0.0:1825 : this would loop back to the docker container as opposed to Docker host where the OneAgent run).
Potential solution : set the Docker container to --network="host" : not really an option : brakes completely the wanted isolation.
Are there any one using StatsD the same way ? I mean : from several different Containers, sending from within a Container to host OneAgent StasD listening port UDP localhost:18125 ? Any known solutions ?
Solved! Go to Solution.
OH NO! I would recommend opening up a support ticket as updates should not break current set ups.
Hello. Support feedback : Between OneAgent versions 1.207.242 and 1.209.169 the StatsD listening port changed from UDP *:18125 to UDP localhost:18125 is a design fix, it works as wanted. Regards.
Ports opened globally for services without any authorization such as statsd ingest are not a good idea. At the moment you cannot modify the listening interface, but you can add an iptables rule and a sysctl config to forward the packets to loopback address.
Hi. Been there done that :
sudo iptables -t nat -A OUTPUT -d <this host ip> -p udp -m udp --dport 18125 -j DNAT --to-destination 127.0.0.1:18125
I.e. intercept outgoing packets sent to <this ip>, and redirect them to "localhost". It technically works.
Modification refused by Sys Admins & Network admins. Back to square one. 🙂
Other (non) solution : send StatsD to ActiveGate 1.227+ which can offer remote StatsD port : Leads to loss of metric host context informations added previously by OneAgent ("...no extra enrichment is added. If you want to add context to your metrics, you'll need to add dimensions of your choice to your StatsD metrics....).
AFAIU the change in OneAgent StatsD port listening only to udp:localhost:18125 was to avoid potentially receiving metrics from other hosts, leading to mangled host context. Fair enough.
Yet, sending to ActiveGate leads to loss of information. Back to square one. 🙂
Other (non) solution : install OneAgent within container : it create a strong dependency between our application and OneAgent life cycle management. Refused here. Back to square one. 😐
Did you ever find a workable solution other than what was suggest with iptables? We have run into same issue. There should be a oneagent config parameter so user (customer) can change listener interface at their discretion and not be force to localhost. We may need to add a proxy (e.g. nginx) to our dynatrace pod simply to have listener interface be the host ip so other containers that are not running in hostNetwork = true (95% of the) to be able to get metrics to the local agent!
Podman containers, not good a solution :
Docker Containers, not good either and did not work for me:
This type of strategy (from within a container, pointing StatsD to an equivalent of "localhost", as opposed to host ip) could apparently also be done using explicit container internal reserved IP (gateway), but this is not a way to go AFAIU : may have unexplored side effects, and brake due to dynamic behaviour. Also, fundamentally, it downgrade container isolation. Hmmm. "Tout ça pour ça".
On our Docker, because we aim at moving away from Docker, we got the support to activate the not recommended flag "debugExtensionDSstatsdacceptremotestatsdtraffic". It works with OneAgent v1.221+. But this is kind on unmanageable : no API nor any UI available for me (the admin) to configure it myself. BTW, AFAIK also no API available to activate OneAgent StatsD : no automation possible at installation time.
To my understanding the iptable solution looked the cleanest : respect isolation, not opening of unwanted avenues.