Installing Dynatrace cluster on hardened CIS image (Ubuntu 22.04)

[Ardhi]

Hi,

We're installing Dynatrace Managed on customer premises but the Nginx service couldn't listen to port 443.
From the log, it seems forbidden. We could try to elevate Nginx user privileges but the concern here is the unknown implication (because it's in the internal steps of dynatrace-managed.sh script).

Right now also Nginx binds to port 8022 and in the Nginx.conf also nowhere to be found the port 443 configurations. Checked with "check" and "status" commands of dynatrace-managed.sh, all services are ok.

We have also pam_faillock(sudo_account):unknown option:unlock error during the installation. Not sure if this is related.

Please advise.

[Radoslaw_Szulgo]

Hi @Ardhi ,

You're struggling to set it up because the "443" port is a "privileged port" in Linux operating systems (pam-related issues) 

The TCP/IP port numbers below 1024 are unique because normal users can't run servers on them. This is a security feature in that if you connect to a service on one of these ports, you can be reasonably sure that you have the real thing and not a fake that some hacker has put up for you.

(source: https://www.w3.org/Daemon/User/Installation/PrivilegedPorts.html)

That's why by default, in Dynatrace Managed installation, Nginx listens on port :8022, and we have iptables (nftables) rules that redirect the traffic from :443 to :8022. We don't recommend changing that configuration. 

If that's needed, then:

setcap 'cap_net_bind_service=+ep' /path/to/dynatrace-managed

And then anytime dynatrace-managed is executed thereafter, it will have the CAP_NET_BIND_SERVICE capability. setcap Is in the Debian package libcap2-bin.

Now for the caveats - You'll need at least a 2.6.24 kernel

Resources:

• capabilities(7) man page. Read this long and hard if you're using capabilities in a production environment. There are some really tricky details of how capabilities are inherited across exec() calls that are detailed here.
• setcap man page
• "Bind ports below 1024 without root on GNU/Linux": The document that first pointed me towards setcap.

Note: RHEL first added this in v6. Not sure when Ubuntu has added this.

[Ardhi]

Hi @Radoslaw_Szulgo ,

Thank you for the answer,
I put the setcap command and reinstalled but still not listening to 443 ...
Just wondering, if is this the case (not listening to 443) when the server is blocked for outbound connection to Dynatrace? I just realized cannot connect to mcsvc.dynatrace.com from the server.

EDIT: can connect to outgoing mcsvc.dynatrace.com, so this is not an issue

[Julius_Loman]

@Ardhi The NGINX does not listen on the 443/tcp for the reasons above, but there is iptables redirect for that purpose:

# ss -alt '( sport = 443 )'
State                 Recv-Q                 Send-Q                                 Local Address:Port                                 Peer Address:Port                 Process                 

# ss -alpt '( sport = :8022 )'
State                  Recv-Q                 Send-Q                                 Local Address:Port                                 Peer Address:Port                Process                 
LISTEN                 0                      511                                          0.0.0.0:8022                                      0.0.0.0:*                    users:(("nginx",pid=1510320,fd=7),("nginx",pid=1510319,fd=7),("nginx",pid=1510318,fd=7),("nginx",pid=1510317,fd=7),("nginx",pid=1510316,fd=7))

# iptables -t nat -L PREROUTING
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
REDIRECT   tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL tcp dpt:https redir ports 8022

[Ardhi]

sorry can you elaborate? are you implying we don't have to listen to port 443?
here's attached

 

my ports
Because I cannot open https://<ip address> / port 443 of the server in the browser

EDIT: now I can access after stopping nftables (checking from CIS list), not sure if this is related

[Julius_Loman · Accepted solution]

@Ardhi yes, in your case it seems perfectly fine.

First you should check if you can connect locally to the cluster on the node:

# curl -v https://localhost -k

If this works, then most likely there is a firewall or something else between your browser and the cluster node preventing connections.