Based on my understanding, it is expected to see slowness due to network in software service of front-end tier and software service of middle tier.
That is because when a tier make a request to any backend tier, there is a request-response interaction, and thus, the existence of network metric like loss rate, latency etc. But since DB tier is the most backend-tier, it wouldn't have any more backend tier to make a request, so why would I still see slowness due to network?
Is my understanding correct, if no can anyone shed some light or correct me?
If not, then what could it means to have slowness due to network in DB tier's software service?
Slowness dose to the network means that caller (app server issuing the query) has been waiting for the response, while server hasn't been taking time to calculate the response - it has sent it already, or request hasn't been fully received already because of the network issue. In the other words, data packets were "in flight".
Typically the network delays occur because of packet loss. When transmitting party doesn't receive an ACK to a transmitted packet, the transmitting party has to retransmit the packet (I'm simplifying here, but we don't have to go into details).
If there is any virtualization involved between the database server an the caller app server, then packet losses are perfectly possible. These may be a real packet loss (buffer overrun because apps/VMs aren't fast enough to pick packets up) or virtual packet loss caused by tiny VM clock skews where time slicing in an oversubscribed virtualization host accumulates to make the TCP stack "think" that it didn't receive ACK on time, thus causing retransmit. When a single packet loss happens in virtual environment, where typical TCP windows are huge and TCP ACK timeouts are 1 sec, a single packet retransmit may mean 1-sec delay due to "network". Like in the attached picture (and old but still valid DC RUM example of how a single packet loss causes delay in database query response time).
In reality it is not a network issue at all. It is a virtualization host oversubscription issue. Dynatrace OneAgent installed on VMs in such environment would pinpoint it with higher precision because it is aware of the vertical stack.
If the network metric of DB tier software service indicate the network performance between app tier and db tier, then
when I look at network metric of App tier software service, how can I know it indicates network performance between web & app or between app & db?