topic Re: Kubernetes Number of EndPoints Custom Metric in Container platforms

Kubernetes Number of EndPoints Custom Metric

marina_pollehn — Fri, 14 Jul 2023 06:48:58 GMT

Hi everyone,

I am currently trying to create a metric event which is creating a custom alert if a Kubernetes service has no valid EndPoints (hence #EP in frontend SVC == 0). I can see the configured ports for this service but I don't see how I can alert on the absence of EndPoints. Anyone any suggestions?

Re: Kubernetes Number of EndPoints Custom Metric

marina_pollehn — Thu, 13 Jul 2023 13:01:40 GMT

My alternative approach was to use the connectivity measure of the process - unfortunately the connectivity % is not shown there - also after restarting the containers. Anyone any idea? 😉

Re: Kubernetes Number of EndPoints Custom Metric

florian_g — Thu, 13 Jul 2023 13:59:48 GMT

Hi @marina_pollehn - by no valid endpoint you mean that there is currently no pod in ready state for this service? Meaning there's no pod accepting traffic.
fyi @stefan_penner

Re: Kubernetes Number of EndPoints Custom Metric

michiel_otten — Wed, 19 Jul 2023 10:57:44 GMT

Hi Florian,

We've just had a meeting about this and the idea is that we want to be informed IF a Pod looses it's endpoint.
By default the service is configured in the Kubernetes deployment. So this alert will be a check that nobody pushed wrong configurations.

For us: a Kubernetes service is using TCP 80 to guide traffic towards our frontend and backend pods. So the idea now is to use the Data Explorer and relationships in it to do a pod count:

builtin:kubernetes.pods:splitBy("dt.entity.cloud_application"):sort(value(auto,descending)):filter(in("dt.entity.cloud_application", entitySelector("type(~"CLOUD_APPLICATION~"),toRelationship.isKubernetesSvcOfCa(type(~"KUBERNETES_SERVICE~"),entityName.contains(~"SERVICE_NAME~"))"))):Avg():Default(0)

If this number drops to 0, all relationships between the pod and it's supporting service are lost, meaning the internal endpoint is not available anymore. Tomorrow we'll start testing it and hope this suits for us.

KR.
Michiel