Kubernetes Number of EndPoints Custom Metric

[marina_pollehn]

Hi everyone,

I am currently trying to create a metric event which is creating a custom alert if a Kubernetes service has no valid EndPoints (hence #EP in frontend SVC == 0). I can see the configured ports for this service but I don't see how I can alert on the absence of EndPoints. Anyone any suggestions?

[marina_pollehn]

My alternative approach was to use the connectivity measure of the process - unfortunately the connectivity % is not shown there - also after restarting the containers. Anyone any idea? 😉

[florian_g]

Hi @marina_pollehn - by no valid endpoint you mean that there is currently no pod in ready state for this service? Meaning there's no pod accepting traffic.
fyi @stefan_penner 

[michiel_otten · Accepted solution]

Hi Florian,

We've just had a meeting about this and the idea is that we want to be informed IF a Pod looses it's endpoint. 
By default the service is configured in the Kubernetes deployment. So this alert will be a check that nobody pushed wrong configurations.

For us: a Kubernetes service is using TCP 80 to guide traffic towards our frontend and backend pods. So the idea now is to use the Data Explorer and relationships in it to do a pod count:

builtin:kubernetes.pods:splitBy("dt.entity.cloud_application"):sort(value(auto,descending)):filter(in("dt.entity.cloud_application", entitySelector("type(~"CLOUD_APPLICATION~"),toRelationship.isKubernetesSvcOfCa(type(~"KUBERNETES_SERVICE~"),entityName.contains(~"SERVICE_NAME~"))"))):Avg():Default(0)

If this number drops to 0, all relationships between the pod and it's supporting service are lost, meaning the internal endpoint is not available anymore. Tomorrow we'll start testing it and hope this suits for us.

KR. 
Michiel