Cannot install Dynatrace operator on GKE

[tamaraflorens]

Hi,

I'm Facing this issue after running kubectl apply -f dynakube.yaml

Error from server (InternalError): error when creating "dynakube.yaml": Internal error occurred: failed calling webhook "webhook.dynatrace.com": failed to call webhook: Post "https://dynatrace-webhook.dynatrace.svc:443/validate?timeout=10s": no endpoints available for service "dynatrace-webhook"

I following this step 

after searching they say that I should create firewalls rule ingress to port 8443. I do. but the error still the same.

this is my firewall rule configuration

please help..

[agonzalez]

Hello is your port 443 also open ? The url that you gave "https://dynatrace-webhook.dynatrace.svc:443" is set to port 443, check that your firewall allow the 443 port.

[agonzalez]

You can find the dynatrace operator network policies/ports here: https://docs.dynatrace.com/docs/setup-and-configuration/setup-on-k8s/reference/network

[tamaraflorens]

Thanks for your reply. I create firewall rule with allow all but unfortunately it didn't work

I'm still facing this error

Error from server (InternalError): error when creating "dynakube.yaml": Internal error occurred: failed calling webhook "webhook.dynatrace.com": failed to call webhook: Post "https://dynatrace-webhook.dynatrace.svc:443/validate?timeout=10s": no endpoints available for service "dynatrace-webhook"

[RazTN7]

Hi @tamaraflorens 

Is this Dynatrace Saas Env.
If yes. Can you curl from each node of GCP cluster and see the status as running.

curl -X GET https://{environmentid}.live.dynatrace.com/rest/health

It looks like 

mutatingwebhookconfiguration.
and
validatingwebhookconfiguration. service is simply not registering properly.

Follow:

$ kubectl get ValidatingWebhookConfiguration -A

or

$ kubectl describe mutatingwebhookconfigurations,validatingwebhookconfigurations > webhooks.txt

review the error message.

If the webhook is calling a service that doesn't exist, the webhook might be leftover from an incomplete or improper removal of a service. follow the steps to uninstall/delete the service.

$ kubectl delete validatingwebhookconfiguration <NAME>

$ kubectl delete mutatingwebhookconfiguration <NAME>

Ref - webhook debugging

Cheers!
RN

[tamaraflorens]

Hi, @RazTN7 

thank you for your reply.

I have curl from every nodes and it running.

I also delete mutatingwebhookconfiguration and validatingwebhookconfiguration.

but error still the same. 

[dortega]

Hi , 

I have the same problem, the firewall rules are applied, but is not working fine when apply kubectl -n dynatrace wait pod --for=condition=ready --selector=app.kubernetes.io/name=dynatrace-operator,app.kubernetes.io/component=webhook --timeout=300s

kubectl -n dynatrace wait pod --for=condition=ready --selector=app.kubernetes.io/name=dynatrace-operator,app.kubernetes.io/component=webhook --timeout=300stimed out waiting for the condition on pods/dynatrace-webhook-7bb4fbbdcf-svvlc
timed out waiting for the condition on pods/dynatrace-webhook-7bb4fbbdcf-xfq5c

Can you help me?

[dortega]

kubectl apply -f dynakube.yaml
secret/gkeproseller created
Error from server (InternalError): error when creating "dynakube.yaml": Internal error occurred: failed calling webhook "webhook.dynatrace.com": failed to call webhook: Post "https://dynatrace-webhook.dynatrace.svc:443/validate?timeout=2s": no endpoints available for service "dynatrace-webhook"

[gWalker_enGen]

@dortega Did you ever find a resolution to this problem? I am also getting the same error message: 
Error from server (InternalError): error when creating "./k8s/development/dynakube.yaml": Internal error occurred: failed calling webhook "v1beta2.dynakube.webhook.dynatrace.com": failed to call webhook: Post "https://dynatrace-webhook.dynatrace.svc:443/validate-dynatrace-com-v1beta2-dynakube?timeout=10s": no endpoints available for service "dynatrace-webhook"

[gopher]

@tamaraflorens ,

Being that you are air gapped - In your CRD you will need to make sure that you set the feature.dynatrace.com/no-proxy:  in your annotations. 
This needs to be for your private repositories and the Dynatrace service address. 
This will prevent the Dynatrace components trying to go through the proxy.

  annotations:
    feature.dynatrace.com/automatic-kubernetes-api-monitoring-cluster-name: "#CONTEXT"  
    feature.dynatrace.com/no-proxy: ".dynatrace.svc,.artifactory.com,.amazonaws.com" #private repo
    feature.dynatrace.com/injection-readonly-volume: "true"
    feature.dynatrace.com/oneagent-initial-connect-retry-ms: "10000"
    feature.dynatrace.com/init-container-seccomp-profile: "true"

[gWalker_enGen]

@gopher The "no-proxy" remedy that you offer is dependent upon the Dynakube being created. However, kubectl apply -f dynakube.yaml command is throwing the error and preventing the Dynakube from being created. So I don't think setting the proxy and/or no-proxy in the Dynakube configuration is the solution to this problem.

[gopher]

@gWalker_enGen , I wouldn't dismiss it.
I've been deploying on GKE for a long time and if you don't set this where you are air gapped and or have a local proxy for egress traffic, you will get this error.

why ? because the call has a HTTPS:// which will go to the proxy first - it's unlikely you have this address registered in the DNS.

The only other scenario is that the operator & webhook pods are not up - in which case your installation of the csi & operator has failed, in which case this will need to be fixed first - however doesn't seem to be the case based on information supplied. 

[gWalker_enGen]

@gopher In my case, I do have a DNS entry for my Dynatrace Host and my Operator and Webhook Pods are NOT up. When I describe my one running Webhook Pod, I get the following:

Liveness probe failed: Get "http://10.121.73.41:10080/livez": dial tcp 10.121.73.41:10080: connect: connection refused

Is this related to "healthchecks" and ports not being open? 

[IzabelaRokita]

Hey @tamaraflorens , did you manage to find the solution to your problem? If so, it would be amazing if you've shared it with the rest of the Community! If not, let me know, and I'll look for some further assistance 😊