Cannot install Dynatrace operator on GKE

[tamaraflorens]

Hi,

I'm Facing this issue after running kubectl apply -f dynakube.yaml

Error from server (InternalError): error when creating "dynakube.yaml": Internal error occurred: failed calling webhook "webhook.dynatrace.com": failed to call webhook: Post "https://dynatrace-webhook.dynatrace.svc:443/validate?timeout=10s": no endpoints available for service "dynatrace-webhook"

I following this step 

after searching they say that I should create firewalls rule ingress to port 8443. I do. but the error still the same.

this is my firewall rule configuration

please help..

[agonzalez]

Hello is your port 443 also open ? The url that you gave "https://dynatrace-webhook.dynatrace.svc:443" is set to port 443, check that your firewall allow the 443 port.

[agonzalez]

You can find the dynatrace operator network policies/ports here: https://docs.dynatrace.com/docs/setup-and-configuration/setup-on-k8s/reference/network

[tamaraflorens]

Thanks for your reply. I create firewall rule with allow all but unfortunately it didn't work

I'm still facing this error

Error from server (InternalError): error when creating "dynakube.yaml": Internal error occurred: failed calling webhook "webhook.dynatrace.com": failed to call webhook: Post "https://dynatrace-webhook.dynatrace.svc:443/validate?timeout=10s": no endpoints available for service "dynatrace-webhook"

[RazTN7]

Hi @tamaraflorens 

Is this Dynatrace Saas Env.
If yes. Can you curl from each node of GCP cluster and see the status as running.

curl -X GET https://{environmentid}.live.dynatrace.com/rest/health

It looks like 

mutatingwebhookconfiguration.
and
validatingwebhookconfiguration. service is simply not registering properly.

Follow:

$ kubectl get ValidatingWebhookConfiguration -A

or

$ kubectl describe mutatingwebhookconfigurations,validatingwebhookconfigurations > webhooks.txt

review the error message.

If the webhook is calling a service that doesn't exist, the webhook might be leftover from an incomplete or improper removal of a service. follow the steps to uninstall/delete the service.

$ kubectl delete validatingwebhookconfiguration <NAME>

$ kubectl delete mutatingwebhookconfiguration <NAME>

Ref - webhook debugging

Cheers!
RN

[tamaraflorens]

Hi, @RazTN7 

thank you for your reply.

I have curl from every nodes and it running.

I also delete mutatingwebhookconfiguration and validatingwebhookconfiguration.

but error still the same. 

[dortega]

Hi , 

I have the same problem, the firewall rules are applied, but is not working fine when apply kubectl -n dynatrace wait pod --for=condition=ready --selector=app.kubernetes.io/name=dynatrace-operator,app.kubernetes.io/component=webhook --timeout=300s

kubectl -n dynatrace wait pod --for=condition=ready --selector=app.kubernetes.io/name=dynatrace-operator,app.kubernetes.io/component=webhook --timeout=300stimed out waiting for the condition on pods/dynatrace-webhook-7bb4fbbdcf-svvlc
timed out waiting for the condition on pods/dynatrace-webhook-7bb4fbbdcf-xfq5c

Can you help me?

[dortega]

kubectl apply -f dynakube.yaml
secret/gkeproseller created
Error from server (InternalError): error when creating "dynakube.yaml": Internal error occurred: failed calling webhook "webhook.dynatrace.com": failed to call webhook: Post "https://dynatrace-webhook.dynatrace.svc:443/validate?timeout=2s": no endpoints available for service "dynatrace-webhook"

[gWalker_enGen]

@dortega Did you ever find a resolution to this problem? I am also getting the same error message: 
Error from server (InternalError): error when creating "./k8s/development/dynakube.yaml": Internal error occurred: failed calling webhook "v1beta2.dynakube.webhook.dynatrace.com": failed to call webhook: Post "https://dynatrace-webhook.dynatrace.svc:443/validate-dynatrace-com-v1beta2-dynakube?timeout=10s": no endpoints available for service "dynatrace-webhook"

[gopher]

@tamaraflorens ,

Being that you are air gapped - In your CRD you will need to make sure that you set the feature.dynatrace.com/no-proxy:  in your annotations. 
This needs to be for your private repositories and the Dynatrace service address. 
This will prevent the Dynatrace components trying to go through the proxy.

  annotations:
    feature.dynatrace.com/automatic-kubernetes-api-monitoring-cluster-name: "#CONTEXT"  
    feature.dynatrace.com/no-proxy: ".dynatrace.svc,.artifactory.com,.amazonaws.com" #private repo
    feature.dynatrace.com/injection-readonly-volume: "true"
    feature.dynatrace.com/oneagent-initial-connect-retry-ms: "10000"
    feature.dynatrace.com/init-container-seccomp-profile: "true"

[gWalker_enGen]

@gopher The "no-proxy" remedy that you offer is dependent upon the Dynakube being created. However, kubectl apply -f dynakube.yaml command is throwing the error and preventing the Dynakube from being created. So I don't think setting the proxy and/or no-proxy in the Dynakube configuration is the solution to this problem.

[gopher]

@gWalker_enGen , I wouldn't dismiss it.
I've been deploying on GKE for a long time and if you don't set this where you are air gapped and or have a local proxy for egress traffic, you will get this error.

why ? because the call has a HTTPS:// which will go to the proxy first - it's unlikely you have this address registered in the DNS.

The only other scenario is that the operator & webhook pods are not up - in which case your installation of the csi & operator has failed, in which case this will need to be fixed first - however doesn't seem to be the case based on information supplied.