How to delete a resource with the error: failed calling webhook

The original mistake

In preparation of the GitLab essentials workshop, I’m using helm to deploy it. After a few tests, I wanted to clean up my cluster, and accidentally deleted the namespace before doing helm uninstall. As a result, the namespace got stuck in the “terminating” state…

Troubleshooting

Now the namespace is stuck, but why ?

However, no resources seem to exist yet in the namespace:

rocky@gitlab-master1:dbi-gitlab-ws:~$ kubectl get all -n gitlab
> No resources found

By default, GitLab installs the cert-manager controller, which comes with CRDs. However, the get all command does not return the CRDs:

rocky@gitlab-master1:dbi-gitlab-ws:~$ kubectl get challenges.acme.cert-manager.io -n gitlab
NAME                                              STATE     DOMAIN                                   AGE
gitlab-gitlab-tls-c5nxj-1256604583-3239988248     invalid   gitlab-workshop.dbi-services.com     27m
gitlab-kas-tls-qghrb-3784695029-3983492218        invalid   kas-workshop.dbi-services.com        27m
gitlab-minio-tls-l8676-2620392232-3964581703      invalid   minio-workshop.dbi-services.com      27m
gitlab-registry-tls-k9j6n-1904257687-1249029966   invalid   registry-workshop.dbi-services.com   27m

CRDs delete does not work because the finalizer does not respond during deletion.

The easiest way to do this is to remove the finalizer from the resource:

rocky@gitlab-master1:dbi-gitlab-ws:~$ kubectl patch challenges.acme.cert-manager.io/gitlab-gitlab-tls-c5nxj-1256604583-3239988248 --type=json --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]' -n gitlab
> Error from server (InternalError): Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://gitlab-certmanager-webhook.gitlab.svc:443/mutate?timeout=10s": service "gitlab-certmanager-webhook" not found

Unfortunately, in this case, the patch doesn’t work because the delete of the namespace has removed some resources needed by the finalizer…

Solution

The cert-manager installs webhooks to manage CRDs:

rocky@gitlab-master1:dbi-gitlab-ws:~$ kubectl get ValidatingWebhookConfiguration
NAME                            WEBHOOKS   AGE
cert-manager-webhook            1          81m

rocky@gitlab-master1:dbi-gitlab-ws:~$ kubectl get MutatingWebhookConfiguration
NAME                         WEBHOOKS   AGE
gitlab-certmanager-webhook   1          81m

Webhooks call services and pods that no longer exist in our case. As a result, the webhook call fails and blocks the finalizer.

To correct the problem, simply delete the webhooks:

rocky@gitlab-master1:dbi-gitlab-ws:~$ kubectl delete ValidatingWebhookConfiguration cert-manager-webhook
rocky@gitlab-master1:dbi-gitlab-ws:~$ kubectl delete MutatingWebhookConfiguration gitlab-certmanager-webhook

After that, it is possible to delete the remaining CRDs:

rocky@gitlab-master1:dbi-gitlab-ws:~$ kubectl patch challenges.acme.cert-manager.io/gitlab-gitlab-tls-c5nxj-1256604583-3239988248 --type=json --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]' -n gitlab
> challenge.acme.cert-manager.io/gitlab-gitlab-tls-c5nxj-1256604583-3239988248 patched

The namespace will be automatically deleted once all CRDs have been cleaned.