Kubernetes namespace deleting stuck in Terminating state

Please try to troubleshoot and delete remaining resources. Do not force removals unless you know what you are doing. See: https://access.redhat.com/solutions/4165791
https://cloud.redhat.com/blog/the-hidden-dangers-of-terminating-namespaces

This usually happens because something is preventing a resource from being deleted, causing namespace deletion to be stuck. It is necessary to troubleshoot which resources are failing to be deleted and why.

Troubleshoot and delete remaining resources

Try to list all the items in the namespace with the following command

kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace>

If the previous one fails, please try this one, which might not return a complete list but has less chance to fail

kubectl api-resources --verbs=list --cached --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace>

Note: If you get the error below, try the solution below as well.

error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

kubectl get apiservice

Look for ones the AVAILABLE is False If you don’t need those APIs any more, delete them:

kubectl delete apiservce <service-name>

Try manually removing every listed resource and, if one fails, troubleshoot why.

Force individual object removal when it has finalizers - USE WITH CAUTION

Sometimes, a resource (specially a custom resource managed by an operator) may stay “terminating” waiting on a finalizer, although any needed cleanup tasks have been already completed, so it would be necessary to force its removal.

However, a very important warning: forcing the removal of an object without having properly cleaned it up may lead to unstable and unpredictable behavior, so you must be 100% sure this is not the case and open a support case if you have even a minimum doubt. The impact would depend on each operator and what object is affected, but can be potentially high.

Only if you know what you are doing and you are 100% sure that any cleanup tasks for the object have been properly completed but still need to force its removal, you do it this way:

patch -n <project-name> <object-kind>/<object-name> --type=merge -p '{"metadata": {"finalizers":null}}'

Forcing a Deletion of a Namespace - USE WITH CAUTION

Sometimes, when a project has been stuck in “Terminating” state, even if all the resources have been properly removed afterwards, namespace may remain stuck in that state forever, so it becomes necessary to force its removal.

However, a very important warning: forcing the removal of a namespace without having properly cleaned it up may lead to unstable and unpredictable cluster behavior, so you must be 100% sure this is not the case and open a support case if you have even a minimum doubt.

Only if you know what you are doing and you are 100% sure that you have properly cleaned up any resource from the namespace but still need to force its removal, you can follow these steps to do so:

The finalizer for kubernetes needs to be removed. The catch is that it can’t be done with kubectl apply -f, it should be done via the cluster API to work.

Step 1: Dump the descriptor as JSON to a file

There is also a scripted version, see bottom of page

kubectl get namespace blaat -o json > blaat.json

Edit the file:

{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2021-06-17T11:00:43Z",
        "name": "blaat",
        "resourceVersion": "35083891",
        "uid": "f12add74-9432-4217-8842-660443b74fd9"
    },
    "spec": {
        "finalizers": [
            "kubernetes"
        ]
    },
    "status": {
        "phase": "Terminating"
    }
}

Remove kubernetes from the finalisers array:

{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2021-06-17T11:00:43Z",
        "name": "blaat",
        "resourceVersion": "35083891",
        "uid": "f12add74-9432-4217-8842-660443b74fd9"
    },
    "spec": {
        "finalizers": [
        ]
    },
    "status": {
        "phase": "Terminating"
    }
}

Step 2: Executing our cleanup command

Instruct the cluster to get rid of the namespace:

kubectl replace --raw "/api/v1/namespaces/blaat/finalize" -f ./blaat.json

Where:/api/v1/namespaces/<your_namespace_here>/finalize

After running the command, the namespace should now be absent from your namespaces list.

The key thing to note here is the resource you are modifying, in our case, it is for namespaces, it could be pods, deployments, services, etc. This same method can be applied to those resources stuck in Terminating state.

Note: This should be done with caution as it may delete the namespace only and leave orphan objects within the, now non-exiting, namespace - a confusing state for Kubernetes. If this happens, the namespace can be re-created manually and sometimes the orphaned objects will re-appear under the just-created namespace which will allow manual cleanup and recovery.

Scripted removal of namespace finalizer

set -eou pipefail
namespace=$1
if [ -z "$namespace" ]
then
  echo "This script requires a namespace argument input. None found. Exiting."
  exit 1
fi
kubectl get namespace $namespace -o json | jq '.spec = {"finalizers":[]}' > rknf_tmp.json
kubectl proxy &
sleep 5
curl -H "Content-Type: application/json" -X PUT --data-binary @rknf_tmp.json http://localhost:8001/api/v1/namespaces/$namespace/finalize
pkill -9 -f "kubectl proxy"
rm rknf_tmp.json

Usage

sh remove_k8s_ns_finalizer.sh <namespace-name>

Description

1: Use some defensive settings to avoid problems as a result of failure in some of the utilities, programs and pipes which are being tied together here.

2–7: Require a single input argument which is the namespace name and exit if this isn’t provided.

8: Get the live namespace configuration, declaratively swap out the finalizers part and save to a temporary file.

9: Start kube proxy in the background.

10: Wait 5 seconds before continuing. This inserts race condition which allows kube proxy to start up. You could improve on this bit.

11: Apply the new namespace configuration by interacting with the kube proxy to the kube API. You could interact with the API directly but kube proxy provides a static interface for the script because it already know the API endpoint via your kube context configuration.

12: Find and kill the background kube proxy process.

13: Delete the temporary file that was created to remove the side-effect after the script has completed.

Source:
https://craignewtondev.medium.com/how-to-fix-kubernetes-namespace-deleting-stuck-in-terminating-state-5ed75792647e
https://kubernetes.io/blog/2021/05/14/using-finalizers-to-control-deletion/
https://medium.com/pareture/script-to-force-remove-kubernetes-namespace-finalizer-57b72bd9460d
https://access.redhat.com/solutions/4165791
https://cloud.redhat.com/blog/the-hidden-dangers-of-terminating-namespaces