Technotes for future me

Upgrade Kubernetes

Kubernetes Upgrade v1.23 to v1.24 – Common Errors and Solutions

Enable the CRI plugin, to make containerd communicate with the kubelet directly.

On all cluster nodes enable cri plugin:

root@master:# vi /etc/containerd/config.toml
#Commented out cri from disabled_plugins
#disabled_plugins: ["cri"]
disabled_plugins: [""]
root@master:# systemctl daemon-reload
root@master:# systemctl restart containerd
root@ubmaster:# systemctl status containerd

root@master:# systemctl restart kubelet
root@master:# systemctl status kubelet

Kubeadm Upgrade (Non-Service Effecting for Pods)

apt-mark unhold kubeadm && \
apt-get update && apt-get install -y kubeadm=1.24.9-00 && \
apt-mark hold kubeadm

Kubelet and Kubectl Upgrade (Service Effecting for Pods)

apt-mark unhold kubelet kubectl && \
apt-get update && apt-get install -y kubelet=1.24.9-00 kubectl=1.24.9-00 && \
apt-mark hold kubelet kubectl


The kubelet service will not come up right away. Below is the error you will encounter, because of which your node will be in NotReady state, even though all packages are upgraded. Below the solution for this error.

root@master:# sudo systemctl status kubelet
  kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
     Active: activating (auto-restart) (Result: exit-code) since 

master kubelet[123456]: Error: failed to parse kubelet flag: unknown flag: --network-plugin

There are a few blogs mentioning removing only “–network-plugin=cni” from the kubeadm-flags.env file in the kubelet directory. But it’s not successful as you will get below error and kubelet will remain down.

root@master:# journalctl -xe | kubelet

W0820 14:15:31.786693  154117 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {  <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial unix: missing address". Reconnecting...
Error: failed to run Kubelet: unable to determine runtime API version: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix: missing address"

Starting Kubelet service after failure

To resolve both Error 2 and 3, we need to comment out the existing environment variable used by the kubelet service and add updated parameters.

root@master:# vi /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--container-runtime=remote --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock"

All services must be up now, wait for pods to come up as well. And repeat the same steps on all nodes including Master and Workers.


Last updated on 27 Feb 2023
Published on 18 Mar 2021
Edit on GitHub