Upgrade Kubernetes
Kubernetes Upgrade v1.23 to v1.24 – Common Errors and Solutions
Enable the CRI plugin, to make containerd communicate with the kubelet directly.
On all cluster nodes enable cri plugin:
root@master:# vi /etc/containerd/config.toml
>>>
#Commented out cri from disabled_plugins
#disabled_plugins: ["cri"]
disabled_plugins: [""]
<<<
root@master:# systemctl daemon-reload
root@master:# systemctl restart containerd
root@ubmaster:# systemctl status containerd
root@master:# systemctl restart kubelet
root@master:# systemctl status kubelet
Kubeadm Upgrade (Non-Service Effecting for Pods)
apt-mark unhold kubeadm && \
apt-get update && apt-get install -y kubeadm=1.24.9-00 && \
apt-mark hold kubeadm
Kubelet and Kubectl Upgrade (Service Effecting for Pods)
apt-mark unhold kubelet kubectl && \
apt-get update && apt-get install -y kubelet=1.24.9-00 kubectl=1.24.9-00 && \
apt-mark hold kubelet kubectl
Error
The kubelet service will not come up right away. Below is the error you will encounter, because of which your node will be in NotReady state, even though all packages are upgraded. Below the solution for this error.
root@master:# sudo systemctl status kubelet
kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since
master kubelet[123456]: Error: failed to parse kubelet flag: unknown flag: --network-plugin
There are a few blogs mentioning removing only “–network-plugin=cni” from the kubeadm-flags.env file in the kubelet directory. But it’s not successful as you will get below error and kubelet will remain down.
root@master:# journalctl -xe | kubelet
W0820 14:15:31.786693 154117 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to { <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial unix: missing address". Reconnecting...
Error: failed to run Kubelet: unable to determine runtime API version: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix: missing address"
Starting Kubelet service after failure
To resolve both Error 2 and 3, we need to comment out the existing environment variable used by the kubelet service and add updated parameters.
root@master:# vi /var/lib/kubelet/kubeadm-flags.env
>>>
#KUBELET_KUBEADM_ARGS="--network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.6"
KUBELET_KUBEADM_ARGS="--container-runtime=remote --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=k8s.gcr.io/pause:3.2"
All services must be up now, wait for pods to come up as well. And repeat the same steps on all nodes including Master and Workers.
Source:
https://technokofe.com/2022/08/22/kubernetes-upgrade-v1-23-to-v1-24-common-errors-and-solutions/
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#urgent-upgrade-notes