Kubernetes CheatSheet

https://kubernetes.io/docs/reference/kubectl/cheatsheet/

https://k8s-examples.container-solutions.com/

Often used Commands

Which container ID belongs to which container

docker ps -a --no-trunc |awk '{print $1, $(NF)}'

CONTAINER ID    CONTAINER NAMES
58bf7e62c1aba6e8cad83afa1116438effe97ea6e399e9f5d02310332f89d388 k8s_kube-controller-manager_kube-controller-manager-k8smaster_kube-system_c51cbe2d8d48e23650ef16139848480e_18
9538a2f4f92ea2f4f8e11ed8d8f6f2c196eda72dec62576db9d0e52366e6b525 k8s_kube-scheduler_kube-scheduler-k8smaster_kube-system_7ee66c7e55e6071f5d478bbafc1eb85a_17
f333093f03eda82138ca4d2ba0e2ff002016ca458f217dacd7ea6e31bff6f634 k8s_coredns_coredns-78fcd69978-bmsf4_kube-system_416048f6-5c97-4ea1-bdb3-54b46310a416_0
1782505a0e9d8cf93ef364565384b8f218130a6207bce028cbefc9f4c5596373 k8s_POD_coredns-78fcd69978-bmsf4_kube-system_416048f6-5c97-4ea1-bdb3-54b46310a416_0
f4416f6b91c29e38e117d6ef02db3bbf31b127059da7b0ab1b9c6287a7591336 focused_faraday

Prepare delete Evicted containers command

k get pod -A |grep Evicted | awk '{print "kubectl delete -n "$1 " pod "$2}'

kubectl delete -n kubernetes-dashboard pod kubernetes-dashboard-9f9799597-84khr
kubectl delete -n kubernetes-dashboard pod kubernetes-dashboard-9f9799597-9prcb
kubectl delete -n kubernetes-dashboard pod kubernetes-dashboard-9f9799597-j2kd5
kubectl delete -n kubernetes-dashboard pod kubernetes-dashboard-9f9799597-zgr6w

Prepare delete Completed containers command

kubectl get pods -A |grep Completed| awk '{print "kubectl delete -n "$1 " pod "$2}'

Drain and cordon (portworx) node

kubectl drain --ignore-daemonsets --delete-local-data <node-name>
pxctl service maintenance --enter
reboot
pxctl service maintenance --exit
watch --color sudo /opt/pwx/bin/pxctl status --color
kubectl uncordon <node-name>

get services in specific namespace and put in yaml files

ns=blaat ; kubectl get svc -n ${ns} | grep -v 10.55 | tail -n +2 | awk '{print "kubectl get svc -n '${ns}' "$1" -oyaml | kubectl-neat > '${ns}-'"$1}'

kubectl get svc -n ${ns} | tail -n +2 | awk '{print "kubectl get svc -n '${ns}' "$1" -oyaml | kubectl-neat > '${ns}-'"$1}'

start debug busybox container

docker pull busybox

kubectl run -i --tty --rm debug --image=ghcr.io/blaat/docker-toolbox:latest --restart=Never -- sh

decode tls secret

kubectl get secrets -n blaat blaat-tls -o json | jq '.data["tls.crt"]' |sed 's/"//g' | base64 -d| openssl x509 --text

update secret

kubectl create secret tls -n blaat blaat-server-tls --key blaat.eu.privkey.pem --cert blaat.eu.fullchain.pem --dry-run=client -o yaml | kubectl apply -f -

Node uptime in cluster

for I in $(kubectl get nodes | tail -n +2 | awk '{print $1}') ; do echo "$I - $(ssh $I uptime)" ; done

Show number of messages in Fluentd buffer

while true ; do echo -n "$(date) - " ; for I in $(kubectl get nodes | cut -d\  -f1 | tail -n +2) ; do echo $I ' - ' $(ssh ${I} du -sk /var/log/fluentd-buffers/kubernetes.forwarder.buffer | cut -f1 ) ; done | awk '{print $3}' | awk '{s+=$1} END {printf "%.0f", s}' ; echo ; sleep 60 ; done

curl service

# Find DNS config
k get cm -n kube-system coredns -oyaml
apiVersion: v1
...
        kubernetes cluster.local in-addr.arpa ip6.arpa {

.. test.blaat.svc.cluster.local

cctl

for I in $(ls ~/.cctl |grep -v config.yml); do cctl generate_inventory -c $I;done

Common Commands

Run curl test temporarily

kubectl run --rm mytest --image=yauritux/busybox-curl -it

Run wget test temporarily

kubectl run --rm mytest --image=busybox -it

Run nginx deployment with 2 replicas

kubectl run my-nginx --image=nginx --replicas=2 --port=80

Run nginx pod and expose it

kubectl run my-nginx --restart=Never --image=nginx --port=80 --expose

Run nginx deployment and expose it

kubectl run my-nginx --image=nginx --port=80 --expose

Set namespace preference

kubectl config set-context <context_name> --namespace=<ns_name>

List pods with nodes info

kubectl get pod -o wide

List everything

kubectl get all --all-namespaces

Get all services

kubectl get service --all-namespaces

Get all deployments

kubectl get deployments --all-namespaces

Show nodes with labels

kubectl get nodes --show-labels

Get resources with json output

kubectl get pods --all-namespaces -o json

Validate yaml file with dry run

kubectl create --dry-run --validate -f pod-dummy.yaml

Start a temporary pod for testing

kubectl run --rm -i -t --image=alpine test-$RANDOM -- sh

kubectl run shell command

kubectl exec -it mytest -- ls -l /etc/hosts

Get system conf via configmap

kubectl -n kube-system get cm kubeadm-config -o yaml

Get deployment yaml

kubectl -n denny-websites get deployment mysql -o yaml

Explain resource

kubectl explain pods=,
kubectl explain svc

Watch pods

kubectl get pods -n wordpress --watch

Query healthcheck endpoint

curl -L http://127.0.0.1:10250/healthz

Open a bash terminal in a pod

kubectl exec -it storage sh

Check pod environment variables

kubectl exec redis-master-ft9ex env

Enable kubectl shell autocompletion

echo "source <(kubectl completion bash)" >>~/.bashrc=, and reload

Use minikube dockerd in your laptop

eval $(minikube docker-env)=, No need to push docker hub any more

Kubectl apply a folder of yaml files

kubectl apply -R -f .

Get services sorted by name

kubectl get services --sort-by=.metadata.name

Get pods sorted by restart count

kubectl get pods --sort-by='.status.containerStatuses[0].restartCount'

List pods and images

kubectl get pods -o='custom-columns=PODS:.metadata.name,Images:.spec.containers[*].image'

OpenShift CheatSheet

Check Performance

Get node resource usage

kubectl top node

Get pod resource usage

kubectl top pod

Get resource usage for a given pod

kubectl top <podname> --containers

List resource utilization for all containers

kubectl top pod --all-namespaces --containers=true

Resources Deletion

Delete pod

kubectl delete pod/<pod-name> -n <my-namespace>

Delete pod by force

kubectl delete pod/<pod-name> --grace-period=0 --force|

Delete pods by labels

kubectl delete pod -l env=test

Delete deployments by labels

kubectl delete deployment -l app=wordpress

Delete all resources filtered by labels

kubectl delete pods,services -l name=myLabel

Delete resources under a namespace

kubectl -n my-ns delete po,svc --all

Delete persist volumes by labels

kubectl delete pvc -l app=wordpress

Delete state fulset only (not pods)

kubectl delete sts/<stateful_set_name> --cascade=false

Log & Conf Files

Config folder: /etc/kubernetes/
Certificate files: /etc/kubernetes/pki/
Credentials to API server: /etc/kubernetes/kubelet.conf Superuser credentials: /etc/kubernetes/admin.conf
kubectl config file: ~/.kube/config
Kubernets working dir: /var/lib/kubelet/
Docker working dir: /var/lib/docker/=,/var/log/containers/
Etcd working dir: /var/lib/etcd/
Network cni: /etc/cni/net.d/
Log files: /var/log/pods/
log in worker node: /var/log/kubelet.log=,/var/log/kube-proxy.log
log in master node: kube-apiserver.log=,kube-scheduler.log=,kube-controller-manager.log
Env: /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Env: export KUBECONFIG=/etc/kubernetes/admin.conf

Pod

List all pods

kubectl get pods

List pods for all namespace

kubectl get pods -all-namespaces

List all critical pods

kubectl get -n kube-system pods -a

List pods with more info

kubectl get pod -o wide=,kubectl get pod/<pod-name> -o yaml

Get pod info

kubectl describe pod/srv-mysql-server

List all pods with labels

kubectl get pods --show-labels

https://github.com/kubernetes/kubernetes/issues/49387 List all unhealthy pods

kubectl get pods --field-selector=status.phase!=Running --all-namespaces

List running pods

kubectl get pods --field-selector=status.phase=Running

Get Pod initContainer status

kubectl get pod --template '{{.status.initContainerStatuses}}' <pod-name>

kubectl run command

kubectl exec -it -n "$ns" "$podname" -- sh -c "echo $msg >>/dev/err.log"

Watch pods

kubectl get pods -n wordpress --watch

Get pod by selector

kubectl get pods --selector="app=syslog" -o jsonpath='{.items[*].metadata.name}'

List pods and images

kubectl get pods -o='custom-columns=PODS:.metadata.name,Images:.spec.containers[*].image'

List pods and containers

-o='custom-columns=PODS:.metadata.name,CONTAINERS:.spec.containers[*].name'

Label & Annontation

Filter pods by label

kubectl get pods -l owner=denny

Manually add label to a pod

kubectl label pods dummy-input owner=denny

Remove label

kubectl label pods dummy-input owner-

Deployment & Scale

Scale out

kubectl scale --replicas=3 deployment/nginx-app

online rolling upgrade

kubectl rollout app-v1 app-v2 --image=img:v2

Roll backup

kubectl rollout app-v1 app-v2 --rollback

List rollout

kubectl get rs

Check update status

kubectl rollout status deployment/nginx-app

Check update history

kubectl rollout history deployment/nginx-app

Pause/Resume

kubectl rollout pause deployment/nginx-deployment=,resume

Rollback to previous version

kubectl rollout undo deployment/nginx-deployment

Quota & Limits & Resource

List Resource Quota

kubectl get resourcequota

List Limit Range

kubectl get limitrange

Customize resource definition

kubectl set resources deployment nginx -c=nginx --limits=cpu=200m

Customize resource definition

kubectl set resources deployment nginx -c=nginx --limits=memory=512Mi

Service

List all services

kubectl get services

List service endpoints

kubectl get endpoints

Get service detail

kubectl get service nginx-service -o yaml

Get service cluster ip

kubectl get service nginx-service -o go-template='{{.spec.clusterIP}}'

Get service cluster port

kubectl get service nginx-service -o go-template='{{(index .spec.ports 0).port}}'

Expose deployment as lb service

kubectl expose deployment/my-app --type=LoadBalancer --name=my-service

Expose service as lb service

kubectl expose service/wordpress-1-svc --type=LoadBalancer --name=ns1

Secrets

List secrets

kubectl get secrets --all-namespaces

Generate secret

echo -n 'mypasswd'=, then redirect tobase64 --decode

Get secret

kubectl get secret denny-cluster-kubeconfig

Get a specific field of a secret

kubectl get secret denny-cluster-kubeconfig -o jsonpath="{.data.value}"

Create secret from cfg file

kubectl create secret generic db-user-pass --from-file=./username.txt

StatefulSet

List statefulset

kubectl get sts

Delete statefulset only (not pods)

kubectl delete sts/<stateful_set_name> --cascade=false

Scale statefulset

kubectl scale sts/<stateful_set_name> --replicas=5

Volumes & Volume Claims

List storage class

kubectl get storageclass

Check the mounted volumes

kubectl exec storage ls /data

Check persist volume

kubectl describe pv/pv0001

Events & Metrics

View all events

kubectl get events --all-namespaces

List Events sorted by timestamp

kubectl get events --sort-by=.metadata.creationTimestamp

Node Maintenance

Mark node as unschedulable

kubectl cordon $NDOE_NAME

Mark node as schedulable

kubectl uncordon $NDOE_NAME

Drain node in preparation for maintenance

kubectl drain $NODE_NAME

Namespace & Security

List authenticated contexts

kubectl config get-contexts=,~/.kube/config

Set namespace preference

kubectl config set-context <context_name> --namespace=<ns_name>

Load context from config file

kubectl get cs --kubeconfig kube_config.yml

Switch context

kubectl config use-context <cluster-name>

Delete the specified context

kubectl config delete-context <cluster-name>

List all namespaces defined

kubectl get namespaces

List certificates

kubectl get csr

https://kubernetes.io/docs/concepts/policy/pod-security-policy/ Check user privilege

kubectl --as=system:serviceaccount:ns-denny:test-privileged-sa -n ns-denny auth can-i use pods/list |

https://kubernetes.io/docs/concepts/policy/pod-security-policy/ Check user privilege

kubectl auth can-i use pods/list

Network

Temporarily add a port-forwarding

kubectl port-forward redis-134 6379:6379

Add port-forwaring for deployment

kubectl port-forward deployment/redis-master 6379:6379

Add port-forwaring for replicaset

kubectl port-forward rs/redis-master 6379:6379

Add port-forwaring for service

kubectl port-forward svc/redis-master 6379:6379

Get network policy

kubectl get NetworkPolicy

Patch

Patch service to loadbalancer

kubectl patch svc $svc_name -p '{"spec": {"type": "LoadBalancer"}}'

Extenstions

Enumerates the resource types available

kubectl api-resources

List api group

kubectl api-versions

List all CRD

kubectl get crd

List storageclass

kubectl get storageclass

Components & Services

Services on Master Nodes

kube-apiserver

exposes the Kubernetes API from master nodes
https://github.com/kubernetes/kubernetes/tree/master/cmd/kube-apiserver

etcd

reliable data store for all k8s cluster data
https://coreos.com/etcd/

kube-scheduler

schedule pods to run on selected nodes
https://github.com/kubernetes/kubernetes/tree/master/cmd/kube-scheduler

kube-controller-manager

node controller, replication controller, endpoints controller, and service account & token controllers
https://github.com/kubernetes/kubernetes/tree/master/cmd/kube-controller-manager

Services on Worker Nodes

kubelet

makes sure that containers are running in a pod
https://github.com/kubernetes/kubernetes/tree/master/cmd/kubelet

kube-proxy

perform connection forwarding
https://github.com/kubernetes/kubernetes/tree/master/cmd/kube-proxy

Container

Runtime Kubernetes supported runtimes: Docker, rkt, runc and any https://github.com/opencontainers/runtime-spec][OCI runtime-spec]] implementation.
https://github.com/docker/engine

Addons: pods and services that implement cluster features

DNS

serves DNS records for Kubernetes services

Web UI

a general purpose, web-based UI for Kubernetes clusters

Container Resource Monitoring

collect, store and serve container metrics

Cluster-level Logging

save container logs to a central log store with search/browsing interface

Tools

More Resources