Kubernetes

Handling multiple clusters

kubectl config view                    # Short list of clusters in ~/.kube/config
kubectl config current-context         # Get current cluster
kubectl config use-context <context>   # Switch to another context

kubectl config unset current-context   # Clear context (useful for safety in scripts)

Cluster commands

kubectl cluster-info

In general query resource types with

kubectl get <type>
kubectl describe <type plural> <name>
kubectl describe svc/webapp1-clusterip-svc

Useful “get” commands

kubectl get nodes
kubectl get pods
kubectl get rc                     # replication controllers
kubectl get namespaces
kubectl get services or kubectl get svc
export CLUSTER_IP=$(kubectl get services/webapp1-clusterip-svc -o go-template='{{(index .spec.clusterIP)}}')

kubectl get deployments <application>

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   0/3     0            0           1s

kubectl get replicasets

NAME                          DESIRED   CURRENT   READY   AGE
nginx-deployment-75675f5897   3         3         3       18s

kubectl get sa                     # secret attachements

A complete list of all resource and CRDs you can get with

kubectl api-resources
kubectl api-resources --namespaced=false      # List only cluster-wide resources

Creating/manipulating resources

kubectl apply -f clusterip.yml
kubectl create -f some.json
kubectl delete -f some.json                   # Useful to delete the same resources with the same YAML!

kubectl edit <type> <resource>                # Edit live, will automatically restart stuff

kubectl get rc <node> -o yaml >some.yaml
kubectl update -f some.yaml

kubectl annotate <type> <name> [--overwrite=true] key=value

kubectl delete pod -l name=<name>
kubectl delete services &lt;service>
kubectl delete deployment &lt;application>

kubectl run-container <name> --image=<image> --port=<port>

kubectl resize --replicas=4 rc <name>

kubectl expose rc <name> --port=<port> --public-ip=<ip>

Cluster Administration

Removing/Readding nodes

kubectl drain <node>         # Evacuates all pods with replication controllers
kubectl uncordon <node>      # Readd node for pod scheduling

Leave cluster (node)

kubeadm reset

Create new join token

kubeadm token create –print-join-command

Testing kubectl Commands

Several commands allow passing “–dry-run” to test impact

kubectl run --dry-run [...]

Debugging in Pods

Running commands in pods

kubectl exec <pod> [-n <namespace>] <command>

kubectl exec <pod> -it bash      # Note some commands need a proper TTY (so add "-it")

Forwarding ports to pods

kubectl port-forward -n <namespace> <pod> <remote port>:<local port>

For debugging network stuff it is always helpful to start a busybox pod

kubectl run -it busybox --replicas=1 --image=busybox --restart=Never

Defining Limits and Quotas

Conceptionally “quotas” limit the resource usage per namespace while “limits” are maximum allocation amounts per resource type (e.g. cpu, memory, storage, network policies…)

apiVersion: v1
kind: Template
objects:
- apiVersion: v1
  kind: BuildConfig      # or any other...
  spec:
    resources:
      requests:
        cpu: 1
        memory: 2Gi
      limits:
        cpu: 2
        memory: 4Gi

JVM and CFS problems/solutions: https://engineering.squarespace.com/blog/2017/understanding-linux-container-scheduling

Downward API

Using the downward API you can expose pod resource infos to the pod itself via environment variables …

spec:
  containers:
  - env:
    - name: MY_CPU_REQUEST
      valueFrom:
        resourceFieldRef:
          resource: requests.cpu
    - name: MY_MEM_REQUEST
      valueFrom:
        resourceFieldRef:
          resource: requests.memory

… or a volume mount

spec:
  containers: 
    [...]
    volumeMounts:
    - name: podinfo
      mountPath: /etc/podinfo
      readOnly: false
  volumes:
  - name: podinfo
    downwardAPI:
      items:
      - path: "labels"
        fieldRef:
          fieldPath: metadata.labels
      - path: "annotations"
        fieldRef:
          fieldPath: metadata.annotations

Cron & Jobs

Trigger cron manually right now

 kubectl create job --from=cronjob/<the name of cron to run> <job name>

Automatically clean up cron job invocations

Either limit the history via optional spec fields:

.spec.successfulJobsHistoryLimit 
.spec.failedJobsHistoryLimit

Shorter CLI

With those two bash lines

ns() {
    kubectl config set-context --current --namespace=$1 >/dev/null
}
alias k='kubectl "--context=${KUBECTL_CONTEXT:-$(kubectl config current-context)}"'

you can do stuff like

ns myapp
k get pods

Online Tutorials

Katacoda

Accessing Kubernetes API from pods

 curl -v --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://<mycluster>

Migration Stories

Saltside: https://engineering.saltside.se/migrating-to-kubernetes-day-20-problems-fbbda4905c23

Use Cases

Authn/Authz

List of auth proxy implementations

Openshift OAuth proxy (forked from bitly, work with plain k8s too)
pusher OAuth2 proxy (continued work of bitly)
Buzzfeed OAuth2 SSO
linkerd
Envoy (Istio)
Ambassador (Envoy based)
Conjur + Auth Proxy
keyclock-proxy (deprecated)

Misc

Dumping nginx-ingress config

Debugging pods without netstat

cat /proc/net/tcp             # gives you raw data with hex numbers :-(

# Local endpoints
grep -v "rem_address" /proc/net/tcp  | awk  '{x=strtonum("0x"substr($2,index($2,":")-2,2)); for (i=5; i>0; i-=2) x = x"."strtonum("0x"substr($2,i,2))}{print x":"strtonum("0x"substr($2,index($2,":")+1,4))}'

# Remote clients
grep -v "rem_address" /proc/net/tcp  | awk  '{x=strtonum("0x"substr($3,index($3,":")-2,2)); for (i=5; i>0; i-=2) x = x"."strtonum("0x"substr($3,i,2))}{print x":"strtonum("0x"substr($3,index($3,":")+1,4))}'

Print all node ports using a Go template

kubectl get svc --all-namespaces -o go-template='{{range .items}}{{range.spec.ports}}{{if .nodePort}}{{.nodePort}}{{"\n"}}{{end}}{{end}}{{end}}'

Last updated on 31 Jan 2021
Published on 25 Dec 2019
Edit on GitHub