Canary testing
Canary testing
Canary testing is a technique to reduce the risk of introducing a new software version in production by slowly rolling out the change to a small subset of users before rolling it out to the entire infrastructure and making it available to everybody. Canary testing is recommended as a best practice for production deployments. In this case it’s done via the ingress controller, which point to a different service/deployment, which has a different image.
The test can be done via a header in this case x-canary: testing, set with the curl command or in the browser with a header plugin for example https://modheader.com/.
Ingress annotations
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "x-canary"
nginx.ingress.kubernetes.io/canary-by-header-value: "testing"
On varnish
varnishlog -g request -q 'ReqHeader ~ "x-canary: testing"' -g session
On k8s cluster
k logs -n ingress-nginx-default -l ingress=default -f | grep -i contain
Test url
The idea is to use a new url which is not available in PRD yet
curl -k https://blaataap.com/containers/docker/docker_cmd_entrypoint/ -H "x-canary: testing"
Example deployment
Create two deployments, one for the canary and one for the production. Both should have the same host and path in the ingress.
Deployment production
Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: blaataap-ingress
namespace: blaataap
labels:
app.kubernetes.io/instance: blaataap
annotations:
nginx.org/server-tokens: "False"
spec:
ingressClassName: nginx
rules:
- host: blaataap.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: blaataap-svc
port:
number: 80
Service
---
apiVersion: v1
kind: Service
metadata:
name: blaataap-svc
namespace: blaataap
labels:
app: blaataap
app.kubernetes.io/instance: blaataap
spec:
ports:
- port: 80
targetPort: 8080
protocol: TCP
name: http
selector:
app: blaataap
Deployment
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: blaataap
app.kubernetes.io/instance: blaataap
name: blaataap
namespace: blaataap
spec:
replicas: 3
selector:
matchLabels:
app: blaataap
template:
metadata:
labels:
app: blaataap
spec:
containers:
- name: blaataap
image: ghcr.io/adelerhof/blaataap-prd:20240104121247
Deployment canary
Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: blaataap-ingress
namespace: blaataap
labels:
app.kubernetes.io/instance: blaataap
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "x-canary"
nginx.ingress.kubernetes.io/canary-by-header-value: "testing"
spec:
ingressClassName: nginx
rules:
- host: blaataap.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: blaataap-svc
port:
number: 80
Service
---
apiVersion: v1
kind: Service
metadata:
name: blaataap-svc
namespace: blaataap
labels:
app: blaataap
app.kubernetes.io/instance: blaataap
spec:
ports:
- port: 80
targetPort: 8080
protocol: TCP
name: http
selector:
app: blaataap
Deployment
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: blaataap
app.kubernetes.io/instance: blaataap
name: blaataap
namespace: blaataap
spec:
replicas: 3
selector:
matchLabels:
app: blaataap
template:
metadata:
labels:
app: blaataap
spec:
containers:
- name: blaataap
image: ghcr.io/adelerhof/blaataap-prd:20240104141414
Source:
https://kubernetes.github.io/ingress-nginx/examples/canary/
https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/nginx-configuration/annotations.md#canary