Linkerd for K8S Canary Deployment/Traffic splitter.

Reference to Jira Ticket number CTDS-234, Below diagram shows upgrade from v1 to v2 while serving the traffic.
Screenshot 2019-12-10 at 08.31.27

What is Canary deployment (in k8s)?

A deployment strategy is using Canaries (a.k.a. incremental rollouts). With canaries, the new version of the application is gradually deployed to the Kubernetes cluster while getting a very small amount of live traffic (i.e. a subset of live users are connecting to the new version while the rest are still using the previous version).

How to achieve canary deployment in Linkerd?

Answer to above question is Flagger, Flagger is a Kubernetes operator that automates the promotion of canary deployments using Istio, Linkerd, App Mesh, NGINXor Gloorouting for traffic shifting and Prometheus metrics for canary analysis. The canary analysis can be extended with webhooks for running system integration/acceptance tests, load tests, or any other custom validation.Flagger implements a control loop that gradually shifts traffic to the canary while measuring key performance indicators like HTTP requests success rate, requests average duration and pods health. Based on analysis of the KPIsa canary is promoted or aborted, and the analysis result is published to Slackor MS Teams.

How to setup Flagger?

Its very simple but depends on the version of Kubernetes you working currently(day of documenting), there are two versions of Kubernetes cluster one is 1.13 and 1.14, As during the time project decided to use 1.14 version in UAT and PROD environment so to setup Flagger just need to fire this command  (Need kubectl  version 1.14)
kubectl apply -k
This would install Flagger in linkerd namespace.
Example implementation (referenced from Flagger):-
Steps to follow:-

  1. setup namespace for implementation >> kubectl create ns test  (can be any namespace).
  2. inject linkerd proxy into newly created namespace >> kubectl annotate namespace test
  3. This is optional but good to have horizontal pod scaler, Refer metrics-server for setting up metric server/heapster.
    kubectl apply -k
  4. create a custom resource “canary” for your deployment object which need canary

deployment please refer the attach file and replace below parameters:-

Place holder                                                                                Description

  1. __NameOfYourChoice__            Name of your canary deployment object (i.e podinfo).
  2. __NameOfYourNameSpace__      Name of the namespace where deployment lives also where canary deployment would live.
  3. __NameOfYourDeployment__ Name of the target deployment (i.e. pod info).
  4. __NameOfYourDeployment__ This is optional and Name of the target deployment (i.e. pod info).
  5. __ClusterIPPORTNumber__ Port number of Cluster IP service deployed.
  6. __PODPortNumber__ Port Number of Pod decoyed underneath Service(Optional).

It’s good to have test which can send request and keep checking the pod deployments going well though its optional.
kubectl apply -f ./canary-podinfo.yaml
On execution of above command few objects will be applied and few will be generated.

# applied
# generated

Here is the trick bit which actually sets canary deployment after bootstrapping actual deployment would go down to zero and another deployment would come up and start serving on this address.Well Canary deployment setup is ready to cater the request for it. Link to video which shows how canary deployment happens for a sample app. Clarity is not at its best but will give some idea of objects moving in the process. The deployment video will canary deploy from version 3.1.1 to 3.1.2.
Below is the traffic split code which gets generated as command runs in step-5

Name:         podinfo
Namespace:    test
API Version:
Kind:         TrafficSplit
Creation Timestamp:  2019-12-08T19:06:15Z
Generation:          67
Owner References:
API Version: 
Block Owner Deletion:  true
Controller:            true
Kind:                  Canary
Name:                  podinfo
UID:                   a57bc070-4a23-42aa-9c35-1d556f8c97de
Resource Version:        270255
Self Link:               /apis/
UID:                     24bd60f7-336e-4ad9-8774-8413b8ef361f
Service:  podinfo-canary
Weight:   0
Service:  podinfo-primary
Weight:   100
Service:    podinfo

NOTE:- Traffic splitter can be modified and created as custom reproduce for traffic splitting pointing to different service.

What is TrafficSpliter?

This resource allows users to incrementally direct percentages of traffic between various services. It will be used by clients such as ingress controllers or service mesh sidecars to split the outgoing traffic to different destinations. For example there are two versions of deployment V1 and V2, It can be done by deploying both the versions and split traffic between, for example V1 takes half the traffic and V2 takes another half or in ratios of 10/90,20/80, 30/70 so on. When a specific deployment version is been preferred to use all the traffic can route to that version with 0/100. Sample file for traffic splitter:-

kind: TrafficSplit
name: service
service: service
– service: service-V1
weight: 50
– service: service-V2
weight: 50
Output of sample implemenattion.

Screenshot 2019-12-10 at 15.14.33

Challenges to be addressed while implementing:-

Database changes and its impact on old version of deployment.
Unlike Blue/green deployments, Canary releases are based on the following assumptions:
Multiple versions of your application can exist together at the same time, getting live traffic.
If you don’t use some kind of sticky session mechanism, some customers might hit a production server in one request and a canary server in another. something like user-agent to identify the source of request and point to this respective server.

References for setup. For Canary deployment using linkerd. Flagger code base.

Published by sbcons

Sbcons has expertise in different areas of SDLC, provide solution, services and consultancy to all type of industry.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: