Canary Release

Reduce the risk of introducing a new software version in production by slowly rolling out the change to a small subset of users. This plugin also enables rolling back to your original upstream service, or shifting all traffic to the new version.

Configuration Reference

Enabling the plugin on a service

1
2
3
4
5
6
$ curl -X POST http://<admin-hostname>:8001/services/<service>/plugins \
    --data "name=canary"  \
    --data "config.percentage=50" \
    --data "config.upstream_host=example.com" \
    --data "config.upstream_fallback=false" \
    --data "config.upstream_port=80"

1
2
3
4
5
6
7
8
9
10
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: <canary-example>
config: 
  percentage: 50
  upstream_host: example.com
  upstream_fallback: false
  upstream_port: 80
plugin: canary

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
apiVersion: v1
kind: Service
metadata:
  name: <service>
  labels:
    app: <service>
  annotations:
    konghq.com/plugins: <canary-example>
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: <service>
  selector:
    app: <service>

1
2
3
4
5
6
7
8
plugins:
- name: canary
  service: <service>
  config: 
    percentage: 50
    upstream_host: example.com
    upstream_fallback: false
    upstream_port: 80

Enabling the plugin on a route

1
2
3
4
5
6
$ curl -X POST http://<admin-hostname>:8001/routes/<route>/plugins \
    --data "name=canary"  \
    --data "config.percentage=50" \
    --data "config.upstream_host=example.com" \
    --data "config.upstream_fallback=false" \
    --data "config.upstream_port=80"

1
2
3
4
5
6
7
8
9
10
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: <canary-example>
config: 
  percentage: 50
  upstream_host: example.com
  upstream_fallback: false
  upstream_port: 80
plugin: canary

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
apiVersion: networking/v1beta1
kind: Ingress
metadata:
  name: <route>
  annotations:
    kubernetes.io/ingress.class: kong
    konghq.com/plugins: <canary-example>
spec:
  rules:
  - host: examplehostname.com
    http:
      paths:
      - path: /bar
        backend:
          serviceName: echo
          servicePort: 80

1
2
3
4
5
6
7
8
plugins:
- name: canary
  route: <route>
  config: 
    percentage: 50
    upstream_host: example.com
    upstream_fallback: false
    upstream_port: 80

Enabling the plugin globally

A plugin which is not associated to any service, route, or consumer is considered global, and will be run on every request. Read the Plugin Reference and the Plugin Precedence sections for more information.

1
2
3
4
5
6
$ curl -X POST http://<admin-hostname>:8001/plugins/ \
    --data "name=canary"  \
    --data "config.percentage=50" \
    --data "config.upstream_host=example.com" \
    --data "config.upstream_fallback=false" \
    --data "config.upstream_port=80"

1
2
3
4
5
6
7
8
9
10
11
12
13
14
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: <global-canary>
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: \"true\"
config: 
  percentage: 50
  upstream_host: example.com
  upstream_fallback: false
  upstream_port: 80
plugin: canary

1
2
3
4
5
6
7
plugins:
- name: canary
  config: 
    percentage: 50
    upstream_host: example.com
    upstream_fallback: false
    upstream_port: 80

Parameters

Here's a list of all the parameters which can be used in this plugin's configuration:

Usage

The Canary Release plugin allows you to route traffic to two separate upstream Services referred to as Service A and Service B. The location of Service A is defined by the service entity for the request being proxied. The location of Service B is defined by the config.upstream_host, config.upstream_port, and/or config.upstream_uri as configured on the plugin.

There are 3 modes of operation:

1. Set a fixed percentage to be routed to Service B. See parameter config.percentage.
2. Define an allow or deny group comprised of Consumers with allowed or denied access to Service B. The Consumer-group association can be configured using the ACL plugin.
3. Set a period (in linear time) over which the traffic will be transferred from Service A to Service B. See parameters config.start and config.duration.

Determining Where to Route a Request

(This does not apply to allowing or denying groups).

The Canary Release plugin defines a number of “buckets” (config.steps). Each of these buckets can be routed to either Service A or Service B.

For example: If you set config.steps to 100 steps, and percentage to 10%, Canary will create 100 “buckets”, 10 of which will be routed to Service B, while the other 90 will remain routed to Service A.

The config.hash parameter determines which requests end up in a specific bucket. When set to consumer, Canary ensures each Consumer will consistently end up in the same bucket. The effect is that once a Consumer’s bucket switches to Service B, it will then always go to Service B, and will not flip-flop between A and B. Alternatively, if it is set to ip, then the same concept applies but on the basis of the originating IP address.

When using either the consumer or ip setting, if any specific Consumer or IP is responsible for more than the average percentage of traffic, the migration may not be evenly distributed, e.g., if the percentage is set to 50%, then 50% of either the Consumers or IPs will be rerouted, but not necessarily 50% of the total requests.

When set to none, the requests will be evenly distributed; each bucket will get the same number of requests, but a Consumer or IP might flip-flop between Service A and Service B on consecutive requests.

Canary provides an automatic fallback if, for some reason, a Consumer or IP can not be identified. The fallback order is consumer->ip->none.

Finalizing the Canary

Once the Canary is complete, either going to 100% for a percentage-based Canary, or when the timed Canary has reached 100%, the configuration will need to be updated. Note: If the plugin is configured on a route, then all routes for the current service must have completed the Canary.

1. Update the service entity to point to Service B by matching it to the URL as specified by config.upstream_host, config.upstream_uri, and config.upstream_port.
2. Remove the Canary plugin with a DELETE request.

Removing or disabling the Canary Release plugin before the Canary is complete will instantly switch all traffic to Service A.

Upstream Healthchecks

The configuration item upstream_fallback uses Upstream Healthchecks to skip applying the Canary upstream if it does not have at least one healthy target. For this configuration to take effect, the following conditions must be met:

• As this configuration relies on Kong’s balancer (and healthchecks), the name specified in config.upstream_host must point to a valid Kong Upstream object
• Healthchecks are enabled in the canary upstream, i.e., the upstream specified in upstream_host needs to have healthchecks enabled it. It works with both passive and active healthchecks.