Canary Release

Reduce the risk of introducing a new software version in production by slowly rolling out the change to a small subset of users. This plugin also enables rolling back to your original upstream service, or shifting all traffic to the new version.

Configuration Reference

This plugin is compatible with DB-less mode.

In DB-less mode, you configure Kong Gateway declaratively. Therefore, the Admin API is mostly read-only. The only tasks it can perform are all related to handling the declarative config, including:

• Setting a target's health status in the load balancer
• Validating configurations against schemas
• Uploading the declarative configuration using the /config endpoint

Example plugin configuration

curl -X POST http://localhost:8001/services/SERVICE_NAME|SERVICE_ID/plugins \
    --data "name=canary"  \
    --data "config.percentage=50" \
    --data "config.upstream_host=example.com" \
    --data "config.upstream_port=80"

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: canary-example
config: 
  percentage: 50
  upstream_host: example.com
  upstream_port: 80
plugin: canary

apiVersion: v1
kind: Service
metadata:
  name: SERVICE_NAME|SERVICE_ID
  labels:
    app: SERVICE_NAME|SERVICE_ID
  annotations:
    konghq.com/plugins: canary-example
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: SERVICE_NAME|SERVICE_ID
  selector:
    app: SERVICE_NAME|SERVICE_ID

plugins:
- name: canary
  service: SERVICE_NAME|SERVICE_ID
  config: 
    percentage: 50
    upstream_host: example.com
    upstream_port: 80

curl -X POST http://localhost:8001/routes/ROUTE_NAME|ROUTE_ID/plugins \
  --data "name=canary"  \
    --data "config.percentage=50" \
    --data "config.upstream_host=example.com" \
    --data "config.upstream_port=80"

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: canary-example
config: 
  percentage: 50
  upstream_host: example.com
  upstream_port: 80
plugin: canary

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ROUTE_NAME|ROUTE_ID
  annotations:
    kubernetes.io/ingress.class: kong
    konghq.com/plugins: canary-example
spec:
  rules:
  - host: examplehostname.com
    http:
      paths:
      - path: /bar
        backend:
          service:
            name: echo
            port:
              number: 80

plugins:
- name: canary
  route: ROUTE_NAME
  config: 
    percentage: 50
    upstream_host: example.com
    upstream_port: 80

curl -X POST http://localhost:8001/plugins/ \
  --data "name=canary"  \
    --data "config.percentage=50" \
    --data "config.upstream_host=example.com" \
    --data "config.upstream_port=80"

apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: <global-canary>
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: \"true\"
config: 
  percentage: 50
  upstream_host: example.com
  upstream_port: 80
plugin: canary

plugins:
- name: canary
  config: 
    percentage: 50
    upstream_host: example.com
    upstream_port: 80

Parameters

Here's a list of all the parameters which can be used in this plugin's configuration:

Usage

The Canary Release plugin allows you to route traffic to two separate upstream Services referred to as Service A and Service B. The location of Service A is defined by the service entity for the request being proxied. The location of Service B is defined by the config.upstream_host, config.upstream_port, and/or config.upstream_uri as configured on the plugin.

There are 3 modes of operation:

1. Set a fixed percentage to be routed to Service B. See parameter config.percentage.
2. Define an allow or deny group comprised of Consumers with allowed or denied access to Service B. The Consumer-group association can be configured using the ACL plugin.
3. Set a period (in linear time) over which the traffic will be transferred from Service A to Service B. See parameters config.start and config.duration.

Determining Where to Route a Request

(This does not apply to allowing or denying groups).

The Canary Release plugin defines a number of “buckets” (config.steps). Each of these buckets can be routed to either Service A or Service B.

For example: If you set config.steps to 100 steps, and percentage to 10%, Canary will create 100 “buckets”, 10 of which will be routed to Service B, while the other 90 will remain routed to Service A.

The config.hash parameter determines which requests end up in a specific bucket. When set to consumer, Canary ensures each Consumer will consistently end up in the same bucket. The effect is that once a Consumer’s bucket switches to Service B, it will then always go to Service B, and will not flip-flop between A and B. Alternatively, if it is set to ip or header, then the same concept applies but on the basis of the originating IP address or header value.

When using either the consumer, header, or ip setting, if any specific Consumer, Header, or IP is responsible for more than the average percentage of traffic, the migration may not be evenly distributed, e.g., if the percentage is set to 50%, then 50% of either the Consumers or IPs will be rerouted, but not necessarily 50% of the total requests.

When set to none, the requests will be evenly distributed; each bucket will get the same number of requests, but a Consumer or IP might flip-flop between Service A and Service B on consecutive requests.

Canary provides an automatic fallback if, for some reason, a Consumer, Header, or IP can not be identified. The fallback order is consumer->ip->none.

Overriding the Canary

In some cases, you may want to allow clients to pick Service A or Service B instead of applying the configured canary rules. By setting the config.canary_by_header_name, clients can send the value always to always use the canary service (Service B) or send the value never to never use the canary service (always use Service A).

Finalizing the Canary

Once the Canary is complete, either going to 100% for a percentage-based Canary, or when the timed Canary has reached 100%, the configuration will need to be updated. Note: If the plugin is configured on a route, then all routes for the current service must have completed the Canary.

1. Update the service entity to point to Service B by matching it to the URL as specified by config.upstream_host, config.upstream_uri, and config.upstream_port.
2. Remove the Canary plugin with a DELETE request.

Removing or disabling the Canary Release plugin before the Canary is complete will instantly switch all traffic to Service A.

Upstream Healthchecks

The configuration item upstream_fallback uses Upstream Healthchecks to skip applying the Canary upstream if it does not have at least one healthy target. For this configuration to take effect, the following conditions must be met:

• As this configuration relies on Kong’s balancer (and healthchecks), the name specified in config.upstream_host must point to a valid Kong Upstream object
• Healthchecks are enabled in the canary upstream, i.e., the upstream specified in upstream_host needs to have healthchecks enabled it. It works with both passive and active healthchecks.

Changelog

Kong Gateway 2.8.x

• Added the canary_by_header_name configuration parameter.

Kong Gateway 2.1.x

• Use allow and deny instead of whitelist and blacklist in the groups parameter.
• Added the hash_header configuration parameter.