You are browsing documentation for an older version. See the latest documentation here.

Upgrading to Kong 3.x

This guide covers the changes required when upgrading an ingress controller-managed Kong instance from 2.x to 3.x.

Prerequisites

Helm v3 is installed
You are familiar with Helm install and upgrade operations. See the documentation for Helm v3.

Note: Deploying and upgrading via the Helm chart is the supported mechanism for production deployments of KIC. If you’re deploying KIC using Kustomize or some other mechanism, you need to develop and test your own upgrade strategy based on the following examples.

Update CRDs

KIC 2.7 includes Kong Gateway 3.x compatibility changes to the controller CRDs. Helm does not update CRDs automatically. You must apply them manually before upgrading KIC:

kubectl apply -f https://raw.githubusercontent.com/Kong/charts/main/charts/kong/crds/custom-resource-definitions.yaml

Upgrade your KIC version

KIC 2.7 is the first version that supports Kong 3.x. You must upgrade to KIC 2.7 before upgrading to Kong 3.x. See the KIC Changelog for all changes in this release.

2.7 includes a minor breaking change that affects the CombinedRoutes feature gate, but is otherwise not expected to require changes to the existing configuration.

Because KIC 2.7 is compatible with all 2.x Kong Gateway releases, you should upgrade it and the chart first:

helm repo update

helm upgrade ${YOUR_RELEASE_NAME} kong/kong \
  --namespace ${YOUR_NAMESPACE} \
  -f ${PATH_TO_YOUR_VALUES_FILE} \
  --version 2.13.0 \
  --set ingressController.image.tag="2.7"

2.13 is the first chart version that supports 2.7. You can use later versions if available.

Update Ingress regular expression paths for Kong 3.x compatibility

Kong 3.x includes a number of its own breaking changes, but most don’t affect its interactions with KIC, or are handled automatically by 2.7 changes. You should still review the changelog for changes that don’t interact with KIC, such as changes to kong.conf or environment variable settings, and changes to the PDK that affect custom plugins.

Kong 3.x includes changes to regular expression path handling, which do require manual updates to Ingress configuration. In Kong 2.x, Kong applied a heuristic based on the presence of special characters in a route path to determine if a path was a regular expression. This heuristic was imperfect and Kong Gateway 3.x removed it, and instead requires that any regular expression begins with a ~ prefix. Ingress does not allow paths that begin with any character other than /, so Ingress rules with a regular expression path must begin with /~ instead.

Ingress rule paths have no way to indicate that a path is a regular expression. The ImplementationSpecific path type can contain either regular expression or non-regular expression paths. If you have existing Ingresses with regular expression paths, those paths will break if you upgrade to 3.x without updating configuration.

Important: Prefix and Exact rules must never use regular expressions. Only use regular expressions in ImplementationSpecific rules.

The new 3.x paths are also incompatible with 2.x. Adding the /~ prefix to Ingress rules directly breaks incremental upgrades. To smooth the migration process and allow users to update rules gradually, KIC 2.7 includes the enableLegacyRegexDetection option to continue applying the 2.x regular expression heuristic on KIC’s end.

If the following sets of rules are met, KIC creates a Kong route path with the ~ prefix:

The enableLegacyRegexDetection option is enabled
The Kong version is 3.0 or higher
A path matches the 2.x heuristic

or:

The path begins with /~

This allows for a mixture of paths that have and have not been migrated to 3.x-style configuration. The heuristic is never applied to paths that already begin with /~.

Note: If you have paths that require a Kong route that actually begins with /~, set the konghq.com/regex-prefix annotation on their Ingress. This overrides the default prefix. For example, setting konghq.com/regex-prefix: /@ will replace leading /@ sequences with ~ in their Kong route, while leading /~ sequences will be preserved as-is in the Kong route.

To enable legacy regex detection, create an IngressClassParameters resource with enableLegacyRegexDetection=true and attach it to your IngressClass. The option is not enabled for IngressClasses by default.

echo '
apiVersion: configuration.konghq.com/v1alpha1
kind: IngressClassParameters
metadata:
  name: 2x-heuristic
spec:
  enableLegacyRegexDetection: true
' | kubectl apply -n kong -f -

kubectl patch ingressclass kong --patch '
{"spec":
		{"parameters":
				{
				"apiGroup": "configuration.konghq.com",
				"kind": "IngressClassParameters",
				"name": "2x-heuristic",
				"namespace": "kong",
				"scope": "Namespace"
				}
		}
}'

If you use a non-standard namespace or Ingress class, you will need to replace the kong Ingress class name and kong namespace with values appropriate to your environment.

After you have upgraded to Kong 3.x, update all of your regular expression paths to include the /~ prefix in the Ingress itself at your earliest convenience. The heuristic is only intended for decreasing downtime during upgrades, so it behaves exactly like the version included in Kong 2.8.x, bugs included. Once you added the prefix to all regular expression paths, you can disable enableLegacyRegexDetection.

Important: The enableLegacyRegexDetection option is meant to be temporary. Only use it for migration.

The Gateway API HTTPRoute resources are not affected by this problem. They do have a dedicated regular expression path type, and KIC inserts the ~ prefix automatically for these.

Testing environment

To avoid issues with the upgrade, run it in a test environment before deploying it to production. Create a Kubernetes cluster using the same tools that deployed your production cluster, or use a local development cluster such as minikube or kind.

Using Helm, check the deployed chart version:

      Command
    

      Response
    
helm list -A
NAME               NAMESPACE   STATUS   CHART       APP VERSION
ingress-controller kong-system deployed kong-2.13.0 2.7

In the above example, kong-2.13.0 is the currently deployed chart version.

Using the existing chart version and the values.yaml configuration for your production environment, deploy a copy to your test cluster with the --version flag:

helm install kong-upgrade-testing kong/kong \
  --version ${YOUR_VERSION} \
  -f ${PATH_TO_YOUR_VALUES_FILE}

Note: You may need to adjust your chart further to work in a development or staging environment. See the Helm chart documentation. Use this testing environment to walk through the following upgrade steps and ensure there are no problems during the upgrade process. Once you’re satisfied everything is ready, switch to the production cluster and work through the upgrade steps again.

Upgrade

Configure Helm repository

Check the local helm installation to make sure it has the Kong Charts Repository loaded:

      Command
    

      Response
    
helm repo list
NAME    URL
kong    https://charts.konghq.com

If the repository is not present, add it:

helm repo add kong https://charts.konghq.com

Update the repository to pull the latest repository updates:

      Command
    

      Response
    
helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kong" chart repository
Update Complete. ⎈Happy Helming!⎈

Perform the upgrade

Run the following command, specifying the old release name, the namespace where you’ve configured Kong Gateway, and the existing values.yaml configuration file:

helm upgrade ${YOUR_RELEASE_NAME} kong/kong \
  --namespace ${YOUR_NAMESPACE} \
  -f ${PATH_TO_YOUR_VALUES_FILE} \
  --set image.tag="3.0"

After the upgrade completes, there is a brief period of time before the new resources are online. You can wait for the relevant Pod resources to complete by watching them in your release namespace:

kubectl -n ${YOUR_RELEASE_NAMESPACE} get pods -w

Once the new pods are in a Ready state, the upgrade is complete. Update your values.yaml file to use the new Kong and Kong Ingress Controller image versions to continue using these through future upgrades.

Rollback

If you run into problems during or after the upgrade, Helm provides a rollback mechanism to revert to a previous revision of the release:

helm rollback --namespace ${YOUR_RELEASE_NAMESPACE} ${YOUR_RELEASE_NAME}

You can wait for the rollback to complete by watching the relevant Pod resources:

kubectl -n ${YOUR_RELEASE_NAMESPACE} get pods -w

After a rollback, if you ran into issues in production, consider using a testing environment to identify and correct these issues, or reference the troubleshooting documentation.