If you are upgrading from Kong Ingress Controller 2.12, read the upgrading to 3.0 section first
Upgrading Kong Ingress Controller using Helm
Prerequisites
-
Ensure that you installed Kong Ingress Controller 3.0 or above, using Helm.
-
Fetch the latest version of the Kong Helm chart using
helm repo update
. -
Update your
values.yaml
file to use the latest version of Kong Ingress Controller. The values to set are different depending on if you’ve installed with thekong/ingress
chart or thekong/kong
chart. You can find which chart you’re using by runninghelm list -A -o yaml | grep chart
.
Update custom resource definitions
Custom resource definitions (CRDs) are not upgraded when running helm upgrade
. To manually upgrade the CRDs in your cluster, run the following command:
kubectl kustomize https://github.com/kong/kubernetes-ingress-controller/config/crd?ref=v3.5 | kubectl apply -f -
Gateway API versions
Gateway API resources are constantly evolving, and Kong Ingress Controller is updated to use the latest version of the Gateway API CRDs with each release.
Kong Ingress Controller 3.2, 3.4, and 3.5 contain Gateway API changes that require operator intervention.
Kong Ingress Controller 3.2
Starting from version 3.2, Kong Ingress Controller supports Gateway API version 1.1.
The primary change in Gateway API v1.1 is the promotion of GRPCRoute
from v1alpha2 to v1.
If you have been using the Standard Channel of the Gateway API, then you don’t need to do anything extra. Download the latest version of the CRD from the Standard Channel and install it in your cluster directly.
If you installed the Experimental Channel of Gateway API v1.0, complete the following steps to upgrade to version v1.1 of the CRD.
-
Install the Experimental Channel of Gateway API v1.1:
kubectl apply --force=true -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.1.0/experimental-install.yaml
Copied to clipboard!--force=true
is necessary because upstream updated the CRD directly for the alpha stage without preserving the definition of the old version. - Update all your manifests to use
v1
instead ofv1alpha2
. - Upgrade to Kong Ingress Controller v3.2.
Kong Ingress Controller 3.4
Starting in Kong Ingress Controller 3.4, Kong Ingress Controller supports Gateway API version 1.2.
Gateway API 1.2 introduces a breaking change by removing the v1alpha2
versions of GRPCRoute
and ReferenceGrant.
If you are using Gateway API v1.1 with Kong Ingress Controller 3.2 or later, and you do not have any GRPCRoute
or ReferenceGrant
resources in the v1alpha2
version, you can upgrade directly to Gateway API v1.2.
You can use the following script to ensure your GRPCRoute
and ReferenceGrant
CRDs aren’t using v1alpha2
storage version:
kubectl get grpcroutes -A -o jsonpath='{.items[*].apiVersion}' | tr ' ' '\n' | sort | uniq -c
kubectl get referencegrants -A -o jsonpath='{.items[*].apiVersion}' | tr ' ' '\n' | sort | uniq -c
If the output contains v1alpha2
, it means that there are one or more GRPCRoute
or ReferenceGrant
(or both) using v1alpha2
storage version and you must update the manifests before upgrading.
Otherwise, upgrade Gateway API and Kong Ingress Controller following these steps:
- Ensure you are using Gateway API 1.1 (you can upgrade by following the steps in the section above).
- Ensure you are using Kong Ingress Controller version 3.2 or later.
- Update all
GRPCRoute
manifests to usev1
instead ofv1alpha2
, and allReferenceGrant
manifests to usev1beta1
instead ofv1alpha2
. You can follow the Gateway API upgrade guide for detailed steps. - Install the standard or experimental channel of Gateway API 1.2.
Kong Ingress Controller 3.5
Starting from Kong Ingress Controller 3.5, several Kong-specific resources have been deprecated in favor of their Gateway API equivalents, and CRDs are now managed separately.
Resource Deprecations
Kong Ingress Controller 3.5 deprecates several Kong-specific ingress resources. While these resources continue to function, you will see deprecation warnings when using them. The recommended migration paths are:
-
KongIngress
→KongUpstreamPolicy
-
TCPIngress
→TCPRoute
(from Gateway API) -
UDPIngress
→UDPRoute
(from Gateway API)
These deprecated resources will be removed in a future version. We recommend migrating to the Gateway API equivalents as soon as possible.
Gateway API Support
Starting from Kong Ingress Controller 3.5, Kong Ingress Controller supports Gateway API version 1.3.
To upgrade Gateway API from 1.2 to 1.3:
- Ensure you are using Kong Ingress Controller version 3.5 or later.
-
Install Gateway API 1.3 CRDs:
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/standard-install.yaml
Copied to clipboard!For experimental features like CORS filters, install the experimental channel:
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/experimental-install.yaml
Copied to clipboard! - Upgrade to Kong Ingress Controller v3.5.
New Features
The Kong Ingress Controller 3.5 release introduces several new features:
-
Combined HTTP Routes (GA): Now generally available, this feature allows you to consolidate multiple
HTTPRoute
resources into a single Kong Gateway service. Enable with the--combined-services-from-different-httproutes
flag. -
Connection Draining: Ensures graceful handling of client connections when pods terminate. Enable with the
--enable-drain-support
flag.
Upgrade Kong Ingress Controller
Run the following command, specifying the old release name, the namespace where you’ve configured Kong Gateway, and the existing values.yaml
configuration file.
After the upgrade completes, there’s a brief period of time before the new resources are online. You can wait for the relevant Pod resources to cycle by watching them in your release namespace:
kubectl -n $YOUR_RELEASE_NAMESPACE get pods -w
Once the new pods are in a Ready
state, the upgrade is complete.
Rollback
If you run into problems during or after the upgrade, Helm provides a rollback mechanism to revert to a previous revision of the release.
helm rollback --namespace $YOUR_RELEASE_NAMESPACE $YOUR_RELEASE_NAME
You can wait for the rollback to complete by watching the relevant Pod resources:
kubectl -n $YOUR_RELEASE_NAMESPACE get pods -w
After a rollback, if you run into issues in production, consider using a testing environment to identify and correct these issues, or reference the troubleshooting documentation.
Upgrade to Kong Ingress Controller 3.0
Upgrading from Kong Ingress Controller 2.12 to 3.x+ is a major version change which contains breaking changes. To safely upgrade, follow these steps.
-
Switch to Helm as your deployment mechanism.
As of Kong Ingress Controller 3.0, Helm is the only officially supported install method.
-
Upgrade Kong to version 3.4.1 or later.
Kong Ingress Controller 3.0 requires Kong Gateway 3.4.1 or later. You must upgrade your Kong Gateway instances to 3.4.1 before you upgrade to Kong Ingress Controller 3.0.
-
Update the Kong Ingress Controller CRDs.
Helm does not upgrade CRDs automatically. You must apply the 3.x CRDs before you upgrade your releases.
kubectl kustomize https://github.com/Kong/kubernetes-ingress-controller/config/crd/?ref=v3.0.0 | kubectl apply -f -
Copied to clipboard! -
Convert
KongIngress
route
andservice
fields to annotations.Route (Ingress) and Service (Service) configuration fields previously available in
KongIngress
are now all handled via dedicated annotations and will not be respected if set inKongIngress
.For example, if you set the
route.https_redirect_status_code
in aKongIngress
resource, you should now use thekonghq.com/https-redirect-status-code
annotation on an Ingress or HTTPRoute resource. -
Remove the
CombinedRoutes
andCombinedServices
feature gates if set.The
CombinedRoutes
andCombinedServices
feature gates have been enabled by default since versions 2.8.0 and 2.11.0, respectively. Version 3.x removes these feature gates and the combined generators are now the only option. You must remove these flags from theCONTROLLER_FEATURE_GATES
environment variable if they are present. -
Remove the
Knative
feature gate if set.As KNative is no longer supported, you need to use another controller for KNative Ingress resources if you use them.
-
Remove or rename outdated CLI arguments and
CONTROLLER_*
environment variables.Version 3.0 removes or renames several flags that were previously deprecated, removed due to other changes, or retained temporarily for compatibility after their functionality was removed.
The CLI argument versions of these flags are listed below. If you use the equivalent
CONTROLLER_
environment variables (for example,CONTROLLER_SYNC_RATE_LIMIT
for--sync-rate-limit
), you must update those as well.-
--sync-rate-limit
is now--proxy-sync-seconds
. -
--konnect-runtime-group-id
is now--konnect-control-plane-id
. -
--stderrthreshold
and--debug-log-reduce-redundancy
were removed following changes to the logging system. -
--log-level
no longer accepts thewarn
,fatal
, andpanic
values due to consolidation of log levels. -
--update-status-on-shutdown
was removed after its earlier functionality was removed. -
--kong-custom-entities-secret
was removed after its functionality was removed in 2.0. -
--leader-elect
was removed. The controller automatically configures its leader election mode based on other settings. -
--enable-controller-ingress-extensionsv1beta1
and--enable-controller-ingress-networkingv1beta1
were removed following removal of support for older Ingress API versions.
-
Notable changes
The following changes are not considered breaking changes. However, they are notable changes from the previous version and are documented here for completeness.
Expression Router
Kong Gateway 3.0 introduced a new expression-based routing engine. This engine allows Kong Ingress Controller to set some match criteria and route matching precedence not possible under the original Kong Gateway routing engine. This functionality is necessary to implement some aspects of the Gateway API specification.
DB-less configurations in the Helm chart now use the expressions
router_flavor
kong.conf setting by default to take advantage of this functionality. DB-backed configurations use traditional_compatible
instead for backwards compatibility, as existing route configuration from older versions cannot yet be migrated in DB mode.
Use of the new routing engine should not change route matching outside of cases where route precedence did not match the Gateway API specification. The new engine does have different performance characteristics than the old engine, but should improve matching and configuration update speed for most configurations.
Logging changes
Kong Ingress Controller 3.0 uses a new logging engine to unify the logging implementation across all its subsystems. Earlier versions used different logging engines in different subsystems, which led to some logs using a different format than others and some logs appearing at incorrect log levels.
The new logging system consolidates log levels into debug
, info
, and error
. Logs that were previously logged at the warn
level are now logged at error
, as the conditions that triggered warn
level logs were infrequent and should not occur under normal circumstances. fatal
and panic
levels were available in configuration, but were not actually used.
The new logging system changes the default console
format. In earlier versions, console logs used a key=value
format:
time="2023-09-21T23:07:26Z" level=info msg="the ingress class name has been set" logger=setup value=kong othervalue=pong
In 3.0, console
is a mixture of unlabeled tab-delimited fields (for standard keys such as timestamp, log level, and log section) and JSON (for fields specific to individual log entries):
2023-09-22T00:38:16.026Z info setup the ingress class name has been set {"value": "kong","othervalue":"pong"}
The json
format is unchanged except for the order of fields. Earlier versions printed fields in alphabetical order:
{"level":"info","logger":"setup","msg":"the ingress class name has been set","time":"2023-09-21T23:15:15Z","value":"kong"}
Kong Ingress Controller 3.0 or later prints standard log fields first and entry-specific fields in the order they were added in code:
{"level":"info","time":"2023-09-22T00:28:13.006Z","logger":"setup","msg":"the ingress class name has been set","value":"kong"}
Although the default log setting is still console
, json
should be used for production systems, or any other systems that need machine-parseable logs.