Edit this page
Report an issueYou are browsing documentation for an older version. See the latest documentation here.
Upgrading to 2.1.x and later versions
This guide walks through the steps needed to upgrade to 2.1.x and later versions from earlier versions, and covers changes from older versions to help operators evaluate whether they need to make changes to their configuration. It also covers creation of a testing environment to test the upgrade.
Prerequisites
- Helm v3 is installed
- You are familiar with Helm
installandupgradeoperations. See the documentation for Helm v3.
Note: Deploying and upgrading via the Helm chart is the supported mechanism for production deployments of KIC. If you’re deploying KIC using Kustomize or some other mechanism, you need to develop and test your own upgrade strategy based on the following examples.
Upgrading from 2.0.x to 2.1.x or later
This document covers both requirements for upgrading from 1.x to 2.x and requirements for upgrading to later 2.x versions past 2.0. If you have already upgraded to 2.0.x, only the steps in this subsection are necessary. If you are still on 1.x, follow the subsequent sections to upgrade to 2.0.x first. You must upgrade to 2.0.x before upgrading to 2.1.x or later.
Once you are on 2.0, you can upgrade to later 2.x versions directly as long as you apply the following CRD and webhook updates and remove the deprecated flag. You do not need to upgrade to 2.1 before upgrading to later 2.x versions.
Update CRDs
The previous KongIngress CRD included several incorrectly-named fields which did not apply correctly to Kong upstreams. 2.1.x CRDs fix these fields:
-
healthchecks.passive.unhealthy.timeoutis nowhealthchecks.passive.unhealthy.timeouts -
healthchecks.active.unhealthy.timeoutis nowhealthchecks.active.unhealthy.timeouts
Before upgrading, review your KongIngresses to see if you use either of the old fields. You will need to manually re-add them with the new field name after upgrading.
Kubernetes does not allow unknown fields in CRDs. Since the old fields don’t exist in 2.1, Kubernetes will strip the old field and its value after updating. There is no automated way to copy the old values into the new field.
Helm does not update CRDs automatically, and 2.1 includes changes to the controller CRDs. You must apply them manually before upgrading:
kubectl apply -f https://raw.githubusercontent.com/Kong/charts/main/charts/kong/crds/custom-resource-definitions.yaml
Updating the admission webhook
2.1 includes changes to the admission webhook to account for changes in the webhook code and to avoid unwanted interaction with Helm Secrets. If you are using Helm, the webhook will update automatically. If not, you should remove your current webhook and follow the admission webhook guide to create a new one.
Deprecated leader election flag
2.1 deprecates the --leader-elect flag (and CONTROLLER_LEADER_ELECT
environment variable). Leader election is now set automatically based on the
database mode (election is enabled for Postgres-backed instances and disabled
for DB-less instances). The flag is still accepted but no longer has any
effect, and will be removed in a future release.
1.x to 2.x Breaking changes
Mechanically the helm upgrade is backwards compatible, but the KIC 2.0.x release
includes some breaking changes for options and controller operations:
- Several controller manager flags were removed or changed
- The format of controller manager logs has changed, and logs are now produced by multiple controllers instead of one
- The admission webhook now requires clients that support TLS 1.2 or later. See the KIC Changelog for all changes in this release.
Flag Changes
If you don’t have a heavily customized KIC deployment (for example,
if you use standard values.yaml options and flags for your Helm deployment
of the KIC), then the following flag changes likely have no impact on you.
However, if you previously set custom arguments for the controller with
options like ingressController.args, pay careful attention to the following
sections and make adjustments to your config.
Removed flags
The following general purpose flags have been removed from the controller manager:
--version--alsologtostderr--logtostderr--v--vmodule
Support for deprecated classless ingress types has been removed:
--process-classless-ingress-v1beta1--process-classless-ingress-v1--process-classless-kong-consumer
Changed flags
The following Ingress controller toggles have been replaced:
-
--disable-ingress-extensionsv1beta1has been replaced by--enable-controller-ingress-extensionsv1beta1=false -
--disable-ingress-networkingv1has been replaced by--enable-controller-ingress-networkingv1=false -
--disable-ingress-networkingv1beta1has been replaced by--enable-controller-ingress-networkingv1beta1=false
If you’re affected by these flag changes, review the Independent Controller Toggling section for more context.
The --dump-config flag is now a boolean:
-
truereplaces the oldenabledvalue -
falsereplaces the olddisabledvalue -
truewith the additional new--dump-sensitive-config=trueflag replaces the oldsensitivevalue
Logging Differences
In versions of the KIC prior to v2.0.0 logging output included a large startup header and the majority of logs were produced by a single logging entity. For example:
-------------------------------------------------------------------------------
Kong Ingress controller
-------------------------------------------------------------------------------
W0825 14:48:18.084560 1 client_config.go:614] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
time="2021-08-25T14:48:18Z" level=info msg="version of kubernetes api-server: 1.21" api-server-host="https://10.96.0.1:443" git_commit=5e58841cce77d4bc13713ad2b91fa0d961e69192 git_tree_state=clean git_version=v1.21.1 major=1 minor=21 platform=linux/amd64
time="2021-08-25T14:48:19Z" level=info msg="kong version: 2.5.0" kong_version=2.5.0
time="2021-08-25T14:48:19Z" level=info msg="datastore strategy for kong: off"
time="2021-08-25T14:48:19Z" level=info msg="chosen Ingress API version: networking.k8s.io/v1"
time="2021-08-25T14:48:55Z" level=info msg="started leading" component=status-syncer
time="2021-08-25T14:48:55Z" level=info msg="syncing configuration" component=controller
time="2021-08-25T14:48:55Z" level=info msg="no configuration change, skipping sync to kong" component=controller
This previous architecture had a single controller that was responsible for all supported resources (for example, Ingress or TCPIngress) and logged for them.
In addition to increasing logging output to help identify problems and conditions during the controller manager runtime, v2.0.0 introduced individual controllers for each supported API type. There is now logging metadata specific to these components. For example:
time="2021-08-25T15:01:37Z" level=info msg="Starting EventSource" logger=controller-runtime.manager.controller.ingress reconciler group=networking.k8s.io reconciler kind=Ingress
time="2021-08-25T15:01:53Z" level=info msg="updating the proxy with new Ingress" NetV1Ingress="{\"Namespace\":\"default\",\"Name\":\"httpbin-ingress-v1beta1\"}" logger=controllers.Ingress.netv1 name=httpbin-ingress-v1beta1 namespace=default
time="2021-08-25T15:01:54Z" level=info msg="successfully synced configuration to kong." subsystem=proxy-cache-resolver
In these example log entries, note the logger=controllers.Ingress.netv1 component.
This helps identify specific components so that operators can more
easily search for these components when reviewing logs.
If you and your team depend on logging output as a significant component of KIC administration in your organization, we recommend deploying v2.0.x in a non-production environment before upgrading. Set up a testing environment and familiarize yourself with the new logging characteristics.
Independent Controller Toggling
In v2.0.x, KIC separates its single monolithic controller into several independent ones focused on specific APIs.
With the v2.0.x release, these independent controllers can be individually
enabled or disabled with the new --enable-controller-{NAME} flags provided
for the controller manager.
Auto-negotiation of the Ingress API version (for example, extensions/v1beta1 or
networking/v1) has been disabled and you now have to set
exactly one option for these specific controllers:
--enable-controller-ingress-extensionsv1beta1--enable-controller-ingress-networkingv1--enable-controller-ingress-networkingv1beta1
In most cases, and with versions of Kubernetes greater than v1.19.x, you can use
the default value of networking/v1.
See the CLI Arguments Reference for a full list of these new options and their default values.
Testing environment
To avoid issues with the upgrade, run it in a test environment before deploying it to production. Create a Kubernetes cluster using the same tools that deployed your production cluster, or use a local development cluster such as minikube or kind.
Using Helm, check the deployed chart version:
In the above example, kong-2.3.0 is the currently deployed chart version.
Using the existing chart version and the values.yaml configuration for
your production environment, deploy a copy to your test cluster
with the --version flag:
$ helm install kong-upgrade-testing kong/kong \
--version ${YOUR_VERSION} \
-f ${PATH_TO_YOUR_VALUES_FILE}
Note: You may need to adjust your chart further to work in a development or staging environment. See the Helm chart documentation. Use this testing environment to walk through the following upgrade steps and ensure there are no problems during the upgrade process. Once you’re satisfied everything is ready, switch to the production cluster and work through the upgrade steps again.
Upgrade
Configure Helm repository
Check the local helm installation to make sure it has the
Kong Charts Repository loaded:
If the repository is not present, add it:
$ helm repo add kong https://charts.konghq.com
Update the repository to pull the latest repository updates:
Perform the upgrade
Run the following command, specifying the old release name, the namespace where
you’ve configured Kong Gateway, and the existing values.yaml configuration file:
$ helm upgrade ${YOUR_RELEASE_NAME} kong/kong \
--namespace ${YOUR_NAMESPACE} \
-f ${PATH_TO_YOUR_VALUES_FILE}
After the upgrade completes there is a brief period of time before the new resources are online. You can wait for the relevant Pod resources to complete by watching them in your release namespace:
$ kubectl -n ${YOUR_RELEASE_NAMESPACE} get pods -w
Once the new pods are in a Ready state, the upgrade is complete.
Rollback
If you run into problems during or after the upgrade, Helm provides a rollback mechanism to revert to a previous revision of the release:
$ helm rollback --namespace ${YOUR_RELEASE_NAMESPACE} ${YOUR_RELEASE_NAME}
You can wait for the rollback to complete by watching the relevant Pod resources:
$ kubectl -n ${YOUR_RELEASE_NAMESPACE} get pods -w
After a rollback, if you ran into issues in production, consider using a testing environment to identify and correct these issues, or reference the troubleshooting documentation.
