Kong Gateway supports direct upgrades between long-term support (LTS) versions of Kong Gateway Enterprise.
This guide walks you through upgrading from Kong Gateway Enterprise 3.4 LTS to Kong Gateway Enterprise 3.10 LTS.
There are three upgrade strategies available when upgrading from an LTS version to a newer LTS version: in-place, dual-cluster, and rolling.
This guide describes the best applicable strategy for each deployment mode that Kong Gateway supports.
Additionally, it lists some fundamental factors that play important roles in the upgrade process, and explains how to back up and recover data.
This guide uses the following terms in the context of Kong Gateway:
Upgrade: The overall process of switching from an older to a newer version of Kong Gateway.
Migration: The migration of your data store data into a new environment.
For example, the process of moving data from an old PostgreSQL instance to a new one is referred to as database migration.
To make sure your upgrade is successful, carefully review all the steps in this guide. It’s very important to understand all the preparation steps and choose the recommended upgrade path based on your deployment type.
Caution: The migration pattern described in this document can only happen between two LTS versions, Kong Gateway Enterprise 3.4 LTS and Kong Gateway Enterprise 3.10 LTS. If you apply this document to other release intervals, database modifications may be run in the wrong sequence and leave the database schema in a broken state.
Starting from Kong Gateway 3.5, Enterprise Developer Portal and Vitals are no longer
included in Kong Gateway Enterprise.
Existing customers who have purchased Kong Enterprise Portal or Vitals can continue to use them and be supported via a dedicated mechanism.
If you have purchased Kong Enterprise Portal or Vitals in the past and would like to continue to use them in 3.10, contact Kong Support for more information.
The actual execution of the upgrade depends on type of deployment you have with Kong Gateway.
In this part of the upgrade journey, you will use the strategy you determined during the preparation phase.
Execute your chosen upgrade strategy on dev.
Move from dev to prod.
Smoke test.
Finish the upgrade or roll back and try again.
Now, let’s move on to preparation, starting with your backup options.
Always back up your database or declarative configuration files before an upgrade.
The kong migrations commands used during upgrade and database migration are not reversible.
There are two main types of backup for Kong Gateway entities:
Database backup: PostgreSQL has native exporting and importing tools that are reliable and performant, and that ensure consistency when backing up or restoring data. If you’re running Kong Gateway in Traditional or Hybrid mode, you should always take a database-native backup.
Declarative backup: Kong ships two declarative backup tools: decK and the Kong CLI, which support managing Kong Gateway entities in the declarative format.
For Traditional and Hybrid mode deployments, use these tools to create secondary backups. For DB-less mode deployments, use the Kong CLI and manually manage declarative configuration files.
We highly recommend backing up your data using both methods if possible, as this offers you recovery flexibility.
The database-native tools are robust and can restore data instantly compared to the declarative tools.
In case of data corruption, try to do a database-level restore first.
Otherwise, bootstrap a new database and use declarative tools to restore configuration from backup files.
Review the Backup and restore guide to
prepare backups of your configuration.
If you run into any issues and need to roll back, you can also reference that guide to restore your old data store.
Here’s a flowchart that breaks down how the decision process works:
Figure 1: Choose an upgrade strategy based on your deployment type. For Traditional mode, choose a dual-cluster upgrade if you have enough resources, or an in-place upgrade if you don’t have enough resources. For DB-less mode and Konnect DPs, use a rolling upgrade. For Hybrid mode, use one of the Traditional mode strategies for CPs, and the rolling upgrade for DPs.
See the following sections for breakdowns of each strategy.
A Traditional mode deployment is when all Kong Gateway components are running in one environment,
and there is no Control Plane/Data Plane separation.
You have two options when upgrading Kong Gateway in Traditional mode:
Dual-cluster upgrade:
A new Kong Gateway cluster of version Y is deployed alongside the current version X, so that two
clusters serve requests concurrently during the upgrade process.
In-place upgrade: An in-place upgrade reuses
the existing database and has to shut down cluster X first, then configure the new cluster Y to point
to the database.
We recommend using a dual-cluster upgrade if you have the resources to run another cluster concurrently.
Use the in-place method only if resources are limited, as it will cause business downtime.
Upgrading Kong Gateway from one LTS version to another LTS version with zero downtime can be achieved through a dual-cluster upgrade strategy.
This approach involves setting up a new cluster running the upgraded version of Kong Gateway alongside the existing cluster running the current version.
At a high level, the process typically involves the following steps:
Provisioning a same-size deployment: You need to ensure that the new cluster, which will run the upgraded version of Kong Gateway, has the same capacity and resources as the existing cluster. This ensures that both clusters can handle the same amount of traffic and workload.
Setting up dual-cluster deployment: Once the new cluster is provisioned, you can start deploying your APIs and configurations to both clusters simultaneously. The dual cluster deployment allows both the old and new clusters to coexist and process requests in parallel.
Data synchronization: During the dual cluster deployment, data synchronization is crucial to ensure that both clusters have the same data. This can involve migrating data from the old cluster to the new one or setting up a shared data storage solution to keep both clusters in sync. Import the database from the old cluster to the new cluster by using a snapshot or pg_restore.
Traffic rerouting: As the new cluster is running alongside the old one, you can start gradually routing incoming traffic to the new cluster. This process can be done gradually or through a controlled switchover mechanism to minimize any impact on users. This can be achieved by any load balancer, like DNS, Nginx, F5, or even a Kong Gateway node with Canary plugin enabled.
Testing and validation: Before performing a complete switchover to the new cluster, it is essential to thoroughly test and validate the functionality of the upgraded version. This includes testing APIs, plugins, authentication mechanisms, and other functionalities to ensure they are working as expected.
Complete switchover: Once you are confident that the upgraded cluster is fully functional and stable, you can redirect all incoming traffic to the new cluster. This step completes the upgrade process and decommissions the old cluster.
By following this dual cluster deployment strategy, you can achieve a smooth and zero-downtime upgrade from one LTS version of Kong Gateway to another. This approach helps ensure high availability and uninterrupted service for your users throughout the upgrade process.
While an in-place upgrade allows you to perform the upgrade on the same infrastructure,
it does require some downtime during the actual upgrade process.
Plan a suitable maintenance or downtime window during which you can perform the upgrade.
During this period, the Kong Gateway will be temporarily unavailable.
For scenarios where zero downtime is critical, consider the dual-cluster upgrade method,
keeping in mind the additional resources and complexities.
In DB-less mode, each independent Kong Gateway node loads a copy of declarative Kong Gateway
configuration data into memory without persistent database storage, so failure of some nodes doesn’t spread to other nodes.
Deployments in this mode should use the rolling upgrade strategy.
You could parse the validity of the declarative YAML contents with version Y, using the deck gateway validate or the kong config parse command.
You must back up your current kong.yaml file before starting the upgrade.
Hybrid mode deployments consist of one or more Control Plane (CP) nodes, and one or more Data Plane (DP) nodes.
CP nodes use a database to store Kong Gateway configuration data, whereas DP nodes don’t, since they get all of the needed information from the CP.
The recommended upgrade process is a combination of different upgrade strategies for each type of node, CP or DP.
The major challenge with a Hybrid mode upgrade is the communication between the CP and DP.
As Hybrid mode requires the minor version of the CP to be no less than that of the DP, you must upgrade CP nodes before DP nodes.
The upgrade must be carried out in two phases:
Upgrade the CP according to the recommendations in the section Traditional mode,
while DP nodes are still serving API requests.
Upgrade DP nodes using the recommendations from the section DB-less mode.
Point the new DP nodes to the new CP to avoid version conflicts.
The role decoupling feature between CP and DP enables DP nodes to serve API requests while upgrading CP.
With this method, there is no business downtime.
Custom plugins (either your own plugins or third-party plugins that are not shipped with Kong Gateway)
need to be installed on both the Control Plane and the Data Planes in Hybrid mode.
Install the plugins on the Control Plane first, and then the Data Planes.
See the following sections for a breakdown of the options for Hybrid mode deployments.
CP nodes must be upgraded before DP nodes. CP nodes serve an admin-only role and require database support.
You can select from the same upgrade strategies nominated for Traditional mode (dual-cluster or in-place),
as described in figure 2 and figure 3 respectively.
flowchart TD
DBA[(Current
database)]
DBB[(New
database)]
CPX(Current Control Plane X)
Admin(No admin
write operations)
CPY(New Control Plane Y)
DPX(fa:fa-layer-group Current Data Plane X nodes)
API(API requests)
DBA -.- CPX -."DP connects to either CP X...".- DPX
Admin -.X.- CPX & CPY
DBB --pg_restore--- CPY -."...OR to CP Y".- DPX
API--> DPX
style API stroke:none!important,fill:none!important
style DBA stroke-dasharray:3
style CPX stroke-dasharray:3
style Admin fill:none!important,stroke:none!important,color:#d44324!important
linkStyle 2,3 stroke:#d44324!important,color:#d44324!important
Figure 2: The diagram shows a CP upgrade using the dual-cluster strategy.The new CP Y is deployed alongside the current CP X, while current DP nodes X are still serving API requests.
flowchart
DBA[(Database)]
CPX(Current Control Plane X #40;inactive#41;)
Admin(No admin write operations)
CPY(New Control Plane Y)
DPX(fa:fa-layer-group Current Data Plane X nodes)
API(API requests)
DBA -..- CPX -."DP connects to either CP X...".- DPX
Admin -.X.- CPX & CPY
DBA --"kong migrations up kong migrations finish"--- CPY -."...OR to CP Y".- DPX
API--> DPX
style API stroke:none!important,fill:none!important
style CPX stroke-dasharray:3
style Admin fill:none!important,stroke:none!important,color:#d44324!important
linkStyle 2,3 stroke:#d44324!important,color:#d44324!important
Figure 3: The diagram shows a CP upgrade using the in-place strategy, where the current CP X is directly replaced by a new CP Y.The database is reused by the new CP Y, and the current CP X is shut down once all nodes are migrated.
From the two diagrams, you can see that DP nodes X remain connected to the current CP node X, or alternatively switch to the new CP node Y.
Kong Gateway guarantees that new minor versions of CPs are compatible with old minor versions of the DP,
so you can temporarily point DP nodes X to the new CP node Y.
This lets you pause the upgrade process if needed, or conduct it over a longer period of time.
This setup is meant to be temporary, to be used only during the upgrade process.
We do not recommend running a combination of new versions of CP nodes and old versions of DP nodes in a long-term production deployment.
After the CP upgrade, cluster X can be decommissioned. You can delay this task to the very end of the DP upgrade.
Once the CP nodes are upgraded, you can move on to upgrade the DP nodes.
The only supported upgrade strategy for DP upgrades is the rolling upgrade.
The following diagrams, figure 4 and 5, are the counterparts of figure 2 and 3 respectively.
flowchart TD
DBX[(Current database)]
DBY[(New database)]
CPX(Current Control Plane X)
CPY(New Control Plane Y)
DPX(Current Data Planes X)
DPY(New Data Planes Y)
API(API requests)
LB(Load balancer)
Admin(No admin write operations)
Admin2(No admin write operations)
subgraph A [ ]
Admin -.X.- CPX
DBX -.- CPX
DBY --- CPY
CPX -."Current DP connects to either CP X...".- DPX
Admin2 -.X.- CPY
CPY -."...OR to CP Y".- DPX
DPX -.90%..- LB
CPY --- DPY --10%---- LB
end
subgraph B [ ]
API --> LB & LB & LB
end
linkStyle 0,4 stroke:#d44324!important,color:#d44324!important
linkStyle 8,9 stroke:#b6d7a8!important
style CPX stroke-dasharray:3
style DPX stroke-dasharray:3
style DBX stroke-dasharray:3
style API stroke:none!important,fill:none!important
style A stroke:none!important,display:none!important
style B stroke:none!important,display:none!important
style Admin fill:none!important,stroke:none!important,color:#d44324!important
style Admin2 fill:none!important,stroke:none!important,color:#d44324!important
Figure 4: The diagram shows a DP upgrade using the dual-cluster and rolling strategies.The new CP Y is deployed alongside with the current CP X, while current DP nodes X are still serving API requests.In the image, the background color of the current database and CP X is grey instead of white, signaling that the old CP is already upgraded and might have been decommissioned.
flowchart
DBA[(Database)]
CPX(Current Control Plane X #40;inactive#41;)
CPY(New Control Plane Y)
DPX(Current Data Planes X)
DPY(New Data Planes Y)
API(API requests)
LB(Load balancer)
Admin(No admin write operations)
Admin2(No admin write operations)
subgraph A [ ]
Admin -.X.- CPX
DBA -.X.- CPX
DBA --- CPY
CPX -."Current DP connects to either CP X...".- DPX
Admin2 -.X.- CPY
CPY -."OR to CP Y".- DPX -.90%..- LB
CPY --- DPY --10%---- LB
end
subgraph B [ ]
API --> LB & LB & LB
end
linkStyle 0,1,4 stroke:#d44324!important,color:#d44324!important
linkStyle 8,9 stroke:#b6d7a8!important
style CPX stroke-dasharray:3,stroke:#c1c6cdff!important
style DPX stroke-dasharray:3
style A stroke:none!important,color:#fff!important
style B stroke:none!important,color:#fff!important
style Admin fill:none!important,stroke:none!important,color:#d44324!important
style Admin2 fill:none!important,stroke:none!important,color:#d44324!important
Figure 5: The diagram shows a DP upgrade using the in-place and rolling strategies.The diagram shows that the database is reused by the new CP Y, while current DP nodes X are still serving API requests.
The following tables categorize all relevant changelog entries from Kong Gateway Enterprise 3.4.0.0 up to 3.10.0.3.
Carefully review each entry and make changes to your configuration accordingly.
The feature or behaviors in the following table have been permanently removed.
By updating settings based on the table below, you can avoid any potential
issues that may arise from using deprecated aliases and ensure
that your Kong Gateway instance functions correctly with the most recent changes and improvements.
It’s essential to keep configurations up to date to maintain the system’s stability,
security, and optimal performance.
Change
Category
Action Required
Deprecated and stopped producing Debian 10 containers and packages, as this version reached its End of Life (EOL) date on June 30, 2024.
As 3.4.3.12, Kong is not building Kong Gateway installation packages or Docker images for Debian 10. Kong is no longer providing official support for any Kong version running on this system.
Deployment
Debian 11 and 12 are available. Upgrade to one of these versions before
upgrading to Kong Gateway 3.10.
Deprecated and stopped producing CentOS containers and packages.
As 3.4.3.12, Kong is not building Kong Gateway installation packages or Docker images for CentOS. Kong is no longer providing official support for any Kong version running on this system.
Deployment
Migrate to a different OS before upgrading to Kong Gateway 3.10.
Deprecated and stopped producing RHEL 7 containers and packages, as this version reached its End of Life (EOL) date on June 30, 2024.
As 3.4.3.12, Kong is not building Kong Gateway installation packages or Docker images for RHEL 7. Kong is no longer providing official support for any Kong version running on this system.
Deployment
RHEL 8 and 9 are available. Upgrade to one of these versions before
upgrading to Kong Gateway 3.10.
The product component known as Kong Enterprise Portal (Developer Portal) is no longer included in Kong Gateway Enterprise. Existing customers who have purchased Kong Enterprise Portal can continue to use it and be supported via a dedicated mechanism.
Developer Portal
If you have purchased Kong Enterprise Portal in the past and would like to continue to use it in 3.10, contact Kong Support for more information.
The product component known as Vitals is no longer included in Kong Gateway Enterprise. Existing customers who have purchased Kong Vitals can continue to use it and be supported via a dedicated mechanism. Konnect users can take advantage of our Advanced Analytics offering, which provides a superset of Vitals functionality.
Vitals
If you have purchased Vitals in the past and would like to continue to use it in 3.10, contact Kong Support for more information.
SAML
The priority of the SAML plugin changed to 1010 to correct the integration
between the SAML plugin and other Consumer-based plugins.
This is important for those who run custom plugins, as it may affect the sequence in which your plugins are executed.
This does not change the order of execution for plugins in a standard Kong Gateway installation.
Plugins
Review custom plugin priorities.
If any of the changes in priorities break expected behaviour, adjust as necessary.
In OpenSSL 3.2, the default SSL/TLS security level has been changed from 1 to 2.
This means the security level is set to 112 bits of security.
As a result, the following are prohibited:
RSA, DSA, and DH keys shorter than 2048 bits
ECC keys shorter than 224 bits
Any cipher suite using RC4
SSL version 3
Additionally, compression is disabled.
SSL
Review your security configurations and keys to ensure that they comply to the new standard.
OpenSSL 3.x now includes TLS 1.3. TLS 1.1 and earlier versions have been deprecated and are disabled by default.
SSL
If you still need to support TLS 1.1, set the ssl_cipher_suite setting to old.
Standardized Redis configuration across plugins.
The Redis configuration now follows a common schema that is shared across other plugins.
Kong has changed and refactored the shared Redis configuration that previously was imported by require "kong.enterprise_edition.redis". If you created a custom plugin that is using this shared configuration or if you have a forked version of a plugin, like rate-limiting-advanced, then you might need to do additional steps before you can upgrade to the new version of this Redis config.
Out of the box, custom plugins should still work since the old shared configuration is still available. The new config adds the cluster_max_redirections option for Redis Cluster, and the cluster_nodes format and sentinel_nodes were changed. Other than that, the initialization step is no longer required.
As part of this change, the following plugins switched cluster_addresses to cluster_nodes and sentinel_addresses to sentinel_nodes for Redis configuration:
For bundled plugins, there is no action required. These fields are converted automatically when you run kong migrations up. Also, any changes uploaded via decK or the Admin API using the old cluster_addresses and sentinel_addresses are respected and properly translated to the new fields.
The Azure Functions plugin now eliminates the upstream/request URI and only uses the config.routeprefix configuration field to construct the request path when requesting the Azure API.
Plugins
Update your plugin configuration to use config.routeprefix instead of the deprecated upstream/request URI fields.
OAS Validation plugin
The plugin now bypasses schema validation when the content type is not application/json.
Plugins
Ensure that all requests have Content-type: application/json.
Kong Manager now uses the session management mechanism in the OpenID Connect plugin.
admin_gui_session_conf is no longer required when authenticating with OIDC. Instead, session-related
configuration parameters are set in admin_gui_auth_conf (like session_secret).
Kong Manager
We recommend reviewing your configuration, as some session-related parameters in admin_gui_auth_conf have different default values compared to the ones in admin_gui_session_conf.
The listing endpoints for Consumer Groups (/consumer_groups) and Consumers (/consumers) now respond
with paginated results.
The JSON key for the list has been changed to data instead of consumer_groups or consumers.
Admin API
Update any CI that requires access to the Consumer and Consumer Groups lists.
If you are using ngx.var.http_* in custom code to access HTTP headers, the behavior of that variable changed slightly when the same header is used multiple times in a single request. Previously, it would return the first value only; now it returns all the values, separated by commas.
Kong Gateway’s PDK header getters and setters work as before.
Nginx Templates
Adjust your custom code accordingly.
There are some changes to the configuration of the HashiCorp Vault entity when using the AppRole authentication method:
A string entirely made of spaces can’t be specified as the role_id or secret_id.
One of secret_id or secret_id_file is required.
Vaults
If you have any strings entirely made of spaces as the role_id or secret_id, change them.
Specify at least one of secret_id or secret_id_file in the HashiCorp Vault entity when using the AppRole authentication method.
The Granular Tracing feature has been deprecated and removed from Kong Gateway.
Tracing
Remove the following tracing-related parameters from your kong.conf file:
The kong.logrotate configuration file will no longer be overwritten during upgrade.
Logging
If running on Debian or Ubuntu, set the environment variable DEBIAN_FRONTEND=noninteractive when upgrading to avoid any interactive prompts and enable fully automatic upgrades.
Changed the encoding of spaces in query arguments from + to %20 in the kong.service.request.clear_query_arg PDK module.
While the + character represents the correct encoding of space in query strings, Kong uses %20 in many other APIs, which is inherited from Nginx/OpenResty.
The following table lists behavior changes that may cause your database configuration
or kong.conf to fail.
This includes deprecated (but not removed) features.
Change
Category
Action Required
GraphQL Rate Limiting Advanced plugin
Schema validation has been updated so that Redis cluster mode is now supported.
This schema change does not impact other implementations of this plugin.
Plugins
No
The default value of the dns_no_sync option has been changed to off.
kong.conf
Check if you need to adjust this value.
Session plugin
Introduced the new configuration field read_body_for_logout with a default value of false.
This change alters the behavior of logout_post_arg in such a way that it is no longer considered, unless read_body_for_logout is explicitly set to true.
This adjustment prevents the Session plugin from automatically reading request bodies for logout detection, particularly on POST requests.
Plugins
If you are using logout_post_arg in your Session plugin configuration, set read_body_for_logout to true to activate it.
Manually specifying a node_id via Kong configuration (kong.conf) is deprecated.
The node_id parameter is planned to be removed in a future version.
kong.conf
We recommend not manually specifying a node_id in configuration.
Upgrades using the following features might require help from Kong.
Kong’s support team provides advanced features and professional services tailored to meet
specific business needs, including data migration, custom plugin integration,
and seamless coordination with existing settings PDK.
Change
Category
Action Required
Kong Gateway now requires an Enterprise license to use dynamic plugin ordering.
Plugins
If you have plugin configurations that use dynamic plugin ordering in a free Gateway installation, they won’t work.
Remove these configurations before upgrading, or reach out to Kong Support.
Enterprise Free mode is no longer available. Running Kong Gateway without a license will now behave the same as running it with an expired license.
Now that you have chosen an upgrade strategy and reviewed all the relevant changes between the 3.4 and 3.10 LTS releases, you can start the upgrade with your chosen strategy:
Traditional mode or control planes in hybrid mode: