Kong Gateway 3.4 to 3.10 LTS upgrade

Uses: Kong Gateway

Kong Gateway supports direct upgrades between long-term support (LTS) versions of Kong Gateway Enterprise. This guide walks you through upgrading from Kong Gateway Enterprise 3.4 LTS to Kong Gateway Enterprise 3.10 LTS.

There are three upgrade strategies available when upgrading from an LTS version to a newer LTS version: in-place, dual-cluster, and rolling. This guide describes the best applicable strategy for each deployment mode that Kong Gateway supports. Additionally, it lists some fundamental factors that play important roles in the upgrade process, and explains how to back up and recover data.

This guide uses the following terms in the context of Kong Gateway:

  • Upgrade: The overall process of switching from an older to a newer version of Kong Gateway.
  • Migration: The migration of your data store data into a new environment. For example, the process of moving data from an old PostgreSQL instance to a new one is referred to as database migration.

To make sure your upgrade is successful, carefully review all the steps in this guide. It’s very important to understand all the preparation steps and choose the recommended upgrade path based on your deployment type.

Caution: The migration pattern described in this document can only happen between two LTS versions, Kong Gateway Enterprise 3.4 LTS and Kong Gateway Enterprise 3.10 LTS. If you apply this document to other release intervals, database modifications may be run in the wrong sequence and leave the database schema in a broken state.

Prerequisites

Read this document thoroughly to successfully complete the upgrade process, as it includes all the necessary operational knowledge for the upgrade.

Starting from Kong Gateway 3.5, Enterprise Developer Portal and Vitals are no longer included in Kong Gateway Enterprise. Existing customers who have purchased Kong Enterprise Portal or Vitals can continue to use them and be supported via a dedicated mechanism. If you have purchased Kong Enterprise Portal or Vitals in the past and would like to continue to use them in 3.10, contact Kong Support for more information.

Upgrade journey overview

Preparation phase

There are a number of steps you must complete before upgrading to Kong Gateway 3.10 LTS:

  1. Work through any listed prerequisites.
  2. Back up your database or your declarative configuration files.
  3. Choose the right strategy for upgrading based on your deployment topology.
  4. Review the Kong Gateway changes from 3.4 to 3.10 for any breaking changes that may affect your deployments.
  5. Review any modifications made to the kong.conf file between the LTS releases.
  6. Using your chosen strategy, test migration in a pre-production environment.

Performing the upgrade

The actual execution of the upgrade depends on type of deployment you have with Kong Gateway. In this part of the upgrade journey, you will use the strategy you determined during the preparation phase.

  1. Execute your chosen upgrade strategy on dev.
  2. Move from dev to prod.
  3. Smoke test.
  4. Finish the upgrade or roll back and try again.

Now, let’s move on to preparation, starting with your backup options.

Preparation: Choose a backup strategy

Always back up your database or declarative configuration files before an upgrade. The kong migrations commands used during upgrade and database migration are not reversible.

There are two main types of backup for Kong Gateway entities:

  • Database backup: PostgreSQL has native exporting and importing tools that are reliable and performant, and that ensure consistency when backing up or restoring data. If you’re running Kong Gateway in Traditional or Hybrid mode, you should always take a database-native backup.
  • Declarative backup: Kong ships two declarative backup tools: decK and the Kong CLI, which support managing Kong Gateway entities in the declarative format. For Traditional and Hybrid mode deployments, use these tools to create secondary backups. For DB-less mode deployments, use the Kong CLI and manually manage declarative configuration files.

We highly recommend backing up your data using both methods if possible, as this offers you recovery flexibility.

The database-native tools are robust and can restore data instantly compared to the declarative tools. In case of data corruption, try to do a database-level restore first. Otherwise, bootstrap a new database and use declarative tools to restore configuration from backup files.

Review the Backup and restore guide to prepare backups of your configuration. If you run into any issues and need to roll back, you can also reference that guide to restore your old data store.

Preparation: Choose an upgrade strategy based on deployment mode

Choose your deployment mode:

Here’s a flowchart that breaks down how the decision process works:

Figure 1: Choose an upgrade strategy based on your deployment type. For Traditional mode, choose a dual-cluster upgrade if you have enough resources, or an in-place upgrade if you don’t have enough resources. For DB-less mode and Konnect DPs, use a rolling upgrade. For Hybrid mode, use one of the Traditional mode strategies for CPs, and the rolling upgrade for DPs.

See the following sections for breakdowns of each strategy.

Traditional mode

A Traditional mode deployment is when all Kong Gateway components are running in one environment, and there is no Control Plane/Data Plane separation.

You have two options when upgrading Kong Gateway in Traditional mode:

  • Dual-cluster upgrade: A new Kong Gateway cluster of version Y is deployed alongside the current version X, so that two clusters serve requests concurrently during the upgrade process.
  • In-place upgrade: An in-place upgrade reuses the existing database and has to shut down cluster X first, then configure the new cluster Y to point to the database.

We recommend using a dual-cluster upgrade if you have the resources to run another cluster concurrently. Use the in-place method only if resources are limited, as it will cause business downtime.

Dual-cluster upgrade

Upgrading Kong Gateway from one LTS version to another LTS version with zero downtime can be achieved through a dual-cluster upgrade strategy. This approach involves setting up a new cluster running the upgraded version of Kong Gateway alongside the existing cluster running the current version.

At a high level, the process typically involves the following steps:

  1. Provisioning a same-size deployment: You need to ensure that the new cluster, which will run the upgraded version of Kong Gateway, has the same capacity and resources as the existing cluster. This ensures that both clusters can handle the same amount of traffic and workload.

  2. Setting up dual-cluster deployment: Once the new cluster is provisioned, you can start deploying your APIs and configurations to both clusters simultaneously. The dual cluster deployment allows both the old and new clusters to coexist and process requests in parallel.

  3. Data synchronization: During the dual cluster deployment, data synchronization is crucial to ensure that both clusters have the same data. This can involve migrating data from the old cluster to the new one or setting up a shared data storage solution to keep both clusters in sync. Import the database from the old cluster to the new cluster by using a snapshot or pg_restore.

  4. Traffic rerouting: As the new cluster is running alongside the old one, you can start gradually routing incoming traffic to the new cluster. This process can be done gradually or through a controlled switchover mechanism to minimize any impact on users. This can be achieved by any load balancer, like DNS, Nginx, F5, or even a Kong Gateway node with Canary plugin enabled.

  5. Testing and validation: Before performing a complete switchover to the new cluster, it is essential to thoroughly test and validate the functionality of the upgraded version. This includes testing APIs, plugins, authentication mechanisms, and other functionalities to ensure they are working as expected.

  6. Complete switchover: Once you are confident that the upgraded cluster is fully functional and stable, you can redirect all incoming traffic to the new cluster. This step completes the upgrade process and decommissions the old cluster.

By following this dual cluster deployment strategy, you can achieve a smooth and zero-downtime upgrade from one LTS version of Kong Gateway to another. This approach helps ensure high availability and uninterrupted service for your users throughout the upgrade process.

In-place upgrade

While an in-place upgrade allows you to perform the upgrade on the same infrastructure, it does require some downtime during the actual upgrade process. Plan a suitable maintenance or downtime window during which you can perform the upgrade. During this period, the Kong Gateway will be temporarily unavailable.

For scenarios where zero downtime is critical, consider the dual-cluster upgrade method, keeping in mind the additional resources and complexities.

DB-less mode

In DB-less mode, each independent Kong Gateway node loads a copy of declarative Kong Gateway configuration data into memory without persistent database storage, so failure of some nodes doesn’t spread to other nodes.

Deployments in this mode should use the rolling upgrade strategy. You could parse the validity of the declarative YAML contents with version Y, using the deck gateway validate or the kong config parse command.

You must back up your current kong.yaml file before starting the upgrade.

Hybrid mode

Hybrid mode deployments consist of one or more Control Plane (CP) nodes, and one or more Data Plane (DP) nodes. CP nodes use a database to store Kong Gateway configuration data, whereas DP nodes don’t, since they get all of the needed information from the CP. The recommended upgrade process is a combination of different upgrade strategies for each type of node, CP or DP.

The major challenge with a Hybrid mode upgrade is the communication between the CP and DP. As Hybrid mode requires the minor version of the CP to be no less than that of the DP, you must upgrade CP nodes before DP nodes.

The upgrade must be carried out in two phases:

  1. Upgrade the CP according to the recommendations in the section Traditional mode, while DP nodes are still serving API requests.
  2. Upgrade DP nodes using the recommendations from the section DB-less mode. Point the new DP nodes to the new CP to avoid version conflicts.

The role decoupling feature between CP and DP enables DP nodes to serve API requests while upgrading CP. With this method, there is no business downtime.

Custom plugins (either your own plugins or third-party plugins that are not shipped with Kong Gateway) need to be installed on both the Control Plane and the Data Planes in Hybrid mode. Install the plugins on the Control Plane first, and then the Data Planes.

See the following sections for a breakdown of the options for Hybrid mode deployments.

Control Planes

CP nodes must be upgraded before DP nodes. CP nodes serve an admin-only role and require database support. You can select from the same upgrade strategies nominated for Traditional mode (dual-cluster or in-place), as described in figure 2 and figure 3 respectively.

Upgrading the CP nodes using the dual-cluster strategy:

 
flowchart TD
    DBA[(Current
    database)]
    DBB[(New 
    database)]
    CPX(Current Control Plane X)
    Admin(No admin 
    write operations)
    CPY(New Control Plane Y)
    DPX(fa:fa-layer-group Current Data Plane X nodes)
    API(API requests)

    DBA -.- CPX -."DP connects to either 
CP X...".- DPX Admin -.X.- CPX & CPY DBB --pg_restore--- CPY -."...OR to CP Y".- DPX API--> DPX style API stroke:none!important,fill:none!important style DBA stroke-dasharray:3 style CPX stroke-dasharray:3 style Admin fill:none!important,stroke:none!important,color:#d44324!important linkStyle 2,3 stroke:#d44324!important,color:#d44324!important

Figure 2: The diagram shows a CP upgrade using the dual-cluster strategy. The new CP Y is deployed alongside the current CP X, while current DP nodes X are still serving API requests.

Upgrading the CP nodes using the in-place strategy:

 
flowchart 
    DBA[(Database)]
    CPX(Current Control Plane X 
#40;inactive#41;) Admin(No admin
write operations) CPY(New Control Plane Y) DPX(fa:fa-layer-group Current Data Plane X nodes) API(API requests) DBA -..- CPX -."DP connects to either
CP X...".- DPX Admin -.X.- CPX & CPY DBA --"kong migrations up
kong migrations finish"--- CPY -."...OR to CP Y".- DPX API--> DPX style API stroke:none!important,fill:none!important style CPX stroke-dasharray:3 style Admin fill:none!important,stroke:none!important,color:#d44324!important linkStyle 2,3 stroke:#d44324!important,color:#d44324!important

Figure 3: The diagram shows a CP upgrade using the in-place strategy, where the current CP X is directly replaced by a new CP Y. The database is reused by the new CP Y, and the current CP X is shut down once all nodes are migrated.

From the two diagrams, you can see that DP nodes X remain connected to the current CP node X, or alternatively switch to the new CP node Y. Kong Gateway guarantees that new minor versions of CPs are compatible with old minor versions of the DP, so you can temporarily point DP nodes X to the new CP node Y. This lets you pause the upgrade process if needed, or conduct it over a longer period of time.

This setup is meant to be temporary, to be used only during the upgrade process. We do not recommend running a combination of new versions of CP nodes and old versions of DP nodes in a long-term production deployment.

After the CP upgrade, cluster X can be decommissioned. You can delay this task to the very end of the DP upgrade.

Data Planes

Once the CP nodes are upgraded, you can move on to upgrade the DP nodes. The only supported upgrade strategy for DP upgrades is the rolling upgrade. The following diagrams, figure 4 and 5, are the counterparts of figure 2 and 3 respectively.

Using the dual-cluster strategy with a rolling upgrade workflow:

 
flowchart TD
    DBX[(Current 
database)] DBY[(New
database)] CPX(Current Control Plane X) CPY(New Control Plane Y) DPX(Current Data Planes X) DPY(New Data Planes Y) API(API requests) LB(Load balancer) Admin(No admin
write operations) Admin2(No admin
write operations) subgraph A [ ] Admin -.X.- CPX DBX -.- CPX DBY --- CPY CPX -."Current DP connects to
either CP X...".- DPX Admin2 -.X.- CPY CPY -."...OR to CP Y".- DPX DPX -.90%..- LB CPY --- DPY --10%---- LB end subgraph B [ ] API --> LB & LB & LB end linkStyle 0,4 stroke:#d44324!important,color:#d44324!important linkStyle 8,9 stroke:#b6d7a8!important style CPX stroke-dasharray:3 style DPX stroke-dasharray:3 style DBX stroke-dasharray:3 style API stroke:none!important,fill:none!important style A stroke:none!important,display:none!important style B stroke:none!important,display:none!important style Admin fill:none!important,stroke:none!important,color:#d44324!important style Admin2 fill:none!important,stroke:none!important,color:#d44324!important

Figure 4: The diagram shows a DP upgrade using the dual-cluster and rolling strategies. The new CP Y is deployed alongside with the current CP X, while current DP nodes X are still serving API requests. In the image, the background color of the current database and CP X is grey instead of white, signaling that the old CP is already upgraded and might have been decommissioned.

Using the in-place strategy strategy with a rolling upgrade workflow:

 
flowchart 
    DBA[(Database)]
    CPX(Current Control Plane X 
#40;inactive#41;) CPY(New Control Plane Y) DPX(Current Data Planes X) DPY(New Data Planes Y) API(API requests) LB(Load balancer) Admin(No admin
write operations) Admin2(No admin
write operations) subgraph A [ ] Admin -.X.- CPX DBA -.X.- CPX DBA --- CPY CPX -."Current DP connects to
either CP X...".- DPX Admin2 -.X.- CPY CPY -."OR to CP Y".- DPX -.90%..- LB CPY --- DPY --10%---- LB end subgraph B [ ] API --> LB & LB & LB end linkStyle 0,1,4 stroke:#d44324!important,color:#d44324!important linkStyle 8,9 stroke:#b6d7a8!important style CPX stroke-dasharray:3,stroke:#c1c6cdff!important style DPX stroke-dasharray:3 style A stroke:none!important,color:#fff!important style B stroke:none!important,color:#fff!important style Admin fill:none!important,stroke:none!important,color:#d44324!important style Admin2 fill:none!important,stroke:none!important,color:#d44324!important

Figure 5: The diagram shows a DP upgrade using the in-place and rolling strategies. The diagram shows that the database is reused by the new CP Y, while current DP nodes X are still serving API requests.

Preparation: Review gateway changes

The following tables categorize all relevant changelog entries from Kong Gateway Enterprise 3.4.0.0 up to 3.10.0.3. Carefully review each entry and make changes to your configuration accordingly.

Removed or deprecated

The feature or behaviors in the following table have been permanently removed. By updating settings based on the table below, you can avoid any potential issues that may arise from using deprecated aliases and ensure that your Kong Gateway instance functions correctly with the most recent changes and improvements.

It’s essential to keep configurations up to date to maintain the system’s stability, security, and optimal performance.

Change

Category

Action Required

Deprecated and stopped producing Debian 10 containers and packages, as this version reached its End of Life (EOL) date on June 30, 2024.

As 3.4.3.12, Kong is not building Kong Gateway installation packages or Docker images for Debian 10. Kong is no longer providing official support for any Kong version running on this system.
Deployment Debian 11 and 12 are available. Upgrade to one of these versions before upgrading to Kong Gateway 3.10.
Deprecated and stopped producing CentOS containers and packages.

As 3.4.3.12, Kong is not building Kong Gateway installation packages or Docker images for CentOS. Kong is no longer providing official support for any Kong version running on this system.
Deployment Migrate to a different OS before upgrading to Kong Gateway 3.10.
Deprecated and stopped producing RHEL 7 containers and packages, as this version reached its End of Life (EOL) date on June 30, 2024.

As 3.4.3.12, Kong is not building Kong Gateway installation packages or Docker images for RHEL 7. Kong is no longer providing official support for any Kong version running on this system.
Deployment RHEL 8 and 9 are available. Upgrade to one of these versions before upgrading to Kong Gateway 3.10.
The product component known as Kong Enterprise Portal (Developer Portal) is no longer included in Kong Gateway Enterprise. Existing customers who have purchased Kong Enterprise Portal can continue to use it and be supported via a dedicated mechanism. Developer Portal If you have purchased Kong Enterprise Portal in the past and would like to continue to use it in 3.10, contact Kong Support for more information.
The product component known as Vitals is no longer included in Kong Gateway Enterprise. Existing customers who have purchased Kong Vitals can continue to use it and be supported via a dedicated mechanism. Konnect users can take advantage of our Advanced Analytics offering, which provides a superset of Vitals functionality. Vitals If you have purchased Vitals in the past and would like to continue to use it in 3.10, contact Kong Support for more information.
SAML

The priority of the SAML plugin changed to 1010 to correct the integration between the SAML plugin and other Consumer-based plugins.

This is important for those who run custom plugins, as it may affect the sequence in which your plugins are executed. This does not change the order of execution for plugins in a standard Kong Gateway installation.

Plugins Review custom plugin priorities. If any of the changes in priorities break expected behaviour, adjust as necessary.
In OpenSSL 3.2, the default SSL/TLS security level has been changed from 1 to 2. This means the security level is set to 112 bits of security.

As a result, the following are prohibited:
  • RSA, DSA, and DH keys shorter than 2048 bits
  • ECC keys shorter than 224 bits
  • Any cipher suite using RC4
  • SSL version 3 Additionally, compression is disabled.
SSL Review your security configurations and keys to ensure that they comply to the new standard.
OpenSSL 3.x now includes TLS 1.3. TLS 1.1 and earlier versions have been deprecated and are disabled by default. SSL If you still need to support TLS 1.1, set the ssl_cipher_suite setting to old.
Standardized Redis configuration across plugins. The Redis configuration now follows a common schema that is shared across other plugins.

Kong has changed and refactored the shared Redis configuration that previously was imported by require "kong.enterprise_edition.redis". If you created a custom plugin that is using this shared configuration or if you have a forked version of a plugin, like rate-limiting-advanced, then you might need to do additional steps before you can upgrade to the new version of this Redis config.

Out of the box, custom plugins should still work since the old shared configuration is still available. The new config adds the cluster_max_redirections option for Redis Cluster, and the cluster_nodes format and sentinel_nodes were changed. Other than that, the initialization step is no longer required.

As part of this change, the following plugins switched cluster_addresses to cluster_nodes and sentinel_addresses to sentinel_nodes for Redis configuration:

The following plugins are also now using standardized Redis configuration:

Plugins For bundled plugins, there is no action required. These fields are converted automatically when you run kong migrations up. Also, any changes uploaded via decK or the Admin API using the old cluster_addresses and sentinel_addresses are respected and properly translated to the new fields.

Forked custom plugins aren’t automatically migrated. For more information about how to migrate custom plugins, see Custom plugins that used shared Redis config.
Azure Functions plugin

The Azure Functions plugin now eliminates the upstream/request URI and only uses the config.routeprefix configuration field to construct the request path when requesting the Azure API.
Plugins Update your plugin configuration to use config.routeprefix instead of the deprecated upstream/request URI fields.
OAS Validation plugin

The plugin now bypasses schema validation when the content type is not application/json.
Plugins Ensure that all requests have Content-type: application/json.
Kong Manager now uses the session management mechanism in the OpenID Connect plugin. admin_gui_session_conf is no longer required when authenticating with OIDC. Instead, session-related configuration parameters are set in admin_gui_auth_conf (like session_secret). Kong Manager We recommend reviewing your configuration, as some session-related parameters in admin_gui_auth_conf have different default values compared to the ones in admin_gui_session_conf.

See the migration FAQ for more information.
The listing endpoints for Consumer Groups (/consumer_groups) and Consumers (/consumers) now respond with paginated results. The JSON key for the list has been changed to data instead of consumer_groups or consumers. Admin API Update any CI that requires access to the Consumer and Consumer Groups lists.
If you are using ngx.var.http_* in custom code to access HTTP headers, the behavior of that variable changed slightly when the same header is used multiple times in a single request. Previously, it would return the first value only; now it returns all the values, separated by commas.

Kong Gateway’s PDK header getters and setters work as before.
Nginx Templates Adjust your custom code accordingly.
There are some changes to the configuration of the HashiCorp Vault entity when using the AppRole authentication method:
  • A string entirely made of spaces can’t be specified as the role_id or secret_id.
  • One of secret_id or secret_id_file is required.
Vaults
  • If you have any strings entirely made of spaces as the role_id or secret_id, change them.
  • Specify at least one of secret_id or secret_id_file in the HashiCorp Vault entity when using the AppRole authentication method.
The Granular Tracing feature has been deprecated and removed from Kong Gateway. Tracing Remove the following tracing-related parameters from your kong.conf file:

  • tracing
  • tracing_write_strategy
  • tracing_write_endpoint
  • tracing_time_threshold
  • tracing_types
  • tracing_debug_header
  • generate_trace_details

We recommend transitioning to OpenTelemetry Instrumentation instead.

The kong.logrotate configuration file will no longer be overwritten during upgrade. Logging If running on Debian or Ubuntu, set the environment variable DEBIAN_FRONTEND=noninteractive when upgrading to avoid any interactive prompts and enable fully automatic upgrades.
Changed the encoding of spaces in query arguments from + to %20 in the kong.service.request.clear_query_arg PDK module. While the + character represents the correct encoding of space in query strings, Kong uses %20 in many other APIs, which is inherited from Nginx/OpenResty. PDK No

Compatible

The following table lists behavior changes that may cause your database configuration or kong.conf to fail. This includes deprecated (but not removed) features.

Change

Category

Action Required

GraphQL Rate Limiting Advanced plugin

Schema validation has been updated so that Redis cluster mode is now supported. This schema change does not impact other implementations of this plugin.
Plugins No
The default value of the dns_no_sync option has been changed to off. kong.conf Check if you need to adjust this value.
Session plugin

Introduced the new configuration field read_body_for_logout with a default value of false. This change alters the behavior of logout_post_arg in such a way that it is no longer considered, unless read_body_for_logout is explicitly set to true.

This adjustment prevents the Session plugin from automatically reading request bodies for logout detection, particularly on POST requests.
Plugins If you are using logout_post_arg in your Session plugin configuration, set read_body_for_logout to true to activate it.
Manually specifying a node_id via Kong configuration (kong.conf) is deprecated. The node_id parameter is planned to be removed in a future version. kong.conf We recommend not manually specifying a node_id in configuration.

Possible support required

Upgrades using the following features might require help from Kong.

Kong’s support team provides advanced features and professional services tailored to meet specific business needs, including data migration, custom plugin integration, and seamless coordination with existing settings PDK.

Change

Category

Action Required

Kong Gateway now requires an Enterprise license to use dynamic plugin ordering. Plugins If you have plugin configurations that use dynamic plugin ordering in a free Gateway installation, they won’t work. Remove these configurations before upgrading, or reach out to Kong Support.
Enterprise Free mode is no longer available. Running Kong Gateway without a license will now behave the same as running it with an expired license. Licensing Try out Konnect, or reach out to Kong Sales for a demo.

Perform upgrade

Now that you have chosen an upgrade strategy and reviewed all the relevant changes between the 3.4 and 3.10 LTS releases, you can start the upgrade with your chosen strategy:

Traditional mode or control planes in hybrid mode:

DB-less mode or data planes in hybrid mode:

Troubleshooting

If you run into issues during the upgrade and need to roll back, restore Kong Gateway based on the backup method.

Did this doc help?

Something wrong?

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!