Kong Gateway supports direct upgrades between long-term support (LTS) versions of Kong Gateway Enterprise.
This guide walks you through upgrading from Kong Gateway Enterprise 2.8 LTS to Kong Gateway Enterprise 3.4 LTS.
There are three upgrade strategies available for the LTS to LTS upgrade: in-place, dual-cluster, and rolling.
This guide nominates the best applicable strategy for each deployment mode that Kong Gateway supports.
Additionally, it lists some fundamental factors that play important roles in the upgrade process, and explains how to back up and recover data.
This guide uses the following terms in the context of Kong Gateway:
Upgrade: The overall process of switching from an older to a newer version of Kong Gateway.
Migration: The migration of your data store data into a new environment.
For example, the process of moving 2.8.x data from an old PostgreSQL instance to a new one for 3.4.x is referred to as database migration.
To make sure your upgrade is successful, carefully review all the steps in this guide. It’s very important to understand all the preparation steps and choose the recommended upgrade path based on your deployment type.
Caution: The migration pattern described in this document can only happen between two LTS versions, Kong Gateway Enterprise 2.8 LTS and Kong Gateway Enterprise 3.4 LTS. If you apply this document to other release intervals, database modifications may be run in the wrong sequence and leave the database schema in a broken state. To migrate between other versions, see the general upgrade guide.
Starting from Kong Gateway 3.4, Cassandra is not supported.
If you’re using Cassandra as your data store, migrate off of Cassandra first and upgrade second.
Work with the Kong support team to migrate from Cassandra to PostgreSQL.
The actual execution of the upgrade depends on type of deployment you have with Kong Gateway.
In this part of the upgrade journey, you will use the strategy you determined during the preparation phase.
Execute your chosen upgrade strategy on dev.
Move from dev to prod.
Smoke test.
Finish the upgrade or roll back and try again.
Now, let’s move on to preparation, starting with your backup options.
Always back up your database or declarative configuration files before an upgrade.
The kong migrations commands used during upgrade and database migration are not reversible.
There are two main types of backup for Kong Gateway entities:
Database backup: PostgreSQL has native exporting and importing tools that are reliable and performant, and that ensure consistency when backing up or restoring data. If you’re running Kong Gateway in Traditional or Hybrid mode, you should always take a database-native backup.
Declarative backup: Kong ships two declarative backup tools: decK and the Kong CLI, which support managing Kong Gateway entities in the declarative format.
For Traditional and Hybrid mode deployments, use these tools to create secondary backups. For DB-less mode deployments, use the Kong CLI and manually manage declarative configuration files.
We highly recommend backing up your data using both methods if possible, as this offers you recovery flexibility.
The database-native tools are robust and can restore data instantly compared to the declarative tools.
In case of data corruption, try to do a database-level restore first.
Otherwise, bootstrap a new database and use declarative tools to restore configuration from backup files.
Review the Backup and restore guide to
prepare backups of your configuration.
If you run into any issues and need to roll back, you can also reference that guide to restore your old data store.
Here’s a flowchart that breaks down how the decision process works:
Figure 1: Choose an upgrade strategy based on your deployment type. For Traditional mode, choose a dual-cluster upgrade if you have enough resources, or an in-place upgrade if you don’t have enough resources. For DB-less mode and Konnect DPs, use a rolling upgrade. For Hybrid mode, use one of the Traditional mode strategies for CPs, and the rolling upgrade for DPs.
See the following sections for breakdowns of each strategy.
A Traditional mode deployment is when all Kong Gateway components are running in one environment,
and there is no Control Plane/Data Plane separation.
You have two options when upgrading Kong Gateway in Traditional mode:
Dual-cluster upgrade:
A new Kong Gateway cluster of version Y is deployed alongside the current version X, so that two
clusters serve requests concurrently during the upgrade process.
In-place upgrade: An in-place upgrade reuses
the existing database and has to shut down cluster X first, then configure the new cluster Y to point
to the database.
We recommend using a dual-cluster upgrade if you have the resources to run another cluster concurrently.
Use the in-place method only if resources are limited, as it will cause business downtime.
Upgrading Kong Gateway from one LTS version to another LTS version with zero downtime can be achieved through a dual-cluster upgrade strategy.
This approach involves setting up a new cluster running the upgraded version of Kong Gateway alongside the existing cluster running the current version.
At a high level, the process typically involves the following steps:
Provisioning a same-size deployment: You need to ensure that the new cluster, which will run the upgraded version of Kong Gateway, has the same capacity and resources as the existing cluster. This ensures that both clusters can handle the same amount of traffic and workload.
Setting up dual-cluster deployment: Once the new cluster is provisioned, you can start deploying your APIs and configurations to both clusters simultaneously. The dual cluster deployment allows both the old and new clusters to coexist and process requests in parallel.
Data synchronization: During the dual cluster deployment, data synchronization is crucial to ensure that both clusters have the same data. This can involve migrating data from the old cluster to the new one or setting up a shared data storage solution to keep both clusters in sync. Import the database from the old cluster to the new cluster by using a snapshot or pg_restore.
Traffic rerouting: As the new cluster is running alongside the old one, you can start gradually routing incoming traffic to the new cluster. This process can be done gradually or through a controlled switchover mechanism to minimize any impact on users. This can be achieved by any load balancer, like DNS, Nginx, F5, or even a Kong Gateway node with Canary plugin enabled.
Testing and validation: Before performing a complete switchover to the new cluster, it is essential to thoroughly test and validate the functionality of the upgraded version. This includes testing APIs, plugins, authentication mechanisms, and other functionalities to ensure they are working as expected.
Complete switchover: Once you are confident that the upgraded cluster is fully functional and stable, you can redirect all incoming traffic to the new cluster. This step completes the upgrade process and decommissions the old cluster.
By following this dual cluster deployment strategy, you can achieve a smooth and zero-downtime upgrade from one LTS version of Kong Gateway to another. This approach helps ensure high availability and uninterrupted service for your users throughout the upgrade process.
While an in-place upgrade allows you to perform the upgrade on the same infrastructure,
it does require some downtime during the actual upgrade process.
Plan a suitable maintenance or downtime window during which you can perform the upgrade.
During this period, the Kong Gateway will be temporarily unavailable.
For scenarios where zero downtime is critical, consider the dual-cluster upgrade method,
keeping in mind the additional resources and complexities.
In DB-less mode, each independent Kong Gateway node loads a copy of declarative Kong Gateway
configuration data into memory without persistent database storage, so failure of some nodes doesn’t spread to other nodes.
Deployments in this mode should use the rolling upgrade strategy.
You could parse the validity of the declarative YAML contents with version Y, using the deck gateway validate or the kong config parse command.
You must back up your current kong.yaml file before starting the upgrade.
Hybrid mode deployments consist of one or more Control Plane (CP) nodes, and one or more Data Plane (DP) nodes.
CP nodes use a database to store Kong Gateway configuration data, whereas DP nodes don’t, since they get all of the needed information from the CP.
The recommended upgrade process is a combination of different upgrade strategies for each type of node, CP or DP.
The major challenge with a Hybrid mode upgrade is the communication between the CP and DP.
As Hybrid mode requires the minor version of the CP to be no less than that of the DP, you must upgrade CP nodes before DP nodes.
The upgrade must be carried out in two phases:
Upgrade the CP according to the recommendations in the section Traditional mode,
while DP nodes are still serving API requests.
Upgrade DP nodes using the recommendations from the section DB-less mode.
Point the new DP nodes to the new CP to avoid version conflicts.
The role decoupling feature between CP and DP enables DP nodes to serve API requests while upgrading CP.
With this method, there is no business downtime.
Custom plugins (either your own plugins or third-party plugins that are not shipped with Kong Gateway)
need to be installed on both the Control Plane and the Data Planes in Hybrid mode.
Install the plugins on the Control Plane first, and then the Data Planes.
See the following sections for a breakdown of the options for Hybrid mode deployments.
CP nodes must be upgraded before DP nodes. CP nodes serve an admin-only role and require database support.
You can select from the same upgrade strategies nominated for Traditional mode (dual-cluster or in-place),
as described in figure 2 and figure 3 respectively.
flowchart TD
DBA[(Current
database)]
DBB[(New
database)]
CPX(Current Control Plane X)
Admin(No admin
write operations)
CPY(New Control Plane Y)
DPX(fa:fa-layer-group Current Data Plane X nodes)
API(API requests)
DBA -.- CPX -."DP connects to either CP X...".- DPX
Admin -.X.- CPX & CPY
DBB --pg_restore--- CPY -."...OR to CP Y".- DPX
API--> DPX
style API stroke:none!important,fill:none!important
style DBA stroke-dasharray:3
style CPX stroke-dasharray:3
style Admin fill:none!important,stroke:none!important,color:#d44324!important
linkStyle 2,3 stroke:#d44324!important,color:#d44324!important
Figure 2: The diagram shows a CP upgrade using the dual-cluster strategy.The new CP Y is deployed alongside the current CP X, while current DP nodes X are still serving API requests.
flowchart
DBA[(Database)]
CPX(Current Control Plane X #40;inactive#41;)
Admin(No admin write operations)
CPY(New Control Plane Y)
DPX(fa:fa-layer-group Current Data Plane X nodes)
API(API requests)
DBA -..- CPX -."DP connects to either CP X...".- DPX
Admin -.X.- CPX & CPY
DBA --"kong migrations up kong migrations finish"--- CPY -."...OR to CP Y".- DPX
API--> DPX
style API stroke:none!important,fill:none!important
style CPX stroke-dasharray:3
style Admin fill:none!important,stroke:none!important,color:#d44324!important
linkStyle 2,3 stroke:#d44324!important,color:#d44324!important
Figure 3: The diagram shows a CP upgrade using the in-place strategy, where the current CP X is directly replaced by a new CP Y.The database is reused by the new CP Y, and the current CP X is shut down once all nodes are migrated.
From the two diagrams, you can see that DP nodes X remain connected to the current CP node X, or alternatively switch to the new CP node Y.
Kong Gateway guarantees that new minor versions of CPs are compatible with old minor versions of the DP,
so you can temporarily point DP nodes X to the new CP node Y.
This lets you pause the upgrade process if needed, or conduct it over a longer period of time.
This setup is meant to be temporary, to be used only during the upgrade process.
We do not recommend running a combination of new versions of CP nodes and old versions of DP nodes in a long-term production deployment.
After the CP upgrade, cluster X can be decommissioned. You can delay this task to the very end of the DP upgrade.
Once the CP nodes are upgraded, you can move on to upgrade the DP nodes.
The only supported upgrade strategy for DP upgrades is the rolling upgrade.
The following diagrams, figure 4 and 5, are the counterparts of figure 2 and 3 respectively.
flowchart TD
DBX[(Current database)]
DBY[(New database)]
CPX(Current Control Plane X)
CPY(New Control Plane Y)
DPX(Current Data Planes X)
DPY(New Data Planes Y)
API(API requests)
LB(Load balancer)
Admin(No admin write operations)
Admin2(No admin write operations)
subgraph A [ ]
Admin -.X.- CPX
DBX -.- CPX
DBY --- CPY
CPX -."Current DP connects to either CP X...".- DPX
Admin2 -.X.- CPY
CPY -."...OR to CP Y".- DPX
DPX -.90%..- LB
CPY --- DPY --10%---- LB
end
subgraph B [ ]
API --> LB & LB & LB
end
linkStyle 0,4 stroke:#d44324!important,color:#d44324!important
linkStyle 8,9 stroke:#b6d7a8!important
style CPX stroke-dasharray:3
style DPX stroke-dasharray:3
style DBX stroke-dasharray:3
style API stroke:none!important,fill:none!important
style A stroke:none!important,display:none!important
style B stroke:none!important,display:none!important
style Admin fill:none!important,stroke:none!important,color:#d44324!important
style Admin2 fill:none!important,stroke:none!important,color:#d44324!important
Figure 4: The diagram shows a DP upgrade using the dual-cluster and rolling strategies.The new CP Y is deployed alongside with the current CP X, while current DP nodes X are still serving API requests.In the image, the background color of the current database and CP X is grey instead of white, signaling that the old CP is already upgraded and might have been decommissioned.
flowchart
DBA[(Database)]
CPX(Current Control Plane X #40;inactive#41;)
CPY(New Control Plane Y)
DPX(Current Data Planes X)
DPY(New Data Planes Y)
API(API requests)
LB(Load balancer)
Admin(No admin write operations)
Admin2(No admin write operations)
subgraph A [ ]
Admin -.X.- CPX
DBA -.X.- CPX
DBA --- CPY
CPX -."Current DP connects to either CP X...".- DPX
Admin2 -.X.- CPY
CPY -."OR to CP Y".- DPX -.90%..- LB
CPY --- DPY --10%---- LB
end
subgraph B [ ]
API --> LB & LB & LB
end
linkStyle 0,1,4 stroke:#d44324!important,color:#d44324!important
linkStyle 8,9 stroke:#b6d7a8!important
style CPX stroke-dasharray:3,stroke:#c1c6cdff!important
style DPX stroke-dasharray:3
style A stroke:none!important,color:#fff!important
style B stroke:none!important,color:#fff!important
style Admin fill:none!important,stroke:none!important,color:#d44324!important
style Admin2 fill:none!important,stroke:none!important,color:#d44324!important
Figure 5: The diagram shows a DP upgrade using the in-place and rolling strategies.The diagram shows that the database is reused by the new CP Y, while current DP nodes X are still serving API requests.
The following tables categorize all relevant changelog entries from Kong Gateway Enterprise 2.8.0.0-2.8.4.1 up to 3.4.0.0.
Carefully review each entry and make changes to your configuration accordingly.
The following table lists new features introduced in a 3.x release prior or equal to 3.4.0.0.
These features may affect existing configurations.
Change
Category
Action Required
Request Validator plugin
The Request Validator plugin now allows requests carrying a
content-type with a parameter to match its content-type without a parameter.
Plugins
No
The Data Plane configuration cache was removed.
Configuration persistence is now done automatically with LMDB.
The Data Plane config cache mechanism and its related configuration options
(data_plane_config_cache_mode and data_plane_config_cache_path) have been removed in favor of LMDB.
DB config
Remove parameters from Kong Gateway configuration.
Bumped the version number of declarative configuration to 3.0 for changes on route.path.
Declarative configurations using older versions are upgraded to 3.0 during migrations.
DB config
If any configurations are stored in a repository (following a GitOps model),
these need to be upgraded using deck file convert.
DB config
No
The default value of lua_ssl_trusted_certificate has changed to system to automatically load the
trusted CA list from the system CA store.
kong.conf
If you don’t have this field explicitly configured, make sure that the new default value’s behavior
of automatically pulling in all certs on the server suits your deployment. Otherwise, adjust the setting.
Rate Limiting, Rate Limiting Advanced, and Response Rate Limiting plugins
The default policy for these plugins is now local for all deployment modes.
Plugins
No
Plugin batch queuing: HTTP Log, StatsD, OpenTelemetry, and Datadog plugins
The queuing system has been reworked, causing some plugin parameters to not function as expected anymore.
Plugins
If you use queues in these plugins, new parameters must be configured.
See each plugin’s documentation for details.
The module kong.tools.batch_queue has been renamed to kong.tools.batch and the API was changed.
If your custom plugin uses queues, it must be updated to use the new parameters.
Applies to 3.4.3.5 and later versions.
In OpenSSL 3.2, the default SSL/TLS security level has been changed from 1 to 2.
This means the security level is set to 112 bits of security and compression is disabled.
As a result, the following are prohibited:
RSA, DSA, and DH keys shorter than 2048 bits
ECC keys shorter than 224 bits
Any cipher suite using RC4
SSL version 3
OpenSSL
Ensure that your configuration complies with the new security level requirements.
The feature or behaviors in the following table have been permanently removed.
By updating settings based on the table below, you can avoid any potential
issues that may arise from using deprecated aliases and ensure
that your Kong Gateway instance functions correctly with the most recent changes and improvements.
It’s essential to keep configurations up to date to maintain the system’s stability,
security, and optimal performance.
Change
Category
Action Required
Deprecated and stopped producing Debian 8 (Jessie) and Debian 10 (Buster) containers and packages.
Deployment
Debian 11 and 12 are available. Upgrade to one of these versions before
upgrading to Kong Gateway 3.4.
Deprecated and stopped producing Amazon Linux 1 containers and packages.
Deployment
We recommend migrating to Amazon Linux 2 or another supported operating
system to continue receiving updates, security patches, and support from AWS.
Deprecated and stopped producing Alpine Linux images and packages.
The underlying operating system (OS) for our convenience Docker tags
(for example, latest, 3.4.0.0, 3.4) has changed from Alpine to Ubuntu.
Deployment
Review your Dockerfiles for OS-specific configuration and adjust as necessary.
If you are using one of the convenience images, adjust your configuration for Ubuntu.
Otherwise, switch to an image using a specific OS tag
(for example, 3.4.0.0-debian).
Deprecated and stopped producing Ubuntu 18.04 (Bionic) packages, as Standard Support for
Ubuntu 18.04 has ended as of June 2023.
Deployment
Ubuntu 20.04 and 22.04 are available. Upgrade to one of these versions before upgrading to Kong Gateway 3.4.
The deprecated shorthands field in Kong plugin or DAO schemas was removed
in favor of the typed shorthand_fields.
Plugins
If your custom schemas still use shorthands, you need to update them to
use shorthand_fields. This update is necessary to ensure
compatibility with the latest version of Kong Gateway.
Support for the legacy = true | false attribute was removed from Kong Gateway
schemas and Kong Gateway field schemas.
PDK
You can no longer use the legacy attribute in your schemas.
Any references to legacy=true | false in your custom schemas should be
updated to conform to the latest schema specifications without the legacy attribute.
The deprecated alias of Kong.serve_admin_api was removed.
Nginx templates
If your custom Nginx templates still use the alias, change it to Kong.admin_content.
The Kong Gateway singletons module kong.singletons was removed in favor of the PDK kong.*.
PDK
Remove the kong.singletons module and use the kong global variable instead.
For example: singletons.db.daos -> kong.db.daos
ngx.ctx.balancer_address was removed in favor of ngx.ctx.balancer_data.
Tracing
If you were previously using ngx.ctx.balancer_address, use ngx.ctx.balancer_data instead.
The normalization rules for route.path have changed.
Kong Gateway now stores the unnormalized path, but the regex path always pattern-matches
with the normalized URI.
Previously, Kong Gateway replaced percent-encoding in the regex path pattern to
ensure different forms of URI matches. That is no longer supported.
Write all characters without percent-encoding, except for the reserved characters defined in
RFC 3986,
Router
After the upgrade, if you configure Routes using the old method,
you will receive an alert and need to reconfigure the affected Routes using the new Route
configuration method.
Kong Gateway no longer uses a heuristic to guess whether a Route path is a regex pattern.
From 3.0 onward, all regex paths must start with the ~ prefix, and all paths that don’t start with ~ are
considered plain text.
Router
No manual migration required.
The migration process should automatically convert the regex paths when upgrading from 2.8 to 3.4.
Going forward, when writing a regex, it must start with the ~ symbol.
Support for the nginx-opentracing module is deprecated as of 3.0 and will be removed from Kong Gateway in 4.0.
Tracing
No
The Admin API endpoints /vitals/reports and /vitals/reports/:entity_type have been removed.
Admin API
After the upgrade, use one of the following endpoints from the Vitals API instead:
For /vitals/reports/consumer, use /<workspace_name>/vitals/status_codes/by_consumer instead
For /vitals/reports/service, use /<workspace_name>/vitals/status_codes/by_service instead
For /vitals/reports/hostname, use /<workspace_name>/vitals/nodes instead
POST requests on /targets endpoints are no longer able to update existing entities.
They are only able to create new ones.
Admin API
If you have scripts that use POST requests to modify /targets, change them to PUT requests to the appropriate
endpoints before updating to Kong Gateway 3.4.
Insert and update operations on duplicated targets return a 409 error.
Admin API
If you have duplicate targets, remove them.
The kong.request.get_path() PDK function now performs path normalization on the string that is returned to the caller.
The raw, non-normalized version of the request path can be fetched via kong.request.get_raw_path().
PDK
No
pdk.response.set_header(), pdk.response.set_headers(), and pdk.response.exit() now ignore and
emit warnings for manually set Transfer-Encoding headers.
PDK
No
The go_pluginserver_exe and go_plugins_dir directives are no longer supported.
Go PDK
If you are using Go plugin server, migrate your plugins to use the Go PDK before upgrading.
The Kong Gateway constant CREDENTIAL_USERNAME with the value of X-Credential-Username has been removed.
Kong plugins also don’t support this constant.
Plugins
Use the constant CREDENTIAL_IDENTIFIER (X-Credential-Identifier) when setting the upstream
headers for a credential.
It is no longer possible to use .lua format to import a declarative configuration file via
the Kong CLI tool. Only JSON and YAML formats are supported.
Declarative config
If your update with Kong Gateway involves executing kong config db_import config.lua,
convert the config.lua file into a config.json or config.yml file before upgrading.
DAOs in plugins must be listed in an array, so that their loading order is explicit.
Loading them in a hash-like table is no longer supported.
Plugins
Review your custom plugins. If you have plugins that use hash-like tables for listing DAOs, convert them into arrays.
Plugins now must have a valid PRIORITY (integer) and VERSION (x.y.z format) field in their handler.lua
file, otherwise the plugin will fail to load.
Plugins
Review your custom plugins. Add a PRIORITY and VERSION in their respective formats to all of your custom plugins.
The kong.plugins.log-serializers.basic library was removed in favor of the PDK function kong.log.serialize.
PDK
Upgrade your plugins to use the new PDK function.
The support for deprecated legacy plugin schemas was removed.
Plugins
If your custom plugins still use the old (0.x era) schemas, you must upgrade them.
Updated the priority for some plugins.
This is important for those who run custom plugins as it may affect the sequence in which your plugins are executed.
This does not change the order of execution for plugins in a standard Kong Gateway installation.
Old and new plugin priority values:
acme changed from 1007 to 1705
basic-auth changed from 1001 to 1100
canary changed from 13 to 20
degraphql changed from 1005 to 1500
graphql-proxy-cache-advanced changed from 100 to 99
hmac-auth changed from 1000 to 1030
jwt changed from 1005 to 1450
jwt-signer changed from 999 to 1020.
key-auth changed from 1003 to 1250
key-auth-advanced changed from 1003 to 1250
ldap-auth changed from 1002 to 1200
ldap-auth-advanced changed from 1002 to 1200
mtls-auth changed from 1006 to 1600
oauth2 changed from 1004 to 1400
openid-connect changed from 1000 to 1050
rate-limiting changed from 901 to 910
rate-limiting-advanced changed from 902 to 910
route-by-header changed from 2000 to 850
route-transformer-advanced changed from 800 to 780
pre-function changed from +inf to 1000000
vault-auth changed from 1003 to 1350
Plugins
Review custom plugin priorities.
If any of the changes in priorities break expected behaviour, adjust as necessary.
JWT plugin
The authenticated JWT is no longer put into the Nginx context (ngx.ctx.authenticated_jwt_token).
Plugins
Custom plugins which depend on that value being set under that name must be updated to use
Kong Gateway’s shared context instead (kong.ctx.shared.authenticated_jwt_token) before upgrading to 3.4.
JWT plugin
The JWT plugin now denies any request that has different tokens in the JWT token search locations.
Plugins
No
StatsD plugin
Any metric name that is related to a Gateway Service now has a service. prefix:
kong.service.<service_identifier>.request.count.
The metric kong.<service_identifier>.request.status.<status> has been renamed to
kong.service.<service_identifier>.status.<status>.
The metric kong.<service_identifier>.user.<consumer_identifier>.request.status.<status> has been renamed to
kong.service.<service_identifier>.user.<consumer_identifier>.status.<status>.
The metric *.status.<status>.total from metrics status_count and status_count_per_user has been removed.
These plugins don’t store response data in ngx.ctx.proxy_cache_hit anymore.
They store it in kong.ctx.shared.proxy_cache_hit.
Plugins
Logging plugins that need the response data must now read it from kong.ctx.shared.proxy_cache_hit.
Added the allow_debug_header configuration property to kong.conf to constrain the Kong-Debug header for debugging.
This option defaults to off.
DB config
If you were previously relying on the Kong-Debug header to provide debugging information,
set allow_debug_header=on to continue doing so.
If you’re using Response Transformer plugin as a workaround to remove headers,
you no longer need the plugin. Disable and remove it.
The lua-resty-session library has been upgraded to v4.0.0. This version includes a full
rewrite of the session library, and is not backwards compatible.
This library is used by the following plugins: Session, OpenID Connect, and SAML.
Many session parameters used by these plugins have been renamed or removed.
This also affects any session configuration that uses the Session or OpenID Connect
plugin in the background, including sessions for Kong Manager and Dev Portal.
Plugins
All existing sessions are invalidated when upgrading to this version.
For sessions to work as expected in this version, all nodes must run Kong Gateway 3.4.x.
For that reason, we recommend that during upgrades, proxy nodes with mixed versions run for as little time as possible.
During that time, the invalid sessions could cause failures and partial downtime.
You can expect the following behavior:
After upgrading the Control Plane: Existing Kong Manager and Dev Portal sessions will be
invalidated and all users will be required to log back in.
After upgrading the Data Planes: Existing proxy sessions will be invalidated.
If you have an IdP configured, users will be required to log back into the IdP.
After you have upgraded all of your CP and DP nodes to 3.4 and ensured that your environment is stable,
you can update parameters to their new renamed versions. Although your configuration will continue to work as
previously configured, we recommend adjusting your configuration to avoid future unexpected behavior.
See the breaking changes doc for all session configuration
changes and guidance on how to convert your existing session configuration.
Cassandra DB support has been removed. It is no longer supported as a data store for Kong Gateway.
DB config
Reach out to Kong Support for help with migrating Cassandra to PostgreSQL.
The following table lists behavior changes that may cause your database configuration
or kong.conf to fail.
This includes deprecated (but not removed) features.
Change
Category
Action Required
Regex look-around and backreference support in the atc-router have been removed.
These are rarely used features and removing support for them improves the speed of our regex matching.
If your current regexes use look-around or backreferences, you will receive an error when attempting to start Kong Gateway,
showing exactly which regex is incompatible.
Router
To ensure consistency of behavior after the upgrade, set router_flavor = traditional,
or change the regex to remove look-around or backreferences.
Review the regex paths and ensure these removed features are not in use.
Update the regex router accordingly.
ACL, Bot Detection, and IP Restriction plugins
Removed the deprecated config.blacklist and config.whitelist configuration parameters.
Plugins
Remove the deprecated config.blacklist and config.whitelist configuration parameters.
Use config.denylist and config.allowlist instead.
ACME plugin
The default value of the config.auth_method configuration parameter is now token.
Plugins
Following the upgrade, the config.auth_method will persist as null
(if this configuration is not used), which deviates from the new version’s default
setting of token. Update the value if necessary.
AWS Lambda plugin
The AWS region is now required. You can set it through the plugin configuration with the config.aws_region field parameter, or with environment variables.
The plugin now allows host and config.aws_region fields to be set at the same time, and always applies the SigV4 signature.
Plugins
Review your configuration or consult with the Kong Support team.
HTTP Log plugin
The config.headers field now only takes a single string per header name,
where it previously took an array of values.
Plugins
Review your configuration or consult with the Kong Support team.
Prometheus plugin
Complete overhaul of the plugin:
High cardinality metrics are now disabled by default.
Decreased performance penalty to proxy traffic when collecting metrics.
The following metric names were adjusted to add units to standardize where possible:
http_status to http_requests_total.
latency to kong_request_latency_ms (HTTP), kong_upstream_latency_ms, kong_kong_latency_ms, and session_duration_ms (stream).
Kong Gateway latency and upstream latency can operate at orders of different magnitudes. Separate these buckets to reduce memory overhead.
kong_bandwidth to kong_bandwidth_bytes.
nginx_http_current_connections and nginx_stream_current_connections were merged into to nginx_hconnections_total
request_count and consumer_status were merged into http_requests_total.
If the per_consumer config is set to false, the consumer label will be empty. If the per_consumer config is true, the consumer label will be filled.
Removed the following metric: http_consumer_status
Constrained the valid values of this configuration option to only accept the following: tlsv1_1, tlsv1_2, tlsv1_3 or any.
This mirrors the setting ssl_min_protocol_version in PostgreSQL 12.x and onward.
See the PostgreSQL documentation
for more information about that parameter.
DB config
Change the value in your kong.conf or environment variables from tlsv1_0 to tlsv1_2.
To use the default setting in kong.conf, verify that your PostgreSQL server supports TLS 1.2 or higher versions, or set the TLS version yourself.
TLS versions lower than tlsv1_2 are already deprecated and considered insecure from PostgreSQL 12.x onward.
The /consumer_groups/:id/overrides endpoint is deprecated in favor of a more generic plugin scoping mechanism.
Admin API
Instead of setting overrides, you can apply a plugin instance to a Consumer Group entity. See the
Rate Limiting Advanced
documentation for an example.
The admin_api_uri property is now deprecated and will be fully removed in a future version of Kong Gateway.
Admin API
Rename the configuration property admin_api_uri to admin_gui_api_url.
Upgrades using the following features might require help from Kong.
Kong’s support team provides advanced features and professional services tailored to meet
specific business needs, including data migration, custom plugin integration,
and seamless coordination with existing settings PDK.
Change
Category
Action Required
As of 3.0, Kong Gateway’s schema library’s process_auto_fields function doesn’t make deep copies of data that is passed to it when the given context is select.
This was done to avoid excessive deep copying of tables where we believe the data most of the time comes from a driver like pgmoon or lmdb.
If a custom plugin relied on process_auto_fields not overriding the given table, it must make its own copy before passing it to the function.
Plugins
If a custom plugin needs to fetch data by deep copying, perform it on the select context before calling the function.
Define context = “select” to trigger deep copying:
The following table lists changes that you should be aware of, but require no action.
Change
Category
Action Required
The PDK is no longer versioned.
PDK
No
The migration helper library (mostly used for Cassandra migrations) is no longer supplied with Kong Gateway.
DB config
No
PostgreSQL migrations can now have an up_f part like Cassandra migrations, designating a function to call.
The up_f part is invoked after the up part has been executed against the database.
DB config
No
Kong plugins are no longer individually versioned.
Starting in 3.0.0, every plugin’s version has been updated to match Kong Gateway’s version.
Plugins
No
HTTP Log plugin
If the log server responds with a 3xx HTTP status code, the plugin now considers it to be
an error and retries according to the retry configuration. Previously, 3xx status codes would be interpreted
as a success, causing the log entries to be dropped.
Plugins
No
Serverless Functions (post-function or pre-function)
kong.cache now points to a cache instance that is
dedicated to the Serverless Functions plugins. It does not provide access to the global Kong Gateway cache.
Access to certain fields in kong.conf has also been restricted.
Plugins
No
Zipkin plugin
This plugin now uses queues for internal buffering.
The standard queue parameter set is available to control queuing behavior.
The following table lists changes to parameters managed in the kong.conf configuration file:
2.8
3.4
data_plane_config_cache_mode = unencrypted
Removed in 3.4
data_plane_config_cache_path
Removed in 3.4
admin_api_uri
Deprecated. Use admin_gui_api_url instead.
database Cassandra support
Accepted values are postgres and off. All Cassandra options have been removed.
pg_keepalive_timeout = 60000
You can now specify the maximal idle timeout (in ms) for the Postgres connections in the pool.
If this value is set to 0 then the timeout interval is unlimited.
If not specified, this value will be the same as lua_socket_keepalive_timeout.
Now that you have chosen an upgrade strategy, reviewed all the relevant changes between the 2.8 and 3.4 LTS releases
you can move on to performing the upgrade with your chosen strategy:
Traditional mode or control planes in hybrid mode: