Kong Enterprise LTS 2.8 to 3.4 upgrade guide

Request Validator plugin

The Request Validator plugin now allows requests carrying a content-type with a parameter to match its content-type without a parameter.

Plugins

No

The data plane configuration cache was removed. Configuration persistence is now done automatically with LMDB.

The data plane config cache mechanism and its related configuration options (data_plane_config_cache_mode and data_plane_config_cache_path) have been removed in favor of LMDB.

DB config

Remove parameters from Kong configuration.

Bumped the version number of declarative configuration to 3.0 for changes on route.path.

Declarative configurations using older versions are upgraded to 3.0 during migrations.

DB config

If any configurations are stored in a repository (following a GitOps model), these need to be upgraded using deck convert.

Tags may now contain space characters.

DB config

No

The default value of lua_ssl_trusted_certificate has changed to system to automatically load the trusted CA list from the system CA store.

kong.conf

If you don’t have this field explicitly configured, make sure that the new default value’s behavior of automatically pulling in all certs on the server suits your deployment. Otherwise, adjust the setting.

Rate Limiting, Rate Limiting Advanced, and Response Rate Limiting plugins

The default policy for these plugins is now local for all deployment modes.

Plugins

No

Plugin batch queuing: HTTP Log, StatsD, OpenTelemetry, and Datadog plugins

The queuing system has been reworked, causing some plugin parameters to not function as expected anymore.

Plugins

If you use queues in these plugins, new parameters must be configured. See each plugin’s documentation for details.

The module kong.tools.batch_queue has been renamed to kong.tools.batch and the API was changed. If your custom plugin uses queues, it must be updated to use the new parameters.

Applies to 3.4.3.5 and later versions.

In OpenSSL 3.2, the default SSL/TLS security level has been changed from 1 to 2.

This means the security level is set to 112 bits of security. As a result, the following are prohibited:

• RSA, DSA, and DH keys shorter than 2048 bits
• ECC keys shorter than 224 bits
• Any cipher suite using RC4
• SSL version 3 Additionally, compression is disabled.

OpenSSL

Ensure that your configuration complies with the new security level requirements.

Deprecated and stopped producing Debian 8 (Jessie) and Debian 10 (Buster) containers and packages.

Deployment

Debian 11 and 12 are available. Upgrade to one of these versions before upgrading to Kong 3.4.

Deprecated and stopped producing Amazon Linux 1 containers and packages.

Deployment

We recommend migrating to Amazon Linux 2 or another supported operating system to continue receiving updates, security patches, and support from AWS.

Deprecated and stopped producing Alpine Linux images and packages.

The underlying operating system (OS) for our convenience Docker tags (for example, latest, 3.4.0.0, 3.4) has changed from Alpine to Ubuntu.

Deployment

Review your Dockerfiles for OS-specific configuration and adjust as necessary.

If you are using one of the convenience images, adjust your configuration for Ubuntu. Otherwise, switch to an image using a specific OS tag (for example, 3.4.0.0-debian).

Deprecated and stopped producing Ubuntu 18.04 (“Bionic”) packages, as Standard Support for Ubuntu 18.04 has ended as of June 2023.

Deployment

Ubuntu 20.04 and 22.04 are available. Upgrade to one of these versions before upgrading to Kong 3.4.

The deprecated shorthands field in Kong plugin or DAO schemas was removed in favor of the typed shorthand_fields.

Plugins

If your custom schemas still use shorthands, you need to update them to use shorthand_fields. This update is necessary to ensure compatibility with the latest version of Kong.

Support for the legacy = true | false attribute was removed from Kong schemas and Kong field schemas.

You can no longer use the legacy attribute in your schemas.

Any references to legacy=true | false in your custom schemas should be updated to conform to the latest schema specifications without the legacy attribute.

The deprecated alias of Kong.serve_admin_api was removed.

Nginx templates

If your custom Nginx templates still use the alias, change it to Kong.admin_content.

The Kong singletons module kong.singletons was removed in favor of the PDK kong.*.

PDK

Remove the kong.singletons module and use the kong global variable instead.

For example: singletons.db.daos -> kong.db.daos

ngx.ctx.balancer_address was removed in favor of ngx.ctx.balancer_data.

Tracing

If you were previously using ngx.ctx.balancer_address, use ngx.ctx.balancer_data instead.

The normalization rules for route.path have changed. Kong Gateway now stores the unnormalized path, but the regex path always pattern-matches with the normalized URI.

Previously, Kong Gateway replaced percent-encoding in the regex path pattern to ensure different forms of URI matches. That is no longer supported. Except for the reserved characters defined in RFC 3986, write all other characters without percent-encoding.

Router

After the upgrade, if you configure routes using the old method, you will receive an alert and need to reconfigure the affected routes using the new route configuration method.

Kong Gateway no longer uses a heuristic to guess whether a route path is a regex pattern. From 3.0 onward, all regex paths must start with the ~ prefix, and all paths that don’t start with ~ are considered plain text.

Router

No manual migration required.
The migration process should automatically convert the regex paths when upgrading from 2.8 to 3.4.

Going forward, when writing a regex, it must start with the ~ symbol.

Support for the nginx-opentracing module is deprecated as of 3.0 and will be removed from Kong in 4.0.

No

The Admin API endpoints /vitals/reports and /vitals/reports/:entity_type have been removed.

Admin API

After the upgrade, use one of the following endpoints from the Vitals API instead:

• For /vitals/reports/consumer, use /<workspace_name>/vitals/status_codes/by_consumer instead
• For /vitals/reports/service, use /<workspace_name>/vitals/status_codes/by_service instead
• For /vitals/reports/hostname, use /<workspace_name>/vitals/nodes instead

POST requests on /targets endpoints are no longer able to update existing entities. They are only able to create new ones.

Admin API

If you have scripts that use POST requests to modify /targets, change them to PUT requests to the appropriate endpoints before updating to Kong Gateway 3.4.

The kong.request.get_path() PDK function now performs path normalization on the string that is returned to the caller. The raw, non-normalized version of the request path can be fetched via kong.request.get_raw_path().

PDK

No

pdk.response.set_header(), pdk.response.set_headers(), and pdk.response.exit() now ignore and emit warnings for manually set Transfer-Encoding headers.

PDK

No

The go_pluginserver_exe and go_plugins_dir directives are no longer supported.

Go PDK

If you are using Go plugin server, migrate your plugins to use the Go PDK before upgrading.

The Kong constant CREDENTIAL_USERNAME with the value of X-Credential-Username has been removed.

Kong plugins also don’t support this constant.

Plugins

Use the constant CREDENTIAL_IDENTIFIER (X-Credential-Identifier) when setting the upstream headers for a credential.

It is no longer possible to use .lua format to import a declarative configuration file via the Kong CLI tool. Only JSON and YAML formats are supported.

Declarative config

If your update procedure with Kong Gateway involves executing kong config db_import config.lua, convert the config.lua file into a config.json or config.yml file before upgrading.

DAOs in plugins must be listed in an array, so that their loading order is explicit. Loading them in a hash-like table is no longer supported.

Plugins

Review your custom plugins. If you have plugins that use hash-like tables for listing DAOs, convert them into arrays.

Plugins now must have a valid PRIORITY (integer) and VERSION (x.y.z format) field in their handler.lua file, otherwise the plugin will fail to load.

Plugins

Review your custom plugins. Add a PRIORITY and VERSION in their respective formats to all of your custom plugins.

The kong.plugins.log-serializers.basic library was removed in favor of the PDK function kong.log.serialize.

PDK

Upgrade your plugins to use the new PDK function.

The support for deprecated legacy plugin schemas was removed.

Plugins

If your custom plugins still use the old (0.x era) schemas, you must upgrade them.

Updated the priority for some plugins.

This is important for those who run custom plugins as it may affect the sequence in which your plugins are executed. This does not change the order of execution for plugins in a standard Kong Gateway installation.

Old and new plugin priority values:

• acme changed from 1007 to 1705
• basic-auth changed from 1001 to 1100
• canary changed from 13 to 20
• degraphql changed from 1005 to 1500
• graphql-proxy-cache-advanced changed from 100 to 99
• hmac-auth changed from 1000 to 1030
• jwt changed from 1005 to 1450
• jwt-signer changed from 999 to 1020.
• key-auth changed from 1003 to 1250
• key-auth-advanced changed from 1003 to 1250
• ldap-auth changed from 1002 to 1200
• ldap-auth-advanced changed from 1002 to 1200
• mtls-auth changed from 1006 to 1600
• oauth2 changed from 1004 to 1400
• openid-connect changed from 1000 to 1050
• rate-limiting changed from 901 to 910
• rate-limiting-advanced changed from 902 to 910
• route-by-header changed from 2000 to 850
• route-transformer-advanced changed from 800 to 780
• pre-function changed from +inf to 1000000
• vault-auth changed from 1003 to 1350

Plugins

Review custom plugin priorities. If any of the changes in priorities break expected behaviour, adjust as necessary.

JWT plugin

The authenticated JWT is no longer put into the Nginx context (ngx.ctx.authenticated_jwt_token).

Plugins

Custom plugins which depend on that value being set under that name must be updated to use Kong’s shared context instead (kong.ctx.shared.authenticated_jwt_token) before upgrading to 3.4.

JWT plugin (jwt)

The JWT plugin now denies any request that has different tokens in the JWT token search locations.

Plugins

No

** StatsD plugin**

Any metric name that is related to a service now has a service. prefix:

kong.service.<service_identifier>.request.count.

• The metric kong.<service_identifier>.request.status.<status> has been renamed to kong.service.<service_identifier>.status.<status>.
• The metric kong.<service_identifier>.user.<consumer_identifier>.request.status.<status> has been renamed to kong.service.<service_identifier>.user.<consumer_identifier>.status.<status>.
• The metric *.status.<status>.total from metrics status_count and status_count_per_user has been removed.

Plugins

No

Proxy Cache, Proxy Cache Advanced, and GraphQL Proxy Cache Advanced plugins

These plugins don’t store response data in ngx.ctx.proxy_cache_hit anymore. They store it in kong.ctx.shared.proxy_cache_hit.

Plugins

Logging plugins that need the response data must now read it from kong.ctx.shared.proxy_cache_hit.

Added the allow_debug_header configuration property to kong.conf to constrain the Kong-Debug header for debugging. This option defaults to off.

DB config

If you were previously relying on the Kong-Debug header to provide debugging information, set allow_debug_header=on to continue doing so.

If you’re using Response Transformer plugin as a workaround to remove headers, you no longer need the plugin. Disable and remove it.

The lua-resty-session library has been upgraded to v4.0.0. This version includes a full rewrite of the session library, and is not backwards compatible.

This library is used by the following plugins: Session, OpenID Connect, and SAML. Many session parameters used by these plugins have been renamed or removed.

This also affects any session configuration that uses the Session or OpenID Connect plugin in the background, including sessions for Kong Manager and Dev Portal.

Plugins

All existing sessions are invalidated when upgrading to this version. For sessions to work as expected in this version, all nodes must run Kong Gateway 3.4.x.

For that reason, we recommend that during upgrades, proxy nodes with mixed versions run for as little time as possible. During that time, the invalid sessions could cause failures and partial downtime.

You can expect the following behavior:

• After upgrading the control plane: Existing Kong Manager and Dev Portal sessions will be invalidated and all users will be required to log back in.
• After upgrading the data planes: Existing proxy sessions will be invalidated. If you have an IdP configured, users will be required to log back into the IdP.

After you have upgraded all of your CP and DP nodes to 3.4 and ensured that your environment is stable, you can update parameters to their new renamed versions. Although your configuration will continue to work as previously configured, we recommend adjusting your configuration to avoid future unexpected behavior.

See the breaking changes doc for all session configuration changes and guidance on how to convert your existing session configuration.

Cassandra DB support has been removed. It is no longer supported as a data store for Kong Gateway.

DB config

Migrate to PostgreSQL.

Regex look-around and backreferences support in the atc-router have been removed. These are rarely used features and removing support for them improves the speed of our regex matching.

If your current regexes use look-around or backreferences you will receive an error when attempting to start Kong, showing exactly what regex is incompatible.

Router

To ensure consistency of behavior after the upgrade, set router_flavor = traditional, or change the regex to remove look-around or backreferences.

Review the regex paths and ensure these removed features are not in use. Update the regex router accordingly.

ACL, Bot Detection, and IP Restriction plugins

Removed the deprecated blacklist and whitelist configuration parameters.

Plugins

Remove the deprecated blacklist and whitelist configuration parameters. Use denylist and allowlist instead.

ACME plugin

The default value of the auth_method configuration parameter is now token.

Plugins

Caution: Following the upgrade, the auth_method will persist as null (if this configuration is not utilized), which deviates from the new version’s default setting of token.

AWS Lambda plugin

• The AWS region is now required. You can set it through the plugin configuration with the aws_region field parameter, or with environment variables.
• The plugin now allows host and aws_region fields to be set at the same time, and always applies the SigV4 signature.

Plugins

Review your configuration or consult with the Kong Support team.

HTTP Log (http-log) plugin

The headers field now only takes a single string per header name, where it previously took an array of values.

Plugins

Review your configuration or consult with the Kong Support team.

Prometheus plugin

Complete overhaul of the plugin:

• High cardinality metrics are now disabled by default.
• Decreased performance penalty to proxy traffic when collecting metrics.
• The following metric names were adjusted to add units to standardize where possible:
  • http_status to http_requests_total.

  • latency to kong_request_latency_ms (HTTP), kong_upstream_latency_ms, kong_kong_latency_ms, and session_duration_ms (stream).

    Kong latency and upstream latency can operate at orders of different magnitudes. Separate these buckets to reduce memory overhead.

  • kong_bandwidth to kong_bandwidth_bytes.
  • nginx_http_current_connections and nginx_stream_current_connections were merged into to nginx_hconnections_total

  • request_count and consumer_status were merged into http_requests_total.

    If the per_consumer config is set to false, the consumer label will be empty. If the per_consumer config is true, the consumer label will be filled.

• Removed the following metric: http_consumer_status
• New metrics:
  • session_duration_ms: monitoring stream connections.
  • node_info: Single gauge set to 1 that outputs the node’s ID and Kong Gateway version.
• http_requests_total has a new label, source. It can be set to exit, error, or service.
• All memory metrics have a new label: node_id.
• Updated the Grafana dashboard that comes packaged with Kong

Plugins

To make sure you don’t miss any data, adjust your plugin configuration to use the new settings.

If you have any custom dashboards or have written any custom PromQL, review them and ensure the name changes haven’t broken anything.

StatsD Advanced plugin

The StatsD Advanced plugin has been deprecated and will be removed in 4.0. All capabilities are now available in the StatsD plugin.

Plugins

We recommend migrating over to the StatsD plugin, however StatsD Advanced continues to function in 3.4.

Serverless Functions (post-function or pre-function)

Removed the deprecated config.functions configuration parameter from the Serverless Functions plugins’ schemas.

Plugins

Use the config.access phase instead.

The default PostgreSQL SSL version has been bumped to TLS 1.2. In kong.conf:

• The default pg_ssl_version is now tlsv1_2.
• Constrained the valid values of this configuration option to only accept the following: tlsv1_1, tlsv1_2, tlsv1_3 or any.

This mirrors the setting ssl_min_protocol_version in PostgreSQL 12.x and onward. See the PostgreSQL documentation for more information about that parameter.

DB config

Change the value in your kong.conf or environment variables from tlsv1_0 to tlsv1_2.

To use the default setting in kong.conf, verify that your Postgres server supports TLS 1.2 or higher versions, or set the TLS version yourself. TLS versions lower than tlsv1_2 are already deprecated and considered insecure from PostgreSQL 12.x onward.

The /consumer_groups/:id/overrides endpoint is deprecated in favor of a more generic plugin scoping mechanism.

Admin API

Instead of setting overrides, you can apply a plugin instance to a consumer group entity. See the Rate Limiting Advanced documentation for an example.

The admin_api_uri property is now deprecated and will be fully removed in a future version of Kong Gateway.

Admin API

Rename the configuration property admin_api_uri to admin_gui_api_url.

As of 3.0, Kong Gateway’s schema library’s process_auto_fields function will not make deep copies of data that is passed to it when the given context is select. This was done to avoid excessive deep copying of tables where we believe the data most of the time comes from a driver like pgmoon or lmdb.

If a custom plugin relied on process_auto_fields not overriding the given table, it must make its own copy before passing it to the function now.

Plugins

If a custom plugin needs to fetch data by deep copying, perform it on the select context before calling the function.

Define context = “select” to trigger deep copying:

local is_select = context == "select"
  if not is_select then
    data = tablex.deepcopy(data)
  end

The list of reported plugins available on the server now returns a table of metadata per plugin instead of the boolean true.

For example:

plugins": {
  "available_on_server": {
      "acl": "3.0.1",
      "acme": "0.4.0",
      "aws-lambda": "3.6.3",
      "azure-functions": "1.0.1",
      "basic-auth": "2.2.0",
      "bot-detection": "2.0.0",
  ...

Plugins

Adapt to use the new metadata instead of boolean values.

The PDK is no longer versioned.

PDK

The migration helper library (mostly used for Cassandra migrations) is no longer supplied with Kong Gateway.

DB config

PostgreSQL migrations can now have an up_f part like Cassandra migrations, designating a function to call. The up_f part is invoked after the up part has been executed against the database.

DB config

Kong Plugins are no longer individually versioned. Starting in 3.0.0, every Kong plugin’s version has been updated to match Kong Gateway’s version.

Plugins

HTTP Log plugin

If the log server responds with a 3xx HTTP status code, the plugin now considers it to be an error and retries according to the retry configuration. Previously, 3xx status codes would be interpreted as a success, causing the log entries to be dropped.

Plugins

Serverless Functions (post-function or pre-function)

kong.cache now points to a cache instance that is dedicated to the Serverless Functions plugins. It does not provide access to the global Kong Gateway cache. Access to certain fields in kong.conf has also been restricted.

Plugins

Zipkin plugin

This plugin now uses queues for internal buffering. The standard queue parameter set is available to control queuing behavior.

Plugins