Kong Gateway 3.0.x breaking changes

Before upgrading, review any configuration or breaking changes in this version and prior versions that affect your current installation.

You may need to adopt different upgrade paths depending on your deployment methods, set of features in use, custom plugins, for example.

Plugins

For breaking changes to plugins, see the Kong Gateway Changelog for your Kong Gateway version.

Kong plugins

If you are adding a new plugin to your installation, you need to run kong migrations up with the plugin name specified. For example, KONG_PLUGINS=tls-handshake-modifier.

The 3.0 release includes the following new plugins:

• OpenTelemetry (opentelemetry)
• TLS Handshake Modifier (tls-handshake-modifier)
• TLS Metadata Headers (tls-metadata-headers)
• WebSocket Size Limit (websocket-size-limit)
• WebSocket Validator (websocket-validator)

Kong plugins no longer support CREDENTIAL_USERNAME (X-Credential-Username). Use the constant CREDENTIAL_IDENTIFIER (X-Credential-Identifier) when setting the upstream headers for a credential.

Deployment

Amazon Linux 1 and Debian 8 (Jessie) containers and packages are deprecated and are no longer produced for new versions of Kong Gateway.

Blue-green deployments

Traditional mode: Blue-green upgrades from versions of 2.8.1 and below to 3.0.0 are not currently supported. This is a known issue planned to be fixed in the next 2.8 release. When that version is released, 2.x users should upgrade to that version before beginning a blue-green upgrade to 3.0.

Hybrid mode: See the upgrade instructions below.

Dependencies

If you are using the provided binary packages (except Debian and RHEL), all necessary dependencies for the gateway are bundled and you can skip this section.

As of Kong Gateway 3.0, Debian and RHEL images are built with minimal dependencies and run through automated security scanners before being published. They only contain the bare minimum required to run Kong Gateway. If you would like further customize the base image and any dependencies, you can build your own Docker images.

If you are using Debian, RHEL, or building your dependencies by hand, there are changes since the previous release, so you will need to rebuild them with the latest patches.

The required OpenResty version for Kong Gateway 3.0.x is 1.21.4.1. In addition to an upgraded OpenResty, you need the correct OpenResty patches for this new version, including the latest release of lua-kong-nginx-module. The kong-build-tools repository contains openresty-build-tools, which allows you to more easily build OpenResty with the necessary patches and modules.

Migrations

The migration helper library (mostly used for Cassandra migrations) is no longer supplied with Kong Gateway.

PostgreSQL migrations can now have an up_f part like Cassandra migrations, designating a function to call. The up_f part is invoked after the up part has been executed against the database for both PostgreSQL and Cassandra.

Deprecations and changed parameters

The StatsD Advanced plugin has been deprecated and will be removed in 4.0. All capabilities are now available in the StatsD plugin.

The following plugins have had configuration parameters changed or removed. You will need to carefully review and update your configuration as needed:

ACL, Bot Detection, and IP Restriction

• Removed the deprecated blacklist and whitelist configuration parameters. Use allow or deny instead.

ACME

• The default value of the auth_method configuration parameter is now token.

AWS Lambda

• The AWS region is now required. You can set it through the plugin configuration with the aws_region field parameter, or with environment variables.
• The plugin now allows host and aws_region fields to be set at the same time, and always applies the SigV4 signature.

HTTP Log

• The headers field now only takes a single string per header name, where it previously took an array of values.

JWT

• The authenticated JWT is no longer put into the nginx context (ngx.ctx.authenticated_jwt_token). Custom plugins which depend on that value being set under that name must be updated to use Kong’s shared context instead (kong.ctx.shared.authenticated_jwt_token) before upgrading to 3.0.

Prometheus

• High cardinality metrics are now disabled by default.

• The following metric names were adjusted to add units to standardize where possible:
  • http_status to http_requests_total.
  • latency to kong_request_latency_ms (HTTP), kong_upstream_latency_ms, kong_kong_latency_ms, and session_duration_ms (stream). Kong latency and upstream latency can operate at orders of different magnitudes. Separate these buckets to reduce memory overhead.
  • kong_bandwidth to kong_bandwidth_bytes.
  • nginx_http_current_connections and nginx_stream_current_connections were merged into nginx_connections_total.
  • request_count and consumer_status were merged into http_requests_total. If the per_consumer config is set to false, the consumer label will be empty. If the per_consumer config is true, the consumer label will be filled.
• Other metric changes:
  • Removed the following metric: http_consumer_status.
  • http_requests_total has a new label, source. It can be set to exit, error, or service.
  • All memory metrics have a new label: node_id.
  • The plugin doesn’t export status codes, latencies, bandwidth and upstream health check metrics by default. They can still be turned on manually by setting status_code_metrics, lantency_metrics, bandwidth_metrics and upstream_health_metrics respectively.

Pre-function and Post-function plugins

• Removed the deprecated config.functions configuration parameter from the post-function and pre-function plugins’ schemas. Use the config.access phase instead.

StatsD

• Any metric name that is related to a service now has a service. prefix: kong.service.<service_identifier>.request.count.
  • The metric kong.<service_identifier>.request.status.<status> has been renamed to kong.service.<service_identifier>.status.<status>.
  • The metric kong.<service_identifier>.user.<consumer_identifier>.request.status.<status> has been renamed to kong.service.<service_identifier>.user.<consumer_identifier>.status.<status>.
• The metric *.status.<status>.total from metrics status_count and status_count_per_user has been removed.

Proxy Cache, Proxy Cache Advanced, and GraphQL Proxy Cache Advanced

• These plugins don’t store response data in ngx.ctx.proxy_cache_hit anymore.
• Logging plugins that need the response data must now read it from kong.ctx.shared.proxy_cache_hit.

Custom plugins and the PDK

• DAOs in plugins must be listed in an array, so that their loading order is explicit. Loading them in a hash-like table is no longer supported.
• Plugins MUST now have a valid PRIORITY (integer) and VERSION (“x.y.z” format) field in their handler.lua file, otherwise the plugin will fail to load.
• The old kong.plugins.log-serializers.basic library was removed in favor of the PDK function kong.log.serialize. Upgrade your plugins to use the PDK.

• The support for deprecated legacy plugin schemas was removed. If your custom plugins still use the old (0.x era) schemas, you are now forced to upgrade them.

• Updated the priority for some plugins.

  This is important for those who run custom plugins as it may affect the sequence in which your plugins are executed. This does not change the order of execution for plugins in a standard Kong Gateway installation.

  Old and new plugin priority values:

  • acme changed from 1007 to 1705
  • basic-auth changed from 1001 to 1100
  • canary changed from 13 to 20
  • degraphql changed from 1005 to 1500
  • graphql-proxy-cache-advanced changed from 100 to 99
  • hmac-auth changed from 1000 to 1030
  • jwt changed from 1005 to 1450
  • jwt-signer changed from 999 to 1020.
  • key-auth changed from 1003 to 1250
  • key-auth-advanced changed from 1003 to 1250
  • ldap-auth changed from 1002 to 1200
  • ldap-auth-advanced changed from 1002 to 1200
  • mtls-auth changed from 1006 to 1600
  • oauth2 changed from 1004 to 1400
  • openid-connect changed from 1000 to 1050
  • rate-limiting changed from 901 to 910
  • rate-limiting-advanced changed from 902 to 910
  • route-by-header changed from 2000 to 850
  • route-transformer-advanced changed from 800 to 780
  • pre-function changed from +inf to 1000000
  • vault-auth change from 1003 to 1350

• The kong.request.get_path() PDK function now performs path normalization on the string that is returned to the caller. The raw, non-normalized version of the request path can be fetched via kong.request.get_raw_path().

• pdk.response.set_header(), pdk.response.set_headers(), pdk.response.exit() now ignore and emit warnings for manually set Transfer-Encoding headers.

• The PDK is no longer versioned.

• The JavaScript PDK now returns Uint8Array for kong.request.getRawBody, kong.response.getRawBody, and kong.service.response.getRawBody. The Python PDK returns bytes for kong.request.get_raw_body, kong.response.get_raw_body, and kong.service.response.get_raw_body. Previously, these functions returned strings.

• The go_pluginserver_exe and go_plugins_dir directives are no longer supported. If you are using Go plugin server, migrate your plugins to use the Go PDK before upgrading.

• As of 3.0, Kong Gateway’s schema library’s process_auto_fields function will not make deep copies of data that is passed to it when the given context is select. This was done to avoid excessive deep copying of tables where Kong believes the data most of the time comes from a driver like pgmoon or lmdb.

  If a custom plugin relied on process_auto_fields not overriding the given table, it must make its own copy before passing it to the function now.

• The deprecated shorthands field in Kong plugin or DAO schemas was removed in favor of the typed shorthand_fields. If your custom schemas still use shorthands, you need to update them to use shorthand_fields.

• The support for legacy = true/false attribute was removed from Kong schemas and Kong field schemas.

• The Kong singletons module kong.singletons was removed in favor of the PDK kong.*.

New router

Kong Gateway no longer uses a heuristic to guess whether a route.path is a regex pattern. From 3.0 onward, all regex paths must start with the "~" prefix, and all paths that don’t start with "~" will be considered plain text. The migration process should automatically convert the regex paths when upgrading from 2.x to 3.0.

The normalization rules for route.path have changed. Kong Gateway now stores the unnormalized path, but the regex path always pattern-matches with the normalized URI. Previously, Kong Gateway replaced percent-encoding in the regex path pattern to ensure different forms of URI matches. That is no longer supported. Except for the reserved characters defined in rfc3986, write all other characters without percent-encoding.

Declarative and DB-less

The version number (_format_version) of declarative configuration has been bumped to 3.0 for changes on route.path. Declarative configurations with older versions will be upgraded to 3.0 during migrations.

Do not sync (deck gateway sync) declarative configuration files from 2.8 or earlier to 3.0. Old configuration files will overwrite the configuration and create compatibility issues. To grab the updated configuration, deck gateway dump the 3.0 file after migrations are completed.

It is no longer possible to use the .lua format to import a declarative configuration file from the kong CLI tool. Only JSON and YAML formats are supported. If your update procedure with Kong Gateway involves executing kong config db_import config.lua, convert the config.lua file into a config.json or config.yml file before upgrading.

Admin API

The Admin API endpoint /vitals/reports has been removed.

POST requests on /targets endpoints are no longer able to update existing entities. They are only able to create new ones. If you have scripts that use POST requests to modify /targets, change them to PUT requests to the appropriate endpoints before updating to Kong 3.0.

The list of reported plugins available on the server now returns a table of metadata per plugin instead of a boolean true.

Configuration

The Kong constant CREDENTIAL_USERNAME with the value of X-Credential-Username has been removed.

The default value of lua_ssl_trusted_certificate has changed to system to automatically load the trusted CA list from the system CA store.

The data plane config cache mechanism and its related configuration options (data_plane_config_cache_mode and data_plane_config_cache_path) have been removed in favor of LMDB.

ngx.ctx.balancer_address was removed in favor of ngx.ctx.balancer_data.

Kong for Kubernetes considerations

The Helm chart automates the upgrade migration process. When running helm upgrade, the chart spawns an initial job to run kong migrations up and then spawns new Kong pods with the updated version. Once these pods become ready, they begin processing traffic and old pods are terminated. Once this is complete, the chart spawns another job to run kong migrations finish.

While the migrations themselves are automated, the chart does not automatically ensure that you follow the recommended upgrade path. If you are upgrading from more than one minor Kong Gateway version back, check the upgrade path recommendations.

Although not required, users should upgrade their chart version and Kong Gateway version independently. In the event of any issues, this will help clarify whether the issue stems from changes in Kubernetes resources or changes in Kong Gateway.

For specific Kong for Kubernetes version upgrade considerations, see Upgrade considerations

Kong deployment split across multiple releases

The standard chart upgrade automation process assumes that there is only a single Kong Gateway release in the Kong Gateway cluster, and runs both migrations up and migrations finish jobs.

If you split your Kong Gateway deployment across multiple Helm releases (to create proxy-only and admin-only nodes, for example), you must set which migration jobs run based on your upgrade order.

To handle clusters split across multiple releases, you should:

1. Upgrade one of the releases with:

   helm upgrade RELEASENAME -f values.yaml \
   --set migrations.preUpgrade=true \
   --set migrations.postUpgrade=false
   

2. Upgrade all but one of the remaining releases with:

   helm upgrade RELEASENAME -f values.yaml \
   --set migrations.preUpgrade=false \
   --set migrations.postUpgrade=false
   

3. Upgrade the final release with:

   helm upgrade RELEASENAME -f values.yaml \
   --set migrations.preUpgrade=false \
   --set migrations.postUpgrade=true
   

This ensures that all instances are using the new Kong Gateway package before running kong migrations finish.

Hybrid mode considerations

Important: If you are currently running in hybrid mode, upgrade the control plane first, and then the data planes.

• If you are currently running 2.8.x in classic (traditional) mode and want to run in hybrid mode instead, follow the hybrid mode installation instructions after running the migration.
• Custom plugins (either your own plugins or third-party plugins that are not shipped with Kong Gateway) need to be installed on both the control plane and the data planes in hybrid mode. Install the plugins on the control plane first, and then the data planes.
• The Rate Limiting Advanced plugin does not support the cluster strategy in hybrid mode. The redis strategy must be used instead.

Template changes

There are changes in the Nginx configuration file between every minor and major version of Kong Gateway starting with 2.0.x.

In 3.0.x, the deprecated alias of Kong.serve_admin_api was removed. If your custom Nginx templates still use it, change it to Kong.admin_content.

git clone https://github.com/kong/kong
cd kong
git diff -w 2.0.0 3.0.0 kong/templates/nginx_kong*.lua

git diff 2.0.0 3.0.0 kong/templates/nginx_kong*.lua > kong_config_changes.diff