Kong Konnect

Troubleshoot Runtime Instances

Out of sync runtime instance

Occasionally, a Kong Gateway runtime instance might get out of sync with the Konnect control plane. If this happens, you will see the status Out of sync on the Runtime Instances page, meaning the control plane can’t communicate with the instance.

Troubleshoot the issue using the following methods:

Ensure the runtime instance is running. If it’s not running, start it; if it is running, restart it.

Check the sync status in the Runtime Manager.
Check the logs of the runtime that’s appearing as Out of sync. The default directory for Kong Gateway logs is /usr/local/kong/logs.

If you find any of the following errors:
- Runtime instance failed to connect to the control plane.
- Runtime instance failed to ping the control plane.
- Runtime instance failed to receive a ping response from the control plane.
You may have an issue on the host network where the instance resides. Diagnose and resolve the issue, then restart the instance and check the sync status in the Runtime Manager.
If the logs show a license issue, contact Kong Support.

If you are unable to resolve sync issues using the above methods, contact Kong Support.

Missing functionality

If a Konnect feature isn’t working on your runtime instance, the version may be out of date.

Verify that your instance versions are up to date:

Open Runtime Manager, then open your runtime group.
Click + Add runtime instance and check the Kong Gateway version in the code block. This is the version that the Konnect control plane is running.
Return to the runtime instances page.
Check the runtime instance versions in the table. If you see an instance running an older version of Kong Gateway, your runtime instance may need upgrading.

If your version is up to date but the feature still isn’t working, contact Kong Support.

Version compatibility

We recommend running one major version (2.x or 3.x) of a runtime instance per runtime group, unless you are in the middle of version upgrades to the data plane.

If you mix major runtime instance versions, the control plane will support the least common subset of configurations across all the versions connected to the Konnect control plane. For example, if you are running 2.8.1.3 on one runtime instance and 3.0.0.0 on another, the control plane will only push configurations that can be used by the 2.8.1.3 runtime instance.

If you experience compatibility errors, upgrade your data planes to match the version of the highest-versioned runtime instance in your runtime group.

Possible compatibility errors:

Error code	Severity	Description	Resolution	References
D100	error	Plugin `jq` is not available in Kong Gateway versions < 2.6.0.0.	Please upgrade Kong Gateway to version `2.6.0.0` or above.	jq plugin
D101	error	For the `canary` plugin, one or more of the following `config` fields are set: `hash_header` but Kong Gateway versions < 2.6.0.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.6.0.0` or above.	Canary plugin
D102	error	For the `canary` plugin, `config.hash` field has been set to `header`. This is not supported in Kong Gateway versions < 2.6.0.0. The plugin configuration has been changed to `config.hash=consumer` in the data-plane.	Please upgrade Kong Gateway to version `2.6.0.0` or above.	Canary plugin
D103	error	For the `kafka-log` plugin, one or more of the following `config` fields are set: `authentication`, `keepalive_enabled`, `security` but Kong Gateway versions < 2.6.0.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.6.0.0` or above.	Kafka Log plugin
D104	error	For the `kafka-upstream` plugin, one or more of the following `config` fields are set: `authentication`, `keepalive_enabled`, `security` but Kong Gateway versions < 2.6.0.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.6.0.0` or above.	Kafka Upstream plugin
D105	error	For the `rate-limiting-advanced` plugin, either `config.strategy` is set to `local` or `config.identifier` is set to `path` or both. These settings are not supported in Kong Gateway versions < 2.6.0.0. In the data-plane, the plugin configuration has one or more of the following changes: (a) `config.strategy` has been updated to `redis`, (b) `config.sync_rate` has been updated to `1`, (c) `config.identifier has been updated to` path`.`	Please upgrade Kong Gateway to version `2.6.0.0` or above.	Rate Limiting Advanced plugin
D106	error	For the `openid-connect` plugin, one or more of the following unsupported `config` fields or values for a `config` field are in use: `by_username_ignore_case`, `disable_session`, `downstream_introspection_jwt_header`, `downstream_user_info_jwt_header`, `introspection_accept`, `introspection_check_active`, `upstream_introspection_jwt_header`, `upstream_user_info_jwt_header`, `userinfo_accept`, `userinfo_headers_client`, `userinfo_headers_names`, `userinfo_headers_values`, `userinfo_query_args_client`, `userinfo_query_args_names`, `userinfo_query_args_values`, `auth_methods` array field with value `userinfo`, `ignore_signature` array field with value `introspection`, `ignore_signature` array field with value `userinfo`, `login_methods` array field with value `userinfo`, `token_headers_grants` array field with value `refresh_token`.	Please upgrade Kong Gateway to version `2.6.0.0` or above.	OpenID Connect plugin
D107	error	For the `rate-limiting-advanced` plugin, one or more of the following `config` fields are set: `path` but Kong Gateway versions < 2.6.0.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.6.0.0` or above.	Rate Limiting Advanced plugin
D108	error	For the `forward-proxy` plugin, one or more of the following `config` fields are set: `auth_username`, `auth_password` but Kong Gateway versions < 2.7.0.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.7.0.0` or above.	Forward Proxy plugin
D109	error	For the `mocking` plugin, one or more of the following `config` fields are set: `random_examples` but Kong Gateway versions < 2.7.0.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.7.0.0` or above.	Mocking plugin
D110	error	For the `rate-limiting-advanced` plugin, one or more of the following `config` fields are set: `enforce_consumer_groups`, `consumer_groups` but Kong Gateway versions < 2.7.0.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.7.0.0` or above.	Rate Limiting Advanced plugin
D111	error	For the `rate-limiting-advanced` plugin, one or more of the following `config` fields are set: `redis.username`, `redis.sentinel_username` but Kong Gateway versions < 2.8.0.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.8.0.0` or above.	Rate Limiting Advanced plugin
D112	error	For the `proxy-cache-advanced` plugin, one or more of the following `config` fields are set: `redis.username`, `redis.sentinel_username` but Kong Gateway versions < 2.8.0.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.8.0.0` or above.	Proxy Cache Advanced plugin
D113	error	For the `canary` plugin, one or more of the following `config` fields are set: `canary_by_header_name` but Kong Gateway versions < 2.8.0.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.8.0.0` or above.	Canary plugin
D114	error	For the `forward-proxy` plugin, one or more of the following `config` fields are set: `https_proxy_host`, `https_proxy_port` but Kong Gateway versions < 2.8.0.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.8.0.0` or above.	Forward Proxy plugin
D115	warning	N/A	N/A
D116	error	For the `openid-connect` plugin, one or more of the following `config` fields are set: `session_redis_username`, `resolve_distributed_claims` but Kong Gateway versions < 2.8.0.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.8.0.0` or above.	OpenID Connect plugin
D118	error	For the `kafka-log` plugin, one or more of the following `config` fields are set: `cluster_name` but Kong Gateway versions < 2.8.0.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.8.0.0` or above.	Kafka Log plugin
D119	error	For the `kafka-upstream` plugin, one or more of the following `config` fields are set: `cluster_name` but Kong Gateway versions < 2.8.0.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.8.0.0` or above.	Kafka Upstream plugin
D120	error	For the `kafka-log` plugin `config.authentication.mechanism` is set to `SCRAM-SHA-512`. This is not supported in Kong Gateway versions < 2.8.1.1. The value for the field has been changed to `SCRAM-SHA-256` in the data-plane.	Please upgrade Kong Gateway to version `2.8.1.1` or above.	Kafka Log plugin
D121	error	For the `kafka-upstream` plugin `config.authentication.mechanism` is set to `SCRAM-SHA-512`. This is not supported in Kong Gateway versions < 2.8.1.1. The value for the field has been changed to `SCRAM-SHA-256` in the data-plane.	Please upgrade Kong Gateway to version `2.8.1.1` or above.	Kafka Upstream plugin
D122	error	For the `mtls-auth` plugin, one or more of the following `config` fields are set: `http_proxy_host`, `http_proxy_port`, `https_proxy_host`, `https_proxy_port` but Kong Gateway versions < 2.8.1.1 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.8.1.1` or above.	mTLS Authentication plugin
D123	error	It seems you are using Konnect Developer Portal with Kong Gateway. For the Developer Portal’s application registration feature to work correctly, `konnect-application-auth` plugin is required but this plugin is not available on Kong Gateway versions < 3.0. The feature is not working as intended and can cause severe security issues including authentication and authorization not being enforced in the Gateway.	Please upgrade Kong Gateway to version `3.0.0.0` or above.	Not applicable
D124	error	For the `ldap-auth-advanced` plugin, one or more of the following `config` fields are set: `group_required` but Kong Gateway versions < 3.0.0.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `3.0.0.0` or above.	LDAP Authentication Advanced plugin
D125	error	For the `opa` plugin, one or more of the following `config` fields are set: `include_body_in_opa_input`, `include_parsed_json_body_in_opa_input`, `ssl_verify` but Kong Gateway versions < 3.0.0.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `3.0.0.0` or above.	OPA plugin
D126	error	For the `statsd-advanced` plugin, one or more of the following `config` fields are set: `consumer_identifier_default`, `service_identifier_default`, `workspace_identifier_default` but Kong Gateway versions < 3.0.0.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `3.0.0.0` or above.	StatsD Advanced plugin
P101	error	For the `acme` plugin, one or more of the following `config` fields are set: `preferred_chain`, `storage_config.vault.auth_method`, `storage_config.vault.auth_path`, `storage_config.vault.auth_role`, `storage_config.vault.jwt_path` but Kong Gateway versions < 2.6 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.6` or above.	ACME plugin
P102	error	For the `aws-lambda` plugin, one or more of the following `config` fields are set: `base64_encode_body` but Kong Gateway versions < 2.6 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.6` or above.	AWS Lambda plugin
P103	error	For the `grpc-web` plugin, one or more of the following `config` fields are set: `allow_origin_header` but Kong Gateway versions < 2.6 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.6` or above.	gRPC Web plugin
P104	error	For the `request-termination` plugin, one or more of the following `config` fields are set: `echo`, `trigger` but Kong Gateway versions < 2.6 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.6` or above.	Request Termination plugin
P105	error	For the `datadog` plugin, one or more of the following `config` fields are set: `service_name_tag`, `status_tag`, `consumer_tag` but Kong Gateway versions < 2.7 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.7` or above.	Datadog plugin
P106	error	For the `datadog` plugin, distribution metric type is not support in Kong gateway versions < 2.7. Distribution metrics will not be emitted by the gateway.	Please upgrade Kong Gateway to version `2.7` or above.	Datadog plugin
P107	error	For the `ip-restriction` plugin, one or more of the following `config` fields are set: `status`, `message` but Kong Gateway versions < 2.7 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.7` or above.	IP Restriction plugin
P108	error	For the `rate-limiting` plugin, one or more of the following `config` fields are set: `redis_ssl`, `redis_ssl_verify`, `redis_server_name` but Kong Gateway versions < 2.7 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.7` or above.	Rate Limiting plugin
P109	error	For the `zipkin` plugin, `config.header_type` field has been set to `ignore`. This is not supported in Kong Gateway versions < 2.7. The plugin configuration has been changed to `config.header_type=preserve` in the data-plane.	Please upgrade Kong Gateway to version `2.7` or above.	Zipkin plugin
P110	error	For the `zipkin` plugin, one or more of the following `config` fields are set: `local_service_name` but Kong Gateway versions < 2.7 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.7` or above.	Zipkin plugin
P111	error	For the `acme` plugin, one or more of the following `config` fields are set: `rsa_key_size` but Kong Gateway versions < 2.8 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.8` or above.	ACME plugin
P112	error	For the `rate-limiting` plugin, one or more of the following `config` fields are set: `redis_username` but Kong Gateway versions < 2.8 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.8` or above.	Rate Limiting plugin
P113	error	For the `response-ratelimiting` plugin, one or more of the following `config` fields are set: `redis_username` but Kong Gateway versions < 2.8 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `2.8` or above.	Response Rate Limiting plugin
P114	error	For the `response-ratelimiting` plugin, one or more of the following `config` fields are set: `redis_username` but Kong Gateway versions < 2.8 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `3.0` or above.	Response Rate Limiting plugin
P115	error	Plugin `opentelemetry` is not available in Kong gateway versions < 3.0.	Please upgrade Kong Gateway to version `3.0` or above.	OpenTelemetry plugin
P116	error	For the `zipkin` plugin, one or more of the following `config` fields are set: `http_span_name`, `connect_timeout`, `send_timeout`, `read_timeout` but Kong Gateway versions < 3.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `3.0` or above.	Zipkin plugin
P117	error	For the `prometheus` plugin, one or more of the following `config` fields are set: `status_code_metrics`, `latency_metrics`, `bandwidth_metrics`, `upstream_health_metrics` but Kong Gateway versions < 3.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `3.0` or above.	Prometheus plugin
P118	error	For the `acme` plugin, one or more of the following `config` fields are set: `allow_any_domain` but Kong Gateway versions < 3.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please upgrade Kong Gateway to version `3.0` or above.	ACME plugin
P119	error	For the `Service` entity, `enabled` field has been set to `true` but Kong Gateway versions < 2.7 do not support this feature. The Service has been left enabled in the Kong Gateway, and the traffic for the Service is being routed by Kong Gateway. This is a critical error and may result in unwanted traffic being routed to the upstream Services via Kong Gateway.	Please upgrade Kong Gateway to version `2.7` or above.	Service entity
P120	error	For the `aws-lambda` plugin, one or more of the following `config` fields are set: `proxy_scheme` but Kong Gateway versions >= 3.0 do not support these fields. Plugin features that rely on these fields are not working as intended.	Please use `config.proxy_url` instead of `config.proxy_scheme` field.	AWS Lambda plugin
P121	error	For the `aws-lambda` plugin, `config.aws_region` and `config.host` fields are set. These fields were mutually exclusive for Kong gateway versions < 3.0. The plugin configuration has been changed to remove the `config.host` field.	Please upgrade Kong Gateway to version `3.0` or above.	AWS Lambda plugin
P122	error	For the `pre-function` plugin, `config.functions` field has been used. This is not supported in Kong Gateway versions >= 3.0. The plugin configuration has been updated to rename `config.functions` to `config.access` in the data-plane.	Please update the configuration to use `config.access` field instead of `config.functions`.	Serverless Functions plugin
P123	error	For the `post-function` plugin, `config.functions` field has been used. This is not supported in Kong Gateway versions >= 3.0. The plugin configuration has been updated to rename `config.functions` to `config.access` in the data-plane.	Please update the configuration to use `config.access` field instead of `config.functions`.	Serverless Functions plugin
P124	warning	For the `pre-function` plugin, `config.functions` field has been used. This field is deprecated and it is no longer supported in Kong Gateway versions >= 3.0.	Please update the plugin configuration to use `config.access` field in place of `config.functions` field	Serverless Functions plugin
P125	warning	For the `post-function` plugin, `config.functions` field has been used. This field is deprecated and it is no longer supported in Kong Gateway versions >= 3.0.	Please update the plugin configuration to use `config.access` field in place of `config.functions` field	Serverless Functions plugin