Fallback configuration

Kong Ingress Controller 3.2.0 introduced the Fallback Configuration feature. It is designed to isolate issues related to individual parts of the configuration, allowing updates to the rest of it to proceed with no interruption. If you’re using Kong Ingress Controller in a multi-team environment, the fallback configuration mechanism can help you avoid lock-ups when one team’s configuration is broken.

Note: Fallback Configuration is an opt-in feature. You must enable it by setting FallbackConfiguration=true in the controller’s feature gates configuration. See Feature Gates to learn how to do that.

How fallback configuration works

Kong Ingress Controller translates Kubernetes objects it gets from the Kubernetes API and pushes the translation result via Kong Gateway’s Admin API to Kong Gateway instances. However, issues can arise at various stages of this process:

  1. Admission Webhook: Validates individual Kubernetes objects against schemas and basic rules.
  2. Translation Process: Detects issues like cross-object validation errors.
  3. Kong Gateway Response: Kong Gateway rejects the configuration and returns an error associated with a specific object.

Fallback Configuration is triggered when an issue is detected in the 3rd stage and provides the following benefits:

  • Allows unaffected objects to be updated even when there are configuration errors.
  • Automatically builds a fallback configuration that Kong Gateway will accept without requiring user intervention by either:
    • Excluding the broken objects along with its dependants.
    • Backfilling the broken object along with its dependants using the last valid Kubernetes objects’ in-memory cache (if CONTROLLER_USE_LAST_VALID_CONFIG_FOR_FALLBACK environment variable is set to true).
  • Enables users to inspect and identify what objects were excluded from or backfilled in the configuration using diagnostic endpoints.

The following table summarizes the behavior of the Fallback Configuration feature based on the configuration:

FallbackConfiguration feature gate value

CONTROLLER_USE_LAST_VALID_CONFIG_FOR_FALLBACK value

Behavior

false Not applicable The last valid configuration is used as a whole to recover (if stored).
true false The Fallback Configuration is triggered, broken objects and their dependents are excluded.
true true The Fallback Configuration is triggered, broken objects and their dependents are excluded and backfilled with their last valid version (if stored).

The diagram below illustrates how the Fallback Configuration feature works in detail:

Inspecting the fallback configuration process

Each time Kong Ingress Controller successfully applies a fallback configuration, it emits a Kubernetes Event with the FallbackKongConfigurationSucceeded reason. It will also emit an Event with the FallbackKongConfigurationApplyFailed reason in case the fallback configuration gets rejected by Kong Gateway. You can monitor these events to track the fallback configuration process.

You can check to see if the Event is emitted by running:

kubectl get events -A --field-selector='reason=FallbackKongConfigurationSucceeded'
Copied to clipboard!

The results should look like this:

NAMESPACE   LAST SEEN   TYPE     REASON                               OBJECT                                 MESSAGE
kong        4m26s       Normal   FallbackKongConfigurationSucceeded   pod/kong-controller-7f4fd47bb7-zdktb   successfully applied fallback Kong configuration to https://192.168.194.11:8444
Copied to clipboard!

Another way to monitor the Fallback Configuration mechanism is by Prometheus metrics. See Prometheus Metrics for more information.

Did this doc help?

Something wrong?

Help us make these docs great!

Kong Developer docs are open source. If you find these useful and want to make them better, contribute today!