Kong Ingress Controller 3.2.0 introduced the Fallback Configuration feature. It is designed to isolate issues related to individual parts of the configuration, allowing updates to the rest of it to proceed with no interruption. If you’re using Kong Ingress Controller in a multi-team environment, the fallback configuration mechanism can help you avoid lock-ups when one team’s
configuration is broken.
Note: Fallback Configuration is an opt-in feature. You must enable it by setting FallbackConfiguration=true in the controller’s feature gates configuration. See Feature Gates to learn how to do that.
Kong Ingress Controller translates Kubernetes objects it gets from the Kubernetes API and pushes the translation result via Kong Gateway’s Admin API to Kong Gateway instances. However, issues can arise at various stages of this process:
Admission Webhook: Validates individual Kubernetes objects against schemas and basic rules.
Translation Process: Detects issues like cross-object validation errors.
Kong Gateway Response: Kong Gateway rejects the configuration and returns an error associated with a specific object.
Fallback Configuration is triggered when an issue is detected in the 3rd stage and provides the following benefits:
Allows unaffected objects to be updated even when there are configuration errors.
Automatically builds a fallback configuration that Kong Gateway will accept without requiring user intervention by
either:
Excluding the broken objects along with its dependants.
Backfilling the broken object along with its dependants using the last valid Kubernetes objects’ in-memory cache (if CONTROLLER_USE_LAST_VALID_CONFIG_FOR_FALLBACK environment variable is set to true).
Enables users to inspect and identify what objects were excluded from or backfilled in the configuration using
diagnostic endpoints.
The following table summarizes the behavior of the Fallback Configuration feature based on the configuration:
FallbackConfiguration feature gate value
CONTROLLER_USE_LAST_VALID_CONFIG_FOR_FALLBACK value
Behavior
false
Not applicable
The last valid configuration is used as a whole to recover (if stored).
true
false
The Fallback Configuration is triggered, broken objects and their dependents are excluded.
true
true
The Fallback Configuration is triggered, broken objects and their dependents are excluded and backfilled with their last valid version (if stored).
The diagram below illustrates how the Fallback Configuration feature works in detail:
Each time Kong Ingress Controller successfully applies a fallback configuration, it emits a Kubernetes Event with the FallbackKongConfigurationSucceeded reason. It will also emit an Event with the FallbackKongConfigurationApplyFailed reason in case the fallback configuration gets rejected by Kong Gateway. You can monitor these events to track the fallback configuration process.
You can check to see if the Event is emitted by running:
kubectl get events -A--field-selector='reason=FallbackKongConfigurationSucceeded'
Copied to clipboard!
The results should look like this:
NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE
kong 4m26s Normal FallbackKongConfigurationSucceeded pod/kong-controller-7f4fd47bb7-zdktb successfully applied fallback Kong configuration to https://192.168.194.11:8444
Copied to clipboard!
Another way to monitor the Fallback Configuration mechanism is by Prometheus metrics. See Prometheus Metrics for more information.