Circuit Breaker
New to Kuma? You don’t need this policy, check
MeshCircuitBreaker
instead. If you want to use theCircuitBreaker
policy, remember that it requires the TrafficRoute policy to function properly.
Circuit Breaker is an outbound policy. Dataplanes whose configuration is modified are in the
sources
matcher.
This policy will look for errors in the live traffic being exchanged between our data plane proxies and it will mark a data proxy as an unhealthy if certain conditions are met and - by doing so - making sure that no additional traffic can reach an unhealthy data plane proxy until it is healthy again.
Circuit breakers - unlike active Health Checks - do not send additional traffic to our data plane proxies but they rather inspect the existing service traffic. They are also commonly used to prevent cascading failures in our services.
Like a real-world circuit breaker when the circuit is closed then traffic between a source and destination data plane proxy is allowed to freely flow through it, and when it is open then the traffic is interrupted.
The conditions that determine when a circuit breaker is closed or open are being configured in what we call “detectors”. This policy provides 5 different types of detectors and they are triggered on some deviations in the upstream service behavior. All detectors could coexist on the same outbound interface.
Once one of the detectors has been triggered the corresponding data plane proxy is ejected from the set of the load balancer for a period equal to baseEjectionTime. Every further ejection of the same data plane proxy will further extend the baseEjectionTime multiplied by the number of ejections: for example the 4th ejection will be lasting for a period of time of 4 * baseEjectionTime
.
Available detectors:
Usage
As usual, we can apply sources
and destinations
selectors to determine how circuit breakers will be applied across our data plane proxies.
For example:
apiVersion: kuma.io/v1alpha1
kind: CircuitBreaker
mesh: default
metadata:
name: circuit-breaker-example
spec:
sources:
- match:
kuma.io/service: web
destinations:
- match:
kuma.io/service: backend
conf:
interval: 5s
baseEjectionTime: 30s
maxEjectionPercent: 20
splitExternalAndLocalErrors: false
thresholds:
maxConnections: 2
maxPendingRequests: 2
maxRequests: 2
maxRetries: 2
detectors:
totalErrors:
consecutive: 20
gatewayErrors:
consecutive: 10
localErrors:
consecutive: 7
standardDeviation:
requestVolume: 10
minimumHosts: 5
factor: 1.9
failure:
requestVolume: 10
minimumHosts: 5
threshold: 85
We will apply the configuration with kubectl apply -f [..]
.
The example demonstrates a complete configuration. A CircuitBreaker
can also be configured in a simpler way by leveraging the default values of Envoy for any property that is not explicitly defined, for instance:
apiVersion: kuma.io/v1alpha1
kind: CircuitBreaker
mesh: default
metadata:
name: circuit-breaker-example
spec:
sources:
- match:
kuma.io/service: web_default_svc_80
destinations:
- match:
kuma.io/service: backend_default_svc_80
conf:
detectors:
totalErrors: {}
standardDeviation: {}
We will apply the configuration with kubectl apply -f [..]
.
In this example when we get five errors in a row of any type (5
is default Envoy value for totalErrors.consecutive
) the data plane proxy will be ejected for 30s
the first time, 60s
for the second time, and so on.
In the current version of Kong Mesh
destinations
only supports theservice
tag.
interval
Time interval between ejection analysis sweeps. Defaults to 10s.
baseEjectionTime
The base time that a data plane proxy is ejected for. The real time is equal to the base time multiplied by the number of times the data plane proxy has been ejected. Defaults to 30s.
maxEjectionPercent
The maximum percent of an upstream Envoy cluster that can be ejected due to outlier detection. Defaults to 10% but will eject at least one data plane proxy regardless of the value.
splitExternalAndLocalErrors
Activates Split Mode.
Split Mode: There are two types of errors that might occur in a circuit breaker:
- Locally originated: errors triggered locally when estabilishing a connection at the TCP layer (ie: connection refused, connection reset).
- Externally originated: errors triggered remotely like a 5xx error in the response.
If Split Mode is off, Kong Mesh won’t distinguish errors by their origin and they will be counted together. If Split Mode is on, different parameters can be used to fine tune the detectors. All detectors counts errors according to the state of this parameter.
Detectors
Below is a list of available detectors that can be configured in Kong Mesh.
Total Errors
Errors with status code 5xx and locally originated errors, in Split Mode just errors with status code 5xx.
-
consecutive
- how many consecutive errors in a row will trigger the detector. Defaults to5
.
Gateway Errors
Subset of totalErrors related to gateway errors (502, 503 or 504 status code).
-
consecutive
- how many consecutive errors in a row will trigger the detector. Defaults to5
.
Local Errors
Taken into account only in Split Mode, number of locally originated errors.
-
consecutive
- how many consecutive errors in a row will trigger the detector. Defaults to5
.
Standard Deviation
Detection based on success rate, aggregated from every data plane proxy in the Envoy cluster.
-
requestVolume
- ignore data plane proxies with a number of requests less thanrequestVolume
. Defaults to100
. -
minimumHosts
- ignore counting the success rate for an Envoy cluster if the number of data plane proxies with requiredrequestVolume
is less thanminimumHosts
. Defaults to5
. -
factor
- resulting threshold equals tomean - (stdev * factor)
. Defaults to1.9
.
Failures
Detection based on success rate with an explicit threshold (unlike standardDeviation).
-
requestVolume
- ignore data plane proxies with a number of requests less thanrequestVolume
. Defaults to50
. -
minimumHosts
- ignore counting the success rate for an Envoy cluster if the number of data plane proxies with requiredrequestVolume
is less thanminimumHosts
. Defaults to5
. -
threshold
- eject the data plane proxy if its percentage of failures is greater than - or equal to - this value. Defaults to85
.
Thresholds
Alongside the detectors, CircuitBreaker allows configuring thresholds:
-
maxConnections
- the maximum number of connections that Envoy will make to the upstream cluster. If not specified, the default is 1024. -
maxPendingRequests
- the maximum number of pending requests that Envoy will allow to the upstream cluster. If not specified, the default is 1024. -
maxRequests
- the maximum number of parallel requests that Envoy will make to the upstream cluster. If not specified, the default is 1024. -
maxRetries
- the maximum number of parallel retries that Envoy will allow to the upstream cluster. If not specified, the default is 3.
Matching
CircuitBreaker
is an Outbound Connection Policy.
The only supported value for destinations.match
is kuma.io/service
.
Builtin Gateway support
Circuit Breaker policies are supported on the builtin gateway like any other dataplane.
Non-mesh traffic
When passthrough mode is activated any non-mesh traffic is passing Envoy without applying the CircuitBreaker policies. Read more about Non-mesh traffic.
All policy options
$schema: http://json-schema.org/draft-04/schema#
$ref: #/definitions/CircuitBreaker
definitions
CircuitBreaker
- ## Circuit Breaker
- CircuitBreaker defines circuit breaking policy for dataplane's outbound
- Type:
object
- This schema accepts additional properties.
- Properties
- sources
- List of selectors to match dataplanes that are sources of traffic.
- Type:
array
- Items
- $ref: #/definitions/kuma.mesh.v1alpha1.Selector
- destinations
- List of selectors to match services that are destinations of traffic.
- Type:
array
- Items
- $ref: #/definitions/kuma.mesh.v1alpha1.Selector
- conf
- sources
kuma.mesh.v1alpha1.CircuitBreaker.Conf
- ## Conf
- Type:
object
- This schema accepts additional properties.
- Properties
- interval
- Time interval between ejection analysis sweeps
- Type:
string
- String format must be a "regex"
- The value must match this pattern:
^([0-9]+\.?[0-9]*|\.[0-9]+)s$
- baseEjectionTime
- The base time that a host is ejected for. The real time is equal to the base time multiplied by the number of times the host has been ejected
- Type:
string
- String format must be a "regex"
- The value must match this pattern:
^([0-9]+\.?[0-9]*|\.[0-9]+)s$
- maxEjectionPercent
- The maximum percent of an upstream cluster that can be ejected due to outlier detection, has to be in [0 - 100] range
- Type:
integer
- splitExternalAndLocalErrors
- Enables Split Mode in which local and external errors are distinguished
- Type:
boolean
- detectors
- thresholds
- interval
kuma.mesh.v1alpha1.CircuitBreaker.Conf.Detectors
- ## Detectors
- Type:
object
- This schema accepts additional properties.
- Properties
- totalErrors
- Errors with status code 5xx and locally originated errors, in Split Mode - just errors with status code 5xx
- $ref: #/definitions/kuma.mesh.v1alpha1.CircuitBreaker.Conf.Detectors.Errors
- gatewayErrors
- Subset of 'total' related to gateway errors (502, 503 or 504 status code)
- $ref: #/definitions/kuma.mesh.v1alpha1.CircuitBreaker.Conf.Detectors.Errors
- localErrors
- Takes into account only in Split Mode, number of locally originated errors
- $ref: #/definitions/kuma.mesh.v1alpha1.CircuitBreaker.Conf.Detectors.Errors
- standardDeviation
- failure
- totalErrors
kuma.mesh.v1alpha1.CircuitBreaker.Conf.Detectors.Errors
- ## Errors
- Detector based on counting consecutive number of errors
- Type:
object
- This schema accepts additional properties.
- Properties
- consecutive
- Type:
integer
- Type:
- consecutive
kuma.mesh.v1alpha1.CircuitBreaker.Conf.Detectors.Failure
- ## Failure
- Detection based on success rate, but threshold is set explicitly (unlike 'standardDeviation')
- Type:
object
- This schema accepts additional properties.
- Properties
- requestVolume
- Ignore hosts with less number of requests than 'requestVolume'
- Type:
integer
- minimumHosts
- Won't count success rate for cluster if number of hosts with required 'requestVolume' is less than 'minimumHosts'
- Type:
integer
- threshold
- Eject host if failure percentage of a given host is greater than or equal to this value, has to be in [0 - 100] range
- Type:
integer
- requestVolume
kuma.mesh.v1alpha1.CircuitBreaker.Conf.Detectors.StandardDeviation
- ## Standard Deviation
- Detection based on success rate, aggregated from every host in the cluser
- Type:
object
- This schema accepts additional properties.
- Properties
- requestVolume
- Ignore hosts with less number of requests than 'requestVolume'
- Type:
integer
- minimumHosts
- Won't count success rate for cluster if number of hosts with required 'requestVolume' is less than 'minimumHosts'
- Type:
integer
- factor
- Resulting threshold = mean - (stdev * factor)
- Type:
number
- requestVolume
kuma.mesh.v1alpha1.CircuitBreaker.Conf.Thresholds
- ## Thresholds
- Type:
object
- This schema accepts additional properties.
- Properties
- maxConnections
- The maximum number of connections that Envoy will make to the upstream cluster. If not specified, the default is 1024.
- Type:
integer
- maxPendingRequests
- The maximum number of pending requests that Envoy will allow to the upstream cluster. If not specified, the default is 1024.
- Type:
integer
- maxRetries
- The maximum number of parallel retries that Envoy will allow to the upstream cluster. If not specified, the default is 3.
- Type:
integer
- maxRequests
- The maximum number of parallel requests that Envoy will make to the upstream cluster. If not specified, the default is 1024.
- Type:
integer
- maxConnections
kuma.mesh.v1alpha1.Selector
- ## Selector
- Selector defines structure for selecting tags for given dataplane
- Type:
object
- This schema accepts additional properties.
- Properties
- match
- Tags to match, can be used for both source and destinations
- Type:
object
- This schema accepts additional properties.
- Properties
- match
Generated with json-schema-md-doc