Skip to content
Kong Logo | Kong Docs Logo
search
  • We're Hiring!
  • Docs
    • Kong Gateway
    • Kong Konnect
    • Kong Mesh
    • Plugin Hub
    • decK
    • Kubernetes Ingress Controller
    • Insomnia
    • Kuma

    • Docs contribution guidelines
  • Plugin Hub
  • Support
  • Community
  • Kong Academy
Get a Demo Start Free Trial
  • Kong Gateway
  • Kong Konnect
  • Kong Mesh
  • Plugin Hub
  • decK
  • Kubernetes Ingress Controller
  • Insomnia
  • Kuma

  • Docs contribution guidelines
  • 2.1.x (latest)
  • 2.0.x
  • 1.9.x
  • 1.8.x
  • 1.7.x
  • 1.6.x
  • 1.5.x
  • 1.4.x
  • 1.3.x
  • 1.2.x
  • 1.1.x
  • 1.0.x
    • Introduction to Kong Mesh
    • What is Service Mesh?
    • How Kong Mesh works
    • Deployments
    • Version support policy
    • Stability
    • Release notes
    • Installation Options
    • Kubernetes
    • Helm
    • OpenShift
    • Docker
    • Amazon ECS
    • Amazon Linux
    • Red Hat
    • CentOS
    • Debian
    • Ubuntu
    • macOS
    • Windows
    • Explore Kong Mesh with the Kubernetes demo app
    • Explore Kong Mesh with the Universal demo app
    • Standalone deployment
    • Deploy a standalone control plane
    • Multi-zone deployment
    • Deploy a multi-zone global control plane
    • License
    • Overview
    • Data plane proxy
    • Data plane on Kubernetes
    • Data plane on Universal
    • Gateway
    • Zone Ingress
    • Zone Egress
    • CLI
    • GUI
    • Observability
    • Inspect API
    • Kubernetes Gateway API
    • Networking
    • Service Discovery
    • DNS
    • Kong Mesh CNI
    • Transparent Proxying
    • IPv6 support
    • Non-mesh traffic
    • Secure access across Kong Mesh components
    • Secrets
    • Kong Mesh API Access Control
    • API server authentication
    • Data plane proxy authentication
    • Zone proxy authentication
    • Data plane proxy membership
    • Dataplane Health
    • Fine-tuning
    • Control Plane Configuration
    • Upgrades
    • Requirements
    • Introduction
    • General notes about Kong Mesh policies
    • Applying Policies
    • How Kong Mesh chooses the right policy to apply
    • Understanding TargetRef policies
    • Protocol support in Kong Mesh
    • Mesh
    • Mutual TLS
    • Traffic Permissions
    • Traffic Route
    • Traffic Metrics
    • Traffic Trace
    • Traffic Log
    • Locality-aware Load Balancing
    • Fault Injection
    • Health Check
    • Circuit Breaker
    • Proxy Template
    • External Service
    • Retry
    • Timeout
    • Rate Limit
    • Virtual Outbound
    • MeshGateway
    • MeshGatewayRoute
    • Service Health Probes
    • MeshAccessLog (Beta)
    • MeshCircuitBreaker (Beta)
    • MeshFaultInjection (Beta)
    • MeshHealthCheck (Beta)
    • MeshHTTPRoute (Beta)
    • MeshProxyPatch (Beta)
    • MeshRateLimit (Beta)
    • MeshRetry (Beta)
    • MeshTimeout (Beta)
    • MeshTrace (Beta)
    • MeshTrafficPermission (Beta)
    • Overview
    • HashiCorp Vault CA
    • Amazon ACM Private CA
    • cert-manager Private CA
    • OPA policy support
    • MeshOPA (beta)
    • Multi-zone authentication
    • FIPS support
    • Certificate Authority rotation
    • Role-Based Access Control
    • UBI Images
    • Windows Support
    • Auditing
    • HTTP API
    • Annotations and labels in Kubernetes mode
    • Kong Mesh data collection
      • Mesh
      • CircuitBreaker
      • ExternalService
      • FaultInjection
      • HealthCheck
      • MeshGateway
      • MeshGatewayRoute
      • ProxyTemplate
      • RateLimit
      • Retry
      • Timeout
      • TrafficLog
      • TrafficPermission
      • TrafficRoute
      • TrafficTrace
      • VirtualOutbound
      • Dataplane
      • ZoneEgress
      • ZoneIngress
      • kuma-cp
      • kuma-dp
      • kumactl
    • Kuma-cp configuration reference
    • Open source License
    • Contribute to Mesh

github-edit-pageEdit this page

report-issueReport an issue

enterprise-switcher-iconSwitch to OSS

On this page
  • TargetRef support matrix
  • Configuration
    • Retry on
    • Backoff
    • Rate limited backoff
  • Examples
    • HTTP web to backend on 5xx
    • gRPC web to backend on DeadlineExceeded
    • TCP web to backend
Kong Mesh
2.1.x (latest)
  • Home
  • Kong Mesh
  • Policies
  • MeshRetry (beta)

MeshRetry (beta)

This policy uses new policy matching algorithm and is in beta state, it should not be mixed with Retry.

This policy enables Kong Mesh to know how to behave if there is a failed scenario (i.e. HTTP request) which could be retried.

TargetRef support matrix

TargetRef type top level to from
Mesh ✅ ✅ ❌
MeshSubset ✅ ❌ ❌
MeshService ✅ ✅ ❌
MeshServiceSubset ✅ ❌ ❌
MeshGatewayRoute ❌ ❌ ❌

To learn more about the information in this table, see the matching docs.

Configuration

The policy let you configure retry behaviour for HTTP, GRPC and TCP protocols. The protocol is selected by picking the most specific protocol.

Each protocol has a separate section under default in the policy YAML. Some sections are common between protocols or have similar meaning.

Retry on

The field retryOn is a list of conditions which will cause a retry.

For HTTP these are related to the response status code or method (“5xx”, “429”, “HttpMethodGet”). For gRPC these are status codes in response headers (“canceled”, “deadline-exceeded”, etc.). There is no equivalent for TCP.

One or more conditions can be specified, for example:

retryOn:
  - "429"
  - "503"

means that it the policy will retry on a status code 429 or 503.

Full list of available HTTP conditions:

retryOn:
  - 5XX
  - GatewayError
  - Reset
  - Retriable4xx
  - ConnectFailure
  - EnvoyRatelimited
  - RefusedStream
  - Http3PostConnectFailure
  - HttpMethodConnect
  - HttpMethodDelete
  - HttpMethodGet
  - HttpMethodHead
  - HttpMethodOptions
  - HttpMethodPatch
  - HttpMethodPost
  - HttpMethodPut
  - HttpMethodTrace
  - "429" # any HTTP status code
  - "503"

Full list of available gRPC conditions:

retryOn:
  - Canceled
  - DeadlineExceeded
  - Internal
  - ResourceExhausted
  - Unavailable

Backoff

This parameter is applicable to both HTTP and GRPC.

It consists of BaseInterval (the amount of time between retries) and MaxInterval (the maximal amount of time taken between retries).

We use a fully jittered exponential back-off algorithm for retries. Given a base interval B and retry number N, the back-off for the retry is in the range [0, (2N - 1) × B).

For example, given a 25ms interval, the first retry will be delayed randomly by 0-24ms, the 2nd by 0-74ms, the 3rd by 0-174ms, and so on.

The interval is capped at a MaxInterval, which defaults to 10 times the BaseInterval.

Rate limited backoff

This parameter is applicable to both HTTP and GRPC.

MeshRetry can be configured in such a way that when the upstream server rate limits the request and responds with a header like retry-after or x-ratelimit-reset it uses the value from the header to determine when to send the retry request instead of the backoff algorithm.

Example

Given this configuration:

retryOn:
  - "503"
rateLimitedBackOff:
  resetHeaders:
    - name: retry-after
      format: Seconds
    - name: x-ratelimit-reset
      format: UnixTimestamp

and an HTTP response:

HTTP/1.1 503 Service Unavailable
retry-after: 15

The retry request will be issued after 15 seconds.

If the response is as follows:

HTTP/1.1 503 Service Unavailable
x-ratelimit-reset: 1706096119

The request will be retried at Wed Jan 24 2024 11:35:19 GMT+0000.

If the response does not contain retry-after or x-ratelimit-reset header (with valid integer value) then the amount of time to wait before issuing a request is determined by backoff algorithm.

Examples

HTTP web to backend on 5xx

Kubernetes
Universal
apiVersion: kuma.io/v1alpha1
kind: MeshRetry
metadata:
  name: web-to-backend-retry-http
  namespace: kong-mesh-system
  labels:
    kuma.io/mesh: default # optional, defaults to `default` if unset
spec:
  targetRef:
    kind: MeshService
    name: web
  to:
    - targetRef:
        kind: MeshService
        name: backend
      default:
        http:
          numRetries: 10
          backOff:
            baseInterval: 15s
            maxInterval: 20m
          retryOn:
            - "5xx"

Apply the configuration with kubectl apply -f [..].

type: MeshRetry
name: web-to-backend-retry-http
mesh: default
spec:
  targetRef:
    kind: MeshService
    name: web
  to:
    - targetRef:
        kind: MeshService
        name: backend
      default:
        http:
          numRetries: 10
          backOff:
            baseInterval: 15s
            maxInterval: 20m
          retryOn:
            - "5xx"

Apply the configuration with kumactl apply -f [..] or with the HTTP API.

gRPC web to backend on DeadlineExceeded

Kubernetes
Universal
apiVersion: kuma.io/v1alpha1
kind: MeshRetry
metadata:
  name: web-to-backend-retry-grpc
  namespace: kong-mesh-system
  labels:
    kuma.io/mesh: default # optional, defaults to `default` if unset
spec:
  targetRef:
    kind: MeshService
    name: web
  to:
    - targetRef:
        kind: MeshService
        name: backend
      default:
        grpc:
          numRetries: 5
          backOff:
            baseInterval: 5s
            maxInterval: 1m
          retryOn:
            - "DeadlineExceeded"

Apply the configuration with kubectl apply -f [..].

type: MeshRetry
name: web-to-backend-retry-grpc
mesh: default
spec:
  targetRef:
    kind: MeshService
    name: web
  to:
    - targetRef:
        kind: MeshService
        name: backend
      default:
        grpc:
          numRetries: 5
          backOff:
            baseInterval: 5s
            maxInterval: 1m
          retryOn:
            - "DeadlineExceeded"

Apply the configuration with kumactl apply -f [..] or with the HTTP API.

TCP web to backend

Kubernetes
Universal
apiVersion: kuma.io/v1alpha1
kind: MeshRetry
metadata:
  name: web-to-backend-retry-tcp
  namespace: kong-mesh-system
  labels:
    kuma.io/mesh: default # optional, defaults to `default` if unset
spec:
  targetRef:
    kind: MeshService
    name: web
  to:
    - targetRef:
        kind: MeshService
        name: backend
      default:
        tcp:
          maxConnectAttempt: 5

Apply the configuration with kubectl apply -f [..].

type: MeshRetry
name: web-to-backend-retry-tcp
mesh: default
spec:
  targetRef:
    kind: MeshService
    name: web
  to:
    - targetRef:
        kind: MeshService
        name: backend
      default:
        tcp:
          maxConnectAttempt: 5

Apply the configuration with kumactl apply -f [..] or with the HTTP API.

Thank you for your feedback.
Was this page useful?
  • Kong
    THE CLOUD CONNECTIVITY COMPANY

    Kong powers reliable digital connections across APIs, hybrid and multi-cloud environments.

    • Company
    • Customers
    • Events
    • Investors
    • Careers Hiring!
    • Partners
    • Press
    • Contact
  • Products
    • Kong Konnect
    • Kong Gateway
    • Kong Mesh
    • Get Started
    • Pricing
  • Resources
    • eBooks
    • Webinars
    • Briefs
    • Blog
    • API Gateway
    • Microservices
  • Open Source
    • Install Kong Gateway
    • Kong Community
    • Kubernetes Ingress
    • Kuma
    • Insomnia
  • Solutions
    • Decentralize
    • Secure & Govern
    • Create a Dev Platform
    • API Gateway
    • Kubernetes
    • Service Mesh
Star
  • Terms•Privacy
© Kong Inc. 2023