MeshRetry

This policy uses new policy matching algorithm. Do not combine with Retry.

This policy enables Kong Mesh to know how to behave if there are failed requests which could be retried.

TargetRef support matrix

`targetRef`	Allowed kinds
`targetRef.kind`	`Mesh`, `Dataplane`, `MeshSubset(deprecated)`
`to[].targetRef.kind`	`Mesh`, `MeshService`, `MeshExternalService`

`targetRef`	Allowed kinds
`targetRef.kind`	`Mesh`, `MeshGateway`, `MeshGateway` with listener `tags`
`to[].targetRef.kind`	`Mesh`

`targetRef`	Allowed kinds
`targetRef.kind`	`Mesh`, `MeshSubset`
`to[].targetRef.kind`	`Mesh`, `MeshService`, `MeshExternalService`

To learn more about the information in this table, see the matching docs.

Configuration

The policy let you configure retry behaviour for HTTP, GRPC and TCP protocols. The protocol is selected by picking the most specific protocol.

Each protocol has a separate section under default in the policy yaml. Some sections are common between protocols or have similar meaning.

Retry on

The field retryOn is a list of conditions which will cause a retry.

For HTTP these are related to the response status code or method (5xx, 429, HttpMethodGet). For gRPC these are status codes in response headers (canceled, deadline-exceeded, etc.). There is no equivalent for TCP.

One or more conditions can be specified, for example:

retryOn:
  - "429"
  - "503"

means that it the policy will retry on a status code 429 or 503.

Full list of available HTTP conditions:

retryOn:
  - 5XX
  - GatewayError
  - Reset
  - Retriable4xx
  - ConnectFailure
  - EnvoyRatelimited
  - RefusedStream
  - Http3PostConnectFailure
  - HttpMethodConnect
  - HttpMethodDelete
  - HttpMethodGet
  - HttpMethodHead
  - HttpMethodOptions
  - HttpMethodPatch
  - HttpMethodPost
  - HttpMethodPut
  - HttpMethodTrace
  - "429" # any HTTP status code
  - "503"

Full list of available gRPC conditions:

retryOn:
  - Canceled
  - DeadlineExceeded
  - Internal
  - ResourceExhausted
  - Unavailable

Back off

This parameter is applicable to both HTTP and GRPC.

It consists of BaseInterval (the amount of time between retries) and MaxInterval (the maximal amount of time taken between retries).

We use an exponential back-off algorithm with jitter for retries. Given a base interval B and retry number N, the back-off for the retry is in the range [0, (2^N - 1) × B).

For example, given a 25 ms interval, the first retry will be delayed randomly by 0-24 ms, the second by 0-74 ms, the third by 0-174 ms, and so on.

The interval is capped at a MaxInterval, which defaults to 10 times the BaseInterval.

Rate limited back off

This parameter is applicable to both HTTP and GRPC.

MeshRetry can be configured in such a way that when the upstream server rate limits the request and responds with a header like retry-after or x-ratelimit-reset it uses the value from the header to determine when to send the retry request instead of the back off algorithm.

Example

Given this configuration:

retryOn:
  - "503"
rateLimitedBackOff:
  resetHeaders:
    - name: retry-after
      format: Seconds
    - name: x-ratelimit-reset
      format: UnixTimestamp

and an HTTP response:

HTTP/1.1 503 Service Unavailable
retry-after: 15

The retry request will be issued after 15 seconds.

If the response is as follows:

HTTP/1.1 503 Service Unavailable
x-ratelimit-reset: 1706096119

The request will be retried at Wed Jan 24 2024 11:35:19 GMT+0000.

If the response does not contain retry-after or x-ratelimit-reset header (with valid integer value) then the amount of time to wait before issuing a request is determined by back off algorithm.

Examples

HTTP frontend to backend on 5xx

Kubernetes

Universal

Terraform

I am using MeshService

apiVersion: kuma.io/v1alpha1
kind: MeshRetry
metadata:
  name: frontend-to-backend-retry-http
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend_kuma-demo_svc_8080
    default:
      http:
        numRetries: 10
        backOff:
          baseInterval: 15s
          maxInterval: 20m
        retryOn:
        - 5xx

apiVersion: kuma.io/v1alpha1
kind: MeshRetry
metadata:
  name: frontend-to-backend-retry-http
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend
      namespace: kuma-demo
      sectionName: http
    default:
      http:
        numRetries: 10
        backOff:
          baseInterval: 15s
          maxInterval: 20m
        retryOn:
        - 5xx

I am using MeshService

type: MeshRetry
name: frontend-to-backend-retry-http
mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend
    default:
      http:
        numRetries: 10
        backOff:
          baseInterval: 15s
          maxInterval: 20m
        retryOn:
        - 5xx

type: MeshRetry
name: frontend-to-backend-retry-http
mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend
      sectionName: http
    default:
      http:
        numRetries: 10
        backOff:
          baseInterval: 15s
          maxInterval: 20m
        retryOn:
        - 5xx

Please adjust konnect_mesh_control_plane.my_meshcontrolplane.id and konnect_mesh.my_mesh.name according to your current configuration

resource "konnect_mesh_retry" "frontend_to_backend_retry_http" {
  provider = konnect-beta
  type = "MeshRetry"
  name = "frontend-to-backend-retry-http"
  spec = {
    target_ref = {
      kind = "Dataplane"
      labels = {
        app = "frontend"
      }
    }
    to = [
      {
        target_ref = {
          kind = "MeshService"
          name = "backend"
          section_name = "http"
        }
        default = {
          http = {
            num_retries = "10"
            back_off = {
              base_interval = "15s"
              max_interval = "20m"
            }
            retry_on = [
               = "5xx"
            ]
          }
        }
      }
    ]
  }
  labels   = {
    "kuma.io/mesh" = konnect_mesh.my_mesh.name
  }
  cp_id    = konnect_mesh_control_plane.my_meshcontrolplane.id
  mesh     = konnect_mesh.my_mesh.name
}

gRPC frontend to backend on DeadlineExceeded

Kubernetes

Universal

Terraform

I am using MeshService

apiVersion: kuma.io/v1alpha1
kind: MeshRetry
metadata:
  name: frontend-to-backend-retry-grpc
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    tags:
      app: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend_kuma-demo_svc_8080
    default:
      grpc:
        numRetries: 5
        backOff:
          baseInterval: 5s
          maxInterval: 1m
        retryOn:
        - DeadlineExceeded

apiVersion: kuma.io/v1alpha1
kind: MeshRetry
metadata:
  name: frontend-to-backend-retry-grpc
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    tags:
      app: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend
      namespace: kuma-demo
      sectionName: http
    default:
      grpc:
        numRetries: 5
        backOff:
          baseInterval: 5s
          maxInterval: 1m
        retryOn:
        - DeadlineExceeded

I am using MeshService

type: MeshRetry
name: frontend-to-backend-retry-grpc
mesh: default
spec:
  targetRef:
    kind: Dataplane
    tags:
      app: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend
    default:
      grpc:
        numRetries: 5
        backOff:
          baseInterval: 5s
          maxInterval: 1m
        retryOn:
        - DeadlineExceeded

type: MeshRetry
name: frontend-to-backend-retry-grpc
mesh: default
spec:
  targetRef:
    kind: Dataplane
    tags:
      app: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend
      sectionName: http
    default:
      grpc:
        numRetries: 5
        backOff:
          baseInterval: 5s
          maxInterval: 1m
        retryOn:
        - DeadlineExceeded

Please adjust konnect_mesh_control_plane.my_meshcontrolplane.id and konnect_mesh.my_mesh.name according to your current configuration

resource "konnect_mesh_retry" "frontend_to_backend_retry_grpc" {
  provider = konnect-beta
  type = "MeshRetry"
  name = "frontend-to-backend-retry-grpc"
  spec = {
    target_ref = {
      kind = "Dataplane"
      tags = {
        app = "frontend"
      }
    }
    to = [
      {
        target_ref = {
          kind = "MeshService"
          name = "backend"
          section_name = "http"
        }
        default = {
          grpc = {
            num_retries = "5"
            back_off = {
              base_interval = "5s"
              max_interval = "1m"
            }
            retry_on = [
               = "DeadlineExceeded"
            ]
          }
        }
      }
    ]
  }
  labels   = {
    "kuma.io/mesh" = konnect_mesh.my_mesh.name
  }
  cp_id    = konnect_mesh_control_plane.my_meshcontrolplane.id
  mesh     = konnect_mesh.my_mesh.name
}

TCP frontend to backend

Kubernetes

Universal

Terraform

I am using MeshService

apiVersion: kuma.io/v1alpha1
kind: MeshRetry
metadata:
  name: frontend-to-backend-retry-tcp
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend_kuma-demo_svc_8080
    default:
      tcp:
        maxConnectAttempt: 5

apiVersion: kuma.io/v1alpha1
kind: MeshRetry
metadata:
  name: frontend-to-backend-retry-tcp
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend
      namespace: kuma-demo
      sectionName: http
    default:
      tcp:
        maxConnectAttempt: 5

I am using MeshService

type: MeshRetry
name: frontend-to-backend-retry-tcp
mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend
    default:
      tcp:
        maxConnectAttempt: 5

type: MeshRetry
name: frontend-to-backend-retry-tcp
mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend
      sectionName: http
    default:
      tcp:
        maxConnectAttempt: 5

Please adjust konnect_mesh_control_plane.my_meshcontrolplane.id and konnect_mesh.my_mesh.name according to your current configuration

resource "konnect_mesh_retry" "frontend_to_backend_retry_tcp" {
  provider = konnect-beta
  type = "MeshRetry"
  name = "frontend-to-backend-retry-tcp"
  spec = {
    target_ref = {
      kind = "Dataplane"
      labels = {
        app = "frontend"
      }
    }
    to = [
      {
        target_ref = {
          kind = "MeshService"
          name = "backend"
          section_name = "http"
        }
        default = {
          tcp = {
            max_connect_attempt = "5"
          }
        }
      }
    ]
  }
  labels   = {
    "kuma.io/mesh" = konnect_mesh.my_mesh.name
  }
  cp_id    = konnect_mesh_control_plane.my_meshcontrolplane.id
  mesh     = konnect_mesh.my_mesh.name
}

MeshRetry

TargetRef support matrix

Configuration

Retry on

Back off

Rate limited back off

Example

Examples

HTTP frontend to backend on 5xx

gRPC frontend to backend on DeadlineExceeded

TCP frontend to backend

All policy options