Configure Service health checks

Uses: Kong Ingress Controller

Deployment Platform:

Prerequisites

Kong Konnect

If you don’t have a Konnect account, you can get started quickly with our onboarding wizard.

The following Konnect items are required to complete this tutorial:
- Personal access token (PAT): Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.
Set the personal access token as an environment variable:
```
export KONNECT_TOKEN='YOUR KONNECT TOKEN'
```
Copied to clipboard!

Enable the Gateway API

Install the Gateway API CRDs before installing Kong Ingress Controller.

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/standard-install.yaml

Copied to clipboard!

Create a Gateway and GatewayClass instance to use.

echo "
apiVersion: v1
kind: Namespace
metadata:
  name: kong
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: kong
  annotations:
    konghq.com/gatewayclass-unmanaged: 'true'
spec:
  controllerName: konghq.com/kic-gateway-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kong
spec:
  gatewayClassName: kong
  listeners:
  - name: proxy
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
         from: All
" | kubectl apply -n kong -f -

      
        
      
    
Copied to clipboard!

Create a KIC Control Plane

Use the Konnect API to create a new CLUSTER_TYPE_K8S_INGRESS_CONTROLLER Control Plane:

CONTROL_PLANE_DETAILS=$(curl -X POST "https://us.api.konghq.com/v2/control-planes" \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "name": "My KIC CP",
       "cluster_type": "CLUSTER_TYPE_K8S_INGRESS_CONTROLLER"
     }')

Copied to clipboard!

We’ll need the id and telemetry_endpoint for the values.yaml file later. Save them as environment variables:

CONTROL_PLANE_ID=$(echo $CONTROL_PLANE_DETAILS | jq -r .id)
CONTROL_PLANE_TELEMETRY=$(echo $CONTROL_PLANE_DETAILS | jq -r '.config.telemetry_endpoint | sub("https://";"")')

Copied to clipboard!

Create mTLS certificates

Kong Ingress Controller talks to Konnect over a connected secured with TLS certificates.

Generate a new certificate using openssl:

openssl req -new -x509 -nodes -newkey rsa:2048 -subj "/CN=kongdp/C=US" -keyout ./tls.key -out ./tls.crt

Copied to clipboard!

The certificate needs to be a single line string to send it to the Konnect API with curl. Use awk to format the certificate:

export CERT=$(awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' tls.crt);

Copied to clipboard!

Next, upload the certificate to Konnect:

curl -X POST "https://us.api.konghq.com/v2/control-planes/$CONTROL_PLANE_ID/dp-client-certificates" \
     -H "Authorization: Bearer $KONNECT_TOKEN" \
     --json '{
       "cert": "'$CERT'"
     }'

Copied to clipboard!

Finally, store the certificate in a Kubernetes secret so that Kong Ingress Controller can read it:

kubectl create namespace kong -o yaml --dry-run=client | kubectl apply -f -
kubectl create secret tls konnect-client-tls -n kong --cert=./tls.crt --key=./tls.key

Copied to clipboard!

Kong Ingress Controller running (attached to Konnect)

Add the Kong Helm charts:

helm repo add kong https://charts.konghq.com
helm repo update

Copied to clipboard!

Create a values.yaml file:

cat <<EOF > values.yaml
controller:
  ingressController:
    image:
      tag: "3.4"
    env:
      feature_gates: "FillIDs=true"
    konnect:
      license:
        enabled: true
      enabled: true
      controlPlaneID: "$CONTROL_PLANE_ID"
      tlsClientCertSecretName: konnect-client-tls
      apiHostname: "us.kic.api.konghq.com"
gateway:
  image:
    repository: kong/kong-gateway
  env:
    konnect_mode: 'on'
    vitals: "off"
    cluster_mtls: pki
    cluster_telemetry_endpoint: "$CONTROL_PLANE_TELEMETRY:443"
    cluster_telemetry_server_name: "$CONTROL_PLANE_TELEMETRY"
    cluster_cert: /etc/secrets/konnect-client-tls/tls.crt
    cluster_cert_key: /etc/secrets/konnect-client-tls/tls.key
    lua_ssl_trusted_certificate: system
    proxy_access_log: "off"
    dns_stale_ttl: "3600"
  secretVolumes:
     - konnect-client-tls
EOF

      
        
      
    
Copied to clipboard!

Install Kong Ingress Controller using Helm:

helm install kong kong/ingress -n kong --create-namespace --values ./values.yaml

Copied to clipboard!

Set $PROXY_IP as an environment variable for future commands:

export PROXY_IP=$(kubectl get svc --namespace kong kong-gateway-proxy -o jsonpath='{range .status.loadBalancer.ingress[0]}{@.ip}{@.hostname}{end}')
echo $PROXY_IP

Copied to clipboard!

Kong Ingress Controller running

Add the Kong Helm charts:

helm repo add kong https://charts.konghq.com
helm repo update

Copied to clipboard!

Install Kong Ingress Controller using Helm:

helm install kong kong/ingress -n kong --create-namespace

Copied to clipboard!

Set $PROXY_IP as an environment variable for future commands:

export PROXY_IP=$(kubectl get svc --namespace kong kong-gateway-proxy -o jsonpath='{range .status.loadBalancer.ingress[0]}{@.ip}{@.hostname}{end}')
echo $PROXY_IP

Copied to clipboard!

Required Kubernetes resources

This how-to requires some Kubernetes services to be available in your cluster. These services will be used by the resources created in this how-to.

kubectl apply -f https://developer.konghq.com/manifests/kic/httpbin-service.yaml -n kong

Copied to clipboard!

This how-to also requires 1 pre-configured route:

echo "
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin
  namespace: kong
  annotations:
    konghq.com/strip-path: 'true'
spec:
  parentRefs:
  - name: kong
    namespace: kong
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /httpbin
    backendRefs:
    - name: httpbin
      kind: Service
      port: 80
" | kubectl apply -f -

      
        
      
    
Copied to clipboard!

echo "
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  namespace: kong
  annotations:
    konghq.com/strip-path: 'true'
spec:
  ingressClassName: kong
  rules:
  - http:
      paths:
      - path: /httpbin
        pathType: ImplementationSpecific
        backend:
          service:
            name: httpbin
            port:
              number: 80
" | kubectl apply -f -

      
        
      
    
Copied to clipboard!

Health check types

Kong Gateway supports active and passive health checks. This allows Kong Gateway to automatically short-circuit requests to specific Pods that are misbehaving in your Kubernetes Cluster. The process to re-enable these pods is different between active and passive health checks.

Passive health checks

Pods that are marked as unhealthy by Kong Gateway are permanently marked as unhealthy.

If a passive health check for a service that runs in a cluster and if the Pod that runs the service reports an error, Kong Gateway returns a 503, indicating that the service is unavailable. Kong Gateway doesn’t proxy any requests to the unhealthy pod.

There is no way to mark the pod as healthy again using Kong Ingress Controller and passive health checks. To resolve the issue, choose on of the following options:

Delete the current Pod: Kong Gateway then sends proxy requests to the new Pod that is in its place.
Scale the deployment: Kong Gateway then sends proxy requests to the new Pods and leaves the short-circuited Pod out of the loop.

Active health checks

Pods that are marked as unhealthy by Kong Gateway are temporarily marked as unhealthy.

Kong Gateway will make a request to the healthcheck path periodically. When it has received enough healthy responses, it will re-enable the Pod in the load balancer and traffic will be routed to the Pod again.

Enable passive health checking

All health checks are done at the Service-level. To configure Kong Gateway to short-circuit requests to a Pod if it throws 3 consecutive errors, add a KongUpstreamPolicy resource:

echo '
apiVersion: configuration.konghq.com/v1beta1
kind: KongUpstreamPolicy
metadata:
    name: demo-health-checking
    namespace: kong
spec:
  healthchecks:
    passive:
      healthy:
        successes: 3
      unhealthy:
        httpFailures: 3
' | kubectl apply -f -

      
        
      
    
Copied to clipboard!

Associate the KongUpstreamPolicy resource with httpbin Service:

 kubectl patch -n kong svc httpbin -p '{"metadata":{"annotations":{"konghq.com/upstream-policy":"demo-health-checking"}}}'

Copied to clipboard!

Test the Ingress rule by sending two requests to /status/500 that simulate a failure from the upstream service:

 curl -i "$PROXY_IP/httpbin/status/500"
 curl -i "$PROXY_IP/httpbin/status/500"

Copied to clipboard!

The results should look like this:

 HTTP/1.1 500 INTERNAL SERVER ERROR
 Content-Type: text/html; charset=utf-8
 Content-Length: 0
 Connection: keep-alive
 Server: gunicorn/19.9.0
 Access-Control-Allow-Origin: *
 Access-Control-Allow-Credentials: true
 X-Kong-Upstream-Latency: 1
 X-Kong-Proxy-Latency: 0
 Via: kong/3.11

      
        
      
    
Copied to clipboard!

Send a third request with status/200. This will reset the circuit breaker counter as it is a healthy response:

 curl -i "$PROXY_IP/httpbin/status/200"

Copied to clipboard!

The results should look like this:

 HTTP/1.1 200 OK
 Content-Type: text/html; charset=utf-8
 Content-Length: 0
 Connection: keep-alive
 Server: gunicorn/19.9.0
 Access-Control-Allow-Origin: *
 Access-Control-Allow-Credentials: true
 X-Kong-Upstream-Latency: 1
 X-Kong-Proxy-Latency: 0
 Via: kong/3.11

      
        
      
    
Copied to clipboard!

Kong Gateway didn’t short-circuit because there were only two failures.

Trip the circuit breaker

Send three requests to /status/500 to mark the pod as unhealthy. We need three requests as this is the number provided in unhealthy.httpFailures in the KongUpstreamPolicy resource:

 curl -i "$PROXY_IP/httpbin/status/500"
 curl -i "$PROXY_IP/httpbin/status/500"
 curl -i "$PROXY_IP/httpbin/status/500"

Copied to clipboard!

The results should look like this:

 HTTP/1.1 500 INTERNAL SERVER ERROR
 Content-Type: text/html; charset=utf-8
 Content-Length: 0
 Connection: keep-alive
 Server: gunicorn/19.9.0
 Access-Control-Allow-Origin: *
 Access-Control-Allow-Credentials: true
 X-Kong-Upstream-Latency: 1
 X-Kong-Proxy-Latency: 0
 Via: kong/3.11

      
        
      
    
Copied to clipboard!

Make a request to /status/200 and note that Kong Gateway returns an HTTP 503:
```
 curl -i "$PROXY_IP/httpbin/status/200"
```
Copied to clipboard!
```
 curl -i "$PROXY_IP/httpbin/status/200"
```
Copied to clipboard!
The results should look like this:
```
 HTTP/1.1 503 Service Temporarily Unavailable
 Content-Type: application/json; charset=utf-8
 Connection: keep-alive
 Content-Length: 62
 X-Kong-Response-Latency: 0
 Server: kong/3.11

 {
   "message":"failure to get a peer from the ring-balancer"
 }%
```
Copied to clipboard!
Because there’s only one Pod of httpbin Service running in the cluster, and that is throwing errors, Kong Gateway doesn’t proxy any additional requests. To get resolve this, you can use active health-check, where each instance of Kong Gateway actively probes Pods to check if they are healthy.

Enable active health checking

Active health checking can automatically mark an upstream service as healthy again once it receives enough healthy responses.

Update the KongUpstreamPolicy resource to use active health checks:

echo '
apiVersion: configuration.konghq.com/v1beta1
kind: KongUpstreamPolicy
metadata:
    name: demo-health-checking
    namespace: kong
spec:
  healthchecks:
    active:
      healthy:
        interval: 5
        successes: 3
      httpPath: /status/200
      type: http
      unhealthy:
        httpFailures: 1
        interval: 5
    passive:
      healthy:
        successes: 3
      unhealthy:
        httpFailures: 3
' | kubectl apply -f -

      
        
      
    
Copied to clipboard!

This configures Kong Gateway to actively probe /status/200 every five seconds. If a Pod is unhealthy from Kong Gateway’s perspective, three successful probes change the status of the Pod to healthy, and Kong Gateway again starts to forward requests to that Pod. Wait 15 seconds for the pod to be marked as healthy before continuing.

Make a request to /status/200 after 15 seconds:

 sleep 15 && curl -i "$PROXY_IP/httpbin/status/200"

Copied to clipboard!

The results should look like this:

 HTTP/1.1 200 OK
 Content-Type: text/html; charset=utf-8
 Content-Length: 0
 Connection: keep-alive
 Server: gunicorn/19.9.0
 Access-Control-Allow-Origin: *
 Access-Control-Allow-Credentials: true
 X-Kong-Upstream-Latency: 1
 X-Kong-Proxy-Latency: 1
 Via: kong/3.11

      
        
      
    
Copied to clipboard!

Trip the circuit again by sending three requests that return the HTTP 500 from httpbin:

curl -i "$PROXY_IP/httpbin/status/500"
curl -i "$PROXY_IP/httpbin/status/500"
curl -i "$PROXY_IP/httpbin/status/500"

Copied to clipboard!

The httpbin pod is now marked as unhealthy for 15 seconds. This is the duration required for active health checks to re-classify the httpbin Pod as healthy again (three requests with a five second interval).

curl -i $PROXY_IP/httpbin/status/200

Copied to clipboard!

The results should look like this:

HTTP/1.1 503 Service Temporarily Unavailable
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Content-Length: 62
X-Kong-Response-Latency: 0
Server: kong/3.11

{
  "message":"failure to get a peer from the ring-balancer"
}%

      
        
      
    
Copied to clipboard!

Wait 15 seconds then make another request:

sleep 15 && curl -i "$PROXY_IP/httpbin/status/200"

Copied to clipboard!

The results should look like this:

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-Kong-Upstream-Latency: 1
X-Kong-Proxy-Latency: 1
Via: kong/3.11

      
        
      
    
Copied to clipboard!

Active health checking has marked the upstream healthy again in Kong Gateway

Cleanup

Delete created Kubernetes resources

kubectl delete -n kong -f https://developer.konghq.com/manifests/kic/httpbin-service.yaml

Copied to clipboard!

Configure Service health checks

Prerequisites

Kong Konnect

Enable the Gateway API

Create a KIC Control Plane

Create mTLS certificates

Kong Ingress Controller running (attached to Konnect)

Kong Ingress Controller running

Required Kubernetes resources

Health check types

Passive health checks

Active health checks

Enable passive health checking

Trip the circuit breaker

Enable active health checking

Cleanup

Delete created Kubernetes resources

Uninstall KIC from your cluster

Next Steps

Did this doc help?

Help us make these docs great!

Still need help