Verify Upstream TLS
You can configure Kong Gateway to verify the certificate it presents by attaching a CA certificate to a Service. This guide shows how to achieve this using the BackendTLSPolicy
(when using Gateway API) or using Kubernetes Service annotations (when using Ingress API).
Prerequisites
Kong Konnect
If you don’t have a Konnect account, you can get started quickly with our onboarding wizard.
- The following Konnect items are required to complete this tutorial:
- Personal access token (PAT): Create a new personal access token by opening the Konnect PAT page and selecting Generate Token.
-
Set the personal access token as an environment variable:
export KONNECT_TOKEN='YOUR KONNECT TOKEN'
Copied to clipboard!
Generate a CA Certificate
Kong Gateway can validate the certificate chain to a specific depth. To showcase all the possible configurations, create a certificate chain with a root CA, an intermediate CA, and a leaf server certificate:
mkdir certs && cd certs
cd certs
openssl req -new -newkey rsa:2048 -nodes -keyout root.key -subj "/CN=root" -x509 -days 365 -out root.crt
openssl req -new -newkey rsa:2048 -nodes -keyout inter.key -subj "/CN=inter" -out inter.csr
openssl x509 -req -in inter.csr -CA root.crt -CAkey root.key -CAcreateserial -days 365 -out inter.crt -extfile <(echo "basicConstraints=CA:TRUE")
openssl req -new -newkey rsa:2048 -nodes -keyout leaf.key -subj "/CN=kong.example" -out leaf.csr
openssl x509 -req -in leaf.csr -CA inter.crt -CAkey inter.key -CAcreateserial -days 365 -out leaf.crt -extfile <(printf "subjectAltName=DNS:kong.example")
cat leaf.crt inter.crt > chain.crt
rm -f *.csr *.srl
cd ..
Running this script generates the following files in certs
directory:
-
root.key
,root.crt
: Root CA key and certificate -
inter.key
,inter.crt
: Intermediate CA key and certificate -
leaf.key
,leaf.crt
: Server key and certificate (valid forkong.example
SAN) -
chain.crt
: Server certificate chain
Configure TLS on the echo service
As part of the prerequisites, you deployed the echo
Service to your cluster. Let’s configure it to serve HTTPS. Create a secret with the server key and the certificate chain (including the intermediate certificate and the leaf certificate).
-
Create a Kubernetes secret containing the certificate:
kubectl create secret -n kong tls goecho-tls --key ./certs/leaf.key --cert ./certs/chain.crt
Copied to clipboard! -
Patch the
echo
deployment to use the secret and serve HTTPS using it:kubectl patch -n kong deployment echo -p '{ "spec": { "template": { "spec": { "containers": [ { "name": "echo", "ports": [ { "containerPort": 443 } ], "env": [ { "name": "HTTPS_PORT", "value": "443" }, { "name": "TLS_CERT_FILE", "value": "/etc/tls/tls.crt" }, { "name": "TLS_KEY_FILE", "value": "/etc/tls/tls.key" } ], "volumeMounts": [ { "mountPath": "/etc/tls", "name": "tls" } ] } ], "volumes": [ { "name": "tls", "secret": { "secretName": "goecho-tls" } } ] } } } }'
Copied to clipboard! -
Patch the Service to use HTTPS by adding the
konghq.com/protocol: https
annotation and thespec.ports
entry:kubectl patch -n kong service echo -p '{ "metadata": { "annotations": { "konghq.com/protocol": "https" } }, "spec": { "ports": [ { "name": "https", "port": 443, "targetPort": 443 } ] } }'
Copied to clipboard!
Expose the echo Service
Now that the echo
Service is serving an HTTPS endpoint, we need to expose it:
Verify connectivity by making an HTTP request to proxy. The Service serves HTTPS but Kong Gateway initiates the connection and proxies it as HTTP in this case, so the request should be made over HTTP. The Host
header has to match the hostname of the Service.
curl "$PROXY_IP/echo" \
-H "Host: kong.example"
curl "$PROXY_IP/echo" \
-H "Host: kong.example"
You should see a response similar to this:
Welcome, you are connected to node orbstack.
Running on Pod echo-bd94b7dcc-qxs2b.
In namespace default.
With IP address 192.168.194.9.
Through HTTPS connection.
That means the Service is up and running and Kong Gateway connects to it successfully over HTTPS, without verification.
Configure the root CA Certificate
Before enabling TLS verification, we need to add the root CA certificate to the Kong Gateway’s CA certificates and associate it with the Service.
Enable TLS verification
Update your Route to verify the certificate of the upstream service:
Kong Gateway is now verifying the certificate of the upstream service and accepting the connection because the certificate is trusted.
Configure verification depth
By default, Kong Gateway verifies the certificate chain up to the root CA certificate with no depth limit. You can configure the verification depth by annotating the service with the konghq.com/tls-verify-depth
annotation.
To test, set the verification depth to 0 to not allow any intermediate certificates.
Now, when you issue the same request as before, you should see an error stating that an invalid response was received from the upstream server.
By default, Kong Gateway keeps upstream connections alive for 60 seconds (
upstream_keepalive_idle_timeout
). Due to this, you may need to wait for 60 seconds to see the TLS verification fail. To speed up the process, you can restart the Kong Gateway pod.
curl "$PROXY_IP/echo" \
-H "Host: kong.example"
curl "$PROXY_IP/echo" \
-H "Host: kong.example"
{
"message":"An invalid response was received from the upstream server",
"request_id":"e2b3182856c96c23d61e880d0a28012f"
}
You can inspect Kong Gateway’s container logs to see the error.
kubectl logs -n kong deploy/kong-gateway | grep "GET /echo"
2024/11/29 11:41:46 [error] 1280#0: *45531 upstream SSL certificate verify error: (22:certificate chain too long) while SSL handshaking to upstream, client: 192.168.194.1, server: kong, request: "GET /echo HTTP/1.1", upstream: "https://192.168.194.19:443/", host: "kong.example", request_id: "678281372fb8907ed06d517cf515de78"
192.168.194.1 - - [29/Nov/2024:11:41:46 +0000] "GET /echo HTTP/1.1" 502 126 "-" "curl/8.7.1" kong_request_id: "678281372fb8907ed06d517cf515de78"
Kong Gateway is now rejecting the connection because the certificate chain is too long. Changing the verification depth to 1 should allow the connection to succeed again.
Now, when you issue the same request as before, you should see a successful response.
curl "$PROXY_IP/echo" \
-H "Host: kong.example"
curl "$PROXY_IP/echo" \
-H "Host: kong.example"
Cleanup
Delete created Kubernetes resources
kubectl delete -n kong -f https://developer.konghq.com/manifests/kic/echo-service.yaml