Multi-zone authentication

To add to the security of your deployments, Kong Mesh provides token generation for authenticating remote control planes to the global control plane.

The control plane token is a JWT that contains:

• The name of the zone the token is generated for
• The token’s serial number, used for token rotation

The control plane token is signed by a signing key that is autogenerated on the global control plane. The signing key is SHA256 encrypted.

You can check for the signing key:

$ kumactl get global-secrets

which returns something like:

NAME                             AGE
control-plane-signing-key-0001   36m

Set up tokens

To generate the tokens you need and configure your clusters:

• Generate a token for each remote control plane.
• Add the token to the configuration for each remote zone.
• Enable authentication on the global control plane.

Generate token for each remote zone

On the global control plane, authenticate and run the following command:

$ kumactl generate control-plane-token --zone=west > /tmp/token
$ cat /tmp/token

The generated token looks like:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJab25lIjoid2VzdCIsIlRva2VuU2VyaWFsTnVtYmVyIjoxfQ.kIrS5W0CPMkEVhuRXcUxk3F_uUoeI3XK1Gw-uguWMpQ

For authentication to the global control plane on Kubernetes, you can port-forward port 5681 to access the API.

Add token to each remote configuration

$ kumactl install control-plane \
  --mode=remote \
  --zone=<zone name> \
  --cp-token-path=/tmp/token \
  --ingress-enabled \
  --kds-global-address grpcs://`<global-kds-address>` | kubectl apply -f - 

$ kubectl create secret generic cp-token -n kong-mesh-system --from-file=/tmp/token

kuma:
  controlPlane:
    secrets:
      - Env: "KMESH_MULTIZONE_REMOTE_KDS_AUTH_CP_TOKEN_INLINE"
        Secret: "cp-token"
        Key: "token"

$ KUMA_MODE=remote \
  KUMA_MULTIZONE_REMOTE_ZONE=<zone-name> \
  KUMA_MULTIZONE_REMOTE_GLOBAL_ADDRESS=grpcs://<global-kds-address> \
  KMESH_MULTIZONE_REMOTE_KDS_AUTH_CP_TOKEN_INLINE="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJab25lIjoid2VzdCIsIlRva2VuU2VyaWFsTnVtYmVyIjoxfQ.kIrS5W0CPMkEVhuRXcUxk3F_uUoeI3XK1Gw-uguWMpQ" \
  ./kuma-cp run

Enable authentication on the global control plane

If you are starting from scratch and not securing existing Kong Mesh deployment, you can do this as a first step.

$ kumactl install control-plane \
  --mode=global \
  --cp-auth=cpToken | kubectl apply -f -

kuma:
  controlPlane:
    envVars:
      KMESH_MULTIZONE_GLOBAL_KDS_AUTH_TYPE: cpToken

$ KUMA_MODE=global \
  KMESH_MULTIZONE_GLOBAL_KDS_AUTH_TYPE=cpToken \
  ./kuma-cp run

Verify the remote control plane is connected with authentication by looking at the global control plane logs:

2021-02-24T14:30:38.596+0100	INFO	kds.auth	Remote CP successfully authenticated using Control Plane Token	{"tokenSerialNumber": 1, "zone": "cluster-2"}

Rotate tokens

If a control plane token or signing key is compromised, you must rotate all tokens.

Generate new signing key

The signing key is stored as a GlobalSecret with a name that looks like control-plane-signing-key-{serialNumber}.

Make sure to generate the new signing key with a serial number greater than the serial number of the current signing key.

$ kubectl get secrets -n kong-mesh-system --field-selector='type=system.kuma.io/global-secret'
NAME                             TYPE                           DATA   AGE
control-plane-signing-key-0001   system.kuma.io/global-secret   1      25m

$ TOKEN="$(kumactl generate signing-key)" && echo "
apiVersion: v1
data:
  value: $TOKEN
kind: Secret
metadata:
  name: control-plane-signing-key-0002
  namespace: kong-mesh-system
type: system.kuma.io/global-secret
" | kubectl apply -f - 

$ kumactl get global-secrets
NAME                             AGE
control-plane-signing-key-0001   36m

echo "
type: GlobalSecret
name: control-plane-signing-key-0002
data: 
" | kumactl apply --var key=$(kumactl generate signing-key) -f -

Regenerate control plane tokens

Create and add a new token for each remote control plane. These tokens are automatically created with the signing key that’s assigned the highest serial number, so they’re created with the new signing key.

Make sure the new signing key is available; otherwise old and new tokens are created with the same signing key and can both provide authentication.

Remove the old signing key

$ kubectl delete secret control-plane-signing-key-0001 -n kong-mesh-system

$ kumactl delete global-secret control-plane-signing-key-0001

All new connections to the global control plane now require tokens signed with the new signing key.

Restart the global control plane

Restart all instances of the global control plane. All connections are now authenticated with the new tokens.

Explore an example token

You can decode the tokens to validate the signature or explore details.

For example, run:

$ kumactl generate control-plane-token --zone=west

which returns:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJab25lIjoid2VzdCIsIlRva2VuU2VyaWFsTnVtYmVyIjoxfQ.kIrS5W0CPMkEVhuRXcUxk3F_uUoeI3XK1Gw-uguWMpQ

Paste the token into the UI at jwt.io, or run

$ kumactl generate control-plane-token --zone=west | jwt

The result looks like:

Additional security

By default, a connection from the remote control plane to the global control plane is secured with TLS. You should also configure the remote control plane to verify the certificate authority (CA) of the global control plane.