PermalinkHybrid Mode Deployment
PermalinkIntroduction
Traditionally, Kong has always required a database, which could be either Postgres or Cassandra, to store its configured entities such as Routes, Services and Plugins. In Kong 1.1 we added the capability to run Kong without a database, using only in-memory storage for entities: we call this DB-less mode.
In Kong 2.0, we introduced a new method of deploying Kong which is called the Hybrid mode, also known as Control Plane / Data Plane Separation (CP/DP).
In this mode Kong nodes in a cluster are separated into two roles: control plane (CP), where configuration is managed and the Admin API is served from, and data plane (DP), which serves traffic for the proxy. Each DP node is connected to one of the CP nodes. Instead of accessing the database contents directly in the traditional deployment method, the DP nodes maintains connection with CP nodes, and receives the latest configuration.
Deploying using Hybrid mode has a number of benefits:
- Drastically reduce the amount of traffic on the database, since only CP nodes need a direct connection to the database.
- Increased security, in an event where one of the DP nodes gets intruded, an attacker won’t be able to affect other nodes in the Kong cluster.
- Easiness of management, since an admin will only need to interact with the CP nodes to control and monitor the status of the entire Kong cluster.
PermalinkConfiguration Properties
Hybrid Mode introduces the following configuration properties:
Parameter | Description |
---|---|
cluster_listen Optional |
List of addresses and ports on which the control plane will listen for incoming data plane connections. Defaults to 0.0.0.0:8005 . Note this port is always protected with Mutual TLS (mTLS) encryption. Ignored on data plane nodes. |
cluster_control_plane Required |
Address and port that the data plane nodes use to connect to the control plane. Must point to the port configured using the cluster_listen property on the control plane node. Ignored on control plane nodes. |
cluster_mtls Optional |
One of "shared" or "pki" . Indicates whether Hybrid Mode will use a shared certificate/key pair for CP/DP mTLS or if PKI mode will be used. Defaults to "shared" . See below sections for differences in mTLS modes. |
The following properties are used differently between “shared” and “PKI” modes:
Parameter | Description | Shared Mode | PKI Mode |
---|---|---|---|
cluster_cert and cluster_cert_key Required |
Certificate/key pair used for mTLS between CP/DP nodes. | Same between CP/DP nodes. | Unique certificate for each node, generated from the CA specified by cluster_ca_cert . |
cluster_ca_cert Required in PKI mode |
The trusted CA certificate file in PEM format used to verify the cluster_cert . |
Ignored | CA certificate used to verify cluster_cert , same between CP/DP nodes. Required |
cluster_server_name Required in PKI mode |
The SNI Server Name presented by the DP node mTLS handshake. | Ignored | In PKI mode the DP nodes will also verify that the Common Name (CN) or Subject Alternative Name (SAN) inside certificate presented by CP matches the cluster_server_name value. |
PermalinkTopology
PermalinkGenerating Certificate/Key Pair
Before using the Hybrid Mode, it is necessary to have a shared certificate/key pair generated so that the communication security between CP and DP nodes can be established.
This certificate/key pair is shared by both CP and DP nodes, mutual TLS handshake (mTLS) is used for authentication so the actual private key is never transferred on the network.
To create a certificate/key pair:
kong hybrid gen_cert
This will generate cluster.crt
and cluster.key
files and save them to the current directory.
By default it is valid for 3 years, but can be set longer or shorter with the --days
option.
See kong hybrid --help
for more usage information.
The cluster.crt
and cluster.key
files need to be transferred to both Kong CP and DP nodes.
Observe proper permission setting on the key file to ensure it can only be read by Kong.
PermalinkSetting Up Kong Control Plane Nodes
Starting the control plane is fairly simple. Aside from the database configuration
which is the same as today, we need to specify the “role” of the node to “control_plane”.
This will cause Kong to listen on 0.0.0.0:8005
by default for data plane
connections. The 8005
port on the control plane will need to be
accessible by all the data plane it controls through any firewalls you may have
in place.
In addition, the cluster_cert
and cluster_cert_key
configuration need to point to
the certificate/key pair that was generated above.
KONG_ROLE=control_plane KONG_CLUSTER_CERT=cluster.crt KONG_CLUSTER_CERT_KEY=cluster.key kong start
Or in kong.conf
:
role = control_plane
cluster_cert = cluster.crt
cluster_cert_key = cluster.key
Note that control plane still needs a database (Postgres or Cassandra) to store the “source of truth” configurations, although the database never needs to be access by data plane nodes. You may run more than a single control plane nodes to provide load balancing and redundancy as long as they points to the same backend database.
PermalinkPKI mode
Starting in Kong 2.1, the Hybrid cluster can use certificates signed by a central certificate authority (CA).
This mode can be activated by setting cluster_mtls
to "pki"
in kong.conf
. The default value is "shared"
.
In PKI mode, the control plane and data plane don’t need to use the same cluster_cert
and cluster_cert_key
.
Instead, Kong validates both sides by checking if they are issued by configured CA. This eliminates the risk of
transporting private keys around.
Note: Certificates on CP and DP must contain the
TLS Web Server Authentication
andTLS Web Client Authentication
as X509v3 Extended Key Usage extension, respectively.
Kong doesn’t validate the CommonName (CN) in the DP certificate; it can take an arbitrary value.
Set the following configuration parameters in kong.conf
on the control plane:
cluster_mtls = pki
cluster_ca_cert = /path/to/ca-cert.crt
cluster_cert = control-plane.crt
cluster_cert_key = control-plane.key
cluster_ca_cert
specifies the root CA certificate for cluster_cert
and cluster_cert_key
. This
certificate must be the root CA certificate and not any of an intermediate CA.
Kong allows at most 3
levels of intermediate CAs to be used between the root CA and the cluster certificate.
Set the following configuration parameters in kong.conf
on the data plane:
cluster_mtls = pki
cluster_server_name = control-plane.kong.yourcorp.tld
cluster_cert = data-plane.crt
cluster_cert_key = data-plane.crt
cluster_server_name
specifies the SNI (Server Name Indication extension) to use for data plane
connections to the control plane through TLS. When not set, data plane will use kong_clustering
as the SNI.
PermalinkRevocation check of Data Plane certificates
When Kong is running Hybrid mode with PKI mode, the Control Plane can be configured to optionally check for revocation status of the connecting Data Plane certificate.
The supported method is through Online Certificate Status Protocol (OCSP) responders. Issued data plane certificates must contain the Certificate Authority Information Access extension that references the URI of OCSP responder that can be reached from the Control Plane.
To enable OCSP checks, set the cluster_ocsp
config on the Control Plane to one of the following values:
on
: OCSP revocation check is enabled and the Data Plane must pass the revocation check to establish connection with the Control Plane. This implies that certificates without the OCSP extension or unreachable OCSP responder also prevents a connection from being established.off
: OCSP revocation check is disabled (default).optional
: OCSP revocation check will be attempted, however, if the OCSP responder URI is not found inside the Data Plane-provided certificate or communication with the OCSP responder failed, then Data Plane is still allowed through.
Note that OCSP checks are only performed on the Control Plane against certificates provided by incoming Data Plane
nodes. The cluster_ocsp
config has no effect on Data Plane nodes.
cluster_oscp
affects all Hybrid mode connections established from a Data Plane to its Control Plane.
PermalinkStarting Data Plane Nodes
Now that a control plane is running, it is not very useful if no data plane nodes are talking to it and serving traffic (remember control plane nodes can not be used for proxying). To start the data plane, all we need to do is to specify the “role” to “data_plane”, give it the address and port of where the control plane can be reached and the node automatically connects and syncs itself up with the current configuration.
Similar to the CP config above, cluster_cert
and cluster_cert_key
configuration need to
point to the same files as the CP has. In addition, the certificate from cluster_cert
(in “shared” mode) or cluster_ca_cert
(in “pki” mode) is automatically added to the trusted
chain in lua_ssl_trusted_certificate
.
Note: In this release of the Hybrid Mode, the data plane receives updates from the Control
Plane via a format that is similar to the Declarative Config, therefore the database
property has to be set to off
for Kong to start up properly.
KONG_ROLE=data_plane KONG_CLUSTER_CONTROL_PLANE=control-plane.example.com:8005 KONG_CLUSTER_CERT=cluster.crt KONG_CLUSTER_CERT_KEY=cluster.key KONG_DATABASE=off kong start
Or in kong.conf
:
role = data_plane
cluster_control_plane = control-plane.example.com:8005
database = off
cluster_cert = cluster.crt
cluster_cert_key = cluster.key
PermalinkDP Node Start Sequence
When set as a DP node, Kong Gateway processes configuration in the following order:
- Config cache: If the file
config.json.gz
exists in thekong_prefix
path (/usr/local/kong
by default), the DP node loads it as configuration. declarative_config
exists: If there is no config cache and thedeclarative_config
parameter is set, the DP node loads the specified file.- Empty config: If there is no config cache or declarative configuration file available, the node starts with empty configuration. In this state, it returns 404 to all requests.
- Contact CP Node: In all cases, the DP node contacts the CP node to retrieve
the latest configuration. If successful, it gets stored in the local config
cache (
config.json.gz
).
PermalinkChecking the status of the cluster
You may want to check the status of the Kong cluster from time to time, such as checking to see the which nodes are actively receiving config updates from control plane, or when was it last updated. This can be achieved by using the control plane’s new Cluster Status API:
# on control plane node
http :8001/clustering/data-planes
{
"data": [
{
"config_hash": "a9a166c59873245db8f1a747ba9a80a7",
"hostname": "data-plane-2",
"id": "ed58ac85-dba6-4946-999d-e8b5071607d4",
"ip": "192.168.10.3",
"last_seen": 1580623199,
"ttl": 1139376,
"version": "2.2.1",
},
{
"config_hash": "a9a166c59873245db8f1a747ba9a80a7",
"hostname": "data-plane-1",
"id": "ed58ac85-dba6-4946-999d-e8b5071607d4",
"ip": "192.168.10.4",
"last_seen": 1580623200,
"ttl": 1139377,
"version": "2.3.0",
}
],
"next": null
}
The Cluster Status API provides helpful information such as the name of the node and last time it synced with the control plane, as well as config version currently running on them.
PermalinkManaging the cluster using Control Plane nodes
Once the nodes are setup, use the Admin API on the control plane as usual, those changes will be synced and updated on the data plane nodes automatically within seconds.
PermalinkVersion compatibility
Kong Gateway (OSS) control planes only allow connections from data planes with the same major version. Control planes won’t allow connections from data planes with newer minor versions.
For example, a Kong Gateway (OSS) v2.5.2 control plane:
- Accepts a Kong Gateway (OSS) 2.5.0, 2.5.1 and 2.5.2 data plane
- Accepts a Kong Gateway (OSS) 2.3.8, 2.2.1 and 2.2.0 data plane
- Accepts a Kong Gateway (OSS) 2.5.3 data plane (newer patch version on the data plane is accepted)
- Rejects a Kong Gateway (OSS) 1.0.0 data plane (major version differs)
- Rejects a Kong Gateway (OSS) 2.6.0 data plane (minor version on data plane is newer)
Furthermore, for every plugin that is configured on the Kong Gateway (OSS) control plane, new configs are only pushed to data planes that have those configured plugins installed and loaded. The major version of those configured plugins must be the same on both the control planes and data planes. Also, the minor versions of the plugins on the data planes could not be newer than versions installed on the control planes. Note that similar to Kong Gateway (OSS) version checks, plugin patch versions are also ignored when determining the compatibility.
Configured plugins means any plugin that is either enabled globally or configured by Services, Routes, or Consumers.
For example, if a Kong Gateway (OSS) control plane has plugin1
v1.1.1
and plugin2
v2.1.0 installed, and plugin1
is configured by a Route
object:
- It accepts Kong Gateway (OSS) data planes with
plugin1
v1.1.2, andplugin2
not installed. - It accepts Kong Gateway (OSS) data planes with
plugin1
v1.1.2,plugin2
v2.1.0, andplugin3
v9.8.1 installed. - It rejects Kong Gateway (OSS) data planes with
plugin1
v1.2.0, andplugin2
v2.1.0 installed (minor version of plugin on data plane is newer). - It accepts Kong Gateway (OSS) data planes with
plugin1
v1.1.1, andplugin3
v9.8.1 installed. - It rejects Kong Gateway (OSS) data planes with
plugin1
not installed (plugin configured on control plane but not installed on data plane).
If the compatibility checks fail, the control plane stops pushing out new configs to the incompatible data planes to avoid breaking them.
When new configs are sent from control planes to data planes, the data planes run their own schema validator against a control plane supplied DB-less config. Data planes reject any config that does not pass the schema validation.
One exception to the validation on data planes is that if the value of a control plane
supplied field is null
and the data plane’s schema does not have information about
this null
valued field, then the field is simply ignored.
If a config can not be pushed to a data plane due to failure of the
compatibility checks, the control plane will contain warn
level lines in the
error.log
similar to the following:
unable to send updated configuration to DP node with hostname: localhost.localdomain ip: 127.0.0.1 reason: version mismatches, CP version: 2.2 DP version: 2.1
unable to send updated configuration to DP node with hostname: localhost.localdomain ip: 127.0.0.1 reason: CP and DP does not have same set of plugins installed or their versions might differ
In addition, the /clustering/data-planes
Admin API endpoint will return
the version of the data plane node and the latest config hash the node is
using. These will help detect version incompatibilities from the
control plane side.
PermalinkFault tolerance
A valid question you may ask is: What would happen if control plane nodes are down, will the data plane keep functioning? The answer is yes. Data plane caches the latest configuration it received from the control plane on the local disk. In case the control plane stops working, the data plane will keep serving requests using cached configurations. It does so while constantly trying to reestablish communication with the control plane.
This means that the Data Plane nodes can be stopped even for extended periods of time, and the Data Plane will still proxy traffic normally. Data Plane nodes can be restarted while in disconnected mode, and will load the last configuration in the cache to start working. When the Control Plane is brought up again, the Data Plane nodes will contact them and resume connected mode.
PermalinkDisconnected Mode
The viability of the Data Plane while disconnected means that Control Plane updates or database restores can be done with peace of mind. First bring down the Control Plane, perform all required downtime processes, and only bring up the Control Plane after verifying the success and correctness of the procedure. During that time, the Data Plane will keep working with the latest configuration.
A new Data Plane node can be provisioned during Control Plane downtime. This
requires either copying the config cache file (config.json.gz
) from another
Data Plane node, or using a declarative configuration. In either case, if it
has the role of "data_plane"
, it will also keep trying to contact the Control
Plane until it’s up again.
To change a disconnected Data Plane node’s configuration, you have to remove
the config cache file (config.json.gz
), ensure the declarative_config
parameter or the KONG_DECLARATIVE_CONFIG
environment variable is set, and set
the whole configuration in the referenced YAML file.
PermalinkLimitations
PermalinkPlugins compatibility
This version of Hybrid Mode uses declarative config as the config sync format which means it has the same limitations as declarative config as of today. Please refer to the Plugin Compatibility section of declarative config documentation for more information.
PermalinkReadonly Status API endpoints on Data Plane
Several readonly endpoints from the Admin API are exposed to the Status API on data planes, including the following:
- GET /upstreams/{upstream}/targets/
- GET /upstreams/{upstream}/health/
- GET /upstreams/{upstream}/targets/all/
- GET /upstreams/{upstream}/targets/{target}
Please refer to Upstream objects in the Admin API documentation for more information about the endpoints.