Data plane on Universal
As mentioned previously in universal you need to create a dataplane definition and pass it to the kuma-dp run
command.
When transparent proxying is not enabled, the outbound service dependencies have to be manually specified in the Dataplane
entity.
This also means that without transparent proxying you must update your codebases to consume those external services on 127.0.0.1
on the port specified in the outbound
section.
To avoid users bypassing the sidecar, have the service listen only on the internal interface (
127.0.0.1
or::1
) instead of all interfaces (0.0.0.0
or::
).
For example, this is how we start a Dataplane
for a hypothetical Redis service and then start the kuma-dp
process:
cat dp.yaml
type: Dataplane
mesh: default
name: redis-1
networking:
address: 23.234.0.1 # IP of the instance
inbound:
- port: 9000
servicePort: 6379
tags:
kuma.io/service: redis
kuma-dp run \
--cp-address=https://127.0.0.1:5678 \
--dataplane-file=dp.yaml
--dataplane-token-file=/tmp/kuma-dp-redis-1-token
In the example above, any external client who wants to consume Redis through the sidecar will have to use 23.234.0.1:9000
, which will redirect to the Redis service listening on address 127.0.0.1:6379
. If your service doesn’t listen on 127.0.0.1
and you can’t change the address it listens on, you can set the serviceAddress
as shown below.
type: Dataplane
...
networking:
...
inbound:
- port: 9000
serviceAddress: 192.168.1.10
servicePort: 6379
...
This configuration indicates that your service is listening on 192.168.1.10
, and incoming traffic will be redirected to that address.
Note that in Universal dataplanes need to start with a token for authentication. You can learn how to generate tokens in the security section.
Now let’s assume that we have another service called “Backend” that listens on port 80
, and that makes outgoing requests to the redis
service:
cat dp.yaml
type: Dataplane
mesh: default
name:
networking:
address:
inbound:
- port: 8000
servicePort: 80
tags:
kuma.io/service: backend
kuma.io/protocol: http
outbound:
- port: 10000
tags:
kuma.io/service: redis
kuma-dp run \
--cp-address=https://127.0.0.1:5678 \
--dataplane-file=dp.yaml \
--dataplane-var name=`hostname -s` \
--dataplane-var address=192.168.0.2 \
--dataplane-token-file=/tmp/kuma-dp-backend-1-token
In order for the backend
service to successfully consume redis
, we specify an outbound
networking section in the Dataplane
configuration instructing the DP to listen on a new port 10000
and to proxy any outgoing request on port 10000
to the redis
service.
For this to work, we must update our application to consume redis
on 127.0.0.1:10000
.
You can parametrize your
Dataplane
definition, so you can reuse the same file for manykuma-dp
instances or even services.
Lifecycle
On Universal you can manage Dataplane
resources either in Direct mode or in Indirect mode.
Direct
This is the recommended way to operate with Dataplane
resources on Universal.
Joining the mesh
Pass Dataplane
resource directly to kuma-dp run
command. Dataplane
resource could be a Mustache template in this case:
backend-dp-tmpl.yaml
type: Dataplane
mesh: default
name: { { name } }
networking:
address: { { address } }
inbound:
- port: 8000
servicePort: 80
tags:
kuma.io/service: backend
kuma.io/protocol: http
The command with template parameters will look like this:
kuma-dp run \
--dataplane-file=backend-dp-tmpl.yaml \
--dataplane-var name=my-backend-dp \
--dataplane-var address=192.168.0.2 \
...
When xDS connection between proxy and kuma-cp is established, Dataplane
resource will be created automatically by kuma-cp.
To join the mesh in a graceful way, we need to first make sure the application is ready to serve traffic before it can be considered a valid traffic destination. By default, a proxy will be considered healthy regardless of its state. Consider using service probes to mark the data plane proxy as healthy only after all health checks are passed.
Leaving the mesh
To leave the mesh in a graceful shutdown, we need to remove the traffic destination from all the clients before shutting it down.
kuma-dp
process upon receiving SIGTERM starts listener draining in Envoy, then it waits for draining time before stopping the process.
During the draining process, Envoy can still accept connections however:
- It is marked as unhealthy on Envoy Admin
/ready
endpoint - It sends
connection: close
for HTTP/1.1 requests and GOAWAY frame for HTTP/2. This forces clients to close a connection and reconnect to the new instance.
If the application next to the kuma-dp
process quits immediately after the SIGTERM signal, there is a high chance that clients will still try to send traffic to this destination.
To mitigate this, we need to support graceful shutdown in the application. For example, the application should wait X seconds to exit after receiving the first SIGTERM signal.
Consider using service probes to mark data plane proxy as unhealthy when it is in draining state.
If data plane proxy is shutdown gracefully, the Dataplane
resource is automatically deleted by kuma-cp.
If the data plane proxy goes down ungracefully, the Dataplane
resource isn’t deleted immediately. The following sequence of the events should happen:
- After
KUMA_METRICS_DATAPLANE_IDLE_TIMEOUT
(default 5mins) the data plane proxy is marked as Offline . This is because there’s no active xDS connection between the proxy and kuma-cp. - After
KUMA_RUNTIME_UNIVERSAL_DATAPLANE_CLEANUP_AGE
(default 72h) offline data plane proxies are deleted.
This guarantees that Dataplane
resources are eventually cleaned up even in the case of ungraceful shutdown.
Indirect
The lifecycle is called “Indirect”, because there is no strict dependency between Dataplane
resource creation and the
startup of the data plane proxy. This is a good way if you have some external components that manages Dataplane
lifecycle.
Joining the mesh
Dataplane
resource is created using HTTP API or kumactl.
Dataplane
resource is created before data plane proxy started. There is no support for templates, resource should be
a valid Dataplane
configuration.
When data plane proxy is started, it takes name
and mesh
as an input arguments. After connection between proxy and
kuma-cp is established, kuma-cp finds the Dataplane
resource with name
and mesh
in the store.
kuma-cp run \
--name=my-backend-dp \
--mesh=default \
...
To join the mesh in a graceful way, you can use service probes just like in Direct section.
Leaving the mesh
Kuma-cp will never delete the Dataplane
resource (with both graceful and ungraceful shutdowns).
If data plane proxy is shutdown gracefully, then Dataplane
resource will be marked as Offline. Offline data plane proxies
deleted automatically after KUMA_RUNTIME_UNIVERSAL_DATAPLANE_CLEANUP_AGE
, by default it’s 72h.
If data plane proxy went down ungracefully, then the following sequence of the events should happen:
- After
KUMA_METRICS_DATAPLANE_IDLE_TIMEOUT
(default 5mins) the data plane proxy is marked as Offline . This is because there’s no active xDS connection between the proxy and kuma-cp. - After
KUMA_RUNTIME_UNIVERSAL_DATAPLANE_CLEANUP_AGE
(default 72h) offline data plane proxies are deleted.
To leave the mesh in a graceful way, you can use service probes just like in Direct section.
Envoy
Envoy
has a powerful Admin API for monitoring and troubleshooting.
By default, kuma-dp
starts Envoy Admin API
on the loopback interface. The port is configured in the Dataplane
entity:
type: Dataplane
mesh: default
name: my-dp
networking:
admin:
port: 1000
# ...
If the admin
section is empty or port is equal to zero then the default value for port will be taken from the Kong Mesh Control Plane configuration.
Dataplane configuration
$schema: http://json-schema.org/draft-04/schema#
$ref: #/definitions/Dataplane
definitions
Dataplane
- ## Dataplane
- Dataplane defines a configuration of a side-car proxy.
- Type:
object
- This schema accepts additional properties.
- Properties
- networking
- Networking describes inbound and outbound interfaces of the data plane proxy.
- $ref: #/definitions/kuma.mesh.v1alpha1.Dataplane.Networking
- metrics
- Configuration for metrics that should be collected and exposed by the data plane proxy. Settings defined here will override their respective defaults defined at a Mesh level.
- $ref: #/definitions/kuma.mesh.v1alpha1.MetricsBackend
- probes
- Probes describe a list of endpoints that will be exposed without mTLS. This is useful to expose the health endpoints of the application so the orchestration system (e.g. Kubernetes) can still health check the application. See https://kuma.io/docs/latest/policies/service-health-probes/#virtual-probes for more information. Deprecated: this feature will be removed for Universal; on Kubernetes, it's not needed anymore.
- $ref: #/definitions/kuma.mesh.v1alpha1.Dataplane.Probes
- networking
kuma.mesh.v1alpha1.Dataplane.Networking
- ## Networking
- Networking describes inbound and outbound interfaces of a data plane proxy.
- Type:
object
- This schema accepts additional properties.
- Properties
- address
- IP on which the data plane proxy is accessible to the control plane and other data plane proxies in the same network. This can also be a hostname, in which case the control plane will periodically resolve it.
- Type:
string
- advertisedAddress
- In some situations, a data plane proxy resides in a private network (e.g. Docker) and is not reachable via
address
to other data plane proxies.advertisedAddress
is configured with a routable address for such data plane proxy so that other proxies in the mesh can connect to it overadvertisedAddress
and not via address. Envoy still binds to theaddress
, notadvertisedAddress
. - Type:
string
- In some situations, a data plane proxy resides in a private network (e.g. Docker) and is not reachable via
- gateway
- Gateway describes a configuration of the gateway of the data plane proxy.
- $ref: #/definitions/kuma.mesh.v1alpha1.Dataplane.Networking.Gateway
- inbound
- Inbound describes a list of inbound interfaces of the data plane proxy. Inbound describes a service implemented by the data plane proxy. All incoming traffic to a data plane proxy is going through inbound listeners. For every defined Inbound there is a corresponding Envoy Listener.
- Type:
array
- outbound
- Outbound describes a list of services consumed by the data plane proxy. For every defined Outbound, there is a corresponding Envoy Listener.
- Type:
array
- transparent_proxying
- TransparentProxying describes the configuration for transparent proxying. It is used by default on Kubernetes.
- $ref: #/definitions/kuma.mesh.v1alpha1.Dataplane.Networking.TransparentProxying
- admin
- Admin describes configuration related to Envoy Admin API. Due to security, all the Envoy Admin endpoints are exposed only on localhost. Additionally, Envoy will expose
/ready
endpoint onnetworking.address
for health checking systems to be able to check the state of Envoy. The rest of the endpoints exposed onnetworking.address
are always protected by mTLS and only meant to be consumed internally by the control plane. - $ref: #/definitions/kuma.mesh.v1alpha1.EnvoyAdmin
- Admin describes configuration related to Envoy Admin API. Due to security, all the Envoy Admin endpoints are exposed only on localhost. Additionally, Envoy will expose
- address
kuma.mesh.v1alpha1.Dataplane.Networking.Gateway
- ## Gateway
- Gateway describes a service that ingress should not be proxied.
- Type:
object
- This schema accepts additional properties.
- Properties
- tags
- Tags associated with a gateway of this data plane to, e.g.
kuma.io/service=gateway
,env=prod
.kuma.io/service
tag is mandatory. - Type:
object
- This schema accepts additional properties.
- Properties
- Tags associated with a gateway of this data plane to, e.g.
- type
- #### Gateway Type
- The value is restricted to the following:
- "DELEGATED"
0
- "BUILTIN"
1
- tags
kuma.mesh.v1alpha1.Dataplane.Networking.Inbound
- ## Inbound
- Inbound describes a service implemented by the data plane proxy. All incoming traffic to a data plane proxy are going through inbound listeners. For every defined Inbound there is a corresponding Envoy Listener.
- Type:
object
- This schema accepts additional properties.
- Properties
- port
- Port of the inbound interface that will forward requests to the service. When transparent proxying is used, it is a port on which the service is listening to. When transparent proxying is not used, Envoy will bind to this port.
- Type:
integer
- servicePort
- Port of the service that requests will be forwarded to. Defaults to the same value as
port
. - Type:
integer
- Port of the service that requests will be forwarded to. Defaults to the same value as
- serviceAddress
- Address of the service that requests will be forwarded to. Defaults to 'inbound.address', since Kuma DP should be deployed next to the service.
- Type:
string
- address
- Address on which inbound listener will be exposed. Defaults to
networking.address
. - Type:
string
- Address on which inbound listener will be exposed. Defaults to
- tags
- Tags associated with an application this data plane proxy is deployed next to, e.g.
kuma.io/service=web
,version=1.0
. You can then reference these tags in policies like MeshTrafficPermission.kuma.io/service
tag is mandatory. - Type:
object
- This schema accepts additional properties.
- Properties
- Tags associated with an application this data plane proxy is deployed next to, e.g.
- health
- Health describes the status of an inbound. If 'health' is nil we consider data plane proxy as healthy. Unhealthy data plane proxies are excluded from Endpoints Discovery Service (EDS). On Kubernetes, it is filled automatically by the control plane if Pod has readiness probe configured. On Universal, it can be set by the external health checking system, but the most common way is to use service probes. See https://kuma.io/docs/latest/documentation/health for more information.
- $ref: #/definitions/kuma.mesh.v1alpha1.Dataplane.Networking.Inbound.Health
- serviceProbe
- ServiceProbe defines parameters for probing the service next to sidecar. When service probe is defined, Envoy will periodically health check the application next to it and report the status to the control plane. On Kubernetes, Kuma deployments rely on Kubernetes probes so this is not used. See https://kuma.io/docs/latest/documentation/health for more information.
- $ref: #/definitions/kuma.mesh.v1alpha1.Dataplane.Networking.Inbound.ServiceProbe
- state
- #### State
- The value is restricted to the following:
- "Ready"
0
- "NotReady"
1
- "Ignored"
2
- name
- Name adds another way of referencing this port, usable with MeshService
- Type:
string
- port
kuma.mesh.v1alpha1.Dataplane.Networking.Inbound.Health
- ## Health
- Health describes the status of an inbound
- Type:
object
- This schema accepts additional properties.
- Properties
- ready
- Ready indicates if the data plane proxy is ready to serve the traffic.
- Type:
boolean
- ready
kuma.mesh.v1alpha1.Dataplane.Networking.Inbound.ServiceProbe
- ## Service Probe
- ServiceProbe defines parameters for probing service's port
- Type:
object
- This schema accepts additional properties.
- Properties
- interval
- Interval between consecutive health checks.
- Type:
string
- String format must be a "regex"
- The value must match this pattern:
^([0-9]+\.?[0-9]*|\.[0-9]+)s$
- timeout
- Maximum time to wait for a health check response.
- Type:
string
- String format must be a "regex"
- The value must match this pattern:
^([0-9]+\.?[0-9]*|\.[0-9]+)s$
- unhealthy_threshold
- Number of consecutive unhealthy checks before considering a host unhealthy.
- Type:
integer
- healthy_threshold
- Number of consecutive healthy checks before considering a host healthy.
- Type:
integer
- tcp
- Tcp checker tries to establish tcp connection with destination
- $ref: #/definitions/kuma.mesh.v1alpha1.Dataplane.Networking.Inbound.ServiceProbe.Tcp
- interval
kuma.mesh.v1alpha1.Dataplane.Networking.Inbound.ServiceProbe.Tcp
- ## Tcp
- Type:
object
- This schema accepts additional properties.
- Properties
kuma.mesh.v1alpha1.Dataplane.Networking.Outbound
- ## Outbound
- Outbound describes a service consumed by the data plane proxy. For every defined Outbound there is a corresponding Envoy Listener.
- Type:
object
- This schema accepts additional properties.
- Properties
- address
- IP on which the consumed service will be available to this data plane proxy. On Kubernetes, it's usually ClusterIP of a Service or PodIP of a Headless Service. Defaults to 127.0.0.1
- Type:
string
- port
- Port on which the consumed service will be available to this data plane proxy. When transparent proxying is not used, Envoy will bind to this port.
- Type:
integer
- tags
- Tags of consumed data plane proxies.
kuma.io/service
tag is required. These tags can then be referenced indestinations
section of policies like TrafficRoute or into
section in policies like MeshAccessLog. It is recommended to only usekuma.io/service
. If you need to consume specific data plane proxy of a service (for example:version=v2
) the better practice is to use TrafficRoute. - Type:
object
- This schema accepts additional properties.
- Properties
- Tags of consumed data plane proxies.
- backendRef
- BackendRef is a way to target MeshService. Experimental. Do not use on production yet.
- $ref: #/definitions/kuma.mesh.v1alpha1.Dataplane.Networking.Outbound.BackendRef
- address
kuma.mesh.v1alpha1.Dataplane.Networking.Outbound.BackendRef
- ## Backend Ref
- Type:
object
- This schema accepts additional properties.
- Properties
- kind
- Kind is a type of the object to target. Allowed: MeshService
- Type:
string
- name
- Name of the targeted object
- Type:
string
- port
- Port of the targeted object. Required when kind is MeshService.
- Type:
integer
- labels
- Labels to select a single object. If no object is selected then outbound is not created. If multiple objects are selected then the oldest one is used.
- Type:
object
- This schema accepts additional properties.
- Properties
- kind
kuma.mesh.v1alpha1.Dataplane.Networking.TransparentProxying
- ## Transparent Proxying
- TransparentProxying describes configuration for transparent proxying.
- Type:
object
- This schema accepts additional properties.
- Properties
- redirectportinbound
- Port on which all inbound traffic is being transparently redirected.
- Type:
integer
- redirectportoutbound
- Port on which all outbound traffic is being transparently redirected.
- Type:
integer
- directaccessservices
- List of services that will be accessed directly via IP:PORT Use
*
to indicate direct access to every service in the Mesh. Using*
to directly access every service is a resource-intensive operation, use it only if needed. - Type:
array
- Items
- Type:
string
- List of services that will be accessed directly via IP:PORT Use
- reachable_services
- List of reachable services (represented by the value of
kuma.io/service
) via transparent proxying. Setting an explicit list can dramatically improve the performance of the mesh. If not specified, all services in the mesh are reachable. - Type:
array
- Items
- Type:
string
- List of reachable services (represented by the value of
- ipfamilymode
- #### Ip Family Mode
- The value is restricted to the following:
- "UnSpecified"
0
- "DualStack"
1
- "IPv4"
2
- "IPv6"
3
- reachable_backends
- Reachable backend via transparent proxy when running with MeshExternalService, MeshService and MeshMultiZoneService. Setting an explicit list of refs can dramatically improve the performance of the mesh. If not specified, all services in the mesh are reachable.
- $ref: #/definitions/kuma.mesh.v1alpha1.Dataplane.Networking.TransparentProxying.ReachableBackends
- redirectportinbound
kuma.mesh.v1alpha1.Dataplane.Networking.TransparentProxying.ReachableBackendRef
- ## Reachable Backend Ref
- Type:
object
- This schema accepts additional properties.
- Properties
- kind
- Type of the backend: MeshService or MeshExternalService +required
- Type:
string
- name
- Name of the backend. +optional
- Type:
string
- namespace
- Namespace of the backend. Might be empty +optional
- Type:
string
- port
- Port of the backend. +optional
- Type:
integer
- labels
- Labels used to select backends +optional
- Type:
object
- This schema accepts additional properties.
- Properties
- kind
kuma.mesh.v1alpha1.Dataplane.Networking.TransparentProxying.ReachableBackends
- ## Reachable Backends
- Type:
object
- This schema accepts additional properties.
- Properties
- refs
- Type:
array
- Type:
- refs
kuma.mesh.v1alpha1.Dataplane.Probes
- ## Probes
- Type:
object
- This schema accepts additional properties.
- Properties
- port
- Port on which the probe endpoints will be exposed. This cannot overlap with any other ports.
- Type:
integer
- endpoints
- List of endpoints to expose without mTLS.
- Type:
array
- port
kuma.mesh.v1alpha1.Dataplane.Probes.Endpoint
- ## Endpoint
- Type:
object
- This schema accepts additional properties.
- Properties
- inbound_port
- Inbound port is a port of the application from which we expose the endpoint.
- Type:
integer
- inbound_path
- Inbound path is a path of the application from which we expose the endpoint. It is recommended to be as specific as possible.
- Type:
string
- path
- Path is a path on which we expose inbound path on the probes port.
- Type:
string
- inbound_port
kuma.mesh.v1alpha1.EnvoyAdmin
- ## Envoy Admin
- Type:
object
- This schema accepts additional properties.
- Properties
- port
- Port on which Envoy Admin API server will be listening
- Type:
integer
- port
kuma.mesh.v1alpha1.MetricsBackend
- ## Metrics Backend
- MetricsBackend defines metric backends
- Type:
object
- This schema accepts additional properties.
- Properties
- name
- Name of the backend, can be then used in Mesh.metrics.enabledBackend
- Type:
string
- type
- Type of the backend (Kuma ships with 'prometheus')
- Type:
string
- conf
- Configuration of the backend
- Type:
object
- This schema accepts additional properties.
- Properties
- name
Generated with json-schema-md-doc