In order to give you better service we use cookies. By continuing to use our website, you agree to the use of cookies as described in our Cookie Policy

Kong Logo
header icon

Kafka Log

Kong Enterprise only: This plugin is only available with a Kong Enterprise subscription.
Please inquire about Kong Enterprise by contacting us, or start a free trial today.

Publish request and response logs to a Kafka topic.

Terminology

  • plugin: a plugin executing actions inside Kong before or after a request has been proxied to the upstream API.
  • Upstream service: this refers to your own API/service sitting behind Kong, to which client requests are forwarded.

Configuration

Global plugins

A plugin which is not associated to any Service, Route, or Consumer (or API, if you are using an older version of Kong) is considered "global", and will be run on every request. Read the Plugin Reference and the Plugin Precedence sections for more information.

Plugins can be configured using the http://kong:8001/plugins/ endpoint.

Parameters

Here's a list of all the parameters which can be used in this plugin's configuration:

Form ParameterDescription
nameThe name of the plugin to use, in this case kafka-log
enabled

default value: true		Whether this plugin will be applied.
config.bootstrap_servers

List of bootstrap brokers in a {host: host, port: port} format.
config.topic

The Kafka topic to publish to
config.timeout
optional

default value:

10000

Socket timeout in milliseconds.
config.keepalive
optional

default value:

60000

Keepalive timeout in milliseconds.
config.producer_request_acks
optional

default value:

1

The number of acknowledgments the producer requires the leader to have received before considering a request complete. Allowed values: 0 for no acknowledgments, 1 for only the leader, and -1 for the full ISR
config.producer_request_timeout
optional

default value:

2000

Time to wait for a Produce response in milliseconds
config.producer_request_limits_messages_per_request
optional

default value:

200

Maximum number of messages to include into a single Produce request
config.producer_request_limits_bytes_per_request
optional

default value:

1048576

Maximum size of a Produce request in bytes
config.producer_request_retries_max_attempts
optional

default value:

10

Maximum number of retry attempts per single Produce request
config.producer_request_retries_backoff_timeout
optional

default value:

100

Backoff interval between retry attempts in milliseconds.
config.producer_async
optional

default value:

true

Flag to enable asynchronous mode
config.producer_async_flush_timeout
optional

default value:

1000

Maximum time interval in milliseconds between buffer flushes in asynchronous mode.
config.producer_async_buffering_limits_messages_in_memory
optional

default value:

50000

Maximum number of messages that can be buffered in memory in asynchronous mode.
config.api_version
optional

default value:

0

API version of a Produce request. Allowed values: 0, 1, or 2.

Installation

Manually download and install:

$ git clone https://github.com/kong/kong-plugin-kafka-log.git /path/to/kong/plugins/kong-plugin-kafka-log
$ cd /path/to/kong/plugins/kong-plugin-kafka-log
$ luarocks make *.rockspec

In both cases, you need to change your Kong plugins configuration option to include this plugin:

plugins = bundled,kong-plugin-kafka-log

Or, if you don’t want to activate any of the bundled plugins:

plugins = kong-plugin-kafka-log

Then reload kong:

kong reload

Configuration

Enabling globally

$ curl -X POST http://kong:8001/plugins \
    --data "name=kafka-log" \
    --data "config.bootstrap_servers[1].host=localhost" \
    --data "config.bootstrap_servers[1].port=9092" \
    --data "config.topic=kong-log" \
    --data "config.timeout=10000" \
    --data "config.keepalive=60000" \
    --data "config.producer_request_acks=1" \
    --data "config.producer_request_timeout=2000" \
    --data "config.producer_request_limits_messages_per_request=200" \
    --data "config.producer_request_limits_bytes_per_request=1048576" \
    --data "config.producer_request_retries_max_attempts=10" \
    --data "config.producer_request_retries_backoff_timeout=100" \
    --data "config.producer_async=true" \
    --data "config.producer_async_flush_timeout=1000" \
    --data "config.producer_async_buffering_limits_messages_in_memory=50000"

Log Format

Similar to HTTP Log Plugin.

Implementation details

This plugin makes use of lua-resty-kafka client under the hood.

Known issues and limitations

Known limitations:

  1. There is no support for TLS
  2. There is no support for Authentication
  3. There is no support for message compression

Quickstart

The following guidelines assume that both Kong and Kafka have been installed on your local machine:

  1. Install kong-plugin-kafka-log as specified in the “Installation” section above.

  2. Create kong-log topic in your Kafka cluster:

     ${KAFKA_HOME}/bin/kafka-topics.sh --create \
     --zookeeper localhost:2181 \
     --replication-factor 1 \
     --partitions 10 \
     --topic kong-log

  3. Add kong-plugin-kafka-log plugin globally:

     curl -X POST http://localhost:8001/plugins \
     --data "name=kafka-log" \
     --data "config.bootstrap_servers[1].host=localhost \
     --data "config.bootstrap_servers[1].port=9092 \
     --data "config.topic=kong-log"

    Alternatively, create a Service with a Route, and attach the plugin to that Service.

  4. Make sample requests:

     for i in {1..50} ; do curl http://localhost:8000/request/$i ; done

  5. Verify the contents of Kafka topic:

     ${KAFKA_HOME}/bin/kafka-console-consumer.sh \
     --bootstrap-server localhost:9092 \
     --topic kong-log \
     --partition 0 \
     --from-beginning \
     --timeout-ms 1000