Skip to content
Kong Logo | Kong Docs Logo
search
  • We're Hiring!
  • Docs
    • Kong Gateway
    • Kong Konnect
    • Kong Mesh
    • Plugin Hub
    • decK
    • Kubernetes Ingress Controller
    • Insomnia
    • Kuma

    • Docs contribution guidelines
  • Plugin Hub
  • Support
  • Community
  • Kong Academy
Get a Demo Start Free Trial
  • Kong Gateway
  • Kong Konnect
  • Kong Mesh
  • Plugin Hub
  • decK
  • Kubernetes Ingress Controller
  • Insomnia
  • Kuma

  • Docs contribution guidelines
  • 3.2.x (latest)
  • 3.1.x
  • 3.0.x
  • 2.8.x
  • 2.7.x
  • 2.6.x
  • Older Enterprise versions (2.1-2.5)
  • Older OSS versions (2.1-2.5)
  • Archive (pre-2.1)
    • Overview of Kong Gateway
      • Version Support Policy
      • Supported Installation Options
      • Supported Linux Distributions
    • Stability
    • Release Notes
      • Services
        • Overview
        • Configure Routes with Expressions
      • Upstreams
      • Plugins
      • Routing Traffic
      • Load Balancing
      • Health Checks and Circuit Breakers
      • Kong Performance Testing
    • Glossary
    • Get Kong
    • Services and Routes
    • Rate Limiting
    • Proxy Caching
    • Key Authentication
    • Load-Balancing
      • Overview
        • Overview
        • Deploy Kong Gateway in Hybrid mode
      • DB-less Deployment
      • Traditional
      • Overview
        • Helm
        • OpenShift with Helm
        • kubectl apply
        • Kubernetes Deployment Options
        • Using docker run
        • Build your own Docker images
        • Amazon Linux
        • Debian
        • Red Hat
        • Ubuntu
      • Running Kong as a non-root user
      • Securing the Admin API
      • Using systemd
      • Start Kong Gateway Securely
      • Programatically Creating Admins
      • Enabling RBAC
      • Overview
      • Download your License
      • Deploy Enterprise License
      • Using the License API
      • Monitor Licenses Usage
      • Default Ports
      • DNS Considerations
      • Network and Firewall
      • CP/DP Communication through a Forward Proxy
    • Kong Configuration File
    • Environment Variables
    • Serving a Website and APIs from Kong
      • Overview
      • Prometheus
      • StatsD
      • Datadog
      • Overview
      • Writing a Custom Trace Exporter
      • Tracing API Reference
    • Resource Sizing Guidelines
    • Security Update Process
    • Blue-Green Deployments
    • Canary Deployments
    • Clustering Reference
      • Log Reference
      • Dynamic log level updates
      • Customize Gateway Logs
      • Upgrade Kong Gateway 3.1.x
      • Migrate from OSS to Enterprise
    • Overview
      • Overview
      • Metrics
      • Analytics with InfluxDB
      • Analytics with Prometheus
      • Estimate Analytics Storage in PostgreSQL
      • Overview
      • Getting Started
      • Advanced Usage
        • Overview
        • Environment Variables
        • AWS Secrets Manager
        • Google Secrets Manager
        • Hashicorp Vault
        • Securing the Database with AWS Secrets Manager
      • Reference Format
      • Overview
      • Get Started with Dynamic Plugin Ordering
      • Overview
      • Enable the Dev Portal
      • Publish an OpenAPI Spec
      • Structure and File Types
      • Themes Files
      • Working with Templates
      • Using the Editor
        • Basic Auth
        • Key Auth
        • OIDC
        • Sessions
        • Adding Custom Registration Fields
        • Manage Developers
        • Developer Roles and Content Permissions
        • Authorization Provider Strategy
        • Enable Application Registration
        • Enable Key Authentication for Application Registration
          • External OAuth2 Support
          • Set up Okta and Kong for External Oauth
          • Set up Azure AD and Kong for External Authentication
        • Manage Applications
        • Theme Editing
        • Migrating Templates Between Workspaces
        • Markdown Rendering Module
        • Customizing Portal Emails
        • Adding and Using JavaScript Assets
        • Single Page App in Dev Portal
        • Alternate OpenAPI Renderer
      • SMTP
      • Workspaces
      • Helpers CLI
      • Portal API Documentation
    • Audit Logging
    • Keyring and Data Encryption
    • Workspaces
    • Consumer Groups
    • Event Hooks
    • FIPS 140-2
    • Overview
    • Enable Kong Manager
      • Services and Routes
      • Rate Limiting
      • Proxy Caching
      • Authentication with Consumers
      • Load Balancing
      • Overview
      • Create a Super Admin
      • Workspaces and Teams
      • Reset Passwords and RBAC Tokens
      • Basic Auth
        • Configure LDAP
        • LDAP Service Directory Mapping
        • Configure OIDC
        • OIDC Authenticated Group Mapping
      • Sessions
        • Overview
        • Enable RBAC
        • Add a Role and Permissions
        • Create a User
        • Create an Admin
    • Networking Configuration
    • Workspaces
    • Create Consumer Groups
    • Sending Email
    • Overview
    • File Structure
    • Implementing Custom Logic
    • Plugin Configuration
    • Accessing the Data Store
    • Storing Custom Entities
    • Caching Custom Entities
    • Extending the Admin API
    • Writing Tests
    • (un)Installing your Plugin
      • Overview
      • kong.client
      • kong.client.tls
      • kong.cluster
      • kong.ctx
      • kong.ip
      • kong.jwe
      • kong.log
      • kong.nginx
      • kong.node
      • kong.request
      • kong.response
      • kong.router
      • kong.service
      • kong.service.request
      • kong.service.response
      • kong.table
      • kong.tracing
      • kong.vault
      • kong.websocket.client
      • kong.websocket.upstream
      • Go
      • Javascript
      • Python
      • Running Plugins in Containers
      • External Plugin Performance
    • Overview
        • Overview
        • OpenID Connect with Curity
        • OpenID Connect with Azure AD
        • OpenID Connect with Google
        • OpenID Connect with Okta
        • OpenID Connect with Auth0
        • OpenID Connect with Cognito
      • Authentication Reference
      • Allow Multiple Authentication Plugins
    • Rate Limiting Plugin
      • Add a Body Value
    • GraphQL
      • gRPC Plugins
      • Configure a gRPC service
    • Overview
    • Information Routes
    • Health Routes
    • Tags
    • Debug Routes
    • Services
    • Routes
    • Consumers
    • Plugins
    • Certificates
    • CA Certificates
    • SNIs
    • Upstreams
    • Targets
    • Vaults
    • Keys
    • Licenses
    • Workspaces
    • RBAC
    • Admins
    • Developers
    • Consumer Groups
    • Event Hooks
    • Keyring and Data Encryption
    • Audit Logs
    • kong.conf
    • Injecting Nginx Directives
    • CLI
    • File Permissions Reference
    • Key Management
    • Performance Testing Framework
    • Router Expressions Language
    • FAQ

github-edit-pageEdit this page

report-issueReport an issue

enterprise-switcher-iconSwitch to OSS

On this page
  • Installation
  • Usage
    • Basic usage
    • Terraform managed instances
    • AWS
    • Charts
  • Drivers
  • Workflow
  • Sample Output
Kong Gateway
3.1.x
  • Home
  • Kong Gateway
  • How Kong Works
  • Kong Gateway Performance Testing
You are browsing documentation for an outdated version. See the latest documentation here.

Kong Gateway Performance Testing

The Kong Gateway codebase includes a performance testing framework. It allows Kong developers and users to evaluate the performance of Kong itself as well as bundled or custom plugins, and plot frame graphs to debug performance bottlenecks. The framework collects RPS (request per second) and latencies of Kong processing the request to represent performance metrics under different workloads.

The framework is implemented as an extension to Kong’s integration test suite.

Installation

The framework uses busted and some Lua development dependencies of Kong. To set up the environment, run make dev on your Kong repository to install all Lua dependencies.

Usage

Basic usage

Install Lua development dependencies of Kong being installed and invoke each test file with busted. The following runs the RPS and latency test using the docker driver.

PERF_TEST_USE_DAILY_IMAGE=true PERF_TEST_VERSIONS=git:master,git:perf/your-other-branch bin/busted -o gtest spec/04-perf/01-rps/01-simple_spec.lua

Terraform managed instances

By default, Terraform doesn’t teardown instances after each test in lazy_teardown, allowing users to run multiple tests consecutively without rebuilding the infrastructure each time.

This behavior can be changed by setting the PERF_TEST_TEARDOWN_ALL environment variable to true. Users can also manually teardown the infrastructure by running:

PERF_TEST_TEARDOWN_ALL=1 PERF_TEST_DRIVER=terraform PERF_TEST_TERRAFORM_PROVIDER=your-provider bin/busted -o gtest -t single spec/04-perf/99-teardown/

AWS

When using the terraform driver and aws-ec2 as the provider, AWS credentials can be provided using environment variables. It’s common to pass the AWS_PROFILE variable to point to a stored AWS credentials profile, or AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to use credentials directly. See the Terraform AWS provider document for further information.

Charts

Charts are not rendered by default. There’s a reference implementation to draw graphs on all JSON data stored in the output directory. Use the following commands to draw graphs:

cwd=$(pwd)
cd spec/helpers/perf/charts/
pip install -r requirements.txt
for i in $(ls ${cwd}/output/*.data.json); do
  python ./charts.py $i -o "${cwd}/output/"
done

Drivers

Three “drivers” are implemented depending on different environments, accuracy requirements, and setup complexity.

Driver Test between git commits Test between binary releases Flame Graph Test unreleased version
Docker yes yes   yes (use_daily_image = true)
Terraform yes yes yes yes (use_daily_image = true)

You must install the following Lua development dependencies to use either of the drivers:

  • Docker Driver is based on Docker images. It requires fewer dependencies. And it doesn’t support FlameGraph generation.

  • terraform Driver runs in a remote VM or on a bare metal machine. It’s most accurate, but it requires Terraform knowledge to operate and setup.

    • Requires the Terraform binary be installed.
    • Requires git binary if testing between git commits. When testing between git commits, the framework assumes the current directory is Kong’s repo. It will stash your working directory and remove the stash after the test is finished. When using the Docker or Terraform driver, the framework derives the base version of each commit and uses the matching Docker image or Kong binary package and puts the local source code inside.

Workflow

Multiple test cases can share the same Lua file.

The following code snippet demonstrates a basic workflow for defining a workload and load testing Kong.

local perf = require("spec.helpers.perf")

perf.use_defaults()

local versions = { "git:master", "3.0.0" }

for _, version in ipairs(versions) do
  describe("perf test for Kong " .. version .. " #simple #no_plugins", function()
    local bp
    lazy_setup(function()
      local helpers = perf.setup_kong(version)

      bp = helpers.get_db_utils("postgres", {
        "routes",
        "services",
      })

      local upstream_uri = perf.start_worker([[
      location = /test {
        return 200;
      }
      ]])

      local service = bp.services:insert {
        url = upstream_uri .. "/test",
      }

      bp.routes:insert {
        paths = { "/s1-r1" },
        service = service,
        strip_path = true,
      }
    end)

    before_each(function()
      perf.start_kong({
        --kong configs
      })
    end)

    after_each(function()
      perf.stop_kong()
    end)

    lazy_teardown(function()
      perf.teardown()
    end)

    it("#single_route", function()
      local results = {}
      for i=1,3 do
        perf.start_load({
          path = "/s1-r1",
          connections = 1000,
          threads = 5,
          duration = 10,
        })

        ngx.sleep(10)

        local result = assert(perf.wait_result())

        print(("### Result for Kong %s (run %d):\n%s"):format(version, i, result))
        results[i] = result
      end

      print(("### Combined result for Kong %s:\n%s"):format(version, assert(perf.combine_results(results))))

      perf.save_error_log("output/" .. version:gsub("[:/]", "#") .. "-single_route.log")
    end)
  end)
end

The test can be run as a normal Busted test, using bin/busted path/to/this_file_spec.lua.

More examples can be found in the spec/04-perf folder in the Kong repository.

Sample Output

### Test Suite: git:origin/master~10 #simple #hybrid #no_plugins #single_route
### Result for Kong git:origin/master~10 (run 1):
Running 30s test @ http://172.17.0.50:8000/s1-r1
  5 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.53ms    3.35ms  87.02ms   89.65%
    Req/Sec     6.42k     1.33k   10.53k    67.29%
  Latency Distribution
     50%    2.65ms
     75%    4.35ms
     90%    6.99ms
     99%   16.86ms
  960330 requests in 30.10s, 228.96MB read
Requests/sec:  31904.97
Transfer/sec:      7.61MB
### Result for Kong git:origin/master~10 (run 2):
Running 30s test @ http://172.17.0.50:8000/s1-r1
  5 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.56ms    6.55ms 236.87ms   91.95%
    Req/Sec     5.79k     1.54k   10.50k    66.18%
  Latency Distribution
     50%    2.82ms
     75%    5.31ms
     90%    9.72ms
     99%   26.33ms
  863963 requests in 30.07s, 206.00MB read
Requests/sec:  28730.72
Transfer/sec:      6.85MB
### Result for Kong git:origin/master~10 (run 3):
Running 30s test @ http://172.17.0.50:8000/s1-r1
  5 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.79ms    3.81ms  82.25ms   89.60%
    Req/Sec     6.14k     1.43k   12.59k    66.49%
  Latency Distribution
     50%    2.77ms
     75%    4.69ms
     90%    7.71ms
     99%   18.88ms
  917366 requests in 30.08s, 218.72MB read
Requests/sec:  30499.09
Transfer/sec:      7.27MB
### Combined result for Kong git:origin/master~10:
RPS     Avg: 30378.26
Latency Avg: 3.94ms    Max: 236.87ms
   P90 (ms): 6.99, 9.72, 7.71
   P99 (ms): 16.86, 26.33, 18.88

With samples in spec/04-perf, the RPS and latency results, Kong error logs, charts and Flame Graph files are saved to the output directory in the current directory.

Thank you for your feedback.
Was this page useful?
  • Kong
    THE CLOUD CONNECTIVITY COMPANY

    Kong powers reliable digital connections across APIs, hybrid and multi-cloud environments.

    • Company
    • Customers
    • Events
    • Investors
    • Careers Hiring!
    • Partners
    • Press
    • Contact
  • Products
    • Kong Konnect
    • Kong Gateway
    • Kong Mesh
    • Get Started
    • Pricing
  • Resources
    • eBooks
    • Webinars
    • Briefs
    • Blog
    • API Gateway
    • Microservices
  • Open Source
    • Install Kong Gateway
    • Kong Community
    • Kubernetes Ingress
    • Kuma
    • Insomnia
  • Solutions
    • Decentralize
    • Secure & Govern
    • Create a Dev Platform
    • API Gateway
    • Kubernetes
    • Service Mesh
Star
  • Terms•Privacy
© Kong Inc. 2023