Skip to content
Kong Logo | Kong Docs Logo
  • Docs
    • Explore the API Specs
      View all API Specs View all API Specs View all API Specs arrow image
    • Documentation
      API Specs
      Kong Gateway
      Lightweight, fast, and flexible cloud-native API gateway
      Kong Konnect
      Single platform for SaaS end-to-end connectivity
      Kong AI Gateway
      Multi-LLM AI Gateway for GenAI infrastructure
      Kong Mesh
      Enterprise service mesh based on Kuma and Envoy
      decK
      Helps manage Kong’s configuration in a declarative fashion
      Kong Ingress Controller
      Works inside a Kubernetes cluster and configures Kong to proxy traffic
      Kong Gateway Operator
      Manage your Kong deployments on Kubernetes using YAML Manifests
      Insomnia
      Collaborative API development platform
  • Plugin Hub
    • Explore the Plugin Hub
      View all plugins View all plugins View all plugins arrow image
    • Functionality View all View all arrow image
      View all plugins
      AI's icon
      AI
      Govern, secure, and control AI traffic with multi-LLM AI Gateway plugins
      Authentication's icon
      Authentication
      Protect your services with an authentication layer
      Security's icon
      Security
      Protect your services with additional security layer
      Traffic Control's icon
      Traffic Control
      Manage, throttle and restrict inbound and outbound API traffic
      Serverless's icon
      Serverless
      Invoke serverless functions in combination with other plugins
      Analytics & Monitoring's icon
      Analytics & Monitoring
      Visualize, inspect and monitor APIs and microservices traffic
      Transformations's icon
      Transformations
      Transform request and responses on the fly on Kong
      Logging's icon
      Logging
      Log request and response data using the best transport for your infrastructure
  • Support
  • Community
  • Kong Academy
Get a Demo Start Free Trial
Kong Ingress Controller
3.1.x
  • Home icon
  • Kong Ingress Controller
  • Production
  • Observability
  • Using Prometheus and Grafana
github-edit-pageEdit this page
report-issueReport an issue
  • Kong Gateway
  • Kong Konnect
  • Kong Mesh
  • Kong AI Gateway
  • Plugin Hub
  • decK
  • Kong Ingress Controller
  • Kong Gateway Operator
  • Insomnia
  • Kuma

  • Docs contribution guidelines
  • unreleased
  • 3.4.x (latest) (LTS)
  • 3.3.x
  • 3.2.x
  • 3.1.x
  • 3.0.x
  • 2.12.x (LTS)
  • 2.11.x
  • 2.10.x
  • 2.9.x
  • 2.8.x
  • 2.7.x
  • 2.6.x
  • 2.5.x (LTS)
  • Introduction
    • Overview
    • Kubernetes Gateway API
    • Version Support Policy
    • Changelog
  • How KIC Works
    • Architecture
    • Gateway API
    • Ingress
    • Custom Resources
    • Using Annotations
    • Admission Webhook
  • Get Started
    • Install KIC
    • Services and Routes
    • Rate Limiting
    • Proxy Caching
    • Key Authentication
  • KIC in Production
    • Deployment Topologies
      • Overview
      • Gateway Discovery
      • Database Backed
      • Traditional (sidecar)
    • Installation Methods
      • Helm
      • Kong Gateway Operator
    • Cloud Deployment
      • Azure
      • Amazon
      • Google
    • Enterprise License
    • Observability
      • Prometheus Metrics
      • Configuring Prometheus and Grafana
      • Kubernetes Events
    • Upgrading
      • Kong Gateway
      • Ingress Controller
  • Guides
    • Service Configuration
      • HTTP Service
      • TCP Service
      • UDP Service
      • gRPC Service
      • TLS
      • External Service
      • HTTPS Redirects
      • Multiple Backend Services
      • Configuring Gateway API resources across namespaces
    • Request Manipulation
      • Rewriting Hosts and Paths
      • Rewrite Annotation
      • Customizing load-balancing behavior
    • High Availability
      • KIC High Availability
      • Service Health Checks
      • Last Known Good Config
    • Security
      • Kong Vaults
      • Using Workspaces
      • Preserving Client IP
      • Kubernetes Secrets in Plugins
    • Migrate
      • KongIngress to KongUpstreamPolicy
      • Migrating from Ingress to Gateway
      • Credential Type Labels
    • Customize Deployments
      • Images
    • Custom Ingress Class
      • Internal / External Traffic
  • Plugins
    • Custom Plugins
    • Authentication
    • ACL
    • Rate Limiting
    • mTLS
    • OIDC
  • Reference
    • Troubleshooting
    • Version Compatibility
    • Annotations
    • Configuration Options
    • Feature Gates
    • FAQ
      • Plugin Compatibility
      • Kong Router
      • Custom nginx.conf
    • Custom Resource Definitions
    • Resources Requiring Setting Ingress Class
    • Gateway API migration
    • Required Permissions for Installation
enterprise-switcher-icon Switch to OSS
On this pageOn this page
  • Prerequisites
    • Install Kong
    • Test connectivity to Kong
  • Install Prometheus and Grafana
  • Enable ServiceMonitor
  • Set Up Port Forwards
  • Access Grafana Dashboard
  • Setup Services
  • Metrics collected
    • Request Latencies of Various Services
    • Kong Proxy Latency
    • Error Rates
    • Request Rate and Bandwidth
You are browsing documentation for an older version. See the latest documentation here.

Using Prometheus and Grafana

The Kong Ingress Controller gives you visibility into how Kong Gateway is performing and how the services in your Kubernetes cluster are responding to the inbound traffic.

Learn to set up monitoring for Kong Gateway with Prometheus.

As of Kong Ingress Controller 2.0, there are additional performance metrics associated with the configuration process rather than the runtime performance of the Gateway. For more information, see the Prometheus metrics reference.

Prerequisites: Install Kong Ingress Controller in your Kubernetes cluster and connect to Kong.

Prerequisites

Install Kong

You can install Kong in your Kubernetes cluster using Helm.

  1. Add the Kong Helm charts:

     helm repo add kong https://charts.konghq.com
     helm repo update
    
  2. Install Kong Ingress Controller and Kong Gateway with Helm:

     helm install kong kong/ingress -n kong --create-namespace 
    

Test connectivity to Kong

Kubernetes exposes the proxy through a Kubernetes service. Run the following commands to store the load balancer IP address in a variable named PROXY_IP:

  1. Populate $PROXY_IP for future commands:

     export PROXY_IP=$(kubectl get svc --namespace kong kong-gateway-proxy -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
     echo $PROXY_IP
    
  2. Ensure that you can call the proxy IP:

     curl -i $PROXY_IP
    

    The results should look like this:

     HTTP/1.1 404 Not Found
     Content-Type: application/json; charset=utf-8
     Connection: keep-alive
     Content-Length: 48
     X-Kong-Response-Latency: 0
     Server: kong/3.0.0
      
     {"message":"no Route matched with those values"}
    

Install Prometheus and Grafana

If you already have the Prometheus Operator and Grafana installed in your cluster, you can skip installing Prometheus and Grafana.

Note: The Prometheus Operator is required, as the Kong Ingress Controller uses its PodMonitor custom resource to configure scrape rules.

Install Prometheus with a scrape interval of 10 seconds to have fine-grained data points for all metrics. And install both Prometheus and Grafana in a dedicated monitoring namespace.

  1. Create a values-monitoring.yaml file to set the scrape interval, use Grafana persistence, and install Kong’s dashboard:
     prometheus:
       prometheusSpec:
         scrapeInterval: 10s
         evaluationInterval: 30s
     grafana:
       persistence:
         enabled: true  # enable persistence using Persistent Volumes
       dashboardProviders:
         dashboardproviders.yaml:
           apiVersion: 1
           providers:
           - name: 'default' # Configure a dashboard provider file to
             orgId: 1        # put Kong dashboard into.
             folder: ''
             type: file
             disableDeletion: false
             editable: true
             options:
               path: /var/lib/grafana/dashboards/default
       dashboards:
         default:
           kong-dash:
             gnetId: 7424  # Install the following Grafana dashboard in the
             revision: 11  # instance: https://grafana.com/dashboards/7424
             datasource: Prometheus
           kic-dash:
             gnetId: 15662
             datasource: Prometheus
        
    
  2. To install Prometheus and Grafana, execute the following, specifying the path to the values-monitoring.yaml file that you created:

     $ kubectl create namespace monitoring
     $ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
     $ helm install promstack prometheus-community/kube-prometheus-stack --namespace monitoring --version 52.1.0 -f values-monitoring.yaml
    

Enable ServiceMonitor

  1. To enable ServiceMonitor, set gateway.serviceMonitor.enabled=true.

     $ helm upgrade kong kong/ingress -n kong --set gateway.serviceMonitor.enabled=true --set gateway.serviceMonitor.labels.release=promstack
    

    By default, kube-prometheus-stack selects ServiceMonitors and PodMonitors by arelease label equal to the release name. The labels setting here adds a label matching the promstack release name from the example.

  2. Enable the Prometheus plugin in Kong at the global level, so that each request that flows into the Kubernetes cluster gets tracked in Prometheus:

     $ echo 'apiVersion: configuration.konghq.com/v1
     kind: KongClusterPlugin
     metadata:
       name: prometheus
       annotations:
         kubernetes.io/ingress.class: kong
       labels:
         global: "true"
     plugin: prometheus
     config:
       status_code_metrics: true
       bandwidth_metrics: true
       upstream_health_metrics: true
       latency_metrics: true
       per_consumer: false
     ' | kubectl apply -f -
    

    The results should look like this:

     kongclusterplugin.configuration.konghq.com/prometheus created
    

Set Up Port Forwards

In a production environment, you would have a Kubernetes Service with an external IP or load balancer, which would allow you to access Prometheus, Grafana, and Kong. For demo purposes, set up port-forwarding using kubectl to get access. It is not advisable to do this in production.

Open a new terminal and execute these commands:

kubectl -n monitoring port-forward services/prometheus-operated 9090 &
kubectl -n monitoring port-forward services/promstack-grafana 3000:80 &

# You can access Prometheus in your browser at localhost:9090
# You can access Grafana in your browser at localhost:3000
# You can access Kong at $PROXY_IP
# We are using plain-text HTTP proxy for this purpose of
# demo.

Access Grafana Dashboard

To access Grafana, you need to get the password for the admin user.

Execute the following to read the password and take note of it:

kubectl get secret --namespace monitoring promstack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Now, browse to http://localhost:3000 and fill in username as “admin” and password that you made a note of. After you log in to Grafana you can notice that Kong’s Grafana Dashboard should already be installed for you.

Setup Services

Spin up some services for demo purposes and setup Ingress routing for them.

  1. Install three services: billing, invoice, and comments.

     kubectl apply -f https://docs.konghq.com/assets/kubernetes-ingress-controller/examples/multiple-services.yaml
    
  2. Create Ingress routing rules in Kubernetes. This configures Kong to proxy traffic destined for these services correctly.

     echo '
     apiVersion: networking.k8s.io/v1
     kind: Ingress
     metadata:
       name: sample-ingresses
       annotations:
         konghq.com/strip-path: "true"
     spec:
       ingressClassName: kong
       rules:
       - http:
          paths:
          - path: /billing
            pathType: ImplementationSpecific
            backend:
              service:
                name: billing
                port:
                  number: 80
          - path: /comments
            pathType: ImplementationSpecific
            backend:
              service:
                name: comments
                port:
                  number: 80
          - path: /invoice
            pathType: ImplementationSpecific
            backend:
              service:
                name: invoice
                port:
                  number: 80
     ' | kubectl apply -f -
    
  3. Create some traffic after configuring the services and proxies.

     while true;
     do
       curl $PROXY_IP/billing/status/200
       curl $PROXY_IP/billing/status/501
       curl $PROXY_IP/invoice/status/201
       curl $PROXY_IP/invoice/status/404
       curl $PROXY_IP/comments/status/200
       curl $PROXY_IP/comments/status/200
       sleep 0.01
     done
    

Because you have already enabled Prometheus plugin in Kong to collect metrics for requests proxied via Kong, you should see metrics coming through in the Grafana dashboard.

You should be able to see metrics related to the traffic flowing through the services. Try tweaking the above script to send different traffic patterns and see how the metrics change. The upstream services are httpbin instances, meaning you can use a variety of endpoints to shape your traffic.

Metrics collected

Request Latencies of Various Services

Request latencies

Kong collects latency data of how long your services take to respond to requests. One can use this data to alert the on-call engineer if the latency goes beyond a certain threshold. For example, let’s say you have an SLA that your APIs will respond with latency of less than 20 millisecond for 95% of the requests. You could configure Prometheus to alert based on the following query:

histogram_quantile(0.95, sum(rate(kong_request_latency_ms_sum{route=~"$route"}[1m])) by (le)) > 20

The query calculates the 95th percentile of the total request latency (or duration) for all of your services and alerts you if it is more than 20 milliseconds. The “type” label in this query is “request”, which tracks the latency added by Kong and the service. You can switch this to “upstream” to track latency added by the service only. Prometheus is highly flexible and well documented, so we won’t go into details of setting up alerts here, but you’ll be able to find them in the Prometheus documentation.

Kong Proxy Latency

Proxy latencies

Kong also collects metrics about its performance. The following query is similar to the previous one but gives us insight into latency added by Kong:

histogram_quantile(0.90, sum(rate(kong_kong_latency_ms_bucket[1m])) by (le,service)) > 2

Error Rates

Error rates

Another important metric to track is the rate of errors and requests your services are serving. The time series kong_http_status collects HTTP status code metrics for each service.

This metric can help you track the rate of errors for each of your service:

sum(rate(kong_http_requests_total{code=~"5[0-9]{2}"}[1m])) by (service)

You can also calculate the percentage of requests in any duration that are errors. Try to come up with a query to derive that result.

Please note that all HTTP status codes are indexed, meaning you could use the data to learn about your typical traffic pattern and identify problems. For example, a sudden rise in 404 response codes could be indicative of client codes requesting an endpoint that was removed in a recent deploy.

Request Rate and Bandwidth

Request rates

One can derive the total request rate for each of your services or across your Kubernetes cluster using the kong_http_status time series.

Bandwidth

Another metric that Kong keeps track of is the amount of network bandwidth (kong_bandwidth) being consumed. This gives you an estimate of how request/response sizes correlate with other behaviors in your infrastructure.

You now have metrics for the services running inside your Kubernetes cluster and have much more visibility into your applications, without making any modifications in your services. You can use Alertmanager or Grafana to now configure alerts based on the metrics observed and your SLOs.

Thank you for your feedback.
Was this page useful?
Too much on your plate? close cta icon
More features, less infrastructure with Kong Konnect. 1M requests per month for free.
Try it for Free
  • Kong
    Powering the API world

    Increase developer productivity, security, and performance at scale with the unified platform for API management, service mesh, and ingress controller.

    • Products
      • Kong Konnect
      • Kong Gateway Enterprise
      • Kong Gateway
      • Kong Mesh
      • Kong Ingress Controller
      • Kong Insomnia
      • Product Updates
      • Get Started
    • Documentation
      • Kong Konnect Docs
      • Kong Gateway Docs
      • Kong Mesh Docs
      • Kong Insomnia Docs
      • Kong Konnect Plugin Hub
    • Open Source
      • Kong Gateway
      • Kuma
      • Insomnia
      • Kong Community
    • Company
      • About Kong
      • Customers
      • Careers
      • Press
      • Events
      • Contact
  • Terms• Privacy• Trust and Compliance
© Kong Inc. 2025