Set up AI Proxy with Hugging Face

This guide walks you through setting up the AI Proxy plugin with Hugging Face.

For all providers, the Kong AI Proxy plugin attaches to route entities.

It can be installed into one route per operation, for example:

• OpenAI chat route
• Cohere chat route
• Cohere completions route

Each of these AI-enabled routes must point to a null service. This service doesn’t need to map to any real upstream URL, it can point somewhere empty (for example, http://localhost:32000), because the AI Proxy plugin overwrites the upstream URL. This requirement will be removed in a later Kong revision.

Prerequisites

• Hugging Face account and subscription
• You need a service to contain the route for the LLM provider. Create a service first:

  curl -X POST http://localhost:8001/services \
    --data "name=ai-proxy" \
    --data "url=http://localhost:32000"
  

  Remember that the upstream URL can point anywhere empty, as it won’t be used by the plugin.

• Hugging Face access token with permissions to make calls to the Inference API
• Text-generation model from Hugging Face

Provider configuration

Set up route and plugin

curl -X POST http://localhost:8001/services/ai-proxy/routes \
  --data "name=huggingface-chat" \
  --data "paths[]=~/huggingface-chat$"

curl -X POST http://localhost:8001/routes/huggingface-chat/plugins \
  --data "name=ai-proxy" \
  --data "config.route_type=llm/v1/chat" \
  --data "config.auth.header_name=Authorization" \
  --data "config.auth.header_value= Bearer <huggingface_token>" \ 
  --data "config.model.provider=huggingface" \
  --data "config.model.name=<huggingface_model>" \
  --data "config.model.options.max_tokens=512" \
  --data "config.model.options.temperature=1.0" \
  --data "config.model.options.top_p=256" \
  --data "config.model.options.top_k=0.5"

routes:
- name: huggingface-chat
  service:
    name: ai-proxy
  paths:
    - "~/huggingface-chat$"
  methods:
    - POST
plugins:
  - name: ai-proxy
    config:
      route_type: "llm/v1/chat"
      auth:
        header_name: "Authorization"
        header_value: "Bearer <huggingface_token>"  # add your Hugging Face access token
      model:
        provider: "huggingface"
        name: "<huggingface_model>" # add the Hugging Face model to use
        options:
          max_tokens: 512
          temperature: 1.0
          top_p: 256
          top_k: 0.5

Test the configuration

Make an llm/v1/chat type request to test your new endpoint:

curl -X POST http://localhost:8000/huggingface-chat \
  -H 'Content-Type: application/json' \
  --data-raw '{ "messages": [ { "role": "system", "content": "You are a mathematician" }, { "role": "user", "content": "What is 1+1?"} ] }'