Load Balance

Intelligent API key management with weighted load balancing, model-specific filtering, and automatic failover. Distribute traffic across multiple keys for optimal performance and reliability.

Weighted Load Balancing

Bifrost uses weighted random selection to distribute requests across multiple keys. This allows you to:

Control Traffic Distribution:

• Assign higher weights to premium keys with better rate limits
• Balance between production and backup keys
• Gradually migrate traffic during key rotation

Weight Calculation Example:

Key 1: Weight 0.7 (70% probability)
Key 2: Weight 0.3 (30% probability)
Total Weight: 1.0

Random selection ensures statistical distribution over time

Algorithm Details:

1. Calculate total weight of all eligible keys
2. Generate random number between 0 and total weight
3. Select key based on cumulative weight ranges
4. A retry after a per-key failure (a rate limit, a rejected credential, an exhausted quota) draws again from the keys not yet tried; a retry after a network error or 5xx reuses the same key. See How retries work.

Weight Rules:

• A weight must be a finite number of 0 or more. The API rejects a negative, NaN or infinite weight with 400; such a value in config.json counts as 0.
• A key with weight 0 is not picked while another eligible key has a positive weight. It still serves when it is the only key left, for example on a retry after the others failed.
• When every eligible key has weight 0, keys are picked uniformly at random.

In Bifrost Enterprise, Adaptive Routing replaces the static weights with measured ones while key selection is on. Either way, it holds a key the provider refused until one request checks it again (sooner when every key is held), and editing a key releases every hold on it, since the provider’s answers applied to the key as it was. See Held keys.

Model Whitelisting and Filtering

Keys can be restricted to specific models for access control and cost management:

Model Filtering Logic:

• Empty models array ([]): Denies ALL models (deny-by-default, v1.5.0+) - use ["*"] to allow all
• Populated models array: Key only supports listed models
• blacklisted_models: Optional per-key denylist. If non-empty and the requested model appears in it, the key is excluded-even if that model is also in models (denylist wins over the allowlist)
• regex: entries: In models or blacklisted_models, an entry such as "regex:^gpt-4.*" is an RE2 pattern matched case-insensitively against the whole model name. Other entries stay exact. A pattern that is empty, *, or does not compile is refused with 400
• Model mismatch: Key is excluded from selection for that request

Use Cases:

• Premium Models: Dedicated keys for expensive models (GPT-4, Claude-3)
• Team Separation: Different keys for different teams or projects
• Cost Control: Restrict access to specific model tiers
• Compliance: Separate keys for different security requirements
• Denylist: Block specific models on a key

Example Model Restrictions:

Each key is created individually via POST /api/providers/{provider}/keys:

// Premium-only key
{
  "name": "openai-pre-key-1",
  "value": "premium-key",
  "models": ["gpt-4o", "o1-preview"],
  "weight": 1.0
}

// Standard-only key
{
  "name": "openai-std-key-1",
  "value": "standard-key",
  "models": ["gpt-4o-mini", "gpt-3.5-turbo"],
  "weight": 1.0
}

// Shared key with denylist
{
  "name": "openai-shared-key",
  "value": "env.OPENAI_API_KEY",
  "models": ["gpt-4o", "gpt-4o-mini"],
  "blacklisted_models": ["gpt-5"],
  "weight": 1.0
}

Deployment Mapping (Azure & Bedrock)

For cloud providers with deployment-based routing, Bifrost validates deployment availability:

Azure:

• Keys must have deployment mappings for specific models
• Deployment name maps to actual Azure deployment identifier
• Missing deployment excludes key from selection

AWS Bedrock:

• Supports model profiles and direct model access
• Deployment mappings enable inference profile routing
• ARN configuration determines URL formation

Deployment Validation Process:

1. Check if provider uses deployments (Azure/Bedrock)
2. Verify deployment exists for requested model
3. Exclude keys without proper deployment mapping
4. Continue with standard weighted selection

Custom Key Usage (By Name or ID)

Bifrost supports referencing a stored provider key by name or by ID instead of sending the raw secret. This can be useful when you want callers to reference logical key names or stable IDs and let the gateway resolve the actual secret from configured provider keys.

When both are provided, ID takes priority over name.

By ID

• Header: send x-bf-api-key-id: <key-id> on the request. The gateway will look up the key with that ID.
• Context (Go SDK):

ctx := context.Background()
ctx = context.WithValue(ctx, schemas.BifrostContextKeyAPIKeyID, "key-uuid-1234")

By Name

• Header: send x-bf-api-key: <key-name> on the request. The gateway will look up the named key and use its secret for the upstream provider call.
• Context (Go SDK):

ctx := context.Background()
ctx = context.WithValue(ctx, schemas.BifrostContextKeyAPIKeyName, "openai-key-1")

Note: Both mechanisms reference a stored key (not the raw secret). The gateway resolves the key against configured provider keys and applies model allowlists, denylists, and deployment mapping. When an explicit key ID or name is supplied, weighted selection is bypassed and the referenced key is used directly. The pin applies only to the first provider of the chain, its retries included, a key pinned by a routing rule takes its place, and a direct key takes precedence over both; see API key selection.

# Example: request referencing a stored key name that doesn't exist
curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-bf-api-key: non_existant_key" \
  -d '{
    "model": "anthropic/claude-haiku-4-5",
    "messages": [{"role": "user", "content": "Hello, Bifrost!"}]
  }'

Response (example):

{"is_bifrost_error":false,"error":{"error":"no key found with name \"non_existant_key\" for provider: anthropic","message":"no key found with name \"non_existant_key\" for provider: anthropic"},"extra_fields":{"provider":"anthropic","model_requested":"claude-haiku-4-5","request_type":"chat_completion"}}

Example: request referencing a stored key name that exists but no configured keys support the requested model

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-bf-api-key: key_with_model_disabled" \
  -d '{
    "model": "anthropic/claude-sonnet-4-5",
    "messages": [{"role": "user", "content": "Hello, Bifrost!"}]
  }'

Response (example):

{"is_bifrost_error":false,"error":{"error":"no keys found that support model: claude-sonnet-4-5","message":"no keys found that support model: claude-sonnet-4-5"},"extra_fields":{"provider":"anthropic","model_requested":"claude-sonnet-4-5","request_type":"chat_completion"}}

Note: This is not a weighted selection, by providing a specific key name you are explicitly telling Bifrost which stored key to use, so weighted distribution is bypassed. The example above demonstrates the error returned when a referenced key name cannot be resolved.

Direct Keys

A caller can send its own provider key instead of using the keys stored in Bifrost. Two things must both be set:

• Allow Direct API Keys turned on under Config > Security in the UI, or client.allow_direct_keys: true in config.json. It is off by default.
• The request header x-bf-direct-key: true. The key is then read from Authorization: Bearer, x-api-key or x-goog-api-key. A virtual key in those headers is never treated as a direct key.

From the Go SDK, set schemas.BifrostContextKeyDirectKey on the context; no flag is needed.

A direct key is used as given: it skips key selection, model filtering and any key pin, and is used on the first provider of the chain, its retries included, and on nothing after it. Every fallback, including one on the same provider, runs on the keys stored in Bifrost. When the provider refuses the caller’s key, Bifrost returns the provider’s own error. See Direct API key for the details.

The Bedrock x-bf-bedrock-* and Azure x-bf-azure-endpoint header paths removed in v1.5 have not come back. Direct keys were reintroduced in v1.5.6 with the x-bf-direct-key header.