> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getbifrost.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Adaptive Load Balancing
> Advanced load balancing algorithms with predictive scaling, health monitoring, and performance optimization for enterprise-grade traffic distribution.
**Looking for comprehensive provider routing documentation?**
For a detailed guide covering how adaptive load balancing works with governance routing, the two-level architecture (provider + key selection), Model Catalog integration, and example scenarios, see the [**Provider Routing Guide**](/providers/provider-routing).
This page focuses on the technical implementation and performance characteristics of adaptive load balancing.
## Overview
**Adaptive Load Balancing** in Bifrost Enterprise automatically optimizes traffic distribution across providers and keys based on real-time performance metrics. The system operates at **two levels** - provider selection (direction) and key selection (route) - continuously monitoring error rates, latency, and throughput to dynamically adjust weights, ensuring optimal performance and reliability.
| Feature | Description |
| ------------------------------------ | -------------------------------------------------------------------------------------------------- |
| **Dynamic Weight Adjustment** | Automatically adjusts key weights based on performance metrics |
| **Real-time Performance Monitoring** | Tracks error rates, latency, and success rates per model-key combination |
| **Cross-Node Coordination** | Nodes share rate-limit (TPM) signals so an overloaded key is backed off fleet-wide within a region |
| **Circuit Breaker Integration** | Temporarily removes poorly performing keys from rotation |
| **Fast Recovery** | Recovering routes are favored so they climb back quickly after transient failures |
**Zero-overhead design**: All route selection logic adds less than **10 microseconds** to hot path latency. Weight calculations happen asynchronously every 5 seconds, so request routing uses pre-computed weights with minimal overhead.
***
## Architecture
The load balancing system operates at two levels:
* **Direction-level** (provider + model): Decides which provider to use for a given model
* **Route-level** (provider + model + key): Decides which API key to use within a provider
This two-tier approach enables both macro-level provider selection and micro-level key optimization.
```mermaid theme={null}
graph TB
Request["Incoming Request
model: gpt-4"]
subgraph DirectionSelection["Direction Selection"]
DS["Provider Selector
Score-based selection"]
DP1["OpenAI
score: 0.92"]
DP2["Azure
score: 0.85"]
DP3["Anthropic
score: 0.78"]
end
subgraph RouteSelection["Route Selection"]
RS["Key Selector
Weighted random"]
K1["Key 1
weight: 850"]
K2["Key 2
weight: 620"]
K3["Key 3
weight: 45"]
end
subgraph Tracker["Metrics Tracker"]
T["Real-time Metrics
5-second recomputation"]
M1["Error Rate"]
M2["Latency Score"]
M3["Utilization"]
end
Request --> DS
DS --> DP1 & DP2 & DP3
DP1 --> RS
RS --> K1 & K2 & K3
K1 --> Response["API Response"]
Response --> T
T --> M1 & M2 & M3
M1 & M2 & M3 -.->|"Update Weights"| DS & RS
```
***
## How Weight Calculation Works
Every 5 seconds, the system recalculates a weight for each route from its recent
performance. Three signals drive the score, in priority order:
| Factor | Role | Purpose |
| ----------------- | --------- | -------------------------------------------------------------------------------- |
| **Error Penalty** | Primary | Penalizes routes with high error rates |
| **Latency Score** | Secondary | Penalizes routes that are slow relative to their peers and to their own baseline |
| **Utilization** | Tuning | Discourages overloading any single high-performing route |
Which signals apply depends on the route's health: healthy routes are scored mainly
on errors and latency, while routes that are actively recovering are scored on latency
and recovery progress so they aren't held back by stale error history. The combined
score maps to a weight on a fixed scale - lower penalties mean higher weight, which
means more traffic - with a floor so no route is ever fully starved while it has a
chance to recover.
```mermaid theme={null}
flowchart LR
subgraph Inputs["Raw Metrics"]
E["Error Rate"]
L["Latency"]
U["Utilization"]
end
subgraph Scoring["Health-aware Scoring"]
EP["Error Penalty
primary"]
LP["Latency Score
peer + baseline"]
US["Utilization
balancing"]
end
subgraph Output["Final Weight"]
NS["Combined Score"]
FW["Route Weight"]
end
E --> EP
L --> LP
U --> US
EP & LP & US --> NS
NS --> FW
```
***
## Key Capabilities
1. **Automatic Route Health Management**: Routes automatically transition between 4 states (Healthy, Degraded, Failed, Recovering) based on error rates and latency. No manual intervention required when a route fails or recovers.
2. **Fair Traffic Distribution**: The system prevents any single route from being overloaded while still favoring better performers. Low-weight routes always get minimum traffic to prove recovery.
3. **Real-time Dashboard**: Provides visibility into weight distribution, performance metrics (error rates, latency), state transitions, and actual vs expected traffic per route.
4. **Multi-Factor Scoring**: Routes are scored from error rate (the primary, time-decayed signal), a token-aware latency score (comparing a route both to its peers and to its own recent baseline), and fair-share utilization. Recovering routes are scored to favor quick, safe recovery.
5. **Smart Key Selection**: Traffic is distributed probabilistically - higher-weight keys get proportionally more requests, but lower-weight keys keep a small share so potentially-recovered routes are continually re-probed instead of always picking the single best route.
6. **Performance Thresholds**: Pre-tuned error-rate and latency triggers drive state transitions - a route is marked Degraded at the first signs of trouble, Failed on sustained errors or a rate-limit hit, and promoted back to Healthy only after it has proven itself on live traffic.
The system is designed to be self-healing: it penalizes failing routes quickly, but also decays those penalties rapidly once issues are fixed, so a recovered route returns to full traffic within seconds.
***
## Configuration
Adaptive load balancing ships **pre-tuned** - the scoring weights, thresholds, and recovery timings are not user-configurable by design. The operator controls are four switches:
| Setting | Default | Effect |
| ----------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------- |
| **Provider selection** (`direction_selection_enabled`) | On | Whether the system picks the provider (Level 1). Off ⇒ the request's provider is honored as-is. |
| **Key selection** (`route_selection_enabled`) | On | Whether per-key (Level 2) selection is adaptive. Off ⇒ keys are chosen by static weighted-random (metrics are still tracked). |
| **Re-route failed providers** (`reroute_failed_directions`) | Off | If a pinned provider's direction is circuit-broken, re-route the request to a healthy provider for the same model. |
| **Prune failed fallbacks** (`prune_failed_fallbacks`) | Off | Drop circuit-broken providers from a request's configured fallback list. |
The two selection switches default **on**; the two failed-direction behaviors are **opt-in**. All four take effect live (no restart) and propagate across the cluster.
All four switches can be changed from the dashboard, via the API, or in `config.json`.
The load balancer settings page exposes the four switches; changes apply immediately across the cluster.
Read and update the settings through `/api/load-balancer-config`:
```bash theme={null}
# Read the current settings
curl http://localhost:8080/api/load-balancer-config
# Update the settings - always send all four fields
curl -X PUT http://localhost:8080/api/load-balancer-config \
-H "Content-Type: application/json" \
-d '{
"direction_selection_enabled": true,
"route_selection_enabled": true,
"reroute_failed_directions": true,
"prune_failed_fallbacks": false
}'
```
The update is persisted, applied to the running nodes immediately, and broadcast to cluster peers.
The `PUT` body is a full replacement, not a patch: any field omitted from the request body is set to `false`. Always send all four fields - sending only the field you want to change silently turns off the others (including the default-on selection switches).
Add a top-level `load_balancer_config` block:
```json theme={null}
{
"load_balancer_config": {
"direction_selection_enabled": true,
"route_selection_enabled": true,
"reroute_failed_directions": true,
"prune_failed_fallbacks": false
}
}
```
Unlike the API, this block is presence-aware: omitted fields keep their current values, so you only need to list the switches you want to change. Settings saved through the dashboard or the API take precedence over `config.json` values.
## Scope & Limitations
* **Per-node weights**: Each node load-balances on its own observed metrics. The only signal shared across nodes is a rate-limit (TPM) backoff, and only within the same region - there is no global weight consensus or cross-region coordination. This is deliberate: latency and error profiles differ per region, so importing another region's metrics would pollute a node's view of route health.
* **\~5-second adaptation**: Weight and state changes lag live traffic by up to one recompute cycle. Immediate per-request resilience (key rotation and fallback failover) is handled separately and is not subject to this delay.
* **Optimistic cold start**: A brand-new key or provider enters at full weight and competes at roughly fair share before it has been measured, then self-corrects within a cycle or two.
* **Relative, not absolute**: Routes are ranked against their peers, not against a fixed latency or cost target. The system is not cost-, org-, or session-aware, and does not accept manual per-key weights for the adaptive path - those concerns are handled by [governance routing](/providers/provider-routing).
***
## Next Steps
* **[Provider Routing](/providers/provider-routing)** - How adaptive load balancing composes with governance rules and the Model Catalog
* **[Circuit Breaker](./circuit-breaker)** - Header-signal-driven failover to a backup provider when a primary endpoint degrades
* **[Clustering](./clustering)** - Multi-node deployments and the gossip layer behind cross-node load balancer signals