Budget and Limits

Enterprise-grade budget management and cost control with hierarchical budget allocation through virtual keys, teams, and customers.

Overview

Budgeting and rate limiting are a core feature of Bifrost’s governance system managed through Virtual Keys.

Bifrost’s budget management system provides comprehensive cost control and financial governance for enterprise AI deployments. It operates through a hierarchical budget structure that enables granular cost management, usage tracking, and financial oversight across your entire organization.

Core Hierarchy:

Customer (has independent budget)
    ↓ (one-to-many)
Team (has independent budget) 
    ↓ (one-to-many)
Virtual Key (has independent budget + rate limits)
    ↓ (one-to-many)
Provider Config (has independent budget + rate limits)

OR

Customer (has independent budget)
    ↓ (direct attachment)
Virtual Key (has independent budget + rate limits)
    ↓ (one-to-many)
Provider Config (has independent budget + rate limits)

OR

Virtual Key (standalone - has independent budget + rate limits)
    ↓ (one-to-many)
Provider Config (has independent budget + rate limits)

Key Capabilities:

• Virtual Keys - Primary access control via x-bf-vk header (exclusive team OR customer attachment)
• Budget Management - Independent budget limits at each hierarchy level with cumulative checking
• Rate Limiting - Request and token-based throttling at both VK and provider config levels
• Provider-Level Governance - Granular budgets and rate limits per AI provider within a virtual key
• Model/Provider Filtering - Granular access control per virtual key
• Usage Tracking - Real-time monitoring and audit trails
• Audit Headers - Optional team and customer identification

---

Budget Management

Cost Calculation

Bifrost automatically calculates costs based on:

• Provider Pricing - Real-time model pricing data
• Token Usage - Input + output tokens from API responses
• Request Type - Different pricing for chat, text, embedding, speech, transcription
• Cache Status - Reduced costs for cached responses
• Batch Operations - Volume discounts for batch requests

All cost calculation details are covered in Architecture > Framework > Model Catalog.

Budget Checking Flow

When a request is made with a virtual key, Bifrost checks all applicable budgets independently in the hierarchy. Each budget must have sufficient remaining balance for the request to proceed.

Checking Sequence:

For VK → Team → Customer:

1. ✓ Provider Config Budget (if provider config has budget)
2. ✓ VK Budget (if VK has budget)
3. ✓ Team Budget (if VK's team has budget)  
4. ✓ Customer Budget (if team's customer has budget)

For VK → Customer (direct):

1. ✓ Provider Config Budget (if provider config has budget)
2. ✓ VK Budget (if VK has budget)
3. ✓ Customer Budget (if VK's customer has budget)

For Standalone VK:

1. ✓ Provider Config Budget (if provider config has budget)
2. ✓ VK Budget (if VK has budget)

Important Notes:

• All applicable budgets must pass - any single budget failure blocks the request
• Budgets are independent - each tracks its own usage and limits
• Costs are deducted from all applicable budgets - same cost applied to each level
• Rate limits checked at provider config and VK levels - teams and customers have no rate limits
• Provider selection - providers that exceed their budget or rate limits are excluded from routing

Example:

- Provider config budget: $4/$5 remaining ✓
- VK budget: $9/$10 remaining ✓
- Team budget: $15/$20 remaining ✓  
- Customer budget: $45/$50 remaining ✓
- Result: Allowed (no budget is exceeded)

- After request: 
    - Request cost: $2 
    - Updated Provider=$6/$5, VK=$11/$10, Team=$17/$20, Customer=$47/$50
    - Then the next request will be blocked (both provider and VK budgets exceeded).

Rate Limiting

Rate limits protect your system from abuse and manage traffic by setting thresholds on request frequency and token usage over a specific time window. Rate limits can be configured at both the Virtual Key level and Provider Config level for granular control.

Bifrost supports two types of rate limits that work in parallel:

• Request Limits: Control the maximum number of API calls that can be made within a set duration (e.g., 100 requests per minute).
• Token Limits: Control the maximum number of tokens (prompt + completion) that can be processed within a set duration (e.g., 50,000 tokens per hour).

Rate Limit Hierarchy

Rate limits are checked in hierarchical order:

1. ✓ Provider Config Rate Limits (if provider config has rate limits)
2. ✓ Virtual Key Rate Limits (if VK has rate limits)

For a request to be allowed, it must pass both the request limit and token limit checks at all applicable levels. If a provider config exceeds its rate limits, that provider is excluded from routing, but other providers within the same virtual key remain available.

Provider-Level Rate Limiting

Provider configs within a virtual key can have independent rate limits, enabling:

• Per-Provider Throttling: Different rate limits for OpenAI vs Anthropic
• Provider Isolation: Rate limit violations on one provider don’t affect others
• Granular Control: Fine-tune limits based on provider capabilities and costs

Reset Durations

Budgets and rate limits support flexible reset durations:

Format Examples:

• 1m - 1 minute
• 5m - 5 minutes
• 1h - 1 hour
• 1d - 1 day
• 1w - 1 week
• 1M - 1 month

Common Patterns:

• Rate Limits: 1m, 1h, 1d for request throttling
• Budgets: 1d, 1w, 1M for cost control

---

Configuration Guide

Configure provider-level budgets and rate limits using any of these methods:

Web UI

The Bifrost Web UI provides an intuitive interface for configuring provider-level governance through the Virtual Keys management page.

Creating Virtual Keys with Provider Configs

1. Navigate to Virtual Keys: Go to Virtual Keys page in the Bifrost dashboard
2. Create New Virtual Key: Click “Create Virtual Key” button
3. Configure Providers: In the “Provider Configurations” section:
   • Add multiple providers with individual weights
   • Set provider-specific budgets and rate limits
   • Configure allowed models per provider

Provider Configuration Interface

Key Features:

• Visual Provider Cards: Each provider displays as an expandable card
• Budget Controls: Set spending limits with reset periods per provider
• Rate Limit Controls: Configure token and request limits independently
• Model Filtering: Specify allowed models for each provider
• Weight Distribution: Visual indicators for load balancing weights
• Real-time Validation: Immediate feedback on configuration errors

Monitoring Provider Usage

The info sheet for the virtual key provides real-time monitoring of:

• Budget consumption per provider
• Rate limit utilization (tokens and requests)
• Provider availability status
• Usage trends and forecasting

Provider-Level Governance Examples

Example 1: Mixed Provider Budgets

A virtual key configured with multiple providers and different budget allocations:

{
  "name": "marketing-team-vk",
  "budget": { "max_limit": 100, "reset_duration": "1M" },
  "provider_configs": [
    {
      "provider": "openai",
      "weight": 0.7,
      "budget": { "max_limit": 50, "reset_duration": "1M" }
    },
    {
      "provider": "anthropic", 
      "weight": 0.3,
      "budget": { "max_limit": 30, "reset_duration": "1M" }
    }
  ]
}

Behavior:

• OpenAI requests limited to 50 dollars/month at provider level + 100 dollars/month at VK level
• Anthropic requests limited to 30 dollars/month at provider level + 100 dollars/month at VK level
• If any provider’s budget is exhausted, all requests to that provider will be blocked

Example 2: Provider-Specific Rate Limits

Different rate limits based on provider capabilities:

{
  "name": "high-volume-vk",
  "provider_configs": [
    {
      "provider": "openai",
      "rate_limit": {
        "request_max_limit": 1000,
        "request_reset_duration": "1h",
        "token_max_limit": 1000000,
        "token_reset_duration": "1h"
      }
    },
    {
      "provider": "anthropic",
      "rate_limit": {
        "request_max_limit": 500,
        "request_reset_duration": "1h",
        "token_max_limit": 500000,
        "token_reset_duration": "1h"
      }
    }
  ]
}

Behavior:

• OpenAI: 1000 requests/hour, 1M tokens/hour
• Anthropic: 500 requests/hour, 500K tokens/hour
• If any provider’s rate limits are exceeded, all requests to that provider will be blocked

Example 3: Failover Strategy

Provider configurations with budget-based failover:

{
  "name": "cost-optimized-vk",
  "provider_configs": [
    {
      "provider": "openai-cheap",
      "weight": 1.0,
      "budget": { "max_limit": 10, "reset_duration": "1d" }
    },
    {
      "provider": "openai-premium",
      "weight": 0.0,
      "budget": { "max_limit": 50, "reset_duration": "1d" },
      "rate_limit": {
        "request_max_limit": 100,
        "request_reset_duration": "1h",
        "token_max_limit": 50000,
        "token_reset_duration": "1h"
      }
    }
  ]
}

Behavior:

• Primary: Use cheap provider until $10 daily budget exhausted
• Fallback: Automatically switch to premium provider when cheap option unavailable. To enable this, you should not send provider name in the request body, read Routing for more details.
• Cost containment: Prevent unexpected overspend on premium resources and limit the number of requests to the premium provider

Key Benefits of Provider-Level Governance

• Granular Control: Set specific spending limits and rate limits per AI provider
• Automatic Fallback: Route to alternative providers when budgets or rate limits are exceeded
• Cost Control: Track and control spending by provider for better financial oversight
• Performance Testing: A/B testing across providers with controlled budgets
• Multi-Provider Strategies: Primary/backup provider configurations
• Cost-Tiered Access: Cheap providers for basic tasks, premium for complex workloads

---

Next Steps

• Routing - Direct requests to specific AI models, providers, and keys using Virtual Keys.
• MCP Tool Filtering - Manage MCP clients/tools for virtual keys.
• Tracing - Audit trails and request tracking