Provider Configuration

Multi-Provider Setup

Configure multiple providers to seamlessly switch between them. This example shows how to configure OpenAI, Anthropic, and Mistral providers.

type MyAccount struct{}

func (a *MyAccount) GetConfiguredProviders() ([]schemas.ModelProvider, error) {
    return []schemas.ModelProvider{schemas.OpenAI, schemas.Anthropic, schemas.Mistral}, nil
}

func (a *MyAccount) GetKeysForProvider(ctx *context.Context, provider schemas.ModelProvider) ([]schemas.Key, error) {
    switch provider {
    case schemas.OpenAI:
        return []schemas.Key{{
            Value:  os.Getenv("OPENAI_API_KEY"),
            Models: []string{},
            Weight: 1.0,
        }}, nil
    case schemas.Anthropic:
        return []schemas.Key{{
            Value:  os.Getenv("ANTHROPIC_API_KEY"),
            Models: []string{},
            Weight: 1.0,
        }}, nil
    case schemas.Mistral:
        return []schemas.Key{{
            Value:  os.Getenv("MISTRAL_API_KEY"),
            Models: []string{},
            Weight: 1.0,
        }}, nil
    }
    return nil, fmt.Errorf("provider %s not supported", provider)
}

func (a *MyAccount) GetConfigForProvider(provider schemas.ModelProvider) (*schemas.ProviderConfig, error) {
    // Return same config for all providers
    return &schemas.ProviderConfig{
            NetworkConfig:            schemas.DefaultNetworkConfig,
            ConcurrencyAndBufferSize: schemas.DefaultConcurrencyAndBufferSize,
    }, nil
}

If Bifrost receives a new provider at runtime (i.e., one that is not returned by GetConfiguredProviders() initially on bifrost.Init()), it will set up the provider at runtime using GetConfigForProvider(), which may cause a delay in the first request to that provider.

Making Requests

Once providers are configured, you can make requests to any specific provider. This example shows how to send a request directly to Mistral’s latest vision model. Bifrost handles the provider-specific API formatting automatically.

response, err := client.ChatCompletionRequest(schemas.NewBifrostContext(context.Background(), schemas.NoDeadline), &schemas.BifrostChatRequest{
    Provider: schemas.Mistral,
    Model:    "pixtral-12b-latest",
    Input:    messages,
})

Environment Variables

Set up your API keys for the providers you want to use:

export OPENAI_API_KEY="your-openai-api-key"
export ANTHROPIC_API_KEY="your-anthropic-api-key"
export CEREBRAS_API_KEY="your-cerebras-api-key"
export DEEPSEEK_API_KEY="your-deepseek-api-key"
export MISTRAL_API_KEY="your-mistral-api-key"
export GROQ_API_KEY="your-groq-api-key"
export COHERE_API_KEY="your-cohere-api-key"

Advanced Configuration

Weighted Load Balancing

Distribute requests across multiple API keys or providers based on custom weights. This example shows how to split traffic 70/30 between two OpenAI keys, useful for managing rate limits or costs across different accounts.

func (a *MyAccount) GetKeysForProvider(ctx *context.Context, provider schemas.ModelProvider) ([]schemas.Key, error) {
    switch provider {
    case schemas.OpenAI:
        return []schemas.Key{{
            Value:  os.Getenv("OPENAI_API_KEY_1"),
            Models: []string{},
            Weight: 0.7, // 70% of requests
        },
        {
            Value:  os.Getenv("OPENAI_API_KEY_2"),
            Models: []string{},
            Weight: 0.3, // 30% of requests
        },
        }, nil
    }
    return nil, fmt.Errorf("provider %s not supported", provider)
}

Model-Specific Keys

Use different API keys for specific models, allowing you to manage access controls and billing separately. This example uses a premium key for advanced reasoning models (o1-preview, o1-mini) and a standard key for regular GPT models.

func (a *MyAccount) GetKeysForProvider(ctx *context.Context, provider schemas.ModelProvider) ([]schemas.Key, error) {
    switch provider {
    case schemas.OpenAI:
        return []schemas.Key{
            {
                Value:  os.Getenv("OPENAI_API_KEY"),
                Models: []string{"gpt-4o", "gpt-4o-mini"},
                Weight: 1.0,
            },
            {
                Value:  os.Getenv("OPENAI_API_KEY_PREMIUM"),
                Models: []string{"o1-preview", "o1-mini"},
                Weight: 1.0,
            },
        }, nil
    }
    return nil, fmt.Errorf("provider %s not supported", provider)
}

Custom Base URL

Override the default API endpoint for a provider. This is useful for connecting to self-hosted models, local development servers, or OpenAI-compatible APIs like vLLM, Ollama, or LiteLLM.

func (a *MyAccount) GetConfigForProvider(provider schemas.ModelProvider) (*schemas.ProviderConfig, error) {
	switch provider {
	case schemas.OpenAI:
		return &schemas.ProviderConfig{
			NetworkConfig: schemas.NetworkConfig{
				BaseURL: "http://localhost:8000/v1", // Custom endpoint
			},
			ConcurrencyAndBufferSize: schemas.DefaultConcurrencyAndBufferSize,
		}, nil
	}
	return nil, fmt.Errorf("provider %s not supported", provider)
}

For self-hosted providers like Ollama and SGL, BaseURL is required. For standard providers, it’s optional and overrides the default endpoint.

Private Network Access

If your provider runs on a private IP address (e.g. a local vLLM or Ollama instance on 192.168.x.x or 10.x.x.x), set AllowPrivateNetwork: true in NetworkConfig. Link-local addresses (169.254.x.x) remain blocked regardless of this setting.

func (a *MyAccount) GetConfigForProvider(provider schemas.ModelProvider) (*schemas.ProviderConfig, error) {
	switch provider {
	case schemas.OpenAI:
		return &schemas.ProviderConfig{
			NetworkConfig: schemas.NetworkConfig{
				BaseURL:              "http://192.168.1.195:8000",
				AllowPrivateNetwork:  true,
			},
			ConcurrencyAndBufferSize: schemas.DefaultConcurrencyAndBufferSize,
		}, nil
	}
	return nil, fmt.Errorf("provider %s not supported", provider)
}

Managing Retries

Configure retry behavior for handling temporary failures and rate limits. This example sets up exponential backoff with up to 5 retries, starting with 1ms delay and capping at 10 seconds - ideal for handling transient network issues.

For a full explanation of how retries work, key rotation on rate limits, and how retries connect with fallbacks, see Retries & Fallbacks.

func (a *MyAccount) GetConfigForProvider(provider schemas.ModelProvider) (*schemas.ProviderConfig, error) {
	switch provider {
	case schemas.OpenAI:
		return &schemas.ProviderConfig{
			NetworkConfig: schemas.NetworkConfig{
				MaxRetries:          5,
				RetryBackoffInitial: 1 * time.Millisecond,
				RetryBackoffMax:     10 * time.Second,
			},
			ConcurrencyAndBufferSize: schemas.DefaultConcurrencyAndBufferSize,
		}, nil
	}
	return nil, fmt.Errorf("provider %s not supported", provider)
}

Custom Concurrency and Buffer Size

Fine-tune performance by adjusting worker concurrency and queue sizes per provider (defaults are 1000 workers and 5000 queue size). This example gives OpenAI higher limits (100 workers, 500 queue) for high throughput, while Anthropic gets conservative limits to respect their rate limits.

func (a *MyAccount) GetConfigForProvider(provider schemas.ModelProvider) (*schemas.ProviderConfig, error) {
    switch provider {
    case schemas.OpenAI:
        return &schemas.ProviderConfig{
            NetworkConfig: schemas.DefaultNetworkConfig,
            ConcurrencyAndBufferSize: schemas.ConcurrencyAndBufferSize{
                MaxConcurrency: 100, // Max number of concurrent requests (no of workers)
                BufferSize:     500, // Max number of requests in the buffer (queue size)
            },
        }, nil
    case schemas.Anthropic:
        return &schemas.ProviderConfig{
            NetworkConfig: schemas.DefaultNetworkConfig,
            ConcurrencyAndBufferSize: schemas.ConcurrencyAndBufferSize{
                MaxConcurrency: 25,
                BufferSize:     100,
            },
        }, nil
    }
    return nil, fmt.Errorf("provider %s not supported", provider)
}

Custom Headers

Bifrost supports two ways to add custom headers to provider requests: static headers configured at the provider level, and dynamic headers passed per-request via context.

Static Headers (Provider Level)

Configure headers that are automatically included in every request to a specific provider using NetworkConfig.ExtraHeaders:

func (a *MyAccount) GetConfigForProvider(provider schemas.ModelProvider) (*schemas.ProviderConfig, error) {
	switch provider {
	case schemas.OpenAI:
		return &schemas.ProviderConfig{
			NetworkConfig: schemas.NetworkConfig{
				ExtraHeaders: map[string]string{
					"x-custom-org":   "my-organization",
					"x-environment":  "production",
				},
			},
			ConcurrencyAndBufferSize: schemas.DefaultConcurrencyAndBufferSize,
		}, nil
	}
	return nil, fmt.Errorf("provider %s not supported", provider)
}

Dynamic Headers (Per Request)

Send custom headers with individual requests by adding them to the request context. Headers are automatically propagated to the provider:

import (
    "context"
    "github.com/maximhq/bifrost/core/schemas"
)

func makeRequestWithCustomHeaders() {
    // Create base context
    ctx := context.Background()

    // Add custom headers using BifrostContextKeyExtraHeaders
    extraHeaders := map[string][]string{
        "user-id":         {"user-123"},
        "session-id":      {"session-abc"},
        "custom-metadata": {"value1", "value2"}, // Multiple values supported
    }
    ctx = context.WithValue(ctx, schemas.BifrostContextKeyExtraHeaders, extraHeaders)

    // Make request with custom headers
    response, err := client.ChatCompletionRequest(schemas.NewBifrostContext(ctx, schemas.NoDeadline), &schemas.BifrostChatRequest{
        Provider: schemas.OpenAI,
        Model:    "gpt-4o-mini",
        Input:    messages,
    })
    if err != nil {
        // Handle error
    }
}

How it works:

Headers are stored as map[string][]string in the context
Multiple values per header name are supported
Header names are case-insensitive and normalized to lowercase
Headers are accessible throughout the request lifecycle

Example use cases:

User identification: user-id, tenant-id
Request tracking: correlation-id, trace-id
Custom metadata: department, cost-center
A/B testing: experiment-id, variant

Security Denylist

Bifrost maintains a security denylist of headers that are never forwarded to providers, regardless of configuration:

denylist := map[string]bool{
    "proxy-authorization": true,
    "cookie":              true,
    "host":                true,
    "content-length":      true,
    "connection":          true,
    "transfer-encoding":   true,

    // prevent auth/key overrides
    "x-api-key":      true,
    "x-goog-api-key": true,
    "x-bf-api-key":   true,
    "x-bf-vk":        true,
}

This denylist is applied to both static and dynamic headers to prevent security vulnerabilities.

Setting Up a Proxy

Route requests through proxies for compliance, security, or geographic requirements. This example shows both HTTP proxy for OpenAI and authenticated SOCKS5 proxy for Anthropic, useful for corporate environments or regional access.

func (a *MyAccount) GetConfigForProvider(provider schemas.ModelProvider) (*schemas.ProviderConfig, error) {
	switch provider {
	case schemas.OpenAI:
		return &schemas.ProviderConfig{
			NetworkConfig:            schemas.DefaultNetworkConfig,
			ConcurrencyAndBufferSize: schemas.DefaultConcurrencyAndBufferSize,
			ProxyConfig: &schemas.ProxyConfig{
				Type: schemas.HttpProxy,
				URL:  "http://localhost:8000", // Proxy URL
			},
		}, nil
	case schemas.Anthropic:
		return &schemas.ProviderConfig{
			NetworkConfig:            schemas.DefaultNetworkConfig,
			ConcurrencyAndBufferSize: schemas.DefaultConcurrencyAndBufferSize,
			ProxyConfig: &schemas.ProxyConfig{
				Type:     schemas.Socks5Proxy,
				URL:      "http://localhost:8000", // Proxy URL
				Username: "user",
				Password: "password",
			},
		}, nil
	}
	return nil, fmt.Errorf("provider %s not supported", provider)
}

Send Back Raw Response

Include the original provider response alongside Bifrost’s standardized response format. Useful for debugging and accessing provider-specific metadata. Provider-level default (applies to all requests for this provider):

func (a *MyAccount) GetConfigForProvider(provider schemas.ModelProvider) (*schemas.ProviderConfig, error) {
    return &schemas.ProviderConfig{
        NetworkConfig: schemas.DefaultNetworkConfig,
        ConcurrencyAndBufferSize: schemas.DefaultConcurrencyAndBufferSize,
        SendBackRawResponse: true,
    }, nil
}

Per-request override (overrides the provider default for a single request):

ctx := context.Background()
ctx = context.WithValue(ctx, schemas.BifrostContextKeySendBackRawResponse, true) // or false to suppress

response, err := client.ChatCompletionRequest(schemas.NewBifrostContext(ctx, schemas.NoDeadline), &schemas.BifrostChatRequest{
    Provider: schemas.OpenAI,
    Model:    "gpt-4o-mini",
    Input:    messages,
})

if response.ChatResponse != nil {
    rawResp := response.ChatResponse.ExtraFields.RawResponse // original provider JSON
}

When enabled, the raw provider response appears in ExtraFields.RawResponse:

type BifrostChatResponse struct {
	ID                string                     `json:"id"`
	Choices           []BifrostResponseChoice    `json:"choices"`
	Created           int                        `json:"created"` // The Unix timestamp (in seconds).
	Model             string                     `json:"model"`
	Object            string                     `json:"object"` // "chat.completion" or "chat.completion.chunk"
	ServiceTier       string                     `json:"service_tier"`
	SystemFingerprint string                     `json:"system_fingerprint"`
	Usage             *BifrostLLMUsage           `json:"usage"`
	ExtraFields       BifrostResponseExtraFields `json:"extra_fields"`
}

type BifrostResponseExtraFields struct {
	RequestType    RequestType        `json:"request_type"`
	Provider       ModelProvider      `json:"provider"`
	ModelRequested string             `json:"model_requested"`
	Latency        int64              `json:"latency"`     // in milliseconds (for streaming responses this will be each chunk latency, and the last chunk latency will be the total latency)
	ChunkIndex     int                `json:"chunk_index"` // used for streaming responses to identify the chunk index, will be 0 for non-streaming responses
	RawResponse    interface{}        `json:"raw_response,omitempty"`
	CacheDebug     *BifrostCacheDebug `json:"cache_debug,omitempty"`
}

Send Back Raw Request

Include the original request sent to the provider alongside Bifrost’s response. Useful for debugging request transformations and verifying what was actually sent to the provider. Provider-level default (applies to all requests for this provider):

func (a *MyAccount) GetConfigForProvider(provider schemas.ModelProvider) (*schemas.ProviderConfig, error) {
    return &schemas.ProviderConfig{
        NetworkConfig: schemas.DefaultNetworkConfig,
        ConcurrencyAndBufferSize: schemas.DefaultConcurrencyAndBufferSize,
        SendBackRawRequest: true,
    }, nil
}

Per-request override (overrides the provider default for a single request):

ctx := context.Background()
ctx = context.WithValue(ctx, schemas.BifrostContextKeySendBackRawRequest, true) // or false to suppress

response, err := client.ChatCompletionRequest(schemas.NewBifrostContext(ctx, schemas.NoDeadline), &schemas.BifrostChatRequest{
    Provider: schemas.OpenAI,
    Model:    "gpt-4o-mini",
    Input:    messages,
})

if response.ChatResponse != nil {
    rawReq := response.ChatResponse.ExtraFields.RawRequest // exact JSON sent to the provider
}

When enabled, the raw provider request appears in ExtraFields.RawRequest:

type BifrostResponseExtraFields struct {
	// ... other fields
	RawRequest     interface{}        `json:"raw_request,omitempty"`
	RawResponse    interface{}        `json:"raw_response,omitempty"`
}

Store Raw Request/Response

Persist the raw provider request and response in the log record without necessarily returning them in the API response. This is orthogonal to the send-back flags - enabling this does not affect what the caller receives, and enabling send-back does not automatically store data in logs. Enable both to do both. Provider-level default (applies to all requests for this provider):

func (a *MyAccount) GetConfigForProvider(provider schemas.ModelProvider) (*schemas.ProviderConfig, error) {
    return &schemas.ProviderConfig{
        NetworkConfig: schemas.DefaultNetworkConfig,
        ConcurrencyAndBufferSize: schemas.DefaultConcurrencyAndBufferSize,
        StoreRawRequestResponse: true,
    }, nil
}

Per-request override (overrides the provider default for a single request):

ctx := context.Background()
ctx = context.WithValue(ctx, schemas.BifrostContextKeyStoreRawRequestResponse, true) // or false to disable

response, err := client.ChatCompletionRequest(schemas.NewBifrostContext(ctx, schemas.NoDeadline), &schemas.BifrostChatRequest{
    Provider: schemas.OpenAI,
    Model:    "gpt-4o-mini",
    Input:    messages,
})
// Raw data is persisted in the log record.
// ExtraFields.RawRequest/RawResponse are nil unless send-back flags are also enabled.

StoreRawRequestResponse only has effect when the logging plugin is active - raw data is written to the log record by the logging plugin. Without it, enabling this flag captures the data but nothing persists it.StoreRawRequestResponse, SendBackRawRequest, and SendBackRawResponse are orthogonal controls - enabling any one does not imply the others. Enable any combination depending on whether you need raw data in logs, in the response, or both.

Best Practices

Performance Considerations

Keys are fetched from your GetKeysForProvider implementation on every request. Ensure your implementation is optimized for speed to avoid adding latency:

func (a *MyAccount) GetKeysForProvider(ctx *context.Context, provider schemas.ModelProvider) ([]schemas.Key, error) {
    // ✅ Good: Fast in-memory lookup
    switch provider {
    case schemas.OpenAI:
        return a.cachedOpenAIKeys, nil  // Pre-cached keys
    }
    
    // ❌ Avoid: Database queries, API calls, complex algorithms
    // This will add latency to every AI request
    // keys := fetchKeysFromDatabase(provider)  // Too slow!
    // return processWithComplexLogic(keys)     // Too slow!
    
    return nil, fmt.Errorf("provider %s not supported", provider)
}

Recommendations:

Cache keys in memory during application startup
Use simple switch statements or map lookups
Avoid database queries, file I/O, or network calls
Keep complex key processing logic outside the request path

Next Steps

Streaming Responses - Real-time response generation
Tool Calling - Enable AI to use external functions
Multimodal AI - Process images, audio, and text
Core Features - Advanced Bifrost capabilities

Overview

Quick Start

Release Cadence

Migration Guides

SDK Integrations

Providers & Guides

MCP Gateway

Custom plugins

Open Source Features

Provider Configuration

Multi-Provider Setup

Making Requests

Environment Variables

Advanced Configuration

Weighted Load Balancing

Model-Specific Keys

Custom Base URL

Private Network Access

Managing Retries

Custom Concurrency and Buffer Size

Custom Headers

Static Headers (Provider Level)

Dynamic Headers (Per Request)

Security Denylist

Setting Up a Proxy

Send Back Raw Response

Send Back Raw Request

Store Raw Request/Response

Best Practices

Performance Considerations

Next Steps

​Multi-Provider Setup

​Making Requests

​Environment Variables

​Advanced Configuration

​Weighted Load Balancing

​Model-Specific Keys

​Custom Base URL

​Private Network Access

​Managing Retries

​Custom Concurrency and Buffer Size

​Custom Headers

​Static Headers (Provider Level)

​Dynamic Headers (Per Request)

​Security Denylist

​Setting Up a Proxy

​Send Back Raw Response

​Send Back Raw Request

​Store Raw Request/Response

​Best Practices

​Performance Considerations

​Next Steps

Multi-Provider Setup

Making Requests

Environment Variables

Advanced Configuration

Weighted Load Balancing

Model-Specific Keys

Custom Base URL

Private Network Access

Managing Retries

Custom Concurrency and Buffer Size

Custom Headers

Static Headers (Provider Level)

Dynamic Headers (Per Request)

Security Denylist

Setting Up a Proxy

Send Back Raw Response

Send Back Raw Request

Store Raw Request/Response

Best Practices

Performance Considerations

Next Steps