Test Harness Coverage

Per-provider matrix of features exercised by Bifrost’s end-to-end Postman/newman test harness.

Overview

Bifrost ships a comprehensive end-to-end test harness (tests/e2e/api/collections/provider-harness.json) that exercises every provider’s translation surface. This page documents which features are covered for each provider, sourced from each upstream’s official docs and verified against what’s in the harness collection today.

Total: 725 requests across 12 folders covering native API, provider-native drop-in routes (/openai, /anthropic, /bedrock, /genai), umbrella SDK drop-in shapes (/langchain, /litellm, /pydanticai, /cursor, /cohere), cross-model routing, passthrough endpoints, an endpoint × provider × modality matrix (chat / embeddings / audio / images), feature variations, and explicit backlog coverage.

[PREVIEW] tag: 25 requests prefixed [PREVIEW] are gated behind INCLUDE_PREVIEW=1 (default-skipped) because they target account/region-scoped resources (preview-model deployments, vector stores, cached content references, MCP servers, audio-capable deployments) that the harness can’t provision in arbitrary environments. Run make run-provider-harness-test INCLUDE_PREVIEW=1 to include them.

Run it with:

make run-provider-harness-test

Output:

• tmp/newman-report.html — rich HTML report
• tmp/harness-failures.md — categorized failures + coverage matrices
• Interactive viewer at http://localhost:8090 with Resend + Copy curl

Status legend

• ✅ — exercised by the harness, expected to pass against a properly-configured upstream
• ✅* — exercised, but needs an environment-side resource the harness can’t manufacture (vector store, cached content reference, real audio bytes, MCP server, preview deployment, etc.). These rows are typically [PREVIEW]-tagged in the collection so they default-skip; opt in with make run-provider-harness-test INCLUDE_PREVIEW=1
• ❌ — provider supports this feature but the harness doesn’t yet exercise it (gap; PRs welcome)

Features a provider doesn’t natively support are simply omitted from that provider’s table — there’s no N/A row, since each table only lists features within that provider’s own API surface.

Per-provider coverage

OpenAI

Feature	Status
Basic chat	✅
System message	✅
Multi-turn conversation	✅
Streaming (SSE)	✅
Stop sequences	✅
Sampling params (temperature / top_p)	✅
Logprobs / top_logprobs	✅
Seed (deterministic outputs)	✅
Predicted outputs	✅
Function calling (custom tool)	✅
Tool choice forced (required)	✅
Tool choice specific function	✅
Parallel tool calls	✅
Strict tool input	✅
MCP toolset	✅*
Web search (basic)	✅
Code interpreter (Responses API)	✅
File search (vector store)	✅*
Computer use (Responses API)	✅*
Vision (image input)	✅
PDF input	✅*
Audio input	✅*
Reasoning effort (reasoning_effort)	✅
Reasoning summary (Responses API)	✅
Structured output (json_schema)	✅
Response format JSON object	✅
Prompt caching (ephemeral)	✅
Stream options w/ usage	✅
Service tier (auto / flex / priority)	✅
Background mode (Responses async)	✅
Truncation strategy (auto)	✅
Include array (Responses)	✅
Custom tool (Responses)	✅
Skills / container	✅
Token counting (/v1/responses/input_tokens)	✅
Batch API (create / list)	✅
Files API (list)	✅
Models list	✅

Anthropic

Feature	Status
Basic chat	✅
System message	✅
Multi-turn conversation	✅
Streaming (SSE)	✅
Stop sequences	✅
Function calling (custom tool)	✅
Tool choice forced (any)	✅
Tool choice specific function	✅
Parallel tool calls	✅
Strict tool input	✅
Tool input examples	✅
Defer loading	✅
Allowed callers	✅
Eager input streaming	✅
Tool search (BM25 / regex)	✅
MCP toolset	✅*
Web search (basic / dynamic filtering / domain filter / user location)	✅
Web fetch	✅
Code execution	✅
Computer use	✅
Text editor tool	✅
Bash tool	✅
Memory tool	✅
Vision (image input)	✅
PDF input (URL or base64 source)	✅
Citations on document blocks	✅
Extended thinking (thinking.budget_tokens)	✅
Adaptive thinking (thinking.adaptive)	✅
Interleaved thinking (beta)	✅
Structured output (json_schema)	✅
Output config / effort	✅
Prompt caching (ephemeral)	✅
Prompt caching (1-hour TTL)	✅
Service tier	✅
anthropic-beta header	✅
Skills / container	✅
Context management / 1M context	✅
Compaction beta	✅
Token-efficient tools beta	✅
Fine-grained tool streaming	✅
Fast mode (Opus 4.6)	✅*
Redact thinking beta	✅
Token counting (/v1/messages/count_tokens)	✅
Batch API (create / list)	✅
Files API (list)	✅
Models list	✅

Bedrock

Feature	Status
Basic chat	✅
System message	✅
Multi-turn conversation	✅
Streaming (SSE / AWS event-stream)	✅
Stop sequences	✅
Sampling params (temperature OR top_p, not both on Anthropic models)	✅
Function calling	✅
Tool choice forced	✅
Tool choice specific function	✅
Parallel tool calls	✅
MCP toolset	✅
Web search (basic / dynamic filtering / domain filter / user location)	✅
Web fetch	✅
Code execution	✅
Computer use	✅
Text editor tool	✅
Bash tool	✅
Memory tool	✅
Vision (image; URLs auto-fetched + inlined by Bifrost)	✅
PDF input (URLs auto-fetched + inlined)	✅
Citations on document blocks	✅
Extended thinking (thinking.budget_tokens)	✅
Adaptive thinking (Opus 4.7 via /v1/chat)	✅
Interleaved thinking (beta)	✅
Structured output (json_schema via tool-mode workaround)	✅
Output config / effort	✅
Prompt caching (ephemeral)	✅
Prompt caching (1-hour TTL)	✅
anthropic-beta header passthrough	✅
Context management / 1M context	✅
Service tier (auto)	✅
Performance config (latency: optimized)	✅
Request metadata	✅
Cross-region inference (global. prefix)	✅
Cross-region inference (us. prefix)	✅
Native Converse (/bedrock/model/{m}/converse)	✅
Native InvokeModel (/bedrock/model/{m}/invoke)	✅
Model invocation jobs (Batch API equivalent)	✅
Batch API	✅

Gemini (Google AI Studio)

Feature	Status
Basic chat	✅
System message	✅
Multi-turn conversation	✅
Streaming (SSE)	✅
Stop sequences	✅
Sampling params (temperature / top_p / top_k)	✅
Presence + frequency penalty	✅
Logprobs	✅
Function calling	✅
Tool choice forced (any)	✅
Tool choice specific function	✅
Parallel tool calls	✅
Web search (basic)	✅
Code execution	✅
URL context tool	✅
Google search grounding	✅
Vision (image input)	✅
PDF input	✅*
Audio input	✅
YouTube URL input	✅
Thinking budget (thinkingConfig)	✅
Structured output (json_schema)	✅
Response MIME type	✅
Prompt caching (implicit, via cached content)	✅
Cached content reference (cachedContent: "cachedContents/{id}")	✅*
Cached contents lifecycle (POST/GET/PATCH/DELETE /v1beta/cachedContents)	❌
Cached contents list (GET /v1beta/cachedContents)	✅
Safety settings	✅
Token counting (:countTokens)	✅
Files API (list)	✅
Models list	✅

Vertex AI

Feature	Status
Basic chat	✅
System message	✅
Multi-turn conversation	✅
Streaming (SSE)	✅
Sampling params (temperature / top_p)	✅
Stop sequences	✅
Function calling	✅
Tool choice forced	✅
Parallel function calls (Gemini)	✅
Defer loading (Anthropic)	✅
Allowed callers (Anthropic)	✅
Tool search (Anthropic BM25)	✅
MCP toolset	✅*
Web search (basic / dynamic filtering / domain filter / user location)	✅
Web fetch	✅
Code execution	✅
Google search grounding	✅
Computer use (Anthropic)	✅
Text editor tool (Anthropic)	✅
Bash tool (Anthropic)	✅
Memory tool (Anthropic)	✅
Vision (image input)	✅
PDF input (URLs auto-fetched + inlined for Vertex-Anthropic)	✅
Audio input (Gemini)	✅*
Citations on document blocks	✅
Extended thinking (thinking.budget_tokens)	✅
Adaptive thinking	✅
Thinking budget (Gemini thinkingConfig)	✅
Interleaved thinking (Anthropic beta)	✅
Structured output (json_schema)	✅
Response MIME type (Gemini)	✅
Output config / effort (Anthropic)	✅
Prompt caching (ephemeral)	✅
Prompt caching (1-hour TTL)	✅
Cached contents lifecycle (POST/GET/PATCH/DELETE /v1/projects/.../cachedContents)	❌
anthropic-beta header passthrough	✅
Safety settings (Gemini)	✅
Context management / 1M context (Anthropic)	✅
Token counting	✅
Model Garden (Llama / Mistral)	✅*

Azure OpenAI

Feature	Status
Basic chat	✅
System message	✅
Multi-turn conversation	✅
Streaming (SSE)	✅
Stop sequences	✅
Sampling params (temperature / top_p)	✅
Logprobs / top_logprobs	❌
Seed (deterministic outputs)	❌
Predicted outputs	❌
Function calling	✅
Tool choice forced	✅
Tool choice specific function	❌
Parallel tool calls	✅
Strict tool input	❌
Code interpreter (Responses preview)	✅
File search (Responses preview)	✅*
Vision (image input)	✅
Audio input (gpt-4o-audio-preview)	✅*
Reasoning effort (o3 deployment)	✅*
Structured output (json_schema)	✅
Response format JSON object	❌
Service tier (auto / flex / priority)	✅
Skills / container	✅
Azure On Your Data (azure_search)	✅

Bifrost-side normalizations applied automatically (don’t appear as separate rows):

• Vision URL images on Bedrock — fetched and inlined as base64 (Bedrock Converse only accepts inline bytes)
• PDF URL documents on Bedrock — same fetch+inline path
• PDF URL documents on Vertex Claude — same (Vertex-Anthropic doesn’t accept URL document sources)
• Anthropic-style {type:"document",source:{...}} blocks on /v1/chat/completions — normalized to {type:"file",file:{...}} at JSON unmarshal so every provider’s converter sees the same shape

Modalities (8. Criss-Cross)

The harness exercises every non-chat modality across providers via both native /v1/<modality> routes and OpenAI / GenAI drop-in shapes. Cells marked N/A are not applicable (provider does not expose the modality through Bifrost, or the cell is the provider talking to itself through its own drop-in shape). Cells the harness deliberately skips at run time (encoded as [SKIP] in the collection) inherit N/A here.

Modality (route)	OpenAI	Anthropic	Bedrock	Gemini	Vertex	Azure
Embeddings, native (/v1/embeddings)	✅ (text-embedding-3-small)	N/A	✅ (titan-embed-text-v2)	✅ (gemini-embedding-001)	✅ (text-embedding-005)	✅
Embeddings via OpenAI drop-in (/openai/v1/embeddings)	N/A	N/A	✅	✅	✅	✅
Embeddings via GenAI drop-in (/genai/v1beta/models/{m}:embedContent)	✅	N/A	✅	N/A	✅	✅
Audio transcription, native (/v1/audio/transcriptions)	✅ (whisper-1)	N/A	N/A	✅ (gemini-2.5-flash)	N/A	✅
Audio transcription via OpenAI drop-in	N/A	N/A	N/A	✅	N/A	✅
Text-to-speech, native (/v1/audio/speech)	✅ (tts-1)	N/A	N/A	N/A	N/A	✅
Text-to-speech via OpenAI drop-in	N/A	N/A	N/A	N/A	N/A	✅
Image generation, native (/v1/images/generations)	✅ (gpt-image-1)	N/A	✅ (titan-image-generator-v2)	✅ (imagen-4.0)	✅ (imagen-4.0)	✅ (gpt-image-2)
Image generation via OpenAI drop-in	N/A	N/A	✅	✅	✅	✅

Folder 8 also contains a 8.6 Feature combinations sub-folder (26 requests) that exercises tool-calling, vision, JSON/structured-output, and reasoning across the /v1/chat/completions × /openai × /anthropic × /bedrock × /genai shape grid - these are tabulated in the cross-cutting matrices below.

Cross-cutting (Bifrost-specific)

These exercise Bifrost’s translation layer between provider shapes — every check uses the unified POST /v1/chat/completions endpoint with provider/model prefix routing.

Translation-layer matrices (/v1/chat/completions with provider/model prefix)

Each row is a dedicated Round N sub-folder under 12. Backlog Coverage. The number in parentheses is the count of (provider, model) cells exercised today.

Matrix	Status
Cross-model routing (50 models × 5 providers)	✅
System message matrix (11)	✅
Multi-turn matrix (11)	✅
Streaming matrix (18)	✅
Stop sequences matrix (11)	✅
Function calling matrix (19)	✅
Tool choice forced matrix (13)	✅
Vision matrix (15)	✅
PDF input matrix (4)	✅
Structured output / json_schema matrix (14)	✅
Extended / adaptive thinking matrix (6)	✅
Prompt caching matrix (6)	✅
Web search matrix (6)	✅
Code execution matrix (3)	✅

Drop-in (native-shape) matrices

These exercise the same features but against the provider-native drop-in routes (/openai, /anthropic, /bedrock, /genai) rather than /v1/chat/completions.

Matrix	Status
Drop-in streaming (9)	✅
Drop-in function calling (9)	✅
Drop-in vision (8)	✅
Drop-in structured output (8)	✅

Umbrella-SDK drop-in shapes

Bifrost exposes umbrella SDK routes that accept the OpenAI / Anthropic / Gemini / Bedrock / Cohere request shape byte-for-byte and dispatch to the right typed provider underneath. Each umbrella has its own smoke folder plus a shared vision matrix.

Umbrella route	Coverage
/cohere	Basic chat, streaming, multi-turn, tools, list models (5)
/langchain	OpenAI / Anthropic / Gemini / Bedrock / Cohere shape chat (5)
/litellm	OpenAI / Anthropic / Gemini / Bedrock / Cohere shape chat (5)
/pydanticai	OpenAI / Anthropic / Gemini / Bedrock / Cohere shape chat (5)
/cursor	OpenAI / Anthropic / Gemini / Bedrock shape chat (4)
Drop-in umbrella vision matrix (/langchain, /litellm, /pydanticai, /cursor × OpenAI/Anthropic/Gemini shapes)	✅ (12)

Bifrost-only behaviors

Feature	Status
Failover via X-Bifrost-Fallback-Models	✅
Virtual key auth via X-Bifrost-VK	✅
Sampling-params auto-strip for Opus 4.7+	✅
Computer-use generation auto-correct (4.5↔4.6 pairing)	✅
Rate limit propagation with Retry-After	❌

Passthrough surface (*_passthrough/*)

Catch-all forwarding routes that strip incoming auth headers and inject Bifrost’s configured provider key.

Feature	OpenAI	Anthropic	Bedrock	Azure	Gemini
Basic chat	✅	✅	N/A	✅	✅
Streaming	✅	✅	N/A	✅	✅
Multi-turn	N/A	✅	N/A	N/A	N/A
Vision	✅	✅	N/A	✅	✅
Function calling / tool use	N/A	✅	N/A	✅	✅
Structured output (json_schema)	✅	N/A	N/A	✅	N/A
Web search	✅	✅	N/A	N/A	✅
Code execution / code interpreter	✅	N/A	N/A	N/A	✅
Computer use	N/A	✅	N/A	N/A	N/A
Extended thinking	N/A	✅	N/A	N/A	N/A
Prompt caching	N/A	✅	N/A	N/A	N/A

Covered across the Passthrough Backlog folder + Cross-Cut Round 28: Passthrough Advanced Matrix (24 requests total).

Bedrock: passthrough is not supported by design. AWS SigV4 signing requires bifrost to sign the request with its own credentials, which fundamentally conflicts with byte-for-byte forwarding. Use the typed /bedrock/model/{modelId}/converse, /converse-stream, or /invoke routes instead — those go through bifrost’s typed Bedrock provider with proper sigv4 handling.

Vertex: no passthrough variant. Google OAuth bearer tokens are rotated per-request and can’t be bridged through a byte-for-byte forward.

Coverage by transport route

Route	What it tests
drop-in (native)	Provider-native shape via /openai, /anthropic, /bedrock, /genai. Most thorough coverage.
drop-in (umbrella)	SDK-umbrella shapes via /langchain, /litellm, /pydanticai, /cursor, /cohere. Smoke + vision matrix.
cross-model	Unified /v1/chat/completions with provider/model prefix. Tests Bifrost’s translation layer.
modality	8.3 – 8.5 exercise /v1/embeddings, /v1/audio/transcriptions, /v1/audio/speech, /v1/images/generations across providers + drop-ins.
passthrough	/*_passthrough/* byte-for-byte forwarding with auth-header strip + Bifrost key injection.

Known gaps that need external setup

These cells stay ❌ even after running because they require provider-side state the harness can’t manufacture:

• OpenAI File Search — needs a real vs_* vector store ([PREVIEW]-tagged)
• OpenAI / Azure Audio Input — needs real base64 audio bytes ([PREVIEW]-tagged)
• OpenAI / Anthropic Batch creation — needs a real input file ID
• Gemini Cached Content reference — the lifecycle endpoints (create/list/retrieve/update/delete) work end-to-end; only the referencing tests (passing cachedContents/{id} to generateContent) need a pre-provisioned cache with 32k+ tokens of content (Gemini’s minimum) ([PREVIEW]-tagged)
• Vertex Anthropic features in us-central1 — region-restricted unless GOOGLE_LOCATION=global
• Vertex preview-model deployments (Gemini-3.x, etc.) — [PREVIEW]-tagged; require account access
• Vertex-Anthropic URL document sources — Vertex doesn’t accept URL document sources upstream; bifrost auto-fetches and inlines via inlineDocumentURLs for parity with direct Anthropic
• Azure preview deployments (o3 / gpt-4o-audio-preview / computer-use-preview) — [PREVIEW]-tagged; require deployment provisioning
• MCP toolset tests — [PREVIEW]-tagged; need a reachable MCP server

Coverage report layout

When you run the harness, tmp/harness-failures.md is generated with three matrices:

1. Feature × Provider — every feature row × every provider column
2. Feature × Route — same features × drop-in / cross-model / passthrough
3. Per-(provider, model) — every distinct provider/model tuple, with the exact features each one exercised

Plus three “missing coverage” lists (per-provider, per-route, per-model) that surface gaps as a backlog.

Extending coverage

The full backlog of candidate test additions lives in tests/e2e/api/HARNESS_COVERAGE_BACKLOG.md — provider-by-provider feature inventory sourced from each provider’s official docs. Adding a new test is a one-line entry in the relevant sub-folder of provider-harness.json.