Log Exports

Offload Bifrost request and response payloads to S3 or GCS object storage while keeping searchable metadata in the logs database.

Bifrost’s log store can be paired with an object storage backend (S3 or GCS) so that large request/response payloads are streamed to durable object storage while the logs database (SQLite or Postgres) keeps only searchable metadata, indexes, and pointers. This keeps the database small and fast, makes payloads cheap to retain for long periods, and lets you query archived traffic from your own data lake.

How it works

object_storage is not a standalone feature. It is a sub-config of logs_store:

logs_store
├── enabled
├── type            (sqlite | postgres)
├── config          (SQLite path or Postgres connection)
├── object_storage              ← S3 or GCS payload offload (optional)
└── object_storage_exclude_fields

Retention is configured separately, at client_config.log_retention_days - see Retention policy below.

When logs_store.object_storage is set:

1. Bifrost writes per-request metadata (timestamps, provider, model, latency, token counts, cost, status, IDs) to the logs database.
2. Large payload fields (request body, response body, streamed chunks, tool call arguments, etc.) are uploaded to the configured bucket under prefix/.
3. The database row stores the object key, so the UI and API can fetch the payload on demand.
4. object_storage_exclude_fields lets you skip specific fields from offload (for example, when you do not want to retain raw user prompts at all).

Only S3 and GCS are supported today. Azure Blob, local filesystem, and data warehouse destinations are not implemented.

Configuration via config.json

Amazon S3

{
  "logs_store": {
    "enabled": true,
    "type": "sqlite",
    "config": {
      "path": "/app/data/logs.db"
    },
    "object_storage": {
      "type": "s3",
      "bucket": "env.AWS_S3_BUCKET",
      "region": "env.AWS_REGION",
      "access_key_id": "env.AWS_ACCESS_KEY_ID",
      "secret_access_key": "env.AWS_SECRET_ACCESS_KEY",
      "prefix": "bifrost/logs",
      "compress": true
    },
    "object_storage_exclude_fields": []
  }
}

S3 fields (all values support the env.<NAME> indirection to read from environment variables):

Field	Required	Notes
type	yes	Must be "s3".
bucket	yes	Target bucket name.
region	yes	AWS region, e.g. us-west-2.
access_key_id	conditional	Required for static credentials. Omit when using role_arn or the default credential chain (IRSA, instance profile, env).
secret_access_key	conditional	Pairs with access_key_id.
session_token	optional	For temporary STS credentials.
role_arn	optional	Assume this role instead of using static keys.
endpoint	optional	Override for S3-compatible stores (MinIO, Cloudflare R2, Wasabi).
force_path_style	optional	Set true for most S3-compatible endpoints.
prefix	optional	Object key prefix. Defaults to bifrost.
compress	optional	Gzip payloads before upload. Defaults to false.

A live example is checked in at examples/configs/withobjectstorages3/config.json.

Google Cloud Storage

{
  "logs_store": {
    "enabled": true,
    "type": "sqlite",
    "config": {
      "path": "/app/data/logs.db"
    },
    "object_storage": {
      "type": "gcs",
      "bucket": "env.GCS_BUCKET",
      "credentials_json": "env.GCS_KEY",
      "project_id": "env.GCP_PROJECT_ID",
      "prefix": "bifrost/logs",
      "compress": true
    },
    "object_storage_exclude_fields": []
  }
}

GCS fields:

Field	Required	Notes
type	yes	Must be "gcs".
bucket	yes	Target bucket name.
credentials_json	conditional	Service-account JSON (or env.GCS_KEY pointing at it). Omit when running on GKE with Workload Identity or any environment where Application Default Credentials work.
credentials	deprecated	Legacy alias for credentials_json. Use credentials_json in new configs.
project_id	optional	Useful when ADC cannot infer the project.
prefix	optional	Object key prefix. Defaults to bifrost.
compress	optional	Gzip payloads before upload. Defaults to false.

A live example is checked in at examples/configs/withobjectstoragegcs/config.json.

Choosing what stays in the database vs. what is offloaded

By default, every offloadable payload field is uploaded to object storage. object_storage_exclude_fields lets you pin specific fields to the database only, so they are never written to the bucket. Listed fields are not dropped - they continue to live in the logs DB row as before. Everything not listed is offloaded.

Values must be database column names (not JSON paths). Common choices:

Column	Contents
raw_request	The verbatim provider request body.
raw_response	The verbatim provider response body.
input_history	The full conversation sent to the model.
output_message	The model’s primary output message.

{
  "logs_store": {
    "object_storage": { "...": "..." },
    "object_storage_exclude_fields": [
      "raw_request",
      "raw_response"
    ]
  }
}

Unknown column names are silently ignored, so a typo will not error - it will just leave that field on the default (offloaded) path. Reference tests covering this behaviour live at framework/logstore/hybrid_test.go.

Typical use cases:

• Data residency: keep raw prompts and responses inside the DB (which may itself sit inside a controlled VPC) while still benefiting from offload for less sensitive fields.
• Operational queries: keep input_history in the DB so SQL queries and the UI’s search can run against the full conversation without paying an object-fetch round trip.

Retention policy

Bifrost ships a background log cleaner that deletes old logs on a fixed schedule.

Configuration

Retention is configured on the client config, not on logs_store. The active key is client_config.log_retention_days:

{
  "client_config": {
    "log_retention_days": 30
  }
}

Aspect	Value
Default	365 days
Minimum	1 day (enforced by validator)
Disable	Set to 0 (cleanup is skipped entirely)
Cleanup cadence	Every 24 hours, plus a random jitter of 15-30 minutes
Startup behaviour	A cleanup pass runs immediately when Bifrost starts
Delete batch size	100 rows per query
Per-pass timeout	30 minutes

Source: framework/logstore/cleaner.go and transports/bifrost-http/server/server.go.

What gets deleted

The cleaner deletes database rows older than now - retention_days (UTC). It deletes from the main Log table as well as the MCP tool logs table.

What does NOT get deleted automatically

Important: the retention cleaner does not delete the corresponding payloads in S3 or GCS. The hybrid log store explicitly delegates object cleanup to the bucket’s own lifecycle policy.

If you have object storage enabled, configure a lifecycle / object-lifecycle-management rule on the bucket to expire objects under your prefix/. Pick a duration that matches (or exceeds) log_retention_days.

Amazon S3 lifecycle rule (example)

{
  "Rules": [
    {
      "ID": "bifrost-logs-expire-30d",
      "Status": "Enabled",
      "Filter": { "Prefix": "bifrost/logs/" },
      "Expiration": { "Days": 30 }
    }
  ]
}

Apply with aws s3api put-bucket-lifecycle-configuration --bucket <bucket> --lifecycle-configuration file://lifecycle.json. See the AWS docs on object lifecycle management.

Google Cloud Storage lifecycle rule (example)

{
  "lifecycle": {
    "rule": [
      {
        "action": { "type": "Delete" },
        "condition": {
          "age": 30,
          "matchesPrefix": ["bifrost/logs/"]
        }
      }
    ]
  }
}

Apply with gcloud storage buckets update gs://<bucket> --lifecycle-file=lifecycle.json. See the GCS docs on Object Lifecycle Management.

Manual deletes still cascade

Single and batch deletes triggered explicitly (via the UI’s Delete log action or the underlying DeleteLog / DeleteLogs APIs) do remove the associated objects from S3 or GCS at the same time. Only the time-based retention sweep is bucket-blind.

Configuration via Helm

The Bifrost Helm chart at helm-charts/bifrost/ does not expose dedicated values for logs_store.object_storage. It renders the same config.json schema documented above, so you supply the block under the chart’s bifrost.* configuration tree (or mount your own config.json via an existing ConfigMap/Secret).

Inline values

bifrost:
  logsStore:
    enabled: true
    type: sqlite
    config:
      path: /app/data/logs.db
    objectStorage:
      type: s3
      bucket: env.AWS_S3_BUCKET
      region: env.AWS_REGION
      accessKeyId: env.AWS_ACCESS_KEY_ID
      secretAccessKey: env.AWS_SECRET_ACCESS_KEY
      prefix: bifrost/logs
      compress: true

For GCS, substitute the objectStorage block:

bifrost:
  logsStore:
    enabled: true
    type: sqlite
    config:
      path: /app/data/logs.db
    objectStorage:
      type: gcs
      bucket: env.GCS_BUCKET
      credentialsJson: env.GCS_KEY
      projectId: env.GCP_PROJECT_ID
      prefix: bifrost/logs
      compress: true

The chart converts camelCase keys to the snake_case form Bifrost expects when it writes the runtime config.json. Provide the referenced env vars (AWS_*, GCS_*) through extraEnv, envFrom, or an existing Kubernetes Secret.

BYO config.json

If you prefer to manage the full config.json yourself, mount it as a Secret and point the chart at it. Use the same logs_store.object_storage blocks shown in the config.json section verbatim.

Configuration via the UI

The web UI does not currently expose a form for logs_store.object_storage. To enable payload offload:

1. Edit config.json (or the Helm values) using the snippets above.
2. Restart Bifrost so the new log store wiring takes effect.
3. Confirm new requests are landing in the bucket by browsing the configured prefix/ path.

Once configured, the existing Logs screen in the UI transparently fetches payloads from object storage when you open a log entry. No UI changes are needed on the read path.

Verifying the setup

1. Send a request through Bifrost.
2. Confirm a row appears in the logs DB (visible via the UI’s Logs screen).
3. List the bucket under your prefix/. You should see one or more objects per request.
4. Open the log entry in the UI. The payload pane should render the content fetched from object storage.

If the bucket stays empty, check the Bifrost logs for objectstore errors. The most common causes are missing credentials, a region/endpoint mismatch, or a bucket policy that blocks the credentials’ principal.