LLM Cost Gateway — A Spend Gate for LLM Providers

⚠ This is an open-source project, not client work. The demo numbers below come from a synthetic 30-request workload against deterministic mock providers — illustrative, not a benchmark and not a promise. The “Verified live” section is the only part that touched a real provider.

The problem

Two failure modes show up in nearly every production LLM system: spend is unbounded (a retry loop or a runaway agent can bill thousands of dollars before anyone opens a dashboard), and providers are fragile (a single vendor’s 503 takes your feature down). A dashboard tells you about the first one tomorrow; a gateway refuses the request today.

What was built

An OpenAI-compatible gateway standing in front of your model providers. Adoption is a base_url change — request and response bodies stay standard OpenAI JSON, while the gateway’s own decisions come back as x-gateway-* headers. The request path: classify into a tier (cheap/mid/frontier) → prompt-cache lookup → pre-flight budget reservation → provider chain with retry and circuit breaking → cost metering → post-flight reconciliation → cache store.

Per-tenant budgets are hard and enforced before the request leaves the building: a tenant past its cap gets a structured 429 without a single upstream call being made. Usage aggregates per tenant, model, and tier at /v1/usage; routing policy lives in hot-reloaded YAML with validate-then-swap — a file that fails validation is rejected and the previous policy stays live.

Engineering decisions

A hard cap must be decided before the upstream call, but the true token count is only known after — so budgeting is two-phase: reserve a deliberately conservative estimate (the estimator uses 3.6 characters per token, skewing high because under-estimating is the failure mode that lets a tenant slip past a cap), then reconcile the signed difference once the provider reports real usage. A failed call releases the whole reservation: a tenant must not be billed for a 502.

Reservation is one atomic operation — a Lua script server-side on Redis, an asyncio.Lock for the in-memory backend — because read-then-write is a lost-update race. One test fires 50 concurrent reservations at a 1,000-token cap and asserts exactly 10 succeed. Retryable and non-retryable errors are distinguished explicitly: a 429/5xx traverses the fallback chain, while a 400/401 aborts the whole chain immediately — replaying a malformed request against three providers only produces three more 400s and a misleading “all providers failed” error.

And one deliberate default: prompt content is never logged. A gateway sits on the path of every prompt an organisation sends; logging it by default would silently make the gateway’s log store the most sensitive datastore in the company.

Offline demo and live verification

make demo runs a 30-request workload against deterministic mock providers: 26/30 succeeded, 4 were served by fallback while mid-primary was down, the circuit breaker opened twice and closed itself after the provider recovered, and 4 requests from the demo-capped tenant were refused with a 429 once its daily budget ran out. The cost comparison on that workload — tiered routing at $0.157879 versus $0.448200 had every request used the frontier tier — is arithmetic over one synthetic workload, stated in the README as illustrative, not a promise.

The OpenAI-compatible adapter was verified live against DeepSeek (deepseek-v4-flash) on 2026-08-16: routing, cost metering over provider-reported usage, caching, and the /v1/usage report were all confirmed end to end. That live run surfaced two things the mock could not: an authentication 401 correctly aborting the chain (rather than silently spending money at a second vendor), and a retired model alias pinned in the policy file. The Anthropic and Gemini adapters are written from documented API contracts and are honestly labelled untested against the live APIs.

Run it yourself

make install
make demo     # scripted workload, offline, no keys, no Redis
make test     # 222 tests, all offline
make run      # serve at http://localhost:8080 as an OpenAI endpoint

Public repo: github.com/ridwanspace/llm-cost-gateway.