LLM Flightdeck — Mission Control for LLM Operations

⚠ This is an open-source project, not client work. Every number quoted below comes from offline runs on the three upstream repos’ deterministic mock providers — not measurements of any production system — and the project README publishes the command that reproduces each one.

The problem

Once a business starts running LLMs, four operational questions arrive at once: how much are we spending, did the last prompt change break anything, which provider is unhealthy right now, and how resistant is this system to prompt injection. The answers are scattered across different tools, or do not exist yet. Flightdeck is a mission-control layer over three of our other engineering repos — llm-eval-harness, llm-cost-gateway, and prompt-injection-bench: it ingests their real run artifacts and renders one dashboard for all four questions.

What was built

Four pages over artifacts validated by Zod schemas:

• Overview — cross-tool KPIs: spend, worst budget headroom, latest eval pass rate, regression count, ASR, and currently open circuits.
• Cost — spend and token burn-down per workload phase, cache hit rate, per-tenant budget meters, and the routing policy panel.
• Evals — run history and a case-level diff: which cases flipped pass → fail, which scorers carried the drop, and which case improved.
• Security — ASR per defense preset (127-attack corpus × 9 presets × 5 repetitions) at three simulated target susceptibilities, with a category × preset matrix and per-repetition spread.

Engineering decisions

Two producers (the eval harness and the injection bench) serialize raw records but not their aggregates, so Flightdeck recomputes mean, pass rate, and ASR from those records. The recomputation formulas are unit-tested against the fixtures, so a formula that drifts from the producers’ definitions fails the test suite. No number on any page is hardcoded.

The dashboard deliberately reads static artifact snapshots rather than scraping anything live: no polling, no websocket, no connection to a running gateway. It is a read-only viewer over files, and the README states that limitation plainly.

The numbers behind the screens

The bundled fixtures deliberately contain stories, not clean data: a planted prompt regression (mean drops 0.917 → 0.860; three cases regress beyond the 0.02 tolerance, one improves), a demo-capped tenant driven into its 1,500-token daily cap, and mid-primary’s outage history (4 failures, 2 rejections, the circuit closed again after recovery). On the security page, at susceptibility 0.8 even the full defense stack lets 22.2% of attacks through; at 0.3 the same stack measured 0.0% — both reported as mock-run results, and every chart carries the label.

Run it yourself

Requires Node ≥ 20, no API keys, no database:

npm install
npm run demo   # open http://localhost:3000

Public repo: github.com/ridwanspace/llm-flightdeck. The three upstream repos are needed only to regenerate the fixtures, not to run the demo.