LLM Eval Harness — Prompt-Regression Diffs for CI

⚠ This is an open-source project, not client work. The demo numbers below are produced by a deterministic mock provider and mock judge — they describe a fixture, not the accuracy of any real system. Numbers from live runs are labelled explicitly.

The problem

There is one question to answer before a prompt change ships: did this make anything worse? Most eval tooling reports a single number per run — and a single number tells you the direction of a change, never its shape. A change that lifts the mean can still break the three cases your largest customer cares about; a change that lowers the mean can be the right call if it eliminates hallucinations.

What was built

A golden-dataset evaluation harness: suites are defined in YAML, run against any OpenAI-compatible endpoint, and scored with deterministic scorers (regex, exact match, JSON Schema, latency/cost budgets) and structured LLM judges (faithfulness, relevance, your own rubric). Judges always request structured output — {"score": 0..1, "reasoning": "..."} — validated before use; nothing regex-scrapes “Score: 4/5” out of prose.

The core of the tool is llm-eval diff: two runs are matched case by case, each case is classified regressed / improved / unchanged against a tolerance band, and a gate policy you choose fails the build. Movement smaller than the tolerance is reported as unchanged, because a tool that cries “regression” over judge jitter is a tool people learn to ignore. Run artifacts are plain JSON with a schema_version, so a baseline can be archived in CI and compared weeks later.

A deliberately honest demo

make demo runs a 6-case suite against prompt v1, then v2, then diffs them. v2 adds an explicit grounding instruction and fixes the hallucinating case (student-discount, +0.450) — yet the mean drops 0.917 → 0.860 because two previously-correct answers became too terse to still be judged grounded. Three cases regressed, one improved, two unchanged; the gate fails with exit code 1 and CI blocks the merge. That specific finding — keep the grounding instruction, tune the terseness instruction — is invisible in the aggregate.

The mock judge scores grounding by content-word overlap with the context, so terser answers score lower even when correct. That is a property of this offline fixture, not a claim about real judges — and it is deliberately kept because it produces an honest mixed result rather than a demo where the new prompt wins everything.

What the live runs found

The openai adapter was verified live against DeepSeek’s OpenAI-compatible endpoint, gemini against gemini-2.5-flash-lite, and anthropic against DeepSeek’s Anthropic-compatible surface — all on 2026-08-16; api.anthropic.com itself is stated as untested. The first live runs found bugs the mock could not: a JSON-mode fallback that was dead because it matched one vendor’s error prose, and reasoning models exhausting max_tokens before answering, truncating judge responses — which looked like judge variance (three identical runs scoring 0.931 / 1.000 / 0.986) but was a token-budget bug that repeat sampling made worse, not better. Both are now regression-tested.

One more lesson from the live runs: the same six-case suite was scored 1.000 by deepseek-v4-flash and 0.944 by gemini-2.5-flash-lite. A judge model is part of your eval’s definition, not an interchangeable detail — which is why the judge model is recorded in every run artifact.

Run it yourself

make install
make demo    # the A/B regression demo above, offline, no API keys
make test    # 179 tests, offline

Public repo: github.com/ridwanspace/llm-eval-harness.