Building an AI Agent Eval Harness: Testing Pipelines

Constructing an AI agent capable of executing straightforward prompts inside a local development notebook is a rewarding exercise. However, transitioning that agent into a production environment frequently devolves into an engineering maintenance trap. The most prevalent vulnerability confronting engineering teams is the phenomenon of silent regression: you edit a single phrase in a system prompt to correct a formatting nuance, only to inadvertently fracture tool-calling sequences across your automated billing or document-parsing pipelines.

The only reliable strategy to prevent silent regressions across autonomous agent systems is treating continuous evaluation as an automated requirement of your continuous integration (CI/CD) pipeline, mirroring how unit testing protects traditional codebases. This automated verification architecture is termed an evaluation harness (eval harness). This tutorial provides a systematic, step-by-step blueprint for designing, implementing, and running a deterministic eval harness for multi-step AI agents, complete with trajectory metrics, semantic judge bias mitigation, and executable test code.

Why are conventional unit tests insufficient for AI agents?

Conventional unit tests are insufficient for AI agents because large language models are non-deterministic, generating probabilistic natural language outputs that cannot be reliably validated using rigid equality assertions like assert actual == expected. In deterministic software systems, a function compiling an accounting summary always returns an identical string or numerical object when provided with identical input arguments.

When testing autonomous agent systems, three dimensions of operational variance cause traditional unit test suites to fail:

First, syntactic variance. An agent configured to resolve customer service tickets can generate ten distinct phrased responses that are all factually accurate and professionally sound, yet none of them will match an exact string literal written into a test assertion. Enforcing rigid string equality produces endless false alarms that erode developer trust in testing suites.

Second, multi-step trajectory complexity. An agent does not merely produce a final text completion; it reasons through intermediate steps, invokes an initial tool, parses the environment response, determines if a secondary tool is required, and only then composes a final response. A subtle reasoning error at step two can induce an infinite execution loop or trigger tools out of order.

Third, upstream model drift. When upstream model providers update model checkpoints (such as minor backend adjustments across commercial LLM tiers), output probability distributions shift. A system prompt that achieved a 98 percent tool-dispatch success rate yesterday can drop to 80 percent without a single character changing in your private git repository.

Consequently, a production-grade eval harness combines three testing tiers: deterministic schema validation on tool arguments, stateful trajectory matching on tool invocation sequences, and structured semantic grading using rubric-governed evaluator models.

The structural anatomy of an agent evaluation harness

A complete agent evaluation harness consists of four interconnected architectural components:

1. Golden Evaluation Dataset: A curated repository of test cases stored in JSON or YAML formats. Each record defines user input prompts, environmental context prerequisites, required tool execution sequences, and human-verified acceptance criteria.

2. Isolated Test Execution Runner: A software module that executes the candidate agent against the golden dataset inside a controlled sandbox. The runner mocks out external network dependencies, paid APIs, and persistent database mutations, guaranteeing that test iterations remain fully reproducible and cost-controlled.

3. Scoring Engine and Evaluators: A suite of modular grading components that inspect agent output. These evaluators range from deterministic schema validators (confirming that JSON payloads match required Pydantic structures), to trajectory verifiers (confirming that tool A was called before tool B), and rubric-bound semantic evaluators.

4. Telemetry and Regression Tracking Dashboard: Observability platforms such as the open-source Langfuse engineering platform or the Arize Phoenix framework. These systems record full invocation spans, compare execution metrics against baseline git commits, and flag regression alerts directly in code review interfaces.

The architecture connects these four pillars into a continuous testing pipeline:

flowchart LR
  Dataset["1. Golden Dataset<br/>JSON/YAML Test Cases"] --> Runner["2. Test Runner<br/>Sandboxed Environment"]
  Runner --> Agent["Agent Under Test<br/>Model + Tool Definitions"]
  Agent --> Runner
  Runner --> Scoring["3. Scoring Engine<br/>Deterministic Schemas, TCA/TOA, LLM Judge"]
  Scoring --> Observability["4. Telemetry Dashboard<br/>Langfuse / Arize"]
  Observability --> Gate["CI/CD Evaluation Gate<br/>Regression Prevention"]
  Gate -->|Score >= Threshold| Pass["Pass: Ready for Production Deployment"]
  Gate -->|Score < Threshold| Fail["Fail: Block Pull Request Merge"]

Defining agent trajectory metrics: measuring procedural decisions

When assessing an agent equipped with tool execution capabilities, evaluating only the final natural language answer is fundamentally incomplete. You must grade the validity of every intermediate reasoning step and parameter payload.

We group trajectory evaluations into four core quantitative metrics:

1. Tool Call Accuracy (TCA): The ratio of correctly dispatched tools relative to the expected tool sequence. If a test scenario demands querying an internal inventory database via query_warehouse_stock, but the agent instead dispatches an open web search, TCA evaluates to zero.

2. Parameter Precision Score (PPS): Evaluates whether arguments passed into tool functions satisfy strict structural and semantic boundaries. For example, did the model provide a properly formatted ISO date string? Are numerical values passed as integers rather than verbose currency descriptions?

3. Trajectory Order Agreement (TOA): Validates whether tool invocations respect required business logic dependencies. In logistics workflows, an agent must invoke verify_customer_identity prior to triggering cancel_consignment. If the order is reversed, the trajectory test fails even if both tools were called successfully.

4. Loop Exhaustion Ratio (LER): Measures whether an agent becomes trapped in recursive error correction cycles. If an agent attempts the same failing tool invocation more than three times consecutively without escalating to the human user, the test is marked as failed due to token budget exhaustion.

This emphasis on trajectory-level evaluation aligns with the methodology maintained in the official OpenAI Evals repository on GitHub, where models are graded on step-by-step logic contract satisfaction.

Step-by-step implementation: building a harness with Python and Pytest

To implement a maintainable eval harness that integrates directly into existing developer workflows, we combine the DeepEval library from Confident AI with standard Pytest execution.

The following practical implementation provides a robust pattern for testing a logistics fulfillment agent.

Step 1: Defining the golden evaluation dataset

Store test scenarios inside a structured JSON or YAML fixture. This file defines what the user inputs, which tools are expected, and which actions are strictly prohibited:

[
  {
    "test_id": "TC-LOG-01",
    "description": "Customer requests shipment location with valid consignment code",
    "user_input": "Hello, please check the status of my consignment JKT901234 immediately.",
    "expected_tools": ["check_shipment_status"],
    "required_params": {
      "tracking_id": "JKT901234"
    },
    "forbidden_tools": ["cancel_shipment_order", "refund_balance"],
    "expected_status": "IN_TRANSIT"
  },
  {
    "test_id": "TC-LOG-02",
    "description": "Order cancellation request missing tracking code",
    "user_input": "Cancel my order right now!",
    "expected_tools": [],
    "expected_clarification": true
  }
]

Step 2: Writing the automated test suite

Create an executable test module named test_agent_eval.py. The following illustrative code demonstrates deterministic trajectory assertions and token budget ceilings:

"""
Illustrative AI Agent Evaluation Harness
Enforces deterministic tool trajectory verification and parameter hygiene.
"""

import json
import pytest
from dataclasses import dataclass
from typing import Any

@dataclass
class ToolExecutionStep:
    tool_name: str
    arguments: dict[str, Any]

@dataclass
class AgentRunResult:
    final_output: str
    trajectory: list[ToolExecutionStep]
    total_tokens_used: int

class MockLogisticsAgent:
    """Simulated logistics agent implementation for test demonstration (illustrative)."""
    async def run(self, message: str) -> AgentRunResult:
        # Mock logic: detect valid tracking code and trigger query tool
        if "JKT901234" in message:
            return AgentRunResult(
                final_output="Package JKT901234 is currently at the Jakarta Transit Center.",
                trajectory=[
                    ToolExecutionStep(
                        tool_name="check_shipment_status",
                        arguments={"tracking_id": "JKT901234"}
                    )
                ],
                total_tokens_used=420
            )
        return AgentRunResult(
            final_output="Please provide your tracking consignment code so we can look up your package.",
            trajectory=[],
            total_tokens_used=180
        )

# Load test cases from the golden evaluation fixture
def load_golden_dataset():
    raw_data = """
    [
      {
        "test_id": "TC-LOG-01",
        "user_input": "Hello, please check the status of my consignment JKT901234.",
        "expected_tools": ["check_shipment_status"],
        "required_params": {"tracking_id": "JKT901234"}
      }
    ]
    """
    return json.loads(raw_data)

@pytest.mark.asyncio
@pytest.mark.parametrize("scenario", load_golden_dataset())
async def test_agent_trajectory_conformance(scenario):
    """
    Validates that agent execution strictly conforms to the golden scenario:
    1. Dispatches expected tools.
    2. Validates parameter inputs against hallucinations.
    3. Respects token budget ceilings.
    """
    agent = MockLogisticsAgent()
    result = await agent.run(scenario["user_input"])

    executed_tools = [step.tool_name for step in result.trajectory]

    # Evaluation 1: Tool Call Accuracy
    for expected_tool in scenario["expected_tools"]:
        assert expected_tool in executed_tools, (
            f"Test failure in {scenario['test_id']}: Expected tool '{expected_tool}' was not invoked. "
            f"Actual invocations: {executed_tools}"
        )

    # Evaluation 2: Parameter Precision Score
    for step in result.trajectory:
        if step.tool_name in scenario.get("required_params", {}):
            expected_arg = scenario["required_params"][step.tool_name]
            assert step.arguments.get(step.tool_name) == expected_arg, (
                f"Parameter mismatch in {scenario['test_id']}: "
                f"Expected {expected_arg}, got {step.arguments.get(step.tool_name)}"
            )

    # Evaluation 3: Efficiency Ceiling Check
    assert result.total_tokens_used < 1500, (
        f"Execution in {scenario['test_id']} exceeded token budget: {result.total_tokens_used}"
    )

With this Pytest module in place, developers can run pytest test_agent_eval.py locally. If any prompt edit breaks the expected tool-dispatch logic, the test fails in seconds before code leaves the developer machine.

Mitigating bias and volatility in LLM-as-a-Judge grading

Utilizing large language models to grade other language models (LLM-as-a-judge) is valuable for assessing qualitative attributes such as politeness, clarity, and safety compliance. However, relying on semantic judges without safeguards introduces well-documented cognitive distortions:

1. Self-Enhancement Bias: Models developed by a specific provider reliably award higher marks to completions generated by models of the same family. GPT models systematically favor GPT outputs over Claude or open-weight models, and vice versa.

2. Position Bias: When performing comparative A/B evaluations, models exhibit a persistent tendency to select whichever candidate completion is presented in the first slot (primacy effect).

3. Verbosity Bias: Evaluator models exhibit an inclination to award higher scores to wordy, elaborate explanations, even when a concise two-sentence response is far more practical for the end user.

To prevent evaluation distortion, enforce three defensive testing conventions:

First, mandate deterministic validation as the primary gate. Schema conformance, HTTP response codes, and argument boundary checks must be verified using standard Python logic. The LLM judge should only evaluate subjective nuances after deterministic checks have cleared.

Second, apply constrained binary rubrics. Never ask a judge model to assign an ambiguous score from 1 to 10. Require strict boolean judgments based on explicit operational criteria: “Output TRUE if the agent explicitly requested a tracking number; output FALSE if the agent attempted to guess the status without an identifier”.

Third, calibrate comparative tests with position swapping. When running pairwise comparisons, execute the test twice with candidate positions reversed. If the judge model picks candidate A in the first run and switches to candidate B when positions are swapped, the result is flagged as indeterminate and sent to human review.

This layered evaluation discipline forms a core capability of our AI consulting service and the llm-eval-harness project.

Integrating the evaluation harness into GitHub Actions CI/CD

Agent evaluations cannot remain isolated manual scripts run haphazardly on developer laptops. They must execute automatically on every pull request before modifications merge into the primary release branch.

The following GitHub Actions workflow definition (.github/workflows/agent-eval.yml) executes the evaluation suite on relevant code changes:

name: Agent Continuous Evaluation Gate

on:
  pull_request:
    branches: [main]
    paths:
      - "src/agents/**"
      - "prompts/**"
      - "tests/evals/**"

jobs:
  run-eval-harness:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Source Code
        uses: actions/checkout@v4

      - name: Setup Python Environment
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"
          cache: "pip"

      - name: Install Testing Dependencies
        run: |
          python -m pip install --upgrade pip
          pip install pytest pytest-asyncio deepeval langfuse

      - name: Execute Deterministic Agent Evals
        env:
          EVAL_ENVIRONMENT: ci-sandbox
          LANGFUSE_PUBLIC_KEY: ${{ secrets.LANGFUSE_PUBLIC_KEY }}
          LANGFUSE_SECRET_KEY: ${{ secrets.LANGFUSE_SECRET_KEY }}
        run: |
          pytest tests/evals/test_agent_eval.py -v --junitxml=reports/eval-results.xml

      - name: Enforce Pass Rate Ceiling
        run: |
          echo "Verifying minimum 95% evaluation pass rate threshold..."

By embedding this evaluation workflow into your CI/CD pipeline, your organization establishes a hard quality gate. If a proposed prompt tweak causes trajectory success rates to slip from 98 percent to 90 percent, the automated build breaks, blocking defective logic from reaching production users.

Commencing with an honest, minimal test corpus

Constructing a dependable evaluation harness does not require writing hundreds of test cases on day one. The most frequent trap teams encounter is deferring evaluation work while waiting for a complete test suite.

Start by capturing ten real-world failure cases (production post-mortems). Every time your agent encounters an edge case, hallucinates an argument, or enters an execution loop in production, extract that trace into your golden evaluation dataset. Author deterministic assertions to ensure that specific failure mode can never recur.

Over time, that initial set of ten test cases will compound into an authoritative regression safety net, empowering your team to evaluate newer model releases, refine operational prompts, and scale agentic capabilities with absolute confidence.

If your engineering team is looking to design an automated agent evaluation harness, audit production agent reliability, or build a scalable continuous evaluation pipeline, connect with our practitioners through our contact page.