Eval harness

What you'll build

Step-by-step build

1. 1

   Build the test set

   Collect 30 to 100 real inputs that represent what your app actually sees, and for each write the expected answer or a clear grading rule. Include the tricky and edge cases that have burned you before. This dataset is the whole foundation, so make it honest and versioned.

2. 2

   Write the runner

   Build a loop that takes each input, runs it through your app or prompt, and records the raw output. Make the prompt and model configurable so you can point the same runner at different versions. Store every output; you will grade it in the next step.

3. 3

   Add scorers

   Implement several grading functions: exact match, substring contains, JSON-schema-valid, and an LLM-as-judge that rates open-ended answers against a rubric. Each case declares which scorer applies. Keep scorers pure and deterministic where possible so runs are reproducible.

4. 4

   Aggregate the score

   Roll the per-case results into one headline number, like percent passing, plus a breakdown by scorer and by slice. Show which specific cases failed, not just the total. A single comparable number is what makes changes decidable.

5. 5

   Save runs by version

   Persist each run to Postgres or Atlas tagged with the prompt version, model, and timestamp. Never overwrite; every run is a historical record. This is what lets you compare today against last week.

6. 6

   Diff two runs

   Build a view that compares any two runs and highlights cases that newly passed or newly failed. The regressions are the point; a change can raise the average while breaking specific important cases. Make new failures impossible to miss.

7. 7

   Wire it into CI

   Add a GitHub Actions job that runs the eval on every pull request and fails the build if the score drops below a set threshold. Now a bad prompt or model change cannot merge silently. This turns the harness from a tool into a guardrail.

8. 8

   Deploy and document

   Host the report UI, run the harness against a real prompt change, and capture the before/after diff. Write a README explaining the dataset, the scorers, and the CI gate. Show a caught regression, because that is the proof it works.

Done when

• ✓One command runs the whole dataset and prints a single comparable score.
• ✓Changing the prompt or model and rerunning shows a clear before/after difference.
• ✓The diff view surfaces cases that newly failed even when the average went up.
• ✓A CI run fails the build when the score drops below the threshold.

Ship it

Run the harness as a CLI or FastAPI service with datasets and runs in Postgres or Atlas, and host the report and diff UI on Vercel. Wire the eval into GitHub Actions so it gates every change. Put the dataset description, the scorers, and a screenshot of a caught regression in the README.

What it proves: You can measure LLM quality systematically and catch regressions before they ship, the discipline that lets a team change prompts and models with confidence.

Paste this into Cursor, Claude, or ChatGPT and build it step by step.

You are my senior AI engineer pair. Help me build "Eval harness" step by step. Stack: a CLI or FastAPI runner, Groq free tier as both the model under test and the LLM-as-judge scorer, Postgres or MongoDB Atlas (free M0) for datasets and runs, Next.js for the report UI, Pydantic or Zod for schema scoring, and GitHub Actions for CI.

An eval harness is a repeatable way to run my app over a fixed test set and grade the outputs so I can compare before and after a change.

Requirements:
1. A versioned dataset of 30 to 100 inputs, each with an expected answer or grading rule.
2. A runner that executes a configurable prompt or model over every case and stores the output.
3. Multiple scorers: exact match, contains, JSON-schema-valid, and LLM-as-judge (a model grading outputs against a rubric).
4. One aggregate score plus a per-case pass/fail breakdown, saved per run and tagged by version.
5. A diff view of two runs that highlights new failures, and a CI gate that fails on a score drop.

Work in this order: dataset, runner, scorers, aggregation, run storage, diff view, then CI. STOP after each step so I can test. Do not write the whole app at once.