Applied AI / Model Evaluation
AI Evaluation Harness
Evaluation infrastructure that turns "it looks good" into a number, a threshold and a decision — reproducible datasets, repeatable runs, model comparison and a failure taxonomy you can act on.
01 — The problem
The demo was impressive. That is not the same as good enough to ship.
Teams put an AI feature in front of a stakeholder, it handles the examples someone thought of, and it ships. Then it meets the long tail: the ambiguous case, the adversarial input, the customer whose data looks nothing like the demo.
Without a baseline, a held-out set and an agreed acceptance threshold, nobody can say whether a prompt change made things better or worse — and every model upgrade becomes a leap of faith.
02 — The system
Same dataset, same config, same numbers — every time.
A run is a versioned dataset plus a pinned configuration plus a set of scorers. Results land in a store where any two runs can be diffed — model against model, prompt against prompt, base against adapted — with the failures grouped by type rather than listed as a wall of examples.
Evaluation datasets
Versioned, frozen, with a held-out split nothing is allowed to train or tune against.
Repeatable runs
Every run records model, parameters, prompt version and dataset hash.
Model comparison
Side-by-side on quality, cost and latency — three axes, not one.
Failure analysis
Errors clustered into a taxonomy so the fix is a decision, not a guess.
Decision thresholds
Ship / don't ship criteria agreed before the run, not negotiated after it.
Evaluation reporting
A report a non-ML stakeholder can read and a CI gate an engineer cannot ignore.
03 — Architecture
How a run executes
04 — Engineering decisions
Measurement design is the hard part.
Held-out means held out
The evaluation split is frozen and never used for prompt iteration or training. The moment you tune against it, it stops measuring anything.
Metrics chosen per decision
If a false negative costs more than a false positive, the metric has to say so. Aggregate accuracy hides exactly the errors that matter in finance and risk.
Determinism where possible
Pinned parameters and seeds; where nondeterminism is unavoidable, runs are repeated and variance is reported instead of hidden behind a single number.
A failure taxonomy, not a list
Errors are labelled by cause — missing context, format violation, hallucinated value, ambiguous ground truth. Each cause has a different fix, and one of them is fixing the dataset.
Cost and latency are first-class
A two-point quality gain that triples cost and doubles latency is a decision, not an upgrade. The harness records all three so the trade-off is explicit.
Evaluation runs in CI
A prompt or model change that drops below threshold fails the build. Evaluation that only happens when someone remembers is not a control.
05 — Reliability & controls
What the harness does when a run goes wrong.
06 — Evidence
Verified results
None published. Client evaluation results stay with the client.
Benchmarked results
- Baseline vs candidate on the public task set — measurement pending
- Cost and p95 latency per 1k items — measurement pending
- Score variance across repeated runs — measurement pending
These metrics have not been measured yet. Nothing is claimed for them until a reproducible run exists.
Simulated results
Fault injection across provider errors, rate limits and malformed outputs to confirm the harness degrades honestly.
Projected business value
Measured against the cost of the errors this catches before release — and against fine-tuning spend it can rule out entirely.
Verified, benchmarked, simulated and projected figures are kept separate on purpose. A benchmarked number is never presented as a verified client result.
07 — Technology
Technology supports the story. The architecture and the controls are the story.
08 — Repository & demo
Shipping AI decisions you can't measure?
We can define the acceptance metric, build the held-out set and find out what your current system actually scores.