Ugo ChukwuAI & Financial Systems
← Work

Applied AI / Model Evaluation

AI Evaluation Harness

Evaluation infrastructure that turns "it looks good" into a number, a threshold and a decision — reproducible datasets, repeatable runs, model comparison and a failure taxonomy you can act on.

StatusBenchmark
DomainApplied AI
RoleDesign & build
Core stackPython · CI · model APIs

01 — The problem

The demo was impressive. That is not the same as good enough to ship.

Teams put an AI feature in front of a stakeholder, it handles the examples someone thought of, and it ships. Then it meets the long tail: the ambiguous case, the adversarial input, the customer whose data looks nothing like the demo.

Without a baseline, a held-out set and an agreed acceptance threshold, nobody can say whether a prompt change made things better or worse — and every model upgrade becomes a leap of faith.

Cost of the gap

Regressions found by customers instead of by a test run.

Risk

Automated decisions about money or risk with no measured error rate.

Blocker

No evidence to justify — or refuse — a fine-tuning budget.

02 — The system

Same dataset, same config, same numbers — every time.

A run is a versioned dataset plus a pinned configuration plus a set of scorers. Results land in a store where any two runs can be diffed — model against model, prompt against prompt, base against adapted — with the failures grouped by type rather than listed as a wall of examples.

Evaluation datasets

Versioned, frozen, with a held-out split nothing is allowed to train or tune against.

Repeatable runs

Every run records model, parameters, prompt version and dataset hash.

Model comparison

Side-by-side on quality, cost and latency — three axes, not one.

Failure analysis

Errors clustered into a taxonomy so the fix is a decision, not a guess.

Decision thresholds

Ship / don't ship criteria agreed before the run, not negotiated after it.

Evaluation reporting

A report a non-ML stakeholder can read and a CI gate an engineer cannot ignore.

03 — Architecture

How a run executes

Dataset registryRun configPrompt / adapter version
Run orchestratorbatching · concurrency · resume
Model adaptershosted · open · fine-tuned
Scorersexact · structured · rubric · human sample
Results storeper-item outputs · metrics · cost · latency
Comparison reportFailure clustersCI gate
Run comparison · held-out setscoring
baselinefailures clustered by typecandidate

04 — Engineering decisions

Measurement design is the hard part.

Held-out means held out

The evaluation split is frozen and never used for prompt iteration or training. The moment you tune against it, it stops measuring anything.

Metrics chosen per decision

If a false negative costs more than a false positive, the metric has to say so. Aggregate accuracy hides exactly the errors that matter in finance and risk.

Determinism where possible

Pinned parameters and seeds; where nondeterminism is unavoidable, runs are repeated and variance is reported instead of hidden behind a single number.

A failure taxonomy, not a list

Errors are labelled by cause — missing context, format violation, hallucinated value, ambiguous ground truth. Each cause has a different fix, and one of them is fixing the dataset.

Cost and latency are first-class

A two-point quality gain that triples cost and doubles latency is a decision, not an upgrade. The harness records all three so the trade-off is explicit.

Evaluation runs in CI

A prompt or model change that drops below threshold fails the build. Evaluation that only happens when someone remembers is not a control.

05 — Reliability & controls

What the harness does when a run goes wrong.

Failure modeBehaviour
Provider error mid-runPer-item retries with backoff; the run resumes from the last completed item rather than restarting.
Rate limitingConcurrency throttled adaptively; a slow run is acceptable, a partial run silently scored is not.
Malformed model outputSchema validation failure is recorded as a scored failure class — not quietly repaired into a pass.
Incomplete runMarked partial and excluded from comparisons and CI gating until it completes.
Disputed ground truthItems flagged for review; the dataset is corrected and versioned, and affected runs are re-scored.

06 — Evidence

Project statusBenchmark

Verified results

None published. Client evaluation results stay with the client.

Benchmarked results

  • Baseline vs candidate on the public task setmeasurement pending
  • Cost and p95 latency per 1k itemsmeasurement pending
  • Score variance across repeated runsmeasurement pending

These metrics have not been measured yet. Nothing is claimed for them until a reproducible run exists.

Simulated results

Fault injection across provider errors, rate limits and malformed outputs to confirm the harness degrades honestly.

Projected business value

Measured against the cost of the errors this catches before release — and against fine-tuning spend it can rule out entirely.

Verified, benchmarked, simulated and projected figures are kept separate on purpose. A benchmarked number is never presented as a verified client result.

07 — Technology

PythonStructured outputsOpenAI / open modelsPostgreSQLGitHub ActionsDocker

Technology supports the story. The architecture and the controls are the story.

Shipping AI decisions you can't measure?

We can define the acceptance metric, build the held-out set and find out what your current system actually scores.

Book a Discovery Call