· 4 min
Eval harness basics for product teams
An evaluation harness turns “the new prompt feels better” into a comparison a team can repeat. It does not need to begin with a large platform or one universal score. Start with the actual task, a small set of representative cases and a rubric that names what a useful answer must do.
Build the test set from work, not prompt demos
Collect examples from permitted, representative inputs. Remove unnecessary identifiers, label expected behavior, and preserve hard cases: missing fields, conflicting instructions, long inputs, unsupported requests and cases that should be escalated. Split examples by meaningful category so one easy majority does not conceal a weak but important slice. Keep the set versioned; otherwise a changed dataset can look like a model improvement.
Hypothetical example: for an internal ticket router, cases might cover account access, billing, product defects and messages too vague to route. The rubric could check the route, required structured fields and whether an ambiguous case is sent to a person. It would not reward a confident guess. The categories and outcomes here are illustrative, not observed product results.
Use a rubric that matches the risk
- Use deterministic checks for schemas, required fields, valid citations, known labels and tool arguments.
- Use an explicit rubric for qualities such as relevance, tone or completeness; provide examples of passing and failing outputs.
- Have a person review a sample of high-impact cases and disagreements, especially when a grader is itself a model.
- Track refusal, escalation and unsupported-claim behavior alongside task completion.
For each run, record the model identifier, prompt or system version, retrieval snapshot, decoding settings and dataset version. Compare against a baseline and inspect regressions by category before approving a change. In production, sample real outcomes under the product’s privacy rules and add newly discovered failure modes to the next version of the set. Do not turn a benchmark into training data and then use the same cases as proof of generalization.
A harness can live in CI, a notebook or an evaluation service. Keep the test cases and results exportable so product decisions survive a vendor change. OpenAI’s dataset evaluation guide describes reusable cases and graders for iterating on prompts; check the current status of any provider-specific tool before depending on it.
When the harness shows a repeatable behavior gap, decide whether prompt, retrieval, product logic or weights are the right intervention. The fine-tuning decision guide walks through that choice. For help translating a task into testable acceptance criteria, share a few representative failure cases.