How to compare AI models using your own tasks

Define the task before comparing models

Public benchmarks can be useful background, but they rarely match your product, data, tone, latency needs and failure tolerance. A model that is excellent at coding may be mediocre at your support classification task. A cheaper model may be good enough for extraction but risky for policy advice. OpenAI's evals guide frames evaluation as testing model outputs against criteria. That is the habit you need: collect representative tasks, define what good means, run models the same way and compare outputs with evidence.

Build a small eval set

Start with 30 to 60 examples, not thousands. Pull them from real work after removing secrets and personal data. Include easy cases, common cases, edge cases and cases where the right answer is to abstain. Each row should have an input, expected behavior, scoring notes and tags. Tags help you see patterns: long input, ambiguous request, math, citation required, safety-sensitive, formatting strict or domain jargon. If you cannot write expected behavior, the task is probably too vague to evaluate fairly.

Score with a rubric

Use a simple 0 to 3 scale. Zero means harmful or unusable. One means partially useful but wrong in an important way. Two means acceptable with edits. Three means ready or nearly ready. Add binary checks for things that must be exact: valid JSON, no unsupported citation, answer under 150 words, includes required fields, refuses disallowed request. The OpenAI models page can help identify candidates, but your rubric decides whether a candidate survives.

Example rubric

Run models fairly

Keep the prompt, tools and temperature as similar as practical. If one model needs a different instruction style, record that as part of the setup. Run examples in a stable script so you can repeat the comparison later. Save raw outputs, scores and notes. If a model fails because your prompt was unclear, fix the prompt and rerun all candidates. Do not quietly improve one model's setup and compare it to stale outputs from another.

Use context deliberately

Anthropic's context engineering guide argues for high-signal context over bloated context. Apply that to evaluations. If production will retrieve three source chunks, evaluate with three source chunks. If production will include a system policy and tool output, include them. Evaluating with cleaner context than production gives false confidence. Evaluating with messier context than production may reject a useful model unfairly.

Problems to check

• Cherry-picked prompts: examples are selected because they flatter a favorite model.
• Single-score thinking: average score hides catastrophic failures in a small tag group.
• Unpriced winner: the best answer is too expensive or slow for expected traffic.
• One-time test: the model or prompt changes later, but the evaluation is not rerun.
• Subjective drift: different reviewers score style instead of the rubric.

Next steps

Create a CSV with 40 examples and five tags. Pick two or three model candidates. Run them through the same prompt and score blindly if possible. Choose the smallest model that meets the quality bar for each task category. Keep the evaluation set in the project so future model swaps become engineering decisions rather than debates about impressive demos.

Keep winners per task

One model does not have to win everything. You might use a smaller model for routing, a stronger model for drafting, and a different model for code analysis. Record the winner by task tag. This avoids overpaying for simple calls and avoids using a cheap model where a rare but serious failure would be costly.

Reviewer calibration

Before scoring the full set, have two reviewers score the same five outputs and discuss disagreements. This exposes vague rubric language early. If reviewers cannot agree on what a two means, the final leaderboard will look precise while hiding subjective noise.

Hold out examples and track critical failures

A small pilot helps you find obvious problems; it does not establish a reliable production failure rate. Reserve some examples that you do not use while tuning prompts. Repeat ambiguous cases to see whether results vary, and review serious failures separately from the average score. For a hypothetical support classifier, an incorrect escalation on one urgent case may matter more than excellent wording on many easy tickets. Record latency distributions as well as the mean so slow responses remain visible. Keep the test input, model identifier and prompt version with each result, then rerun the held-out set before approving a change.

Sources and further reading

Documentation checked October 4, 2026. Product features and availability can change.

• OpenAI evals guide
• OpenAI models documentation
• Anthropic context engineering guide