Eval
Eval
Measure model quality on your own data. Compare models automatically with AutoEvals, or define your own rubrics with Direct Evals.
Model providers ship updates constantly and prompts drift. Eval gives you a repeatable way to measure model quality before and after every change, and to compare model options for a given task.
Inference platform offers two ways to run evals:
| Setup | Best for | |
|---|---|---|
| AutoEvals | None. Inference platform samples your live traffic and scores it with two fixed rubrics. | Finding out whether a better model exists for a task you already run in production. |
| Direct Evals | You pick the dataset, write the rubric, and select the models. | Measuring a quality dimension specific to your product, on a curated benchmark. |
AutoEvals: compare models on your traffic
AutoEvals answer one question: is there a better model for this workload than the one you run today? A run samples your recent production traffic from a Gateway task or a traced agent, replays each sample against a set of candidate models, scores every output with LLM judges, and publishes a model recommendation with the evidence behind it.
You do not build a dataset or write a rubric. Start a run from the Compare Models tab on any task or agent, and read the results down to the individual judged sample.
Run an AutoEval
From integration to a completed run: configuration, sampling, scoring, and limits.
Interpret the results
The recommendation, the leaderboard, and the Sample Viewer.
Direct Evals: define your own benchmark
Direct Evals put you in control of all three inputs: the dataset, the rubric, and the models. Use them when you need to measure something the fixed AutoEval rubrics do not cover, or when you want a stable benchmark that does not change as your traffic does.
If you're planning to fine-tune a custom model, run Direct Evals first. A validated rubric and eval dataset are prerequisites for training. They're the measuring stick that determines when the model has learned enough, or when to stop to prevent overfitting.
How it works
- Define a rubric - describe what "good" looks like in plain English
- Pick a dataset - samples from captured traffic or uploaded JSONL
- Select models - the candidates you want to compare
- Run the eval - each sample goes through each model, and an LLM judge scores every output
- Compare results - side-by-side scores show which model wins
Key concepts
| Concept | Description |
|---|---|
| LLM-as-a-judge | A capable LLM reads your rubric, examines the model output, and returns a scored judgment. |
| Rubric | A plain English description of a quality dimension, scored numerically. Defines what "good" means for your use case. |
| Direct rubric | The LLM judge grades the model output directly against the rubric, without comparing it to a reference answer. |
| Adherence rubric | The LLM judge grades the model output based on how closely it matches a reference response. |
| Eval dataset | A stable, curated set of challenging examples that acts as your benchmark. Pick the hard cases. |
| Offline vs online | Offline evals run against collected samples. Online evals score live traffic as it flows through. Offline is available today; online is coming soon. |
| Train-eval splits | Training and eval data must never overlap. If a model trains on eval examples, the eval becomes meaningless. See the zero-overlap rule. |