Define a completed job
Consider a field-service dispatcher assigning repairs. A plausible-looking schedule is not enough. Technicians need appropriate qualifications, travel time must be possible, and appointments must fit customer availability. Some jobs require parts that have not arrived.
A task should state what an agent receives, what it may change, and what counts as completion. For scheduling, inputs might include jobs, skills, stock, availability, and travel estimates. The output might be a proposed schedule with unresolved jobs explicitly identified.
This is a task-design example, not a reported experiment. Its purpose is to make the evaluation requirements concrete.
Separate costly mistakes
Useful scoring preserves distinctions that matter in operations. Missing a safety requirement should not disappear inside an average alongside a few minutes saved on travel. Hard constraints should be reported separately from efficiency measures.
Some checks are executable: did two appointments overlap, or was a required qualification absent? Others need expert judgment, such as whether an escalation included enough information to be actionable. An automated judge needs its own validation against that judgment.
A good benchmark tells you which work an agent can handle, which work needs review, and which mistakes remain unacceptable.
Make comparisons useful
Compare systems under stated conditions: the same task version, available tools, reference material, and execution limits. Record the model version, agent configuration, retries, and human assistance. A result without those details is difficult to reproduce.
Quality is one part of the decision. Measure completion time, model and tool costs, and how much review a person still performs. Report failed and unfinished tasks rather than quietly removing them from the denominator.
Before post-training, establish a strong baseline with suitable instructions, tools, and retrieval. That reveals which gaps require better access to information and which call for learned behavior.
Connect evaluation to training
Use observed failures to shape training examples and practice environments. Demonstrations can teach a consistent way to use tools. Reinforcement learning can reward successful outcomes when tasks have reliable scoring and an environment supports repeated attempts.
Keep final evaluation cases out of training. Where records share customers, assets, templates, or repeated incidents, a random row split may leave near-duplicates on both sides. Split along boundaries that reflect the unfamiliar work an agent will face.
Repeatedly checking one test set can also influence development. Keep a separate development set for iteration and reserve a held-out set for final comparisons. Report uncertainty when a small set cannot support a broad conclusion.
Keep the benchmark useful
A benchmark should survive the first training run. Version its cases and scoring rules, preserve comparison conditions, and add new operational exceptions deliberately. When the test changes, say so; scores from different versions are not automatically comparable.
For a software partner, this becomes a practical asset: a way to compare agent suppliers, check model upgrades, and decide where automation belongs. It connects post-training to an operational decision instead of ending at a model score.