One distinction worth drawing out: for single-turn systems evals measure output quality, but for multi-step agents you also need to score the trajectory. Two runs can return the same correct final answer via very different tool-call sequences, and the riskier path only surfaces if you evaluate intermediate steps rather than just the terminal score. That makes the 'score you can put next to last week's' harder to interpret — a number went up, but did the agent get there more safely or less?
Trajectory scoring is the piece most eval setups skip. A correct final answer can hide a reckless path, and you only see it when you score the intermediate tool calls. It also breaks the clean week-over-week number, because a higher terminal score can come from a riskier route. The honest dashboard tracks path safety alongside output quality.
One distinction worth drawing out: for single-turn systems evals measure output quality, but for multi-step agents you also need to score the trajectory. Two runs can return the same correct final answer via very different tool-call sequences, and the riskier path only surfaces if you evaluate intermediate steps rather than just the terminal score. That makes the 'score you can put next to last week's' harder to interpret — a number went up, but did the agent get there more safely or less?
Trajectory scoring is the piece most eval setups skip. A correct final answer can hide a reckless path, and you only see it when you score the intermediate tool calls. It also breaks the clean week-over-week number, because a higher terminal score can come from a riskier route. The honest dashboard tracks path safety alongside output quality.