Anthropic
Claude Code measures what a plugin adds by running without it
The eval command in Claude Code 2.1.269 runs each test case with a plugin and again without it, then reports the gap as a single number.
Anthropic added an eval command to Claude Code that runs each test case with a plugin and again without it, then reports the difference. A high score alone does not show the plugin helped, according to the documentation, because the model might do as well without it.
The command, claude plugin eval, shipped in Claude Code 2.1.269 on Sept. 11, 2026. That version is on the latest release channel. As of Sept. 12, npm’s stable tag still pointed at 2.1.236.
How it scores
A suite is a folder of cases. Each case is a prompt a user might type plus one or more graders, according to the plugin evals documentation. A grader is a pass-fail check, such as a regular expression over the reply, whether a given tool was called or a rubric that a second model judges.
Every case runs three times by default, each in a fresh, isolated session. A run’s score is the share of graders that passed, and a case’s score is the mean across its runs. The default pass threshold is 1.0, so a case has to pass every grader on every run.
When a plugin is present, the command repeats each case with no plugin loaded and reports WITH, W/OUT and the difference between them. “If a case scores 1.0 both with and without the plugin, the plugin isn’t what made it pass,” the documentation says. One case is six runs.
A companion command, claude plugin eval init, drafts a suite. It asks the author what a good result looks like and proposes cases and graders to match. Each one gets a trial run before any files are written.
Built for CI
The command writes a JSON result and an HTML report, according to the 2.1.269 changelog. In CI it signals through its exit code: 0 when every case clears the threshold, 1 when a case falls short and 2 for a partial run.
Every run and every judged grader is a model call billed to the account, the documentation says. --max-cost-usd caps the list-price estimate of those calls. The cap is checked before each run starts, so runs already in flight can carry spend past it, and a suite cut short this way exits with code 2.
The documentation also tells teams to pin the agent model in CI “so a model rollout isn’t mistaken for a plugin regression,” and to pin the judge model too.
Runs are locked down by default. Bash, Write, Edit, WebFetch and WebSearch are removed from the session unless the author grants them. MCP tool calls are answered from recorded mocks, and the plugin’s own servers are not started.
Analysis
A pass rate cannot tell an author whether the plugin did anything. A plugin that passes 90% of its cases looks like a success until the same prompts pass 90% of the time with it switched off.
Running the baseline by default puts that comparison in front of the author on the first run, rather than leaving it as an option to find later. It is not free. The baseline roughly doubles the agent runs, and the documentation suggests --ablation none for quick suites run on every change where the difference is not needed. Teams watching spend may take that advice further than intended and end up back at a score that does not say what it measures.