How the numbers are produced

Lite — 300 hand-selected GitHub issues from 12 open-source Python repos. Each instance ships as a repo snapshot + a failing test. The agent has to produce a patch that makes the failing test pass without breaking the passing tests.

Verified — the reproducible subset. Issues that were manually audited to confirm the test is a fair measure of "did the agent fix the reported bug."

Full — the entire 2 294-instance dataset.

The Docker eval harness runs the failing test before and after the agent's patch. Only instances where the agent's patch makes every test in the pass_to_pass list pass, and every test in the fail_to_pass list transition from fail → pass, count as resolved. Everything else is a fail, regardless of how close the patch was.

For each instance the harness spins up an isolated Docker container mirroring the instance's Python environment, checks out the pre-fix commit, and launches b0 in headless mode with the issue text as the prompt.

Model routing is driven by ANTHROPIC_MODEL / OPENROUTER_API_KEY env vars, not by the -m flag. The CLI -m flag is a display override; the model that actually runs is the one the env vars select.

• resolved — patch applied, all tests pass
• patched — patch applied, some tests still fail (partial credit for reasoning, no credit on the leaderboard)
• no_changes — b0 ran but didn't modify any files
• kepler_failed — CLI errored (crash, timeout, credit exhaustion)

The operator-level recipe lives in benchmark/AZURE-VM-EVAL.md in the b0 npm package repo. Once the public benchmarks repo goes up (see PRD-083), this page will link to the three-command quickstart.