SWE-bench Lite · 300 instances · Docker eval
176/300 resolved(58.7%)
Real GitHub issues. Official SWE-bench evaluation. Any model you choose.
176
Resolved
95%
Cache hit
—
Total cost
Agent Comparison
| Agent | Score | Model | BYOK | OSS |
|---|---|---|---|---|
| Claude Code | 72.0% | Claude Sonnet 4 | No | No |
| Codex CLI | 69.1% | GPT-4.1 | No | Yes |
| Aider | 53.0% | Various | Yes | Yes |
| Bahulam Codeyou | 58.7% | MiMo v2.5 (BYOK) | Yes | No |
| SWE-agent | 23.0% | Various | Yes | Yes |
Bahulam Code 2.6.14 · MiMo v2.5 · Updated 2026-07-30
Bahulam Code across models
Same agent, same harness. Only the model changes.
| Metric | MiMo v2.5now | DeepSeek V4 Flashnow | GLM 5.2now | Tencent HY3now |
|---|---|---|---|---|
| Vendor | Xiaomi | DeepSeek | Zhipu | Tencent (free) |
| SWE-bench | 58.7% | 54.7% | 61.0% | 44.3% |
| Resolved | 176 / 300 | 164 / 300 | 183 / 300 | 133 / 300 |
| Patched | 244 (81%) | 216 (72%) | 235 (78%) | 230 (77%) |
| Cost / 300 | — | $11.60 | $26 | $0 (free) |
| Per instance | — | $0.071 | $0.086 | $0 |
| Cache hit | 95% | 94% | 94% | 93% |
| Date | 2026-08-19 | 2026-08-20 | 2026-07-17 | 2026-07-15 |
Every model runs the same harness · Docker eval · SWE-bench Lite 300
How Bahulam Code solves issues
1
Search
Find relevant files
2
Understand
Read the root cause
3
Fix
Surgical source edit
4
Verify
Run project tests
Resolved by project
django/django 80sympy/sympy 36matplotlib/matplotlib 14scikit 12pytest 9psf/requests 6sphinx 6astropy/astropy 4mwaskom/seaborn 3pallets/flask 2pydata/xarray 2pylint 2
All 176 resolved instances
✓astropy__astropy-12907✓astropy__astropy-14365✓astropy__astropy-14995✓astropy__astropy-6938✓django__django-10914✓django__django-10924✓django__django-11001✓django__django-11039✓django__django-11049✓django__django-11099✓django__django-11133✓django__django-11179✓django__django-11283✓django__django-11583✓django__django-11620✓django__django-11630✓django__django-11742✓django__django-11815✓django__django-11848✓django__django-11910✓django__django-11999✓django__django-12184✓django__django-12284✓django__django-12286✓django__django-12453✓django__django-12497✓django__django-12700✓django__django-12708✓django__django-12747✓django__django-12856✓django__django-12908✓django__django-12915✓django__django-12983✓django__django-13028✓django__django-13033✓django__django-13158✓django__django-13220✓django__django-13230✓django__django-13315✓django__django-13401✓django__django-13447✓django__django-13590✓django__django-13658✓django__django-13710✓django__django-13757✓django__django-13925✓django__django-13933✓django__django-14016✓django__django-14017✓django__django-14155✓django__django-14238✓django__django-14382✓django__django-14411✓django__django-14580✓django__django-14608✓django__django-14672✓django__django-14730✓django__django-14752✓django__django-14787✓django__django-14855✓django__django-14915✓django__django-14997✓django__django-14999✓django__django-15347✓django__django-15388✓django__django-15498✓django__django-15781✓django__django-15814✓django__django-15851✓django__django-15902✓django__django-15996✓django__django-16041✓django__django-16046✓django__django-16139✓django__django-16229✓django__django-16255✓django__django-16379✓django__django-16527✓django__django-16595✓django__django-16820✓django__django-16873✓django__django-16910✓django__django-17051✓django__django-17087✓matplotlib__matplotlib-23314✓matplotlib__matplotlib-23562✓matplotlib__matplotlib-23913✓matplotlib__matplotlib-23964✓matplotlib__matplotlib-23987✓matplotlib__matplotlib-24149✓matplotlib__matplotlib-24265✓matplotlib__matplotlib-24970✓matplotlib__matplotlib-25332✓matplotlib__matplotlib-25433✓matplotlib__matplotlib-25442✓matplotlib__matplotlib-25498✓matplotlib__matplotlib-26011✓matplotlib__matplotlib-26020✓mwaskom__seaborn-3010✓mwaskom__seaborn-3190✓mwaskom__seaborn-3407✓pallets__flask-4992✓pallets__flask-5063✓psf__requests-1963✓psf__requests-2148✓psf__requests-2317✓psf__requests-2674✓psf__requests-3362✓psf__requests-863✓pydata__xarray-4094✓pydata__xarray-5131✓pylint-dev__pylint-5859✓pylint-dev__pylint-7993✓pytest-dev__pytest-11143✓pytest-dev__pytest-11148✓pytest-dev__pytest-5227✓pytest-dev__pytest-5692✓pytest-dev__pytest-6116✓pytest-dev__pytest-7432✓pytest-dev__pytest-8365✓pytest-dev__pytest-8906✓pytest-dev__pytest-9359✓scikit-learn__scikit-learn-10297✓scikit-learn__scikit-learn-11040✓scikit-learn__scikit-learn-11281✓scikit-learn__scikit-learn-12471✓scikit-learn__scikit-learn-13142✓scikit-learn__scikit-learn-13241✓scikit-learn__scikit-learn-13496✓scikit-learn__scikit-learn-13497✓scikit-learn__scikit-learn-13584✓scikit-learn__scikit-learn-13779✓scikit-learn__scikit-learn-14092✓scikit-learn__scikit-learn-15512✓sphinx-doc__sphinx-10325✓sphinx-doc__sphinx-11445✓sphinx-doc__sphinx-7975✓sphinx-doc__sphinx-8595✓sphinx-doc__sphinx-8713✓sphinx-doc__sphinx-8721✓sympy__sympy-12419✓sympy__sympy-12481✓sympy__sympy-13043✓sympy__sympy-13471✓sympy__sympy-13480✓sympy__sympy-13647✓sympy__sympy-13773✓sympy__sympy-13971✓sympy__sympy-14396✓sympy__sympy-14774✓sympy__sympy-15011✓sympy__sympy-15308✓sympy__sympy-15345✓sympy__sympy-15346✓sympy__sympy-15609✓sympy__sympy-15678✓sympy__sympy-16792✓sympy__sympy-17022✓sympy__sympy-18057✓sympy__sympy-18189✓sympy__sympy-18532✓sympy__sympy-18621✓sympy__sympy-19487✓sympy__sympy-20154✓sympy__sympy-20212✓sympy__sympy-20442✓sympy__sympy-20590✓sympy__sympy-21055✓sympy__sympy-21379✓sympy__sympy-21614✓sympy__sympy-21847✓sympy__sympy-22714✓sympy__sympy-23117✓sympy__sympy-23262✓sympy__sympy-24152✓sympy__sympy-24213
Methodology
Harness: Official swebench v4.1.0 with Docker containers. Each instance runs in isolation matching the repo's Python version.
Scoring: Resolved = ALL failing tests pass AND ALL existing tests still pass. No partial credit.
Score: 176/300 (58.7%) against all instances. 95% cache hit rate, — total run cost.
Artifacts: full predictions, evaluator reports, and costs for this and every other run are public at github.com/BahulamAI/bahulam-benchmarks.
Try Bahulam Code
Bring your own key. Pick any model.
npm i -g @bahulam/codeWeb app