Nejumi 4 - SWE-Bench (Japanese): leaderboard
Eighty issues sampled from a Japanese translation of SWE-bench Verified, kept under 7000 input tokens. The prompt carries the issue text and the relevant file snippets and the model replies with a unified diff rather than exploring the repository; the patch is applied and the project's tests are run, and an issue counts as resolved only when they all pass. Higher is better. Source: github.com/wandb/llm-leaderboard.
Metric: Score (%). Source: nejumi.ai. Saturation forecast: Around January 2027. 139 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Fable 5 | 80 | #16 |
| 2 | Claude Opus 5 | 75 | #12 |
| 3 | GPT-5.6 Sol | 73.75 | #19 |
| 4 | GPT-5.6 Terra | 70 | #41 |
| 5 | Qwen3.8-Flash-Next | 70 | #72 |
| 6 | Hy4 preview | 70 | #38 |
| 7 | Qwen3.8 2.4T A95B | 70 | #55 |
| 8 | Claude Opus 4.8 | 68.75 | #37 |
| 9 | GPT-5.5 (2026-04-23) | 68.75 | #22 |
| 10 | Kimi K3 | 67.5 | #30 |
| 11 | Gemini 3 Flash (Preview) | 66.25 | #82 |
| 12 | DeepSeek V4.1 Flash | 66.25 | #36 |
| 13 | Grok 4.5 | 66.25 | #48 |
| 14 | Claude Opus 4.7 | 65 | #49 |
| 15 | GLM-5.3 | 63.75 | #39 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=nejumi-4-swe-bench-japanese · How It Works · Data refreshed daily, snapshot 2026-10-20.