JetBrains Mellum2.1 Card - Terminal-Bench 2.1 (Pi harness): leaderboard
Terminal-Bench 2.1 command-line tasks, run in the Pi v0.73.1 agent harness. JetBrains evaluated Mellum2.1 Thinking, Mellum2 Thinking, Gemma 4 E4B and Qwen3.5 9B with the same pipeline in thinking mode (each model's default sampling, a 114K-token context and up to 16K tokens per turn) and reported the results on the Mellum2.1 model card (October 2026). Vendor-reported. Metric: Resolved (%). Higher is better. Source: https://huggingface.co/JetBrains/Mellum2.1-12B-A2.5B-Thinking/raw/92ddae9fc7665e9f801d141d2e5a6b2caf2460c4/README.md
Metric: Resolved (%). Source: huggingface.co. Saturation forecast: Around September 2028. 4 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3.5 9B (Thinking) | 21.7 | #377 (Qwen 3.5 9B) |
| 2 | Gemma 4 E4B (IT) (Thinking) | 3.4 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=jetbrains-mellum2-1-card-terminal-bench-2-1-pi-harness · How It Works · Data refreshed daily, snapshot 2026-10-19.