ISA-Bench (First Attempt): leaderboard
ISA-Bench (2026) on the first attempt only: 92 programming-game tasks in five constrained instruction sets, solved when the first program passes every verification case, with no worked solutions in the prompt. Share of tasks solved, in percent; higher is better.
Metric: Tasks solved (%; share of the 92 tasks of five programming games with constrained instruction sets (SIC-1, Human Resource Machine, TIS-100, MHRD, BOX-256) whose program passes every verification case in the game's own parser, virtual machine and verifier; first attempt only, no worked solutions; Ollama default builds, temperature 0.7, 4,096 output tokens). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 14 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | ISA-Bench Gemma-4 31B (Thinking, Ollama build) | 25 | |
| 2 | ISA-Bench Qwen3 14B (Thinking, Ollama build) | 15.22 | |
| 3 | ISA-Bench Qwen3 32B (Thinking, Ollama build) | 14.13 | |
| 4 | ISA-Bench Qwen3 8B (Thinking, Ollama build) | 14.13 | |
| 5 | ISA-Bench DeepSeek-R1 32B (Ollama build) | 13.04 | |
| 6 | ISA-Bench Devstral-Small-2 (Ollama build) | 13.04 | |
| 7 | ISA-Bench Qwen3-Coder 30B (Ollama build) | 13.04 | |
| 8 | ISA-Bench Qwen2.5-Coder 32B (Ollama build) | 10.87 | |
| 9 | ISA-Bench Granite-4.1 8B (Ollama build) | 6.52 | |
| 10 | ISA-Bench Qwen2.5-Coder 14B (Ollama build) | 3.26 |
Interactive version: theaggregate.ai/benchmark?slug=isa-bench-first-attempt · How It Works · Data refreshed daily, snapshot 2026-11-12.