llmbenchmarks.io Local GPU - ARC-AGI-1 Public Eval: leaderboard
ARC-AGI-1 on local hardware: visual grid puzzles in which the model infers a transformation from a few input-output grid pairs and writes the output grid for a new input, graded by exact match. This board uses the public evaluation set (419 test inputs) rather than ARC Prize's semi-private set; ARC Prize puts the gap between the two sets at up to 10 points. Score: pass@2 (%), two attempts per test input with the lab's sampling settings (temperature above 0, a separate seed per attempt); runs before 2026-10-05 were pass@1 at temperature 0 and are not read. llmbenchmarks.io runs this on its own hardware (2026): open-weight models, mostly 4-bit and 8-bit quantised checkpoints (NVFP4, AWQ, QAT) pinned to a Hugging Face repository and revision, served by vLLM 0.29.0 on one consumer machine (2x RTX 5060 Ti, 32 GB VRAM, tensor parallel 2), each with thinking on or off. Unlike vendor or leaderboard scores for the same test, a row measures the quantised deployment a home user would run, so a quantised checkpoint is its own model, not the full-precision original. Labels carry the mode as (Thinking) or (Non-thinking); Llama 3.1 rows (no thinking mode) and Ministral 3 14B Reasoning rows (always reasoning) stay bare. A run on a subset of the test is not read. Higher is better. Source: https://llmbenchmarks.io/local-gpu-benchmarks/
Metric: pass@2 on the public evaluation set (%). Source: llmbenchmarks.io. Saturation forecast: Around 2029. 12 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3.5 4B (Non-reasoning) | 3.25 | #514 (Qwen 3.5 4B) |
| 2 | Gemma 4 E4B (IT) (Non-reasoning) | 2.88 | |
| 3 | Llama 3.1 8B Instruct | 1.88 | #1017 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=llmbenchmarks-io-local-gpu-arc-agi-1-public-eval · How It Works · Data refreshed daily, snapshot 2026-11-05.