llmbenchmarks.io Local GPU - IFBench: leaderboard
IFBench (Allen AI, 2025) on local hardware: 300 prompts with new, out-of-distribution verifiable output constraints checked by code. Dataset, prompt, checker, headline metric, sampling and a 32,768-token answer limit follow the Swallow LLM Leaderboard's setup (Swallow counts the limit over input plus output, this run over the output only, and top_k is not sent). Score: instruction-level strict accuracy (%), the share of individual constraints satisfied. Runs before 2026-10-22, when the token limit was introduced, are not read. llmbenchmarks.io runs this on its own hardware (2026): open-weight models, mostly 4-bit and 8-bit quantised checkpoints (NVFP4, AWQ, QAT) pinned to a Hugging Face repository and revision, served by vLLM 0.29.0 on one consumer machine (2x RTX 5060 Ti, 32 GB VRAM, tensor parallel 2), each with thinking on or off. Unlike vendor or leaderboard scores for the same test, a row measures the quantised deployment a home user would run, so a quantised checkpoint is its own model, not the full-precision original. Labels carry the mode as (Thinking) or (Non-thinking); Llama 3.1 rows (no thinking mode) and Ministral 3 14B Reasoning rows (always reasoning) stay bare. A run on a subset of the test is not read. Higher is better. Source: https://llmbenchmarks.io/local-gpu-benchmarks/
Metric: Instruction-level strict accuracy (%). Source: llmbenchmarks.io. Saturation forecast: Around February 2027. 24 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3.5 4B (Thinking) | 46.22 | #514 (Qwen 3.5 4B) |
| 2 | Gemma 4 E4B (IT) (Thinking) | 40.12 | |
| 3 | Gemma 4 E4B (IT) (Non-reasoning) | 34.3 | |
| 4 | Qwen 3.5 4B (Non-reasoning) | 31.1 | #514 (Qwen 3.5 4B) |
| 5 | Llama 3.1 8B Instruct | 26.16 | #1017 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=llmbenchmarks-io-local-gpu-ifbench · How It Works · Data refreshed daily, snapshot 2026-11-05.