Structured Local-Deployment MCQ: leaderboard
A benchmark from Binghamton University (2026) of 1,085 multiple-choice items with two to four options in 16 topic families, such as rule application, extraction, counting, formatting and short scenario decisions, built to test small language models for structured local deployment. The score is strict accuracy in percent, which requires exactly the correct option letter in a format-valid output; higher is better.
Metric: Strict accuracy (%): share of the 1,085 items answered with the correct option letter in a format-valid output (a correct but malformed answer scores zero), deterministic decoding; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 2.5 Coder 3B Instruct | 75.67 | #3583 |
| 2 | Qwen 2.5 1.5B Instruct | 67.1 | #7470 |
| 3 | Qwen 3.5 2B | 64.98 | #4810 |
| 4 | stablelm-zephyr-3B | 54.19 | #8551 |
| 5 | SmolLM2-1.7B-Instruct | 42.76 | #9173 |
| 6 | SmolLM2-360M-Instruct | 32.07 | #9586 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=structured-local-deployment-mcq · How It Works · Data refreshed daily, snapshot 2026-11-13.