Weight Space Programming - Held-out - MMLU-Pro: leaderboard
Weight Space Programming v1 Table2 held-out test percentages, Qwen3.5-0.8B and Qwen3.5-9B base versus exact task-specific edited candidates. Starred edits were discarded and remain explicitly labeled, never substituted for the final original checkpoint. Thinking enabled, T0.6, top_p0.95, top_k20, presence penalty1.5, 32768 output cap, seed0. Source Table4 fixes test populations 500/198/2450/12032/271/580/65 chains (288 steps) by named task. Do not admit development scores, gains or search traces as capability outcomes.
Metric: Accuracy / task score (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 4 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen3.5-9B (WSP Run: task=MMLU-Pro;candidate=base;thinking=enabled;temperature=0.6;top_p=0.95;top_k=20;max_new_tokens=32768;seed=0) | 82.89 | |
| 2 | Qwen3.5-9B (WSP Run: task=MMLU-Pro;candidate=edited-discarded;thinking=enabled;temperature=0.6;top_p=0.95;top_k=20;max_new_tokens=32768;seed=0) | 82.15 | |
| 3 | Qwen3.5-0.8B (WSP Run: task=MMLU-Pro;candidate=edited-discarded;thinking=enabled;temperature=0.6;top_p=0.95;top_k=20;max_new_tokens=32768;seed=0) | 35.07 | |
| 4 | Qwen3.5-0.8B (WSP Run: task=MMLU-Pro;candidate=base;thinking=enabled;temperature=0.6;top_p=0.95;top_k=20;max_new_tokens=32768;seed=0) | 34.17 |
Interactive version: theaggregate.ai/benchmark?slug=weight-space-programming-held-out-mmlu-pro · How It Works · Data refreshed daily, snapshot 2026-11-12.