Compression-Bench - Qwen3.8 27B / 12 GiB: leaderboard
Compression-Bench (arXiv 2611.06858) measures how well a coding agent can compress a large language model to fit a fixed storage limit. The agent gets a BF16 checkpoint, a size limit covering both disk and GPU memory, one GPU, offline data and papers, and eight hours, and must deliver compressed weights with its own inference code; all six agents run in Claude Code at maximum reasoning effort. A verifier scores the frozen submission once. The reward is ten times the ratio of the original model's perplexity to the compressed model's, on text the original model generated, and it is zero if the submission exceeds the size limit, fails to run, or keeps less than 80% of the original's accuracy on math, code and question-answering items. Unlike MLE-bench, where agents train models for Kaggle tasks graded by each competition's metric, every task has one objective, fidelity to a given model under a memory budget. This board is the reward for compressing Qwen3.8 27B to 12 GiB; the uncompressed model scores 10.
Metric: Fidelity reward F = 10 x PPL_bf16 / PPL, 0 when a gate fails (BF16 = 10). Source: github.com. Saturation forecast: Around December 2027. 6 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Sonnet 5.5 (Max) | 9.9 | #18 (Claude Sonnet 5.5) |
| 2 | Claude Opus 5.5 (Max) | 9.84 | #5 (Claude Opus 5.5) |
| 3 | Kimi K3.1 Pro (Max) | 9.66 | #28 (Kimi K3.1 Pro) |
| 4 | GLM-5.3 (Max) | 9.52 | #62 (GLM-5.3) |
| 5 | DeepSeek V4.1 Flash (1017) (Max) | 9.48 | #38 (DeepSeek V4.1 Flash (1017)) |
| 6 | MiniMax-M3.1-Flash (Max) | 9.48 | #104 (MiniMax-M3.1-Flash) |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #6 GPT-6 Astra.
Interactive version: theaggregate.ai/benchmark?slug=compression-bench-qwen3-8-27b-12-gib · How It Works · Data refreshed daily, snapshot 2026-11-12.