Compression-Bench - Qwen3.5 122B-A10B / 48 GiB (perplexity): leaderboard
Compression-Bench (arXiv 2611.06858) measures how well a coding agent can compress a large language model to fit a fixed storage limit. The agent gets a BF16 checkpoint, a size limit covering both disk and GPU memory, one GPU, offline data and papers, and eight hours, and must deliver compressed weights with its own inference code; all six agents run in Claude Code at maximum reasoning effort. A verifier scores the frozen submission once: it measures the compressed model's perplexity on text the original model generated, and the submission fails if it exceeds the size limit, fails to run, or keeps less than 80% of the original's accuracy on math, code and question-answering items. Unlike MLE-bench, where agents train models for Kaggle tasks graded by each competition's metric, every task has one objective, fidelity to a given model under a memory budget. This board is the published perplexity of each agent's compressed Qwen3.5 122B-A10B at 48 GiB, lower is better (the uncompressed model scores 2.755); an agent whose submission failed a gate is not ranked.
Metric: Perplexity of the compressed model on held-out BF16-generated text (lower is better; BF16 2.755); submissions that fail a gate are not ranked. Source: github.com. Saturation forecast: Around July 2028. 6 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Sonnet 5.5 (Max) | 2.79 | #19 (Claude Sonnet 5.5) |
| 2 | DeepSeek V4.1 Flash (1017) (Max) | 2.8 | #39 (DeepSeek V4.1 Flash (1017)) |
| 3 | GLM-5.3 (Max) | 2.81 | #66 (GLM-5.3) |
| 4 | MiniMax-M3.1-Flash (Max) | 2.81 | #107 (MiniMax-M3.1-Flash) |
| 5 | Claude Opus 5.5 (Max) | 2.82 | #5 (Claude Opus 5.5) |
| 6 | Kimi K3.1 Pro (Max) | 2.84 | #29 (Kimi K3.1 Pro) |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #6 GPT-6 Astra.
Interactive version: theaggregate.ai/benchmark?slug=compression-bench-qwen3-5-122b-a10b-48-gib-perplexity · How It Works · Data refreshed daily, snapshot 2026-11-13.