Compression-Bench - Qwen3.8 27B / 8 GiB (perplexity): leaderboard
Compression-Bench (arXiv 2611.06858) measures how well a coding agent can compress a large language model to fit a fixed storage limit. The agent gets a BF16 checkpoint, a size limit covering both disk and GPU memory, one GPU, offline data and papers, and eight hours, and must deliver compressed weights with its own inference code; all six agents run in Claude Code at maximum reasoning effort. A verifier scores the frozen submission once: it measures the compressed model's perplexity on text the original model generated, and the submission fails if it exceeds the size limit, fails to run, or keeps less than 80% of the original's accuracy on math, code and question-answering items. Unlike MLE-bench, where agents train models for Kaggle tasks graded by each competition's metric, every task has one objective, fidelity to a given model under a memory budget. This board is the published perplexity of each agent's compressed Qwen3.8 27B at 8 GiB, lower is better (the uncompressed model scores 3.221); an agent whose submission failed a gate is not ranked.
Metric: Perplexity of the compressed model on held-out BF16-generated text (lower is better; BF16 3.221); submissions that fail a gate are not ranked. Source: github.com. Saturation forecast: Around October 2028. 3 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Sonnet 5.5 (Max) | 3.36 | #19 (Claude Sonnet 5.5) |
| 2 | Claude Opus 5.5 (Max) | 3.37 | #5 (Claude Opus 5.5) |
| 3 | DeepSeek V4.1 Flash (1017) (Max) | 4.28 | #39 (DeepSeek V4.1 Flash (1017)) |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #6 GPT-6 Astra.
Interactive version: theaggregate.ai/benchmark?slug=compression-bench-qwen3-8-27b-8-gib-perplexity · How It Works · Data refreshed daily, snapshot 2026-11-13.