Compression-Bench - Nemotron-3.5 120B-A12B / 48 GiB (perplexity): leaderboard
Compression-Bench (arXiv 2611.06858) measures how well a coding agent can compress a large language model to fit a fixed storage limit. The agent gets a BF16 checkpoint, a size limit covering both disk and GPU memory, one GPU, offline data and papers, and eight hours, and must deliver compressed weights with its own inference code; all six agents run in Claude Code at maximum reasoning effort. A verifier scores the frozen submission once: it measures the compressed model's perplexity on text the original model generated, and the submission fails if it exceeds the size limit, fails to run, or keeps less than 80% of the original's accuracy on math, code and question-answering items. Unlike MLE-bench, where agents train models for Kaggle tasks graded by each competition's metric, every task has one objective, fidelity to a given model under a memory budget. This board is the published perplexity of each agent's compressed Nemotron-3.5 120B-A12B at 48 GiB, lower is better (the uncompressed model scores 1.688); an agent whose submission failed a gate is not ranked.
Metric: Perplexity of the compressed model on held-out BF16-generated text (lower is better; BF16 1.688); submissions that fail a gate are not ranked. Source: github.com. Saturation forecast: Around March 2028. 6 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 5.5 (Max) | 1.73 | #5 (Claude Opus 5.5) |
| 2 | Claude Sonnet 5.5 (Max) | 1.73 | #19 (Claude Sonnet 5.5) |
| 3 | Kimi K3.1 Pro (Max) | 1.73 | #29 (Kimi K3.1 Pro) |
| 4 | DeepSeek V4.1 Flash (1017) (Max) | 1.75 | #39 (DeepSeek V4.1 Flash (1017)) |
| 5 | MiniMax-M3.1-Flash (Max) | 1.75 | #107 (MiniMax-M3.1-Flash) |
| 6 | GLM-5.3 (Max) | 1.78 | #66 (GLM-5.3) |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #6 GPT-6 Astra.
Interactive version: theaggregate.ai/benchmark?slug=compression-bench-nemotron-3-5-120b-a12b-48-gib-perplexity · How It Works · Data refreshed daily, snapshot 2026-11-13.