SpecCompact - RULER 16k: leaderboard
Agent task completion and long-context performance with distinct compaction configurations. SpecCompact v1 reports the original implementation, before later latency optimizations. AppWorld uses the complete Test-Normal and Test-Challenge sets; tau3 uses three trials on 80 randomly selected tasks. LongBench GovReport and QMSum use ROUGE-L; RULER uses five task types at 8k and 16k with official matching metrics. Compaction occurs every eight model calls with a 2,048-token summary budget. SpecCompact uses six slots, an acceptance threshold of 0.7, and a separately fine-tuned Qwen3-0.6B drafter for each target. Full History retains the complete history. Exact checkpoint revisions and reasoning settings are unreported. BF16 Gemma and FP8 Qwen are expressly described for latency experiments and are not imputed to task-score runs. Every compaction configuration retains a distinct source identity. Question access: Primary result reviewed; exact scored release availability not confirmed.
Metric: Published task score (0–100). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 6 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemma4-31B-IT (Benchmark Sweep Run: source=SpecCompact; configuration=Full History; original implementation) | 92.12 | |
| 2 | Qwen3.5-122B-A10B (Benchmark Sweep Run: source=SpecCompact; configuration=Full History; original implementation) | 90.92 | |
| 3 | Qwen3.5-122B-A10B (Benchmark Sweep Run: source=SpecCompact; configuration=SpecCompact; original implementation) | 72.6 | |
| 4 | Qwen3.5-122B-A10B (Benchmark Sweep Run: source=SpecCompact; configuration=Target-self; original implementation) | 72.15 | |
| 5 | Gemma4-31B-IT (Benchmark Sweep Run: source=SpecCompact; configuration=Target-self; original implementation) | 71.02 | |
| 6 | Gemma4-31B-IT (Benchmark Sweep Run: source=SpecCompact; configuration=SpecCompact; original implementation) | 68.55 |
Interactive version: theaggregate.ai/benchmark?slug=speccompact-ruler-16k · How It Works · Data refreshed daily, snapshot 2026-11-06.