ServerBench - Gemma4 E4B IT 1xL40: leaderboard
Coding agents construct complete verified serving systems under frozen constraints. ServerBench v1 gives each coding agent one eight-hour development run in a network-disabled Harbor sandbox, followed by a 1,800-second replay on held-out seeds. All agents use maximum reasoning effort; GPT models use Codex and other models use Claude Code. The reported score belongs to the strongest verified final submission, allowing at most one rerun for a clearly anomalous failure. It measures latency-qualified token value per GPU dollar of the serving code the agent produced, under fixed checkpoint, hardware, latency targets and prices. Submission concurrency is retained. Native Not passed cells have no numeric score and are not converted to zero. Question access: Serving task code and development procedures are public; independent correctness-test details are withheld.
Metric: Latency-qualified dollars per GPU dollar. Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Sonnet 5.5 (Benchmark Sweep Run: source=ServerBench; harness=Claude Code; maximum reasoning; serving=Gemma4 E4B IT 1xL40; concurrency=32) | 2.97 | |
| 2 | Claude Opus 5.5 (Benchmark Sweep Run: source=ServerBench; harness=Claude Code; maximum reasoning; serving=Gemma4 E4B IT 1xL40; concurrency=40) | 2.47 | |
| 3 | GPT 6 Astra (Benchmark Sweep Run: source=ServerBench; harness=Codex; maximum reasoning; serving=Gemma4 E4B IT 1xL40; concurrency=32) | 1.75 | |
| 4 | GPT 6.1 Sol (Benchmark Sweep Run: source=ServerBench; harness=Codex; maximum reasoning; serving=Gemma4 E4B IT 1xL40; concurrency=24) | 1.59 | |
| 5 | Claude Opus 5 (Benchmark Sweep Run: source=ServerBench; harness=Claude Code; maximum reasoning; serving=Gemma4 E4B IT 1xL40; concurrency=24) | 1.15 | |
| 6 | Qwen3.8-Max-0902 (Benchmark Sweep Run: source=ServerBench; harness=Claude Code; maximum reasoning; serving=Gemma4 E4B IT 1xL40; concurrency=14) | 0.87 | |
| 7 | GPT 6 Sol (Benchmark Sweep Run: source=ServerBench; harness=Codex; maximum reasoning; serving=Gemma4 E4B IT 1xL40; concurrency=5) | 0.55 | |
| 8 | Kimi K3 (Benchmark Sweep Run: source=ServerBench; harness=Claude Code; maximum reasoning; serving=Gemma4 E4B IT 1xL40; concurrency=6) | 0.52 | |
| 9 | MiMo V2.6 Pro (Benchmark Sweep Run: source=ServerBench; harness=Claude Code; maximum reasoning; serving=Gemma4 E4B IT 1xL40; concurrency=10) | 0.22 |
Interactive version: theaggregate.ai/benchmark?slug=serverbench-gemma4-e4b-it-1xl40 · How It Works · Data refreshed daily, snapshot 2026-11-06.