XStack - Target + Correctness Judge: leaderboard
Whole-repository PyTorch-on-GPU to JAX-on-TPU migration by coding agents, graded by rerunning the migrated main experiment on a fresh TPU. XStack v1: 20 PyTorch-on-GPU repositories migrated whole to JAX on TPU v6e/v5p/v4 machines, one cell per agent and repository. All seven agents use high reasoning effort through the developer API (GPT models in Codex CLI; Claude Opus 5 and three other models in Claude Code CLI with their own backends). A fresh TPU reruns the migrated main experiment; checks apply in turn: grading target reached, GPT-5.6-sol correctness judge, GPT-5.6-sol cheat judge, and weights loading into the source model. Each board is the cumulative share of the 20 repositories passing up to that check; the full pass is all four. Cost, tokens and hours are not capability scores; the pooled "all" row is not a model. Question access: The paper links a dataset; the 20 repositories are public projects, but the graded targets/recipes release was not verified in this audit.
Metric: Repositories passing target and correctness judge (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 7 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 5 (Benchmark Sweep Run: source=XStack; harness=Claude Code CLI (Anthropic); effort=high; official API; 20 repositories) | 80 | |
| 2 | GPT-5.6 Sol (Benchmark Sweep Run: source=XStack; harness=Codex CLI; effort=high; official API; 20 repositories) | 70 | |
| 3 | DeepSeek V4.1 Flash (Benchmark Sweep Run: source=XStack; harness=Claude Code CLI (DeepSeek); effort=high; official API; 20 repositories) | 60 | |
| 4 | GLM-5.3 Flash (Benchmark Sweep Run: source=XStack; harness=Claude Code CLI (Z.ai); effort=high; official API; 20 repositories) | 50 | |
| 5 | Muse Spark 1.3 (Benchmark Sweep Run: source=XStack; harness=Claude Code CLI (Meta); effort=high; official API; 20 repositories) | 35 | |
| 6 | GPT-5.6 Terra (Benchmark Sweep Run: source=XStack; harness=Codex CLI; effort=high; official API; 20 repositories) | 25 | |
| 7 | GPT-5.6 Luna (Benchmark Sweep Run: source=XStack; harness=Codex CLI; effort=high; official API; 20 repositories) | 15 |
Interactive version: theaggregate.ai/benchmark?slug=xstack-target-plus-correctness-judge · How It Works · Data refreshed daily, snapshot 2026-11-07.