MoveBench - Judge reference all-attempt agreement: leaderboard

MoveBench (2610.26185v1, Figure 1) uses the exact printed values. Its 809 questions comprise 113 single-select, 197 multi-select, and 499 open-ended questions, with 5012 criteria. The domain counts are 257 program-generation, 291 data-analysis, and 261 QA questions. Multiple-choice answers use exact match. For open-ended answers, earned rubric points are pooled over possible points within each format. The overall score weights the three format scores by successfully scored question counts; domain scores separately average per-question normalized points. The judges are Gemini 3.1 Pro, Claude 4.6 Sonnet, and GPT 5.2, with weights .35/.35/.30. An item receives zero when 2 of 3 judges flag a safety violation. Failed requests are retried; unresolved failures are excluded, and per-model scored denominators are not supplied. Native model names retain the study prefix; source GPT 6 is not resolved to Astra or Luna. The public 80-question GitHub release includes rubrics, but is a distinct subset/version and does not establish that all 809 scored questions are released. Table 2 and Section 4.3.2 use an adjudicated reference of 300 criteria per judge. Claude has 193 agreements, 87 disagreements, and 20 unclassified cases; Gemini has 229 agreements and 71 disagreements; GPT has 187 agreements and 113 disagreements. This all-attempt yield is derived from native counts and does not treat missing classifications as valid comparisons. The cause of the 20 unclassified cases remains unresolved. Metric: Normalized score.

Metric: Normalized score. Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 3 models tracked.

Top models

Interactive version: theaggregate.ai/benchmark?slug=movebench-judge-reference-all-attempt-agreement · How It Works · Data refreshed daily, snapshot 2026-10-28.