MoveBench - Judge committee reference MSE: leaderboard

MoveBench (2610.26185v1, Figure 1) uses the exact printed values. Its 809 questions comprise 113 single-select, 197 multi-select, and 499 open-ended questions, with 5012 criteria. The domain counts are 257 program-generation, 291 data-analysis, and 261 QA questions. Multiple-choice answers use exact match. For open-ended answers, earned rubric points are pooled over possible points within each format. The overall score weights the three format scores by successfully scored question counts; domain scores separately average per-question normalized points. The judges are Gemini 3.1 Pro, Claude 4.6 Sonnet, and GPT 5.2, with weights .35/.35/.30. An item receives zero when 2 of 3 judges flag a safety violation. Failed requests are retried; unresolved failures are excluded, and per-model scored denominators are not supplied. Native model names retain the study prefix; source GPT 6 is not resolved to Astra or Luna. The public 80-question GitHub release includes rubrics, but is a distinct subset/version and does not establish that all 809 scored questions are released. Table 4 reports normalized Gemini/Claude/GPT committee weights and reference MSE. The source does not report held-out independent committee validation; this is an in-study calibration diagnostic. The third closest-to-baseline row has weights summing to 0.9978, and the source identifies only two fully specified normalized configurations. The unresolved third configuration is excluded from this intake without inventing a normalization. MSE belongs to the committee, not an individual judge. Metric: Reference MSE.

Metric: Reference MSE. Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 2 models tracked.

Top models

Interactive version: theaggregate.ai/benchmark?slug=movebench-judge-committee-reference-mse · How It Works · Data refreshed daily, snapshot 2026-10-28.