AJ-Bench (LLM-as-a-Judge) - Word: leaderboard

Verifying agent trajectories on AJ-Bench (2026), the Word subdomain (24 trajectories of OSWorld document-editing tasks): the judge decides whether each trajectory succeeded and is scored against binary human-verified labels. In this setting the judge reads the trajectory only, with no tools or environment access. F1 in percent; higher is better.

Metric: F1 (%) on Word GUI tasks (OSWorld; 12 tasks, 24 trajectories), averaged over three runs, with the judge reading the trajectory only (no tools or environment access), of AJ-Bench, judging whether agent trajectories succeeded against binary human-verified labels (155 tasks, 516 trajectories), default model configurations; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 13 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Gemini 3 Pro (Preview)72.14#119
2DeepSeek V3.269.77#419
3Grok 465.26#319
4Claude Sonnet 4.561.56#261
5Gemini 2.5 Pro60.13#276
6Kimi K2 090555.39#570
7Claude Opus 4.551.45#160
8GLM-4.650.07#498
9GPT-5 MiniLow48.41#364
10GPT-544.81#178
11GPT-5.139.54#240
12Qwen 3 235B A22B36.82#628
13LongCat-Flash-Chat30.35#818

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=aj-bench-llm-as-a-judge-word · How It Works · Data refreshed daily, snapshot 2026-11-21.