AJ-Bench (Agent-as-a-Judge): leaderboard

Verifying agent behavior on AJ-Bench (2026): the judge decides whether each of 516 agent trajectories from search, data-system and GUI environments succeeded, and is scored against binary human-verified labels. In this setting the judge runs as an MCPMark agent that can call tools and inspect the live environment to gather evidence. Mean F1 over the seven subdomains in percent; higher is better.

Metric: Overall F1 (%), mean of the seven subdomain F1 scores, with the judge running as an MCPMark agent that calls tools and inspects the live environment for evidence, of AJ-Bench, judging whether agent trajectories succeeded against binary human-verified labels (155 tasks, 516 trajectories), default model configurations; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 2 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1DeepSeek V3.277.34#425
2GPT-5 Mini75.99#373

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=aj-bench-agent-as-a-judge · How It Works · Data refreshed daily, snapshot 2026-11-22.