The Aggregate Digest

The Aggregate Digest — 2026-10-19

2026-10-19T07:04:34.833918+00:00

Daily

New Benchmarks (30)

LiveBench AMPS Hard (Score): leader Claude Opus 5 (Max) (99.01), 71 models
Items are machine-generated symbolic maths problems built from the ten hardest AMPS categories with a harder draw of random primitives. The model answers in one turn and its final boxed expression is compared with the ground truth in SymPy; if SymPy cannot confirm a match, the grading script asks a language model to compare the two. This is the AMPS Hard task in LiveBench's maths category, and higher is better.
LiveBench Connections (Score): leader Gemini 3.1 Pro (Preview) (High) (100.0), 71 models
One item gives a list of 8, 12 or 16 words to be sorted into groups of four that share a hidden connection, such as a category or a common associated word. This is the connections task in LiveBench's language comprehension category, checked against a fixed answer key with no model judge, on puzzles that LiveBench replaces with newer ones in dated releases. Scores run 0 to 100 and higher is better.
LiveBench Math Comp (Score): leader Claude Opus 4.7 (xHigh) (98.04), 71 models
The competition task in LiveBench's math category: high school contest problems with the prose and answer order changed to limit contamination. The first release drew on AMC 12 2023, SMC 2023 and AIME 2024, and later releases swapped in newer competitions. Answers are checked against ground truth with no model judge; the number is the share answered correctly, higher is better. Source: arxiv.org/abs/2406.19314.
LiveBench Olympiad (Score): leader Claude Opus 5.5 (Max) (93.25), 71 models
The LiveBench math task built from proof-based USAMO and IMO problems, refreshed over time with newer competitions. Formulae are cut out of a written solution and handed back to the model in scrambled order, and it must say which formula fills each gap. Scoring is automatic, by edit distance between its ordering and the true one with no judge model, and higher is better.
LiveBench Paraphrase (Score): leader Gemini 3.8 Flash (High) (80.0), 71 models
The model is given a recent article from The Guardian and asked to paraphrase it while obeying randomly drawn constraints such as a word limit or elements it must include. It is a single-turn task with the article supplied in context, and scoring is an automatic check of whether the constraints were met rather than a judgement of writing quality. Higher is better.
LiveBench Plot Unscrambling (Score): leader GPT-6 Astra (Max) (86.29), 71 models
LiveBench shuffles the sentences of a plot synopsis taken from a recently released film and asks the model to put them back in the original order. Grading compares the model's ordering with the true one and gives partial credit from the edit distance between the two sequences. Source material is refreshed to limit contamination, and higher is better. Source: arxiv.org/abs/2406.19314.
LiveBench Simplify (Score): leader Gemini 3.6 Flash (High) (75.7), 71 models
The model is given a recent article from The Guardian and asked to simplify it while obeying randomly drawn constraints such as a word count, required keywords or banned words. This is the simplify task in LiveBench's instruction following category, and the score counts only adherence to those instructions, checked without a model judge. Scores run 0 to 100 and higher is better.
LiveBench Story Generation (Score): leader Gemini 3.8 Flash (High) (82.75), 71 models
One of four instruction-following subtasks in LiveBench. The model writes a story about a recent Guardian article while obeying a randomly drawn subset of automatically verifiable constraints such as word limits, required keywords and formatting rules. Grading checks only whether those constraints were met, combining whole-prompt and per-instruction accuracy; the score runs from 0 to 100, higher is better.
LiveBench Summarize (Score): leader Gemini 3.1 Pro (Preview) (High) (88.23), 71 models
The summarize task in LiveBench's instruction following category: the model is given a recent Guardian article and must summarise it while obeying several automatically checkable instructions, such as word limits. Scoring is adherence to those instructions rather than summary quality, and the board shows the task score in percent, higher is better. Source: arxiv.org/abs/2406.19314.
LiveBench Table Join (Score): leader GPT-6 Astra (Max) (58.37), 71 models
Each item shows two tables from a recent public dataset whose columns partly overlap, with the second table's names anonymized, and asks for a valid join mapping from the first to the second. Scoring is automatic: an F1 over the proposed column pairs, counting extra pairs as false positives and missing ones as false negatives. This board is that task's LiveBench score, higher is better.
LiveBench Table Reformat (Score): leader DeepSeek V4 Pro (100.0), 71 models
The table reformatting task inside LiveBench's data analysis category: the model is handed a table taken from a recent Kaggle or Socrata dataset in one serialisation and has to rewrite it in a requested target format among JSON, JSONL, Markdown, CSV, TSV and HTML. Output is checked automatically against the expected table with no model judge. Higher is better. Source: arxiv.org/abs/2406.19314.
LiveBench Typos (Score): leader Claude Fable 5 (Max) (94.0), 71 models
LiveBench injects common human typos into a recent arXiv abstract and asks the model to return the passage with the misspellings fixed and nothing else altered. Scoring is all or nothing: the answer counts only if the exact corrected text appears inside it, so any other edit loses the point. The board is the percent of passages scored correct, higher is better. Source: arxiv.org/abs/2406.19314.
LiveBench Zebra Puzzle (Score): leader GPT-5.5 (xHigh) (100.0), 71 models
Logic grid puzzles in LiveBench's reasoning category, where constraints about people and their attributes must be combined to work out a requested assignment. They are generated procedurally and have been replaced twice with harder sets, first with larger grids and more complex constraints. Answers in solution tags are checked against ground truth automatically, and the score is a percentage where higher is better.
KillSwitch-Bench (Pass rate (%; attempts whose program passed the hidden verification, every task attempted the same number of times)): leader Claude Opus 5 (70.0), 20 models
Coding agents write and interpret programs in KillSwitch, an esoteric language built to be hard for LLMs, working only from its published specification. Five tasks (hello world, a temperature conversion, max, and two program-understanding tasks), each attempted twice with a $5 budget; an attempt passes when its program passes the hidden verification. Each model runs under its vendor's agent (Claude Code, Codex) or OpenCode. Score: pass rate (%) over the ten attempts; the site's own ranking also subtracts small cost and code-size penalties, which are left out here. Higher is better. Source: https://bench.killswitch-lang.org/
TAC (Travel Agent Compassion) (Welfare rate (%; ticket-booking scenarios in which the agent avoided the option that harms animals, nothing in the request mentioning welfare)): leader Claude Opus 4.8 (64.7), 26 models
Travel Agent Compassion (CaML, arXiv 2606.18142; inspect_evals/tac): an AI booking agent chooses activities for users in 13 ticket-booking scenarios where the obvious or best-reviewed option harms animals (bullfights, marine shows, animal rides and races) and the request never mentions welfare. Scoring is programmatic on what the agent books. Score: welfare rate, the share of runs (%) that avoid the harmful option; picking at random avoids it about 64.5% of the time and many models do worse. Higher is better. Source: https://compassionbench.com/tac
TAC (Travel Agent Compassion) - With Welfare Guidance (Welfare rate (%; the same scenarios with one sentence of animal-welfare guidance in the system prompt)): leader Claude Opus 5 (100.0), 26 models
The Travel Agent Compassion scenarios (CaML, arXiv 2606.18142; inspect_evals/tac_welfare) rerun with one sentence of animal-welfare guidance in the agent's system prompt, so the gap to the unguided board separates default behaviour from instruction-following. Score: welfare rate, the share of runs (%) that avoid the option harming animals. Higher is better. Source: https://compassionbench.com/tac
CrueltyBench (Welfare score (0-100; two-judge panel, green 1 / yellow 0.5 / red 0, mean over the implicit animal-cruelty requests and epochs)): leader GPT-6 Astra (81.7), 14 models
CrueltyBench (CaML): single-turn, realistic in-persona requests with implicit animal-cruelty implications that never mention the animal's welfare (18 harm scenarios plus controls, 5 epochs). A fixed two-judge panel (Claude Opus 5, Gemini 3.7 Flash) grades each response green (clearly reduces or mitigates the cruelty), yellow or red; control items penalise moralising at ordinary requests. Score: mean welfare score (0-100; green 1, yellow 0.5, red 0). Higher is better. Source: https://compassionbench.com/crueltybench
LatentGUIWorld - Overall (Episode success rate (%; macro-average over six interactive GUI environments, 150 test episodes each, at most 12 model responses per episode)): leader GPT-6 Astra (63.3), 11 models
LatentGUIWorld (Learn from Move, OpenMOSS, arXiv 2610.16720): six interactive GUI environments in matched pairs (Drag, Ten-Choice, Rotation), 150 test episodes each at a 1280x720 viewport. In one Drag and one Ten-Choice variant the target follows from what is on screen or revealed by hovering; in the other variants the agent must work out hidden movement dynamics from the effects of its own actions. At most 12 model responses per episode, Rotation scored at the first release, each agent on its official computer-use or tool interface where one exists and on the authors' six-action JSON schema otherwise (DeepSeek-V4.1-Flash); GUI-specialised agents get additional tools where their official action space is insufficient. Score: episode success rate (%), macro-average over the six environments. Higher is better. Source: https://arxiv.org/abs/2610.16720
LatentGUIWorld - Dynamics-Dependent (I2) Average (Episode success rate (%; macro-average over the four environments whose hidden movement dynamics must be identified from the agent's own actions: egocentric Drag, egocentric Ten-Choice, inner and outer Rotation)): leader GPT-6 Astra (45.0), 11 models
The four LatentGUIWorld environments (arXiv 2610.16720) whose hidden movement dynamics can only be identified from the effects of the agent's own actions: egocentric Drag, egocentric Ten-Choice, and inner- and outer-ring Rotation, 150 episodes each, at most 12 model responses per episode. Score: episode success rate (%), macro-average over the four. Higher is better. Source: https://arxiv.org/abs/2610.16720
Elorian Glimpse Card - WGO Bench (Score): leader Glimpse (Max) (42.7), 5 models
WGO Bench, in Elorian's robotics category: end-to-end understanding of manipulation events, identifying completed actions in robot footage. Elorian reported this for its Glimpse research preview (October 16, 2026), running each model with its native tools: Glimpse at max, GPT-6 Astra at max with Code Interpreter, GPT-6.1 Sol and Claude Opus 5.5 at high effort with Code Interpreter (the post says max-effort results are not available for these video evaluations), and Gemini 3.8 Flash at high with its Video Agent. Vendor-reported. Metric: Score. Higher is better. Source: https://elorian.ai/blog/introducing-glimpse
Elorian Glimpse Card - ERQA (Accuracy (%)): leader Glimpse (Max) (83.75), 5 models
ERQA embodied spatial reasoning questions about images. Elorian reported this for its Glimpse research preview (October 16, 2026), running every model at its highest thinking effort with its native tools: Glimpse at max, GPT-6 Astra at max with Code Interpreter, GPT-6.1 Sol and Claude Opus 5.5 at max effort with Code Interpreter, and Gemini 3.8 Flash at high with Code Interpreter. Vendor-reported. Metric: Accuracy (%). Higher is better. Source: https://elorian.ai/blog/introducing-glimpse
Elorian Glimpse Card - BabyVision (Accuracy (%)): leader GPT-6 Astra (Max) (97.68), 4 models
BabyVision fine-grained visual reasoning questions. Elorian reported this for its Glimpse research preview (October 16, 2026), running every model at its highest thinking effort with its native tools: Glimpse at max, GPT-6 Astra at max with Code Interpreter, GPT-6.1 Sol and Claude Opus 5.5 at max effort with Code Interpreter, and Gemini 3.8 Flash at high with Code Interpreter. Vendor-reported. Metric: Accuracy (%). Higher is better. Source: https://elorian.ai/blog/introducing-glimpse
Elorian Glimpse Card - CharXiv-R (Accuracy (%)): leader Claude Opus 5.5 (Max) (93.4), 5 models
CharXiv reasoning questions about charts from scientific papers. Elorian reported this for its Glimpse research preview (October 16, 2026), running every model at its highest thinking effort with its native tools: Glimpse at max, GPT-6 Astra at max with Code Interpreter, GPT-6.1 Sol and Claude Opus 5.5 at max effort with Code Interpreter, and Gemini 3.8 Flash at high with Code Interpreter. Vendor-reported. Metric: Accuracy (%). Higher is better. Source: https://elorian.ai/blog/introducing-glimpse
Elorian Glimpse Card - UGC-VideoCap (Score): leader Glimpse (Max) (82.66), 5 models
UGC-VideoCap captioning of short user-generated videos. Elorian reported this for its Glimpse research preview (October 16, 2026), running each model with its native tools: Glimpse at max, GPT-6 Astra at max with Code Interpreter, GPT-6.1 Sol and Claude Opus 5.5 at high effort with Code Interpreter (the post says max-effort results are not available for these video evaluations), and Gemini 3.8 Flash at high with its Video Agent. Vendor-reported. Metric: Score. Higher is better. Source: https://elorian.ai/blog/introducing-glimpse
Elorian Glimpse Card - DREAM-1K (Score): leader GPT-6 Astra (Max) (79.05), 5 models
DREAM-1K dense, detailed video captioning. Elorian reported this for its Glimpse research preview (October 16, 2026), running each model with its native tools: Glimpse at max, GPT-6 Astra at max with Code Interpreter, GPT-6.1 Sol and Claude Opus 5.5 at high effort with Code Interpreter (the post says max-effort results are not available for these video evaluations), and Gemini 3.8 Flash at high with its Video Agent. Vendor-reported. Metric: Score. Higher is better. Source: https://elorian.ai/blog/introducing-glimpse
Elorian Glimpse Card - TOMATO (Accuracy (%)): leader GPT-6 Astra (Max) (96.02), 5 models
TOMATO fine-grained temporal reasoning over video. Elorian reported this for its Glimpse research preview (October 16, 2026), running each model with its native tools: Glimpse at max, GPT-6 Astra at max with Code Interpreter, GPT-6.1 Sol and Claude Opus 5.5 at high effort with Code Interpreter (the post says max-effort results are not available for these video evaluations), and Gemini 3.8 Flash at high with its Video Agent. Vendor-reported. Metric: Accuracy (%). Higher is better. Source: https://elorian.ai/blog/introducing-glimpse
Elorian Glimpse Card - Video-MME v2 (with subtitles) (Grouped rubric score): leader GPT-6.1 Sol (Max) (65.27), 4 models
Video-MME v2 video understanding with subtitles, scored with its grouped rubric. Elorian reported this for its Glimpse research preview (October 16, 2026), running each model with its native tools: Glimpse at max, GPT-6 Astra at max with Code Interpreter, GPT-6.1 Sol at max and Claude Opus 5.5 at high effort with Code Interpreter (the post says max-effort results are not available for its other video evaluations), and Gemini 3.8 Flash at high with its Video Agent. Vendor-reported. Metric: Grouped rubric score. Higher is better. Source: https://elorian.ai/blog/introducing-glimpse
Elorian Glimpse Card - E.T. Bench (Score): leader Glimpse (Max) (63.55), 3 models
E.T. Bench event-level understanding of video. Elorian reported this for its Glimpse research preview (October 16, 2026), running each model with its native tools: Glimpse at max, GPT-6 Astra at max with Code Interpreter, GPT-6.1 Sol and Claude Opus 5.5 at high effort with Code Interpreter (the post says max-effort results are not available for these video evaluations), and Gemini 3.8 Flash at high with its Video Agent. Vendor-reported. Metric: Score. Higher is better. Source: https://elorian.ai/blog/introducing-glimpse
Elorian Glimpse Card - FSC-147 (Score): leader Glimpse (Max) (83.36), 5 models
FSC-147 open-vocabulary object counting in images (Elorian's higher-is-better score; the post does not define the metric further). Elorian reported this for its Glimpse research preview (October 16, 2026), running every model at its highest thinking effort with its native tools: Glimpse at max, GPT-6 Astra at max with Code Interpreter, GPT-6.1 Sol and Claude Opus 5.5 at max effort with Code Interpreter, and Gemini 3.8 Flash at high with Code Interpreter. Vendor-reported. Metric: Score. Higher is better. Source: https://elorian.ai/blog/introducing-glimpse
Elorian Glimpse Card - CountQA (Accuracy (%)): leader GPT-6 Astra (Max) (86.42), 5 models
CountQA counting questions about product images. Elorian reported this for its Glimpse research preview (October 16, 2026), running every model at its highest thinking effort with its native tools: Glimpse at max, GPT-6 Astra at max with Code Interpreter, GPT-6.1 Sol and Claude Opus 5.5 at max effort with Code Interpreter, and Gemini 3.8 Flash at high with Code Interpreter. Vendor-reported. Metric: Accuracy (%). Higher is better. Source: https://elorian.ai/blog/introducing-glimpse

Top-10 New Scores (56)

Claude Fable 5.1 on KernelBench CUDA: 0.2188 (#9)
Claude Opus 5.5 on CAIS Vision Capabilities Index: 68.1 (#2)
Claude Opus 5.5 on KernelBench CUDA: 0.5167 (#2)
Claude Opus 5.5 on Lech Mazur Writing: 3.715 (#3)
Claude Opus 5.5 on MedXpertQA MM (Natomy): 86.55 (#4)
Claude Opus 5.5 on MedXpertQA MM (Natomy) - Basic Science: 89.52 (#2)
Claude Opus 5.5 on MedXpertQA MM (Natomy) - Diagnosis: 85.99 (#4)
Claude Opus 5.5 on MedXpertQA MM (Natomy) - Treatment: 85.71 (#3)
GPT-6 Astra Pro on KernelBench CUDA: 0.2068 (#10)
GPT-6.1 Sol on CAIS Vision Capabilities Index: 68.0 (#3)
GPT-6.1 Sol on KernelBench CUDA: 0.1975 (#12)
GPT-6.1 Sol on OpenAI GPT-6 Astra Launch - Internal Computer Use Safety Benchmark: 4.32 (#1)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Adversarial User Simulations - Mental Health: 100.0 (#3)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Chat Plugins: 100.0 (#2)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Non-Violent Wrongdoing: 98.9 (#3)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Self-Harm: 92.0 (#4)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Sensitive Personal Data: 74.4 (#4)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Human Red-Teaming Chat Plugins: 95.5 (#2)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Biology Refusal - Benign: 98.2 (#1)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Biology Refusal - Dual Use: 98.0 (#2)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Biology Refusal - Severe: 99.8 (#3)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Challenging Prompts - Extremism: 97.9 (#3)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Challenging Prompts - Gore: 88.9 (#3)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Challenging Prompts - Hate: 100.0 (#3)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Challenging Prompts - Sexual/Minors: 98.3 (#2)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Challenging Prompts - Violent Illicit Behavior: 98.3 (#4)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Cyber Safety - Production Chat: 0.987 (#1)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Cyber Safety - Semi-Synthetic Agentic Environments: 0.98 (#4)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Cyber Safety - Synthetic Agentic Environments: 0.997 (#3)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - HealthBench Hard: 36.2 (#2)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - HealthBench Professional: 64.2 (#2)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Image Input - Erotic: 100.0 (#2)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Image Input - Extremism: 98.4 (#2)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Image Input - Hate: 99.8 (#3)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Image Input - Self-Harm: 99.4 (#3)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Multimodal Troubleshooting Virology: 55.34 (#2)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - ProtocolQA Open-Ended: 40.74 (#3)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - SHP2 Protein Function Prediction: 0.332 (#2)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Static Jailbreaks - Bio High Risk: 93.8 (#2)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Static Jailbreaks - Bio Severe: 94.3 (#2)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Static Jailbreaks - Cyber: 85.8 (#3)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Static Jailbreaks - Violence Moderate: 88.8 (#3)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Static Jailbreaks - Violence Severe: 95.0 (#2)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - Tacit Knowledge and Troubleshooting: 88.5 (#2)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - TroubleshootingBench: 47.96 (#2)
GPT-6.1 Sol on OpenAI GPT-6 Astra System Card - U18 - Sexual Content: 98.4 (#2)
GPT-6.1 Sol on OpenAI GPT-6 Sol & Luna Launch - Broken Search Tool Non-Disclosure: 2.1 (#2)
GPT-6.1 Sol on OpenAI GPT-6 Sol & Luna Launch - Factual Errors on User-Flagged Conversations: 4.12 (#4)
GPT-6.1 Sol on OpenAI GPT-6 Sol & Luna Launch - Reviewer Bypass Attempts: 0.0 (#1)
GPT-6.1 Sol on OpenAI GPT-6 Sol & Luna Launch - Warning Circumvention: 23.48 (#2)
Gemini 4 Argon on CAIS Risk Index: 56.2 (#33)
Gemini 4 Argon on KernelBench CUDA: 0.2732 (#6)
Gemini 4 Argon on Lech Mazur Writing: 0.8 (#19)
Muse Wonder Full on AA-LCR: 83.67 (#11)
Muse Wonder Full on CritPt: 32.9 (#1)
Muse Wonder Full on Finance Agent v2: 59.1 (#6)

New #1 Leaders (29)

OpenAI GPT-6 Astra System Card - TroubleshootingBench: GPT-6 Astra (63.46) beat GPT-6 Sol by 18.16
OpenAI GPT-6 Astra System Card - Tacit Knowledge and Troubleshooting: GPT-6 Astra (92.55) beat GPT-6 Sol by 13.35
OpenAI GPT-6 Astra System Card - Multimodal Troubleshooting Virology: GPT-6 Astra (63.11) beat GPT-6 Sol by 12.51
ToolSandbox: LFM2.5-24B-A2B (80.71) beat GPT-4o (2024-05-13) by 7.71
OpenAI GPT-6 Astra System Card - U18 - Eating Disorders: GPT-6.1 Sol (96.2) beat GPT-6 Astra by 4.1
AI Coding Daily (OpenCode) - Total: DeepSeek V4.1 Flash (1017) (Max) (60.97) beat Space Bunny (Max) by 3.78
MedXpertQA MM (Natomy) - Basic Science: Gemini 4 Argon (92.92) beat GPT-6 Astra by 3.68
MedXpertQA MM (Natomy) - Treatment: Gemini 4 Argon (90.18) beat GPT-6 Astra by 3.13
MedXpertQA MM (Natomy): Gemini 4 Argon (90.5) beat GPT-6 Astra by 3.1
MedXpertQA MM (Natomy) - Diagnosis: Gemini 4 Argon (89.91) beat GPT-6 Astra by 2.92
Crosby micro1 RedlineBench: Gemini 4 Argon (59.5) beat Claude Opus 5 by 2.6
ScreenSpot-Pro: GroundCascade (Indeed-UI-32B + KV-Ground-8B) ScreenSpot-Pro official grounding leaderboard (85.14) beat Indeed-UI-32B-zoomin ScreenSpot-Pro official grounding leaderboard by 2.41
OpenAI GPT-6 Astra System Card - U18 - Emotional Reliance: GPT-6.1 Sol (96.9) beat GPT-6 Sol by 2.1
OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Age-Restricted Actions: GPT-6.1 Sol (83.0) beat GPT-6 Astra by 1.9
OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Violent Wrongdoing: GPT-6.1 Sol (92.6) beat GPT-6 Astra by 1.9
OpenAI GPT-6 Astra System Card - U18 - Age-Restricted Goods and Dangerous Activities: GPT-6.1 Sol (93.4) beat GPT-6 Astra by 1.6
MCP Atlas: Muse Spark Open (xhigh effort, full 1,000-task set) (xHigh) (89.2) beat Muse Spark 1.1 by 1.1
OpenAI GPT-6 Astra System Card - ProtocolQA Open-Ended: GPT-6 Astra (45.37) beat GPT-6 Sol by 0.97
OpenAI GPT-6 Astra System Card - Adversarial User Simulations - Self-Harm: GPT-6.1 Sol (99.6) beat GPT-6 Astra by 0.7
GameDevBench: Claude Opus 5.5 (Claude Code, xHigh) (70.6) beat GPT-6.1 Sol (Codex, xHigh) by 0.63
CritPt: Muse Wonder Full (Max) (32.9) beat GPT-5.6 Sol (Max) by 0.6
OpenAI GPT-6 Astra System Card - Challenging Prompts - Non-Violent Illicit Behavior: GPT-6.1 Sol (100.0) beat GPT-6 Astra by 0.3
OpenAI GPT-6 Astra System Card - Challenging Prompts - Sexual: GPT-6.1 Sol (98.7) beat GPT-6 Sol by 0.3
OpenAI GPT-6 Astra System Card - HealthBench: GPT-6.1 Sol (58.5) beat GPT-6 Astra by 0.2
OpenAI GPT-6 Astra System Card - Challenging Prompts - Self-Harm: GPT-6.1 Sol (99.4) beat GPT-6 Astra by 0.2
OpenAI GPT-6 Astra System Card - U18 - Self-Harm: GPT-6.1 Sol (99.7) beat GPT-6 Astra by 0.2
OpenAI GPT-6 Astra System Card - Adversarial User Simulations - Emotional Reliance: GPT-6.1 Sol (99.5) beat GPT-6 Astra by 0.2
CAIS Risk Index: Claude Opus 5.5 (27.1) beat Claude Fable 5 by 0.2
OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Human Red-Teaming Codex: GPT-6.1 Sol (97.8) beat GPT-6 Astra by 0.1

The Aggregate Digest — 2026-10-18

2026-10-18T06:10:22.705382+00:00

Frontier

Best available model: DeepSeek V4.1 Flash (1017) enters at #15 (Aggregate Rating 1911) on 42 benchmarks

Daily

New Benchmarks (33)

Wiz Cyber Model Arena (Antigravity) - Overall (Macro average of the four category pass@1 rates (%)): leader Gemini 4 Argon (77.0), 5 models
Wiz Cyber Model Arena (Antigravity) - Code Vulnerabilities (Code Vulnerabilities challenges solved, pass@1 (%)): leader Gemini 4 Argon (64.9), 5 models
Wiz Cyber Model Arena (Antigravity) - API Security (API Security challenges solved, pass@1 (%)): leader Gemini 4 Argon (87.0), 5 models
Wiz Cyber Model Arena (Antigravity) - Websec.fr CTFs (Websec.fr CTFs challenges solved, pass@1 (%)): leader Gemini 3.8 Flash Cyber (77.0), 5 models
Wiz Cyber Model Arena (Antigravity) - Cloud Security (Cloud Security challenges solved, pass@1 (%)): leader Gemini 4 Argon (87.0), 5 models
DeepSeek V4.1 Flash 1017 Card - HLE w/o tools (full set) (Accuracy (%)): leader DeepSeek V4.1 Flash (1017) (42.6), 2 models
Humanity's Last Exam without tools, full question set. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Accuracy (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - HLE w/o tools (text-only subset) (Accuracy (%)): leader DeepSeek V4.1 Flash (1017) (44.0), 2 models
Humanity's Last Exam without tools, text-only subset of the questions. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Accuracy (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - Terminal-Bench 4.0 (Claude Code harness) (Score (%)): leader DeepSeek V4.1 Flash (1017) (42.3), 2 models
Terminal-Bench 4.0 command-line tasks, run in the Claude Code harness. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Score (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - DeepSWE v1.1 (Mini-SWE-V2 harness) (Pass@1 (%)): leader DeepSeek V4.1 Flash (1017) (77.0), 2 models
DeepSWE v1.1 software-engineering tasks, run in the Mini-SWE-V2 harness, one attempt per task. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Pass@1 (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - ProgramBench (Score (%)): leader DeepSeek V4.1 Flash (1017) (29.1), 2 models
ProgramBench program-construction tasks. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Score (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - NL2Repo-Bench (Score (%)): leader DeepSeek V4.1 Flash (1017) (68.2), 2 models
NL2Repo-Bench: building a complete code repository from a natural-language specification. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Score (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - Automation-Bench v1.0.6 (Score (%)): leader DeepSeek V4.1 Flash (1017) (60.5), 2 models
Automation-Bench version 1.0.6 workflow-automation tasks. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Score (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - Agents' Last Exam (Score (%)): leader DeepSeek V4.1 Flash (1017) (35.4), 2 models
Agents' Last Exam agentic tasks. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Score (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - Apex Bench v1.1 (Score (%)): leader DeepSeek V4.1 Flash (1017) (72.1), 2 models
Apex Bench version 1.1. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Score (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - Chartography (with tools) (Score (%)): leader DeepSeek V4.1 Flash (1017) (83.0), 2 models
Chartography chart-understanding questions, with tools allowed. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Score (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - BabyVision (with tools) (Score (%)): leader DeepSeek V4.1 Flash (1017) (92.5), 2 models
BabyVision visual-reasoning questions, with tools allowed. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Score (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - ZeroBench-main (with tools, Pass@5) (Pass@5 (%)): leader DeepSeek V4.1 Flash (1017) (51.0), 2 models
ZeroBench main set of hard visual-reasoning questions, with tools allowed, scored pass@5. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Pass@5 (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
Meta Muse Wonder Scorecard - MemoryBench (Rubric score (%)): leader Muse Wonder (Max) (85.6), 5 models
Meta's internal MemoryBench: recalling critical details across long multi-turn conversations of 64K to 100K tokens, with distractor conversations and turns that conflict with or update earlier facts; an LLM judge scores each answer against weighted, question-specific rubrics. The questions are not public. Meta ran this for its October 12, 2026 Muse Wonder release scorecard, with every model at max reasoning effort; competitor runs are Meta's best-effort internal runs. Metric: Rubric score (%). Higher is better. Source: https://research.meta.ai/blog/muse-wonder
Meta Muse Wonder Scorecard - OSWorld V2.1 (mean partial reward) (Mean per-task partial reward (%)): leader Claude Fable 5.1 (Max) (80.7), 4 models
OSWorld V2.1: 108 long-horizon real-world workflows on a full Ubuntu desktop, controlled through screenshots, clicks and typing, run in Meta's common internal framework with the official task environments and graders. Mean partial reward per task. Meta ran this for its October 12, 2026 Muse Wonder release scorecard, with every model at max reasoning effort; competitor runs are Meta's best-effort internal runs. Claude Opus 5.5's provider-reported score is left out. Metric: Mean per-task partial reward (%). Higher is better. Source: https://research.meta.ai/blog/muse-wonder
Meta Muse Wonder Scorecard - OSWorld V2.1 (strict binary completion) (Strict task completion (%)): leader Claude Fable 5.1 (Max) (43.1), 4 models
OSWorld V2.1: 108 long-horizon real-world workflows on a full Ubuntu desktop, controlled through screenshots, clicks and typing, run in Meta's common internal framework with the official task environments and graders. Share of tasks fully completed. Meta ran this for its October 12, 2026 Muse Wonder release scorecard, with every model at max reasoning effort; competitor runs are Meta's best-effort internal runs. Metric: Strict task completion (%). Higher is better. Source: https://research.meta.ai/blog/muse-wonder
HoldemBench (All-in-adjusted win rate (mbb/hand)): leader DeepSeek V4.1 Flash (5368.1), 12 models
HoldemBench: duplicate heads-up no-limit Texas hold'em (blinds 50/100, 100 big blinds, stacks reset every hand). Every deal is played twice with seats swapped and all-in hands before the river are scored by exact equity, so most card luck cancels. LLMs, OpenRouter decision models and a local model play through one identical action menu. Score: all-in-adjusted win rate in milli-big-blinds per hand against the whole field, from one additive model over all pairings of six leagues (95% deal-cluster bootstrap intervals upstream). Unbounded; 0 is break-even. Reference bots are excluded. Higher is better. Source: https://huggingface.co/datasets/chrystians/poker-holdembench
GACC-Bench (5% budget) - Micro accuracy (Question accuracy (%)): leader GPT-6.1 Sol (42.86), 10 models
GACC-Bench-full (500 goals, 2,571 questions over 281 contexts from seven long-document and agent-trace collections). The listed model is the goal-conditioned compressor; questions are withheld from it. A fixed DeepSeek-V4-Pro reader answers each question from the compressed context and a fixed DeepSeek-V4-Pro judge scores it against the reference (98.25% human agreement on 114 sampled responses). Retention budget 5% of source-context tokens. Micro accuracy is the percentage of individual questions answered correctly. Model labels as printed; reasoning or effort settings are not disclosed. Full-context reference: 99.38 Micro / 96.80 Macro; no-context: 6.46 / 0.20. Metric: Question accuracy (%). Higher is better. Source: https://arxiv.org/html/2610.21893v1
GACC-Bench (5% budget) - Macro accuracy (Goal accuracy, all questions correct (%)): leader GPT-6 Astra (6.0), 10 models
GACC-Bench-full (500 goals, 2,571 questions over 281 contexts from seven long-document and agent-trace collections). The listed model is the goal-conditioned compressor; questions are withheld from it. A fixed DeepSeek-V4-Pro reader answers each question from the compressed context and a fixed DeepSeek-V4-Pro judge scores it against the reference (98.25% human agreement on 114 sampled responses). Retention budget 5% of source-context tokens. Macro accuracy is the percentage of goals for which every associated question is answered correctly, computed over all 500 goals. Model labels as printed; reasoning or effort settings are not disclosed. Full-context reference: 99.38 Micro / 96.80 Macro; no-context: 6.46 / 0.20. Metric: Goal accuracy, all questions correct (%). Higher is better. Source: https://arxiv.org/html/2610.21893v1
GACC-Bench (10% budget) - Micro accuracy (Question accuracy (%)): leader GPT-6 Astra (57.41), 10 models
GACC-Bench-full (500 goals, 2,571 questions over 281 contexts from seven long-document and agent-trace collections). The listed model is the goal-conditioned compressor; questions are withheld from it. A fixed DeepSeek-V4-Pro reader answers each question from the compressed context and a fixed DeepSeek-V4-Pro judge scores it against the reference (98.25% human agreement on 114 sampled responses). Retention budget 10% of source-context tokens. Micro accuracy is the percentage of individual questions answered correctly. Model labels as printed; reasoning or effort settings are not disclosed. Full-context reference: 99.38 Micro / 96.80 Macro; no-context: 6.46 / 0.20. Metric: Question accuracy (%). Higher is better. Source: https://arxiv.org/html/2610.21893v1
GACC-Bench (10% budget) - Macro accuracy (Goal accuracy, all questions correct (%)): leader GPT-6 Astra (14.8), 10 models
GACC-Bench-full (500 goals, 2,571 questions over 281 contexts from seven long-document and agent-trace collections). The listed model is the goal-conditioned compressor; questions are withheld from it. A fixed DeepSeek-V4-Pro reader answers each question from the compressed context and a fixed DeepSeek-V4-Pro judge scores it against the reference (98.25% human agreement on 114 sampled responses). Retention budget 10% of source-context tokens. Macro accuracy is the percentage of goals for which every associated question is answered correctly, computed over all 500 goals. Model labels as printed; reasoning or effort settings are not disclosed. Full-context reference: 99.38 Micro / 96.80 Macro; no-context: 6.46 / 0.20. Metric: Goal accuracy, all questions correct (%). Higher is better. Source: https://arxiv.org/html/2610.21893v1
Hy-Memory Writer Matrix - Mem0 (Six-metric mean score (%)): leader Hy-Memory-30B-A3B (54.02), 7 models
Long-term memory writing: the listed model is the memory writer inside an unchanged memory framework; a frozen answer model and judge are shared. Each harness score is the equal-weight mean of six metrics (LoCoMo judge, LongMemEval, PersonaMem-v2, Travel SPS, Shopping PS, Code Memory Bench) on fixed deterministic 20% benchmark subsamples. Mem0 harness (trained on). Checkpoint identifiers are omitted by the paper. Hy3 is the paper's Hy3 flagship label (reasoning mode unspecified); Hy3-A3B Base is the paper's label for the Hy3-30B-A3B instruction-tuned starting checkpoint of Hy-Memory-30B-A3B. Metric: Six-metric mean score (%). Higher is better. Source: https://arxiv.org/html/2610.22196v1
Hy-Memory Writer Matrix - Hy-Memory (Six-metric mean score (%)): leader Hy-Memory-30B-A3B (60.55), 7 models
Long-term memory writing: the listed model is the memory writer inside an unchanged memory framework; a frozen answer model and judge are shared. Each harness score is the equal-weight mean of six metrics (LoCoMo judge, LongMemEval, PersonaMem-v2, Travel SPS, Shopping PS, Code Memory Bench) on fixed deterministic 20% benchmark subsamples. Hy-Memory harness (trained on). Checkpoint identifiers are omitted by the paper. Hy3 is the paper's Hy3 flagship label (reasoning mode unspecified); Hy3-A3B Base is the paper's label for the Hy3-30B-A3B instruction-tuned starting checkpoint of Hy-Memory-30B-A3B. Metric: Six-metric mean score (%). Higher is better. Source: https://arxiv.org/html/2610.22196v1
Hy-Memory Writer Matrix - MemOS (Six-metric mean score (%)): leader Hy-Memory-30B-A3B (50.51), 7 models
Long-term memory writing: the listed model is the memory writer inside an unchanged memory framework; a frozen answer model and judge are shared. Each harness score is the equal-weight mean of six metrics (LoCoMo judge, LongMemEval, PersonaMem-v2, Travel SPS, Shopping PS, Code Memory Bench) on fixed deterministic 20% benchmark subsamples. MemOS harness (trained on). Checkpoint identifiers are omitted by the paper. Hy3 is the paper's Hy3 flagship label (reasoning mode unspecified); Hy3-A3B Base is the paper's label for the Hy3-30B-A3B instruction-tuned starting checkpoint of Hy-Memory-30B-A3B. Metric: Six-metric mean score (%). Higher is better. Source: https://arxiv.org/html/2610.22196v1
Hy-Memory Writer Matrix - EverMemOS (Six-metric mean score (%)): leader Hy-Memory-30B-A3B (56.89), 7 models
Long-term memory writing: the listed model is the memory writer inside an unchanged memory framework; a frozen answer model and judge are shared. Each harness score is the equal-weight mean of six metrics (LoCoMo judge, LongMemEval, PersonaMem-v2, Travel SPS, Shopping PS, Code Memory Bench) on fixed deterministic 20% benchmark subsamples. EverMemOS harness, held out from Hy-Memory training. Checkpoint identifiers are omitted by the paper. Hy3 is the paper's Hy3 flagship label (reasoning mode unspecified); Hy3-A3B Base is the paper's label for the Hy3-30B-A3B instruction-tuned starting checkpoint of Hy-Memory-30B-A3B. Metric: Six-metric mean score (%). Higher is better. Source: https://arxiv.org/html/2610.22196v1
Hy-Memory Writer Matrix - Macro average (Six-metric mean score (%)): leader Hy-Memory-30B-A3B (55.49), 7 models
Long-term memory writing: the listed model is the memory writer inside an unchanged memory framework; a frozen answer model and judge are shared. Each harness score is the equal-weight mean of six metrics (LoCoMo judge, LongMemEval, PersonaMem-v2, Travel SPS, Shopping PS, Code Memory Bench) on fixed deterministic 20% benchmark subsamples. Equal-weight average of the four harness scores. Checkpoint identifiers are omitted by the paper. Hy3 is the paper's Hy3 flagship label (reasoning mode unspecified); Hy3-A3B Base is the paper's label for the Hy3-30B-A3B instruction-tuned starting checkpoint of Hy-Memory-30B-A3B. Metric: Six-metric mean score (%). Higher is better. Source: https://arxiv.org/html/2610.22196v1
PremiseSQL - Base prompting - Flaw detection (DET (fraction)): leader Claude Opus 5.5 (0.492), 16 models
PremiseSQL premise-critique evaluation over Spider and BIRD Text-to-SQL queries with flawed-premise variants and their paired originals (3,600 requests per model and prompting setting), temperature 0 and default API configuration, two-judge labels. Base prompting: standard Text-to-SQL instructions with no premise warning. DET: share of flawed queries for which the model detects a premise flaw. Fraction from 0 to 1. Model labels as printed. Metric: DET (fraction). Higher is better. Source: https://arxiv.org/html/2610.22158v1
PremiseSQL - Implicit prompting - Flaw detection (DET (fraction)): leader DeepSeek V3.2 (0.898), 16 models
PremiseSQL premise-critique evaluation over Spider and BIRD Text-to-SQL queries with flawed-premise variants and their paired originals (3,600 requests per model and prompting setting), temperature 0 and default API configuration, two-judge labels. Implicit prompting: the model is told that the question may contain errors and asked to assess its validity before answering. DET: share of flawed queries for which the model detects a premise flaw. Fraction from 0 to 1. Model labels as printed. Metric: DET (fraction). Higher is better. Source: https://arxiv.org/html/2610.22158v1
PremiseSQL - Implicit prompting - Valid-query recognition (VAL (fraction)): leader Gemini 3.1 Pro (Preview) (0.99), 16 models
PremiseSQL premise-critique evaluation over Spider and BIRD Text-to-SQL queries with flawed-premise variants and their paired originals (3,600 requests per model and prompting setting), temperature 0 and default API configuration, two-judge labels. Implicit prompting: the model is told that the question may contain errors and asked to assess its validity before answering. VAL: share of the paired valid original queries that the model explicitly recognises as valid, the guard against rejecting everything. Fraction from 0 to 1. Model labels as printed. Metric: VAL (fraction). Higher is better. Source: https://arxiv.org/html/2610.22158v1

New Models (1)

DeepSeek V4.1 Flash (1017) — Aggregate Rating 1911, #15
- SelfBench - PostHog/posthog: 65.71 (#2/14)
- SelfBench - mupt-ai/self-bench: 71.05 (#3/7)
- SelfBench - vercel/next.js: 71.05 (#3/13)
- SelfBench - getsentry/sentry: 79.49 (#4/15)
- NL2Repo: 68.2 (#5/51)
- LLM2014 Logic 2026-10: 59.46 (#9/45)
- VoxelBench: 2052.0 (#10/59)
- Bug Hunt Bench - LMS: 21.2 (#11/117)
- Bug Hunt Bench: 39.4 (#12/117)
- Bug Hunt Bench - VS Code Extension: 18.2 (#12/117)
- ...and 20 more

Top-10 New Scores (17)

Claude Opus 5.5 on HiL-Bench: 59.0 (#7)
GPT-6 Astra on HiL-Bench: 65.32 (#3)
GPT-6.1 Sol on HiL-Bench: 62.33 (#4)
Gemini 4 Argon on Bug Hunt Bench: 40.7 (#11)
Gemini 4 Argon on Bug Hunt Bench - LMS: 27.7 (#2)
Gemini 4 Argon on Bug Hunt Bench - VS Code Extension: 13.0 (#31)
Gemini 4 Argon on HiL-Bench: 62.0 (#5)
Gemini 4 Argon on MathArena - ARXIVLEAN September: 39.62 (#2)
Gemini 4 Argon on UGI - Natural Intelligence: 81.36 (#3)
Gemini 4 Argon on UGI - Willingness (W/10): 1.8 (#1247)
Gemini 4 Argon on UGI - Writing: 79.71 (#1)
Gemini 4 Argon on UGI Leaderboard: 49.92 (#150)
Gemini 4 Argon on Vectara Hallucination Leaderboard: 87.9 (#83)
Gemini 4 Argon on Wiz Cyber Model Arena (ADK React) - Websec.fr CTFs: 74.7 (#7)
Gemini 4 Argon on Wiz Cyber Model Arena (Claude Code) - API Security: 82.7 (#3)
Gemini 4 Argon on Wiz Cyber Model Arena (Claude Code) - Overall: 70.8 (#2)
Gemini 4 Argon on Wiz Cyber Model Arena (Claude Code) - Websec.fr CTFs: 66.7 (#12)

New #1 Leaders (10)

KNOWS - Accessibility Tree + Screenshot: GPT-6 Astra (72.5) beat GPT-5.5 by 23.0
HiL-Bench: Muse Wonder (79.67) beat Claude Fable 5.1 by 18.17
Wiz Cyber Model Arena (ADK React) - Cloud Security: Gemini 4 Argon (82.6) beat Gemini 3.8 Flash Cyber by 15.9
Wiz Cyber Model Arena (Claude Code) - Cloud Security: Gemini 4 Argon (73.9) beat Claude Opus 5 by 10.1
Wiz Cyber Model Arena (ADK React) - Code Vulnerabilities: Gemini 4 Argon (65.8) beat Gemini 3.8 Flash Cyber by 8.8
Wiz Cyber Model Arena (Claude Code) - Websec.fr CTFs: GPT-6 Sol (88.5) beat Claude Opus 5 by 6.9
Wiz Cyber Model Arena (Claude Code) - Code Vulnerabilities: Gemini 4 Argon (59.9) beat Claude Opus 5 by 4.7
Wiz Cyber Model Arena (ADK React) - Overall: Gemini 4 Argon (77.2) beat Gemini 3.8 Flash Cyber by 2.3
UGI - Writing: gemini-4-argon (Thinking, low) (79.71) beat Gemini 3.8 Flash (Thinking, Medium) by 1.16
Wiz Cyber Model Arena (ADK React) - API Security: Gemini 4 Argon (85.8) beat Gemini 3.8 Flash Cyber by 0.6

Weekly

New Benchmarks (38)

AgenticBBO Frontier - Digits MLP HPO (Normalized anytime/final score (0-1)): leader GPT-6 Astra (Max) (0.617), 7 models
AgenticBBO-Bench frontier challenge on the single Digits MLP HPO task, with 5 initial + 25 new black-box evaluations and seed 2. An LLM agent uses the Codex harness to propose and adapt optimization queries. The normalized score is 70% anytime trajectory quality plus 30% final quality, bounded 0–1; higher is better. This is one run per model/task, not a mean over the broad 77-task benchmark. Published reproduction profiles request max reasoning for six models and high for Gemini 3.1 Pro Preview; original historical run catalogs are not distributed. The paper’s DeepSeek V4.1 Flash identity is retained despite its reproduction API alias lacking .1. The numerical Gaussian-process reference is not an LLM. Single-run results do not measure run-to-run uncertainty.
AgenticBBO Frontier - Sysbench Database Tuning (Normalized anytime/final score (0-1)): leader GPT-6 Astra (Max) (0.501), 7 models
AgenticBBO-Bench frontier challenge on the single Sysbench Database Tuning task, with 50 initial + 200 new black-box evaluations and seed 2. An LLM agent uses the Codex harness to propose and adapt optimization queries. The normalized score is 70% anytime trajectory quality plus 30% final quality, bounded 0–1; higher is better. This is one run per model/task, not a mean over the broad 77-task benchmark. Published reproduction profiles request max reasoning for six models and high for Gemini 3.1 Pro Preview; original historical run catalogs are not distributed. The paper’s DeepSeek V4.1 Flash identity is retained despite its reproduction API alias lacking .1. The numerical Gaussian-process reference is not an LLM. Single-run results do not measure run-to-run uncertainty.
AgenticBBO Frontier - Median1 Molecular Design (Normalized anytime/final score (0-1)): leader DeepSeek V4.1 Flash (Max) (0.823), 7 models
AgenticBBO-Bench frontier challenge on the single Median1 Molecular Design task, with 50 initial + 200 new black-box evaluations and seed 2. An LLM agent uses the Codex harness to propose and adapt optimization queries. The normalized score is 70% anytime trajectory quality plus 30% final quality, bounded 0–1; higher is better. This is one run per model/task, not a mean over the broad 77-task benchmark. Published reproduction profiles request max reasoning for six models and high for Gemini 3.1 Pro Preview; original historical run catalogs are not distributed. The paper’s DeepSeek V4.1 Flash identity is retained despite its reproduction API alias lacking .1. The numerical Gaussian-process reference is not an LLM. Single-run results do not measure run-to-run uncertainty.
AgenticBBO Frontier - BBOB f15 d10 (Normalized anytime/final score (0-1)): leader GPT-6 Astra (Max) (0.664), 7 models
AgenticBBO-Bench frontier challenge on the single BBOB f15 d10 task, with 20 initial + 100 new black-box evaluations and seed 2. An LLM agent uses the Codex harness to propose and adapt optimization queries. The normalized score is 70% anytime trajectory quality plus 30% final quality, bounded 0–1; higher is better. This is one run per model/task, not a mean over the broad 77-task benchmark. Published reproduction profiles request max reasoning for six models and high for Gemini 3.1 Pro Preview; original historical run catalogs are not distributed. The paper’s DeepSeek V4.1 Flash identity is retained despite its reproduction API alias lacking .1. The numerical Gaussian-process reference is not an LLM. Single-run results do not measure run-to-run uncertainty.
AgenticBBO Frontier - Bigblue1 Chip Placement (Normalized anytime/final score (0-1)): leader DeepSeek V4.1 Flash (Max) (0.673), 7 models
AgenticBBO-Bench frontier challenge on the single Bigblue1 Chip Placement task, with 50 initial + 200 new black-box evaluations and seed 2. An LLM agent uses the Codex harness to propose and adapt optimization queries. The normalized score is 70% anytime trajectory quality plus 30% final quality, bounded 0–1; higher is better. This is one run per model/task, not a mean over the broad 77-task benchmark. Published reproduction profiles request max reasoning for six models and high for Gemini 3.1 Pro Preview; original historical run catalogs are not distributed. The paper’s DeepSeek V4.1 Flash identity is retained despite its reproduction API alias lacking .1. The numerical Gaussian-process reference is not an LLM. Single-run results do not measure run-to-run uncertainty.
Wiz Cyber Model Arena (Antigravity) - Overall (Macro average of the four category pass@1 rates (%)): leader Gemini 4 Argon (77.0), 5 models
Wiz Cyber Model Arena (Antigravity) - Code Vulnerabilities (Code Vulnerabilities challenges solved, pass@1 (%)): leader Gemini 4 Argon (64.9), 5 models
Wiz Cyber Model Arena (Antigravity) - API Security (API Security challenges solved, pass@1 (%)): leader Gemini 4 Argon (87.0), 5 models
Wiz Cyber Model Arena (Antigravity) - Websec.fr CTFs (Websec.fr CTFs challenges solved, pass@1 (%)): leader Gemini 3.8 Flash Cyber (77.0), 5 models
Wiz Cyber Model Arena (Antigravity) - Cloud Security (Cloud Security challenges solved, pass@1 (%)): leader Gemini 4 Argon (87.0), 5 models
DeepSeek V4.1 Flash 1017 Card - HLE w/o tools (full set) (Accuracy (%)): leader DeepSeek V4.1 Flash (1017) (42.6), 2 models
Humanity's Last Exam without tools, full question set. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Accuracy (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - HLE w/o tools (text-only subset) (Accuracy (%)): leader DeepSeek V4.1 Flash (1017) (44.0), 2 models
Humanity's Last Exam without tools, text-only subset of the questions. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Accuracy (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - Terminal-Bench 4.0 (Claude Code harness) (Score (%)): leader DeepSeek V4.1 Flash (1017) (42.3), 2 models
Terminal-Bench 4.0 command-line tasks, run in the Claude Code harness. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Score (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - DeepSWE v1.1 (Mini-SWE-V2 harness) (Pass@1 (%)): leader DeepSeek V4.1 Flash (1017) (77.0), 2 models
DeepSWE v1.1 software-engineering tasks, run in the Mini-SWE-V2 harness, one attempt per task. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Pass@1 (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - ProgramBench (Score (%)): leader DeepSeek V4.1 Flash (1017) (29.1), 2 models
ProgramBench program-construction tasks. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Score (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - NL2Repo-Bench (Score (%)): leader DeepSeek V4.1 Flash (1017) (68.2), 2 models
NL2Repo-Bench: building a complete code repository from a natural-language specification. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Score (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - Automation-Bench v1.0.6 (Score (%)): leader DeepSeek V4.1 Flash (1017) (60.5), 2 models
Automation-Bench version 1.0.6 workflow-automation tasks. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Score (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - Agents' Last Exam (Score (%)): leader DeepSeek V4.1 Flash (1017) (35.4), 2 models
Agents' Last Exam agentic tasks. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Score (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - Apex Bench v1.1 (Score (%)): leader DeepSeek V4.1 Flash (1017) (72.1), 2 models
Apex Bench version 1.1. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Score (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - Chartography (with tools) (Score (%)): leader DeepSeek V4.1 Flash (1017) (83.0), 2 models
Chartography chart-understanding questions, with tools allowed. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Score (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - BabyVision (with tools) (Score (%)): leader DeepSeek V4.1 Flash (1017) (92.5), 2 models
BabyVision visual-reasoning questions, with tools allowed. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Score (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
DeepSeek V4.1 Flash 1017 Card - ZeroBench-main (with tools, Pass@5) (Pass@5 (%)): leader DeepSeek V4.1 Flash (1017) (51.0), 2 models
ZeroBench main set of hard visual-reasoning questions, with tools allowed, scored pass@5. DeepSeek reported this on the DeepSeek-V4.1-Flash-1017 model card and API change log (October 17, 2026). Only DeepSeek's own runs of the dated 1017 checkpoint and the original V4.1 Flash are included; the card's comparator columns are left out, and reasoning effort is not stated. Metric: Pass@5 (%). Higher is better. Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md
Meta Muse Wonder Scorecard - MemoryBench (Rubric score (%)): leader Muse Wonder (Max) (85.6), 5 models
Meta's internal MemoryBench: recalling critical details across long multi-turn conversations of 64K to 100K tokens, with distractor conversations and turns that conflict with or update earlier facts; an LLM judge scores each answer against weighted, question-specific rubrics. The questions are not public. Meta ran this for its October 12, 2026 Muse Wonder release scorecard, with every model at max reasoning effort; competitor runs are Meta's best-effort internal runs. Metric: Rubric score (%). Higher is better. Source: https://research.meta.ai/blog/muse-wonder
Meta Muse Wonder Scorecard - OSWorld V2.1 (mean partial reward) (Mean per-task partial reward (%)): leader Claude Fable 5.1 (Max) (80.7), 4 models
OSWorld V2.1: 108 long-horizon real-world workflows on a full Ubuntu desktop, controlled through screenshots, clicks and typing, run in Meta's common internal framework with the official task environments and graders. Mean partial reward per task. Meta ran this for its October 12, 2026 Muse Wonder release scorecard, with every model at max reasoning effort; competitor runs are Meta's best-effort internal runs. Claude Opus 5.5's provider-reported score is left out. Metric: Mean per-task partial reward (%). Higher is better. Source: https://research.meta.ai/blog/muse-wonder
Meta Muse Wonder Scorecard - OSWorld V2.1 (strict binary completion) (Strict task completion (%)): leader Claude Fable 5.1 (Max) (43.1), 4 models
OSWorld V2.1: 108 long-horizon real-world workflows on a full Ubuntu desktop, controlled through screenshots, clicks and typing, run in Meta's common internal framework with the official task environments and graders. Share of tasks fully completed. Meta ran this for its October 12, 2026 Muse Wonder release scorecard, with every model at max reasoning effort; competitor runs are Meta's best-effort internal runs. Metric: Strict task completion (%). Higher is better. Source: https://research.meta.ai/blog/muse-wonder
HoldemBench (All-in-adjusted win rate (mbb/hand)): leader DeepSeek V4.1 Flash (5368.1), 12 models
HoldemBench: duplicate heads-up no-limit Texas hold'em (blinds 50/100, 100 big blinds, stacks reset every hand). Every deal is played twice with seats swapped and all-in hands before the river are scored by exact equity, so most card luck cancels. LLMs, OpenRouter decision models and a local model play through one identical action menu. Score: all-in-adjusted win rate in milli-big-blinds per hand against the whole field, from one additive model over all pairings of six leagues (95% deal-cluster bootstrap intervals upstream). Unbounded; 0 is break-even. Reference bots are excluded. Higher is better. Source: https://huggingface.co/datasets/chrystians/poker-holdembench
GACC-Bench (5% budget) - Micro accuracy (Question accuracy (%)): leader GPT-6.1 Sol (42.86), 10 models
GACC-Bench-full (500 goals, 2,571 questions over 281 contexts from seven long-document and agent-trace collections). The listed model is the goal-conditioned compressor; questions are withheld from it. A fixed DeepSeek-V4-Pro reader answers each question from the compressed context and a fixed DeepSeek-V4-Pro judge scores it against the reference (98.25% human agreement on 114 sampled responses). Retention budget 5% of source-context tokens. Micro accuracy is the percentage of individual questions answered correctly. Model labels as printed; reasoning or effort settings are not disclosed. Full-context reference: 99.38 Micro / 96.80 Macro; no-context: 6.46 / 0.20. Metric: Question accuracy (%). Higher is better. Source: https://arxiv.org/html/2610.21893v1
GACC-Bench (5% budget) - Macro accuracy (Goal accuracy, all questions correct (%)): leader GPT-6 Astra (6.0), 10 models
GACC-Bench-full (500 goals, 2,571 questions over 281 contexts from seven long-document and agent-trace collections). The listed model is the goal-conditioned compressor; questions are withheld from it. A fixed DeepSeek-V4-Pro reader answers each question from the compressed context and a fixed DeepSeek-V4-Pro judge scores it against the reference (98.25% human agreement on 114 sampled responses). Retention budget 5% of source-context tokens. Macro accuracy is the percentage of goals for which every associated question is answered correctly, computed over all 500 goals. Model labels as printed; reasoning or effort settings are not disclosed. Full-context reference: 99.38 Micro / 96.80 Macro; no-context: 6.46 / 0.20. Metric: Goal accuracy, all questions correct (%). Higher is better. Source: https://arxiv.org/html/2610.21893v1
GACC-Bench (10% budget) - Micro accuracy (Question accuracy (%)): leader GPT-6 Astra (57.41), 10 models
GACC-Bench-full (500 goals, 2,571 questions over 281 contexts from seven long-document and agent-trace collections). The listed model is the goal-conditioned compressor; questions are withheld from it. A fixed DeepSeek-V4-Pro reader answers each question from the compressed context and a fixed DeepSeek-V4-Pro judge scores it against the reference (98.25% human agreement on 114 sampled responses). Retention budget 10% of source-context tokens. Micro accuracy is the percentage of individual questions answered correctly. Model labels as printed; reasoning or effort settings are not disclosed. Full-context reference: 99.38 Micro / 96.80 Macro; no-context: 6.46 / 0.20. Metric: Question accuracy (%). Higher is better. Source: https://arxiv.org/html/2610.21893v1
GACC-Bench (10% budget) - Macro accuracy (Goal accuracy, all questions correct (%)): leader GPT-6 Astra (14.8), 10 models
GACC-Bench-full (500 goals, 2,571 questions over 281 contexts from seven long-document and agent-trace collections). The listed model is the goal-conditioned compressor; questions are withheld from it. A fixed DeepSeek-V4-Pro reader answers each question from the compressed context and a fixed DeepSeek-V4-Pro judge scores it against the reference (98.25% human agreement on 114 sampled responses). Retention budget 10% of source-context tokens. Macro accuracy is the percentage of goals for which every associated question is answered correctly, computed over all 500 goals. Model labels as printed; reasoning or effort settings are not disclosed. Full-context reference: 99.38 Micro / 96.80 Macro; no-context: 6.46 / 0.20. Metric: Goal accuracy, all questions correct (%). Higher is better. Source: https://arxiv.org/html/2610.21893v1
Hy-Memory Writer Matrix - Mem0 (Six-metric mean score (%)): leader Hy-Memory-30B-A3B (54.02), 7 models
Long-term memory writing: the listed model is the memory writer inside an unchanged memory framework; a frozen answer model and judge are shared. Each harness score is the equal-weight mean of six metrics (LoCoMo judge, LongMemEval, PersonaMem-v2, Travel SPS, Shopping PS, Code Memory Bench) on fixed deterministic 20% benchmark subsamples. Mem0 harness (trained on). Checkpoint identifiers are omitted by the paper. Hy3 is the paper's Hy3 flagship label (reasoning mode unspecified); Hy3-A3B Base is the paper's label for the Hy3-30B-A3B instruction-tuned starting checkpoint of Hy-Memory-30B-A3B. Metric: Six-metric mean score (%). Higher is better. Source: https://arxiv.org/html/2610.22196v1
Hy-Memory Writer Matrix - Hy-Memory (Six-metric mean score (%)): leader Hy-Memory-30B-A3B (60.55), 7 models
Long-term memory writing: the listed model is the memory writer inside an unchanged memory framework; a frozen answer model and judge are shared. Each harness score is the equal-weight mean of six metrics (LoCoMo judge, LongMemEval, PersonaMem-v2, Travel SPS, Shopping PS, Code Memory Bench) on fixed deterministic 20% benchmark subsamples. Hy-Memory harness (trained on). Checkpoint identifiers are omitted by the paper. Hy3 is the paper's Hy3 flagship label (reasoning mode unspecified); Hy3-A3B Base is the paper's label for the Hy3-30B-A3B instruction-tuned starting checkpoint of Hy-Memory-30B-A3B. Metric: Six-metric mean score (%). Higher is better. Source: https://arxiv.org/html/2610.22196v1
Hy-Memory Writer Matrix - MemOS (Six-metric mean score (%)): leader Hy-Memory-30B-A3B (50.51), 7 models
Long-term memory writing: the listed model is the memory writer inside an unchanged memory framework; a frozen answer model and judge are shared. Each harness score is the equal-weight mean of six metrics (LoCoMo judge, LongMemEval, PersonaMem-v2, Travel SPS, Shopping PS, Code Memory Bench) on fixed deterministic 20% benchmark subsamples. MemOS harness (trained on). Checkpoint identifiers are omitted by the paper. Hy3 is the paper's Hy3 flagship label (reasoning mode unspecified); Hy3-A3B Base is the paper's label for the Hy3-30B-A3B instruction-tuned starting checkpoint of Hy-Memory-30B-A3B. Metric: Six-metric mean score (%). Higher is better. Source: https://arxiv.org/html/2610.22196v1
Hy-Memory Writer Matrix - EverMemOS (Six-metric mean score (%)): leader Hy-Memory-30B-A3B (56.89), 7 models
Long-term memory writing: the listed model is the memory writer inside an unchanged memory framework; a frozen answer model and judge are shared. Each harness score is the equal-weight mean of six metrics (LoCoMo judge, LongMemEval, PersonaMem-v2, Travel SPS, Shopping PS, Code Memory Bench) on fixed deterministic 20% benchmark subsamples. EverMemOS harness, held out from Hy-Memory training. Checkpoint identifiers are omitted by the paper. Hy3 is the paper's Hy3 flagship label (reasoning mode unspecified); Hy3-A3B Base is the paper's label for the Hy3-30B-A3B instruction-tuned starting checkpoint of Hy-Memory-30B-A3B. Metric: Six-metric mean score (%). Higher is better. Source: https://arxiv.org/html/2610.22196v1
Hy-Memory Writer Matrix - Macro average (Six-metric mean score (%)): leader Hy-Memory-30B-A3B (55.49), 7 models
Long-term memory writing: the listed model is the memory writer inside an unchanged memory framework; a frozen answer model and judge are shared. Each harness score is the equal-weight mean of six metrics (LoCoMo judge, LongMemEval, PersonaMem-v2, Travel SPS, Shopping PS, Code Memory Bench) on fixed deterministic 20% benchmark subsamples. Equal-weight average of the four harness scores. Checkpoint identifiers are omitted by the paper. Hy3 is the paper's Hy3 flagship label (reasoning mode unspecified); Hy3-A3B Base is the paper's label for the Hy3-30B-A3B instruction-tuned starting checkpoint of Hy-Memory-30B-A3B. Metric: Six-metric mean score (%). Higher is better. Source: https://arxiv.org/html/2610.22196v1
PremiseSQL - Base prompting - Flaw detection (DET (fraction)): leader Claude Opus 5.5 (0.492), 16 models
PremiseSQL premise-critique evaluation over Spider and BIRD Text-to-SQL queries with flawed-premise variants and their paired originals (3,600 requests per model and prompting setting), temperature 0 and default API configuration, two-judge labels. Base prompting: standard Text-to-SQL instructions with no premise warning. DET: share of flawed queries for which the model detects a premise flaw. Fraction from 0 to 1. Model labels as printed. Metric: DET (fraction). Higher is better. Source: https://arxiv.org/html/2610.22158v1
PremiseSQL - Implicit prompting - Flaw detection (DET (fraction)): leader DeepSeek V3.2 (0.898), 16 models
PremiseSQL premise-critique evaluation over Spider and BIRD Text-to-SQL queries with flawed-premise variants and their paired originals (3,600 requests per model and prompting setting), temperature 0 and default API configuration, two-judge labels. Implicit prompting: the model is told that the question may contain errors and asked to assess its validity before answering. DET: share of flawed queries for which the model detects a premise flaw. Fraction from 0 to 1. Model labels as printed. Metric: DET (fraction). Higher is better. Source: https://arxiv.org/html/2610.22158v1
PremiseSQL - Implicit prompting - Valid-query recognition (VAL (fraction)): leader Gemini 3.1 Pro (Preview) (0.99), 16 models
PremiseSQL premise-critique evaluation over Spider and BIRD Text-to-SQL queries with flawed-premise variants and their paired originals (3,600 requests per model and prompting setting), temperature 0 and default API configuration, two-judge labels. Implicit prompting: the model is told that the question may contain errors and asked to assess its validity before answering. VAL: share of the paired valid original queries that the model explicitly recognises as valid, the guard against rejecting everything. Fraction from 0 to 1. Model labels as printed. Metric: VAL (fraction). Higher is better. Source: https://arxiv.org/html/2610.22158v1

New Models (109)

GPT-6 Astra Pro — Aggregate Rating 1989, #3
- Chess Puzzles (Epoch AI): 76.0 (#1/153)
Claude Opus 5.5 — Aggregate Rating 1988, #4
- Agent Arena: 15.53 (#1/51)
- Agent Arena - Confirmed Success: 17.29 (#1/51)
- Anthropic Child Safety Single-Turn Benign Requests - Over-Refusal Rate (Claude.ai): 0.05 (#1/7)
- Anthropic Claude Code Dual-Use Requests - Success Rate (Mythos 5.1 System Card): 99.8 (#1/5)
- SWE-bench Pro (Opus 5 & Mythos 5.1 System Cards): 89.9 (#1/3)
- ParseBench: 79.85 (#1/113)
- ParseBench - Tables: 94.25 (#1/113)
- ParseBench - Charts: 70.89 (#1/113)
- STEMBench120 v1.0.0 (Closed Book) - Physics: 2.808 (#1/8)
- STEMBench120 v1.0.0 (Closed Book) - Earth science: 2.76 (#1/8)
- ...and 38 more
GPT-6 Astra — Aggregate Rating 1979, #6
- SciLaws-Bench: 51.33 (#1/14)
- Design Arena (3D): 1509.0 (#1/166)
- STEMBench120 v1.0.0 (Tools) - Overall: 2.809 (#1/2)
- STEMBench120 v1.0.0 (Tools) - Biology: 2.709 (#1/2)
- STEMBench120 v1.0.0 (Tools) - Chemistry: 2.777 (#1/2)
- STEMBench120 v1.0.0 (Tools) - Physics: 3.025 (#1/2)
- STEMBench120 v1.0.0 (Tools) - Astronomy: 2.723 (#1/2)
- STEMBench120 v1.0.0 (Tools) - Earth science: 2.726 (#1/2)
- STEMBench120 v1.0.0 (Tools) - Engineering: 2.894 (#1/2)
- KNOWS - Accessibility Tree + Screenshot: 72.5 (#1/4)
- ...and 3 more
GPT-6.1 Sol — Aggregate Rating 1967, #7
- GDP.pdf: 35.0 (#1/6)
- Kaggle FACTS (Google): 75.29 (#2/39)
- Kaggle FACTS Grounding: 82.42 (#2/52)
- Kaggle FACTS Parametric: 82.11 (#2/44)
- Kaggle FACTS Multimodal: 48.46 (#3/38)
- HiL-Bench: 62.33 (#4/28)
- HLE-Diamond (With Tools): 69.1 (#4/8)
- Design Arena (3D): 1458.0 (#5/166)
- Design Arena (UI Components): 1325.0 (#16/172)
- Kaggle Open Benchmarks - SciCode Subproblems: 37.8 (#16/60)
- ...and 23 more
Gemini 4 Argon — Aggregate Rating 1965, #8
- OTIS Mock AIME 2024-25: 100.0 (#1/304)
- SimpleQA Verified: 76.0 (#1/91)
- HieroglyphBench: 69.7 (#1/33)
- STEMBench120 v1.0.0 (Closed Book) - Astronomy: 2.778 (#1/8)
- Humanity's Sixth Sense: 55.4 (#1/27)
- Argo-Bench - Solved: 35.71 (#1/50)
- Icelandic LLM - Famous Phrases: 36.19 (#1/68)
- Icelandic LLM - Idiom Completion: 75.96 (#1/68)
- NYT Connections Extended: 98.6 (#1/128)
- Kaggle Global-MMLU-Lite - Bengali: 96.0 (#1/44)
- ...and 186 more
Claude Fable 5.1 — Aggregate Rating 1958, #9
- BBQ Disambiguated Accuracy (Mythos 5.1 System Card): 89.92 (#1/6)
- Anthropic Election Integrity Multi-Turn - Appropriate Response Rate (API): 90.0 (#4/8)
Muse Wonder Full — Aggregate Rating 1943, #10
- LLM Stats (DeepSWE 1.1): 79.0 (#1/45)
- LLM Stats (SWE-Marathon): 57.5 (#1/8)
- Vals AI Vibe Code Bench 1-100: 33.98 (#1/25)
- AA CritPt: 32.86 (#1/581)
- AA GDP.pdf: 32.6 (#1/224)
- Harvey LAB-AA: 13.06 (#1/27)
- Harvey LAB-AA - Corporate M&A: 98.18 (#1/27)
- Harvey LAB-AA - Funds & Asset Management: 95.72 (#1/27)
- Harvey LAB-AA - Intellectual Property: 97.86 (#1/27)
- Harvey LAB-AA - International Trade & Sanctions: 96.41 (#1/27)
- ...and 108 more
Claude Sonnet 5.5 — Aggregate Rating 1936, #11
- Conceptual Reasoning Index - Argument Evaluation (LMCA): 68.26 (#1/211)
- ReLE - Coding: 78.0 (#1/90)
- Conceptual Reasoning Index: 77.08 (#2/210)
- SecIT Bench (Claude Code): 69.93 (#2/4)
- Design Arena (3D): 1463.0 (#4/166)
- SEAL - Humanity's Last Exam: 46.12 (#4/55)
- SEAL - Humanity's Last Exam (Text Only): 46.62 (#4/65)
- GDP.pdf: 31.4 (#4/6)
- Conceptual Reasoning Index - Consistency (ACCoRD): 85.66 (#4/210)
- ReLE - Reasoning and Mathematics: 85.7 (#8/187)
- ...and 19 more
Claude Opus 5 — Aggregate Rating 1930, #12
- EnterpriseBench (CoreCraft Agents): 68.7 (#4/6)
- BBQ Disambiguated Accuracy (Mythos 5.1 System Card): 82.14 (#5/6)
Muse Wonder — Aggregate Rating 1911, #14
- Design Arena (Website): 1361.0 (#1/192)
- HiL-Bench: 79.67 (#1/28)
- STEMBench120 v1.0.0 (Closed Book) - Overall: 2.683 (#1/8)
- STEMBench120 v1.0.0 (Closed Book) - Biology: 2.615 (#1/8)
- STEMBench120 v1.0.0 (Closed Book) - Chemistry: 2.703 (#1/8)
- Vals AI CyberBench: 80.36 (#1/47)
- AA GDP.pdf - Legal - Criterion Pass Rate: 88.03 (#1/224)
- Harvey LAB-AA - Data Privacy & Cybersecurity: 98.12 (#1/27)
- Harvey LAB-AA - Healthcare & Life Sciences: 96.09 (#1/27)
- Harvey LAB-AA - Immigration: 95.39 (#1/27)
- ...and 176 more
DeepSeek V4.1 Flash (1017) — Aggregate Rating 1911, #15
- SelfBench - PostHog/posthog: 65.71 (#2/14)
- SelfBench - mupt-ai/self-bench: 71.05 (#3/7)
- SelfBench - vercel/next.js: 71.05 (#3/13)
- SelfBench - getsentry/sentry: 79.49 (#4/15)
- NL2Repo: 68.2 (#5/51)
- LLM2014 Logic 2026-10: 59.46 (#9/45)
- VoxelBench: 2052.0 (#10/59)
- Bug Hunt Bench - LMS: 21.2 (#11/117)
- Bug Hunt Bench: 39.4 (#12/117)
- Bug Hunt Bench - VS Code Extension: 18.2 (#12/117)
- ...and 20 more
GPT-6 Sol — Aggregate Rating 1910, #16
- Wiz Cyber Model Arena (Claude Code) - Websec.fr CTFs: 88.5 (#1/20)
- Wiz Cyber Model Arena (Claude Code) - Overall: 69.3 (#3/20)
- Wiz Cyber Model Arena (Claude Code) - Code Vulnerabilities: 50.8 (#3/20)
- Design Arena (SVG): 1328.0 (#5/129)
- AndroidWorld: 100.0 (#5/48)
- AutomationBench: 33.2 (#7/109)
- Wiz Cyber Model Arena (Claude Code) - Cloud Security: 59.4 (#7/20)
- Wiz Cyber Model Arena (Claude Code) - API Security: 78.4 (#8/20)
- Chess Puzzles (Epoch AI): 56.0 (#15/153)
- Epoch AI - Scicode: 57.64 (#26/220)
- ...and 1 more
Claude Fable 5 — Aggregate Rating 1907, #17
- OmniDocBench 1.5: 89.8 (#32/99)
GPT-5.5 Pro — Aggregate Rating 1899, #18
- Chess Puzzles (Epoch AI): 62.0 (#8/153)
GPT-5.6 Sol — Aggregate Rating 1889, #20
- SciLaws-Bench: 40.11 (#12/14)
- Epoch AI - Scicode: 57.75 (#23/220)
- OmniDocBench 1.5: 85.8 (#62/99)
- DuelLab Overall: 57.431351352674 (#62/208)
Gemini 3.8 Flash — Aggregate Rating 1873, #22
- SEAL - AudioMultiChallenge - Text Output: 60.4 (#1/21)
- AA-AnalystAgent: 46.25 (#15/50)
- AA-AnalystAgent - Pass@1: 61.25 (#17/50)
- AA-AnalystAgent - Pass@K: 72.5 (#28/50)
- Epoch AI - Scicode: 56.6 (#35/220)
Gemini 3.7 Flash — Aggregate Rating 1871, #23
- Epoch AI - Scicode: 59.84 (#11/220)
Muse Spark 1.3 — Aggregate Rating 1865, #25
- Epoch AI - Scicode: 59.72 (#12/220)
- Epoch AI - Frontierswe: 25.8 (#13/21)
- AA-AnalystAgent: 37.5 (#26/50)
- AA-AnalystAgent - Pass@1: 56.5 (#28/50)
- DuelLab Overall: 70.707351473382 (#29/208)
- AA-AnalystAgent - Pass@K: 71.25 (#30/50)
- Epoch AI - Apex Agents: 47.6 (#31/54)
Grok 4.6 — Aggregate Rating 1864, #26
- EnterpriseBench (CoreCraft Agents): 65.6 (#6/6)
- Epoch AI - Scicode: 56.48 (#36/220)
- Kaggle WWTP Engineering: 0.0 (#37/37)
Grok 4.7 — Aggregate Rating 1864, #27
- AA-AnalystAgent - Pass@1: 67.0 (#4/50)
- EnterpriseBench (CoreCraft Agents): 68.2 (#5/6)
- AA-AnalystAgent: 55.0 (#5/50)
- MineBench: 1840.0 (#14/75)
- AA-AnalystAgent - Pass@K: 72.5 (#25/50)
- Epoch AI - Scicode: 57.41 (#29/220)
Qwen 3.8 Max (0902) — Aggregate Rating 1854, #28
- DuelLab Overall: 70.09754911653 (#30/208)
Kimi K3 — Aggregate Rating 1852, #29
- SciLaws-Bench: 47.36 (#4/14)
- Epoch AI - Scicode: 59.49 (#13/220)
- HiL-Bench: 42.67 (#13/28)
- OmniDocBench 1.5: 91.1 (#17/99)
- HLE-Diamond - Knowledge: 24.0 (#19/28)
- HLE-Diamond: 28.3 (#22/28)
- HLE-Diamond - Reasoning: 32.6 (#23/28)
- DuelLab Overall: 44.992307148713 (#108/208)
GPT-5.5 — Aggregate Rating 1850, #31
- Chess Puzzles (Epoch AI): 63.0 (#7/153)
- HiL-Bench: 39.67 (#16/28)
- CyberGym: 81.8 (#16/52)
- OmniDocBench 1.5: 89.4 (#33/99)
- Epoch AI - Scicode: 56.13 (#40/220)
- DuelLab Overall: 37.096104955095 (#141/208)
Qwen 3.8 Max — Aggregate Rating 1849, #32
- Vals AI SAFE-Teen - Urgent Safety: 26.0 (#5/18)
- Vals AI SAFE-Teen - Health Advice: 35.0 (#5/18)
- Vals AI SAFE-Teen - Relationship Boundaries: 51.0 (#7/18)
DeepSeek V4.1 Flash — Aggregate Rating 1842, #35
- Wiz Cyber Model Arena (ADK React) - API Security: 80.9 (#4/18)
- AA-AnalystAgent - Pass@K: 80.0 (#5/50)
- Wiz Cyber Model Arena (Claude Code) - Cloud Security: 56.5 (#9/20)
- Wiz Cyber Model Arena (ADK React) - Cloud Security: 50.7 (#10/18)
- Wiz Cyber Model Arena (Claude Code) - API Security: 78.4 (#10/20)
- SciLaws-Bench: 40.59 (#11/14)
- SelfBench - getsentry/sentry: 57.69 (#13/15)
- Wiz Cyber Model Arena (ADK React) - Overall: 47.4 (#13/18)
- Wiz Cyber Model Arena (ADK React) - Code Vulnerabilities: 16.4 (#15/18)
- Wiz Cyber Model Arena (ADK React) - Websec.fr CTFs: 41.4 (#15/18)
- ...and 9 more
Claude Haiku 5.5 — Aggregate Rating 1840, #36
- AA-AnalystAgent - Pass@K: 83.75 (#2/50)
- AA-AnalystAgent - Pass@1: 68.25 (#3/50)
- DRACO: 81.5 (#6/17)
- AA-AnalystAgent: 53.75 (#7/50)
- Epoch AI - Frontierswe: 43.77 (#8/21)
- Epoch AI - Ebr Bench: 43.81 (#9/27)
- Epoch AI - Proofbench: 89.0 (#9/47)
- KernelBench Hub - CUDA: 34.02 (#11/21)
- BridgeBench Overall: 601.0 (#11/30)
- Design Arena (3D): 1348.0 (#14/166)
- ...and 58 more
Claude Opus 4.8 — Aggregate Rating 1840, #37
- OmniDocBench 1.5: 87.9 (#51/99)
Hy4 preview — Aggregate Rating 1838, #38
- Wolfram LLM Benchmarking Project: 66.8 (#54/598)
MiMo-V2.6-Pro — Aggregate Rating 1837, #39
- STEMBench120 v1.0.0 (Closed Book) - Biology: 2.533 (#4/8)
- STEMBench120 v1.0.0 (Closed Book) - Earth science: 2.623 (#4/8)
- AA-AnalystAgent - Pass@K: 80.0 (#4/50)
- STEMBench120 v1.0.0 (Closed Book) - Chemistry: 2.498 (#5/8)
- STEMBench120 v1.0.0 (Closed Book) - Overall: 2.172 (#8/8)
- STEMBench120 v1.0.0 (Closed Book) - Physics: 0.983 (#8/8)
- STEMBench120 v1.0.0 (Closed Book) - Astronomy: 2.223 (#8/8)
- STEMBench120 v1.0.0 (Closed Book) - Engineering: 2.172 (#8/8)
- Wiz Cyber Model Arena (ADK React) - API Security: 77.8 (#9/18)
- Wiz Cyber Model Arena (Claude Code) - API Security: 76.5 (#13/20)
- ...and 12 more
GLM-5.3 — Aggregate Rating 1835, #40
- HiL-Bench: 58.33 (#8/28)
- Epoch AI - Scicode: 59.03 (#15/220)
- Last Translation Benchmark: 8.562019758507136 (#19/50)
- AA-AnalystAgent: 38.75 (#24/50)
- DuelLab Overall: 73.216004317629 (#25/208)
- AA-AnalystAgent - Pass@1: 55.25 (#29/50)
- AA-AnalystAgent - Pass@K: 71.25 (#31/50)
GPT-5.6 Terra — Aggregate Rating 1834, #41
- Epoch AI - Scicode: 54.98 (#52/220)
- DuelLab Overall: 36.70263882537 (#142/208)
Step 5 Preview — Aggregate Rating 1833, #43
- Vals AI ReverseEngBench: 0.38 (#31/35)
- SvelteBench: 92.2 (#84/240)
- SpeechMap Compliance: 59.7 (#228/411)
GPT-5 Pro (2025-10-06) — Aggregate Rating 1826, #45
- Chess Puzzles (Epoch AI): 48.0 (#20/153)
Muse Spark 1.1 — Aggregate Rating 1824, #48
- Chess Puzzles (Epoch AI): 34.0 (#59/153)
Grok 4.5 — Aggregate Rating 1823, #49
- Kaggle WWTP Engineering: 0.0 (#36/37)
- Epoch AI - Scicode: 54.98 (#51/220)
Gemini 3.6 Flash — Aggregate Rating 1823, #51
- Epoch AI - Scicode: 53.36 (#77/220)
GPT-6 Luna — Aggregate Rating 1820, #52
- AA-AnalystAgent - Pass@K: 85.0 (#1/50)
- KNOWS - Accessibility Tree + Screenshot: 48.0 (#3/4)
- Wiz Cyber Model Arena (Claude Code) - Code Vulnerabilities: 43.4 (#6/20)
- AA-AnalystAgent - Pass@1: 65.75 (#7/50)
- Wiz Cyber Model Arena (Claude Code) - Overall: 60.5 (#9/20)
- Wiz Cyber Model Arena (Claude Code) - Websec.fr CTFs: 71.3 (#9/20)
- Wiz Cyber Model Arena (Claude Code) - Cloud Security: 52.2 (#11/20)
- Wiz Cyber Model Arena (Claude Code) - API Security: 75.3 (#16/20)
- AA-AnalystAgent: 43.75 (#21/50)
- AutomationBench: 20.7 (#27/109)
- ...and 2 more
Qwen3.8 2.4T A95B — Aggregate Rating 1814, #54
- DuelLab Overall: 68.574824916157 (#31/208)
Claude Sonnet 5 — Aggregate Rating 1813, #55
- BBQ Disambiguated Accuracy (Mythos 5.1 System Card): 72.36 (#6/6)
Muse Spark Open — Aggregate Rating 1812, #56
- Nejumi 4 - MT-Bench (Japanese) - Coding: 100.0 (#1/143)
- Nejumi 4 - MT-Bench (Japanese) - STEM: 100.0 (#1/143)
- Nejumi 4 - MT-Bench (Japanese) - Writing: 99.5 (#1/143)
- AA GDP.pdf - Legal - Criterion Pass Rate: 87.81 (#2/224)
- Nejumi 4 - GLP - Logical Reasoning: 97.0 (#3/138)
- Vals AI Finance Agent v2: 61.34 (#3/79)
- APEX v1: 73.9 (#4/7)
- Nejumi 4 - GLP - Coding: 71.21 (#4/138)
- Nejumi 4 - jaster (2-shot) - NIILC: 61.58 (#4/143)
- Nejumi 4 - HLE (Japanese) - Calibration Error: 41.0 (#4/143)
- ...and 182 more
Gemini 3.1 Pro (Preview) — Aggregate Rating 1810, #57
- OmniDocBench 1.5: 90.0 (#31/99)
Gemini 3.5 Flash — Aggregate Rating 1809, #58
- Epoch AI - Scicode: 53.94 (#68/220)
Muse Spark 1.2 — Aggregate Rating 1807, #61
- Epoch AI - Scicode: 57.41 (#28/220)
- DuelLab Overall: 59.63986214168 (#58/208)
- Chess Puzzles (Epoch AI): 27.0 (#75/153)
Claude Opus 4.6 — Aggregate Rating 1806, #62
- OmniDocBench 1.5: 86.6 (#61/99)
GLM-5.3 Flash — Aggregate Rating 1801, #63
- Wiz Cyber Model Arena (ADK React) - API Security: 78.4 (#8/18)
- AA-AnalystAgent - Pass@K: 78.75 (#8/50)
- Wiz Cyber Model Arena (Claude Code) - API Security: 77.2 (#11/20)
- Vals AI Terminal-Bench 4.0: 37.37 (#14/49)
- Wiz Cyber Model Arena (ADK React) - Websec.fr CTFs: 35.6 (#16/18)
- Wiz Cyber Model Arena (ADK React) - Cloud Security: 40.6 (#16/18)
- Wiz Cyber Model Arena (ADK React) - Overall: 41.8 (#17/18)
- Wiz Cyber Model Arena (Claude Code) - Code Vulnerabilities: 24.1 (#17/20)
- Wiz Cyber Model Arena (Claude Code) - Cloud Security: 42.0 (#17/20)
- Wiz Cyber Model Arena (ADK React) - Code Vulnerabilities: 12.4 (#18/18)
- ...and 12 more
MiMo-V2.6-Flash — Aggregate Rating 1800, #65
- Wolfram LLM Benchmarking Project: 50.4 (#214/598)
GPT-5.6 Luna — Aggregate Rating 1798, #68
- Epoch AI - Scicode: 53.59 (#72/220)
- DuelLab Overall: 26.223645310319 (#189/208)
GPT-5.3 Codex — Aggregate Rating 1796, #70
- HiL-Bench: 4.33 (#28/28)
GLM-5.2 — Aggregate Rating 1790, #71
- CyberGym: 90.84 (#5/52)
- Epoch AI - Scicode: 51.16 (#98/220)
Ling 3.1 Flash — Aggregate Rating 1790, #72
- OpenRouter Tau2-Bench Airline: 79.3 (#2/147)
- LLM2014 Logic 2026-10: 28.67 (#28/45)
- Wolfram LLM Benchmarking Project: 64.9 (#66/598)
- OpenRouter GPQA Diamond: 81.3 (#91/163)
- SpeechMap Compliance: 57.4 (#242/411)
Claude Opus 4.5 (20251101) — Aggregate Rating 1787, #73
- Chess Puzzles (Epoch AI): 14.0 (#114/153)
DeepSeek V4 Flash (0731) — Aggregate Rating 1784, #76
- CyberGym: 93.17 (#1/52)
GPT-5.4 — Aggregate Rating 1782, #79
- CyberGym: 79.0 (#17/52)
- HiL-Bench: 9.67 (#26/28)
- OmniDocBench 1.5: 85.5 (#69/99)
- DuelLab Overall: 47.949879446397 (#99/208)
Qwen 3.7 Plus — Aggregate Rating 1769, #89
- OmniDocBench 1.5: 91.4 (#15/99)
GPT-5.2 Codex — Aggregate Rating 1764, #93
- BenchGecko Score: 80.7 (#20/20)
GPT-5 — Aggregate Rating 1763, #95
- CyberGym: 60.2 (#26/52)
- DuelLab Overall: 41.830431578697 (#125/208)
Beam 1.0 — Aggregate Rating 1763, #97
- AutomationBench-AA - Support - Raw Objective Completion: 84.2 (#68/225)
- AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 78.29 (#72/224)
- AutomationBench-AA - Finance - Raw Objective Completion: 78.83 (#73/225)
- AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 67.37 (#76/224)
- AutomationBench-AA - Marketing - Raw Objective Completion: 77.7 (#76/225)
- AutomationBench-AA: 48.46 (#80/225)
- AutomationBench-AA - Operations - Raw Objective Completion: 85.03 (#80/225)
- AutomationBench-AA - Strict: 17.05 (#81/225)
- AutomationBench-AA - Sales - Raw Objective Completion: 66.34 (#81/225)
- AutomationBench-AA - HR - Raw Objective Completion: 65.55 (#84/225)
- ...and 47 more
Mistral Large 4 — Aggregate Rating 1763, #99
- AI Coding Daily (OpenCode) - React-TS Code Quality: 19.33 (#1/21)
- AI Coding Daily (OpenCode) - Laravel Code Quality: 18.13 (#2/21)
- AI Coding Daily (OpenCode) - Total: 56.86 (#3/21)
- SecIT Bench (Pydantic AI): 76.38 (#10/27)
- MathArena - ARXIV_FALSE August: 11.31 (#16/16)
- MathArena - ARXIV August: 24.56 (#16/16)
- MathArena - ARXIV_FALSE June: 22.22 (#20/30)
- MathArena - ARXIV June: 65.97 (#20/31)
- OTIS Mock AIME 2024-25: 98.33 (#24/304)
- Agent Arena - Bash Recovery: 1.78 (#26/51)
- ...and 23 more
Qwen 3.8 27B — Aggregate Rating 1762, #100
- SWE-bench Live Go (138 tasks, AMI Agent): 29.71 (#6/6)
- OmniDocBench 1.5: 91.1 (#18/99)
- DuelLab Overall: 40.694320328634 (#131/208)
- Epoch AI - Scicode: 46.64 (#133/220)
DeepSeek V4 Pro — Aggregate Rating 1761, #102
- HiL-Bench: 38.46 (#17/28)
- HLE-Diamond - Reasoning: 20.4 (#27/28)
- HLE-Diamond - Knowledge: 12.8 (#27/28)
Kimi K2.6 — Aggregate Rating 1758, #105
- HiL-Bench: 18.67 (#25/28)
K2 Horizon 375B A23B — Aggregate Rating 1756, #108
- MLCR-AA - Conciseness: 98.89 (#8/96)
- MLCR-AA - Accuracy: 67.98 (#57/96)
- MLCR-AA: 6.67 (#75/96)
- MLCR-AA - Completeness: 10.11 (#81/96)
Qwen 3.6 Plus — Aggregate Rating 1750, #118
- OmniDocBench 1.5: 91.2 (#16/99)
O3 — Aggregate Rating 1743, #123
- DuelLab Overall: 42.445279604729 (#123/208)
Inkling — Aggregate Rating 1742, #124
- Epoch AI - Scicode: 46.99 (#129/220)
- DuelLab Overall: 19.0345386711 (#203/208)
DeepSeek V4 Flash — Aggregate Rating 1735, #134
- HiL-Bench: 47.81 (#11/28)
MiniMax-M3 — Aggregate Rating 1734, #137
- HiL-Bench: 38.26 (#19/28)
- Last Translation Benchmark: 6.256860592755214 (#22/50)
Claude Sonnet 4.5 — Aggregate Rating 1732, #142
- Chess Puzzles (Epoch AI): 10.0 (#123/153)
Kimi K2.5 — Aggregate Rating 1732, #143
- CyberGym: 41.3 (#32/52)
Nemotron 3.5 Super VL — Aggregate Rating 1727, #152
- The Aggregate - Puzzle Portfolio Ladder: 92.6 (#5/20)
- The Aggregate - Inversion and Execution Ladder: 91.7 (#6/18)
- The Aggregate - Day-one Recall Check: 29.4 (#15/34)
- AI Coding Daily (OpenCode) - React-TS Code Quality: 15.17 (#19/21)
- AI Coding Daily (OpenCode) - Laravel Code Quality: 15.01 (#20/21)
- AI Coding Daily (OpenCode) - Total: 38.61 (#21/21)
- OpenRouter Tau2-Bench Airline: 75.3 (#38/147)
- OpenRouter GPQA Diamond: 88.0 (#45/163)
- AA-Omniscience Hallucination Rate: 35.85 (#68/578)
- AI Chess Leaderboard (Reasoning): 1066.0 (#81/344)
- ...and 64 more
Solar Open2 250B — Aggregate Rating 1721, #158
- AA GDPval: 970.77 (#172/296)
Qwen 3.6 27B — Aggregate Rating 1712, #168
- OmniDocBench 1.5: 89.4 (#34/99)
Grok 4.20 — Aggregate Rating 1710, #173
- HiL-Bench: 20.0 (#24/28)
Muse Glimmer 30B — Aggregate Rating 1706, #179
- AA-AnalystAgent: 20.0 (#37/50)
- AA-AnalystAgent - Pass@1: 37.25 (#42/50)
- AA-AnalystAgent - Pass@K: 58.75 (#42/50)
- OmniDocBench 1.5: 75.8 (#84/99)
- Tinybird AI SQL Benchmark - Exactness: 51.96 (#105/387)
- Tinybird AI SQL Benchmark - Success Rate: 100.0 (#186/394)
- Tinybird AI SQL Benchmark - First-Attempt Success Rate: 90.0 (#267/394)
Grok 4.3 — Aggregate Rating 1703, #188
- Epoch AI - Scicode: 48.26 (#121/220)
MiMo-V2.5 — Aggregate Rating 1700, #193
- AA GDP.pdf - Engineering - Criterion Pass Rate: 40.25 (#135/224)
- AA GDP.pdf: 4.0 (#171/224)
- AA GDP.pdf - Legal - Criterion Pass Rate: 52.86 (#176/224)
- AA GDP.pdf - STEM/Research - Criterion Pass Rate: 44.67 (#191/224)
- AA GDP.pdf - Insurance - Criterion Pass Rate: 44.72 (#192/224)
- AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 34.68 (#195/224)
- AA GDP.pdf - HR - Criterion Pass Rate: 38.18 (#202/224)
- AA GDP.pdf - Criterion Pass Rate: 37.42 (#204/224)
- AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 28.92 (#210/224)
- AA GDP.pdf - Healthcare - Criterion Pass Rate: 28.02 (#215/224)
- ...and 2 more
Gemma 4 31B — Aggregate Rating 1697, #199
- OmniDocBench 1.5: 0.13 (#99/99)
- DuelLab Overall: 38.266838188441 (#139/208)
GPT-5.4 Mini — Aggregate Rating 1692, #213
- Epoch AI - Scicode: 52.08 (#87/220)
- DuelLab Overall: 30.412700420227 (#174/208)
Solar Pro 4 — Aggregate Rating 1692, #215
- AA-Briefcase - XLSX - Rubric Pass Rate: 33.17 (#94/231)
- AA-Briefcase - PPTX - Rubric Pass Rate: 31.22 (#96/231)
- AA-Briefcase - Analytical Quality Elo: 1160.71 (#97/231)
- AA-Briefcase: 33.55 (#98/231)
- AA-Briefcase - PDF - Rubric Pass Rate: 33.62 (#105/231)
- AA-Briefcase - DOCX - Rubric Pass Rate: 32.54 (#107/231)
- AA-Briefcase - Presentation Elo: 1010.17 (#123/231)
- Tinybird AI SQL Benchmark - First-Attempt Success Rate: 96.0 (#141/394)
- Tinybird AI SQL Benchmark - Exactness: 47.48 (#218/387)
- Tinybird AI SQL Benchmark - Success Rate: 98.0 (#263/394)
Solar Mini 4 — Aggregate Rating 1688, #224
- Design Arena (Game Dev): 1138.0 (#109/180)
Step 3.7 Flash — Aggregate Rating 1683, #237
- Gert Labs Rankings: 23.67 (#100/100)
- DuelLab Overall: 30.675572146764 (#171/208)
GPT-4.1 — Aggregate Rating 1679, #245
- DuelLab Overall: 41.868615462324 (#124/208)
llm-jp-4.1-33B — Aggregate Rating 1678, #251
- Open Japanese LLM v2 - JBLiMP: 85.8 (#1/12)
- Open Japanese LLM v2 - M-IFEval-Ja: 53.49 (#2/26)
- Open Japanese LLM v2 - JCulture-MCQ: 55.15 (#2/10)
- Open Japanese LLM v2 - JCoLA In-Domain: 78.73 (#3/12)
- Nejumi 4 - jaster (0-shot) - JSICK: 85.0 (#3/143)
- Open Japanese LLM v2 - JamC-QA: 55.61 (#4/24)
- Open Japanese LLM v2 - StructEval: 80.27 (#4/12)
- Open Japanese LLM v2 - JCoLA Out-of-Domain: 79.51 (#5/12)
- Nejumi 4 - jaster (0-shot) - CommonsenseMoralJA: 97.0 (#7/143)
- Nejumi 4 - jaster (0-shot) - MAWPS: 98.0 (#7/143)
- ...and 64 more
Claude Haiku 4.5 (20251001) — Aggregate Rating 1674, #259
- Chess Puzzles (Epoch AI): 8.0 (#128/153)
GPT-5.4 Nano (2026-03-17) — Aggregate Rating 1670, #266
- Chess Puzzles (Epoch AI): 45.0 (#28/153)
O3 Mini — Aggregate Rating 1669, #270
- Epoch AI - Scicode: 42.82 (#151/220)
Qwen 3 235B A22B 2507 Instruct — Aggregate Rating 1656, #298
- ContextEcho: 40.0 (#11/23)
MiniMax-M2.5 — Aggregate Rating 1654, #300
- HiL-Bench: 6.33 (#27/28)
GPT-5.4 Nano — Aggregate Rating 1639, #329
- Epoch AI - Scicode: 47.22 (#126/220)
- DuelLab Overall: 27.172941650905 (#185/208)
Hermes 4 405B — Aggregate Rating 1639, #330
- EQ-Bench Creative Writing v3: 1270.7 (#44/106)
GPT-OSS-120B — Aggregate Rating 1636, #337
- Last Translation Benchmark: 2.305159165751921 (#30/50)
- Epoch AI - Scicode: 34.03 (#196/220)
Gemma 4 26B A4B — Aggregate Rating 1631, #351
- OmniDocBench 1.5: 0.15 (#98/99)
llm-jp-4.1-32B-a3B — Aggregate Rating 1629, #355
- Open Japanese LLM v2 - M-IFEval-Ja: 56.4 (#1/26)
- Nejumi 4 - jaster (2-shot) - MAWPS: 99.0 (#1/143)
- Open Japanese LLM v2 - JCulture-MCQ: 52.24 (#3/10)
- Open Japanese LLM v2 - JFinQA: 65.6 (#3/11)
- Open Japanese LLM v2 - JCoLA Out-of-Domain: 81.35 (#3/12)
- Open Japanese LLM v2 - JCoLA In-Domain: 76.88 (#5/12)
- Open Japanese LLM v2 - JBLiMP: 79.15 (#8/12)
- Nejumi 4 - jaster (0-shot) - MAWPS: 98.0 (#8/143)
- Open Japanese LLM v2 - JamC-QA: 52.75 (#9/24)
- Open Japanese LLM v2 - StructEval: 77.68 (#9/12)
- ...and 64 more
Qwen 3 Next 80B A3B Instruct — Aggregate Rating 1627, #364
- ContextEcho: 30.0 (#7/23)
llm-jp-4.1-8B — Aggregate Rating 1614, #399
- Open Japanese LLM v2 - JCoLA Out-of-Domain: 82.87 (#1/12)
- Open Japanese LLM v2 - JCoLA In-Domain: 79.19 (#2/12)
- Open Japanese LLM v2 - M-IFEval-Ja: 50.58 (#3/26)
- Open Japanese LLM v2 - JCulture-MCQ: 48.7 (#5/10)
- Open Japanese LLM v2 - StructEval: 79.03 (#5/12)
- Open Japanese LLM v2 - JBLiMP: 82.48 (#6/12)
- Open Japanese LLM v2 - JFinQA: 60.9 (#7/11)
- Nejumi 4 - BFCL - Irrelevance Detection: 93.33 (#9/143)
- Nejumi 4 - jaster (2-shot) - MAWPS: 98.0 (#13/143)
- Open Japanese LLM v2 - JamC-QA: 48.25 (#15/24)
- ...and 64 more
Mercury 2 — Aggregate Rating 1602, #434
- AA GDP.pdf: 3.8 (#176/224)
- AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 52.67 (#177/224)
- AA GDP.pdf - STEM/Research - Criterion Pass Rate: 48.34 (#177/224)
- AA GDP.pdf - Insurance - Criterion Pass Rate: 49.2 (#180/224)
- AA GDP.pdf - Legal - Criterion Pass Rate: 47.28 (#187/224)
- AA GDP.pdf - Real Estate - Criterion Pass Rate: 44.92 (#187/224)
- AA GDP.pdf - Criterion Pass Rate: 43.14 (#188/224)
- AA GDP.pdf - Construction - Criterion Pass Rate: 46.17 (#189/224)
- AA GDP.pdf - Healthcare - Criterion Pass Rate: 45.76 (#189/224)
- AA GDP.pdf - HR - Criterion Pass Rate: 42.45 (#195/224)
- ...and 2 more
Qwen 3 32B — Aggregate Rating 1600, #442
- AA GDP.pdf - HR - Criterion Pass Rate: 49.98 (#178/224)
- AA GDP.pdf: 1.6 (#196/224)
- AA GDP.pdf - STEM/Research - Criterion Pass Rate: 40.28 (#199/224)
- AA GDP.pdf - Healthcare - Criterion Pass Rate: 40.49 (#200/224)
- AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 34.16 (#200/224)
- AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 37.79 (#201/224)
- AA GDP.pdf - Legal - Criterion Pass Rate: 36.28 (#201/224)
- AA GDP.pdf - Real Estate - Criterion Pass Rate: 40.76 (#202/224)
- AA GDP.pdf - Criterion Pass Rate: 38.26 (#203/224)
- AA GDP.pdf - Insurance - Criterion Pass Rate: 41.35 (#203/224)
- ...and 2 more
Llama 4 Maverick — Aggregate Rating 1593, #465
- AA GDP.pdf: 0.8 (#212/224)
- AA GDP.pdf - Engineering - Criterion Pass Rate: 9.75 (#220/224)
- AA GDP.pdf - Criterion Pass Rate: 3.97 (#223/224)
- AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 0.0 (#223/224)
- AA GDP.pdf - Healthcare - Criterion Pass Rate: 3.86 (#223/224)
- AA GDP.pdf - HR - Criterion Pass Rate: 0.0 (#223/224)
- AA GDP.pdf - Insurance - Criterion Pass Rate: 1.67 (#223/224)
- AA GDP.pdf - Legal - Criterion Pass Rate: 0.0 (#223/224)
- AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 0.92 (#223/224)
- AA GDP.pdf - STEM/Research - Criterion Pass Rate: 0.44 (#223/224)
- ...and 2 more
Nemotron 3.5 Lightning — Aggregate Rating 1585, #488
- Tinybird AI SQL Benchmark - Exactness: 49.35 (#177/387)
- Tinybird AI SQL Benchmark - First-Attempt Success Rate: 94.0 (#186/394)
- Tinybird AI SQL Benchmark - Success Rate: 98.0 (#253/394)
Mistral Small 4 — Aggregate Rating 1579, #516
- Last Translation Benchmark: 1.3172338090010975 (#40/50)
Qwen 3 14B — Aggregate Rating 1572, #535
- AA GDP.pdf - Real Estate - Criterion Pass Rate: 41.42 (#200/224)
- AA GDP.pdf: 1.2 (#202/224)
- AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 37.53 (#202/224)
- AA GDP.pdf - Healthcare - Criterion Pass Rate: 38.0 (#202/224)
- AA GDP.pdf - STEM/Research - Criterion Pass Rate: 39.37 (#203/224)
- AA GDP.pdf - Engineering - Criterion Pass Rate: 24.45 (#204/224)
- AA GDP.pdf - HR - Criterion Pass Rate: 37.47 (#204/224)
- AA GDP.pdf - Criterion Pass Rate: 35.63 (#209/224)
- AA GDP.pdf - Legal - Criterion Pass Rate: 28.87 (#210/224)
- AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 30.59 (#210/224)
- ...and 2 more
Mistral Small 3.2 — Aggregate Rating 1554, #602
- AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 21.62 (#216/224)
- AA GDP.pdf - STEM/Research - Criterion Pass Rate: 6.48 (#220/224)
- AA GDP.pdf - Criterion Pass Rate: 12.61 (#221/224)
- AA GDP.pdf - Healthcare - Criterion Pass Rate: 18.87 (#221/224)
- AA GDP.pdf - HR - Criterion Pass Rate: 5.78 (#221/224)
- AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 7.98 (#221/224)
- AA GDP.pdf - Real Estate - Criterion Pass Rate: 22.42 (#221/224)
- AA GDP.pdf: 0.0 (#222/224)
- AA GDP.pdf - Construction - Criterion Pass Rate: 26.85 (#222/224)
- AA GDP.pdf - Insurance - Criterion Pass Rate: 8.48 (#222/224)
- ...and 2 more
Llama 4 Scout — Aggregate Rating 1536, #652
- AA GDP.pdf - Engineering - Criterion Pass Rate: 8.89 (#221/224)
- AA GDP.pdf: 0.0 (#223/224)
- AA GDP.pdf - Construction - Criterion Pass Rate: 21.3 (#223/224)
- AA GDP.pdf - Real Estate - Criterion Pass Rate: 2.5 (#223/224)
- AA GDP.pdf - Criterion Pass Rate: 3.68 (#224/224)
- AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 0.0 (#224/224)
- AA GDP.pdf - Healthcare - Criterion Pass Rate: 1.82 (#224/224)
- AA GDP.pdf - HR - Criterion Pass Rate: 0.0 (#224/224)
- AA GDP.pdf - Insurance - Criterion Pass Rate: 1.39 (#224/224)
- AA GDP.pdf - Legal - Criterion Pass Rate: 0.0 (#224/224)
- ...and 2 more
Qwen 3 8B — Aggregate Rating 1532, #671
- AA GDP.pdf: 1.4 (#198/224)
- AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 30.58 (#208/224)
- AA GDP.pdf - STEM/Research - Criterion Pass Rate: 38.14 (#208/224)
- AA GDP.pdf - Healthcare - Criterion Pass Rate: 29.6 (#211/224)
- AA GDP.pdf - Legal - Criterion Pass Rate: 24.64 (#211/224)
- AA GDP.pdf - Real Estate - Criterion Pass Rate: 34.02 (#213/224)
- AA GDP.pdf - Criterion Pass Rate: 30.37 (#214/224)
- AA GDP.pdf - Engineering - Criterion Pass Rate: 21.97 (#214/224)
- AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 28.81 (#214/224)
- AA GDP.pdf - HR - Criterion Pass Rate: 24.41 (#215/224)
- ...and 2 more
GPT-4.1 Nano — Aggregate Rating 1515, #731
- Chess Puzzles (Epoch AI): 0.0 (#152/153)
Gemma 4 E4B — Aggregate Rating 1504, #766
- OmniDocBench 1.5: 0.18 (#96/99)
LFM2.5-2.6B — Aggregate Rating 1492, #804
- Tinybird AI SQL Benchmark - Success Rate: 90.0 (#342/394)
- Tinybird AI SQL Benchmark - First-Attempt Success Rate: 60.0 (#361/394)
- Tinybird AI SQL Benchmark - Exactness: 9.34 (#374/387)
Gemma 3 4B — Aggregate Rating 1432, #1082
- OpenRouter GPQA Diamond: 27.1 (#161/163)
Gemma 4 E2B — Aggregate Rating 1428, #1103
- OmniDocBench 1.5: 0.29 (#95/99)

Top-10 New Scores (373)

Claude Fable 5.1 on Anthropic Election Integrity Multi-Turn - Appropriate Response Rate (API): 90.0 (#4)
Claude Fable 5.1 on BBQ Disambiguated Accuracy (Mythos 5.1 System Card): 89.92 (#1)
Claude Opus 5.5 on Agent Arena: 15.53 (#1)
Claude Opus 5.5 on Agent Arena - Bash Recovery: 15.01 (#2)
Claude Opus 5.5 on Agent Arena - Confirmed Success: 17.29 (#1)
Claude Opus 5.5 on Agent Arena - Praise vs Complaint: 36.78 (#3)
Claude Opus 5.5 on Agent Arena - Steerability: 8.2 (#8)
Claude Opus 5.5 on Agent Arena - Tool Hallucination: -0.37 (#15)
Claude Opus 5.5 on Anthropic Child Safety Multi-Turn - Appropriate Response Rate (API): 84.0 (#8)
Claude Opus 5.5 on Anthropic Child Safety Multi-Turn - Appropriate Response Rate (Claude.ai): 99.0 (#2)
Claude Opus 5.5 on Anthropic Claude Code Malicious Requests - Refusal Rate (Mythos 5.1 System Card): 79.8 (#5)
Claude Opus 5.5 on Anthropic Disordered Eating Single-Turn Risk Requests - Harmless Rate (API): 95.64 (#9)
Claude Opus 5.5 on Anthropic Disordered Eating Single-Turn Risk Requests - Harmless Rate (Claude.ai): 99.26 (#6)
Claude Opus 5.5 on Anthropic Election Integrity Multi-Turn - Appropriate Response Rate (API): 87.0 (#8)
Claude Opus 5.5 on Anthropic Election Integrity Multi-Turn - Appropriate Response Rate (Claude.ai): 85.0 (#6)
Claude Opus 5.5 on Anthropic Malicious Computer Use - Refusal Rate: 79.46 (#8)
Claude Opus 5.5 on Anthropic Single-Turn Benign Requests - Over-Refusal Rate (API): 0.03 (#4)
Claude Opus 5.5 on Anthropic Single-Turn Benign Requests - Over-Refusal Rate (Claude.ai): 0.38 (#2)
Claude Opus 5.5 on Anthropic Single-Turn Harmful Requests - Harmless Rate (API): 94.5 (#9)
Claude Opus 5.5 on Anthropic Single-Turn Harmful Requests - Harmless Rate (Claude.ai): 99.51 (#3)
Claude Opus 5.5 on Anthropic Suicide and Self-Harm Multi-Turn - Appropriate Response Rate (API): 66.0 (#4)
Claude Opus 5.5 on Anthropic Suicide and Self-Harm Multi-Turn - Appropriate Response Rate (Claude.ai): 94.0 (#2)
Claude Opus 5.5 on Anthropic Suicide and Self-Harm Single-Turn Risk Requests - Harmless Rate (API): 99.16 (#8)
Claude Opus 5.5 on BBQ Disambiguated Accuracy (Mythos 5.1 System Card): 89.65 (#3)
Claude Opus 5.5 on Chess Puzzles (Epoch AI): 67.0 (#5)
Claude Opus 5.5 on Conceptual Reasoning Index: 76.07 (#3)
Claude Opus 5.5 on Conceptual Reasoning Index - Argument Evaluation (LMCA): 66.18 (#4)
Claude Opus 5.5 on Conceptual Reasoning Index - Consistency (ACCoRD): 88.42 (#1)
Claude Opus 5.5 on Conceptual Reasoning Index - Decision Theory (DTBench): 95.55 (#10)
Claude Opus 5.5 on DRACO: 87.4 (#2)
Claude Opus 5.5 on HiL-Bench: 59.0 (#7)
Claude Opus 5.5 on ParseBench: 79.85 (#1)
Claude Opus 5.5 on ParseBench - Charts: 70.89 (#1)
Claude Opus 5.5 on ParseBench - Content Faithfulness: 91.72 (#2)
Claude Opus 5.5 on ParseBench - Semantic Formatting: 77.04 (#4)
Claude Opus 5.5 on ParseBench - Tables: 94.25 (#1)
Claude Opus 5.5 on ParseBench - Visual Grounding: 65.33 (#35)
Claude Opus 5.5 on STEMBench120 v1.0.0 (Closed Book) - Astronomy: 2.759 (#2)
Claude Opus 5.5 on STEMBench120 v1.0.0 (Closed Book) - Biology: 1.865 (#8)
Claude Opus 5.5 on STEMBench120 v1.0.0 (Closed Book) - Chemistry: 2.666 (#2)
Claude Opus 5.5 on STEMBench120 v1.0.0 (Closed Book) - Overall: 2.621 (#2)
Claude Opus 5.5 on SWE-bench Pro (Opus 5 & Mythos 5.1 System Cards): 89.9 (#1)
Claude Opus 5.5 on SciLaws-Bench: 48.87 (#3)
Claude Opus 5.5 on Toolathlon Verified Pass@3 (Anthropic Internal Harness): 82.4 (#5)
Claude Opus 5.5 on Toolathlon Verified Pass^3 (Anthropic Internal Harness): 72.2 (#4)
GPT-6 Astra on AutomationBench: 41.4 (#3)
GPT-6 Astra on Design Arena (3D): 1509.0 (#1)
GPT-6 Astra on Epoch AI - Scicode: 56.48 (#37)
GPT-6 Astra on HiL-Bench: 65.32 (#3)
GPT-6 Astra on STEMBench120 v1.0.0 (Tools) - Astronomy: 2.723 (#1)
GPT-6 Astra on STEMBench120 v1.0.0 (Tools) - Biology: 2.709 (#1)
GPT-6 Astra on STEMBench120 v1.0.0 (Tools) - Chemistry: 2.777 (#1)
GPT-6 Astra on STEMBench120 v1.0.0 (Tools) - Earth science: 2.726 (#1)
GPT-6 Astra on STEMBench120 v1.0.0 (Tools) - Engineering: 2.894 (#1)
GPT-6 Astra on STEMBench120 v1.0.0 (Tools) - Overall: 2.809 (#1)
GPT-6 Astra on STEMBench120 v1.0.0 (Tools) - Physics: 3.025 (#1)
GPT-6 Astra Pro on Chess Puzzles (Epoch AI): 76.0 (#1)
GPT-6.1 Sol on Conceptual Reasoning Index: 72.68 (#22)
GPT-6.1 Sol on Conceptual Reasoning Index - Argument Evaluation (LMCA): 62.78 (#27)
GPT-6.1 Sol on Conceptual Reasoning Index - Consistency (ACCoRD): 80.87 (#26)
GPT-6.1 Sol on Conceptual Reasoning Index - Decision Theory (DTBench): 94.22 (#26)
GPT-6.1 Sol on Design Arena (3D): 1458.0 (#5)
GPT-6.1 Sol on Design Arena (UI Components): 1325.0 (#16)
GPT-6.1 Sol on Epoch AI - Scicode: 55.79 (#43)
GPT-6.1 Sol on GDP.pdf: 35.0 (#1)
GPT-6.1 Sol on HLE-Diamond (With Tools): 69.1 (#4)
GPT-6.1 Sol on HiL-Bench: 62.33 (#4)
GPT-6.1 Sol on Kaggle FACTS (Google): 75.29 (#2)
GPT-6.1 Sol on Kaggle FACTS Grounding: 82.42 (#2)
GPT-6.1 Sol on Kaggle FACTS Multimodal: 48.46 (#3)
GPT-6.1 Sol on Kaggle FACTS Parametric: 82.11 (#2)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Arabic: 69.5 (#40)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Bengali: 70.0 (#40)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Burmese: 74.5 (#38)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Chinese: 82.75 (#34)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Culturally Agnostic: 78.31 (#36)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Culturally Sensitive: 79.47 (#33)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - English: 91.0 (#23)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - French: 76.5 (#35)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - German: 85.25 (#31)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Hindi: 65.0 (#40)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Indonesian: 77.0 (#35)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Italian: 81.75 (#34)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Japanese: 87.5 (#28)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Korean: 90.25 (#23)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Portuguese: 88.0 (#29)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Spanish: 82.0 (#33)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Swahili: 77.0 (#35)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Yoruba: 64.25 (#43)
GPT-6.1 Sol on Kaggle Open Benchmarks - SciCode Subproblems: 37.8 (#16)
Gemini 4 Argon on AA-AnalystAgent: 48.75 (#12)
Gemini 4 Argon on AA-AnalystAgent - Pass@1: 57.75 (#25)
Gemini 4 Argon on AA-AnalystAgent - Pass@K: 66.25 (#40)
Gemini 4 Argon on AI Chess Leaderboard (Continuation): 1641.0 (#9)
Gemini 4 Argon on AI Chess Leaderboard (Reasoning): 1632.0 (#16)
Gemini 4 Argon on AIMultiple - FinanceReasoning (Hard): 90.76 (#3)
Gemini 4 Argon on AIMultiple - Text-to-SQL (Adjudicated): 81.4 (#8)
Gemini 4 Argon on AIMultiple - Text-to-SQL (Strict Execution Match): 64.3 (#2)
Gemini 4 Argon on ARC-AGI-1: 98.5 (#9)
Gemini 4 Argon on ARC-AGI-2: 91.25 (#13)
Gemini 4 Argon on ARC-AGI-3: 49.79 (#5)
Gemini 4 Argon on Arabic Broad Leaderboard: 9.128 (#10)
Gemini 4 Argon on Arabic Broad Leaderboard - General Knowledge: 9.175 (#13)
Gemini 4 Argon on Arabic Broad Leaderboard - MMLU: 9.752 (#12)
Gemini 4 Argon on Arabic Broad Leaderboard - RAG QA: 8.61 (#19)
Gemini 4 Argon on Arabic Broad Leaderboard - Reasoning & Math: 9.465 (#10)
Gemini 4 Argon on Arabic Broad Leaderboard - Translation (incl Dialects): 8.0 (#8)
Gemini 4 Argon on Arabic Broad Leaderboard - Trust & Safety: 9.0 (#43)
Gemini 4 Argon on Arabic Broad Leaderboard - Writing (incl Dialects): 7.864 (#17)
Gemini 4 Argon on Argo-Bench: 54.99 (#3)
Gemini 4 Argon on Argo-Bench - Solved: 35.71 (#1)
Gemini 4 Argon on BenchGecko Score: 88.6 (#9)
Gemini 4 Argon on BridgeBench Overall: 623.0 (#8)
Gemini 4 Argon on Bug Hunt Bench: 40.7 (#11)
Gemini 4 Argon on Bug Hunt Bench - LMS: 27.7 (#2)
Gemini 4 Argon on Bug Hunt Bench - VS Code Extension: 13.7 (#30)
Gemini 4 Argon on Chess Bench LLM: 2022.0 (#4)
Gemini 4 Argon on Chess Puzzles (Epoch AI): 59.0 (#11)
Gemini 4 Argon on ChessBench: 24.41 (#3)
Gemini 4 Argon on ChessBench - Accuracy: 80.1 (#2)
Gemini 4 Argon on ChessBench - Coherence: 85.8 (#3)
Gemini 4 Argon on ChessBench - Elo: 1356.0 (#3)
Gemini 4 Argon on ChessBench GitHub - Elo: 2430.0 (#12)
Gemini 4 Argon on Conceptual Reasoning Index: 72.41 (#24)
Gemini 4 Argon on Conceptual Reasoning Index - Argument Evaluation (LMCA): 63.69 (#19)
Gemini 4 Argon on Conceptual Reasoning Index - Consistency (ACCoRD): 77.3 (#48)
Gemini 4 Argon on Conceptual Reasoning Index - Decision Theory (DTBench): 96.0 (#8)
Gemini 4 Argon on Creative Writing (Lechmazur): 0.8 (#19)
Gemini 4 Argon on CursorBench 4.0: 52.9 (#6)
Gemini 4 Argon on Deep20Bench - Edition 1.1: 13.666666666667 (#2)
Gemini 4 Argon on Design Arena (3D): 1392.0 (#9)
Gemini 4 Argon on Design Arena (Data Viz): 1347.0 (#4)
Gemini 4 Argon on Design Arena (Game Dev): 1370.0 (#9)
Gemini 4 Argon on Design Arena (UI Components): 1338.0 (#12)
Gemini 4 Argon on Design Arena (Website): 1314.0 (#11)
Gemini 4 Argon on EKI Estonian LLM - Book Recommendations: 63.47 (#21)
Gemini 4 Argon on EKI Estonian LLM - Idioms: 70.0 (#8)
Gemini 4 Argon on EKI Estonian LLM - Overall: 72.48 (#3)
Gemini 4 Argon on EKI Estonian LLM - Propaganda Resistance: 79.52 (#51)
Gemini 4 Argon on EKI Estonian LLM - Terminology: 57.54 (#4)
Gemini 4 Argon on EKI Estonian LLM - Trivia Hard: 50.62 (#2)
Gemini 4 Argon on EnigmaForge: 81.2 (#2)
Gemini 4 Argon on EnigmaForge - Fact F1: 98.3 (#2)
Gemini 4 Argon on EnigmaForge - Intuition: 72.9 (#4)
Gemini 4 Argon on Epoch AI - Cursorbench: 52.9 (#6)
Gemini 4 Argon on Epoch AI - ECI: 164.56 (#5)
Gemini 4 Argon on Epoch AI - Ebr Bench: 64.76 (#3)
Gemini 4 Argon on Epoch AI - Frontiermath Tier 4 V2: 78.05 (#15)
Gemini 4 Argon on Epoch AI - Frontiermath Tiers 1 3 V2: 85.96 (#10)
Gemini 4 Argon on Epoch AI - Furniture Assembly: 72.5 (#5)
Gemini 4 Argon on Epoch AI - GPQA Diamond: 96.4 (#1)
Gemini 4 Argon on Epoch AI - Mystery Game Puzzles: 64.0 (#5)
Gemini 4 Argon on ErdosBench: 3.04 (#6)
Gemini 4 Argon on FrontierMath - Tier 4 (v2): 78.05 (#15)
Gemini 4 Argon on FrontierMath - Tiers 1-3 (v2): 85.96 (#10)
Gemini 4 Argon on Gert Labs Rankings: 69.63 (#7)
Gemini 4 Argon on HLE-Diamond: 56.9 (#5)
Gemini 4 Argon on HLE-Diamond - Knowledge: 52.0 (#2)
Gemini 4 Argon on HLE-Diamond - Reasoning: 61.8 (#8)
Gemini 4 Argon on HiL-Bench: 62.0 (#5)
Gemini 4 Argon on Humanity's Sixth Sense: 55.4 (#1)
Gemini 4 Argon on Icelandic LLM - Case Government: 98.75 (#2)
Gemini 4 Argon on Icelandic LLM - Flora & Fauna: 49.95 (#7)
Gemini 4 Argon on Icelandic LLM - GEC: 80.73 (#5)
Gemini 4 Argon on Icelandic LLM - Generative Quality: 74.91 (#15)
Gemini 4 Argon on Icelandic LLM - Idiom Meanings: 81.67 (#13)
Gemini 4 Argon on Icelandic LLM - Song Lyrics: 21.83 (#2)
Gemini 4 Argon on Icelandic LLM - Textile Terms: 70.4 (#7)
Gemini 4 Argon on Icelandic LLM - WikiQA: 71.26 (#3)
Gemini 4 Argon on Icelandic LLM Leaderboard - Average: 70.52 (#2)
Gemini 4 Argon on Kaggle ExtractBench: 89.81 (#4)
Gemini 4 Argon on Kaggle FACTS Grounding: 78.63 (#6)
Gemini 4 Argon on Kaggle FACTS Parametric: 71.55 (#11)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Arabic: 94.75 (#6)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Burmese: 94.0 (#5)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Chinese: 95.0 (#8)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Culturally Agnostic: 95.91 (#3)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Culturally Sensitive: 94.84 (#4)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - English: 96.0 (#7)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - French: 95.25 (#7)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - German: 96.75 (#2)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Hindi: 95.5 (#2)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Indonesian: 94.75 (#9)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Italian: 95.5 (#7)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Korean: 94.5 (#8)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Portuguese: 96.0 (#5)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Spanish: 96.5 (#4)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Yoruba: 93.5 (#5)
Gemini 4 Argon on Kaggle Open Benchmarks - AIME 2025: 100.0 (#10)
Gemini 4 Argon on Kaggle Open Benchmarks - MATH-500: 99.0 (#6)
Gemini 4 Argon on Kaggle Open Benchmarks - SciCode Subproblems: 45.02 (#2)
Gemini 4 Argon on Kaggle SimpleQA: 71.18 (#3)
Gemini 4 Argon on Kaggle SimpleQA Verified: 75.6 (#3)
Gemini 4 Argon on Kaggle VideoQA: 87.0 (#3)
Gemini 4 Argon on KernelBench Hub - CUDA: 42.2 (#9)
Gemini 4 Argon on KernelBench Hub - Mega: 27.03 (#3)
Gemini 4 Argon on LLM2014 Logic 2026-10: 69.35 (#5)
Gemini 4 Argon on Last Translation Benchmark: 40.944017563117455 (#5)
Gemini 4 Argon on MathArena - ARXIV August: 82.46 (#7)
Gemini 4 Argon on MathArena - ARXIV_FALSE August: 54.17 (#11)
Gemini 4 Argon on MineBench: 2250.0 (#2)
Gemini 4 Argon on Multi-turn Debate (Lechmazur): 1672.7 (#6)
Gemini 4 Argon on NYT Connections Extended: 98.6 (#1)
Gemini 4 Argon on Next.js Agent Evals (OpenCode) - Success Rate with AGENTS.md: 97.0 (#2)
Gemini 4 Argon on NonoBench: 86.666666666667 (#11)
Gemini 4 Argon on OTIS Mock AIME 2024-25: 100.0 (#1)
Gemini 4 Argon on OpenRouter Tau2-Bench Airline: 78.7 (#5)
Gemini 4 Argon on PLCC - Art & Entertainment: 95.0 (#5)
Gemini 4 Argon on PLCC - Culture & Tradition: 98.0 (#11)
Gemini 4 Argon on PLCC - Geography: 99.0 (#10)
Gemini 4 Argon on PLCC - Grammar: 96.0 (#5)
Gemini 4 Argon on PLCC - History: 98.0 (#5)
Gemini 4 Argon on PUMA Polish Multimodal - Contemporary Life: 84.0 (#2)
Gemini 4 Argon on PUMA Polish Multimodal - Geography and Environment: 89.0 (#3)
Gemini 4 Argon on PUMA Polish Multimodal - History and Culture: 83.0 (#3)
Gemini 4 Argon on PUMA Polish Multimodal - Structured Extraction: 69.0 (#7)
Gemini 4 Argon on ReLE - Agents - BFCL-V3: 72.5 (#8)
Gemini 4 Argon on ReLE - Agents and Tool Use: 73.7 (#2)
Gemini 4 Argon on ReLE - Coding: 60.5 (#47)
Gemini 4 Argon on ReLE - Education - Gaokao: 68.7 (#2)
Gemini 4 Argon on ReLE - Education - Middle School Subjects: 58.0 (#9)
Gemini 4 Argon on ReLE - Education - Primary School Subjects: 71.3 (#2)
Gemini 4 Argon on ReLE - Finance: 90.0 (#3)
Gemini 4 Argon on ReLE - Law - Lawyer Qualification Exam: 77.3 (#47)
Gemini 4 Argon on ReLE - Law and Civil Service: 80.7 (#65)
Gemini 4 Argon on ReLE - Medicine - Basic Medical Knowledge: 88.5 (#4)
Gemini 4 Argon on ReLE - Medicine - Physician Exams: 88.3 (#17)
Gemini 4 Argon on ReLE - Medicine and Mental Health: 88.5 (#21)
Gemini 4 Argon on ReLE - Overall: 77.3 (#2)
Gemini 4 Argon on ReLE - Reasoning - BBH: 87.5 (#4)
Gemini 4 Argon on ReLE - Reasoning and Mathematics: 85.4 (#10)
Gemini 4 Argon on RuneBench: 6.906727309216 (#3)
Gemini 4 Argon on SEAL - AudioMultiChallenge - Text Output: 58.41 (#3)
Gemini 4 Argon on SEAL - Professional Reasoning Benchmark - Finance: 49.9 (#11)
Gemini 4 Argon on SEAL - Professional Reasoning Benchmark - Legal: 49.41 (#11)
Gemini 4 Argon on STEMBench120 v1.0.0 (Closed Book) - Biology: 2.348 (#6)
Gemini 4 Argon on STEMBench120 v1.0.0 (Closed Book) - Chemistry: 2.336 (#8)
Gemini 4 Argon on STEMBench120 v1.0.0 (Closed Book) - Earth science: 2.647 (#3)
Gemini 4 Argon on STEMBench120 v1.0.0 (Closed Book) - Engineering: 2.471 (#6)
Gemini 4 Argon on STEMBench120 v1.0.0 (Closed Book) - Overall: 2.444 (#6)
Gemini 4 Argon on STEMBench120 v1.0.0 (Closed Book) - Physics: 2.083 (#6)
Gemini 4 Argon on SimpleQA Verified: 76.0 (#1)
Gemini 4 Argon on SpeechMap Compliance: 87.0 (#55)
Gemini 4 Argon on SvelteBench: 100.0 (#10)
Gemini 4 Argon on Tinybird AI SQL Benchmark - Exactness: 56.91 (#26)
Gemini 4 Argon on Tinybird AI SQL Benchmark - First-Attempt Success Rate: 100.0 (#41)
Gemini 4 Argon on Tinybird AI SQL Benchmark - Success Rate: 100.0 (#178)
Gemini 4 Argon on UGI - Natural Intelligence: 81.36 (#3)
Gemini 4 Argon on UGI - Willingness (W/10): 1.8 (#1247)
Gemini 4 Argon on UGI - Writing: 79.71 (#1)
Gemini 4 Argon on UGI Leaderboard: 49.92 (#150)
Gemini 4 Argon on Vals AI Vibe Code Bench 1-100: 21.2 (#8)
Gemini 4 Argon on Vectara Hallucination Leaderboard: 87.9 (#83)
Gemini 4 Argon on VoxelBench: 2229.0 (#6)
Gemini 4 Argon on Wiz Cyber Model Arena (ADK React) - Websec.fr CTFs: 74.7 (#7)
Gemini 4 Argon on Wiz Cyber Model Arena (Claude Code) - API Security: 82.7 (#3)
Gemini 4 Argon on Wiz Cyber Model Arena (Claude Code) - Overall: 70.8 (#2)
Gemini 4 Argon on Wiz Cyber Model Arena (Claude Code) - Websec.fr CTFs: 66.7 (#12)
Muse Wonder Full on AA CritPt: 32.86 (#1)
Muse Wonder Full on AA GDP.pdf: 32.6 (#1)
Muse Wonder Full on AA GDP.pdf - Construction - Criterion Pass Rate: 75.97 (#13)
Muse Wonder Full on AA GDP.pdf - Criterion Pass Rate: 80.35 (#16)
Muse Wonder Full on AA GDP.pdf - Engineering - Criterion Pass Rate: 75.05 (#29)
Muse Wonder Full on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 83.08 (#30)
Muse Wonder Full on AA GDP.pdf - HR - Criterion Pass Rate: 72.48 (#73)
Muse Wonder Full on AA GDP.pdf - Healthcare - Criterion Pass Rate: 89.84 (#17)
Muse Wonder Full on AA GDP.pdf - Insurance - Criterion Pass Rate: 84.25 (#20)
Muse Wonder Full on AA GDP.pdf - Legal - Criterion Pass Rate: 87.8 (#3)
Muse Wonder Full on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 79.93 (#8)
Muse Wonder Full on AA GDP.pdf - Real Estate - Criterion Pass Rate: 75.78 (#17)
Muse Wonder Full on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 80.46 (#3)
Muse Wonder Full on AA GDPval: 1857.09 (#2)
Muse Wonder Full on AA Humanity's Last Exam: 53.61 (#16)
Muse Wonder Full on AA Long Context Reasoning: 83.67 (#15)
Muse Wonder Full on AA Omniscience: 30.15 (#32)
Muse Wonder Full on AA SciCode: 62.62 (#4)
Muse Wonder Full on AA Terminal-Bench 4.0: 57.07 (#6)
Muse Wonder Full on AA-Briefcase: 61.31 (#5)
Muse Wonder Full on AA-Briefcase - Analytical Quality Elo: 2116.26 (#2)
Muse Wonder Full on AA-Briefcase - DOCX - Rubric Pass Rate: 60.36 (#5)
Muse Wonder Full on AA-Briefcase - PDF - Rubric Pass Rate: 67.24 (#4)
Muse Wonder Full on AA-Briefcase - PPTX - Rubric Pass Rate: 57.14 (#12)
Muse Wonder Full on AA-Briefcase - Presentation Elo: 1524.05 (#14)
Muse Wonder Full on AA-Briefcase - XLSX - Rubric Pass Rate: 61.36 (#3)
Muse Wonder Full on AA-Omniscience Accuracy: 43.22 (#99)
Muse Wonder Full on AA-Omniscience Hallucination Rate: 23.01 (#18)
Muse Wonder Full on AA-Omniscience Index - Business: 23.5 (#33)
Muse Wonder Full on AA-Omniscience Index - Health: 16.4 (#28)
Muse Wonder Full on AA-Omniscience Index - Humanities & Social Sciences: 31.5 (#26)
Muse Wonder Full on AA-Omniscience Index - Law: 24.8 (#54)
Muse Wonder Full on AA-Omniscience Index - Science, Engineering & Mathematics: 28.6 (#13)
Muse Wonder Full on AA-Omniscience Index - Software Engineering (SWE): 56.1 (#53)
Muse Wonder Full on AA-Omniscience Index - Software Engineering (SWE) - C: 79.0 (#41)
Muse Wonder Full on AA-Omniscience Index - Software Engineering (SWE) - Dart: 36.0 (#71)
Muse Wonder Full on AA-Omniscience Index - Software Engineering (SWE) - Go: 64.0 (#50)
Muse Wonder Full on AA-Omniscience Index - Software Engineering (SWE) - HTML: 74.0 (#30)
Muse Wonder Full on AA-Omniscience Index - Software Engineering (SWE) - Java: 32.0 (#49)
Muse Wonder Full on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 74.55 (#35)
Muse Wonder Full on AA-Omniscience Index - Software Engineering (SWE) - Julia: 24.0 (#90)
Muse Wonder Full on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 46.0 (#59)
Muse Wonder Full on AA-Omniscience Index - Software Engineering (SWE) - PHP: 52.0 (#79)
Muse Wonder Full on AA-Omniscience Index - Software Engineering (SWE) - Python: 48.5 (#85)
Muse Wonder Full on AA-Omniscience Index - Software Engineering (SWE) - R: 48.0 (#40)
Muse Wonder Full on AA-Omniscience Index - Software Engineering (SWE) - Rust: 70.0 (#53)
Muse Wonder Full on AA-Omniscience Index - Software Engineering (SWE) - Swift: 68.0 (#34)
Muse Wonder Full on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 58.89 (#60)
Muse Wonder Full on Artificial Analysis Intelligence Index: 55.8 (#4)
Muse Wonder Full on AutomationBench-AA: 71.33 (#3)
Muse Wonder Full on AutomationBench-AA - Finance - Raw Objective Completion: 85.68 (#35)
Muse Wonder Full on AutomationBench-AA - HR - Raw Objective Completion: 83.32 (#1)
Muse Wonder Full on AutomationBench-AA - Marketing - Raw Objective Completion: 84.58 (#37)
Muse Wonder Full on AutomationBench-AA - Operations - Raw Objective Completion: 96.21 (#11)
Muse Wonder Full on AutomationBench-AA - Sales - Raw Objective Completion: 88.19 (#6)
Muse Wonder Full on AutomationBench-AA - Strict: 45.36 (#3)
Muse Wonder Full on AutomationBench-AA - Support - Raw Objective Completion: 92.3 (#3)
Muse Wonder Full on BenchLM: 74.3 (#11)
Muse Wonder Full on Bug Hunt Bench: 42.0 (#8)
Muse Wonder Full on Bug Hunt Bench - LMS: 25.5 (#4)
Muse Wonder Full on Bug Hunt Bench - VS Code Extension: 21.5 (#6)
Muse Wonder Full on GDP.pdf: 33.2 (#3)
Muse Wonder Full on Harvey LAB-AA: 13.06 (#1)
Muse Wonder Full on Harvey LAB-AA - All Pass: 25.56 (#2)
Muse Wonder Full on Harvey LAB-AA - Antitrust & Competition: 93.6 (#4)
Muse Wonder Full on Harvey LAB-AA - Arbitration & International Dispute Resolution: 96.06 (#2)
Muse Wonder Full on Harvey LAB-AA - Banking & Finance: 96.64 (#3)
Muse Wonder Full on Harvey LAB-AA - Bankruptcy & Restructuring: 94.46 (#2)
Muse Wonder Full on Harvey LAB-AA - Capital Markets: 96.86 (#2)
Muse Wonder Full on Harvey LAB-AA - Corporate Governance: 96.69 (#2)
Muse Wonder Full on Harvey LAB-AA - Corporate M&A: 98.18 (#1)
Muse Wonder Full on Harvey LAB-AA - Data Privacy & Cybersecurity: 97.88 (#3)
Muse Wonder Full on Harvey LAB-AA - Emerging Companies & Venture Capital: 95.79 (#2)
Muse Wonder Full on Harvey LAB-AA - Employment & Labor: 91.51 (#2)
Muse Wonder Full on Harvey LAB-AA - Energy & Natural Resources: 97.36 (#3)
Muse Wonder Full on Harvey LAB-AA - Environmental & ESG: 92.95 (#2)
Muse Wonder Full on Harvey LAB-AA - Funds & Asset Management: 95.72 (#1)
Muse Wonder Full on Harvey LAB-AA - Healthcare & Life Sciences: 95.19 (#3)
Muse Wonder Full on Harvey LAB-AA - Immigration: 94.55 (#2)
Muse Wonder Full on Harvey LAB-AA - Insurance: 94.12 (#3)
Muse Wonder Full on Harvey LAB-AA - Intellectual Property: 97.86 (#1)
Muse Wonder Full on Harvey LAB-AA - International Trade & Sanctions: 96.41 (#1)
Muse Wonder Full on Harvey LAB-AA - Litigation & Dispute Resolution: 94.86 (#2)
Muse Wonder Full on Harvey LAB-AA - Real Estate: 98.06 (#1)
Muse Wonder Full on Harvey LAB-AA - Structured Finance & Securitization: 94.52 (#2)
Muse Wonder Full on Harvey LAB-AA - Tax: 90.97 (#3)
Muse Wonder Full on Harvey LAB-AA - Trusts, Estates & Private Client: 95.04 (#3)
Muse Wonder Full on Harvey LAB-AA - White Collar Defense & Investigations: 95.74 (#1)
Muse Wonder Full on LLM Stats (Artificial Analysis): 56.0 (#3)
Muse Wonder Full on LLM Stats (GDP.pdf): 33.2 (#4)
Muse Wonder Full on LLM Stats (HealthBench Professional): 62.3 (#7)
Muse Wonder Full on LLM Stats (Program Bench): 59.0 (#6)
Muse Wonder Full on LLM Stats Score: 52.06 (#13)
Muse Wonder Full on Vals AI (Vals Index): 64.57 (#5)
Muse Wonder Full on Vals AI BioMysteryBench: 73.33 (#8)
Muse Wonder Full on Vals AI Code Migration: 61.03 (#7)
Muse Wonder Full on Vals AI CyberBench: 73.63 (#12)
Muse Wonder Full on Vals AI Excel Modeling: 71.92 (#9)
Muse Wonder Full on Vals AI Finance Agent v2: 59.06 (#6)
Muse Wonder Full on Vals AI Harvey Legal Agent Bench: 21.67 (#4)
Muse Wonder Full on Vals AI IOI: 79.83 (#11)
Muse Wonder Full on Vals AI Legal Research Bench: 53.37 (#5)
Muse Wonder Full on Vals AI LegalBench: 87.42 (#4)
Muse Wonder Full on Vals AI MedCode: 49.93 (#18)
Muse Wonder Full on Vals AI MedScribe: 89.33 (#8)
Muse Wonder Full on Vals AI MysteryMechanism: 35.13 (#11)
Muse Wonder Full on Vals AI ProgramBench: 0.5 (#27)
Muse Wonder Full on Vals AI ProofBench: 94.0 (#10)
Muse Wonder Full on Vals AI Public Benefits Bench: 73.75 (#3)
Muse Wonder Full on Vals AI ReverseEngBench: 11.83 (#12)
Muse Wonder Full on Vals AI Tax Agent Bench: 43.0 (#5)
Muse Wonder Full on Vals AI Terminal-Bench 4.0: 58.08 (#5)
Muse Wonder Full on Vals AI Terminal-Bench Science: 37.14 (#7)
Muse Wonder Full on Vals AI Vibe Code Bench: 86.0 (#12)
Muse Wonder Full on Vending-Bench 2: 12291.558333333334 (#4)

New #1 Leaders (117)

Design Arena (3D): GPT-6 Astra (Max) (1509.0) beat Claude Opus 5.5 by 29.0
KNOWS - Accessibility Tree + Screenshot: GPT-6 Astra (72.5) beat GPT-5.5 by 23.0
HiL-Bench: Muse Wonder (79.67) beat Claude Fable 5.1 by 18.17
Wiz Cyber Model Arena (ADK React) - Cloud Security: Gemini 4 Argon (82.6) beat Gemini 3.8 Flash Cyber by 15.9
HieroglyphBench: Gemini 4 Argon (69.7) beat Gemini 3.7 Flash by 14.7
Lean AI Formalization Leaderboard: Axiom Prover (Axiom Math) (263.0) beat Humanifa + GPT 5.6 sol by 12.0
WDCD R3 Pressure Integrity: Claude Opus 4.7 (150.0) beat Grok 4 by 12.0
WebArena: Narada AI Code (86.1) beat DeepSeek V3.2 by 11.8
LLM Stats (SWE-Marathon): Muse Wonder Full (57.5) beat Grok 4.7 by 11.5
Wiz Cyber Model Arena (Claude Code) - Cloud Security: Gemini 4 Argon (73.9) beat Claude Opus 5 by 10.1
Wiz Cyber Model Arena (ADK React) - Code Vulnerabilities: Gemini 4 Argon (65.8) beat Gemini 3.8 Flash Cyber by 8.8
Open Japanese LLM v2 - M-IFEval-Ja: llm-jp-4.1-32B-a3B (Thinking) (56.4) beat llm-jp-4-kappa-32B-a3B-v0.1 (Thinking) by 8.73
SWE-bench Pro (Opus 5 & Mythos 5.1 System Cards): Claude Opus 5.5 (Max) (89.9) beat Claude Fable 5.1 (Max) by 8.7
BenchX - Pass Rate: gemma-4-31B-it · T=1.0 (90.14) beat GPT-5.6 Sol by 7.2
Wiz Cyber Model Arena (Claude Code) - Websec.fr CTFs: GPT-6 Sol (88.5) beat Claude Opus 5 by 6.9
TaxCalcBench: Meta Muse Wonder w/ Web Search (49/50) (71.43) beat GPT-6 Sol Web Search by 5.43
Icelandic LLM - Famous Phrases: Gemini 4 Argon (36.19) beat Gemini 3.8 Flash by 5.31
EKI Estonian LLM - Trivia: Gemini 4 Argon (90.0) beat Gemini 3.1 Pro (Preview) by 5.0
Wiz Cyber Model Arena (Claude Code) - Code Vulnerabilities: Gemini 4 Argon (59.9) beat Claude Opus 5 by 4.7
Senior SWE-Bench: Claude Opus 5.5 (Mini-SWE-Agent, High) (39.325842696629216) beat GPT-5.6 Sol (Mini-SWE-Agent, xHigh) by 4.59
PinchBench - 147 Tasks - OpenClaw 2026.6.5 (5181e4f): nemotron-3.5-bolt-74B-a9B-bf16 (91.68252517006803) beat Nemotron Lightning 3.5 30B A3B by 4.51
Chess Puzzles (Epoch AI): GPT-6 Astra Pro (Max) (76.0) beat GPT-6 Astra (Max) by 4.0
PUMA Polish Multimodal - Document QA: Gemini 4 Argon (93.0) beat Claude Opus 5.5 by 4.0
AIMultiple - Agentic RAG Routing: Gemini 4 Argon (94.02) beat Claude Opus 5.5 by 3.8
SEAL - AudioMultiChallenge - Text Output: Gemini 3.8 Flash (High) (60.4) beat Inkling (Thinking) by 3.76
AA-AnalystAgent - Pass@K: GPT-6 Luna (Max) (85.0) beat Gemini 3.1 Pro (Preview) by 3.75
Vals AI Vibe Code Bench 1-100: Muse Wonder Full (33.98) beat Claude Opus 5.5 by 3.62
Harvey LAB-AA: Muse Wonder Full (Max) (13.06) beat Grok 4.7 (xHigh) by 3.62
AA-Briefcase - DOCX - Rubric Pass Rate: Muse Wonder (Max) (65.68) beat Gemini 4 Argon (High) by 3.55
Harvey LAB-AA - Funds & Asset Management: Muse Wonder Full (Max) (95.72) beat Muse Spark 1.3 (Max) by 3.4
FEV-Bench - WAPE: Timestral-209M (88.9) beat timesfm-3 by 3.19
Next.js Agent Evals (OpenCode) - Success Rate: Gemini 4 Argon (97.0) beat Grok 4.7 by 3.0
FEV-Bench - MASE: Timestral-209M (88.84) beat timesfm-3 by 2.88
ReLE - Language and Instruction Following: Gemini 4 Argon (81.6) beat Hy4 preview by 2.8
FEV-Bench: Timestral-209M (89.59) beat timesfm-3 by 2.75
FEV-Bench - SQL: Timestral-209M (90.84) beat timesfm-3 by 2.63
Vals AI CyberBench: Muse Wonder (80.36) beat GPT-6 Sol by 2.38
FEV-Bench - WQL: Timestral-209M (89.77) beat timesfm-3 by 2.31
Wiz Cyber Model Arena (ADK React) - Overall: Gemini 4 Argon (77.2) beat Gemini 3.8 Flash Cyber by 2.3
BALSAM - Summarization: Nojoom-AQL-120B (58.82) beat Nojoom-AQL-42B by 2.24
Harvey LAB-AA - Immigration: Muse Wonder (Max) (95.39) beat Grok 4.7 (xHigh) by 2.18
DuelLab Overall: Claude Opus 5.5 (xHigh) (95.16780469798) beat GPT-6 Astra (xHigh) by 2.05
PLCC - Vocabulary: Gemini 4 Argon (98.0) beat Gemini 3.1 Pro (Preview) by 2.0
SloBench - Slovene SuperGLUE: Gizela-31B-it-vit-think (79.87) beat Gemma 3 27B (IT) by 1.93
SWE-Marathon v1.1: Muse Wonder Full (Thinking, max) (Muse Code) (53.125) beat Claude Sonnet 5.5 (Thinking, Max) (Claude Code) by 1.88
Open Japanese LLM v2 - JBLiMP: llm-jp-4.1-33B (Thinking) (85.8) beat llm-jp-4-8B-Thinking by 1.81
Humanity's Sixth Sense: Gemini 4 Argon (High) (55.4) beat GPT-6 Astra (Max) by 1.8
ReLE - Coding: Claude Sonnet 5.5 (78.0) beat Claude Opus 5.5 by 1.7
Open Japanese LLM v2 - JCoLA Out-of-Domain: llm-jp-4.1-8B (Thinking) (82.87) beat llm-jp-4-8B-Thinking by 1.52
Kaggle Open Benchmarks - MMLU-Pro: Gemini 4 Argon (93.19) beat Claude Opus 5 by 1.51
EKI Estonian LLM - Language Advice: Gemini 4 Argon (77.5) beat GPT-6 Astra by 1.46
AutomationBench-AA - HR - Raw Objective Completion: Muse Wonder Full (Max) (83.32) beat Gemini 4 Argon (High) by 1.33
ReLE - Education - High School Subjects: Gemini 4 Argon (79.3) beat Gemini 3.1 Pro (Preview) by 1.2
UGI - Writing: gemini-4-argon (Thinking, low) (79.71) beat Gemini 3.8 Flash (Thinking, Medium) by 1.16
MMEB Leaderboard: Wianmu-5B-v0 (60.45) beat WeMM-Embedding-9B by 1.12
Agent Arena - Confirmed Success: Claude Opus 5.5 (Max) (17.29) beat Claude Fable 5.1 (Max) by 1.1
LLM Stats (DeepSWE 1.1): Muse Wonder Full (79.0) beat Gemini 4 Argon by 1.1
Harvey LAB-AA - Corporate M&A: Muse Wonder Full (Max) (98.18) beat Mistral Large 4 Preview by 1.07
Design Arena (ASCII Art): GPT-6 Astra (1353.0) beat Claude Opus 5 by 1.0
PUMA Polish Multimodal - OCR: Gemini 4 Argon (91.0) beat Claude Fable 5 by 1.0
Argo-Bench - Solved: Gemini 4 Argon (High) (35.71) beat Claude Opus 5.5 (xHigh) by 0.95
Icelandic LLM - Idiom Completion: Gemini 4 Argon (75.96) beat Claude Opus 5.5 by 0.93
OpenRouter GPQA Diamond: Gemini 4 Argon (96.5) beat Gemini 3.8 Flash by 0.9
PUMA Polish Multimodal - Vision Mean: Gemini 4 Argon (84.83) beat Gemini 3.7 Flash by 0.83
GDP.pdf: GPT-6.1 Sol (Max) (35.0) beat GPT-6 Astra (Max) by 0.8
Harvey LAB-AA - Healthcare & Life Sciences: Muse Wonder (Max) (96.09) beat Muse Spark 1.3 (Max) by 0.79
AA GDP.pdf - Legal - Criterion Pass Rate: Muse Wonder (Max) (88.03) beat DeepSeek V4.1 Flash (Max) by 0.75
Kaggle Open Benchmarks - MMLU: Gemini 4 Argon (95.25) beat Grok 4.5 by 0.71
ReLE - Education: Gemini 4 Argon (69.3) beat Gemini 3.1 Pro (Preview) by 0.7
AI Coding Daily (OpenCode) - React-TS Code Quality: Mistral Large 4 (High) (19.33) beat DeepSeek V4.1 Flash (High) by 0.66
Epoch AI - GPQA Diamond: Gemini 4 Argon (High) (96.4) beat GPT-6 Astra (Max) by 0.63
EnterpriseRAG Bench - Correctness: Flexor.ai (+ GPT-6.1 Sol) (93.0) beat CDL (Causal Dynamics Lab) by 0.6
Wiz Cyber Model Arena (ADK React) - API Security: Gemini 4 Argon (85.8) beat Gemini 3.8 Flash Cyber by 0.6
EnterpriseRAG Bench: Flexor.ai (+ GPT-6.1 Sol) (91.48) beat CDL (Causal Dynamics Lab) by 0.57
AA CritPt: Muse Wonder Full (Max) (32.86) beat GPT-5.6 Sol (Max) by 0.57
SciLaws-Bench: GPT-6 Astra (51.33) beat GPT-5.5 by 0.56
NYT Connections Extended: Gemini 4 Argon (Thinking, High) (98.6) beat GPT-6 Astra (Thinking, xHigh) by 0.5
Harvey LAB-AA - Real Estate: Muse Wonder Full (Max) (98.06) beat Muse Spark 1.3 (Max) by 0.49
Epoch AI - Rli: Claude Opus 5.5 (Unknown) (21.25) beat GPT-6 Astra by 0.42
SimpleQA Verified: Gemini 4 Argon (High) (76.0) beat GPT-6 Astra (Max) by 0.4
AA GDP.pdf: Muse Wonder Full (Max) (32.6) beat GPT-6 Astra (xHigh) by 0.4
Harvey LAB-AA - White Collar Defense & Investigations: Muse Wonder Full (Max) (95.74) beat Muse Spark 1.3 (Max) by 0.34
GAIA: Shadow Harness V8.01 (94.02) beat Ops-Agentic-Search-2.0 by 0.33
PLCC - Overall: Gemini 4 Argon (97.33) beat Gemini 3.1 Pro (Preview) by 0.33
STEMBench120 v1.0.0 (Closed Book) - Physics: Claude Opus 5.5 (2.808) beat Hy4 preview by 0.29
STEMBench120 v1.0.0 (Closed Book) - Engineering: Claude Opus 5.5 (2.872) beat Hy4 preview by 0.26
Kaggle Global-MMLU-Lite - Bengali: Gemini 4 Argon (96.0) beat Claude Opus 4.8 by 0.25
Kaggle Global-MMLU-Lite - Japanese: Gemini 4 Argon (96.25) beat Claude Opus 5 by 0.25
Kaggle Global-MMLU-Lite - Swahili: Gemini 4 Argon (95.75) beat Claude Opus 5.5 by 0.25
AI Forever RAG - Set Generation: Qwen3.8-27B (kb graphRAG) (aa045f) (0.5046) beat RuadaptQwen2.5-32B-Instruct (9449f3) by 0.22
AI Forever RAG - Simple Generation: Qwen3.8-27B (kb graphRAG) (aa045f) (0.6265) beat RuadaptQwen2.5-32B-Instruct (9449f3) by 0.2
AI Forever RAG - Set Retrieval: Qwen3.8-27B (kb graphRAG) (aa045f) (0.8139) beat Qwen2.5-7B-Instruct (d6ccf3) by 0.19
AI Forever RAG - Multi-Hop Generation: gemma4-25B (kb graphRAG) (fe39ea) (0.6374) beat RuadaptQwen2.5-32B-Instruct (9449f3) by 0.19
STEMBench120 v1.0.0 (Closed Book) - Astronomy: Gemini 4 Argon (2.778) beat DeepSeek V4.1 Flash by 0.18
AI Forever RAG - Generation: Qwen3.8-27B (kb graphRAG) (aa045f) (0.6474) beat RuadaptQwen2.5-32B-Instruct (9449f3) by 0.18
STEMBench120 v1.0.0 (Closed Book) - Earth science: Claude Opus 5.5 (2.76) beat MiMo-V2.6-Pro by 0.14
Harvey LAB-AA - International Trade & Sanctions: Muse Wonder Full (Max) (96.41) beat Muse Spark 1.3 (Max) by 0.13
Anthropic Child Safety Single-Turn Benign Requests - Over-Refusal Rate (Claude.ai): Claude Opus 5.5 (0.05) beat Claude Fable 5.1 by 0.12
Harvey LAB-AA - Intellectual Property: Muse Wonder Full (Max) (97.86) beat Muse Spark 1.3 (Max) by 0.11
Anthropic Claude Code Dual-Use Requests - Success Rate (Mythos 5.1 System Card): Claude Opus 5.5 (99.8) beat Claude Opus 5 by 0.1
STEMBench120 v1.0.0 (Closed Book) - Overall: Muse Wonder (2.683) beat Hy4 preview by 0.1
AI Forever RAG - Conditional Generation: Qwen3.8-27B (kb graphRAG) (aa045f) (0.8285) beat RuadaptQwen2.5-32B-Instruct (9449f3) by 0.1
STEMBench120 v1.0.0 (Closed Book) - Chemistry: Muse Wonder (2.703) beat Hy4 preview by 0.09
AI Forever RAG - Retrieval: Qwen3.8-27B (kb graphRAG) (aa045f) (0.7716) beat RuadaptQwen2.5-32B-Instruct (9449f3) by 0.09
STEMBench120 v1.0.0 (Tools) - Engineering: GPT-6 Astra (xHigh) (2.894) beat GPT-6.1 Sol by 0.08
AI Forever RAG - Multi-Hop Retrieval: Qwen3.8-27B (kb graphRAG) (aa045f) (0.7529) beat RuadaptQwen2.5-32B-Instruct (9449f3) by 0.07
AI Forever RAG - Simple Retrieval: Qwen3.8-27B (kb graphRAG) (aa045f) (0.7688) beat RuadaptQwen2.5-32B-Instruct (9449f3) by 0.06
STEMBench120 v1.0.0 (Tools) - Overall: GPT-6 Astra (xHigh) (2.809) beat GPT-6.1 Sol by 0.05
STEMBench120 v1.0.0 (Tools) - Biology: GPT-6 Astra (xHigh) (2.709) beat GPT-6.1 Sol by 0.05
STEMBench120 v1.0.0 (Tools) - Chemistry: GPT-6 Astra (xHigh) (2.777) beat GPT-6.1 Sol by 0.05
STEMBench120 v1.0.0 (Tools) - Earth science: GPT-6 Astra (xHigh) (2.726) beat GPT-6.1 Sol by 0.05
STEMBench120 v1.0.0 (Tools) - Physics: GPT-6 Astra (xHigh) (3.025) beat GPT-6.1 Sol by 0.04
STEMBench120 v1.0.0 (Closed Book) - Biology: Muse Wonder (2.615) beat Hy4 preview by 0.03
STEMBench120 v1.0.0 (Tools) - Astronomy: GPT-6 Astra (xHigh) (2.723) beat GPT-6.1 Sol by 0.03
Conceptual Reasoning Index - Argument Evaluation (LMCA): Claude Sonnet 5.5 (68.26) beat Claude Opus 5.5 by 0.03
AI Forever RAG - Conditional Retrieval: gemma4-25B (kb graphRAG) (fe39ea) (0.7644) beat RuadaptQwen2.5-32B-Instruct (9449f3) by 0.02
Sonar LLM Leaderboard - Java - Vulnerability Density: GPT-5.4 (Medium) (0.04) beat GPT-5.2 (High) by 0.01

The Aggregate Digest — 2026-10-17

2026-10-17T09:49:11.425731+00:00

Daily

New Benchmarks (24)

MathArena - ARXIVLEAN September (Accuracy (%)): leader Claude Opus 5.5 (High) (47.17), 5 models
Accuracy on proving or disproving 53 September 2026 arXiv conjectures in Lean4.34.1. Submissions are checked against immutable reference statements with the strict comparator. The exact53 informal problems and formal statements are released by MathArena. Published percentage preserves the source model and effort settings; separate from informal final-answer and false-statement behavior cohorts. Source: https://matharena.ai/?comp=arxivlean--september
MathArena - ARXIV_FALSE September (Accuracy (%)): leader GPT-6.1 Sol (Max) (82.7), 7 models
Normalized 0–3 rubric score on 53 plausible but false mathematical statements from September 2026 arXiv papers. Full credit requires explicitly identifying the statement as false; a directly contradictory proposed correction caps the score at1. The judge classifies response behavior, not proof validity. September harness: coding tools without internet, 12-hour and $100 limit per attempt. Published percentage, with source model and effort settings retained; separate from final-answer and Lean proof scores. Source: https://matharena.ai/?comp=arxiv_false--september
MathArena - ARXIV September (Accuracy (%)): leader GPT-6.1 Sol (Max) (91.51), 7 models
Accuracy on 53 research mathematics final-answer questions from September 2026 arXiv papers. An LLM judge checks mathematical equivalence to the reference answer; individual answers receive binary scores. The September harness permits coding tools without internet, with a 12-hour and $100 model-cost limit per attempt. Published percentages retain the source model and effort settings; distinct from earlier monthly cohorts. Source: https://matharena.ai/?comp=arxiv--september
FDE-Gym L0 original 20 (Mean delivery score): leader Claude Opus 5.5 (0.82), 13 models
FDE-Gym primary paper, 13 configurations in one minimal five-tool agent framework; scripted stakeholders and held-out production data. Native business uplift minus stakeholder-contact cost; negative scores are valid, not percentages. Reference solution 1 is not a theoretical ceiling. No effort inferred where not reported; Haiku extended-thinking run stays separate. L0 original 20 task cohort; same three scenarios have identical L2/L3 instructions. Metric: Mean delivery score. Higher is better. Source: https://arxiv.org/html/2610.22350v1
FDE-Gym L1 original 20 (Mean delivery score): leader Claude Opus 5.5 (0.62), 13 models
FDE-Gym primary paper, 13 configurations in one minimal five-tool agent framework; scripted stakeholders and held-out production data. Native business uplift minus stakeholder-contact cost; negative scores are valid, not percentages. Reference solution 1 is not a theoretical ceiling. No effort inferred where not reported; Haiku extended-thinking run stays separate. L1 original 20 task cohort; same three scenarios have identical L2/L3 instructions. Metric: Mean delivery score. Higher is better. Source: https://arxiv.org/html/2610.22350v1
FDE-Gym L2 original 20 (Mean delivery score): leader Claude Opus 5.5 (0.54), 13 models
FDE-Gym primary paper, 13 configurations in one minimal five-tool agent framework; scripted stakeholders and held-out production data. Native business uplift minus stakeholder-contact cost; negative scores are valid, not percentages. Reference solution 1 is not a theoretical ceiling. No effort inferred where not reported; Haiku extended-thinking run stays separate. L2 original 20 task cohort; same three scenarios have identical L2/L3 instructions. Metric: Mean delivery score. Higher is better. Source: https://arxiv.org/html/2610.22350v1
FDE-Gym L3 original 20 (Mean delivery score): leader Claude Opus 5.5 (0.55), 13 models
FDE-Gym primary paper, 13 configurations in one minimal five-tool agent framework; scripted stakeholders and held-out production data. Native business uplift minus stakeholder-contact cost; negative scores are valid, not percentages. Reference solution 1 is not a theoretical ceiling. No effort inferred where not reported; Haiku extended-thinking run stays separate. L3 original 20 task cohort; same three scenarios have identical L2/L3 instructions. Metric: Mean delivery score. Higher is better. Source: https://arxiv.org/html/2610.22350v1
FDE-Gym L3 full 95 (Mean delivery score): leader Claude Opus 5.5 (0.56), 13 models
FDE-Gym primary paper, 13 configurations in one minimal five-tool agent framework; scripted stakeholders and held-out production data. Native business uplift minus stakeholder-contact cost; negative scores are valid, not percentages. Reference solution 1 is not a theoretical ceiling. No effort inferred where not reported; Haiku extended-thinking run stays separate. L3 full 95 task cohort; same three scenarios have identical L2/L3 instructions. Metric: Mean delivery score. Higher is better. Source: https://arxiv.org/html/2610.22350v1
FDE-Gym L3 resolved full 95 (Resolved tasks (out of 95)): leader Claude Sonnet 5.5 (15.0), 13 models
FDE-Gym primary paper, 13 configurations in one minimal five-tool agent framework; scripted stakeholders and held-out production data. Native business uplift minus stakeholder-contact cost; negative scores are valid, not percentages. Reference solution 1 is not a theoretical ceiling. No effort inferred where not reported; Haiku extended-thinking run stays separate. Full 95-task cohort; resolved means score at least 0.8 and no incident. Metric: Resolved tasks (out of 95). Higher is better. Source: https://arxiv.org/html/2610.22350v1
OpenGCUA OSWorld-v2 offline GUI batch partial score (Partial Score (%)): leader GPT-6 Astra (67.1), 8 models
OpenGCUA paper, OSWorld-v2 v2.1 82-task offline subset excluding26 internet tasks. Latest evaluated attempt per task, never best-of selection; failures retained. Source model identities as printed; effort unspecified for aggregate tables. Partial evaluator score and full-task success are separate metrics. GUI batch GUI-only batched actions. Metric: Partial Score (%). Higher is better. Source: https://arxiv.org/html/2610.22200v1
OpenGCUA OSWorld-v2 offline GUI batch success (Success (%)): leader GPT-6 Astra (36.6), 8 models
OpenGCUA paper, OSWorld-v2 v2.1 82-task offline subset excluding26 internet tasks. Latest evaluated attempt per task, never best-of selection; failures retained. Source model identities as printed; effort unspecified for aggregate tables. Partial evaluator score and full-task success are separate metrics. GUI batch GUI-only batched actions. Metric: Success (%). Higher is better. Source: https://arxiv.org/html/2610.22200v1
OpenGCUA OSWorld-v2 offline OpenGCUA partial score (Partial Score (%)): leader GPT-6 Astra (75.7), 8 models
OpenGCUA paper, OSWorld-v2 v2.1 82-task offline subset excluding26 internet tasks. Latest evaluated attempt per task, never best-of selection; failures retained. Source model identities as printed; effort unspecified for aggregate tables. Partial evaluator score and full-task success are separate metrics. OpenGCUA code execution and model-initiated workflow control. Metric: Partial Score (%). Higher is better. Source: https://arxiv.org/html/2610.22200v1
OpenGCUA OSWorld-v2 offline OpenGCUA success (Success (%)): leader GPT-6 Astra (46.3), 8 models
OpenGCUA paper, OSWorld-v2 v2.1 82-task offline subset excluding26 internet tasks. Latest evaluated attempt per task, never best-of selection; failures retained. Source model identities as printed; effort unspecified for aggregate tables. Partial evaluator score and full-task success are separate metrics. OpenGCUA code execution and model-initiated workflow control. Metric: Success (%). Higher is better. Source: https://arxiv.org/html/2610.22200v1
OpenGCUA OSWorld-v2 offline GUI CLI partial score (Partial score (%)): leader Claude Opus 5 (65.7), 4 models
OpenGCUA paper, OSWorld-v2 v2.1 82-task offline subset excluding26 internet tasks. Latest evaluated attempt per task, never best-of selection; failures retained. Source model identities as printed; effort unspecified for aggregate tables. Partial evaluator score and full-task success are separate metrics. Batched GUI+CLI comparison; full-GCUA cells already present in Table2. Metric: Partial score (%). Higher is better. Source: https://arxiv.org/html/2610.22200v1
OpenGCUA macOS four-task completion (Complete tasks (out of 4)): leader GPT-6 Astra (4.0), 5 models
OpenGCUA real-device macOS harness, four tasks, selected run1 per task; task-level capability, tiny preliminary cohort, not a general ranking claim. Workbook, PDF, shopping research and hotel comparison. DeepSeek omitted by source because hotel trajectory is missing; no invented aggregate. Metric: Complete tasks (out of 4). Higher is better. Source: https://arxiv.org/html/2610.22200v1
MindEval2 MHQR GPT-6.1 Sol High (MHQR (0–10, lower better)): leader GPT-5.6 Terra (0.84), 18 models
MindEval2 MHQR clinical response quality: weighted issue severity0/1/5/10, lower is better; 50 patients over3 sessions, 150 patient-session scores, 61819 replies. Counsellors temperature1 and provider defaults; DeepSeek V4.1 Flash thinking patient simulator. GPT-6.1 Sol High judge; same replies across judge variants. Shared-family self-preference and clinician disagreement are limitations; not a clinical-effectiveness claim. Metric: MHQR (0–10, lower better). Lower is better. Source: https://arxiv.org/html/2610.22314v1
MindEval2 MHQR DeepSeek V4.1 Flash (MHQR (0–10, lower better)): leader GPT-5.6 Terra (0.15), 18 models
MindEval2 MHQR clinical response quality: weighted issue severity0/1/5/10, lower is better; 50 patients over3 sessions, 150 patient-session scores, 61819 replies. Counsellors temperature1 and provider defaults; DeepSeek V4.1 Flash thinking patient simulator. DeepSeek V4.1 Flash judge; same replies across judge variants. Shared-family self-preference and clinician disagreement are limitations; not a clinical-effectiveness claim. Metric: MHQR (0–10, lower better). Lower is better. Source: https://arxiv.org/html/2610.22314v1
Diverge - Creative Utility (Normalized creative utility (0–1)): leader Diverge Qwen3-8B SFT-RL (Non-reasoning) (0.64), 11 models
Diverge v1 held-out1,740 instances from2026 papers,16domains; three idea cards generated as one set except DARLING three single-card draws. Qwen custom variants separate, proposed checkpoint600; generic Nemotron120B revision unresolved, never inferred as current Nemotron3Super. Impact eval GPT5.4mini both candidate/gold orderings; distance eval Gemini3.5Flash,temp0,max1024 (training uses different judges); LC win rate length-controlled pooled logistic adjustment. CU greedy-order patience0.9 quality-times-distance sum divided by1+0.9+0.9². Grounding/distance are diagnostics, not monotonically better capability. Dataset/code release currently coming soon. Metric: Normalized creative utility (0–1). Higher is better. Source: https://arxiv.org/html/2610.22324v1
Diverge - At Least One Win (Fraction of instances with at least one winning idea (0–1)): leader Diverge Qwen3-8B SFT-RL (Non-reasoning) (0.78), 11 models
Diverge v1 held-out1,740 instances from2026 papers,16domains; three idea cards generated as one set except DARLING three single-card draws. Qwen custom variants separate, proposed checkpoint600; generic Nemotron120B revision unresolved, never inferred as current Nemotron3Super. Impact eval GPT5.4mini both candidate/gold orderings; distance eval Gemini3.5Flash,temp0,max1024 (training uses different judges); LC win rate length-controlled pooled logistic adjustment. CU greedy-order patience0.9 quality-times-distance sum divided by1+0.9+0.9². Grounding/distance are diagnostics, not monotonically better capability. Dataset/code release currently coming soon. Metric: Fraction of instances with at least one winning idea (0–1). Higher is better. Source: https://arxiv.org/html/2610.22324v1
Diverge - Length-controlled Win Rate (Length-controlled impact win rate (0–1)): leader Diverge Qwen3-8B SFT-RL (Non-reasoning) (0.5), 11 models
Diverge v1 held-out1,740 instances from2026 papers,16domains; three idea cards generated as one set except DARLING three single-card draws. Qwen custom variants separate, proposed checkpoint600; generic Nemotron120B revision unresolved, never inferred as current Nemotron3Super. Impact eval GPT5.4mini both candidate/gold orderings; distance eval Gemini3.5Flash,temp0,max1024 (training uses different judges); LC win rate length-controlled pooled logistic adjustment. CU greedy-order patience0.9 quality-times-distance sum divided by1+0.9+0.9². Grounding/distance are diagnostics, not monotonically better capability. Dataset/code release currently coming soon. Metric: Length-controlled impact win rate (0–1). Higher is better. Source: https://arxiv.org/html/2610.22324v1
Diverge Independent Sampling - Creative Utility (Normalized creative utility (0–1)): leader Claude Opus 4.8 (0.34), 4 models
Diverge v1 held-out1,740 instances from2026 papers,16domains; three idea cards generated as one set except DARLING three single-card draws. Qwen custom variants separate, proposed checkpoint600; generic Nemotron120B revision unresolved, never inferred as current Nemotron3Super. Impact eval GPT5.4mini both candidate/gold orderings; distance eval Gemini3.5Flash,temp0,max1024 (training uses different judges); LC win rate length-controlled pooled logistic adjustment. CU greedy-order patience0.9 quality-times-distance sum divided by1+0.9+0.9². Grounding/distance are diagnostics, not monotonically better capability. Dataset/code release currently coming soon. Table12 only k1,n3 independent single-idea samples, separate board from one k3 set; repeated set-mode cells not duplicated. Metric: Normalized creative utility (0–1). Higher is better. Source: https://arxiv.org/html/2610.22324v1
Diverge Independent Sampling - At Least One Win (Fraction of instances with at least one winning idea (0–1)): leader Claude Opus 4.8 (0.59), 4 models
Diverge v1 held-out1,740 instances from2026 papers,16domains; three idea cards generated as one set except DARLING three single-card draws. Qwen custom variants separate, proposed checkpoint600; generic Nemotron120B revision unresolved, never inferred as current Nemotron3Super. Impact eval GPT5.4mini both candidate/gold orderings; distance eval Gemini3.5Flash,temp0,max1024 (training uses different judges); LC win rate length-controlled pooled logistic adjustment. CU greedy-order patience0.9 quality-times-distance sum divided by1+0.9+0.9². Grounding/distance are diagnostics, not monotonically better capability. Dataset/code release currently coming soon. Table12 only k1,n3 independent single-idea samples, separate board from one k3 set; repeated set-mode cells not duplicated. Metric: Fraction of instances with at least one winning idea (0–1). Higher is better. Source: https://arxiv.org/html/2610.22324v1
Diverge Independent Sampling - Length-controlled Win Rate (Length-controlled impact win rate (0–1)): leader Claude Opus 4.8 (0.33), 4 models
Diverge v1 held-out1,740 instances from2026 papers,16domains; three idea cards generated as one set except DARLING three single-card draws. Qwen custom variants separate, proposed checkpoint600; generic Nemotron120B revision unresolved, never inferred as current Nemotron3Super. Impact eval GPT5.4mini both candidate/gold orderings; distance eval Gemini3.5Flash,temp0,max1024 (training uses different judges); LC win rate length-controlled pooled logistic adjustment. CU greedy-order patience0.9 quality-times-distance sum divided by1+0.9+0.9². Grounding/distance are diagnostics, not monotonically better capability. Dataset/code release currently coming soon. Table12 only k1,n3 independent single-idea samples, separate board from one k3 set; repeated set-mode cells not duplicated. Metric: Length-controlled impact win rate (0–1). Higher is better. Source: https://arxiv.org/html/2610.22324v1
Vals AI Science Index (Science Index (%)): leader GPT-6 Astra (65.09), 21 models
Vals AI Science Index v1, launched October16 2026: equal-weight mean of BioMysteryBench, MysteryMechanism and Terminal-Bench Science scores. Measures agentic scientific discovery and execution across biology, hidden mathematical mechanisms and scientific terminal tasks. Native0–100 index, higher is better; component overlap does not create three independent new experiments. The source labels its question set proprietary. https://www.vals.ai/benchmarks/science_index

New Models (1)

Beam 1.0 — Aggregate Rating 1763, #97
- AutomationBench-AA - Support - Raw Objective Completion: 84.2 (#68/225)
- AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 78.29 (#72/224)
- AutomationBench-AA - Finance - Raw Objective Completion: 78.83 (#73/225)
- AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 67.37 (#76/224)
- AutomationBench-AA - Marketing - Raw Objective Completion: 77.7 (#76/225)
- AutomationBench-AA: 48.46 (#80/225)
- AutomationBench-AA - Operations - Raw Objective Completion: 85.03 (#80/225)
- AutomationBench-AA - Strict: 17.05 (#81/225)
- AutomationBench-AA - Sales - Raw Objective Completion: 66.34 (#81/225)
- AutomationBench-AA - HR - Raw Objective Completion: 65.55 (#84/225)
- ...and 47 more

Top-10 New Scores (14)

GPT-6 Astra on AutomationBench: 41.4 (#3)
GPT-6 Astra Pro on Chess Puzzles (Epoch AI): 76.0 (#1)
Gemini 4 Argon on AA-3D: 1153.9782 (#4)
Gemini 4 Argon on AA-AnalystAgent: 48.75 (#12)
Gemini 4 Argon on AA-AnalystAgent - Pass@1: 57.75 (#25)
Gemini 4 Argon on AA-AnalystAgent - Pass@K: 66.25 (#40)
Gemini 4 Argon on Argo-Bench: 54.99 (#3)
Gemini 4 Argon on Argo-Bench - Solved: 35.71 (#1)
Gemini 4 Argon on ChessBench GitHub - Elo: 2430.0 (#12)
Gemini 4 Argon on MathArena - ARXIV August: 82.46 (#7)
Gemini 4 Argon on MathArena - ARXIV_FALSE August: 54.17 (#11)
Gemini 4 Argon on SEAL - AudioMultiChallenge - Text Output: 58.41 (#3)
Gemini 4 Argon on SEAL - Professional Reasoning Benchmark - Finance: 49.9 (#11)
Gemini 4 Argon on SEAL - Professional Reasoning Benchmark - Legal: 49.41 (#11)

New #1 Leaders (14)

WebArena: Narada AI Code (86.1) beat DeepSeek V3.2 by 11.8
Open Japanese LLM v2 - M-IFEval-Ja: llm-jp-4.1-32B-a3B (Thinking) (56.4) beat llm-jp-4-kappa-32B-a3B-v0.1 (Thinking) by 8.73
SEAL - AudioMultiChallenge - Text Output: Gemini 3.8 Flash (High) (60.4) beat Inkling (Thinking) by 3.76
BALSAM - Summarization: Nojoom-AQL-120B (58.82) beat Nojoom-AQL-42B by 2.24
Chess Puzzles (Epoch AI): GPT-6 Astra Pro (Max) (76.0) beat GPT-6 Astra (Max) by 2.0
FEV-Bench - WAPE: Timestral-209M (88.9) beat t0-gamma by 2.0
SloBench - Slovene SuperGLUE: Gizela-31B-it-vit-think (79.87) beat Gemma 3 27B (IT) by 1.93
Open Japanese LLM v2 - JBLiMP: llm-jp-4.1-33B (Thinking) (85.8) beat llm-jp-4-8B-Thinking by 1.81
Open Japanese LLM v2 - JCoLA Out-of-Domain: llm-jp-4.1-8B (Thinking) (82.87) beat llm-jp-4-8B-Thinking by 1.52
MMEB Leaderboard: Wianmu-5B-v0 (60.45) beat WeMM-Embedding-9B by 1.12
Argo-Bench - Solved: Gemini 4 Argon (High) (35.71) beat Claude Opus 5.5 (xHigh) by 0.95
FEV-Bench - WQL: Timestral-209M (89.77) beat t0-gamma by 0.94
FEV-Bench: Timestral-209M (89.59) beat t0-gamma by 0.76
FEV-Bench - MASE: Timestral-209M (88.84) beat t0-gamma by 0.47

The Aggregate Digest — 2026-10-16

2026-10-16T07:35:27.820115+00:00

Daily

New Benchmarks (40)

Doc2Harness - Permissions (Pass rate (%)): leader GPT-6 Astra (71.6), 19 models
Table 2; three independent builds per model in the same fixed builder harness; development document plus interface skeleton; 328 end-to-end test cases; frozen artifact tested by scripted mock model; category means and overall weighted by test count. Metric: Pass rate (%). Higher is better. Source: https://arxiv.org/html/2610.20344v1
Doc2Harness - File operations (Pass rate (%)): leader Grok 4.7 (50.7), 19 models
Table 2; three independent builds per model in the same fixed builder harness; development document plus interface skeleton; 328 end-to-end test cases; frozen artifact tested by scripted mock model; category means and overall weighted by test count. Metric: Pass rate (%). Higher is better. Source: https://arxiv.org/html/2610.20344v1
Doc2Harness - Session state (Pass rate (%)): leader GPT-6 Astra (44.0), 19 models
Table 2; three independent builds per model in the same fixed builder harness; development document plus interface skeleton; 328 end-to-end test cases; frozen artifact tested by scripted mock model; category means and overall weighted by test count. Metric: Pass rate (%). Higher is better. Source: https://arxiv.org/html/2610.20344v1
Doc2Harness - Direct drive (Pass rate (%)): leader GPT-6 Astra (39.4), 19 models
Table 2; three independent builds per model in the same fixed builder harness; development document plus interface skeleton; 328 end-to-end test cases; frozen artifact tested by scripted mock model; category means and overall weighted by test count. Metric: Pass rate (%). Higher is better. Source: https://arxiv.org/html/2610.20344v1
Doc2Harness - Headless (Pass rate (%)): leader GPT-6 Astra (45.0), 19 models
Table 2; three independent builds per model in the same fixed builder harness; development document plus interface skeleton; 328 end-to-end test cases; frozen artifact tested by scripted mock model; category means and overall weighted by test count. Metric: Pass rate (%). Higher is better. Source: https://arxiv.org/html/2610.20344v1
Doc2Harness - Interactive terminal (Pass rate (%)): leader Qwen 3.8 Max (0902) (93.3), 19 models
Table 2; three independent builds per model in the same fixed builder harness; development document plus interface skeleton; 328 end-to-end test cases; frozen artifact tested by scripted mock model; category means and overall weighted by test count. Metric: Pass rate (%). Higher is better. Source: https://arxiv.org/html/2610.20344v1
Doc2Harness - Other (Pass rate (%)): leader DeepSeek V4.1 Flash (80.0), 19 models
Table 2; three independent builds per model in the same fixed builder harness; development document plus interface skeleton; 328 end-to-end test cases; frozen artifact tested by scripted mock model; category means and overall weighted by test count. Metric: Pass rate (%). Higher is better. Source: https://arxiv.org/html/2610.20344v1
Doc2Harness - Overall (Pass rate (%)): leader GPT-6 Astra (53.05), 19 models
Table 2; three independent builds per model in the same fixed builder harness; development document plus interface skeleton; 328 end-to-end test cases; frozen artifact tested by scripted mock model; category means and overall weighted by test count. Metric: Pass rate (%). Higher is better. Source: https://arxiv.org/html/2610.20344v1
ClaimSpy Pilot - Card - Materials (Sign accuracy (0–1)): leader Gemini 3.1 Pro (Preview) (0.6), 3 models
Table 16; 20-claim pilot, 10 per dataset, one run per backbone/condition; ClaimSpy multi-agent scaffold with fixed Claude search/coding agents; agreement with expert gold verdict sign; ambiguity and small sample retained; abstention is not capability score. Metric: Sign accuracy (0–1). Higher is better. Source: https://arxiv.org/html/2610.20457v1
ClaimSpy Pilot - Card - Biomedicine (Sign accuracy (0–1)): leader GPT-5.4 (1.0), 3 models
Table 16; 20-claim pilot, 10 per dataset, one run per backbone/condition; ClaimSpy multi-agent scaffold with fixed Claude search/coding agents; agreement with expert gold verdict sign; ambiguity and small sample retained; abstention is not capability score. Metric: Sign accuracy (0–1). Higher is better. Source: https://arxiv.org/html/2610.20457v1
ClaimSpy Pilot - No card - Materials (Sign accuracy (0–1)): leader Gemini 3.1 Pro (Preview) (0.7), 3 models
Table 16; 20-claim pilot, 10 per dataset, one run per backbone/condition; ClaimSpy multi-agent scaffold with fixed Claude search/coding agents; agreement with expert gold verdict sign; ambiguity and small sample retained; abstention is not capability score. Metric: Sign accuracy (0–1). Higher is better. Source: https://arxiv.org/html/2610.20457v1
ClaimSpy Pilot - No card - Biomedicine (Sign accuracy (0–1)): leader GPT-5.4 (1.0), 3 models
Table 16; 20-claim pilot, 10 per dataset, one run per backbone/condition; ClaimSpy multi-agent scaffold with fixed Claude search/coding agents; agreement with expert gold verdict sign; ambiguity and small sample retained; abstention is not capability score. Metric: Sign accuracy (0–1). Higher is better. Source: https://arxiv.org/html/2610.20457v1
Voice Standard v0.1.3 (claude-opus-5-5; 30s) - Dictation (Published dictation task_quality (1-100), canonical v0.1.3, 30s audio-length bucket; mechanical comparison with truth_text, normalized exact-match/WER/punctuation/capitalization/formatting components; judge name identifies separate run cohort only; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader Claude Opus 4.5 (20251101) (100.0), 44 models
Published dictation task_quality (1-100), canonical v0.1.3, 30s audio-length bucket; mechanical comparison with truth_text, normalized exact-match/WER/punctuation/capitalization/formatting components; judge name identifies separate run cohort only; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (claude-opus-5-5; 30s) - Information Extraction (Published information_extraction task_quality (1-100), canonical v0.1.3, 30s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader GPT-5.6 Sol (94.0), 48 models
Published information_extraction task_quality (1-100), canonical v0.1.3, 30s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (claude-opus-5-5; 30s) - Summarization (Published summarization task_quality (1-100), canonical v0.1.3, 30s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader GPT-5.6 Terra (99.0), 48 models
Published summarization task_quality (1-100), canonical v0.1.3, 30s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (claude-opus-5-5; 30s) - Text Correction (Published text_correction task_quality (1-100), canonical v0.1.3, 30s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Corrections are judged against corrupted missed_text, not the original transcript; old reference-based metrics are excluded. Latency, cost and mixed-judge composites are not scored here.): leader GPT-5.6 Sol (98.0), 54 models
Published text_correction task_quality (1-100), canonical v0.1.3, 30s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Corrections are judged against corrupted missed_text, not the original transcript; old reference-based metrics are excluded. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (claude-opus-5-5; 120s) - Dictation (Published dictation task_quality (1-100), canonical v0.1.3, 120s audio-length bucket; mechanical comparison with truth_text, normalized exact-match/WER/punctuation/capitalization/formatting components; judge name identifies separate run cohort only; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader Gemini 3.5 Flash (99.0), 44 models
Published dictation task_quality (1-100), canonical v0.1.3, 120s audio-length bucket; mechanical comparison with truth_text, normalized exact-match/WER/punctuation/capitalization/formatting components; judge name identifies separate run cohort only; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (claude-opus-5-5; 120s) - Information Extraction (Published information_extraction task_quality (1-100), canonical v0.1.3, 120s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader GPT-6 Luna (High) (93.0), 48 models
Published information_extraction task_quality (1-100), canonical v0.1.3, 120s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (claude-opus-5-5; 120s) - Summarization (Published summarization task_quality (1-100), canonical v0.1.3, 120s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader Claude Sonnet 5.5 (95.0), 48 models
Published summarization task_quality (1-100), canonical v0.1.3, 120s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (claude-opus-5-5; 120s) - Text Correction (Published text_correction task_quality (1-100), canonical v0.1.3, 120s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Corrections are judged against corrupted missed_text, not the original transcript; old reference-based metrics are excluded. Latency, cost and mixed-judge composites are not scored here.): leader Claude Opus 5 (94.0), 54 models
Published text_correction task_quality (1-100), canonical v0.1.3, 120s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Corrections are judged against corrupted missed_text, not the original transcript; old reference-based metrics are excluded. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (claude-opus-5-5; 900s) - Information Extraction (Published information_extraction task_quality (1-100), canonical v0.1.3, 900s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader GPT-6 Luna (86.0), 48 models
Published information_extraction task_quality (1-100), canonical v0.1.3, 900s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (claude-opus-5-5; 900s) - Summarization (Published summarization task_quality (1-100), canonical v0.1.3, 900s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader GPT-6 Luna (Non-reasoning) (91.0), 47 models
Published summarization task_quality (1-100), canonical v0.1.3, 900s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (claude-opus-5-5; 900s) - Text Correction (Published text_correction task_quality (1-100), canonical v0.1.3, 900s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Corrections are judged against corrupted missed_text, not the original transcript; old reference-based metrics are excluded. Latency, cost and mixed-judge composites are not scored here.): leader Claude Opus 5 (94.0), 53 models
Published text_correction task_quality (1-100), canonical v0.1.3, 900s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Corrections are judged against corrupted missed_text, not the original transcript; old reference-based metrics are excluded. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (claude-opus-5-5; 3600s) - Information Extraction (Published information_extraction task_quality (1-100), canonical v0.1.3, 3600s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader Kimi K3 (86.0), 48 models
Published information_extraction task_quality (1-100), canonical v0.1.3, 3600s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (claude-opus-5-5; 3600s) - Summarization (Published summarization task_quality (1-100), canonical v0.1.3, 3600s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader GPT-6 Astra (92.0), 47 models
Published summarization task_quality (1-100), canonical v0.1.3, 3600s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (claude-opus-5-5; 3600s) - Text Correction (Published text_correction task_quality (1-100), canonical v0.1.3, 3600s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Corrections are judged against corrupted missed_text, not the original transcript; old reference-based metrics are excluded. Latency, cost and mixed-judge composites are not scored here.): leader GPT-6 Astra (89.0), 51 models
Published text_correction task_quality (1-100), canonical v0.1.3, 3600s audio-length bucket; reference-free claude-opus-5-5 rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Corrections are judged against corrupted missed_text, not the original transcript; old reference-based metrics are excluded. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (gpt-5.6-sol; 30s) - Dictation (Published dictation task_quality (1-100), canonical v0.1.3, 30s audio-length bucket; mechanical comparison with truth_text, normalized exact-match/WER/punctuation/capitalization/formatting components; judge name identifies separate run cohort only; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader Claude Opus 4.5 (20251101) (100.0), 52 models
Published dictation task_quality (1-100), canonical v0.1.3, 30s audio-length bucket; mechanical comparison with truth_text, normalized exact-match/WER/punctuation/capitalization/formatting components; judge name identifies separate run cohort only; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (gpt-5.6-sol; 30s) - Information Extraction (Published information_extraction task_quality (1-100), canonical v0.1.3, 30s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader Claude Haiku 5.5 (94.0), 53 models
Published information_extraction task_quality (1-100), canonical v0.1.3, 30s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (gpt-5.6-sol; 30s) - Summarization (Published summarization task_quality (1-100), canonical v0.1.3, 30s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader Gemini 3.8 Flash (100.0), 52 models
Published summarization task_quality (1-100), canonical v0.1.3, 30s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (gpt-5.6-sol; 30s) - Text Correction (Published text_correction task_quality (1-100), canonical v0.1.3, 30s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Corrections are judged against corrupted missed_text, not the original transcript; old reference-based metrics are excluded. Latency, cost and mixed-judge composites are not scored here.): leader DeepSeek V4.1 Flash (99.0), 56 models
Published text_correction task_quality (1-100), canonical v0.1.3, 30s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Corrections are judged against corrupted missed_text, not the original transcript; old reference-based metrics are excluded. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (gpt-5.6-sol; 120s) - Dictation (Published dictation task_quality (1-100), canonical v0.1.3, 120s audio-length bucket; mechanical comparison with truth_text, normalized exact-match/WER/punctuation/capitalization/formatting components; judge name identifies separate run cohort only; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader Gemini 4 Argon (99.0), 53 models
Published dictation task_quality (1-100), canonical v0.1.3, 120s audio-length bucket; mechanical comparison with truth_text, normalized exact-match/WER/punctuation/capitalization/formatting components; judge name identifies separate run cohort only; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (gpt-5.6-sol; 120s) - Information Extraction (Published information_extraction task_quality (1-100), canonical v0.1.3, 120s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader nemotron-lightning-3p5-30B-a3B (100.0), 53 models
Published information_extraction task_quality (1-100), canonical v0.1.3, 120s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (gpt-5.6-sol; 120s) - Summarization (Published summarization task_quality (1-100), canonical v0.1.3, 120s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader GPT-5 (100.0), 52 models
Published summarization task_quality (1-100), canonical v0.1.3, 120s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (gpt-5.6-sol; 120s) - Text Correction (Published text_correction task_quality (1-100), canonical v0.1.3, 120s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Corrections are judged against corrupted missed_text, not the original transcript; old reference-based metrics are excluded. Latency, cost and mixed-judge composites are not scored here.): leader GPT-5.6 Luna (95.0), 56 models
Published text_correction task_quality (1-100), canonical v0.1.3, 120s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Corrections are judged against corrupted missed_text, not the original transcript; old reference-based metrics are excluded. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (gpt-5.6-sol; 900s) - Information Extraction (Published information_extraction task_quality (1-100), canonical v0.1.3, 900s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader GPT-6 Luna (High) (89.0), 53 models
Published information_extraction task_quality (1-100), canonical v0.1.3, 900s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (gpt-5.6-sol; 900s) - Summarization (Published summarization task_quality (1-100), canonical v0.1.3, 900s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader GPT-6 Astra (100.0), 52 models
Published summarization task_quality (1-100), canonical v0.1.3, 900s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (gpt-5.6-sol; 900s) - Text Correction (Published text_correction task_quality (1-100), canonical v0.1.3, 900s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Corrections are judged against corrupted missed_text, not the original transcript; old reference-based metrics are excluded. Latency, cost and mixed-judge composites are not scored here.): leader GPT-6 Sol (91.0), 55 models
Published text_correction task_quality (1-100), canonical v0.1.3, 900s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Corrections are judged against corrupted missed_text, not the original transcript; old reference-based metrics are excluded. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (gpt-5.6-sol; 3600s) - Information Extraction (Published information_extraction task_quality (1-100), canonical v0.1.3, 3600s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader Claude Opus 5 (85.0), 53 models
Published information_extraction task_quality (1-100), canonical v0.1.3, 3600s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (gpt-5.6-sol; 3600s) - Summarization (Published summarization task_quality (1-100), canonical v0.1.3, 3600s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.): leader GPT-6 Astra (98.0), 52 models
Published summarization task_quality (1-100), canonical v0.1.3, 3600s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Latency, cost and mixed-judge composites are not scored here.
Voice Standard v0.1.3 (gpt-5.6-sol; 3600s) - Text Correction (Published text_correction task_quality (1-100), canonical v0.1.3, 3600s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Corrections are judged against corrupted missed_text, not the original transcript; old reference-based metrics are excluded. Latency, cost and mixed-judge composites are not scored here.): leader GLM-5.3 (88.0), 54 models
Published text_correction task_quality (1-100), canonical v0.1.3, 3600s audio-length bucket; reference-free gpt-5.6-sol rubric, source transcript as context; normalized 1-5 rubric components; higher is better. Corrections are judged against corrupted missed_text, not the original transcript; old reference-based metrics are excluded. Latency, cost and mixed-judge composites are not scored here.

New Models (2)

Muse Spark Open — Aggregate Rating 1875, #22
- AA GDP.pdf - Legal - Criterion Pass Rate: 87.81 (#2/223)
- AA GDP.pdf - HR - Criterion Pass Rate: 85.57 (#10/223)
- AA-Omniscience Hallucination Rate: 24.07 (#21/577)
- AA SciCode: 58.45 (#23/232)
- AA-Omniscience Index - Science, Engineering & Mathematics: 25.8 (#26/577)
- AA GDP.pdf - Insurance - Criterion Pass Rate: 80.12 (#30/223)
- AA-Omniscience Index - Software Engineering (SWE) - Rust: 78.0 (#34/577)
- AA-Omniscience Index - Health: 13.2 (#37/577)
- AA-Briefcase - PDF - Rubric Pass Rate: 53.45 (#37/229)
- AA GDP.pdf - Criterion Pass Rate: 75.59 (#41/223)
- ...and 51 more
Nemotron 3.5 Super VL — Aggregate Rating 1750, #117
- The Aggregate - Puzzle Portfolio Ladder: 92.6 (#5/20)
- The Aggregate - Inversion and Execution Ladder: 91.7 (#6/18)
- The Aggregate - Day-one Recall Check: 29.4 (#15/34)
- AA-Omniscience Hallucination Rate: 35.85 (#68/577)
- AA-Briefcase - PDF - Rubric Pass Rate: 35.34 (#96/229)
- AA-Omniscience Index - Humanities & Social Sciences: 4.0 (#113/577)
- AA-Briefcase - XLSX - Rubric Pass Rate: 27.05 (#115/229)
- AA-Briefcase - Analytical Quality Elo: 996.67 (#116/229)
- AA CritPt: 10.57 (#117/580)
- AA-Briefcase - DOCX - Rubric Pass Rate: 30.18 (#118/229)
- ...and 50 more

Top-10 New Scores (68)

Claude Opus 5.5 on Conceptual Reasoning Index: 76.07 (#3)
Claude Opus 5.5 on Conceptual Reasoning Index - Argument Evaluation (LMCA): 66.18 (#4)
Claude Opus 5.5 on Conceptual Reasoning Index - Consistency (ACCoRD): 88.42 (#1)
Claude Opus 5.5 on Conceptual Reasoning Index - Decision Theory (DTBench): 95.55 (#10)
Claude Opus 5.5 on ParseBench: 79.85 (#1)
Claude Opus 5.5 on ParseBench - Charts: 70.89 (#1)
Claude Opus 5.5 on ParseBench - Content Faithfulness: 91.72 (#2)
Claude Opus 5.5 on ParseBench - Semantic Formatting: 77.04 (#4)
Claude Opus 5.5 on ParseBench - Tables: 94.25 (#1)
Claude Opus 5.5 on ParseBench - Visual Grounding: 65.33 (#35)
Claude Sonnet 5.5 on Conceptual Reasoning Index: 77.08 (#2)
Claude Sonnet 5.5 on Conceptual Reasoning Index - Consistency (ACCoRD): 85.66 (#4)
Claude Sonnet 5.5 on Conceptual Reasoning Index - Decision Theory (DTBench): 95.55 (#9)
Claude Sonnet 5.5 on ReLE - Agents - BFCL-V3: 63.6 (#50)
Claude Sonnet 5.5 on ReLE - Agents and Tool Use: 64.3 (#46)
Claude Sonnet 5.5 on ReLE - Education: 59.5 (#48)
Claude Sonnet 5.5 on ReLE - Education - Gaokao: 60.3 (#35)
Claude Sonnet 5.5 on ReLE - Education - High School Subjects: 66.5 (#51)
Claude Sonnet 5.5 on ReLE - Education - Middle School Subjects: 49.7 (#55)
Claude Sonnet 5.5 on ReLE - Education - Primary School Subjects: 61.3 (#74)
Claude Sonnet 5.5 on ReLE - Finance: 79.5 (#72)
Claude Sonnet 5.5 on ReLE - Language and Instruction Following: 66.8 (#70)
Claude Sonnet 5.5 on ReLE - Law - Lawyer Qualification Exam: 74.0 (#72)
Claude Sonnet 5.5 on ReLE - Law and Civil Service: 81.0 (#62)
Claude Sonnet 5.5 on ReLE - Medicine - Basic Medical Knowledge: 84.4 (#30)
Claude Sonnet 5.5 on ReLE - Medicine - Physician Exams: 85.5 (#48)
Claude Sonnet 5.5 on ReLE - Medicine and Mental Health: 85.2 (#47)
Claude Sonnet 5.5 on ReLE - Overall: 74.5 (#18)
Claude Sonnet 5.5 on ReLE - Reasoning - BBH: 84.4 (#21)
Claude Sonnet 5.5 on ReLE - Reasoning and Mathematics: 85.7 (#8)
GPT-6.1 Sol on Conceptual Reasoning Index: 72.68 (#22)
GPT-6.1 Sol on Conceptual Reasoning Index - Argument Evaluation (LMCA): 62.78 (#27)
GPT-6.1 Sol on Conceptual Reasoning Index - Consistency (ACCoRD): 80.87 (#26)
GPT-6.1 Sol on Conceptual Reasoning Index - Decision Theory (DTBench): 94.22 (#26)
Gemini 4 Argon on ARC-AGI-1: 98.5 (#9)
Gemini 4 Argon on ARC-AGI-2: 91.25 (#13)
Gemini 4 Argon on ARC-AGI-3: 49.79 (#5)
Gemini 4 Argon on Agents on Rails - Atomic Task Score: 81.15789473684211 (#1)
Gemini 4 Argon on Agents on Rails - Feature Ticket Score: 53.1 (#3)
Gemini 4 Argon on Agents on Rails - Multi-step Task Score: 26.142857142857142 (#6)
Gemini 4 Argon on BenchGecko Score: 88.6 (#9)
Gemini 4 Argon on BridgeBench Overall: 623.0 (#8)
Gemini 4 Argon on Conceptual Reasoning Index: 72.41 (#24)
Gemini 4 Argon on Conceptual Reasoning Index - Argument Evaluation (LMCA): 63.69 (#19)
Gemini 4 Argon on Conceptual Reasoning Index - Consistency (ACCoRD): 77.3 (#48)
Gemini 4 Argon on Conceptual Reasoning Index - Decision Theory (DTBench): 96.0 (#8)
Gemini 4 Argon on Design Arena (Data Viz): 1328.0 (#9)
Gemini 4 Argon on Epoch AI - ECI: 164.56 (#5)
Gemini 4 Argon on Epoch AI - Ebr Bench: 64.76 (#3)
Gemini 4 Argon on Epoch AI - Frontiermath Tiers 1 3 V2: 85.96 (#10)
Gemini 4 Argon on FrontierMath - Tiers 1-3 (v2): 85.96 (#10)
Gemini 4 Argon on Multi-turn Debate (Lechmazur): 1676.4 (#6)
Gemini 4 Argon on ReLE - Agents - BFCL-V3: 72.5 (#8)
Gemini 4 Argon on ReLE - Agents and Tool Use: 73.7 (#2)
Gemini 4 Argon on ReLE - Coding: 60.5 (#47)
Gemini 4 Argon on ReLE - Education - Gaokao: 68.7 (#2)
Gemini 4 Argon on ReLE - Education - Middle School Subjects: 58.0 (#9)
Gemini 4 Argon on ReLE - Education - Primary School Subjects: 71.3 (#2)
Gemini 4 Argon on ReLE - Finance: 90.0 (#3)
Gemini 4 Argon on ReLE - Law - Lawyer Qualification Exam: 77.3 (#47)
Gemini 4 Argon on ReLE - Law and Civil Service: 80.7 (#65)
Gemini 4 Argon on ReLE - Medicine - Basic Medical Knowledge: 88.5 (#4)
Gemini 4 Argon on ReLE - Medicine - Physician Exams: 88.3 (#17)
Gemini 4 Argon on ReLE - Medicine and Mental Health: 88.5 (#21)
Gemini 4 Argon on ReLE - Overall: 77.3 (#2)
Gemini 4 Argon on ReLE - Reasoning - BBH: 87.5 (#4)
Gemini 4 Argon on ReLE - Reasoning and Mathematics: 85.4 (#10)
Gemini 4 Argon on Vals AI Vibe Code Bench 1-100: 21.2 (#8)

New #1 Leaders (11)

BenchX - Pass Rate: gemma-4-31B-it · T=1.0 (90.14) beat GPT-5.6 Sol by 7.2
PinchBench - 147 Tasks - OpenClaw 2026.6.5 (5181e4f): nemotron-3.5-bolt-74B-a9B-bf16 (91.68252517006803) beat Nemotron Lightning 3.5 30B A3B by 4.51
Agents on Rails - Atomic Task Score: Gemini 4 Argon (High) (81.15789473684211) beat Grok 4.7 (High) by 4.05
ReLE - Language and Instruction Following: Gemini 4 Argon (81.6) beat Hy4 preview by 2.8
ReLE - Coding: Claude Sonnet 5.5 (78.0) beat Claude Opus 5.5 by 1.7
ReLE - Education - High School Subjects: Gemini 4 Argon (79.3) beat Gemini 3.1 Pro (Preview) by 1.2
ReLE - Education: Gemini 4 Argon (69.3) beat Gemini 3.1 Pro (Preview) by 0.7
AI Coding Daily (OpenCode) - React-TS Code Quality: Mistral Large 4 (High) (19.33) beat DeepSeek V4.1 Flash (High) by 0.66
EnterpriseRAG Bench - Correctness: Flexor.ai (+ GPT-6.1 Sol) (93.0) beat CDL (Causal Dynamics Lab) by 0.6
EnterpriseRAG Bench: Flexor.ai (+ GPT-6.1 Sol) (91.48) beat CDL (Causal Dynamics Lab) by 0.57
Conceptual Reasoning Index - Argument Evaluation (LMCA): Claude Sonnet 5.5 (68.26) beat Claude Opus 5.5 by 0.03

The Aggregate Digest — 2026-10-15

2026-10-15T10:33:01.475276+00:00

Daily

New Benchmarks (110)

Epoch AI - Spireclimb (Score): leader GPT-6 Astra (Max) (25.67), 7 models
Epoch AI's computer-use evaluation of Slay the Spire 1 and 2 at maximum difficulty. Models act through screenshots, keyboard and mouse, with the game wiki available. Score is the equally weighted mean of both games' win percentages; higher is better. Source: epoch.ai/benchmarks/spireclimb.
Epoch AI - Spireclimb Sts1 A20 Heart (Score): leader GPT-6.1 Sol (Max) (20.0), 7 models
The Slay the Spire 1 track of Epoch AI's SpireClimb: win percentage across 60 seeded runs, with four characters at Ascension 20 and the Heart defeated. Models use a computer-use harness at maximum reasoning effort. Higher is better. Source: epoch.ai/benchmarks/spireclimb.
Epoch AI - Spireclimb Sts2 A10 (Score): leader GPT-6 Astra (Max) (34.67), 7 models
The Slay the Spire 2 track of Epoch AI's SpireClimb: win percentage across 75 seeded runs, with five characters at Ascension 10. The game is fixed at version 0.107.1 and models use a computer-use harness at maximum reasoning effort. Higher is better. Source: epoch.ai/benchmarks/spireclimb.
Agents on Rails - Atomic Task Score (Mean Task Score (%)): leader Grok 4.7 (High) (77.10526315789474), 11 models
Partial-credit hidden-test scores over 19 atomic Rails tasks, using the fixed lemans/miniswen agent. Runs are averaged within each task, then tasks receive equal weight; each model retains its explicit reasoning effort. This October 2026 protocol is distinct from the legacy binary-success board.
Agents on Rails - Feature Ticket Score (Mean Task Score (%)): leader Claude Opus 5.5 (xHigh) (59.15), 11 models
Partial-credit hidden-test scores over 20 feature-size Rails tickets, using lemans/miniswen. Every task has equal weight after its runs are averaged; native run counts and explicit effort are retained. This is distinct from the legacy successful-run percentage.
Agents on Rails - Multi-step Task Score (Mean Task Score (%)): leader Claude Opus 5.5 (xHigh) (72.14285714285714), 11 models
Mean partial-credit score on seven long-horizon tasks in new and existing Rails apps using lemans/miniswen. Required-check failures earn zero; optional checks add credit. Task means have equal weight, with run counts and reasoning effort retained.
Guesswork 2026-10 (MAE (z-score units)): leader GPT-6.1 Sol (0.5687), 33 models
The October 2026 round of this site's benchmark-score forecasting contest. Models predict new model and benchmark results before publication and are evaluated on common cells against The Aggregate statistical baseline. Score is mean absolute error in per-benchmark z-score units; lower is better. Missing usable predictions receive the benchmark mean. Source: theaggregate.ai/prediction-duel.
Patch Performance - SWE-bench Measurable Subset (Resolved (%)): leader Claude Opus 5 (95.652174), 4 models
Resolution on a pre-filtered measurable subset: 230 SWE-bench and 109 SWE-bench-Live tasks, mini-SWE-agent, 250 steps, trajectories August 17–18 2026. This is the full 339-task selection before the later solved-patch runtime analysis; it is not either benchmark’s full leaderboard. Percentages are computed from the printed resolved counts and their explicit denominators.
Patch Performance - SWE-bench-Live Measurable Subset (Resolved (%)): leader Claude Opus 5 (69.724771), 4 models
Resolution on a pre-filtered measurable subset: 230 SWE-bench and 109 SWE-bench-Live tasks, mini-SWE-agent, 250 steps, trajectories August 17–18 2026. This is the full 339-task selection before the later solved-patch runtime analysis; it is not either benchmark’s full leaderboard. Percentages are computed from the printed resolved counts and their explicit denominators.
Patch Performance - Combined Measurable Subset (Resolved (%)): leader Claude Opus 5 (87.315634), 4 models
Resolution on a pre-filtered measurable subset: 230 SWE-bench and 109 SWE-bench-Live tasks, mini-SWE-agent, 250 steps, trajectories August 17–18 2026. This is the full 339-task selection before the later solved-patch runtime analysis; it is not either benchmark’s full leaderboard. Percentages are computed from the printed resolved counts and their explicit denominators.
APEX-Harm - Task Utility - Static Prompt Injection (Mean Task Score (0–1)): leader Gemini 3.8 Flash (High) (0.871), 12 models
Task utility under injection attacks in the APEX-Agents harness: 180 instances derived from 15 finance, consulting and legal tasks. Each run has 150 steps. Scores are 0–1 task-grader means, not attack success or exposure. Eight attack conditions stay separate; the reported overall score is also retained.
APEX-Harm - Task Utility - Static Script Injection (Mean Task Score (0–1)): leader Gemini 3.8 Flash (High) (0.871), 12 models
Task utility under injection attacks in the APEX-Agents harness: 180 instances derived from 15 finance, consulting and legal tasks. Each run has 150 steps. Scores are 0–1 task-grader means, not attack success or exposure. Eight attack conditions stay separate; the reported overall score is also retained.
APEX-Harm - Task Utility - Dynamic Prompt Injection (Mean Task Score (0–1)): leader Gemini 3.8 Flash (High) (0.882), 12 models
Task utility under injection attacks in the APEX-Agents harness: 180 instances derived from 15 finance, consulting and legal tasks. Each run has 150 steps. Scores are 0–1 task-grader means, not attack success or exposure. Eight attack conditions stay separate; the reported overall score is also retained.
APEX-Harm - Task Utility - Dynamic Script Injection Without Permission (Mean Task Score (0–1)): leader Gemini 3.8 Flash (High) (0.897), 12 models
Task utility under injection attacks in the APEX-Agents harness: 180 instances derived from 15 finance, consulting and legal tasks. Each run has 150 steps. Scores are 0–1 task-grader means, not attack success or exposure. Eight attack conditions stay separate; the reported overall score is also retained.
APEX-Harm - Task Utility - Dynamic Script Injection With Permission (Mean Task Score (0–1)): leader Gemini 3.8 Flash (High) (0.879), 12 models
Task utility under injection attacks in the APEX-Agents harness: 180 instances derived from 15 finance, consulting and legal tasks. Each run has 150 steps. Scores are 0–1 task-grader means, not attack success or exposure. Eight attack conditions stay separate; the reported overall score is also retained.
APEX-Harm - Task Utility - Task-Specific Replacement Direct Action (Mean Task Score (0–1)): leader Gemini 3.8 Flash (High) (0.877), 12 models
Task utility under injection attacks in the APEX-Agents harness: 180 instances derived from 15 finance, consulting and legal tasks. Each run has 150 steps. Scores are 0–1 task-grader means, not attack success or exposure. Eight attack conditions stay separate; the reported overall score is also retained.
APEX-Harm - Task Utility - Task-Specific Replacement Runtime Information (Mean Task Score (0–1)): leader Gemini 3.8 Flash (High) (0.886), 12 models
Task utility under injection attacks in the APEX-Agents harness: 180 instances derived from 15 finance, consulting and legal tasks. Each run has 150 steps. Scores are 0–1 task-grader means, not attack success or exposure. Eight attack conditions stay separate; the reported overall score is also retained.
APEX-Harm - Task Utility - Attacker-Provided Generic Script (Mean Task Score (0–1)): leader Gemini 3.8 Flash (High) (0.867), 12 models
Task utility under injection attacks in the APEX-Agents harness: 180 instances derived from 15 finance, consulting and legal tasks. Each run has 150 steps. Scores are 0–1 task-grader means, not attack success or exposure. Eight attack conditions stay separate; the reported overall score is also retained.
APEX-Harm - Task Utility - Overall (Mean Task Score (0–1)): leader Gemini 3.8 Flash (High) (0.877), 12 models
Task utility under injection attacks in the APEX-Agents harness: 180 instances derived from 15 finance, consulting and legal tasks. Each run has 150 steps. Scores are 0–1 task-grader means, not attack success or exposure. Eight attack conditions stay separate; the reported overall score is also retained.
OmniStepBench - Text Single-Step (Audited Accuracy (%)): leader Claude Opus 5.5 (99.2), 42 models
Audited task accuracy in a shared OpenClaw harness over 233 tasks selected from 22 benchmarks. Six modality/execution groups and task-weighted text/multimodal means remain separate. API and local deployment provenance is retained; text-only models have no multimodal score.
OmniStepBench - Text Multi-Step (Audited Accuracy (%)): leader Claude Opus 5.5 (84.4), 42 models
Audited task accuracy in a shared OpenClaw harness over 233 tasks selected from 22 benchmarks. Six modality/execution groups and task-weighted text/multimodal means remain separate. API and local deployment provenance is retained; text-only models have no multimodal score.
OmniStepBench - Text Multi-Step with Tools (Audited Accuracy (%)): leader GPT-6.1 Sol (71.3), 42 models
Audited task accuracy in a shared OpenClaw harness over 233 tasks selected from 22 benchmarks. Six modality/execution groups and task-weighted text/multimodal means remain separate. API and local deployment provenance is retained; text-only models have no multimodal score.
OmniStepBench - Text Average (Audited Accuracy (%)): leader GPT-6.1 Sol (81.2), 42 models
Audited task accuracy in a shared OpenClaw harness over 233 tasks selected from 22 benchmarks. Six modality/execution groups and task-weighted text/multimodal means remain separate. API and local deployment provenance is retained; text-only models have no multimodal score.
OmniStepBench - Multimodal Single-Step (Audited Accuracy (%)): leader GPT-5.5 (90.0), 34 models
Audited task accuracy in a shared OpenClaw harness over 233 tasks selected from 22 benchmarks. Six modality/execution groups and task-weighted text/multimodal means remain separate. API and local deployment provenance is retained; text-only models have no multimodal score.
OmniStepBench - Multimodal Multi-Step (Audited Accuracy (%)): leader GPT-6 Astra (94.2), 34 models
Audited task accuracy in a shared OpenClaw harness over 233 tasks selected from 22 benchmarks. Six modality/execution groups and task-weighted text/multimodal means remain separate. API and local deployment provenance is retained; text-only models have no multimodal score.
OmniStepBench - Multimodal Multi-Step with Tools (Audited Accuracy (%)): leader GPT-6 Astra (67.6), 34 models
Audited task accuracy in a shared OpenClaw harness over 233 tasks selected from 22 benchmarks. Six modality/execution groups and task-weighted text/multimodal means remain separate. API and local deployment provenance is retained; text-only models have no multimodal score.
OmniStepBench - Multimodal Average (Audited Accuracy (%)): leader GPT-6 Astra (78.0), 34 models
Audited task accuracy in a shared OpenClaw harness over 233 tasks selected from 22 benchmarks. Six modality/execution groups and task-weighted text/multimodal means remain separate. API and local deployment provenance is retained; text-only models have no multimodal score.
ReplayBench T01 - LangGraph - Banking (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - LangGraph - Calendar (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - LangGraph - DevOps (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - LangGraph - Email (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - LangGraph - Files (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - LangGraph - Identity (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - LangGraph - Security (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - LlamaIndex - Banking (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - LlamaIndex - Calendar (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - LlamaIndex - DevOps (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - LlamaIndex - Email (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - LlamaIndex - Files (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - LlamaIndex - Identity (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - LlamaIndex - Security (Final State Error Rate (%)): leader Claude Opus 5 (20.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - OpenAI SDK - Banking (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - OpenAI SDK - Calendar (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - OpenAI SDK - DevOps (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - OpenAI SDK - Email (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - OpenAI SDK - Files (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - OpenAI SDK - Identity (Final State Error Rate (%)): leader Claude Opus 5 (0.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
ReplayBench T01 - OpenAI SDK - Security (Final State Error Rate (%)): leader Claude Opus 5 (20.0), 8 models
Final-state error after replaying an authentic tool result in the wrong task context. T01 attribution experiment: eight models, three separate frameworks, seven separate scenarios and ten valid trials per cell. Lower error is better. This is tool-result misbinding, not injected instructions.
SPaRC Inspect 1-A Full Test - Overall (Accuracy (%)): leader GPT-6.1 Sol (Medium) (83.6), 4 models
Solve all 500 spatial pathfinding test puzzles under Inspect task version 1-A at a pinned dataset revision. A path must satisfy every rule. Overall and five difficulty levels are accuracy; invalid paths are errors. Only complete, error-free runs enter this protocol; explicit API identities, reasoning and output limits remain attached.
SPaRC Inspect 1-A Full Test - Level 1 (Accuracy (%)): leader GPT-6.1 Sol (Medium) (100.0), 4 models
Solve all 500 spatial pathfinding test puzzles under Inspect task version 1-A at a pinned dataset revision. A path must satisfy every rule. Overall and five difficulty levels are accuracy; invalid paths are errors. Only complete, error-free runs enter this protocol; explicit API identities, reasoning and output limits remain attached.
SPaRC Inspect 1-A Full Test - Level 2 (Accuracy (%)): leader GPT-6.1 Sol (Medium) (94.1), 4 models
Solve all 500 spatial pathfinding test puzzles under Inspect task version 1-A at a pinned dataset revision. A path must satisfy every rule. Overall and five difficulty levels are accuracy; invalid paths are errors. Only complete, error-free runs enter this protocol; explicit API identities, reasoning and output limits remain attached.
SPaRC Inspect 1-A Full Test - Level 3 (Accuracy (%)): leader GPT-6.1 Sol (Medium) (87.6), 4 models
Solve all 500 spatial pathfinding test puzzles under Inspect task version 1-A at a pinned dataset revision. A path must satisfy every rule. Overall and five difficulty levels are accuracy; invalid paths are errors. Only complete, error-free runs enter this protocol; explicit API identities, reasoning and output limits remain attached.
SPaRC Inspect 1-A Full Test - Level 4 (Accuracy (%)): leader GPT-6.1 Sol (Medium) (82.6), 4 models
Solve all 500 spatial pathfinding test puzzles under Inspect task version 1-A at a pinned dataset revision. A path must satisfy every rule. Overall and five difficulty levels are accuracy; invalid paths are errors. Only complete, error-free runs enter this protocol; explicit API identities, reasoning and output limits remain attached.
SPaRC Inspect 1-A Full Test - Level 5 (Accuracy (%)): leader GPT-6.1 Sol (Medium) (49.4), 4 models
Solve all 500 spatial pathfinding test puzzles under Inspect task version 1-A at a pinned dataset revision. A path must satisfy every rule. Overall and five difficulty levels are accuracy; invalid paths are errors. Only complete, error-free runs enter this protocol; explicit API identities, reasoning and output limits remain attached.
SPaRC Inspect 1-A Full Test - Invalid Path (Invalid Path Rate (%)): leader GPT-6.1 Sol (Medium) (7.2), 4 models
Solve all 500 spatial pathfinding test puzzles under Inspect task version 1-A at a pinned dataset revision. A path must satisfy every rule. Overall and five difficulty levels are accuracy; invalid paths are errors. Only complete, error-free runs enter this protocol; explicit API identities, reasoning and output limits remain attached.
Google Argon Report - PostTrainBench v1.1 OpenCode 10h H100 (Score (%)): leader Claude Opus 5.5 (49.3), 4 models
Google Gemini 4 Argon October 2026 release report. All models: OpenCode, ten hours, one NVIDIA H100; weighted aggregate over four base models and seven benchmarks. All four models evaluated by Google. Maximum available or best available reasoning, with exact peer API effort unreported; not interchangeable with borrowed leaderboard rows.
Google Argon Report - LABBench 2 Terminal and Internet (Score (%)): leader Gemini 4 Argon (88.8), 4 models
Google Gemini 4 Argon October 2026 release report. All models: Linux terminal, bioinformatics tools, Python, R and internet access. All four models evaluated by Google. Maximum available or best available reasoning, with exact peer API effort unreported; not interchangeable with borrowed leaderboard rows.
Google Argon Report - GraphWalks BFS Up to 128k (Score (%)): leader Gemini 4 Argon (99.7), 4 models
Google Gemini 4 Argon October 2026 release report. All models: 650 GraphWalks items up to 128k tokens; BFS F1. All four models evaluated by Google. Maximum available or best available reasoning, with exact peer API effort unreported; not interchangeable with borrowed leaderboard rows.
Google Argon Report - GraphWalks BFS 256k to 1M (Score (%)): leader Gemini 4 Argon (84.2), 4 models
Google Gemini 4 Argon October 2026 release report. All models: 200 GraphWalks items from 256k to 1M tokens; BFS F1. All four models evaluated by Google. Maximum available or best available reasoning, with exact peer API effort unreported; not interchangeable with borrowed leaderboard rows.
Google Argon Report - LVBench API Frame Limits (Score (%)): leader Gemini 4 Argon (91.7), 4 models
Google Gemini 4 Argon October 2026 release report. No tools. Gemini: 1 FPS; Astra: 800 frames; Fable: 300 frames; Opus: 600 frames, reflecting provider API limits. All four models evaluated by Google. Maximum available or best available reasoning, with exact peer API effort unreported; not interchangeable with borrowed leaderboard rows.
AA-3D (Elo): leader Claude Sonnet 5.5 (Max, Default Fallback) (1194.2268), 21 models
Build interactive three.js scenes from 55 briefs in an offline Stirrup sandbox. Blind panel votes yield an unbounded Elo score. Preserve the model’s printed reasoning and fallback setting.
ArXivMath August 2026 No Tools (Anthropic Internal Harness) (Accuracy (%)): leader Claude Opus 5.5 (Max) (91.2), 16 models
Accuracy on MathArena's August 2026 set of 57 research-mathematics problems, averaged over four attempts per problem in Anthropic's internal harness without safeguard classifiers. No tools. Distinct from MathArena's two-attempt vendor-harness leaderboard and earlier monthly sets.
ArXivMath August 2026 Tools (Anthropic Internal Harness) (Accuracy (%)): leader Claude Opus 5.5 (Max) (96.9), 16 models
Accuracy on MathArena's August 2026 set of 57 research-mathematics problems, averaged over four attempts per problem in Anthropic's internal harness without safeguard classifiers. Code execution sandbox without internet access. Distinct from MathArena's two-attempt vendor-harness leaderboard and earlier monthly sets.
Anthropic Child Safety Single-Turn Harmful Requests - Harmless Rate (API) (Harmless Response Rate (%)): leader Claude Opus 5 (100.0), 4 models
Percentage of harmless responses to single-turn harmful child-safety requests across tested languages, using the API without a system prompt. Higher is better. Source: Claude Opus 5.5 system card, Table 4.2.A; distinct from benign-request over-refusal.
Terminal-Bench-Science 0.1 (Anthropic Internal Harness) (Resolved (%)): leader Claude Opus 5.5 (Max) (58.7), 3 models
Task success on 70 scientific research workflows in terminal environments, graded by hidden tests of produced artifacts. These Anthropic internal-harness results use max effort. The Opus 5.5 run averages ten trials per task with safeguards and its documented server-side fallback. Higher is better. Source: Claude Opus 5.5 system card.
Toolathlon Verified Pass@1 (Anthropic Internal Harness) (Pass@1 (%)): leader Claude Opus 5 (Max) (80.6), 6 models
Mean single-trial success over 108 Toolathlon-Verified tasks, with three trials per task in Anthropic’s internal harness at max effort. Execution-based checks assess tool use across 32 applications. Higher is better; model labels retain documented classifier-fallback conditions. Source: Claude Opus 5.5 system card. Distinct from pass@3 and pass^3.
DKGI Pi - Amazon (Accuracy (%)): leader GPT-6 Astra (79.4), 2 models
DKGI Pi - Amazon. Knowledge graph question answering via file tools;Pi harness;source reported accuracy. Codex/Luna only row is not pooled into this protocol. Metric: Accuracy (%). Primary published table; source configurations remain distinct.
DKGI Pi - MAG (Accuracy (%)): leader GPT-6 Astra (91.8), 2 models
DKGI Pi - MAG. Knowledge graph question answering via file tools;Pi harness;source reported accuracy. Codex/Luna only row is not pooled into this protocol. Metric: Accuracy (%). Primary published table; source configurations remain distinct.
DKGI Pi - Prime (Accuracy (%)): leader GPT-6 Astra (74.5), 2 models
DKGI Pi - Prime. Knowledge graph question answering via file tools;Pi harness;source reported accuracy. Codex/Luna only row is not pooled into this protocol. Metric: Accuracy (%). Primary published table; source configurations remain distinct.
LLM Stats (AutomationBench; API default fallbacks enabled) (Score (%)): leader Claude Sonnet 5.5 (44.7), 1 models
LLM Stats source-attributed result collection. Protocol: API default fallbacks enabled. Scores and source method notes retained without rescaling beyond the published percent conversion. Unverified protocols remain separate source observations; they are not assumed equivalent to another model or benchmark setting.
LLM Stats (AutomationBench; No fallback models; safeguards enabled) (Score (%)): leader Claude Opus 5.5 (40.0), 1 models
LLM Stats source-attributed result collection. Protocol: No fallback models; safeguards enabled. Scores and source method notes retained without rescaling beyond the published percent conversion. Unverified protocols remain separate source observations; they are not assumed equivalent to another model or benchmark setting.
LLM Stats (AutomationBench; Private held-out set) (Score (%)): leader Gemini 4 Argon (51.3), 5 models
LLM Stats source-attributed result collection. Protocol: Private held-out set. Scores and source method notes retained without rescaling beyond the published percent conversion. Unverified protocols remain separate source observations; they are not assumed equivalent to another model or benchmark setting.
LLM Stats (AutomationBench; Protocol unverified — claude-opus-5) (Score (%)): leader Claude Opus 5 (26.0), 1 models
LLM Stats source-attributed result collection. Protocol: Protocol unverified — claude-opus-5. Scores and source method notes retained without rescaling beyond the published percent conversion. Unverified protocols remain separate source observations; they are not assumed equivalent to another model or benchmark setting.
LLM Stats (AutomationBench; Protocol unverified — gpt-6-astra) (Score (%)): leader GPT-6 Astra (41.4), 1 models
LLM Stats source-attributed result collection. Protocol: Protocol unverified — gpt-6-astra. Scores and source method notes retained without rescaling beyond the published percent conversion. Unverified protocols remain separate source observations; they are not assumed equivalent to another model or benchmark setting.
LLM Stats (Graphwalks BFS >128k; 256K–1M range (F1)) (Score (%)): leader Gemini 4 Argon (84.2), 3 models
LLM Stats source-attributed result collection. Protocol: 256K–1M range (F1). Scores and source method notes retained without rescaling beyond the published percent conversion. Unverified protocols remain separate source observations; they are not assumed equivalent to another model or benchmark setting.
LLM Stats (HealthBench; Length-adjusted) (Score (%)): leader Claude Sonnet 5.5 (65.4), 8 models
LLM Stats source-attributed result collection. Protocol: Length-adjusted. Scores and source method notes retained without rescaling beyond the published percent conversion. Unverified protocols remain separate source observations; they are not assumed equivalent to another model or benchmark setting.
OmniCompass-VL - Temporal Grounding - Consistency Score (Consistency score (%)): leader Gemini 3.8 Flash (27.1), 36 models
OmniCompass-VL - Temporal Grounding - Consistency Score. Matched text-only/text-plus-image target specifications; consistency requires both target forms correct. 820 visual-language videos, 2460 task instances. Source-specific frame budgets retained in primary table; no fusion with single-condition accuracy. Metric: Consistency score (%). Primary published table; source configurations remain distinct.
OmniCompass-VL - Spatial Grounding - Consistency Score (Consistency score (%)): leader Seed-2.1-Pro-260915 (34.9), 36 models
OmniCompass-VL - Spatial Grounding - Consistency Score. Matched text-only/text-plus-image target specifications; consistency requires both target forms correct. 820 visual-language videos, 2460 task instances. Source-specific frame budgets retained in primary table; no fusion with single-condition accuracy. Metric: Consistency score (%). Primary published table; source configurations remain distinct.
OmniCompass-VL - Question Answering - Consistency Score (Consistency score (%)): leader Gemini 3.8 Flash (69.1), 36 models
OmniCompass-VL - Question Answering - Consistency Score. Matched text-only/text-plus-image target specifications; consistency requires both target forms correct. 820 visual-language videos, 2460 task instances. Source-specific frame budgets retained in primary table; no fusion with single-condition accuracy. Metric: Consistency score (%). Primary published table; source configurations remain distinct.
OmniCompass-VL - Overall - Consistency Score (Consistency score (%)): leader Gemini 3.8 Flash (42.8), 36 models
OmniCompass-VL - Overall - Consistency Score. Matched text-only/text-plus-image target specifications; consistency requires both target forms correct. 820 visual-language videos, 2460 task instances. Source-specific frame budgets retained in primary table; no fusion with single-condition accuracy. Metric: Consistency score (%). Primary published table; source configurations remain distinct.
Physics Expert Audit - PHYBench - mean@4 (Validated/repaired accuracy (%, mean@4)): leader GPT-5.6 Sol (High) (90.23), 3 models
Expert-validated and repaired physics questions, with the published mean@4 scoring protocol. Distinct corrected population; public-source tasks use no tools, expert tasks use Claude Code/Codex for Fable/Sol and no tools for Gemini. Not fused into the original benchmark.
Physics Expert Audit - PHYBench - pass@4 (Validated/repaired accuracy (%, pass@4)): leader GPT-5.6 Sol (High) (95.4), 3 models
Expert-validated and repaired physics questions, with the published pass@4 scoring protocol. Distinct corrected population; public-source tasks use no tools, expert tasks use Claude Code/Codex for Fable/Sol and no tools for Gemini. Not fused into the original benchmark.
Physics Expert Audit - PRISM-Physics - mean@4 (Validated/repaired accuracy (%, mean@4)): leader GPT-5.6 Sol (High) (94.59), 3 models
Expert-validated and repaired physics questions, with the published mean@4 scoring protocol. Distinct corrected population; public-source tasks use no tools, expert tasks use Claude Code/Codex for Fable/Sol and no tools for Gemini. Not fused into the original benchmark.
Physics Expert Audit - PRISM-Physics - pass@4 (Validated/repaired accuracy (%, pass@4)): leader GPT-5.6 Sol (High) (95.95), 3 models
Expert-validated and repaired physics questions, with the published pass@4 scoring protocol. Distinct corrected population; public-source tasks use no tools, expert tasks use Claude Code/Codex for Fable/Sol and no tools for Gemini. Not fused into the original benchmark.
Physics Expert Audit - UGPhysics - mean@4 (Validated/repaired accuracy (%, mean@4)): leader GPT-5.6 Sol (High) (92.07), 3 models
Expert-validated and repaired physics questions, with the published mean@4 scoring protocol. Distinct corrected population; public-source tasks use no tools, expert tasks use Claude Code/Codex for Fable/Sol and no tools for Gemini. Not fused into the original benchmark.
Physics Expert Audit - UGPhysics - pass@4 (Validated/repaired accuracy (%, pass@4)): leader GPT-5.6 Sol (High) (93.9), 3 models
Expert-validated and repaired physics questions, with the published pass@4 scoring protocol. Distinct corrected population; public-source tasks use no tools, expert tasks use Claude Code/Codex for Fable/Sol and no tools for Gemini. Not fused into the original benchmark.
PutnamBench - Coq, Answer Supplied ($45/task) (Problems solved; Coq, Answer Supplied; reported compute budget: $45/task): leader Claude Fable 5.1 (Max) (8.0), 1 models
Number of Putnam competition problems reported solved in Coq, with numerical answers supplied in the theorem statement where required. The source reports compute budget '$45/task'. Higher is better; counts stay separate from other languages, answer conditions and budgets. Reported corpus revision: 4dbe26ef21563af851eedaeb82d936fe1f94fc52. Five proofs use existing statements; three use corrected formalizations documented by the source. The proof-artifact commit is distinct from the benchmark baseline.
RSI4EO - OpenEarthMap - 6 labeled tiles (mIoU (%)): leader GPT-6 Astra (60.0), 2 models
RSI4EO - OpenEarthMap - 6 labeled tiles. Agent designs/trains a remote-sensing model;500 held-out OpenEarthMap tiles. Training labels=6, unlabeled=4,194. Source Table4; full pipeline result, not direct image classification by the language model. Metric: mIoU (%). Primary published table; source configurations remain distinct.
RSI4EO - OpenEarthMap - 300 labeled tiles (mIoU (%)): leader Claude Opus 5.5 (67.8), 2 models
RSI4EO - OpenEarthMap - 300 labeled tiles. Agent designs/trains a remote-sensing model;500 held-out OpenEarthMap tiles. Training labels=300, unlabeled=0. Source Table4; full pipeline result, not direct image classification by the language model. Metric: mIoU (%). Primary published table; source configurations remain distinct.
RSI4EO - OpenEarthMap - 600 labeled tiles (mIoU (%)): leader GPT-6 Astra (69.4), 2 models
RSI4EO - OpenEarthMap - 600 labeled tiles. Agent designs/trains a remote-sensing model;500 held-out OpenEarthMap tiles. Training labels=600, unlabeled=0. Source Table4; full pipeline result, not direct image classification by the language model. Metric: mIoU (%). Primary published table; source configurations remain distinct.
RSI4EO - OpenEarthMap - 2,700 labeled tiles (mIoU (%)): leader GPT-6 Astra (71.6), 2 models
RSI4EO - OpenEarthMap - 2,700 labeled tiles. Agent designs/trains a remote-sensing model;500 held-out OpenEarthMap tiles. Training labels=2,700, unlabeled=1,500. Source Table4; full pipeline result, not direct image classification by the language model. Metric: mIoU (%). Primary published table; source configurations remain distinct.
SEAR - robosuite - Lineage (Test success (%)): leader Claude Opus 5 (97.1), 5 models
SEAR - robosuite - Lineage. Mean over three runs (two marked in paper); Lineage code-library protocol. RoboLab full task population; its 58-task DeepSeek lineage result is not pooled.. Reported metric: Test success (%). Primary published table, with source configuration boundaries preserved.
SEAR - robosuite - Vanilla (Test success (%)): leader Claude Opus 5 (93.8), 5 models
SEAR - robosuite - Vanilla. Mean over three runs (two marked in paper); Vanilla code-library protocol. RoboLab full task population; its 58-task DeepSeek lineage result is not pooled.. Reported metric: Test success (%). Primary published table, with source configuration boundaries preserved.
SEAR - LIBERO - Lineage (Test success (%)): leader Claude Opus 5 (94.4), 5 models
SEAR - LIBERO - Lineage. Mean over three runs (two marked in paper); Lineage code-library protocol. RoboLab full task population; its 58-task DeepSeek lineage result is not pooled.. Reported metric: Test success (%). Primary published table, with source configuration boundaries preserved.
SEAR - LIBERO - Vanilla (Test success (%)): leader Claude Opus 5 (95.2), 5 models
SEAR - LIBERO - Vanilla. Mean over three runs (two marked in paper); Vanilla code-library protocol. RoboLab full task population; its 58-task DeepSeek lineage result is not pooled.. Reported metric: Test success (%). Primary published table, with source configuration boundaries preserved.
SEAR - LIBERO-PRO - Lineage (Test success (%)): leader Claude Opus 5 (91.8), 5 models
SEAR - LIBERO-PRO - Lineage. Mean over three runs (two marked in paper); Lineage code-library protocol. RoboLab full task population; its 58-task DeepSeek lineage result is not pooled.. Reported metric: Test success (%). Primary published table, with source configuration boundaries preserved.
SEAR - LIBERO-PRO - Vanilla (Test success (%)): leader Claude Opus 5 (93.0), 5 models
SEAR - LIBERO-PRO - Vanilla. Mean over three runs (two marked in paper); Vanilla code-library protocol. RoboLab full task population; its 58-task DeepSeek lineage result is not pooled.. Reported metric: Test success (%). Primary published table, with source configuration boundaries preserved.
SEAR - LIBERO-RIG - Lineage (Test success (%)): leader Claude Opus 5 (97.0), 5 models
SEAR - LIBERO-RIG - Lineage. Mean over three runs (two marked in paper); Lineage code-library protocol. RoboLab full task population; its 58-task DeepSeek lineage result is not pooled.. Reported metric: Test success (%). Primary published table, with source configuration boundaries preserved.
SEAR - LIBERO-RIG - Vanilla (Test success (%)): leader Claude Opus 5 (96.7), 5 models
SEAR - LIBERO-RIG - Vanilla. Mean over three runs (two marked in paper); Vanilla code-library protocol. RoboLab full task population; its 58-task DeepSeek lineage result is not pooled.. Reported metric: Test success (%). Primary published table, with source configuration boundaries preserved.
SEAR - RoboLab - Lineage (Test success (%)): leader Claude Opus 5 (68.0), 4 models
SEAR - RoboLab - Lineage. Mean over three runs (two marked in paper); Lineage code-library protocol. RoboLab full task population; its 58-task DeepSeek lineage result is not pooled.. Reported metric: Test success (%). Primary published table, with source configuration boundaries preserved.
SEAR - RoboLab - Vanilla (Test success (%)): leader Claude Opus 5 (61.9), 5 models
SEAR - RoboLab - Vanilla. Mean over three runs (two marked in paper); Vanilla code-library protocol. RoboLab full task population; its 58-task DeepSeek lineage result is not pooled.. Reported metric: Test success (%). Primary published table, with source configuration boundaries preserved.
STEMBench120 - Average (Research progress (0–4)): leader GPT-6 Astra (Harness: Codex) (xHigh) (2.809), 10 models
STEMBench120 - Average. v1.0.0; 120 tasks across six STEM domains; Codex tool runs distinct from closed-book rows; correctness-adjusted research progress, not answer accuracy. Reported metric: Research progress (0–4). Primary published table, with source configuration boundaries preserved.
STEMBench120 - Biology (Research progress (0–4)): leader GPT-6 Astra (Harness: Codex) (xHigh) (2.709), 10 models
STEMBench120 - Biology. v1.0.0; 120 tasks across six STEM domains; Codex tool runs distinct from closed-book rows; correctness-adjusted research progress, not answer accuracy. Reported metric: Research progress (0–4). Primary published table, with source configuration boundaries preserved.
STEMBench120 - Chemistry (Research progress (0–4)): leader GPT-6 Astra (Harness: Codex) (xHigh) (2.777), 10 models
STEMBench120 - Chemistry. v1.0.0; 120 tasks across six STEM domains; Codex tool runs distinct from closed-book rows; correctness-adjusted research progress, not answer accuracy. Reported metric: Research progress (0–4). Primary published table, with source configuration boundaries preserved.
STEMBench120 - Physics (Research progress (0–4)): leader GPT-6 Astra (Harness: Codex) (xHigh) (3.025), 10 models
STEMBench120 - Physics. v1.0.0; 120 tasks across six STEM domains; Codex tool runs distinct from closed-book rows; correctness-adjusted research progress, not answer accuracy. Reported metric: Research progress (0–4). Primary published table, with source configuration boundaries preserved.
STEMBench120 - Astronomy (Research progress (0–4)): leader Gemini 4 Argon (2.778), 10 models
STEMBench120 - Astronomy. v1.0.0; 120 tasks across six STEM domains; Codex tool runs distinct from closed-book rows; correctness-adjusted research progress, not answer accuracy. Reported metric: Research progress (0–4). Primary published table, with source configuration boundaries preserved.
STEMBench120 - Earth science (Research progress (0–4)): leader Claude Opus 5.5 (2.76), 10 models
STEMBench120 - Earth science. v1.0.0; 120 tasks across six STEM domains; Codex tool runs distinct from closed-book rows; correctness-adjusted research progress, not answer accuracy. Reported metric: Research progress (0–4). Primary published table, with source configuration boundaries preserved.
STEMBench120 - Engineering (Research progress (0–4)): leader GPT-6 Astra (Harness: Codex) (xHigh) (2.894), 10 models
STEMBench120 - Engineering. v1.0.0; 120 tasks across six STEM domains; Codex tool runs distinct from closed-book rows; correctness-adjusted research progress, not answer accuracy. Reported metric: Research progress (0–4). Primary published table, with source configuration boundaries preserved.
Vals AI Robo-bench (Accuracy (%)): leader GPT-6.1 Sol (Max) (28.33), 7 models
Vals AI evaluates models controlling a robot arm and humanoid in real-time physics simulation from motor commands and camera images. Accuracy measures success on robot tasks, most of which are held out; this is an agent-control protocol, not static visual question answering.
Vals AI SWE-bench Multimodal (Resolved (%)): leader Claude Opus 5.5 (70.21), 18 models
Resolution rate on 480 visual JavaScript repository issues using mini-swe-agent. Screenshots are attached before the run, internet access is disabled, and the repository test harness grades the final patch. Network-enabled diagnostic results are separate from this ranking.

Top-10 New Scores (107)

Claude Fable 5.1 on Anthropic Election Integrity Multi-Turn - Appropriate Response Rate (API): 90.0 (#4)
Claude Fable 5.1 on BBQ Disambiguated Accuracy (Mythos 5.1 System Card): 89.92 (#1)
Claude Opus 5.5 on Anthropic Child Safety Multi-Turn - Appropriate Response Rate (API): 84.0 (#8)
Claude Opus 5.5 on Anthropic Child Safety Multi-Turn - Appropriate Response Rate (Claude.ai): 99.0 (#2)
Claude Opus 5.5 on Anthropic Claude Code Malicious Requests - Refusal Rate (Mythos 5.1 System Card): 79.8 (#5)
Claude Opus 5.5 on Anthropic Disordered Eating Single-Turn Risk Requests - Harmless Rate (API): 95.64 (#9)
Claude Opus 5.5 on Anthropic Disordered Eating Single-Turn Risk Requests - Harmless Rate (Claude.ai): 99.26 (#6)
Claude Opus 5.5 on Anthropic Election Integrity Multi-Turn - Appropriate Response Rate (API): 87.0 (#8)
Claude Opus 5.5 on Anthropic Election Integrity Multi-Turn - Appropriate Response Rate (Claude.ai): 85.0 (#6)
Claude Opus 5.5 on Anthropic Malicious Computer Use - Refusal Rate: 79.46 (#8)
Claude Opus 5.5 on Anthropic Single-Turn Benign Requests - Over-Refusal Rate (API): 0.03 (#4)
Claude Opus 5.5 on Anthropic Single-Turn Benign Requests - Over-Refusal Rate (Claude.ai): 0.38 (#2)
Claude Opus 5.5 on Anthropic Single-Turn Harmful Requests - Harmless Rate (API): 94.5 (#9)
Claude Opus 5.5 on Anthropic Single-Turn Harmful Requests - Harmless Rate (Claude.ai): 99.51 (#3)
Claude Opus 5.5 on Anthropic Suicide and Self-Harm Multi-Turn - Appropriate Response Rate (API): 66.0 (#4)
Claude Opus 5.5 on Anthropic Suicide and Self-Harm Multi-Turn - Appropriate Response Rate (Claude.ai): 94.0 (#2)
Claude Opus 5.5 on Anthropic Suicide and Self-Harm Single-Turn Risk Requests - Harmless Rate (API): 99.16 (#8)
Claude Opus 5.5 on BBQ Disambiguated Accuracy (Mythos 5.1 System Card): 89.65 (#3)
Claude Opus 5.5 on Chess Puzzles (Epoch AI): 67.0 (#4)
Claude Opus 5.5 on STEMBench120 v1.0.0 (Closed Book) - Astronomy: 2.759 (#2)
Claude Opus 5.5 on STEMBench120 v1.0.0 (Closed Book) - Biology: 1.865 (#8)
Claude Opus 5.5 on STEMBench120 v1.0.0 (Closed Book) - Chemistry: 2.666 (#2)
Claude Opus 5.5 on STEMBench120 v1.0.0 (Closed Book) - Overall: 2.621 (#2)
Claude Opus 5.5 on SWE-bench Pro (Opus 5 & Mythos 5.1 System Cards): 89.9 (#1)
Claude Opus 5.5 on Toolathlon Verified Pass@3 (Anthropic Internal Harness): 82.4 (#5)
Claude Opus 5.5 on Toolathlon Verified Pass^3 (Anthropic Internal Harness): 72.2 (#4)
Claude Sonnet 5.5 on Chess Puzzles (Epoch AI): 59.0 (#12)
Claude Sonnet 5.5 on SecIT Bench (Claude Code): 69.93 (#2)
Claude Sonnet 5.5 on SecIT Bench (Pydantic AI): 75.13 (#14)
GPT-6 Astra on Epoch AI - Scicode: 56.48 (#37)
GPT-6 Astra on STEMBench120 v1.0.0 (Tools) - Astronomy: 2.723 (#1)
GPT-6 Astra on STEMBench120 v1.0.0 (Tools) - Biology: 2.709 (#1)
GPT-6 Astra on STEMBench120 v1.0.0 (Tools) - Chemistry: 2.777 (#1)
GPT-6 Astra on STEMBench120 v1.0.0 (Tools) - Earth science: 2.726 (#1)
GPT-6 Astra on STEMBench120 v1.0.0 (Tools) - Engineering: 2.894 (#1)
GPT-6 Astra on STEMBench120 v1.0.0 (Tools) - Overall: 2.809 (#1)
GPT-6 Astra on STEMBench120 v1.0.0 (Tools) - Physics: 3.025 (#1)
GPT-6.1 Sol on Epoch AI - Scicode: 55.79 (#43)
Gemini 4 Argon on AA-AnalystAgent: 48.75 (#11)
Gemini 4 Argon on AA-AnalystAgent - Pass@1: 57.75 (#24)
Gemini 4 Argon on AA-AnalystAgent - Pass@K: 66.25 (#39)
Gemini 4 Argon on Arabic Broad Leaderboard: 9.128 (#10)
Gemini 4 Argon on Arabic Broad Leaderboard - General Knowledge: 9.175 (#13)
Gemini 4 Argon on Arabic Broad Leaderboard - MMLU: 9.752 (#12)
Gemini 4 Argon on Arabic Broad Leaderboard - RAG QA: 8.61 (#19)
Gemini 4 Argon on Arabic Broad Leaderboard - Reasoning & Math: 9.465 (#10)
Gemini 4 Argon on Arabic Broad Leaderboard - Translation (incl Dialects): 8.0 (#8)
Gemini 4 Argon on Arabic Broad Leaderboard - Trust & Safety: 9.0 (#43)
Gemini 4 Argon on Arabic Broad Leaderboard - Writing (incl Dialects): 7.864 (#17)
Gemini 4 Argon on Bug Hunt Bench: 39.3 (#12)
Gemini 4 Argon on Bug Hunt Bench - LMS: 25.7 (#3)
Gemini 4 Argon on Bug Hunt Bench - VS Code Extension: 13.7 (#28)
Gemini 4 Argon on Chess Bench LLM: 1974.0 (#5)
Gemini 4 Argon on Chess Puzzles (Epoch AI): 59.0 (#10)
Gemini 4 Argon on ChessBench: 24.41 (#3)
Gemini 4 Argon on ChessBench - Accuracy: 80.1 (#2)
Gemini 4 Argon on ChessBench - Coherence: 85.8 (#3)
Gemini 4 Argon on ChessBench - Elo: 1356.0 (#3)
Gemini 4 Argon on Creative Writing (Lechmazur): 0.8 (#19)
Gemini 4 Argon on Deep20Bench - Edition 1.1: 13.666666666667 (#2)
Gemini 4 Argon on Design Arena (3D): 1378.0 (#11)
Gemini 4 Argon on Design Arena (Game Dev): 1369.0 (#9)
Gemini 4 Argon on Design Arena (UI Components): 1279.0 (#36)
Gemini 4 Argon on Design Arena (Website): 1322.0 (#10)
Gemini 4 Argon on EKI Estonian LLM - Book Recommendations: 63.47 (#21)
Gemini 4 Argon on EKI Estonian LLM - Idioms: 70.0 (#8)
Gemini 4 Argon on EKI Estonian LLM - Overall: 72.48 (#3)
Gemini 4 Argon on EKI Estonian LLM - Propaganda Resistance: 79.52 (#51)
Gemini 4 Argon on EKI Estonian LLM - Terminology: 57.54 (#4)
Gemini 4 Argon on EKI Estonian LLM - Trivia Hard: 50.62 (#2)
Gemini 4 Argon on Epoch AI - Cursorbench: 52.9 (#6)
Gemini 4 Argon on Epoch AI - Frontiermath Tier 4 V2: 78.05 (#15)
Gemini 4 Argon on Epoch AI - Mystery Game Puzzles: 64.0 (#5)
Gemini 4 Argon on ErdosBench: 3.04 (#6)
Gemini 4 Argon on FrontierMath - Tier 4 (v2): 78.05 (#15)
Gemini 4 Argon on Gert Labs Rankings: 69.77 (#7)
Gemini 4 Argon on HLE-Diamond: 56.9 (#5)
Gemini 4 Argon on HLE-Diamond - Knowledge: 52.0 (#2)
Gemini 4 Argon on HLE-Diamond - Reasoning: 61.8 (#8)
Gemini 4 Argon on Humanity's Sixth Sense: 55.4 (#1)
Gemini 4 Argon on Icelandic LLM - Case Government: 98.75 (#2)
Gemini 4 Argon on Icelandic LLM - Flora & Fauna: 49.95 (#7)
Gemini 4 Argon on Icelandic LLM - GEC: 80.73 (#5)
Gemini 4 Argon on Icelandic LLM - Generative Quality: 74.91 (#15)
Gemini 4 Argon on Icelandic LLM - Idiom Meanings: 81.67 (#13)
Gemini 4 Argon on Icelandic LLM - Song Lyrics: 21.83 (#2)
Gemini 4 Argon on Icelandic LLM - Textile Terms: 70.4 (#7)
Gemini 4 Argon on Icelandic LLM - WikiQA: 71.26 (#3)
Gemini 4 Argon on Icelandic LLM Leaderboard - Average: 70.52 (#2)
Gemini 4 Argon on KernelBench Hub - CUDA: 42.2 (#9)
Gemini 4 Argon on KernelBench Hub - Mega: 27.03 (#3)
Gemini 4 Argon on LLM2014 Logic 2026-10: 69.35 (#5)
Gemini 4 Argon on MineBench: 2269.0 (#1)
Gemini 4 Argon on NYT Connections Extended: 98.6 (#1)
Gemini 4 Argon on Next.js Agent Evals (OpenCode) - Success Rate with AGENTS.md: 97.0 (#2)
Gemini 4 Argon on OTIS Mock AIME 2024-25: 100.0 (#1)
Gemini 4 Argon on OpenRouter Tau2-Bench Airline: 78.7 (#6)
Gemini 4 Argon on STEMBench120 v1.0.0 (Closed Book) - Biology: 2.348 (#6)
Gemini 4 Argon on STEMBench120 v1.0.0 (Closed Book) - Chemistry: 2.336 (#8)
Gemini 4 Argon on STEMBench120 v1.0.0 (Closed Book) - Earth science: 2.647 (#3)
Gemini 4 Argon on STEMBench120 v1.0.0 (Closed Book) - Engineering: 2.471 (#6)
Gemini 4 Argon on STEMBench120 v1.0.0 (Closed Book) - Overall: 2.444 (#6)
Gemini 4 Argon on STEMBench120 v1.0.0 (Closed Book) - Physics: 2.083 (#6)
Gemini 4 Argon on SpeechMap Compliance: 87.0 (#55)
Gemini 4 Argon on Tinybird AI SQL Benchmark - Exactness: 56.91 (#26)
Gemini 4 Argon on Tinybird AI SQL Benchmark - First-Attempt Success Rate: 100.0 (#41)
Gemini 4 Argon on Tinybird AI SQL Benchmark - Success Rate: 100.0 (#175)

New #1 Leaders (27)

SWE-bench Pro (Opus 5 & Mythos 5.1 System Cards): Claude Opus 5.5 (Max) (89.9) beat Claude Fable 5.1 (Max) by 8.7
Icelandic LLM - Famous Phrases: Gemini 4 Argon (36.19) beat Gemini 3.8 Flash by 5.31
EKI Estonian LLM - Trivia: Gemini 4 Argon (90.0) beat Gemini 3.1 Pro (Preview) by 5.0
Vals AI Vibe Code Bench 1-100: Muse Wonder Full (33.98) beat Claude Opus 5.5 by 3.62
Next.js Agent Evals (OpenCode) - Success Rate: Gemini 4 Argon (97.0) beat Grok 4.7 by 3.0
Humanity's Sixth Sense: Gemini 4 Argon (High) (55.4) beat GPT-6 Astra (Max) by 1.8
EKI Estonian LLM - Language Advice: Gemini 4 Argon (77.5) beat GPT-6 Astra by 1.46
Icelandic LLM - Idiom Completion: Gemini 4 Argon (75.96) beat Claude Opus 5.5 by 0.93
OpenRouter GPQA Diamond: Gemini 4 Argon (96.5) beat Gemini 3.8 Flash by 0.9
NYT Connections Extended: Gemini 4 Argon (Thinking, High) (98.6) beat GPT-6 Astra (Thinking, xHigh) by 0.5
Epoch AI - Rli: Claude Opus 5.5 (Unknown) (21.25) beat GPT-6 Astra by 0.42
STEMBench120 v1.0.0 (Closed Book) - Physics: Claude Opus 5.5 (2.808) beat Hy4 preview by 0.29
STEMBench120 v1.0.0 (Closed Book) - Engineering: Claude Opus 5.5 (2.872) beat Hy4 preview by 0.26
STEMBench120 v1.0.0 (Closed Book) - Astronomy: Gemini 4 Argon (2.778) beat DeepSeek V4.1 Flash by 0.18
STEMBench120 v1.0.0 (Closed Book) - Earth science: Claude Opus 5.5 (2.76) beat MiMo-V2.6-Pro by 0.14
Anthropic Child Safety Single-Turn Benign Requests - Over-Refusal Rate (Claude.ai): Claude Opus 5.5 (0.05) beat Claude Fable 5.1 by 0.12
STEMBench120 v1.0.0 (Closed Book) - Overall: Muse Wonder (2.683) beat Hy4 preview by 0.1
Anthropic Claude Code Dual-Use Requests - Success Rate (Mythos 5.1 System Card): Claude Opus 5.5 (99.8) beat Claude Opus 5 by 0.1
STEMBench120 v1.0.0 (Closed Book) - Chemistry: Muse Wonder (2.703) beat Hy4 preview by 0.09
STEMBench120 v1.0.0 (Tools) - Engineering: GPT-6 Astra (xHigh) (2.894) beat GPT-6.1 Sol by 0.08
STEMBench120 v1.0.0 (Tools) - Overall: GPT-6 Astra (xHigh) (2.809) beat GPT-6.1 Sol by 0.05
STEMBench120 v1.0.0 (Tools) - Biology: GPT-6 Astra (xHigh) (2.709) beat GPT-6.1 Sol by 0.05
STEMBench120 v1.0.0 (Tools) - Chemistry: GPT-6 Astra (xHigh) (2.777) beat GPT-6.1 Sol by 0.05
STEMBench120 v1.0.0 (Tools) - Earth science: GPT-6 Astra (xHigh) (2.726) beat GPT-6.1 Sol by 0.05
STEMBench120 v1.0.0 (Tools) - Physics: GPT-6 Astra (xHigh) (3.025) beat GPT-6.1 Sol by 0.04
STEMBench120 v1.0.0 (Closed Book) - Biology: Muse Wonder (2.615) beat Hy4 preview by 0.03
STEMBench120 v1.0.0 (Tools) - Astronomy: GPT-6 Astra (xHigh) (2.723) beat GPT-6.1 Sol by 0.03

The Aggregate Digest — 2026-10-14

2026-10-14T06:20:51.469101+00:00

Daily

New Benchmarks (7)

LiveBench Design Layout (Score): leader Claude Opus 5.5 (86.9), 36 models
LiveBench task column. Design and layout tasks across presentations, PDFs and documents, checked for visual and structural correctness. Scores follow the served release CSV; exact model effort and release provenance require the source run metadata.
LiveBench Research (Score): leader Claude Opus 5 (88.9), 36 models
LiveBench task column. Open-ended research reports assessed against a rubric. Scores follow the served release CSV; exact model effort and release provenance require the source run metadata.
LiveBench Browser (Score): leader Claude Fable 5.1 (100.0), 36 models
LiveBench task column. Multi-step tasks in working web applications, executed through a browser; share of tasks passed. Scores follow the served release CSV; exact model effort and release provenance require the source run metadata.
LiveBench Coding (Score): leader Claude Fable 5 (98.3), 36 models
LiveBench task column. Real merged-PR bug fixes in a frozen monorepo, graded deterministically; regressions score zero. Scores follow the served release CSV; exact model effort and release provenance require the source run metadata.
LiveBench Long Horizon (Score): leader Claude Opus 5.5 (85.0), 36 models
LiveBench task column. Extended coding and optimization, business applications with changing requirements, and codebase operations over simulated months; deterministic grading. Scores follow the served release CSV; exact model effort and release provenance require the source run metadata.
LiveBench Data Analysis (Score): leader Gemini 4 Argon (85.2), 36 models
LiveBench task column. Short table-analysis questions involving consecutive events, joins and reformatting, with a single correct answer. Scores follow the served release CSV; exact model effort and release provenance require the source run metadata.
LiveBench Reasoning (Score): leader Gemini 4 Argon (95.1), 36 models
LiveBench task column. Short logic, spatial and theory-of-mind questions with a single correct answer. Scores follow the served release CSV; exact model effort and release provenance require the source run metadata.

Top-10 New Scores (105)

GPT-6.1 Sol on HLE-Diamond (With Tools): 69.1 (#4)
GPT-6.1 Sol on Kaggle FACTS (Google): 75.29 (#2)
GPT-6.1 Sol on Kaggle FACTS Grounding: 82.42 (#2)
GPT-6.1 Sol on Kaggle FACTS Multimodal: 48.46 (#3)
GPT-6.1 Sol on Kaggle FACTS Parametric: 82.11 (#2)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Arabic: 69.5 (#40)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Bengali: 70.0 (#40)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Burmese: 74.5 (#38)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Chinese: 82.75 (#34)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Culturally Agnostic: 78.31 (#36)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Culturally Sensitive: 79.47 (#33)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - English: 91.0 (#23)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - French: 76.5 (#35)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - German: 85.25 (#31)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Hindi: 65.0 (#40)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Indonesian: 77.0 (#35)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Italian: 81.75 (#34)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Japanese: 87.5 (#28)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Korean: 90.25 (#23)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Portuguese: 88.0 (#29)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Spanish: 82.0 (#33)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Swahili: 77.0 (#35)
GPT-6.1 Sol on Kaggle Global-MMLU-Lite - Yoruba: 64.25 (#43)
GPT-6.1 Sol on Kaggle Open Benchmarks - SciCode Subproblems: 37.8 (#16)
Gemini 4 Argon on AI Chess Leaderboard (Continuation): 1641.0 (#9)
Gemini 4 Argon on AI Chess Leaderboard (Reasoning): 1599.0 (#19)
Gemini 4 Argon on AIMultiple - FinanceReasoning (Hard): 90.76 (#2)
Gemini 4 Argon on AIMultiple - Text-to-SQL (Adjudicated): 81.4 (#8)
Gemini 4 Argon on AIMultiple - Text-to-SQL (Strict Execution Match): 64.3 (#2)
Gemini 4 Argon on Bug Hunt Bench: 33.0 (#18)
Gemini 4 Argon on Bug Hunt Bench - LMS: 18.0 (#18)
Gemini 4 Argon on Bug Hunt Bench - VS Code Extension: 15.0 (#23)
Gemini 4 Argon on CursorBench 4.0: 52.9 (#6)
Gemini 4 Argon on EnigmaForge: 81.2 (#2)
Gemini 4 Argon on EnigmaForge - Fact F1: 98.3 (#2)
Gemini 4 Argon on EnigmaForge - Intuition: 72.9 (#4)
Gemini 4 Argon on Epoch AI - Furniture Assembly: 72.5 (#5)
Gemini 4 Argon on Epoch AI - GPQA Diamond: 96.4 (#1)
Gemini 4 Argon on Kaggle ExtractBench: 89.81 (#4)
Gemini 4 Argon on Kaggle FACTS Grounding: 78.63 (#6)
Gemini 4 Argon on Kaggle FACTS Parametric: 71.55 (#11)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Arabic: 94.75 (#6)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Burmese: 94.0 (#5)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Chinese: 95.0 (#8)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Culturally Agnostic: 95.91 (#3)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Culturally Sensitive: 94.84 (#4)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - English: 96.0 (#7)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - French: 95.25 (#7)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - German: 96.75 (#2)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Hindi: 95.5 (#2)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Indonesian: 94.75 (#9)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Italian: 95.5 (#7)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Korean: 94.5 (#8)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Portuguese: 96.0 (#5)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Spanish: 96.5 (#4)
Gemini 4 Argon on Kaggle Global-MMLU-Lite - Yoruba: 93.5 (#5)
Gemini 4 Argon on Kaggle Open Benchmarks - AIME 2025: 100.0 (#10)
Gemini 4 Argon on Kaggle Open Benchmarks - MATH-500: 99.0 (#6)
Gemini 4 Argon on Kaggle Open Benchmarks - SciCode Subproblems: 45.02 (#2)
Gemini 4 Argon on Kaggle SimpleQA: 71.18 (#3)
Gemini 4 Argon on Kaggle SimpleQA Verified: 75.6 (#3)
Gemini 4 Argon on Kaggle VideoQA: 87.0 (#3)
Gemini 4 Argon on LM Market Cap LMC Score: 40.0 (#203)
Gemini 4 Argon on Last Translation Benchmark: 40.944017563117455 (#5)
Gemini 4 Argon on NonoBench: 86.666666666667 (#11)
Gemini 4 Argon on PLCC - Art & Entertainment: 95.0 (#5)
Gemini 4 Argon on PLCC - Culture & Tradition: 98.0 (#11)
Gemini 4 Argon on PLCC - Geography: 99.0 (#10)
Gemini 4 Argon on PLCC - Grammar: 96.0 (#5)
Gemini 4 Argon on PLCC - History: 98.0 (#5)
Gemini 4 Argon on PUMA Polish Multimodal - Contemporary Life: 84.0 (#2)
Gemini 4 Argon on PUMA Polish Multimodal - Geography and Environment: 89.0 (#3)
Gemini 4 Argon on PUMA Polish Multimodal - History and Culture: 83.0 (#3)
Gemini 4 Argon on PUMA Polish Multimodal - Structured Extraction: 69.0 (#6)
Gemini 4 Argon on RuneBench: 6.906727309216 (#3)
Gemini 4 Argon on SimpleQA Verified: 76.0 (#1)
Gemini 4 Argon on SvelteBench: 100.0 (#10)
Gemini 4 Argon on VoxelBench: 1703.0 (#27)
Muse Wonder Full on Harvey LAB-AA: 13.06 (#1)
Muse Wonder Full on Harvey LAB-AA - All Pass: 25.56 (#2)
Muse Wonder Full on Harvey LAB-AA - Antitrust & Competition: 93.6 (#4)
Muse Wonder Full on Harvey LAB-AA - Arbitration & International Dispute Resolution: 96.06 (#2)
Muse Wonder Full on Harvey LAB-AA - Banking & Finance: 96.64 (#3)
Muse Wonder Full on Harvey LAB-AA - Bankruptcy & Restructuring: 94.46 (#2)
Muse Wonder Full on Harvey LAB-AA - Capital Markets: 96.86 (#2)
Muse Wonder Full on Harvey LAB-AA - Corporate Governance: 96.69 (#2)
Muse Wonder Full on Harvey LAB-AA - Corporate M&A: 98.18 (#1)
Muse Wonder Full on Harvey LAB-AA - Data Privacy & Cybersecurity: 97.88 (#3)
Muse Wonder Full on Harvey LAB-AA - Emerging Companies & Venture Capital: 95.79 (#2)
Muse Wonder Full on Harvey LAB-AA - Employment & Labor: 91.51 (#2)
Muse Wonder Full on Harvey LAB-AA - Energy & Natural Resources: 97.36 (#3)
Muse Wonder Full on Harvey LAB-AA - Environmental & ESG: 92.95 (#2)
Muse Wonder Full on Harvey LAB-AA - Funds & Asset Management: 95.72 (#1)
Muse Wonder Full on Harvey LAB-AA - Healthcare & Life Sciences: 95.19 (#3)
Muse Wonder Full on Harvey LAB-AA - Immigration: 94.55 (#2)
Muse Wonder Full on Harvey LAB-AA - Insurance: 94.12 (#3)
Muse Wonder Full on Harvey LAB-AA - Intellectual Property: 97.86 (#1)
Muse Wonder Full on Harvey LAB-AA - International Trade & Sanctions: 96.41 (#1)
Muse Wonder Full on Harvey LAB-AA - Litigation & Dispute Resolution: 94.86 (#2)
Muse Wonder Full on Harvey LAB-AA - Real Estate: 98.06 (#1)
Muse Wonder Full on Harvey LAB-AA - Structured Finance & Securitization: 94.52 (#2)
Muse Wonder Full on Harvey LAB-AA - Tax: 90.97 (#3)
Muse Wonder Full on Harvey LAB-AA - Trusts, Estates & Private Client: 95.04 (#3)
Muse Wonder Full on Harvey LAB-AA - White Collar Defense & Investigations: 95.74 (#1)
Muse Wonder Full on Vending-Bench 2: 12291.558333333334 (#4)

New #1 Leaders (42)

HieroglyphBench: Gemini 4 Argon (69.7) beat Gemini 3.7 Flash by 14.7
Design Arena (UI Components): muse-wonder-1.0 (1405.0) beat GPT-6 Astra by 6.0
TaxCalcBench: Meta Muse Wonder w/ Web Search (49/50) (71.43) beat GPT-6 Sol Web Search by 5.43
PUMA Polish Multimodal - Document QA: Gemini 4 Argon (93.0) beat Claude Opus 5.5 by 4.0
AIMultiple - Agentic RAG Routing: Gemini 4 Argon (94.02) beat Claude Opus 5.5 by 3.8
AA-AnalystAgent - Pass@K: GPT-6 Luna (Max) (85.0) beat Gemini 3.1 Pro (Preview) by 3.75
Harvey LAB-AA: Muse Wonder Full (Max) (13.06) beat Grok 4.7 (xHigh) by 3.62
Harvey LAB-AA - Funds & Asset Management: Muse Wonder Full (Max) (95.72) beat Muse Spark 1.3 (Max) by 3.4
FEV-Bench - SQL: t0-gamma (91.59) beat timesfm-3 by 3.38
FEV-Bench - MASE: t0-gamma (88.72) beat timesfm-3 by 2.76
FEV-Bench: t0-gamma (89.22) beat timesfm-3 by 2.38
Harvey LAB-AA - Immigration: Muse Wonder (Max) (95.39) beat Grok 4.7 (xHigh) by 2.18
PLCC - Vocabulary: Gemini 4 Argon (98.0) beat Gemini 3.1 Pro (Preview) by 2.0
FEV-Bench - WQL: t0-gamma (89.28) beat timesfm-3 by 1.82
FEV-Bench - WAPE: t0-gamma (87.28) beat timesfm-3 by 1.57
Kaggle Open Benchmarks - MMLU-Pro: Gemini 4 Argon (93.19) beat Claude Opus 5 by 1.51
Harvey LAB-AA - Corporate M&A: Muse Wonder Full (Max) (98.18) beat Mistral Large 4 Preview by 1.07
PUMA Polish Multimodal - OCR: Gemini 4 Argon (91.0) beat Claude Fable 5 by 1.0
PUMA Polish Multimodal - Vision Mean: Gemini 4 Argon (84.83) beat Gemini 3.7 Flash by 0.83
Harvey LAB-AA - Healthcare & Life Sciences: Muse Wonder (Max) (96.09) beat Muse Spark 1.3 (Max) by 0.79
Kaggle Open Benchmarks - MMLU: Gemini 4 Argon (95.25) beat Grok 4.5 by 0.71
Epoch AI - GPQA Diamond: Gemini 4 Argon (High) (96.4) beat GPT-6 Astra (Max) by 0.63
Harvey LAB-AA - Real Estate: Muse Wonder Full (Max) (98.06) beat Muse Spark 1.3 (Max) by 0.49
SimpleQA Verified: Gemini 4 Argon (High) (76.0) beat GPT-6 Astra (Max) by 0.4
Harvey LAB-AA - White Collar Defense & Investigations: Muse Wonder Full (Max) (95.74) beat Muse Spark 1.3 (Max) by 0.34
GAIA: Shadow Harness V8.01 (94.02) beat Ops-Agentic-Search-2.0 by 0.33
PLCC - Overall: Gemini 4 Argon (97.33) beat Gemini 3.1 Pro (Preview) by 0.33
Kaggle Global-MMLU-Lite - Bengali: Gemini 4 Argon (96.0) beat Claude Opus 4.8 by 0.25
Kaggle Global-MMLU-Lite - Japanese: Gemini 4 Argon (96.25) beat Claude Opus 5 by 0.25
Kaggle Global-MMLU-Lite - Swahili: Gemini 4 Argon (95.75) beat Claude Opus 5.5 by 0.25
AI Forever RAG - Set Generation: Qwen3.8-27B (kb graphRAG) (aa045f) (0.5046) beat RuadaptQwen2.5-32B-Instruct (9449f3) by 0.22
AI Forever RAG - Simple Generation: Qwen3.8-27B (kb graphRAG) (aa045f) (0.6265) beat RuadaptQwen2.5-32B-Instruct (9449f3) by 0.2
AI Forever RAG - Set Retrieval: Qwen3.8-27B (kb graphRAG) (aa045f) (0.8139) beat Qwen2.5-7B-Instruct (d6ccf3) by 0.19
AI Forever RAG - Multi-Hop Generation: gemma4-25B (kb graphRAG) (fe39ea) (0.6374) beat RuadaptQwen2.5-32B-Instruct (9449f3) by 0.19
AI Forever RAG - Generation: Qwen3.8-27B (kb graphRAG) (aa045f) (0.6474) beat RuadaptQwen2.5-32B-Instruct (9449f3) by 0.18
Harvey LAB-AA - International Trade & Sanctions: Muse Wonder Full (Max) (96.41) beat Muse Spark 1.3 (Max) by 0.13
Harvey LAB-AA - Intellectual Property: Muse Wonder Full (Max) (97.86) beat Muse Spark 1.3 (Max) by 0.11
AI Forever RAG - Conditional Generation: Qwen3.8-27B (kb graphRAG) (aa045f) (0.8285) beat RuadaptQwen2.5-32B-Instruct (9449f3) by 0.1
AI Forever RAG - Retrieval: Qwen3.8-27B (kb graphRAG) (aa045f) (0.7716) beat RuadaptQwen2.5-32B-Instruct (9449f3) by 0.09
AI Forever RAG - Multi-Hop Retrieval: Qwen3.8-27B (kb graphRAG) (aa045f) (0.7529) beat RuadaptQwen2.5-32B-Instruct (9449f3) by 0.07
AI Forever RAG - Simple Retrieval: Qwen3.8-27B (kb graphRAG) (aa045f) (0.7688) beat RuadaptQwen2.5-32B-Instruct (9449f3) by 0.06
AI Forever RAG - Conditional Retrieval: gemma4-25B (kb graphRAG) (fe39ea) (0.7644) beat RuadaptQwen2.5-32B-Instruct (9449f3) by 0.02

The Aggregate Digest — 2026-10-13

2026-10-13T14:39:48.977499+00:00

Frontier

Best available model: Muse Wonder Full enters at #12 (Aggregate Rating 1931) on 66 benchmarks
Best available model: Muse Wonder enters at #13 (Aggregate Rating 1913) on 69 benchmarks

Daily

New Benchmarks (12)

Clinical Dentistry - Deployment Accuracy (Accuracy (%); 30 questions, one trial, Opus 4.8 judge; refusals count as incorrect): leader GPT-5.2 (96.7), 8 models
Eight model runs on 30 clinical dental questions, judged by Claude Opus 4.8 with independently published GPT-5.2/5.5 judge checks. One temperature-zero trial; refusals count as incorrect. Native percentage accuracy is not conditional accuracy or its bootstrap confidence interval. Small differences at this sample size are not established.
Paydirt - Held Pair Accuracy (Correct pairs (%); both answers must pass, raw arm, omp harness, 18 held pairs): leader GPT-6 Astra (Max) (94.44), 22 models
Paydirt 2026-10 measures white-box security code review over 18 held vulnerable/fixed or accurate/overclaimed pairs. Both answers must be right for pair credit. The raw arm uses the common omp 18.4.4 harness without Bounty Operator profiles, one trial per input and the highest requested reasoning effort. Source-declared incomplete runs are unranked; mixed high/xhigh runs and dated IDs remain explicit.
Taste-Bench - Average (Accuracy (%)): leader GPT-5.6 Sol (xHigh) (59.7), 14 models
Taste-Bench evaluates decisions between alternative engineering and research trajectories without revealing their future outcomes. 502 forks are shown in both option orders; primary accuracy requires both presentations correct, while Mean Presentation Accuracy measures individual presentations. Average is the equal-weight mean of engineering and research accuracies; construction cells retain their source sample counts. Table 3 pins reasoning settings; failures to parse remain incorrect. Reported column: Average.
Taste-Bench - Engineering (Accuracy (%)): leader GPT-5.6 Sol (xHigh) (56.9), 14 models
Taste-Bench evaluates decisions between alternative engineering and research trajectories without revealing their future outcomes. 502 forks are shown in both option orders; primary accuracy requires both presentations correct, while Mean Presentation Accuracy measures individual presentations. Average is the equal-weight mean of engineering and research accuracies; construction cells retain their source sample counts. Table 3 pins reasoning settings; failures to parse remain incorrect. Reported column: Engineering.
Taste-Bench - Research (Accuracy (%)): leader Claude Opus 5 (64.3), 14 models
Taste-Bench evaluates decisions between alternative engineering and research trajectories without revealing their future outcomes. 502 forks are shown in both option orders; primary accuracy requires both presentations correct, while Mean Presentation Accuracy measures individual presentations. Average is the equal-weight mean of engineering and research accuracies; construction cells retain their source sample counts. Table 3 pins reasoning settings; failures to parse remain incorrect. Reported column: Research.
Taste-Bench - Parallel Engineering (Accuracy (%)): leader GPT-5.6 Sol (xHigh) (75.8), 14 models
Taste-Bench evaluates decisions between alternative engineering and research trajectories without revealing their future outcomes. 502 forks are shown in both option orders; primary accuracy requires both presentations correct, while Mean Presentation Accuracy measures individual presentations. Average is the equal-weight mean of engineering and research accuracies; construction cells retain their source sample counts. Table 3 pins reasoning settings; failures to parse remain incorrect. Reported column: Parallel Engineering.
Taste-Bench - Parallel Research (Accuracy (%)): leader Claude Opus 5 (64.6), 14 models
Taste-Bench evaluates decisions between alternative engineering and research trajectories without revealing their future outcomes. 502 forks are shown in both option orders; primary accuracy requires both presentations correct, while Mean Presentation Accuracy measures individual presentations. Average is the equal-weight mean of engineering and research accuracies; construction cells retain their source sample counts. Table 3 pins reasoning settings; failures to parse remain incorrect. Reported column: Parallel Research.
Taste-Bench - Detour Engineering (Accuracy (%)): leader GPT-5.6 Sol (xHigh) (48.1), 14 models
Taste-Bench evaluates decisions between alternative engineering and research trajectories without revealing their future outcomes. 502 forks are shown in both option orders; primary accuracy requires both presentations correct, while Mean Presentation Accuracy measures individual presentations. Average is the equal-weight mean of engineering and research accuracies; construction cells retain their source sample counts. Table 3 pins reasoning settings; failures to parse remain incorrect. Reported column: Detour Engineering.
Taste-Bench - Detour Research (Accuracy (%)): leader GPT-5.5 (xHigh) (68.8), 14 models
Taste-Bench evaluates decisions between alternative engineering and research trajectories without revealing their future outcomes. 502 forks are shown in both option orders; primary accuracy requires both presentations correct, while Mean Presentation Accuracy measures individual presentations. Average is the equal-weight mean of engineering and research accuracies; construction cells retain their source sample counts. Table 3 pins reasoning settings; failures to parse remain incorrect. Reported column: Detour Research.
Taste-Bench - Mean Presentation Accuracy (Accuracy (%)): leader GPT-5.6 Sol (xHigh) (65.2), 14 models
Taste-Bench evaluates decisions between alternative engineering and research trajectories without revealing their future outcomes. 502 forks are shown in both option orders; primary accuracy requires both presentations correct, while Mean Presentation Accuracy measures individual presentations. Average is the equal-weight mean of engineering and research accuracies; construction cells retain their source sample counts. Table 3 pins reasoning settings; failures to parse remain incorrect. Reported column: Mean Presentation Accuracy.
The Aggregate - Lichess Rating Prediction (25 Games) (MAE (rating points)): leader Gemini 3.1 Pro (Preview) (Medium) (269.66), 16 models
Predict each player’s pre-game Lichess rapid Glicko-2 rating from an anonymized standard 10+0 game. All 25 games involve 50 distinct established players, each with at least 1,000 rapid games and rating deviation at most 60 at collection. Every model sees the same frozen moves, result and time control, with no identities, ratings, engine analysis or tools. Medium reasoning is requested with an 8,192-token output cap. The score is mean absolute error across all 50 player estimates, in rating points; lower is better. Only complete 25-game panels are ranked. The original twenty games span five mean-rating bands, with five further games deliberately adding players rated below 1000; this convenience sample is not representative of the Lichess population. Reference predictors are not AI observations.
The Aggregate - Lichess Pairwise Strength (25 Pairs) (Accuracy (%)): leader Claude Opus 5.5 (Medium) (88.0), 16 models
Choose which of two target players had the higher established pre-game Lichess rapid Glicko-2 rating, using one anonymized game per player. The frozen sample pairs all 50 players from 25 games into 25 comparisons. Every model sees the same target colors, moves, result and time control, with identities, ratings, links and earlier guesses hidden. Medium reasoning is requested with an 8,192-token output cap and no tools. Accuracy uses all 25 categorical choices equally; higher is better. Five reversed-order repeats are diagnostics, not scored items. This task reuses the games from the numerical rating benchmark and is not independent evidence. Only complete panels are ranked. The public evidence includes every pair, label, choice and explanation, plus the recorded amendment retaining choices when a confidence number contradicted them.

New Models (2)

Muse Wonder Full — Aggregate Rating 1931, #12
- AA CritPt: 32.86 (#1/576)
- AA GDP.pdf: 32.6 (#1/212)
- AutomationBench-AA - HR - Raw Objective Completion: 83.32 (#1/221)
- LLM Stats (DeepSWE 1.1): 79.0 (#1/45)
- LLM Stats (SWE-Marathon): 57.5 (#1/8)
- AA GDPval: 1857.09 (#2/290)
- AA GDP.pdf - Legal - Criterion Pass Rate: 87.8 (#2/212)
- AutomationBench-AA: 71.33 (#3/221)
- AutomationBench-AA - Strict: 45.36 (#3/221)
- AA-Briefcase - Analytical Quality Elo: 2116.26 (#3/226)
- ...and 56 more
Muse Wonder — Aggregate Rating 1913, #13
- AA GDP.pdf - Legal - Criterion Pass Rate: 88.03 (#1/212)
- AA-Briefcase - DOCX - Rubric Pass Rate: 65.68 (#1/226)
- LLM Stats (DeepSWE 1.1): 78.8 (#2/45)
- LLM Stats (SWE-Marathon): 53.1 (#2/8)
- AutomationBench-AA - HR - Raw Objective Completion: 78.45 (#4/221)
- AA GDPval: 1770.34 (#5/290)
- AA-Briefcase - PDF - Rubric Pass Rate: 65.52 (#5/226)
- GDP.pdf: 30.8 (#5/6)
- AA-Briefcase - XLSX - Rubric Pass Rate: 61.14 (#6/226)
- LLM Stats (GDP.pdf): 30.8 (#6/11)
- ...and 59 more

Top-10 New Scores (7)

Claude Opus 5.5 on DRACO: 87.4 (#2)
Claude Opus 5.5 on SciLaws-Bench: 48.87 (#3)
Claude Sonnet 5.5 on Design Arena (3D): 1485.0 (#2)
Claude Sonnet 5.5 on GDP.pdf: 31.4 (#4)
GPT-6 Astra on Design Arena (3D): 1516.0 (#1)
GPT-6.1 Sol on Design Arena (UI Components): 1352.0 (#6)
GPT-6.1 Sol on GDP.pdf: 35.0 (#1)

New #1 Leaders (14)

Design Arena (3D): GPT-6 Astra (Max) (1516.0) beat Claude Opus 5.5 by 34.0
LLM Stats (SWE-Marathon): Muse Wonder Full (57.5) beat Grok 4.7 by 11.5
LiveBench Consecutive Events: muse-wonder-1.0-max (98.53) beat Claude Sonnet 4.6 (Thinking, Medium) by 5.85
Senior SWE-Bench: Claude Opus 5.5 (Mini-SWE-Agent, High) (39.325842696629216) beat GPT-5.6 Sol (Mini-SWE-Agent, xHigh) by 4.59
AA-Briefcase - DOCX - Rubric Pass Rate: Muse Wonder (Max) (65.68) beat Gemini 4 Argon (High) by 3.55
Vals AI CyberBench: muse-wonder (80.36) beat GPT-6 Sol by 2.38
SWE-Marathon v1.1: Muse Wonder Full (Thinking, max) (Muse Code) (53.125) beat Claude Sonnet 5.5 (Thinking, Max) (Claude Code) by 1.88
AutomationBench-AA - HR - Raw Objective Completion: Muse Wonder Full (Max) (83.32) beat Gemini 4 Argon (High) by 1.33
LLM Stats (DeepSWE 1.1): Muse Wonder Full (79.0) beat Gemini 4 Argon by 1.1
GDP.pdf: GPT-6.1 Sol (Max) (35.0) beat GPT-6 Astra (Max) by 0.8
AA GDP.pdf - Legal - Criterion Pass Rate: Muse Wonder (Max) (88.03) beat DeepSeek V4.1 Flash (Max) by 0.75
AA CritPt: Muse Wonder Full (Max) (32.86) beat GPT-5.6 Sol (Max) by 0.57
SciLaws-Bench: GPT-6 Astra (51.33) beat GPT-5.5 by 0.56
AA GDP.pdf: Muse Wonder Full (Max) (32.6) beat GPT-6 Astra (xHigh) by 0.4

The Aggregate Digest — 2026-10-12

2026-10-12T06:33:34.463643+00:00

Daily

New Benchmarks (2)

DataSense-Bench - BFCL Data Selection (Selected-data BFCL accuracy (%)): leader DeepSeek V4 Pro (36.53), 6 models
DataSense-Bench measures training-data selection by an AI agent. Selects 50 EnvScaler tool-use trajectories per subset; outcome is BFCL V3 accuracy in percent (0–100). The selecting agent has four hours to inspect data, execute analysis code and run forward passes, without training or evaluation-task access. A separate Qwen3-4B is fine-tuned with a fixed recipe within each task. The score is the mean outcome of the first-ranked subset across three selection runs, not the agent’s direct task execution, maximum checkpoint, gain or ranking accuracy. Higher is better. Gains over random selection are small relative to reported standard deviations across selection runs; score differences do not establish statistically reliable superiority. Public code alone does not establish access to the complete scored question set.
DataSense-Bench - TBLite Data Selection (Selected-data TBLite average reward): leader DeepSeek V4 Pro (0.1254), 10 models
DataSense-Bench measures training-data selection by an AI agent. Selects 1,000 OpenThoughts-Agent terminal trajectories per subset; outcome is TBLite average reward in its original 0–1 units. The selecting agent has four hours to inspect data, execute analysis code and run forward passes, without training or evaluation-task access. A separate Qwen3-4B is fine-tuned with a fixed recipe within each task. The score is the mean outcome of the first-ranked subset across three selection runs, not the agent’s direct task execution, maximum checkpoint, gain or ranking accuracy. Higher is better. Gains over random selection are small relative to reported standard deviations across selection runs; score differences do not establish statistically reliable superiority. Public code alone does not establish access to the complete scored question set.

Top-10 New Scores (8)

Claude Opus 5.5 on Agent Arena: 15.53 (#1)
Claude Opus 5.5 on Agent Arena - Bash Recovery: 15.01 (#2)
Claude Opus 5.5 on Agent Arena - Confirmed Success: 17.29 (#1)
Claude Opus 5.5 on Agent Arena - Praise vs Complaint: 36.78 (#3)
Claude Opus 5.5 on Agent Arena - Steerability: 8.2 (#9)
Claude Opus 5.5 on Agent Arena - Tool Hallucination: -0.37 (#17)
GPT-6.1 Sol on Design Arena (3D): 1476.0 (#2)
GPT-6.1 Sol on Design Arena (Data Viz): 1338.0 (#8)

New #1 Leaders (2)

Agent Arena: Claude Opus 5.5 (Max) (15.53) beat Claude Opus 5.5 (High) by 1.2
Agent Arena - Confirmed Success: Claude Opus 5.5 (Max) (17.29) beat Claude Fable 5.1 (Max) by 1.1

The Aggregate Digest — 2026-10-11

2026-10-11T07:02:17.540972+00:00

Daily

New Benchmarks (1)

The Aggregate - Day-one Recall Check (Accuracy (%)): leader Claude Fable 5 (61.8), 33 models
The Aggregate runs this evaluation itself. A short version of the book recall bank: 34 questions about 12 public-domain books in three tiers, famous, mid and obscure. It is the first step of the battery we run on every new free model the day it appears, so it is the one recall board where free models sit next to the flagships. Graded by matching the key, with no second-pass judge. Saying “unknown” counts as wrong; an empty reply does not count at all. Reasoning effort was set to high where the route accepts it. The pipeline ingests the published model scores with their original precision and recorded run settings. Acceptance of the high-effort request was not recorded, so ingestion does not assert a confirmed High variant.

Top-10 New Scores (45)

Claude Fable 5.1 on Chatbot Arena (Vision - Humor): 1281.0 (#16)
Claude Opus 5.5 on Chatbot Arena (Image-to-WebDev): 1749.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Vision - Creative Writing): 1301.0 (#5)
Claude Opus 5.5 on HealthBench-Psych - Hard: 42.281 (#2)
Claude Opus 5.5 on Kaggle Cipher Frontlines: 72.73 (#21)
Claude Opus 5.5 on Kaggle Game Arena (Unified): 400.79 (#7)
Claude Opus 5.5 on Kaggle Game Arena Chess: 948.16 (#12)
Claude Sonnet 5.5 on Arabic Broad Leaderboard: 8.911 (#23)
Claude Sonnet 5.5 on Arabic Broad Leaderboard - General Knowledge: 8.857 (#39)
Claude Sonnet 5.5 on Arabic Broad Leaderboard - MMLU: 9.587 (#22)
Claude Sonnet 5.5 on Arabic Broad Leaderboard - RAG QA: 7.268 (#44)
Claude Sonnet 5.5 on Arabic Broad Leaderboard - Reasoning & Math: 9.302 (#12)
Claude Sonnet 5.5 on Arabic Broad Leaderboard - Translation (incl Dialects): 7.889 (#14)
Claude Sonnet 5.5 on Arabic Broad Leaderboard - Trust & Safety: 9.667 (#7)
Claude Sonnet 5.5 on Arabic Broad Leaderboard - Writing (incl Dialects): 7.727 (#33)
Claude Sonnet 5.5 on Chatbot Arena (Image-to-WebDev): 1740.0 (#2)
Claude Sonnet 5.5 on Kaggle Cipher Frontlines: 81.82 (#19)
Claude Sonnet 5.5 on Kaggle Game Arena (Unified): 328.5 (#14)
Claude Sonnet 5.5 on Kaggle Game Arena Chess: 761.4 (#18)
Claude Sonnet 5.5 on LLM Chess (Saplin): 1194.567 (#22)
GPT-6 Astra on LLM Chess (Saplin): 1613.842 (#1)
GPT-6.1 Sol on Chatbot Arena (Image-to-WebDev): 1703.0 (#5)
GPT-6.1 Sol on HealthBench-Psych - Hard: 39.269 (#8)
GPT-6.1 Sol on Kaggle Game Arena (Unified): 438.2 (#3)
GPT-6.1 Sol on Kaggle Game Arena Chess: 1186.75 (#9)
GPT-6.1 Sol on Kaggle Game Arena Four in a Row: 574.26 (#6)
GPT-6.1 Sol on LLM Chess (Saplin): 1611.585 (#2)
GPT-6.1 Sol on ReLE - Agents - BFCL-V3: 55.6 (#99)
GPT-6.1 Sol on ReLE - Agents and Tool Use: 55.3 (#106)
GPT-6.1 Sol on ReLE - Coding: 61.0 (#43)
GPT-6.1 Sol on ReLE - Education: 63.4 (#22)
GPT-6.1 Sol on ReLE - Education - Gaokao: 59.6 (#39)
GPT-6.1 Sol on ReLE - Education - High School Subjects: 72.4 (#24)
GPT-6.1 Sol on ReLE - Education - Middle School Subjects: 50.9 (#46)
GPT-6.1 Sol on ReLE - Education - Primary School Subjects: 70.7 (#9)
GPT-6.1 Sol on ReLE - Finance: 85.1 (#20)
GPT-6.1 Sol on ReLE - Language and Instruction Following: 74.5 (#12)
GPT-6.1 Sol on ReLE - Law - Lawyer Qualification Exam: 84.7 (#6)
GPT-6.1 Sol on ReLE - Law and Civil Service: 87.3 (#5)
GPT-6.1 Sol on ReLE - Medicine - Basic Medical Knowledge: 86.0 (#18)
GPT-6.1 Sol on ReLE - Medicine - Physician Exams: 85.7 (#45)
GPT-6.1 Sol on ReLE - Medicine and Mental Health: 86.1 (#39)
GPT-6.1 Sol on ReLE - Overall: 71.7 (#35)
GPT-6.1 Sol on ReLE - Reasoning - BBH: 62.6 (#173)
GPT-6.1 Sol on ReLE - Reasoning and Mathematics: 81.2 (#38)

New #1 Leaders (6)

Chatbot Arena (Image-to-WebDev): Claude Opus 5.5 (Max) (1749.0) beat GPT-6 Astra (Max) by 16.0
Kaggle Cipher Frontlines: GPT-6.1 Sol (100.0) beat Gemini 3 Flash (Preview) by 9.09
SuperCLUE-Terminal - Overall: MiMo-V2.6-Pro (64.65) beat Kimi K3 (Max) by 4.04
HealthBench-Psych - Hard: Claude Sonnet 5.5 (45.159) beat Claude Opus 5 by 3.65
Chatbot Arena (Vision - Chinese): Gemini 3.8 Flash (High) (1380.0) beat Gemini 3.7 Flash (High) by 3.0
Vals AI SAFE-Teen - Relationship Boundaries: Mistral Large 4 (8.0) beat GPT-6.1 Sol by 2.0

Weekly

New Benchmarks (1)

The Aggregate - Day-one Recall Check (Accuracy (%)): leader Claude Fable 5 (61.8), 33 models
The Aggregate runs this evaluation itself. A short version of the book recall bank: 34 questions about 12 public-domain books in three tiers, famous, mid and obscure. It is the first step of the battery we run on every new free model the day it appears, so it is the one recall board where free models sit next to the flagships. Graded by matching the key, with no second-pass judge. Saying “unknown” counts as wrong; an empty reply does not count at all. Reasoning effort was set to high where the route accepts it. The pipeline ingests the published model scores with their original precision and recorded run settings. Acceptance of the high-effort request was not recorded, so ingestion does not assert a confirmed High variant.

New Models (315)

Claude Opus 5.5 — Aggregate Rating 1996, #3
- CursorBench 4.0: 57.8 (#1/68)
- Chatbot Arena (Image-to-WebDev): 1749.0 (#1/64)
- Epoch AI - Frontiermath Erdos: 2.94 (#1/7)
- YC-Bench: 2873.72509 (#1/53)
- Terminal-Bench 4.0 (Claude Code): 64.85 (#1/18)
- SEAL - Remote Labor Index (RLI): 21.25 (#1/20)
- PUMA Polish Multimodal - Document QA: 89.0 (#1/76)
- React Native Evals: 88.57 (#2/32)
- AA-Omniscience Accuracy: 66.22 (#2/571)
- HealthBench-Psych - Hard: 42.281 (#2/25)
- ...and 28 more
GPT-6 Astra — Aggregate Rating 1984, #5
- LLMConfBench: 0.67 (#1/28)
- LLM Chess (Saplin): 1613.842 (#1/178)
- RuneBench: 7.258225267108 (#1/90)
- VANTAGE-Bench: 75.26 (#1/18)
- VANTAGE-Bench - Spatial: 83.12 (#1/18)
- VANTAGE-Bench - Temporal: 51.64 (#1/18)
- VANTAGE-Bench - Semantic: 85.08 (#1/18)
- VANTAGE-Bench - Video QA: 83.26 (#1/18)
- VANTAGE-Bench - Event Verification: 86.9 (#1/18)
- VANTAGE-Bench - Single Object Tracking: 81.2 (#1/18)
- ...and 49 more
Gemini 4 Argon — Aggregate Rating 1973, #7
- Epoch AI - Apex Agents: 82.2 (#1/52)
- Epoch AI - Blueprint Bench 2: 0.544 (#1/31)
- LLM Stats (DeepSWE 1.1): 77.9 (#1/43)
- LLM Stats (Graphwalks BFS <128k): 99.7 (#1/13)
- LLM Stats (LVBench): 91.7 (#1/31)
- LLM Stats (PostTrainBench): 45.3 (#1/8)
- LLM Stats (Graphwalks BFS >128k): 84.2 (#2/12)
- ComplexConstraints: 53.3 (#2/44)
- Epoch AI - Vending Bench 2: 13718.16 (#3/66)
- Epoch AI - Scicode: 61.81 (#4/213)
- ...and 12 more
Claude Fable 5.1 — Aggregate Rating 1967, #8
- React Native Evals: 89.05 (#1/32)
- AA-Omniscience Accuracy: 67.23 (#1/571)
- Clerk LLM Leaderboard: 91.7 (#2/25)
- Benchmarks.bio - MetagenomicsBench: 61.7 (#2/22)
- Agents on Rails: 92.06349206349206 (#2/17)
- FoodTruckBench: 62732.17 (#2/57)
- Terminal-Bench 4.0 (Claude Code): 57.88 (#3/18)
- CursorBench 4.0: 51.8 (#7/68)
- RuneBench: 6.143277369387 (#11/90)
- Chatbot Arena (Vision - Humor): 1281.0 (#16/91)
- ...and 1 more
GPT-6.1 Sol — Aggregate Rating 1961, #9
- LLM Stats (APEX-Agents): 60.0 (#1/11)
- SEAL - MultiChallenge: 81.95 (#1/32)
- Kaggle Cipher Frontlines: 100.0 (#1/46)
- Epoch AI - Weirdml V3: 35.47 (#2/17)
- Harvey LAB-AA - Capital Markets: 96.47 (#2/25)
- Chatbot Arena (Text - Mathematical): 1539.0 (#2/397)
- LLM Chess (Saplin): 1611.585 (#2/178)
- Legal Benchmarks - Data Extraction - Form: 83.0 (#2/24)
- Epoch AI - ECI: 166.09 (#3/274)
- LLM Stats (Humanity's Last Exam (no tools, text-only)): 52.9 (#3/12)
- ...and 103 more
Claude Sonnet 5.5 — Aggregate Rating 1942, #10
- HealthBench-Psych - Hard: 45.159 (#1/25)
- Chatbot Arena (Image-to-WebDev): 1740.0 (#2/64)
- Design Arena (Data Viz): 1382.0 (#2/169)
- DRACO: 87.0 (#2/15)
- Terminal-Bench 4.0 (Claude Code): 61.82 (#2/18)
- Clerk LLM Leaderboard: 91.7 (#3/25)
- WebDev Arena: 1786.26 (#3/134)
- Epoch AI - Frontiermath Erdos: 2.9 (#3/7)
- CursorBench 4.0: 55.5 (#4/68)
- Design Arena (Game Dev): 1440.0 (#4/175)
- ...and 45 more
Claude Opus 5 — Aggregate Rating 1934, #11
- Agents on Rails: 92.06349206349206 (#1/17)
- FoodTruckBench: 75263.63 (#1/57)
- LLMConfBench: 0.61 (#2/28)
- DeepSWE: 73.6 (#3/70)
- Clerk LLM Leaderboard: 91.5 (#4/25)
- YC-Bench: 1911.5916200000001 (#6/53)
- Terminal-Bench 4.0 (Claude Code): 53.94 (#7/18)
- DeepSWE-mini (16 Tasks): 75.0 (#8/70)
- CursorBench 4.0: 46.6 (#13/68)
- AA-Omniscience Accuracy: 60.87 (#15/571)
- ...and 3 more
GPT-6.1 Pro Sol — Aggregate Rating 1912, #12
- Arabic Broad Leaderboard - MMLU: 9.917 (#4/120)
- PRISM (1C:Enterprise) - Platform Tasks (B): 90.0 (#5/60)
- Arabic Broad Leaderboard - Translation (incl Dialects): 8.056 (#7/120)
- PRISM (1C:Enterprise) - Algorithmic Tasks (A): 93.0 (#10/60)
- Arabic Broad Leaderboard: 9.053 (#11/120)
- Arabic Broad Leaderboard - Trust & Safety: 9.667 (#13/120)
- Arabic Broad Leaderboard - RAG QA: 8.39 (#20/120)
- Arabic Broad Leaderboard - Reasoning & Math: 8.837 (#30/120)
- Arabic Broad Leaderboard - Writing (incl Dialects): 7.727 (#35/120)
- Arabic Broad Leaderboard - General Knowledge: 8.524 (#58/120)
Claude Fable 5 — Aggregate Rating 1912, #13
- MirrorCode: 63.9 (#1/6)
- SolarBench: 53.75 (#1/11)
- Agents on Rails: 90.47619047619048 (#4/17)
- AA-Omniscience Accuracy: 65.35 (#5/571)
- Clerk LLM Leaderboard: 90.5 (#7/25)
- Terminal-Bench 4.0 (Claude Code): 44.55 (#11/18)
- DeepSWE: 69.9 (#12/70)
- RuneBench: 6.006744738811 (#12/90)
- DeepSWE-mini (16 Tasks): 67.71 (#21/70)
GPT-6 Sol — Aggregate Rating 1908, #14
- Chatbot Arena (Image-to-WebDev): 1661.0 (#6/64)
- Epoch AI - ECI: 162.72 (#7/274)
- AutomationBench: 33.2 (#7/109)
- Vals AI ReverseEngBench: 30.53 (#7/33)
- Kaggle Open Benchmarks - MMLU: 93.94 (#7/55)
- Kaggle Open Benchmarks - MATH-500: 98.64 (#7/21)
- Harvey LAB-AA: 3.61 (#8/25)
- WDCD: 83.7 (#8/15)
- WDCD R1 Understanding: 100.0 (#8/15)
- WDCD R3 Pressure Integrity: 100.0 (#8/15)
- ...and 53 more
GPT-5.6 Sol — Aggregate Rating 1892, #16
- MirrorCode: 20.0 (#3/6)
- Thinkingbox: 61.91 (#3/17)
- SolarBench: 33.75 (#4/11)
- LLMConfBench: 0.4 (#4/28)
- DeepSWE-mini (16 Tasks): 78.12 (#5/70)
- Agents on Rails: 84.12698412698413 (#5/17)
- LLM Chess (Saplin): 1549.708 (#5/178)
- FoodTruckBench: 53229.1 (#6/57)
- DeepSWE: 72.7 (#9/70)
- AutomationBench: 28.77 (#14/109)
- ...and 5 more
Gemini 3.8 Flash — Aggregate Rating 1884, #18
- Chatbot Arena (Vision - Chinese): 1380.0 (#1/125)
- PUMA Polish Multimodal - Contemporary Life: 84.0 (#1/76)
- PUMA Polish Multimodal - Geography and Environment: 91.0 (#1/76)
- DeepSWE: 73.8 (#2/70)
- PUMA Polish Multimodal - History and Culture: 85.0 (#2/76)
- PUMA Polish Multimodal - Vision Mean: 84.0 (#2/76)
- PUMA Polish Multimodal - Document QA: 88.0 (#3/76)
- Arabic Broad Leaderboard - Reasoning & Math: 9.767 (#3/120)
- Harvey LAB-AA - Tax: 88.41 (#5/25)
- PUMA Polish Multimodal - OCR: 88.0 (#5/76)
- ...and 40 more
Gemini 3.7 Flash — Aggregate Rating 1874, #20
- AI Chess Leaderboard (Reasoning): 1851.0 (#2/340)
- AI Chess Leaderboard (Continuation): 1746.0 (#4/281)
- Chatbot Arena (Vision - Humor): 1299.0 (#5/91)
- AutomationBench: 30.44 (#10/109)
- Chatbot Arena (Image-to-WebDev): 1603.0 (#14/64)
- Agents on Rails: 71.42857142857143 (#14/17)
- Clerk LLM Leaderboard: 88.3 (#17/25)
- DeepSWE-mini (16 Tasks): 68.75 (#20/70)
- DeepSWE: 65.5 (#25/70)
- ForecastBench: 66.1 (#59/297)
- ...and 1 more
GPT-5.5 (2026-04-23) — Aggregate Rating 1874, #21
- Embodied-BenchForge - Static Scene: 57.67 (#1/11)
- Embodied-BenchForge - Success Rate: 83.18 (#1/11)
- Embodied-BenchForge - Embodied QA: 73.64 (#1/11)
- FoodTruckBench: 61408.09 (#3/57)
- YC-Bench: 1206.0448766666666 (#23/53)
Muse Spark 1.3 — Aggregate Rating 1870, #22
- Harvey LAB-AA - All Pass: 26.67 (#1/25)
- Harvey LAB-AA - Antitrust & Competition: 95.17 (#1/25)
- Harvey LAB-AA - Arbitration & International Dispute Resolution: 96.65 (#1/25)
- Harvey LAB-AA - Banking & Finance: 97.64 (#1/25)
- Harvey LAB-AA - Bankruptcy & Restructuring: 95.62 (#1/25)
- Harvey LAB-AA - Capital Markets: 97.65 (#1/25)
- Harvey LAB-AA - Corporate Governance: 97.37 (#1/25)
- Harvey LAB-AA - Data Privacy & Cybersecurity: 98.12 (#1/25)
- Harvey LAB-AA - Emerging Companies & Venture Capital: 96.7 (#1/25)
- Harvey LAB-AA - Employment & Labor: 93.63 (#1/25)
- ...and 49 more
Grok 4.7 — Aggregate Rating 1869, #23
- Harvey LAB-AA: 9.44 (#1/25)
- Harvey LAB-AA - Immigration: 93.21 (#1/25)
- Harvey LAB-AA - All Pass: 16.67 (#2/25)
- Harvey LAB-AA - International Trade & Sanctions: 94.02 (#2/25)
- Harvey LAB-AA - Litigation & Dispute Resolution: 94.36 (#2/25)
- Harvey LAB-AA - Real Estate: 96.31 (#2/25)
- Harvey LAB-AA - White Collar Defense & Investigations: 93.38 (#2/25)
- Harvey LAB-AA - Antitrust & Competition: 93.12 (#3/25)
- Harvey LAB-AA - Banking & Finance: 95.4 (#3/25)
- Harvey LAB-AA - Funds & Asset Management: 90.48 (#3/25)
- ...and 69 more
Grok 4.6 — Aggregate Rating 1867, #24
- Kaggle Open Benchmarks - MATH-500: 99.08 (#3/21)
- FoodTruckBench: 59537.27 (#4/57)
- Agents on Rails: 82.53968253968253 (#6/17)
- Kaggle Open Benchmarks - MMLU: 93.97 (#6/55)
- YC-Bench: 1568.2406299999998 (#14/53)
- Kaggle SimpleQA Verified: 52.86 (#14/52)
- Kaggle SimpleQA: 49.49 (#17/45)
- Kaggle VideoQA: 60.0 (#17/17)
- RuneBench: 5.833968889621 (#18/90)
- DeepSWE-mini (16 Tasks): 70.31 (#19/70)
- ...and 3 more
Kimi K3 — Aggregate Rating 1858, #25
- Agents on Rails: 90.47619047619048 (#3/17)
- YC-Bench: 2053.29564 (#4/53)
- LLMConfBench: 0.38 (#5/28)
- SolarBench: 26.25 (#6/11)
- Thinkingbox: 57.37 (#7/17)
- Clerk LLM Leaderboard: 90.1 (#11/25)
- DeepSWE-mini (16 Tasks): 70.31 (#16/70)
- TaxCalcBench: 32.0 (#16/45)
- DeepSWE: 68.5 (#19/70)
- AutomationBench: 22.68 (#23/109)
- ...and 1 more
GPT-5.5 — Aggregate Rating 1853, #26
- CoffeeBench: 3109.19 (#1/7)
- WeaveBench: 33.3 (#2/10)
- ContractScrub: 0.632 (#3/12)
- MirrorCode: 10.0 (#5/6)
- TERMS-Bench: 10.145702598770821 (#9/15)
- RuneBench: 5.679067544577 (#24/90)
- DeepSWE-mini (16 Tasks): 65.62 (#28/70)
- DeepSWE: 64.4 (#29/70)
- AutomationBench: 16.59 (#37/109)
Qwen 3.8 Max (0902) — Aggregate Rating 1853, #27
- Last Translation Benchmark: 12.513721185510429 (#12/45)
- Epoch AI - Frontierswe: 17.81 (#16/19)
- Terminal-Bench 4.0 (Claude Code): 26.97 (#16/18)
- FrontierSWE V2: 17.809649411764706 (#18/22)
- Tinybird AI SQL Benchmark - Success Rate: 100.0 (#108/380)
- Tinybird AI SQL Benchmark - Exactness: 49.77 (#165/373)
- Tinybird AI SQL Benchmark - First-Attempt Success Rate: 94.0 (#183/380)
Qwen 3.8 Max — Aggregate Rating 1852, #28
- YC-Bench: 2095.8380700000002 (#3/53)
- Epoch AI - Apex Agents: 63.3 (#11/52)
- FrontierSWE V2: 15.809350588235294 (#19/22)
- DeepSWE: 57.5 (#37/70)
- DeepSWE-mini (16 Tasks): 45.31 (#51/70)
Step 5 Preview — Aggregate Rating 1844, #31
- AI Chess Leaderboard (Reasoning): 1800.0 (#7/340)
- RSI-Exam: 0.352 (#12/18)
- ReLE - Law - Lawyer Qualification Exam: 81.3 (#17/189)
- ReLE - Medicine - Physician Exams: 87.8 (#21/189)
- ReLE - Law and Civil Service: 84.7 (#23/189)
- ReLE - Education - Primary School Subjects: 68.7 (#24/190)
- ReLE - Coding: 68.7 (#27/88)
- Chatbot Arena (Image-to-WebDev): 1541.0 (#30/64)
- WebDev Arena: 1570.15 (#30/134)
- Chatbot Arena (Vision - English): 1276.0 (#32/165)
- ...and 22 more
Claude Haiku 5.5 — Aggregate Rating 1843, #32
- LLM Stats (HealthBench): 61.6 (#2/14)
- LLM Stats (Program Bench): 82.0 (#2/13)
- LiveBench Math Comp: 98.04 (#3/69)
- Vals AI Vibe Code Bench: 90.44 (#3/110)
- LLM Stats (BenchCAD): 67.0 (#4/7)
- LLM Stats (BenchCAD (with Python tool)): 87.0 (#4/7)
- Featherbench: 100.0 (#4/29)
- MLCR-AA: 63.33 (#5/95)
- LLM Stats (HealthBench Professional): 64.8 (#5/16)
- LLM Stats (Humanity's Last Exam (no tools, text-only)): 45.9 (#5/12)
- ...and 242 more
Claude Opus 4.8 — Aggregate Rating 1843, #33
- Agents on Rails: 79.36507936507937 (#9/17)
- Terminal-Bench 4.0 (Claude Code): 23.64 (#17/18)
- DeepSWE: 59.0 (#35/70)
- DeepSWE-mini (16 Tasks): 55.0 (#36/70)
- AutomationBench: 16.89 (#36/109)
- RuneBench: 5.083611863145 (#43/90)
- LLM Chess (Saplin): 449.807 (#56/178)
- AA-Omniscience Accuracy: 48.83 (#67/571)
MiMo-V2.6-Pro — Aggregate Rating 1841, #34
- SuperCLUE-Terminal - Overall: 64.65 (#1/18)
- ErdosBench: 3.02 (#5/16)
- Epoch AI - Scicode: 60.88 (#8/213)
- FoodTruckBench: 49373.25 (#9/57)
- F5 Labs AI Security Leaderboard - ARS: 84.69 (#9/10)
- Epoch AI - Apex Agents: 59.5 (#14/52)
- Epoch AI - Proofbench: 70.0 (#15/44)
- Chatbot Arena (Image-to-WebDev): 1595.0 (#18/64)
- WebDev Arena: 1618.41 (#18/134)
- Epoch AI - Critpt: 26.57 (#35/224)
- ...and 5 more
DeepSeek V4.1 Flash — Aggregate Rating 1840, #35
- Clerk LLM Leaderboard: 92.0 (#1/25)
- Harvey LAB-AA - Antitrust & Competition: 94.2 (#2/25)
- Harvey LAB-AA - Emerging Companies & Venture Capital: 94.2 (#2/25)
- Harvey LAB-AA - Employment & Labor: 90.51 (#2/25)
- Arabic Broad Leaderboard - Writing (incl Dialects): 8.136 (#2/120)
- Harvey LAB-AA - Corporate M&A: 95.71 (#4/25)
- Harvey LAB-AA - Healthcare & Life Sciences: 94.41 (#4/25)
- Harvey LAB-AA - International Trade & Sanctions: 93.63 (#4/25)
- Harvey LAB-AA - Corporate Governance: 92.69 (#5/25)
- Harvey LAB-AA - Environmental & ESG: 90.56 (#5/25)
- ...and 61 more
GLM-5.3 — Aggregate Rating 1840, #36
- Harvey LAB-AA - Bankruptcy & Restructuring: 92.39 (#2/25)
- Harvey LAB-AA - Environmental & ESG: 90.92 (#2/25)
- Harvey LAB-AA - Healthcare & Life Sciences: 95.08 (#2/25)
- Harvey LAB-AA - Immigration: 92.97 (#2/25)
- Harvey LAB-AA - Intellectual Property: 96.2 (#2/25)
- Harvey LAB-AA - International Trade & Sanctions: 94.02 (#3/25)
- Harvey LAB-AA - Litigation & Dispute Resolution: 93.98 (#3/25)
- Harvey LAB-AA - Tax: 89.52 (#3/25)
- Harvey LAB-AA - Antitrust & Competition: 93.12 (#4/25)
- Harvey LAB-AA - Corporate Governance: 93.72 (#4/25)
- ...and 27 more
Hy4 preview — Aggregate Rating 1837, #37
- Clerk LLM Leaderboard: 90.3 (#9/25)
- FoodTruckBench: 39037.43 (#13/57)
GPT-5.6 Terra — Aggregate Rating 1836, #38
- SolarBench: 6.25 (#10/11)
- Agents on Rails: 77.77777777777779 (#10/17)
- Vals AI ReverseEngBench: 9.54 (#13/33)
- DeepSWE-mini (16 Tasks): 70.31 (#14/70)
- DeepSWE: 69.6 (#14/70)
- LLM Chess (Saplin): 1293.005 (#16/178)
- RuneBench: 5.881387994547 (#17/90)
- AutomationBench: 23.59 (#22/109)
- CursorBench 4.0: 41.3 (#28/68)
- YC-Bench: 636.14367 (#35/53)
Ember-1 — Aggregate Rating 1831, #39
- LLM Stats (DeepSWE 1.1): 75.2 (#4/43)
- LLM Stats (Terminal-Bench 2.1): 82.0 (#22/43)
- DuelLab Overall: 46.145337370657 (#41/191)
- LLM Stats Score: 43.14 (#51/383)
Gemini 3.6 Flash — Aggregate Rating 1828, #41
- LLMConfBench: 0.43 (#3/28)
- Clerk LLM Leaderboard: 88.7 (#16/25)
- YC-Bench: 968.1231300000001 (#31/53)
- AutomationBench: 17.05 (#34/109)
- DeepSWE-mini (16 Tasks): 45.31 (#52/70)
- DeepSWE: 46.7 (#53/70)
Muse Spark 1.1 — Aggregate Rating 1827, #43
- SolarBench: 42.5 (#2/11)
- ProgramBench Almost: 4.0 (#13/25)
- ProgramBench: 0.0 (#14/25)
- Epoch AI - Scicode: 58.8 (#17/213)
- DeepSWE-mini (16 Tasks): 54.69 (#40/70)
- Epoch AI - Apex Agents: 31.8 (#46/52)
- DeepSWE: 53.3 (#47/70)
- YC-Bench: 28.636860000000002 (#51/53)
- Epoch AI - Critpt: 15.14 (#83/224)
Grok 4.5 — Aggregate Rating 1826, #44
- YC-Bench: 2629.90251 (#2/53)
- SolarBench: 38.75 (#3/11)
- FoodTruckBench: 34585.92 (#15/57)
- Clerk LLM Leaderboard: 73.1 (#24/25)
- RuneBench: 5.349501689433 (#36/90)
- DeepSWE-mini (16 Tasks): 50.0 (#44/70)
- DeepSWE: 53.8 (#45/70)
Claude Opus 4.7 — Aggregate Rating 1826, #45
- WeaveBench: 35.1 (#1/10)
- MirrorCode: 31.1 (#2/6)
- CoffeeBench: 2781.933333333333 (#2/7)
- Embodied-BenchForge - Static Scene: 56.61 (#2/11)
- Embodied-BenchForge - Success Rate: 77.73 (#2/11)
- Embodied-BenchForge - Embodied QA: 70.0 (#2/11)
- TERMS-Bench: 11.09280671414272 (#3/15)
- ContractScrub: 0.621 (#4/12)
- PrinzBench: 25.0 (#22/29)
- LLM Chess (Saplin): 691.058 (#49/178)
- ...and 3 more
Qwen 3.8 Flash — Aggregate Rating 1824, #47
- DuelLab Overall: 42.50970698502 (#52/191)
DeepSeek V4 Pro (0813) — Aggregate Rating 1821, #48
- Arcee MMLU-Pro (BenchLM): 87.5 (#6/62)
- FoodTruckBench: 37349.92 (#14/57)
- YC-Bench: 1459.74346 (#16/53)
- Vals AI ReverseEngBench: 1.91 (#24/33)
- DuelLab Overall: 26.441089776165 (#133/191)
- Epoch AI - Scicode: 39.93 (#160/213)
- Epoch AI - Critpt: 0.0 (#224/224)
GPT-6 Luna — Aggregate Rating 1819, #49
- F5 Labs AI Security Leaderboard - CASI: 93.52 (#6/10)
- F5 Labs AI Security Leaderboard - ARS: 86.6 (#7/10)
- o11y-bench - Pass@3: 87.3 (#8/58)
- Harvey LAB-AA: 3.33 (#9/25)
- o11y-bench - Pass^3: 66.67 (#9/58)
- Harvey LAB-AA - Real Estate: 93.78 (#11/25)
- WDCD R1 Understanding: 100.0 (#12/15)
- WDCD R3 Pressure Integrity: 88.0 (#12/15)
- HealthBench-Psych - Hard: 33.593 (#12/25)
- PUMA Polish Multimodal - Contemporary Life: 64.0 (#13/76)
- ...and 56 more
Qwen3.8 2.4T A95B — Aggregate Rating 1817, #50
- Tinybird AI SQL Benchmark - Exactness: 56.03 (#31/373)
- Tinybird AI SQL Benchmark - Success Rate: 100.0 (#127/380)
- Tinybird AI SQL Benchmark - First-Attempt Success Rate: 96.0 (#139/380)
Ling 3.1 Flash — Aggregate Rating 1817, #51
- LLM Stats (Finance Agent v2): 57.87 (#2/29)
- LLM Stats (SkillsBench): 68.7 (#2/12)
- LLM Stats (AutomationBench): 52.5 (#4/25)
- LLM Stats (CyberGym): 87.9 (#4/22)
- LLM Stats (FrontierSWE): 75.16 (#4/17)
- LLM Stats (HealthBench Professional): 65.35 (#4/16)
- LLM Stats (Multi-Challenge): 69.78 (#4/33)
- LLM Stats (Terminal-Bench 4.0): 40.4 (#10/25)
- LLM Stats Score: 50.99 (#15/383)
- BenchLM: 62.1 (#46/200)
- ...and 1 more
Muse Spark 1.2 — Aggregate Rating 1816, #53
- Chatbot Arena (Text - French): 1534.0 (#1/293)
- Chatbot Arena (Text - Spanish): 1505.0 (#1/298)
- ProgramBench Almost: 4.5 (#10/25)
- ProgramBench: 0.0 (#12/25)
- Agents on Rails: 76.19047619047619 (#12/17)
- FoodTruckBench: 41904.3 (#12/57)
- Vals AI ReverseEngBench: 0.0 (#31/33)
- DeepSWE-mini (16 Tasks): 54.69 (#38/70)
- DeepSWE: 54.9 (#39/70)
- YC-Bench: 9.07025 (#53/53)
Gemini 3.1 Pro (Preview) — Aggregate Rating 1813, #54
- ContractScrub: 0.655 (#1/12)
- TsuGO: 80.0 (#1/9)
- Fibble3 Arena: 100.0 (#2/35)
- Fibble4 Arena: 100.0 (#2/37)
- AuthMem-Bench (Task Success Rate): 52.9 (#2/7)
- TRAPSBench: 82.0 (#2/14)
- OrchBench: 0.801 (#2/9)
- Fibble Arena: 100.0 (#3/39)
- CoffeeBench: 1695.0533333333333 (#4/7)
- TERMS-Bench: 10.577861108852488 (#5/15)
- ...and 15 more
Gemini 3.5 Flash — Aggregate Rating 1812, #55
- AI Chess Leaderboard (Continuation): 1706.0 (#8/281)
- SolarBench: 11.25 (#9/11)
- AI Chess Leaderboard (Reasoning): 1721.0 (#13/340)
- FoodTruckBench: 22311.44 (#28/57)
- WebDev Arena (HTML): 1568.0 (#32/141)
- Chatbot Arena (Image-to-WebDev): 1512.0 (#41/64)
- WebDev Arena (Simulations): 1558.0 (#41/141)
- RuneBench: 4.942324501573 (#46/90)
- WebDev Arena (Gaming): 1553.0 (#47/141)
- WebDev Arena (Frontend): 1525.0 (#48/142)
- ...and 9 more
Claude Sonnet 5 — Aggregate Rating 1812, #56
- LLMConfBench: 0.37 (#6/28)
- Benchmarks.bio - scBench: 59.52 (#6/23)
- SolarBench: 12.5 (#8/11)
- Benchmarks.bio - SpatialBench: 69.28 (#10/22)
- Agents on Rails: 69.84126984126983 (#15/17)
- Terminal-Bench 4.0 (Claude Code): 12.42 (#18/18)
- Clerk LLM Leaderboard: 85.7 (#19/25)
- Vals AI ReverseEngBench: 0.38 (#29/33)
- LLM Chess (Saplin): 805.861 (#40/178)
- DeepSWE: 53.8 (#42/70)
- ...and 8 more
Qwen3.8 Omni Flash — Aggregate Rating 1810, #58
- LLM Stats (WildClawBench): 71.0 (#1/6)
- LLM Stats Score: 37.83 (#79/383)
- Tinybird AI SQL Benchmark - Exactness: 52.72 (#91/373)
- Tinybird AI SQL Benchmark - First-Attempt Success Rate: 92.0 (#228/380)
- Tinybird AI SQL Benchmark - Success Rate: 98.0 (#263/380)
GPT-5.4 (2026-03-05) — Aggregate Rating 1809, #59
- LiveMathematicianBench: 45.243619489559165 (#2/19)
Claude Opus 4.6 — Aggregate Rating 1809, #60
- MageBench S1: 1747.0 (#1/35)
- TERMS-Bench: 11.615915923770823 (#2/15)
- FoodTruckBench: 49519.37 (#8/57)
- PrinzBench: 19.0 (#27/29)
- LLM Chess (Saplin): 198.192 (#75/178)
- AA-Omniscience Accuracy: 46.98 (#76/571)
GLM-5.3 Flash — Aggregate Rating 1803, #61
- Harvey LAB-AA - Emerging Companies & Venture Capital: 93.97 (#3/25)
- Harvey LAB-AA - Intellectual Property: 96.09 (#4/25)
- Harvey LAB-AA - Funds & Asset Management: 89.7 (#5/25)
- Harvey LAB-AA - Antitrust & Competition: 92.39 (#6/25)
- Harvey LAB-AA - Corporate Governance: 92.01 (#6/25)
- Harvey LAB-AA - Environmental & ESG: 90.08 (#6/25)
- Harvey LAB-AA - Tax: 86.96 (#6/25)
- Agents on Rails: 82.53968253968253 (#7/17)
- Harvey LAB-AA - All Pass: 10.0 (#8/25)
- Harvey LAB-AA - Immigration: 92.36 (#8/25)
- ...and 97 more
MiMo-V2.6-Flash — Aggregate Rating 1803, #62
- SuperCLUE-Terminal - Overall: 53.54 (#5/18)
- WebDev Arena (HTML): 1643.0 (#14/141)
- WebDev Arena (Gaming): 1708.0 (#14/141)
- Chatbot Arena (Code): 1637.0 (#16/142)
- WebDev Arena (Frontend): 1655.0 (#16/142)
- Chatbot Arena (Image-to-WebDev): 1597.0 (#17/64)
- Epoch AI - Proofbench: 63.0 (#17/44)
- WebDev Arena (Reference-Based Design): 1654.0 (#17/141)
- WebDev Arena (React): 1652.0 (#18/126)
- WebDev Arena (Brand & Marketing): 1630.0 (#18/141)
- ...and 12 more
Gemini 3 Pro (Preview) — Aggregate Rating 1801, #64
- MageBench S1: 1722.0 (#3/35)
- Spreadsheet Arena: 54.651162790698 (#5/14)
- MageBench: 1617.0 (#8/22)
- Multivac Evaluation: 7.2 (#18/19)
- FoodTruckBench: 17199.34 (#31/57)
GPT-5.6 Luna — Aggregate Rating 1800, #66
- Agents on Rails: 73.01587301587301 (#13/17)
- FoodTruckBench: 32931.72 (#16/57)
- DeepSWE: 67.2 (#21/70)
- LLM Chess (Saplin): 1185.297 (#23/178)
- Vals AI ReverseEngBench: 1.53 (#25/33)
- DeepSWE-mini (16 Tasks): 60.94 (#31/70)
- AutomationBench: 17.05 (#33/109)
- Epoch AI - Apex Agents: 43.0 (#38/52)
- RuneBench: 5.285493006366 (#39/90)
- CursorBench 4.0: 35.9 (#40/68)
- ...and 1 more
Qwen3.8-Flash-Next — Aggregate Rating 1799, #67
- SEA-HELM (English): 70.84868726454363 (#2/58)
- SEA-HELM (Indonesian): 79.04193333844408 (#2/58)
- SEA-HELM (Filipino): 75.59006552978165 (#3/58)
- SEA-HELM (Thai): 72.99170696782426 (#5/58)
- SEA-HELM (Malay): 74.92644503595338 (#7/58)
- SEA-HELM (Burmese): 66.77668337666796 (#8/58)
- SEA-HELM (Vietnamese): 74.93247788363739 (#8/58)
- SEA-HELM (Tamil): 67.04010644507768 (#16/58)
GPT-5.3 Codex — Aggregate Rating 1798, #68
- WeaveBench: 18.4 (#4/10)
- MageBench S1: 1717.0 (#4/35)
- MageBench S2: 1616.0 (#5/11)
GLM-5.2 — Aggregate Rating 1794, #69
- SolarBench: 23.75 (#7/11)
- Agents on Rails: 66.66666666666666 (#16/17)
- Epoch AI - Apex Agents: 45.2 (#34/52)
- DeepSWE: 43.8 (#56/70)
- DeepSWE-mini (16 Tasks): 40.62 (#57/70)
- AutomationBench: 7.76 (#71/109)
Claude Opus 4.5 (20251101) — Aggregate Rating 1791, #70
- Spreadsheet Arena: 73.563218390805 (#1/14)
Qwen 3.7 Max — Aggregate Rating 1788, #71
- Arcee MMLU-Pro (BenchLM): 89.6 (#1/62)
- LLMConfBench: 0.32 (#9/28)
- FoodTruckBench: 27826.69 (#20/57)
- DuelLab Overall: 30.899796153952 (#105/191)
DeepSeek-V4-Flash-Vision-Exp — Aggregate Rating 1788, #73
- FrontierSWE V2: 14.831252647058824 (#20/22)
DeepSeek V4 Flash (0731) — Aggregate Rating 1787, #74
- V-FiLLM - Simplified Statements (Multi-Turn): 88.0 (#4/6)
- V-FiLLM - 10-Q Filings: 96.4 (#5/6)
- LLMConfBench: 0.33 (#7/28)
- Arcee MMLU-Pro (BenchLM): 86.2 (#13/62)
- FoodTruckBench: 24038.94 (#25/57)
GPT-5 Pro — Aggregate Rating 1786, #75
- TRAPSBench: 80.7 (#5/14)
GPT-5.4 — Aggregate Rating 1785, #76
- Logical Consistency Reformulation - Base Accuracy: 98.9 (#2/7)
- IH-Benchmark: 97.6 (#2/37)
- Thinkingbox: 65.36 (#2/17)
- F5 Labs AI Security Leaderboard - CASI: 95.76 (#2/10)
- WeaveBench: 22.8 (#3/10)
- Poker Arena: 18.0 (#3/7)
- MirrorCode: 15.6 (#4/6)
- MageBench S1: 1658.0 (#7/35)
- AutomationBench: 11.57 (#57/109)
Gemini 3 Flash (Preview) — Aggregate Rating 1782, #78
- CLEM TextMapWorld: 91.3458 (#1/31)
- MageBench: 1694.0 (#2/22)
- CLEM GuessWhat: 92.78 (#2/28)
- CLEM ReferenceGame: 100.0 (#3/31)
- CLEM TextMapWorld GraphReasoning: 83.082366 (#3/27)
- CLEM Wordle with Clue: 80.11 (#3/31)
- Embodied-BenchForge - Static Scene: 51.97 (#3/11)
- Embodied-BenchForge - Success Rate: 72.27 (#3/11)
- Embodied-BenchForge - Embodied QA: 59.55 (#3/11)
- CLEM Wordle with Critic: 73.889361 (#4/30)
- ...and 17 more
Claude Opus 4.5 — Aggregate Rating 1781, #79
- Arcee MMLU-Pro (BenchLM): 89.5 (#2/62)
- Kernel Arena - KernelBench HIP: 3.112439024390244 (#3/7)
- PrinzBench: 14.0 (#28/29)
Kimi K2.7 Code — Aggregate Rating 1776, #81
- SolarBench: 5.0 (#11/11)
- AutomationBench: 3.2 (#92/109)
- DuelLab Overall: 28.442446138644 (#122/191)
Seed 2.1 Turbo — Aggregate Rating 1772, #84
- Tinybird AI SQL Benchmark - Success Rate: 100.0 (#48/380)
- Tinybird AI SQL Benchmark - First-Attempt Success Rate: 94.0 (#174/380)
- Tinybird AI SQL Benchmark - Exactness: 43.58 (#268/373)
Claude Sonnet 4.6 — Aggregate Rating 1771, #85
- ContractScrub: 0.637 (#2/12)
- CoffeeBench: 2235.9366666666665 (#3/7)
- TimeSage-EV: 82.38 (#3/6)
- Embodied-BenchForge - Static Scene: 42.54 (#7/11)
- Embodied-BenchForge - Success Rate: 16.82 (#7/11)
- Embodied-BenchForge - Embodied QA: 42.27 (#10/11)
- PrinzBench: 8.0 (#29/29)
- FoodTruckBench: 17425.83 (#29/57)
- Arcee MMLU-Pro (BenchLM): 79.2 (#39/62)
- DeepSWE: 29.9 (#65/70)
- ...and 3 more
Qwen 3.7 Plus — Aggregate Rating 1771, #86
- Arcee MMLU-Pro (BenchLM): 88.5 (#3/62)
- DuelLab Overall: 32.704812491587 (#100/191)
Hy3 — Aggregate Rating 1771, #88
- EarthVerse: 72.77 (#9/25)
- FoodTruckBench: 28055.39 (#18/57)
- Chatbot Arena (Text - Japanese): 1415.0 (#73/278)
GPT-5.2 Codex — Aggregate Rating 1769, #89
- WeaveBench: 6.1 (#5/10)
- Multivac Evaluation: 8.7 (#14/19)
GPT-5 — Aggregate Rating 1766, #91
- TRAPSBench: 82.2 (#1/14)
- Spreadsheet Arena: 37.5 (#7/14)
- LiveMathematicianBench: 28.306264501160094 (#8/19)
- LLMConfBench: 0.3 (#12/28)
- MageBench S1: 1536.0 (#30/35)
- DuelLab Overall: 34.148507411632 (#84/191)
GLM-5.1 — Aggregate Rating 1765, #92
- TERMS-Bench: 11.696256743843419 (#1/15)
- EdgeBench: 37.4 (#1/2)
- CoffeeBench: 1597.1166666666668 (#5/7)
- Thinkingbox: 33.19 (#11/17)
- AutomationBench: 6.2 (#76/109)
Qwen 3.8 27B — Aggregate Rating 1765, #95
- LLMConfBench: 0.31 (#10/28)
- Agents on Rails: 76.19047619047619 (#11/17)
- AA-AnalystAgent - Pass@K: 75.0 (#14/39)
- AA-AnalystAgent - Pass@1: 48.0 (#24/39)
- AA-AnalystAgent: 18.75 (#26/39)
- Epoch AI - Apex Agents: 47.5 (#30/52)
- Wolfram LLM Benchmarking Project: 45.6 (#271/589)
DeepSeek V4 Pro — Aggregate Rating 1764, #96
- PIJ - Criminal Profiling: 51.38 (#1/10)
- PredAct-Bench: 60.7 (#2/13)
- PIJ - Criminal Process Summarization: 44.81 (#4/10)
- PIJ - Sentence Prediction: 44.39 (#5/10)
- TERMS-Bench: 10.525148581399497 (#6/15)
- PIJ - Criminal Process Reconstruction: 53.19 (#7/10)
- Thinkingbox: 43.26 (#10/17)
- FoodTruckBench: 27142.02 (#22/57)
- DeepSWE: 62.8 (#31/70)
- DeepSWE-mini (16 Tasks): 59.38 (#33/70)
- ...and 1 more
GPT-5.1 (2025-11-13) — Aggregate Rating 1763, #97
- Spreadsheet Arena: 33.980582524272 (#9/14)
Kimi K2.6 — Aggregate Rating 1760, #99
- CoffeeBench: 453.93 (#6/7)
- TERMS-Bench: 10.166852532923977 (#8/15)
- AutomationBench: 2.59 (#94/109)
Mistral Large 4 — Aggregate Rating 1760, #100
- LLM Stats (AutomationBench): 59.9 (#1/25)
- Polish Knowledge Benchmark - Economy & Business: 84.39 (#3/69)
- Polish Knowledge Benchmark - Education & Science: 86.61 (#3/69)
- LLM Stats (Finance Agent v2): 54.7 (#6/29)
- Vals AI Harvey Legal Agent Bench: 15.83 (#6/76)
- Polish Knowledge Benchmark - Law & Administration: 92.15 (#6/69)
- Polish Knowledge Benchmark - Ecology & Environment: 86.39 (#6/69)
- Polish Knowledge Benchmark - Media & Journalism: 81.58 (#6/69)
- Polish Knowledge Benchmark - Polish Symbols: 87.23 (#8/69)
- Polish Knowledge Benchmark - Military: 88.79 (#8/69)
- ...and 171 more
Grok Build 0.1 — Aggregate Rating 1757, #104
- DuelLab Overall: 24.10934469322 (#148/191)
GPT-5.2 — Aggregate Rating 1756, #105
- CLEM ImageGame: 99.92 (#1/26)
- RepoProbe: 62.7 (#1/20)
- MageBench S1: 1737.0 (#2/35)
- CLEM Clean Up: 98.24 (#3/27)
- Spreadsheet Arena: 60.810810810811 (#4/14)
- CLEM Codenames: 71.534907 (#4/30)
- CLEM Taboo: 95.56 (#4/31)
- F5 Labs AI Security Leaderboard - CASI: 94.33 (#4/10)
- CLEM Wordle: 62.33 (#5/28)
- CLEM Clemscore: 81.66 (#6/31)
- ...and 16 more
O3 Pro — Aggregate Rating 1755, #107
- Thinkingbox: 19.31 (#13/17)
Seed 2.0 Pro — Aggregate Rating 1754, #111
- TERMS-Bench: 8.61154484726998 (#11/15)
Qwen 3.6 Plus — Aggregate Rating 1753, #113
- Arcee MMLU-Pro (BenchLM): 88.5 (#4/62)
- TERMS-Bench: 10.303802705852293 (#7/15)
- FoodTruckBench: 7668.06 (#37/57)
- AutomationBench: 4.3 (#84/109)
O3 (2025-04-16) — Aggregate Rating 1751, #117
- ResearchCodeBench: 59.43396226415094 (#2/32)
- LiveMathematicianBench: 24.361948955916475 (#15/19)
Nex N2 Pro — Aggregate Rating 1751, #118
- FoodTruckBench: 24923.12 (#23/57)
- DuelLab Overall: 15.775972719151 (#187/191)
Inkling — Aggregate Rating 1750, #119
- PACT - Rule-Scope Discernment: 0.909 (#1/22)
- PACT - Pushback Resistance: 0.974 (#5/22)
- PACT: 0.931 (#6/22)
- PACT - Transparency: 0.177 (#6/22)
- PACT - Pressure Resistance: 0.94 (#9/22)
- PACT - Default Compliance: 0.949 (#11/22)
- Harvey LAB-AA - Corporate Governance: 87.9 (#12/25)
- Harvey LAB-AA - White Collar Defense & Investigations: 82.04 (#18/25)
- Harvey LAB-AA - Environmental & ESG: 82.08 (#19/25)
- Harvey LAB-AA - International Trade & Sanctions: 86.59 (#19/25)
- ...and 26 more
Gemini 2.5 Pro (Preview 03-25) — Aggregate Rating 1747, #120
- PlatinumBench (MIT): 0.53 (#1/34)
- VideoGameBench: 0.48 (#1/5)
- SudokuBench (Multi-step, 4x4): 73.33 (#2/14)
- ResearchCodeBench: 58.9622641509434 (#3/32)
- SudokuBench (Single-shot, 6x6): 13.33 (#3/14)
- SudokuBench (Single-shot, 4x4): 60.0 (#5/14)
- SudokuBench (Multi-step, 9x9): 0.0 (#5/12)
- SudokuBench (Multi-step, 6x6): 0.0 (#6/14)
- SudokuBench (Single-shot, 9x9): 0.0 (#7/14)
O3 — Aggregate Rating 1745, #121
- DuelLab Overall: 33.844714557036 (#87/191)
Claude Opus 4.1 (20250805) — Aggregate Rating 1739, #126
- Spreadsheet Arena: 64.285714285714 (#3/14)
MiniMax-M3 — Aggregate Rating 1738, #129
- Arabic Broad Leaderboard - RAG QA: 9.195 (#12/120)
- Arabic Broad Leaderboard - General Knowledge: 9.175 (#13/120)
- Arabic Broad Leaderboard: 8.715 (#39/120)
- Arabic Broad Leaderboard - Translation (incl Dialects): 7.444 (#39/120)
- Arabic Broad Leaderboard - Reasoning & Math: 8.605 (#41/120)
- Chatbot Arena (Vision - Entity Recognition): 1162.0 (#43/51)
- Arabic Broad Leaderboard - Writing (incl Dialects): 7.591 (#44/120)
- FoodTruckBench: 881.99 (#45/57)
- Arabic Broad Leaderboard - Trust & Safety: 9.0 (#45/120)
- Arabic Broad Leaderboard - MMLU: 9.256 (#54/120)
- ...and 2 more
DeepSeek V4 Flash — Aggregate Rating 1736, #134
- PredAct-Bench: 52.4 (#8/13)
- Agents on Rails: 65.07936507936508 (#17/17)
- AutomationBench: 18.11 (#32/109)
- FoodTruckBench: 5504.13 (#38/57)
- DeepSWE: 53.3 (#46/70)
- DeepSWE-mini (16 Tasks): 48.44 (#47/70)
- DuelLab Overall: 25.206643405162 (#141/191)
Inkling Small — Aggregate Rating 1736, #135
- Harvey LAB-AA - Corporate Governance: 87.33 (#13/25)
- Harvey LAB-AA - Capital Markets: 86.27 (#20/25)
- Harvey LAB-AA - All Pass: 1.67 (#22/25)
- Harvey LAB-AA - Corporate M&A: 80.39 (#22/25)
- Harvey LAB-AA - Healthcare & Life Sciences: 81.32 (#22/25)
- Harvey LAB-AA - Structured Finance & Securitization: 75.27 (#22/25)
- Harvey LAB-AA: 0.0 (#23/25)
- Harvey LAB-AA - Antitrust & Competition: 73.67 (#23/25)
- Harvey LAB-AA - Arbitration & International Dispute Resolution: 72.52 (#23/25)
- Harvey LAB-AA - Banking & Finance: 82.46 (#23/25)
- ...and 16 more
GLM-5 — Aggregate Rating 1735, #137
- FoodTruckBench: -210.33 (#53/57)
Claude Sonnet 4.5 — Aggregate Rating 1734, #138
- F5 Labs AI Security Leaderboard - ARS: 94.59 (#1/10)
- Spreadsheet Arena: 69.318181818182 (#2/14)
- MageBench: 1674.0 (#3/22)
- HalluScore: 1.21 (#7/17)
- LLMConfBench: 0.14 (#18/28)
- MageBench S1: 1589.0 (#20/35)
- PrinzBench: 20.0 (#26/29)
- FoodTruckBench: 1388.31 (#41/57)
Kimi K2.5 — Aggregate Rating 1734, #139
- Arcee MMLU-Pro (BenchLM): 87.1 (#7/62)
- MageBench S1: 1652.0 (#8/35)
- MageBench: 1595.0 (#12/22)
- FoodTruckBench: 29.98 (#52/57)
- BenchLM: 56.4 (#59/200)
Qwen 3.5 397B A17B — Aggregate Rating 1733, #141
- Arcee MMLU-Pro (BenchLM): 87.8 (#5/62)
- ParliamentBench: 56.0 (#6/13)
- WeaveBench: 0.9 (#8/10)
- ContractScrub: 0.316 (#12/12)
- FoodTruckBench: -218.31 (#54/57)
- BenchLM: 55.5 (#60/200)
Qwen 3.5 Plus (2026-04-20) — Aggregate Rating 1732, #142
- LLMConfBench: 0.18 (#16/28)
Gemini 2.5 Pro (Preview 05-06) — Aggregate Rating 1732, #143
- ResearchCodeBench: 64.15094339622641 (#1/32)
O1 (2024-12-17) — Aggregate Rating 1730, #144
- PlatinumBench (MIT): 0.74 (#2/34)
- ResearchCodeBench: 48.113207547169814 (#9/32)
Gemini 2.5 Pro — Aggregate Rating 1730, #145
- MageBench: 1655.0 (#4/22)
- ContractScrub: 0.557 (#5/12)
- Spreadsheet Arena: 35.714285714286 (#8/14)
- TRAPSBench: 78.1 (#8/14)
- MageBench S1: 1540.0 (#29/35)
Claude Opus 4 (20250514) — Aggregate Rating 1727, #148
- HELM Safety SimpleSafetyTests: 100.0 (#6/87)
- HELM Safety BBQ: 97.1 (#11/87)
- HELM Safety Mean score: 96.9 (#13/87)
- HELM Safety HarmBench: 91.7 (#20/87)
- HELM Safety XSTest: 97.0 (#27/87)
- HELM Safety Anthropic Red Team: 98.9 (#44/87)
Gemini 3.5 Flash Lite — Aggregate Rating 1726, #149
- Clerk LLM Leaderboard: 79.9 (#23/25)
- Harvey LAB-AA - Antitrust & Competition: 59.3 (#24/25)
- Harvey LAB-AA - Arbitration & International Dispute Resolution: 58.54 (#24/25)
- Harvey LAB-AA - Banking & Finance: 68.78 (#24/25)
- Harvey LAB-AA - Bankruptcy & Restructuring: 58.71 (#24/25)
- Harvey LAB-AA - Capital Markets: 70.33 (#24/25)
- Harvey LAB-AA - Corporate Governance: 75.0 (#24/25)
- Harvey LAB-AA - Corporate M&A: 64.09 (#24/25)
- Harvey LAB-AA - Data Privacy & Cybersecurity: 74.79 (#24/25)
- Harvey LAB-AA - Emerging Companies & Venture Capital: 68.94 (#24/25)
- ...and 19 more
Hy3-preview — Aggregate Rating 1726, #150
- YC-Bench: 124.87983 (#46/53)
MiMo-V2.5-Pro — Aggregate Rating 1725, #152
- FoodTruckBench: 22387.99 (#27/57)
- DuelLab Overall: 34.39029124682 (#81/191)
GPT-5.1 Codex — Aggregate Rating 1723, #154
- WeaveBench: 1.8 (#6/10)
Claude Opus 4 — Aggregate Rating 1721, #155
- HalluScore: 0.85 (#4/17)
- AI Chess Leaderboard (Continuation): 889.0 (#75/281)
- AI Chess Leaderboard (Reasoning): 704.0 (#166/340)
GLM-5V Turbo — Aggregate Rating 1720, #157
- Chatbot Arena (Text - German): 1413.0 (#117/312)
DeepSeek V4 Pro Preview — Aggregate Rating 1718, #159
- EdgeBench: 31.0 (#2/2)
Qwen 3.6 27B — Aggregate Rating 1715, #163
- SEA-HELM (Malay): 76.25965023295286 (#3/58)
- SEA-HELM (Thai): 73.21879609819743 (#3/58)
- SEA-HELM (Indonesian): 77.96390053028469 (#4/58)
- SEA-HELM (Burmese): 68.37156802775759 (#4/58)
- SEA-HELM (Tamil): 75.3154180958033 (#4/58)
- SEA-HELM (Vietnamese): 76.12380366068943 (#4/58)
- SEA-HELM (Filipino): 75.23034107034779 (#5/58)
- SEA-HELM (English): 64.48197916515696 (#11/58)
- Arcee MMLU-Pro (BenchLM): 86.2 (#12/62)
- LLMConfBench: 0.12 (#19/28)
GPT-5.4 Mini (2026-03-17) — Aggregate Rating 1714, #164
- AuthMem-Bench (Task Success Rate): 56.6 (#1/7)
- Embodied-BenchForge - Static Scene: 47.23 (#5/11)
- Embodied-BenchForge - Success Rate: 16.82 (#6/11)
- AuthMem-Bench (Unauthorized Action Rate): 56.3 (#7/7)
- Embodied-BenchForge - Embodied QA: 46.82 (#8/11)
- FoodTruckBench: 470.21 (#48/57)
GPT-5 Mini (2025-08-07) — Aggregate Rating 1714, #165
- Epoch AI - Critpt: 0.0 (#222/224)
Muse Glimmer 30B — Aggregate Rating 1713, #166
- SEA-HELM (English): 70.73389748624143 (#3/58)
- SEA-HELM (Tamil): 72.02743562606177 (#8/58)
- SEA-HELM (Burmese): 65.94571329470007 (#9/58)
- SEA-HELM (Indonesian): 75.6131549907547 (#10/58)
- SEA-HELM (Malay): 74.13933949886231 (#10/58)
- SEA-HELM (Thai): 69.38230507388069 (#10/58)
- SEA-HELM (Filipino): 74.28471591404333 (#10/58)
- SEA-HELM (Vietnamese): 73.51851357944497 (#10/58)
- LLMConfBench: 0.3 (#11/28)
- FoodTruckBench: 10424.54 (#36/57)
- ...and 3 more
Grok 4.20 — Aggregate Rating 1713, #167
- TERMS-Bench: 10.033958430119476 (#10/15)
- IH-Benchmark: 74.9 (#21/37)
- FoodTruckBench: 1337.76 (#42/57)
Grok 4 — Aggregate Rating 1711, #169
- Poker Arena: 19.0 (#2/7)
- MageBench S1: 1459.0 (#35/35)
Qwen 3.5 122B A10B — Aggregate Rating 1711, #170
- SEA-HELM (Malay): 76.31148733755832 (#1/58)
- SEA-HELM (Vietnamese): 76.39220650636878 (#2/58)
- SEA-HELM (Burmese): 69.16455660325089 (#3/58)
- SEA-HELM (English): 69.77094098437527 (#4/58)
- SEA-HELM (Filipino): 75.3659664396205 (#4/58)
- SEA-HELM (Indonesian): 77.92674102235782 (#5/58)
- SEA-HELM (Tamil): 74.70735721066367 (#7/58)
- SEA-HELM (Thai): 72.74847786800697 (#8/58)
- Arcee MMLU-Pro (BenchLM): 86.7 (#10/62)
Gemma 4 31B (IT) — Aggregate Rating 1711, #171
- SEA-HELM (Indonesian): 79.75223550457908 (#1/58)
- SEA-HELM (Burmese): 70.96834827749052 (#1/58)
- SEA-HELM (Tamil): 76.39848719621575 (#1/58)
- SEA-HELM (Thai): 74.2951168102322 (#1/58)
- SEA-HELM (Filipino): 78.44463592943431 (#1/58)
- SEA-HELM (Vietnamese): 77.08759710440066 (#1/58)
- V-FiLLM - 10-Q Filings: 98.4 (#1/6)
- V-FiLLM - Simplified Statements: 97.6 (#1/6)
- SEA-HELM (Malay): 76.26643145452898 (#2/58)
- V-FiLLM - Simplified Statements (Multi-Turn): 96.4 (#3/6)
- ...and 3 more
O4 Mini — Aggregate Rating 1709, #172
- ResearchCodeBench: 58.490566037735846 (#4/32)
- ContractScrub: 0.453 (#10/12)
- LiveMathematicianBench: 24.593967517401392 (#14/19)
- LLMConfBench: 0.084 (#20/28)
Gemini 3.1 Flash Lite — Aggregate Rating 1707, #174
- DuelLab Overall: 12.05135749479 (#191/191)
GPT-5 Mini — Aggregate Rating 1707, #176
- IH-Benchmark: 88.6 (#9/37)
- MageBench: 1586.0 (#13/22)
- MageBench S1: 1516.0 (#32/35)
- FoodTruckBench: 49.91 (#51/57)
- Epoch AI - Scicode: 39.24 (#163/213)
O1 — Aggregate Rating 1706, #177
- gg-bench: 36.275680268646 (#1/7)
- PlatinumBench (MIT): 1.27 (#9/34)
Qwen 3.5 27B — Aggregate Rating 1704, #183
- SEA-HELM (Thai): 73.52566573735001 (#2/58)
- SEA-HELM (Vietnamese): 76.26567753911479 (#3/58)
- SEA-HELM (English): 66.96635121099732 (#5/58)
- SEA-HELM (Indonesian): 77.59582653355096 (#6/58)
- SEA-HELM (Burmese): 67.55397472849607 (#6/58)
- SEA-HELM (Tamil): 74.71279531250052 (#6/58)
- SEA-HELM (Malay): 74.90266658725403 (#8/58)
- SEA-HELM (Filipino): 74.61172267681118 (#8/58)
- LLMConfBench: 0.19 (#15/28)
- Arcee MMLU-Pro (BenchLM): 86.1 (#15/62)
Grok 4.3 — Aggregate Rating 1704, #184
- Thinkingbox: 14.38 (#14/17)
- FoodTruckBench: 27880.06 (#19/57)
- Opus Magnum Bench: 0.0 (#23/23)
GLM-4.7 — Aggregate Rating 1704, #185
- MageBench S1: 1675.0 (#6/35)
- RepoProbe: 50.7 (#13/20)
- MageBench: 1576.0 (#15/22)
- Arcee MMLU-Pro (BenchLM): 84.3 (#24/62)
Gemini 3.1 Flash Lite (Preview) — Aggregate Rating 1703, #186
- MageBench S2: 1599.0 (#6/11)
- MageBench S1: 1578.0 (#23/35)
- YC-Bench: 202.89933333333335 (#43/53)
Ling-3.0-flash-VL — Aggregate Rating 1703, #189
- Tinybird AI SQL Benchmark - First-Attempt Success Rate: 98.0 (#92/380)
- Tinybird AI SQL Benchmark - Success Rate: 100.0 (#180/380)
- Tinybird AI SQL Benchmark - Exactness: 44.56 (#257/373)
Claude Sonnet 4 — Aggregate Rating 1699, #194
- LLMConfBench: 0.058 (#22/28)
Gemma 4 31B — Aggregate Rating 1699, #195
- LLMConfBench: 0.27 (#14/28)
- IH-Benchmark: 84.8 (#16/37)
- Arcee MMLU-Pro (BenchLM): 85.2 (#19/62)
- AutomationBench: 1.7 (#97/109)
Claude 3.7 Sonnet (20250219) — Aggregate Rating 1698, #196
- VideoGameBench: 0.0 (#3/5)
- PlatinumBench (MIT): 0.93 (#4/34)
- Mizan LLM Leaderboard: 71.27 (#4/45)
- ResearchCodeBench: 51.886792452830186 (#6/32)
DeepSeek V3.2 — Aggregate Rating 1697, #198
- MageBench S2: 1616.0 (#2/11)
- MageBench: 1650.0 (#5/22)
- MageBench S1: 1682.0 (#5/35)
- LLMConfBench: 0.14 (#17/28)
- FoodTruckBench: 2057.52 (#40/57)
- Epoch AI - Apex Agents: 21.3 (#51/52)
O1 Preview (2024-09-12) — Aggregate Rating 1697, #199
- PlatinumBench (MIT): 2.33 (#17/34)
Qwen 3 Max — Aggregate Rating 1695, #201
- PIJ - Criminal Process Reconstruction: 59.51 (#2/10)
- PIJ - Criminal Profiling: 47.2 (#4/10)
- PIJ - Criminal Process Summarization: 44.23 (#5/10)
- PIJ - Sentence Prediction: 39.7 (#7/10)
GPT-5.4 Mini — Aggregate Rating 1695, #202
- MageBench S2: 1617.0 (#1/11)
- Logical Consistency Reformulation - Base Accuracy: 98.9 (#1/7)
Grok 4.1 Fast — Aggregate Rating 1694, #208
- Spreadsheet Arena: 49.230769230769 (#6/14)
- IH-Benchmark: 89.9 (#7/37)
- MageBench S1: 1637.0 (#9/35)
- Chess Bench LLM: 1442.0 (#20/133)
- FoodTruckBench: 816.82 (#46/57)
Gemini 2.5 Flash (Preview 05-20) — Aggregate Rating 1693, #210
- MedRoundsQA - Single-Turn Diagnosis: 65.75 (#4/15)
- MedRoundsQA - Multi-Turn Diagnosis: 26.68 (#8/15)
Claude Sonnet 4 (20250514) — Aggregate Rating 1693, #211
- HELM Safety Mean score: 98.1 (#3/87)
- HELM Safety HarmBench: 98.0 (#6/87)
- HELM Safety SimpleSafetyTests: 100.0 (#7/87)
- HELM Safety BBQ: 96.6 (#17/87)
- HELM Safety XSTest: 96.6 (#32/87)
- HELM Safety Anthropic Red Team: 99.2 (#33/87)
DeepSeek V3.1 Terminus — Aggregate Rating 1692, #212
- ParliamentBench: 66.0 (#4/13)
Qwen 3.6 35B A3B — Aggregate Rating 1691, #214
- SEA-HELM (Tamil): 75.68833321973936 (#2/58)
- Embodied-BenchForge - Static Scene: 49.86 (#4/11)
- Embodied-BenchForge - Success Rate: 46.36 (#4/11)
- SEA-HELM (Vietnamese): 76.11423363451094 (#5/58)
- SEA-HELM (Malay): 75.05124334515837 (#6/58)
- SEA-HELM (Thai): 72.83759309841926 (#7/58)
- Embodied-BenchForge - Embodied QA: 50.91 (#7/11)
- SEA-HELM (Indonesian): 77.29443083660205 (#8/58)
- SEA-HELM (Filipino): 74.40816727905933 (#9/58)
- SEA-HELM (English): 64.62325191125416 (#10/58)
- ...and 4 more
DeepSeek R1 0528 — Aggregate Rating 1690, #217
- TsuGO: 32.3 (#7/9)
Solar Pro 4 — Aggregate Rating 1689, #222
- Arcee MMLU-Pro (BenchLM): 86.3 (#11/62)
- FoodTruckBench: 13984.57 (#33/57)
Qwen 3 VL 235B A22B — Aggregate Rating 1686, #228
- TsuGO: 40.0 (#3/9)
- GameWorld Computer-Use: 31.4 (#6/8)
- GameWorld Generalist: 30.8 (#9/10)
- TRAPSBench: 72.3 (#13/14)
Solar Mini 4 — Aggregate Rating 1685, #229
- LLM Stats (Humanity's Last Exam (no tools, text-only)): 25.8 (#11/12)
- LLM Stats (Terminal-Bench 4.0): 1.0 (#25/25)
- Design Arena (Website): 1168.0 (#110/189)
- LLM Stats Score: 31.9 (#122/383)
- Tinybird AI SQL Benchmark - First-Attempt Success Rate: 78.0 (#322/380)
- Tinybird AI SQL Benchmark - Success Rate: 92.0 (#327/380)
- Tinybird AI SQL Benchmark - Exactness: 30.35 (#336/373)
Gemini 2.5 Flash (Preview 04-17) — Aggregate Rating 1685, #230
- ResearchCodeBench: 45.28301886792453 (#11/32)
O3 Mini (2025-01-31) — Aggregate Rating 1684, #232
- ResearchCodeBench: 52.358490566037744 (#5/32)
- PlatinumBench (MIT): 3.18 (#24/34)
Nemotron 3 Ultra — Aggregate Rating 1684, #234
- Arcee MMLU-Pro (BenchLM): 86.8 (#9/62)
- FoodTruckBench: 3516.9 (#39/57)
- Epoch AI - Apex Agents: 22.7 (#50/52)
- DuelLab Overall: 23.046263924168 (#156/191)
Kimi K2 — Aggregate Rating 1683, #236
- Arcee MMLU-Pro (BenchLM): 81.1 (#35/62)
Gemini 2.5 Flash — Aggregate Rating 1682, #237
- TRAPSBench: 76.1 (#10/14)
- MageBench: 1596.0 (#11/22)
- LLMConfBench: 0.051 (#23/28)
- MageBench S1: 1572.0 (#24/35)
MiMo-V2-Omni — Aggregate Rating 1682, #238
- FoodTruckBench: 597.93 (#47/57)
GPT-4.5 (Preview) — Aggregate Rating 1681, #239
- PlatinumBench (MIT): 0.99 (#5/34)
GPT-4.1 — Aggregate Rating 1681, #240
- ResearchCodeBench: 50.0 (#7/32)
- LiveMathematicianBench: 25.754060324825982 (#12/19)
- DuelLab Overall: 27.74281744852 (#128/191)
Claude 3.7 Sonnet — Aggregate Rating 1681, #241
- gg-bench: 9.532963985071 (#4/7)
- AI Chess Leaderboard (Reasoning): 520.0 (#283/340)
Grok 4 Fast — Aggregate Rating 1681, #242
- MageBench: 1696.0 (#1/22)
- HalluScore: 0.97 (#5/17)
- MageBench S2: 1584.0 (#10/11)
- MageBench S1: 1620.0 (#12/35)
GPT-4.1 (2025-04-14) — Aggregate Rating 1680, #244
- HELM Safety XSTest: 97.9 (#12/87)
- HELM Safety SimpleSafetyTests: 100.0 (#13/87)
- HELM Safety Mean score: 96.3 (#19/87)
- HELM Safety HarmBench: 91.7 (#21/87)
- HELM Safety Anthropic Red Team: 99.3 (#30/87)
- HELM AIR-Bench: 64.8 (#47/87)
- HELM Safety BBQ: 92.6 (#49/87)
DeepSeek R1 — Aggregate Rating 1680, #245
- gg-bench: 32.503551509584 (#2/7)
- Arcee MMLU-Pro (BenchLM): 84.0 (#25/62)
MiniMax-M2.7 — Aggregate Rating 1678, #248
- AutomationBench: 0.46 (#104/109)
Qwen 3.5 35B A3B — Aggregate Rating 1677, #250
- SEA-HELM (Tamil): 74.75599026047023 (#5/58)
- SEA-HELM (Indonesian): 77.55064911934181 (#7/58)
- SEA-HELM (Burmese): 66.82213061799975 (#7/58)
- SEA-HELM (Filipino): 74.76782862558025 (#7/58)
- SEA-HELM (Vietnamese): 75.00333487320026 (#7/58)
- SEA-HELM (Malay): 74.43470967876267 (#9/58)
- SEA-HELM (Thai): 71.48679203163559 (#9/58)
- PredAct-Bench: 49.8 (#10/13)
- SEA-HELM (English): 64.37052791866016 (#12/58)
- Arcee MMLU-Pro (BenchLM): 85.3 (#17/62)
- ...and 1 more
Claude Haiku 4.5 (20251001) — Aggregate Rating 1677, #251
- FoodTruckBench: 166.37 (#49/57)
Qwen 3 235B A22B 2507 — Aggregate Rating 1676, #253
- MageBench: 1625.0 (#7/22)
- MageBench S2: 1584.0 (#8/11)
- MageBench S1: 1594.0 (#18/35)
- Arcee MMLU-Pro (BenchLM): 84.4 (#23/62)
GPT-5.4 Nano (2026-03-17) — Aggregate Rating 1675, #254
- Embodied-BenchForge - Success Rate: 20.45 (#5/11)
- Embodied-BenchForge - Static Scene: 42.96 (#6/11)
- Embodied-BenchForge - Embodied QA: 53.64 (#6/11)
DeepSeek V3.1 — Aggregate Rating 1673, #260
- Arcee MMLU-Pro (BenchLM): 84.8 (#22/62)
Qwen 3 VL 235B A22B Instruct — Aggregate Rating 1671, #264
- EviSafe - Natural Severity Accuracy: 51.9 (#2/11)
- TRAPSBench: 81.0 (#4/14)
- EvoGUI-Bench - Inverse Action and Value: 55.1 (#6/28)
- EvoGUI-Bench - Temporal Ordering: 30.9 (#15/28)
- EvoGUI-Bench: 37.5 (#16/28)
- EvoGUI-Bench - Successor Discrimination: 70.2 (#21/28)
- FoodTruckBench: -525.3 (#56/57)
O3 Mini — Aggregate Rating 1670, #266
- gg-bench: 31.081750143791 (#3/7)
- LiveMathematicianBench: 20.649651972157773 (#17/19)
GPT-4.5 — Aggregate Rating 1669, #267
- HELM Safety SimpleSafetyTests: 100.0 (#14/87)
- HELM Safety HarmBench: 95.8 (#16/87)
- HELM Safety Mean score: 96.5 (#17/87)
- HELM Safety Anthropic Red Team: 99.4 (#27/87)
- HELM AIR-Bench: 74.1 (#31/87)
- HELM Safety BBQ: 92.0 (#54/87)
- HELM Safety XSTest: 95.1 (#55/87)
Claude Haiku 4.5 — Aggregate Rating 1667, #271
- Embodied-BenchForge - Embodied QA: 55.91 (#4/11)
- CoffeeBench: -630.04 (#7/7)
- ContractScrub: 0.492 (#8/12)
- Embodied-BenchForge - Success Rate: 5.0 (#8/11)
- Embodied-BenchForge - Static Scene: 34.73 (#9/11)
- MageBench S1: 1637.0 (#10/35)
- MageBench: 1514.0 (#22/22)
- AutomationBench: 0.46 (#103/109)
- Bug Hunt Bench - VS Code Extension: 0.5 (#105/108)
- Bug Hunt Bench: 1.5 (#106/108)
- ...and 1 more
Qwen 3 Coder 480B A35B — Aggregate Rating 1667, #273
- MageBench S1: 1601.0 (#16/35)
Grok 3 Mini — Aggregate Rating 1666, #274
- ResearchCodeBench: 19.81132075471698 (#27/32)
A.X-K2 — Aggregate Rating 1664, #281
- AA-Briefcase - Analytical Quality Elo: 766.51 (#144/224)
- AA-Briefcase - DOCX - Rubric Pass Rate: 21.3 (#144/224)
- AA-Briefcase - PDF - Rubric Pass Rate: 20.69 (#144/224)
- AA-Briefcase - XLSX - Rubric Pass Rate: 17.41 (#148/224)
- AA-Briefcase: 17.55 (#150/224)
- AA-Briefcase - PPTX - Rubric Pass Rate: 12.65 (#160/224)
- AA-Briefcase - Presentation Elo: 623.58 (#164/224)
Kimi K2 0905 — Aggregate Rating 1663, #282
- Spreadsheet Arena: 16.666666666667 (#11/14)
Gemma 4 26B A4B (IT) — Aggregate Rating 1663, #284
- SEA-HELM (English): 72.36988893488001 (#1/58)
- SEA-HELM (Burmese): 69.36778041265907 (#2/58)
- SEA-HELM (Filipino): 76.0097429772278 (#2/58)
- SEA-HELM (Indonesian): 78.25960322445724 (#3/58)
- SEA-HELM (Tamil): 75.68397632640644 (#3/58)
- SEA-HELM (Malay): 76.0413469052993 (#4/58)
- SEA-HELM (Thai): 72.9449302581879 (#6/58)
- SEA-HELM (Vietnamese): 75.33052878558718 (#6/58)
Claude 3.5 Sonnet (20241022) — Aggregate Rating 1660, #286
- PlatinumBench (MIT): 1.09 (#7/34)
- ResearchCodeBench: 48.58490566037736 (#8/32)
Grok 3 Mini Beta — Aggregate Rating 1659, #290
- SudokuBench (Multi-step, 4x4): 53.33 (#6/14)
- SudokuBench (Single-shot, 4x4): 40.0 (#10/14)
- SudokuBench (Multi-step, 9x9): 0.0 (#12/12)
- SudokuBench (Single-shot, 6x6): 0.0 (#13/14)
- SudokuBench (Multi-step, 6x6): 0.0 (#14/14)
- SudokuBench (Single-shot, 9x9): 0.0 (#14/14)
Qwen 3 235B A22B 2507 Instruct — Aggregate Rating 1658, #291
- IH-Benchmark: 31.6 (#37/37)
Claude 3.5 Sonnet (20240620) — Aggregate Rating 1657, #293
- CanAiCode: 100.0 (#6/189)
MiniMax-M2.5 — Aggregate Rating 1656, #295
- TsuGO: 37.7 (#5/9)
- FoodTruckBench: -316.87 (#55/57)
Grok 3 — Aggregate Rating 1655, #296
- ResearchCodeBench: 42.924528301886795 (#12/32)
DeepSeek V3 — Aggregate Rating 1644, #312
- MedRoundsQA - Single-Turn Diagnosis: 66.1 (#3/15)
- MedRoundsQA - Multi-Turn Diagnosis: 27.14 (#7/15)
- Arcee MMLU-Pro (BenchLM): 75.9 (#44/62)
Gemma 4 12B (IT) — Aggregate Rating 1643, #315
- SEA-HELM (English): 65.37328150875015 (#8/58)
- SEA-HELM (Indonesian): 74.51484886789345 (#11/58)
- SEA-HELM (Burmese): 62.531340014775274 (#11/58)
- SEA-HELM (Tamil): 69.59851557652829 (#11/58)
- SEA-HELM (Thai): 68.35359912529967 (#13/58)
- SEA-HELM (Filipino): 70.35697332500594 (#14/58)
- SEA-HELM (Vietnamese): 69.90308807019576 (#17/58)
- SEA-HELM (Malay): 69.94014339133665 (#19/58)
GPT-4o (2024-05-13) — Aggregate Rating 1643, #316
- CanAiCode: 98.9 (#43/189)
GPT-5.4 Nano — Aggregate Rating 1642, #320
- MageBench S2: 1584.0 (#7/11)
Grok Code Fast 1 — Aggregate Rating 1640, #324
- Multivac Evaluation: 9.12 (#5/19)
- Spreadsheet Arena: 25.773195876289 (#10/14)
GPT-4o (2024-08-06) — Aggregate Rating 1638, #326
- VideoGameBench: 0.09 (#2/5)
- MedRoundsQA - Single-Turn Diagnosis: 67.47 (#2/15)
- MedRoundsQA - Multi-Turn Diagnosis: 31.55 (#4/15)
- Spreadsheet Arena: 11.578947368421 (#12/14)
- ResearchCodeBench: 41.0377358490566 (#15/32)
- PlatinumBench (MIT): 2.24 (#16/34)
Nemotron 3 Super — Aggregate Rating 1638, #327
- FoodTruckBench: 961.84 (#44/57)
GPT-OSS-120B — Aggregate Rating 1638, #330
- CreditCardQA: 70.0 (#4/11)
- SEA-HELM (Filipino): 71.37706037669942 (#11/58)
- SEA-HELM (English): 62.18669819298428 (#15/58)
- SEA-HELM (Thai): 66.88288703507443 (#16/58)
- SEA-HELM (Indonesian): 72.28304061335028 (#17/58)
- SEA-HELM (Tamil): 66.05106890557987 (#18/58)
- SEA-HELM (Vietnamese): 69.81013126052632 (#18/58)
- SEA-HELM (Malay): 69.68243909777829 (#20/58)
- SEA-HELM (Burmese): 46.8981197209996 (#24/58)
- MageBench S1: 1516.0 (#31/35)
- ...and 2 more
Gemini 2.0 Flash — Aggregate Rating 1637, #331
- VideoGameBench: 0.0 (#5/5)
- ResearchCodeBench: 37.264150943396224 (#17/32)
DeepSeek V3 (0324) — Aggregate Rating 1637, #332
- ResearchCodeBench: 42.45283018867924 (#14/32)
GPT-4o — Aggregate Rating 1636, #333
- gg-bench: 8.935303930277 (#5/7)
- IH-Benchmark: 63.1 (#29/37)
Gemma 4 26B A4B — Aggregate Rating 1636, #335
- ContractScrub: 0.326 (#11/12)
- IH-Benchmark: 80.1 (#20/37)
- Arcee MMLU-Pro (BenchLM): 82.6 (#30/62)
Qwen 3 VL 30B A3B — Aggregate Rating 1634, #338
- GameWorld Computer-Use: 30.8 (#8/8)
- GameWorld Generalist: 30.6 (#10/10)
Mistral Medium 3.5 — Aggregate Rating 1633, #341
- SEA-HELM (English): 63.37863561486068 (#13/58)
- SEA-HELM (Tamil): 68.8046691622515 (#14/58)
- SEA-HELM (Filipino): 70.1232823750849 (#16/58)
- SEA-HELM (Vietnamese): 68.82162306511292 (#21/58)
- SEA-HELM (Malay): 68.92707742347204 (#22/58)
- SEA-HELM (Thai): 64.34988573598403 (#22/58)
- SEA-HELM (Indonesian): 70.08851715431774 (#24/58)
- SEA-HELM (Burmese): 41.82917260447859 (#26/58)
GPT-4.1 Mini — Aggregate Rating 1632, #346
- ResearchCodeBench: 42.45283018867924 (#13/32)
- MageBench: 1580.0 (#14/22)
- LiveMathematicianBench: 20.417633410672853 (#18/19)
Seed-OSS-36B-Instruct — Aggregate Rating 1631, #348
- PACT - Steerability: 0.386 (#14/22)
- PACT - Pushback Resistance: 0.946 (#16/22)
- PACT - Default Compliance: 0.934 (#17/22)
- PACT - Rule-Scope Discernment: 0.799 (#18/22)
- PACT: 0.834 (#19/22)
- PACT - Pressure Resistance: 0.822 (#19/22)
- PACT - Transparency: 0.051 (#20/22)
Qwen 3 Next 80B A3B Instruct — Aggregate Rating 1629, #352
- HalluScore: 1.81 (#10/17)
MiMo-V2-Flash — Aggregate Rating 1627, #359
- MageBench: 1615.0 (#9/22)
- MageBench S2: 1584.0 (#11/11)
- MageBench S1: 1586.0 (#21/35)
- Arcee MMLU-Pro (BenchLM): 84.9 (#21/62)
Qwen 3.5 9B — Aggregate Rating 1625, #363
- SEA-HELM (Tamil): 70.89031509418923 (#10/58)
- SEA-HELM (Thai): 68.77001509953925 (#11/58)
- SEA-HELM (Vietnamese): 71.93084933549117 (#11/58)
- SEA-HELM (Indonesian): 74.23327259771202 (#12/58)
- SEA-HELM (Burmese): 60.4066300966198 (#12/58)
- PredAct-Bench: 43.9 (#13/13)
- SEA-HELM (Malay): 71.10721017105067 (#15/58)
- SEA-HELM (English): 59.998796641839164 (#16/58)
- SEA-HELM (Filipino): 66.58671831335069 (#25/58)
- FoodTruckBench: -678.62 (#57/57)
GPT-4o (2024-11-20) — Aggregate Rating 1623, #369
- PlatinumBench (MIT): 2.46 (#18/34)
- LiveMathematicianBench: 19.489559164733176 (#19/19)
NVIDIA-Nemotron-3-Super-120B-A12B-BF16 — Aggregate Rating 1622, #371
- SEA-HELM (English): 65.15057239425114 (#9/58)
- SEA-HELM (Indonesian): 70.77813667589457 (#21/58)
- SEA-HELM (Filipino): 66.91766511689791 (#23/58)
- SEA-HELM (Malay): 68.65915173731034 (#24/58)
- SEA-HELM (Thai): 56.610180304631214 (#43/58)
- SEA-HELM (Vietnamese): 55.31010580047352 (#46/58)
- SEA-HELM (Tamil): 21.828581304791875 (#51/58)
- SEA-HELM (Burmese): 4.98049185324847 (#52/58)
Mistral Large 3 — Aggregate Rating 1614, #388
- MageBench S1: 1501.0 (#33/35)
- NYT Connections Older Models: 4.8 (#101/114)
GPT-4 Turbo — Aggregate Rating 1613, #389
- CanAiCode: 100.0 (#10/189)
Claude 3 Opus (20240229) — Aggregate Rating 1612, #391
- ENAMEL: 40.1 (#5/30)
- CanAiCode: 100.0 (#7/189)
Command A+ — Aggregate Rating 1611, #397
- Arabic Broad Leaderboard - Writing (incl Dialects): 7.545 (#48/120)
- Arabic Broad Leaderboard - RAG QA: 6.927 (#54/120)
- Arabic Broad Leaderboard - Trust & Safety: 8.667 (#55/120)
- Arabic Broad Leaderboard - Reasoning & Math: 8.093 (#57/120)
- Arabic Broad Leaderboard: 8.023 (#58/120)
- Arabic Broad Leaderboard - General Knowledge: 8.143 (#66/120)
- Arabic Broad Leaderboard - MMLU: 8.843 (#68/120)
- Arabic Broad Leaderboard - Translation (incl Dialects): 6.639 (#69/120)
- Tinybird AI SQL Benchmark - Exactness: 46.67 (#220/373)
- Tinybird AI SQL Benchmark - First-Attempt Success Rate: 82.0 (#298/380)
- ...and 1 more
Qwen 3 Coder 30B A3B Instruct — Aggregate Rating 1610, #399
- AA-Omniscience Accuracy: 16.4 (#401/571)
Mistral Medium 3.1 — Aggregate Rating 1610, #400
- MageBench S2: 1616.0 (#4/11)
- MageBench S1: 1569.0 (#25/35)
Qwen 3 VL 8B Instruct — Aggregate Rating 1610, #401
- EviSafe - Natural Severity Accuracy: 50.3 (#3/11)
- TRAPSBench: 58.9 (#14/14)
Mercury 2.5 — Aggregate Rating 1610, #402
- Epoch AI - Scicode: 38.54 (#169/213)
- Epoch AI - Critpt: 0.0 (#223/224)
- Tinybird AI SQL Benchmark - Exactness: 41.89 (#287/373)
- Tinybird AI SQL Benchmark - First-Attempt Success Rate: 76.0 (#326/380)
- Tinybird AI SQL Benchmark - Success Rate: 90.0 (#333/380)
Llama 4 Maverick Instruct FP8 — Aggregate Rating 1609, #403
- VideoGameBench: 0.0 (#4/5)
- HalluScore: 1.33 (#8/17)
- PlatinumBench (MIT): 2.12 (#15/34)
Sarvam 105B — Aggregate Rating 1609, #404
- Arcee MMLU-Pro (BenchLM): 81.7 (#33/62)
GPT-4 Preview (1106) — Aggregate Rating 1607, #409
- ENAMEL: 47.0 (#1/30)
- CanAiCode: 100.0 (#5/189)
QwQ-32B — Aggregate Rating 1607, #410
- CreditCardQA: 63.6 (#8/11)
GPT-5 Nano — Aggregate Rating 1606, #415
- MageBench: 1571.0 (#16/22)
- IH-Benchmark: 64.6 (#28/37)
- MageBench S1: 1499.0 (#34/35)
- Chatbot Arena (Text - Japanese): 1244.0 (#199/278)
Mercury 2 — Aggregate Rating 1604, #422
- Enkrypt AI - Insecure Code Risk: 1.33 (#12/300)
- Enkrypt AI - Bias Risk: 69.51 (#64/300)
- Enkrypt AI - Risk Score: 24.33 (#69/300)
- Enkrypt AI - Toxicity Risk: 2.36 (#109/298)
- Enkrypt AI - Safety Risk: 32.72 (#209/298)
- Enkrypt AI - Jailbreak Risk: 23.45 (#253/298)
GPT-4 — Aggregate Rating 1603, #423
- Open Medical LLM - MMLU College Biology: 95.1 (#3/193)
- Open Medical LLM - MedQA (USMLE): 78.8 (#4/193)
- Open Medical LLM - MMLU Anatomy: 80.0 (#4/193)
- Open Medical LLM - MMLU Medical Genetics: 91.0 (#4/193)
- Open Medical LLM - MMLU Professional Medicine: 93.8 (#4/193)
- Open Medical LLM: 82.96666666666667 (#5/193)
- Open Medical LLM - MedMCQA: 69.5 (#5/193)
- Open Medical LLM - MMLU Clinical Knowledge: 86.4 (#5/193)
- Open Medical LLM - MMLU College Medicine: 76.9 (#5/193)
- Open Medical LLM - PubMedQA: 75.2 (#59/193)
Qwen 3 32B — Aggregate Rating 1603, #424
- SEA-HELM (Malay): 71.78997131426175 (#12/58)
- SEA-HELM (Tamil): 69.2200988188048 (#13/58)
- SEA-HELM (Indonesian): 73.3861533104353 (#14/58)
- SEA-HELM (Vietnamese): 70.7264661287249 (#14/58)
- SEA-HELM (Burmese): 53.89817568118165 (#15/58)
- SEA-HELM (Thai): 66.94868763001324 (#15/58)
- SEA-HELM (Filipino): 68.18010537294035 (#19/58)
- Mizan LLM Leaderboard: 62.24 (#19/45)
- SEA-HELM (English): 55.34101507090405 (#23/58)
- LLMConfBench: 0.001 (#27/28)
Qwen 2.5 72B Instruct — Aggregate Rating 1599, #436
- HELM Safety XSTest: 97.9 (#13/87)
- HELM Safety Anthropic Red Team: 99.6 (#18/87)
- HELM Safety SimpleSafetyTests: 100.0 (#21/87)
- HELM Safety BBQ: 95.4 (#29/87)
- HELM Safety Mean score: 93.1 (#34/87)
- HELM Safety HarmBench: 72.8 (#44/87)
- HELM AIR-Bench: 59.0 (#57/87)
Gemini 2.0 Flash Lite — Aggregate Rating 1599, #438
- ResearchCodeBench: 30.660377358490564 (#21/32)
Llama 4 Maverick Instruct — Aggregate Rating 1598, #439
- BenCzechMark: 72.9177710045057 (#8/78)
- Spreadsheet Arena: 3.174603174603 (#14/14)
Llama 3.1 405B Instruct — Aggregate Rating 1597, #447
- HELM Safety BBQ: 94.5 (#38/87)
- HELM Safety SimpleSafetyTests: 98.8 (#43/87)
- HELM Safety XSTest: 95.9 (#43/87)
- HELM Safety Mean score: 89.7 (#54/87)
- HELM AIR-Bench: 58.6 (#58/87)
- HELM Safety HarmBench: 62.7 (#63/87)
- HELM Safety Anthropic Red Team: 96.5 (#72/87)
Llama 4 Maverick — Aggregate Rating 1595, #451
- MageBench S1: 1590.0 (#19/35)
- MageBench: 1526.0 (#21/22)
- ResearchCodeBench: 27.358490566037734 (#22/32)
GPT-4 Preview (0125) — Aggregate Rating 1591, #463
- CanAiCode: 100.0 (#9/189)
Granite 4.2 30B — Aggregate Rating 1589, #469
- Arcee MMLU-Pro (BenchLM): 77.6 (#41/62)
- AA-Briefcase - DOCX - Rubric Pass Rate: 11.26 (#176/224)
- AA-Briefcase - XLSX - Rubric Pass Rate: 8.74 (#176/224)
- AA-Briefcase: 9.79 (#177/224)
- AA-Briefcase - PPTX - Rubric Pass Rate: 7.43 (#178/224)
- AA-Briefcase - Analytical Quality Elo: 271.68 (#179/224)
- AA-Briefcase - PDF - Rubric Pass Rate: 8.08 (#183/224)
- AA-Briefcase - Presentation Elo: 225.66 (#191/224)
Qwen 3.5 4B — Aggregate Rating 1589, #470
- Embodied-BenchForge - Embodied QA: 55.91 (#5/11)
- Embodied-BenchForge - Static Scene: 35.97 (#8/11)
- Embodied-BenchForge - Success Rate: 0.0 (#9/11)
- SEA-HELM (Burmese): 52.40528866374238 (#17/58)
- SEA-HELM (English): 57.420799191406246 (#19/58)
- SEA-HELM (Thai): 65.58070937814212 (#19/58)
- SEA-HELM (Vietnamese): 68.97163788845825 (#20/58)
- SEA-HELM (Tamil): 63.497553597175475 (#22/58)
- SEA-HELM (Indonesian): 69.65825670422308 (#25/58)
- SEA-HELM (Malay): 68.31969903250607 (#27/58)
- ...and 1 more
Mistral Large 2 (Jul) — Aggregate Rating 1588, #474
- HELM AIR-Bench: 35.3 (#82/87)
GPT-4 (0613) — Aggregate Rating 1588, #475
- ENAMEL: 45.4 (#2/30)
- CanAiCode: 100.0 (#4/189)
Grok 2 (1212) — Aggregate Rating 1586, #480
- PlatinumBench (MIT): 3.21 (#25/34)
- ResearchCodeBench: 22.169811320754718 (#26/32)
Mistral Medium 3 — Aggregate Rating 1585, #483
- ResearchCodeBench: 35.37735849056604 (#18/32)
Nova Pro (v1) — Aggregate Rating 1585, #485
- ResearchCodeBench: 25.0 (#24/32)
MiniCPM5-2B — Aggregate Rating 1585, #487
- Arcee MMLU-Pro (BenchLM): 70.8 (#48/62)
Qwen 3 30B A3B — Aggregate Rating 1585, #488
- Spreadsheet Arena: 3.27868852459 (#13/14)
- Mizan LLM Leaderboard: 59.39 (#24/45)
GLM-4.7 Flash — Aggregate Rating 1583, #496
- SEA-HELM (English): 57.35955713947848 (#20/58)
- SEA-HELM (Indonesian): 67.88596329049608 (#30/58)
- SEA-HELM (Filipino): 62.969395950629796 (#30/58)
- SEA-HELM (Vietnamese): 64.36330105579914 (#32/58)
- SEA-HELM (Malay): 65.4984627541354 (#34/58)
- SEA-HELM (Thai): 59.29238981742403 (#36/58)
- SEA-HELM (Burmese): 17.896831979512097 (#43/58)
- SEA-HELM (Tamil): 38.42031052524894 (#43/58)
Command A — Aggregate Rating 1583, #498
- ResearchCodeBench: 31.132075471698112 (#20/32)
GPT-OSS-20B — Aggregate Rating 1582, #499
- CreditCardQA: 66.8 (#5/11)
- SEA-HELM (English): 57.10074386772906 (#21/58)
- SEA-HELM (Tamil): 63.233754530551835 (#23/58)
- SEA-HELM (Thai): 61.83051111525851 (#27/58)
- SEA-HELM (Filipino): 65.03408994507146 (#28/58)
- SEA-HELM (Indonesian): 68.38470093159587 (#29/58)
- SEA-HELM (Burmese): 36.14713976893347 (#30/58)
- SEA-HELM (Malay): 66.10609021786549 (#31/58)
- SEA-HELM (Vietnamese): 63.74957477580932 (#36/58)
Qwen 3 VL 4B Instruct — Aggregate Rating 1580, #506
- EviSafe - Natural Severity Accuracy: 52.8 (#1/11)
Qwen 2 72B — Aggregate Rating 1579, #508
- RABBITS: 99.05 (#5/23)
Gemma 3 27B (IT) — Aggregate Rating 1578, #509
- ParliamentBench: 39.0 (#12/13)
- LLM-WikiRace: 36.67 (#16/30)
Nemotron 3 Nano Omni 30B A3B — Aggregate Rating 1577, #510
- Arcee MMLU-Pro (BenchLM): 77.3 (#42/62)
Qwen 2.5 Coder 32B Instruct — Aggregate Rating 1575, #516
- ResearchCodeBench: 25.943396226415093 (#23/32)
- SecRepoBench (Standalone) - Pass@1: 25.2 (#23/29)
- SecRepoBench (Standalone) - Secure-Pass@1: 13.5 (#25/29)
Llama 3.3 70B Instruct — Aggregate Rating 1574, #520
- V-FiLLM - 10-Q Filings: 97.2 (#4/6)
- ParliamentBench: 58.0 (#5/13)
- V-FiLLM - Simplified Statements: 80.8 (#5/6)
- V-FiLLM - Simplified Statements (Multi-Turn): 86.8 (#5/6)
- WorldBench (Cultural Agents): 24.1 (#6/9)
- gg-bench: 7.424813158764 (#7/7)
- PACT - Pressure Resistance: 0.928 (#14/22)
- PACT - Transparency: 0.093 (#15/22)
- PACT - Rule-Scope Discernment: 0.82 (#15/22)
- PACT - Default Compliance: 0.941 (#16/22)
- ...and 11 more
Qwen 3 14B — Aggregate Rating 1573, #524
- SEA-HELM (Tamil): 65.07595815347933 (#19/58)
- SEA-HELM (Vietnamese): 69.07226871089203 (#19/58)
- SEA-HELM (Indonesian): 71.2672297035327 (#20/58)
- SEA-HELM (Burmese): 47.7446320768115 (#21/58)
- SEA-HELM (Thai): 65.46644788440315 (#21/58)
- SEA-HELM (English): 54.499871952000625 (#24/58)
- SEA-HELM (Malay): 68.56705514500611 (#25/58)
- Mizan LLM Leaderboard: 59.12 (#25/45)
- SEA-HELM (Filipino): 66.04380796863231 (#26/58)
Llama 4 Scout Instruct — Aggregate Rating 1572, #526
- BenCzechMark: 68.87479017581059 (#17/78)
Gemma 4 12B — Aggregate Rating 1572, #529
- Arcee MMLU-Pro (BenchLM): 77.2 (#43/62)
Llama 3.1 70B Instruct — Aggregate Rating 1566, #548
- ParliamentBench: 42.0 (#11/13)
- HELM Safety BBQ: 95.4 (#28/87)
- LOL Arena: 42.5 (#36/36)
- HELM Safety XSTest: 94.5 (#57/87)
- HELM Safety Mean score: 84.5 (#76/87)
- HELM Safety SimpleSafetyTests: 92.5 (#76/87)
- HELM AIR-Bench: 42.5 (#79/87)
- HELM Safety Anthropic Red Team: 93.2 (#80/87)
- HELM Safety HarmBench: 46.9 (#80/87)
gemma-4-E4B-it — Aggregate Rating 1566, #549
- SEA-HELM (Burmese): 58.40039121428377 (#14/58)
- SEA-HELM (Tamil): 67.06350331319739 (#15/58)
- SEA-HELM (Thai): 66.33437695669534 (#18/58)
- SEA-HELM (Filipino): 67.0972930780243 (#22/58)
- SEA-HELM (English): 53.79759546358751 (#25/58)
- SEA-HELM (Vietnamese): 67.14093620000683 (#25/58)
- SEA-HELM (Indonesian): 69.64622223106228 (#26/58)
- SEA-HELM (Malay): 68.44147787325001 (#26/58)
Claude 3.5 Haiku — Aggregate Rating 1566, #553
- ResearchCodeBench: 37.735849056603776 (#16/32)
- AI Chess Leaderboard (Reasoning): 576.0 (#243/340)
- AI Chess Leaderboard (Continuation): 395.0 (#250/281)
Celeris-1 — Aggregate Rating 1564, #563
- Arcee MMLU-Pro (BenchLM): 75.9 (#45/62)
Llama 3.3 70B — Aggregate Rating 1562, #569
- CreditCardQA: 62.0 (#10/11)
Seed-X-PPO-7B — Aggregate Rating 1562, #570
- Last Translation Benchmark: 1.756311745334797 (#31/45)
Sailor2-20B-Chat — Aggregate Rating 1559, #580
- Wolfram LLM Benchmarking Project: 28.8 (#420/589)
Mistral Small 3.2 — Aggregate Rating 1556, #587
- PredAct-Bench: 57.0 (#5/13)
GPT-4o Mini — Aggregate Rating 1556, #588
- MageBench: 1631.0 (#6/22)
- gg-bench: 7.637179711886 (#6/7)
- ResearchCodeBench: 23.11320754716981 (#25/32)
- MageBench S1: 1546.0 (#28/35)
Qwen2.5-Omni-7B — Aggregate Rating 1552, #601
- AGL1K - Speech Distance Error: 4540.06 (#10/16)
- AGL1K: 1317.74 (#13/16)
- AGL1K - Non-Speech Distance Error: 7777.2 (#14/16)
Mistral Small 3 — Aggregate Rating 1546, #616
- ParliamentBench: 43.0 (#10/13)
Llama 3 70B — Aggregate Rating 1545, #621
- RABBITS: 98.57 (#7/23)
OLMo 3.1 32B — Aggregate Rating 1544, #622
- Multivac Evaluation: 7.23 (#17/19)
EXAONE 4.0 32B — Aggregate Rating 1544, #625
- Arcee MMLU-Pro (BenchLM): 81.8 (#32/62)
Granite 4.2 8B — Aggregate Rating 1542, #631
- Arcee MMLU-Pro (BenchLM): 74.0 (#47/62)
- Arabic Broad Leaderboard - Reasoning & Math: 6.977 (#75/120)
- Arabic Broad Leaderboard - Trust & Safety: 7.333 (#79/120)
- Arabic Broad Leaderboard - General Knowledge: 7.397 (#84/120)
- Arabic Broad Leaderboard: 6.879 (#87/120)
- Arabic Broad Leaderboard - Writing (incl Dialects): 6.227 (#89/120)
- Arabic Broad Leaderboard - Translation (incl Dialects): 5.472 (#90/120)
- Arabic Broad Leaderboard - MMLU: 7.521 (#94/120)
- Arabic Broad Leaderboard - RAG QA: 5.488 (#95/120)
- Tinybird AI SQL Benchmark - First-Attempt Success Rate: 90.0 (#241/380)
- ...and 2 more
Ministral 3 14B — Aggregate Rating 1540, #636
- PredAct-Bench: 49.2 (#11/13)
Qwen 2.5 VL 7B Instruct — Aggregate Rating 1538, #643
- EviSafe - Natural Severity Accuracy: 36.2 (#7/11)
Ministral-3-8B-Instruct-2512 — Aggregate Rating 1538, #644
- MedRoundsQA - Multi-Turn Diagnosis: 19.92 (#12/15)
- MedRoundsQA - Single-Turn Diagnosis: 36.3 (#14/15)
Llama 4 Scout — Aggregate Rating 1537, #646
- ResearchCodeBench: 18.39622641509434 (#28/32)
Gemma 3 12B (IT) — Aggregate Rating 1535, #655
- LLM-WikiRace: 32.07 (#18/30)
NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 — Aggregate Rating 1534, #660
- SEA-HELM (English): 56.90273315369458 (#22/58)
- SEA-HELM (Thai): 62.89854666720549 (#24/58)
- SEA-HELM (Indonesian): 65.35509494528031 (#36/58)
- SEA-HELM (Vietnamese): 60.718984183283595 (#41/58)
- SEA-HELM (Filipino): 55.529080027207605 (#44/58)
- SEA-HELM (Malay): 57.99902290106085 (#47/58)
- SEA-HELM (Tamil): 22.499281182009927 (#50/58)
- SEA-HELM (Burmese): 3.2342529158570685 (#58/58)
GPT-3.5 Turbo Instruct — Aggregate Rating 1534, #663
- Chess Bench LLM: 1586.0 (#15/133)
Gemma 3 12B — Aggregate Rating 1533, #666
- OpenRouter Tau2-Bench Airline: 48.0 (#116/145)
- OpenRouter GPQA Diamond: 37.9 (#153/159)
Qwen 3 8B — Aggregate Rating 1533, #667
- SEA-HELM (Thai): 63.37090792138304 (#23/58)
- SEA-HELM (Vietnamese): 68.16586365904755 (#23/58)
- SEA-HELM (English): 52.07237292032307 (#27/58)
- SEA-HELM (Indonesian): 69.35592376166638 (#27/58)
- SEA-HELM (Burmese): 40.712613953368695 (#27/58)
- SEA-HELM (Malay): 67.48621857061528 (#28/58)
- SEA-HELM (Filipino): 60.7727157813888 (#33/58)
- Mizan LLM Leaderboard: 52.52 (#33/45)
- SEA-HELM (Tamil): 52.4509690235333 (#36/58)
Claude 3 Sonnet (20240229) — Aggregate Rating 1532, #669
- CanAiCode: 100.0 (#8/189)
- ENAMEL: 34.5 (#9/30)
ALLaM-7B-Instruct-preview — Aggregate Rating 1523, #698
- HalluScore: 3.87 (#14/17)
Sarvam 30B — Aggregate Rating 1522, #705
- Arcee MMLU-Pro (BenchLM): 80.0 (#36/62)
Mixtral-8x22B-v0.1 — Aggregate Rating 1521, #706
- RABBITS: 98.66 (#6/23)
GPT-4.1 Nano — Aggregate Rating 1518, #716
- ResearchCodeBench: 15.09433962264151 (#29/32)
c4ai-command-r-plus — Aggregate Rating 1517, #720
- RABBITS: 98.28 (#8/23)
Gemma 4 E4B — Aggregate Rating 1512, #740
- Arcee MMLU-Pro (BenchLM): 69.4 (#49/62)
Olmo 3.1 32B Instruct — Aggregate Rating 1507, #754
- ParliamentBench: 53.0 (#7/13)
Claude 3 Haiku (20240307) — Aggregate Rating 1504, #759
- ENAMEL: 38.6 (#7/30)
- CanAiCode: 98.9 (#41/189)
Fanar-1-9B — Aggregate Rating 1502, #765
- HalluScore: 10.04 (#16/17)
Gemma 4 E2B (IT) — Aggregate Rating 1501, #768
- SEA-HELM (Burmese): 46.94846306876102 (#23/58)
- SEA-HELM (Tamil): 59.02905936602496 (#29/58)
- SEA-HELM (English): 45.26135026115748 (#30/58)
- SEA-HELM (Filipino): 60.675377569931804 (#35/58)
- SEA-HELM (Indonesian): 64.13173228357995 (#39/58)
- SEA-HELM (Vietnamese): 61.18584409738889 (#39/58)
- SEA-HELM (Thai): 58.32289944253636 (#40/58)
- SEA-HELM (Malay): 61.45962567016155 (#42/58)
Sailor2-8B-Chat — Aggregate Rating 1501, #769
- Wolfram LLM Benchmarking Project: 16.7 (#518/589)
MedGemma 1.5 4B — Aggregate Rating 1500, #775
- RadMatch - Radiologist Agreement: 0.47 (#11/11)
Claude 3 Haiku — Aggregate Rating 1498, #784
- AI Chess Leaderboard (Reasoning): 578.0 (#241/340)
- AI Chess Leaderboard (Continuation): 384.0 (#255/281)
Granite 4.2 3B — Aggregate Rating 1498, #785
- Arcee MMLU-Pro (BenchLM): 67.8 (#52/62)
Yi 1.5 34B — Aggregate Rating 1493, #794
- RABBITS: 97.81 (#9/23)
Qwen 2 VL 7B Instruct — Aggregate Rating 1486, #816
- BALROG MiniHack (VLM): 5.0 (#5/10)
- BALROG NetHack (VLM): 0.0 (#6/10)
- BALROG BabaIsAI (VLM): 11.864406779661017 (#7/10)
- BALROG BabyAI (VLM): 2.0 (#10/10)
- BALROG Crafter (VLM): 5.454545454545454 (#10/10)
- BALROG NetHack (LLM): 0.0 (#33/39)
- BALROG MiniHack (LLM): 2.5 (#35/39)
- BALROG TextWorld (LLM): 1.5686274509803921 (#37/39)
- BALROG BabyAI (LLM): 4.0 (#39/39)
- BALROG Crafter (LLM): 6.363636363636363 (#39/39)
- ...and 1 more
Qwen 3 4B — Aggregate Rating 1485, #823
- Mizan LLM Leaderboard: 47.91 (#39/45)
MedGemma-4B-IT — Aggregate Rating 1480, #842
- Longitudinal MRI Progression Reasoning: 23.5 (#12/16)
Qwen 2.5 7B Instruct — Aggregate Rating 1478, #846
- BALROG NetHack (LLM): 0.0 (#32/39)
- BALROG TextWorld (LLM): 3.9215686274509802 (#33/39)
- BALROG Crafter (LLM): 16.363636363636363 (#34/39)
- BALROG BabaIsAI (LLM): 12.5 (#35/39)
- BALROG BabyAI (LLM): 14.000000000000002 (#37/39)
- BALROG MiniHack (LLM): 0.0 (#39/39)
aya-23-35B — Aggregate Rating 1472, #866
- RABBITS: 95.81 (#13/23)
Granite 4.0 Micro — Aggregate Rating 1469, #882
- VLoc Bench: 0.0 (#26/27)
Phi-3-medium-4k-instruct — Aggregate Rating 1459, #939
- RABBITS: 87.12 (#19/23)
GPT-3.5 Turbo (1106) — Aggregate Rating 1457, #948
- Nexus Function Calling - VirusTotal: 81.0 (#2/4)
- Nexus Function Calling - Overall: 37.45333333333333 (#3/3)
- Nexus Function Calling - OTX: 89.13 (#3/4)
- Nexus Function Calling - CVECPE: 48.0 (#3/4)
- Nexus Function Calling - CVECPE Multi APIs: 7.140000000000001 (#3/4)
- Nexus Function Calling - VT Multi Dependency: 2.04 (#3/4)
- Nexus Function Calling - VT Multi Disconnected: 14.29 (#3/4)
- Nexus Function Calling - Climate: 25.53 (#3/4)
- Nexus Function Calling - Places API: 25.0 (#3/4)
- CanAiCode: 98.9 (#40/189)
- ...and 10 more
GPT-3.5 Turbo (0125) — Aggregate Rating 1456, #958
- CanAiCode: 98.9 (#42/189)
Gemma 3 4B (IT) — Aggregate Rating 1453, #971
- LLM-WikiRace: 16.9 (#25/30)
Phi-3-medium-128k-instruct — Aggregate Rating 1449, #997
- RULER: 81.5 (#32/45)
Llama 2 70B Base — Aggregate Rating 1445, #1013
- RABBITS: 93.61 (#14/23)
Llama 3.1 8B Instruct — Aggregate Rating 1444, #1018
- WorldBench (Cultural Agents): 12.5 (#7/9)
- PACT - Steerability: 0.399 (#12/22)
- PACT: 0.562 (#21/22)
- PACT - Default Compliance: 0.766 (#21/22)
- PACT - Pressure Resistance: 0.599 (#21/22)
- PACT - Transparency: 0.051 (#21/22)
- PACT - Pushback Resistance: 0.448 (#22/22)
- PACT - Rule-Scope Discernment: 0.51 (#22/22)
- BALROG Crafter (LLM): 25.454545454545453 (#28/39)
- BALROG BabaIsAI (LLM): 18.333333333333332 (#29/39)
- ...and 12 more
EuroLLM-9B-Instruct — Aggregate Rating 1439, #1043
- WorldBench (Cultural Agents): 10.8 (#9/9)
Apertus-8B-Instruct-2509 — Aggregate Rating 1439, #1048
- BenCzechMark: 40.98087286862797 (#40/78)
Gemma 4 E2B — Aggregate Rating 1435, #1065
- Arcee MMLU-Pro (BenchLM): 60.0 (#53/62)
Mixtral 8x7B (v0.1) — Aggregate Rating 1435, #1071
- RABBITS: 97.14 (#11/23)
Qwen 2 7B — Aggregate Rating 1432, #1081
- RABBITS: 93.13 (#16/23)
GPT-3.5 Turbo (0613) — Aggregate Rating 1429, #1097
- CanAiCode: 96.7 (#96/189)
Qwen 3.5 2B — Aggregate Rating 1421, #1134
- Embodied-BenchForge - Embodied QA: 46.82 (#9/11)
- Embodied-BenchForge - Static Scene: 32.07 (#10/11)
- Embodied-BenchForge - Success Rate: 0.0 (#10/11)
Granite 4.0 H 1B — Aggregate Rating 1409, #1181
- Arcee MMLU-Pro (BenchLM): 32.9 (#57/62)
Llama 3 8B UltraMedical — Aggregate Rating 1408, #1188
- MedRoundsQA - Single-Turn Diagnosis: 50.0 (#11/15)
- MedRoundsQA - Multi-Turn Diagnosis: 22.21 (#11/15)
Llama 3 8B — Aggregate Rating 1395, #1231
- RABBITS: 96.18 (#12/23)
Phi-3 Mini 128K Instruct — Aggregate Rating 1380, #1281
- RULER: 84.8 (#28/45)
Tiny Aya Global — Aggregate Rating 1372, #1314
- LKValues: 0.288 (#18/19)
Granite 4.0 1B — Aggregate Rating 1371, #1317
- VLoc Bench: 0.0 (#27/27)
- Arcee MMLU-Pro (BenchLM): 34.0 (#56/62)
Mistral-7B-v0.3 — Aggregate Rating 1364, #1340
- RABBITS: 90.65 (#18/23)
Mistral 7B Instruct — Aggregate Rating 1361, #1345
- PACT - Pushback Resistance: 0.583 (#21/22)
- PACT - Steerability: 0.224 (#21/22)
- PACT - Rule-Scope Discernment: 0.568 (#21/22)
- PACT: 0.484 (#22/22)
- PACT - Default Compliance: 0.745 (#22/22)
- PACT - Pressure Resistance: 0.469 (#22/22)
- PACT - Transparency: 0.033 (#22/22)
Gemma 1.1 7B (IT) — Aggregate Rating 1358, #1352
- CyberMetric: 75.4 (#10/25)
NexusRaven-V2-13B — Aggregate Rating 1354, #1362
- Nexus Function Calling - Overall: 58.196666666666665 (#1/3)
- Nexus Function Calling - OTX: 90.22 (#1/4)
- Nexus Function Calling - CVECPE Multi APIs: 25.0 (#1/4)
- Nexus Function Calling - VT Multi Dependency: 38.78 (#1/4)
- Nexus Function Calling - VT Multi Disconnected: 42.86 (#1/4)
- Nexus Function Calling - Climate: 70.21 (#1/4)
- Nexus Function Calling - Places API: 50.0 (#1/4)
- Nexus Function Calling - CVECPE: 66.67 (#2/4)
- Nexus Function Calling - VirusTotal: 80.13 (#3/4)
Granite 4.0 350M — Aggregate Rating 1346, #1374
- VLoc Bench: 0.001 (#25/27)
- Arcee MMLU-Pro (BenchLM): 12.1 (#62/62)
Yarn-Mistral-7B-128k — Aggregate Rating 1343, #1382
- InfiniteBench: 19.96 (#4/4)
granite-3.0-3B-a800m-instruct — Aggregate Rating 1334, #1399
- BenCzechMark: 15.970933828076683 (#66/78)
Qwen 3.5 0.8B — Aggregate Rating 1333, #1404
- Embodied-BenchForge - Static Scene: 30.34 (#11/11)
- Embodied-BenchForge - Success Rate: 0.0 (#11/11)
- Embodied-BenchForge - Embodied QA: 22.73 (#11/11)
MiniCPM5-1B — Aggregate Rating 1288, #1482
- Arcee MMLU-Pro (BenchLM): 48.9 (#55/62)
Granite 4.0 H 350M — Aggregate Rating 1286, #1486
- Arcee MMLU-Pro (BenchLM): 14.4 (#61/62)
Phi-2 — Aggregate Rating 1280, #1492
- RABBITS: 71.66 (#20/23)
Llama 2 7B Base — Aggregate Rating 1272, #1500
- RABBITS: 68.61 (#21/23)
Sailor2-1B-Chat — Aggregate Rating 1270, #1501
- Wolfram LLM Benchmarking Project: 3.3 (#582/589)
Phi-1.5 — Aggregate Rating 1243, #1525
- RABBITS: 37.88 (#22/23)
Phi-1 — Aggregate Rating 1126, #1619
- RABBITS: 25.76 (#23/23)

Top-10 New Scores (279)

Claude Fable 5.1 on AA-Omniscience Accuracy: 67.23 (#1)
Claude Fable 5.1 on Agents on Rails: 92.06349206349206 (#2)
Claude Fable 5.1 on AutomationBench: 22.37 (#24)
Claude Fable 5.1 on Benchmarks.bio - MetagenomicsBench: 61.7 (#2)
Claude Fable 5.1 on Chatbot Arena (Vision - Humor): 1281.0 (#16)
Claude Fable 5.1 on Clerk LLM Leaderboard: 91.7 (#2)
Claude Fable 5.1 on CursorBench 4.0: 51.8 (#7)
Claude Fable 5.1 on FoodTruckBench: 62732.17 (#2)
Claude Fable 5.1 on React Native Evals: 89.05 (#1)
Claude Fable 5.1 on RuneBench: 6.143277369387 (#11)
Claude Fable 5.1 on Terminal-Bench 4.0 (Claude Code): 57.88 (#3)
Claude Opus 5.5 on AA-Omniscience Accuracy: 66.22 (#2)
Claude Opus 5.5 on AI Chess Leaderboard (Reasoning): 1783.0 (#9)
Claude Opus 5.5 on Chatbot Arena (Image-to-WebDev): 1749.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text - French): 1511.0 (#11)
Claude Opus 5.5 on Chatbot Arena (Text - Spanish): 1491.0 (#8)
Claude Opus 5.5 on Chatbot Arena (Vision - Creative Writing): 1301.0 (#5)
Claude Opus 5.5 on Chatbot Arena (Vision - Diagram): 1324.0 (#9)
Claude Opus 5.5 on Chatbot Arena (Vision - English): 1286.0 (#10)
Claude Opus 5.5 on Chatbot Arena (Vision - Homework): 1348.0 (#3)
Claude Opus 5.5 on Chatbot Arena (Vision - OCR): 1307.0 (#9)
Claude Opus 5.5 on Chatbot Arena (Vision): 1293.0 (#11)
Claude Opus 5.5 on Clerk LLM Leaderboard: 91.1 (#5)
Claude Opus 5.5 on CursorBench 4.0: 57.8 (#1)
Claude Opus 5.5 on Epoch AI - Apex Agents: 52.5 (#24)
Claude Opus 5.5 on Epoch AI - Critpt: 30.86 (#12)
Claude Opus 5.5 on Epoch AI - Cursorbench: 56.0 (#3)
Claude Opus 5.5 on Epoch AI - Frontiermath Erdos: 2.94 (#1)
Claude Opus 5.5 on Epoch AI - Scicode: 60.42 (#9)
Claude Opus 5.5 on F5 Labs AI Security Leaderboard - CASI: 93.06 (#8)
Claude Opus 5.5 on HealthBench-Psych - Hard: 42.281 (#2)
Claude Opus 5.5 on Kaggle Cipher Frontlines: 72.73 (#21)
Claude Opus 5.5 on Kaggle Game Arena (Unified): 400.79 (#7)
Claude Opus 5.5 on Kaggle Game Arena Chess: 948.16 (#12)
Claude Opus 5.5 on NYT Connections Extended: 87.8 (#32)
Claude Opus 5.5 on PUMA Polish Multimodal - Contemporary Life: 58.0 (#16)
Claude Opus 5.5 on PUMA Polish Multimodal - Geography and Environment: 82.0 (#8)
Claude Opus 5.5 on PUMA Polish Multimodal - History and Culture: 71.0 (#11)
Claude Opus 5.5 on PUMA Polish Multimodal - OCR: 90.0 (#3)
Claude Opus 5.5 on PUMA Polish Multimodal - Structured Extraction: 62.0 (#12)
Claude Opus 5.5 on PUMA Polish Multimodal - Vision Mean: 75.33 (#9)
Claude Opus 5.5 on React Native Evals: 88.57 (#2)
Claude Opus 5.5 on RuneBench: 6.66164334639 (#6)
Claude Opus 5.5 on Terminal-Bench 4.0 (Claude Code): 64.85 (#1)
Claude Opus 5.5 on Vals AI ReverseEngBench: 33.59 (#4)
Claude Opus 5.5 on WebDev Arena (HTML): 1761.0 (#3)
Claude Opus 5.5 on YC-Bench: 2873.72509 (#1)
Claude Sonnet 5.5 on AA-Omniscience Accuracy: 53.95 (#42)
Claude Sonnet 5.5 on Arabic Broad Leaderboard: 8.911 (#23)
Claude Sonnet 5.5 on Arabic Broad Leaderboard - General Knowledge: 8.857 (#39)
Claude Sonnet 5.5 on Arabic Broad Leaderboard - MMLU: 9.587 (#22)
Claude Sonnet 5.5 on Arabic Broad Leaderboard - RAG QA: 7.268 (#44)
Claude Sonnet 5.5 on Arabic Broad Leaderboard - Reasoning & Math: 9.302 (#12)
Claude Sonnet 5.5 on Arabic Broad Leaderboard - Translation (incl Dialects): 7.889 (#14)
Claude Sonnet 5.5 on Arabic Broad Leaderboard - Trust & Safety: 9.667 (#7)
Claude Sonnet 5.5 on Arabic Broad Leaderboard - Writing (incl Dialects): 7.727 (#33)
Claude Sonnet 5.5 on Blueprint-Bench 2: 0.393 (#5)
Claude Sonnet 5.5 on Chatbot Arena (Image-to-WebDev): 1740.0 (#2)
Claude Sonnet 5.5 on Chatbot Arena (Text - Math): 1512.0 (#8)
Claude Sonnet 5.5 on Chatbot Arena (Text - Mathematical): 1527.0 (#4)
Claude Sonnet 5.5 on ChessBench GitHub - Elo: 2350.0 (#15)
Claude Sonnet 5.5 on Clerk LLM Leaderboard: 91.7 (#3)
Claude Sonnet 5.5 on CursorBench 4.0: 55.5 (#4)
Claude Sonnet 5.5 on DRACO: 87.0 (#2)
Claude Sonnet 5.5 on Design Arena (Data Viz): 1382.0 (#2)
Claude Sonnet 5.5 on Design Arena (Game Dev): 1440.0 (#4)
Claude Sonnet 5.5 on Design Arena (Website): 1317.0 (#11)
Claude Sonnet 5.5 on DuelLab Overall: 83.600748204028 (#5)
Claude Sonnet 5.5 on EKI Estonian LLM - Book Recommendations: 61.31 (#22)
Claude Sonnet 5.5 on EKI Estonian LLM - Idioms: 47.33 (#34)
Claude Sonnet 5.5 on EKI Estonian LLM - Language Advice: 62.5 (#13)
Claude Sonnet 5.5 on EKI Estonian LLM - Overall: 45.57 (#38)
Claude Sonnet 5.5 on EKI Estonian LLM - Propaganda Resistance: 93.42 (#10)
Claude Sonnet 5.5 on EKI Estonian LLM - Terminology: 38.99 (#37)
Claude Sonnet 5.5 on EKI Estonian LLM - Trivia: 50.1 (#48)
Claude Sonnet 5.5 on EKI Estonian LLM - Trivia Hard: 10.88 (#29)
Claude Sonnet 5.5 on Epoch AI - Frontiermath Erdos: 2.9 (#3)
Claude Sonnet 5.5 on Epoch AI - Weirdml V3: 20.0 (#5)
Claude Sonnet 5.5 on Featherbench: 100.0 (#6)
Claude Sonnet 5.5 on Featherbench - Rubric Quality: 9.2 (#8)
Claude Sonnet 5.5 on Kaggle Cipher Frontlines: 81.82 (#19)
Claude Sonnet 5.5 on Kaggle Game Arena (Unified): 328.5 (#14)
Claude Sonnet 5.5 on Kaggle Game Arena Chess: 761.4 (#18)
Claude Sonnet 5.5 on LLM Chess (Saplin): 1194.567 (#22)
Claude Sonnet 5.5 on Maze-Bench: 58.47 (#5)
Claude Sonnet 5.5 on NYT Connections Extended: 80.5 (#44)
Claude Sonnet 5.5 on PRISM (1C:Enterprise) - Algorithmic Tasks (A): 93.0 (#13)
Claude Sonnet 5.5 on PRISM (1C:Enterprise) - Platform Tasks (B): 75.0 (#23)
Claude Sonnet 5.5 on PUMA Polish Multimodal - Contemporary Life: 43.0 (#26)
Claude Sonnet 5.5 on PUMA Polish Multimodal - Document QA: 87.0 (#5)
Claude Sonnet 5.5 on PUMA Polish Multimodal - Geography and Environment: 67.0 (#18)
Claude Sonnet 5.5 on PUMA Polish Multimodal - History and Culture: 61.0 (#18)
Claude Sonnet 5.5 on PUMA Polish Multimodal - OCR: 85.0 (#10)
Claude Sonnet 5.5 on PUMA Polish Multimodal - Structured Extraction: 65.0 (#11)
Claude Sonnet 5.5 on PUMA Polish Multimodal - Vision Mean: 68.0 (#15)
Claude Sonnet 5.5 on RuneBench: 5.524272394171 (#31)
Claude Sonnet 5.5 on Terminal-Bench 4.0 (Claude Code): 61.82 (#2)
Claude Sonnet 5.5 on Vending-Bench 2: 11165.27833333333 (#5)
Claude Sonnet 5.5 on WebDev Arena: 1786.26 (#3)
Claude Sonnet 5.5 on WebDev Arena (Fullstack): 1700.0 (#4)
Claude Sonnet 5.5 on YC-Bench: 1741.3148899999999 (#8)
GPT-6 Astra on AutomationBench: 41.4 (#3)
GPT-6 Astra on Chatbot Arena (Text - German): 1473.0 (#39)
GPT-6 Astra on Chatbot Arena (Text - Korean): 1449.0 (#21)
GPT-6 Astra on Clerk LLM Leaderboard: 90.5 (#8)
GPT-6 Astra on DeepSWE: 73.2 (#4)
GPT-6 Astra on DeepSWE-mini (16 Tasks): 79.69 (#2)
GPT-6 Astra on F5 Labs AI Security Leaderboard - ARS: 85.21 (#8)
GPT-6 Astra on FoodTruckBench: 57760.9 (#5)
GPT-6 Astra on Harvey LAB-AA: 8.61 (#3)
GPT-6 Astra on Harvey LAB-AA - All Pass: 8.89 (#11)
GPT-6 Astra on Harvey LAB-AA - Antitrust & Competition: 87.32 (#14)
GPT-6 Astra on Harvey LAB-AA - Arbitration & International Dispute Resolution: 88.53 (#15)
GPT-6 Astra on Harvey LAB-AA - Banking & Finance: 94.15 (#7)
GPT-6 Astra on Harvey LAB-AA - Bankruptcy & Restructuring: 89.16 (#9)
GPT-6 Astra on Harvey LAB-AA - Capital Markets: 95.95 (#3)
GPT-6 Astra on Harvey LAB-AA - Corporate Governance: 89.04 (#10)
GPT-6 Astra on Harvey LAB-AA - Corporate M&A: 94.75 (#7)
GPT-6 Astra on Harvey LAB-AA - Data Privacy & Cybersecurity: 95.29 (#8)
GPT-6 Astra on Harvey LAB-AA - Emerging Companies & Venture Capital: 90.1 (#14)
GPT-6 Astra on Harvey LAB-AA - Employment & Labor: 88.26 (#4)
GPT-6 Astra on Harvey LAB-AA - Energy & Natural Resources: 94.72 (#10)
GPT-6 Astra on Harvey LAB-AA - Environmental & ESG: 86.86 (#12)
GPT-6 Astra on Harvey LAB-AA - Funds & Asset Management: 87.46 (#14)
GPT-6 Astra on Harvey LAB-AA - Healthcare & Life Sciences: 88.48 (#18)
GPT-6 Astra on Harvey LAB-AA - Immigration: 92.85 (#4)
GPT-6 Astra on Harvey LAB-AA - Insurance: 92.89 (#5)
GPT-6 Astra on Harvey LAB-AA - Intellectual Property: 93.36 (#17)
GPT-6 Astra on Harvey LAB-AA - International Trade & Sanctions: 89.77 (#12)
GPT-6 Astra on Harvey LAB-AA - Litigation & Dispute Resolution: 91.35 (#10)
GPT-6 Astra on Harvey LAB-AA - Real Estate: 94.56 (#7)
GPT-6 Astra on Harvey LAB-AA - Structured Finance & Securitization: 89.89 (#8)
GPT-6 Astra on Harvey LAB-AA - Tax: 82.39 (#15)
GPT-6 Astra on Harvey LAB-AA - Trusts, Estates & Private Client: 91.48 (#12)
GPT-6 Astra on Harvey LAB-AA - White Collar Defense & Investigations: 87.09 (#12)
GPT-6 Astra on LLM Chess (Saplin): 1613.842 (#1)
GPT-6 Astra on LLMConfBench: 0.67 (#1)
GPT-6 Astra on PUMA Polish Multimodal - Contemporary Life: 73.0 (#9)
GPT-6 Astra on PUMA Polish Multimodal - Document QA: 85.0 (#9)
GPT-6 Astra on PUMA Polish Multimodal - Geography and Environment: 85.0 (#7)
GPT-6 Astra on PUMA Polish Multimodal - History and Culture: 75.0 (#8)
GPT-6 Astra on PUMA Polish Multimodal - OCR: 88.0 (#6)
GPT-6 Astra on PUMA Polish Multimodal - Structured Extraction: 58.0 (#17)
GPT-6 Astra on PUMA Polish Multimodal - Vision Mean: 77.33 (#7)
GPT-6 Astra on React Native Evals: 86.37 (#6)
GPT-6 Astra on RuneBench: 7.258225267108 (#1)
GPT-6 Astra on Thinkingbox: 58.31 (#6)
GPT-6 Astra on WDCD: 79.6 (#12)
GPT-6 Astra on WDCD R1 Understanding: 100.0 (#9)
GPT-6 Astra on WDCD R2 In-Document Resistance: 88.0 (#3)
GPT-6 Astra on WDCD R3 Pressure Integrity: 113.0 (#5)
GPT-6 Astra on Wolfram LLM Benchmarking Project: 69.3 (#28)
GPT-6 Astra on YC-Bench: 1336.46611 (#19)
GPT-6.1 Sol on BenchGecko Score: 93.4 (#6)
GPT-6.1 Sol on Chatbot Arena (Image-to-WebDev): 1703.0 (#5)
GPT-6.1 Sol on Chatbot Arena (Text - Math): 1488.0 (#29)
GPT-6.1 Sol on Chatbot Arena (Text - Mathematical): 1539.0 (#2)
GPT-6.1 Sol on Clerk LLM Leaderboard: 90.0 (#12)
GPT-6.1 Sol on DataBench: 58.0 (#6)
GPT-6.1 Sol on Design Arena (Game Dev): 1366.0 (#8)
GPT-6.1 Sol on Design Arena (Website): 1323.0 (#8)
GPT-6.1 Sol on DuelLab Overall: 87.712621275278 (#4)
GPT-6.1 Sol on Epoch AI - ECI: 166.09 (#3)
GPT-6.1 Sol on Epoch AI - Weirdml V3: 35.47 (#2)
GPT-6.1 Sol on Featherbench: 93.0 (#21)
GPT-6.1 Sol on Featherbench - Rubric Quality: 8.8 (#16)
GPT-6.1 Sol on Harvey LAB-AA: 6.94 (#4)
GPT-6.1 Sol on Harvey LAB-AA - All Pass: 7.5 (#12)
GPT-6.1 Sol on Harvey LAB-AA - Antitrust & Competition: 88.77 (#11)
GPT-6.1 Sol on Harvey LAB-AA - Arbitration & International Dispute Resolution: 90.8 (#10)
GPT-6.1 Sol on Harvey LAB-AA - Banking & Finance: 93.91 (#8)
GPT-6.1 Sol on Harvey LAB-AA - Bankruptcy & Restructuring: 86.27 (#16)
GPT-6.1 Sol on Harvey LAB-AA - Capital Markets: 96.47 (#2)
GPT-6.1 Sol on Harvey LAB-AA - Corporate Governance: 86.76 (#14)
GPT-6.1 Sol on Harvey LAB-AA - Corporate M&A: 93.57 (#12)
GPT-6.1 Sol on Harvey LAB-AA - Data Privacy & Cybersecurity: 94.94 (#10)
GPT-6.1 Sol on Harvey LAB-AA - Emerging Companies & Venture Capital: 89.99 (#16)
GPT-6.1 Sol on Harvey LAB-AA - Employment & Labor: 85.39 (#12)
GPT-6.1 Sol on Harvey LAB-AA - Energy & Natural Resources: 95.27 (#8)
GPT-6.1 Sol on Harvey LAB-AA - Environmental & ESG: 87.69 (#11)
GPT-6.1 Sol on Harvey LAB-AA - Funds & Asset Management: 88.34 (#9)
GPT-6.1 Sol on Harvey LAB-AA - Healthcare & Life Sciences: 90.49 (#12)
GPT-6.1 Sol on Harvey LAB-AA - Immigration: 91.39 (#11)
GPT-6.1 Sol on Harvey LAB-AA - Insurance: 91.42 (#10)
GPT-6.1 Sol on Harvey LAB-AA - Intellectual Property: 95.26 (#6)
GPT-6.1 Sol on Harvey LAB-AA - International Trade & Sanctions: 89.51 (#13)
GPT-6.1 Sol on Harvey LAB-AA - Litigation & Dispute Resolution: 89.35 (#18)
GPT-6.1 Sol on Harvey LAB-AA - Real Estate: 94.27 (#8)
GPT-6.1 Sol on Harvey LAB-AA - Structured Finance & Securitization: 89.68 (#10)
GPT-6.1 Sol on Harvey LAB-AA - Tax: 83.28 (#11)
GPT-6.1 Sol on Harvey LAB-AA - Trusts, Estates & Private Client: 91.48 (#13)
GPT-6.1 Sol on Harvey LAB-AA - White Collar Defense & Investigations: 89.79 (#9)
GPT-6.1 Sol on HealthBench-Psych - Hard: 39.269 (#8)
GPT-6.1 Sol on HieroglyphBench: 31.4 (#8)
GPT-6.1 Sol on Kaggle Game Arena (Unified): 438.2 (#3)
GPT-6.1 Sol on Kaggle Game Arena Chess: 1186.75 (#9)
GPT-6.1 Sol on Kaggle Game Arena Four in a Row: 574.26 (#6)
GPT-6.1 Sol on Kaggle Open Benchmarks - AIME 2025: 100.0 (#9)
GPT-6.1 Sol on Kaggle Open Benchmarks - MATH-500: 99.01 (#4)
GPT-6.1 Sol on Kaggle Open Benchmarks - MMLU: 93.83 (#9)
GPT-6.1 Sol on Kaggle Open Benchmarks - MMLU-Pro: 89.44 (#8)
GPT-6.1 Sol on Kaggle SimpleQA: 71.08 (#3)
GPT-6.1 Sol on Kaggle SimpleQA Verified: 73.77 (#4)
GPT-6.1 Sol on KernelBench Hub - CUDA: 30.52 (#10)
GPT-6.1 Sol on KernelBench Hub - Mega: 4.99 (#11)
GPT-6.1 Sol on LLM Chess (Saplin): 1611.585 (#2)
GPT-6.1 Sol on LLM Stats (FrontierCode 1.1): 50.2 (#6)
GPT-6.1 Sol on LLM Stats (Humanity's Last Exam (no tools, text-only)): 52.9 (#3)
GPT-6.1 Sol on Legal Benchmarks - Contract Workflows - Criteria Pass Rate: 86.3 (#6)
GPT-6.1 Sol on Legal Benchmarks - Contract Workflows - Form: 90.8 (#5)
GPT-6.1 Sol on Legal Benchmarks - Contract Workflows - Task Pass Rate: 25.8 (#10)
GPT-6.1 Sol on Legal Benchmarks - Data Extraction - Criteria Pass Rate: 84.5 (#7)
GPT-6.1 Sol on Legal Benchmarks - Data Extraction - Form: 83.0 (#2)
GPT-6.1 Sol on Legal Benchmarks - Data Extraction - Task Pass Rate: 53.3 (#10)
GPT-6.1 Sol on NYT Connections Extended: 95.5 (#6)
GPT-6.1 Sol on PRISM (1C:Enterprise) - Algorithmic Tasks (A): 87.0 (#19)
GPT-6.1 Sol on PRISM (1C:Enterprise) - Platform Tasks (B): 90.0 (#6)
GPT-6.1 Sol on PUMA Polish Multimodal - Contemporary Life: 71.0 (#11)
GPT-6.1 Sol on PUMA Polish Multimodal - Document QA: 85.0 (#8)
GPT-6.1 Sol on PUMA Polish Multimodal - Geography and Environment: 81.0 (#9)
GPT-6.1 Sol on PUMA Polish Multimodal - History and Culture: 78.0 (#6)
GPT-6.1 Sol on PUMA Polish Multimodal - OCR: 87.0 (#7)
GPT-6.1 Sol on PUMA Polish Multimodal - Structured Extraction: 56.0 (#18)
GPT-6.1 Sol on PUMA Polish Multimodal - Vision Mean: 76.33 (#8)
GPT-6.1 Sol on ParseBench: 70.48 (#28)
GPT-6.1 Sol on ParseBench - Charts: 37.16 (#57)
GPT-6.1 Sol on ParseBench - Content Faithfulness: 89.25 (#23)
GPT-6.1 Sol on ParseBench - Semantic Formatting: 74.78 (#11)
GPT-6.1 Sol on ParseBench - Tables: 92.44 (#6)
GPT-6.1 Sol on ParseBench - Visual Grounding: 58.76 (#49)
GPT-6.1 Sol on ReLE - Agents - BFCL-V3: 55.6 (#99)
GPT-6.1 Sol on ReLE - Agents and Tool Use: 55.3 (#106)
GPT-6.1 Sol on ReLE - Coding: 61.0 (#43)
GPT-6.1 Sol on ReLE - Education: 63.4 (#22)
GPT-6.1 Sol on ReLE - Education - Gaokao: 59.6 (#39)
GPT-6.1 Sol on ReLE - Education - High School Subjects: 72.4 (#24)
GPT-6.1 Sol on ReLE - Education - Middle School Subjects: 50.9 (#46)
GPT-6.1 Sol on ReLE - Education - Primary School Subjects: 70.7 (#9)
GPT-6.1 Sol on ReLE - Finance: 85.1 (#20)
GPT-6.1 Sol on ReLE - Language and Instruction Following: 74.5 (#12)
GPT-6.1 Sol on ReLE - Law - Lawyer Qualification Exam: 84.7 (#6)
GPT-6.1 Sol on ReLE - Law and Civil Service: 87.3 (#5)
GPT-6.1 Sol on ReLE - Medicine - Basic Medical Knowledge: 86.0 (#18)
GPT-6.1 Sol on ReLE - Medicine - Physician Exams: 85.7 (#45)
GPT-6.1 Sol on ReLE - Medicine and Mental Health: 86.1 (#39)
GPT-6.1 Sol on ReLE - Overall: 71.7 (#35)
GPT-6.1 Sol on ReLE - Reasoning - BBH: 62.6 (#173)
GPT-6.1 Sol on ReLE - Reasoning and Mathematics: 81.2 (#38)
GPT-6.1 Sol on React Native Evals: 86.34 (#8)
GPT-6.1 Sol on RuneBench: 6.875920012269 (#3)
GPT-6.1 Sol on SEAL - Professional Reasoning Benchmark - Finance: 46.21 (#17)
GPT-6.1 Sol on SEAL - Professional Reasoning Benchmark - Legal: 47.87 (#15)
GPT-6.1 Sol on Sonar LLM Leaderboard - Java - Bug Density: 0.61 (#10)
GPT-6.1 Sol on Sonar LLM Leaderboard - Java - Code Smell Density: 18.6 (#38)
GPT-6.1 Sol on Sonar LLM Leaderboard - Java - Pass Rate: 85.29 (#5)
GPT-6.1 Sol on Sonar LLM Leaderboard - Java - Vulnerability Density: 0.19 (#17)
GPT-6.1 Sol on SpeechMap Compliance: 69.5 (#153)
GPT-6.1 Sol on WDCD: 81.9 (#9)
GPT-6.1 Sol on WDCD R1 Understanding: 100.0 (#10)
GPT-6.1 Sol on WDCD R2 In-Document Resistance: 75.0 (#9)
GPT-6.1 Sol on WDCD R3 Pressure Integrity: 100.0 (#9)
GPT-6.1 Sol on Wolfram LLM Benchmarking Project: 69.3 (#29)
GPT-6.1 Sol on YC-Bench: 936.2673199999999 (#32)
Gemini 4 Argon on ComplexConstraints: 53.3 (#2)
Gemini 4 Argon on Epoch AI - Critpt: 27.14 (#33)
Gemini 4 Argon on Epoch AI - Frontierswe: 54.96 (#5)
Gemini 4 Argon on Epoch AI - Proofbench: 99.0 (#7)
Gemini 4 Argon on Epoch AI - Scicode: 61.81 (#4)
Gemini 4 Argon on Epoch AI - Vending Bench 2: 13718.16 (#3)
Gemini 4 Argon on GDP.pdf: 28.4 (#5)
Gemini 4 Argon on LLM Stats (Agents' Last Exam): 39.5 (#12)
Gemini 4 Argon on LLM Stats (AutomationBench): 51.3 (#5)
Gemini 4 Argon on LLM Stats (Graphwalks BFS >128k): 84.2 (#2)
Gemini 4 Argon on LLM Stats (OSWorld 2.0): 69.2 (#6)
Gemini 4 Argon on LLM Stats (Terminal-Bench 4.0): 57.4 (#5)
Gemini 4 Argon on LLM Stats (Terminal-Bench-Science 0.1): 57.6 (#4)
Gemini 4 Argon on LLM Stats Score: 55.09 (#4)
Gemini 4 Argon on Vals AI Time Horizon Index: 52.5 (#4)
Gemini 4 Argon on WebDev Arena: 1679.5 (#9)

New #1 Leaders (95)

Kaggle Game Arena (Unified): GPT-6 Astra (496.11) beat Claude Opus 5 by 141.64
ContextEcho: Command R7B (-15.0) beat Kimi K2.6 by 115.0
gwBenchmarks: Claude Sonnet 4.6 (14.22) beat Claude Haiku 4.5 by 45.09
Agents on Rails: Claude Opus 5 (High) (92.06349206349206) beat GPT-6 Astra (Max) by 38.76
OpenVLM Video - MMBench-Video: GPT-4o (2024-08-06) (Frames: 1 fps) (71.66666666666667) beat GPT-4o (2024-08-06) by 28.67
HalluWorld: GPT-5.5 (5.0) beat GPT-4o Mini by 23.1
Chatbot Arena (Image-to-WebDev): Claude Opus 5.5 (Max) (1749.0) beat GPT-6 Astra (Max) by 16.0
Lean AI Formalization Leaderboard: Humanifa + GPT 5.6 sol (251.0) beat Axiom Prover (Axiom Math) by 13.0
Design Arena (Game Dev): GPT-6 Astra (Max) (1491.0) beat Claude Opus 5.5 by 12.0
Wolfram LLM Benchmarking Project: Claude Opus 5.5 thinking max (84.5) beat Claude Opus 5 thinking max by 10.9
Evals for Every Language - Language nus: Claude Opus 4.8 (60.447266703333334) beat Claude Opus 4.7 by 9.41
Kaggle Cipher Frontlines: GPT-6.1 Sol (100.0) beat Gemini 3 Flash (Preview) by 9.09
Evals for Every Language - Language yue: Claude Opus 4.8 (71.77526519999999) beat Claude Opus 4.7 by 8.3
LLM Stats (WildClawBench): Qwen3.8 Omni Flash (71.0) beat Seed 2.1 Turbo by 8.2
Evals for Every Language - Language sa: Gemini 3.1 Pro (Preview) (71.74656785666666) beat step-3.7-flash-20260528 by 7.35
Evals for Every Language - Language sg: Claude Opus 4.7 (68.85848491333333) beat Gemini 3.1 Pro (Preview) by 7.23
WebDev Arena (Data & Analytics): Claude Opus 5.5 (Max) (1694.0) beat GPT-6 Astra (Max) by 7.0
Evals for Every Language - Language gom: Gemini 3.1 Flash Lite (73.817165625) beat Gemini 3.1 Pro (Preview) by 6.8
Epoch AI - Apex Agents: Gemini 4 Argon (82.2) beat Claude Sonnet 5.5 (Max) by 6.7
Evals for Every Language - Language am: Gemini 3.1 Pro (Preview) (75.25738270333333) beat Gemini 3.1 Flash Lite by 6.66
SEAL - MultiChallenge: GPT-6.1 Sol (81.95) beat Muse Spark by 6.43
TrackingAI IQ Test (Offline): Claude Opus 5 (Max) (87.5) beat GPT-5 Pro by 6.25
Evals for Every Language - Language bho: Claude Sonnet 4 (75.19950074666666) beat Gemini 2.5 Pro by 6.22
Evals for Every Language - Language chm: Claude Opus 4.8 (69.72101609833332) beat Claude Opus 4.7 by 6.12
Evals for Every Language - Language rn: Claude Sonnet 4.6 (70.95658003666667) beat Gemini 3.1 Pro (Preview) by 6.1
Agent Arena - Praise vs Complaint: Gemini 4 Argon (High) (40.6) beat GPT-6 Astra (Max) by 5.97
Evals for Every Language - Language so: Claude Opus 4.8 (75.37049721666666) beat Claude Opus 4.7 by 5.89
VANTAGE-Bench: GPT-6 Astra (75.26) beat Gemini 3.6 Flash by 5.74
LLM Stats (Graphwalks BFS <128k): Gemini 4 Argon (99.7) beat GPT-5.2 by 5.7
VANTAGE-Bench - Semantic: GPT-6 Astra (85.08) beat Gemini 3.6 Flash by 5.67
Evals for Every Language - Language lij: Gemini 3.1 Pro (Preview) (77.85008922833333) beat Claude Sonnet 4.5 by 5.66
Evals for Every Language - Language luo: Claude Opus 4.7 (70.18815189833335) beat Gemini 3.1 Pro (Preview) by 5.62
SAHARA - Machine Translation (African to African): Sunflower-Qwen3.8-27B (17.81) beat Gemini 3 Pro (Preview) by 5.58
LLM Stats (PostTrainBench): Gemini 4 Argon (45.3) beat GLM-5.3 by 5.5
Evals for Every Language - Language tk: Claude Opus 4.7 (75.78373375666666) beat Gemini 3.1 Pro (Preview) by 5.47
Evals for Every Language - Language gd: Gemini 3.1 Pro (Preview) (78.26956672166666) beat Claude Opus 4.8 by 5.42
Evals for Every Language - Language bn: Claude Sonnet 4 (78.70639349833333) beat Gemini 3.1 Pro (Preview) by 5.39
Evals for Every Language - Language hy: Gemini 3.1 Pro (Preview) (75.66078052499999) beat Claude Sonnet 4.5 by 5.38
VANTAGE-Bench - Single Object Tracking: GPT-6 Astra (81.2) beat Gemini 3.6 Flash by 5.21
VANTAGE-Bench - Video QA: GPT-6 Astra (83.26) beat GPT-5.6 Sol by 5.18
VANTAGE-Bench - Temporal: GPT-6 Astra (51.64) beat GPT-5.6 Sol by 5.14
LLM Stats (AutomationBench): Mistral Large 4 (59.9) beat DeepSeek V4.1 Flash by 5.1
Evals for Every Language - Language or: Claude Opus 4.7 (76.12501505666667) beat Claude Sonnet 4.5 by 5.04
VANTAGE-Bench - Event Verification: GPT-6 Astra (86.9) beat Gemini 3.6 Flash by 4.9
LLM Stats (LVBench): Gemini 4 Argon (91.7) beat Gemini 3.8 Flash by 4.6
Evals for Every Language - Language ace: Gemini 3.1 Pro (Preview) (76.82400763) beat step-3.7-flash-20260528 by 4.34
Evals for Every Language - Language hi: Gemini 3.1 Flash Lite (81.45005576) beat Claude Opus 4.7 by 4.27
Evals for Every Language - Language nl: Gemini 3.1 Pro (Preview) (79.20258365833332) beat Gemini 3.1 Flash Lite by 4.17
Evals for Every Language - Language eo: Gemini 3.1 Flash Lite (80.62078474) beat Claude Sonnet 4.5 by 4.08
VANTAGE-Bench - Spatial: GPT-6 Astra (83.12) beat Qwen 3.5 27B by 4.08
Evals for Every Language - Language et: Gemini 3.1 Pro (Preview) (76.65396632333334) beat Gemini 2.5 Pro by 4.05
SuperCLUE-Terminal - Overall: MiMo-V2.6-Pro (64.65) beat Kimi K3 (Max) by 4.04
Design Arena (Text to Speech): google-gemini-3.8-flash-tts (1446.0) beat google-gemini-3.1-flash-tts by 4.0
Evals for Every Language - Language ha: Claude Opus 4.8 (74.69938437666667) beat Gemini 3.1 Pro (Preview) by 3.92
Evals for Every Language - Language ban: Gemini 3.1 Pro (Preview) (72.70350369333333) beat step-3.7-flash-20260528 by 3.67
Evals for Every Language - Language ka: Claude Opus 4.8 (74.80095421) beat step-3.7-flash-20260528 by 3.67
HealthBench-Psych - Hard: Claude Sonnet 5.5 (45.159) beat Claude Opus 5 by 3.65
Evals for Every Language - Language tg: Gemini 3.1 Pro (Preview) (76.03920221) beat step-3.7-flash-20260528 by 3.49
Evals for Every Language - Language da: DeepSeek V3.1 (80.26741193333334) beat Gemini 3.1 Pro (Preview) by 3.42
Evals for Every Language - Language ceb: Gemini 3.1 Pro (Preview) (81.41321678333334) beat Gemini 3.1 Flash Lite by 3.35
Evals for Every Language - Language sw: Claude Sonnet 4.6 (79.88856511499999) beat Gemini 3.1 Pro (Preview) by 3.18
Evals for Every Language - Language id: Gemini 3.1 Pro (Preview) (78.05458235333333) beat Claude Sonnet 4.5 by 3.07
Evals for Every Language - Language dz: Claude Sonnet 4.5 (64.94176956833333) beat GPT-5 by 3.0
Chatbot Arena (Vision - Chinese): Gemini 3.8 Flash (High) (1380.0) beat Gemini 3.7 Flash (High) by 3.0
Evals for Every Language - Language cy: Gemini 3.1 Pro (Preview) (84.91070339833334) beat Gemini 3.1 Flash Lite by 2.88
Evals for Every Language - Language ms: Gemini 3.1 Pro (Preview) (81.87736843666666) beat Gemini 3.1 Flash Lite by 2.83
Harvey LAB-AA - Structured Finance & Securitization: Muse Spark 1.3 (Max) (94.84) beat Muse Spark 1.3 (xHigh) by 2.78
Benchmarks.bio - MetagenomicsBench: Claude Opus 5.5 (62.7) beat Claude Sonnet 5.5 by 2.7
LLM Stats (APEX-Agents): GPT-6.1 Sol (60.0) beat Grok 4.6 by 2.5
LLM Stats (DeepSWE 1.1): Gemini 4 Argon (77.9) beat Muse Spark 1.3 by 2.5
Evals for Every Language - Language crh: Claude Opus 4.8 (75.16963044333333) beat step-3.7-flash-20260528 by 2.12
PUMA Polish Multimodal - Geography and Environment: Gemini 3.8 Flash (91.0) beat Gemini 3.7 Flash by 2.0
Harvey LAB-AA - Employment & Labor: Muse Spark 1.3 (Max) (93.63) beat Muse Spark 1.3 (xHigh) by 1.5
Harvey LAB-AA - Corporate Governance: Muse Spark 1.3 (Max) (97.37) beat Muse Spark 1.3 (xHigh) by 1.48
PUMA Polish Multimodal - Contemporary Life: Gemini 3.8 Flash (84.0) beat Gemini 3.1 Pro (Preview) by 1.0
PUMA Polish Multimodal - Document QA: Claude Opus 5.5 (89.0) beat Gemini 3.5 Flash by 1.0
Evals for Every Language - Language cv: Gemini 3.1 Pro (Preview) (70.18880007666668) beat gemma-4-31B-it-20260402 by 0.89
Medical Information Response Audit (MIRA): DeepSeek V4 Pro (2.16) beat Llama 3.3 70B by 0.79
Harvey LAB-AA - Litigation & Dispute Resolution: Muse Spark 1.3 (Max) (97.37) beat Qwen 3.8 Max (0902) by 0.74
Clerk LLM Leaderboard: DeepSeek V4.1 Flash (92.0) beat Claude Opus 4.8 by 0.7
SWE-bench Live: TianxiCode + Deepseek-v4.1-Flash (71.0) beat Slingshot-v3.4.0 + GPT-5.6-Sol by 0.67
Harvey LAB-AA - Trusts, Estates & Private Client: Muse Spark 1.3 (Max) (96.44) beat Claude Sonnet 5.5 (Xhigh, Default Fallback) by 0.64
Harvey LAB-AA - Intellectual Property: Muse Spark 1.3 (Max) (97.75) beat Claude Fable 5.1 (High, Default Fallback) by 0.58
Arcee MMLU-Pro (BenchLM): Qwen 3.7 Max (89.6) beat Claude Opus 4.6 by 0.5
Harvey LAB-AA - Real Estate: Muse Spark 1.3 (Max) (97.57) beat Muse Spark 1.3 (xHigh) by 0.49
SEAL - Remote Labor Index (RLI): Claude Opus 5.5 (21.25) beat GPT-6 Astra by 0.42
Harvey LAB-AA - Capital Markets: Muse Spark 1.3 (Max) (97.65) beat Claude Fable 5.1 (Xhigh, Default Fallback) by 0.4
Open Medical LLM - MedQA (USMLE): medllama3-v20 (Medical precision bfloat16) (81.38256087981148) beat medllama3-v20 by 0.31
Open Medical LLM - PubMedQA: Med-PaLM 2 (5 Shots) (79.2) beat Flan-PaLM by 0.2
OSWorld 2.0 Partial: Sai w/ Claude Opus 5 (79.38) beat Claude Opus 5 by 0.18
Epoch AI - Blueprint Bench 2: Gemini 4 Argon (0.544) beat GPT-6 Astra by 0.05
EdiTikZ: EdiTikZ-9B-RL (0.726) beat GPT-5.5 by 0.04
Senior SWE-Bench: GPT-5.6 Sol (Mini-SWE-Agent, xHigh) (34.73684210526316) beat Claude Fable 5.1 by 0.04
PACT - Rule-Scope Discernment: Inkling (0.909) beat Nemotron 3 Ultra by 0.02
Agent Arena: Claude Opus 5.5 (High) (14.33) beat Claude Fable 5.1 (Max) by 0.02

The Aggregate Digest — 2026-10-10

2026-10-10T06:28:05.131489+00:00

Daily

New Benchmarks (22)

Anthropic ECI - October 2026 Fit (Haiku 5.5 System Card) (AECI Score): leader Claude Opus 5.5 (174.56), 15 models
Anthropic's internal Epoch Capabilities Index (AECI), an IRT composite of broad benchmark performance, from section 2.3.2 of the October 7, 2026 Haiku 5.5 System Card. Higher is better, with no fixed score bounds. This leaderboard includes all 15 models in Figure 2.3.2.A on page 20. Haiku 5.5 and Opus 5.5 use exact printed scores; June 2024 Sonnet uses the declared 130 anchor. The other 12 scores are approximate, digitized from the original chart to one decimal; this extraction precision is separate from the chart's statistical confidence intervals. June and October 2024 Sonnet remain distinct releases. Individual effort settings are not specified in the AECI paragraph. Scores from this fit are not directly comparable to earlier AECI fits or public Epoch ECI; the chart anchors June 2024 Claude 3.5 Sonnet at 130. The complete current benchmark basket and scored questions are not disclosed.
STEMBench120 v1.0.0 (Tools) - Overall (Mean fractional scientific progress (0–4; 20 tasks/domain; zero-credit failures included)): leader GPT-6.1 Sol (2.759), 1 models
Ulam STEMBench120 v1.0.0 evaluates fractional scientific progress on 120 frontier problems, 20 each in six STEM domains. Multi-day tool-assisted runs are scored separately from closed-book runs. This board reports overall mean rubric credit from 0 to 4, including zero-credit failures; higher is better.
STEMBench120 v1.0.0 (Tools) - Biology (Mean fractional scientific progress (0–4; 20 tasks/domain; zero-credit failures included)): leader GPT-6.1 Sol (2.656), 1 models
Ulam STEMBench120 v1.0.0 evaluates fractional scientific progress on 120 frontier problems, 20 each in six STEM domains. Multi-day tool-assisted runs are scored separately from closed-book runs. This board reports biology mean rubric credit from 0 to 4, including zero-credit failures; higher is better.
STEMBench120 v1.0.0 (Tools) - Chemistry (Mean fractional scientific progress (0–4; 20 tasks/domain; zero-credit failures included)): leader GPT-6.1 Sol (2.73), 1 models
Ulam STEMBench120 v1.0.0 evaluates fractional scientific progress on 120 frontier problems, 20 each in six STEM domains. Multi-day tool-assisted runs are scored separately from closed-book runs. This board reports chemistry mean rubric credit from 0 to 4, including zero-credit failures; higher is better.
STEMBench120 v1.0.0 (Tools) - Physics (Mean fractional scientific progress (0–4; 20 tasks/domain; zero-credit failures included)): leader GPT-6.1 Sol (2.986), 1 models
Ulam STEMBench120 v1.0.0 evaluates fractional scientific progress on 120 frontier problems, 20 each in six STEM domains. Multi-day tool-assisted runs are scored separately from closed-book runs. This board reports physics mean rubric credit from 0 to 4, including zero-credit failures; higher is better.
STEMBench120 v1.0.0 (Tools) - Astronomy (Mean fractional scientific progress (0–4; 20 tasks/domain; zero-credit failures included)): leader GPT-6.1 Sol (2.692), 1 models
Ulam STEMBench120 v1.0.0 evaluates fractional scientific progress on 120 frontier problems, 20 each in six STEM domains. Multi-day tool-assisted runs are scored separately from closed-book runs. This board reports astronomy mean rubric credit from 0 to 4, including zero-credit failures; higher is better.
STEMBench120 v1.0.0 (Tools) - Earth science (Mean fractional scientific progress (0–4; 20 tasks/domain; zero-credit failures included)): leader GPT-6.1 Sol (2.676), 1 models
Ulam STEMBench120 v1.0.0 evaluates fractional scientific progress on 120 frontier problems, 20 each in six STEM domains. Multi-day tool-assisted runs are scored separately from closed-book runs. This board reports earth science mean rubric credit from 0 to 4, including zero-credit failures; higher is better.
STEMBench120 v1.0.0 (Tools) - Engineering (Mean fractional scientific progress (0–4; 20 tasks/domain; zero-credit failures included)): leader GPT-6.1 Sol (2.811), 1 models
Ulam STEMBench120 v1.0.0 evaluates fractional scientific progress on 120 frontier problems, 20 each in six STEM domains. Multi-day tool-assisted runs are scored separately from closed-book runs. This board reports engineering mean rubric credit from 0 to 4, including zero-credit failures; higher is better.
A2Z GameSpec-Bench - GDD Fidelity (GDD Fidelity (%; coding-agent game design fidelity; Table 2)): leader Claude Fable 5.1 (77.0), 9 models
A2Z GameSpec-Bench evaluates coding agents on fidelity to 100 small and large browser-game design documents. Source-code checks, scenario replay and adaptive playtests score compliance with dependency-aware design requirements. This board reports the published Table 2 GDD Fidelity percentage; higher is better. The published overall score is retained, rather than averaging rounded component scores.
A2Z GameSpec-Bench - GDD Fidelity Small (GDD Fidelity Small (%; coding-agent game design fidelity; Table 2)): leader Claude Fable 5.1 (82.8), 9 models
A2Z GameSpec-Bench evaluates coding agents on fidelity to 100 small and large browser-game design documents. Source-code checks, scenario replay and adaptive playtests score compliance with dependency-aware design requirements. This board reports the published Table 2 GDD Fidelity Small percentage; higher is better. The published overall score is retained, rather than averaging rounded component scores.
A2Z GameSpec-Bench - GDD Fidelity Big (GDD Fidelity Big (%; coding-agent game design fidelity; Table 2)): leader Claude Fable 5.1 (71.1), 9 models
A2Z GameSpec-Bench evaluates coding agents on fidelity to 100 small and large browser-game design documents. Source-code checks, scenario replay and adaptive playtests score compliance with dependency-aware design requirements. This board reports the published Table 2 GDD Fidelity Big percentage; higher is better. The published overall score is retained, rather than averaging rounded component scores.
A2Z GameSpec-Bench - Source Code Small (Source Code Small (%; coding-agent game design fidelity; Table 2)): leader Claude Fable 5.1 (89.7), 9 models
A2Z GameSpec-Bench evaluates coding agents on fidelity to 100 small and large browser-game design documents. Source-code checks, scenario replay and adaptive playtests score compliance with dependency-aware design requirements. This board reports the published Table 2 Source Code Small percentage; higher is better. The published overall score is retained, rather than averaging rounded component scores.
A2Z GameSpec-Bench - Source Code Big (Source Code Big (%; coding-agent game design fidelity; Table 2)): leader Claude Fable 5.1 (83.5), 9 models
A2Z GameSpec-Bench evaluates coding agents on fidelity to 100 small and large browser-game design documents. Source-code checks, scenario replay and adaptive playtests score compliance with dependency-aware design requirements. This board reports the published Table 2 Source Code Big percentage; higher is better. The published overall score is retained, rather than averaging rounded component scores.
A2Z GameSpec-Bench - Scenario Replay Small (Scenario Replay Small (%; coding-agent game design fidelity; Table 2)): leader Claude Fable 5.1 (73.5), 9 models
A2Z GameSpec-Bench evaluates coding agents on fidelity to 100 small and large browser-game design documents. Source-code checks, scenario replay and adaptive playtests score compliance with dependency-aware design requirements. This board reports the published Table 2 Scenario Replay Small percentage; higher is better. The published overall score is retained, rather than averaging rounded component scores.
A2Z GameSpec-Bench - Scenario Replay Big (Scenario Replay Big (%; coding-agent game design fidelity; Table 2)): leader Claude Fable 5.1 (55.1), 9 models
A2Z GameSpec-Bench evaluates coding agents on fidelity to 100 small and large browser-game design documents. Source-code checks, scenario replay and adaptive playtests score compliance with dependency-aware design requirements. This board reports the published Table 2 Scenario Replay Big percentage; higher is better. The published overall score is retained, rather than averaging rounded component scores.
A2Z GameSpec-Bench - Adaptive Playtest Small (Adaptive Playtest Small (%; coding-agent game design fidelity; Table 2)): leader Claude Fable 5.1 (85.2), 9 models
A2Z GameSpec-Bench evaluates coding agents on fidelity to 100 small and large browser-game design documents. Source-code checks, scenario replay and adaptive playtests score compliance with dependency-aware design requirements. This board reports the published Table 2 Adaptive Playtest Small percentage; higher is better. The published overall score is retained, rather than averaging rounded component scores.
A2Z GameSpec-Bench - Adaptive Playtest Big (Adaptive Playtest Big (%; coding-agent game design fidelity; Table 2)): leader Claude Fable 5.1 (74.7), 9 models
A2Z GameSpec-Bench evaluates coding agents on fidelity to 100 small and large browser-game design documents. Source-code checks, scenario replay and adaptive playtests score compliance with dependency-aware design requirements. This board reports the published Table 2 Adaptive Playtest Big percentage; higher is better. The published overall score is retained, rather than averaging rounded component scores.
BrickBench (Overall, BrickAgent) - Validity (Validity (fraction); published Overall cohort): leader GPT-6 Astra (1.0), 11 models
BrickBench evaluates text-conditioned LEGO design with the BrickAgent construction, inspection and validation environment. The published Overall cohort combines model, set and alternate-build tasks. This board reports Validity; higher is better. Validity and VQA retain their native 0–1 scale; the design and alignment ratings retain the printed Elo point estimate rather than its adjacent uncertainty. Specialist single-setting rows and no-BrickAgent ablations are separate protocols.
BrickBench (Overall, BrickAgent) - VQA (VQA (fraction); published Overall cohort): leader GPT-6 Astra (0.954), 11 models
BrickBench evaluates text-conditioned LEGO design with the BrickAgent construction, inspection and validation environment. The published Overall cohort combines model, set and alternate-build tasks. This board reports VQA; higher is better. Validity and VQA retain their native 0–1 scale; the design and alignment ratings retain the printed Elo point estimate rather than its adjacent uncertainty. Specialist single-setting rows and no-BrickAgent ablations are separate protocols.
BrickBench (Overall, BrickAgent) - Elo (Elo (rating); published Overall cohort): leader GPT-6 Astra (1297.0), 11 models
BrickBench evaluates text-conditioned LEGO design with the BrickAgent construction, inspection and validation environment. The published Overall cohort combines model, set and alternate-build tasks. This board reports Elo; higher is better. Validity and VQA retain their native 0–1 scale; the design and alignment ratings retain the printed Elo point estimate rather than its adjacent uncertainty. Specialist single-setting rows and no-BrickAgent ablations are separate protocols.
BrickBench (Overall, BrickAgent) - Alignment Elo (Alignment Elo (rating); published Overall cohort): leader GPT-6.1 Sol (1301.0), 11 models
BrickBench evaluates text-conditioned LEGO design with the BrickAgent construction, inspection and validation environment. The published Overall cohort combines model, set and alternate-build tasks. This board reports Alignment Elo; higher is better. Validity and VQA retain their native 0–1 scale; the design and alignment ratings retain the printed Elo point estimate rather than its adjacent uncertainty. Specialist single-setting rows and no-BrickAgent ablations are separate protocols.
BrickBench (Overall, BrickAgent) - Design Elo (Design Elo (rating); published Overall cohort): leader GPT-6 Astra (1299.0), 11 models
BrickBench evaluates text-conditioned LEGO design with the BrickAgent construction, inspection and validation environment. The published Overall cohort combines model, set and alternate-build tasks. This board reports Design Elo; higher is better. Validity and VQA retain their native 0–1 scale; the design and alignment ratings retain the printed Elo point estimate rather than its adjacent uncertainty. Specialist single-setting rows and no-BrickAgent ablations are separate protocols.

Top-10 New Scores (22)

Claude Fable 5.1 on Benchmarks.bio - MetagenomicsBench: 61.7 (#2)
Claude Opus 5.5 on Chatbot Arena (Vision - Diagram): 1325.0 (#8)
Claude Opus 5.5 on Chatbot Arena (Vision - English): 1290.0 (#9)
Claude Opus 5.5 on Chatbot Arena (Vision - Homework): 1366.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Vision - OCR): 1305.0 (#12)
Claude Opus 5.5 on Chatbot Arena (Vision): 1294.0 (#8)
Claude Sonnet 5.5 on Blueprint-Bench 2: 0.393 (#5)
Claude Sonnet 5.5 on EKI Estonian LLM - Book Recommendations: 61.31 (#22)
Claude Sonnet 5.5 on EKI Estonian LLM - Idioms: 47.33 (#34)
Claude Sonnet 5.5 on EKI Estonian LLM - Language Advice: 62.5 (#13)
Claude Sonnet 5.5 on EKI Estonian LLM - Overall: 45.57 (#38)
Claude Sonnet 5.5 on EKI Estonian LLM - Propaganda Resistance: 93.42 (#10)
Claude Sonnet 5.5 on EKI Estonian LLM - Terminology: 38.99 (#37)
Claude Sonnet 5.5 on EKI Estonian LLM - Trivia: 50.1 (#48)
Claude Sonnet 5.5 on EKI Estonian LLM - Trivia Hard: 10.88 (#29)
Claude Sonnet 5.5 on Vending-Bench 2: 11165.27833333333 (#5)
GPT-6 Astra on BrowseComp (Mercor 130): 94.2 (#3)
GPT-6 Astra on Mercor APEX: 64.7 (#8)
GPT-6.1 Sol on BrowseComp (Mercor 130): 95.4 (#2)
GPT-6.1 Sol on Mercor APEX: 60.0 (#12)
GPT-6.1 Sol on SpeechMap Compliance: 69.5 (#153)
GPT-6.1 Sol on Wolfram LLM Benchmarking Project: 69.3 (#29)

New #1 Leaders (3)

Last Translation Benchmark: Lara 3 - Think (56.09220636663008) beat GPT-6 Astra Pro by 10.76
Chatbot Arena (Vision - Homework): Claude Opus 5.5 (High) (1366.0) beat GPT-6.1 Sol (Max) by 5.0
BrowseComp (Mercor 130): GPT-5.6 Sol (Max) (95.7) beat GPT-6.1 Sol (Max) by 0.3

The Aggregate Digest — 2026-10-09

2026-10-09T06:12:16.808175+00:00

Daily

New Benchmarks (28)

PinchBench - 147 Tasks - OpenClaw 2026.5.4 (325df3e) (Best Official Run Score (%)): leader Claude Opus 4.8 Fast (94.4910231292517), 48 models
PinchBench v2 best official run per model within one exact task-set and OpenClaw runtime cohort. The primary card uses the 147-task May 4 runtime; 123-task, sanity-only and other runtime cohorts remain separate. Native run scores and source submission identities are preserved. This board reports PinchBench - 147 Tasks - OpenClaw 2026.5.4 (325df3e).
PinchBench - 147 Tasks - OpenClaw 2026.5.20 (e510042) (Best Official Run Score (%)): leader MiMo-V2.5 (91.87411836734695), 39 models
PinchBench v2 best official run per model within one exact task-set and OpenClaw runtime cohort. The primary card uses the 147-task May 4 runtime; 123-task, sanity-only and other runtime cohorts remain separate. Native run scores and source submission identities are preserved. This board reports PinchBench - 147 Tasks - OpenClaw 2026.5.20 (e510042).
CHI-Bench (Erius) (Pass@1 (%)): leader Claude Opus 5 (54.666666666667), 2 models
CHI-Bench (Erius) reported pass rate across 75 healthcare tasks in three domains on dataset v1.0.0, judged by Claude Opus 4.7. Framework and submitted model checkpoints are retained separately.
CHI-Bench (Claude Code) (Pass@1 (%)): leader Claude Opus 5 (37.333333333333), 8 models
CHI-Bench (Claude Code) reported pass rate across 75 healthcare tasks in three domains on dataset v1.0.0, judged by Claude Opus 4.7. Framework and submitted model checkpoints are retained separately.
CHI-Bench (Codex) (Pass@1 (%)): leader GPT-5.6 Sol (25.333333333333), 6 models
CHI-Bench (Codex) reported pass rate across 75 healthcare tasks in three domains on dataset v1.0.0, judged by Claude Opus 4.7. Framework and submitted model checkpoints are retained separately.
LLM Stats (FrontierSWE V2) (Score (%)): leader Claude Opus 5.5 (62.3), 5 models
AgenticVBench (Codex) (Average Normalized Score (%)): leader GPT-5.6 Sol (38.4), 3 models
AgenticVBench v1.0 published average normalized score with Codex over100 video-production tasks. Keep all model/harness submissions and native component provenance. Penalty-bearing Repurpose scores can be negative; the displayed aggregate is retained rather than recomputed from rounded components.
AgenticVBench (Claude Code) (Average Normalized Score (%)): leader Claude Fable 5 (32.4), 3 models
AgenticVBench v1.0 published average normalized score with Claude Code over100 video-production tasks. Keep all model/harness submissions and native component provenance. Penalty-bearing Repurpose scores can be negative; the displayed aggregate is retained rather than recomputed from rounded components.
Land or Water? (Celeste) (Area-weighted accuracy (%)): leader Claude Opus 5.5 (Low) (99.0), 12 models
Earth reconstruction from geographic knowledge: Celeste (@celestepoasts) asks twelve Claude models whether each of 16,200 latitude/longitude coordinates on a 2-degree grid is land or water, without images or tools. The score is area-weighted accuracy against a 1-km land mask, on a 0–100% scale; higher is better. These are the September 24, 2026 chart results. Fable 5, Fable 5.1 and Opus 5.5 use low-effort thinking; other run settings are unspecified. Exact prompts, the reference mask and per-point answers were not available for independent verification.
LOL Arena - Meme URL Only (Human judge-choice agreement (%), paired meme rounds with image URL only): leader Claude Opus 5.5 (50.6), 21 models
Select the funniest response in Cards Against Humanity-style rounds, scored by agreement with the human judge. Agreement with human judge on 1,910 paired meme rounds; image URL supplied as text, no image input.
LOL Arena - Meme Image (Human judge-choice agreement (%), paired meme rounds with image content): leader Claude Fable 5.1 (49.5), 21 models
Select the funniest response in Cards Against Humanity-style rounds, scored by agreement with the human judge. Agreement with human judge on same 1,910 paired meme rounds; actual image input.
ExploitGym (version unspecified; 6h timeout) (Successful intended exploits (#); version unspecified; 6h timeout): leader Claude Mythos 5 (247.0), 3 models
ExploitGym counts successful exploits that use the intended vulnerability across userspace, V8 and kernel tasks. This board reports Successful intended exploits (#); version unspecified; 6h timeout. Distinct versions, time budgets, throughput rescaling and selected subsets remain separate. The full v1 task set contains 869 vulnerabilities; source-version-unspecified or selected-subset runs are not assumed to use that denominator.
ExploitGym (version unspecified; 2h timeout cutoff) (Successful intended exploits (#); version unspecified; 2h timeout cutoff): leader Claude Mythos 5 (181.0), 3 models
ExploitGym counts successful exploits that use the intended vulnerability across userspace, V8 and kernel tasks. This board reports Successful intended exploits (#); version unspecified; 2h timeout cutoff. Distinct versions, time budgets, throughput rescaling and selected subsets remain separate. The full v1 task set contains 869 vulnerabilities; source-version-unspecified or selected-subset runs are not assumed to use that denominator.
ExploitBench v8-bench (AutoNudge) (Capability coverage (%), all-runs view with reported seed/turn budgets; AutoNudge): leader Claude Mythos Preview (78.0), 10 models
ExploitBench evaluates agents on patched V8 vulnerabilities. Capability coverage (%), all-runs view with reported seed/turn budgets; AutoNudge. This is capability-flag coverage, not mean capability count. Baseline and AutoNudge prompting are separate boards; each source run retains its scaffold and reported seed/turn budget. Scores use the published all-runs view.
BOTTLED - MAVE (F1 (0-1)): leader Claude Opus 5 (0.461), 10 models
BOTTLED measures agents building reusable programs or small models for whole unlabelled workloads under fixed time, compute and API budgets. Table 1 averages two runs on product-attribute extraction (MAVE F1), product-query relevance (ESCI macro F1), and machine-generated text detection (RAID AUROC). These are the quality scores of the agent-produced artifact; cost and normalized gain are separate quantities.
BOTTLED - ESCI (Macro F1 (0-1)): leader Claude Opus 5 (0.499), 10 models
BOTTLED measures agents building reusable programs or small models for whole unlabelled workloads under fixed time, compute and API budgets. Table 1 averages two runs on product-attribute extraction (MAVE F1), product-query relevance (ESCI macro F1), and machine-generated text detection (RAID AUROC). These are the quality scores of the agent-produced artifact; cost and normalized gain are separate quantities.
BOTTLED - RAID (AUROC (0-1)): leader Gemini 3.1 Pro (Preview) (0.822), 10 models
BOTTLED measures agents building reusable programs or small models for whole unlabelled workloads under fixed time, compute and API budgets. Table 1 averages two runs on product-attribute extraction (MAVE F1), product-query relevance (ESCI macro F1), and machine-generated text detection (RAID AUROC). These are the quality scores of the agent-produced artifact; cost and normalized gain are separate quantities.
WorldSolver (Overall score (0-100)): leader GPT-5.6 Sol (48.7), 7 models
WorldSolver evaluates agent-generated physical simulation solvers on 168 tasks spanning seven physics domains. The overall 0–100 score combines execution checks, VLM-assessed visual fidelity and task-specific physical plausibility. The paper uses provider-native coding harnesses, high effort when supported or thinking enabled, 1,800 seconds generation and 600 seconds execution per solver.
SciTrue - Claim Validation (PNG Only) (F1 (%)): leader GPT-5.6 Sol (84.5), 1 models
Honest per-sample development results on NTCIR SciClaimEval. Author runs include structured table data as an auxiliary text channel; official baseline runs and the PNG-only extension are kept on separate protocol boards. Claim validation is macro-F1, evidence pairing requires both paired claims correct, and attribution is Subtask 2 accuracy. This board is the later PNG-only GPT-5.6 Sol extension.
SciTrue - Evidence Pairing (PNG Only) (Accuracy (%)): leader GPT-5.6 Sol (70.2), 1 models
Honest per-sample development results on NTCIR SciClaimEval. Author runs include structured table data as an auxiliary text channel; official baseline runs and the PNG-only extension are kept on separate protocol boards. Claim validation is macro-F1, evidence pairing requires both paired claims correct, and attribution is Subtask 2 accuracy. This board is the later PNG-only GPT-5.6 Sol extension.
Humanity's Sixth Sense (Pass@1 (%), all rubric criteria; three attempts/task): leader GPT-6 Astra (Max) (53.6), 25 models
Scale AI's Humanity’s Sixth Sense evaluates multi-step agent tasks with verifiable all-criteria success. The published pass@1 estimates average three attempts per task, preserve model reasoning effort, and report percent success. Confidence bounds are not scores.
Vals AI SAFE-Teen - Urgent Safety (Failure rate (%), no teen-aware system prompt): leader GPT-6.1 Sol (3.0), 11 models
Vals AI SAFE-Teen measures failure rates on 101 clinician-designed adolescent mental-health scenarios in simulated six-to-twenty-turn conversations. This is the static no-teen-aware-prompt condition for Urgent Safety; lower failure percentages are better. Policy compliance and boundaries are not general capability.
Vals AI SAFE-Teen - Health Advice (Failure rate (%), no teen-aware system prompt): leader GPT-6.1 Sol (4.0), 11 models
Vals AI SAFE-Teen measures failure rates on 101 clinician-designed adolescent mental-health scenarios in simulated six-to-twenty-turn conversations. This is the static no-teen-aware-prompt condition for Health Advice; lower failure percentages are better. Policy compliance and boundaries are not general capability.
Vals AI SAFE-Teen - Relationship Boundaries (Failure rate (%), no teen-aware system prompt): leader GPT-6.1 Sol (10.0), 11 models
Vals AI SAFE-Teen measures failure rates on 101 clinician-designed adolescent mental-health scenarios in simulated six-to-twenty-turn conversations. This is the static no-teen-aware-prompt condition for Relationship Boundaries; lower failure percentages are better. Policy compliance and boundaries are not general capability.
SciExam-ENSO - Composite (Composite score (0–1), final submitted model, six-hour budget): leader Claude Fable 5.1 (0.5154889786306145), 13 models
SciExam-ENSO gives coding agents six hours to construct executable ENSO dynamical models, judged on statistical fit, physical mechanisms and predictive skill. This board uses the final submitted program's Composite score on a zero-to-one scale, not the best intermediate checkpoint. The composite weights statistics 0.3, physics 0.3 and predictivity 0.4. The five-dimensional reference is a fixed scientific model, not an LLM.
SciExam-ENSO - Statistics (Statistics score (0–1), final submitted model, six-hour budget): leader 5D reference model (0.7274632411531605), 13 models
SciExam-ENSO gives coding agents six hours to construct executable ENSO dynamical models, judged on statistical fit, physical mechanisms and predictive skill. This board uses the final submitted program's Statistics score on a zero-to-one scale, not the best intermediate checkpoint. The composite weights statistics 0.3, physics 0.3 and predictivity 0.4. The five-dimensional reference is a fixed scientific model, not an LLM.
SciExam-ENSO - Physics (Physics score (0–1), final submitted model, six-hour budget): leader Claude Fable 5.1 (0.6387), 13 models
SciExam-ENSO gives coding agents six hours to construct executable ENSO dynamical models, judged on statistical fit, physical mechanisms and predictive skill. This board uses the final submitted program's Physics score on a zero-to-one scale, not the best intermediate checkpoint. The composite weights statistics 0.3, physics 0.3 and predictivity 0.4. The five-dimensional reference is a fixed scientific model, not an LLM.
SciExam-ENSO - Predictivity (Predictivity score (0–1), final submitted model, six-hour budget): leader GPT-5.6 Sol (0.30631844773647077), 13 models
SciExam-ENSO gives coding agents six hours to construct executable ENSO dynamical models, judged on statistical fit, physical mechanisms and predictive skill. This board uses the final submitted program's Predictivity score on a zero-to-one scale, not the best intermediate checkpoint. The composite weights statistics 0.3, physics 0.3 and predictivity 0.4. The five-dimensional reference is a fixed scientific model, not an LLM.

Top-10 New Scores (142)

Claude Fable 5.1 on Agents on Rails: 92.06349206349206 (#2)
Claude Fable 5.1 on AutomationBench: 22.37 (#24)
Claude Fable 5.1 on CursorBench 4.0: 51.8 (#7)
Claude Fable 5.1 on FoodTruckBench: 62732.17 (#2)
Claude Fable 5.1 on RuneBench: 6.143277369387 (#11)
Claude Fable 5.1 on Terminal-Bench 4.0 (Claude Code): 57.88 (#3)
Claude Opus 5.5 on Chatbot Arena (Text - French): 1511.0 (#11)
Claude Opus 5.5 on Chatbot Arena (Text - Spanish): 1491.0 (#8)
Claude Opus 5.5 on CursorBench 4.0: 57.8 (#1)
Claude Opus 5.5 on Epoch AI - Frontiermath Erdos: 2.94 (#1)
Claude Opus 5.5 on F5 Labs AI Security Leaderboard - CASI: 93.06 (#8)
Claude Opus 5.5 on NYT Connections Extended: 87.8 (#32)
Claude Opus 5.5 on PUMA Polish Multimodal - Contemporary Life: 58.0 (#16)
Claude Opus 5.5 on PUMA Polish Multimodal - Geography and Environment: 82.0 (#8)
Claude Opus 5.5 on PUMA Polish Multimodal - History and Culture: 71.0 (#11)
Claude Opus 5.5 on PUMA Polish Multimodal - OCR: 90.0 (#3)
Claude Opus 5.5 on PUMA Polish Multimodal - Structured Extraction: 62.0 (#12)
Claude Opus 5.5 on PUMA Polish Multimodal - Vision Mean: 75.33 (#9)
Claude Opus 5.5 on RuneBench: 6.66164334639 (#6)
Claude Opus 5.5 on Terminal-Bench 4.0 (Claude Code): 64.85 (#1)
Claude Opus 5.5 on YC-Bench: 2873.72509 (#1)
Claude Sonnet 5.5 on Chatbot Arena (Text - Math): 1512.0 (#8)
Claude Sonnet 5.5 on Chatbot Arena (Text - Mathematical): 1527.0 (#4)
Claude Sonnet 5.5 on CursorBench 4.0: 55.5 (#4)
Claude Sonnet 5.5 on Epoch AI - Frontiermath Erdos: 2.9 (#3)
Claude Sonnet 5.5 on Maze-Bench: 58.47 (#5)
Claude Sonnet 5.5 on NYT Connections Extended: 80.5 (#44)
Claude Sonnet 5.5 on PUMA Polish Multimodal - Contemporary Life: 43.0 (#26)
Claude Sonnet 5.5 on PUMA Polish Multimodal - Document QA: 87.0 (#5)
Claude Sonnet 5.5 on PUMA Polish Multimodal - Geography and Environment: 67.0 (#18)
Claude Sonnet 5.5 on PUMA Polish Multimodal - History and Culture: 61.0 (#18)
Claude Sonnet 5.5 on PUMA Polish Multimodal - OCR: 85.0 (#10)
Claude Sonnet 5.5 on PUMA Polish Multimodal - Structured Extraction: 65.0 (#11)
Claude Sonnet 5.5 on PUMA Polish Multimodal - Vision Mean: 68.0 (#15)
Claude Sonnet 5.5 on RuneBench: 5.524272394171 (#31)
Claude Sonnet 5.5 on Terminal-Bench 4.0 (Claude Code): 61.82 (#2)
Claude Sonnet 5.5 on YC-Bench: 1741.3148899999999 (#8)
GPT-6 Astra on AutomationBench: 41.4 (#3)
GPT-6 Astra on Chatbot Arena (Text - German): 1473.0 (#39)
GPT-6 Astra on Chatbot Arena (Text - Korean): 1449.0 (#21)
GPT-6 Astra on DeepSWE: 73.2 (#4)
GPT-6 Astra on DeepSWE-mini (16 Tasks): 79.69 (#2)
GPT-6 Astra on F5 Labs AI Security Leaderboard - ARS: 85.21 (#8)
GPT-6 Astra on FoodTruckBench: 57760.9 (#5)
GPT-6 Astra on Harvey LAB-AA: 8.61 (#3)
GPT-6 Astra on Harvey LAB-AA - All Pass: 8.89 (#11)
GPT-6 Astra on Harvey LAB-AA - Antitrust & Competition: 87.32 (#14)
GPT-6 Astra on Harvey LAB-AA - Arbitration & International Dispute Resolution: 88.53 (#15)
GPT-6 Astra on Harvey LAB-AA - Banking & Finance: 94.15 (#7)
GPT-6 Astra on Harvey LAB-AA - Bankruptcy & Restructuring: 89.16 (#9)
GPT-6 Astra on Harvey LAB-AA - Capital Markets: 95.95 (#3)
GPT-6 Astra on Harvey LAB-AA - Corporate Governance: 89.04 (#10)
GPT-6 Astra on Harvey LAB-AA - Corporate M&A: 94.75 (#7)
GPT-6 Astra on Harvey LAB-AA - Data Privacy & Cybersecurity: 95.29 (#8)
GPT-6 Astra on Harvey LAB-AA - Emerging Companies & Venture Capital: 90.1 (#14)
GPT-6 Astra on Harvey LAB-AA - Employment & Labor: 88.26 (#4)
GPT-6 Astra on Harvey LAB-AA - Energy & Natural Resources: 94.72 (#10)
GPT-6 Astra on Harvey LAB-AA - Environmental & ESG: 86.86 (#12)
GPT-6 Astra on Harvey LAB-AA - Funds & Asset Management: 87.46 (#14)
GPT-6 Astra on Harvey LAB-AA - Healthcare & Life Sciences: 88.48 (#18)
GPT-6 Astra on Harvey LAB-AA - Immigration: 92.85 (#4)
GPT-6 Astra on Harvey LAB-AA - Insurance: 92.89 (#5)
GPT-6 Astra on Harvey LAB-AA - Intellectual Property: 93.36 (#17)
GPT-6 Astra on Harvey LAB-AA - International Trade & Sanctions: 89.77 (#12)
GPT-6 Astra on Harvey LAB-AA - Litigation & Dispute Resolution: 91.35 (#10)
GPT-6 Astra on Harvey LAB-AA - Real Estate: 94.56 (#7)
GPT-6 Astra on Harvey LAB-AA - Structured Finance & Securitization: 89.89 (#8)
GPT-6 Astra on Harvey LAB-AA - Tax: 82.39 (#15)
GPT-6 Astra on Harvey LAB-AA - Trusts, Estates & Private Client: 91.48 (#12)
GPT-6 Astra on Harvey LAB-AA - White Collar Defense & Investigations: 87.09 (#12)
GPT-6 Astra on LLMConfBench: 0.67 (#1)
GPT-6 Astra on PUMA Polish Multimodal - Contemporary Life: 73.0 (#9)
GPT-6 Astra on PUMA Polish Multimodal - Document QA: 85.0 (#9)
GPT-6 Astra on PUMA Polish Multimodal - Geography and Environment: 85.0 (#7)
GPT-6 Astra on PUMA Polish Multimodal - History and Culture: 75.0 (#8)
GPT-6 Astra on PUMA Polish Multimodal - OCR: 88.0 (#6)
GPT-6 Astra on PUMA Polish Multimodal - Structured Extraction: 58.0 (#17)
GPT-6 Astra on PUMA Polish Multimodal - Vision Mean: 77.33 (#7)
GPT-6 Astra on RuneBench: 7.258225267108 (#1)
GPT-6 Astra on Thinkingbox: 58.31 (#6)
GPT-6 Astra on Wolfram LLM Benchmarking Project: 69.3 (#28)
GPT-6 Astra on YC-Bench: 1336.46611 (#19)
GPT-6.1 Sol on BenchGecko Score: 93.4 (#6)
GPT-6.1 Sol on Chatbot Arena (Text - Math): 1488.0 (#29)
GPT-6.1 Sol on Chatbot Arena (Text - Mathematical): 1539.0 (#2)
GPT-6.1 Sol on Design Arena (Game Dev): 1399.0 (#5)
GPT-6.1 Sol on Harvey LAB-AA: 6.94 (#4)
GPT-6.1 Sol on Harvey LAB-AA - All Pass: 7.5 (#12)
GPT-6.1 Sol on Harvey LAB-AA - Antitrust & Competition: 88.77 (#11)
GPT-6.1 Sol on Harvey LAB-AA - Arbitration & International Dispute Resolution: 90.8 (#10)
GPT-6.1 Sol on Harvey LAB-AA - Banking & Finance: 93.91 (#8)
GPT-6.1 Sol on Harvey LAB-AA - Bankruptcy & Restructuring: 86.27 (#16)
GPT-6.1 Sol on Harvey LAB-AA - Capital Markets: 96.47 (#2)
GPT-6.1 Sol on Harvey LAB-AA - Corporate Governance: 86.76 (#14)
GPT-6.1 Sol on Harvey LAB-AA - Corporate M&A: 93.57 (#12)
GPT-6.1 Sol on Harvey LAB-AA - Data Privacy & Cybersecurity: 94.94 (#10)
GPT-6.1 Sol on Harvey LAB-AA - Emerging Companies & Venture Capital: 89.99 (#16)
GPT-6.1 Sol on Harvey LAB-AA - Employment & Labor: 85.39 (#12)
GPT-6.1 Sol on Harvey LAB-AA - Energy & Natural Resources: 95.27 (#8)
GPT-6.1 Sol on Harvey LAB-AA - Environmental & ESG: 87.69 (#11)
GPT-6.1 Sol on Harvey LAB-AA - Funds & Asset Management: 88.34 (#9)
GPT-6.1 Sol on Harvey LAB-AA - Healthcare & Life Sciences: 90.49 (#12)
GPT-6.1 Sol on Harvey LAB-AA - Immigration: 91.39 (#11)
GPT-6.1 Sol on Harvey LAB-AA - Insurance: 91.42 (#10)
GPT-6.1 Sol on Harvey LAB-AA - Intellectual Property: 95.26 (#6)
GPT-6.1 Sol on Harvey LAB-AA - International Trade & Sanctions: 89.51 (#13)
GPT-6.1 Sol on Harvey LAB-AA - Litigation & Dispute Resolution: 89.35 (#18)
GPT-6.1 Sol on Harvey LAB-AA - Real Estate: 94.27 (#8)
GPT-6.1 Sol on Harvey LAB-AA - Structured Finance & Securitization: 89.68 (#10)
GPT-6.1 Sol on Harvey LAB-AA - Tax: 83.28 (#11)
GPT-6.1 Sol on Harvey LAB-AA - Trusts, Estates & Private Client: 91.48 (#13)
GPT-6.1 Sol on Harvey LAB-AA - White Collar Defense & Investigations: 89.79 (#9)
GPT-6.1 Sol on HieroglyphBench: 31.4 (#8)
GPT-6.1 Sol on NYT Connections Extended: 95.5 (#6)
GPT-6.1 Sol on PUMA Polish Multimodal - Contemporary Life: 71.0 (#11)
GPT-6.1 Sol on PUMA Polish Multimodal - Document QA: 85.0 (#8)
GPT-6.1 Sol on PUMA Polish Multimodal - Geography and Environment: 81.0 (#9)
GPT-6.1 Sol on PUMA Polish Multimodal - History and Culture: 78.0 (#6)
GPT-6.1 Sol on PUMA Polish Multimodal - OCR: 87.0 (#7)
GPT-6.1 Sol on PUMA Polish Multimodal - Structured Extraction: 56.0 (#18)
GPT-6.1 Sol on PUMA Polish Multimodal - Vision Mean: 76.33 (#8)
GPT-6.1 Sol on ParseBench: 70.48 (#25)
GPT-6.1 Sol on ParseBench - Charts: 37.16 (#54)
GPT-6.1 Sol on ParseBench - Content Faithfulness: 89.25 (#22)
GPT-6.1 Sol on ParseBench - Semantic Formatting: 74.78 (#11)
GPT-6.1 Sol on ParseBench - Tables: 92.44 (#6)
GPT-6.1 Sol on ParseBench - Visual Grounding: 58.76 (#46)
GPT-6.1 Sol on RuneBench: 6.875920012269 (#3)
GPT-6.1 Sol on Sonar LLM Leaderboard - Java - Bug Density: 0.61 (#10)
GPT-6.1 Sol on Sonar LLM Leaderboard - Java - Code Smell Density: 18.6 (#38)
GPT-6.1 Sol on Sonar LLM Leaderboard - Java - Pass Rate: 85.29 (#5)
GPT-6.1 Sol on Sonar LLM Leaderboard - Java - Vulnerability Density: 0.19 (#17)
GPT-6.1 Sol on YC-Bench: 936.2673199999999 (#32)
Gemini 4 Argon on ComplexConstraints: 53.3 (#2)
Gemini 4 Argon on GDP.pdf: 28.4 (#5)
Gemini 4 Argon on LLM Stats (Agents' Last Exam): 39.5 (#12)
Gemini 4 Argon on LLM Stats (AutomationBench): 51.3 (#5)
Gemini 4 Argon on LLM Stats (Graphwalks BFS >128k): 84.2 (#2)
Gemini 4 Argon on LLM Stats (OSWorld 2.0): 69.2 (#6)
Gemini 4 Argon on LLM Stats (Terminal-Bench 4.0): 57.4 (#5)
Gemini 4 Argon on LLM Stats (Terminal-Bench-Science 0.1): 57.6 (#4)
Gemini 4 Argon on LLM Stats Score: 55.09 (#4)

New #1 Leaders (24)

Agents on Rails: Claude Opus 5 (High) (92.06349206349206) beat GPT-6 Astra (Max) by 38.76
OpenVLM Video - MMBench-Video: GPT-4o (2024-08-06) (Frames: 1 fps) (71.66666666666667) beat GPT-4o (2024-08-06) by 28.67
Wolfram LLM Benchmarking Project: Claude Opus 5.5 thinking max (84.5) beat Claude Opus 5 thinking max by 10.9
LLM Stats (Graphwalks BFS <128k): Gemini 4 Argon (99.7) beat GPT-5.2 by 5.7
SAHARA - Machine Translation (African to African): Sunflower-Qwen3.8-27B (17.81) beat Gemini 3 Pro (Preview) by 5.58
LLM Stats (PostTrainBench): Gemini 4 Argon (45.3) beat GLM-5.3 by 5.5
LLM Stats (LVBench): Gemini 4 Argon (91.7) beat Gemini 3.8 Flash by 4.6
Harvey LAB-AA - Structured Finance & Securitization: Muse Spark 1.3 (Max) (94.84) beat Muse Spark 1.3 (xHigh) by 2.78
LLM Stats (DeepSWE 1.1): Gemini 4 Argon (77.9) beat Muse Spark 1.3 by 2.5
PUMA Polish Multimodal - Geography and Environment: Gemini 3.8 Flash (91.0) beat Gemini 3.7 Flash by 2.0
Harvey LAB-AA - Employment & Labor: Muse Spark 1.3 (Max) (93.63) beat Muse Spark 1.3 (xHigh) by 1.5
Harvey LAB-AA - Corporate Governance: Muse Spark 1.3 (Max) (97.37) beat Muse Spark 1.3 (xHigh) by 1.48
PUMA Polish Multimodal - Contemporary Life: Gemini 3.8 Flash (84.0) beat Gemini 3.1 Pro (Preview) by 1.0
PUMA Polish Multimodal - Document QA: Claude Opus 5.5 (89.0) beat Gemini 3.5 Flash by 1.0
Harvey LAB-AA - Litigation & Dispute Resolution: Muse Spark 1.3 (Max) (97.37) beat Qwen 3.8 Max (0902) by 0.74
Harvey LAB-AA - Trusts, Estates & Private Client: Muse Spark 1.3 (Max) (96.44) beat Claude Sonnet 5.5 (Xhigh, Default Fallback) by 0.64
Harvey LAB-AA - Intellectual Property: Muse Spark 1.3 (Max) (97.75) beat Claude Fable 5.1 (High, Default Fallback) by 0.58
Harvey LAB-AA - Real Estate: Muse Spark 1.3 (Max) (97.57) beat Muse Spark 1.3 (xHigh) by 0.49
Harvey LAB-AA - Capital Markets: Muse Spark 1.3 (Max) (97.65) beat Claude Fable 5.1 (Xhigh, Default Fallback) by 0.4
Open Medical LLM - MedQA (USMLE): medllama3-v20 (Medical precision bfloat16) (81.38256087981148) beat medllama3-v20 by 0.31
Open Medical LLM - PubMedQA: Med-PaLM 2 (5 Shots) (79.2) beat Flan-PaLM by 0.2
Senior SWE-Bench: GPT-5.6 Sol (Mini-SWE-Agent, xHigh) (34.73684210526316) beat Claude Fable 5.1 by 0.04
EdiTikZ: EdiTikZ-9B-RL (0.726) beat GPT-5.5 by 0.04
PACT - Rule-Scope Discernment: Inkling (0.909) beat Nemotron 3 Ultra by 0.02

The Aggregate Digest — 2026-10-08

2026-10-08T06:17:15.178829+00:00

Daily

New Benchmarks (7)

LLM Stats (AA-Briefcase v1.1) (Score): leader Claude Opus 5.5 (1822.0), 5 models
Artificial Analysis Briefcase v1.1 evaluates agents on multi-hour professional office work over real files. LLM Stats mirrors the reported quality Elo, higher is better; this version and scale stay separate from unversioned rubric pass rates.
LLM Stats (Chartography) (Score (%)): leader Claude Sonnet 5.5 (90.2), 5 models
Chartography evaluates professional chart understanding on 100 expert-written tasks across scientific, engineering, financial and medical charts. LLM Stats mirrors reported scores, shown here as percentages; tool configuration is not specified by this parent board.
LLM Stats (FrontierMath Tier 4 (v2)) (Score (%)): leader GPT-6.1 Sol (100.0), 5 models
FrontierMath Tier 4 v2 evaluates research-level mathematical problem solving at the frontier of active mathematics. LLM Stats mirrors the v2 subset results as percentages; the version is kept distinct from earlier Tier 4 releases.
LLM Stats (GDPval-AA 2.1) (Score): leader Claude Opus 5.5 (1846.0), 6 models
Artificial Analysis GDPval-AA 2.1 evaluates agentic professional knowledge work across occupations. LLM Stats mirrors the reported Elo rating, higher is better, separately from earlier versions and percentage-valued GDPval results.
LLM Stats (LifeSciBench) (Score (%)): leader GPT-6 Astra (60.3), 5 models
LifeSciBench evaluates 750 open-ended life-science research tasks across seven workflows and biological domains. Expert rubric grading covers scientific reasoning and attached figures, PDFs and sequence files. LLM Stats mirrors reported performance as a percentage.
LLM Stats (SimpleQA Verified) (Score (%)): leader GPT-6.1 Sol (73.9), 5 models
SimpleQA Verified evaluates short-form factual knowledge with 1,000 curated questions with verifiable answers, correcting label noise, redundancy and topical imbalance in SimpleQA. LLM Stats mirrors reported performance as percentages; this is distinct from original SimpleQA.
ChartBmkAgent (Accuracy (%), 300 generated chart questions): leader GPT-5.6 Sol (84.3), 17 models
ChartBmkAgent tests chart reading and reasoning on 300 synthesized chart-question pairs targeting 60 pre-specified failure descriptions across four domains. Accuracy is reference-answer correctness; provider-default thinking is retained except for eight documented timeout/revision cases. This is a targeted generated stress set, not a population sample of chart tasks.

New Models (1)

Claude Haiku 5.5 — Aggregate Rating 1845, #30
- LLM Stats (HealthBench): 61.6 (#2/14)
- LLM Stats (Program Bench): 82.0 (#2/13)
- LiveBench Math Comp: 98.04 (#3/69)
- LLM Stats (BenchCAD): 67.0 (#4/7)
- LLM Stats (BenchCAD (with Python tool)): 87.0 (#4/7)
- LLM Stats (HealthBench Professional): 64.8 (#5/16)
- LLM Stats (Humanity's Last Exam (no tools, text-only)): 45.9 (#5/12)
- LLM Stats (SWE-bench Multilingual): 83.7 (#5/48)
- LLM Stats (SWE-Bench Multimodal): 30.7 (#6/7)
- LLM Stats (Terminal-Bench-Science 0.1): 20.6 (#6/6)
- ...and 120 more

Top-10 New Scores (30)

Claude Fable 5.1 on Clerk LLM Leaderboard: 91.7 (#2)
Claude Opus 5.5 on Clerk LLM Leaderboard: 91.1 (#5)
Claude Opus 5.5 on WebDev Arena (HTML): 1758.0 (#3)
Claude Sonnet 5.5 on Clerk LLM Leaderboard: 91.7 (#3)
Claude Sonnet 5.5 on DRACO: 87.0 (#2)
Claude Sonnet 5.5 on Design Arena (Game Dev): 1428.0 (#4)
Claude Sonnet 5.5 on DuelLab Overall: 83.6 (#5)
Claude Sonnet 5.5 on WebDev Arena (Fullstack): 1699.0 (#4)
GPT-6 Astra on Clerk LLM Leaderboard: 90.5 (#8)
GPT-6 Astra on Mercor APEX: 64.7 (#8)
GPT-6 Astra on WDCD: 79.6 (#12)
GPT-6 Astra on WDCD R1 Understanding: 100.0 (#9)
GPT-6 Astra on WDCD R2 In-Document Resistance: 88.0 (#3)
GPT-6 Astra on WDCD R3 Pressure Integrity: 113.0 (#5)
GPT-6 Astra on YC-Bench: 1336.5 (#11)
GPT-6.1 Sol on Clerk LLM Leaderboard: 90.0 (#12)
GPT-6.1 Sol on Design Arena (Website): 1314.0 (#14)
GPT-6.1 Sol on DuelLab Overall: 87.7 (#4)
GPT-6.1 Sol on KernelBench Hub - CUDA: 30.52 (#10)
GPT-6.1 Sol on KernelBench Hub - Mega: 4.99 (#11)
GPT-6.1 Sol on LLM Stats (FrontierCode 1.1): 50.2 (#6)
GPT-6.1 Sol on LLM Stats (Humanity's Last Exam (no tools, text-only)): 52.9 (#3)
GPT-6.1 Sol on Mercor APEX: 60.0 (#12)
GPT-6.1 Sol on SEAL - Professional Reasoning Benchmark - Finance: 46.21 (#17)
GPT-6.1 Sol on SEAL - Professional Reasoning Benchmark - Legal: 47.87 (#15)
GPT-6.1 Sol on WDCD: 81.9 (#9)
GPT-6.1 Sol on WDCD R1 Understanding: 100.0 (#10)
GPT-6.1 Sol on WDCD R2 In-Document Resistance: 75.0 (#9)
GPT-6.1 Sol on WDCD R3 Pressure Integrity: 100.0 (#9)
Gemini 4 Argon on Vals AI Time Horizon Index: 52.5 (#4)

New #1 Leaders (16)

YC-Bench: Grok 4.5 (2629.9) beat Claude Fable 5 by 652.3
LLM Stats (WildClawBench): Qwen3.8 Omni Flash (71.0) beat Seed 2.1 Turbo by 8.2
Mercor APEX: Gemini 4 Argon (82.4) beat Claude Sonnet 5.5 by 6.9
SEAL - MultiChallenge: GPT-6.1 Sol (81.95) beat Muse Spark by 6.43
VANTAGE-Bench: GPT-6 Astra (75.26) beat Gemini 3.6 Flash by 5.74
VANTAGE-Bench - Semantic: GPT-6 Astra (85.08) beat Gemini 3.6 Flash by 5.67
VANTAGE-Bench - Single Object Tracking: GPT-6 Astra (81.2) beat Gemini 3.6 Flash by 5.21
VANTAGE-Bench - Video QA: GPT-6 Astra (83.26) beat GPT-5.6 Sol by 5.18
VANTAGE-Bench - Temporal: GPT-6 Astra (51.64) beat GPT-5.6 Sol by 5.14
LLM Stats (AutomationBench): Mistral Large 4 (59.9) beat DeepSeek V4.1 Flash by 5.1
VANTAGE-Bench - Event Verification: GPT-6 Astra (86.9) beat Gemini 3.6 Flash by 4.9
VANTAGE-Bench - Spatial: GPT-6 Astra (83.12) beat Qwen 3.5 27B by 4.08
LLM Stats (APEX-Agents): GPT-6.1 Sol (60.0) beat Grok 4.6 by 2.5
Clerk LLM Leaderboard: DeepSeek V4.1 Flash (92.0) beat Claude Opus 4.8 by 0.7
SEAL - Remote Labor Index (RLI): Claude Opus 5.5 (21.25) beat GPT-6 Astra by 0.42
OSWorld 2.0 Partial: Sai w/ Claude Opus 5 (79.38) beat Claude Opus 5 by 0.18

The Aggregate Digest — 2026-10-07

2026-10-07T12:56:41.643997+00:00

Daily

New Benchmarks (17)

Knossos (Edge F1) (Edge F1 (%), macro-averaged over six diagram domains (1,200 test diagrams), direct full-graph generation): leader GPT-5.6 Sol (81.1), 10 models
Knossos (2026) evaluates complete diagram-to-graph topology extraction: from a rendered diagram, the model must output every entity and every relation between them. Its 1,200-diagram test set spans six symbolically generated domains (food web, network, workflow, natural process, circuit, map route) with exact ground-truth topology. The board is the Knossos Avg. column of the paper's Table 1: Edge F1 (%) macro-averaged over the six domains, with each VLM baseline prompted for direct full-graph generation in one response. Node extraction is near ceiling for current models (97%+ Node F1 for the strongest), so Edge F1 carries the signal.
PureDocBench (Three-track average (0-100): mean of the clean, digitally degraded and real degraded track Overall scores, each the mean of text accuracy (100 times one minus the text normalized edit distance), formula CDM and table TEDS, on 1,475 source pages in 10 domains and 66 subcategories, rendered from LLM-written HTML/CSS with source-linked annotations for text, formulas, tables and reading order; higher is better): leader GLM-5.3 Flash (79.69), 58 models
PureDocBench (2026) scores document parsers and vision-language models on 1,475 pages in 10 domains rendered from freshly generated HTML/CSS, whose annotations come from the same source, presented clean, digitally degraded and physically recaptured. The score is the mean of the three track Overall scores, each averaging text accuracy, formula CDM and table TEDS on a 0 to 100 scale; higher is better.
PureDocBench - Clean (Overall parsing score (0-100): mean of text accuracy (100 times one minus the text normalized edit distance), formula CDM and table TEDS, on the clean track (pages rendered straight from their HTML/CSS source) of 1,475 source pages in 10 domains and 66 subcategories, rendered from LLM-written HTML/CSS with source-linked annotations for text, formulas, tables and reading order; higher is better): leader TeleOCR (87.16), 58 models
The clean track of PureDocBench (2026), a source-traceable document-parsing benchmark of 1,475 pages: pages rendered straight from their HTML/CSS source. The Overall score averages text accuracy, formula CDM and table TEDS on a 0 to 100 scale; higher is better.
PureDocBench - Clean - Text Edit Distance (Text normalized edit distance (%): edit distance between the predicted and reference text divided by the longer length, times 100, on the clean track (pages rendered straight from their HTML/CSS source) of 1,475 source pages in 10 domains and 66 subcategories, rendered from LLM-written HTML/CSS with source-linked annotations for text, formulas, tables and reading order; lower is better): leader TeleOCR (12.0), 58 models
Text normalized edit distance on the clean track of PureDocBench (2026), a document-parsing benchmark of 1,475 source-traceable pages (pages rendered straight from their HTML/CSS source). Percent; lower is better.
PureDocBench - Clean - Formula CDM (Formula Character Detection Matching score (0-100) between the rendered predicted and reference LaTeX formulas, on the clean track (pages rendered straight from their HTML/CSS source) of 1,475 source pages in 10 domains and 66 subcategories, rendered from LLM-written HTML/CSS with source-linked annotations for text, formulas, tables and reading order; higher is better): leader TeleOCR (82.33), 58 models
Formula recognition on the clean track of PureDocBench (2026), a document-parsing benchmark of 1,475 source-traceable pages (pages rendered straight from their HTML/CSS source), scored by Character Detection Matching of rendered formulas from 0 to 100. Higher is better.
PureDocBench - Clean - Table TEDS (Table Tree-Edit-Distance-based Similarity (0-100) between the predicted and reference HTML tables, on the clean track (pages rendered straight from their HTML/CSS source) of 1,475 source pages in 10 domains and 66 subcategories, rendered from LLM-written HTML/CSS with source-linked annotations for text, formulas, tables and reading order; higher is better): leader Claude Opus 5 (92.17), 58 models
Table recognition on the clean track of PureDocBench (2026), a document-parsing benchmark of 1,475 source-traceable pages (pages rendered straight from their HTML/CSS source), scored by tree-edit-distance similarity (TEDS) from 0 to 100. Higher is better.
PureDocBench - Clean - Reading Order Edit Distance (Reading-order normalized edit distance (%) between the predicted and reference block sequences, times 100, on the clean track (pages rendered straight from their HTML/CSS source) of 1,475 source pages in 10 domains and 66 subcategories, rendered from LLM-written HTML/CSS with source-linked annotations for text, formulas, tables and reading order; lower is better): leader TeleOCR (15.7), 58 models
Reading-order normalized edit distance on the clean track of PureDocBench (2026), a document-parsing benchmark of 1,475 source-traceable pages (pages rendered straight from their HTML/CSS source). Percent; lower is better.
PureDocBench - Digital Degraded (Overall parsing score (0-100): mean of text accuracy (100 times one minus the text normalized edit distance), formula CDM and table TEDS, on the digitally degraded track (the same pages under ten simulated scene templates such as aging, binding, photocopying, compression, lighting and geometric artifacts) of 1,475 source pages in 10 domains and 66 subcategories, rendered from LLM-written HTML/CSS with source-linked annotations for text, formulas, tables and reading order; higher is better): leader GLM-5.3 Flash (81.03), 58 models
The digital degraded track of PureDocBench (2026), a source-traceable document-parsing benchmark of 1,475 pages: pages under simulated aging, binding, photocopy, compression and lighting degradations. The Overall score averages text accuracy, formula CDM and table TEDS on a 0 to 100 scale; higher is better.
PureDocBench - Digital Degraded - Text Edit Distance (Text normalized edit distance (%): edit distance between the predicted and reference text divided by the longer length, times 100, on the digitally degraded track (the same pages under ten simulated scene templates such as aging, binding, photocopying, compression, lighting and geometric artifacts) of 1,475 source pages in 10 domains and 66 subcategories, rendered from LLM-written HTML/CSS with source-linked annotations for text, formulas, tables and reading order; lower is better): leader OvisOCR2 (19.6), 58 models
Text normalized edit distance on the digital degraded track of PureDocBench (2026), a document-parsing benchmark of 1,475 source-traceable pages (pages under simulated aging, binding, photocopy, compression and lighting degradations). Percent; lower is better.
PureDocBench - Digital Degraded - Formula CDM (Formula Character Detection Matching score (0-100) between the rendered predicted and reference LaTeX formulas, on the digitally degraded track (the same pages under ten simulated scene templates such as aging, binding, photocopying, compression, lighting and geometric artifacts) of 1,475 source pages in 10 domains and 66 subcategories, rendered from LLM-written HTML/CSS with source-linked annotations for text, formulas, tables and reading order; higher is better): leader Gemini 3.6 Flash (78.88), 58 models
Formula recognition on the digital degraded track of PureDocBench (2026), a document-parsing benchmark of 1,475 source-traceable pages (pages under simulated aging, binding, photocopy, compression and lighting degradations), scored by Character Detection Matching of rendered formulas from 0 to 100. Higher is better.
PureDocBench - Digital Degraded - Table TEDS (Table Tree-Edit-Distance-based Similarity (0-100) between the predicted and reference HTML tables, on the digitally degraded track (the same pages under ten simulated scene templates such as aging, binding, photocopying, compression, lighting and geometric artifacts) of 1,475 source pages in 10 domains and 66 subcategories, rendered from LLM-written HTML/CSS with source-linked annotations for text, formulas, tables and reading order; higher is better): leader GLM-5.3 Flash (88.84), 58 models
Table recognition on the digital degraded track of PureDocBench (2026), a document-parsing benchmark of 1,475 source-traceable pages (pages under simulated aging, binding, photocopy, compression and lighting degradations), scored by tree-edit-distance similarity (TEDS) from 0 to 100. Higher is better.
PureDocBench - Digital Degraded - Reading Order Edit Distance (Reading-order normalized edit distance (%) between the predicted and reference block sequences, times 100, on the digitally degraded track (the same pages under ten simulated scene templates such as aging, binding, photocopying, compression, lighting and geometric artifacts) of 1,475 source pages in 10 domains and 66 subcategories, rendered from LLM-written HTML/CSS with source-linked annotations for text, formulas, tables and reading order; lower is better): leader TeleOCR (23.7), 58 models
Reading-order normalized edit distance on the digital degraded track of PureDocBench (2026), a document-parsing benchmark of 1,475 source-traceable pages (pages under simulated aging, binding, photocopy, compression and lighting degradations). Percent; lower is better.
PureDocBench - Real Degraded (Overall parsing score (0-100): mean of text accuracy (100 times one minus the text normalized edit distance), formula CDM and table TEDS, on the real degraded track (the same pages through four physical acquisition chains: phone photos of prints and of photocopies, screen photographs, and screenshots with social-media compression) of 1,475 source pages in 10 domains and 66 subcategories, rendered from LLM-written HTML/CSS with source-linked annotations for text, formulas, tables and reading order; higher is better): leader Gemini 3.6 Flash (75.73), 58 models
The real degraded track of PureDocBench (2026), a source-traceable document-parsing benchmark of 1,475 pages: pages captured by phone from prints, photocopies and screens, or as compressed screenshots. The Overall score averages text accuracy, formula CDM and table TEDS on a 0 to 100 scale; higher is better.
PureDocBench - Real Degraded - Text Edit Distance (Text normalized edit distance (%): edit distance between the predicted and reference text divided by the longer length, times 100, on the real degraded track (the same pages through four physical acquisition chains: phone photos of prints and of photocopies, screen photographs, and screenshots with social-media compression) of 1,475 source pages in 10 domains and 66 subcategories, rendered from LLM-written HTML/CSS with source-linked annotations for text, formulas, tables and reading order; lower is better): leader Qwen 3.8 27B (27.4), 58 models
Text normalized edit distance on the real degraded track of PureDocBench (2026), a document-parsing benchmark of 1,475 source-traceable pages (pages captured by phone from prints, photocopies and screens, or as compressed screenshots). Percent; lower is better.
PureDocBench - Real Degraded - Formula CDM (Formula Character Detection Matching score (0-100) between the rendered predicted and reference LaTeX formulas, on the real degraded track (the same pages through four physical acquisition chains: phone photos of prints and of photocopies, screen photographs, and screenshots with social-media compression) of 1,475 source pages in 10 domains and 66 subcategories, rendered from LLM-written HTML/CSS with source-linked annotations for text, formulas, tables and reading order; higher is better): leader Gemini 3.6 Flash (76.27), 58 models
Formula recognition on the real degraded track of PureDocBench (2026), a document-parsing benchmark of 1,475 source-traceable pages (pages captured by phone from prints, photocopies and screens, or as compressed screenshots), scored by Character Detection Matching of rendered formulas from 0 to 100. Higher is better.
PureDocBench - Real Degraded - Table TEDS (Table Tree-Edit-Distance-based Similarity (0-100) between the predicted and reference HTML tables, on the real degraded track (the same pages through four physical acquisition chains: phone photos of prints and of photocopies, screen photographs, and screenshots with social-media compression) of 1,475 source pages in 10 domains and 66 subcategories, rendered from LLM-written HTML/CSS with source-linked annotations for text, formulas, tables and reading order; higher is better): leader Claude Opus 5 (84.01), 58 models
Table recognition on the real degraded track of PureDocBench (2026), a document-parsing benchmark of 1,475 source-traceable pages (pages captured by phone from prints, photocopies and screens, or as compressed screenshots), scored by tree-edit-distance similarity (TEDS) from 0 to 100. Higher is better.
PureDocBench - Real Degraded - Reading Order Edit Distance (Reading-order normalized edit distance (%) between the predicted and reference block sequences, times 100, on the real degraded track (the same pages through four physical acquisition chains: phone photos of prints and of photocopies, screen photographs, and screenshots with social-media compression) of 1,475 source pages in 10 domains and 66 subcategories, rendered from LLM-written HTML/CSS with source-linked annotations for text, formulas, tables and reading order; lower is better): leader TeleOCR (35.2), 58 models
Reading-order normalized edit distance on the real degraded track of PureDocBench (2026), a document-parsing benchmark of 1,475 source-traceable pages (pages captured by phone from prints, photocopies and screens, or as compressed screenshots). Percent; lower is better.

New Models (1)

Mistral Large 4 — Aggregate Rating 1744, #121
- Vals AI Harvey Legal Agent Bench: 15.83 (#6/75)
- Vals AI Vibe Code Bench 1-100: 14.26 (#16/21)
- Vals AI BioMysteryBench: 67.04 (#17/24)
- Vals AI Terminal-Bench 4.0: 22.73 (#20/44)
- Vals AI Finance Agent v2: 54.68 (#22/75)
- Vals AI MysteryMechanism: 12.16 (#24/24)
- Vals AI Tax Agent Bench: 63.29 (#25/66)
- Vals AI Vibe Code Bench: 78.4 (#25/109)
- Vals AI IOI: 45.28 (#28/41)
- LLM2014 Logic 2026-10: 8.84 (#31/40)
- ...and 17 more

Top-10 New Scores (7)

Claude Opus 5.5 on Vals AI ReverseEngBench: 33.59 (#4)
GPT-6.1 Sol on Kaggle Open Benchmarks - MATH-500: 99.01 (#4)
GPT-6.1 Sol on Kaggle Open Benchmarks - MMLU: 93.83 (#9)
GPT-6.1 Sol on Kaggle Open Benchmarks - MMLU-Pro: 89.44 (#8)
GPT-6.1 Sol on Kaggle SimpleQA: 71.08 (#3)
GPT-6.1 Sol on Kaggle SimpleQA Verified: 73.77 (#4)
GPT-6.1 Sol on Terminal-Bench 4.0: 58.2 (#4)

New #1 Leaders (3)

Terminal-Bench 4.0 (Claude Code): Claude Opus 5.5 (64.8) beat Claude Fable 5.1 by 6.9
Terminal-Bench 4.0: Claude Opus 5.5 (64.8) beat GPT-6 Astra by 6.6
Arcee MMLU-Pro (BenchLM): Qwen 3.7 Max (89.6) beat Claude Opus 4.6 by 0.5

The Aggregate Digest — 2026-10-06

2026-10-06T09:43:19.688584+00:00

Daily

New Benchmarks (1)

MintEval (200-Task Subset) (ActionMatch (%), open setting, compile failures scored 0, 200 stratified BTCUSDT tasks): leader Claude Opus 5.5 (88.9), 4 models
MintEval v0 (arXiv 2610.03080): the model turns a trader's colloquial instruction into a trading-strategy program, which is executed bar by bar against the reference program on identical BTCUSDT 15-minute data and frictions. ActionMatch is the share of active bars (either program in or targeting a position) on which the two programs target the same position; compile failures score 0. Open setting (instruction and interface only), temperature 0, on the stratified 200-task subset (40 per state-span quintile) that the frontier models ran. Percent, higher is better.

Top-10 New Scores (23)

Claude Opus 5.5 on AI Chess Leaderboard (Reasoning): 1775.0 (#9)
Claude Opus 5.5 on Epoch AI - Apex Agents: 52.5 (#24)
Claude Opus 5.5 on Epoch AI - Critpt: 30.86 (#12)
Claude Opus 5.5 on Epoch AI - Cursorbench: 56.0 (#3)
Claude Opus 5.5 on Epoch AI - Scicode: 60.42 (#9)
GPT-6.1 Sol on DataBench: 58.0 (#6)
GPT-6.1 Sol on Epoch AI - ECI: 166.09 (#3)
GPT-6.1 Sol on Epoch AI - Weirdml V3: 35.47 (#2)
GPT-6.1 Sol on Kaggle Open Benchmarks - AIME 2025: 100.0 (#9)
GPT-6.1 Sol on Legal Benchmarks - Contract Workflows - Criteria Pass Rate: 86.3 (#6)
GPT-6.1 Sol on Legal Benchmarks - Contract Workflows - Form: 90.8 (#5)
GPT-6.1 Sol on Legal Benchmarks - Contract Workflows - Task Pass Rate: 25.8 (#9)
GPT-6.1 Sol on Legal Benchmarks - Data Extraction - Criteria Pass Rate: 84.5 (#7)
GPT-6.1 Sol on Legal Benchmarks - Data Extraction - Form: 83.0 (#2)
GPT-6.1 Sol on Legal Benchmarks - Data Extraction - Task Pass Rate: 53.3 (#9)
GPT-6.1 Sol on PRISM (1C:Enterprise) - Algorithmic Tasks (A): 87.0 (#19)
GPT-6.1 Sol on PRISM (1C:Enterprise) - Platform Tasks (B): 90.0 (#6)
Gemini 4 Argon on Epoch AI - Critpt: 27.14 (#33)
Gemini 4 Argon on Epoch AI - Frontierswe: 54.96 (#5)
Gemini 4 Argon on Epoch AI - Proofbench: 99.0 (#7)
Gemini 4 Argon on Epoch AI - Scicode: 61.81 (#4)
Gemini 4 Argon on Epoch AI - Vending Bench 2: 13718.16 (#3)
Gemini 4 Argon on WebDev Arena: 1679.5 (#9)

New #1 Leaders (2)

Epoch AI - Apex Agents: Gemini 4 Argon (82.2) beat Claude Sonnet 5.5 (Max) by 6.7
Epoch AI - Blueprint Bench 2: Gemini 4 Argon (0.544) beat GPT-6 Astra by 0.05

The Aggregate Digest — 2026-10-05

2026-10-05T08:59:26.406719+00:00

Daily

Top-10 New Scores (12)

Claude Fable 5.1 on AA-Omniscience Accuracy: 67.23 (#1)
Claude Fable 5.1 on React Native Evals: 89.05 (#1)
Claude Opus 5.5 on AA-Omniscience Accuracy: 66.22 (#2)
Claude Opus 5.5 on React Native Evals: 88.57 (#2)
Claude Sonnet 5.5 on AA-Omniscience Accuracy: 53.95 (#42)
Claude Sonnet 5.5 on ChessBench GitHub - Elo: 2350.0 (#15)
Claude Sonnet 5.5 on Featherbench: 100.0 (#5)
Claude Sonnet 5.5 on Featherbench - Rubric Quality: 9.2 (#7)
GPT-6 Astra on React Native Evals: 86.37 (#6)
GPT-6.1 Sol on Featherbench: 93.0 (#20)
GPT-6.1 Sol on Featherbench - Rubric Quality: 8.8 (#15)
GPT-6.1 Sol on React Native Evals: 86.34 (#8)

The Aggregate Digest — 2026-10-04

2026-10-04T08:44:10.150484+00:00

Daily

New Benchmarks (9)

KaliBench (Exact Correct (%), Unrestricted mode: share of the 5,000 verified queries whose generated Kali Linux command matches the reference after canonicalization (documented aliases and reordered optional flags allowed, positional order enforced), with only the query given; higher is better): leader GPT-5.6 Sol (61.68), 24 models
Cybersecurity tool use on Kali Linux: from an analyst's natural-language request, the model writes the exact command for the right Kali tool, with no tool list or documentation given (5,000 verified queries). Scored by exact-correct rate after canonicalization, which allows documented flag aliases and reordered optional flags but enforces positional order; blocked or missing answers count as wrong. Higher is better (KaliBench, 2026).
Vision2Web S4-2026 (Claude Code) (Score (0-100)): leader Claude Fable 5 (82.3), 6 models
Vision2Web (He et al., 2026) has coding agents build working websites from UI prototype images, requirements and assets: static responsive pages, interactive multi-page frontends and full-stack applications (193 tasks). This board is the overall score, the mean of the three level scores on a 0-100 scale, under the Claude Code agent framework; higher is better. Season S4-2026 (Oct to Dec 2026) uses Gemini-3.5-flash as the visual judge and GLM-5V-Turbo (Claude Code) as the GUI testing agent; scores compare only within a season.
Vision2Web S4-2026 (Claude Code) - Level 1 Static Webpage (Score (0-100)): leader Claude Opus 5 (90.1), 6 models
Vision2Web Level 1: the agent reproduces static responsive webpages from prototype images, and a VLM judge scores visual fidelity of the desktop, tablet and mobile renderings; the board is their mean on a 0-100 scale under the Claude Code framework, higher is better (He et al., 2026). Season S4-2026 (Oct to Dec 2026) uses Gemini-3.5-flash as the visual judge and GLM-5V-Turbo (Claude Code) as the GUI testing agent; scores compare only within a season.
Vision2Web S4-2026 (Claude Code) - Level 2 Interactive (Score (0-100)): leader Claude Fable 5 (83.2), 6 models
Vision2Web Level 2: the agent builds an interactive multi-page frontend from prototypes; the score is the mean of a VLM-judged visual score and a functional score from workflow-driven GUI tests, on a 0-100 scale under the Claude Code framework, higher is better (He et al., 2026). Season S4-2026 (Oct to Dec 2026) uses Gemini-3.5-flash as the visual judge and GLM-5V-Turbo (Claude Code) as the GUI testing agent; scores compare only within a season.
Vision2Web S4-2026 (Claude Code) - Level 3 Full-Stack (Score (0-100)): leader Claude Fable 5 (76.3), 6 models
Vision2Web Level 3: the agent develops a full-stack web application from requirements and prototypes; the score is the mean of a VLM-judged visual score and a functional score from workflow-driven GUI tests, on a 0-100 scale under the Claude Code framework, higher is better (He et al., 2026). Season S4-2026 (Oct to Dec 2026) uses Gemini-3.5-flash as the visual judge and GLM-5V-Turbo (Claude Code) as the GUI testing agent; scores compare only within a season.
Vision2Web S4-2026 (Codex) (Score (0-100)): leader GPT-5.6 Sol (73.5), 3 models
Vision2Web (He et al., 2026) has coding agents build working websites from UI prototype images, requirements and assets: static responsive pages, interactive multi-page frontends and full-stack applications (193 tasks). This board is the overall score, the mean of the three level scores on a 0-100 scale, under the Codex agent framework; higher is better. Season S4-2026 (Oct to Dec 2026) uses Gemini-3.5-flash as the visual judge and GLM-5V-Turbo (Claude Code) as the GUI testing agent; scores compare only within a season.
Vision2Web S4-2026 (Codex) - Level 1 Static Webpage (Score (0-100)): leader GPT-5.6 Sol (85.7), 3 models
Vision2Web Level 1: the agent reproduces static responsive webpages from prototype images, and a VLM judge scores visual fidelity of the desktop, tablet and mobile renderings; the board is their mean on a 0-100 scale under the Codex framework, higher is better (He et al., 2026). Season S4-2026 (Oct to Dec 2026) uses Gemini-3.5-flash as the visual judge and GLM-5V-Turbo (Claude Code) as the GUI testing agent; scores compare only within a season.
Vision2Web S4-2026 (Codex) - Level 2 Interactive (Score (0-100)): leader GPT-5.6 Sol (74.0), 3 models
Vision2Web Level 2: the agent builds an interactive multi-page frontend from prototypes; the score is the mean of a VLM-judged visual score and a functional score from workflow-driven GUI tests, on a 0-100 scale under the Codex framework, higher is better (He et al., 2026). Season S4-2026 (Oct to Dec 2026) uses Gemini-3.5-flash as the visual judge and GLM-5V-Turbo (Claude Code) as the GUI testing agent; scores compare only within a season.
Vision2Web S4-2026 (Codex) - Level 3 Full-Stack (Score (0-100)): leader GPT-5.6 Sol (60.6), 3 models
Vision2Web Level 3: the agent develops a full-stack web application from requirements and prototypes; the score is the mean of a VLM-judged visual score and a functional score from workflow-driven GUI tests, on a 0-100 scale under the Codex framework, higher is better (He et al., 2026). Season S4-2026 (Oct to Dec 2026) uses Gemini-3.5-flash as the visual judge and GLM-5V-Turbo (Claude Code) as the GUI testing agent; scores compare only within a season.

Top-10 New Scores (115)

Claude Fable 5.1 on AA CritPt: 31.14 (#15)
Claude Fable 5.1 on AA GPQA Diamond: 93.74 (#15)
Claude Fable 5.1 on AA Humanity's Last Exam: 59.13 (#4)
Claude Fable 5.1 on AA Long Context Reasoning: 85.33 (#5)
Claude Fable 5.1 on AA Omniscience: 43.45 (#5)
Claude Fable 5.1 on AA SciCode: 63.08 (#6)
Claude Fable 5.1 on AA-AnalystAgent: 57.5 (#4)
Claude Fable 5.1 on AA-LCR: 85.33 (#4)
Claude Fable 5.1 on AA-Omniscience Accuracy: 67.23 (#1)
Claude Fable 5.1 on Artificial Analysis Intelligence Index: 53.35 (#10)
Claude Fable 5.1 on CAIS Risk Index: 29.0 (#2)
Claude Fable 5.1 on CAIS Vision Capabilities Index: 63.1 (#8)
Claude Fable 5.1 on CritPt: 31.1 (#8)
Claude Fable 5.1 on Finance Agent v2: 58.9 (#9)
Claude Fable 5.1 on FrontierCode: 63.6 (#14)
Claude Fable 5.1 on GPQA Diamond (BenchmarkList): 93.7 (#2)
Claude Fable 5.1 on Gray Swan IPI — Combined Attack Set, Transfer Only, No Computer Use: 1.0 (#3)
Claude Fable 5.1 on Guesswork 2026-09: 0.5512 (#4)
Claude Fable 5.1 on KernelBench Mega: 22.946 (#6)
Claude Fable 5.1 on LABBench2 Aggregate: 68.6 (#4)
Claude Fable 5.1 on LVBench: 79.7 (#8)
Claude Fable 5.1 on Lech Mazur Writing: 3.796 (#1)
Claude Fable 5.1 on MCP Atlas: 87.2 (#2)
Claude Fable 5.1 on MILU: 93.0 (#3)
Claude Fable 5.1 on MazeBench: 10.0 (#5)
Claude Fable 5.1 on PRBench Finance: 50.8 (#8)
Claude Fable 5.1 on Professional Reasoning Bench - Legal: 51.6 (#6)
Claude Fable 5.1 on ReactBench: 45.6 (#2)
Claude Fable 5.1 on Tau3 Banking: 47.22 (#12)
Claude Fable 5.1 on Terminal-Bench-Science: 40.0 (#3)
Claude Fable 5.1 on Vals AI MMMU: 90.64 (#1)
Claude Fable 5.1 on Vals AI MortgageTax: 70.79 (#2)
Claude Fable 5.1 on Vals AI ProgramBench: 7.0 (#2)
Claude Fable 5.1 on Vals AI Time Horizon Index: 63.33 (#3)
Claude Fable 5.1 on Vibe Code Bench v1.1: 90.3 (#2)
Claude Opus 5.5 on AA CritPt: 31.71 (#7)
Claude Opus 5.5 on AA Humanity's Last Exam: 61.35 (#2)
Claude Opus 5.5 on AA Long Context Reasoning: 84.67 (#9)
Claude Opus 5.5 on AA MMMU-Pro: 87.69 (#2)
Claude Opus 5.5 on AA Omniscience: 46.42 (#2)
Claude Opus 5.5 on AA SciCode: 66.9 (#2)
Claude Opus 5.5 on AA-AnalystAgent: 56.25 (#7)
Claude Opus 5.5 on AA-LCR: 84.67 (#5)
Claude Opus 5.5 on AA-Omniscience Accuracy: 66.22 (#2)
Claude Opus 5.5 on AA-Omniscience Net Score: 58.0 (#1)
Claude Opus 5.5 on Artificial Analysis Intelligence Index: 57.62 (#2)
Claude Opus 5.5 on CritPt: 31.7 (#5)
Claude Opus 5.5 on Crosby micro1 RedlineBench: 53.5 (#5)
Claude Opus 5.5 on Finance Agent v2: 58.6 (#10)
Claude Opus 5.5 on FrontierCode: 65.3 (#1)
Claude Opus 5.5 on Gray Swan IPI — Combined Attack Set, Transfer Only, No Computer Use: 1.0 (#2)
Claude Opus 5.5 on LABBench2 Aggregate: 73.1 (#3)
Claude Opus 5.5 on LVBench: 83.7 (#6)
Claude Opus 5.5 on Lech Mazur Writing: 3.767 (#3)
Claude Opus 5.5 on MILU: 93.1 (#2)
Claude Opus 5.5 on Professional Reasoning Bench - Legal: 47.58 (#16)
Claude Opus 5.5 on Terminal-Bench-Science: 63.3333 (#2)
Claude Opus 5.5 on Terminal-Bench-Science 0.1: 63.3 (#2)
Claude Opus 5.5 on Vals AI ProgramBench: 18.5 (#1)
Claude Opus 5.5 on Vals AI Time Horizon Index: 91.33 (#1)
Claude Opus 5.5 on Vibe Code Bench v1.1: 90.3 (#3)
Claude Sonnet 5.5 on AA CritPt: 31.43 (#11)
Claude Sonnet 5.5 on AA Humanity's Last Exam: 54.96 (#17)
Claude Sonnet 5.5 on AA Long Context Reasoning: 82.67 (#43)
Claude Sonnet 5.5 on AA Omniscience: 32.3 (#41)
Claude Sonnet 5.5 on AA SciCode: 61.0 (#10)
Claude Sonnet 5.5 on AA-AnalystAgent: 57.5 (#5)
Claude Sonnet 5.5 on AA-LCR: 82.67 (#23)
Claude Sonnet 5.5 on AA-Omniscience Accuracy: 53.95 (#42)
Claude Sonnet 5.5 on AA-Omniscience Net Score: 35.0 (#8)
Claude Sonnet 5.5 on Artificial Analysis Intelligence Index: 56.0 (#4)
Claude Sonnet 5.5 on CritPt: 31.4 (#7)
Claude Sonnet 5.5 on Guesswork 2026-09: 0.5361 (#2)
Claude Sonnet 5.5 on KernelBench Mega: 32.131 (#3)
Claude Sonnet 5.5 on MILU: 91.6 (#9)
Claude Sonnet 5.5 on Vals AI ProgramBench: 6.5 (#3)
GPT-6 Astra on AA-LCR: 80.67 (#36)
GPT-6 Astra on AIIQ Composite IQ: 142.0 (#2)
GPT-6 Astra on BioSecBench-Function: 46.0 (#1)
GPT-6 Astra on BrowseComp: 91.5 (#1)
GPT-6 Astra on CAIS Risk Index: 45.8 (#14)
GPT-6 Astra on CritPt: 31.7 (#4)
GPT-6 Astra on Finance Agent v2: 53.5 (#21)
GPT-6 Astra on FrontierCode: 64.5 (#6)
GPT-6 Astra on GPQA Diamond (BenchmarkList): 96.1 (#1)
GPT-6 Astra on Gray Swan IPI — Combined Attack Set, Transfer Only, No Computer Use: 8.5 (#7)
GPT-6 Astra on Guesswork 2026-09: 0.5423 (#3)
GPT-6 Astra on LABBench2 Aggregate: 85.4 (#2)
GPT-6 Astra on LVBench: 87.5 (#3)
GPT-6 Astra on Lech Mazur Writing: 3.455 (#4)
GPT-6 Astra on MazeBench: 14.0 (#1)
GPT-6 Astra on PRBench Finance: 47.54 (#14)
GPT-6 Astra on Professional Reasoning Bench - Legal: 48.36 (#14)
GPT-6 Astra on SWE Atlas: 60.9 (#9)
GPT-6 Astra on Terminal-Bench-Science: 68.0952 (#1)
GPT-6 Astra on Thinkingbox: 58.31 (#6)
GPT-6 Astra on Vals AI ProgramBench: 5.5 (#4)
GPT-6 Astra on Vals AI Time Horizon Index: 90.5 (#2)
GPT-6 Astra on VariantBench: 47.7 (#4)
GPT-6 Astra on Vibe Code Bench v1.1: 89.6 (#4)
GPT-6.1 Sol on AA-LCR: 84.0 (#7)
GPT-6.1 Sol on AIIQ Composite IQ: 141.0 (#3)
GPT-6.1 Sol on CritPt: 31.7 (#2)
GPT-6.1 Sol on ProphetArena: 0.9773 (#3)
GPT-6.1 Sol on SecIT Bench (Pydantic AI): 79.91 (#5)
GPT-6.1 Sol on UGI - Natural Intelligence: 78.26 (#7)
GPT-6.1 Sol on UGI - Willingness (W/10): 3.8 (#957)
GPT-6.1 Sol on UGI - Writing: 67.72 (#43)
GPT-6.1 Sol on UGI Leaderboard: 58.75 (#23)
GPT-6.1 Sol on Vals AI ProgramBench: 3.0 (#5)
Gemini 4 Argon on AA-LCR: 79.67 (#53)
Gemini 4 Argon on CritPt: 27.1 (#15)
Gemini 4 Argon on Gray Swan IPI — Combined Attack Set, Transfer Only, No Computer Use: 0.7 (#1)
Gemini 4 Argon on Terminal-Bench-Science 0.1: 57.6 (#3)
Gemini 4 Argon on Vals AI ProgramBench: 2.5 (#7)

New #1 Leaders (17)

Codeforces ELO: DeepSeek V4.1 Flash (Max Effort) (3471.0) beat Gemma 4 31B by 1321.0
Terminal-Bench-Science: GPT-6 Astra (Max) (68.0952) beat Claude Opus 5 (Max) by 38.1
MultiChallenge: DeepSeek V4.1 Flash (72.12) beat DeepSeek V4 Pro by 8.02
AIIQ Composite IQ: opus-5.5 (143.0) beat Claude Fable 5 by 6.0
CAIS Vision Capabilities Index: GPT-6 Astra (71.5) beat Gemini 3.5 Flash by 5.6
PRBench Finance: Muse Spark 1.3 (59.54) beat Muse Spark 1.1 by 4.53
Professional Reasoning Bench - Legal: Muse Spark 1.3 (61.56) beat Muse Spark 1.1 by 4.51
Finance Agent v2: Gemini 4 Argon (65.4) beat Gemini 3.8 Flash by 4.0
LVBench: Gemini 4 Argon (91.7) beat Gemini 3.8 Flash by 3.9
NL2Repo: Claude Opus 5 (max effort) (75.3) beat Claude Opus 5 by 3.0
GameDevBench: GPT-6.1 Sol (Codex, xHigh) (69.97) beat Claude Fable 5 (Claude Code, xHigh) by 2.67
LABBench2 Aggregate: Gemini 4 Argon (88.8) beat Gemini 3.8 Flash by 2.6
Vibe Code Bench v1.1: Gemini 4 Argon (91.9) beat Claude Fable 5 by 1.55
MazeBench: GPT-6 Astra (Max) (14.0) beat GPT-5.6 Sol (Max) by 1.0
FrontierCode: Claude Opus 5.5 (Medium) (65.3) beat Claude Fable 5 (xHigh) by 0.36
GPQA Diamond (BenchmarkList): GPT-6 Astra (Max) (96.1) beat GPT-6 Astra by 0.1
KernelBench Mega: Claude Opus 5.5 (35.46) beat Claude Fable 5 by 0.02

Weekly

New Benchmarks (9)

KaliBench (Exact Correct (%), Unrestricted mode: share of the 5,000 verified queries whose generated Kali Linux command matches the reference after canonicalization (documented aliases and reordered optional flags allowed, positional order enforced), with only the query given; higher is better): leader GPT-5.6 Sol (61.68), 24 models
Cybersecurity tool use on Kali Linux: from an analyst's natural-language request, the model writes the exact command for the right Kali tool, with no tool list or documentation given (5,000 verified queries). Scored by exact-correct rate after canonicalization, which allows documented flag aliases and reordered optional flags but enforces positional order; blocked or missing answers count as wrong. Higher is better (KaliBench, 2026).
Vision2Web S4-2026 (Claude Code) (Score (0-100)): leader Claude Fable 5 (82.3), 6 models
Vision2Web (He et al., 2026) has coding agents build working websites from UI prototype images, requirements and assets: static responsive pages, interactive multi-page frontends and full-stack applications (193 tasks). This board is the overall score, the mean of the three level scores on a 0-100 scale, under the Claude Code agent framework; higher is better. Season S4-2026 (Oct to Dec 2026) uses Gemini-3.5-flash as the visual judge and GLM-5V-Turbo (Claude Code) as the GUI testing agent; scores compare only within a season.
Vision2Web S4-2026 (Claude Code) - Level 1 Static Webpage (Score (0-100)): leader Claude Opus 5 (90.1), 6 models
Vision2Web Level 1: the agent reproduces static responsive webpages from prototype images, and a VLM judge scores visual fidelity of the desktop, tablet and mobile renderings; the board is their mean on a 0-100 scale under the Claude Code framework, higher is better (He et al., 2026). Season S4-2026 (Oct to Dec 2026) uses Gemini-3.5-flash as the visual judge and GLM-5V-Turbo (Claude Code) as the GUI testing agent; scores compare only within a season.
Vision2Web S4-2026 (Claude Code) - Level 2 Interactive (Score (0-100)): leader Claude Fable 5 (83.2), 6 models
Vision2Web Level 2: the agent builds an interactive multi-page frontend from prototypes; the score is the mean of a VLM-judged visual score and a functional score from workflow-driven GUI tests, on a 0-100 scale under the Claude Code framework, higher is better (He et al., 2026). Season S4-2026 (Oct to Dec 2026) uses Gemini-3.5-flash as the visual judge and GLM-5V-Turbo (Claude Code) as the GUI testing agent; scores compare only within a season.
Vision2Web S4-2026 (Claude Code) - Level 3 Full-Stack (Score (0-100)): leader Claude Fable 5 (76.3), 6 models
Vision2Web Level 3: the agent develops a full-stack web application from requirements and prototypes; the score is the mean of a VLM-judged visual score and a functional score from workflow-driven GUI tests, on a 0-100 scale under the Claude Code framework, higher is better (He et al., 2026). Season S4-2026 (Oct to Dec 2026) uses Gemini-3.5-flash as the visual judge and GLM-5V-Turbo (Claude Code) as the GUI testing agent; scores compare only within a season.
Vision2Web S4-2026 (Codex) (Score (0-100)): leader GPT-5.6 Sol (73.5), 3 models
Vision2Web (He et al., 2026) has coding agents build working websites from UI prototype images, requirements and assets: static responsive pages, interactive multi-page frontends and full-stack applications (193 tasks). This board is the overall score, the mean of the three level scores on a 0-100 scale, under the Codex agent framework; higher is better. Season S4-2026 (Oct to Dec 2026) uses Gemini-3.5-flash as the visual judge and GLM-5V-Turbo (Claude Code) as the GUI testing agent; scores compare only within a season.
Vision2Web S4-2026 (Codex) - Level 1 Static Webpage (Score (0-100)): leader GPT-5.6 Sol (85.7), 3 models
Vision2Web Level 1: the agent reproduces static responsive webpages from prototype images, and a VLM judge scores visual fidelity of the desktop, tablet and mobile renderings; the board is their mean on a 0-100 scale under the Codex framework, higher is better (He et al., 2026). Season S4-2026 (Oct to Dec 2026) uses Gemini-3.5-flash as the visual judge and GLM-5V-Turbo (Claude Code) as the GUI testing agent; scores compare only within a season.
Vision2Web S4-2026 (Codex) - Level 2 Interactive (Score (0-100)): leader GPT-5.6 Sol (74.0), 3 models
Vision2Web Level 2: the agent builds an interactive multi-page frontend from prototypes; the score is the mean of a VLM-judged visual score and a functional score from workflow-driven GUI tests, on a 0-100 scale under the Codex framework, higher is better (He et al., 2026). Season S4-2026 (Oct to Dec 2026) uses Gemini-3.5-flash as the visual judge and GLM-5V-Turbo (Claude Code) as the GUI testing agent; scores compare only within a season.
Vision2Web S4-2026 (Codex) - Level 3 Full-Stack (Score (0-100)): leader GPT-5.6 Sol (60.6), 3 models
Vision2Web Level 3: the agent develops a full-stack web application from requirements and prototypes; the score is the mean of a VLM-judged visual score and a functional score from workflow-driven GUI tests, on a 0-100 scale under the Codex framework, higher is better (He et al., 2026). Season S4-2026 (Oct to Dec 2026) uses Gemini-3.5-flash as the visual judge and GLM-5V-Turbo (Claude Code) as the GUI testing agent; scores compare only within a season.

New Models (231)

Gemini 4 Argon — Aggregate Rating 2028, #3
- Agent Arena - Steerability: 13.48 (#1/51)
- Chatbot Arena (Text): 1525.0 (#1/413)
- AutomationBench-AA: 77.51 (#1/213)
- AutomationBench-AA - Strict: 53.12 (#1/213)
- AA-Omniscience Index - Law: 53.0 (#1/565)
- AA-Briefcase - DOCX - Rubric Pass Rate: 62.13 (#1/216)
- AA-Briefcase - PDF - Rubric Pass Rate: 72.41 (#1/216)
- AA-Briefcase - PPTX - Rubric Pass Rate: 67.36 (#1/216)
- AutomationBench-AA - Finance - Raw Objective Completion: 94.62 (#1/213)
- AutomationBench-AA - HR - Raw Objective Completion: 81.99 (#1/213)
- ...and 125 more
Claude Opus 5.5 — Aggregate Rating 2027, #4
- SpacetimeDB LLM Benchmark (TypeScript): 99.5 (#1/19)
- SpacetimeDB LLM Benchmark (Rust): 100.0 (#1/19)
- Design Arena (Data Viz): 1398.0 (#1/166)
- ProphetArena: 0.9812 (#1/16)
- Epoch AI - ECI: 167.35 (#1/270)
- WebDev Arena: 1820.31 (#1/129)
- Epoch AI - Mirrorcode: 77.4 (#1/9)
- Epoch AI - Cursorbench: 57.8 (#1/54)
- Epoch AI - Scicode: 66.9 (#1/195)
- Epoch AI - Lmca: 68.23 (#1/175)
- ...and 182 more
GPT-6 Astra — Aggregate Rating 2018, #5
- DeepSWE: 74.1 (#1/31)
- DecBench: 60.0 (#1/3)
- MedXpertQA MM (Natomy): 87.4 (#1/12)
- MedXpertQA MM (Natomy) - Diagnosis: 86.99 (#1/9)
- MedXpertQA MM (Natomy) - Treatment: 87.05 (#1/9)
- MedXpertQA MM (Natomy) - Basic Science: 89.24 (#1/9)
- DeepSWE-mini (16 Tasks): 81.25 (#1/31)
- Benchmarks.bio - scBench: 64.79 (#1/24)
- Benchmarks.bio - TxBench-AD: 56.9 (#1/14)
- Benchmarks.bio - TxBench-OD: 55.46 (#1/13)
- ...and 127 more
GPT-6 Astra Pro — Aggregate Rating 2010, #7
- OpenRouter GPQA Diamond: 95.5 (#2/154)
- OpenRouter Tau2-Bench Airline: 72.7 (#52/143)
GPT-6.1 Sol — Aggregate Rating 2005, #8
- FrontierMath - Tier 4 (v2): 100.0 (#1/70)
- FrontierMath - Tiers 1-3 (v2): 93.68 (#1/111)
- Explicit Edit Benchmark: 100.0 (#1/25)
- Agent Arena - Tool Hallucination: -0.49 (#1/51)
- MathArena - ARXIV_FALSE August: 86.31 (#1/13)
- MathArena Arxiv: 95.0 (#1/26)
- MathArena - ARXIV June: 94.44 (#1/29)
- MathArena - ARXIV August: 96.49 (#1/13)
- OTIS Mock AIME 2024-25: 100.0 (#1/297)
- Epoch AI - Frontiermath Tiers 1 3 V2: 93.68 (#1/114)
- ...and 249 more
Claude Fable 5.1 — Aggregate Rating 1997, #9
- GSO-Bench: 88.24 (#1/34)
- MLCR-AA - Completeness: 89.03 (#1/94)
- AA-Omniscience Index - Software Engineering (SWE) - Dart: 96.0 (#1/565)
- AA-Omniscience Index - Software Engineering (SWE) - R: 74.0 (#1/565)
- AA-Omniscience Index - Software Engineering (SWE) - Swift: 100.0 (#1/565)
- AA GDP.pdf - Insurance - Criterion Pass Rate: 89.15 (#1/204)
- AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 82.56 (#1/204)
- Harvey LAB-AA - Capital Markets: 97.25 (#1/55)
- Harvey LAB-AA - Intellectual Property: 97.17 (#1/55)
- Chatbot Arena (Text - Korean): 1514.0 (#1/278)
- ...and 119 more
Claude Sonnet 5.5 — Aggregate Rating 1967, #10
- Agent Arena - Bash Recovery: 15.05 (#1/51)
- Design Arena (UI Components): 1415.0 (#1/166)
- Epoch AI - Apex Agents: 75.5 (#1/39)
- AA-Briefcase: 65.15 (#1/216)
- MLCR-AA: 75.0 (#1/94)
- AA-Briefcase - Presentation Elo: 1716.42 (#1/216)
- AA-Omniscience Index - Software Engineering (SWE) - Julia: 92.0 (#1/565)
- Harvey LAB-AA - Data Privacy & Cybersecurity: 98.94 (#1/55)
- Harvey LAB-AA - Trusts, Estates & Private Client: 95.8 (#1/55)
- AA-Briefcase - XLSX - Rubric Pass Rate: 66.36 (#1/216)
- ...and 316 more
Claude Opus 5 — Aggregate Rating 1963, #11
- TrackingAI IQ Test (Vision): 100.0 (#1/28)
- EnterpriseOps-Gym-AA - Calendar: 51.28 (#1/50)
- Vals AI Public Benefits Bench: 76.93 (#1/47)
- NL2Repo: 75.3 (#1/50)
- JobBench: 65.7 (#1/8)
- Harvey LAB-AA - Energy & Natural Resources: 99.01 (#2/55)
- Vals AI MMMU: 89.88 (#2/93)
- SWE Atlas: 66.0 (#2/58)
- BioSecBench-Function: 44.8 (#2/4)
- AA-Omniscience Index - Software Engineering (SWE) - Swift: 96.0 (#3/565)
- ...and 120 more
Claude Fable 5 — Aggregate Rating 1942, #12
- EnterpriseOps-Gym-AA: 51.12 (#1/50)
- AA-Omniscience Index - Business: 34.5 (#1/565)
- EnterpriseOps-Gym-AA - HR: 50.0 (#1/50)
- EnterpriseOps-Gym-AA - Teams: 52.0 (#1/50)
- K-Bench - Risk: 96.11 (#1/62)
- AA Terminal-Bench Hard: 62.88 (#2/441)
- AA-Omniscience Index - Software Engineering (SWE) - Swift: 96.0 (#2/565)
- EnterpriseOps-Gym-AA - Calendar: 50.96 (#2/50)
- EnterpriseOps-Gym-AA - ITSM: 48.07 (#2/50)
- SWE-Together: 85.16 (#2/20)
- ...and 108 more
GPT-6 Sol — Aggregate Rating 1941, #13
- MLCR-AA - Conciseness: 100.0 (#1/94)
- Vals AI CyberBench: 77.98 (#1/44)
- Kaggle FACTS Grounding: 85.22 (#1/49)
- Epoch AI - Vending Bench 2: 14427.85 (#2/63)
- MLCR-AA - Accuracy: 96.67 (#2/94)
- AutomationBench-AA - Sales - Raw Objective Completion: 90.07 (#2/213)
- SpacetimeDB LLM Benchmark (TypeScript): 99.5 (#3/19)
- OTIS Mock AIME 2024-25: 100.0 (#3/297)
- WebDev Arena (Fullstack): 1706.0 (#3/77)
- Supabase Evals (No Skills): 78.3 (#3/6)
- ...and 118 more
GPT-5.5 Pro — Aggregate Rating 1938, #14
- AA CritPt: 30.57 (#19/603)
GPT-5.6 Sol — Aggregate Rating 1921, #15
- AA Terminal-Bench Hard: 65.91 (#1/441)
- AA CritPt: 32.29 (#1/603)
- EnterpriseOps-Gym-AA - CSM: 55.38 (#1/50)
- CritPt: 32.3 (#1/278)
- AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 92.0 (#2/565)
- PostTrainBench: 36.23 (#3/13)
- AA-Briefcase - Presentation Elo: 1651.64 (#3/216)
- AA-Omniscience Index - Software Engineering (SWE) - HTML: 92.0 (#3/565)
- Vals AI CyberBench: 76.31 (#3/44)
- MultiChallenge: 66.32 (#3/7)
- ...and 123 more
Gemini 3.8 Flash — Aggregate Rating 1914, #16
- MEGA-Bench Task - Move Position To Position Hanoi 4 Pole: 58.2 (#1/1)
- OpenRouter GPQA Diamond: 95.6 (#1/154)
- FrontierFinance: 50.01 (#2/13)
- FrontierFinance - Sector, Industry & Macro: 41.52 (#2/13)
- ProphetArena: 0.9802 (#2/16)
- Chatbot Arena (Vision - Diagram): 1341.0 (#2/114)
- FrontierFinance - Financial Data & Modeling: 56.16 (#3/13)
- FrontierFinance - Company Research: 46.58 (#3/13)
- AA GPQA Diamond: 95.25 (#3/636)
- Chatbot Arena (Vision - OCR): 1315.0 (#3/115)
- ...and 105 more
Gemini 3.7 Flash — Aggregate Rating 1903, #19
- AA-AnalystAgent: 60.0 (#1/47)
- AA-AnalystAgent - Pass@1: 70.5 (#1/38)
- EnterpriseOps-Gym-AA - ITSM: 50.28 (#1/50)
- Chatbot Arena (Vision - Chinese): 1377.0 (#1/123)
- EnterpriseOps-Gym-AA: 50.4 (#2/50)
- EnterpriseOps-Gym-AA - Teams: 51.0 (#2/50)
- EnterpriseOps-Gym-AA - HR: 44.02 (#3/50)
- EnterpriseOps-Gym-AA - Calendar: 49.04 (#4/50)
- EnterpriseOps-Gym-AA - Email: 67.31 (#5/50)
- EnterpriseOps-Gym-AA - Hybrid: 42.48 (#5/50)
- ...and 110 more
Grok 4.7 — Aggregate Rating 1900, #20
- AutomationBench-AA - Finance - Raw Objective Completion: 93.49 (#2/213)
- Supabase Evals (Skills): 89.9 (#2/6)
- EKI Estonian LLM - Propaganda Resistance: 97.38 (#2/94)
- SWE-Together: 84.45 (#3/20)
- BioSecBench-Function: 43.3 (#3/4)
- SpacetimeDB LLM Benchmark (TypeScript): 99.5 (#4/19)
- Supabase Evals (No Skills): 78.3 (#4/6)
- SpacetimeDB LLM Benchmark (Rust): 98.4 (#5/19)
- AutomationBench-AA - Strict: 37.75 (#6/213)
- EKI Estonian LLM - Book Recommendations: 86.25 (#6/94)
- ...and 102 more
Muse Spark 1.3 — Aggregate Rating 1896, #21
- Harvey LAB-AA: 95.48 (#1/55)
- Harvey LAB-AA - All Pass: 30.83 (#1/55)
- Harvey LAB-AA - Antitrust & Competition: 96.01 (#1/55)
- Harvey LAB-AA - Arbitration & International Dispute Resolution: 98.92 (#1/55)
- Harvey LAB-AA - Banking & Finance: 97.77 (#1/55)
- Harvey LAB-AA - Bankruptcy & Restructuring: 97.92 (#1/55)
- Harvey LAB-AA - Corporate Governance: 95.89 (#1/55)
- Harvey LAB-AA - Corporate M&A: 97.12 (#1/55)
- Harvey LAB-AA - Emerging Companies & Venture Capital: 97.27 (#1/55)
- Harvey LAB-AA - Employment & Labor: 92.13 (#1/55)
- ...and 106 more
Grok 4.6 — Aggregate Rating 1891, #22
- Tau3 Banking: 50.72 (#2/224)
- EnterpriseOps-Gym-AA - CSM: 55.38 (#2/50)
- EnterpriseOps-Gym-AA - HR: 45.65 (#2/50)
- MLCR-AA - Conciseness: 99.44 (#3/94)
- EnterpriseOps-Gym-AA - ITSM: 44.2 (#4/50)
- BioSecBench-Function: 39.9 (#4/4)
- AA GPQA Diamond: 94.95 (#5/636)
- EnterpriseOps-Gym-AA: 48.34 (#5/50)
- AutomationBench-AA - Finance - Raw Objective Completion: 92.16 (#5/213)
- AutomationBench-AA - HR - Raw Objective Completion: 76.94 (#6/213)
- ...and 80 more
Qwen 3.8 Max (0902) — Aggregate Rating 1886, #23
- Harvey LAB-AA - Litigation & Dispute Resolution: 96.63 (#1/55)
- AA APEX-Agents: 42.4 (#2/35)
- Harvey LAB-AA - Corporate M&A: 96.79 (#2/55)
- Harvey LAB-AA - Environmental & ESG: 93.19 (#2/55)
- Harvey LAB-AA - White Collar Defense & Investigations: 95.62 (#2/55)
- Harvey LAB-AA: 93.65 (#3/55)
- EnterpriseOps-Gym-AA - Calendar: 50.96 (#3/50)
- Harvey LAB-AA - Banking & Finance: 95.17 (#3/55)
- Harvey LAB-AA - Emerging Companies & Venture Capital: 95.56 (#3/55)
- Harvey LAB-AA - Employment & Labor: 90.26 (#3/55)
- ...and 37 more
GPT-5.5 — Aggregate Rating 1881, #24
- Vals AI Terminal-Bench 2.0: 73.2 (#1/67)
- K-Bench: 98.96 (#1/62)
- K-Bench - Risk: 96.02 (#2/62)
- AA-AnalystAgent - Pass@1: 66.25 (#3/38)
- EnterpriseOps-Gym-AA - CSM: 53.94 (#4/50)
- DGEval - IMDG Code MCQ: 90.7 (#4/35)
- DGEval - Regulatory Recall: 30.5 (#4/35)
- AA-AnalystAgent - Pass@K: 78.75 (#5/38)
- AA APEX-Agents: 37.68 (#6/35)
- AA-LCR: 84.33 (#6/202)
- ...and 111 more
Step 5 Preview — Aggregate Rating 1881, #25
- EnterpriseOps-Gym-AA - Email: 70.19 (#1/50)
- Harvey LAB-AA - Healthcare & Life Sciences: 96.04 (#1/55)
- Harvey LAB-AA - Antitrust & Competition: 96.01 (#2/55)
- Harvey LAB-AA - Corporate Governance: 95.21 (#2/55)
- Harvey LAB-AA - Tax: 91.97 (#2/55)
- AA-LCR: 88.33 (#2/202)
- Harvey LAB-AA - Environmental & ESG: 92.11 (#3/55)
- Harvey LAB-AA - International Trade & Sanctions: 93.25 (#3/55)
- GPQA Diamond (BenchmarkList): 93.5 (#3/6)
- Harvey LAB-AA - All Pass: 15.83 (#4/55)
- ...and 80 more
Kimi K3 — Aggregate Rating 1880, #26
- AA Long Context Reasoning: 88.67 (#1/599)
- Harvey LAB-AA - Energy & Natural Resources: 100.0 (#1/55)
- AA-LCR: 88.67 (#1/202)
- Harvey LAB-AA: 94.64 (#2/55)
- Harvey LAB-AA - All Pass: 26.67 (#2/55)
- Harvey LAB-AA - Arbitration & International Dispute Resolution: 96.77 (#2/55)
- Harvey LAB-AA - Bankruptcy & Restructuring: 94.81 (#2/55)
- Harvey LAB-AA - Employment & Labor: 91.01 (#2/55)
- Harvey LAB-AA - Funds & Asset Management: 93.0 (#2/55)
- Harvey LAB-AA - Healthcare & Life Sciences: 95.38 (#2/55)
- ...and 117 more
Qwen 3.8 Max — Aggregate Rating 1877, #27
- OSWorld-Verified: 86.1 (#2/141)
- FrontierSWE V2: 15.81 (#17/20)
- ReactBench: 32.3 (#21/72)
- KernelBench Hard: 0.0545 (#41/62)
- Vals AI CyberBench: 28.57 (#44/44)
- NonoBench: 60.0 (#55/130)
Claude Opus 4.8 — Aggregate Rating 1869, #29
- Vals AI Terminal-Bench 2.0: 70.04 (#2/67)
- AA-AnalystAgent - Pass@K: 78.75 (#4/38)
- PostTrainBench: 33.84 (#5/13)
- Harvey LAB-AA - Bankruptcy & Restructuring: 93.43 (#5/55)
- Vals AI MortgageTax: 69.91 (#5/98)
- K-Bench: 98.58 (#5/62)
- Harvey LAB-AA - Banking & Finance: 94.8 (#6/55)
- Harvey LAB-AA - Corporate Governance: 92.81 (#7/55)
- Vals AI Time Horizon Index: 11.83 (#7/12)
- AA-AnalystAgent - Pass@1: 63.5 (#8/38)
- ...and 103 more
DeepSeek V4.1 Flash — Aggregate Rating 1865, #30
- AA GDP.pdf - Legal - Criterion Pass Rate: 87.28 (#1/204)
- Codeforces ELO: 3471.0 (#1/8)
- MultiChallenge: 72.12 (#1/7)
- Finance Agent v2: 61.57 (#3/50)
- ProphetArena: 0.9633 (#4/16)
- AutomationBench-AA: 68.89 (#4/213)
- MathArena Apex: 65.6 (#5/52)
- NL2Repo: 65.4 (#5/50)
- JobBench: 45.8 (#6/8)
- FoodTruckBench: 50179.0 (#7/10)
- ...and 77 more
GLM-5.3 — Aggregate Rating 1863, #31
- EnterpriseOps-Gym-AA - CSM: 54.84 (#3/50)
- Tau3 Banking: 50.31 (#4/224)
- MCP Atlas: 86.8 (#4/72)
- CAIS Risk Index: 38.7 (#4/52)
- Vals AI Tax Agent Bench: 73.09 (#5/65)
- MultiChallenge: 63.89 (#5/7)
- Finance Agent v2: 60.49 (#6/50)
- GPQA Diamond (BenchmarkList): 91.7 (#6/6)
- AutomationBench-AA - Finance - Raw Objective Completion: 91.17 (#7/213)
- KernelBench Mega: 19.435 (#7/53)
- ...and 82 more
Ember-1 — Aggregate Rating 1863, #33
- Vals AI Tax Agent Bench: 68.04 (#12/65)
- SecIT Bench (Pydantic AI): 74.53 (#14/25)
- Vals AI Vibe Code Bench: 83.87 (#15/108)
- Vals AI BioMysteryBench: 67.04 (#16/23)
- Vals AI MysteryMechanism: 22.52 (#16/23)
- Vals AI Public Benefits Bench: 66.17 (#17/47)
- Vals AI Legal Research Bench: 44.71 (#17/73)
- Vals AI Harvey Legal Agent Bench: 9.17 (#19/74)
- Vals AI ProofBench: 61.0 (#20/48)
- Vals AI Terminal-Bench 4.0: 19.7 (#21/43)
- ...and 11 more
GPT-5.6 Terra — Aggregate Rating 1862, #34
- AA GDP.pdf - Legal - Criterion Pass Rate: 86.0 (#2/204)
- AA Terminal-Bench Hard: 62.88 (#4/441)
- AA APEX-Agents: 38.94 (#4/35)
- MLCR-AA - Accuracy: 93.68 (#6/94)
- K-Bench - Risk: 94.59 (#6/62)
- AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 79.16 (#10/204)
- Vals AI SkillsBench: 58.9 (#11/35)
- AA-Briefcase - Presentation Elo: 1528.02 (#12/216)
- AA GDP.pdf - Construction - Criterion Pass Rate: 75.79 (#14/204)
- EnterpriseOps-Gym-AA - CSM: 48.92 (#14/50)
- ...and 104 more
MiMo-V2.6-Pro — Aggregate Rating 1859, #35
- JobBench: 62.0 (#2/8)
- AA-LCR: 86.33 (#3/202)
- AI Coding Daily (OpenCode) - React-TS Code Quality: 18.0 (#4/18)
- AI Coding Daily (OpenCode) - Total: 53.81 (#5/18)
- Explicit Edit Benchmark: 98.6 (#6/25)
- AI Coding Daily (OpenCode) - Laravel Code Quality: 17.41 (#7/18)
- Agent Arena - Confirmed Success: 7.74 (#9/51)
- FoodTruckBench: 49373.0 (#9/10)
- Vals AI MedScribe: 88.31 (#11/106)
- BenchLM: 75.5 (#11/200)
- ...and 33 more
Gemini 3.6 Flash — Aggregate Rating 1854, #37
- AIMultiple - Text-to-SQL (Strict Execution Match): 61.4 (#3/72)
- AIMultiple - Text-to-SQL (Adjudicated): 83.1 (#3/72)
- Vals AI MMMU: 88.38 (#7/93)
- MLCR-AA - Conciseness: 98.33 (#14/94)
- AIMultiple - FinanceReasoning (Hard): 88.66 (#17/87)
- MazeBench: 0.0 (#19/44)
- Harvey LAB-AA - Trusts, Estates & Private Client: 91.22 (#23/55)
- Vals AI MortgageTax: 67.61 (#24/98)
- Harvey LAB-AA - Funds & Asset Management: 85.71 (#25/55)
- Harvey LAB-AA - Immigration: 88.73 (#26/55)
- ...and 91 more
Grok 4.5 — Aggregate Rating 1854, #38
- SWE-Marathon: 29.0 (#1/12)
- Harvey LAB-AA - Banking & Finance: 95.54 (#2/55)
- Harvey LAB-AA - Tax: 91.64 (#4/55)
- Harvey LAB-AA - Funds & Asset Management: 91.84 (#5/55)
- Harvey LAB-AA - Healthcare & Life Sciences: 94.39 (#5/55)
- Harvey LAB-AA - Intellectual Property: 96.47 (#5/55)
- Harvey LAB-AA - Arbitration & International Dispute Resolution: 95.7 (#6/55)
- Harvey LAB-AA - Capital Markets: 96.08 (#7/55)
- Harvey LAB-AA - Bankruptcy & Restructuring: 92.39 (#8/55)
- Harvey LAB-AA - International Trade & Sanctions: 91.67 (#8/55)
- ...and 104 more
Hy4 preview — Aggregate Rating 1851, #41
- AIMultiple - FinanceReasoning (Hard): 89.92 (#10/87)
- AIMultiple - Agentic RAG Routing: 82.61 (#13/72)
- AIMultiple - Text-to-SQL (Adjudicated): 77.6 (#16/72)
- Vals AI CyberBench: 65.36 (#29/44)
- AIMultiple - Text-to-SQL (Strict Execution Match): 39.4 (#35/72)
- AIIQ Composite IQ: 116.0 (#51/156)
Claude Opus 4.7 — Aggregate Rating 1850, #42
- Vals AI MortgageTax: 70.27 (#3/98)
- Vals AI Terminal-Bench 2.0: 68.54 (#3/67)
- NoteVQA: 43.7 (#6/10)
- SWE-Together: 81.28 (#7/20)
- PostTrainBench: 28.56 (#9/13)
- DGEval - Regulatory Recall: 20.2 (#9/35)
- DGEval - IMDG Code MCQ: 72.1 (#12/35)
- Lech Mazur Writing: 1.782 (#14/56)
- AA-AnalystAgent - Pass@1: 59.5 (#16/38)
- AA Terminal-Bench Hard: 54.55 (#17/441)
- ...and 50 more
Muse Spark 1.1 — Aggregate Rating 1845, #44
- AA GDP.pdf - Legal - Criterion Pass Rate: 84.9 (#8/204)
- Vals AI Time Horizon Index: 1.67 (#12/12)
- AA-Omniscience Index - Science, Engineering & Mathematics: 28.08 (#14/565)
- AA-Omniscience Index - Health: 21.02 (#16/565)
- Vals AI MMMU: 86.59 (#18/93)
- AA SciCode: 58.8 (#25/248)
- Vals AI Public Benefits Bench: 61.3 (#27/47)
- AA GDP.pdf - Real Estate - Criterion Pass Rate: 72.3 (#28/204)
- AA-Omniscience Index - Humanities & Social Sciences: 30.13 (#31/565)
- AIMultiple - Text-to-SQL (Adjudicated): 74.1 (#32/72)
- ...and 58 more
GPT-6 Luna — Aggregate Rating 1842, #45
- Vals AI CyberBench: 76.25 (#4/44)
- Coarena - Task Completion: 77.4 (#5/25)
- MLCR-AA - Accuracy: 93.75 (#5/94)
- ProphetArena: 0.958 (#6/16)
- Supabase Evals (Skills): 75.4 (#6/6)
- Supabase Evals (No Skills): 63.8 (#6/6)
- Coarena: 1000.2 (#9/25)
- EKI Estonian LLM - Idioms: 64.67 (#9/94)
- Epoch AI - Blueprint Bench 2: 0.312 (#11/28)
- SWE-Together: 79.96 (#12/20)
- ...and 108 more
Muse Spark 1.2 — Aggregate Rating 1842, #46
- EnterpriseOps-Gym-AA - Teams: 51.0 (#3/50)
- EnterpriseOps-Gym-AA - Hybrid: 42.48 (#6/50)
- EnterpriseOps-Gym-AA - ITSM: 43.65 (#6/50)
- EnterpriseOps-Gym-AA - Email: 66.35 (#8/50)
- EnterpriseOps-Gym-AA: 47.27 (#9/50)
- EnterpriseOps-Gym-AA - Calendar: 47.12 (#9/50)
- Vals AI Public Benefits Bench: 68.47 (#9/47)
- EnterpriseOps-Gym-AA - HR: 37.5 (#13/50)
- MLCR-AA - Completeness: 51.66 (#16/94)
- EnterpriseOps-Gym-AA - Drive: 60.0 (#16/50)
- ...and 77 more
Qwen 3.8 Flash — Aggregate Rating 1842, #47
- AIMultiple - FinanceReasoning (Hard): 87.39 (#25/87)
- AIMultiple - Text-to-SQL (Adjudicated): 75.6 (#25/72)
- AIMultiple - Agentic RAG Routing: 79.35 (#27/72)
- THOR Finding Triage - CW%: 58.7 (#31/85)
- THOR Finding Triage - Balanced OTS: 60.1 (#33/85)
- THOR Finding Triage - Critical Miss Rate: 0.0 (#38/85)
- AIMultiple - Text-to-SQL (Strict Execution Match): 38.0 (#40/72)
- NonoBench: 66.7 (#49/130)
- THOR Finding Triage - False Review Load: 48.09 (#56/85)
- AIIQ Composite IQ: 113.0 (#60/156)
- ...and 1 more
Gemini 3.1 Pro (Preview) — Aggregate Rating 1840, #48
- Vals AI AIME: 98.12 (#1/96)
- GAMUT: 58.7 (#1/14)
- DGEval - IMDG Code MCQ: 95.8 (#1/35)
- DGEval - Regulatory Recall: 36.8 (#1/35)
- NoteVQA: 52.8 (#1/10)
- Vals AI Terminal-Bench 2.0: 67.42 (#5/67)
- Vals AI MortgageTax: 69.4 (#6/98)
- MazeBench: 1.0 (#9/44)
- Vals AI MMMU: 88.21 (#10/93)
- PostTrainBench: 21.99 (#12/13)
- ...and 13 more
Claude Sonnet 5 — Aggregate Rating 1839, #50
- Vals AI MortgageTax: 70.03 (#4/98)
- EnterpriseOps-Gym-AA - Teams: 49.0 (#6/50)
- AA-AnalystAgent - Pass@K: 77.5 (#8/38)
- EnterpriseOps-Gym-AA - Calendar: 48.08 (#8/50)
- AA-AnalystAgent - Pass@1: 61.5 (#9/38)
- MLCR-AA: 55.0 (#9/94)
- Harvey LAB-AA - Corporate Governance: 92.12 (#9/55)
- MLCR-AA - Completeness: 68.24 (#10/94)
- EnterpriseOps-Gym-AA - Drive: 60.63 (#11/50)
- EnterpriseOps-Gym-AA - Email: 65.38 (#11/50)
- ...and 101 more
DeepSeek V4 Pro (0813) — Aggregate Rating 1839, #51
- EnterpriseOps-Gym-AA - Drive: 66.67 (#1/50)
- EnterpriseOps-Gym-AA - Hybrid: 45.75 (#1/50)
- Codeforces ELO: 3348.0 (#2/8)
- EnterpriseOps-Gym-AA: 49.6 (#4/50)
- EnterpriseOps-Gym-AA - Email: 68.27 (#4/50)
- MathArena Apex: 65.3 (#6/52)
- EnterpriseOps-Gym-AA - HR: 41.85 (#7/50)
- EnterpriseOps-Gym-AA - ITSM: 43.65 (#7/50)
- NL2Repo: 61.5 (#7/50)
- EnterpriseOps-Gym-AA - CSM: 51.61 (#8/50)
- ...and 73 more
Gemini 3.5 Flash — Aggregate Rating 1835, #52
- AA APEX-Agents: 47.05 (#1/35)
- EnterpriseOps-Gym-AA - Hybrid: 44.66 (#2/50)
- EnterpriseOps-Gym-AA: 50.07 (#3/50)
- EnterpriseOps-Gym-AA - ITSM: 47.33 (#3/50)
- Vals AI Terminal-Bench 2.0: 67.42 (#4/67)
- EnterpriseOps-Gym-AA - HR: 42.39 (#5/50)
- EnterpriseOps-Gym-AA - Drive: 62.22 (#6/50)
- EnterpriseOps-Gym-AA - CSM: 52.69 (#7/50)
- EnterpriseOps-Gym-AA - Teams: 48.67 (#7/50)
- Vals AI MMMU: 88.27 (#8/93)
- ...and 106 more
Qwen3.8 2.4T A95B — Aggregate Rating 1834, #53
- MCP Atlas: 84.5 (#8/72)
- AIMultiple - Text-to-SQL (Adjudicated): 77.5 (#17/72)
- AIMultiple - Agentic RAG Routing: 81.52 (#20/72)
- AIMultiple - Text-to-SQL (Strict Execution Match): 43.2 (#24/72)
- AIMultiple - FinanceReasoning (Hard): 86.97 (#34/87)
Claude Opus 4.6 — Aggregate Rating 1833, #54
- AA Global-MMLU-Lite - English: 95.17 (#1/111)
- AA Global-MMLU-Lite - Korean: 93.0 (#1/111)
- AA Global-MMLU-Lite - Chinese: 93.58 (#1/111)
- AA Global-MMLU-Lite - Bengali: 92.08 (#2/111)
- AA Global-MMLU-Lite - German: 93.42 (#2/111)
- AA Global-MMLU-Lite - Hindi: 91.92 (#2/111)
- AA Global-MMLU-Lite - Portuguese: 93.92 (#2/111)
- AA Global-MMLU-Lite: 92.15 (#3/111)
- AA Global-MMLU-Lite - Arabic: 92.25 (#3/111)
- AA Global-MMLU-Lite - Spanish: 94.0 (#3/111)
- ...and 50 more
Gemini 3 Pro (Preview) — Aggregate Rating 1825, #56
- AA MMLU-Pro: 89.76 (#1/349)
- AA LiveCodeBench: 91.75 (#1/343)
- Vals AI MATH 500: 96.4 (#1/60)
- AA Global-MMLU-Lite: 92.22 (#2/111)
- AA Global-MMLU-Lite - Arabic: 92.67 (#2/111)
- AA Global-MMLU-Lite - Spanish: 94.17 (#2/111)
- AA Global-MMLU-Lite - Indonesian: 93.5 (#2/111)
- AA Global-MMLU-Lite - Italian: 94.33 (#2/111)
- AA Global-MMLU-Lite - Japanese: 93.17 (#2/111)
- AA Global-MMLU-Lite - Swahili: 91.0 (#2/111)
- ...and 52 more
MiMo-V2.6-Flash — Aggregate Rating 1824, #57
- AI Coding Daily (OpenCode) - Laravel Code Quality: 17.83 (#2/18)
- JobBench: 61.2 (#3/8)
- AutomationBench-AA - Marketing - Raw Objective Completion: 90.12 (#6/213)
- Explicit Edit Benchmark: 98.3 (#7/25)
- AI Coding Daily (OpenCode) - Total: 53.29 (#8/18)
- AI Coding Daily (OpenCode) - React-TS Code Quality: 17.5 (#9/18)
- Vals AI ReverseEngBench: 4.2 (#10/16)
- SecIT Bench (Pydantic AI): 75.98 (#10/25)
- AutomationBench-AA - Sales - Raw Objective Completion: 86.26 (#13/213)
- Vals AI BioMysteryBench: 69.26 (#13/23)
- ...and 88 more
GPT-5.6 Luna — Aggregate Rating 1822, #58
- EnterpriseOps-Gym-AA - Calendar: 49.04 (#5/50)
- AA APEX-Agents: 35.84 (#7/35)
- Vals AI SkillsBench: 60.45 (#7/35)
- Vals AI CyberBench: 73.63 (#9/44)
- EnterpriseOps-Gym-AA - CSM: 49.46 (#12/50)
- K-Bench - Risk: 94.05 (#12/62)
- AA-LCR: 83.67 (#12/202)
- AA GDP.pdf - Legal - Criterion Pass Rate: 84.07 (#18/204)
- Harvey LAB-AA - Corporate Governance: 89.73 (#19/55)
- Harvey LAB-AA - Energy & Natural Resources: 95.71 (#20/55)
- ...and 102 more
GPT-5.3 Codex — Aggregate Rating 1819, #60
- Vals AI Terminal-Bench 2.0: 64.05 (#6/67)
- AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 84.0 (#14/565)
- AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 82.73 (#16/565)
- AA-LCR: 83.33 (#16/202)
- AA Terminal-Bench Hard: 53.03 (#20/441)
- AA-Omniscience Index - Software Engineering (SWE) - Julia: 76.0 (#24/565)
- AA-Omniscience Index - Software Engineering (SWE) - Python: 80.5 (#24/565)
- AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 86.67 (#25/565)
- AA Long Context Reasoning: 83.33 (#27/599)
- OpenRouter Tau2-Bench Airline: 75.3 (#31/143)
- ...and 32 more
GLM-5.3 Flash — Aggregate Rating 1818, #61
- NoteVQA: 48.0 (#3/10)
- MultiChallenge: 62.6 (#6/7)
- Finance Agent v2: 59.26 (#7/50)
- Chatbot Arena (Vision - Chinese): 1349.0 (#8/123)
- SWE Atlas: 61.29 (#8/58)
- K-Bench: 98.36 (#12/62)
- FrontierSWE V2: 18.13 (#15/20)
- KernelBench Mega: 13.642 (#17/53)
- Vals AI MMMU: 86.01 (#23/93)
- ReactBench: 31.3 (#23/72)
- ...and 19 more
GLM-5.2 — Aggregate Rating 1816, #62
- AA TAU-2 Bench: 99.12 (#1/449)
- EnterpriseOps-Gym-AA - Drive: 61.9 (#7/50)
- PostTrainBench: 31.7 (#8/13)
- AA APEX-Agents: 33.7 (#8/35)
- Harvey LAB-AA - Corporate Governance: 92.47 (#8/55)
- Harvey LAB-AA - Healthcare & Life Sciences: 93.73 (#8/55)
- Harvey LAB-AA - Antitrust & Competition: 92.39 (#10/55)
- Harvey LAB-AA - Energy & Natural Resources: 96.7 (#11/55)
- Harvey LAB-AA - Funds & Asset Management: 90.38 (#11/55)
- EnterpriseOps-Gym-AA - Teams: 47.33 (#12/50)
- ...and 98 more
Qwen3.8-Flash-Next — Aggregate Rating 1816, #63
- BRIDGE Medical Leaderboard: 49.39 (#1/112)
- BRIDGE Medical Leaderboard - Zero-Shot: 46.95 (#1/112)
- BRIDGE Medical Leaderboard - Few-Shot: 56.03 (#1/112)
- BRIDGE Medical Leaderboard - CoT: 45.92 (#1/112)
- ERQA: 72.3 (#1/5)
- IFBench: 81.3 (#2/6)
- NoteVQA: 44.0 (#5/10)
DeepSeek V4 Flash (0731) — Aggregate Rating 1812, #66
- Codeforces ELO: 3289.0 (#3/8)
- IFBench: 79.2 (#4/6)
- MathArena Apex: 58.6 (#8/52)
- Vals AI CyberBench: 72.08 (#14/44)
- NL2Repo: 54.2 (#16/50)
- AIMultiple - Agentic RAG Routing: 82.07 (#17/72)
- MazeBench: 0.0 (#17/44)
- KernelBench Mega: 5.24 (#24/53)
- Vals AI Tax Agent Bench: 62.56 (#25/65)
- AIMultiple - FinanceReasoning (Hard): 87.39 (#28/87)
- ...and 67 more
DeepSeek-V4-Flash-Vision-Exp — Aggregate Rating 1812, #67
- FrontierSWE V2: 14.83 (#18/20)
- OSWorld-Verified: 76.7 (#28/141)
- AIIQ Composite IQ: 122.0 (#40/156)
Claude Opus 4.5 (20251101) — Aggregate Rating 1811, #69
- Vals AI Terminal-Bench 2.0: 58.43 (#10/67)
- Vals AI MortgageTax: 68.68 (#14/98)
- Vals AI MMMU: 82.95 (#36/93)
GPT-5.4 — Aggregate Rating 1809, #71
- AA APEX-Agents: 33.26 (#9/35)
- TempCloze: 56.89 (#9/32)
- K-Bench: 98.41 (#10/62)
- PostTrainBench: 19.0 (#13/13)
- AA Terminal-Bench Hard: 57.58 (#14/441)
- Vellum - GPQA: 92.8 (#14/72)
- K-Bench - Risk: 93.93 (#14/62)
- Vellum - Humanity's Last Exam: 52.1 (#15/41)
- AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 80.91 (#20/565)
- Chatbot Arena (Vision - Entity Recognition): 1215.0 (#23/50)
- ...and 33 more
Qwen 3.7 Max — Aggregate Rating 1808, #73
- Vals AI Tax Agent Bench: 47.41 (#44/65)
Claude Opus 4.5 — Aggregate Rating 1805, #74
- VibeCodingBench: 89.15 (#1/15)
Gemini 3 Flash (Preview) — Aggregate Rating 1802, #75
- Vals AI AIME: 95.62 (#8/96)
- Vals AI MGSM: 93.31 (#10/75)
- Vals AI MortgageTax: 68.72 (#11/98)
- Vals AI MMMU: 87.63 (#13/93)
- Vals AI Terminal-Bench 2.0: 51.69 (#18/67)
- Medical STT Benchmark: 0.1133 (#19/42)
- Vals AI CaseLaw v2: 55.84 (#33/54)
- Vals AI Tax Agent Bench: 37.56 (#51/65)
- Vals AI ProgramBench: 0.0 (#53/62)
Ling 3.1 Flash — Aggregate Rating 1801, #77
- MultiChallenge: 69.78 (#2/7)
- AutomationBench-AA - Marketing - Raw Objective Completion: 89.72 (#7/213)
- AutomationBench-AA - Finance - Raw Objective Completion: 90.3 (#9/213)
- AutomationBench-AA - HR - Raw Objective Completion: 76.18 (#10/213)
- Finance Agent v2: 57.87 (#11/50)
- SWE Atlas: 55.92 (#18/58)
- AA-LCR: 83.0 (#19/202)
- AA GDPval: 1621.75 (#22/282)
- AutomationBench-AA: 61.74 (#24/213)
- AA Long Context Reasoning: 83.0 (#35/599)
- ...and 50 more
Kimi K2.7 Code — Aggregate Rating 1796, #80
- Sci-MMR (Direct Visual Reasoning): 50.6 (#3/8)
- Sci-MMR (Agentic Tool-Use): 52.8 (#3/8)
- GBA Eval: 0.0085 (#18/23)
- GBA Eval - Gameplay Replays: 0.0142 (#18/21)
- GBA Eval - Procedural Tests: 0.0 (#18/21)
- GBA Eval - Audio: 0.0 (#18/21)
- KernelBench Mega: 2.586 (#37/53)
Claude Sonnet 4.6 — Aggregate Rating 1795, #81
- AA Global-MMLU-Lite - Bengali: 90.92 (#5/111)
- AA Global-MMLU-Lite - Japanese: 92.58 (#5/111)
- AA Global-MMLU-Lite - Portuguese: 92.5 (#5/111)
- AA Global-MMLU-Lite - Italian: 92.92 (#6/111)
- AA Global-MMLU-Lite - French: 92.75 (#7/111)
- AA Global-MMLU-Lite - Korean: 91.08 (#8/111)
- Vals AI Terminal-Bench 2.0: 59.55 (#8/67)
- AA Global-MMLU-Lite - Indonesian: 91.5 (#9/111)
- AA Global-MMLU-Lite: 90.48 (#10/111)
- AA Global-MMLU-Lite - Arabic: 90.17 (#10/111)
- ...and 114 more
Qwen 3.7 Plus — Aggregate Rating 1792, #83
- RealWorldQA: 86.9 (#3/51)
- ERQA: 69.8 (#3/5)
- IFBench: 79.1 (#5/6)
- LVBench: 76.2 (#13/49)
- OpenRouter GPQA Diamond: 87.9 (#42/154)
- Vals AI Tax Agent Bench: 38.71 (#49/65)
- OpenRouter Tau2-Bench Airline: 70.3 (#74/143)
GPT-5.2 Codex — Aggregate Rating 1792, #84
- AA IFBench: 77.62 (#17/459)
- BenchGecko Score: 80.7 (#20/20)
- AA-LCR: 82.33 (#24/202)
- AA Long Context Reasoning: 82.33 (#50/599)
- AA TAU-2 Bench: 92.11 (#58/449)
- AA GPQA Diamond: 89.9 (#78/636)
- AA Terminal-Bench Hard: 37.12 (#79/441)
- AA MMMU-Pro: 76.3 (#101/304)
- AA-Omniscience Accuracy: 41.07 (#113/563)
- AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 26.0 (#114/565)
- ...and 24 more
Ling-3.0-flash-fin — Aggregate Rating 1791, #86
- FrontierFinance - Company Research: 47.79 (#2/13)
- FrontierFinance - Earnings & Events: 61.33 (#3/13)
- FrontierFinance - Coverage & Catalyst Monitoring: 48.24 (#3/13)
- FrontierFinance: 46.59 (#5/13)
- FrontierFinance - Sector, Industry & Macro: 36.67 (#5/13)
- FrontierFinance - Screening & Discovery: 31.83 (#8/13)
- FrontierFinance - Financial Data & Modeling: 46.81 (#9/13)
- OpenRouter Tau2-Bench Airline: 70.7 (#71/143)
- AA-LCR: 73.67 (#106/202)
GPT-5 — Aggregate Rating 1789, #87
- Vals AI MATH 500: 96.0 (#3/60)
- AA Global-MMLU-Lite - Bengali: 91.33 (#4/111)
- AA Global-MMLU-Lite - Swahili: 90.17 (#4/111)
- AA Global-MMLU-Lite - French: 93.17 (#5/111)
- AA Global-MMLU-Lite - Hindi: 91.08 (#5/111)
- AA Global-MMLU-Lite: 90.67 (#6/111)
- AA Global-MMLU-Lite - German: 92.5 (#6/111)
- AA Global-MMLU-Lite - Yoruba: 82.83 (#6/111)
- Vals AI CaseLaw v2: 66.45 (#6/54)
- AA Global-MMLU-Lite - Indonesian: 92.08 (#7/111)
- ...and 52 more
QRM-Gemma-2-27B — Aggregate Rating 1788, #88
- RewardBench 2 Safety: 95.78 (#6/249)
- RewardBench 2 Focus: 95.35 (#11/249)
- RewardBench 2 Factuality: 78.53 (#26/249)
- RewardBench 2 Math: 69.95 (#64/249)
- RewardBench 2 Precise IF: 37.19 (#138/249)
Gemini 3 Flash — Aggregate Rating 1786, #89
- TempCloze: 49.86 (#12/32)
- VibeCodingBench: 83.44 (#15/15)
Seed 2.1 Turbo — Aggregate Rating 1786, #90
- NonoBench: 53.3 (#62/130)
Hy3 — Aggregate Rating 1784, #91
- AIMultiple - FinanceReasoning (Hard): 87.39 (#32/87)
- Chatbot Arena (Text - German): 1457.0 (#58/307)
INF-ORM-Llama3.1-70B — Aggregate Rating 1784, #92
- RewardBench 2 Safety: 96.44 (#4/249)
- RewardBench 2 Focus: 90.3 (#42/249)
- RewardBench 2 Factuality: 74.11 (#56/249)
- RewardBench 2 Math: 69.95 (#63/249)
- RewardBench 2 Precise IF: 41.88 (#70/249)
Qwen 3.8 27B — Aggregate Rating 1781, #94
- IFBench: 79.5 (#3/6)
- ERQA: 65.5 (#4/5)
- Tau3 Banking: 48.04 (#6/224)
- EnterpriseOps-Gym-AA - Email: 67.31 (#6/50)
- MLCR-AA - Accuracy: 89.29 (#13/94)
- EnterpriseOps-Gym-AA: 44.23 (#16/50)
- EnterpriseOps-Gym-AA - HR: 36.41 (#16/50)
- EnterpriseOps-Gym-AA - ITSM: 37.57 (#17/50)
- EnterpriseOps-Gym-AA - Hybrid: 38.56 (#18/50)
- EnterpriseOps-Gym-AA - CSM: 45.7 (#24/50)
- ...and 80 more
Composer 2.5 — Aggregate Rating 1781, #96
- KernelBench Mega: 2.482 (#41/53)
DeepSeek V4 Pro — Aggregate Rating 1780, #97
- Explicit Edit Benchmark: 99.7 (#3/25)
- Terminal-Bench-Science: 3.8095 (#16/17)
- NonoBench: 83.3 (#18/130)
- Vals AI Public Benefits Bench: 62.92 (#23/47)
- Vals AI CaseLaw v2: 59.38 (#27/54)
- Vals AI Terminal-Bench 4.0: 11.11 (#30/43)
- Vals AI Tax Agent Bench: 58.5 (#37/65)
- Vals AI ProofBench: 16.0 (#41/48)
- Vals AI ProgramBench: 0.0 (#52/62)
Kimi K2.6 — Aggregate Rating 1780, #98
- NoteVQA: 47.6 (#4/10)
- AA APEX-Agents: 28.47 (#14/35)
- Harvey LAB-AA - Corporate Governance: 90.41 (#16/55)
- AA TAU-2 Bench: 95.91 (#18/449)
- K-Bench: 98.21 (#19/62)
- EnterpriseOps-Gym-AA - Hybrid: 37.04 (#23/50)
- AA IFBench: 75.99 (#24/459)
- EnterpriseOps-Gym-AA - Teams: 44.67 (#25/50)
- Harvey LAB-AA - Tax: 83.95 (#26/55)
- EnterpriseOps-Gym-AA - Calendar: 39.74 (#27/50)
- ...and 93 more
GPT-5.2 — Aggregate Rating 1779, #99
- AA AIME 2025: 99.0 (#1/270)
- Vals AI AIME: 96.88 (#2/96)
- VibeCodingBench: 88.75 (#4/15)
- AA LiveCodeBench: 89.42 (#4/343)
- K-Bench: 98.59 (#4/62)
- K-Bench - Risk: 94.71 (#5/62)
- AA Global-MMLU-Lite - Bengali: 90.92 (#6/111)
- Vals AI MGSM: 94.0 (#6/75)
- AA Global-MMLU-Lite - Italian: 92.58 (#8/111)
- Vals AI CaseLaw v2: 66.02 (#8/54)
- ...and 52 more
Qwen 3.6 Plus — Aggregate Rating 1775, #102
- Vals AI Tax Agent Bench: 32.84 (#56/65)
GPT-5.4 (2026-03-05) — Aggregate Rating 1773, #103
- Vals AI AIME: 96.67 (#5/96)
- Vals AI Terminal-Bench 2.0: 58.43 (#12/67)
- Vals AI MMMU: 87.51 (#15/93)
- Vals AI CaseLaw v2: 63.77 (#16/54)
- Vals AI MortgageTax: 68.32 (#17/98)
- Vals AI ProgramBench: 0.0 (#38/62)
Seed 2.0 Pro — Aggregate Rating 1771, #107
- NoteVQA: 49.6 (#2/10)
Grok 4.20 0309 — Aggregate Rating 1770, #109
- Vals AI Tax Agent Bench: 31.24 (#57/65)
Inkling — Aggregate Rating 1766, #112
- AA-AnalystAgent - Pass@K: 78.75 (#6/38)
- EnterpriseOps-Gym-AA - Email: 62.5 (#17/50)
- FrontierSWE V2: 4.12 (#20/20)
- AA-AnalystAgent - Pass@1: 52.25 (#22/38)
- EnterpriseOps-Gym-AA - Drive: 58.1 (#24/50)
- EnterpriseOps-Gym-AA - Hybrid: 36.6 (#27/50)
- Vals AI CyberBench: 66.79 (#27/44)
- EnterpriseOps-Gym-AA - Teams: 43.0 (#30/50)
- AA-AnalystAgent: 23.75 (#31/47)
- EnterpriseOps-Gym-AA - Calendar: 36.54 (#31/50)
- ...and 70 more
O3 (2025-04-16) — Aggregate Rating 1765, #113
- Vals AI MATH 500: 94.6 (#8/60)
- Vals AI MGSM: 91.75 (#26/75)
- Vals AI AIME: 85.28 (#37/96)
- Vals AI MortgageTax: 65.7 (#40/98)
- Vals AI MMMU: 80.42 (#44/93)
GPT-5.1 — Aggregate Rating 1761, #115
- AA Global-MMLU-Lite - French: 93.5 (#2/111)
- Vals AI CaseLaw v2: 73.42 (#2/54)
- AA Global-MMLU-Lite - English: 94.08 (#5/111)
- AA Global-MMLU-Lite - Bengali: 90.92 (#7/111)
- AA Global-MMLU-Lite - Hindi: 90.75 (#7/111)
- AA Global-MMLU-Lite - Swahili: 89.58 (#7/111)
- AA Global-MMLU-Lite - Yoruba: 81.42 (#7/111)
- AA Global-MMLU-Lite - Chinese: 91.33 (#8/111)
- AA LiveCodeBench: 86.77 (#9/343)
- AA Global-MMLU-Lite: 90.56 (#9/111)
- ...and 51 more
Claude Sonnet 4.5 — Aggregate Rating 1759, #117
- VibeCodingBench: 88.56 (#6/15)
Qwen 3.5 Plus — Aggregate Rating 1759, #119
- TempCloze: 85.74 (#2/32)
MiniMax-M3 — Aggregate Rating 1757, #120
- KernelBench Mega: 4.151 (#29/53)
- Vals AI CyberBench: 65.0 (#30/44)
- Vals AI Terminal-Bench 4.0: 1.01 (#39/43)
DeepSeek V4 Pro Preview — Aggregate Rating 1757, #122
- KernelBench Mega: 4.645 (#25/53)
GPT-5 Codex — Aggregate Rating 1755, #123
- AA AIME 2025: 98.67 (#2/270)
- AA MMLU-Pro: 86.49 (#16/349)
- AA LiveCodeBench: 84.02 (#17/343)
- AA IFBench: 74.15 (#37/459)
- AA Terminal-Bench Hard: 37.88 (#73/441)
- AA TAU-2 Bench: 86.84 (#91/449)
- AA-Omniscience Index - Software Engineering (SWE) - Swift: 40.0 (#117/565)
- AA-Omniscience Index - Software Engineering (SWE) - Java: -5.0 (#122/565)
- AA-Omniscience Accuracy: 38.7 (#128/563)
- AA-Omniscience Index - Business: -4.2 (#128/565)
- ...and 26 more
LDL-Reward-Gemma-2-27B-v0.1 — Aggregate Rating 1755, #124
- RewardBench 2 Safety: 92.22 (#22/249)
- RewardBench 2 Focus: 91.31 (#32/249)
- RewardBench 2 Factuality: 75.58 (#42/249)
- RewardBench 2 Math: 64.48 (#88/249)
- RewardBench 2 Precise IF: 35.0 (#169/249)
Qwen 3.5 397B A17B — Aggregate Rating 1753, #127
- TempCloze: 68.27 (#6/32)
- NoteVQA: 35.7 (#9/10)
Inkling Small — Aggregate Rating 1753, #128
- Vals AI CyberBench: 64.11 (#31/44)
- Vals AI Tax Agent Bench: 41.46 (#48/65)
- AA GDPval: 1124.45 (#122/282)
DeepSeek V4 Flash — Aggregate Rating 1752, #129
- K-Bench - Risk: 87.91 (#42/62)
- K-Bench: 96.96 (#45/62)
Kimi K2.5 — Aggregate Rating 1750, #131
- TempCloze: 59.41 (#8/32)
- Lech Mazur Writing: -1.071 (#40/56)
Gemini 2.5 Pro — Aggregate Rating 1748, #133
- Medical STT Benchmark: 0.0815 (#1/42)
- TempCloze: 74.92 (#4/32)
- RewardBench 2 Math: 89.8 (#10/249)
- RewardBench 2 Precise IF: 61.9 (#26/249)
- RewardBench 2 Factuality: 75.5 (#44/249)
- RewardBench 2 Safety: 88.1 (#60/249)
- RewardBench 2 Focus: 80.5 (#107/249)
MiMo-V2.5-Pro — Aggregate Rating 1744, #135
- JobBench: 25.0 (#8/8)
- AA IFBench: 79.86 (#10/459)
- AA-Omniscience Hallucination Rate: 24.7 (#21/565)
- AA-AnalystAgent - Pass@1: 47.0 (#25/38)
- AA-AnalystAgent - Pass@K: 68.75 (#27/38)
- AA APEX-Agents: 2.43 (#33/35)
- AA-AnalystAgent: 20.0 (#33/47)
- AA TAU-2 Bench: 94.15 (#37/449)
- Vals AI Terminal-Bench 4.0: 0.51 (#41/43)
- Harvey LAB-AA - Antitrust & Competition: 77.54 (#42/55)
- ...and 89 more
Gemini 3.5 Flash Lite — Aggregate Rating 1744, #136
- Vals AI MortgageTax: 68.68 (#12/98)
- Vals AI MMMU: 83.58 (#30/93)
- Vals AI CyberBench: 59.11 (#34/44)
- Vals AI Public Benefits Bench: 51.83 (#41/47)
- Vals AI Tax Agent Bench: 37.45 (#52/65)
- Vals AI ProgramBench: 0.0 (#54/62)
- AIMultiple - FinanceReasoning (Hard): 78.15 (#64/87)
O1 (2024-12-17) — Aggregate Rating 1742, #139
- Vals AI MATH 500: 90.4 (#25/60)
- Vals AI MGSM: 89.31 (#48/75)
- Vals AI AIME: 71.46 (#53/96)
- Vals AI MMMU: 77.41 (#53/93)
Claude Opus 4 (20250514) — Aggregate Rating 1742, #140
- RewardBench 2 Factuality: 82.67 (#14/249)
- RewardBench 2 Math: 74.91 (#43/249)
- RewardBench 2 Safety: 89.54 (#45/249)
- RewardBench 2 Precise IF: 41.88 (#65/249)
- RewardBench 2 Focus: 86.16 (#75/249)
Gemini 2.5 Pro (Preview 05-06) — Aggregate Rating 1740, #142
- RewardBench 2 Precise IF: 46.88 (#42/249)
- RewardBench 2 Safety: 88.06 (#61/249)
- RewardBench 2 Focus: 83.08 (#95/249)
- RewardBench 2 Factuality: 65.32 (#150/249)
- RewardBench 2 Math: 53.42 (#213/249)
Quasar 438B — Aggregate Rating 1740, #143
- AA-Omniscience Hallucination Rate: 21.44 (#13/565)
- AA-Briefcase - XLSX - Rubric Pass Rate: 35.45 (#79/216)
- AA-Briefcase - PDF - Rubric Pass Rate: 37.93 (#80/216)
- AA-LCR: 76.33 (#84/202)
- AA-Briefcase: 34.34 (#85/216)
- AA-Briefcase - Presentation Elo: 1186.68 (#85/216)
- Tau3 Banking: 28.66 (#90/224)
- AA-Briefcase - DOCX - Rubric Pass Rate: 33.73 (#91/216)
- AA-Briefcase - Analytical Quality Elo: 1067.77 (#95/216)
- BenchLM: 49.8 (#95/200)
- ...and 50 more
Claude Opus 4 — Aggregate Rating 1738, #144
- AA-LCR: 69.33 (#147/202)
GPT-5.1 Codex — Aggregate Rating 1738, #145
- AA AIME 2025: 95.67 (#8/270)
- AA Global-MMLU-Lite - French: 92.75 (#8/111)
- AA Global-MMLU-Lite - Hindi: 90.58 (#8/111)
- AA Global-MMLU-Lite - Swahili: 89.5 (#8/111)
- AA Global-MMLU-Lite - Japanese: 91.08 (#10/111)
- AA Global-MMLU-Lite - Spanish: 92.75 (#12/111)
- AA Global-MMLU-Lite - Bengali: 89.92 (#13/111)
- AA Global-MMLU-Lite - Yoruba: 79.75 (#13/111)
- AA LiveCodeBench: 84.87 (#15/343)
- AA Global-MMLU-Lite: 89.7 (#15/111)
- ...and 43 more
Grok 4.20 — Aggregate Rating 1735, #148
- K-Bench: 98.32 (#16/62)
- K-Bench - Risk: 91.14 (#31/62)
Hy3-preview — Aggregate Rating 1734, #150
- KernelBench Mega: 0.31 (#51/53)
Qwen 3.6 27B — Aggregate Rating 1730, #151
- Medmarks - LongHealth Task 2: 91.33 (#1/89)
- Medmarks - PubMedQA: 81.13 (#2/89)
- Medmarks - MedCalc-Bench: 81.64 (#3/89)
- Medmarks - SuperGPQA Medicine Easy: 67.77 (#3/89)
- Medmarks - MedHallu Hard: 62.99 (#4/89)
- Medmarks - MedHallu Medium: 78.62 (#4/89)
- Medmarks - SuperGPQA Medicine Hard: 57.45 (#4/89)
- Medmarks - MedHallu Easy: 82.67 (#5/89)
- Medmarks - Medbullets OP4: 87.99 (#6/89)
- Medmarks - HEAD-QA v2: 90.58 (#9/89)
- ...and 17 more
Seed 1.8 — Aggregate Rating 1730, #152
- TempCloze: 88.58 (#1/32)
Grok 4 — Aggregate Rating 1728, #156
- VibeCodingBench: 88.0 (#9/15)
Llama 3.1 70B Instruct RM RB2 — Aggregate Rating 1725, #161
- RewardBench 2 Factuality: 81.26 (#18/249)
- RewardBench 2 Safety: 88.44 (#54/249)
- RewardBench 2 Math: 69.95 (#60/249)
- RewardBench 2 Precise IF: 41.88 (#67/249)
- RewardBench 2 Focus: 86.46 (#72/249)
Gemma 4 31B (IT) — Aggregate Rating 1723, #163
- Medmarks - MedHallu Easy: 84.12 (#2/89)
- Medmarks - MedHallu Hard: 64.22 (#2/89)
- Medmarks - MedHallu Medium: 80.35 (#2/89)
- Medmarks - MedXpertQA Reasoning: 56.6 (#3/89)
- Medmarks - MedXpertQA Understanding: 50.54 (#3/89)
- Medmarks - SuperGPQA Medicine Hard: 60.22 (#3/89)
- Medmarks - PubHealthBench Reviewed: 93.11 (#4/89)
- Medmarks - SCTpublic: 74.08 (#6/89)
- Medmarks - PubMedQA: 79.73 (#7/89)
- Medmarks - MedMCQA: 78.67 (#9/89)
- ...and 25 more
Muse Glimmer 30B — Aggregate Rating 1723, #164
- Medmarks - M-ARC: 86.67 (#1/89)
- Medmarks - Med-HALT Reasoning FCT: 91.57 (#4/89)
- Medmarks - Med-HALT Reasoning NOTA: 71.63 (#6/89)
- Medmarks - MetaMedQA: 79.95 (#6/89)
- Medmarks - MedConceptsQA Hard: 83.68 (#7/89)
- Medmarks - MedConceptsQA Medium: 88.45 (#7/89)
- Medmarks - MedConceptsQA Easy: 99.28 (#10/89)
- Medmarks - SCTpublic: 71.91 (#12/89)
- Medmarks - MedCalc-Bench: 72.0 (#13/89)
- Medmarks - Medbullets OP4: 86.04 (#14/89)
- ...and 95 more
GPT-5 Mini — Aggregate Rating 1722, #165
- Vals AI CaseLaw v2: 68.49 (#4/54)
- Vals AI MATH 500: 94.8 (#6/60)
- Vals AI MGSM: 92.58 (#16/75)
- AA LiveCodeBench: 83.81 (#18/343)
- AA Global-MMLU-Lite - Korean: 88.67 (#22/111)
- AA AIME 2025: 90.67 (#23/270)
- AA Global-MMLU-Lite - Arabic: 88.5 (#23/111)
- AA Global-MMLU-Lite - Italian: 91.0 (#23/111)
- Vals AI AIME: 91.46 (#24/96)
- AA Global-MMLU-Lite: 87.43 (#25/111)
- ...and 80 more
O4 Mini (2025-04-16) — Aggregate Rating 1722, #166
- Vals AI MGSM: 93.42 (#9/75)
- Vals AI MATH 500: 94.2 (#10/60)
- Vals AI AIME: 83.67 (#41/96)
- Vals AI MortgageTax: 64.83 (#46/98)
- Vals AI MMMU: 79.67 (#46/93)
MiMo-V2.5 — Aggregate Rating 1722, #167
- Vals AI Tax Agent Bench: 29.16 (#58/65)
GPT-5.4 Mini — Aggregate Rating 1720, #170
- MLCR-AA - Conciseness: 99.44 (#5/94)
- AA Global-MMLU-Lite - Burmese: 88.42 (#8/111)
- DGEval - IMDG Code MCQ: 76.1 (#9/35)
- Vals AI AIME: 95.62 (#10/96)
- AA Global-MMLU-Lite - Swahili: 88.33 (#12/111)
- AA Global-MMLU-Lite - Chinese: 90.92 (#14/111)
- AA APEX-Agents: 28.17 (#15/35)
- AA Global-MMLU-Lite - Bengali: 89.42 (#16/111)
- AA Global-MMLU-Lite - Indonesian: 90.83 (#17/111)
- AA Global-MMLU-Lite - Korean: 89.08 (#17/111)
- ...and 125 more
Qwen 3.5 27B — Aggregate Rating 1719, #173
- Medmarks - MedQA: 94.61 (#3/89)
- Medmarks - PubMedQA: 80.93 (#3/89)
- Medmarks - LongHealth Task 1: 91.42 (#5/89)
- Medmarks - Med-HALT Reasoning FCT: 90.87 (#6/89)
- Medmarks - SuperGPQA Medicine Easy: 67.07 (#7/89)
- Medmarks - SuperGPQA Medicine Hard: 54.69 (#7/89)
- Medmarks - CareQA EN: 93.44 (#8/89)
- Medmarks - HEAD-QA v2: 91.04 (#8/89)
- Medmarks - Medbullets OP5: 83.87 (#8/89)
- Medmarks - MetaMedQA: 79.66 (#8/89)
- ...and 17 more
Grok 4.3 — Aggregate Rating 1717, #176
- AA IFBench: 83.33 (#1/459)
- AA-Omniscience Hallucination Rate: 16.94 (#7/565)
- MLCR-AA - Conciseness: 98.89 (#9/94)
- AA TAU-2 Bench: 97.66 (#12/449)
- AA APEX-Agents: 17.04 (#23/35)
- AA-AnalystAgent - Pass@1: 31.75 (#31/38)
- AA-AnalystAgent - Pass@K: 57.5 (#31/38)
- EnterpriseOps-Gym-AA - Teams: 37.67 (#36/50)
- K-Bench - Risk: 88.05 (#41/62)
- EnterpriseOps-Gym-AA - Email: 53.85 (#42/50)
- ...and 101 more
Motif 3 — Aggregate Rating 1716, #177
- AA GDPval: 1122.7 (#125/282)
O4 Mini — Aggregate Rating 1715, #178
- AA LiveCodeBench: 85.93 (#12/343)
- AA AIME 2025: 90.67 (#24/270)
- AA MMLU-Pro: 83.19 (#58/349)
- AA IFBench: 68.71 (#80/459)
- AA MMMU-Pro: 69.25 (#170/304)
- AA-LCR: 61.0 (#179/202)
- AA TAU-2 Bench: 55.56 (#205/449)
- AA Terminal-Bench Hard: 15.15 (#214/441)
- AA-Omniscience Accuracy: 24.8 (#233/563)
- AA-Omniscience Index - Software Engineering (SWE) - Swift: 0.0 (#237/565)
- ...and 26 more
GLM-4.7 — Aggregate Rating 1714, #179
- VibeCodingBench: 83.9 (#13/15)
DeepSeek V3.2 — Aggregate Rating 1712, #183
- Text-to-SQL under RBAC: 95.8 (#1/13)
- VibeCodingBench: 88.19 (#8/15)
- Vals AI CaseLaw v2: 55.41 (#34/54)
- Vals AI AIME: 84.58 (#39/96)
- Vals AI Terminal-Bench 2.0: 35.95 (#41/67)
Gemini 3.1 Flash Lite — Aggregate Rating 1712, #184
- DGEval - IMDG Code MCQ: 79.2 (#7/35)
- DGEval - Regulatory Recall: 22.0 (#7/35)
- K-Bench - Risk: 93.31 (#18/62)
- K-Bench: 98.13 (#21/62)
Solar Mini 4 — Aggregate Rating 1711, #185
- AA-LCR: 83.33 (#13/202)
- AA Long Context Reasoning: 83.33 (#28/599)
- AA-Omniscience Hallucination Rate: 35.83 (#63/565)
- AA-Briefcase - PDF - Rubric Pass Rate: 31.03 (#104/216)
- AA-Briefcase - DOCX - Rubric Pass Rate: 27.22 (#119/216)
- AutomationBench-AA - HR - Raw Objective Completion: 41.4 (#120/213)
- AutomationBench-AA: 22.25 (#121/213)
- AutomationBench-AA - Marketing - Raw Objective Completion: 38.74 (#127/213)
- AA-Briefcase: 22.42 (#128/216)
- AutomationBench-AA - Strict: 2.44 (#128/213)
- ...and 48 more
Gemini 3.1 Flash Lite (Preview) — Aggregate Rating 1710, #186
- Vals AI MortgageTax: 68.04 (#20/98)
- Vals AI CaseLaw v2: 54.98 (#35/54)
- Vals AI MMMU: 82.49 (#37/93)
- Vals AI AIME: 83.33 (#43/96)
- Vals AI Terminal-Bench 2.0: 24.72 (#53/67)
- Vals AI ProgramBench: 0.0 (#61/62)
- Vals AI Tax Agent Bench: 20.09 (#63/65)
Ling-3.0-flash-VL — Aggregate Rating 1710, #187
- AA-LCR: 78.33 (#66/202)
- BenchLM: 47.7 (#105/200)
Claude 3.7 Sonnet (20250219) — Aggregate Rating 1709, #190
- RewardBench 2 Focus: 92.12 (#26/249)
- RewardBench 2 Precise IF: 54.37 (#33/249)
- RewardBench 2 Safety: 90.33 (#37/249)
- RewardBench 2 Math: 75.0 (#42/249)
- RewardBench 2 Factuality: 73.26 (#63/249)
Gemma 4 31B — Aggregate Rating 1708, #192
- Medmarks - SuperGPQA Medicine Hard: 62.52 (#1/89)
- Medmarks - MedXpertQA Reasoning: 59.77 (#2/89)
- Medmarks - MedXpertQA Understanding: 53.54 (#2/89)
- Medmarks - MedHallu Hard: 63.85 (#3/89)
- Medmarks - MedHallu Medium: 79.25 (#3/89)
- Medmarks - MedHallu Easy: 83.76 (#4/89)
- Medmarks - MedQA: 94.19 (#4/89)
- Medmarks - Medbullets OP5: 84.96 (#5/89)
- Medmarks - PubHealthBench Reviewed: 92.98 (#5/89)
- Medmarks - PubMedQA: 80.47 (#5/89)
- ...and 31 more
Claude Sonnet 4 (20250514) — Aggregate Rating 1708, #193
- RewardBench 2 Factuality: 76.12 (#41/249)
- RewardBench 2 Safety: 89.09 (#51/249)
- RewardBench 2 Math: 70.49 (#56/249)
- RewardBench 2 Focus: 75.96 (#128/249)
- RewardBench 2 Precise IF: 35.94 (#157/249)
Grok 4.1 Fast — Aggregate Rating 1707, #196
- VibeCodingBench: 86.8 (#11/15)
Skywork-Reward-Gemma-2-27B-v0.2 — Aggregate Rating 1706, #202
- RewardBench 2 Safety: 96.89 (#2/249)
- RewardBench 2 Focus: 91.72 (#29/249)
- RewardBench 2 Factuality: 76.74 (#37/249)
- RewardBench 2 Math: 67.21 (#77/249)
- RewardBench 2 Precise IF: 37.5 (#126/249)
Qwen 3 Max — Aggregate Rating 1705, #204
- VibeCodingBench: 88.6 (#5/15)
Ling-3.0-flash — Aggregate Rating 1702, #211
- AA GDPval: 948.35 (#163/282)
Seed-1.6 — Aggregate Rating 1702, #212
- TempCloze: 45.93 (#13/32)
Qwen 3.6 35B A3B — Aggregate Rating 1701, #214
- Medmarks - LongHealth Task 2: 90.75 (#4/89)
- Medmarks - MedHallu Hard: 62.26 (#5/89)
- Medmarks - MedHallu Medium: 75.32 (#5/89)
- Medmarks - SuperGPQA Medicine Hard: 55.45 (#5/89)
- Medmarks - MedHallu Easy: 82.67 (#6/89)
- Medmarks - PubMedQA: 80.47 (#6/89)
- Medmarks - SuperGPQA Medicine Easy: 67.29 (#6/89)
- Medmarks - MedCalc-Bench: 78.73 (#8/89)
- Medmarks - MMLU-Pro Health: 79.38 (#10/89)
- Medmarks - SCTpublic: 73.45 (#10/89)
- ...and 19 more
Gemini 2.5 Flash — Aggregate Rating 1697, #216
- TempCloze: 72.28 (#5/32)
- Medical STT Benchmark: 0.0945 (#9/42)
- RewardBench 2 Precise IF: 57.5 (#28/249)
- K-Bench: 97.62 (#28/62)
- RewardBench 2 Math: 85.2 (#29/249)
- RewardBench 2 Safety: 90.9 (#32/249)
- K-Bench - Risk: 87.39 (#44/62)
- RewardBench 2 Focus: 84.1 (#89/249)
- RewardBench 2 Factuality: 67.4 (#130/249)
QRM-Llama3.1-8B-v2 — Aggregate Rating 1696, #221
- RewardBench 2 Safety: 94.67 (#8/249)
- RewardBench 2 Focus: 89.09 (#56/249)
- RewardBench 2 Precise IF: 40.62 (#85/249)
- RewardBench 2 Factuality: 66.53 (#136/249)
- RewardBench 2 Math: 61.2 (#157/249)
Llama-3.1-Tulu-3-70B-SFT-RM-RB2 — Aggregate Rating 1696, #222
- RewardBench 2 Factuality: 80.84 (#20/249)
- RewardBench 2 Safety: 86.89 (#73/249)
- RewardBench 2 Math: 67.76 (#74/249)
- RewardBench 2 Focus: 77.78 (#117/249)
- RewardBench 2 Precise IF: 36.88 (#139/249)
Nemotron 3 Ultra — Aggregate Rating 1695, #223
- K-Bench - Risk: 88.17 (#40/62)
- K-Bench: 96.94 (#46/62)
- MCP Atlas: 63.1 (#54/72)
Solar Pro 4 — Aggregate Rating 1694, #227
- AIMultiple - Text-to-SQL (Adjudicated): 57.0 (#65/72)
- AIMultiple - Text-to-SQL (Strict Execution Match): 30.3 (#66/72)
- AIMultiple - Agentic RAG Routing: 34.24 (#68/72)
- AIMultiple - FinanceReasoning (Hard): 71.43 (#76/87)
- AA GDPval: 1109.82 (#129/282)
URM-LLaMa-3.1-8B — Aggregate Rating 1694, #228
- RewardBench 2 Focus: 97.58 (#2/249)
- RewardBench 2 Safety: 91.78 (#26/249)
- RewardBench 2 Precise IF: 45.0 (#46/249)
- RewardBench 2 Math: 63.93 (#97/249)
- RewardBench 2 Factuality: 68.84 (#115/249)
DeepSeek R1 — Aggregate Rating 1693, #229
- AIIQ Composite IQ: 94.0 (#137/156)
O3 Mini (2025-01-31) — Aggregate Rating 1693, #230
- Vals AI MATH 500: 91.8 (#19/60)
- Vals AI MGSM: 91.35 (#29/75)
- Vals AI AIME: 86.46 (#32/96)
Qwen 3 VL 235B A22B — Aggregate Rating 1693, #231
- GAMUT: 33.0 (#9/14)
Grok 4 Fast — Aggregate Rating 1693, #232
- VibeCodingBench: 88.8 (#3/15)
MiniMax-M2.7 — Aggregate Rating 1692, #234
- Vals AI Tax Agent Bench: 25.38 (#62/65)
GPT-4.1 — Aggregate Rating 1689, #240
- RewardBench 2 Factuality: 82.89 (#12/249)
- RewardBench 2 Safety: 87.26 (#71/249)
- RewardBench 2 Math: 65.21 (#87/249)
- RewardBench 2 Precise IF: 39.74 (#98/249)
- RewardBench 2 Focus: 73.38 (#143/249)
Gemini 2.5 Flash (Preview 04-17) — Aggregate Rating 1689, #242
- RewardBench 2 Precise IF: 55.31 (#31/249)
- RewardBench 2 Safety: 90.94 (#31/249)
- RewardBench 2 Math: 81.15 (#34/249)
- RewardBench 2 Focus: 86.72 (#69/249)
- RewardBench 2 Factuality: 65.74 (#143/249)
Qwen 3.5 35B A3B — Aggregate Rating 1687, #247
- Medmarks - LongHealth Task 1: 91.83 (#2/89)
- Medmarks - LongHealth Task 2: 90.75 (#3/89)
- Medmarks - PubMedQA: 80.6 (#4/89)
- Medmarks - MedQA: 93.72 (#5/89)
- TempCloze: 65.54 (#7/32)
- Medmarks - MedMCQA: 78.88 (#8/89)
- Medmarks - Medbullets OP4: 87.77 (#8/89)
- Medmarks - CareQA EN: 92.77 (#10/89)
- Medmarks - HEAD-QA v2: 90.45 (#10/89)
- Medmarks - Medbullets OP5: 83.66 (#10/89)
- ...and 19 more
Claude Haiku 4.5 (20251001) — Aggregate Rating 1687, #248
- Vals AI Tax Agent Bench: 28.31 (#59/65)
Claude Haiku 4.5 — Aggregate Rating 1684, #250
- VibeCodingBench: 88.97 (#2/15)
- K-Bench: 98.35 (#14/62)
- K-Bench - Risk: 90.35 (#33/62)
Grok 4.1 — Aggregate Rating 1684, #252
- TempCloze: 24.11 (#29/32)
O3 Mini — Aggregate Rating 1682, #255
- AA LiveCodeBench: 73.44 (#55/343)
- AA IFBench: 67.14 (#91/459)
- AA MMLU-Pro: 80.19 (#113/349)
- AA GDP.pdf - STEM/Research - Criterion Pass Rate: 49.5 (#158/204)
- AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 43.97 (#160/204)
- AA GDP.pdf - Healthcare - Criterion Pass Rate: 51.52 (#161/204)
- AA GDP.pdf - Insurance - Criterion Pass Rate: 50.01 (#167/204)
- AA GDP.pdf - Construction - Criterion Pass Rate: 47.7 (#168/204)
- AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 51.73 (#168/204)
- AA GDP.pdf - Engineering - Criterion Pass Rate: 27.71 (#170/204)
- ...and 53 more
Grok 3 Mini — Aggregate Rating 1680, #260
- AA-Omniscience Hallucination Rate: 25.82 (#27/565)
- AA AIME 2025: 84.67 (#48/270)
- AA MMLU-Pro: 82.78 (#67/349)
- AA TAU-2 Bench: 90.35 (#70/449)
- AA LiveCodeBench: 69.63 (#71/343)
- AA-Omniscience Index - Business: -3.2 (#125/565)
- AA-Omniscience Index - Software Engineering (SWE) - Java: -9.0 (#136/565)
- AA-Omniscience Index - Humanities & Social Sciences: -5.4 (#148/565)
- AA-Omniscience Index - Health: -13.0 (#149/565)
- AA-Omniscience Index - Science, Engineering & Mathematics: -2.7 (#152/565)
- ...and 25 more
MiniMax-M2.1 — Aggregate Rating 1674, #268
- VibeCodingBench: 87.42 (#10/15)
GPT-5.1 Codex Mini — Aggregate Rating 1674, #269
- AA AIME 2025: 91.67 (#19/270)
- AA LiveCodeBench: 83.6 (#19/343)
- AA Global-MMLU-Lite - Yoruba: 69.92 (#29/111)
- AA Global-MMLU-Lite - Swahili: 80.08 (#35/111)
- AA Global-MMLU-Lite: 85.02 (#39/111)
- AA Global-MMLU-Lite - Arabic: 85.67 (#39/111)
- AA Global-MMLU-Lite - Bengali: 84.58 (#39/111)
- AA Global-MMLU-Lite - Portuguese: 88.92 (#39/111)
- AA Global-MMLU-Lite - Hindi: 84.92 (#41/111)
- AA Global-MMLU-Lite - Spanish: 89.42 (#42/111)
- ...and 43 more
Qwen 3 VL 32B — Aggregate Rating 1670, #273
- TempCloze: 25.38 (#21/32)
Qwen 3 VL 32B Instruct — Aggregate Rating 1669, #276
- TempCloze: 41.84 (#14/32)
GPT-5.4 Nano — Aggregate Rating 1665, #280
- AA APEX-Agents: 24.93 (#20/35)
- AA IFBench: 75.92 (#26/459)
- Vellum - Humanity's Last Exam: 37.7 (#27/41)
- Vals AI AIME: 88.75 (#28/96)
- Vals AI Terminal-Bench 2.0: 39.89 (#34/67)
- EnterpriseOps-Gym-AA - Email: 56.73 (#36/50)
- Vellum - GPQA: 82.8 (#36/72)
- EnterpriseOps-Gym-AA - Drive: 49.84 (#38/50)
- EnterpriseOps-Gym-AA - Calendar: 30.45 (#39/50)
- EnterpriseOps-Gym-AA - CSM: 29.93 (#39/50)
- ...and 101 more
Claude 3.5 Sonnet (20240620) — Aggregate Rating 1662, #287
- RewardBench 2 Focus: 86.97 (#68/249)
- RewardBench 2 Safety: 85.19 (#92/249)
- RewardBench 2 Precise IF: 38.75 (#108/249)
- RewardBench 2 Math: 56.83 (#201/249)
- RewardBench 2 Factuality: 52.84 (#206/249)
A.X-K2 — Aggregate Rating 1659, #293
- AutomationBench-AA - Sales - Raw Objective Completion: 13.61 (#158/213)
- AutomationBench-AA - Support - Raw Objective Completion: 16.18 (#158/213)
- AutomationBench-AA - Marketing - Raw Objective Completion: 7.34 (#160/213)
- AA GDPval: 965.08 (#161/282)
- AutomationBench-AA - HR - Raw Objective Completion: 3.43 (#163/213)
- AutomationBench-AA - Operations - Raw Objective Completion: 6.09 (#163/213)
- AutomationBench-AA: 2.48 (#166/213)
- AutomationBench-AA - Finance - Raw Objective Completion: 2.24 (#171/213)
- AutomationBench-AA - Strict: 0.0 (#178/213)
Qwen 3 Coder 480B A35B Instruct — Aggregate Rating 1658, #295
- Pencil Puzzle Bench - Kurodoko: 0.0 (#4/51)
- Pencil Puzzle Bench - Firefly: 0.0 (#6/51)
- Pencil Puzzle Bench - Sashigane: 0.0 (#6/51)
- Pencil Puzzle Bench - Sudoku: 0.0 (#12/51)
- BaxBench - No Security Reminder - Correct & Secure: 29.69 (#12/38)
- BaxBench - No Security Reminder - Correct: 53.42 (#12/38)
- Pencil Puzzle Bench - Country: 0.0 (#13/51)
- Pencil Puzzle Bench - Shakashaka: 0.0 (#15/51)
- Pencil Puzzle Bench - Double Choco: 0.0 (#17/51)
- Pencil Puzzle Bench - Heyawake: 0.0 (#18/51)
- ...and 26 more
GPT-OSS-120B — Aggregate Rating 1651, #305
- AA LiveCodeBench: 87.83 (#7/343)
- AA AIME 2025: 93.44 (#15/270)
- AA APEX-Agents: 3.1 (#32/35)
- EnterpriseOps-Gym-AA - Calendar: 33.65 (#35/50)
- AA Global-MMLU-Lite - Yoruba: 62.58 (#40/111)
- EnterpriseOps-Gym-AA - CSM: 27.24 (#41/50)
- EnterpriseOps-Gym-AA - Teams: 27.0 (#45/50)
- EnterpriseOps-Gym-AA - Email: 38.78 (#46/50)
- EnterpriseOps-Gym-AA - Hybrid: 26.14 (#46/50)
- EnterpriseOps-Gym-AA - Drive: 44.13 (#47/50)
- ...and 112 more
DeepSeek V3 (0324) — Aggregate Rating 1650, #311
- BenchLM: 34.2 (#153/200)
GPT-4o (2024-08-06) — Aggregate Rating 1649, #314
- RewardBench 2 Safety: 86.19 (#87/249)
- RewardBench 2 Math: 62.3 (#137/249)
- RewardBench 2 Focus: 72.93 (#145/249)
- RewardBench 2 Factuality: 56.84 (#196/249)
- RewardBench 2 Precise IF: 33.12 (#201/249)
GLM-4.6V — Aggregate Rating 1648, #317
- OpenRouter Tau2-Bench Airline: 60.7 (#103/143)
- OpenRouter GPQA Diamond: 66.8 (#118/154)
Mistral Medium 3.5 — Aggregate Rating 1645, #322
- Vals AI Terminal-Bench 2.0: 30.34 (#45/67)
- Icelandic LLM - Famous Phrases: 4.58 (#51/64)
- K-Bench: 95.92 (#51/62)
- Icelandic LLM - Flora & Fauna: 6.39 (#52/64)
- Icelandic LLM - WikiQA: 22.45 (#53/64)
- Vals AI CaseLaw v2: 44.16 (#53/54)
- K-Bench - Risk: 85.37 (#53/62)
- Icelandic LLM - GEC: 20.18 (#55/64)
- Icelandic LLM - Song Lyrics: 10.96 (#57/64)
- Icelandic LLM - Idiom Completion: 5.44 (#57/64)
- ...and 8 more
Nemotron 3 Super — Aggregate Rating 1645, #323
- K-Bench - Risk: 88.83 (#38/62)
- K-Bench: 96.98 (#44/62)
Gemma 4 26B A4B — Aggregate Rating 1640, #330
- Medmarks - SCTpublic: 75.34 (#3/89)
- Medmarks - MedXpertQA Reasoning: 53.13 (#4/89)
- Medmarks - MedXpertQA Understanding: 44.2 (#5/89)
- Medmarks - MedHallu Hard: 61.76 (#6/89)
- Medmarks - MedQA: 93.53 (#6/89)
- Medmarks - SuperGPQA Medicine Hard: 54.84 (#6/89)
- Medmarks - Medbullets OP5: 83.66 (#9/89)
- Medmarks - MedHallu Medium: 72.96 (#11/89)
- Medmarks - PubMedQA: 79.0 (#11/89)
- Medmarks - LongHealth Task 1: 90.67 (#12/89)
- ...and 32 more
Qwen 3 Coder Plus — Aggregate Rating 1637, #336
- GSO-Bench: 4.9 (#25/34)
GPT-4.1 Mini — Aggregate Rating 1635, #338
- RewardBench 2 Math: 72.13 (#50/249)
- RewardBench 2 Precise IF: 41.25 (#76/249)
- RewardBench 2 Focus: 73.54 (#142/249)
- RewardBench 2 Factuality: 60.84 (#177/249)
- RewardBench 2 Safety: 72.65 (#196/249)
Qwen 3.5 9B — Aggregate Rating 1635, #339
- DGEval - IMDG Code MCQ: 54.9 (#28/35)
- DGEval - Regulatory Recall: 1.0 (#33/35)
- AA GDP.pdf - Real Estate - Criterion Pass Rate: 54.61 (#131/204)
- AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 64.74 (#132/204)
- AA GDP.pdf: 6.6 (#139/204)
- AA GDP.pdf - HR - Criterion Pass Rate: 59.59 (#140/204)
- AA GDP.pdf - Healthcare - Criterion Pass Rate: 56.78 (#147/204)
- AA GDP.pdf - Engineering - Criterion Pass Rate: 29.8 (#150/204)
- AA GDP.pdf - Criterion Pass Rate: 49.79 (#157/204)
- AA GDP.pdf - STEM/Research - Criterion Pass Rate: 48.99 (#163/204)
- ...and 19 more
GPT-4o (2024-11-20) — Aggregate Rating 1633, #343
- AA GDPval: -13.07 (#256/282)
Nova 2.0 Pro Preview — Aggregate Rating 1631, #349
- AA IFBench: 79.59 (#11/459)
- AA AIME 2025: 89.0 (#33/270)
- AA TAU-2 Bench: 92.69 (#55/449)
- AA LiveCodeBench: 73.02 (#56/343)
- AA MMLU-Pro: 82.95 (#61/349)
- MLCR-AA - Completeness: 10.74 (#76/94)
- MLCR-AA: 3.89 (#81/94)
- MLCR-AA - Conciseness: 82.78 (#86/94)
- MLCR-AA - Accuracy: 16.78 (#92/94)
- AA Terminal-Bench Hard: 24.24 (#163/441)
- ...and 31 more
Llama 3.1 8B Base RM RB2 — Aggregate Rating 1627, #361
- RewardBench 2 Factuality: 72.0 (#82/249)
- RewardBench 2 Focus: 83.23 (#93/249)
- RewardBench 2 Safety: 82.67 (#117/249)
- RewardBench 2 Math: 61.2 (#147/249)
- RewardBench 2 Precise IF: 36.25 (#149/249)
Llama-3-OffsetBias-RM-8B — Aggregate Rating 1627, #362
- RewardBench 2 Focus: 95.96 (#8/249)
- RewardBench 2 Precise IF: 40.0 (#87/249)
- RewardBench 2 Factuality: 60.84 (#173/249)
- RewardBench 2 Safety: 72.22 (#198/249)
- RewardBench 2 Math: 51.91 (#214/249)
Mistral Large 3 — Aggregate Rating 1624, #370
- Icelandic LLM - Song Lyrics: 11.89 (#43/64)
- Icelandic LLM - Generative Quality: 55.17 (#44/64)
- Icelandic LLM - Flora & Fauna: 14.95 (#45/64)
- Icelandic LLM - WikiQA: 30.53 (#48/64)
- Icelandic LLM - Case Government: 26.56 (#50/64)
- Icelandic LLM Leaderboard - Average: 25.7 (#52/64)
- Icelandic LLM - Idiom Meanings: 43.36 (#52/64)
- Icelandic LLM - Textile Terms: 18.8 (#53/64)
- Icelandic LLM - Idiom Completion: 6.64 (#54/64)
- Icelandic LLM - Famous Phrases: 2.05 (#56/64)
- ...and 1 more
EXAONE 4.5 33B — Aggregate Rating 1623, #373
- AA IFBench: 57.96 (#140/459)
- AA TAU-2 Bench: 78.07 (#147/449)
- AA Terminal-Bench Hard: 20.45 (#183/441)
- AA MMMU-Pro: 67.34 (#184/304)
- AA-LCR: 54.67 (#199/202)
- AA GDPval: 470.73 (#213/282)
- AA GPQA Diamond: 79.39 (#228/636)
- AA-Omniscience Hallucination Rate: 79.92 (#270/565)
- AA Humanity's Last Exam: 12.88 (#307/688)
- AA-Omniscience Index - Science, Engineering & Mathematics: -29.6 (#314/565)
- ...and 24 more
Gemini 2.5 Flash Lite — Aggregate Rating 1621, #376
- K-Bench: 97.23 (#38/62)
- K-Bench - Risk: 87.07 (#48/62)
Claude 3 Opus (20240229) — Aggregate Rating 1618, #385
- RewardBench 2 Safety: 83.78 (#107/249)
- RewardBench 2 Focus: 66.46 (#174/249)
- RewardBench 2 Precise IF: 33.12 (#193/249)
- RewardBench 2 Factuality: 53.89 (#202/249)
- RewardBench 2 Math: 51.37 (#218/249)
GPT-5 Nano — Aggregate Rating 1617, #387
- Vals AI MATH 500: 93.8 (#15/60)
- AA LiveCodeBench: 78.94 (#32/343)
- Vals AI CaseLaw v2: 52.63 (#44/54)
- Vals AI AIME: 81.18 (#45/96)
- AA Global-MMLU-Lite - Swahili: 76.58 (#46/111)
- Vals AI MGSM: 89.31 (#49/75)
- AA Global-MMLU-Lite - Burmese: 75.58 (#51/111)
- AA Global-MMLU-Lite - Yoruba: 56.67 (#51/111)
- AA AIME 2025: 83.67 (#53/270)
- AA Global-MMLU-Lite: 81.23 (#59/111)
- ...and 48 more
GLM-4 Plus — Aggregate Rating 1617, #389
- VibeCodingBench: 88.2 (#7/15)
InternVL3-38B — Aggregate Rating 1616, #391
- TempCloze: 27.74 (#18/32)
Qwen 3 VL 8B Instruct — Aggregate Rating 1616, #392
- TempCloze: 30.55 (#16/32)
Nova 2.0 Lite — Aggregate Rating 1616, #393
- Evals for Every Language - Language bew: 100.0 (#1/68)
- Evals for Every Language - Language br: 100.0 (#1/68)
- Evals for Every Language - Language dv: 100.0 (#1/68)
- Evals for Every Language - Language fy: 100.0 (#1/68)
- Evals for Every Language - Language cgg: 100.0 (#1/67)
- Evals for Every Language - Language co: 100.0 (#2/69)
- Evals for Every Language - Language qu: 91.67 (#9/68)
- Evals for Every Language - MMLU: 95.33 (#12/69)
- Evals for Every Language - Language en: 81.54 (#12/71)
- Evals for Every Language - Language nus: 30.36 (#13/71)
- ...and 260 more
Mercury 2.5 — Aggregate Rating 1615, #399
- Vals AI CyberBench: 45.48 (#37/44)
- Vals AI (Vals Index): 8.86 (#43/43)
- Vals AI Public Benefits Bench: 45.26 (#44/47)
- Vals AI Tax Agent Bench: 12.41 (#65/65)
- Vals AI Legal Research Bench: 4.33 (#68/73)
- Vals AI Excel Modeling: 8.79 (#69/70)
- Vals AI Code Migration: 4.52 (#69/73)
- Vals AI Finance Agent v2: 18.89 (#71/74)
- Vals AI Terminal-Bench 2.1: 34.46 (#71/76)
- AIIQ Composite IQ: 101.0 (#110/156)
- ...and 2 more
Sarvam 105B — Aggregate Rating 1613, #404
- AA TAU-2 Bench: 46.78 (#226/449)
- AA GPQA Diamond: 73.84 (#301/636)
- AA Humanity's Last Exam: 11.03 (#337/688)
- AA-Omniscience Index - Science, Engineering & Mathematics: -35.6 (#345/565)
- AA IFBench: 34.35 (#346/459)
- AA-Omniscience Accuracy: 17.6 (#366/563)
- AA Terminal-Bench Hard: 1.52 (#368/441)
- AA-Omniscience Index - Software Engineering (SWE) - R: -72.0 (#393/565)
- AA-Omniscience Index - Software Engineering (SWE) - Julia: -72.0 (#396/565)
- AA-Omniscience Index - Software Engineering (SWE) - HTML: -40.0 (#415/565)
- ...and 21 more
Granite 4.2 30B — Aggregate Rating 1611, #408
- AIIQ Composite IQ: 98.0 (#125/156)
Llama 4 Maverick — Aggregate Rating 1609, #413
- K-Bench: 94.95 (#57/62)
- K-Bench - Risk: 81.25 (#61/62)
Qwen 3 32B — Aggregate Rating 1607, #417
- K-Bench: 97.48 (#32/62)
- K-Bench - Risk: 90.8 (#32/62)
- THOR Finding Triage - Critical Miss Rate: 0.0 (#36/85)
- THOR Finding Triage - CW%: 42.4 (#75/85)
- THOR Finding Triage - False Review Load: 76.42 (#77/85)
- THOR Finding Triage - Balanced OTS: 42.2 (#79/85)
Qwen 3 VL 8B — Aggregate Rating 1607, #418
- TempCloze: 24.87 (#25/32)
Command A+ — Aggregate Rating 1600, #440
- AIMultiple - Text-to-SQL (Strict Execution Match): 37.8 (#42/72)
- AIMultiple - FinanceReasoning (Hard): 79.41 (#62/87)
- AIMultiple - Text-to-SQL (Adjudicated): 58.7 (#64/72)
- AIMultiple - Agentic RAG Routing: 40.76 (#66/72)
Qwen 3 VL 4B — Aggregate Rating 1597, #447
- TempCloze: 25.2 (#24/32)
GRM-llama3-8B-sftreg — Aggregate Rating 1597, #451
- RewardBench 2 Precise IF: 38.75 (#110/249)
- RewardBench 2 Safety: 78.67 (#153/249)
- RewardBench 2 Focus: 68.28 (#165/249)
- RewardBench 2 Factuality: 61.89 (#168/249)
- RewardBench 2 Math: 57.92 (#189/249)
FsfairX-LLaMA3-RM-v0.1 — Aggregate Rating 1593, #463
- RewardBench 2 Precise IF: 41.88 (#71/249)
- RewardBench 2 Math: 62.84 (#127/249)
- RewardBench 2 Focus: 70.51 (#161/249)
- RewardBench 2 Safety: 76.67 (#172/249)
- RewardBench 2 Factuality: 59.16 (#186/249)
Command A — Aggregate Rating 1592, #464
- CritPt: 0.0 (#267/278)
GLM-4.7 Flash — Aggregate Rating 1585, #486
- VibeCodingBench: 83.83 (#14/15)
GPT-OSS-20B — Aggregate Rating 1584, #488
- AA AIME 2025: 89.33 (#32/270)
- AA APEX-Agents: 0.74 (#35/35)
- AA LiveCodeBench: 77.67 (#38/343)
- AA Global-MMLU-Lite - Yoruba: 49.25 (#67/111)
- MLCR-AA - Conciseness: 94.44 (#69/94)
- AA Global-MMLU-Lite - Burmese: 65.42 (#76/111)
- AA Global-MMLU-Lite - Hindi: 78.17 (#79/111)
- AA Global-MMLU-Lite - Bengali: 76.0 (#81/111)
- AA Global-MMLU-Lite - Korean: 78.92 (#84/111)
- AA Global-MMLU-Lite: 75.16 (#85/111)
- ...and 76 more
Mistral Small 4 — Aggregate Rating 1584, #489
- K-Bench: 96.21 (#50/62)
- Icelandic LLM - GEC: 33.94 (#51/64)
- Icelandic LLM - Case Government: 25.31 (#51/64)
- Icelandic LLM - WikiQA: 22.82 (#52/64)
- Icelandic LLM - Generative Quality: 45.59 (#52/64)
- Icelandic LLM Leaderboard - Average: 21.47 (#53/64)
- Icelandic LLM - Famous Phrases: 3.5 (#53/64)
- Icelandic LLM - Flora & Fauna: 6.1 (#53/64)
- Icelandic LLM - Idiom Completion: 6.24 (#55/64)
- Icelandic LLM - Song Lyrics: 11.02 (#56/64)
- ...and 4 more
MedGemma-27B-IT — Aggregate Rating 1581, #501
- MedLayXPlain: 67.2 (#2/32)
- PetQA-Bench - Text Helpfulness: 4.872 (#4/18)
- CardioLens: 52.69 (#4/24)
- PetQA-Bench - Multimodal Helpfulness: 4.634 (#5/11)
- PetQA-Bench - Multimodal Factuality: 44.2 (#11/11)
- MedQ-Bench - Perception: 57.16 (#11/14)
- CheXpercept: 76.8 (#11/14)
- PetQA-Bench - Text Factuality: 50.3 (#12/18)
- MedQ-Bench - No-Reference Reasoning: 4.05 (#13/14)
- MedQ-Bench - Comparison Reasoning: 3.5 (#13/14)
Qwen 3 14B — Aggregate Rating 1578, #507
- THOR Finding Triage - Critical Miss Rate: 0.0 (#37/85)
- THOR Finding Triage - CW%: 41.2 (#76/85)
- THOR Finding Triage - Balanced OTS: 43.2 (#77/85)
- THOR Finding Triage - False Review Load: 78.86 (#78/85)
Solar Pro 3 — Aggregate Rating 1574, #518
- BenchLM: 30.4 (#168/200)
- Wolfram LLM Benchmarking Project: 33.3 (#348/550)
gemma-4-E4B-it — Aggregate Rating 1573, #520
- RealDataAgentBench: 55.54 (#14/14)
Qwen 3 VL 4B Instruct — Aggregate Rating 1573, #522
- MobileGym-Bench: 9.4 (#10/10)
- TempCloze: 24.19 (#28/32)
InternLM2-20B-Reward — Aggregate Rating 1570, #535
- RewardBench 2 Focus: 72.53 (#151/249)
- RewardBench 2 Precise IF: 36.25 (#156/249)
- RewardBench 2 Factuality: 55.58 (#198/249)
- RewardBench 2 Math: 57.38 (#200/249)
- RewardBench 2 Safety: 61.11 (#223/249)
GPT-4o Mini (2024-07-18) — Aggregate Rating 1567, #543
- RewardBench 2 Focus: 74.14 (#139/249)
- RewardBench 2 Safety: 76.67 (#171/249)
- RewardBench 2 Precise IF: 34.38 (#181/249)
- RewardBench 2 Math: 51.91 (#216/249)
- RewardBench 2 Factuality: 41.05 (#226/249)
Llama 3.3 70B — Aggregate Rating 1566, #547
- DGEval - Regulatory Recall: 6.6 (#26/35)
- DGEval - IMDG Code MCQ: 36.3 (#32/35)
Granite 4.2 8B — Aggregate Rating 1562, #559
- OpenRouter Tau2-Bench Airline: 58.3 (#107/143)
- OpenRouter GPQA Diamond: 65.7 (#120/154)
- AIIQ Composite IQ: 95.0 (#134/156)
Molmo2-8B — Aggregate Rating 1559, #576
- TempCloze: 23.21 (#32/32)
- Chatbot Arena (Vision - Homework): 1089.0 (#110/110)
InternVL3.5-8B — Aggregate Rating 1555, #586
- TempCloze: 28.05 (#17/32)
GPT-4o Mini — Aggregate Rating 1554, #588
- K-Bench: 95.67 (#54/62)
- K-Bench - Risk: 85.27 (#54/62)
Qwen2.5-Omni-7B — Aggregate Rating 1552, #594
- OmniGAIA - Finance: 4.0 (#10/16)
- OmniGAIA - Science: 7.7 (#10/16)
- OmniGAIA - History: 7.5 (#12/16)
- OmniGAIA - Food: 5.6 (#12/16)
- OmniGAIA - Tech: 4.1 (#13/16)
- OmniGAIA - Art: 2.8 (#13/16)
- OmniGAIA - Overall: 3.6 (#15/18)
- OmniGAIA - Geo: 1.5 (#15/16)
- OmniGAIA - Sport: 0.0 (#16/16)
- OmniGAIA - Movie: 0.0 (#16/16)
Ministral 3 14B — Aggregate Rating 1539, #622
- BenchCompass - Payment Knowledge (Closed Book): 49.5 (#13/16)
- BenchCompass - Context-Grounded Reasoning (Closed Book): 2.1 (#14/16)
- BenchCompass - Task-Input Attack (Open Book): 37.4 (#14/16)
- BenchCompass - Context-Grounded Reasoning (Open Book): 57.3 (#15/16)
Qwen 2.5 VL 7B Instruct — Aggregate Rating 1533, #645
- TempCloze: 25.87 (#20/32)
InternLM2-7B-Reward — Aggregate Rating 1529, #662
- RewardBench 2 Precise IF: 40.0 (#97/249)
- RewardBench 2 Focus: 70.51 (#160/249)
- RewardBench 2 Math: 56.28 (#204/249)
- RewardBench 2 Safety: 59.56 (#224/249)
- RewardBench 2 Factuality: 42.11 (#225/249)
Ministral 3 8B — Aggregate Rating 1528, #665
- BenchCompass - Payment Knowledge (Closed Book): 51.6 (#12/16)
- BenchCompass - Context-Grounded Reasoning (Closed Book): 2.1 (#15/16)
- BenchCompass - Context-Grounded Reasoning (Open Book): 50.0 (#16/16)
- BenchCompass - Task-Input Attack (Open Book): 29.6 (#16/16)
- Mizan: 80.3 (#24/27)
- Mizan - Iraqi Arabic: 73.9 (#24/27)
- Mizan - MSA: 90.0 (#24/27)
Starling-RM-34B — Aggregate Rating 1522, #683
- RewardBench 2 Math: 61.75 (#138/249)
- RewardBench 2 Safety: 75.56 (#180/249)
- RewardBench 2 Precise IF: 31.87 (#209/249)
- RewardBench 2 Factuality: 45.89 (#222/249)
- RewardBench 2 Focus: 48.08 (#224/249)
Phi-4 — Aggregate Rating 1515, #702
- K-Bench: 95.51 (#55/62)
- K-Bench - Risk: 84.56 (#55/62)
Eurus-RM-7B — Aggregate Rating 1515, #704
- RewardBench 2 Focus: 74.75 (#133/249)
- RewardBench 2 Precise IF: 34.38 (#182/249)
- RewardBench 2 Factuality: 60.0 (#183/249)
- RewardBench 2 Math: 56.83 (#202/249)
- RewardBench 2 Safety: 62.67 (#222/249)
Granite 4.2 3B — Aggregate Rating 1514, #707
- AIIQ Composite IQ: 90.0 (#145/156)
GPT-4.1 Nano — Aggregate Rating 1512, #715
- RewardBench 2 Safety: 71.56 (#205/249)
- RewardBench 2 Factuality: 46.46 (#220/249)
- RewardBench 2 Math: 50.41 (#221/249)
- RewardBench 2 Focus: 46.6 (#226/249)
- RewardBench 2 Precise IF: 25.78 (#240/249)
Sarvam 30B — Aggregate Rating 1512, #719
- AA TAU-2 Bench: 34.5 (#264/449)
- AA CritPt: 0.29 (#344/603)
- AA Terminal-Bench Hard: 2.27 (#359/441)
- AA GPQA Diamond: 63.33 (#399/636)
- AA Humanity's Last Exam: 7.51 (#402/688)
- AA IFBench: 26.46 (#416/459)
- AA-Omniscience Index - Software Engineering (SWE) - Julia: -80.0 (#439/565)
- AA-Omniscience Index - Software Engineering (SWE) - Go: -66.0 (#451/565)
- AA-Omniscience Index - Software Engineering (SWE) - Rust: -14.0 (#458/565)
- AA-Omniscience Index - Science, Engineering & Mathematics: -52.5 (#473/565)
- ...and 21 more
MedGemma-4B-IT — Aggregate Rating 1508, #725
- Medmarks - MedHallu Hard: 50.37 (#25/89)
- Medmarks - PubMedQA: 72.67 (#58/89)
- Medmarks - MedHallu Medium: 52.67 (#59/89)
- Medmarks - M-ARC: 23.67 (#69/89)
- Medmarks - MedHallu Easy: 56.02 (#72/89)
- Medmarks - MedMCQA: 55.41 (#76/89)
- Medmarks - MedQA: 64.91 (#78/89)
- Medmarks - MedXpertQA Reasoning: 14.79 (#78/89)
- Medmarks - PubHealthBench Reviewed: 78.2 (#78/89)
- Medmarks - Med-HALT Reasoning FCT: 17.33 (#79/89)
- ...and 17 more
Gemma 4 E4B — Aggregate Rating 1505, #736
- AA GDP.pdf - STEM/Research - Criterion Pass Rate: 45.16 (#174/204)
- AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 37.15 (#177/204)
- AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 45.77 (#178/204)
- AA GDP.pdf - Legal - Criterion Pass Rate: 40.66 (#180/204)
- AA GDP.pdf - Criterion Pass Rate: 39.41 (#184/204)
- AA GDP.pdf - Construction - Criterion Pass Rate: 44.29 (#186/204)
- AA GDP.pdf - HR - Criterion Pass Rate: 41.05 (#186/204)
- AA GDP.pdf - Engineering - Criterion Pass Rate: 24.82 (#188/204)
- AA GDP.pdf: 1.0 (#189/204)
- AA GDP.pdf - Insurance - Criterion Pass Rate: 40.57 (#190/204)
- ...and 18 more
Claude 3 Haiku (20240307) — Aggregate Rating 1502, #744
- RewardBench 2 Focus: 50.1 (#221/249)
- RewardBench 2 Safety: 59.5 (#225/249)
- RewardBench 2 Factuality: 40.42 (#227/249)
- RewardBench 2 Precise IF: 28.12 (#228/249)
- RewardBench 2 Math: 35.52 (#239/249)
Gemini 1.5 Flash-8B — Aggregate Rating 1485, #799
- RewardBench 2 Precise IF: 36.25 (#153/249)
- RewardBench 2 Focus: 67.47 (#166/249)
- RewardBench 2 Safety: 66.22 (#215/249)
- RewardBench 2 Math: 50.82 (#220/249)
- RewardBench 2 Factuality: 46.11 (#221/249)
Qwen 2.5 Coder 7B Instruct — Aggregate Rating 1456, #915
- Text-to-SQL under RBAC: 66.91 (#11/13)
Granite 4.1 8B — Aggregate Rating 1453, #937
- K-Bench: 94.74 (#58/62)
- K-Bench - Risk: 82.54 (#59/62)
internlm2-1.8B-reward — Aggregate Rating 1433, #1036
- RewardBench 2 Precise IF: 36.25 (#155/249)
- RewardBench 2 Focus: 59.6 (#193/249)
- RewardBench 2 Math: 44.26 (#226/249)
- RewardBench 2 Safety: 47.11 (#233/249)
- RewardBench 2 Factuality: 27.58 (#242/249)
granite-4.0-h-micro — Aggregate Rating 1430, #1048
- K-Bench: 81.19 (#62/62)
- K-Bench - Risk: 52.39 (#62/62)
command-r7B-12-2024 — Aggregate Rating 1396, #1181
- AI Chess Leaderboard (Continuation): 366.0 (#266/279)
Qwen-Audio-Chat — Aggregate Rating 1392, #1197
- Multimodal Hallucination - AVH Audio RefCheck Resistance: 65.69 (#4/9)
- Multimodal Hallucination - AVH Audio SelfCheckGPT Resistance: 63.43 (#5/9)
- Multimodal Hallucination - AVH Audio CrossCheck-explicit Resistance: 56.34 (#7/9)
UltraRM-13B — Aggregate Rating 1390, #1206
- RewardBench 2 Focus: 60.81 (#187/249)
- RewardBench 2 Precise IF: 33.12 (#202/249)
- RewardBench 2 Math: 55.19 (#208/249)
- RewardBench 2 Factuality: 50.63 (#211/249)
- RewardBench 2 Safety: 50.89 (#230/249)
Llama 3.2 3B — Aggregate Rating 1337, #1355
- DGEval - IMDG Code MCQ: 14.6 (#35/35)
- DGEval - Regulatory Recall: 0.3 (#35/35)
ArmoRM-Llama3-8B-v0.1 — Aggregate Rating 1332, #1365
- RewardBench 2 Precise IF: 41.88 (#66/249)
- RewardBench 2 Math: 66.12 (#81/249)
- RewardBench 2 Safety: 82.22 (#120/249)
- RewardBench 2 Focus: 76.57 (#124/249)
- RewardBench 2 Factuality: 65.68 (#144/249)
pythia-12B — Aggregate Rating 1160, #1571
- InstructEval - Alignment (HHH): 45.1 (#27/28)
- InstructEval - Problem Solving: 23.9 (#38/54)

Top-10 New Scores (1144)

Claude Fable 5.1 on AA CritPt: 31.14 (#12)
Claude Fable 5.1 on AA GDP.pdf: 28.0 (#11)
Claude Fable 5.1 on AA GDP.pdf - Construction - Criterion Pass Rate: 78.67 (#10)
Claude Fable 5.1 on AA GDP.pdf - Criterion Pass Rate: 80.18 (#16)
Claude Fable 5.1 on AA GDP.pdf - Engineering - Criterion Pass Rate: 73.2 (#37)
Claude Fable 5.1 on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 82.18 (#31)
Claude Fable 5.1 on AA GDP.pdf - HR - Criterion Pass Rate: 82.31 (#19)
Claude Fable 5.1 on AA GDP.pdf - Healthcare - Criterion Pass Rate: 92.29 (#4)
Claude Fable 5.1 on AA GDP.pdf - Insurance - Criterion Pass Rate: 89.15 (#1)
Claude Fable 5.1 on AA GDP.pdf - Legal - Criterion Pass Rate: 85.2 (#5)
Claude Fable 5.1 on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 82.56 (#1)
Claude Fable 5.1 on AA GDP.pdf - Real Estate - Criterion Pass Rate: 74.32 (#16)
Claude Fable 5.1 on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 79.75 (#6)
Claude Fable 5.1 on AA GDPval: 1757.77 (#4)
Claude Fable 5.1 on AA GPQA Diamond: 93.74 (#11)
Claude Fable 5.1 on AA Humanity's Last Exam: 59.13 (#3)
Claude Fable 5.1 on AA Long Context Reasoning: 85.33 (#4)
Claude Fable 5.1 on AA Omniscience: 43.45 (#4)
Claude Fable 5.1 on AA SciCode: 63.08 (#5)
Claude Fable 5.1 on AA-AnalystAgent: 57.5 (#3)
Claude Fable 5.1 on AA-AnalystAgent - Pass@1: 65.75 (#4)
Claude Fable 5.1 on AA-AnalystAgent - Pass@K: 72.5 (#18)
Claude Fable 5.1 on AA-Briefcase: 61.52 (#4)
Claude Fable 5.1 on AA-Briefcase - Analytical Quality Elo: 1977.95 (#6)
Claude Fable 5.1 on AA-Briefcase - DOCX - Rubric Pass Rate: 57.4 (#10)
Claude Fable 5.1 on AA-Briefcase - PDF - Rubric Pass Rate: 68.1 (#3)
Claude Fable 5.1 on AA-Briefcase - PPTX - Rubric Pass Rate: 63.27 (#2)
Claude Fable 5.1 on AA-Briefcase - Presentation Elo: 1488.9 (#22)
Claude Fable 5.1 on AA-Briefcase - XLSX - Rubric Pass Rate: 60.45 (#5)
Claude Fable 5.1 on AA-LCR: 85.33 (#4)
Claude Fable 5.1 on AA-Omniscience Hallucination Rate: 65.63 (#182)
Claude Fable 5.1 on AA-Omniscience Index - Business: 29.9 (#14)
Claude Fable 5.1 on AA-Omniscience Index - Health: 23.5 (#7)
Claude Fable 5.1 on AA-Omniscience Index - Humanities & Social Sciences: 41.0 (#6)
Claude Fable 5.1 on AA-Omniscience Index - Law: 50.1 (#2)
Claude Fable 5.1 on AA-Omniscience Index - Science, Engineering & Mathematics: 33.0 (#5)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE): 86.2 (#2)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - C: 92.0 (#4)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - Dart: 96.0 (#1)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - Go: 90.0 (#2)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - HTML: 84.0 (#12)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - Java: 69.0 (#10)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 87.27 (#3)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - Julia: 88.0 (#6)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 86.0 (#6)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - PHP: 90.0 (#7)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - Python: 91.0 (#2)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - R: 74.0 (#1)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - Rust: 84.0 (#9)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - Swift: 100.0 (#1)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 92.22 (#10)
Claude Fable 5.1 on Artificial Analysis Intelligence Index: 53.35 (#9)
Claude Fable 5.1 on AutomationBench-AA: 59.38 (#35)
Claude Fable 5.1 on AutomationBench-AA - Finance - Raw Objective Completion: 79.67 (#64)
Claude Fable 5.1 on AutomationBench-AA - HR - Raw Objective Completion: 74.92 (#25)
Claude Fable 5.1 on AutomationBench-AA - Marketing - Raw Objective Completion: 90.12 (#5)
Claude Fable 5.1 on AutomationBench-AA - Operations - Raw Objective Completion: 96.39 (#7)
Claude Fable 5.1 on AutomationBench-AA - Sales - Raw Objective Completion: 85.19 (#23)
Claude Fable 5.1 on AutomationBench-AA - Strict: 32.12 (#20)
Claude Fable 5.1 on AutomationBench-AA - Support - Raw Objective Completion: 91.7 (#7)
Claude Fable 5.1 on BenchGecko Score: 88.2 (#9)
Claude Fable 5.1 on CAIS Risk Index: 29.0 (#2)
Claude Fable 5.1 on CAIS Vision Capabilities Index: 63.1 (#8)
Claude Fable 5.1 on Chatbot Arena (Text - Korean): 1514.0 (#1)
Claude Fable 5.1 on Chatbot Arena (Vision - Chinese): 1328.0 (#23)
Claude Fable 5.1 on CritPt: 31.1 (#8)
Claude Fable 5.1 on Finance Agent v2: 58.9 (#9)
Claude Fable 5.1 on FrontierCode: 63.6 (#14)
Claude Fable 5.1 on FrontierSWE V2: 56.29 (#4)
Claude Fable 5.1 on GPQA Diamond (BenchmarkList): 93.7 (#2)
Claude Fable 5.1 on GeoBench ACW: 4302.89 (#4)
Claude Fable 5.1 on GeoBench Photos: 4259.62 (#2)
Claude Fable 5.1 on Gray Swan IPI — Combined Attack Set, Transfer Only, No Computer Use: 1.0 (#3)
Claude Fable 5.1 on Guesswork 2026-09: 0.5512 (#4)
Claude Fable 5.1 on HANDBOOK.md Agents: 38.8 (#1)
Claude Fable 5.1 on Harvey LAB-AA: 93.27 (#7)
Claude Fable 5.1 on Harvey LAB-AA - All Pass: 16.67 (#3)
Claude Fable 5.1 on Harvey LAB-AA - Antitrust & Competition: 95.29 (#3)
Claude Fable 5.1 on Harvey LAB-AA - Arbitration & International Dispute Resolution: 95.7 (#5)
Claude Fable 5.1 on Harvey LAB-AA - Banking & Finance: 94.8 (#5)
Claude Fable 5.1 on Harvey LAB-AA - Bankruptcy & Restructuring: 92.73 (#7)
Claude Fable 5.1 on Harvey LAB-AA - Capital Markets: 97.25 (#1)
Claude Fable 5.1 on Harvey LAB-AA - Corporate Governance: 92.81 (#6)
Claude Fable 5.1 on Harvey LAB-AA - Corporate M&A: 95.83 (#4)
Claude Fable 5.1 on Harvey LAB-AA - Data Privacy & Cybersecurity: 98.94 (#2)
Claude Fable 5.1 on Harvey LAB-AA - Emerging Companies & Venture Capital: 95.9 (#2)
Claude Fable 5.1 on Harvey LAB-AA - Employment & Labor: 89.89 (#5)
Claude Fable 5.1 on Harvey LAB-AA - Energy & Natural Resources: 97.36 (#6)
Claude Fable 5.1 on Harvey LAB-AA - Environmental & ESG: 91.4 (#5)
Claude Fable 5.1 on Harvey LAB-AA - Funds & Asset Management: 90.67 (#7)
Claude Fable 5.1 on Harvey LAB-AA - Healthcare & Life Sciences: 93.4 (#9)
Claude Fable 5.1 on Harvey LAB-AA - Immigration: 93.45 (#6)
Claude Fable 5.1 on Harvey LAB-AA - Insurance: 94.87 (#3)
Claude Fable 5.1 on Harvey LAB-AA - Intellectual Property: 97.17 (#1)
Claude Fable 5.1 on Harvey LAB-AA - International Trade & Sanctions: 92.86 (#4)
Claude Fable 5.1 on Harvey LAB-AA - Litigation & Dispute Resolution: 95.13 (#4)
Claude Fable 5.1 on Harvey LAB-AA - Real Estate: 96.79 (#2)
Claude Fable 5.1 on Harvey LAB-AA - Structured Finance & Securitization: 91.11 (#2)
Claude Fable 5.1 on Harvey LAB-AA - Tax: 90.64 (#7)
Claude Fable 5.1 on Harvey LAB-AA - Trusts, Estates & Private Client: 95.04 (#4)
Claude Fable 5.1 on Harvey LAB-AA - White Collar Defense & Investigations: 94.95 (#3)
Claude Fable 5.1 on KernelBench Mega: 22.946 (#6)
Claude Fable 5.1 on LABBench2 Aggregate: 68.6 (#4)
Claude Fable 5.1 on LVBench: 79.7 (#8)
Claude Fable 5.1 on Lech Mazur Writing: 3.796 (#1)
Claude Fable 5.1 on MCP Atlas: 87.2 (#2)
Claude Fable 5.1 on MILU: 93.0 (#3)
Claude Fable 5.1 on MLCR-AA: 71.11 (#2)
Claude Fable 5.1 on MLCR-AA - Accuracy: 94.84 (#4)
Claude Fable 5.1 on MLCR-AA - Completeness: 89.03 (#1)
Claude Fable 5.1 on MLCR-AA - Conciseness: 86.11 (#82)
Claude Fable 5.1 on MazeBench: 10.0 (#5)
Claude Fable 5.1 on NonoBench: 76.7 (#28)
Claude Fable 5.1 on PRBench Finance: 50.8 (#8)
Claude Fable 5.1 on Professional Reasoning Bench - Legal: 51.6 (#6)
Claude Fable 5.1 on ReactBench: 45.6 (#2)
Claude Fable 5.1 on SWE-Together: 82.72 (#4)
Claude Fable 5.1 on Tau3 Banking: 47.22 (#9)
Claude Fable 5.1 on Terminal-Bench-Science: 40.0 (#3)
Claude Fable 5.1 on Vals AI CyberBench: 70.42 (#18)
Claude Fable 5.1 on Vals AI MMMU: 90.64 (#1)
Claude Fable 5.1 on Vals AI MortgageTax: 70.79 (#2)
Claude Fable 5.1 on Vals AI ProgramBench: 7.0 (#2)
Claude Fable 5.1 on Vals AI Public Benefits Bench: 74.9 (#2)
Claude Fable 5.1 on Vals AI ReverseEngBench: 22.9 (#7)
Claude Fable 5.1 on Vals AI Tax Agent Bench: 77.64 (#1)
Claude Fable 5.1 on Vals AI Time Horizon Index: 63.33 (#3)
Claude Fable 5.1 on Vibe Code Bench v1.1: 90.3 (#2)
Claude Opus 5.5 on AA CritPt: 31.71 (#2)
Claude Opus 5.5 on AA GDP.pdf: 28.8 (#10)
Claude Opus 5.5 on AA GDP.pdf - Construction - Criterion Pass Rate: 75.95 (#13)
Claude Opus 5.5 on AA GDP.pdf - Criterion Pass Rate: 82.25 (#2)
Claude Opus 5.5 on AA GDP.pdf - Engineering - Criterion Pass Rate: 84.35 (#1)
Claude Opus 5.5 on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 91.25 (#1)
Claude Opus 5.5 on AA GDP.pdf - HR - Criterion Pass Rate: 90.16 (#1)
Claude Opus 5.5 on AA GDP.pdf - Healthcare - Criterion Pass Rate: 93.1 (#1)
Claude Opus 5.5 on AA GDP.pdf - Insurance - Criterion Pass Rate: 88.05 (#5)
Claude Opus 5.5 on AA GDP.pdf - Legal - Criterion Pass Rate: 84.79 (#11)
Claude Opus 5.5 on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 79.36 (#8)
Claude Opus 5.5 on AA GDP.pdf - Real Estate - Criterion Pass Rate: 72.66 (#27)
Claude Opus 5.5 on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 81.47 (#1)
Claude Opus 5.5 on AA GDPval: 1867.05 (#1)
Claude Opus 5.5 on AA Humanity's Last Exam: 61.35 (#1)
Claude Opus 5.5 on AA Long Context Reasoning: 84.67 (#6)
Claude Opus 5.5 on AA MMMU-Pro: 87.69 (#1)
Claude Opus 5.5 on AA Omniscience: 46.42 (#1)
Claude Opus 5.5 on AA SciCode: 66.9 (#1)
Claude Opus 5.5 on AA-AnalystAgent: 56.25 (#6)
Claude Opus 5.5 on AA-AnalystAgent - Pass@1: 64.25 (#5)
Claude Opus 5.5 on AA-AnalystAgent - Pass@K: 75.0 (#13)
Claude Opus 5.5 on AA-Briefcase: 61.21 (#5)
Claude Opus 5.5 on AA-Briefcase - Analytical Quality Elo: 2162.22 (#1)
Claude Opus 5.5 on AA-Briefcase - DOCX - Rubric Pass Rate: 60.36 (#3)
Claude Opus 5.5 on AA-Briefcase - PDF - Rubric Pass Rate: 64.66 (#4)
Claude Opus 5.5 on AA-Briefcase - PPTX - Rubric Pass Rate: 62.04 (#4)
Claude Opus 5.5 on AA-Briefcase - Presentation Elo: 1710.53 (#2)
Claude Opus 5.5 on AA-Briefcase - XLSX - Rubric Pass Rate: 61.14 (#3)
Claude Opus 5.5 on AA-LCR: 84.67 (#5)
Claude Opus 5.5 on AA-Omniscience Hallucination Rate: 58.61 (#144)
Claude Opus 5.5 on AA-Omniscience Index - Business: 32.7 (#11)
Claude Opus 5.5 on AA-Omniscience Index - Health: 26.4 (#2)
Claude Opus 5.5 on AA-Omniscience Index - Humanities & Social Sciences: 44.7 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Law: 48.1 (#4)
Claude Opus 5.5 on AA-Omniscience Index - Science, Engineering & Mathematics: 37.8 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE): 88.8 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - C: 96.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Dart: 90.0 (#2)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Go: 88.0 (#7)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - HTML: 88.0 (#5)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Java: 80.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 88.18 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Julia: 84.0 (#10)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 96.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - PHP: 96.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Python: 91.5 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - R: 72.0 (#2)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Rust: 92.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Swift: 92.0 (#4)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 93.33 (#6)
Claude Opus 5.5 on AA-Omniscience Net Score: 58.0 (#1)
Claude Opus 5.5 on Agent Arena: 13.82 (#2)
Claude Opus 5.5 on Agent Arena - Bash Recovery: 12.78 (#3)
Claude Opus 5.5 on Agent Arena - Confirmed Success: 14.12 (#4)
Claude Opus 5.5 on Agent Arena - Praise vs Complaint: 31.23 (#3)
Claude Opus 5.5 on Agent Arena - Steerability: 10.48 (#4)
Claude Opus 5.5 on Agent Arena - Tool Hallucination: -0.47 (#6)
Claude Opus 5.5 on Artificial Analysis Intelligence Index: 57.62 (#1)
Claude Opus 5.5 on AutomationBench-AA: 69.54 (#3)
Claude Opus 5.5 on AutomationBench-AA - Finance - Raw Objective Completion: 88.25 (#18)
Claude Opus 5.5 on AutomationBench-AA - HR - Raw Objective Completion: 77.27 (#4)
Claude Opus 5.5 on AutomationBench-AA - Marketing - Raw Objective Completion: 91.07 (#3)
Claude Opus 5.5 on AutomationBench-AA - Operations - Raw Objective Completion: 96.74 (#4)
Claude Opus 5.5 on AutomationBench-AA - Sales - Raw Objective Completion: 89.49 (#3)
Claude Opus 5.5 on AutomationBench-AA - Strict: 41.25 (#4)
Claude Opus 5.5 on AutomationBench-AA - Support - Raw Objective Completion: 91.11 (#9)
Claude Opus 5.5 on BenchGecko Score: 96.8 (#3)
Claude Opus 5.5 on Bullshit Benchmark: 52.7 (#39)
Claude Opus 5.5 on Chatbot Arena (Text - Math): 1510.0 (#8)
Claude Opus 5.5 on Chatbot Arena (Text - Mathematical): 1521.0 (#6)
Claude Opus 5.5 on Chatbot Arena (Text - Medicine & Healthcare): 1539.0 (#2)
Claude Opus 5.5 on ChessBench GitHub - Elo: 2280.0 (#20)
Claude Opus 5.5 on Creative Writing (Lechmazur): 3.767 (#3)
Claude Opus 5.5 on CritPt: 31.7 (#5)
Claude Opus 5.5 on Crosby micro1 RedlineBench: 53.5 (#5)
Claude Opus 5.5 on Design Arena (SVG): 1335.0 (#4)
Claude Opus 5.5 on Design Arena (UI Components): 1381.0 (#3)
Claude Opus 5.5 on Drone-Bench - Follow: 82.8 (#3)
Claude Opus 5.5 on Drone-Bench - Localize: 95.0 (#2)
Claude Opus 5.5 on Drone-Bench - Navigate: 96.7 (#2)
Claude Opus 5.5 on EKI Estonian LLM - Book Recommendations: 79.64 (#10)
Claude Opus 5.5 on EKI Estonian LLM - Idioms: 62.0 (#14)
Claude Opus 5.5 on EKI Estonian LLM - Language Advice: 68.75 (#4)
Claude Opus 5.5 on EKI Estonian LLM - Overall: 63.68 (#11)
Claude Opus 5.5 on EKI Estonian LLM - Propaganda Resistance: 93.24 (#10)
Claude Opus 5.5 on EKI Estonian LLM - Terminology: 56.45 (#8)
Claude Opus 5.5 on EKI Estonian LLM - Trivia: 64.4 (#18)
Claude Opus 5.5 on EKI Estonian LLM - Trivia Hard: 21.25 (#17)
Claude Opus 5.5 on Epoch AI - Apex Agents: 73.5 (#2)
Claude Opus 5.5 on Epoch AI - Critpt: 31.71 (#3)
Claude Opus 5.5 on Epoch AI - Cursorbench: 57.8 (#1)
Claude Opus 5.5 on Epoch AI - Dtbench: 98.93 (#1)
Claude Opus 5.5 on Epoch AI - Frontiermath Tier 4 V2: 95.0 (#6)
Claude Opus 5.5 on Epoch AI - Frontiermath Tiers 1 3 V2: 91.23 (#3)
Claude Opus 5.5 on Epoch AI - Frontierswe: 62.33 (#2)
Claude Opus 5.5 on Epoch AI - GPQA Diamond: 90.59 (#40)
Claude Opus 5.5 on Epoch AI - Lmca: 68.23 (#1)
Claude Opus 5.5 on Epoch AI - Mirrorcode: 77.4 (#1)
Claude Opus 5.5 on Epoch AI - Mystery Game Puzzles: 71.0 (#3)
Claude Opus 5.5 on Epoch AI - Proofbench: 100.0 (#2)
Claude Opus 5.5 on Epoch AI - Scicode: 66.9 (#1)
Claude Opus 5.5 on Finance Agent v2: 58.6 (#10)
Claude Opus 5.5 on FrontierCode: 65.3 (#1)
Claude Opus 5.5 on FrontierMath - Tier 4 (v2): 95.0 (#6)
Claude Opus 5.5 on FrontierMath - Tiers 1-3 (v2): 91.23 (#3)
Claude Opus 5.5 on FrontierSWE V2: 62.33 (#2)
Claude Opus 5.5 on Gray Swan IPI — Combined Attack Set, Transfer Only, No Computer Use: 1.0 (#2)
Claude Opus 5.5 on Harvey LAB-AA: 91.24 (#17)
Claude Opus 5.5 on Harvey LAB-AA - All Pass: 8.33 (#16)
Claude Opus 5.5 on Harvey LAB-AA - Antitrust & Competition: 92.03 (#11)
Claude Opus 5.5 on Harvey LAB-AA - Arbitration & International Dispute Resolution: 94.62 (#9)
Claude Opus 5.5 on Harvey LAB-AA - Banking & Finance: 94.42 (#7)
Claude Opus 5.5 on Harvey LAB-AA - Bankruptcy & Restructuring: 92.04 (#9)
Claude Opus 5.5 on Harvey LAB-AA - Capital Markets: 94.9 (#11)
Claude Opus 5.5 on Harvey LAB-AA - Corporate Governance: 90.41 (#15)
Claude Opus 5.5 on Harvey LAB-AA - Corporate M&A: 94.87 (#12)
Claude Opus 5.5 on Harvey LAB-AA - Data Privacy & Cybersecurity: 98.59 (#3)
Claude Opus 5.5 on Harvey LAB-AA - Emerging Companies & Venture Capital: 94.88 (#5)
Claude Opus 5.5 on Harvey LAB-AA - Employment & Labor: 88.39 (#11)
Claude Opus 5.5 on Harvey LAB-AA - Energy & Natural Resources: 96.04 (#17)
Claude Opus 5.5 on Harvey LAB-AA - Environmental & ESG: 86.74 (#21)
Claude Opus 5.5 on Harvey LAB-AA - Funds & Asset Management: 89.8 (#12)
Claude Opus 5.5 on Harvey LAB-AA - Healthcare & Life Sciences: 93.07 (#10)
Claude Opus 5.5 on Harvey LAB-AA - Immigration: 94.91 (#1)
Claude Opus 5.5 on Harvey LAB-AA - Insurance: 94.14 (#5)
Claude Opus 5.5 on Harvey LAB-AA - Intellectual Property: 96.11 (#7)
Claude Opus 5.5 on Harvey LAB-AA - International Trade & Sanctions: 91.27 (#9)
Claude Opus 5.5 on Harvey LAB-AA - Litigation & Dispute Resolution: 94.38 (#8)
Claude Opus 5.5 on Harvey LAB-AA - Real Estate: 94.46 (#16)
Claude Opus 5.5 on Harvey LAB-AA - Structured Finance & Securitization: 88.89 (#11)
Claude Opus 5.5 on Harvey LAB-AA - Tax: 85.28 (#22)
Claude Opus 5.5 on Harvey LAB-AA - Trusts, Estates & Private Client: 93.51 (#12)
Claude Opus 5.5 on Harvey LAB-AA - White Collar Defense & Investigations: 88.89 (#20)
Claude Opus 5.5 on LABBench2 Aggregate: 73.1 (#3)
Claude Opus 5.5 on LLM Chess (Saplin): 1575.3 (#3)
Claude Opus 5.5 on LVBench: 83.7 (#6)
Claude Opus 5.5 on Lech Mazur Writing: 3.767 (#3)
Claude Opus 5.5 on MILU: 93.1 (#2)
Claude Opus 5.5 on MLCR-AA: 66.67 (#3)
Claude Opus 5.5 on MLCR-AA - Accuracy: 93.15 (#7)
Claude Opus 5.5 on MLCR-AA - Completeness: 88.36 (#3)
Claude Opus 5.5 on MLCR-AA - Conciseness: 81.11 (#88)
Claude Opus 5.5 on NonoBench: 93.3 (#3)
Claude Opus 5.5 on OTIS Mock AIME 2024-25: 100.0 (#4)
Claude Opus 5.5 on Professional Reasoning Bench - Legal: 47.58 (#16)
Claude Opus 5.5 on RSI-Exam: 0.536 (#1)
Claude Opus 5.5 on ReLE - Agents - BFCL-V3: 74.1 (#6)
Claude Opus 5.5 on ReLE - Agents and Tool Use: 70.8 (#8)
Claude Opus 5.5 on ReLE - Education: 62.5 (#31)
Claude Opus 5.5 on ReLE - Education - Gaokao: 66.4 (#7)
Claude Opus 5.5 on ReLE - Education - High School Subjects: 75.1 (#10)
Claude Opus 5.5 on ReLE - Education - Middle School Subjects: 57.7 (#10)
Claude Opus 5.5 on ReLE - Education - Primary School Subjects: 50.7 (#115)
Claude Opus 5.5 on ReLE - Finance: 84.6 (#22)
Claude Opus 5.5 on ReLE - Language and Instruction Following: 74.7 (#10)
Claude Opus 5.5 on ReLE - Law - Lawyer Qualification Exam: 76.7 (#47)
Claude Opus 5.5 on ReLE - Law and Civil Service: 83.3 (#32)
Claude Opus 5.5 on ReLE - Medicine - Physician Exams: 91.7 (#3)
Claude Opus 5.5 on ReLE - Medicine and Mental Health: 89.0 (#14)
Claude Opus 5.5 on ReLE - Reasoning - BBH: 83.2 (#36)
Claude Opus 5.5 on ReLE - Reasoning and Mathematics: 90.6 (#2)
Claude Opus 5.5 on SimpleQA Verified: 72.2 (#4)
Claude Opus 5.5 on SpacetimeDB LLM Benchmark (C#): 97.4 (#4)
Claude Opus 5.5 on SpacetimeDB LLM Benchmark (TypeScript): 99.5 (#1)
Claude Opus 5.5 on Supabase Evals (No Skills): 91.3 (#1)
Claude Opus 5.5 on Supabase Evals (Skills): 89.9 (#1)
Claude Opus 5.5 on Terminal-Bench-Science: 63.3333 (#2)
Claude Opus 5.5 on Terminal-Bench-Science 0.1: 63.3 (#2)
Claude Opus 5.5 on Vals AI CUA-bench: 14.0 (#2)
Claude Opus 5.5 on Vals AI CyberBench: 74.58 (#7)
Claude Opus 5.5 on Vals AI ProgramBench: 18.5 (#1)
Claude Opus 5.5 on Vals AI Public Benefits Bench: 70.64 (#3)
Claude Opus 5.5 on Vals AI ReverseEngBench: 33.59 (#4)
Claude Opus 5.5 on Vibe Code Bench v1.1: 90.3 (#3)
Claude Opus 5.5 on WebDev Arena: 1820.31 (#1)
Claude Opus 5.5 on WebDev Arena (Fullstack): 1736.0 (#2)
Claude Sonnet 5.5 on AA CritPt: 31.43 (#9)
Claude Sonnet 5.5 on AA GDP.pdf: 25.8 (#24)
Claude Sonnet 5.5 on AA GDP.pdf - Construction - Criterion Pass Rate: 69.11 (#46)
Claude Sonnet 5.5 on AA GDP.pdf - Criterion Pass Rate: 80.6 (#13)
Claude Sonnet 5.5 on AA GDP.pdf - Engineering - Criterion Pass Rate: 82.17 (#6)
Claude Sonnet 5.5 on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 90.78 (#3)
Claude Sonnet 5.5 on AA GDP.pdf - HR - Criterion Pass Rate: 87.47 (#5)
Claude Sonnet 5.5 on AA GDP.pdf - Healthcare - Criterion Pass Rate: 90.36 (#14)
Claude Sonnet 5.5 on AA GDP.pdf - Insurance - Criterion Pass Rate: 86.98 (#10)
Claude Sonnet 5.5 on AA GDP.pdf - Legal - Criterion Pass Rate: 83.42 (#25)
Claude Sonnet 5.5 on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 78.73 (#13)
Claude Sonnet 5.5 on AA GDP.pdf - Real Estate - Criterion Pass Rate: 71.97 (#29)
Claude Sonnet 5.5 on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 79.54 (#7)
Claude Sonnet 5.5 on AA GDPval: 1840.08 (#2)
Claude Sonnet 5.5 on AA Humanity's Last Exam: 54.96 (#16)
Claude Sonnet 5.5 on AA Long Context Reasoning: 82.67 (#40)
Claude Sonnet 5.5 on AA Omniscience: 32.3 (#40)
Claude Sonnet 5.5 on AA SciCode: 61.0 (#8)
Claude Sonnet 5.5 on AA-AnalystAgent: 57.5 (#2)
Claude Sonnet 5.5 on AA-AnalystAgent - Pass@1: 69.5 (#2)
Claude Sonnet 5.5 on AA-AnalystAgent - Pass@K: 78.75 (#3)
Claude Sonnet 5.5 on AA-Briefcase: 65.15 (#1)
Claude Sonnet 5.5 on AA-Briefcase - Analytical Quality Elo: 2133.71 (#2)
Claude Sonnet 5.5 on AA-Briefcase - DOCX - Rubric Pass Rate: 60.95 (#2)
Claude Sonnet 5.5 on AA-Briefcase - PDF - Rubric Pass Rate: 69.83 (#2)
Claude Sonnet 5.5 on AA-Briefcase - PPTX - Rubric Pass Rate: 62.86 (#3)
Claude Sonnet 5.5 on AA-Briefcase - Presentation Elo: 1716.42 (#1)
Claude Sonnet 5.5 on AA-Briefcase - XLSX - Rubric Pass Rate: 66.36 (#1)
Claude Sonnet 5.5 on AA-LCR: 82.67 (#23)
Claude Sonnet 5.5 on AA-Omniscience Accuracy: 53.95 (#42)
Claude Sonnet 5.5 on AA-Omniscience Hallucination Rate: 47.01 (#96)
Claude Sonnet 5.5 on AA-Omniscience Index - Business: 19.3 (#47)
Claude Sonnet 5.5 on AA-Omniscience Index - Health: 9.7 (#42)
Claude Sonnet 5.5 on AA-Omniscience Index - Humanities & Social Sciences: 30.8 (#29)
Claude Sonnet 5.5 on AA-Omniscience Index - Law: 26.6 (#48)
Claude Sonnet 5.5 on AA-Omniscience Index - Science, Engineering & Mathematics: 29.1 (#11)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE): 78.3 (#20)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - C: 87.0 (#18)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Dart: 82.0 (#9)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Go: 82.0 (#21)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - HTML: 74.0 (#29)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Java: 58.0 (#20)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 78.18 (#22)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Julia: 92.0 (#1)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 72.0 (#30)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - PHP: 76.0 (#27)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Python: 85.5 (#11)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - R: 62.0 (#10)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Rust: 86.0 (#6)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Swift: 80.0 (#20)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 85.56 (#26)
Claude Sonnet 5.5 on AA-Omniscience Net Score: 35.0 (#8)
Claude Sonnet 5.5 on AI Chess Leaderboard (Continuation): 1249.0 (#39)
Claude Sonnet 5.5 on AI Chess Leaderboard (Reasoning): 1365.0 (#37)
Claude Sonnet 5.5 on AI Coding Daily (Claude Code) - Laravel Code Quality: 18.68 (#2)
Claude Sonnet 5.5 on AI Coding Daily (Claude Code) - React-TS Code Quality: 18.5 (#3)
Claude Sonnet 5.5 on AI Coding Daily (Claude Code) - Total: 64.31 (#3)
Claude Sonnet 5.5 on AIMultiple - Agentic RAG Routing: 85.33 (#2)
Claude Sonnet 5.5 on AIMultiple - FinanceReasoning (Hard): 89.08 (#13)
Claude Sonnet 5.5 on AIMultiple - Text-to-SQL (Adjudicated): 83.2 (#2)
Claude Sonnet 5.5 on AIMultiple - Text-to-SQL (Strict Execution Match): 59.4 (#4)
Claude Sonnet 5.5 on ActiveVision: 41.2 (#2)
Claude Sonnet 5.5 on ActiveVision (Agents): 78.8 (#3)
Claude Sonnet 5.5 on Agent Arena: 12.52 (#3)
Claude Sonnet 5.5 on Agent Arena - Bash Recovery: 15.05 (#1)
Claude Sonnet 5.5 on Agent Arena - Confirmed Success: 13.91 (#5)
Claude Sonnet 5.5 on Agent Arena - Praise vs Complaint: 26.27 (#5)
Claude Sonnet 5.5 on Agent Arena - Steerability: 6.9 (#8)
Claude Sonnet 5.5 on Agent Arena - Tool Hallucination: -0.47 (#7)
Claude Sonnet 5.5 on Aiera Leaderboard - Depth: 62.0 (#24)
Claude Sonnet 5.5 on Aiera Leaderboard - Facts: 54.34 (#29)
Claude Sonnet 5.5 on Aiera Leaderboard - Grounding: 75.7 (#28)
Claude Sonnet 5.5 on Aiera Leaderboard - Research Score: 64.01 (#29)
Claude Sonnet 5.5 on Artificial Analysis Intelligence Index: 56.0 (#3)
Claude Sonnet 5.5 on AutomationBench-AA: 71.77 (#2)
Claude Sonnet 5.5 on AutomationBench-AA - Finance - Raw Objective Completion: 92.47 (#3)
Claude Sonnet 5.5 on AutomationBench-AA - HR - Raw Objective Completion: 79.22 (#2)
Claude Sonnet 5.5 on AutomationBench-AA - Marketing - Raw Objective Completion: 88.65 (#11)
Claude Sonnet 5.5 on AutomationBench-AA - Operations - Raw Objective Completion: 97.47 (#1)
Claude Sonnet 5.5 on AutomationBench-AA - Sales - Raw Objective Completion: 89.45 (#4)
Claude Sonnet 5.5 on AutomationBench-AA - Strict: 45.51 (#2)
Claude Sonnet 5.5 on AutomationBench-AA - Support - Raw Objective Completion: 92.02 (#3)
Claude Sonnet 5.5 on BenchGecko Score: 89.2 (#7)
Claude Sonnet 5.5 on BenchLM: 83.4 (#3)
Claude Sonnet 5.5 on BridgeBench Overall: 647.0 (#4)
Claude Sonnet 5.5 on Bug Hunt Bench: 51.3 (#1)
Claude Sonnet 5.5 on Bug Hunt Bench - LMS: 28.7 (#1)
Claude Sonnet 5.5 on Bug Hunt Bench - VS Code Extension: 22.7 (#2)
Claude Sonnet 5.5 on Chatbot Arena (Code): 1786.0 (#3)
Claude Sonnet 5.5 on Chatbot Arena (Text - Business, Management & Financial Ops): 1464.0 (#53)
Claude Sonnet 5.5 on Chatbot Arena (Text - Chinese): 1533.0 (#16)
Claude Sonnet 5.5 on Chatbot Arena (Text - Coding): 1536.0 (#13)
Claude Sonnet 5.5 on Chatbot Arena (Text - Creative Writing): 1447.0 (#42)
Claude Sonnet 5.5 on Chatbot Arena (Text - English): 1492.0 (#17)
Claude Sonnet 5.5 on Chatbot Arena (Text - Entertainment, Sports & Media): 1441.0 (#51)
Claude Sonnet 5.5 on Chatbot Arena (Text - Expert): 1556.0 (#1)
Claude Sonnet 5.5 on Chatbot Arena (Text - Hard Prompts (English)): 1518.0 (#10)
Claude Sonnet 5.5 on Chatbot Arena (Text - Hard Prompts): 1504.0 (#24)
Claude Sonnet 5.5 on Chatbot Arena (Text - Instruction Following): 1484.0 (#18)
Claude Sonnet 5.5 on Chatbot Arena (Text - Legal & Government): 1483.0 (#35)
Claude Sonnet 5.5 on Chatbot Arena (Text - Life, Physical & Social Science): 1488.0 (#45)
Claude Sonnet 5.5 on Chatbot Arena (Text - Longer Query): 1493.0 (#24)
Claude Sonnet 5.5 on Chatbot Arena (Text - Medicine & Healthcare): 1483.0 (#52)
Claude Sonnet 5.5 on Chatbot Arena (Text - Multi-Turn): 1473.0 (#51)
Claude Sonnet 5.5 on Chatbot Arena (Text - Non-English): 1450.0 (#55)
Claude Sonnet 5.5 on Chatbot Arena (Text - Russian): 1451.0 (#73)
Claude Sonnet 5.5 on Chatbot Arena (Text - Software & IT Services): 1522.0 (#20)
Claude Sonnet 5.5 on Chatbot Arena (Text - Writing, Literature & Language): 1461.0 (#34)
Claude Sonnet 5.5 on Chatbot Arena (Text): 1471.0 (#45)
Claude Sonnet 5.5 on Chatbot Arena (Vision - Diagram): 1306.0 (#31)
Claude Sonnet 5.5 on Chatbot Arena (Vision - English): 1267.0 (#39)
Claude Sonnet 5.5 on Chatbot Arena (Vision - Homework): 1317.0 (#21)
Claude Sonnet 5.5 on Chatbot Arena (Vision - OCR): 1277.0 (#41)
Claude Sonnet 5.5 on Chatbot Arena (Vision): 1268.0 (#35)
Claude Sonnet 5.5 on ChessBench: 32.1 (#24)
Claude Sonnet 5.5 on ChessBench - Accuracy: 73.5 (#10)
Claude Sonnet 5.5 on ChessBench - Coherence: 50.7 (#34)
Claude Sonnet 5.5 on ChessBench - Elo: 802.0 (#28)
Claude Sonnet 5.5 on Coarena: 1003.2 (#8)
Claude Sonnet 5.5 on Coarena - Task Completion: 53.3 (#18)
Claude Sonnet 5.5 on CritPt: 31.4 (#7)
Claude Sonnet 5.5 on CursorBench 4.0: 55.5 (#2)
Claude Sonnet 5.5 on Epoch AI - Apex Agents: 75.5 (#1)
Claude Sonnet 5.5 on Epoch AI - Critpt: 31.43 (#8)
Claude Sonnet 5.5 on Epoch AI - Cursorbench: 55.5 (#3)
Claude Sonnet 5.5 on Epoch AI - ECI: 165.2 (#3)
Claude Sonnet 5.5 on Epoch AI - Frontiermath Tier 4 V2: 80.49 (#13)
Claude Sonnet 5.5 on Epoch AI - Frontiermath Tiers 1 3 V2: 88.77 (#7)
Claude Sonnet 5.5 on Epoch AI - Frontierswe: 61.89 (#3)
Claude Sonnet 5.5 on Epoch AI - Furniture Assembly: 75.0 (#4)
Claude Sonnet 5.5 on Epoch AI - GPQA Diamond: 95.58 (#2)
Claude Sonnet 5.5 on Epoch AI - Mystery Game Puzzles: 65.0 (#4)
Claude Sonnet 5.5 on Epoch AI - Proofbench: 100.0 (#3)
Claude Sonnet 5.5 on Epoch AI - Scicode: 61.0 (#5)
Claude Sonnet 5.5 on FrontierMath - Tier 4 (v2): 80.49 (#13)
Claude Sonnet 5.5 on FrontierMath - Tiers 1-3 (v2): 88.77 (#7)
Claude Sonnet 5.5 on FrontierSWE V2: 61.89 (#3)
Claude Sonnet 5.5 on Gert Labs Rankings: 72.14 (#6)
Claude Sonnet 5.5 on Guesswork 2026-09: 0.5361 (#2)
Claude Sonnet 5.5 on HLE-Diamond: 43.3 (#9)
Claude Sonnet 5.5 on HLE-Diamond - Knowledge: 32.6 (#10)
Claude Sonnet 5.5 on HLE-Diamond - Reasoning: 54.0 (#10)
Claude Sonnet 5.5 on Harvey LAB-AA: 93.07 (#8)
Claude Sonnet 5.5 on Harvey LAB-AA - All Pass: 11.67 (#10)
Claude Sonnet 5.5 on Harvey LAB-AA - Antitrust & Competition: 92.03 (#12)
Claude Sonnet 5.5 on Harvey LAB-AA - Arbitration & International Dispute Resolution: 95.34 (#7)
Claude Sonnet 5.5 on Harvey LAB-AA - Banking & Finance: 94.8 (#4)
Claude Sonnet 5.5 on Harvey LAB-AA - Bankruptcy & Restructuring: 92.73 (#6)
Claude Sonnet 5.5 on Harvey LAB-AA - Capital Markets: 96.86 (#3)
Claude Sonnet 5.5 on Harvey LAB-AA - Corporate Governance: 91.1 (#13)
Claude Sonnet 5.5 on Harvey LAB-AA - Corporate M&A: 95.83 (#3)
Claude Sonnet 5.5 on Harvey LAB-AA - Data Privacy & Cybersecurity: 98.94 (#1)
Claude Sonnet 5.5 on Harvey LAB-AA - Emerging Companies & Venture Capital: 94.88 (#4)
Claude Sonnet 5.5 on Harvey LAB-AA - Employment & Labor: 89.89 (#4)
Claude Sonnet 5.5 on Harvey LAB-AA - Energy & Natural Resources: 96.7 (#10)
Claude Sonnet 5.5 on Harvey LAB-AA - Environmental & ESG: 90.68 (#8)
Claude Sonnet 5.5 on Harvey LAB-AA - Funds & Asset Management: 90.38 (#8)
Claude Sonnet 5.5 on Harvey LAB-AA - Healthcare & Life Sciences: 94.06 (#6)
Claude Sonnet 5.5 on Harvey LAB-AA - Immigration: 94.18 (#2)
Claude Sonnet 5.5 on Harvey LAB-AA - Insurance: 93.77 (#10)
Claude Sonnet 5.5 on Harvey LAB-AA - Intellectual Property: 95.41 (#13)
Claude Sonnet 5.5 on Harvey LAB-AA - International Trade & Sanctions: 91.67 (#5)
Claude Sonnet 5.5 on Harvey LAB-AA - Litigation & Dispute Resolution: 94.76 (#6)
Claude Sonnet 5.5 on Harvey LAB-AA - Real Estate: 95.04 (#10)
Claude Sonnet 5.5 on Harvey LAB-AA - Structured Finance & Securitization: 90.16 (#6)
Claude Sonnet 5.5 on Harvey LAB-AA - Tax: 89.97 (#8)
Claude Sonnet 5.5 on Harvey LAB-AA - Trusts, Estates & Private Client: 95.8 (#1)
Claude Sonnet 5.5 on Harvey LAB-AA - White Collar Defense & Investigations: 93.6 (#7)
Claude Sonnet 5.5 on Icelandic LLM - Case Government: 95.31 (#16)
Claude Sonnet 5.5 on Icelandic LLM - Famous Phrases: 17.97 (#21)
Claude Sonnet 5.5 on Icelandic LLM - Flora & Fauna: 36.28 (#19)
Claude Sonnet 5.5 on Icelandic LLM - GEC: 67.43 (#16)
Claude Sonnet 5.5 on Icelandic LLM - Generative Quality: 74.93 (#14)
Claude Sonnet 5.5 on Icelandic LLM - Idiom Completion: 59.76 (#19)
Claude Sonnet 5.5 on Icelandic LLM - Idiom Meanings: 79.31 (#22)
Claude Sonnet 5.5 on Icelandic LLM - Song Lyrics: 16.28 (#14)
Claude Sonnet 5.5 on Icelandic LLM - Textile Terms: 65.4 (#16)
Claude Sonnet 5.5 on Icelandic LLM - WikiQA: 59.79 (#18)
Claude Sonnet 5.5 on Icelandic LLM Leaderboard - Average: 62.12 (#18)
Claude Sonnet 5.5 on Kaggle Adversarial Customer Service (Gert Labs): 77.5 (#3)
Claude Sonnet 5.5 on Kaggle AssetOpsBench: 67.42 (#5)
Claude Sonnet 5.5 on Kaggle ExtractBench: 89.01 (#5)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Arabic: 93.0 (#13)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Bengali: 91.75 (#14)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Burmese: 90.25 (#18)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Chinese: 93.75 (#13)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Culturally Agnostic: 92.75 (#15)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Culturally Sensitive: 92.53 (#15)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - English: 94.75 (#12)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - French: 93.75 (#14)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - German: 95.0 (#10)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Hindi: 92.0 (#16)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Indonesian: 93.75 (#12)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Italian: 93.75 (#16)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Japanese: 92.75 (#17)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Korean: 92.75 (#15)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Portuguese: 92.5 (#17)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Spanish: 95.25 (#8)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Swahili: 92.5 (#13)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Yoruba: 84.75 (#26)
Claude Sonnet 5.5 on Kaggle Open Benchmarks - AIME 2025: 96.67 (#11)
Claude Sonnet 5.5 on Kaggle Open Benchmarks - SciCode Subproblems: 43.3 (#3)
Claude Sonnet 5.5 on Kaggle SimpleQA: 46.05 (#16)
Claude Sonnet 5.5 on Kaggle SimpleQA Verified: 44.75 (#14)
Claude Sonnet 5.5 on KernelBench Hub - CUDA: 94.42 (#2)
Claude Sonnet 5.5 on KernelBench Hub - Mega: 32.13 (#2)
Claude Sonnet 5.5 on KernelBench Mega: 32.131 (#3)
Claude Sonnet 5.5 on LLM Stats (AutomationBench): 44.7 (#6)
Claude Sonnet 5.5 on LLM Stats (BioMysteryBench): 89.2 (#3)
Claude Sonnet 5.5 on LLM Stats (DeepSWE 1.1): 71.0 (#9)
Claude Sonnet 5.5 on LLM Stats (FrontierCode 1.1): 52.1 (#5)
Claude Sonnet 5.5 on LLM Stats (Humanity's Last Exam (no tools, text-only)): 56.9 (#2)
Claude Sonnet 5.5 on LLM Stats (Legal Agent Benchmark): 11.7 (#2)
Claude Sonnet 5.5 on LLM Stats (OfficeQA Pro): 65.6 (#6)
Claude Sonnet 5.5 on LLM Stats (Program Bench): 79.7 (#2)
Claude Sonnet 5.5 on LLM Stats (SWE-Bench Multimodal): 54.3 (#3)
Claude Sonnet 5.5 on LLM Stats (SWE-bench Multilingual): 90.3 (#2)
Claude Sonnet 5.5 on LLM Stats Score: 58.04 (#3)
Claude Sonnet 5.5 on LLM2014 Logic 2026-09: 66.5 (#6)
Claude Sonnet 5.5 on LM Market Cap LMC Score: 85.2 (#66)
Claude Sonnet 5.5 on LOL Arena: 46.7 (#28)
Claude Sonnet 5.5 on Legal Benchmarks - Contract Workflows - Criteria Pass Rate: 85.3 (#6)
Claude Sonnet 5.5 on Legal Benchmarks - Contract Workflows - Form: 91.1 (#3)
Claude Sonnet 5.5 on Legal Benchmarks - Contract Workflows - Task Pass Rate: 29.0 (#6)
Claude Sonnet 5.5 on Legal Benchmarks - Data Extraction - Criteria Pass Rate: 83.3 (#7)
Claude Sonnet 5.5 on Legal Benchmarks - Data Extraction - Form: 77.6 (#6)
Claude Sonnet 5.5 on Legal Benchmarks - Data Extraction - Task Pass Rate: 40.0 (#16)
Claude Sonnet 5.5 on LiveBench AMPS Hard: 98.0 (#41)
Claude Sonnet 5.5 on LiveBench Code Completion: 86.96 (#5)
Claude Sonnet 5.5 on LiveBench Code Generation: 95.78 (#1)
Claude Sonnet 5.5 on LiveBench Connections: 99.33 (#31)
Claude Sonnet 5.5 on LiveBench Consecutive Events: 90.94 (#4)
Claude Sonnet 5.5 on LiveBench Integrals With Game: 100.0 (#4)
Claude Sonnet 5.5 on LiveBench JavaScript: 77.27 (#8)
Claude Sonnet 5.5 on LiveBench Logic With Navigation: 82.0 (#13)
Claude Sonnet 5.5 on LiveBench Math Comp: 97.06 (#19)
Claude Sonnet 5.5 on LiveBench Olympiad: 92.47 (#5)
Claude Sonnet 5.5 on LiveBench Paraphrase: 62.28 (#48)
Claude Sonnet 5.5 on LiveBench Plot Unscrambling: 67.79 (#19)
Claude Sonnet 5.5 on LiveBench Python: 35.0 (#64)
Claude Sonnet 5.5 on LiveBench Simplify: 72.37 (#5)
Claude Sonnet 5.5 on LiveBench Spatial: 98.0 (#35)
Claude Sonnet 5.5 on LiveBench Story Generation: 77.08 (#11)
Claude Sonnet 5.5 on LiveBench Summarize: 70.33 (#33)
Claude Sonnet 5.5 on LiveBench Table Join: 46.88 (#41)
Claude Sonnet 5.5 on LiveBench Table Reformat: 98.04 (#51)
Claude Sonnet 5.5 on LiveBench Theory of Mind: 86.54 (#6)
Claude Sonnet 5.5 on LiveBench TypeScript: 56.67 (#8)
Claude Sonnet 5.5 on LiveBench Typos: 86.0 (#12)
Claude Sonnet 5.5 on LiveBench Zebra Puzzle: 100.0 (#16)
Claude Sonnet 5.5 on MILU: 91.6 (#9)
Claude Sonnet 5.5 on MLCR-AA: 75.0 (#1)
Claude Sonnet 5.5 on MLCR-AA - Accuracy: 90.75 (#9)
Claude Sonnet 5.5 on MLCR-AA - Completeness: 88.44 (#2)
Claude Sonnet 5.5 on MLCR-AA - Conciseness: 96.11 (#51)
Claude Sonnet 5.5 on Mercor APEX: 75.5 (#2)
Claude Sonnet 5.5 on MineBench: 2047.0 (#5)
Claude Sonnet 5.5 on NonoBench: 80.0 (#24)
Claude Sonnet 5.5 on OTIS Mock AIME 2024-25: 100.0 (#2)
Claude Sonnet 5.5 on OpenRouter GPQA Diamond: 92.1 (#17)
Claude Sonnet 5.5 on OpenRouter Tau2-Bench Airline: 74.7 (#35)
Claude Sonnet 5.5 on PLCC - Art & Entertainment: 72.0 (#72)
Claude Sonnet 5.5 on PLCC - Culture & Tradition: 82.0 (#72)
Claude Sonnet 5.5 on PLCC - Geography: 92.0 (#55)
Claude Sonnet 5.5 on PLCC - Grammar: 91.0 (#24)
Claude Sonnet 5.5 on PLCC - History: 95.0 (#19)
Claude Sonnet 5.5 on PLCC - Overall: 85.0 (#49)
Claude Sonnet 5.5 on PLCC - Vocabulary: 78.0 (#61)
Claude Sonnet 5.5 on ParseBench: 70.18 (#24)
Claude Sonnet 5.5 on ParseBench - Charts: 37.69 (#52)
Claude Sonnet 5.5 on ParseBench - Content Faithfulness: 91.64 (#4)
Claude Sonnet 5.5 on ParseBench - Semantic Formatting: 71.63 (#17)
Claude Sonnet 5.5 on ParseBench - Tables: 91.04 (#10)
Claude Sonnet 5.5 on ParseBench - Visual Grounding: 58.92 (#44)
Claude Sonnet 5.5 on RuneBench: 3394.0 (#25)
Claude Sonnet 5.5 on SimpleBench: 75.9 (#15)
Claude Sonnet 5.5 on SimpleQA Verified: 46.5 (#35)
Claude Sonnet 5.5 on SpeechMap Compliance: 81.6 (#83)
Claude Sonnet 5.5 on SvelteBench: 100.0 (#6)
Claude Sonnet 5.5 on TaxCalcBench: 18.0 (#26)
Claude Sonnet 5.5 on Tinybird AI SQL Benchmark - Exactness: 60.18 (#6)
Claude Sonnet 5.5 on Tinybird AI SQL Benchmark - First-Attempt Success Rate: 92.0 (#206)
Claude Sonnet 5.5 on Tinybird AI SQL Benchmark - Success Rate: 100.0 (#52)
Claude Sonnet 5.5 on Vals AI (Vals Index): 67.04 (#2)
Claude Sonnet 5.5 on Vals AI CyberBench: 59.58 (#33)
Claude Sonnet 5.5 on Vals AI Excel Modeling: 75.71 (#3)
Claude Sonnet 5.5 on Vals AI Finance Agent v2: 58.1 (#10)
Claude Sonnet 5.5 on Vals AI Harvey Legal Agent Bench: 2.92 (#40)
Claude Sonnet 5.5 on Vals AI IOI: 83.06 (#9)
Claude Sonnet 5.5 on Vals AI Legal Research Bench: 48.08 (#10)
Claude Sonnet 5.5 on Vals AI MedCode: 52.92 (#12)
Claude Sonnet 5.5 on Vals AI MedScribe: 91.1 (#3)
Claude Sonnet 5.5 on Vals AI MysteryMechanism: 49.1 (#3)
Claude Sonnet 5.5 on Vals AI ProgramBench: 6.5 (#3)
Claude Sonnet 5.5 on Vals AI ProofBench: 100.0 (#1)
Claude Sonnet 5.5 on Vals AI Public Benefits Bench: 67.19 (#13)
Claude Sonnet 5.5 on Vals AI ReverseEngBench: 30.15 (#6)
Claude Sonnet 5.5 on Vals AI SAGE: 51.77 (#11)
Claude Sonnet 5.5 on Vals AI Tax Agent Bench: 73.39 (#4)
Claude Sonnet 5.5 on Vals AI Terminal-Bench 2.1: 83.15 (#7)
Claude Sonnet 5.5 on Vals AI Terminal-Bench 4.0: 64.14 (#2)
Claude Sonnet 5.5 on Vals AI Terminal-Bench Science: 45.71 (#4)
Claude Sonnet 5.5 on Vellum - Humanity's Last Exam: 64.5 (#6)
Claude Sonnet 5.5 on VoxelBench: 1000.0 (#55)
Claude Sonnet 5.5 on WebDev Arena: 1698.82 (#5)
Claude Sonnet 5.5 on WebDev Arena (Brand & Marketing): 1767.0 (#2)
Claude Sonnet 5.5 on WebDev Arena (Consumer Product): 1671.0 (#4)
Claude Sonnet 5.5 on WebDev Arena (Content Creation Tools): 1730.0 (#4)
Claude Sonnet 5.5 on WebDev Arena (Data & Analytics): 1621.0 (#6)
Claude Sonnet 5.5 on WebDev Arena (Frontend): 1822.0 (#2)
Claude Sonnet 5.5 on WebDev Arena (Fullstack): 1646.0 (#10)
Claude Sonnet 5.5 on WebDev Arena (Gaming): 1892.0 (#2)
Claude Sonnet 5.5 on WebDev Arena (HTML): 1735.0 (#3)
Claude Sonnet 5.5 on WebDev Arena (React): 1836.0 (#2)
Claude Sonnet 5.5 on WebDev Arena (Reference-Based Design): 1832.0 (#2)
Claude Sonnet 5.5 on WebDev Arena (Simulations): 1862.0 (#3)
GPT-6 Astra on AA CritPt: 31.71 (#4)
GPT-6 Astra on AA GDP.pdf: 32.2 (#1)
GPT-6 Astra on AA GDP.pdf - Construction - Criterion Pass Rate: 84.51 (#2)
GPT-6 Astra on AA GDP.pdf - Criterion Pass Rate: 82.05 (#3)
GPT-6 Astra on AA GDP.pdf - Engineering - Criterion Pass Rate: 82.86 (#3)
GPT-6 Astra on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 89.19 (#7)
GPT-6 Astra on AA GDP.pdf - HR - Criterion Pass Rate: 85.49 (#10)
GPT-6 Astra on AA GDP.pdf - Healthcare - Criterion Pass Rate: 92.51 (#3)
GPT-6 Astra on AA GDP.pdf - Insurance - Criterion Pass Rate: 79.9 (#28)
GPT-6 Astra on AA GDP.pdf - Legal - Criterion Pass Rate: 85.11 (#6)
GPT-6 Astra on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 73.73 (#36)
GPT-6 Astra on AA GDP.pdf - Real Estate - Criterion Pass Rate: 79.15 (#7)
GPT-6 Astra on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 79.89 (#5)
GPT-6 Astra on AA GDPval: 1541.89 (#40)
GPT-6 Astra on AA GPQA Diamond: 96.26 (#1)
GPT-6 Astra on AA Humanity's Last Exam: 54.68 (#22)
GPT-6 Astra on AA Long Context Reasoning: 80.67 (#73)
GPT-6 Astra on AA MMMU-Pro: 86.88 (#3)
GPT-6 Astra on AA Omniscience: 43.73 (#3)
GPT-6 Astra on AA SciCode: 56.48 (#45)
GPT-6 Astra on AA-AnalystAgent: 51.25 (#10)
GPT-6 Astra on AA-AnalystAgent - Pass@1: 60.0 (#13)
GPT-6 Astra on AA-AnalystAgent - Pass@K: 67.5 (#28)
GPT-6 Astra on AA-Briefcase: 52.32 (#28)
GPT-6 Astra on AA-Briefcase - Analytical Quality Elo: 1776.98 (#22)
GPT-6 Astra on AA-Briefcase - DOCX - Rubric Pass Rate: 53.85 (#22)
GPT-6 Astra on AA-Briefcase - PDF - Rubric Pass Rate: 50.86 (#37)
GPT-6 Astra on AA-Briefcase - PPTX - Rubric Pass Rate: 48.98 (#30)
GPT-6 Astra on AA-Briefcase - Presentation Elo: 1535.47 (#10)
GPT-6 Astra on AA-Briefcase - XLSX - Rubric Pass Rate: 54.09 (#21)
GPT-6 Astra on AA-LCR: 80.67 (#36)
GPT-6 Astra on AA-Omniscience Accuracy: 62.6 (#11)
GPT-6 Astra on AA-Omniscience Hallucination Rate: 44.77 (#88)
GPT-6 Astra on AA-Omniscience Index - Business: 34.2 (#2)
GPT-6 Astra on AA-Omniscience Index - Health: 27.0 (#1)
GPT-6 Astra on AA-Omniscience Index - Humanities & Social Sciences: 43.1 (#2)
GPT-6 Astra on AA-Omniscience Index - Law: 48.5 (#3)
GPT-6 Astra on AA-Omniscience Index - Science, Engineering & Mathematics: 29.3 (#10)
GPT-6 Astra on AA-Omniscience Index - Software Engineering (SWE): 84.2 (#4)
GPT-6 Astra on AA-Omniscience Index - Software Engineering (SWE) - C: 91.0 (#7)
GPT-6 Astra on AA-Omniscience Index - Software Engineering (SWE) - Dart: 82.0 (#10)
GPT-6 Astra on AA-Omniscience Index - Software Engineering (SWE) - Go: 92.0 (#1)
GPT-6 Astra on AA-Omniscience Index - Software Engineering (SWE) - HTML: 92.0 (#1)
GPT-6 Astra on AA-Omniscience Index - Software Engineering (SWE) - Java: 75.0 (#3)
GPT-6 Astra on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 88.18 (#2)
GPT-6 Astra on AA-Omniscience Index - Software Engineering (SWE) - Julia: 92.0 (#2)
GPT-6 Astra on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 88.0 (#4)
GPT-6 Astra on AA-Omniscience Index - Software Engineering (SWE) - PHP: 86.0 (#9)
GPT-6 Astra on AA-Omniscience Index - Software Engineering (SWE) - Python: 89.0 (#4)
GPT-6 Astra on AA-Omniscience Index - Software Engineering (SWE) - R: 62.0 (#11)
GPT-6 Astra on AA-Omniscience Index - Software Engineering (SWE) - Rust: 86.0 (#7)
GPT-6 Astra on AA-Omniscience Index - Software Engineering (SWE) - Swift: 88.0 (#11)
GPT-6 Astra on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 97.78 (#1)
GPT-6 Astra on AIIQ Composite IQ: 142.0 (#2)
GPT-6 Astra on Agents' Last Exam: 34.2 (#2)
GPT-6 Astra on Artificial Analysis Intelligence Index: 52.67 (#13)
GPT-6 Astra on AutomationBench: 41.4 (#5)
GPT-6 Astra on AutomationBench-AA: 68.49 (#5)
GPT-6 Astra on AutomationBench-AA - Finance - Raw Objective Completion: 89.24 (#14)
GPT-6 Astra on AutomationBench-AA - HR - Raw Objective Completion: 77.08 (#5)
GPT-6 Astra on AutomationBench-AA - Marketing - Raw Objective Completion: 84.42 (#36)
GPT-6 Astra on AutomationBench-AA - Operations - Raw Objective Completion: 97.01 (#2)
GPT-6 Astra on AutomationBench-AA - Sales - Raw Objective Completion: 85.91 (#15)
GPT-6 Astra on AutomationBench-AA - Strict: 41.55 (#3)
GPT-6 Astra on AutomationBench-AA - Support - Raw Objective Completion: 92.43 (#2)
GPT-6 Astra on BenchGecko Score: 96.3 (#4)
GPT-6 Astra on Benchmarks.bio - BioSecBench-Function: 48.6 (#2)
GPT-6 Astra on Benchmarks.bio - BioSecBench-Refusal: 44.6 (#4)
GPT-6 Astra on Benchmarks.bio - BioSecBench-Surveillance: 48.5 (#2)
GPT-6 Astra on Benchmarks.bio - SpatialBench: 70.14 (#7)
GPT-6 Astra on Benchmarks.bio - TxBench-AD: 56.9 (#1)
GPT-6 Astra on Benchmarks.bio - TxBench-OD: 55.46 (#1)
GPT-6 Astra on Benchmarks.bio - VariantBench: 48.6 (#2)
GPT-6 Astra on Benchmarks.bio - scBench: 64.79 (#1)
GPT-6 Astra on BioSecBench-Function: 46.0 (#1)
GPT-6 Astra on BrowseComp: 91.5 (#1)
GPT-6 Astra on CAIS Risk Index: 45.8 (#14)
GPT-6 Astra on Chatbot Arena (Text - Japanese): 1418.0 (#68)
GPT-6 Astra on Chatbot Arena (Vision - Chinese): 1358.0 (#3)
GPT-6 Astra on Chatbot Arena (Vision - Creative Writing): 1285.0 (#15)
GPT-6 Astra on Chatbot Arena (Vision - Homework): 1342.0 (#3)
GPT-6 Astra on CritPt: 31.7 (#4)
GPT-6 Astra on DeepSWE: 74.1 (#1)
GPT-6 Astra on DeepSWE-mini (16 Tasks): 81.25 (#1)
GPT-6 Astra on Design Arena (ASCII Art): 1340.0 (#3)
GPT-6 Astra on Design Arena (Agents - Web Apps): 1397.0 (#1)
GPT-6 Astra on Design Arena (Data Viz): 1324.0 (#9)
GPT-6 Astra on Design Arena (Game Dev): 1479.0 (#2)
GPT-6 Astra on Design Arena (UI Components): 1380.0 (#4)
GPT-6 Astra on Design Arena (Website): 1340.0 (#5)
GPT-6 Astra on Finance Agent v2: 53.5 (#21)
GPT-6 Astra on FrontierCode: 64.5 (#6)
GPT-6 Astra on FrontierSWE V2: 65.51 (#1)
GPT-6 Astra on GPQA Diamond (BenchmarkList): 96.1 (#1)
GPT-6 Astra on GSO-Bench: 79.41 (#2)
GPT-6 Astra on Gray Swan IPI — Combined Attack Set, Transfer Only, No Computer Use: 8.5 (#7)
GPT-6 Astra on Guesswork 2026-09: 0.5423 (#3)
GPT-6 Astra on Kaggle Adversarial Customer Service (Gert Labs): 75.0 (#8)
GPT-6 Astra on LABBench2 Aggregate: 85.4 (#2)
GPT-6 Astra on LVBench: 87.5 (#3)
GPT-6 Astra on Lech Mazur Writing: 3.455 (#4)
GPT-6 Astra on MLCR-AA: 35.0 (#16)
GPT-6 Astra on MLCR-AA - Accuracy: 98.28 (#1)
GPT-6 Astra on MLCR-AA - Completeness: 36.78 (#22)
GPT-6 Astra on MLCR-AA - Conciseness: 96.67 (#34)
GPT-6 Astra on MazeBench: 14.0 (#1)
GPT-6 Astra on MedXpertQA MM (Natomy): 87.4 (#1)
GPT-6 Astra on MedXpertQA MM (Natomy) - Basic Science: 89.24 (#1)
GPT-6 Astra on MedXpertQA MM (Natomy) - Diagnosis: 86.99 (#1)
GPT-6 Astra on MedXpertQA MM (Natomy) - Treatment: 87.05 (#1)
GPT-6 Astra on NonoBench: 100.0 (#1)
GPT-6 Astra on PRBench Finance: 47.54 (#14)
GPT-6 Astra on ParseBench: 70.67 (#23)
GPT-6 Astra on ParseBench - Charts: 35.08 (#55)
GPT-6 Astra on ParseBench - Content Faithfulness: 89.6 (#18)
GPT-6 Astra on ParseBench - Semantic Formatting: 75.52 (#8)
GPT-6 Astra on ParseBench - Tables: 93.17 (#3)
GPT-6 Astra on ParseBench - Visual Grounding: 59.99 (#39)
GPT-6 Astra on Professional Reasoning Bench - Legal: 48.36 (#14)
GPT-6 Astra on SWE Atlas: 60.9 (#9)
GPT-6 Astra on SWE-Together: 80.09 (#11)
GPT-6 Astra on SnakeBench: 37.9 (#3)
GPT-6 Astra on Tau3 Banking: 43.09 (#26)
GPT-6 Astra on Terminal-Bench 4.0: 58.2 (#1)
GPT-6 Astra on Terminal-Bench-Science: 68.0952 (#1)
GPT-6 Astra on Thinkingbox: 58.31 (#6)
GPT-6 Astra on Vals AI ProgramBench: 5.5 (#4)
GPT-6 Astra on Vals AI ReverseEngBench: 56.87 (#1)
GPT-6 Astra on Vals AI Tax Agent Bench: 63.34 (#24)
GPT-6 Astra on Vals AI Time Horizon Index: 90.5 (#2)
GPT-6 Astra on VariantBench: 47.7 (#4)
GPT-6 Astra on Vectara Hallucination Leaderboard: 91.3 (#45)
GPT-6 Astra on Vibe Code Bench v1.1: 89.6 (#4)
GPT-6 Astra on WolfBench: 86.0 (#2)
GPT-6 Astra on You're Absolutely Right!: 3.0 (#11)
GPT-6 Astra Pro on OpenRouter GPQA Diamond: 95.5 (#2)
GPT-6 Astra Pro on OpenRouter Tau2-Bench Airline: 72.7 (#52)
GPT-6.1 Sol on AA CritPt: 31.71 (#5)
GPT-6.1 Sol on AA GDP.pdf: 32.0 (#2)
GPT-6.1 Sol on AA GDP.pdf - Construction - Criterion Pass Rate: 84.94 (#1)
GPT-6.1 Sol on AA GDP.pdf - Criterion Pass Rate: 82.86 (#1)
GPT-6.1 Sol on AA GDP.pdf - Engineering - Criterion Pass Rate: 81.27 (#10)
GPT-6.1 Sol on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 90.99 (#2)
GPT-6.1 Sol on AA GDP.pdf - HR - Criterion Pass Rate: 83.07 (#15)
GPT-6.1 Sol on AA GDP.pdf - Healthcare - Criterion Pass Rate: 91.78 (#5)
GPT-6.1 Sol on AA GDP.pdf - Insurance - Criterion Pass Rate: 78.78 (#34)
GPT-6.1 Sol on AA GDP.pdf - Legal - Criterion Pass Rate: 85.86 (#4)
GPT-6.1 Sol on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 77.89 (#18)
GPT-6.1 Sol on AA GDP.pdf - Real Estate - Criterion Pass Rate: 81.5 (#1)
GPT-6.1 Sol on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 79.37 (#9)
GPT-6.1 Sol on AA GDPval: 1575.09 (#35)
GPT-6.1 Sol on AA Humanity's Last Exam: 52.92 (#29)
GPT-6.1 Sol on AA Long Context Reasoning: 84.0 (#17)
GPT-6.1 Sol on AA MMMU-Pro: 85.95 (#8)
GPT-6.1 Sol on AA Omniscience: 41.53 (#16)
GPT-6.1 Sol on AA SciCode: 55.79 (#54)
GPT-6.1 Sol on AA-AnalystAgent: 50.0 (#11)
GPT-6.1 Sol on AA-AnalystAgent - Pass@1: 59.75 (#14)
GPT-6.1 Sol on AA-AnalystAgent - Pass@K: 72.5 (#19)
GPT-6.1 Sol on AA-Briefcase: 55.66 (#18)
GPT-6.1 Sol on AA-Briefcase - Analytical Quality Elo: 1806.84 (#19)
GPT-6.1 Sol on AA-Briefcase - DOCX - Rubric Pass Rate: 52.07 (#31)
GPT-6.1 Sol on AA-Briefcase - PDF - Rubric Pass Rate: 53.45 (#30)
GPT-6.1 Sol on AA-Briefcase - PPTX - Rubric Pass Rate: 50.2 (#26)
GPT-6.1 Sol on AA-Briefcase - Presentation Elo: 1437.87 (#34)
GPT-6.1 Sol on AA-Briefcase - XLSX - Rubric Pass Rate: 59.77 (#9)
GPT-6.1 Sol on AA-LCR: 84.0 (#7)
GPT-6.1 Sol on AA-Omniscience Accuracy: 62.1 (#12)
GPT-6.1 Sol on AA-Omniscience Hallucination Rate: 49.36 (#104)
GPT-6.1 Sol on AA-Omniscience Index - Business: 33.1 (#7)
GPT-6.1 Sol on AA-Omniscience Index - Health: 25.6 (#5)
GPT-6.1 Sol on AA-Omniscience Index - Humanities & Social Sciences: 40.3 (#8)
GPT-6.1 Sol on AA-Omniscience Index - Law: 44.0 (#12)
GPT-6.1 Sol on AA-Omniscience Index - Science, Engineering & Mathematics: 28.0 (#16)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE): 82.2 (#12)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - C: 90.0 (#10)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Dart: 64.0 (#29)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Go: 88.0 (#10)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - HTML: 92.0 (#4)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Java: 77.0 (#2)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 87.27 (#5)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Julia: 92.0 (#4)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 84.0 (#12)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - PHP: 84.0 (#12)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Python: 86.0 (#10)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - R: 64.0 (#8)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Rust: 88.0 (#5)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Swift: 84.0 (#16)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 95.56 (#3)
GPT-6.1 Sol on AI Chess Leaderboard (Continuation): 1540.0 (#11)
GPT-6.1 Sol on AI Chess Leaderboard (Reasoning): 1535.0 (#22)
GPT-6.1 Sol on AI Coding Daily (Codex CLI) - Laravel Code Quality: 19.1 (#1)
GPT-6.1 Sol on AI Coding Daily (Codex CLI) - React-TS Code Quality: 19.0 (#1)
GPT-6.1 Sol on AI Coding Daily (Codex CLI) - Total: 68.1 (#1)
GPT-6.1 Sol on AIIQ Composite IQ: 141.0 (#3)
GPT-6.1 Sol on AIMultiple - Agentic RAG Routing: 80.98 (#21)
GPT-6.1 Sol on AIMultiple - FinanceReasoning (Hard): 89.92 (#9)
GPT-6.1 Sol on AIMultiple - Text-to-SQL (Adjudicated): 81.8 (#7)
GPT-6.1 Sol on AIMultiple - Text-to-SQL (Strict Execution Match): 52.4 (#11)
GPT-6.1 Sol on ARC-AGI-1: 98.5 (#7)
GPT-6.1 Sol on ARC-AGI-2: 94.17 (#3)
GPT-6.1 Sol on ARC-AGI-3: 52.73 (#4)
GPT-6.1 Sol on Agent Arena: 11.23 (#5)
GPT-6.1 Sol on Agent Arena - Bash Recovery: 3.47 (#24)
GPT-6.1 Sol on Agent Arena - Confirmed Success: 15.47 (#2)
GPT-6.1 Sol on Agent Arena - Praise vs Complaint: 25.53 (#6)
GPT-6.1 Sol on Agent Arena - Steerability: 11.18 (#3)
GPT-6.1 Sol on Agent Arena - Tool Hallucination: -0.49 (#1)
GPT-6.1 Sol on Agent Security League - Functional Correctness: 77.7 (#9)
GPT-6.1 Sol on Agent Security League - Security Correctness: 34.1 (#3)
GPT-6.1 Sol on Aiera Leaderboard - Depth: 73.6 (#15)
GPT-6.1 Sol on Aiera Leaderboard - Facts: 61.58 (#23)
GPT-6.1 Sol on Aiera Leaderboard - Grounding: 74.5 (#30)
GPT-6.1 Sol on Aiera Leaderboard - Research Score: 69.89 (#22)
GPT-6.1 Sol on Artificial Analysis Intelligence Index: 51.83 (#18)
GPT-6.1 Sol on AutomationBench-AA: 66.57 (#10)
GPT-6.1 Sol on AutomationBench-AA - Finance - Raw Objective Completion: 89.3 (#12)
GPT-6.1 Sol on AutomationBench-AA - HR - Raw Objective Completion: 76.37 (#9)
GPT-6.1 Sol on AutomationBench-AA - Marketing - Raw Objective Completion: 85.42 (#32)
GPT-6.1 Sol on AutomationBench-AA - Operations - Raw Objective Completion: 96.82 (#3)
GPT-6.1 Sol on AutomationBench-AA - Sales - Raw Objective Completion: 85.28 (#21)
GPT-6.1 Sol on AutomationBench-AA - Strict: 37.14 (#9)
GPT-6.1 Sol on AutomationBench-AA - Support - Raw Objective Completion: 92.0 (#4)
GPT-6.1 Sol on BenchLM: 77.6 (#9)
GPT-6.1 Sol on BridgeBench Overall: 646.0 (#5)
GPT-6.1 Sol on Bug Hunt Bench: 44.3 (#3)
GPT-6.1 Sol on Bug Hunt Bench - LMS: 22.0 (#6)
GPT-6.1 Sol on Bug Hunt Bench - VS Code Extension: 22.3 (#3)
GPT-6.1 Sol on Chatbot Arena (Code): 1758.0 (#4)
GPT-6.1 Sol on Chatbot Arena (Text - Business, Management & Financial Ops): 1475.0 (#32)
GPT-6.1 Sol on Chatbot Arena (Text - Chinese): 1500.0 (#59)
GPT-6.1 Sol on Chatbot Arena (Text - Coding): 1542.0 (#8)
GPT-6.1 Sol on Chatbot Arena (Text - Creative Writing): 1460.0 (#27)
GPT-6.1 Sol on Chatbot Arena (Text - English): 1496.0 (#13)
GPT-6.1 Sol on Chatbot Arena (Text - Entertainment, Sports & Media): 1462.0 (#16)
GPT-6.1 Sol on Chatbot Arena (Text - Expert): 1543.0 (#4)
GPT-6.1 Sol on Chatbot Arena (Text - Hard Prompts (English)): 1519.0 (#8)
GPT-6.1 Sol on Chatbot Arena (Text - Hard Prompts): 1509.0 (#18)
GPT-6.1 Sol on Chatbot Arena (Text - Instruction Following): 1490.0 (#13)
GPT-6.1 Sol on Chatbot Arena (Text - Legal & Government): 1511.0 (#4)
GPT-6.1 Sol on Chatbot Arena (Text - Life, Physical & Social Science): 1503.0 (#21)
GPT-6.1 Sol on Chatbot Arena (Text - Longer Query): 1496.0 (#19)
GPT-6.1 Sol on Chatbot Arena (Text - Medicine & Healthcare): 1491.0 (#31)
GPT-6.1 Sol on Chatbot Arena (Text - Multi-Turn): 1493.0 (#20)
GPT-6.1 Sol on Chatbot Arena (Text - Non-English): 1469.0 (#26)
GPT-6.1 Sol on Chatbot Arena (Text - Russian): 1481.0 (#31)
GPT-6.1 Sol on Chatbot Arena (Text - Software & IT Services): 1536.0 (#5)
GPT-6.1 Sol on Chatbot Arena (Text - Writing, Literature & Language): 1472.0 (#20)
GPT-6.1 Sol on Chatbot Arena (Text): 1483.0 (#21)
GPT-6.1 Sol on Chatbot Arena (Vision - Diagram): 1308.0 (#28)
GPT-6.1 Sol on Chatbot Arena (Vision - English): 1294.0 (#7)
GPT-6.1 Sol on Chatbot Arena (Vision - Homework): 1361.0 (#1)
GPT-6.1 Sol on Chatbot Arena (Vision - OCR): 1296.0 (#22)
GPT-6.1 Sol on Chatbot Arena (Vision): 1291.0 (#10)
GPT-6.1 Sol on Chess Bench LLM: 1490.0 (#17)
GPT-6.1 Sol on Chess Puzzles (Epoch AI): 61.0 (#4)
GPT-6.1 Sol on ChessBench: 71.6 (#3)
GPT-6.1 Sol on ChessBench - Accuracy: 76.6 (#6)
GPT-6.1 Sol on ChessBench - Coherence: 83.7 (#5)
GPT-6.1 Sol on ChessBench - Elo: 1336.0 (#3)
GPT-6.1 Sol on ChessBench GitHub - Elo: 2550.0 (#7)
GPT-6.1 Sol on ClockBench: 74.4 (#5)
GPT-6.1 Sol on Coarena: 975.5 (#19)
GPT-6.1 Sol on Coarena - Task Completion: 54.5 (#17)
GPT-6.1 Sol on CritPt: 31.7 (#2)
GPT-6.1 Sol on Epoch AI - Apex Agents: 60.0 (#10)
GPT-6.1 Sol on Epoch AI - Critpt: 31.71 (#5)
GPT-6.1 Sol on Epoch AI - Ebr Bench: 54.29 (#4)
GPT-6.1 Sol on Epoch AI - Frontiermath Tier 4 V2: 100.0 (#1)
GPT-6.1 Sol on Epoch AI - Frontiermath Tiers 1 3 V2: 93.68 (#1)
GPT-6.1 Sol on Epoch AI - Furniture Assembly: 80.0 (#2)
GPT-6.1 Sol on Epoch AI - GPQA Diamond: 95.39 (#3)
GPT-6.1 Sol on Epoch AI - Mystery Game Puzzles: 80.0 (#2)
GPT-6.1 Sol on Epoch AI - Proofbench: 99.0 (#6)
GPT-6.1 Sol on Epoch AI - Scicode: 55.79 (#33)
GPT-6.1 Sol on FrontierMath - Tier 4 (v2): 100.0 (#1)
GPT-6.1 Sol on FrontierMath - Tiers 1-3 (v2): 93.68 (#1)
GPT-6.1 Sol on Gert Labs Rankings: 78.24 (#2)
GPT-6.1 Sol on Guesswork 2026-09: 0.5706 (#5)
GPT-6.1 Sol on HLE-Diamond: 61.1 (#2)
GPT-6.1 Sol on HLE-Diamond - Knowledge: 47.2 (#3)
GPT-6.1 Sol on HLE-Diamond - Reasoning: 75.0 (#2)
GPT-6.1 Sol on Icelandic LLM - Case Government: 97.19 (#8)
GPT-6.1 Sol on Icelandic LLM - Famous Phrases: 29.07 (#4)
GPT-6.1 Sol on Icelandic LLM - Flora & Fauna: 53.49 (#5)
GPT-6.1 Sol on Icelandic LLM - GEC: 74.31 (#8)
GPT-6.1 Sol on Icelandic LLM - Generative Quality: 78.81 (#5)
GPT-6.1 Sol on Icelandic LLM - Idiom Completion: 71.18 (#4)
GPT-6.1 Sol on Icelandic LLM - Idiom Meanings: 82.82 (#8)
GPT-6.1 Sol on Icelandic LLM - Song Lyrics: 14.05 (#24)
GPT-6.1 Sol on Icelandic LLM - Textile Terms: 72.9 (#4)
GPT-6.1 Sol on Icelandic LLM - WikiQA: 65.92 (#11)
GPT-6.1 Sol on Icelandic LLM Leaderboard - Average: 68.17 (#6)
GPT-6.1 Sol on LLM Chess (Saplin): 1454.4 (#11)
GPT-6.1 Sol on LLM Stats (DeepSWE 1.1): 75.22 (#2)
GPT-6.1 Sol on LLM Stats (GDP.pdf): 32.0 (#4)
GPT-6.1 Sol on LLM Stats (HealthBench Hard): 36.2 (#2)
GPT-6.1 Sol on LLM Stats (HealthBench Professional): 64.2 (#4)
GPT-6.1 Sol on LLM Stats (HealthBench): 58.5 (#4)
GPT-6.1 Sol on LLM Stats (OSWorld 2.0): 71.42 (#4)
GPT-6.1 Sol on LLM Stats Score: 52.04 (#11)
GPT-6.1 Sol on LM Market Cap LMC Score: 40.0 (#246)
GPT-6.1 Sol on LiveBench AMPS Hard: 98.0 (#43)
GPT-6.1 Sol on LiveBench Code Completion: 82.61 (#14)
GPT-6.1 Sol on LiveBench Code Generation: 80.28 (#24)
GPT-6.1 Sol on LiveBench Connections: 100.0 (#18)
GPT-6.1 Sol on LiveBench Consecutive Events: 91.29 (#3)
GPT-6.1 Sol on LiveBench Integrals With Game: 100.0 (#5)
GPT-6.1 Sol on LiveBench JavaScript: 63.64 (#46)
GPT-6.1 Sol on LiveBench Logic With Navigation: 86.0 (#7)
GPT-6.1 Sol on LiveBench Math Comp: 97.06 (#21)
GPT-6.1 Sol on LiveBench Olympiad: 92.27 (#6)
GPT-6.1 Sol on LiveBench Paraphrase: 72.58 (#18)
GPT-6.1 Sol on LiveBench Plot Unscrambling: 84.38 (#2)
GPT-6.1 Sol on LiveBench Python: 60.0 (#23)
GPT-6.1 Sol on LiveBench Simplify: 71.97 (#8)
GPT-6.1 Sol on LiveBench Spatial: 98.0 (#37)
GPT-6.1 Sol on LiveBench Story Generation: 70.82 (#36)
GPT-6.1 Sol on LiveBench Summarize: 81.25 (#8)
GPT-6.1 Sol on LiveBench Table Join: 57.42 (#2)
GPT-6.1 Sol on LiveBench Table Reformat: 100.0 (#24)
GPT-6.1 Sol on LiveBench Theory of Mind: 86.54 (#7)
GPT-6.1 Sol on LiveBench TypeScript: 46.67 (#25)
GPT-6.1 Sol on LiveBench Typos: 86.0 (#13)
GPT-6.1 Sol on LiveBench Zebra Puzzle: 100.0 (#18)
GPT-6.1 Sol on MLCR-AA: 33.89 (#17)
GPT-6.1 Sol on MLCR-AA - Accuracy: 95.4 (#3)
GPT-6.1 Sol on MLCR-AA - Completeness: 36.21 (#23)
GPT-6.1 Sol on MLCR-AA - Conciseness: 96.67 (#35)
GPT-6.1 Sol on MathArena - ARXIV August: 96.49 (#1)
GPT-6.1 Sol on MathArena - ARXIV June: 94.44 (#1)
GPT-6.1 Sol on MathArena - ARXIV_FALSE August: 86.31 (#1)
GPT-6.1 Sol on MathArena - ARXIV_FALSE June: 100.0 (#2)
GPT-6.1 Sol on MathArena Arxiv: 95.0 (#1)
GPT-6.1 Sol on MathArena Arxiv False: 90.0 (#3)
GPT-6.1 Sol on Maze-Bench: 88.57 (#2)
GPT-6.1 Sol on Mercor APEX: 60.0 (#12)
GPT-6.1 Sol on NonoBench: 93.3 (#5)
GPT-6.1 Sol on OTIS Mock AIME 2024-25: 100.0 (#1)
GPT-6.1 Sol on OpenRouter GPQA Diamond: 94.4 (#6)
GPT-6.1 Sol on OpenRouter Tau2-Bench Airline: 70.0 (#79)
GPT-6.1 Sol on PLCC - Art & Entertainment: 93.0 (#12)
GPT-6.1 Sol on PLCC - Culture & Tradition: 98.0 (#10)
GPT-6.1 Sol on PLCC - Geography: 98.0 (#15)
GPT-6.1 Sol on PLCC - Grammar: 98.0 (#2)
GPT-6.1 Sol on PLCC - History: 99.0 (#2)
GPT-6.1 Sol on PLCC - Overall: 96.83 (#3)
GPT-6.1 Sol on PLCC - Vocabulary: 95.0 (#5)
GPT-6.1 Sol on ProphetArena: 0.9773 (#3)
GPT-6.1 Sol on RuneBench: 21772.0 (#2)
GPT-6.1 Sol on SecIT Bench (Pydantic AI): 79.91 (#5)
GPT-6.1 Sol on SimpleBench: 82.9 (#5)
GPT-6.1 Sol on SimpleQA Verified: 73.9 (#2)
GPT-6.1 Sol on SvelteBench: 100.0 (#18)
GPT-6.1 Sol on TaxCalcBench: 36.0 (#15)
GPT-6.1 Sol on Tinybird AI SQL Benchmark - Exactness: 50.71 (#124)
GPT-6.1 Sol on Tinybird AI SQL Benchmark - First-Attempt Success Rate: 84.0 (#270)
GPT-6.1 Sol on Tinybird AI SQL Benchmark - Success Rate: 96.0 (#270)
GPT-6.1 Sol on UGI - Natural Intelligence: 78.26 (#7)
GPT-6.1 Sol on UGI - Willingness (W/10): 3.8 (#957)
GPT-6.1 Sol on UGI - Writing: 67.72 (#43)
GPT-6.1 Sol on UGI Leaderboard: 58.75 (#23)
GPT-6.1 Sol on Vals AI (Vals Index): 61.15 (#8)
GPT-6.1 Sol on Vals AI BioMysteryBench: 79.63 (#2)
GPT-6.1 Sol on Vals AI Code Migration: 65.12 (#5)
GPT-6.1 Sol on Vals AI CyberBench: 39.29 (#42)
GPT-6.1 Sol on Vals AI Excel Modeling: 70.77 (#12)
GPT-6.1 Sol on Vals AI Finance Agent v2: 52.03 (#31)
GPT-6.1 Sol on Vals AI Harvey Legal Agent Bench: 5.42 (#31)
GPT-6.1 Sol on Vals AI IOI: 96.89 (#3)
GPT-6.1 Sol on Vals AI Legal Research Bench: 38.46 (#31)
GPT-6.1 Sol on Vals AI MedCode: 48.84 (#26)
GPT-6.1 Sol on Vals AI MedScribe: 86.45 (#21)
GPT-6.1 Sol on Vals AI MysteryMechanism: 46.4 (#5)
GPT-6.1 Sol on Vals AI ProgramBench: 3.0 (#5)
GPT-6.1 Sol on Vals AI ProofBench: 99.0 (#5)
GPT-6.1 Sol on Vals AI Public Benefits Bench: 59.34 (#32)
GPT-6.1 Sol on Vals AI ReverseEngBench: 50.76 (#2)
GPT-6.1 Sol on Vals AI SAGE: 46.54 (#35)
GPT-6.1 Sol on Vals AI Tax Agent Bench: 62.31 (#27)
GPT-6.1 Sol on Vals AI Terminal-Bench 4.0: 55.05 (#6)
GPT-6.1 Sol on Vals AI Terminal-Bench Science: 52.86 (#2)
GPT-6.1 Sol on Vals AI Vibe Code Bench: 88.93 (#7)
GPT-6.1 Sol on VoxelBench: 2518.0 (#3)
GPT-6.1 Sol on WebDev Arena: 1758.72 (#3)
GPT-6.1 Sol on WebDev Arena (Brand & Marketing): 1690.0 (#5)
GPT-6.1 Sol on WebDev Arena (Consumer Product): 1721.0 (#1)
GPT-6.1 Sol on WebDev Arena (Content Creation Tools): 1739.0 (#3)
GPT-6.1 Sol on WebDev Arena (Data & Analytics): 1627.0 (#4)
GPT-6.1 Sol on WebDev Arena (Frontend): 1774.0 (#5)
GPT-6.1 Sol on WebDev Arena (Gaming): 1848.0 (#5)
GPT-6.1 Sol on WebDev Arena (HTML): 1800.0 (#1)
GPT-6.1 Sol on WebDev Arena (React): 1762.0 (#5)
GPT-6.1 Sol on WebDev Arena (Reference-Based Design): 1801.0 (#5)
GPT-6.1 Sol on WebDev Arena (Simulations): 1828.0 (#5)
Gemini 4 Argon on AA CritPt: 27.14 (#46)
Gemini 4 Argon on AA GDP.pdf: 21.8 (#49)
Gemini 4 Argon on AA GDP.pdf - Construction - Criterion Pass Rate: 74.97 (#17)
Gemini 4 Argon on AA GDP.pdf - Criterion Pass Rate: 79.91 (#17)
Gemini 4 Argon on AA GDP.pdf - Engineering - Criterion Pass Rate: 81.7 (#9)
Gemini 4 Argon on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 80.24 (#49)
Gemini 4 Argon on AA GDP.pdf - HR - Criterion Pass Rate: 79.63 (#30)
Gemini 4 Argon on AA GDP.pdf - Healthcare - Criterion Pass Rate: 85.57 (#36)
Gemini 4 Argon on AA GDP.pdf - Insurance - Criterion Pass Rate: 88.24 (#4)
Gemini 4 Argon on AA GDP.pdf - Legal - Criterion Pass Rate: 82.77 (#29)
Gemini 4 Argon on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 80.12 (#7)
Gemini 4 Argon on AA GDP.pdf - Real Estate - Criterion Pass Rate: 73.21 (#24)
Gemini 4 Argon on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 75.8 (#33)
Gemini 4 Argon on AA GDPval: 1627.02 (#21)
Gemini 4 Argon on AA Humanity's Last Exam: 57.09 (#9)
Gemini 4 Argon on AA Long Context Reasoning: 79.67 (#98)
Gemini 4 Argon on AA Omniscience: 42.35 (#14)
Gemini 4 Argon on AA SciCode: 61.81 (#7)
Gemini 4 Argon on AA-Briefcase: 64.63 (#2)
Gemini 4 Argon on AA-Briefcase - Analytical Quality Elo: 1566.4 (#46)
Gemini 4 Argon on AA-Briefcase - DOCX - Rubric Pass Rate: 62.13 (#1)
Gemini 4 Argon on AA-Briefcase - PDF - Rubric Pass Rate: 72.41 (#1)
Gemini 4 Argon on AA-Briefcase - PPTX - Rubric Pass Rate: 67.36 (#1)
Gemini 4 Argon on AA-Briefcase - Presentation Elo: 1307.56 (#61)
Gemini 4 Argon on AA-Briefcase - XLSX - Rubric Pass Rate: 61.14 (#4)
Gemini 4 Argon on AA-LCR: 79.67 (#53)
Gemini 4 Argon on AA-Omniscience Accuracy: 49.9 (#61)
Gemini 4 Argon on AA-Omniscience Hallucination Rate: 15.07 (#5)
Gemini 4 Argon on AA-Omniscience Index - Business: 33.3 (#5)
Gemini 4 Argon on AA-Omniscience Index - Health: 22.4 (#12)
Gemini 4 Argon on AA-Omniscience Index - Humanities & Social Sciences: 39.6 (#11)
Gemini 4 Argon on AA-Omniscience Index - Law: 53.0 (#1)
Gemini 4 Argon on AA-Omniscience Index - Science, Engineering & Mathematics: 34.4 (#2)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE): 71.4 (#27)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - C: 90.0 (#8)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - Dart: 68.0 (#24)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - Go: 76.0 (#27)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - HTML: 82.0 (#22)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - Java: 54.0 (#23)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 72.73 (#39)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - Julia: 52.0 (#44)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 54.0 (#49)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - PHP: 76.0 (#28)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - Python: 68.5 (#48)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - R: 58.0 (#16)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - Rust: 84.0 (#10)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - Swift: 76.0 (#22)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 80.0 (#34)
Gemini 4 Argon on Agent Arena: 7.57 (#10)
Gemini 4 Argon on Agent Arena - Bash Recovery: -18.93 (#48)
Gemini 4 Argon on Agent Arena - Confirmed Success: 15.44 (#3)
Gemini 4 Argon on Agent Arena - Praise vs Complaint: 27.72 (#4)
Gemini 4 Argon on Agent Arena - Steerability: 13.48 (#1)
Gemini 4 Argon on Agent Arena - Tool Hallucination: -0.13 (#31)
Gemini 4 Argon on Artificial Analysis Intelligence Index: 52.56 (#14)
Gemini 4 Argon on AutomationBench: 51.3 (#1)
Gemini 4 Argon on AutomationBench-AA: 77.51 (#1)
Gemini 4 Argon on AutomationBench-AA - Finance - Raw Objective Completion: 94.62 (#1)
Gemini 4 Argon on AutomationBench-AA - HR - Raw Objective Completion: 81.99 (#1)
Gemini 4 Argon on AutomationBench-AA - Marketing - Raw Objective Completion: 94.66 (#1)
Gemini 4 Argon on AutomationBench-AA - Operations - Raw Objective Completion: 96.39 (#8)
Gemini 4 Argon on AutomationBench-AA - Sales - Raw Objective Completion: 92.0 (#1)
Gemini 4 Argon on AutomationBench-AA - Strict: 53.12 (#1)
Gemini 4 Argon on AutomationBench-AA - Support - Raw Objective Completion: 94.72 (#1)
Gemini 4 Argon on BenchLM: 77.2 (#10)
Gemini 4 Argon on Chartography: 71.6 (#1)
Gemini 4 Argon on Chatbot Arena (Code): 1680.0 (#9)
Gemini 4 Argon on Chatbot Arena (Text - Business, Management & Financial Ops): 1522.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text - Chinese): 1587.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text - Coding): 1560.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text - Creative Writing): 1519.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text - English): 1533.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text - Entertainment, Sports & Media): 1521.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text - Expert): 1539.0 (#7)
Gemini 4 Argon on Chatbot Arena (Text - Hard Prompts (English)): 1554.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text - Hard Prompts): 1551.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text - Instruction Following): 1528.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text - Legal & Government): 1533.0 (#2)
Gemini 4 Argon on Chatbot Arena (Text - Life, Physical & Social Science): 1542.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text - Longer Query): 1544.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text - Math): 1530.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text - Mathematical): 1542.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text - Medicine & Healthcare): 1547.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text - Multi-Turn): 1554.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text - Non-English): 1512.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text - Russian): 1533.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text - Software & IT Services): 1563.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text - Spanish): 1497.0 (#4)
Gemini 4 Argon on Chatbot Arena (Text - Writing, Literature & Language): 1522.0 (#1)
Gemini 4 Argon on Chatbot Arena (Text): 1525.0 (#1)
Gemini 4 Argon on CritPt: 27.1 (#15)
Gemini 4 Argon on FrontierSWE V2: 54.96 (#5)
Gemini 4 Argon on Gray Swan IPI — Combined Attack Set, Transfer Only, No Computer Use: 0.7 (#1)
Gemini 4 Argon on Terminal-Bench-Science 0.1: 57.6 (#3)
Gemini 4 Argon on Vals AI (Vals Index): 68.9 (#1)
Gemini 4 Argon on Vals AI BioMysteryBench: 76.3 (#6)
Gemini 4 Argon on Vals AI CUA-bench: 4.83 (#7)
Gemini 4 Argon on Vals AI Code Migration: 68.17 (#2)
Gemini 4 Argon on Vals AI CyberBench: 77.86 (#2)
Gemini 4 Argon on Vals AI Excel Modeling: 75.24 (#4)
Gemini 4 Argon on Vals AI Harvey Legal Agent Bench: 19.58 (#5)
Gemini 4 Argon on Vals AI IOI: 100.0 (#1)
Gemini 4 Argon on Vals AI Legal Research Bench: 54.81 (#4)
Gemini 4 Argon on Vals AI LegalBench: 88.3 (#3)
Gemini 4 Argon on Vals AI MedCode: 58.8 (#3)
Gemini 4 Argon on Vals AI MedScribe: 87.43 (#15)
Gemini 4 Argon on Vals AI MysteryMechanism: 45.49 (#6)
Gemini 4 Argon on Vals AI ProgramBench: 2.5 (#7)
Gemini 4 Argon on Vals AI ProofBench: 99.0 (#7)
Gemini 4 Argon on Vals AI Public Benefits Bench: 69.76 (#5)
Gemini 4 Argon on Vals AI ReverseEngBench: 44.27 (#3)
Gemini 4 Argon on Vals AI SAGE: 53.65 (#4)
Gemini 4 Argon on Vals AI Tax Agent Bench: 76.23 (#2)
Gemini 4 Argon on Vals AI Terminal-Bench 4.0: 57.58 (#5)
Gemini 4 Argon on Vals AI Terminal-Bench Science: 44.29 (#5)
Gemini 4 Argon on Vals AI Vibe Code Bench: 91.91 (#2)
Gemini 4 Argon on Vending-Bench 2: 13718.16 (#3)
Gemini 4 Argon on WebDev Arena (Brand & Marketing): 1603.0 (#27)
Gemini 4 Argon on WebDev Arena (Consumer Product): 1609.0 (#12)
Gemini 4 Argon on WebDev Arena (Content Creation Tools): 1672.0 (#7)
Gemini 4 Argon on WebDev Arena (Data & Analytics): 1625.0 (#5)
Gemini 4 Argon on WebDev Arena (Frontend): 1691.0 (#9)
Gemini 4 Argon on WebDev Arena (Fullstack): 1681.0 (#5)
Gemini 4 Argon on WebDev Arena (Gaming): 1770.0 (#9)
Gemini 4 Argon on WebDev Arena (HTML): 1646.0 (#12)
Gemini 4 Argon on WebDev Arena (React): 1693.0 (#9)
Gemini 4 Argon on WebDev Arena (Reference-Based Design): 1723.0 (#8)
Gemini 4 Argon on WebDev Arena (Simulations): 1749.0 (#9)

New #1 Leaders (153)

Codeforces ELO: DeepSeek V4.1 Flash (Max Effort) (3471.0) beat Gemma 4 31B by 1321.0
Design Arena (SVG): arrow-2-telos (1593.0) beat GPT-6 Astra by 147.0
Benchmarking Large Language Models for Graphem: Gemini 3.1 Pro (Preview) (0.53) beat Qwen 3 8B by 99.62
Design Arena (Agents - Web Apps): GPT-6 Astra (Max) (1397.0) beat GPT-6 Astra by 66.0
MEGA-Bench Task - Move Position To Position Hanoi 4 Pole: Gemini 3.8 Flash (58.2) beat GPT-4o Mini by 47.5
MuPPET: Gemini 2.5 Pro (23.46) beat Qwen 3 8B by 46.91
GSO-Bench: Claude Fable 5.1 (88.24) beat Claude Opus 4.8 by 41.18
Design Arena (Agents - Mobile Apps): Claude Opus 5.5 (1328.0) beat Claude Fable 5 by 41.0
Design Arena (Data Viz): Claude Opus 5.5 (1398.0) beat Claude Fable 5.1 by 40.0
Design Arena (UI Components): Claude Sonnet 5.5 (1415.0) beat GPT-6 Astra by 39.0
Scammer4U: Claude Haiku 4.5 (54.5) beat Gemini 3 Flash by 38.6
Terminal-Bench-Science: GPT-6 Astra (Max) (68.0952) beat Claude Opus 5 (Max) by 38.1
NonoBench: GPT-6 Astra (xHigh) (100.0) beat Gemini 3.1 Pro (Preview) (High) by 36.7
Chatbot Arena (Text - Multi-Turn): Gemini 4 Argon (High) (1554.0) beat Claude Opus 4.6 (High) by 35.0
ClockBench: Top Human Performance (95.0) beat GPT-5.6 Sol (Max) by 28.3
Chatbot Arena (Text - Entertainment, Sports & Media): Gemini 4 Argon (High) (1521.0) beat Claude Fable 5 (High) by 27.0
WebDev Arena (HTML): GPT-6.1 Sol (Max) (1800.0) beat GPT-6 Astra (Max) by 26.0
Chatbot Arena (Text - Software & IT Services): Gemini 4 Argon (High) (1563.0) beat Claude Opus 4.7 (High) by 22.0
AA GDPval: Claude Opus 5.5 (Max, Default Fallback) (1867.05) beat Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) by 20.88
WebDev Arena: Claude Opus 5.5 (Max) (1820.31) beat GPT-6 Astra (Max) by 20.03
Chatbot Arena (Text - English): Gemini 4 Argon (High) (1533.0) beat Claude Opus 4.6 (High) by 19.0
Chatbot Arena (Text - Hard Prompts (English)): Gemini 4 Argon (High) (1554.0) beat Claude Opus 4.6 (High) by 18.0
Chatbot Arena (Vision - Homework): GPT-6.1 Sol (Max) (1361.0) beat GLM-5.3 Flash by 17.0
Chatbot Arena (Text): Gemini 4 Argon (High) (1525.0) beat Claude Opus 5.5 (High) by 16.0
Chatbot Arena (Text - Chinese): Gemini 4 Argon (High) (1587.0) beat Claude Fable 5.1 (Max) by 14.0
Chatbot Arena (Text - Longer Query): Gemini 4 Argon (High) (1544.0) beat Claude Opus 5.5 (High) by 14.0
WebDev Arena (Consumer Product): GPT-6.1 Sol (Max) (1721.0) beat GPT-6 Astra (Max) by 13.0
Chatbot Arena (Text - Instruction Following): Gemini 4 Argon (High) (1528.0) beat Claude Opus 5.5 (High) by 12.0
AutomationBench-AA - Strict: Gemini 4 Argon (High) (53.12) beat GPT-6 Astra (Max) by 11.57
Chatbot Arena (Text - Non-English): Gemini 4 Argon (High) (1512.0) beat Claude Opus 5.5 (High) by 11.0
Chatbot Arena (Text - Medicine & Healthcare): Gemini 4 Argon (High) (1547.0) beat Muse Spark 1.2 (xHigh) by 11.0
Chatbot Arena (Text - Hard Prompts): Gemini 4 Argon (High) (1551.0) beat Claude Opus 5.5 (High) by 10.0
Chatbot Arena (Text - Coding): Gemini 4 Argon (High) (1560.0) beat Claude Opus 4.6 (High) by 9.0
AutomationBench: Gemini 4 Argon (High) (51.3) beat Claude Opus 5.5 (default fallbacks, Max) by 8.8
Mercor APEX: Gemini 4 Argon (82.2) beat Claude Opus 5.5 by 8.7
MultiChallenge: DeepSeek V4.1 Flash (72.12) beat DeepSeek V4 Pro by 8.02
Chatbot Arena (Text - Russian): Gemini 4 Argon (High) (1533.0) beat Claude Opus 5.5 (High) by 8.0
AutomationBench-AA: Gemini 4 Argon (High) (77.51) beat Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) by 7.97
SWE-bench Live: Slingshot-v3.4.0 + GPT-5.6-Sol (70.33) beat AMI Agent + Claude-4.6-Opus by 7.33
Chatbot Arena (Text - Math): Gemini 4 Argon (High) (1530.0) beat Claude Fable 5 (High) by 7.0
Chatbot Arena (Text - Life, Physical & Social Science): Gemini 4 Argon (High) (1542.0) beat Claude Opus 5.5 (High) by 7.0
Chatbot Arena (Text - Mathematical): Gemini 4 Argon (High) (1542.0) beat Claude Opus 5 (High) by 7.0
Epoch AI - Apex Agents: Claude Sonnet 5.5 (Max) (75.5) beat Claude Fable 5.1 (Unknown) by 6.9
Medmarks - M-ARC: Muse Glimmer 30B (High) (86.67) beat Grok 4 by 6.67
AA-Briefcase - Presentation Elo: Claude Sonnet 5.5 (Max, Default Fallback) (1716.42) beat Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) by 6.51
Bug Hunt Bench: Claude Sonnet 5.5 (Max) (51.3) beat GPT-6 Astra (Max) by 6.3
Epoch AI - Cursorbench: Claude Opus 5.5 (Max) (57.8) beat Claude Fable 5.1 (Max) by 6.0
AIIQ Composite IQ: opus-5.5 (143.0) beat Claude Fable 5 by 6.0
CAIS Vision Capabilities Index: GPT-6 Astra (71.5) beat Gemini 3.5 Flash by 5.6
AA-LCR: Kimi K3 (Max) (88.67) beat Muse Spark 1.2 (xHigh) by 5.34
AA-Briefcase - XLSX - Rubric Pass Rate: Claude Sonnet 5.5 (Max, Default Fallback) (66.36) beat Claude Opus 5.5 (Adaptive Reasoning, Xhigh Effort, Default Fallback) by 5.22
LLM Stats (HealthBench): Claude Sonnet 5.5 (65.4) beat Claude Opus 5.5 by 4.8
AutomationBench-AA - HR - Raw Objective Completion: Gemini 4 Argon (High) (81.99) beat Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) by 4.72
PRBench Finance: Muse Spark 1.3 (59.54) beat Muse Spark 1.1 by 4.53
Professional Reasoning Bench - Legal: Muse Spark 1.3 (61.56) beat Muse Spark 1.1 by 4.51
MathArena - ARXIV_FALSE August: GPT-6.1 Sol (Max) (86.31) beat GPT-6 Astra (Max) by 4.37
EnterpriseRAG Bench: CDL (Causal Dynamics Lab) (90.91) beat Mixedbread (+ Opus 5) by 4.33
AA-Briefcase - PDF - Rubric Pass Rate: Gemini 4 Argon (High) (72.41) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 4.31
SWE-Together: Claude Opus 5.5 (85.18) beat Claude Opus 4.6 by 4.29
LiveBench Code Generation: Claude Sonnet 5.5 (Max) (95.78) beat Claude Fable 5 (Max) by 4.23
LLM Stats (Terminal-Bench 4.0): Claude Sonnet 5.5 (70.6) beat Claude Opus 5.5 by 4.2
Bug Hunt Bench - LMS: Claude Sonnet 5.5 (Max) (28.7) beat GPT-5.6 Sol (Max) by 4.2
Epoch AI - Mirrorcode: Claude Opus 5.5 (Max) (77.4) beat Claude Fable 5.1 (High) by 4.1
AA-Briefcase - PPTX - Rubric Pass Rate: Gemini 4 Argon (High) (67.36) beat Claude Fable 5.1 (Adaptive Reasoning, Xhigh Effort, Default Fallback) by 4.09
Chatbot Arena (Text - Korean): Claude Fable 5.1 (Max) (1514.0) beat Muse Spark 1.3 (Max) by 4.0
Finance Agent v2: Gemini 4 Argon (65.4) beat Gemini 3.8 Flash by 4.0
Vals AI Finance Agent v2: Gemini 4 Argon (65.4) beat Gemini 3.8 Flash by 3.96
LVBench: Gemini 4 Argon (91.7) beat Gemini 3.8 Flash by 3.9
MLCR-AA: Claude Sonnet 5.5 (Max, Default Fallback) (75.0) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 3.89
Epoch AI - Scicode: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (66.9) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 3.82
EnterpriseRAG Bench - Completeness: CDL (Causal Dynamics Lab) (94.37) beat Mixedbread (+ Opus 5) by 3.75
SpreadsheetBench v2: arito (54.51) beat WPS AI by 3.65
AA-Briefcase: Claude Sonnet 5.5 (Max, Default Fallback) (65.15) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 3.63
AutomationBench-AA - Marketing - Raw Objective Completion: Gemini 4 Argon (High) (94.66) beat MiMo-V2.6-Pro by 3.55
MathArena - ARXIV August: GPT-6.1 Sol (Max) (96.49) beat GPT-6 Sol (Max) by 3.51
Drone-Bench - Reconstruct: Claude Opus 5.5 (71.0) beat GPT-6 Astra by 3.5
Kaggle FACTS Grounding: GPT-6 Sol (85.22) beat GPT-6 Astra by 3.24
DecBench: GPT-6 Astra (60.0) beat Claude Opus 4.8 by 3.2
LLM Stats (HealthBench Professional): Claude Sonnet 5.5 (69.2) beat Claude Fable 5 by 3.2
SWE-Marathon: Grok 4.5 (29.0) beat Claude Opus 4.8 by 3.0
Chatbot Arena (Text - Business, Management & Financial Ops): Gemini 4 Argon (High) (1522.0) beat Muse Spark 1.2 (xHigh) by 3.0
NL2Repo: Claude Opus 5 (max effort) (75.3) beat Claude Opus 5 by 3.0
AA-Omniscience Index - Law: Gemini 4 Argon (High) (53.0) beat Claude Fable 5.1 (Adaptive Reasoning, Xhigh Effort, Default Fallback) by 2.9
Epoch AI - Lmca: Claude Opus 5.5 (Max) (68.23) beat Claude Fable 5.1 (Unknown) by 2.77
Vellum - Humanity's Last Exam: Claude Opus 5.5 (67.7) beat Claude Fable 5.1 by 2.7
GameDevBench: GPT-6.1 Sol (Codex, xHigh) (69.97) beat Claude Fable 5 (Claude Code, xHigh) by 2.67
Vals AI CyberBench: GPT-6 Sol (77.98) beat MiMo-V2.6-Flash by 2.62
EnterpriseRAG Bench - Correctness: CDL (Causal Dynamics Lab) (92.4) beat Mixedbread (+ Opus 5) by 2.6
HANDBOOK.md Agents: Claude Fable 5.1 (Max) (38.8) beat Claude Fable 5 (Adaptive Reasoning, Max Effort) by 2.6
LABBench2 Aggregate: Gemini 4 Argon (88.8) beat Gemini 3.8 Flash by 2.6
BALSAM - Summarization: Nojoom-AQL-42B (56.58) beat allam-2-34B-prod (Reasoning) by 2.57
Agent Arena - Bash Recovery: Claude Sonnet 5.5 (Max) (15.05) beat Claude Opus 5 (Max) by 2.47
FrontierMath - Tier 4 (v2): GPT-6.1 Sol (Max) (100.0) beat GPT-6 Astra (High) by 2.4
Epoch AI - Frontiermath Tier 4 V2: GPT-6.1 Sol (Max) (100.0) beat GPT-6 Astra (High) by 2.4
AutomationBench-AA - Support - Raw Objective Completion: Gemini 4 Argon (High) (94.72) beat GPT-6 Astra (xHigh) by 2.29
Drone-Bench: Claude Opus 5.5 (97.3) beat GPT-6 Astra by 2.2
Vals AI Code Migration: Claude Sonnet 5.5 (69.83) beat GPT-6 Astra by 2.09
Vals AI Vibe Code Bench: Claude Sonnet 5.5 (92.39) beat Claude Fable 5 by 2.04
Chatbot Arena (Text - Writing, Literature & Language): Gemini 4 Argon (High) (1522.0) beat Claude Opus 5.5 (High) by 2.0
TrackingAI IQ Test (Vision): Claude Opus 5 (Max) (100.0) beat GPT-5.6 Sol (Ultra) by 1.96
AutomationBench-AA - Sales - Raw Objective Completion: Gemini 4 Argon (High) (92.0) beat GPT-6 Sol (xHigh) by 1.93
ReLE - Medicine - Basic Medical Knowledge: Claude Opus 5.5 (91.1) beat GPT-6 Astra by 1.9
Vals AI BioMysteryBench: Claude Sonnet 5.5 (81.11) beat GPT-6 Astra by 1.85
Medmarks - SuperGPQA Medicine Hard: Gemma 4 31B (Thinking) (62.52) beat Gemini 3 Pro (Preview) by 1.84
AA-Briefcase - DOCX - Rubric Pass Rate: Gemini 4 Argon (High) (62.13) beat Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) by 1.77
LLM Stats (BenchCAD): Claude Sonnet 5.5 (74.7) beat Claude Opus 5.5 by 1.7
ReLE - Overall: Claude Opus 5.5 (78.6) beat Qwen 3.7 Max by 1.7
AA GDP.pdf - Real Estate - Criterion Pass Rate: GPT-6.1 Sol (xHigh) (81.5) beat GPT-6 Sol (High) by 1.66
SpacetimeDB LLM Benchmark (Rust): Claude Opus 5.5 (100.0) beat GPT-6 Astra by 1.6
Vibe Code Bench v1.1: Gemini 4 Argon (91.9) beat Claude Fable 5 by 1.55
Supabase Evals (No Skills): Claude Opus 5.5 (High) (91.3) beat Claude Opus 5 (High) by 1.4
VLA-Arena Extrapolation: Paimon (68.89) beat MolmoAct2 by 1.26
AutomationBench-AA - Finance - Raw Objective Completion: Gemini 4 Argon (High) (94.62) beat Grok 4.7 (xHigh) by 1.13
Explicit Edit Benchmark: GPT-6.1 Sol (100.0) beat DeepSeek V4.1 Flash by 1.0
OpenRouter GPQA Diamond: Gemini 3.8 Flash (95.6) beat Fugu Ultra by 1.0
MazeBench: GPT-6 Astra (Max) (14.0) beat GPT-5.6 Sol (Max) by 1.0
VLA-Arena Overall: Paimon (73.08) beat MolmoAct2 by 0.96
Drone-Bench - Detect: Claude Opus 5.5 (88.7) beat Claude Fable 5 by 0.9
Vals AI Time Horizon Index: Claude Opus 5.5 (91.33) beat GPT-6 Astra (Max) by 0.83
AI Coding Daily (Codex CLI) - React-TS Code Quality: GPT-6.1 Sol (High) (19.0) beat GPT-6 Astra (High) by 0.83
Medmarks - LongHealth Task 1: Qwen3.8 27B Thinking (xHigh) (92.08) beat Gemini 3 Pro (Preview) by 0.75
Epoch AI - ECI: Claude Opus 5.5 (167.35) beat GPT-6 Astra by 0.75
Harvey LAB-AA - Healthcare & Life Sciences: Step 5 Preview (96.04) beat Kimi K3 (Max) by 0.66
AA GDP.pdf - Criterion Pass Rate: GPT-6.1 Sol (High) (82.86) beat Claude Opus 5.5 (Adaptive Reasoning, High Effort, Default Fallback) by 0.61
Chartography: Gemini 4 Argon (High) (71.6) beat GPT-6 Astra (Max) by 0.6
ReLE - Coding: Claude Opus 5.5 (76.3) beat Qwen 3.6 Max Preview by 0.6
Epoch AI - Dtbench: Claude Opus 5.5 (Max) (98.93) beat Claude Fable 5 (Max) by 0.53
AI Coding Daily (Codex CLI) - Laravel Code Quality: GPT-6.1 Sol (High) (19.1) beat GPT-6 Astra (High) by 0.5
AutomationBench-AA - Operations - Raw Objective Completion: Claude Sonnet 5.5 (Max, Default Fallback) (97.47) beat GPT-6 Astra (Max) by 0.46
AA GDP.pdf - Construction - Criterion Pass Rate: GPT-6.1 Sol (High) (84.94) beat GPT-6 Astra (High) by 0.43
Harvey LAB-AA - Trusts, Estates & Private Client: Claude Sonnet 5.5 (Xhigh, Default Fallback) (95.8) beat Kimi K3 (Max) by 0.38
Medmarks - LongHealth Task 2: Qwen 3.6 27B (Thinking) (91.33) beat Gemini 3 Pro (Preview) by 0.37
BRIDGE Medical Leaderboard - Few-Shot: Qwen3.8-Flash-Next (Non-reasoning) (56.03) beat Qwen 3.8 27B (Non-reasoning) by 0.36
FrontierCode: Claude Opus 5.5 (Medium) (65.3) beat Claude Fable 5 (xHigh) by 0.36
GAIA: Ops-Agentic-Search-2.0 (93.69) beat CustomGPT.ai Research Lab v44 by 0.33
EnterpriseOps-Gym-AA - Email: Step 5 Preview (70.19) beat Qwen 3.7 Max by 0.32
Wolfram LLM Benchmarking Project: Claude Opus 5 thinking max (73.6) beat Claude Fable 5 thinking max by 0.3
Kaggle Open Benchmarks - MATH-500: Claude Sonnet 5.5 (99.41) beat Gemini 3.8 Flash by 0.3
BRIDGE Medical Leaderboard: Qwen3.8-Flash-Next (Non-reasoning) (49.39) beat Gemma 4 31B (IT) by 0.22
BRIDGE Medical Leaderboard - Zero-Shot: Qwen3.8-Flash-Next (Thinking) (46.95) beat Gemma 4 31B (IT) by 0.21
SnakeBench: GPT-5.1 Codex Max (38.2) beat Gemini 3 Pro (Preview) by 0.2
CADGenBench: build123d-mcp 0.3.85 MCP server + Claude Opus 5.5 (max effort) r2 (0.8298) beat build123d-mcp-v0381-claude-opus-5-xhigh-full-r6 by 0.19
EnterpriseRAG Bench - Valid Extra Docs Resistance: CDL (Causal Dynamics Lab) (99.76) beat Mixedbread (+ Opus 5) by 0.15
Agent Arena - Tool Hallucination: GPT-6.1 Sol (Max) (-0.49) beat GPT-6 Astra (Max) by 0.13
LLM Stats (BenchCAD (with Python tool)): Claude Sonnet 5.5 (96.3) beat Claude Opus 5.5 by 0.1
GPQA Diamond (BenchmarkList): GPT-6 Astra (Max) (96.1) beat GPT-6 Astra by 0.1
Guesswork 2026-09: Claude Opus 5.5 (0.5126) beat Claude Sonnet 5 by 0.09
BRIDGE Medical Leaderboard - CoT: Qwen3.8-Flash-Next (Non-reasoning) (45.92) beat Gemma 4 31B (IT) by 0.04
Blueprint-Bench 2: Gemini 4 Argon (0.544) beat Claude Opus 5.5 by 0.03
ProphetArena: Claude Opus 5.5 (0.9812) beat GPT-6 Astra by 0.02
RSI-Exam: Claude Opus 5.5 (Max) (0.536) beat GPT-6 Astra (Max) by 0.02
KernelBench Mega: Claude Opus 5.5 (35.46) beat Claude Fable 5 by 0.02
FrontierSWE V2: GPT-6 Astra (Max) (65.51) beat GPT-6 Astra by 0.01

The Aggregate Digest — 2026-10-03

2026-10-03T07:59:31.155371+00:00

Daily

New Benchmarks (1)

HoneyBench - Overall Hack Rate (Overall hack rate (%) over nine reward-hacking tasks, 10 runs per task (lower is better)): leader GPT-6 Astra (30.0), 9 models
Share of agent rollouts in which the model commits to reward hacking across nine realistic tasks, such as consulting a chess engine during a chess-ability test or copying wrong numbers during an audit, where the hack is worse than failing honestly. Automated graders and reviewer models flag the hacks, with disagreements checked by hand. Ten runs per task. Lower is better. Source: goodhartlabs.com/honeybench.

Top-10 New Scores (350)

Claude Fable 5.1 on AA CritPt: 31.14 (#9)
Claude Fable 5.1 on AA GDP.pdf: 28.0 (#11)
Claude Fable 5.1 on AA GDP.pdf - Construction - Criterion Pass Rate: 78.67 (#10)
Claude Fable 5.1 on AA GDP.pdf - Criterion Pass Rate: 80.18 (#16)
Claude Fable 5.1 on AA GDP.pdf - Engineering - Criterion Pass Rate: 73.2 (#37)
Claude Fable 5.1 on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 82.18 (#31)
Claude Fable 5.1 on AA GDP.pdf - HR - Criterion Pass Rate: 82.31 (#19)
Claude Fable 5.1 on AA GDP.pdf - Healthcare - Criterion Pass Rate: 92.29 (#4)
Claude Fable 5.1 on AA GDP.pdf - Insurance - Criterion Pass Rate: 89.15 (#1)
Claude Fable 5.1 on AA GDP.pdf - Legal - Criterion Pass Rate: 85.2 (#5)
Claude Fable 5.1 on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 82.56 (#1)
Claude Fable 5.1 on AA GDP.pdf - Real Estate - Criterion Pass Rate: 74.32 (#16)
Claude Fable 5.1 on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 79.75 (#6)
Claude Fable 5.1 on AA GDPval: 1757.77 (#4)
Claude Fable 5.1 on AA GPQA Diamond: 93.74 (#11)
Claude Fable 5.1 on AA Humanity's Last Exam: 59.13 (#2)
Claude Fable 5.1 on AA Long Context Reasoning: 85.33 (#4)
Claude Fable 5.1 on AA Omniscience: 43.45 (#3)
Claude Fable 5.1 on AA SciCode: 63.08 (#3)
Claude Fable 5.1 on AA-AnalystAgent: 57.5 (#3)
Claude Fable 5.1 on AA-AnalystAgent - Pass@1: 65.75 (#4)
Claude Fable 5.1 on AA-AnalystAgent - Pass@K: 72.5 (#18)
Claude Fable 5.1 on AA-Briefcase: 61.52 (#4)
Claude Fable 5.1 on AA-Briefcase - Analytical Quality Elo: 1977.95 (#6)
Claude Fable 5.1 on AA-Briefcase - DOCX - Rubric Pass Rate: 57.4 (#10)
Claude Fable 5.1 on AA-Briefcase - PDF - Rubric Pass Rate: 68.1 (#3)
Claude Fable 5.1 on AA-Briefcase - PPTX - Rubric Pass Rate: 63.27 (#2)
Claude Fable 5.1 on AA-Briefcase - Presentation Elo: 1488.9 (#22)
Claude Fable 5.1 on AA-Briefcase - XLSX - Rubric Pass Rate: 60.45 (#5)
Claude Fable 5.1 on AA-Omniscience Accuracy: 67.23 (#1)
Claude Fable 5.1 on AA-Omniscience Hallucination Rate: 65.63 (#181)
Claude Fable 5.1 on AA-Omniscience Index - Business: 29.9 (#14)
Claude Fable 5.1 on AA-Omniscience Index - Health: 23.5 (#7)
Claude Fable 5.1 on AA-Omniscience Index - Humanities & Social Sciences: 41.0 (#6)
Claude Fable 5.1 on AA-Omniscience Index - Law: 50.1 (#2)
Claude Fable 5.1 on AA-Omniscience Index - Science, Engineering & Mathematics: 33.0 (#5)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE): 86.2 (#2)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - C: 92.0 (#4)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - Dart: 96.0 (#1)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - Go: 90.0 (#2)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - HTML: 84.0 (#12)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - Java: 69.0 (#10)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 87.27 (#3)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - Julia: 88.0 (#6)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 86.0 (#6)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - PHP: 90.0 (#7)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - Python: 91.0 (#2)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - R: 74.0 (#1)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - Rust: 84.0 (#9)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - Swift: 100.0 (#1)
Claude Fable 5.1 on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 92.22 (#10)
Claude Fable 5.1 on Artificial Analysis Intelligence Index: 53.35 (#5)
Claude Fable 5.1 on AutomationBench-AA: 59.38 (#34)
Claude Fable 5.1 on AutomationBench-AA - Finance - Raw Objective Completion: 79.67 (#63)
Claude Fable 5.1 on AutomationBench-AA - HR - Raw Objective Completion: 74.92 (#24)
Claude Fable 5.1 on AutomationBench-AA - Marketing - Raw Objective Completion: 90.12 (#5)
Claude Fable 5.1 on AutomationBench-AA - Operations - Raw Objective Completion: 96.39 (#7)
Claude Fable 5.1 on AutomationBench-AA - Sales - Raw Objective Completion: 85.19 (#23)
Claude Fable 5.1 on AutomationBench-AA - Strict: 32.12 (#20)
Claude Fable 5.1 on AutomationBench-AA - Support - Raw Objective Completion: 91.7 (#7)
Claude Fable 5.1 on BenchGecko Score: 88.2 (#9)
Claude Fable 5.1 on Chatbot Arena (Text - Korean): 1514.0 (#1)
Claude Fable 5.1 on Harvey LAB-AA: 93.27 (#7)
Claude Fable 5.1 on Harvey LAB-AA - All Pass: 16.67 (#3)
Claude Fable 5.1 on Harvey LAB-AA - Antitrust & Competition: 95.29 (#3)
Claude Fable 5.1 on Harvey LAB-AA - Arbitration & International Dispute Resolution: 95.7 (#5)
Claude Fable 5.1 on Harvey LAB-AA - Banking & Finance: 94.8 (#5)
Claude Fable 5.1 on Harvey LAB-AA - Bankruptcy & Restructuring: 92.73 (#7)
Claude Fable 5.1 on Harvey LAB-AA - Capital Markets: 97.25 (#1)
Claude Fable 5.1 on Harvey LAB-AA - Corporate Governance: 92.81 (#6)
Claude Fable 5.1 on Harvey LAB-AA - Corporate M&A: 95.83 (#4)
Claude Fable 5.1 on Harvey LAB-AA - Data Privacy & Cybersecurity: 98.94 (#2)
Claude Fable 5.1 on Harvey LAB-AA - Emerging Companies & Venture Capital: 95.9 (#2)
Claude Fable 5.1 on Harvey LAB-AA - Employment & Labor: 89.89 (#5)
Claude Fable 5.1 on Harvey LAB-AA - Energy & Natural Resources: 97.36 (#6)
Claude Fable 5.1 on Harvey LAB-AA - Environmental & ESG: 91.4 (#5)
Claude Fable 5.1 on Harvey LAB-AA - Funds & Asset Management: 90.67 (#7)
Claude Fable 5.1 on Harvey LAB-AA - Healthcare & Life Sciences: 93.4 (#9)
Claude Fable 5.1 on Harvey LAB-AA - Immigration: 93.45 (#6)
Claude Fable 5.1 on Harvey LAB-AA - Insurance: 94.87 (#3)
Claude Fable 5.1 on Harvey LAB-AA - Intellectual Property: 97.17 (#1)
Claude Fable 5.1 on Harvey LAB-AA - International Trade & Sanctions: 92.86 (#4)
Claude Fable 5.1 on Harvey LAB-AA - Litigation & Dispute Resolution: 95.13 (#4)
Claude Fable 5.1 on Harvey LAB-AA - Real Estate: 96.79 (#2)
Claude Fable 5.1 on Harvey LAB-AA - Structured Finance & Securitization: 91.11 (#2)
Claude Fable 5.1 on Harvey LAB-AA - Tax: 90.64 (#7)
Claude Fable 5.1 on Harvey LAB-AA - Trusts, Estates & Private Client: 95.04 (#4)
Claude Fable 5.1 on Harvey LAB-AA - White Collar Defense & Investigations: 94.95 (#3)
Claude Fable 5.1 on MLCR-AA: 71.11 (#2)
Claude Fable 5.1 on MLCR-AA - Accuracy: 94.84 (#4)
Claude Fable 5.1 on MLCR-AA - Completeness: 89.03 (#1)
Claude Fable 5.1 on MLCR-AA - Conciseness: 86.11 (#82)
Claude Fable 5.1 on Tau3 Banking: 47.22 (#9)
Claude Opus 5.5 on AA CritPt: 31.71 (#2)
Claude Opus 5.5 on AA GDP.pdf: 28.8 (#10)
Claude Opus 5.5 on AA GDP.pdf - Construction - Criterion Pass Rate: 75.95 (#13)
Claude Opus 5.5 on AA GDP.pdf - Criterion Pass Rate: 82.25 (#2)
Claude Opus 5.5 on AA GDP.pdf - Engineering - Criterion Pass Rate: 84.35 (#1)
Claude Opus 5.5 on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 91.25 (#1)
Claude Opus 5.5 on AA GDP.pdf - HR - Criterion Pass Rate: 90.16 (#1)
Claude Opus 5.5 on AA GDP.pdf - Healthcare - Criterion Pass Rate: 93.1 (#1)
Claude Opus 5.5 on AA GDP.pdf - Insurance - Criterion Pass Rate: 88.05 (#5)
Claude Opus 5.5 on AA GDP.pdf - Legal - Criterion Pass Rate: 84.79 (#11)
Claude Opus 5.5 on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 79.36 (#8)
Claude Opus 5.5 on AA GDP.pdf - Real Estate - Criterion Pass Rate: 72.66 (#27)
Claude Opus 5.5 on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 81.47 (#1)
Claude Opus 5.5 on AA GDPval: 1867.05 (#1)
Claude Opus 5.5 on AA Humanity's Last Exam: 61.35 (#1)
Claude Opus 5.5 on AA Long Context Reasoning: 84.67 (#5)
Claude Opus 5.5 on AA MMMU-Pro: 87.69 (#1)
Claude Opus 5.5 on AA Omniscience: 46.42 (#1)
Claude Opus 5.5 on AA SciCode: 66.9 (#1)
Claude Opus 5.5 on AA-AnalystAgent: 56.25 (#4)
Claude Opus 5.5 on AA-AnalystAgent - Pass@1: 64.25 (#5)
Claude Opus 5.5 on AA-AnalystAgent - Pass@K: 75.0 (#13)
Claude Opus 5.5 on AA-Briefcase: 61.21 (#5)
Claude Opus 5.5 on AA-Briefcase - Analytical Quality Elo: 2162.22 (#1)
Claude Opus 5.5 on AA-Briefcase - DOCX - Rubric Pass Rate: 60.36 (#3)
Claude Opus 5.5 on AA-Briefcase - PDF - Rubric Pass Rate: 64.66 (#4)
Claude Opus 5.5 on AA-Briefcase - PPTX - Rubric Pass Rate: 62.04 (#4)
Claude Opus 5.5 on AA-Briefcase - Presentation Elo: 1710.53 (#2)
Claude Opus 5.5 on AA-Briefcase - XLSX - Rubric Pass Rate: 61.14 (#3)
Claude Opus 5.5 on AA-Omniscience Accuracy: 66.22 (#2)
Claude Opus 5.5 on AA-Omniscience Hallucination Rate: 58.61 (#143)
Claude Opus 5.5 on AA-Omniscience Index - Business: 32.7 (#11)
Claude Opus 5.5 on AA-Omniscience Index - Health: 26.4 (#2)
Claude Opus 5.5 on AA-Omniscience Index - Humanities & Social Sciences: 44.7 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Law: 48.1 (#4)
Claude Opus 5.5 on AA-Omniscience Index - Science, Engineering & Mathematics: 37.8 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE): 88.8 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - C: 96.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Dart: 90.0 (#2)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Go: 88.0 (#7)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - HTML: 88.0 (#5)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Java: 80.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 88.18 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Julia: 84.0 (#10)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 96.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - PHP: 96.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Python: 91.5 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - R: 72.0 (#2)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Rust: 92.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Swift: 92.0 (#4)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 93.33 (#6)
Claude Opus 5.5 on Artificial Analysis Intelligence Index: 57.62 (#1)
Claude Opus 5.5 on AutomationBench-AA: 69.54 (#3)
Claude Opus 5.5 on AutomationBench-AA - Finance - Raw Objective Completion: 88.25 (#17)
Claude Opus 5.5 on AutomationBench-AA - HR - Raw Objective Completion: 77.27 (#4)
Claude Opus 5.5 on AutomationBench-AA - Marketing - Raw Objective Completion: 91.07 (#3)
Claude Opus 5.5 on AutomationBench-AA - Operations - Raw Objective Completion: 96.74 (#4)
Claude Opus 5.5 on AutomationBench-AA - Sales - Raw Objective Completion: 89.49 (#3)
Claude Opus 5.5 on AutomationBench-AA - Strict: 41.25 (#4)
Claude Opus 5.5 on AutomationBench-AA - Support - Raw Objective Completion: 91.11 (#9)
Claude Opus 5.5 on BenchGecko Score: 96.8 (#3)
Claude Opus 5.5 on Harvey LAB-AA: 91.24 (#17)
Claude Opus 5.5 on Harvey LAB-AA - All Pass: 8.33 (#16)
Claude Opus 5.5 on Harvey LAB-AA - Antitrust & Competition: 92.03 (#11)
Claude Opus 5.5 on Harvey LAB-AA - Arbitration & International Dispute Resolution: 94.62 (#9)
Claude Opus 5.5 on Harvey LAB-AA - Banking & Finance: 94.42 (#7)
Claude Opus 5.5 on Harvey LAB-AA - Bankruptcy & Restructuring: 92.04 (#9)
Claude Opus 5.5 on Harvey LAB-AA - Capital Markets: 94.9 (#11)
Claude Opus 5.5 on Harvey LAB-AA - Corporate Governance: 90.41 (#15)
Claude Opus 5.5 on Harvey LAB-AA - Corporate M&A: 94.87 (#12)
Claude Opus 5.5 on Harvey LAB-AA - Data Privacy & Cybersecurity: 98.59 (#3)
Claude Opus 5.5 on Harvey LAB-AA - Emerging Companies & Venture Capital: 94.88 (#5)
Claude Opus 5.5 on Harvey LAB-AA - Employment & Labor: 88.39 (#11)
Claude Opus 5.5 on Harvey LAB-AA - Energy & Natural Resources: 96.04 (#17)
Claude Opus 5.5 on Harvey LAB-AA - Environmental & ESG: 86.74 (#21)
Claude Opus 5.5 on Harvey LAB-AA - Funds & Asset Management: 89.8 (#12)
Claude Opus 5.5 on Harvey LAB-AA - Healthcare & Life Sciences: 93.07 (#10)
Claude Opus 5.5 on Harvey LAB-AA - Immigration: 94.91 (#1)
Claude Opus 5.5 on Harvey LAB-AA - Insurance: 94.14 (#5)
Claude Opus 5.5 on Harvey LAB-AA - Intellectual Property: 96.11 (#7)
Claude Opus 5.5 on Harvey LAB-AA - International Trade & Sanctions: 91.27 (#9)
Claude Opus 5.5 on Harvey LAB-AA - Litigation & Dispute Resolution: 94.38 (#8)
Claude Opus 5.5 on Harvey LAB-AA - Real Estate: 94.46 (#16)
Claude Opus 5.5 on Harvey LAB-AA - Structured Finance & Securitization: 88.89 (#11)
Claude Opus 5.5 on Harvey LAB-AA - Tax: 85.28 (#22)
Claude Opus 5.5 on Harvey LAB-AA - Trusts, Estates & Private Client: 93.51 (#12)
Claude Opus 5.5 on Harvey LAB-AA - White Collar Defense & Investigations: 88.89 (#20)
Claude Opus 5.5 on LLM Chess (Saplin): 1575.3 (#3)
Claude Opus 5.5 on MLCR-AA: 66.67 (#3)
Claude Opus 5.5 on MLCR-AA - Accuracy: 93.15 (#7)
Claude Opus 5.5 on MLCR-AA - Completeness: 88.36 (#3)
Claude Opus 5.5 on MLCR-AA - Conciseness: 81.11 (#88)
Claude Opus 5.5 on Vals AI Time Horizon Index: 91.33 (#1)
Claude Sonnet 5.5 on AA CritPt: 31.43 (#7)
Claude Sonnet 5.5 on AA GDP.pdf: 25.8 (#24)
Claude Sonnet 5.5 on AA GDP.pdf - Construction - Criterion Pass Rate: 69.11 (#46)
Claude Sonnet 5.5 on AA GDP.pdf - Criterion Pass Rate: 80.6 (#13)
Claude Sonnet 5.5 on AA GDP.pdf - Engineering - Criterion Pass Rate: 82.17 (#6)
Claude Sonnet 5.5 on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 90.78 (#3)
Claude Sonnet 5.5 on AA GDP.pdf - HR - Criterion Pass Rate: 87.47 (#5)
Claude Sonnet 5.5 on AA GDP.pdf - Healthcare - Criterion Pass Rate: 90.36 (#14)
Claude Sonnet 5.5 on AA GDP.pdf - Insurance - Criterion Pass Rate: 86.98 (#10)
Claude Sonnet 5.5 on AA GDP.pdf - Legal - Criterion Pass Rate: 83.42 (#25)
Claude Sonnet 5.5 on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 78.73 (#13)
Claude Sonnet 5.5 on AA GDP.pdf - Real Estate - Criterion Pass Rate: 71.97 (#29)
Claude Sonnet 5.5 on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 79.54 (#7)
Claude Sonnet 5.5 on AA GDPval: 1840.08 (#2)
Claude Sonnet 5.5 on AA Humanity's Last Exam: 54.96 (#9)
Claude Sonnet 5.5 on AA Long Context Reasoning: 82.67 (#32)
Claude Sonnet 5.5 on AA Omniscience: 32.3 (#26)
Claude Sonnet 5.5 on AA SciCode: 61.0 (#5)
Claude Sonnet 5.5 on AA-AnalystAgent: 57.5 (#2)
Claude Sonnet 5.5 on AA-AnalystAgent - Pass@1: 69.5 (#2)
Claude Sonnet 5.5 on AA-AnalystAgent - Pass@K: 78.75 (#3)
Claude Sonnet 5.5 on AA-Briefcase: 65.15 (#1)
Claude Sonnet 5.5 on AA-Briefcase - Analytical Quality Elo: 2133.71 (#2)
Claude Sonnet 5.5 on AA-Briefcase - DOCX - Rubric Pass Rate: 60.95 (#2)
Claude Sonnet 5.5 on AA-Briefcase - PDF - Rubric Pass Rate: 69.83 (#2)
Claude Sonnet 5.5 on AA-Briefcase - PPTX - Rubric Pass Rate: 62.86 (#3)
Claude Sonnet 5.5 on AA-Briefcase - Presentation Elo: 1716.42 (#1)
Claude Sonnet 5.5 on AA-Briefcase - XLSX - Rubric Pass Rate: 66.36 (#1)
Claude Sonnet 5.5 on AA-Omniscience Accuracy: 53.95 (#42)
Claude Sonnet 5.5 on AA-Omniscience Hallucination Rate: 47.01 (#95)
Claude Sonnet 5.5 on AA-Omniscience Index - Business: 19.3 (#47)
Claude Sonnet 5.5 on AA-Omniscience Index - Health: 9.7 (#42)
Claude Sonnet 5.5 on AA-Omniscience Index - Humanities & Social Sciences: 30.8 (#29)
Claude Sonnet 5.5 on AA-Omniscience Index - Law: 26.6 (#48)
Claude Sonnet 5.5 on AA-Omniscience Index - Science, Engineering & Mathematics: 29.1 (#11)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE): 78.3 (#20)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - C: 87.0 (#18)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Dart: 82.0 (#9)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Go: 82.0 (#21)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - HTML: 74.0 (#29)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Java: 58.0 (#20)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 78.18 (#22)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Julia: 92.0 (#1)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 72.0 (#30)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - PHP: 76.0 (#27)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Python: 85.5 (#11)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - R: 62.0 (#10)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Rust: 86.0 (#6)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Swift: 80.0 (#20)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 85.56 (#26)
Claude Sonnet 5.5 on Agent Arena: 12.52 (#3)
Claude Sonnet 5.5 on Agent Arena - Bash Recovery: 15.05 (#1)
Claude Sonnet 5.5 on Agent Arena - Confirmed Success: 13.91 (#5)
Claude Sonnet 5.5 on Agent Arena - Praise vs Complaint: 26.27 (#5)
Claude Sonnet 5.5 on Agent Arena - Steerability: 6.9 (#8)
Claude Sonnet 5.5 on Agent Arena - Tool Hallucination: -0.47 (#7)
Claude Sonnet 5.5 on Artificial Analysis Intelligence Index: 56.0 (#2)
Claude Sonnet 5.5 on AutomationBench-AA: 71.77 (#2)
Claude Sonnet 5.5 on AutomationBench-AA - Finance - Raw Objective Completion: 92.47 (#3)
Claude Sonnet 5.5 on AutomationBench-AA - HR - Raw Objective Completion: 79.22 (#2)
Claude Sonnet 5.5 on AutomationBench-AA - Marketing - Raw Objective Completion: 88.65 (#10)
Claude Sonnet 5.5 on AutomationBench-AA - Operations - Raw Objective Completion: 97.47 (#1)
Claude Sonnet 5.5 on AutomationBench-AA - Sales - Raw Objective Completion: 89.45 (#4)
Claude Sonnet 5.5 on AutomationBench-AA - Strict: 45.51 (#2)
Claude Sonnet 5.5 on AutomationBench-AA - Support - Raw Objective Completion: 92.02 (#3)
Claude Sonnet 5.5 on BenchGecko Score: 89.2 (#7)
Claude Sonnet 5.5 on Chatbot Arena (Text - Business, Management & Financial Ops): 1464.0 (#53)
Claude Sonnet 5.5 on Chatbot Arena (Text - Chinese): 1533.0 (#16)
Claude Sonnet 5.5 on Chatbot Arena (Text - Coding): 1536.0 (#13)
Claude Sonnet 5.5 on Chatbot Arena (Text - Creative Writing): 1447.0 (#42)
Claude Sonnet 5.5 on Chatbot Arena (Text - English): 1492.0 (#17)
Claude Sonnet 5.5 on Chatbot Arena (Text - Entertainment, Sports & Media): 1441.0 (#51)
Claude Sonnet 5.5 on Chatbot Arena (Text - Expert): 1556.0 (#1)
Claude Sonnet 5.5 on Chatbot Arena (Text - Hard Prompts (English)): 1518.0 (#10)
Claude Sonnet 5.5 on Chatbot Arena (Text - Hard Prompts): 1504.0 (#24)
Claude Sonnet 5.5 on Chatbot Arena (Text - Instruction Following): 1484.0 (#18)
Claude Sonnet 5.5 on Chatbot Arena (Text - Legal & Government): 1483.0 (#35)
Claude Sonnet 5.5 on Chatbot Arena (Text - Life, Physical & Social Science): 1488.0 (#45)
Claude Sonnet 5.5 on Chatbot Arena (Text - Longer Query): 1493.0 (#24)
Claude Sonnet 5.5 on Chatbot Arena (Text - Medicine & Healthcare): 1483.0 (#52)
Claude Sonnet 5.5 on Chatbot Arena (Text - Multi-Turn): 1473.0 (#51)
Claude Sonnet 5.5 on Chatbot Arena (Text - Non-English): 1450.0 (#55)
Claude Sonnet 5.5 on Chatbot Arena (Text - Russian): 1451.0 (#73)
Claude Sonnet 5.5 on Chatbot Arena (Text - Software & IT Services): 1522.0 (#20)
Claude Sonnet 5.5 on Chatbot Arena (Text - Writing, Literature & Language): 1461.0 (#34)
Claude Sonnet 5.5 on Chatbot Arena (Text): 1471.0 (#45)
Claude Sonnet 5.5 on Chatbot Arena (Vision - Diagram): 1306.0 (#31)
Claude Sonnet 5.5 on Chatbot Arena (Vision - English): 1267.0 (#39)
Claude Sonnet 5.5 on Chatbot Arena (Vision - Homework): 1317.0 (#21)
Claude Sonnet 5.5 on Chatbot Arena (Vision - OCR): 1277.0 (#41)
Claude Sonnet 5.5 on Chatbot Arena (Vision): 1268.0 (#35)
Claude Sonnet 5.5 on Harvey LAB-AA: 93.07 (#8)
Claude Sonnet 5.5 on Harvey LAB-AA - All Pass: 11.67 (#10)
Claude Sonnet 5.5 on Harvey LAB-AA - Antitrust & Competition: 92.03 (#12)
Claude Sonnet 5.5 on Harvey LAB-AA - Arbitration & International Dispute Resolution: 95.34 (#7)
Claude Sonnet 5.5 on Harvey LAB-AA - Banking & Finance: 94.8 (#4)
Claude Sonnet 5.5 on Harvey LAB-AA - Bankruptcy & Restructuring: 92.73 (#6)
Claude Sonnet 5.5 on Harvey LAB-AA - Capital Markets: 96.86 (#3)
Claude Sonnet 5.5 on Harvey LAB-AA - Corporate Governance: 91.1 (#13)
Claude Sonnet 5.5 on Harvey LAB-AA - Corporate M&A: 95.83 (#3)
Claude Sonnet 5.5 on Harvey LAB-AA - Data Privacy & Cybersecurity: 98.94 (#1)
Claude Sonnet 5.5 on Harvey LAB-AA - Emerging Companies & Venture Capital: 94.88 (#4)
Claude Sonnet 5.5 on Harvey LAB-AA - Employment & Labor: 89.89 (#4)
Claude Sonnet 5.5 on Harvey LAB-AA - Energy & Natural Resources: 96.7 (#10)
Claude Sonnet 5.5 on Harvey LAB-AA - Environmental & ESG: 90.68 (#8)
Claude Sonnet 5.5 on Harvey LAB-AA - Funds & Asset Management: 90.38 (#8)
Claude Sonnet 5.5 on Harvey LAB-AA - Healthcare & Life Sciences: 94.06 (#6)
Claude Sonnet 5.5 on Harvey LAB-AA - Immigration: 94.18 (#2)
Claude Sonnet 5.5 on Harvey LAB-AA - Insurance: 93.77 (#10)
Claude Sonnet 5.5 on Harvey LAB-AA - Intellectual Property: 95.41 (#13)
Claude Sonnet 5.5 on Harvey LAB-AA - International Trade & Sanctions: 91.67 (#5)
Claude Sonnet 5.5 on Harvey LAB-AA - Litigation & Dispute Resolution: 94.76 (#6)
Claude Sonnet 5.5 on Harvey LAB-AA - Real Estate: 95.04 (#10)
Claude Sonnet 5.5 on Harvey LAB-AA - Structured Finance & Securitization: 90.16 (#6)
Claude Sonnet 5.5 on Harvey LAB-AA - Tax: 89.97 (#8)
Claude Sonnet 5.5 on Harvey LAB-AA - Trusts, Estates & Private Client: 95.8 (#1)
Claude Sonnet 5.5 on Harvey LAB-AA - White Collar Defense & Investigations: 93.6 (#7)
Claude Sonnet 5.5 on MLCR-AA: 75.0 (#1)
Claude Sonnet 5.5 on MLCR-AA - Accuracy: 90.75 (#9)
Claude Sonnet 5.5 on MLCR-AA - Completeness: 88.44 (#2)
Claude Sonnet 5.5 on MLCR-AA - Conciseness: 96.11 (#51)
Claude Sonnet 5.5 on Vals AI ProgramBench: 76.3 (#10)
GPT-6 Astra on BenchGecko Score: 96.3 (#4)
GPT-6 Astra on Chatbot Arena (Vision - Chinese): 1358.0 (#3)
GPT-6.1 Sol on AA-AnalystAgent: 50.0 (#7)
GPT-6.1 Sol on AA-AnalystAgent - Pass@1: 59.75 (#14)
GPT-6.1 Sol on AA-AnalystAgent - Pass@K: 72.5 (#19)
GPT-6.1 Sol on Agent Arena: 11.23 (#5)
GPT-6.1 Sol on Agent Arena - Bash Recovery: 3.47 (#24)
GPT-6.1 Sol on Agent Arena - Confirmed Success: 15.47 (#2)
GPT-6.1 Sol on Agent Arena - Praise vs Complaint: 25.53 (#6)
GPT-6.1 Sol on Agent Arena - Steerability: 11.18 (#3)
GPT-6.1 Sol on Agent Arena - Tool Hallucination: -0.49 (#1)
GPT-6.1 Sol on Agent Security League - Functional Correctness: 77.7 (#9)
GPT-6.1 Sol on Agent Security League - Security Correctness: 34.1 (#3)
GPT-6.1 Sol on Chatbot Arena (Text - Business, Management & Financial Ops): 1475.0 (#32)
GPT-6.1 Sol on Chatbot Arena (Text - Chinese): 1500.0 (#59)
GPT-6.1 Sol on Chatbot Arena (Text - Coding): 1542.0 (#8)
GPT-6.1 Sol on Chatbot Arena (Text - Creative Writing): 1460.0 (#27)
GPT-6.1 Sol on Chatbot Arena (Text - English): 1496.0 (#13)
GPT-6.1 Sol on Chatbot Arena (Text - Entertainment, Sports & Media): 1462.0 (#16)
GPT-6.1 Sol on Chatbot Arena (Text - Expert): 1543.0 (#4)
GPT-6.1 Sol on Chatbot Arena (Text - Hard Prompts (English)): 1519.0 (#8)
GPT-6.1 Sol on Chatbot Arena (Text - Hard Prompts): 1509.0 (#18)
GPT-6.1 Sol on Chatbot Arena (Text - Instruction Following): 1490.0 (#13)
GPT-6.1 Sol on Chatbot Arena (Text - Legal & Government): 1511.0 (#4)
GPT-6.1 Sol on Chatbot Arena (Text - Life, Physical & Social Science): 1503.0 (#21)
GPT-6.1 Sol on Chatbot Arena (Text - Longer Query): 1496.0 (#19)
GPT-6.1 Sol on Chatbot Arena (Text - Medicine & Healthcare): 1491.0 (#31)
GPT-6.1 Sol on Chatbot Arena (Text - Multi-Turn): 1493.0 (#20)
GPT-6.1 Sol on Chatbot Arena (Text - Non-English): 1469.0 (#26)
GPT-6.1 Sol on Chatbot Arena (Text - Russian): 1481.0 (#31)
GPT-6.1 Sol on Chatbot Arena (Text - Software & IT Services): 1536.0 (#5)
GPT-6.1 Sol on Chatbot Arena (Text - Writing, Literature & Language): 1472.0 (#20)
GPT-6.1 Sol on Chatbot Arena (Text): 1483.0 (#21)
GPT-6.1 Sol on Chatbot Arena (Vision - Diagram): 1308.0 (#28)
GPT-6.1 Sol on Chatbot Arena (Vision - English): 1294.0 (#7)
GPT-6.1 Sol on Chatbot Arena (Vision - Homework): 1361.0 (#1)
GPT-6.1 Sol on Chatbot Arena (Vision - OCR): 1296.0 (#22)
GPT-6.1 Sol on Chatbot Arena (Vision): 1291.0 (#10)
GPT-6.1 Sol on Guesswork 2026-09: 0.5706 (#2)
GPT-6.1 Sol on LLM Chess (Saplin): 1454.4 (#11)
GPT-6.1 Sol on Vals AI ProgramBench: 85.25 (#3)
GPT-6.1 Sol on Vals AI Terminal-Bench Science: 52.86 (#2)

New #1 Leaders (14)

AA GDPval: Claude Opus 5.5 (Max, Default Fallback) (1867.05) beat Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) by 20.88
Chatbot Arena (Text - Korean): Claude Fable 5.1 (Max) (1514.0) beat Muse Spark 1.3 (Max) by 11.0
Chatbot Arena (Vision - Homework): GPT-6.1 Sol (Max) (1361.0) beat Claude Fable 5 (High) by 9.0
Chatbot Arena (Text - Expert): Claude Sonnet 5.5 (xHigh) (1556.0) beat Claude Fable 5 (High) by 7.0
AA-Briefcase - Presentation Elo: Claude Sonnet 5.5 (Max, Default Fallback) (1716.42) beat Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) by 6.51
Coarena - Task Completion: MiMo-V2.6-Pro (100.0) beat GPT-6 Luna by 5.0
Design Arena (UI Components): Claude Sonnet 5.5 (1404.0) beat GPT-6 Astra by 3.0
Agent Arena - Bash Recovery: Claude Sonnet 5.5 (Max) (15.05) beat Claude Opus 5 (Max) by 2.0
AA-Briefcase - XLSX - Rubric Pass Rate: Claude Sonnet 5.5 (Max, Default Fallback) (66.36) beat Claude Sonnet 5.5 (Adaptive Reasoning, Xhigh Effort, Default Fallback) by 1.59
Vals AI Time Horizon Index: Claude Opus 5.5 (Max) (91.33) beat GPT-6 Astra (Max) by 0.83
AA-Briefcase: Claude Sonnet 5.5 (Max, Default Fallback) (65.15) beat Gemini 4 Argon (High) by 0.52
AutomationBench-AA - Operations - Raw Objective Completion: Claude Sonnet 5.5 (Max, Default Fallback) (97.47) beat Claude Sonnet 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) by 0.42
Agent Arena - Tool Hallucination: GPT-6.1 Sol (Max) (-0.49) beat Claude Fable 5.1 (Max) by 0.17
Guesswork 2026-09: Claude Opus 5.5 (0.5126) beat Claude Sonnet 5 by 0.09

The Aggregate Digest — 2026-10-02

2026-10-02T21:34:32.169727+00:00

Daily

New Benchmarks (1)

Uruqi (Overall accuracy (%): mean of the three stage scores (self-motion, object mapping, state operations), subclasses weighted equally within each stage, over 52k questions on 2.7k simulated camera-trajectory episodes; higher is better): leader GPT-6 Astra (50.08), 12 models
Spatial cognition from continuous visual experience: over 52k questions on 2.7k simulated camera-trajectory episodes, a vision-language model must track its own motion, keep a persistent map of the objects it has seen and apply spatial state operations (updates, relations, events) as the view changes. Scored by overall accuracy, the mean of the three stage scores; higher is better (Uruqi, 2026).

Top-10 New Scores (93)

Claude Opus 5.5 on RSI-Exam: 0.536 (#1)
Claude Sonnet 5.5 on AIMultiple - Agentic RAG Routing: 85.33 (#2)
Claude Sonnet 5.5 on AIMultiple - FinanceReasoning (Hard): 89.08 (#13)
Claude Sonnet 5.5 on AIMultiple - Text-to-SQL (Adjudicated): 83.2 (#2)
Claude Sonnet 5.5 on AIMultiple - Text-to-SQL (Strict Execution Match): 59.4 (#4)
Claude Sonnet 5.5 on Chatbot Arena (Code): 1786.0 (#3)
Claude Sonnet 5.5 on Kaggle Adversarial Customer Service (Gert Labs): 77.5 (#3)
Claude Sonnet 5.5 on Kaggle AssetOpsBench: 67.42 (#5)
Claude Sonnet 5.5 on Kaggle ExtractBench: 89.01 (#5)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Arabic: 93.0 (#13)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Bengali: 91.75 (#14)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Burmese: 90.25 (#18)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Chinese: 93.75 (#13)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Culturally Agnostic: 92.75 (#15)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Culturally Sensitive: 92.53 (#15)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - English: 94.75 (#12)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - French: 93.75 (#14)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - German: 95.0 (#10)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Hindi: 92.0 (#16)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Indonesian: 93.75 (#12)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Italian: 93.75 (#16)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Japanese: 92.75 (#17)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Korean: 92.75 (#15)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Portuguese: 92.5 (#17)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Spanish: 95.25 (#8)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Swahili: 92.5 (#13)
Claude Sonnet 5.5 on Kaggle Global-MMLU-Lite - Yoruba: 84.75 (#26)
Claude Sonnet 5.5 on Kaggle Open Benchmarks - AIME 2025: 96.67 (#11)
Claude Sonnet 5.5 on Kaggle Open Benchmarks - SciCode Subproblems: 43.3 (#3)
Claude Sonnet 5.5 on Kaggle SimpleQA: 46.05 (#16)
Claude Sonnet 5.5 on Kaggle SimpleQA Verified: 44.75 (#14)
Claude Sonnet 5.5 on PLCC - Art & Entertainment: 72.0 (#72)
Claude Sonnet 5.5 on PLCC - Culture & Tradition: 82.0 (#72)
Claude Sonnet 5.5 on PLCC - Geography: 92.0 (#55)
Claude Sonnet 5.5 on PLCC - Grammar: 91.0 (#24)
Claude Sonnet 5.5 on PLCC - History: 95.0 (#19)
Claude Sonnet 5.5 on PLCC - Overall: 85.0 (#49)
Claude Sonnet 5.5 on PLCC - Vocabulary: 78.0 (#61)
Claude Sonnet 5.5 on SimpleBench: 75.9 (#15)
Claude Sonnet 5.5 on WebDev Arena (Brand & Marketing): 1767.0 (#2)
Claude Sonnet 5.5 on WebDev Arena (Consumer Product): 1671.0 (#4)
Claude Sonnet 5.5 on WebDev Arena (Content Creation Tools): 1730.0 (#4)
Claude Sonnet 5.5 on WebDev Arena (Data & Analytics): 1621.0 (#6)
Claude Sonnet 5.5 on WebDev Arena (Frontend): 1822.0 (#2)
Claude Sonnet 5.5 on WebDev Arena (Fullstack): 1646.0 (#10)
Claude Sonnet 5.5 on WebDev Arena (Gaming): 1892.0 (#2)
Claude Sonnet 5.5 on WebDev Arena (HTML): 1735.0 (#3)
Claude Sonnet 5.5 on WebDev Arena (React): 1836.0 (#2)
Claude Sonnet 5.5 on WebDev Arena (Reference-Based Design): 1832.0 (#2)
Claude Sonnet 5.5 on WebDev Arena (Simulations): 1862.0 (#3)
GPT-6 Astra on Agents' Last Exam: 34.2 (#2)
GPT-6 Astra on AutomationBench: 41.4 (#5)
GPT-6 Astra on Benchmarks.bio - BioSecBench-Function: 48.6 (#2)
GPT-6 Astra on Benchmarks.bio - BioSecBench-Refusal: 44.6 (#4)
GPT-6 Astra on Benchmarks.bio - BioSecBench-Surveillance: 48.5 (#2)
GPT-6 Astra on Benchmarks.bio - MetagenomicsBench: 49.7 (#4)
GPT-6 Astra on Benchmarks.bio - SpatialBench: 70.14 (#7)
GPT-6 Astra on Benchmarks.bio - TxBench-AD: 56.9 (#1)
GPT-6 Astra on Benchmarks.bio - TxBench-OD: 55.46 (#1)
GPT-6 Astra on Benchmarks.bio - VariantBench: 48.6 (#2)
GPT-6 Astra on Benchmarks.bio - scBench: 64.79 (#1)
GPT-6 Astra on DeepSWE: 74.1 (#1)
GPT-6 Astra on DeepSWE-mini (16 Tasks): 81.25 (#1)
GPT-6 Astra on MedXpertQA MM (Natomy): 87.4 (#1)
GPT-6 Astra on MedXpertQA MM (Natomy) - Basic Science: 89.24 (#1)
GPT-6 Astra on MedXpertQA MM (Natomy) - Diagnosis: 86.99 (#1)
GPT-6 Astra on MedXpertQA MM (Natomy) - Treatment: 87.05 (#1)
GPT-6 Astra on ParseBench: 70.67 (#23)
GPT-6 Astra on ParseBench - Charts: 35.08 (#55)
GPT-6 Astra on ParseBench - Content Faithfulness: 89.6 (#18)
GPT-6 Astra on ParseBench - Semantic Formatting: 75.52 (#8)
GPT-6 Astra on ParseBench - Tables: 93.17 (#3)
GPT-6 Astra on ParseBench - Visual Grounding: 59.99 (#39)
GPT-6 Astra on Terminal-Bench 4.0: 58.2 (#1)
GPT-6 Astra on WolfBench: 86.0 (#2)
GPT-6 Astra on You're Absolutely Right!: 3.0 (#11)
GPT-6 Astra Pro on OpenRouter GPQA Diamond: 95.5 (#2)
GPT-6 Astra Pro on OpenRouter Tau2-Bench Airline: 72.7 (#52)
GPT-6.1 Sol on AIMultiple - Agentic RAG Routing: 80.98 (#21)
GPT-6.1 Sol on AIMultiple - FinanceReasoning (Hard): 89.92 (#9)
GPT-6.1 Sol on AIMultiple - Text-to-SQL (Adjudicated): 81.8 (#7)
GPT-6.1 Sol on AIMultiple - Text-to-SQL (Strict Execution Match): 52.4 (#11)
GPT-6.1 Sol on Chess Bench LLM: 1490.0 (#17)
GPT-6.1 Sol on ClockBench: 74.4 (#5)
GPT-6.1 Sol on PLCC - Art & Entertainment: 93.0 (#12)
GPT-6.1 Sol on PLCC - Culture & Tradition: 98.0 (#10)
GPT-6.1 Sol on PLCC - Geography: 98.0 (#15)
GPT-6.1 Sol on PLCC - Grammar: 98.0 (#2)
GPT-6.1 Sol on PLCC - History: 99.0 (#2)
GPT-6.1 Sol on PLCC - Overall: 96.83 (#3)
GPT-6.1 Sol on PLCC - Vocabulary: 95.0 (#5)
GPT-6.1 Sol on SimpleBench: 82.9 (#5)
GPT-6.1 Sol on WebDev Arena (Content Creation Tools): 1739.0 (#3)

New #1 Leaders (11)

ClockBench: Top Human Performance (95.0) beat GPT-5.6 Sol (Max) by 28.3
Medmarks - M-ARC: Muse Glimmer 30B (High) (86.67) beat Grok 4 by 6.67
Kaggle FACTS Grounding: GPT-6 Sol (85.22) beat GPT-6 Astra by 3.24
Medmarks - SuperGPQA Medicine Hard: Gemma 4 31B (Thinking) (62.52) beat Gemini 3 Pro (Preview) by 1.84
OpenRouter GPQA Diamond: Gemini 3.8 Flash (95.6) beat Fugu Ultra by 1.0
Medmarks - LongHealth Task 1: Qwen3.8 27B Thinking (xHigh) (92.08) beat Gemini 3 Pro (Preview) by 0.75
Medmarks - LongHealth Task 2: Qwen 3.6 27B (Thinking) (91.33) beat Gemini 3 Pro (Preview) by 0.37
Wolfram LLM Benchmarking Project: Claude Opus 5 thinking max (73.6) beat Claude Fable 5 thinking max by 0.3
Kaggle Open Benchmarks - MATH-500: Claude Sonnet 5.5 (99.41) beat Gemini 3.8 Flash by 0.3
RSI-Exam: Claude Opus 5.5 (Max) (0.536) beat GPT-6 Astra (Max) by 0.02
ProphetArena: Claude Opus 5.5 (0.9701) beat GPT-6 Astra by 0.01

The Aggregate Digest — 2026-10-01

2026-10-01T08:36:33.558041+00:00

Frontier

Best available model: Gemini 4 Argon enters at #3 (Aggregate Rating 2031) on 69 benchmarks

Daily

New Benchmarks (2)

Benchmarks.bio - MetagenomicsBench (Pass Rate (%)): leader Claude Sonnet 5.5 (60.0), 21 models
MetagenomicsBench (benchmarks.bio) — agentic microbial-community analysis over real metagenomic data (shotgun, 16S/amplicon, long-read, metatranscriptomic and spatial assays): community structure, host associations, function and metabolism, longitudinal dynamics and interventions across 100 evaluations with three runs each, scored as pass rate with deterministic grading.
LabBench (Gamow Labs) (Mean share of binary rubric criteria passed (%) over 20 held-out wet-lab decision tasks (406 criteria), one run per task in each vendor's harness, judged by Codex with GPT-5.6 Sol): leader GPT-6 (40.8), 5 models
LabBench (Gamow Labs, 2026) — 20 held-out wet-lab decision tasks built from real drug-discovery and genomics lab records. The agent receives the records that existed at a decision point, with the lab's interpretation withheld, and must commit to the next step; it is graded on 20-22 binary criteria per task tied to what the lab actually decided. Mean share of criteria passed. Unrelated to FutureHouse's LAB-Bench.

New Models (1)

Gemini 4 Argon — Aggregate Rating 2031, #3
- Agent Arena - Steerability: 15.88 (#1/46)
- AA-Briefcase: 64.63 (#1/211)
- AutomationBench-AA: 77.51 (#1/210)
- AutomationBench-AA - Strict: 53.12 (#1/210)
- AA-Omniscience Index - Law: 53.0 (#1/563)
- AA-Briefcase - DOCX - Rubric Pass Rate: 62.13 (#1/211)
- AA-Briefcase - PDF - Rubric Pass Rate: 72.41 (#1/211)
- AA-Briefcase - PPTX - Rubric Pass Rate: 67.36 (#1/211)
- AutomationBench-AA - Finance - Raw Objective Completion: 94.62 (#1/210)
- AutomationBench-AA - HR - Raw Objective Completion: 81.99 (#1/210)
- ...and 59 more

Top-10 New Scores (313)

Claude Opus 5.5 on Chatbot Arena (Text - Math): 1510.0 (#9)
Claude Opus 5.5 on Chatbot Arena (Text - Mathematical): 1515.0 (#7)
Claude Opus 5.5 on Chatbot Arena (Text - Medicine & Healthcare): 1538.0 (#2)
Claude Sonnet 5.5 on ChessBench: 32.1 (#24)
Claude Sonnet 5.5 on ChessBench - Accuracy: 73.5 (#10)
Claude Sonnet 5.5 on ChessBench - Coherence: 50.7 (#34)
Claude Sonnet 5.5 on ChessBench - Elo: 802.0 (#28)
Claude Sonnet 5.5 on Epoch AI - ECI: 165.2 (#3)
Claude Sonnet 5.5 on Icelandic LLM - Case Government: 95.31 (#16)
Claude Sonnet 5.5 on Icelandic LLM - Famous Phrases: 17.97 (#21)
Claude Sonnet 5.5 on Icelandic LLM - Flora & Fauna: 36.28 (#19)
Claude Sonnet 5.5 on Icelandic LLM - GEC: 67.43 (#16)
Claude Sonnet 5.5 on Icelandic LLM - Generative Quality: 74.93 (#14)
Claude Sonnet 5.5 on Icelandic LLM - Idiom Completion: 59.76 (#19)
Claude Sonnet 5.5 on Icelandic LLM - Idiom Meanings: 79.31 (#22)
Claude Sonnet 5.5 on Icelandic LLM - Song Lyrics: 16.28 (#14)
Claude Sonnet 5.5 on Icelandic LLM - Textile Terms: 65.4 (#16)
Claude Sonnet 5.5 on Icelandic LLM - WikiQA: 59.79 (#18)
Claude Sonnet 5.5 on Icelandic LLM Leaderboard - Average: 62.12 (#18)
Claude Sonnet 5.5 on SpeechMap Compliance: 81.6 (#83)
Claude Sonnet 5.5 on WebDev Arena (Content Creation Tools): 1644.0 (#7)
GPT-6 on AA CritPt: 31.71 (#4)
GPT-6 on AA GDP.pdf: 32.2 (#1)
GPT-6 on AA GDP.pdf - Construction - Criterion Pass Rate: 84.51 (#2)
GPT-6 on AA GDP.pdf - Criterion Pass Rate: 82.05 (#3)
GPT-6 on AA GDP.pdf - Engineering - Criterion Pass Rate: 82.86 (#3)
GPT-6 on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 89.19 (#7)
GPT-6 on AA GDP.pdf - HR - Criterion Pass Rate: 85.49 (#10)
GPT-6 on AA GDP.pdf - Healthcare - Criterion Pass Rate: 92.51 (#3)
GPT-6 on AA GDP.pdf - Insurance - Criterion Pass Rate: 79.9 (#28)
GPT-6 on AA GDP.pdf - Legal - Criterion Pass Rate: 85.11 (#6)
GPT-6 on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 73.73 (#37)
GPT-6 on AA GDP.pdf - Real Estate - Criterion Pass Rate: 79.15 (#7)
GPT-6 on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 79.89 (#5)
GPT-6 on AA GDPval: 1541.89 (#35)
GPT-6 on AA GPQA Diamond: 96.26 (#1)
GPT-6 on AA Humanity's Last Exam: 54.68 (#12)
GPT-6 on AA Long Context Reasoning: 80.67 (#58)
GPT-6 on AA MMMU-Pro: 86.88 (#2)
GPT-6 on AA Omniscience: 43.73 (#2)
GPT-6 on AA SciCode: 56.48 (#33)
GPT-6 on AA-AnalystAgent: 51.25 (#4)
GPT-6 on AA-AnalystAgent - Pass@1: 60.0 (#11)
GPT-6 on AA-AnalystAgent - Pass@K: 67.5 (#25)
GPT-6 on AA-Briefcase: 52.32 (#28)
GPT-6 on AA-Briefcase - Analytical Quality Elo: 1776.98 (#22)
GPT-6 on AA-Briefcase - DOCX - Rubric Pass Rate: 53.85 (#21)
GPT-6 on AA-Briefcase - PDF - Rubric Pass Rate: 50.86 (#36)
GPT-6 on AA-Briefcase - PPTX - Rubric Pass Rate: 48.98 (#30)
GPT-6 on AA-Briefcase - Presentation Elo: 1535.47 (#10)
GPT-6 on AA-Briefcase - XLSX - Rubric Pass Rate: 54.09 (#21)
GPT-6 on AA-Omniscience Accuracy: 62.6 (#11)
GPT-6 on AA-Omniscience Hallucination Rate: 44.77 (#86)
GPT-6 on AA-Omniscience Index - Business: 34.2 (#2)
GPT-6 on AA-Omniscience Index - Health: 27.0 (#1)
GPT-6 on AA-Omniscience Index - Humanities & Social Sciences: 43.1 (#2)
GPT-6 on AA-Omniscience Index - Law: 48.5 (#3)
GPT-6 on AA-Omniscience Index - Science, Engineering & Mathematics: 29.3 (#10)
GPT-6 on AA-Omniscience Index - Software Engineering (SWE): 84.2 (#4)
GPT-6 on AA-Omniscience Index - Software Engineering (SWE) - C: 91.0 (#7)
GPT-6 on AA-Omniscience Index - Software Engineering (SWE) - Dart: 82.0 (#10)
GPT-6 on AA-Omniscience Index - Software Engineering (SWE) - Go: 92.0 (#1)
GPT-6 on AA-Omniscience Index - Software Engineering (SWE) - HTML: 92.0 (#1)
GPT-6 on AA-Omniscience Index - Software Engineering (SWE) - Java: 75.0 (#3)
GPT-6 on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 88.18 (#2)
GPT-6 on AA-Omniscience Index - Software Engineering (SWE) - Julia: 92.0 (#2)
GPT-6 on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 88.0 (#4)
GPT-6 on AA-Omniscience Index - Software Engineering (SWE) - PHP: 86.0 (#9)
GPT-6 on AA-Omniscience Index - Software Engineering (SWE) - Python: 89.0 (#4)
GPT-6 on AA-Omniscience Index - Software Engineering (SWE) - R: 62.0 (#11)
GPT-6 on AA-Omniscience Index - Software Engineering (SWE) - Rust: 86.0 (#7)
GPT-6 on AA-Omniscience Index - Software Engineering (SWE) - Swift: 88.0 (#11)
GPT-6 on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 97.78 (#1)
GPT-6 on Artificial Analysis Intelligence Index: 52.67 (#7)
GPT-6 on AutomationBench-AA: 68.49 (#5)
GPT-6 on AutomationBench-AA - Finance - Raw Objective Completion: 89.24 (#12)
GPT-6 on AutomationBench-AA - HR - Raw Objective Completion: 77.08 (#4)
GPT-6 on AutomationBench-AA - Marketing - Raw Objective Completion: 84.42 (#36)
GPT-6 on AutomationBench-AA - Operations - Raw Objective Completion: 97.01 (#2)
GPT-6 on AutomationBench-AA - Sales - Raw Objective Completion: 85.91 (#15)
GPT-6 on AutomationBench-AA - Strict: 41.55 (#3)
GPT-6 on AutomationBench-AA - Support - Raw Objective Completion: 92.43 (#2)
GPT-6 on Chatbot Arena (Text - Japanese): 1421.0 (#66)
GPT-6 on Design Arena (ASCII Art): 1343.0 (#2)
GPT-6 on Design Arena (Agents - Web Apps): 1406.0 (#1)
GPT-6 on Design Arena (Data Viz): 1310.0 (#13)
GPT-6 on Design Arena (Game Dev): 1458.0 (#2)
GPT-6 on Design Arena (UI Components): 1337.0 (#11)
GPT-6 on Design Arena (Website): 1313.0 (#12)
GPT-6 on Kaggle Adversarial Customer Service (Gert Labs): 75.0 (#7)
GPT-6 on MLCR-AA: 35.0 (#16)
GPT-6 on MLCR-AA - Accuracy: 98.28 (#1)
GPT-6 on MLCR-AA - Completeness: 36.78 (#22)
GPT-6 on MLCR-AA - Conciseness: 96.67 (#34)
GPT-6 on Tau3 Banking: 43.09 (#22)
GPT-6.1 Sol on AA CritPt: 31.71 (#5)
GPT-6.1 Sol on AA GDP.pdf: 32.0 (#2)
GPT-6.1 Sol on AA GDP.pdf - Construction - Criterion Pass Rate: 84.94 (#1)
GPT-6.1 Sol on AA GDP.pdf - Criterion Pass Rate: 82.86 (#1)
GPT-6.1 Sol on AA GDP.pdf - Engineering - Criterion Pass Rate: 81.27 (#10)
GPT-6.1 Sol on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 90.99 (#2)
GPT-6.1 Sol on AA GDP.pdf - HR - Criterion Pass Rate: 83.07 (#15)
GPT-6.1 Sol on AA GDP.pdf - Healthcare - Criterion Pass Rate: 91.78 (#5)
GPT-6.1 Sol on AA GDP.pdf - Insurance - Criterion Pass Rate: 78.78 (#34)
GPT-6.1 Sol on AA GDP.pdf - Legal - Criterion Pass Rate: 85.86 (#4)
GPT-6.1 Sol on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 77.89 (#18)
GPT-6.1 Sol on AA GDP.pdf - Real Estate - Criterion Pass Rate: 81.5 (#1)
GPT-6.1 Sol on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 79.37 (#9)
GPT-6.1 Sol on AA GDPval: 1575.09 (#32)
GPT-6.1 Sol on AA Humanity's Last Exam: 52.92 (#17)
GPT-6.1 Sol on AA Long Context Reasoning: 84.0 (#12)
GPT-6.1 Sol on AA MMMU-Pro: 85.95 (#6)
GPT-6.1 Sol on AA Omniscience: 41.53 (#11)
GPT-6.1 Sol on AA SciCode: 55.79 (#40)
GPT-6.1 Sol on AA-Briefcase: 55.66 (#18)
GPT-6.1 Sol on AA-Briefcase - Analytical Quality Elo: 1806.84 (#20)
GPT-6.1 Sol on AA-Briefcase - DOCX - Rubric Pass Rate: 52.07 (#30)
GPT-6.1 Sol on AA-Briefcase - PDF - Rubric Pass Rate: 53.45 (#29)
GPT-6.1 Sol on AA-Briefcase - PPTX - Rubric Pass Rate: 50.2 (#26)
GPT-6.1 Sol on AA-Briefcase - Presentation Elo: 1437.87 (#33)
GPT-6.1 Sol on AA-Briefcase - XLSX - Rubric Pass Rate: 59.77 (#8)
GPT-6.1 Sol on AA-Omniscience Accuracy: 62.1 (#12)
GPT-6.1 Sol on AA-Omniscience Hallucination Rate: 49.36 (#102)
GPT-6.1 Sol on AA-Omniscience Index - Business: 33.1 (#7)
GPT-6.1 Sol on AA-Omniscience Index - Health: 25.6 (#5)
GPT-6.1 Sol on AA-Omniscience Index - Humanities & Social Sciences: 40.3 (#8)
GPT-6.1 Sol on AA-Omniscience Index - Law: 44.0 (#12)
GPT-6.1 Sol on AA-Omniscience Index - Science, Engineering & Mathematics: 28.0 (#16)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE): 82.2 (#12)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - C: 90.0 (#10)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Dart: 64.0 (#29)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Go: 88.0 (#10)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - HTML: 92.0 (#4)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Java: 77.0 (#2)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 87.27 (#5)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Julia: 92.0 (#4)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 84.0 (#12)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - PHP: 84.0 (#12)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Python: 86.0 (#10)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - R: 64.0 (#8)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Rust: 88.0 (#5)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Swift: 84.0 (#16)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 95.56 (#3)
GPT-6.1 Sol on AI Chess Leaderboard (Continuation): 1560.0 (#10)
GPT-6.1 Sol on AI Chess Leaderboard (Reasoning): 1527.0 (#24)
GPT-6.1 Sol on AI Coding Daily (Codex CLI) - Laravel Code Quality: 19.1 (#1)
GPT-6.1 Sol on AI Coding Daily (Codex CLI) - React-TS Code Quality: 19.0 (#1)
GPT-6.1 Sol on AI Coding Daily (Codex CLI) - Total: 58.1 (#1)
GPT-6.1 Sol on ARC-AGI-1: 98.5 (#7)
GPT-6.1 Sol on ARC-AGI-2: 94.17 (#3)
GPT-6.1 Sol on ARC-AGI-3: 52.73 (#4)
GPT-6.1 Sol on Aiera Leaderboard - Depth: 73.6 (#15)
GPT-6.1 Sol on Aiera Leaderboard - Facts: 61.58 (#23)
GPT-6.1 Sol on Aiera Leaderboard - Grounding: 74.5 (#30)
GPT-6.1 Sol on Aiera Leaderboard - Research Score: 69.89 (#22)
GPT-6.1 Sol on Artificial Analysis Intelligence Index: 51.83 (#11)
GPT-6.1 Sol on AutomationBench-AA: 66.57 (#10)
GPT-6.1 Sol on AutomationBench-AA - Finance - Raw Objective Completion: 89.3 (#10)
GPT-6.1 Sol on AutomationBench-AA - HR - Raw Objective Completion: 76.37 (#9)
GPT-6.1 Sol on AutomationBench-AA - Marketing - Raw Objective Completion: 85.42 (#32)
GPT-6.1 Sol on AutomationBench-AA - Operations - Raw Objective Completion: 96.82 (#3)
GPT-6.1 Sol on AutomationBench-AA - Sales - Raw Objective Completion: 85.28 (#20)
GPT-6.1 Sol on AutomationBench-AA - Strict: 37.14 (#9)
GPT-6.1 Sol on AutomationBench-AA - Support - Raw Objective Completion: 92.0 (#3)
GPT-6.1 Sol on Bug Hunt Bench: 36.5 (#10)
GPT-6.1 Sol on Bug Hunt Bench - LMS: 19.5 (#12)
GPT-6.1 Sol on Bug Hunt Bench - VS Code Extension: 17.0 (#13)
GPT-6.1 Sol on Chatbot Arena (Code): 1759.0 (#3)
GPT-6.1 Sol on Chess Puzzles (Epoch AI): 61.0 (#4)
GPT-6.1 Sol on ChessBench: 71.6 (#3)
GPT-6.1 Sol on ChessBench - Accuracy: 76.6 (#6)
GPT-6.1 Sol on ChessBench - Coherence: 83.7 (#5)
GPT-6.1 Sol on ChessBench - Elo: 1336.0 (#3)
GPT-6.1 Sol on Epoch AI - Apex Agents: 60.0 (#10)
GPT-6.1 Sol on Epoch AI - Critpt: 31.71 (#5)
GPT-6.1 Sol on Epoch AI - Ebr Bench: 54.29 (#4)
GPT-6.1 Sol on Epoch AI - Frontiermath Tier 4 V2: 100.0 (#1)
GPT-6.1 Sol on Epoch AI - Frontiermath Tiers 1 3 V2: 93.68 (#1)
GPT-6.1 Sol on Epoch AI - Furniture Assembly: 80.0 (#2)
GPT-6.1 Sol on Epoch AI - GPQA Diamond: 95.39 (#3)
GPT-6.1 Sol on Epoch AI - Mystery Game Puzzles: 80.0 (#2)
GPT-6.1 Sol on Epoch AI - Proofbench: 99.0 (#6)
GPT-6.1 Sol on Epoch AI - Scicode: 55.79 (#33)
GPT-6.1 Sol on FrontierMath - Tier 4 (v2): 100.0 (#1)
GPT-6.1 Sol on FrontierMath - Tiers 1-3 (v2): 93.68 (#1)
GPT-6.1 Sol on Gert Labs Rankings: 78.21 (#2)
GPT-6.1 Sol on Icelandic LLM - Case Government: 97.19 (#8)
GPT-6.1 Sol on Icelandic LLM - Famous Phrases: 29.07 (#4)
GPT-6.1 Sol on Icelandic LLM - Flora & Fauna: 53.49 (#5)
GPT-6.1 Sol on Icelandic LLM - GEC: 74.31 (#8)
GPT-6.1 Sol on Icelandic LLM - Generative Quality: 78.81 (#5)
GPT-6.1 Sol on Icelandic LLM - Idiom Completion: 71.18 (#4)
GPT-6.1 Sol on Icelandic LLM - Idiom Meanings: 82.82 (#8)
GPT-6.1 Sol on Icelandic LLM - Song Lyrics: 14.05 (#24)
GPT-6.1 Sol on Icelandic LLM - Textile Terms: 72.9 (#4)
GPT-6.1 Sol on Icelandic LLM - WikiQA: 65.92 (#11)
GPT-6.1 Sol on Icelandic LLM Leaderboard - Average: 68.17 (#6)
GPT-6.1 Sol on MLCR-AA: 33.89 (#17)
GPT-6.1 Sol on MLCR-AA - Accuracy: 95.4 (#3)
GPT-6.1 Sol on MLCR-AA - Completeness: 36.21 (#23)
GPT-6.1 Sol on MLCR-AA - Conciseness: 96.67 (#35)
GPT-6.1 Sol on MathArena - ARXIV August: 96.49 (#1)
GPT-6.1 Sol on MathArena - ARXIV June: 94.44 (#1)
GPT-6.1 Sol on MathArena - ARXIV_FALSE August: 86.31 (#1)
GPT-6.1 Sol on MathArena - ARXIV_FALSE June: 100.0 (#2)
GPT-6.1 Sol on MathArena Arxiv: 95.0 (#1)
GPT-6.1 Sol on MathArena Arxiv False: 90.0 (#3)
GPT-6.1 Sol on Mercor APEX: 60.0 (#12)
GPT-6.1 Sol on NonoBench: 93.3 (#5)
GPT-6.1 Sol on OTIS Mock AIME 2024-25: 100.0 (#1)
GPT-6.1 Sol on OpenRouter GPQA Diamond: 93.8 (#6)
GPT-6.1 Sol on OpenRouter Tau2-Bench Airline: 66.7 (#84)
GPT-6.1 Sol on SimpleQA Verified: 73.9 (#2)
GPT-6.1 Sol on TaxCalcBench: 36.0 (#15)
GPT-6.1 Sol on Tinybird AI SQL Benchmark - Exactness: 50.71 (#121)
GPT-6.1 Sol on Tinybird AI SQL Benchmark - First-Attempt Success Rate: 84.0 (#263)
GPT-6.1 Sol on Tinybird AI SQL Benchmark - Success Rate: 96.0 (#263)
GPT-6.1 Sol on Vals AI (Vals Index): 61.15 (#8)
GPT-6.1 Sol on Vals AI BioMysteryBench: 79.63 (#2)
GPT-6.1 Sol on Vals AI Code Migration: 65.12 (#5)
GPT-6.1 Sol on Vals AI CyberBench: 39.29 (#41)
GPT-6.1 Sol on Vals AI Excel Modeling: 70.77 (#12)
GPT-6.1 Sol on Vals AI Finance Agent v2: 52.03 (#31)
GPT-6.1 Sol on Vals AI Harvey Legal Agent Bench: 5.42 (#30)
GPT-6.1 Sol on Vals AI IOI: 96.89 (#3)
GPT-6.1 Sol on Vals AI Legal Research Bench: 38.46 (#30)
GPT-6.1 Sol on Vals AI MedCode: 48.84 (#26)
GPT-6.1 Sol on Vals AI MedScribe: 86.45 (#21)
GPT-6.1 Sol on Vals AI MysteryMechanism: 46.4 (#5)
GPT-6.1 Sol on Vals AI ProofBench: 99.0 (#5)
GPT-6.1 Sol on Vals AI Public Benefits Bench: 59.34 (#31)
GPT-6.1 Sol on Vals AI ReverseEngBench: 50.76 (#2)
GPT-6.1 Sol on Vals AI SAGE: 46.54 (#35)
GPT-6.1 Sol on Vals AI Tax Agent Bench: 62.31 (#27)
GPT-6.1 Sol on Vals AI Terminal-Bench 4.0: 55.05 (#6)
GPT-6.1 Sol on Vals AI Vibe Code Bench: 88.93 (#7)
GPT-6.1 Sol on WebDev Arena: 1758.72 (#3)
GPT-6.1 Sol on WebDev Arena (Brand & Marketing): 1677.0 (#6)
GPT-6.1 Sol on WebDev Arena (Consumer Product): 1738.0 (#1)
GPT-6.1 Sol on WebDev Arena (Data & Analytics): 1632.0 (#3)
GPT-6.1 Sol on WebDev Arena (Frontend): 1775.0 (#4)
GPT-6.1 Sol on WebDev Arena (Gaming): 1853.0 (#4)
GPT-6.1 Sol on WebDev Arena (HTML): 1809.0 (#1)
GPT-6.1 Sol on WebDev Arena (React): 1762.0 (#4)
GPT-6.1 Sol on WebDev Arena (Reference-Based Design): 1793.0 (#4)
GPT-6.1 Sol on WebDev Arena (Simulations): 1846.0 (#3)
Gemini 4 Argon on AA CritPt: 27.14 (#31)
Gemini 4 Argon on AA GDP.pdf: 21.8 (#47)
Gemini 4 Argon on AA GDP.pdf - Construction - Criterion Pass Rate: 74.97 (#18)
Gemini 4 Argon on AA GDP.pdf - Criterion Pass Rate: 79.91 (#17)
Gemini 4 Argon on AA GDP.pdf - Engineering - Criterion Pass Rate: 81.7 (#9)
Gemini 4 Argon on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 80.24 (#53)
Gemini 4 Argon on AA GDP.pdf - HR - Criterion Pass Rate: 79.63 (#30)
Gemini 4 Argon on AA GDP.pdf - Healthcare - Criterion Pass Rate: 85.57 (#36)
Gemini 4 Argon on AA GDP.pdf - Insurance - Criterion Pass Rate: 88.24 (#4)
Gemini 4 Argon on AA GDP.pdf - Legal - Criterion Pass Rate: 82.77 (#28)
Gemini 4 Argon on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 80.12 (#7)
Gemini 4 Argon on AA GDP.pdf - Real Estate - Criterion Pass Rate: 73.21 (#23)
Gemini 4 Argon on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 75.8 (#34)
Gemini 4 Argon on AA GDPval: 1611.33 (#21)
Gemini 4 Argon on AA Humanity's Last Exam: 57.09 (#5)
Gemini 4 Argon on AA Long Context Reasoning: 79.67 (#79)
Gemini 4 Argon on AA Omniscience: 42.35 (#9)
Gemini 4 Argon on AA SciCode: 61.81 (#4)
Gemini 4 Argon on AA-Briefcase: 64.63 (#1)
Gemini 4 Argon on AA-Briefcase - Analytical Quality Elo: 1575.53 (#45)
Gemini 4 Argon on AA-Briefcase - DOCX - Rubric Pass Rate: 62.13 (#1)
Gemini 4 Argon on AA-Briefcase - PDF - Rubric Pass Rate: 72.41 (#1)
Gemini 4 Argon on AA-Briefcase - PPTX - Rubric Pass Rate: 67.36 (#1)
Gemini 4 Argon on AA-Briefcase - Presentation Elo: 1308.29 (#57)
Gemini 4 Argon on AA-Briefcase - XLSX - Rubric Pass Rate: 61.14 (#4)
Gemini 4 Argon on AA-Omniscience Accuracy: 49.9 (#61)
Gemini 4 Argon on AA-Omniscience Hallucination Rate: 15.07 (#5)
Gemini 4 Argon on AA-Omniscience Index - Business: 33.3 (#5)
Gemini 4 Argon on AA-Omniscience Index - Health: 22.4 (#12)
Gemini 4 Argon on AA-Omniscience Index - Humanities & Social Sciences: 39.6 (#11)
Gemini 4 Argon on AA-Omniscience Index - Law: 53.0 (#1)
Gemini 4 Argon on AA-Omniscience Index - Science, Engineering & Mathematics: 34.4 (#2)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE): 71.4 (#27)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - C: 90.0 (#8)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - Dart: 68.0 (#24)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - Go: 76.0 (#27)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - HTML: 82.0 (#22)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - Java: 54.0 (#23)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 72.73 (#39)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - Julia: 52.0 (#44)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 54.0 (#49)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - PHP: 76.0 (#28)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - Python: 68.5 (#48)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - R: 58.0 (#16)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - Rust: 84.0 (#10)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - Swift: 76.0 (#22)
Gemini 4 Argon on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 80.0 (#34)
Gemini 4 Argon on Agent Arena: 7.92 (#8)
Gemini 4 Argon on Agent Arena - Bash Recovery: -18.07 (#43)
Gemini 4 Argon on Agent Arena - Confirmed Success: 14.15 (#2)
Gemini 4 Argon on Agent Arena - Praise vs Complaint: 27.74 (#4)
Gemini 4 Argon on Agent Arena - Steerability: 15.88 (#1)
Gemini 4 Argon on Agent Arena - Tool Hallucination: 0.1 (#34)
Gemini 4 Argon on Artificial Analysis Intelligence Index: 52.56 (#8)
Gemini 4 Argon on AutomationBench: 51.3 (#1)
Gemini 4 Argon on AutomationBench-AA: 77.51 (#1)
Gemini 4 Argon on AutomationBench-AA - Finance - Raw Objective Completion: 94.62 (#1)
Gemini 4 Argon on AutomationBench-AA - HR - Raw Objective Completion: 81.99 (#1)
Gemini 4 Argon on AutomationBench-AA - Marketing - Raw Objective Completion: 94.66 (#1)
Gemini 4 Argon on AutomationBench-AA - Operations - Raw Objective Completion: 96.39 (#9)
Gemini 4 Argon on AutomationBench-AA - Sales - Raw Objective Completion: 92.0 (#1)
Gemini 4 Argon on AutomationBench-AA - Strict: 53.12 (#1)
Gemini 4 Argon on AutomationBench-AA - Support - Raw Objective Completion: 94.72 (#1)
Gemini 4 Argon on BenchLM: 77.2 (#10)
Gemini 4 Argon on Chartography: 71.6 (#1)
Gemini 4 Argon on FrontierSWE V2: 54.96 (#5)
Gemini 4 Argon on Vending-Bench 2: 13718.16 (#3)

New #1 Leaders (56)

Design Arena (Agents - Web Apps): GPT-6 (Max) (1406.0) beat Claude Fable 5.1 by 85.0
Chatbot Arena (Text - Multi-Turn): gemini-4-argon-high (1552.0) beat Claude Opus 4.6 (High) by 33.0
Chatbot Arena (Text - Entertainment, Sports & Media): gemini-4-argon-high (1522.0) beat Claude Fable 5 (High) by 28.0
WebDev Arena (HTML): GPT-6.1 Sol (Max) (1809.0) beat GPT-6 (Max) by 25.0
WebDev Arena (Consumer Product): GPT-6.1 Sol (Max) (1738.0) beat GPT-6 (Max) by 24.0
Design Arena (Agents - Fullstack): GPT-6 (1355.0) beat Claude Fable 5.1 by 21.0
Chatbot Arena (Text - Software & IT Services): gemini-4-argon-high (1562.0) beat Claude Opus 4.7 (High) by 21.0
Chatbot Arena (Text - English): gemini-4-argon-high (1532.0) beat Claude Opus 4.6 (High) by 18.0
Chatbot Arena (Text): gemini-4-argon-high (1525.0) beat Claude Opus 5.5 (High) by 16.0
Chatbot Arena (Text - Hard Prompts (English)): gemini-4-argon-high (1552.0) beat Claude Opus 4.6 (High) by 16.0
Chatbot Arena (Text - Chinese): gemini-4-argon-high (1588.0) beat Claude Fable 5.1 (Max) by 15.0
Chatbot Arena (Text - Longer Query): gemini-4-argon-high (1544.0) beat Claude Opus 5.5 (High) by 14.0
Chatbot Arena (Text - Instruction Following): gemini-4-argon-high (1529.0) beat Claude Opus 5.5 (High) by 13.0
Chatbot Arena (Text - Medicine & Healthcare): gemini-4-argon-high (1549.0) beat Muse Spark 1.2 (xHigh) by 13.0
Chatbot Arena (Text - Non-English): gemini-4-argon-high (1512.0) beat Claude Opus 5.5 (High) by 11.0
Chatbot Arena (Text - Hard Prompts): gemini-4-argon-high (1550.0) beat Claude Opus 5.5 (High) by 9.0
AutomationBench-AA - Strict: Gemini 4 Argon (High) (53.12) beat Claude Sonnet 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) by 8.37
Chatbot Arena (Text - Coding): gemini-4-argon-high (1559.0) beat Claude Opus 4.6 (High) by 8.0
Chatbot Arena (Text - Mathematical): gemini-4-argon-high (1542.0) beat Claude Opus 5 (High) by 7.0
Mercor APEX: Gemini 4 Argon (82.2) beat Claude Sonnet 5.5 by 6.7
AutomationBench: Gemini 4 Argon (High) (51.3) beat Claude Sonnet 5.5 (default fallbacks, Max) by 6.5
AutomationBench-AA: Gemini 4 Argon (High) (77.51) beat Claude Sonnet 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) by 6.18
Chatbot Arena (Text - Russian): gemini-4-argon-high (1531.0) beat Claude Opus 5.5 (High) by 6.0
Chatbot Arena (Text - Math): gemini-4-argon-high (1528.0) beat Claude Fable 5 (High) by 5.0
MathArena - ARXIV_FALSE August: GPT-6.1 Sol (Max) (86.31) beat GPT-6 (Max) by 4.37
AA-Briefcase - PDF - Rubric Pass Rate: Gemini 4 Argon (High) (72.41) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 4.31
AA-Briefcase - PPTX - Rubric Pass Rate: Gemini 4 Argon (High) (67.36) beat Claude Fable 5.1 (Adaptive Reasoning, Xhigh Effort, Default Fallback) by 4.09
Chatbot Arena (Text - Life, Physical & Social Science): gemini-4-argon-high (1539.0) beat Claude Opus 5.5 (High) by 4.0
Chatbot Arena (Text - Business, Management & Financial Ops): gemini-4-argon-high (1523.0) beat Muse Spark 1.2 (xHigh) by 4.0
Vals AI Finance Agent v2: gemini-4-argon (65.4) beat Gemini 3.8 Flash by 3.96
AutomationBench-AA - Marketing - Raw Objective Completion: Gemini 4 Argon (High) (94.66) beat MiMo-V2.6-Pro by 3.55
MathArena - ARXIV August: GPT-6.1 Sol (Max) (96.49) beat GPT-6 Sol (Max) by 3.51
Chatbot Arena (Text - Writing, Literature & Language): gemini-4-argon-high (1523.0) beat Claude Opus 5.5 (High) by 3.0
AA-Omniscience Index - Law: Gemini 4 Argon (High) (53.0) beat Claude Fable 5.1 (Adaptive Reasoning, Xhigh Effort, Default Fallback) by 2.9
BALSAM - Summarization: Nojoom-AQL-42B (56.58) beat allam-2-34B-prod (Reasoning) by 2.57
FrontierMath - Tier 4 (v2): GPT-6.1 Sol (Max) (100.0) beat GPT-6 (High) by 2.4
Epoch AI - Frontiermath Tier 4 V2: GPT-6.1 Sol (Max) (100.0) beat GPT-6 (High) by 2.4
AutomationBench-AA - HR - Raw Objective Completion: Gemini 4 Argon (High) (81.99) beat Claude Sonnet 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) by 2.39
AutomationBench-AA - Support - Raw Objective Completion: Gemini 4 Argon (High) (94.72) beat GPT-6 (xHigh) by 2.29
AutomationBench-AA - Sales - Raw Objective Completion: Gemini 4 Argon (High) (92.0) beat GPT-6 Sol (xHigh) by 1.93
Vals AI (Vals Index): gemini-4-argon (68.9) beat Claude Sonnet 5.5 by 1.86
AA-Briefcase - DOCX - Rubric Pass Rate: Gemini 4 Argon (High) (62.13) beat Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) by 1.77
AA-Briefcase: Gemini 4 Argon (High) (64.63) beat Claude Sonnet 5.5 (Adaptive Reasoning, Xhigh Effort, Default Fallback) by 1.4
Agent Arena - Steerability: Gemini 4 Argon (High) (15.88) beat GPT-6 Sol (Max) by 1.34
AI Coding Daily (Codex CLI) - Total: GPT-6.1 Sol (High) (58.1) beat GPT-6 (High) by 1.33
VLA-Arena Extrapolation: Paimon (68.89) beat MolmoAct2 by 1.26
AutomationBench-AA - Finance - Raw Objective Completion: Gemini 4 Argon (High) (94.62) beat Grok 4.7 (xHigh) by 1.13
Chatbot Arena (Text - Creative Writing): gemini-4-argon-high (1522.0) beat Claude Opus 5.5 (High) by 1.0
VLA-Arena Overall: Paimon (73.08) beat MolmoAct2 by 0.96
AI Coding Daily (Codex CLI) - React-TS Code Quality: GPT-6.1 Sol (High) (19.0) beat GPT-6 (High) by 0.83
Epoch AI - ECI: Claude Opus 5.5 (167.35) beat GPT-6 by 0.75
Harvey LAB-AA - Healthcare & Life Sciences: Step 5 Preview (96.04) beat Kimi K3 (Max) by 0.66
Chartography: Gemini 4 Argon (High) (71.6) beat GPT-6 (Max) by 0.6
AI Coding Daily (Codex CLI) - Laravel Code Quality: GPT-6.1 Sol (High) (19.1) beat GPT-6 (High) by 0.5
EnterpriseOps-Gym-AA - Email: Step 5 Preview (70.19) beat Qwen 3.7 Max by 0.32
Blueprint-Bench 2: Gemini 4 Argon (0.544) beat Claude Opus 5.5 by 0.03

The Aggregate Digest — 2026-09-30

2026-09-30T12:28:44.651636+00:00

Frontier

Best available model: GPT-6.1 Sol enters at #3 (Aggregate Rating 2035) on 114 benchmarks

Daily

New Benchmarks (3)

LLM Stats (AutomationBench v1.0.6) (Score (%)): leader MiMo-V2.6-Pro (53.1), 5 models
Zapier's AutomationBench v1.0.6 as listed by LLM Stats: multi-step business-workflow automation through tool calls in simulated SaaS apps. Scored 0–1 as the share of objectives completed; results are vendor self-reported.
LLM Stats (HealthBench Consensus) (Score (%)): leader GPT-6.1 Sol (96.0), 5 models
OpenAI's HealthBench Consensus as listed by LLM Stats: the HealthBench subset graded only on rubric criteria with high physician agreement, over multi-turn conversations with users and clinicians. Scored 0–1; higher is better.
LLM Stats (Terminal-Bench-Science 0.1) (Score (%)): leader GPT-6 (64.6), 5 models
Terminal-Bench-Science 0.1 as listed by LLM Stats: 70 outcome-verified scientific research workflows across the life, physical, earth, mathematical and engineering sciences, completed by agents in containerized terminals. Accuracy 0–1; higher is better.

New Models (1)

GPT-6.1 Sol — Aggregate Rating 2035, #3
- Explicit Edit Benchmark: 100.0 (#1/24)
- AA GDP.pdf - Criterion Pass Rate: 82.86 (#1/195)
- AA GDP.pdf - Construction - Criterion Pass Rate: 84.94 (#1/195)
- AA GDP.pdf - Real Estate - Criterion Pass Rate: 81.5 (#1/195)
- EnigmaForge: 84.5 (#1/24)
- EnigmaForge - Intuition: 83.8 (#1/24)
- EnigmaForge - Fact F1: 98.5 (#1/24)
- Bug Hunt Bench - VS Code Extension: 24.0 (#1/98)
- AA GDP.pdf: 32.0 (#2/195)
- AA-Omniscience Index - Software Engineering (SWE) - Java: 77.0 (#2/559)
- ...and 101 more

Top-10 New Scores (226)

Claude Fable 5 on GSO-Bench: 78.43 (#3)
Claude Fable 5 on Vals AI Public Benefits Bench: 70.43 (#4)
Claude Fable 5.1 on Vals AI Public Benefits Bench: 74.9 (#2)
Claude Fable 5.1 on Vals AI ReverseEngBench: 22.9 (#5)
Claude Fable 5.1 on Vals AI Tax Agent Bench: 77.64 (#1)
Claude Opus 5 on PostTrainBench: 35.04 (#4)
Claude Opus 5 on TrackingAI IQ Test (Vision): 100.0 (#1)
Claude Opus 5 on Vals AI Public Benefits Bench: 76.93 (#1)
Claude Opus 5 on Vals AI ReverseEngBench: 12.21 (#6)
Claude Opus 5 on Vals AI Tax Agent Bench: 75.06 (#2)
Claude Opus 5.5 on Epoch AI - Critpt: 31.71 (#4)
Claude Opus 5.5 on Epoch AI - Cursorbench: 56.0 (#2)
Claude Opus 5.5 on Epoch AI - Frontiermath Tier 4 V2: 95.0 (#5)
Claude Opus 5.5 on Epoch AI - Frontiermath Tiers 1 3 V2: 91.23 (#2)
Claude Opus 5.5 on Epoch AI - GPQA Diamond: 90.59 (#37)
Claude Opus 5.5 on Epoch AI - Mirrorcode: 77.4 (#1)
Claude Opus 5.5 on Epoch AI - Mystery Game Puzzles: 71.0 (#2)
Claude Opus 5.5 on Epoch AI - Scicode: 65.05 (#2)
Claude Opus 5.5 on FrontierMath - Tier 4 (v2): 95.0 (#5)
Claude Opus 5.5 on FrontierMath - Tiers 1-3 (v2): 91.23 (#2)
Claude Opus 5.5 on OTIS Mock AIME 2024-25: 100.0 (#2)
Claude Opus 5.5 on ReLE - Agents - BFCL-V3: 74.1 (#6)
Claude Opus 5.5 on ReLE - Agents and Tool Use: 70.8 (#8)
Claude Opus 5.5 on ReLE - Education: 62.5 (#29)
Claude Opus 5.5 on ReLE - Education - Gaokao: 66.4 (#7)
Claude Opus 5.5 on ReLE - Education - High School Subjects: 75.1 (#9)
Claude Opus 5.5 on ReLE - Education - Middle School Subjects: 57.7 (#9)
Claude Opus 5.5 on ReLE - Education - Primary School Subjects: 50.7 (#113)
Claude Opus 5.5 on ReLE - Finance: 84.6 (#21)
Claude Opus 5.5 on ReLE - Language and Instruction Following: 74.7 (#10)
Claude Opus 5.5 on ReLE - Law - Lawyer Qualification Exam: 76.7 (#45)
Claude Opus 5.5 on ReLE - Law and Civil Service: 83.3 (#30)
Claude Opus 5.5 on ReLE - Medicine - Physician Exams: 91.7 (#2)
Claude Opus 5.5 on ReLE - Medicine and Mental Health: 89.0 (#12)
Claude Opus 5.5 on ReLE - Reasoning - BBH: 83.2 (#36)
Claude Opus 5.5 on ReLE - Reasoning and Mathematics: 90.6 (#2)
Claude Opus 5.5 on SimpleQA Verified: 72.2 (#3)
Claude Opus 5.5 on Supabase Evals (No Skills): 91.3 (#1)
Claude Opus 5.5 on Supabase Evals (Skills): 89.9 (#1)
Claude Opus 5.5 on Vals AI Public Benefits Bench: 70.64 (#3)
Claude Opus 5.5 on Vals AI ReverseEngBench: 33.59 (#2)
Claude Opus 5.5 on WebDev Arena: 1820.31 (#1)
Claude Opus 5.5 on WebDev Arena (Fullstack): 1741.0 (#2)
Claude Sonnet 5.5 on AI Coding Daily (Claude Code) - Laravel Code Quality: 18.68 (#2)
Claude Sonnet 5.5 on AI Coding Daily (Claude Code) - React-TS Code Quality: 18.5 (#3)
Claude Sonnet 5.5 on AI Coding Daily (Claude Code) - Total: 56.18 (#3)
Claude Sonnet 5.5 on Aiera Leaderboard - Depth: 62.0 (#23)
Claude Sonnet 5.5 on Aiera Leaderboard - Facts: 54.34 (#28)
Claude Sonnet 5.5 on Aiera Leaderboard - Grounding: 75.7 (#28)
Claude Sonnet 5.5 on Aiera Leaderboard - Research Score: 64.01 (#28)
Claude Sonnet 5.5 on BridgeBench Overall: 647.0 (#4)
Claude Sonnet 5.5 on Bug Hunt Bench: 34.3 (#13)
Claude Sonnet 5.5 on Bug Hunt Bench - LMS: 18.7 (#12)
Claude Sonnet 5.5 on Bug Hunt Bench - VS Code Extension: 15.7 (#18)
Claude Sonnet 5.5 on Chatbot Arena (Code): 1699.0 (#4)
Claude Sonnet 5.5 on Coarena: 976.1 (#17)
Claude Sonnet 5.5 on Coarena - Task Completion: 0.0 (#23)
Claude Sonnet 5.5 on Epoch AI - Apex Agents: 75.5 (#1)
Claude Sonnet 5.5 on Epoch AI - Critpt: 31.43 (#6)
Claude Sonnet 5.5 on Epoch AI - Cursorbench: 55.5 (#3)
Claude Sonnet 5.5 on Epoch AI - Frontiermath Tier 4 V2: 80.49 (#11)
Claude Sonnet 5.5 on Epoch AI - Frontiermath Tiers 1 3 V2: 88.77 (#5)
Claude Sonnet 5.5 on Epoch AI - Frontierswe: 61.89 (#3)
Claude Sonnet 5.5 on Epoch AI - Furniture Assembly: 75.0 (#3)
Claude Sonnet 5.5 on Epoch AI - GPQA Diamond: 95.58 (#2)
Claude Sonnet 5.5 on Epoch AI - Mystery Game Puzzles: 65.0 (#3)
Claude Sonnet 5.5 on Epoch AI - Proofbench: 100.0 (#3)
Claude Sonnet 5.5 on Epoch AI - Scicode: 61.0 (#5)
Claude Sonnet 5.5 on FrontierMath - Tier 4 (v2): 80.49 (#11)
Claude Sonnet 5.5 on FrontierMath - Tiers 1-3 (v2): 88.77 (#5)
Claude Sonnet 5.5 on FrontierSWE V2: 61.89 (#3)
Claude Sonnet 5.5 on Gert Labs Rankings: 69.92 (#6)
Claude Sonnet 5.5 on HLE-Diamond: 43.3 (#9)
Claude Sonnet 5.5 on HLE-Diamond - Knowledge: 32.6 (#10)
Claude Sonnet 5.5 on HLE-Diamond - Reasoning: 54.0 (#10)
Claude Sonnet 5.5 on KernelBench Hub - CUDA: 94.42 (#2)
Claude Sonnet 5.5 on KernelBench Hub - Mega: 32.13 (#2)
Claude Sonnet 5.5 on LLM2014 Logic 2026-09: 66.5 (#6)
Claude Sonnet 5.5 on LOL Arena: 46.7 (#28)
Claude Sonnet 5.5 on Legal Benchmarks - Contract Workflows - Criteria Pass Rate: 85.3 (#6)
Claude Sonnet 5.5 on Legal Benchmarks - Contract Workflows - Form: 91.1 (#3)
Claude Sonnet 5.5 on Legal Benchmarks - Contract Workflows - Task Pass Rate: 29.0 (#6)
Claude Sonnet 5.5 on Legal Benchmarks - Data Extraction - Criteria Pass Rate: 83.3 (#7)
Claude Sonnet 5.5 on Legal Benchmarks - Data Extraction - Form: 77.6 (#6)
Claude Sonnet 5.5 on Legal Benchmarks - Data Extraction - Task Pass Rate: 40.0 (#16)
Claude Sonnet 5.5 on NonoBench: 80.0 (#19)
Claude Sonnet 5.5 on OTIS Mock AIME 2024-25: 100.0 (#1)
Claude Sonnet 5.5 on OpenRouter GPQA Diamond: 92.9 (#10)
Claude Sonnet 5.5 on OpenRouter Tau2-Bench Airline: 76.7 (#12)
Claude Sonnet 5.5 on ParseBench: 70.18 (#24)
Claude Sonnet 5.5 on ParseBench - Charts: 37.69 (#52)
Claude Sonnet 5.5 on ParseBench - Content Faithfulness: 91.64 (#4)
Claude Sonnet 5.5 on ParseBench - Semantic Formatting: 71.63 (#17)
Claude Sonnet 5.5 on ParseBench - Tables: 91.04 (#10)
Claude Sonnet 5.5 on ParseBench - Visual Grounding: 58.92 (#44)
Claude Sonnet 5.5 on SimpleQA Verified: 46.5 (#32)
Claude Sonnet 5.5 on SvelteBench: 100.0 (#6)
Claude Sonnet 5.5 on TaxCalcBench: 18.0 (#24)
Claude Sonnet 5.5 on Tinybird AI SQL Benchmark - Exactness: 60.18 (#6)
Claude Sonnet 5.5 on Tinybird AI SQL Benchmark - First-Attempt Success Rate: 92.0 (#201)
Claude Sonnet 5.5 on Tinybird AI SQL Benchmark - Success Rate: 100.0 (#53)
Claude Sonnet 5.5 on Vals AI Public Benefits Bench: 67.19 (#12)
Claude Sonnet 5.5 on Vals AI ReverseEngBench: 30.15 (#4)
Claude Sonnet 5.5 on WebDev Arena: 1698.82 (#4)
Claude Sonnet 5.5 on WebDev Arena (Brand & Marketing): 1659.0 (#9)
Claude Sonnet 5.5 on WebDev Arena (Consumer Product): 1621.0 (#7)
Claude Sonnet 5.5 on WebDev Arena (Data & Analytics): 1570.0 (#13)
Claude Sonnet 5.5 on WebDev Arena (Frontend): 1727.0 (#4)
Claude Sonnet 5.5 on WebDev Arena (Gaming): 1823.0 (#4)
Claude Sonnet 5.5 on WebDev Arena (HTML): 1707.0 (#2)
Claude Sonnet 5.5 on WebDev Arena (React): 1726.0 (#4)
Claude Sonnet 5.5 on WebDev Arena (Reference-Based Design): 1761.0 (#4)
Claude Sonnet 5.5 on WebDev Arena (Simulations): 1823.0 (#4)
GPT-6 on GSO-Bench: 79.41 (#2)
GPT-6 on Vals AI ReverseEngBench: 56.87 (#1)
GPT-6 on Vals AI Tax Agent Bench: 63.34 (#23)
GPT-6.1 Sol on AA CritPt: 31.71 (#5)
GPT-6.1 Sol on AA GDP.pdf: 32.0 (#2)
GPT-6.1 Sol on AA GDP.pdf - Construction - Criterion Pass Rate: 84.94 (#1)
GPT-6.1 Sol on AA GDP.pdf - Criterion Pass Rate: 82.86 (#1)
GPT-6.1 Sol on AA GDP.pdf - Engineering - Criterion Pass Rate: 81.27 (#9)
GPT-6.1 Sol on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 90.99 (#2)
GPT-6.1 Sol on AA GDP.pdf - HR - Criterion Pass Rate: 83.07 (#15)
GPT-6.1 Sol on AA GDP.pdf - Healthcare - Criterion Pass Rate: 91.78 (#5)
GPT-6.1 Sol on AA GDP.pdf - Insurance - Criterion Pass Rate: 78.78 (#33)
GPT-6.1 Sol on AA GDP.pdf - Legal - Criterion Pass Rate: 85.86 (#4)
GPT-6.1 Sol on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 77.89 (#17)
GPT-6.1 Sol on AA GDP.pdf - Real Estate - Criterion Pass Rate: 81.5 (#1)
GPT-6.1 Sol on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 79.37 (#9)
GPT-6.1 Sol on AA GDPval: 1575.09 (#31)
GPT-6.1 Sol on AA Humanity's Last Exam: 52.92 (#16)
GPT-6.1 Sol on AA Long Context Reasoning: 84.0 (#12)
GPT-6.1 Sol on AA MMMU-Pro: 85.95 (#6)
GPT-6.1 Sol on AA Omniscience: 41.53 (#10)
GPT-6.1 Sol on AA SciCode: 55.79 (#39)
GPT-6.1 Sol on AA-Briefcase: 55.66 (#17)
GPT-6.1 Sol on AA-Briefcase - Analytical Quality Elo: 1806.84 (#20)
GPT-6.1 Sol on AA-Briefcase - DOCX - Rubric Pass Rate: 52.07 (#29)
GPT-6.1 Sol on AA-Briefcase - PDF - Rubric Pass Rate: 53.45 (#28)
GPT-6.1 Sol on AA-Briefcase - PPTX - Rubric Pass Rate: 50.2 (#25)
GPT-6.1 Sol on AA-Briefcase - Presentation Elo: 1437.87 (#33)
GPT-6.1 Sol on AA-Briefcase - XLSX - Rubric Pass Rate: 59.77 (#7)
GPT-6.1 Sol on AA-Omniscience Accuracy: 62.1 (#12)
GPT-6.1 Sol on AA-Omniscience Hallucination Rate: 49.36 (#100)
GPT-6.1 Sol on AA-Omniscience Index - Business: 33.1 (#6)
GPT-6.1 Sol on AA-Omniscience Index - Health: 25.6 (#5)
GPT-6.1 Sol on AA-Omniscience Index - Humanities & Social Sciences: 40.3 (#8)
GPT-6.1 Sol on AA-Omniscience Index - Law: 44.0 (#11)
GPT-6.1 Sol on AA-Omniscience Index - Science, Engineering & Mathematics: 28.0 (#15)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE): 82.2 (#12)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - C: 90.0 (#9)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Dart: 64.0 (#28)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Go: 88.0 (#10)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - HTML: 92.0 (#4)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Java: 77.0 (#2)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 87.27 (#5)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Julia: 92.0 (#4)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 84.0 (#12)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - PHP: 84.0 (#12)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Python: 86.0 (#10)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - R: 64.0 (#8)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Rust: 88.0 (#5)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - Swift: 84.0 (#16)
GPT-6.1 Sol on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 95.56 (#3)
GPT-6.1 Sol on Artificial Analysis Intelligence Index: 51.83 (#10)
GPT-6.1 Sol on AutomationBench-AA: 66.57 (#9)
GPT-6.1 Sol on AutomationBench-AA - Finance - Raw Objective Completion: 89.3 (#9)
GPT-6.1 Sol on AutomationBench-AA - HR - Raw Objective Completion: 76.37 (#8)
GPT-6.1 Sol on AutomationBench-AA - Marketing - Raw Objective Completion: 85.42 (#31)
GPT-6.1 Sol on AutomationBench-AA - Operations - Raw Objective Completion: 96.82 (#3)
GPT-6.1 Sol on AutomationBench-AA - Sales - Raw Objective Completion: 85.28 (#19)
GPT-6.1 Sol on AutomationBench-AA - Strict: 37.14 (#8)
GPT-6.1 Sol on AutomationBench-AA - Support - Raw Objective Completion: 92.0 (#2)
GPT-6.1 Sol on BenchLM: 66.9 (#23)
GPT-6.1 Sol on BridgeBench Overall: 646.0 (#5)
GPT-6.1 Sol on Bug Hunt Bench: 44.0 (#3)
GPT-6.1 Sol on Bug Hunt Bench - LMS: 23.0 (#5)
GPT-6.1 Sol on Bug Hunt Bench - VS Code Extension: 24.0 (#1)
GPT-6.1 Sol on ChessBench GitHub - Elo: 2550.0 (#7)
GPT-6.1 Sol on Coarena: 1000.0 (#8)
GPT-6.1 Sol on Coarena - Task Completion: 0.0 (#22)
GPT-6.1 Sol on EnigmaForge: 84.5 (#1)
GPT-6.1 Sol on EnigmaForge - Fact F1: 98.5 (#1)
GPT-6.1 Sol on EnigmaForge - Intuition: 83.8 (#1)
GPT-6.1 Sol on HLE-Diamond: 61.1 (#2)
GPT-6.1 Sol on HLE-Diamond - Knowledge: 47.2 (#3)
GPT-6.1 Sol on HLE-Diamond - Reasoning: 75.0 (#2)
GPT-6.1 Sol on LLM Stats (DeepSWE 1.1): 75.22 (#2)
GPT-6.1 Sol on LLM Stats (GDP.pdf): 32.0 (#4)
GPT-6.1 Sol on LLM Stats (HealthBench Hard): 36.2 (#2)
GPT-6.1 Sol on LLM Stats (HealthBench Professional): 64.2 (#4)
GPT-6.1 Sol on LLM Stats (HealthBench): 58.5 (#4)
GPT-6.1 Sol on LLM Stats (OSWorld 2.0): 71.42 (#4)
GPT-6.1 Sol on LLM Stats Score: 52.04 (#11)
GPT-6.1 Sol on LM Market Cap LMC Score: 40.0 (#244)
GPT-6.1 Sol on LiveBench AMPS Hard: 98.0 (#43)
GPT-6.1 Sol on LiveBench Code Completion: 82.61 (#14)
GPT-6.1 Sol on LiveBench Code Generation: 80.28 (#24)
GPT-6.1 Sol on LiveBench Connections: 100.0 (#18)
GPT-6.1 Sol on LiveBench Consecutive Events: 91.29 (#3)
GPT-6.1 Sol on LiveBench Integrals With Game: 100.0 (#5)
GPT-6.1 Sol on LiveBench JavaScript: 63.64 (#46)
GPT-6.1 Sol on LiveBench Logic With Navigation: 86.0 (#7)
GPT-6.1 Sol on LiveBench Math Comp: 97.06 (#21)
GPT-6.1 Sol on LiveBench Olympiad: 92.27 (#6)
GPT-6.1 Sol on LiveBench Paraphrase: 72.58 (#18)
GPT-6.1 Sol on LiveBench Plot Unscrambling: 84.38 (#2)
GPT-6.1 Sol on LiveBench Python: 60.0 (#23)
GPT-6.1 Sol on LiveBench Simplify: 71.97 (#8)
GPT-6.1 Sol on LiveBench Spatial: 98.0 (#37)
GPT-6.1 Sol on LiveBench Story Generation: 70.82 (#36)
GPT-6.1 Sol on LiveBench Summarize: 81.25 (#8)
GPT-6.1 Sol on LiveBench Table Join: 57.42 (#2)
GPT-6.1 Sol on LiveBench Table Reformat: 100.0 (#24)
GPT-6.1 Sol on LiveBench Theory of Mind: 86.54 (#7)
GPT-6.1 Sol on LiveBench TypeScript: 46.67 (#25)
GPT-6.1 Sol on LiveBench Typos: 86.0 (#13)
GPT-6.1 Sol on LiveBench Zebra Puzzle: 100.0 (#18)
GPT-6.1 Sol on MLCR-AA: 33.89 (#17)
GPT-6.1 Sol on MLCR-AA - Accuracy: 95.4 (#3)
GPT-6.1 Sol on MLCR-AA - Completeness: 36.21 (#23)
GPT-6.1 Sol on MLCR-AA - Conciseness: 96.67 (#35)
GPT-6.1 Sol on Maze-Bench: 88.57 (#2)
GPT-6.1 Sol on RuneBench: 21772.0 (#2)
GPT-6.1 Sol on SvelteBench: 100.0 (#18)
GPT-6.1 Sol on VoxelBench: 2564.0 (#2)

New #1 Leaders (21)

GSO-Bench: Claude Fable 5.1 (88.24) beat Claude Opus 4.8 by 41.18
WebDev Arena: Claude Opus 5.5 (Max) (1820.31) beat GPT-6 (Max) by 20.03
Coarena - Task Completion: GPT-6 Luna (100.0) beat GPT-5.6 Luna by 9.7
EnigmaForge: GPT-6.1 Sol (Low) (84.5) beat GPT-6 (Low) by 4.2
Epoch AI - Mirrorcode: Claude Opus 5.5 (Max) (77.4) beat Claude Fable 5.1 (High) by 4.1
EnigmaForge - Intuition: GPT-6.1 Sol (Low) (83.8) beat GPT-6 (Low) by 3.7
DecBench: GPT-6 (60.0) beat Claude Opus 4.8 by 3.2
AutomationBench: Claude Sonnet 5.5 (default fallbacks, Max) (44.8) beat Claude Opus 5.5 (default fallbacks, Max) by 2.3
Epoch AI - Apex Agents: Claude Sonnet 5.5 (Max) (75.5) beat Claude Opus 5.5 (Max) by 2.0
TrackingAI IQ Test (Vision): Claude Opus 5 (Max) (100.0) beat GPT-5.6 Sol (Ultra) by 1.96
EnigmaForge - Fact F1: GPT-6.1 Sol (Low) (98.5) beat GPT-6 Sol (Low) by 1.9
ReLE - Medicine - Basic Medical Knowledge: Claude Opus 5.5 (91.1) beat GPT-6 by 1.9
ReLE - Overall: Claude Opus 5.5 (78.6) beat Qwen 3.7 Max by 1.7
AA GDP.pdf - Real Estate - Criterion Pass Rate: GPT-6.1 Sol (xHigh) (81.5) beat GPT-6 Sol (High) by 1.66
Supabase Evals (No Skills): Claude Opus 5.5 (High) (91.3) beat Claude Opus 5 (High) by 1.4
Explicit Edit Benchmark: GPT-6.1 Sol (100.0) beat Muse Spark 1.3 Contributor by 1.0
Design Arena (UI Components): Claude Opus 5.5 (1376.0) beat GPT-6 by 1.0
AA GDP.pdf - Criterion Pass Rate: GPT-6.1 Sol (High) (82.86) beat Claude Opus 5.5 (Adaptive Reasoning, High Effort, Default Fallback) by 0.61
ReLE - Coding: Claude Opus 5.5 (76.3) beat Qwen 3.6 Max Preview by 0.6
AA GDP.pdf - Construction - Criterion Pass Rate: GPT-6.1 Sol (High) (84.94) beat GPT-6 (High) by 0.43
CADGenBench: build123d-mcp 0.3.85 MCP server + Claude Opus 5.5 (max effort) r2 (0.8298) beat build123d-mcp-v0381-claude-opus-5-xhigh-full-r6 by 0.19

The Aggregate Digest — 2026-09-29

2026-09-29T09:34:51.786134+00:00

Frontier

Best available model: Claude Sonnet 5.5 enters at #7 (Aggregate Rating 1985) on 162 benchmarks

Daily

New Benchmarks (83)

LLM Stats (ArXivMath) (Score (%)): leader Claude Opus 5.5 (91.2), 5 models
MathArena's ArXivMath as listed by LLM Stats: final-answer research mathematics problems extracted monthly from recent arXiv abstracts and checked to be self-contained and verifiable. Accuracy in percent; higher is better.
LongRCA-Mini (All-at-Once) (Root-step accuracy (%; exact match to the earliest decisive root-cause step; LongRCA-Mini, 200 fixed human-annotated failed agent trajectories, 40 from each of SWE-bench Pro, Terminal-Bench 2, TravelPlanner, VitaBench and WebArena Verified (macro-average of the five sources); All-at-once protocol: the whole execution log in one attribution prompt; invalid outputs count as errors)): leader GPT-5.6 Sol (17.0), 5 models
LongRCA Bench (2026) collects failed agent trajectories from five benchmarks, with the earliest decisive root-cause step annotated by humans; LongRCA-Mini is its fixed 200-trajectory sample, 40 per source. The model reads the whole execution log in one prompt and names the root-cause step; the score is the percentage of exact matches, macro-averaged over sources. Higher is better.
LongRCA-Mini - Responsible-Role Accuracy (All-at-Once) (Responsible-role accuracy (%; the role that took the root-cause step; LongRCA-Mini, 200 fixed human-annotated failed agent trajectories, 40 from each of SWE-bench Pro, Terminal-Bench 2, TravelPlanner, VitaBench and WebArena Verified (macro-average of the five sources); All-at-once protocol: the whole execution log in one attribution prompt; invalid outputs count as errors)): leader Kimi K3 (54.5), 5 models
Responsible-role accuracy on LongRCA-Mini (2026), 200 human-annotated failed agent trajectories from five benchmarks: the model reads the whole execution log in one prompt and names the agent role that took the root-cause step. Percentage correct, macro-averaged over sources; higher is better.
KnowHal (Overall score (%; mean of the positive-question and negative-question average accuracies; 1,800 entity-image samples in 10 domains with paired positive and false-premise negative questions; positive answers scored by standard VQA answer normalization and soft matching, negative answers by a GPT-4o-mini judge (option matching for relation questions); zero-shot)): leader Gemini 3 Flash (Preview) (67.9), 14 models
KnowHal (Shandong University, 2026) evaluates hallucination in multimodal LLMs with paired questions over 1,800 entity images in 10 domains: a positive question about evidence that is present and a negative question carrying a false premise, across entity, attribute, relation and knowledge dimensions. The overall score is the mean of positive and negative accuracy in percent; higher is better.
KnowHal - Positive Questions (Accuracy (%; mean over the entity, attribute, relation and knowledge dimensions on questions about valid visual evidence or facts; 1,800 entity-image samples in 10 domains with paired positive and false-premise negative questions; positive answers scored by standard VQA answer normalization and soft matching, negative answers by a GPT-4o-mini judge (option matching for relation questions); zero-shot)): leader Gemini 3 Flash (Preview) (86.6), 14 models
Accuracy of multimodal LLMs on the positive questions of KnowHal (Shandong University, 2026), which ask about visual evidence or factual knowledge that is actually present for the depicted entity, averaged over the entity, attribute, relation and knowledge dimensions. Answers are scored by standard VQA answer matching; percent, higher is better.
KnowHal - Negative Questions (Accuracy (%; mean over the entity, attribute, relation and knowledge dimensions on questions carrying a misleading premise the model must reject; 1,800 entity-image samples in 10 domains with paired positive and false-premise negative questions; positive answers scored by standard VQA answer normalization and soft matching, negative answers by a GPT-4o-mini judge (option matching for relation questions); zero-shot)): leader Qwen-VL-Max (KnowHal checkpoint unspecified) (57.6), 14 models
Accuracy on the negative questions of KnowHal (Shandong University, 2026), which embed a misleading premise, such as a substituted attribute or object, a false spatial relation or a fabricated action, that the model must reject. Averaged over the four dimensions and scored by a GPT-4o-mini judge or option matching; percent, higher is better.
KnowHal - Entity (Positive) (Accuracy (%; entity-dimension positive questions; 1,800 entity-image samples in 10 domains with paired positive and false-premise negative questions; positive answers scored by standard VQA answer normalization and soft matching, negative answers by a GPT-4o-mini judge (option matching for relation questions); zero-shot)): leader Seed 2.0 Mini (93.0), 14 models
The entity dimension of KnowHal (Shandong University, 2026), positive questions: recognizing the entity shown in the image. Accuracy in percent; higher is better.
KnowHal - Attribute (Positive) (Accuracy (%; attribute-dimension positive questions; 1,800 entity-image samples in 10 domains with paired positive and false-premise negative questions; positive answers scored by standard VQA answer normalization and soft matching, negative answers by a GPT-4o-mini judge (option matching for relation questions); zero-shot)): leader Seed 2.0 Mini (91.8), 14 models
The attribute dimension of KnowHal (Shandong University, 2026), positive questions: fine-grained visual properties of the entity such as color and shape. Accuracy in percent; higher is better.
KnowHal - Relation (Positive) (Accuracy (%; relation-dimension positive questions; 1,800 entity-image samples in 10 domains with paired positive and false-premise negative questions; positive answers scored by standard VQA answer normalization and soft matching, negative answers by a GPT-4o-mini judge (option matching for relation questions); zero-shot)): leader Gemini 3 Flash (Preview) (89.6), 14 models
The relation dimension of KnowHal (Shandong University, 2026), positive questions: spatial and behavioral relations among entities, asked as multiple choice. Accuracy in percent; higher is better.
KnowHal - Knowledge (Positive) (Accuracy (%; knowledge-dimension positive questions; 1,800 entity-image samples in 10 domains with paired positive and false-premise negative questions; positive answers scored by standard VQA answer normalization and soft matching, negative answers by a GPT-4o-mini judge (option matching for relation questions); zero-shot)): leader Gemini 3 Flash (Preview) (79.2), 14 models
The knowledge dimension of KnowHal (Shandong University, 2026), positive questions: external world knowledge about the depicted entity beyond what is visible. Accuracy in percent; higher is better.
KnowHal - Entity (Negative) (Accuracy (%; entity-dimension negative (false-premise) questions; 1,800 entity-image samples in 10 domains with paired positive and false-premise negative questions; positive answers scored by standard VQA answer normalization and soft matching, negative answers by a GPT-4o-mini judge (option matching for relation questions); zero-shot)): leader Qwen 3 VL 32B Instruct (52.9), 14 models
The entity dimension of KnowHal (Shandong University, 2026), negative questions: recognizing the entity shown in the image, asked with a false premise the model must reject. Accuracy in percent; higher is better.
KnowHal - Attribute (Negative) (Accuracy (%; attribute-dimension negative (false-premise) questions; 1,800 entity-image samples in 10 domains with paired positive and false-premise negative questions; positive answers scored by standard VQA answer normalization and soft matching, negative answers by a GPT-4o-mini judge (option matching for relation questions); zero-shot)): leader Qwen-VL-Max (KnowHal checkpoint unspecified) (74.7), 14 models
The attribute dimension of KnowHal (Shandong University, 2026), negative questions: fine-grained visual properties of the entity such as color and shape, asked with a false premise the model must reject. Accuracy in percent; higher is better.
KnowHal - Relation (Negative) (Accuracy (%; relation-dimension negative (false-premise) questions; 1,800 entity-image samples in 10 domains with paired positive and false-premise negative questions; positive answers scored by standard VQA answer normalization and soft matching, negative answers by a GPT-4o-mini judge (option matching for relation questions); zero-shot)): leader Qwen 2.5 VL 72B Instruct (73.9), 14 models
The relation dimension of KnowHal (Shandong University, 2026), negative questions: spatial and behavioral relations among entities, asked as multiple choice, asked with a false premise the model must reject. Accuracy in percent; higher is better.
KnowHal - Knowledge (Negative) (Accuracy (%; knowledge-dimension negative (false-premise) questions; 1,800 entity-image samples in 10 domains with paired positive and false-premise negative questions; positive answers scored by standard VQA answer normalization and soft matching, negative answers by a GPT-4o-mini judge (option matching for relation questions); zero-shot)): leader Qwen-VL-Max (KnowHal checkpoint unspecified) (37.7), 14 models
The knowledge dimension of KnowHal (Shandong University, 2026), negative questions: external world knowledge about the depicted entity beyond what is visible, asked with a false premise the model must reject. Accuracy in percent; higher is better.
DataSpace (Task accuracy (%; share of the 410 DataSpace tasks, each a question over a task-local heterogeneous workspace of CSV, JSON, SQLite, Markdown, PDF and video artifacts, whose returned table matches the gold table under the deterministic evaluator: header-invariant column alignment, type- and precision-aware normalization and order-aware row comparison; DataSpace-Agent reference harness unless the row names another harness)): leader Grok 4.5 (66.34), 10 models
DataSpace (Li et al., 2026) asks a data agent to answer analytics questions over task-local workspaces that mix databases, CSV and JSON files, long documents and videos, returning the complete result table. A task counts as solved only when a deterministic evaluator matches the returned table to the gold table after header-invariant column alignment and type- and precision-aware normalization; the score is the percentage of 410 tasks solved, higher is better. Rows use the DataSpace-Agent reference harness unless the label names another agent harness.
FinPerMA (Accuracy (%; share of the 2,494 six-option multiple-choice questions answered with the gold option and the 500 open-ended questions judged correct by majority vote of a three-judge cross-vendor ensemble; full_context setting: the model receives every prior dialogue turn of the investor persona verbatim; 2,994 questions from 276 synthetic investor personas with frozen event timelines, including Post-Shock checkpoints after material events; provider-default system prompt, single seed)): leader FinPerMA Qwen3.8 (checkpoint unspecified) (46.9), 7 models
FinPerMA (Wang et al., 2026) tests whether an LLM agent keeps and updates a personalized model of an investor over long, event-driven advisory dialogues, including checkpoints after material events that should change the preferences of the investor. Accuracy in percent over 2,994 questions, 2,494 six-option multiple-choice and 500 open-ended ones judged by a three-judge ensemble, with the full dialogue history supplied; higher is better.
FinPerMA - Multiple Choice (Accuracy (%; share of the 2,494 six-option multiple-choice questions whose selected option content matches the gold option, scored deterministically; chance 16.7%; full_context setting: the model receives every prior dialogue turn of the investor persona verbatim; 2,994 questions from 276 synthetic investor personas with frozen event timelines, including Post-Shock checkpoints after material events; provider-default system prompt, single seed)): leader Claude Opus 4.8 (39.1), 7 models
Part of FinPerMA (Wang et al., 2026): accuracy in percent on 2,494 six-option multiple-choice questions about a synthetic investor persona, answered with the full advisory dialogue history supplied and scored deterministically. Higher is better; chance is about 17 percent.
UA-JudgeExam (Full-condition accuracy (%; question and four options shown, one letter answered, on a 400-item sample of the 8,128 items that remain after the Claude Haiku 4.5 option-only gate; denominators exclude unparseable answers (DeepSeek R1 352, Pixtral Large 365, Nova Micro 388, Llama 3.1 8B 202); greedy decoding where the provider accepts a temperature; chance is 25)): leader GPT-5.6 Sol (95.8), 13 models
UA-JudgeExam (2026) is built from the question bank the Ukrainian High Qualification Commission of Judges publishes for appellate-judge candidates: four-option legal items with official keys, filtered to remove items a model can answer from the options alone. Score is multiple-choice accuracy with the question shown, on a 400-item sample of the filtered set, in percent; chance is 25 and higher is better.
AffectSim (Initial View) (Macro-F1 (%; P-Init: the agent's initial camera pose, held fixed; five-class emotion recognition (angry, fearful, happy, neutral, sad) on the 3,941 test episodes of AffectSim, full-body affective motions replayed in Habitat indoor scenes; frozen recognizer, eight uniformly sampled frames for most open models, full video or eight frames for closed models)): leader Claude Opus 5 (25.46), 24 models
AffectSim (2026) tests emotion recognition from body movement in simulated homes: emotion-labelled full-body motions are replayed in Habitat indoor scenes and a frozen vision-language model classifies each of 3,941 test episodes as angry, fearful, happy, neutral or sad. The observation is the agent's initial camera pose, held fixed. Score is macro-F1 in percent; higher is better.
AffectSim (Reference View) (Macro-F1 (%; P-Ref: an informative camera trajectory chosen by a privileged reference observer from body pose and scene geometry; five-class emotion recognition (angry, fearful, happy, neutral, sad) on the 3,941 test episodes of AffectSim, full-body affective motions replayed in Habitat indoor scenes; frozen recognizer, eight uniformly sampled frames for most open models, full video or eight frames for closed models)): leader Claude Opus 5 (35.68), 24 models
AffectSim (2026) tests emotion recognition from body movement in simulated homes: emotion-labelled full-body motions are replayed in Habitat indoor scenes and a frozen vision-language model classifies each of 3,941 test episodes as angry, fearful, happy, neutral or sad. The observation is an informative camera trajectory chosen by a privileged reference observer from body pose and scene geometry. Score is macro-F1 in percent; higher is better.
AffectSim (Monitor View) (Macro-F1 (%; P-Mon: a fixed room-level monitoring camera; five-class emotion recognition (angry, fearful, happy, neutral, sad) on the 3,941 test episodes of AffectSim, full-body affective motions replayed in Habitat indoor scenes; frozen recognizer, eight uniformly sampled frames for most open models, full video or eight frames for closed models)): leader Claude Opus 5 (29.27), 24 models
AffectSim (2026) tests emotion recognition from body movement in simulated homes: emotion-labelled full-body motions are replayed in Habitat indoor scenes and a frozen vision-language model classifies each of 3,941 test episodes as angry, fearful, happy, neutral or sad. The observation is a fixed room-level monitoring camera. Score is macro-F1 in percent; higher is better.
AffectSim (Active Observation) (Macro-F1 (%; A-Obs: the view reached by the authors' two-stage active-observation baseline (person search, then target following), falling back to P-Init when no new view is acquired; five-class emotion recognition (angry, fearful, happy, neutral, sad) on the 3,941 test episodes of AffectSim, full-body affective motions replayed in Habitat indoor scenes; frozen recognizer, eight uniformly sampled frames for most open models, full video or eight frames for closed models)): leader Claude Opus 5 (28.37), 24 models
AffectSim (2026) tests emotion recognition from body movement in simulated homes: emotion-labelled full-body motions are replayed in Habitat indoor scenes and a frozen vision-language model classifies each of 3,941 test episodes as angry, fearful, happy, neutral or sad. The observation is the view reached by the authors' two-stage active-observation baseline (person search, then target following), falling back to P-Init when no new view is acquired. Score is macro-F1 in percent; higher is better.
CodeAssay (Final hidden-test correctness (%; share of 185 audited Python tasks in ten software-engineering categories whose scored solution passes every hidden test; one generation from the description, signature and a public-test excerpt plus at most one repair guided by public-test failures; provider-default reasoning, temperature 0 where accepted, 16,384-token output limit; standard system prompt)): leader GPT-5.6 Sol (98.9), 7 models
CodeAssay (2026) is a Python code-generation benchmark of 185 audited tasks in ten software-engineering categories. A model gets the description, signature and public-test excerpt, may repair once after public-test failures, and is graded on disjoint hidden tests; the score is the percent of tasks passed, higher is better. This board uses the standard system prompt.
CodeAssay (Security-Focused Prompt) (Final hidden-test correctness (%; share of 185 audited Python tasks in ten software-engineering categories whose scored solution passes every hidden test; one generation from the description, signature and a public-test excerpt plus at most one repair guided by public-test failures; provider-default reasoning, temperature 0 where accepted, 16,384-token output limit; security-focused system prompt that adds input validation, no eval/exec, secrets-module randomness, sanitized external data and error handling that exposes no internals)): leader GPT-5.6 Sol (98.4), 7 models
CodeAssay (2026) hidden-test correctness on 185 audited Python tasks when the system prompt asks for security-conscious code (input validation, no eval or exec, secure randomness, sanitized external data). Percent of tasks passed after at most one public-test-guided repair; higher is better.
GMA (Success rate (%; a task succeeds only when all its subtasks are satisfied; 300 tasks in seven custom Android applications, vision-only agent seeing raw screenshots (two most recent kept), MobileWorld action space plus call_user and answer, Qwen3.7-Plus user simulator, 150-step budget; all four difficulty tiers)): leader Qwen 3.8 Max (67.33), 8 models
GMA (General Mobile Assistants, 2026) evaluates vision-only mobile GUI agents on 300 tasks across seven custom Android applications, from atomic actions to realistic cross-application workflows. Strict success rate in percent, a task counting only when all its verified subtasks are complete; higher is better.
GMA - Subtask Completion (Average score (%; share of each task's programmatically verified subtasks completed, averaged over tasks; 300 tasks in seven custom Android applications, vision-only agent seeing raw screenshots (two most recent kept), MobileWorld action space plus call_user and answer, Qwen3.7-Plus user simulator, 150-step budget; all four difficulty tiers)): leader Qwen 3.8 Max (80.35), 8 models
Average share of programmatically verified subtasks a mobile GUI agent completes per task on the 300 GMA tasks across seven custom Android applications, in percent; higher is better.
GMA - Atomic (Success rate (%; 300 tasks in seven custom Android applications, vision-only agent seeing raw screenshots (two most recent kept), MobileWorld action space plus call_user and answer, Qwen3.7-Plus user simulator, 150-step budget; 73 atomic single-action tasks)): leader Qwen 3.8 Max (93.15), 8 models
Strict success rate of mobile GUI agents on the 73 atomic single-action tasks of GMA (2026), seven custom Android applications under a vision-only agent; percent, higher is better.
GMA - Compositional (Success rate (%; 300 tasks in seven custom Android applications, vision-only agent seeing raw screenshots (two most recent kept), MobileWorld action space plus call_user and answer, Qwen3.7-Plus user simulator, 150-step budget; 129 compositional multi-step tasks in one application)): leader Qwen 3.8 Max (70.54), 8 models
Strict success rate of mobile GUI agents on the 129 compositional multi-step tasks within one application of GMA (2026), seven custom Android applications under a vision-only agent; percent, higher is better.
GMA - Cross-Application (Success rate (%; 300 tasks in seven custom Android applications, vision-only agent seeing raw screenshots (two most recent kept), MobileWorld action space plus call_user and answer, Qwen3.7-Plus user simulator, 150-step budget; 78 tasks with cross-application dependencies)): leader Qwen 3.8 Max (52.56), 8 models
Strict success rate of mobile GUI agents on the 78 tasks with dependencies across applications of GMA (2026), seven custom Android applications under a vision-only agent; percent, higher is better.
GMA - Realistic (Success rate (%; 300 tasks in seven custom Android applications, vision-only agent seeing raw screenshots (two most recent kept), MobileWorld action space plus call_user and answer, Qwen3.7-Plus user simulator, 150-step budget; 20 realistic underspecified workflows over four to seven applications with distractors)): leader Seed 2.1 Pro (20.0), 8 models
Strict success rate of mobile GUI agents on the 20 realistic, underspecified workflows spanning four to seven applications of GMA (2026), seven custom Android applications under a vision-only agent; percent, higher is better.
SciFigQual-Bench (Mean absolute error of the overall figure-quality score against the mean expert score (1-10 rating scale, lower is better; 1,200 figures from 254 ACL, EMNLP, ICML and NeurIPS papers, each rated with its caption and citing paragraphs; one direct VLM call per figure)): leader Claude Opus 4.8 (0.443), 11 models
SciFigQual-Bench (University of Sydney and University of Hong Kong, 2026) asks vision-language models to rate published computer-science paper figures for visual clarity, layout, caption consistency, context consistency and misleading risk, reading each figure with its caption and citing paragraphs. Mean absolute error of the overall 1 to 10 score against the mean expert rating on 1,200 figures, one direct call per figure; lower is better.
MerchantBench (ReAct) (Final net assets (thousand RMB; 365-day order-level simulation of a drop-shipping store grounded in 98,843 real product records, 26 tools for sourcing, listing, pricing and cash flow; the store starts with RMB 2,000 cash and a RMB 1,000 deposit; mean of three runs; the authors minimal ReAct controller over the 26 tools)): leader GPT-5.6 Sol (40.89), 8 models
MerchantBench (Alibaba and Zhejiang University, 2026) tests long-term coherence by having an LLM agent run a drop-shipping online store for 365 simulated days, sourcing products from real catalog records, pricing listings, managing cash flow and reacting to delayed order outcomes. The agent uses a minimal ReAct controller over the 26 store tools; final net assets in thousands of RMB, mean of three runs, higher is better.
CAPA (Executable success (%; share of the 300 held-out coding sessions solved within an eight-turn budget, a submission counting when the execution-grounded judge accepts it from the hidden tests and the reference solution, and a simulated user agent answering clarification questions; same-user history: the model also sees the dialogue traces of five resolved sessions from the same user, whose recurring ambiguity pattern the held-out request repeats)): leader Claude Opus 4.8 (90.0), 12 models
CAPA tests whether a coding assistant learns the recurring ambiguity of one user: each held-out session opens with an ambiguous coding request whose missing detail the same user has resolved the same way before, and the assistant may ask for clarification or submit code. This board is the share of sessions solved within eight turns when the assistant is also given five resolved sessions from the same user. A simulated user answers clarifying questions and an execution-grounded judge decides whether a submission is correct.
CAPA - First-Turn Success (First-turn executable success (%; share of the 300 held-out coding sessions whose first assistant turn is an accepted submission, with no clarification asked; same-user history: the model also sees the dialogue traces of five resolved sessions from the same user, whose recurring ambiguity pattern the held-out request repeats)): leader Claude Opus 4.8 (60.3), 12 models
CAPA tests whether a coding assistant learns the recurring ambiguity of one user: each held-out session opens with an ambiguous coding request whose missing detail the same user has resolved the same way before, and the assistant may ask for clarification or submit code. This board is the share of sessions solved at the first turn, with no clarification asked, when the assistant is also given five resolved sessions from the same user. A simulated user answers clarifying questions and an execution-grounded judge decides whether a submission is correct.
CAPA (No History) (Executable success (%; share of the 300 held-out coding sessions solved within an eight-turn budget, a submission counting when the execution-grounded judge accepts it from the hidden tests and the reference solution, and a simulated user agent answering clarification questions; no history: the model sees only the new ambiguous request)): leader Claude Opus 4.8 (88.0), 12 models
CAPA tests whether a coding assistant learns the recurring ambiguity of one user: each held-out session opens with an ambiguous coding request whose missing detail the same user has resolved the same way before, and the assistant may ask for clarification or submit code. This board is the share of sessions solved within eight turns when the assistant sees only the new request. A simulated user answers clarifying questions and an execution-grounded judge decides whether a submission is correct.
CAPA (No History) - First-Turn Success (First-turn executable success (%; share of the 300 held-out coding sessions whose first assistant turn is an accepted submission, with no clarification asked; no history: the model sees only the new ambiguous request)): leader Claude Opus 4.8 (24.3), 12 models
CAPA tests whether a coding assistant learns the recurring ambiguity of one user: each held-out session opens with an ambiguous coding request whose missing detail the same user has resolved the same way before, and the assistant may ask for clarification or submit code. This board is the share of sessions solved at the first turn, with no clarification asked, when the assistant sees only the new request. A simulated user answers clarifying questions and an execution-grounded judge decides whether a submission is correct.
MemeBench (Success rate (%; visual, identity, knowledge and reasoning checklists all passed) on 1,253 Chinese and English memes centred on anime, comics, games and online subcultures, explained closed-book from the image; scored against VIKR checklists under the intersection of Gemini-3.1-Pro and GPT-5.1 judges, unscored responses counted as failures): leader Gemini 3.1 Pro (Preview) (60.3), 26 models
MemeBench (2026) asks vision-language models to explain 1,253 Chinese and English memes about anime, comics, games and online subcultures from the image alone, checked against VIKR checklists of visual clues, identity links, knowledge units and reasoning under two judges. Score is the success rate, the share of memes whose explanation passes all four checklists, in percent; higher is better.
MemeBench - Visual Clues (Visual-clue coverage (%) on 1,253 Chinese and English memes centred on anime, comics, games and online subcultures, explained closed-book from the image; scored against VIKR checklists under the intersection of Gemini-3.1-Pro and GPT-5.1 judges, unscored responses counted as failures): leader Gemini 3.1 Pro (Preview) (92.3), 26 models
MemeBench (2026) asks vision-language models to explain 1,253 Chinese and English memes about anime, comics, games and online subcultures from the image alone, checked against VIKR checklists of visual clues, identity links, knowledge units and reasoning under two judges. This board is the coverage of the visual-clue checklist items, in percent; higher is better.
MemeBench - Identity Links (Identity-link coverage (%) on 1,253 Chinese and English memes centred on anime, comics, games and online subcultures, explained closed-book from the image; scored against VIKR checklists under the intersection of Gemini-3.1-Pro and GPT-5.1 judges, unscored responses counted as failures): leader Gemini 3.1 Pro (Preview) (74.6), 26 models
MemeBench (2026) asks vision-language models to explain 1,253 Chinese and English memes about anime, comics, games and online subcultures from the image alone, checked against VIKR checklists of visual clues, identity links, knowledge units and reasoning under two judges. This board is the coverage of the identity-link checklist items, naming the characters, works and people a meme relies on, in percent; higher is better.
MemeBench - Knowledge Units (Knowledge-unit coverage (%) on 1,253 Chinese and English memes centred on anime, comics, games and online subcultures, explained closed-book from the image; scored against VIKR checklists under the intersection of Gemini-3.1-Pro and GPT-5.1 judges, unscored responses counted as failures): leader Gemini 3.1 Pro (Preview) (69.7), 26 models
MemeBench (2026) asks vision-language models to explain 1,253 Chinese and English memes about anime, comics, games and online subcultures from the image alone, checked against VIKR checklists of visual clues, identity links, knowledge units and reasoning under two judges. This board is the coverage of the background-knowledge checklist items, in percent; higher is better.
MemeBench - Reasoning (Reasoning-mechanism coverage (%) on 1,253 Chinese and English memes centred on anime, comics, games and online subcultures, explained closed-book from the image; scored against VIKR checklists under the intersection of Gemini-3.1-Pro and GPT-5.1 judges, unscored responses counted as failures): leader Gemini 3.1 Pro (Preview) (76.7), 26 models
MemeBench (2026) asks vision-language models to explain 1,253 Chinese and English memes about anime, comics, games and online subcultures from the image alone, checked against VIKR checklists of visual clues, identity links, knowledge units and reasoning under two judges. This board is the coverage of the reasoning-mechanism checklist items, in percent; higher is better.
MemeBench (Chinese) (Success rate (%) on the 768 Chinese memes of 1,253 Chinese and English memes centred on anime, comics, games and online subcultures, explained closed-book from the image; scored against VIKR checklists under the intersection of Gemini-3.1-Pro and GPT-5.1 judges, unscored responses counted as failures): leader Gemini 3.1 Pro (Preview) (49.9), 26 models
MemeBench (2026) asks vision-language models to explain 1,253 Chinese and English memes about anime, comics, games and online subcultures from the image alone, checked against VIKR checklists of visual clues, identity links, knowledge units and reasoning under two judges. This board is the success rate on the 768 Chinese memes, in percent; higher is better.
MemeBench (English) (Success rate (%) on the 485 English memes of 1,253 Chinese and English memes centred on anime, comics, games and online subcultures, explained closed-book from the image; scored against VIKR checklists under the intersection of Gemini-3.1-Pro and GPT-5.1 judges, unscored responses counted as failures): leader Gemini 3.1 Pro (Preview) (76.7), 26 models
MemeBench (2026) asks vision-language models to explain 1,253 Chinese and English memes about anime, comics, games and online subcultures from the image alone, checked against VIKR checklists of visual clues, identity links, knowledge units and reasoning under two judges. This board is the success rate on the 485 English memes, in percent; higher is better.
HiEviDR-Bench (RAG) (Overall score (0-100): sum of five 0-20 dimension scores (multimodal report quality, evidence traceability, and progressively gated citation, claim and answer scores) judged by Qwen3-VL-235B-A22B-Instruct against the hierarchical evidence graph, averaged over the Wikipedia and arXiv subsets; MLLM with RAG: one-pass report generation over the top-15 retrieved multimodal evidence items; higher is better): leader GPT-5.6 Sol (38.35), 16 models
HiEviDR-Bench (UCAS, Tsinghua and others, 2026) tests multimodal deep-research reports on its Wikipedia and arXiv question sets under the one-pass retrieval-augmented pipeline. A Qwen3-VL-235B-A22B judge scores each report against a hierarchical evidence graph on five 0-20 dimensions (report quality, evidence traceability, and gated citation, claim and answer scores); the score is the mean over the Wikipedia and arXiv sets of that 0 to 100 sum, higher is better.
HiEviDR-Bench - Wiki (RAG) (Overall score (0-100): sum of five 0-20 dimension scores (multimodal report quality, evidence traceability, and progressively gated citation, claim and answer scores) judged by Qwen3-VL-235B-A22B-Instruct against the hierarchical evidence graph, on the 1,000-question Wikipedia subset; MLLM with RAG: one-pass report generation over the top-15 retrieved multimodal evidence items; higher is better): leader GPT-5.6 Sol (42.67), 16 models
HiEviDR-Bench (UCAS, Tsinghua and others, 2026) tests multimodal deep-research reports on 1,000 Wikipedia questions under the one-pass retrieval-augmented pipeline. A Qwen3-VL-235B-A22B judge scores each report against a hierarchical evidence graph on five 0-20 dimensions (report quality, evidence traceability, and gated citation, claim and answer scores); the score is their sum from 0 to 100, higher is better.
HiEviDR-Bench - Arxiv (RAG) (Overall score (0-100): sum of five 0-20 dimension scores (multimodal report quality, evidence traceability, and progressively gated citation, claim and answer scores) judged by Qwen3-VL-235B-A22B-Instruct against the hierarchical evidence graph, on the 1,000-question arXiv subset; MLLM with RAG: one-pass report generation over the top-15 retrieved multimodal evidence items; higher is better): leader GPT-5.6 Sol (34.03), 16 models
HiEviDR-Bench (UCAS, Tsinghua and others, 2026) tests multimodal deep-research reports on 1,000 arXiv paper questions under the one-pass retrieval-augmented pipeline. A Qwen3-VL-235B-A22B judge scores each report against a hierarchical evidence graph on five 0-20 dimensions (report quality, evidence traceability, and gated citation, claim and answer scores); the score is their sum from 0 to 100, higher is better.
HiEviDR-Bench (Deep Research) (Overall score (0-100): sum of five 0-20 dimension scores (multimodal report quality, evidence traceability, and progressively gated citation, claim and answer scores) judged by Qwen3-VL-235B-A22B-Instruct against the hierarchical evidence graph, averaged over the Wikipedia and arXiv subsets; MLLM with Deep Research: up to three rounds of question refinement with top-20 retrieval, re-ranked to 25 text chunks and 5 images; higher is better): leader Grok 4.5 (41.18), 16 models
HiEviDR-Bench (UCAS, Tsinghua and others, 2026) tests multimodal deep-research reports on its Wikipedia and arXiv question sets under the iterative deep-research pipeline. A Qwen3-VL-235B-A22B judge scores each report against a hierarchical evidence graph on five 0-20 dimensions (report quality, evidence traceability, and gated citation, claim and answer scores); the score is the mean over the Wikipedia and arXiv sets of that 0 to 100 sum, higher is better.
HiEviDR-Bench - Wiki (Deep Research) (Overall score (0-100): sum of five 0-20 dimension scores (multimodal report quality, evidence traceability, and progressively gated citation, claim and answer scores) judged by Qwen3-VL-235B-A22B-Instruct against the hierarchical evidence graph, on the 1,000-question Wikipedia subset; MLLM with Deep Research: up to three rounds of question refinement with top-20 retrieval, re-ranked to 25 text chunks and 5 images; higher is better): leader Grok 4.5 (46.41), 16 models
HiEviDR-Bench (UCAS, Tsinghua and others, 2026) tests multimodal deep-research reports on 1,000 Wikipedia questions under the iterative deep-research pipeline. A Qwen3-VL-235B-A22B judge scores each report against a hierarchical evidence graph on five 0-20 dimensions (report quality, evidence traceability, and gated citation, claim and answer scores); the score is their sum from 0 to 100, higher is better.
HiEviDR-Bench - Arxiv (Deep Research) (Overall score (0-100): sum of five 0-20 dimension scores (multimodal report quality, evidence traceability, and progressively gated citation, claim and answer scores) judged by Qwen3-VL-235B-A22B-Instruct against the hierarchical evidence graph, on the 1,000-question arXiv subset; MLLM with Deep Research: up to three rounds of question refinement with top-20 retrieval, re-ranked to 25 text chunks and 5 images; higher is better): leader Grok 4.5 (35.95), 16 models
HiEviDR-Bench (UCAS, Tsinghua and others, 2026) tests multimodal deep-research reports on 1,000 arXiv paper questions under the iterative deep-research pipeline. A Qwen3-VL-235B-A22B judge scores each report against a hierarchical evidence graph on five 0-20 dimensions (report quality, evidence traceability, and gated citation, claim and answer scores); the score is their sum from 0 to 100, higher is better.
ClawArena-Team (Subagent-Management Score (%): task correctness multiplied by a least-privilege and modality-routing management factor in [0, 1]; 41 multi-turn multimodal scenarios with 258 evaluation rounds; a text-only main agent with partial workspace access creates, empowers and schedules a fixed locally served pool of LLM, VLM and omni subagents; execution-based checks, no LLM judge; higher is better): leader Claude Fable 5 (60.0), 12 models
ClawArena-Team (2026) measures how well one language model manages a team of subagents: over 41 multi-turn multimodal scenarios, a text-only main agent must create, empower and schedule a fixed local pool of text, vision and audio subagents, with execution-based checks and no LLM judge. The board is the Subagent-Management Score in percent, task correctness multiplied by a least-privilege and modality-routing factor; higher is better.
ClawArena-Team - Task Completion (Task completion rate (%): mean pass rate over user questions; 41 multi-turn multimodal scenarios with 258 evaluation rounds; a text-only main agent with partial workspace access creates, empowers and schedules a fixed locally served pool of LLM, VLM and omni subagents; execution-based checks, no LLM judge; higher is better): leader Claude Fable 5 (74.4), 12 models
ClawArena-Team (2026) measures how well one language model manages a team of subagents: over 41 multi-turn multimodal scenarios, a text-only main agent must create, empower and schedule a fixed local pool of text, vision and audio subagents, with execution-based checks and no LLM judge. The board is the task completion rate, the mean pass rate over user questions in percent; higher is better.
ClawArena-Team - Tool-Permission Precision (Tool-permission precision (%): per subagent, the share of granted tool types actually used, averaged; 41 multi-turn multimodal scenarios with 258 evaluation rounds; a text-only main agent with partial workspace access creates, empowers and schedules a fixed locally served pool of LLM, VLM and omni subagents; execution-based checks, no LLM judge; higher is better): leader GPT-5.5 (79.9), 12 models
ClawArena-Team (2026) measures how well one language model manages a team of subagents: over 41 multi-turn multimodal scenarios, a text-only main agent must create, empower and schedule a fixed local pool of text, vision and audio subagents, with execution-based checks and no LLM judge. The board is tool-permission precision, the share of the tool types granted to each subagent that it actually uses, in percent; higher is better.
ClawArena-Team - Workspace-Permission Precision (Workspace-permission precision (%): per subagent, files accessed divided by files granted, averaged; 41 multi-turn multimodal scenarios with 258 evaluation rounds; a text-only main agent with partial workspace access creates, empowers and schedules a fixed locally served pool of LLM, VLM and omni subagents; execution-based checks, no LLM judge; higher is better): leader Claude Fable 5 (49.2), 12 models
ClawArena-Team (2026) measures how well one language model manages a team of subagents: over 41 multi-turn multimodal scenarios, a text-only main agent must create, empower and schedule a fixed local pool of text, vision and audio subagents, with execution-based checks and no LLM judge. The board is workspace-permission precision, the files each subagent accesses divided by the files it was granted, in percent; higher is better.
Plan2Map (Mean IoU (%) between the predicted and the held-out reference boundary polygons, times 100, direct VLM-to-GeoJSON prediction from the planning document with no tool access, provider default temperature 1, on the paper 40-case stratified subset of the 208 UK planning records; higher is better): leader Gemini 3.1 Pro (Preview) (11.2), 4 models
Plan2Map (2026) tests document-grounded geospatial reconstruction: from a UK planning record alone (notice text, schedules, map plates and annotations) a model must produce the restricted area as a GeoJSON boundary. Scored by mean IoU with the held-out reference polygon times 100; higher is better. Frontier vision-language models predict GeoJSON directly from the planning document without tools, on a 40-case stratified subset.
Plan2Map - Any Overlap (Share of cases (%) whose predicted boundary overlaps the reference at all (IoU above 0), direct VLM-to-GeoJSON prediction from the planning document with no tool access, provider default temperature 1, on the paper 40-case stratified subset of the 208 UK planning records; higher is better): leader GPT-5.5 Pro (50.0), 4 models
The share of Plan2Map (2026) cases in which the predicted planning boundary overlaps the reference polygon at all (IoU above 0), in percent; higher is better. Frontier vision-language models predict GeoJSON directly from the planning document without tools, on a 40-case stratified subset.
Plan2Map - Localization Accuracy (Localization accuracy (%): share of cases whose centroid error is at most 10 percent of the reference boundary Feret diameter (Acc@0.1D), direct VLM-to-GeoJSON prediction from the planning document with no tool access, provider default temperature 1, on the paper 40-case stratified subset of the 208 UK planning records; higher is better): leader GPT-5.5 Pro (10.0), 4 models
Localization accuracy on Plan2Map (2026): the share of cases whose predicted boundary centroid lies within 10 percent of the reference boundary Feret diameter, in percent; higher is better. Frontier vision-language models predict GeoJSON directly from the planning document without tools, on a 40-case stratified subset.
Plan2Map - Centroid Error (Median centroid distance in metres between the predicted and the reference boundary, direct VLM-to-GeoJSON prediction from the planning document with no tool access, provider default temperature 1, on the paper 40-case stratified subset of the 208 UK planning records; lower is better): leader GPT-5.5 Pro (386.0), 4 models
The median distance in metres between the predicted and the reference boundary centroids on Plan2Map (2026), a benchmark for reconstructing planning-record boundaries; lower is better. Frontier vision-language models predict GeoJSON directly from the planning document without tools, on a 40-case stratified subset.
Causal-Plan-Bench (Overall score (0-100): macro-average of the 12 task scores, six four-choice accuracy (%) after answer normalization tasks and six open-ended tasks scored by rubric score (0-100): a blind GPT-5.6-sol judge scores each open-ended answer against a human-verified reference from 0 to 1, times 100; 100 held-out expert-reviewed items per task drawn from egocentric and robot video (EPIC-KITCHENS, Ego4D, HoloAssist and others), the same instruction, visual evidence and answer format for every model, temperature 0.2, mean of three regenerated runs; higher is better): leader GPT-6 (43.04), 14 models
Causal-Plan-Bench (2026) tests physically grounded embodied planning from video: whether a model can judge preconditions, predict action effects, compose multi-step plans and recover from failures, over 1,200 held-out items in 12 tasks. The overall score is the macro-average of six multiple-choice accuracies and six open-ended scores from a GPT-5.6-sol rubric judge, on a 0-100 scale; higher is better.
Causal-Plan-Bench - Executability (Executability score: mean four-choice accuracy (%) after answer normalization over the spatial precondition, affordance precondition and physical feasibility tasks; 100 held-out expert-reviewed items per task drawn from egocentric and robot video (EPIC-KITCHENS, Ego4D, HoloAssist and others), the same instruction, visual evidence and answer format for every model, temperature 0.2, mean of three regenerated runs; higher is better): leader GPT-6 (46.33), 14 models
The executability dimension of Causal-Plan-Bench (2026), the mean of three tasks on whether an action is executable: spatial and affordance preconditions and physical feasibility, as four-choice accuracy (%). Items come from egocentric and robot video and every model sees the same instruction and evidence; higher is better.
Causal-Plan-Bench - Effects (Effects score: mean four-choice accuracy (%) after answer normalization over the affordance visual semantics, spatial postcondition and affordance postcondition tasks; 100 held-out expert-reviewed items per task drawn from egocentric and robot video (EPIC-KITCHENS, Ego4D, HoloAssist and others), the same instruction, visual evidence and answer format for every model, temperature 0.2, mean of three regenerated runs; higher is better): leader GPT-6 (39.67), 14 models
The effects dimension of Causal-Plan-Bench (2026), the mean of three tasks on what an action changes: affordance visual semantics and spatial and functional postconditions, as four-choice accuracy (%). Items come from egocentric and robot video and every model sees the same instruction and evidence; higher is better.
Causal-Plan-Bench - Composition (Composition score: mean rubric score (0-100): a blind GPT-5.6-sol judge scores each open-ended answer against a human-verified reference from 0 to 1, times 100, over the state evolution, strategic rationale and inter-step dependency tasks; 100 held-out expert-reviewed items per task drawn from egocentric and robot video (EPIC-KITCHENS, Ego4D, HoloAssist and others), the same instruction, visual evidence and answer format for every model, temperature 0.2, mean of three regenerated runs; higher is better): leader GPT-6 (42.0), 14 models
The composition dimension of Causal-Plan-Bench (2026), the mean of three tasks on multi-step plan composition: state evolution, strategic rationale and inter-step dependencies, as open-ended answers scored 0-100 by a GPT-5.6-sol rubric judge. Items come from egocentric and robot video and every model sees the same instruction and evidence; higher is better.
Causal-Plan-Bench - Robustness (Robustness score: mean rubric score (0-100): a blind GPT-5.6-sol judge scores each open-ended answer against a human-verified reference from 0 to 1, times 100, over the plan diagnosis and repair, counterfactual outcome and failure recovery tasks; 100 held-out expert-reviewed items per task drawn from egocentric and robot video (EPIC-KITCHENS, Ego4D, HoloAssist and others), the same instruction, visual evidence and answer format for every model, temperature 0.2, mean of three regenerated runs; higher is better): leader GPT-6 (44.17), 14 models
The robustness dimension of Causal-Plan-Bench (2026), the mean of three tasks on robust planning: plan diagnosis and repair, counterfactual outcomes and failure recovery, as open-ended answers scored 0-100 by a GPT-5.6-sol rubric judge. Items come from egocentric and robot video and every model sees the same instruction and evidence; higher is better.
Causal-Plan-Bench - Spatial Precondition (Accuracy (%) on the spatial precondition (conditions that must hold before the action) task (four-choice accuracy (%) after answer normalization); 100 held-out expert-reviewed items per task drawn from egocentric and robot video (EPIC-KITCHENS, Ego4D, HoloAssist and others), the same instruction, visual evidence and answer format for every model, temperature 0.2, mean of three regenerated runs; higher is better): leader RoboBrain-2.5-8B (42.0), 14 models
A four-choice task of Causal-Plan-Bench (2026) on the spatial conditions (proximity, contact, space) that must hold before an observed action starts, 100 held-out video items scored by accuracy (%). Items come from egocentric and robot video and every model sees the same instruction and evidence; higher is better.
Causal-Plan-Bench - Affordance Precondition (Accuracy (%) on the affordance precondition (functional object properties needed before the action) task (four-choice accuracy (%) after answer normalization); 100 held-out expert-reviewed items per task drawn from egocentric and robot video (EPIC-KITCHENS, Ego4D, HoloAssist and others), the same instruction, visual evidence and answer format for every model, temperature 0.2, mean of three regenerated runs; higher is better): leader RynnBrain1.1-9B (48.0), 14 models
A four-choice task of Causal-Plan-Bench (2026) on the functional properties an object needs before an observed action (graspable, openable, cuttable), 100 held-out video items scored by accuracy (%). Items come from egocentric and robot video and every model sees the same instruction and evidence; higher is better.
Causal-Plan-Bench - Physical Feasibility (Accuracy (%) on the physical feasibility (why the action is executable now) task (four-choice accuracy (%) after answer normalization); 100 held-out expert-reviewed items per task drawn from egocentric and robot video (EPIC-KITCHENS, Ego4D, HoloAssist and others), the same instruction, visual evidence and answer format for every model, temperature 0.2, mean of three regenerated runs; higher is better): leader GPT-6 (50.0), 14 models
A four-choice task of Causal-Plan-Bench (2026) on why an observed action is physically executable at that moment, 100 held-out video items scored by accuracy (%). Items come from egocentric and robot video and every model sees the same instruction and evidence; higher is better.
Causal-Plan-Bench - Affordance Visual Semantics (Accuracy (%) on the affordance visual semantics (manipulated object, hotspot and mechanism in a keyframe) task (four-choice accuracy (%) after answer normalization); 100 held-out expert-reviewed items per task drawn from egocentric and robot video (EPIC-KITCHENS, Ego4D, HoloAssist and others), the same instruction, visual evidence and answer format for every model, temperature 0.2, mean of three regenerated runs; higher is better): leader RoboBrain-2.5-8B (35.0), 14 models
A four-choice task of Causal-Plan-Bench (2026) on the manipulated object, interaction hotspot and physical mechanism in a keyframe, 100 held-out video items scored by accuracy (%). Items come from egocentric and robot video and every model sees the same instruction and evidence; higher is better.
Causal-Plan-Bench - Spatial Postcondition (Accuracy (%) on the spatial postcondition (spatial result after the action) task (four-choice accuracy (%) after answer normalization); 100 held-out expert-reviewed items per task drawn from egocentric and robot video (EPIC-KITCHENS, Ego4D, HoloAssist and others), the same instruction, visual evidence and answer format for every model, temperature 0.2, mean of three regenerated runs; higher is better): leader Seed 2.1 Pro (42.0), 14 models
A four-choice task of Causal-Plan-Bench (2026) on the spatial result an action causes (position, contact, support), 100 held-out video items scored by accuracy (%). Items come from egocentric and robot video and every model sees the same instruction and evidence; higher is better.
Causal-Plan-Bench - Affordance Postcondition (Accuracy (%) on the affordance postcondition (functional state change after the action) task (four-choice accuracy (%) after answer normalization); 100 held-out expert-reviewed items per task drawn from egocentric and robot video (EPIC-KITCHENS, Ego4D, HoloAssist and others), the same instruction, visual evidence and answer format for every model, temperature 0.2, mean of three regenerated runs; higher is better): leader GPT-6 (43.0), 14 models
A four-choice task of Causal-Plan-Bench (2026) on the functional state change an action causes, such as closed to open, 100 held-out video items scored by accuracy (%). Items come from egocentric and robot video and every model sees the same instruction and evidence; higher is better.
Causal-Plan-Bench - State Evolution (Open-ended state evolution (the ongoing action and its immediate state change) task, rubric score (0-100): a blind GPT-5.6-sol judge scores each open-ended answer against a human-verified reference from 0 to 1, times 100; 100 held-out expert-reviewed items per task drawn from egocentric and robot video (EPIC-KITCHENS, Ego4D, HoloAssist and others), the same instruction, visual evidence and answer format for every model, temperature 0.2, mean of three regenerated runs; higher is better): leader GPT-6 (42.8), 14 models
An open-ended task of Causal-Plan-Bench (2026): models describe the ongoing action in a keyframe and the immediate state change it causes, over 100 held-out video items; a GPT-5.6-sol rubric judge scores each answer against a human-verified reference on a 0-100 scale. Items come from egocentric and robot video and every model sees the same instruction and evidence; higher is better.
Causal-Plan-Bench - Strategic Rationale (Open-ended strategic rationale (why the step serves the high-level goal) task, rubric score (0-100): a blind GPT-5.6-sol judge scores each open-ended answer against a human-verified reference from 0 to 1, times 100; 100 held-out expert-reviewed items per task drawn from egocentric and robot video (EPIC-KITCHENS, Ego4D, HoloAssist and others), the same instruction, visual evidence and answer format for every model, temperature 0.2, mean of three regenerated runs; higher is better): leader GPT-6 (43.0), 14 models
An open-ended task of Causal-Plan-Bench (2026): models explain why the current step is necessary for the high-level goal, over 100 held-out video items; a GPT-5.6-sol rubric judge scores each answer against a human-verified reference on a 0-100 scale. Items come from egocentric and robot video and every model sees the same instruction and evidence; higher is better.
Causal-Plan-Bench - Inter-Step Dependency (Open-ended inter-step dependency (how one step enables the next) task, rubric score (0-100): a blind GPT-5.6-sol judge scores each open-ended answer against a human-verified reference from 0 to 1, times 100; 100 held-out expert-reviewed items per task drawn from egocentric and robot video (EPIC-KITCHENS, Ego4D, HoloAssist and others), the same instruction, visual evidence and answer format for every model, temperature 0.2, mean of three regenerated runs; higher is better): leader GPT-6 (40.2), 14 models
An open-ended task of Causal-Plan-Bench (2026): models explain how the result of one step satisfies the precondition of the next, over 100 held-out video items; a GPT-5.6-sol rubric judge scores each answer against a human-verified reference on a 0-100 scale. Items come from egocentric and robot video and every model sees the same instruction and evidence; higher is better.
Causal-Plan-Bench - Plan Diagnosis and Repair (Open-ended bad plan diagnosis and repair (find the wrong step and repair the plan) task, rubric score (0-100): a blind GPT-5.6-sol judge scores each open-ended answer against a human-verified reference from 0 to 1, times 100; 100 held-out expert-reviewed items per task drawn from egocentric and robot video (EPIC-KITCHENS, Ego4D, HoloAssist and others), the same instruction, visual evidence and answer format for every model, temperature 0.2, mean of three regenerated runs; higher is better): leader GPT-6 (39.8), 14 models
An open-ended task of Causal-Plan-Bench (2026): models find the wrong step and error type in a plan and repair the step sequence, over 100 held-out video items; a GPT-5.6-sol rubric judge scores each answer against a human-verified reference on a 0-100 scale. Items come from egocentric and robot video and every model sees the same instruction and evidence; higher is better.
Causal-Plan-Bench - Counterfactual Outcome (Open-ended counterfactual outcome (the consequence if a key condition fails) task, rubric score (0-100): a blind GPT-5.6-sol judge scores each open-ended answer against a human-verified reference from 0 to 1, times 100; 100 held-out expert-reviewed items per task drawn from egocentric and robot video (EPIC-KITCHENS, Ego4D, HoloAssist and others), the same instruction, visual evidence and answer format for every model, temperature 0.2, mean of three regenerated runs; higher is better): leader GPT-6 (46.3), 14 models
An open-ended task of Causal-Plan-Bench (2026): models state the direct consequence if a key condition does not hold, over 100 held-out video items; a GPT-5.6-sol rubric judge scores each answer against a human-verified reference on a 0-100 scale. Items come from egocentric and robot video and every model sees the same instruction and evidence; higher is better.
Causal-Plan-Bench - Failure Recovery (Open-ended failure recovery (a strategy to re-establish the conditions after a failure) task, rubric score (0-100): a blind GPT-5.6-sol judge scores each open-ended answer against a human-verified reference from 0 to 1, times 100; 100 held-out expert-reviewed items per task drawn from egocentric and robot video (EPIC-KITCHENS, Ego4D, HoloAssist and others), the same instruction, visual evidence and answer format for every model, temperature 0.2, mean of three regenerated runs; higher is better): leader GPT-6 (46.4), 14 models
An open-ended task of Causal-Plan-Bench (2026): models propose a recovery strategy that re-establishes the conditions a failed step needs, over 100 held-out video items; a GPT-5.6-sol rubric judge scores each answer against a human-verified reference on a 0-100 scale. Items come from egocentric and robot video and every model sees the same instruction and evidence; higher is better.
WorkBench 2026 (Successful task completion (%) on the corrected 2026 release of WorkBench: 690 templated workplace tasks over a sandbox of five databases (calendar, email, website analytics, CRM, project board) and 26 tools, graded by comparing the final sandbox state with the ground truth; one native tool-calling agent loop for every model, up to 20 steps, temperature 0 where allowed; higher is better): leader Claude Fable 5 (97.7), 24 models
WorkBench (Styles et al., 2024; corrected release 2026) evaluates workplace agents on 690 templated tasks in a sandbox of calendar, email, website-analytics, CRM and project-management databases with 26 tools, graded by comparing the final sandbox state with the ground truth. This board is the task completion rate (%) with one native tool-calling agent loop of up to 20 steps; higher is better.
WorkBench 2026 - Side Effects (Side-effect rate (%): share of tasks in which the agent took a wrong, harmful action (for example emailing the wrong person) on the corrected 2026 release of WorkBench: 690 templated workplace tasks over a sandbox of five databases (calendar, email, website analytics, CRM, project board) and 26 tools, graded by comparing the final sandbox state with the ground truth; one native tool-calling agent loop for every model, up to 20 steps, temperature 0 where allowed; lower is better): leader Claude Fable 5 (1.9), 24 models
WorkBench (Styles et al., 2024; corrected release 2026) evaluates workplace agents on 690 templated tasks in a sandbox of calendar, email, website-analytics, CRM and project-management databases with 26 tools, graded by comparing the final sandbox state with the ground truth. This board is the share (%) of tasks in which the agent took a wrong, harmful action such as emailing the wrong person; lower is better.
LoSoNA (Accuracy@3 (%) under the naive (write the next chat message, no instruction to infer norms) prompt condition on 38 curated group-chat scenarios in which other participants demonstrate a hidden local norm and a final elicitor turn forces a revealing reply; three sampled replies per scenario judged for compliance by a fixed Gemini 3.1 Pro Preview judge; accuracy@3 (%) is the share of scenarios where the majority of the three replies comply; higher is better): leader Claude Fable 5 (47.4), 8 models
LoSoNA (2026) tests local social norm adaptation: in 38 curated group chats other participants demonstrate an unstated local norm, and a final turn forces a reply that shows whether the model inferred it, judged by an LLM. This board is accuracy at 3 (%), the share of scenarios where most of three sampled replies comply, under naive prompting with no instruction to infer norms; higher is better.
LoSoNA - Elicitor-Only (Accuracy@3 (%) under the elicitor-only (reply only to the final turn, prior turns as ordinary context) prompt condition on 38 curated group-chat scenarios in which other participants demonstrate a hidden local norm and a final elicitor turn forces a revealing reply; three sampled replies per scenario judged for compliance by a fixed Gemini 3.1 Pro Preview judge; accuracy@3 (%) is the share of scenarios where the majority of the three replies comply; higher is better): leader Gemini 3.1 Pro (Preview) (47.4), 8 models
LoSoNA (2026) tests local social norm adaptation: in 38 curated group chats other participants demonstrate an unstated local norm, and a final turn forces a reply that shows whether the model inferred it, judged by an LLM. This board is accuracy at 3 (%) when the model is told to reply only to the final turn and use earlier turns as ordinary context; higher is better.
LoSoNA - Style Adaptation (Accuracy@3 (%) under the style-adaptation (fit the local context, tone and habits, without naming norms) prompt condition on 38 curated group-chat scenarios in which other participants demonstrate a hidden local norm and a final elicitor turn forces a revealing reply; three sampled replies per scenario judged for compliance by a fixed Gemini 3.1 Pro Preview judge; accuracy@3 (%) is the share of scenarios where the majority of the three replies comply; higher is better): leader Gemini 3.1 Pro (Preview) (47.4), 8 models
LoSoNA (2026) tests local social norm adaptation: in 38 curated group chats other participants demonstrate an unstated local norm, and a final turn forces a reply that shows whether the model inferred it, judged by an LLM. This board is accuracy at 3 (%) when the model is softly told to fit the local context, tone and habits; higher is better.
LoSoNA - Norm-Informed (Accuracy@3 (%) under the norm-informed (told a repeated local pattern or norm may be relevant) prompt condition on 38 curated group-chat scenarios in which other participants demonstrate a hidden local norm and a final elicitor turn forces a revealing reply; three sampled replies per scenario judged for compliance by a fixed Gemini 3.1 Pro Preview judge; accuracy@3 (%) is the share of scenarios where the majority of the three replies comply; higher is better): leader Gemini 3.1 Pro (Preview) (84.2), 8 models
LoSoNA (2026) tests local social norm adaptation: in 38 curated group chats other participants demonstrate an unstated local norm, and a final turn forces a reply that shows whether the model inferred it, judged by an LLM. This board is accuracy at 3 (%) when the model is told a repeated local pattern or norm may be relevant; higher is better.
OpenTumorBoard - Board Simulation (Conclusion alignment (1-5) of a simulated multidisciplinary discussion's consensus with the real tumor board's conclusion, mean over 184 test cases (malformed outputs score 1; judged by Qwen3.8-27B); higher is better): leader Gemini 3.7 Flash (2.78), 14 models
The model simulates a whole tumor-board discussion for a real case and reaches a consensus on therapy, surgery, next steps and trial matching. An LLM judge scores alignment with the real board's conclusion on a 1-5 scale, higher is better (OpenTumorBoard, 2026).
World Agent (Maintenance completion score (%): share of a world's contracted events, structures and probes realized, mean per run over the nine-case common subset (27 runs per model, easy/hard/extreme tiers, PhyAgentOS harness, isolated LLM judge plus programmatic audit); higher is better): leader Qwen 3.8 Max (0902) (81.3), 8 models
Tests whether a language-model agent can keep a delivered interactive world running: over rounds it uses world-interface tools to realize the contracted events, structures and probes of a world specification. Scored by the maintenance completion score, the share of contract items realized, higher is better (World Agent, 2026).
World Agent - Relation (Maintenance relation score (%): share of authored causal, temporal and concurrency relations between realized events that hold, computed by program, mean per run over the nine-case common subset (27 runs per model, PhyAgentOS harness); higher is better): leader Qwen 3.8 Max (0902) (73.8), 8 models
World Agent's maintenance relation score: the share of authored causal, temporal and concurrency relations between realized events that hold, computed by program from the agent's run; higher is better.

New Models (1)

Claude Sonnet 5.5 — Aggregate Rating 1985, #7
- AA-Briefcase: 63.23 (#1/202)
- AutomationBench-AA: 71.33 (#1/198)
- AutomationBench-AA - Strict: 44.75 (#1/198)
- MLCR-AA: 75.0 (#1/93)
- AA-Omniscience Index - Software Engineering (SWE) - Julia: 92.0 (#1/554)
- Harvey LAB-AA - Data Privacy & Cybersecurity: 98.94 (#1/53)
- Harvey LAB-AA - Trusts, Estates & Private Client: 95.8 (#1/53)
- AA-Briefcase - XLSX - Rubric Pass Rate: 64.77 (#1/202)
- AutomationBench-AA - HR - Raw Objective Completion: 79.6 (#1/198)
- AutomationBench-AA - Operations - Raw Objective Completion: 97.05 (#1/198)
- ...and 151 more

Top-10 New Scores (224)

Claude Fable 5 on EnterpriseBench (CoreCraft Agents): 70.3 (#3)
Claude Fable 5 on FrontierSWE V2: 46.96 (#5)
Claude Fable 5 on Vals AI ProofBench: 95.0 (#7)
Claude Fable 5 on Vals AI Tax Agent Bench: 69.82 (#9)
Claude Fable 5.1 on Chatbot Arena (Vision - Chinese): 1325.0 (#24)
Claude Fable 5.1 on FrontierSWE V2: 56.29 (#3)
Claude Fable 5.1 on GeoBench ACW: 4302.89 (#4)
Claude Fable 5.1 on GeoBench Photos: 4259.62 (#2)
Claude Fable 5.1 on HANDBOOK.md Agents: 38.8 (#1)
Claude Fable 5.1 on NonoBench: 76.7 (#22)
Claude Fable 5.1 on SWE-Together: 82.72 (#4)
Claude Fable 5.1 on Vals AI CyberBench: 70.42 (#17)
Claude Opus 5 on FrontierSWE V2: 52.01 (#4)
Claude Opus 5 on SWE-Together: 80.32 (#11)
Claude Opus 5 on Vals AI CyberBench: 65.36 (#26)
Claude Opus 5.5 on Drone-Bench - Follow: 82.8 (#3)
Claude Opus 5.5 on Drone-Bench - Localize: 95.0 (#2)
Claude Opus 5.5 on Drone-Bench - Navigate: 96.7 (#2)
Claude Opus 5.5 on EKI Estonian LLM - Book Recommendations: 79.64 (#10)
Claude Opus 5.5 on EKI Estonian LLM - Idioms: 62.0 (#14)
Claude Opus 5.5 on EKI Estonian LLM - Language Advice: 68.75 (#4)
Claude Opus 5.5 on EKI Estonian LLM - Overall: 63.68 (#11)
Claude Opus 5.5 on EKI Estonian LLM - Propaganda Resistance: 93.24 (#10)
Claude Opus 5.5 on EKI Estonian LLM - Terminology: 56.45 (#8)
Claude Opus 5.5 on EKI Estonian LLM - Trivia: 64.4 (#18)
Claude Opus 5.5 on EKI Estonian LLM - Trivia Hard: 21.25 (#17)
Claude Opus 5.5 on Epoch AI - Apex Agents: 73.5 (#1)
Claude Opus 5.5 on Epoch AI - Critpt: 31.71 (#3)
Claude Opus 5.5 on Epoch AI - Cursorbench: 57.8 (#1)
Claude Opus 5.5 on Epoch AI - Dtbench: 98.93 (#1)
Claude Opus 5.5 on Epoch AI - Frontierswe: 62.33 (#2)
Claude Opus 5.5 on Epoch AI - Lmca: 68.23 (#1)
Claude Opus 5.5 on Epoch AI - Proofbench: 100.0 (#2)
Claude Opus 5.5 on Epoch AI - Scicode: 66.9 (#1)
Claude Opus 5.5 on FrontierSWE V2: 62.33 (#2)
Claude Opus 5.5 on NonoBench: 93.3 (#3)
Claude Opus 5.5 on SWE-Together: 84.33 (#2)
Claude Opus 5.5 on SpacetimeDB LLM Benchmark (C#): 97.4 (#4)
Claude Opus 5.5 on SpacetimeDB LLM Benchmark (TypeScript): 99.5 (#1)
Claude Opus 5.5 on Vals AI CUA-bench: 14.0 (#2)
Claude Opus 5.5 on Vals AI CyberBench: 74.58 (#6)
Claude Sonnet 5.5 on AA CritPt: 31.43 (#5)
Claude Sonnet 5.5 on AA GDP.pdf: 25.8 (#20)
Claude Sonnet 5.5 on AA GDP.pdf - Construction - Criterion Pass Rate: 69.11 (#38)
Claude Sonnet 5.5 on AA GDP.pdf - Criterion Pass Rate: 80.6 (#10)
Claude Sonnet 5.5 on AA GDP.pdf - Engineering - Criterion Pass Rate: 82.17 (#6)
Claude Sonnet 5.5 on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 90.78 (#2)
Claude Sonnet 5.5 on AA GDP.pdf - HR - Criterion Pass Rate: 87.47 (#5)
Claude Sonnet 5.5 on AA GDP.pdf - Healthcare - Criterion Pass Rate: 90.36 (#11)
Claude Sonnet 5.5 on AA GDP.pdf - Insurance - Criterion Pass Rate: 86.98 (#9)
Claude Sonnet 5.5 on AA GDP.pdf - Legal - Criterion Pass Rate: 83.42 (#21)
Claude Sonnet 5.5 on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 78.73 (#12)
Claude Sonnet 5.5 on AA GDP.pdf - Real Estate - Criterion Pass Rate: 71.97 (#23)
Claude Sonnet 5.5 on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 79.54 (#7)
Claude Sonnet 5.5 on AA GDPval: 1844.21 (#2)
Claude Sonnet 5.5 on AA Humanity's Last Exam: 54.96 (#8)
Claude Sonnet 5.5 on AA Long Context Reasoning: 82.67 (#28)
Claude Sonnet 5.5 on AA Omniscience: 32.3 (#20)
Claude Sonnet 5.5 on AA SciCode: 61.0 (#4)
Claude Sonnet 5.5 on AA-Briefcase: 63.23 (#1)
Claude Sonnet 5.5 on AA-Briefcase - Analytical Quality Elo: 2160.93 (#2)
Claude Sonnet 5.5 on AA-Briefcase - DOCX - Rubric Pass Rate: 59.76 (#2)
Claude Sonnet 5.5 on AA-Briefcase - PDF - Rubric Pass Rate: 64.66 (#3)
Claude Sonnet 5.5 on AA-Briefcase - PPTX - Rubric Pass Rate: 60.82 (#4)
Claude Sonnet 5.5 on AA-Briefcase - Presentation Elo: 1693.64 (#2)
Claude Sonnet 5.5 on AA-Briefcase - XLSX - Rubric Pass Rate: 64.77 (#1)
Claude Sonnet 5.5 on AA-Omniscience Accuracy: 53.95 (#37)
Claude Sonnet 5.5 on AA-Omniscience Hallucination Rate: 47.01 (#92)
Claude Sonnet 5.5 on AA-Omniscience Index - Business: 19.3 (#40)
Claude Sonnet 5.5 on AA-Omniscience Index - Health: 9.7 (#35)
Claude Sonnet 5.5 on AA-Omniscience Index - Humanities & Social Sciences: 30.8 (#22)
Claude Sonnet 5.5 on AA-Omniscience Index - Law: 26.6 (#41)
Claude Sonnet 5.5 on AA-Omniscience Index - Science, Engineering & Mathematics: 29.1 (#10)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE): 78.3 (#15)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - C: 87.0 (#15)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Dart: 82.0 (#9)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Go: 82.0 (#18)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - HTML: 74.0 (#23)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Java: 58.0 (#15)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 78.18 (#17)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Julia: 92.0 (#1)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 72.0 (#25)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - PHP: 76.0 (#24)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Python: 85.5 (#10)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - R: 62.0 (#9)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Rust: 86.0 (#5)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Swift: 80.0 (#19)
Claude Sonnet 5.5 on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 85.56 (#21)
Claude Sonnet 5.5 on AI Chess Leaderboard (Continuation): 1759.0 (#3)
Claude Sonnet 5.5 on AI Chess Leaderboard (Reasoning): 1158.0 (#62)
Claude Sonnet 5.5 on ActiveVision: 41.2 (#2)
Claude Sonnet 5.5 on ActiveVision (Agents): 78.8 (#3)
Claude Sonnet 5.5 on Artificial Analysis Intelligence Index: 55.98 (#3)
Claude Sonnet 5.5 on AutomationBench-AA: 71.33 (#1)
Claude Sonnet 5.5 on AutomationBench-AA - Finance - Raw Objective Completion: 91.96 (#4)
Claude Sonnet 5.5 on AutomationBench-AA - HR - Raw Objective Completion: 79.6 (#1)
Claude Sonnet 5.5 on AutomationBench-AA - Marketing - Raw Objective Completion: 90.28 (#4)
Claude Sonnet 5.5 on AutomationBench-AA - Operations - Raw Objective Completion: 97.05 (#1)
Claude Sonnet 5.5 on AutomationBench-AA - Sales - Raw Objective Completion: 89.67 (#2)
Claude Sonnet 5.5 on AutomationBench-AA - Strict: 44.75 (#1)
Claude Sonnet 5.5 on AutomationBench-AA - Support - Raw Objective Completion: 91.22 (#5)
Claude Sonnet 5.5 on BenchLM: 80.5 (#5)
Claude Sonnet 5.5 on Bug Hunt Bench: 57.0 (#1)
Claude Sonnet 5.5 on Bug Hunt Bench - LMS: 33.0 (#1)
Claude Sonnet 5.5 on Bug Hunt Bench - VS Code Extension: 24.0 (#1)
Claude Sonnet 5.5 on CursorBench 4.0: 55.5 (#2)
Claude Sonnet 5.5 on Harvey LAB-AA: 93.07 (#6)
Claude Sonnet 5.5 on Harvey LAB-AA - All Pass: 11.67 (#9)
Claude Sonnet 5.5 on Harvey LAB-AA - Antitrust & Competition: 92.03 (#10)
Claude Sonnet 5.5 on Harvey LAB-AA - Arbitration & International Dispute Resolution: 95.34 (#6)
Claude Sonnet 5.5 on Harvey LAB-AA - Banking & Finance: 94.8 (#3)
Claude Sonnet 5.5 on Harvey LAB-AA - Bankruptcy & Restructuring: 92.73 (#6)
Claude Sonnet 5.5 on Harvey LAB-AA - Capital Markets: 96.86 (#3)
Claude Sonnet 5.5 on Harvey LAB-AA - Corporate Governance: 91.1 (#11)
Claude Sonnet 5.5 on Harvey LAB-AA - Corporate M&A: 95.83 (#2)
Claude Sonnet 5.5 on Harvey LAB-AA - Data Privacy & Cybersecurity: 98.94 (#1)
Claude Sonnet 5.5 on Harvey LAB-AA - Emerging Companies & Venture Capital: 94.88 (#4)
Claude Sonnet 5.5 on Harvey LAB-AA - Employment & Labor: 89.89 (#3)
Claude Sonnet 5.5 on Harvey LAB-AA - Energy & Natural Resources: 96.7 (#8)
Claude Sonnet 5.5 on Harvey LAB-AA - Environmental & ESG: 90.68 (#6)
Claude Sonnet 5.5 on Harvey LAB-AA - Funds & Asset Management: 90.38 (#7)
Claude Sonnet 5.5 on Harvey LAB-AA - Healthcare & Life Sciences: 94.06 (#5)
Claude Sonnet 5.5 on Harvey LAB-AA - Immigration: 94.18 (#2)
Claude Sonnet 5.5 on Harvey LAB-AA - Insurance: 93.77 (#9)
Claude Sonnet 5.5 on Harvey LAB-AA - Intellectual Property: 95.41 (#12)
Claude Sonnet 5.5 on Harvey LAB-AA - International Trade & Sanctions: 91.67 (#4)
Claude Sonnet 5.5 on Harvey LAB-AA - Litigation & Dispute Resolution: 94.76 (#5)
Claude Sonnet 5.5 on Harvey LAB-AA - Real Estate: 95.04 (#9)
Claude Sonnet 5.5 on Harvey LAB-AA - Structured Finance & Securitization: 90.16 (#5)
Claude Sonnet 5.5 on Harvey LAB-AA - Tax: 89.97 (#6)
Claude Sonnet 5.5 on Harvey LAB-AA - Trusts, Estates & Private Client: 95.8 (#1)
Claude Sonnet 5.5 on Harvey LAB-AA - White Collar Defense & Investigations: 93.6 (#5)
Claude Sonnet 5.5 on LLM Stats (AutomationBench): 44.7 (#6)
Claude Sonnet 5.5 on LLM Stats (BioMysteryBench): 89.2 (#3)
Claude Sonnet 5.5 on LLM Stats (DeepSWE 1.1): 71.0 (#8)
Claude Sonnet 5.5 on LLM Stats (FrontierCode 1.1): 52.1 (#5)
Claude Sonnet 5.5 on LLM Stats (Humanity's Last Exam (no tools, text-only)): 56.9 (#2)
Claude Sonnet 5.5 on LLM Stats (Legal Agent Benchmark): 11.7 (#2)
Claude Sonnet 5.5 on LLM Stats (OfficeQA Pro): 65.6 (#6)
Claude Sonnet 5.5 on LLM Stats (Program Bench): 79.7 (#2)
Claude Sonnet 5.5 on LLM Stats (SWE-Bench Multimodal): 54.3 (#3)
Claude Sonnet 5.5 on LLM Stats (SWE-bench Multilingual): 90.3 (#2)
Claude Sonnet 5.5 on LLM Stats Score: 58.06 (#3)
Claude Sonnet 5.5 on LM Market Cap LMC Score: 85.2 (#66)
Claude Sonnet 5.5 on LiveBench AMPS Hard: 98.0 (#41)
Claude Sonnet 5.5 on LiveBench Code Completion: 86.96 (#5)
Claude Sonnet 5.5 on LiveBench Code Generation: 95.78 (#1)
Claude Sonnet 5.5 on LiveBench Connections: 99.33 (#29)
Claude Sonnet 5.5 on LiveBench Consecutive Events: 90.94 (#3)
Claude Sonnet 5.5 on LiveBench Integrals With Game: 100.0 (#4)
Claude Sonnet 5.5 on LiveBench JavaScript: 77.27 (#8)
Claude Sonnet 5.5 on LiveBench Logic With Navigation: 82.0 (#12)
Claude Sonnet 5.5 on LiveBench Math Comp: 97.06 (#19)
Claude Sonnet 5.5 on LiveBench Olympiad: 92.47 (#5)
Claude Sonnet 5.5 on LiveBench Paraphrase: 62.28 (#46)
Claude Sonnet 5.5 on LiveBench Plot Unscrambling: 67.79 (#17)
Claude Sonnet 5.5 on LiveBench Python: 35.0 (#62)
Claude Sonnet 5.5 on LiveBench Simplify: 72.37 (#5)
Claude Sonnet 5.5 on LiveBench Spatial: 98.0 (#35)
Claude Sonnet 5.5 on LiveBench Story Generation: 77.08 (#11)
Claude Sonnet 5.5 on LiveBench Summarize: 70.33 (#31)
Claude Sonnet 5.5 on LiveBench Table Join: 46.88 (#39)
Claude Sonnet 5.5 on LiveBench Table Reformat: 98.04 (#49)
Claude Sonnet 5.5 on LiveBench Theory of Mind: 86.54 (#6)
Claude Sonnet 5.5 on LiveBench TypeScript: 56.67 (#8)
Claude Sonnet 5.5 on LiveBench Typos: 86.0 (#12)
Claude Sonnet 5.5 on LiveBench Zebra Puzzle: 100.0 (#16)
Claude Sonnet 5.5 on MLCR-AA: 75.0 (#1)
Claude Sonnet 5.5 on MLCR-AA - Accuracy: 90.75 (#8)
Claude Sonnet 5.5 on MLCR-AA - Completeness: 88.44 (#2)
Claude Sonnet 5.5 on MLCR-AA - Conciseness: 96.11 (#50)
Claude Sonnet 5.5 on MineBench: 1948.0 (#8)
Claude Sonnet 5.5 on RuneBench: 3394.0 (#24)
Claude Sonnet 5.5 on Vals AI (Vals Index): 69.22 (#2)
Claude Sonnet 5.5 on Vals AI CyberBench: 59.58 (#31)
Claude Sonnet 5.5 on Vals AI Excel Modeling: 75.71 (#3)
Claude Sonnet 5.5 on Vals AI Finance Agent v2: 58.1 (#9)
Claude Sonnet 5.5 on Vals AI Harvey Legal Agent Bench: 2.92 (#36)
Claude Sonnet 5.5 on Vals AI IOI: 83.06 (#7)
Claude Sonnet 5.5 on Vals AI Legal Research Bench: 48.08 (#9)
Claude Sonnet 5.5 on Vals AI MedCode: 52.92 (#11)
Claude Sonnet 5.5 on Vals AI MedScribe: 91.1 (#3)
Claude Sonnet 5.5 on Vals AI MysteryMechanism: 49.1 (#3)
Claude Sonnet 5.5 on Vals AI ProofBench: 100.0 (#1)
Claude Sonnet 5.5 on Vals AI Public Benefits Bench: 67.19 (#12)
Claude Sonnet 5.5 on Vals AI ReverseEngBench: 30.15 (#4)
Claude Sonnet 5.5 on Vals AI SAGE: 51.77 (#10)
Claude Sonnet 5.5 on Vals AI Tax Agent Bench: 73.39 (#3)
Claude Sonnet 5.5 on Vals AI Terminal-Bench 2.1: 83.15 (#7)
Claude Sonnet 5.5 on Vals AI Terminal-Bench 4.0: 53.03 (#3)
Claude Sonnet 5.5 on Vals AI Terminal-Bench Science: 38.57 (#3)
Claude Sonnet 5.5 on Vellum - Humanity's Last Exam: 64.5 (#6)
Claude Sonnet 5.5 on VoxelBench: 1000.0 (#54)
GPT-6 on Chatbot Arena (Vision - Creative Writing): 1284.0 (#16)
GPT-6 on Chatbot Arena (Vision - Homework): 1335.0 (#5)
GPT-6 on FrontierSWE V2: 65.51 (#1)
GPT-6 on NonoBench: 100.0 (#1)
GPT-6 on SWE-Together: 78.47 (#15)
GPT-6 Sol on ARC-AGI-1: 95.5 (#30)
GPT-6 Sol on ARC-AGI-2: 89.58 (#14)
GPT-6 Sol on ARC-AGI-3: 4.62 (#12)
GPT-6 Sol on EKI Estonian LLM - Book Recommendations: 86.25 (#5)
GPT-6 Sol on EKI Estonian LLM - Idioms: 70.67 (#6)
GPT-6 Sol on EKI Estonian LLM - Language Advice: 67.08 (#5)
GPT-6 Sol on EKI Estonian LLM - Overall: 69.16 (#4)
GPT-6 Sol on EKI Estonian LLM - Propaganda Resistance: 89.05 (#26)
GPT-6 Sol on EKI Estonian LLM - Terminology: 57.54 (#4)
GPT-6 Sol on EKI Estonian LLM - Trivia: 76.4 (#9)
GPT-6 Sol on EKI Estonian LLM - Trivia Hard: 37.12 (#9)
GPT-6 Sol on Epoch AI - Apex Agents: 54.3 (#16)
GPT-6 Sol on Epoch AI - Critpt: 30.86 (#6)
GPT-6 Sol on Epoch AI - Dtbench: 97.33 (#11)
GPT-6 Sol on Epoch AI - Lmca: 59.08 (#23)
GPT-6 Sol on Epoch AI - Scicode: 57.64 (#15)
GPT-6 Sol on Explicit Edit Benchmark: 85.2 (#18)
GPT-6 Sol on NonoBench: 86.7 (#12)
GPT-6 Sol on SWE-Together: 80.34 (#10)
GPT-6 Sol on SpacetimeDB LLM Benchmark (C#): 93.2 (#14)
GPT-6 Sol on SpacetimeDB LLM Benchmark (Rust): 98.4 (#4)
GPT-6 Sol on SpacetimeDB LLM Benchmark (TypeScript): 99.5 (#3)
GPT-6 Sol on Vals AI MysteryMechanism: 30.18 (#10)
GPT-6 Sol on Vals AI RSI Index: 0.281 (#4)
GPT-6 Sol on Vals AI Terminal-Bench 4.0: 34.34 (#6)
GPT-6 Sol on Vals AI Terminal-Bench Science: 31.43 (#5)

New #1 Leaders (44)

NonoBench: GPT-6 (xHigh) (100.0) beat Gemini 3.1 Pro (Preview) (High) by 36.7
Design Arena (Data Viz): Claude Opus 5.5 (1374.0) beat Muse Spark 1.2 by 16.0
Bug Hunt Bench: Claude Sonnet 5.5 (Max) (57.0) beat GPT-6 (Max) by 12.0
Bug Hunt Bench - LMS: Claude Sonnet 5.5 (Max) (33.0) beat GPT-5.6 Sol (Max) by 8.5
Chatbot Arena (Vision - Chinese): Gemini 3.7 Flash (High) (1387.0) beat Claude Fable 5 (High) by 8.0
SWE-bench Live: Slingshot-v3.4.0 + GPT-5.6-Sol (70.33) beat AMI Agent + Claude-4.6-Opus by 7.33
Epoch AI - Cursorbench: Claude Opus 5.5 (Max) (57.8) beat Claude Fable 5.1 (Max) by 6.0
Epoch AI - Apex Agents: Claude Opus 5.5 (Max) (73.5) beat Claude Fable 5.1 (Unknown) by 4.9
LLM Stats (HealthBench): Claude Sonnet 5.5 (65.4) beat Claude Opus 5.5 by 4.8
SWE-Together: Claude Fable 5 (85.16) beat Claude Opus 4.6 by 4.27
LiveBench Code Generation: Claude Sonnet 5.5 (Max) (95.78) beat Claude Fable 5 (Max) by 4.23
LLM Stats (Terminal-Bench 4.0): Claude Sonnet 5.5 (70.6) beat Claude Opus 5.5 by 4.2
MLCR-AA: Claude Sonnet 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (75.0) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 3.89
Epoch AI - Scicode: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (66.9) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 3.82
AA-Briefcase - XLSX - Rubric Pass Rate: Claude Sonnet 5.5 (Adaptive Reasoning, Xhigh Effort, Default Fallback) (64.77) beat Claude Opus 5.5 (Adaptive Reasoning, Xhigh Effort, Default Fallback) by 3.63
Drone-Bench - Reconstruct: Claude Opus 5.5 (71.0) beat GPT-6 by 3.5
AutomationBench-AA - Strict: Claude Sonnet 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (44.75) beat GPT-6 (Max) by 3.2
LLM Stats (HealthBench Professional): Claude Sonnet 5.5 (69.2) beat Claude Fable 5 by 3.2
SWE-Marathon: Grok 4.5 (29.0) beat Claude Opus 4.8 by 3.0
Epoch AI - Lmca: Claude Opus 5.5 (Max) (68.23) beat Claude Fable 5.1 (Unknown) by 2.77
Vellum - Humanity's Last Exam: Claude Opus 5.5 (67.7) beat Claude Fable 5.1 by 2.7
Vals AI CyberBench: GPT-6 Sol (77.98) beat MiMo-V2.6-Flash by 2.62
HANDBOOK.md Agents: Claude Fable 5.1 (Max) (38.8) beat Claude Fable 5 (Adaptive Reasoning, Max Effort) by 2.6
AutomationBench-AA - HR - Raw Objective Completion: Claude Sonnet 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (79.6) beat Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) by 2.33
Drone-Bench: Claude Opus 5.5 (97.3) beat GPT-6 by 2.2
Vals AI Code Migration: Claude Sonnet 5.5 (69.83) beat GPT-6 by 2.09
Vals AI Vibe Code Bench: Claude Sonnet 5.5 (92.39) beat Claude Fable 5 by 2.04
Mercor APEX: Claude Sonnet 5.5 (75.5) beat Claude Opus 5.5 by 2.0
Vals AI BioMysteryBench: Claude Sonnet 5.5 (81.11) beat GPT-6 by 1.85
AutomationBench-AA: Claude Sonnet 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (71.33) beat Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) by 1.79
AA-Briefcase: Claude Sonnet 5.5 (Adaptive Reasoning, Xhigh Effort, Default Fallback) (63.23) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 1.71
LLM Stats (BenchCAD): Claude Sonnet 5.5 (74.7) beat Claude Opus 5.5 by 1.7
SpacetimeDB LLM Benchmark (Rust): Claude Opus 5.5 (100.0) beat GPT-6 by 1.6
Bug Hunt Bench - VS Code Extension: Claude Sonnet 5.5 (Max) (24.0) beat GPT-6 (xHigh) by 1.0
Drone-Bench - Detect: Claude Opus 5.5 (88.7) beat Claude Fable 5 by 0.9
Epoch AI - Dtbench: Claude Opus 5.5 (Max) (98.93) beat Claude Fable 5 (Max) by 0.53
Harvey LAB-AA - Trusts, Estates & Private Client: Claude Sonnet 5.5 (Adaptive Reasoning, Xhigh Effort, Default Fallback) (95.8) beat Kimi K3 (Max) by 0.38
BRIDGE Medical Leaderboard - Few-Shot: Qwen3.8-Flash-Next (Non-reasoning) (56.03) beat Qwen 3.8 27B (Non-reasoning) by 0.36
BRIDGE Medical Leaderboard: Qwen3.8-Flash-Next (Non-reasoning) (49.39) beat Gemma 4 31B (IT) by 0.22
BRIDGE Medical Leaderboard - Zero-Shot: Qwen3.8-Flash-Next (Thinking) (46.95) beat Gemma 4 31B (IT) by 0.21
LLM Stats (BenchCAD (with Python tool)): Claude Sonnet 5.5 (96.3) beat Claude Opus 5.5 by 0.1
AutomationBench-AA - Operations - Raw Objective Completion: Claude Sonnet 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (97.05) beat GPT-6 (Max) by 0.04
BRIDGE Medical Leaderboard - CoT: Qwen3.8-Flash-Next (Non-reasoning) (45.92) beat Gemma 4 31B (IT) by 0.04
FrontierSWE V2: GPT-6 (Max) (65.51) beat GPT-6 by 0.01

The Aggregate Digest — 2026-09-28

2026-09-28T06:11:20.367562+00:00

Daily

Top-10 New Scores (17)

Claude Opus 5 on Creative Writing (Lechmazur): 3.454 (#4)
Claude Opus 5.5 on Agent Arena: 12.15 (#2)
Claude Opus 5.5 on Agent Arena - Bash Recovery: 10.64 (#4)
Claude Opus 5.5 on Agent Arena - Confirmed Success: 15.5 (#2)
Claude Opus 5.5 on Agent Arena - Praise vs Complaint: 19.8 (#3)
Claude Opus 5.5 on Agent Arena - Steerability: 14.5 (#1)
Claude Opus 5.5 on Agent Arena - Tool Hallucination: -0.32 (#23)
Claude Opus 5.5 on Bullshit Benchmark: 52.7 (#39)
Claude Opus 5.5 on ChessBench GitHub - Elo: 2280.0 (#19)
Claude Opus 5.5 on Creative Writing (Lechmazur): 3.757 (#3)
Claude Opus 5.5 on Design Arena (SVG): 1337.0 (#4)
GPT-6 on SnakeBench: 37.9 (#3)
GPT-6 on Vectara Hallucination Leaderboard: 91.3 (#45)
GPT-6 Sol on Bullshit Benchmark: 41.8 (#58)
GPT-6 Sol on Design Arena (3D): 1307.0 (#20)
GPT-6 Sol on Design Arena (ASCII Art): 1278.0 (#16)
GPT-6 Sol on Vectara Hallucination Leaderboard: 93.5 (#29)

New #1 Leaders (3)

Design Arena (SVG): arrow-2-telos (1625.0) beat GPT-6 by 179.0
MEGA-Bench Task - Move Position To Position Hanoi 4 Pole: Gemini 3.8 Flash (58.2) beat GPT-4o Mini by 47.5
GAIA: Ops-Agentic-Search-2.0 (93.69) beat CustomGPT.ai Research Lab v44 by 0.33

The Aggregate Digest — 2026-09-27

2026-09-27T12:07:13.199270+00:00

Daily

Top-10 New Scores (74)

Claude Fable 5.1 on Chatbot Arena (Text - German): 1486.0 (#20)
Claude Fable 5.1 on Chatbot Arena (Text - Japanese): 1555.0 (#1)
Claude Fable 5.1 on Chatbot Arena (Text - Polish): 1485.0 (#23)
Claude Fable 5.1 on Chatbot Arena (Text - Spanish): 1479.0 (#17)
Claude Fable 5.1 on Design Arena (Agents - Mobile Apps): 1268.0 (#4)
Claude Fable 5.1 on UGI - Natural Intelligence: 70.11 (#39)
Claude Fable 5.1 on UGI - Willingness (W/10): 4.2 (#899)
Claude Fable 5.1 on UGI - Writing: 63.86 (#75)
Claude Fable 5.1 on UGI Leaderboard: 52.08 (#95)
Claude Opus 5.5 on AI Coding Daily (Claude Code) - Laravel Code Quality: 17.52 (#6)
Claude Opus 5.5 on AI Coding Daily (Claude Code) - React-TS Code Quality: 17.17 (#4)
Claude Opus 5.5 on AI Coding Daily (Claude Code) - Total: 53.69 (#4)
Claude Opus 5.5 on Chatbot Arena (Text - Business, Management & Financial Ops): 1499.0 (#7)
Claude Opus 5.5 on Chatbot Arena (Text - Chinese): 1559.0 (#2)
Claude Opus 5.5 on Chatbot Arena (Text - Coding): 1547.0 (#5)
Claude Opus 5.5 on Chatbot Arena (Text - Creative Writing): 1521.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text - English): 1508.0 (#4)
Claude Opus 5.5 on Chatbot Arena (Text - Entertainment, Sports & Media): 1493.0 (#2)
Claude Opus 5.5 on Chatbot Arena (Text - Expert): 1556.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text - Hard Prompts (English)): 1536.0 (#2)
Claude Opus 5.5 on Chatbot Arena (Text - Hard Prompts): 1541.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text - Instruction Following): 1516.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text - Legal & Government): 1495.0 (#16)
Claude Opus 5.5 on Chatbot Arena (Text - Life, Physical & Social Science): 1535.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text - Longer Query): 1530.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text - Multi-Turn): 1519.0 (#2)
Claude Opus 5.5 on Chatbot Arena (Text - Non-English): 1501.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text - Russian): 1525.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text - Software & IT Services): 1541.0 (#2)
Claude Opus 5.5 on Chatbot Arena (Text - Writing, Literature & Language): 1520.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text): 1509.0 (#1)
Claude Opus 5.5 on Design Arena (3D): 1450.0 (#2)
Claude Opus 5.5 on Design Arena (Agents - Mobile Apps): 1268.0 (#5)
Claude Opus 5.5 on SpeechMap Compliance: 82.8 (#79)
Claude Opus 5.5 on UGI - Natural Intelligence: 73.28 (#23)
Claude Opus 5.5 on UGI - Willingness (W/10): 4.2 (#898)
Claude Opus 5.5 on UGI - Writing: 70.01 (#22)
Claude Opus 5.5 on UGI Leaderboard: 54.46 (#55)
Claude Opus 5.5 on ZeroBench: 43.0 (#2)
GPT-6 on AI Coding Daily (Codex CLI) - Laravel Code Quality: 18.6 (#1)
GPT-6 on AI Coding Daily (Codex CLI) - React-TS Code Quality: 18.17 (#1)
GPT-6 on AI Coding Daily (Codex CLI) - Total: 56.77 (#1)
GPT-6 on Chatbot Arena (Text - Chinese): 1516.0 (#33)
GPT-6 on Chatbot Arena (Text - French): 1510.0 (#9)
GPT-6 on Chatbot Arena (Text - Math): 1486.0 (#28)
GPT-6 on Chatbot Arena (Text - Mathematical): 1486.0 (#34)
GPT-6 on Chatbot Arena (Text - Spanish): 1441.0 (#78)
GPT-6 on OpenRouter Tau2-Bench Airline: 68.0 (#76)
GPT-6 on ZeroBench: 52.0 (#1)
GPT-6 Sol on Chatbot Arena (Text - Business, Management & Financial Ops): 1465.0 (#47)
GPT-6 Sol on Chatbot Arena (Text - Chinese): 1495.0 (#67)
GPT-6 Sol on Chatbot Arena (Text - Coding): 1524.0 (#26)
GPT-6 Sol on Chatbot Arena (Text - Creative Writing): 1451.0 (#34)
GPT-6 Sol on Chatbot Arena (Text - English): 1462.0 (#72)
GPT-6 Sol on Chatbot Arena (Text - Entertainment, Sports & Media): 1445.0 (#42)
GPT-6 Sol on Chatbot Arena (Text - Expert): 1507.0 (#29)
GPT-6 Sol on Chatbot Arena (Text - Hard Prompts (English)): 1494.0 (#40)
GPT-6 Sol on Chatbot Arena (Text - Hard Prompts): 1485.0 (#54)
GPT-6 Sol on Chatbot Arena (Text - Instruction Following): 1447.0 (#65)
GPT-6 Sol on Chatbot Arena (Text - Legal & Government): 1464.0 (#67)
GPT-6 Sol on Chatbot Arena (Text - Life, Physical & Social Science): 1470.0 (#77)
GPT-6 Sol on Chatbot Arena (Text - Longer Query): 1467.0 (#59)
GPT-6 Sol on Chatbot Arena (Text - Math): 1450.0 (#69)
GPT-6 Sol on Chatbot Arena (Text - Mathematical): 1455.0 (#71)
GPT-6 Sol on Chatbot Arena (Text - Medicine & Healthcare): 1460.0 (#89)
GPT-6 Sol on Chatbot Arena (Text - Multi-Turn): 1461.0 (#71)
GPT-6 Sol on Chatbot Arena (Text - Non-English): 1448.0 (#55)
GPT-6 Sol on Chatbot Arena (Text - Russian): 1466.0 (#49)
GPT-6 Sol on Chatbot Arena (Text - Software & IT Services): 1507.0 (#38)
GPT-6 Sol on Chatbot Arena (Text - Spanish): 1430.0 (#93)
GPT-6 Sol on Chatbot Arena (Text - Writing, Literature & Language): 1440.0 (#68)
GPT-6 Sol on Chatbot Arena (Text): 1457.0 (#60)
GPT-6 Sol on SpeechMap Compliance: 64.0 (#185)
GPT-6 Sol on ZeroBench: 41.0 (#3)

New #1 Leaders (18)

Chatbot Arena (Text - Japanese): Claude Fable 5.1 (Max) (1555.0) beat Claude Fable 5 (High) by 32.0
ZeroBench: GPT-6 (Max) (52.0) beat GPT-5.6 Sol (Max) by 22.0
AppWorld Challenge: Claude Opus 4.7 (85.6) beat Qwen 3 14B by 18.0
Chatbot Arena (Text - Creative Writing): Claude Opus 5.5 (High) (1521.0) beat Claude Fable 5 (High) by 17.0
Chatbot Arena (Text - French): Muse Spark 1.3 (Max) (1541.0) beat Claude Fable 5.1 (Max) by 9.0
Chatbot Arena (Text - Writing, Literature & Language): Claude Opus 5.5 (High) (1520.0) beat Claude Fable 5 (High) by 9.0
Chatbot Arena (Text - Hard Prompts): Claude Opus 5.5 (High) (1541.0) beat Claude Opus 4.6 (High) by 8.0
Chatbot Arena (Text - Expert): Claude Opus 5.5 (High) (1556.0) beat Claude Fable 5 (High) by 8.0
Chatbot Arena (Text - Life, Physical & Social Science): Claude Opus 5.5 (High) (1535.0) beat Claude Fable 5 (High) by 7.0
Chatbot Arena (Text - Russian): Claude Opus 5.5 (High) (1525.0) beat Claude Fable 5 (High) by 6.0
Chatbot Arena (Text - Longer Query): Claude Opus 5.5 (High) (1530.0) beat Claude Opus 4.6 (High) by 6.0
Chatbot Arena (Text - Non-English): Claude Opus 5.5 (High) (1501.0) beat Claude Fable 5 (High) by 5.0
AppWorld Normal: Claude Opus 4.6 (91.1) beat Qwen 3 14B by 4.2
Chatbot Arena (Text): Claude Opus 5.5 (High) (1509.0) beat Claude Fable 5 (High) by 3.0
Chatbot Arena (Text - Instruction Following): Claude Opus 5.5 (High) (1516.0) beat Claude Opus 4.6 (High) by 3.0
AI Coding Daily (Codex CLI) - Total: GPT-6 (High) (56.77) beat GPT-6 (Medium) by 0.74
AI Coding Daily (Codex CLI) - Laravel Code Quality: GPT-6 (High) (18.6) beat GPT-6 (Medium) by 0.4
AI Coding Daily (Codex CLI) - React-TS Code Quality: GPT-6 (High) (18.17) beat GPT-6 (Medium) by 0.34

Weekly

New Benchmarks (28)

xDailyBench - Task-level Overall (Score (%)): leader Seed-2.1-Evolving (75.6), 11 models
Professional consultation on real daily problems: each task is scored against a rubric, then averaged equally across tasks. Higher is better. Source: arxiv.org/abs/2609.07784.
xDailyBench - Personal Life (Score (%)): leader Kimi K3 (75.3), 11 models
xDailyBench task-level score over the personal-life domain. Higher is better. Source: arxiv.org/abs/2609.07784.
xDailyBench - White-collar (Score (%)): leader Seed-2.1-Evolving (78.8), 11 models
xDailyBench task-level score over the white-collar work domain. Higher is better. Source: arxiv.org/abs/2609.07784.
xDailyBench - Study & Research (Score (%)): leader Kimi K3 (80.5), 11 models
xDailyBench task-level score over the study and research domain. Higher is better. Source: arxiv.org/abs/2609.07784.
xDailyBench - Other Domains (Score (%)): leader GPT-5.6 Sol (82.4), 11 models
xDailyBench task-level score over the remaining domains outside personal life, white-collar work and study. Higher is better. Source: arxiv.org/abs/2609.07784.
xDailyBench - Explicit Rubrics (Score (%)): leader Seed-2.1-Evolving (81.0), 11 models
xDailyBench rubric-level score pooled over requirements the user stated explicitly. Higher is better. Source: arxiv.org/abs/2609.07784.
xDailyBench - Implicit Rubrics (Score (%)): leader Kimi K3 (67.6), 11 models
xDailyBench rubric-level score pooled over requirements the user left implicit -- the harder half, and the one that separates the field. Higher is better. Source: arxiv.org/abs/2609.07784.
xDailyBench - Information Understanding (Score (%)): leader Kimi K3 (77.1), 12 models
xDailyBench rubric items aggregated by capability: understanding the information supplied with the request. Higher is better. Source: arxiv.org/abs/2609.07784.
xDailyBench - Instruction Following (Score (%)): leader Seed-2.1-Evolving (78.4), 12 models
xDailyBench rubric items aggregated by capability: following the instructions given. Higher is better. Source: arxiv.org/abs/2609.07784.
xDailyBench - Analytical Reasoning (Score (%)): leader GPT-5.5 (75.3), 12 models
xDailyBench rubric items aggregated by capability: analytical reasoning. Higher is better. Source: arxiv.org/abs/2609.07784.
xDailyBench - Quantitative Reasoning (Score (%)): leader Seed-2.1-Evolving (74.7), 12 models
xDailyBench rubric items aggregated by capability: quantitative reasoning. Higher is better. Source: arxiv.org/abs/2609.07784.
xDailyBench - Artifact Quality (Score (%)): leader Seed-2.1-Evolving (81.5), 12 models
xDailyBench rubric items aggregated by capability: the quality of the artifact the consultation produces. Higher is better. Source: arxiv.org/abs/2609.07784.
PerfReasoning - Low Effort (Both-Exact Pass Rate (%)): leader GPT-5.6 Sol (88.4), 11 models
Reasoning about hardware performance: the model writes a performance model for a kernel and gets three self-revision rounds. Both-exact pass rate requires predicted runtime and scaling to be right together, at low reasoning effort. Higher is better. Source: arxiv.org/abs/2609.04476.
PerfReasoning - Medium Effort (Both-Exact Pass Rate (%)): leader GPT-5.6 Sol (88.4), 10 models
PerfReasoning both-exact pass rate at medium reasoning effort. Higher is better. Source: arxiv.org/abs/2609.04476.
PerfReasoning - High Effort (Both-Exact Pass Rate (%)): leader GPT-5.5 (88.4), 12 models
PerfReasoning both-exact pass rate at high reasoning effort. Higher is better. Source: arxiv.org/abs/2609.04476.
PerfReasoning - Max Effort (Both-Exact Pass Rate (%)): leader GPT-5.6 Sol (88.4), 4 models
PerfReasoning both-exact pass rate at maximum reasoning effort, for the four models that expose it. Higher is better. Source: arxiv.org/abs/2609.04476.
HAE-GEO - Clean Answer Quality (Rubric Score (0-2)): leader GPT-5.6 Sol (1.53), 10 models
Deep-search agents on clean web evidence, before any poisoning: semantic rubric score 0-2. Higher is better. Source: arxiv.org/abs/2609.06027.
HAE-GEO - Fooled Rate (L1) (Fooled Rate (%)): leader Qwen 3.8 27B (26.6), 10 models
Deep-search agents against hierarchical web evidence poisoning at level 1: how often the planted evidence lands in the answer. Lower is better. Source: arxiv.org/abs/2609.06027.
HAE-GEO - Fooled Rate (L2) (Fooled Rate (%)): leader Kimi K3 (32.3), 10 models
HAE-GEO fooled rate at poisoning level 2. Lower is better. Source: arxiv.org/abs/2609.06027.
HAE-GEO - Fooled Rate (L3) (Fooled Rate (%)): leader Qwen 3.8 27B (31.7), 10 models
HAE-GEO fooled rate at poisoning level 3, the most aggressive. Lower is better. Source: arxiv.org/abs/2609.06027.
HAE-GEO - Risk Awareness (Rubric Score (0-2)): leader GPT-5.6 Sol (1.02), 10 models
HAE-GEO: whether the agent flags the risk that its sources may be manipulated, rubric 0-2. Higher is better. Source: arxiv.org/abs/2609.06027.
HAE-GEO - Poisoned Source Recognition (Rubric Score (0-2)): leader Kimi K3 (1.32), 10 models
HAE-GEO: whether the agent identifies the poisoned source specifically, rubric 0-2. Higher is better. Source: arxiv.org/abs/2609.06027.
Search-to-World - Request Resolution (Resolution Rate (%)): leader DeepSeek V4 Flash (88.3), 12 models
An agent is given a user request, searches the web for assets, and must deliver a navigable 3D world. Overall Request Resolution is the share of requests it resolves at all. Higher is better. Source: arxiv.org/abs/2609.07605.
Search-to-World - World Delivery (Delivery Rate (%)): leader DeepSeek V4 Flash (78.7), 12 models
Search-to-World's World Delivery Rate: the share of requests that end in an actually delivered, navigable 3D world rather than a resolved-but-undelivered attempt. Higher is better. Source: arxiv.org/abs/2609.07605.
SciTrue - Claim Validation (F1 (%)): leader Claude Fable 5 (86.3), 15 models
Validating scientific claims against the paper they cite: F1 on the claim verdict over frontier and open models. Higher is better. Source: arxiv.org/abs/2609.00654.
SciTrue - Evidence Pairing (Accuracy (%)): leader Claude Fable 5 (74.4), 15 models
SciTrue's paired condition: the verdict counts only when the model also identifies the supporting evidence, so it separates a right answer from a right answer for the right reason. Higher is better. Source: arxiv.org/abs/2609.00654.
SciTrue - Claim Attribution (Accuracy (%)): leader Claude Fable 5 (97.7), 14 models
SciTrue Task 2: attributing a claim to its source, multiple choice. Higher is better. Source: arxiv.org/abs/2609.00654.
Explicit Edit Benchmark (Explicit Edit Score v2 (%)): leader DeepSeek V4.1 Flash (99.0), 16 models
226 deterministic exact-edit tasks over eleven families - block, cross-file, distinct, multi-file and subset edits, replace-all, select-one, unique-match, literal edge cases, Unicode fixes and non-Latin-script text - replayed through coding agents; an observation passes only when the edited file matches the expected bytes. Score v2 = coverage × (0.75 first-exact + 0.25 final-exact), repetitions averaged inside a configuration and task and configurations weighted equally inside a task. Published here per model family, upstream's median over that model's complete eligible configurations, so a model with many harnesses is not ranked by its luckiest one. Higher is better. Source: huggingface.co/datasets/alexshpunt/explicit-edit-benchmark.

New Models (172)

Claude Opus 5.5 — Aggregate Rating 2052, #3
- CursorBench 4.0: 57.8 (#1/44)
- InferenceBench: 12.08 (#1/38)
- KernelBench Hub - Mega: 35.46 (#1/14)
- MathArena Arxiv False: 95.0 (#1/22)
- MathArena - ARXIV_FALSE June: 100.0 (#1/27)
- Chatbot Arena (Text): 1509.0 (#1/409)
- Chatbot Arena (Code): 1827.0 (#1/134)
- Design Arena (Game Dev): 1481.0 (#1/169)
- Chatbot Arena (Text - Russian): 1525.0 (#1/373)
- Chatbot Arena (Text - Creative Writing): 1521.0 (#1/407)
- ...and 320 more
GPT-6 — Aggregate Rating 2039, #4
- WebDev Arena (Fullstack): 1755.0 (#1/70)
- Epoch AI - Frontierswe: 65.51 (#1/13)
- Epoch AI - Rli: 20.83 (#1/15)
- Epoch AI - Vending Bench 2: 15514.7 (#1/62)
- Epoch AI - Blueprint Bench 2: 0.497 (#1/26)
- ActiveVision: 76.5 (#1/11)
- ZeroBench: 52.0 (#1/74)
- Agents on Rails: 53.3 (#1/25)
- HalluHard: 0.3652 (#1/33)
- DuelLab Overall: 94.8 (#1/200)
- ...and 41 more
Claude Fable 5.1 — Aggregate Rating 2021, #6
- Chatbot Arena (Text - Japanese): 1555.0 (#1/271)
- Epoch AI - Lmca: 65.32 (#2/172)
- AIMultiple - Text-to-SQL (Adjudicated): 82.8 (#2/49)
- ActiveVision (Agents): 76.5 (#3/8)
- APEX v1: 73.8 (#3/5)
- Design Arena (Agents - Mobile Apps): 1268.0 (#4/48)
- ActiveVision: 15.3 (#4/11)
- Vals AI IOI: 90.78 (#4/33)
- SuperCLUE-VLM - Overall: 75.13 (#4/15)
- Agents on Rails: 31.7 (#5/25)
- ...and 12 more
Claude Opus 5 — Aggregate Rating 1988, #7
- BALROG BabyAI (LLM): 100.0 (#1/39)
- Epoch AI - Dtbench: 97.87 (#2/210)
- BALROG Crafter (LLM): 68.2 (#2/39)
- BALROG TextWorld (LLM): 71.6 (#2/39)
- BALROG NetHack (LLM): 7.3 (#2/39)
- Epoch AI - Frontierswe: 52.01 (#3/13)
- ActiveVision (Agents): 68.2 (#4/8)
- BALROG BabaIsAI (LLM): 95.8 (#4/39)
- BALROG MiniHack (LLM): 37.5 (#4/39)
- Epoch AI - Lmca: 64.45 (#5/172)
- ...and 7 more
GPT-6 Sol — Aggregate Rating 1968, #8
- OpenAI GPT-6 Astra System Card - Biology Refusal - Dual Use: 98.0 (#1/5)
- OpenAI GPT-6 Astra System Card - Cyber Safety - Synthetic Agentic Environments: 0.998 (#1/5)
- OpenAI GPT-6 Astra System Card - Challenging Prompts - Gore: 90.0 (#1/6)
- OpenAI GPT-6 Astra System Card - Challenging Prompts - Sexual: 98.4 (#1/6)
- OpenAI GPT-6 Astra System Card - Challenging Prompts - Sexual/Minors: 99.1 (#1/6)
- OpenAI GPT-6 Astra System Card - U18 - Emotional Reliance: 94.8 (#1/7)
- OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Sensitive Personal Data: 85.4 (#1/4)
- MathArena - ARXIV August: 92.98 (#1/12)
- Agent Arena - Steerability: 15.7 (#1/44)
- MLCR-AA - Conciseness: 100.0 (#1/92)
- ...and 323 more
Claude Fable 5 — Aggregate Rating 1962, #9
- Epoch AI - Frontierswe: 46.96 (#4/13)
- Chatbot Arena (Search): 1230.0 (#5/34)
- FrontierCode: 64.25 (#5/60)
- WebDev Arena: 1627.98 (#8/124)
- Epoch AI - Dtbench: 97.33 (#9/210)
- Epoch AI - Lmca: 61.15 (#16/172)
- ForecastBench Tournament - Market: 72.0 (#44/255)
- ForecastBench: 65.9 (#64/255)
GPT-5.6 Sol — Aggregate Rating 1945, #11
- Benchouse Analytics Agent Benchmark: 74.43 (#2/5)
- Benchouse Analytics Agent Benchmark - Descriptive: 97.83 (#2/5)
- Benchouse Analytics Agent Benchmark - Diagnostic: 68.67 (#2/5)
- Benchouse Analytics Agent Benchmark - Prescriptive: 55.17 (#2/5)
- ActiveVision: 23.5 (#3/11)
- Vals AI IOI: 91.17 (#3/33)
- APEX v1: 73.1 (#5/5)
- DeepSearchQA: 93.0 (#6/16)
- ParseBench: 75.35 (#7/106)
- Agents on Rails: 28.3 (#8/25)
- ...and 2 more
Gemini 3.8 Flash — Aggregate Rating 1944, #12
- SuperCLUE-VLM - Overall: 79.15 (#2/15)
- AIMultiple - Text-to-SQL (Strict Execution Match): 63.5 (#2/49)
- Chatbot Arena (Text - Math): 1522.0 (#3/393)
- Kaggle AssetOpsBench: 69.02 (#3/21)
- Benchmarks.bio - BioSecBench-Surveillance: 44.1 (#4/5)
- Epoch AI - Blueprint Bench 2: 0.386 (#4/26)
- AIMultiple - Text-to-SQL (Adjudicated): 82.2 (#4/49)
- Chatbot Arena (Text - Japanese): 1505.0 (#6/271)
- Epoch AI - Apex Agents: 64.3 (#6/33)
- ARC-AGI-1: 98.5 (#6/237)
- ...and 16 more
Muse Spark 1.3 — Aggregate Rating 1925, #15
- Chatbot Arena (Text - French): 1541.0 (#1/287)
- Chatbot Arena (Text - Korean): 1510.0 (#1/276)
- APEX v1: 74.0 (#2/5)
- Vals AI CyberBench: 72.74 (#4/9)
- Epoch AI - Scicode: 59.72 (#5/171)
- Agents' Last Exam: 32.2 (#5/29)
- WebDev Arena: 1651.94 (#7/124)
- SuperCLUE-VLM - Overall: 71.8 (#7/15)
- DeepSearchQA: 89.4 (#9/16)
- Maze-Bench: 40.75 (#9/37)
- ...and 29 more
Gemini 3.7 Flash — Aggregate Rating 1924, #16
- Epoch AI - Apex Agents: 67.8 (#2/33)
- AIMultiple - Agentic RAG Routing: 85.3 (#2/49)
- Chatbot Arena (Text - Spanish): 1501.0 (#3/292)
- Chatbot Arena (Text - Polish): 1508.0 (#3/229)
- AIMultiple - Text-to-SQL (Strict Execution Match): 59.4 (#3/49)
- Chatbot Arena (Text - Japanese): 1508.0 (#4/271)
- Benchmarks.bio - BioSecBench-Surveillance: 41.5 (#5/5)
- Chatbot Arena (Text - Korean): 1482.0 (#6/276)
- AIMultiple - Text-to-SQL (Adjudicated): 79.1 (#7/49)
- Vals AI IOI: 67.83 (#10/33)
- ...and 4 more
Grok 4.7 — Aggregate Rating 1916, #18
- AutomationBench-AA - Finance - Raw Objective Completion: 93.49 (#1/191)
- LLM Stats (SWE-Marathon): 46.0 (#1/6)
- Next.js Agent Evals (OpenCode) - Success Rate: 94.0 (#1/13)
- AutomationBench-AA - Strict: 37.75 (#4/191)
- AA-Briefcase - DOCX - Rubric Pass Rate: 57.99 (#4/195)
- FrontierSWE V2: 29.5 (#4/10)
- ChessBench GitHub - Elo: 2600.0 (#4/38)
- Enkrypt AI - Risk Score: 9.78 (#4/298)
- AA-Briefcase - Analytical Quality Elo: 1994.09 (#5/195)
- Vending-Bench 2: 10536.83 (#5/66)
- ...and 228 more
Grok 4.6 — Aggregate Rating 1910, #19
- Supabase Evals (No Skills): 81.2 (#3/6)
- Supabase Evals (Skills): 84.1 (#4/6)
- Vals AI BioMysteryBench: 72.22 (#5/16)
- Vals AI CyberBench: 66.01 (#6/9)
- SuperCLUE-VLM - Overall: 63.55 (#9/15)
- Agents on Rails: 16.7 (#18/25)
- Vals AI IOI: 47.61 (#21/33)
- Kaggle Cipher Frontlines: 0.0 (#42/42)
Qwen 3.8 Max (0902) — Aggregate Rating 1908, #20
- WebDev Arena (Fullstack): 1681.0 (#3/70)
- SuperCLUE-VLM - Overall: 76.45 (#3/15)
- NatureBench: 17.78 (#5/23)
- MLCR-AA - Accuracy: 86.63 (#16/92)
- MLCR-AA: 20.0 (#27/92)
- MLCR-AA - Completeness: 22.09 (#35/92)
- Epoch AI - Critpt: 17.71 (#47/181)
- Epoch AI - Scicode: 52.08 (#53/171)
- MLCR-AA - Conciseness: 95.56 (#58/92)
- DuelLab Overall: 33.6 (#106/200)
GPT-5.5 — Aggregate Rating 1904, #22
- FrontierCode: 56.66 (#18/60)
- ParseBench: 67.76 (#36/106)
- SpeechMap Compliance: 40.1 (#325/398)
Kimi K3 — Aggregate Rating 1899, #24
- ForecastBench Baseline - Zero-Shot - Dataset: 61.1 (#2/81)
- ForecastBench Baseline - Zero-Shot - Overall: 61.2 (#3/81)
- SuperCLUE-VLM - Overall: 73.8 (#6/15)
- ForecastBench Baseline - Zero-Shot - Market: 61.2 (#9/81)
- Vals AI IOI: 48.94 (#20/33)
- DuelLab Overall: 39.6 (#70/200)
- ForecastBench: 65.6 (#75/255)
- ForecastBench Tournament - Market: 70.9 (#94/255)
Qwen 3.8 Max — Aggregate Rating 1897, #25
- E-Commerce Bench: 472.0 (#5/21)
- Mercor APEX: 63.3 (#9/46)
Step 5 Preview — Aggregate Rating 1896, #26
- Epoch AI - Scicode: 58.91 (#9/171)
- LLM2014 Logic 2026-09: 49.87 (#22/56)
- WebDev Arena (Brand & Marketing): 1592.0 (#26/133)
- MLCR-AA - Accuracy: 82.76 (#26/92)
- WebDev Arena (Gaming): 1626.0 (#27/133)
- WebDev Arena (React): 1578.0 (#28/118)
- Chatbot Arena (Code): 1565.0 (#29/134)
- WebDev Arena (Frontend): 1575.0 (#29/134)
- WebDev Arena (Reference-Based Design): 1580.0 (#29/133)
- WebDev Arena (Simulations): 1549.0 (#36/133)
- ...and 7 more
Claude Opus 4.8 — Aggregate Rating 1892, #27
- FrontierCode: 59.6 (#16/60)
DeepSeek V4.1 Flash — Aggregate Rating 1886, #29
- Vals AI CyberBench: 73.69 (#2/9)
- Design Arena (ASCII Art): 1312.0 (#5/89)
- Chatbot Arena (Text - Spanish): 1500.0 (#5/292)
- Chatbot Arena (Text - Math): 1509.0 (#7/393)
- ComplexConstraints: 49.7 (#8/83)
- Vals AI BioMysteryBench: 67.78 (#10/16)
- SuperCLUE-VLM - Overall: 56.49 (#11/15)
- RuneBench: 7761.0 (#12/58)
- Chatbot Arena (Text - Mathematical): 1504.0 (#13/390)
- WebDev Arena (Fullstack): 1622.0 (#13/70)
- ...and 112 more
GPT-5.6 Terra — Aggregate Rating 1885, #30
- BALROG BabaIsAI (LLM): 97.5 (#2/39)
- BALROG BabyAI (LLM): 98.0 (#4/39)
- MedXpertQA MM (Natomy) - Treatment: 78.12 (#5/9)
- BALROG MiniHack (LLM): 37.5 (#5/39)
- BALROG NetHack (LLM): 3.3 (#5/39)
- Vals AI IOI: 87.61 (#5/33)
- Riemann-bench: 66.4 (#5/44)
- MedXpertQA MM (Natomy): 72.0 (#6/12)
- MedXpertQA MM (Natomy) - Diagnosis: 70.64 (#6/9)
- MedXpertQA MM (Natomy) - Basic Science: 68.84 (#6/9)
- ...and 6 more
GLM-5.3 — Aggregate Rating 1884, #31
- E-Commerce Bench: 391.0 (#7/21)
- Vals AI IOI: 68.44 (#9/33)
- Epoch AI - Apex Agents: 56.6 (#11/33)
- AIMultiple - Agentic RAG Routing: 79.3 (#14/49)
- AIMultiple - Text-to-SQL (Adjudicated): 74.6 (#18/49)
- Epoch AI - Lmca: 55.51 (#29/172)
- AIMultiple - Text-to-SQL (Strict Execution Match): 35.2 (#33/49)
- DataBench: 16.3 (#45/64)
- AutomationBench-AA - Sales - Raw Objective Completion: 76.88 (#47/191)
- AutomationBench-AA - Finance - Raw Objective Completion: 80.33 (#48/191)
- ...and 55 more
MiMo-V2.6-Pro — Aggregate Rating 1882, #32
- AutomationBench-AA - Marketing - Raw Objective Completion: 91.11 (#1/191)
- AA GDP.pdf - HR - Criterion Pass Rate: 90.13 (#2/183)
- LLM Stats (CyberGym): 94.0 (#2/20)
- LLM Stats (Terminal-Bench 2.1): 89.9 (#2/42)
- AA Long Context Reasoning: 86.33 (#3/547)
- LLM Stats (SEC-bench Pro): 66.3 (#3/7)
- Vals AI CyberBench: 72.86 (#3/9)
- LLM Stats (ExploitGym): 17.8 (#4/8)
- Chatbot Arena (Text - Expert): 1535.0 (#5/359)
- Vals AI Public Benefits Bench: 68.94 (#5/42)
- ...and 156 more
Gemini 3.6 Flash — Aggregate Rating 1878, #33
- Kaggle Cipher Frontlines: 90.91 (#5/42)
- Epoch AI - Apex Agents: 46.9 (#22/33)
- ParseBench: 70.03 (#26/106)
- Vals AI IOI: 35.06 (#28/33)
- ForecastBench: 65.7 (#68/255)
- ForecastBench Tournament - Market: 70.8 (#99/255)
GPT-5.6 Sol (Codex) — Aggregate Rating 1876, #34
- Android Bench: 19.3 (#35/43)
Claude Opus 4.7 — Aggregate Rating 1875, #35
- AppWorld Challenge: 85.6 (#1/12)
- AppWorld Normal: 88.7 (#2/12)
- LLM Arena RU: 1151.0 (#7/112)
- Kaggle Cipher Frontlines: 81.82 (#11/42)
- FrontierCode: 53.93 (#26/60)
Muse Spark 1.1 — Aggregate Rating 1868, #37
- APEX v1: 73.8 (#4/5)
- Mercor APEX: 30.3 (#40/46)
Gemini 3.1 Pro (Preview) — Aggregate Rating 1867, #38
- LLMEval-Logic Formalization Free: 44.3 (#2/14)
- LLMEval-Logic Base: 73.3 (#4/14)
- LLMEval-Logic Hard: 33.5 (#4/14)
- LLMEval-Logic Hard Sub-Q: 71.3 (#4/14)
- LLMEval-Logic Formalization Fixed: 54.5 (#7/14)
- Epoch AI - Dtbench: 97.07 (#13/210)
- Epoch AI - Lmca: 52.8 (#36/172)
- Mercor APEX: 35.3 (#38/46)
- ParseBench: 60.82 (#62/106)
GPT-6 Luna — Aggregate Rating 1866, #39
- OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Non-Violent Wrongdoing: 100.0 (#1/4)
- OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Self-Harm: 96.0 (#1/4)
- OpenAI GPT-6 Astra System Card - Image Input - Self-Harm: 99.9 (#1/6)
- OpenAI GPT-6 Astra System Card - Static Jailbreaks - Cyber: 87.5 (#2/5)
- OpenAI GPT-6 Astra System Card - Cyber Safety - Semi-Synthetic Agentic Environments: 0.987 (#2/5)
- OpenAI GPT-6 Astra System Card - Challenging Prompts - Non-Violent Illicit Behavior: 99.7 (#2/6)
- OpenAI GPT-6 Astra System Card - Challenging Prompts - Extremism: 97.9 (#2/6)
- OpenAI GPT-6 Astra System Card - U18 - Eating Disorders: 87.1 (#2/7)
- OpenAI GPT-6 Astra System Card - U18 - Emotional Reliance: 94.6 (#2/7)
- OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Sensitive Personal Data: 79.5 (#2/4)
- ...and 304 more
Grok 4.5 — Aggregate Rating 1865, #40
- Vals AI IOI: 40.56 (#24/33)
- Kaggle Cipher Frontlines: 0.0 (#40/42)
Gemini 3.5 Flash — Aggregate Rating 1864, #41
- Kaggle Cipher Frontlines: 90.91 (#4/42)
- ParseBench: 74.02 (#12/106)
- Epoch AI - Apex Agents: 27.5 (#31/33)
- DuelLab Overall: 24.1 (#161/200)
Hy4 preview — Aggregate Rating 1863, #42
- ReLE - Language and Instruction Following: 78.8 (#1/185)
- ReLE - Finance: 87.9 (#5/184)
- ReLE - Education - Middle School Subjects: 58.8 (#5/185)
- Epoch AI - Proofbench: 75.0 (#8/33)
- ReLE - Education - Gaokao: 64.9 (#9/185)
- ReLE - Education - High School Subjects: 74.0 (#12/185)
- ReLE - Overall: 74.7 (#15/168)
- ReLE - Agents and Tool Use: 68.2 (#15/158)
- ReLE - Law and Civil Service: 85.3 (#15/184)
- ReLE - Medicine - Basic Medical Knowledge: 85.7 (#15/184)
- ...and 19 more
Muse Spark 1.2 — Aggregate Rating 1860, #44
- APEX v1: 75.6 (#1/5)
- DeepSearchQA: 85.9 (#12/16)
- WebDev Arena (Fullstack): 1580.0 (#28/70)
- Epoch AI - Apex Agents: 36.4 (#28/33)
- Vals AI IOI: 21.78 (#30/33)
Claude Sonnet 5 — Aggregate Rating 1855, #46
- Kaggle Cipher Frontlines: 63.64 (#19/42)
- FrontierCode: 56.18 (#21/60)
- Vals AI IOI: 45.0 (#22/33)
- Epoch AI - Lmca: 50.02 (#47/172)
- Epoch AI - Dtbench: 89.51 (#72/210)
DeepSeek V4 Pro (0813) — Aggregate Rating 1854, #47
- Vals AI IOI: 51.61 (#19/33)
- Epoch AI - Apex Agents: 47.3 (#21/33)
- Epoch AI - Dtbench: 93.87 (#45/210)
- Epoch AI - Lmca: 45.45 (#68/172)
Seed 2.1 Pro — Aggregate Rating 1854, #48
- SuperCLUE-VLM - Overall: 73.84 (#5/15)
Claude Opus 4.6 — Aggregate Rating 1853, #49
- AppWorld Normal: 91.1 (#1/12)
- AppWorld Challenge: 72.4 (#3/12)
- MCP Atlas: 76.8 (#21/65)
- FrontierCode: 43.67 (#38/60)
MiMo-V2.6-Flash — Aggregate Rating 1851, #51
- LLM Stats (CyberGym): 95.1 (#1/20)
- Vals AI CyberBench: 75.36 (#1/9)
- LLM Stats (SEC-bench Pro): 47.5 (#7/7)
- LLM Stats (ExploitBench): 25.3 (#8/8)
- LLM Stats (ExploitGym): 6.0 (#8/8)
- LLM Stats (OSWorld-Verified): 80.8 (#8/26)
- LLM Stats (Program Bench): 26.0 (#8/11)
- Vals AI Harvey Legal Agent Bench: 11.25 (#8/69)
- LLM Stats (Terminal-Bench 2.1): 87.6 (#9/42)
- Vals AI Public Benefits Bench: 67.59 (#11/42)
- ...and 80 more
GPT-5.3 Codex — Aggregate Rating 1847, #55
- Vals AI IOI: 53.83 (#17/33)
GPT-5.4 (OpenClaw) — Aggregate Rating 1844, #57
- WildClawBench: 50.3 (#20/72)
Qwen3.8 2.4T A95B — Aggregate Rating 1842, #58
- Icelandic LLM - Case Government: 80.31 (#25/55)
- Icelandic LLM - Famous Phrases: 11.34 (#29/55)
- Icelandic LLM - Generative Quality: 62.77 (#32/55)
- Icelandic LLM - GEC: 44.95 (#33/55)
- Icelandic LLM - Idiom Completion: 29.61 (#34/55)
- Icelandic LLM Leaderboard - Average: 44.14 (#36/55)
- Icelandic LLM - Textile Terms: 49.0 (#36/55)
- Icelandic LLM - WikiQA: 35.05 (#40/55)
- Icelandic LLM - Song Lyrics: 11.78 (#41/55)
- Icelandic LLM - Idiom Meanings: 57.41 (#41/55)
- ...and 1 more
GPT-5.6 Luna — Aggregate Rating 1841, #59
- OpenAI GPT-6 Astra System Card - U18 - Self-Harm: 99.0 (#2/7)
- AppWorld Challenge: 73.4 (#2/12)
- OpenAI GPT-6 Astra System Card - U18 - Age-Restricted Goods and Dangerous Activities: 76.0 (#4/7)
- AppWorld Normal: 85.1 (#4/12)
- OpenAI GPT-6 Astra System Card - U18 - Eating Disorders: 70.2 (#5/7)
- OpenAI GPT-6 Astra System Card - U18 - Emotional Reliance: 93.1 (#6/7)
- OpenAI GPT-6 Astra System Card - U18 - Sexual Content: 92.2 (#7/7)
- MedXpertQA MM (Natomy): 69.0 (#7/12)
- MedXpertQA MM (Natomy) - Diagnosis: 68.06 (#7/9)
- MedXpertQA MM (Natomy) - Treatment: 71.65 (#7/9)
- ...and 7 more
GPT-5.4 — Aggregate Rating 1838, #61
- MCP Atlas: 70.6 (#34/65)
- WebDev Arena (Fullstack): 1417.0 (#66/70)
Gemini 3 Pro (Preview) — Aggregate Rating 1833, #62
- MCP Atlas: 70.3 (#36/65)
GLM-5.2 — Aggregate Rating 1833, #63
- Mercor APEX: 45.2 (#29/46)
- DataBench: 13.3 (#58/64)
GLM-5.3 Flash — Aggregate Rating 1833, #65
- SuperCLUE-VLM - Overall: 69.94 (#8/15)
- E-Commerce Bench: 258.0 (#12/21)
- NatureBench: 11.11 (#12/23)
- Epoch AI - Apex Agents: 52.8 (#15/33)
- Vals AI IOI: 52.5 (#18/33)
- Icelandic LLM - Song Lyrics: 13.46 (#24/55)
- Chatbot Arena (Text - Korean): 1443.0 (#29/276)
- Enkrypt AI - Bias Risk: 52.45 (#29/298)
- Icelandic LLM - Famous Phrases: 8.2 (#36/55)
- Icelandic LLM - WikiQA: 35.05 (#41/55)
- ...and 14 more
Qwen3.8-Flash-Next — Aggregate Rating 1832, #66
- SEA-HELM (English) - MMLU Redux: 92.44 (#1/58)
- SEA-HELM (English) - MuSR: 84.16 (#1/58)
- SEA-HELM (English) - Knowledge: 77.84 (#1/58)
- SEA-HELM (Indonesian) - Syntax (LLM Judge): 88.17 (#1/58)
- SEA-HELM (Indonesian) - Global MMLU Lite: 87.46 (#1/58)
- SEA-HELM (Indonesian) - Question Answering: 85.9 (#1/58)
- SEA-HELM (Indonesian) - Syntax: 76.39 (#1/58)
- SEA-HELM (Indonesian) - Toxicity Detection: 57.58 (#1/58)
- SEA-HELM (Indonesian) - Linguistic Diagnostics: 79.7 (#1/58)
- SEA-HELM (Indonesian) - Safety: 64.72 (#1/58)
- ...and 112 more
DeepSeek-V4-Flash-Vision-Exp — Aggregate Rating 1828, #69
- Tinybird AI SQL Benchmark - Exactness: 50.0 (#135/334)
- Tinybird AI SQL Benchmark - Success Rate: 100.0 (#183/340)
- Tinybird AI SQL Benchmark - First-Attempt Success Rate: 92.0 (#211/340)
Qwen 3.7 Max — Aggregate Rating 1826, #70
- OpenRouter GPQA Diamond: 90.9 (#14/145)
- OpenRouter Tau2-Bench Airline: 72.0 (#55/132)
GPT-5 Pro — Aggregate Rating 1823, #72
- Architectural RedlineBench: 24.2 (#17/28)
DeepSeek V4 Flash (0731) — Aggregate Rating 1822, #73
- Vals AI IOI: 32.72 (#29/33)
Gemini 3 Flash (Preview) — Aggregate Rating 1821, #74
- MCP Atlas: 62.0 (#48/65)
Claude Opus 4.5 — Aggregate Rating 1820, #75
- Architectural RedlineBench: 26.1 (#15/28)
- MCP Atlas: 69.8 (#38/65)
- DuelLab Overall: 39.7 (#68/200)
Kimi K2.7 Code — Aggregate Rating 1819, #77
- Mercor APEX: 37.6 (#36/46)
Claude Sonnet 4.6 — Aggregate Rating 1813, #80
- Mercor APEX: 43.0 (#31/46)
- MCP Atlas: 69.5 (#40/65)
- FrontierCode: 40.0 (#43/60)
Gemini 3 Flash — Aggregate Rating 1810, #83
- ParseBench: 75.05 (#8/106)
- OmniDocBench 1.5: 90.37 (#21/81)
GPT-5 — Aggregate Rating 1805, #85
- Architectural RedlineBench: 23.3 (#18/28)
- DuelLab Overall: 28.7 (#133/200)
GPT-5.2 — Aggregate Rating 1801, #87
- Architectural RedlineBench: 32.1 (#13/28)
- AIMultiple - Agentic RAG Routing: 74.5 (#21/49)
- AIMultiple - Text-to-SQL (Adjudicated): 70.2 (#33/49)
- MCP Atlas: 67.6 (#43/65)
- AIMultiple - Text-to-SQL (Strict Execution Match): 29.6 (#46/49)
- OmniDocBench 1.5: 85.75 (#52/81)
Kimi K2.6 — Aggregate Rating 1801, #88
- APEX-Agents-AA: 28.5 (#19/57)
GLM-5.1 — Aggregate Rating 1798, #89
- Epoch AI - Apex Agents: 40.9 (#25/33)
- MCP Atlas: 75.6 (#26/65)
- Mercor APEX: 40.9 (#33/46)
Seed 2.0 Pro — Aggregate Rating 1796, #92
- LLMEval-Logic Formalization Free: 30.9 (#9/14)
- LLMEval-Logic Base: 56.2 (#11/14)
- LLMEval-Logic Hard: 6.1 (#11/14)
- LLMEval-Logic Hard Sub-Q: 39.4 (#11/14)
- LLMEval-Logic Formalization Fixed: 47.6 (#11/14)
Qwen 3.8 27B — Aggregate Rating 1795, #94
- Mercor APEX: 47.5 (#25/46)
- Vals AI IOI: 39.06 (#27/33)
- Epoch AI - ECI: 149.38 (#52/268)
- Epoch AI - Dtbench: 88.0 (#76/210)
- Epoch AI - Lmca: 41.36 (#81/172)
- Chatbot Arena (Text - Polish): 1438.0 (#83/229)
- AA GDP.pdf - Construction - Criterion Pass Rate: 56.51 (#103/183)
- AA GDP.pdf - Engineering - Criterion Pass Rate: 44.79 (#103/183)
- AA GDP.pdf - Real Estate - Criterion Pass Rate: 58.54 (#105/183)
- AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 50.9 (#118/183)
- ...and 8 more
Hy3 — Aggregate Rating 1792, #96
- AIMultiple - Agentic RAG Routing: 77.7 (#16/49)
- AIMultiple - Text-to-SQL (Strict Execution Match): 40.8 (#21/49)
- AIMultiple - Text-to-SQL (Adjudicated): 72.0 (#25/49)
- Chatbot Arena (Text - Polish): 1461.0 (#50/229)
- Chatbot Arena (Text - Korean): 1406.0 (#67/276)
O3 Pro — Aggregate Rating 1783, #105
- MCP Atlas: 44.5 (#62/65)
O3 — Aggregate Rating 1782, #106
- Architectural RedlineBench: 15.8 (#20/28)
- DuelLab Overall: 22.2 (#169/200)
GPT-5.1 — Aggregate Rating 1780, #108
- M3-Bench - PD: 97.0 (#1/12)
- M3-Bench - UG: 95.0 (#1/12)
- M3-Bench - RPD: 96.0 (#1/12)
- M3-Bench - PGG: 93.0 (#1/12)
- M3-Bench - CPR: 94.0 (#1/12)
- M3-Bench - AC: 91.0 (#1/12)
- M3-Bench - KP: 90.0 (#1/12)
- M3-Bench - L1: 93.0 (#1/12)
- M3-Bench - L2: 89.0 (#1/12)
- M3-Bench - L3: 90.3 (#1/12)
- ...and 9 more
Inkling — Aggregate Rating 1780, #109
- Icelandic LLM - Idiom Completion: 48.87 (#23/55)
- Icelandic LLM - Flora & Fauna: 29.79 (#25/55)
- Icelandic LLM - WikiQA: 47.63 (#27/55)
- Icelandic LLM - Famous Phrases: 12.18 (#27/55)
- Icelandic LLM - Song Lyrics: 12.81 (#28/55)
- Epoch AI - Apex Agents: 33.8 (#30/33)
- Icelandic LLM - Case Government: 71.25 (#32/55)
- Icelandic LLM Leaderboard - Average: 46.52 (#34/55)
- Vals AI Terminal-Bench 4.0: 0.0 (#34/34)
- Icelandic LLM - Idiom Meanings: 65.9 (#35/55)
- ...and 4 more
GPT-5.4 (2026-03-05) — Aggregate Rating 1780, #110
- Vals AI Harvey Legal Agent Bench: 0.0 (#66/69)
MiniMax-M3 — Aggregate Rating 1773, #115
- Epoch AI - Apex Agents: 37.7 (#26/33)
- Mercor APEX: 37.7 (#35/46)
Claude Sonnet 4.5 — Aggregate Rating 1772, #116
- Architectural RedlineBench: 27.0 (#14/28)
- MCP Atlas: 59.5 (#51/65)
Claude Opus 4.1 — Aggregate Rating 1771, #117
- BenchLM: 36.6 (#124/194)
DeepSeek V4 Flash — Aggregate Rating 1770, #120
- Mercor APEX: 55.3 (#14/46)
Qwen 3.5 397B A17B — Aggregate Rating 1769, #122
- Epoch AI - Apex Agents: 24.9 (#32/33)
Kimi K2.5 — Aggregate Rating 1767, #124
- MCP Atlas: 64.4 (#46/65)
DeepSeek V4 Pro Preview — Aggregate Rating 1766, #126
- SWE-Milestone: 26.38 (#22/27)
MiMo-V2.5-Pro — Aggregate Rating 1765, #128
- DuelLab Overall: 39.6 (#69/200)
Inkling Small — Aggregate Rating 1764, #129
- Vals AI Terminal-Bench 4.0: 1.51 (#31/34)
- Icelandic LLM - Idiom Completion: 16.73 (#41/55)
- Icelandic LLM - Case Government: 43.12 (#42/55)
- Icelandic LLM - Famous Phrases: 6.03 (#43/55)
- Icelandic LLM - Textile Terms: 33.6 (#43/55)
- Icelandic LLM Leaderboard - Average: 29.48 (#44/55)
- Icelandic LLM - WikiQA: 26.5 (#45/55)
- Icelandic LLM - Flora & Fauna: 7.18 (#45/55)
- Icelandic LLM - Idiom Meanings: 50.38 (#45/55)
- Icelandic LLM - Generative Quality: 43.35 (#47/55)
- ...and 4 more
Gemini 2.5 Pro — Aggregate Rating 1763, #130
- OmniDocBench 1.5: 88.03 (#41/81)
GLM-5V Turbo — Aggregate Rating 1762, #133
- Chatbot Arena (Text - German): 1411.0 (#112/306)
Gemini 3.5 Flash Lite — Aggregate Rating 1758, #136
- Mercor APEX: 29.3 (#41/46)
- Kaggle Cipher Frontlines: 0.0 (#41/42)
- ParseBench: 60.64 (#63/106)
Grok 4.20 — Aggregate Rating 1753, #143
- Kaggle Cipher Frontlines: 81.82 (#12/42)
Qwen 3.6 27B — Aggregate Rating 1747, #154
- SEA-HELM (Indonesian) - Natural Language Inference: 83.57 (#1/58)
- SEA-HELM (Malay) - SEA-NLI: 71.14 (#1/58)
- SEA-HELM (Burmese) - Natural Language Inference: 60.72 (#1/58)
- SEA-HELM (Burmese) - SEA-NLI: 68.1 (#1/58)
- SEA-HELM (Tamil) - Causal Reasoning: 95.87 (#1/58)
- SEA-HELM (Tamil) - Natural Language Inference: 57.76 (#1/58)
- SEA-HELM (Tamil) - NLR: 76.81 (#1/58)
- SEA-HELM (Thai) - NLR: 79.45 (#1/58)
- SEA-HELM (Thai) - NLU: 64.67 (#1/58)
- SEA-HELM (Vietnamese) - Causal Reasoning: 95.24 (#1/58)
- ...and 112 more
Muse Glimmer 30B — Aggregate Rating 1746, #155
- SEA-HELM (English) - AA-LCR: 81.42 (#1/58)
- SEA-HELM (Indonesian) - Metaphor Understanding: 89.85 (#1/58)
- SEA-HELM (Indonesian) - SEA-Safeguard: 78.42 (#1/58)
- SEA-HELM (Indonesian) - NLU: 83.99 (#1/58)
- SEA-HELM (Malay) - SEA-Safeguard: 78.95 (#1/58)
- SEA-HELM (Burmese) - Belebele QA: 85.88 (#1/58)
- SEA-HELM (Burmese) - NLU: 67.54 (#1/58)
- SEA-HELM (Tamil) - Sentiment Analysis: 98.19 (#1/58)
- SEA-HELM (English) - MathArena: 51.92 (#2/58)
- SEA-HELM (Indonesian) - SEA-MT-Bench (LLM Judge): 87.98 (#2/58)
- ...and 113 more
GPT-5.4 Mini — Aggregate Rating 1741, #159
- Vision2Code: 4.1 (#2/13)
- FrontierCode: 43.01 (#41/60)
- MCP Atlas: 56.7 (#57/65)
Gemini 3.1 Flash Lite — Aggregate Rating 1736, #165
- Beyond Function Calling: 41.6 (#6/12)
- ParseBench: 58.13 (#69/106)
GPT-5 Mini — Aggregate Rating 1735, #167
- ParseBench: 51.52 (#81/106)
Qwen 3.5 122B A10B — Aggregate Rating 1735, #168
- SEA-HELM (English) - SuperGPQA: 63.33 (#1/58)
- SEA-HELM (Indonesian) - SEA-MT-Bench (LLM Judge): 88.03 (#1/58)
- SEA-HELM (Malay) - SEA-MT-Bench (LLM Judge): 86.35 (#1/58)
- SEA-HELM (Thai) - Sentiment Analysis: 42.82 (#1/58)
- SEA-HELM (English) - MMLU Redux: 92.25 (#2/58)
- SEA-HELM (English) - Knowledge: 77.79 (#2/58)
- SEA-HELM (Indonesian) - Causal Reasoning: 97.04 (#2/58)
- SEA-HELM (Indonesian) - Global MMLU Lite: 86.01 (#2/58)
- SEA-HELM (Indonesian) - NLR: 90.07 (#2/58)
- SEA-HELM (Malay) - Belebele QA: 93.94 (#2/58)
- ...and 112 more
Gemma 4 31B (IT) — Aggregate Rating 1730, #173
- SEA-HELM (English) - IFEval: 94.92 (#1/58)
- SEA-HELM (English) - LiveCodeBench v6: 80.09 (#1/58)
- SEA-HELM (English) - MathArena: 56.21 (#1/58)
- SEA-HELM (Indonesian) - SEA-IFEval: 97.75 (#1/58)
- SEA-HELM (Indonesian) - Causal Reasoning: 97.54 (#1/58)
- SEA-HELM (Indonesian) - NLR: 90.35 (#1/58)
- SEA-HELM (Malay) - Belebele QA: 94.85 (#1/58)
- SEA-HELM (Malay) - SEA-IFEval: 90.72 (#1/58)
- SEA-HELM (Malay) - Sentiment Analysis: 53.45 (#1/58)
- SEA-HELM (Malay) - NLU: 74.15 (#1/58)
- ...and 114 more
Ling-3.0-flash-VL — Aggregate Rating 1730, #175
- OpenRouter Tau2-Bench Airline: 72.0 (#54/132)
- OpenRouter GPQA Diamond: 82.2 (#71/145)
O1 — Aggregate Rating 1729, #178
- Architectural RedlineBench: 11.2 (#22/28)
Qwen 3.5 27B — Aggregate Rating 1727, #182
- SEA-HELM (Burmese) - SEA-Safeguard: 89.24 (#1/58)
- SEA-HELM (Tamil) - SEA-IFEval: 95.7 (#1/58)
- SEA-HELM (Indonesian) - Natural Language Inference: 83.23 (#2/58)
- SEA-HELM (Burmese) - SEA-IFEval: 85.71 (#2/58)
- SEA-HELM (Tamil) - NLR: 75.31 (#2/58)
- SEA-HELM (Thai) - NLU: 64.55 (#2/58)
- SEA-HELM (English) - SuperGPQA: 60.01 (#3/58)
- SEA-HELM (English) - Knowledge: 75.65 (#3/58)
- SEA-HELM (Indonesian) - SEA-MT-Bench (LLM Judge): 87.29 (#3/58)
- SEA-HELM (Indonesian) - Global MMLU Lite: 85.23 (#3/58)
- ...and 112 more
GLM-4.7 — Aggregate Rating 1726, #183
- Chatbot Arena (Text - Japanese): 1440.0 (#42/271)
- MCP Atlas: 58.1 (#53/65)
Motif 3 — Aggregate Rating 1724, #185
- AA-Briefcase - Presentation Elo: 1077.56 (#93/195)
- AA-Briefcase - DOCX - Rubric Pass Rate: 28.99 (#96/195)
- AA-Briefcase - Analytical Quality Elo: 909.0 (#101/195)
- AA-Briefcase - XLSX - Rubric Pass Rate: 23.64 (#101/195)
- AA-Briefcase: 24.46 (#104/195)
- AA-Briefcase - PDF - Rubric Pass Rate: 23.28 (#109/195)
- AA-Briefcase - PPTX - Rubric Pass Rate: 19.4 (#114/195)
- AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 49.65 (#123/183)
- AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 59.93 (#129/183)
- AA GDP.pdf - HR - Criterion Pass Rate: 58.56 (#130/183)
- ...and 9 more
DeepSeek V3.2 — Aggregate Rating 1723, #187
- M3-Bench - GE: 88.0 (#2/12)
- M3-Bench - CPR: 92.0 (#2/12)
- M3-Bench - AC: 84.0 (#5/12)
- M3-Bench - PGG: 87.0 (#6/12)
- M3-Bench - SH: 90.0 (#7/12)
- M3-Bench - KP: 80.0 (#7/12)
- M3-Bench - VD: 81.0 (#9/12)
- M3-Bench - L2: 84.2 (#9/12)
- M3-Bench - L3: 83.8 (#9/12)
- M3-Bench - UG: 85.0 (#10/12)
- ...and 7 more
Gemma 4 31B — Aggregate Rating 1722, #191
- Kaggle Cipher Frontlines: 63.64 (#17/42)
Gemini 3.1 Flash Lite (Preview) — Aggregate Rating 1722, #192
- Vision2Code: 3.17 (#7/13)
- CheXpercept: 87.1 (#8/14)
- CC-OCR V2: 61.36 (#8/15)
- BehaviorBench: 31.3 (#19/20)
Step 3.7 Flash — Aggregate Rating 1720, #195
- SuperCLUE-VLM - Overall: 60.41 (#10/15)
Nemotron 3 Ultra — Aggregate Rating 1715, #206
- SEAL - MCP Atlas: 63.1 (#26/34)
- Mercor APEX: 22.7 (#44/46)
Solar Pro 4 — Aggregate Rating 1709, #217
- WebDev Arena (Fullstack): 1428.0 (#63/70)
- WebDev Arena (Content Creation Tools): 1362.0 (#96/129)
- Chatbot Arena (Text - Korean): 1370.0 (#109/276)
- Chatbot Arena (Text - Math): 1417.0 (#125/393)
- Chatbot Arena (Text - Mathematical): 1418.0 (#133/390)
- Chatbot Arena (Text - Spanish): 1396.0 (#147/292)
- Chatbot Arena (Text - French): 1402.0 (#155/287)
- Chatbot Arena (Text - Japanese): 1303.0 (#159/271)
- Chatbot Arena (Text - Chinese): 1417.0 (#165/385)
- Chatbot Arena (Text - German): 1344.0 (#175/306)
- ...and 2 more
Kimi K2 — Aggregate Rating 1703, #226
- BenchLM: 26.4 (#174/194)
Qwen 3.6 35B A3B — Aggregate Rating 1702, #228
- SEA-HELM (Indonesian) - Sentiment Analysis: 81.84 (#1/58)
- SEA-HELM (Tamil) - Pragmatics: 80.83 (#1/58)
- SEA-HELM (Tamil) - Linguistic Diagnostics: 85.18 (#1/58)
- SEA-HELM (Thai) - Natural Language Inference: 61.28 (#1/58)
- SEA-HELM (Thai) - SEA-NLI: 77.4 (#2/58)
- SEA-HELM (Filipino) - SEA-NLI: 66.26 (#2/58)
- SEA-HELM (Vietnamese) - Natural Language Inference: 62.92 (#2/58)
- SEA-HELM (Vietnamese) - SEA-NLI: 78.22 (#2/58)
- SEA-HELM (Vietnamese) - NLR: 78.54 (#2/58)
- SEA-HELM (Tamil) - SEA-Safeguard: 81.76 (#3/58)
- ...and 112 more
Qwen 3.5 35B A3B — Aggregate Rating 1698, #236
- SEA-HELM (Indonesian) - Pragmatics: 77.65 (#1/58)
- SEA-HELM (Burmese) - SEA-IFEval: 86.01 (#1/58)
- SEA-HELM (Vietnamese) - SEA-MT-Bench (LLM Judge): 87.3 (#1/58)
- SEA-HELM (Indonesian) - Question Answering: 84.73 (#2/58)
- SEA-HELM (Vietnamese) - SEA-Safeguard: 81.0 (#2/58)
- SEA-HELM (Indonesian) - SEA-IFEval: 97.05 (#3/58)
- SEA-HELM (Tamil) - SEA-IFEval: 95.04 (#3/58)
- SEA-HELM (Tamil) - Pragmatics: 73.2 (#3/58)
- SEA-HELM (Filipino) - SEA-IFEval: 92.94 (#3/58)
- SEA-HELM (Vietnamese) - SEA-IFEval: 95.75 (#3/58)
- ...and 112 more
Claude 3.7 Sonnet — Aggregate Rating 1697, #239
- PandaBench: 95.9432 (#15/490)
GPT-4.1 — Aggregate Rating 1694, #247
- Architectural RedlineBench: 7.6 (#26/28)
Claude Haiku 4.5 — Aggregate Rating 1690, #250
- MCP Atlas: 40.2 (#64/65)
O3 Mini — Aggregate Rating 1689, #256
- PandaBench: 95.9286 (#16/490)
Gemma 4 26B A4B (IT) — Aggregate Rating 1676, #270
- SEA-HELM (Thai) - Causal Reasoning: 98.06 (#1/58)
- SEA-HELM (English) - IFEval: 94.8 (#2/58)
- SEA-HELM (English) - LiveCodeBench v6: 78.29 (#2/58)
- SEA-HELM (Indonesian) - SEA-IFEval: 97.16 (#2/58)
- SEA-HELM (Indonesian) - Syntax (LLM Judge): 85.65 (#2/58)
- SEA-HELM (Indonesian) - Toxicity Detection: 54.83 (#2/58)
- SEA-HELM (Malay) - SEA-IFEval: 90.27 (#2/58)
- SEA-HELM (Malay) - SEA-MT-Bench (LLM Judge): 86.3 (#2/58)
- SEA-HELM (Burmese) - Toxicity Detection: 69.82 (#2/58)
- SEA-HELM (Burmese) - Safety: 77.96 (#2/58)
- ...and 112 more
Qwen 3 235B A22B — Aggregate Rating 1676, #272
- VinCLAT (Catalan): 17.48 (#2/18)
A.X-K2 — Aggregate Rating 1674, #275
- AA GDP.pdf - HR - Criterion Pass Rate: 66.15 (#100/183)
- AA GDP.pdf - Healthcare - Criterion Pass Rate: 62.22 (#118/183)
- AA GDP.pdf - Legal - Criterion Pass Rate: 64.51 (#122/183)
- AA GDP.pdf: 6.4 (#123/183)
- AA GDP.pdf - Real Estate - Criterion Pass Rate: 53.72 (#123/183)
- AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 62.91 (#125/183)
- AA GDP.pdf - Construction - Criterion Pass Rate: 52.03 (#126/183)
- AA GDP.pdf - Insurance - Criterion Pass Rate: 57.19 (#130/183)
- AA GDP.pdf - Criterion Pass Rate: 52.18 (#132/183)
- AA GDP.pdf - Engineering - Criterion Pass Rate: 30.63 (#132/183)
- ...and 2 more
Qwen3 Coder Next — Aggregate Rating 1673, #276
- OrgForge-IT: 80.0 (#5/10)
Gemma 4 12B (IT) — Aggregate Rating 1670, #283
- SEA-HELM (Filipino) - SEA-IFEval: 93.6 (#2/58)
- SEA-HELM (Filipino) - NLU: 76.4 (#2/58)
- SEA-HELM (English) - LiveCodeBench v6: 74.24 (#3/58)
- SEA-HELM (Indonesian) - Metaphor Understanding: 85.23 (#3/58)
- SEA-HELM (Malay) - SEA-IFEval: 90.12 (#3/58)
- SEA-HELM (Malay) - Sentiment Analysis: 52.86 (#3/58)
- SEA-HELM (Burmese) - Sentiment Analysis: 47.56 (#3/58)
- SEA-HELM (Burmese) - NLG: 56.23 (#3/58)
- SEA-HELM (Thai) - SEA-IFEval: 94.64 (#3/58)
- SEA-HELM (Filipino) - Causal Reasoning: 94.18 (#3/58)
- ...and 112 more
Qwen 3 Coder 480B A35B Instruct — Aggregate Rating 1670, #284
- Do Coding Agents Understand Least-Privilege Au: 63.3 (#7/11)
- ALE-Bench: 461.45 (#69/89)
- Tinybird AI SQL Benchmark - Exactness: 51.48 (#102/334)
- Bullshit Benchmark: 20.0 (#119/194)
- Tinybird AI SQL Benchmark - First-Attempt Success Rate: 94.0 (#152/340)
- Tinybird AI SQL Benchmark - Success Rate: 98.0 (#204/340)
GLM-4.6V — Aggregate Rating 1659, #301
- Chatbot Arena (Text - Medicine & Healthcare): 1402.0 (#175/379)
GPT-OSS-120B — Aggregate Rating 1658, #304
- SEA-HELM (English) - MathArena: 45.48 (#4/58)
- SEA-HELM (English) - LiveCodeBench v6: 68.18 (#5/58)
- SEA-HELM (Tamil) - SEA-MT-Bench (LLM Judge): 79.98 (#7/58)
- SEA-HELM (Filipino) - SEA-Safeguard: 75.7 (#8/58)
- SEA-HELM (English) - MuSR: 77.64 (#9/58)
- SEA-HELM (Thai) - SEA-MT-Bench (LLM Judge): 83.21 (#10/58)
- SEA-HELM (Indonesian) - Syntax (LLM Judge): 73.99 (#11/58)
- SEA-HELM (Indonesian) - SEA-MT-Bench (LLM Judge): 85.28 (#11/58)
- SEA-HELM (Indonesian) - Sentiment Analysis: 79.13 (#11/58)
- SEA-HELM (Thai) - Sentiment Analysis: 39.37 (#11/58)
- ...and 114 more
Mistral Medium 3.5 — Aggregate Rating 1657, #308
- SEA-HELM (Filipino) - SEA-Safeguard: 79.65 (#1/58)
- SEA-HELM (Vietnamese) - Question Answering: 82.49 (#3/58)
- SEA-HELM (Vietnamese) - Summarization: 18.14 (#3/58)
- SEA-HELM (Vietnamese) - NLU: 71.23 (#4/58)
- SEA-HELM (Vietnamese) - NLG: 55.73 (#4/58)
- SEA-HELM (Indonesian) - NLG: 55.78 (#6/58)
- SEA-HELM (Thai) - SEA-Safeguard: 74.96 (#6/58)
- SEA-HELM (Filipino) - NLG: 56.98 (#6/58)
- SEA-HELM (English) - IFEval: 91.25 (#7/58)
- SEA-HELM (Malay) - Belebele QA: 93.44 (#7/58)
- ...and 114 more
Grok Code Fast 1 — Aggregate Rating 1653, #315
- BenchLM: 32.0 (#144/194)
LongCat-Flash-Chat — Aggregate Rating 1652, #319
- NYT Connections Older Models: 5.2 (#99/114)
Claude 3.5 Sonnet — Aggregate Rating 1644, #334
- PandaBench: 98.25 (#1/490)
- BenchLM: 29.9 (#162/194)
Gemini 2.0 Flash — Aggregate Rating 1642, #337
- PandaBench: 89.9545 (#133/490)
GPT-4.1 Mini — Aggregate Rating 1638, #346
- AppWorld Challenge: 37.6 (#8/12)
- AppWorld Normal: 61.9 (#9/12)
- BenchLM: 29.1 (#164/194)
Qwen 3.5 9B — Aggregate Rating 1636, #349
- SEA-HELM (Tamil) - SEA-Safeguard: 83.91 (#1/58)
- SEA-HELM (Malay) - SEA-Safeguard: 77.17 (#4/58)
- SEA-HELM (Burmese) - SEA-Safeguard: 83.01 (#7/58)
- SEA-HELM (Indonesian) - Natural Language Inference: 81.45 (#8/58)
- SEA-HELM (English) - AA-LCR: 64.65 (#9/58)
- SEA-HELM (Burmese) - SEA-IFEval: 80.7 (#9/58)
- SEA-HELM (Vietnamese) - Question Answering: 80.01 (#9/58)
- SEA-HELM (English) - SuperGPQA: 46.07 (#10/58)
- SEA-HELM (Indonesian) - SEA-IFEval: 94.7 (#10/58)
- SEA-HELM (Indonesian) - NLR: 88.16 (#10/58)
- ...and 113 more
NVIDIA-Nemotron-3-Super-120B-A12B-BF16 — Aggregate Rating 1636, #352
- SEA-HELM (Filipino) - Kalahi (LLM Judge): 74.27 (#6/58)
- SEA-HELM (English) - LiveCodeBench v6: 66.89 (#7/58)
- SEA-HELM (English) - MuSR: 78.36 (#7/58)
- SEA-HELM (English) - MMLU Redux: 90.66 (#9/58)
- SEA-HELM (English) - Knowledge: 68.23 (#10/58)
- SEA-HELM (Filipino) - Cultural: 71.72 (#10/58)
- SEA-HELM (English) - SuperGPQA: 45.81 (#11/58)
- SEA-HELM (Filipino) - Kalahi: 87.56 (#11/58)
- SEA-HELM (English) - MathArena: 37.63 (#12/58)
- SEA-HELM (Indonesian) - Global MMLU Lite: 82.57 (#12/58)
- ...and 112 more
Mercury 2 — Aggregate Rating 1635, #355
- WebDev Arena (Simulations): 1212.0 (#125/133)
Mistral Large 3 — Aggregate Rating 1630, #368
- LM Market Cap LMC Score: 67.0 (#189/468)
Gemma 4 26B A4B — Aggregate Rating 1628, #371
- Kaggle Cipher Frontlines: 63.64 (#18/42)
GPT-5 Nano — Aggregate Rating 1623, #381
- RealWorldQA: 74.12 (#20/49)
Mercury 2.5 — Aggregate Rating 1622, #389
- Epoch AI - Proofbench: 3.0 (#32/33)
- Vals AI IOI: 2.44 (#33/33)
- Vals AI Harvey Legal Agent Bench: 0.0 (#54/69)
- AA GDP.pdf - Insurance - Criterion Pass Rate: 53.86 (#135/183)
- AA GDP.pdf - HR - Criterion Pass Rate: 55.91 (#137/183)
- AA GDP.pdf - Legal - Criterion Pass Rate: 57.19 (#137/183)
- AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 54.75 (#142/183)
- AutomationBench-AA - Operations - Raw Objective Completion: 8.25 (#142/191)
- AA GDP.pdf - Criterion Pass Rate: 47.63 (#145/183)
- AA GDP.pdf - STEM/Research - Criterion Pass Rate: 49.0 (#145/183)
- ...and 49 more
GPT-4 Turbo — Aggregate Rating 1620, #394
- BenchLM: 21.6 (#181/194)
MiniMax M1 80k — Aggregate Rating 1616, #402
- BenchLM: 24.9 (#177/194)
Qwen 3 32B — Aggregate Rating 1614, #406
- SEA-HELM (English) - LiveCodeBench v6: 69.34 (#4/58)
- SEA-HELM (Tamil) - Pragmatics: 72.3 (#4/58)
- SEA-HELM (Indonesian) - Metaphor Understanding: 84.5 (#5/58)
- VinCLAT (Catalan): 6.1 (#5/18)
- SEA-HELM (Thai) - Toxicity Detection: 45.63 (#7/58)
- SEA-HELM (English) - MathArena: 40.01 (#8/58)
- SEA-HELM (Malay) - Safety: 42.82 (#8/58)
- SEA-HELM (Thai) - Safety: 57.76 (#9/58)
- SEA-HELM (Indonesian) - Causal Reasoning: 95.49 (#11/58)
- SEA-HELM (Tamil) - Causal Reasoning: 89.72 (#11/58)
- ...and 114 more
Granite 4.2 30B — Aggregate Rating 1610, #420
- AutomationBench-AA - Strict: 0.15 (#150/191)
- AutomationBench-AA: 1.51 (#156/191)
- AutomationBench-AA - HR - Raw Objective Completion: 1.43 (#157/191)
- AutomationBench-AA - Marketing - Raw Objective Completion: 1.45 (#158/191)
- AutomationBench-AA - Sales - Raw Objective Completion: 3.03 (#161/191)
- AutomationBench-AA - Support - Raw Objective Completion: 5.78 (#161/191)
- AutomationBench-AA - Operations - Raw Objective Completion: 1.0 (#166/191)
- AutomationBench-AA - Finance - Raw Objective Completion: 0.4 (#167/191)
Llama 4 Maverick — Aggregate Rating 1609, #422
- BenchLM: 23.0 (#180/194)
Gemini 1.5 Pro — Aggregate Rating 1603, #438
- BenchLM: 27.9 (#171/194)
Qwen 3 30B A3B — Aggregate Rating 1598, #452
- VinCLAT (Catalan): 2.38 (#7/18)
GLM-4.7 Flash — Aggregate Rating 1592, #467
- SEA-HELM (Filipino) - Summarization: 22.53 (#9/58)
- SEA-HELM (English) - SuperGPQA: 39.42 (#16/58)
- SEA-HELM (English) - MathArena: 32.38 (#17/58)
- SEA-HELM (English) - AA-LCR: 43.26 (#19/58)
- SEA-HELM (English) - Knowledge: 61.71 (#19/58)
- SEA-HELM (Indonesian) - Global MMLU Lite: 73.97 (#20/58)
- SEA-HELM (English) - LiveCodeBench v6: 54.31 (#21/58)
- SEA-HELM (Filipino) - Sentiment Analysis: 59.76 (#22/58)
- SEA-HELM (English) - MMLU Redux: 84.0 (#23/58)
- SEA-HELM (English) - MuSR: 67.8 (#23/58)
- ...and 112 more
Command A+ — Aggregate Rating 1590, #471
- LM Market Cap LMC Score: 50.8 (#234/468)
Mistral Medium 3 — Aggregate Rating 1589, #477
- BenchLM: 28.3 (#169/194)
GPT-OSS-20B — Aggregate Rating 1587, #480
- SEA-HELM (English) - MathArena: 42.63 (#6/58)
- SEA-HELM (English) - LiveCodeBench v6: 64.91 (#8/58)
- SEA-HELM (Indonesian) - Pragmatics: 70.88 (#9/58)
- SEA-HELM (Tamil) - SEA-MT-Bench (LLM Judge): 73.75 (#14/58)
- SEA-HELM (Tamil) - Natural Language Inference: 47.71 (#16/58)
- SEA-HELM (Thai) - Toxicity Detection: 41.73 (#16/58)
- SEA-HELM (Filipino) - SEA-NLI: 58.05 (#16/58)
- SEA-HELM (Tamil) - SEA-NLI: 53.75 (#17/58)
- SEA-HELM (Tamil) - Translations: 86.29 (#18/58)
- SEA-HELM (Tamil) - Pragmatics: 59.32 (#18/58)
- ...and 113 more
Qwen 3 14B — Aggregate Rating 1583, #492
- SEA-HELM (Tamil) - Question Answering: 72.39 (#2/58)
- SEA-HELM (Tamil) - NLU: 82.79 (#3/58)
- SEA-HELM (Thai) - Toxicity Detection: 46.54 (#5/58)
- SEA-HELM (Thai) - Safety: 58.64 (#5/58)
- SEA-HELM (English) - LiveCodeBench v6: 68.05 (#6/58)
- VinCLAT (Catalan): 3.03 (#6/18)
- SEA-HELM (Filipino) - Sentiment Analysis: 64.6 (#7/58)
- SEA-HELM (Indonesian) - Sentiment Analysis: 79.52 (#10/58)
- SEA-HELM (English) - MathArena: 38.37 (#11/58)
- SEA-HELM (Indonesian) - Metaphor Understanding: 79.23 (#12/58)
- ...and 114 more
Qwen 3.5 4B — Aggregate Rating 1576, #510
- SEA-HELM (Burmese) - Summarization: 23.46 (#9/58)
- SEA-HELM (English) - AA-LCR: 63.08 (#10/58)
- SEA-HELM (Indonesian) - Natural Language Inference: 80.52 (#12/58)
- SEA-HELM (Tamil) - SEA-IFEval: 85.17 (#14/58)
- SEA-HELM (Burmese) - SEA-IFEval: 71.84 (#15/58)
- SEA-HELM (Vietnamese) - SEA-IFEval: 90.9 (#15/58)
- SEA-HELM (Thai) - Thai Exam: 65.89 (#16/58)
- SEA-HELM (Thai) - Safety: 54.56 (#16/58)
- SEA-HELM (English) - Knowledge: 61.83 (#17/58)
- SEA-HELM (Indonesian) - NLR: 85.63 (#17/58)
- ...and 112 more
gemma-4-E4B-it — Aggregate Rating 1570, #527
- SEA-HELM (Burmese) - Sentiment Analysis: 47.56 (#4/58)
- SEA-HELM (Filipino) - Sentiment Analysis: 64.45 (#8/58)
- SEA-HELM (Thai) - Natural Language Inference: 53.93 (#10/58)
- SEA-HELM (Burmese) - Toxicity Detection: 55.53 (#11/58)
- SEA-HELM (Burmese) - SEA-Safeguard: 75.96 (#11/58)
- SEA-HELM (Burmese) - Safety: 65.74 (#11/58)
- SEA-HELM (Tamil) - SEA-NLI: 55.26 (#11/58)
- SEA-HELM (Filipino) - NLU: 74.03 (#11/58)
- SEA-HELM (English) - LiveCodeBench v6: 62.7 (#12/58)
- SEA-HELM (Indonesian) - Sentiment Analysis: 79.1 (#12/58)
- ...and 112 more
Qwen 2.5 Coder 32B Instruct — Aggregate Rating 1566, #541
- BenchLM: 27.1 (#172/194)
Nova Pro — Aggregate Rating 1565, #545
- BenchLM: 20.8 (#183/194)
Claude 3.5 Haiku — Aggregate Rating 1564, #552
- PandaBench: 97.9886 (#3/490)
Granite 4.2 8B — Aggregate Rating 1554, #580
- Chatbot Arena (Text - Chinese): 1324.0 (#250/385)
- AI Chess Leaderboard (Reasoning): 468.0 (#302/332)
GPT-4o Mini — Aggregate Rating 1552, #582
- BenchLM: 26.9 (#173/194)
Llama 4 Scout — Aggregate Rating 1541, #608
- BenchLM: 29.4 (#163/194)
Ministral 3 14B — Aggregate Rating 1541, #609
- BenchLM: 25.3 (#176/194)
Nemotron 3 Nano Omni 30B-A3B — Aggregate Rating 1533, #636
- ParseBench: 48.51 (#83/106)
Llama 3 70B — Aggregate Rating 1532, #637
- BenchLM: 23.6 (#179/194)
DeepSeek R1 Distill Qwen 32B — Aggregate Rating 1531, #640
- BenchLM: 30.4 (#156/194)
Ministral 3 8B — Aggregate Rating 1530, #643
- BenchLM: 19.3 (#185/194)
Qwen 3 8B — Aggregate Rating 1528, #652
- SEA-HELM (English) - LiveCodeBench v6: 64.9 (#9/58)
- SEA-HELM (Thai) - Toxicity Detection: 44.46 (#9/58)
- VinCLAT (Catalan): 1.63 (#9/18)
- SEA-HELM (Vietnamese) - Question Answering: 79.2 (#10/58)
- SEA-HELM (English) - MathArena: 35.54 (#13/58)
- SEA-HELM (Indonesian) - Sentiment Analysis: 78.89 (#13/58)
- SEA-HELM (Vietnamese) - SEA-NLI: 71.87 (#14/58)
- SEA-HELM (Filipino) - Sentiment Analysis: 61.37 (#16/58)
- SEA-HELM (Thai) - Safety: 53.98 (#19/58)
- SEA-HELM (English) - MuSR: 71.93 (#20/58)
- ...and 114 more
GPT-4.1 Nano — Aggregate Rating 1512, #698
- DuelLab Overall: 40.1 (#67/200)
- BenchLM: 24.9 (#178/194)
Phi-4 — Aggregate Rating 1511, #702
- BenchLM: 25.9 (#175/194)
Mixtral 8x22B Instruct (v0.1) — Aggregate Rating 1508, #711
- BenchLM: 21.1 (#182/194)
OLMo 3 32B — Aggregate Rating 1503, #728
- Chatbot Arena (Text - French): 1303.0 (#228/287)
Granite 4.2 3B — Aggregate Rating 1502, #732
- Chatbot Arena (Text - Chinese): 1257.0 (#309/385)
Gemma 4 E2B (IT) — Aggregate Rating 1498, #742
- SEA-HELM (Tamil) - Question Answering: 69.04 (#7/58)
- SEA-HELM (Burmese) - Sentiment Analysis: 42.33 (#15/58)
- SEA-HELM (Burmese) - SEA-Safeguard: 66.67 (#15/58)
- SEA-HELM (Tamil) - NLU: 81.51 (#16/58)
- SEA-HELM (Indonesian) - SEA-IFEval: 90.23 (#19/58)
- SEA-HELM (Burmese) - Global MMLU Lite: 47.38 (#19/58)
- SEA-HELM (Burmese) - Safety: 47.72 (#19/58)
- SEA-HELM (Tamil) - Pragmatics: 58.66 (#20/58)
- SEA-HELM (Thai) - Toxicity Detection: 40.85 (#20/58)
- SEA-HELM (Burmese) - SEA-IFEval: 66.21 (#21/58)
- ...and 112 more
Claude 3 Haiku — Aggregate Rating 1488, #776
- BenchLM: 14.8 (#190/194)
Qwen 3 4B — Aggregate Rating 1480, #795
- VinCLAT (Catalan): 0.75 (#12/18)
Apertus-70B-Instruct-2509 — Aggregate Rating 1480, #796
- SEA-HELM (Malay) - Toxicity Detection: 16.57 (#3/58)
- SEA-HELM (Burmese) - Translations: 58.72 (#32/58)
- SEA-HELM (Burmese) - NLG: 37.28 (#32/58)
- SEA-HELM (Burmese) - Summarization: 15.85 (#34/58)
- SEA-HELM (Burmese) - SEA-Safeguard: 0.0 (#35/58)
- SEA-HELM (Tamil) - Summarization: 9.75 (#36/58)
- SEA-HELM (Burmese) - Causal Reasoning: 0.0 (#37/58)
- SEA-HELM (Filipino) - Kalahi (LLM Judge): 46.89 (#39/58)
- SEA-HELM (Tamil) - NLG: 40.24 (#40/58)
- SEA-HELM (Filipino) - Translations: 80.61 (#40/58)
- ...and 112 more
LFM2.5-2.6B — Aggregate Rating 1480, #797
- OpenUI Generative UI Benchmark: 3.3 (#31/31)
Gemini 1.0 Pro — Aggregate Rating 1461, #858
- BenchLM: 12.9 (#191/194)
Apertus-8B-Instruct-2509 — Aggregate Rating 1423, #1052
- Icelandic LLM - Generative Quality: 29.14 (#53/55)
- Icelandic LLM - Song Lyrics: 9.96 (#54/55)
- Icelandic LLM Leaderboard - Average: 7.73 (#55/55)
- Icelandic LLM - WikiQA: 6.39 (#55/55)
- Icelandic LLM - Famous Phrases: 0.12 (#55/55)
- Icelandic LLM - Textile Terms: 2.6 (#55/55)
- Icelandic LLM - Flora & Fauna: 0.2 (#55/55)
- Icelandic LLM - GEC: 2.29 (#55/55)
- Icelandic LLM - Case Government: 2.19 (#55/55)
- Icelandic LLM - Idiom Completion: 0.53 (#55/55)
- ...and 1 more
Ministral 3 3B — Aggregate Rating 1423, #1053
- BenchLM: 18.1 (#189/194)
Qwen 3 1.7B — Aggregate Rating 1379, #1214
- VinCLAT (Catalan): 0.15 (#15/18)
Mistral-7B-v0.3 — Aggregate Rating 1361, #1278
- BenchLM: 8.1 (#193/194)
LFM2 24B A2B — Aggregate Rating 1345, #1322
- BenchLM: 19.9 (#184/194)
Qwen 3 0.6B — Aggregate Rating 1293, #1419
- VinCLAT (Catalan): 0.05 (#17/18)
LFM2.5-1.2B Instruct — Aggregate Rating 1291, #1424
- BenchLM: 18.3 (#188/194)
LFM2.5-1.2B — Aggregate Rating 1276, #1447
- BenchLM: 18.6 (#187/194)
MiniCPM5-1B — Aggregate Rating 1276, #1449
- BenchLM: 4.1 (#194/194)

Top-10 New Scores (699)

Claude Fable 5 on Chatbot Arena (Search): 1230.0 (#5)
Claude Fable 5 on Epoch AI - Dtbench: 97.33 (#9)
Claude Fable 5 on Epoch AI - Frontierswe: 46.96 (#4)
Claude Fable 5 on Epoch AI - Lmca: 61.15 (#16)
Claude Fable 5 on ForecastBench: 65.9 (#64)
Claude Fable 5 on ForecastBench Tournament - Market: 72.0 (#44)
Claude Fable 5 on FrontierCode: 64.25 (#5)
Claude Fable 5 on WebDev Arena: 1627.98 (#8)
Claude Fable 5.1 on AIMultiple - Agentic RAG Routing: 83.7 (#6)
Claude Fable 5.1 on AIMultiple - Text-to-SQL (Adjudicated): 82.8 (#2)
Claude Fable 5.1 on AIMultiple - Text-to-SQL (Strict Execution Match): 54.7 (#5)
Claude Fable 5.1 on APEX v1: 73.8 (#3)
Claude Fable 5.1 on ActiveVision: 15.3 (#4)
Claude Fable 5.1 on ActiveVision (Agents): 76.5 (#3)
Claude Fable 5.1 on Agents on Rails: 31.7 (#5)
Claude Fable 5.1 on Chatbot Arena (Text - German): 1486.0 (#20)
Claude Fable 5.1 on Chatbot Arena (Text - Japanese): 1555.0 (#1)
Claude Fable 5.1 on Chatbot Arena (Text - Polish): 1485.0 (#23)
Claude Fable 5.1 on Chatbot Arena (Text - Spanish): 1479.0 (#17)
Claude Fable 5.1 on DataBench: 27.0 (#31)
Claude Fable 5.1 on Design Arena (Agents - Mobile Apps): 1268.0 (#4)
Claude Fable 5.1 on Epoch AI - Dtbench: 97.6 (#6)
Claude Fable 5.1 on Epoch AI - Lmca: 65.32 (#2)
Claude Fable 5.1 on SuperCLUE-VLM - Overall: 75.13 (#4)
Claude Fable 5.1 on UGI - Natural Intelligence: 70.11 (#39)
Claude Fable 5.1 on UGI - Willingness (W/10): 4.2 (#899)
Claude Fable 5.1 on UGI - Writing: 63.86 (#75)
Claude Fable 5.1 on UGI Leaderboard: 52.08 (#95)
Claude Fable 5.1 on Vals AI CyberBench: 70.42 (#5)
Claude Fable 5.1 on Vals AI IOI: 90.78 (#4)
Claude Opus 5 on ActiveVision: 10.6 (#6)
Claude Opus 5 on ActiveVision (Agents): 68.2 (#4)
Claude Opus 5 on Agents on Rails: 31.7 (#6)
Claude Opus 5 on BALROG BabaIsAI (LLM): 95.8 (#4)
Claude Opus 5 on BALROG BabyAI (LLM): 100.0 (#1)
Claude Opus 5 on BALROG Crafter (LLM): 68.2 (#2)
Claude Opus 5 on BALROG MiniHack (LLM): 37.5 (#4)
Claude Opus 5 on BALROG NetHack (LLM): 7.3 (#2)
Claude Opus 5 on BALROG TextWorld (LLM): 71.6 (#2)
Claude Opus 5 on DeepSearchQA: 90.4 (#7)
Claude Opus 5 on Epoch AI - Dtbench: 97.87 (#2)
Claude Opus 5 on Epoch AI - Frontierswe: 52.01 (#3)
Claude Opus 5 on Epoch AI - Lmca: 64.45 (#5)
Claude Opus 5 on ForecastBench: 66.1 (#53)
Claude Opus 5 on ForecastBench Tournament - Market: 72.2 (#38)
Claude Opus 5 on Vals AI CyberBench: 65.36 (#7)
Claude Opus 5 on Vals AI IOI: 84.33 (#6)
Claude Opus 5.5 on AA CritPt: 31.71 (#2)
Claude Opus 5.5 on AA GDP.pdf: 28.8 (#6)
Claude Opus 5.5 on AA GDP.pdf - Construction - Criterion Pass Rate: 75.95 (#10)
Claude Opus 5.5 on AA GDP.pdf - Criterion Pass Rate: 82.25 (#1)
Claude Opus 5.5 on AA GDP.pdf - Engineering - Criterion Pass Rate: 84.35 (#1)
Claude Opus 5.5 on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 91.25 (#1)
Claude Opus 5.5 on AA GDP.pdf - HR - Criterion Pass Rate: 90.16 (#1)
Claude Opus 5.5 on AA GDP.pdf - Healthcare - Criterion Pass Rate: 93.1 (#1)
Claude Opus 5.5 on AA GDP.pdf - Insurance - Criterion Pass Rate: 88.05 (#4)
Claude Opus 5.5 on AA GDP.pdf - Legal - Criterion Pass Rate: 84.79 (#10)
Claude Opus 5.5 on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 79.36 (#7)
Claude Opus 5.5 on AA GDP.pdf - Real Estate - Criterion Pass Rate: 72.66 (#21)
Claude Opus 5.5 on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 81.47 (#1)
Claude Opus 5.5 on AA GDPval: 1846.17 (#1)
Claude Opus 5.5 on AA Humanity's Last Exam: 61.35 (#1)
Claude Opus 5.5 on AA Long Context Reasoning: 84.67 (#5)
Claude Opus 5.5 on AA MMMU-Pro: 87.69 (#1)
Claude Opus 5.5 on AA Omniscience: 46.42 (#1)
Claude Opus 5.5 on AA SciCode: 66.9 (#1)
Claude Opus 5.5 on AA-Briefcase: 61.21 (#2)
Claude Opus 5.5 on AA-Briefcase - Analytical Quality Elo: 2206.92 (#1)
Claude Opus 5.5 on AA-Briefcase - DOCX - Rubric Pass Rate: 60.36 (#1)
Claude Opus 5.5 on AA-Briefcase - PDF - Rubric Pass Rate: 64.66 (#2)
Claude Opus 5.5 on AA-Briefcase - PPTX - Rubric Pass Rate: 62.04 (#2)
Claude Opus 5.5 on AA-Briefcase - Presentation Elo: 1709.91 (#1)
Claude Opus 5.5 on AA-Briefcase - XLSX - Rubric Pass Rate: 61.14 (#1)
Claude Opus 5.5 on AA-Omniscience Accuracy: 66.22 (#2)
Claude Opus 5.5 on AA-Omniscience Hallucination Rate: 58.61 (#130)
Claude Opus 5.5 on AA-Omniscience Index - Business: 32.7 (#7)
Claude Opus 5.5 on AA-Omniscience Index - Health: 26.4 (#2)
Claude Opus 5.5 on AA-Omniscience Index - Humanities & Social Sciences: 44.7 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Law: 48.1 (#3)
Claude Opus 5.5 on AA-Omniscience Index - Science, Engineering & Mathematics: 37.8 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE): 88.8 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - C: 96.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Dart: 90.0 (#2)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Go: 88.0 (#7)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - HTML: 88.0 (#4)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Java: 80.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 88.18 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Julia: 84.0 (#8)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 96.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - PHP: 96.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Python: 91.5 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - R: 72.0 (#2)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Rust: 92.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Swift: 92.0 (#4)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 93.33 (#3)
Claude Opus 5.5 on AI Chess Leaderboard (Continuation): 1486.0 (#13)
Claude Opus 5.5 on AI Coding Daily (Claude Code) - Laravel Code Quality: 18.95 (#1)
Claude Opus 5.5 on AI Coding Daily (Claude Code) - React-TS Code Quality: 19.67 (#1)
Claude Opus 5.5 on AI Coding Daily (Claude Code) - Total: 57.83 (#1)
Claude Opus 5.5 on ARC-AGI-1: 98.5 (#5)
Claude Opus 5.5 on ARC-AGI-2: 93.33 (#4)
Claude Opus 5.5 on ActiveVision: 41.2 (#2)
Claude Opus 5.5 on ActiveVision (Agents): 82.4 (#2)
Claude Opus 5.5 on Agent Security League - Functional Correctness: 68.7 (#16)
Claude Opus 5.5 on Agent Security League - Security Correctness: 33.5 (#3)
Claude Opus 5.5 on Agents on Rails: 33.3 (#3)
Claude Opus 5.5 on Aiera Leaderboard - Depth: 71.1 (#16)
Claude Opus 5.5 on Aiera Leaderboard - Facts: 61.08 (#23)
Claude Opus 5.5 on Aiera Leaderboard - Grounding: 76.2 (#27)
Claude Opus 5.5 on Aiera Leaderboard - Research Score: 69.46 (#22)
Claude Opus 5.5 on Artificial Analysis Intelligence Index: 57.62 (#1)
Claude Opus 5.5 on AutomationBench-AA: 69.54 (#1)
Claude Opus 5.5 on AutomationBench-AA - Finance - Raw Objective Completion: 88.25 (#11)
Claude Opus 5.5 on AutomationBench-AA - HR - Raw Objective Completion: 77.27 (#1)
Claude Opus 5.5 on AutomationBench-AA - Marketing - Raw Objective Completion: 91.07 (#2)
Claude Opus 5.5 on AutomationBench-AA - Operations - Raw Objective Completion: 96.74 (#2)
Claude Opus 5.5 on AutomationBench-AA - Sales - Raw Objective Completion: 89.49 (#2)
Claude Opus 5.5 on AutomationBench-AA - Strict: 41.25 (#2)
Claude Opus 5.5 on AutomationBench-AA - Support - Raw Objective Completion: 91.11 (#5)
Claude Opus 5.5 on BenchLM: 87.0 (#2)
Claude Opus 5.5 on Bug Hunt Bench: 41.7 (#5)
Claude Opus 5.5 on Bug Hunt Bench - LMS: 22.7 (#4)
Claude Opus 5.5 on Bug Hunt Bench - VS Code Extension: 19.0 (#5)
Claude Opus 5.5 on Chartography: 66.3 (#3)
Claude Opus 5.5 on Chatbot Arena (Code): 1827.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text - Business, Management & Financial Ops): 1499.0 (#7)
Claude Opus 5.5 on Chatbot Arena (Text - Chinese): 1559.0 (#2)
Claude Opus 5.5 on Chatbot Arena (Text - Coding): 1547.0 (#5)
Claude Opus 5.5 on Chatbot Arena (Text - Creative Writing): 1521.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text - English): 1508.0 (#4)
Claude Opus 5.5 on Chatbot Arena (Text - Entertainment, Sports & Media): 1493.0 (#2)
Claude Opus 5.5 on Chatbot Arena (Text - Expert): 1556.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text - Hard Prompts (English)): 1536.0 (#2)
Claude Opus 5.5 on Chatbot Arena (Text - Hard Prompts): 1541.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text - Instruction Following): 1516.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text - Legal & Government): 1495.0 (#16)
Claude Opus 5.5 on Chatbot Arena (Text - Life, Physical & Social Science): 1535.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text - Longer Query): 1530.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text - Multi-Turn): 1519.0 (#2)
Claude Opus 5.5 on Chatbot Arena (Text - Non-English): 1501.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text - Russian): 1525.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text - Software & IT Services): 1541.0 (#2)
Claude Opus 5.5 on Chatbot Arena (Text - Writing, Literature & Language): 1520.0 (#1)
Claude Opus 5.5 on Chatbot Arena (Text): 1509.0 (#1)
Claude Opus 5.5 on Chess Bench LLM: 1197.0 (#30)
Claude Opus 5.5 on ChessBench: 52.8 (#9)
Claude Opus 5.5 on ChessBench - Accuracy: 77.6 (#3)
Claude Opus 5.5 on ChessBench - Coherence: 62.9 (#22)
Claude Opus 5.5 on ChessBench - Elo: 1147.0 (#6)
Claude Opus 5.5 on ClockBench: 66.7 (#2)
Claude Opus 5.5 on Coarena: 991.6 (#11)
Claude Opus 5.5 on Coarena - Task Completion: 53.0 (#14)
Claude Opus 5.5 on ComplexConstraints: 49.9 (#7)
Claude Opus 5.5 on DataBench: 70.5 (#1)
Claude Opus 5.5 on DeepsecBench: 26.68 (#9)
Claude Opus 5.5 on Design Arena (3D): 1450.0 (#2)
Claude Opus 5.5 on Design Arena (Agents - Mobile Apps): 1268.0 (#5)
Claude Opus 5.5 on Design Arena (Website): 1361.0 (#2)
Claude Opus 5.5 on DuelLab Overall: 81.9 (#3)
Claude Opus 5.5 on EQ-Bench Longform Writing: 84.6 (#3)
Claude Opus 5.5 on Enkrypt AI - Bias Risk: 25.58 (#5)
Claude Opus 5.5 on Enkrypt AI - Insecure Code Risk: 0.44 (#4)
Claude Opus 5.5 on Enkrypt AI - Jailbreak Risk: 4.46 (#62)
Claude Opus 5.5 on Enkrypt AI - Risk Score: 8.19 (#2)
Claude Opus 5.5 on Enkrypt AI - Safety Risk: 12.03 (#5)
Claude Opus 5.5 on Enkrypt AI - Toxicity Risk: 3.09 (#136)
Claude Opus 5.5 on EnterpriseBench (CoreCraft Agents): 74.4 (#2)
Claude Opus 5.5 on Epoch AI - Ebr Bench: 71.43 (#2)
Claude Opus 5.5 on ErdosBench: 3.24 (#2)
Claude Opus 5.5 on Featherbench: 100.0 (#3)
Claude Opus 5.5 on Featherbench - Rubric Quality: 9.5 (#1)
Claude Opus 5.5 on FrontierSWE V2: 62.3 (#2)
Claude Opus 5.5 on GDP.pdf: 31.4 (#3)
Claude Opus 5.5 on Gert Labs Rankings: 73.65 (#3)
Claude Opus 5.5 on Harvey LAB-AA: 91.24 (#11)
Claude Opus 5.5 on Harvey LAB-AA - All Pass: 8.33 (#11)
Claude Opus 5.5 on Harvey LAB-AA - Antitrust & Competition: 92.03 (#9)
Claude Opus 5.5 on Harvey LAB-AA - Arbitration & International Dispute Resolution: 94.62 (#7)
Claude Opus 5.5 on Harvey LAB-AA - Banking & Finance: 94.42 (#5)
Claude Opus 5.5 on Harvey LAB-AA - Bankruptcy & Restructuring: 92.04 (#8)
Claude Opus 5.5 on Harvey LAB-AA - Capital Markets: 94.9 (#7)
Claude Opus 5.5 on Harvey LAB-AA - Corporate Governance: 90.41 (#12)
Claude Opus 5.5 on Harvey LAB-AA - Corporate M&A: 94.87 (#8)
Claude Opus 5.5 on Harvey LAB-AA - Data Privacy & Cybersecurity: 98.59 (#2)
Claude Opus 5.5 on Harvey LAB-AA - Emerging Companies & Venture Capital: 94.88 (#3)
Claude Opus 5.5 on Harvey LAB-AA - Employment & Labor: 88.39 (#8)
Claude Opus 5.5 on Harvey LAB-AA - Energy & Natural Resources: 96.04 (#13)
Claude Opus 5.5 on Harvey LAB-AA - Environmental & ESG: 86.74 (#15)
Claude Opus 5.5 on Harvey LAB-AA - Funds & Asset Management: 89.8 (#10)
Claude Opus 5.5 on Harvey LAB-AA - Healthcare & Life Sciences: 93.07 (#8)
Claude Opus 5.5 on Harvey LAB-AA - Immigration: 94.91 (#1)
Claude Opus 5.5 on Harvey LAB-AA - Insurance: 94.14 (#5)
Claude Opus 5.5 on Harvey LAB-AA - Intellectual Property: 96.11 (#7)
Claude Opus 5.5 on Harvey LAB-AA - International Trade & Sanctions: 91.27 (#7)
Claude Opus 5.5 on Harvey LAB-AA - Litigation & Dispute Resolution: 94.38 (#5)
Claude Opus 5.5 on Harvey LAB-AA - Real Estate: 94.46 (#11)
Claude Opus 5.5 on Harvey LAB-AA - Structured Finance & Securitization: 88.89 (#7)
Claude Opus 5.5 on Harvey LAB-AA - Tax: 85.28 (#16)
Claude Opus 5.5 on Harvey LAB-AA - Trusts, Estates & Private Client: 93.51 (#8)
Claude Opus 5.5 on Harvey LAB-AA - White Collar Defense & Investigations: 88.89 (#14)
Claude Opus 5.5 on HieroglyphBench: 37.4 (#7)
Claude Opus 5.5 on Icelandic LLM - Case Government: 97.5 (#6)
Claude Opus 5.5 on Icelandic LLM - Famous Phrases: 27.62 (#6)
Claude Opus 5.5 on Icelandic LLM - GEC: 80.73 (#4)
Claude Opus 5.5 on Icelandic LLM - Generative Quality: 74.54 (#13)
Claude Opus 5.5 on Icelandic LLM - Idiom Meanings: 84.04 (#3)
Claude Opus 5.5 on Icelandic LLM - Song Lyrics: 20.53 (#2)
Claude Opus 5.5 on Icelandic LLM - Textile Terms: 73.3 (#3)
Claude Opus 5.5 on InferenceBench: 12.08 (#1)
Claude Opus 5.5 on Kaggle FACTS (Google): 65.3 (#8)
Claude Opus 5.5 on Kaggle FACTS Grounding: 68.46 (#20)
Claude Opus 5.5 on Kaggle FACTS Multimodal: 48.37 (#3)
Claude Opus 5.5 on Kaggle FACTS Parametric: 57.87 (#11)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Arabic: 95.5 (#3)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Bengali: 94.75 (#4)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Burmese: 92.5 (#10)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Chinese: 95.5 (#5)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Culturally Agnostic: 95.53 (#4)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Culturally Sensitive: 95.19 (#2)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - French: 96.25 (#2)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - German: 96.0 (#2)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Hindi: 94.75 (#7)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Indonesian: 95.25 (#5)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Italian: 96.5 (#2)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Japanese: 95.0 (#5)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Korean: 95.25 (#4)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Portuguese: 96.5 (#2)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Spanish: 96.5 (#4)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Yoruba: 93.25 (#5)
Claude Opus 5.5 on Kaggle Open Benchmarks - AIME 2025: 100.0 (#6)
Claude Opus 5.5 on KernelBench Hub - CUDA: 79.38 (#2)
Claude Opus 5.5 on LLM Stats (AutomationBench): 40.0 (#7)
Claude Opus 5.5 on LLM Stats (BioMysteryBench): 89.3 (#2)
Claude Opus 5.5 on LLM Stats (DeepSWE 1.1): 74.2 (#2)
Claude Opus 5.5 on LLM Stats (HealthBench Professional): 65.6 (#2)
Claude Opus 5.5 on LLM Stats (Legal Agent Benchmark): 8.3 (#3)
Claude Opus 5.5 on LLM Stats (OfficeQA Pro): 67.7 (#3)
Claude Opus 5.5 on LM Market Cap LMC Score: 95.1 (#5)
Claude Opus 5.5 on LOL Arena: 49.3 (#5)
Claude Opus 5.5 on Legal Benchmarks - Contract Workflows - Task Pass Rate: 48.4 (#1)
Claude Opus 5.5 on Legal Benchmarks - Data Extraction - Form: 82.2 (#2)
Claude Opus 5.5 on LiveBench AMPS Hard: 99.0 (#11)
Claude Opus 5.5 on LiveBench Code Completion: 86.96 (#3)
Claude Opus 5.5 on LiveBench Code Generation: 91.55 (#2)
Claude Opus 5.5 on LiveBench Connections: 99.33 (#27)
Claude Opus 5.5 on LiveBench Consecutive Events: 90.56 (#5)
Claude Opus 5.5 on LiveBench Integrals With Game: 100.0 (#3)
Claude Opus 5.5 on LiveBench JavaScript: 81.82 (#2)
Claude Opus 5.5 on LiveBench Logic With Navigation: 86.0 (#6)
Claude Opus 5.5 on LiveBench Math Comp: 97.06 (#17)
Claude Opus 5.5 on LiveBench Olympiad: 93.25 (#1)
Claude Opus 5.5 on LiveBench Paraphrase: 64.68 (#39)
Claude Opus 5.5 on LiveBench Plot Unscrambling: 77.48 (#6)
Claude Opus 5.5 on LiveBench Python: 70.0 (#8)
Claude Opus 5.5 on LiveBench Simplify: 63.0 (#33)
Claude Opus 5.5 on LiveBench Spatial: 98.0 (#34)
Claude Opus 5.5 on LiveBench Story Generation: 71.18 (#32)
Claude Opus 5.5 on LiveBench Summarize: 69.33 (#35)
Claude Opus 5.5 on LiveBench Table Join: 52.33 (#8)
Claude Opus 5.5 on LiveBench Table Reformat: 98.04 (#47)
Claude Opus 5.5 on LiveBench Theory of Mind: 84.61 (#13)
Claude Opus 5.5 on LiveBench TypeScript: 63.33 (#1)
Claude Opus 5.5 on LiveBench Typos: 82.0 (#31)
Claude Opus 5.5 on LiveBench Zebra Puzzle: 100.0 (#13)
Claude Opus 5.5 on MLCR-AA: 66.67 (#2)
Claude Opus 5.5 on MLCR-AA - Accuracy: 93.15 (#6)
Claude Opus 5.5 on MLCR-AA - Completeness: 88.36 (#2)
Claude Opus 5.5 on MLCR-AA - Conciseness: 81.11 (#86)
Claude Opus 5.5 on MathArena - ARXIV August: 85.96 (#4)
Claude Opus 5.5 on MathArena - ARXIV June: 83.33 (#8)
Claude Opus 5.5 on MathArena - ARXIV_FALSE August: 77.38 (#5)
Claude Opus 5.5 on MathArena - ARXIV_FALSE June: 100.0 (#1)
Claude Opus 5.5 on MathArena Arxiv: 81.67 (#6)
Claude Opus 5.5 on MathArena Arxiv False: 95.0 (#1)
Claude Opus 5.5 on Maze-Bench: 85.17 (#2)
Claude Opus 5.5 on MineBench: 2145.0 (#2)
Claude Opus 5.5 on Multi-turn Debate (Lechmazur): 1701.0 (#4)
Claude Opus 5.5 on NYT Connections Extended: 88.5 (#28)
Claude Opus 5.5 on Next.js Agent Evals (Claude Code) - Success Rate: 97.0 (#2)
Claude Opus 5.5 on Next.js Agent Evals (Claude Code) - Success Rate with AGENTS.md: 97.0 (#3)
Claude Opus 5.5 on OpenRouter GPQA Diamond: 90.6 (#17)
Claude Opus 5.5 on OpenRouter Tau2-Bench Airline: 73.3 (#47)
Claude Opus 5.5 on PLCC - Art & Entertainment: 86.0 (#29)
Claude Opus 5.5 on PLCC - Culture & Tradition: 93.0 (#27)
Claude Opus 5.5 on PLCC - Geography: 94.0 (#46)
Claude Opus 5.5 on PLCC - Grammar: 94.0 (#10)
Claude Opus 5.5 on PLCC - History: 97.0 (#8)
Claude Opus 5.5 on PLCC - Overall: 92.5 (#14)
Claude Opus 5.5 on PLCC - Vocabulary: 91.0 (#16)
Claude Opus 5.5 on PRISM (1C:Enterprise) - Algorithmic Tasks (A): 93.0 (#7)
Claude Opus 5.5 on ParseBench: 78.01 (#3)
Claude Opus 5.5 on Riemann-bench: 69.6 (#3)
Claude Opus 5.5 on RuneBench: 18333.0 (#2)
Claude Opus 5.5 on SEAL - Fortress: 13.18 (#61)
Claude Opus 5.5 on SEAL - Professional Reasoning Benchmark - Legal: 47.58 (#16)
Claude Opus 5.5 on SpeechMap Compliance: 82.8 (#79)
Claude Opus 5.5 on SvelteBench: 93.3 (#62)
Claude Opus 5.5 on TaxCalcBench: 40.0 (#9)
Claude Opus 5.5 on Tinybird AI SQL Benchmark - Exactness: 61.33 (#4)
Claude Opus 5.5 on Tinybird AI SQL Benchmark - First-Attempt Success Rate: 90.0 (#216)
Claude Opus 5.5 on Tinybird AI SQL Benchmark - Success Rate: 100.0 (#40)
Claude Opus 5.5 on UGI - Natural Intelligence: 73.28 (#23)
Claude Opus 5.5 on UGI - Willingness (W/10): 4.2 (#898)
Claude Opus 5.5 on UGI - Writing: 70.01 (#22)
Claude Opus 5.5 on UGI Leaderboard: 54.46 (#55)
Claude Opus 5.5 on Vals AI BioMysteryBench: 79.26 (#2)
Claude Opus 5.5 on Vals AI Code Migration: 66.65 (#2)
Claude Opus 5.5 on Vals AI Excel Modeling: 75.94 (#2)
Claude Opus 5.5 on Vals AI Finance Agent v2: 58.59 (#8)
Claude Opus 5.5 on Vals AI Harvey Legal Agent Bench: 3.75 (#33)
Claude Opus 5.5 on Vals AI IOI: 95.06 (#2)
Claude Opus 5.5 on Vals AI Legal Research Bench: 50.48 (#4)
Claude Opus 5.5 on Vals AI MedCode: 49.8 (#15)
Claude Opus 5.5 on Vals AI MedScribe: 91.43 (#1)
Claude Opus 5.5 on Vals AI MysteryMechanism: 49.55 (#2)
Claude Opus 5.5 on Vals AI ProgramBench: 86.99 (#1)
Claude Opus 5.5 on Vals AI ProofBench: 100.0 (#1)
Claude Opus 5.5 on Vals AI Public Benefits Bench: 70.64 (#3)
Claude Opus 5.5 on Vals AI RSI Index: 0.371 (#1)
Claude Opus 5.5 on Vals AI ReverseEngBench: 33.59 (#2)
Claude Opus 5.5 on Vals AI SAGE: 45.83 (#34)
Claude Opus 5.5 on Vals AI Tax Agent Bench: 70.5 (#6)
Claude Opus 5.5 on Vals AI Terminal-Bench Science: 48.57 (#2)
Claude Opus 5.5 on Vals AI Vibe Code Bench: 90.29 (#2)
Claude Opus 5.5 on Vending-Bench 2: 9235.25 (#7)
Claude Opus 5.5 on VoxelBench: 2662.0 (#1)
Claude Opus 5.5 on WebDev Arena (Brand & Marketing): 1799.0 (#1)
Claude Opus 5.5 on WebDev Arena (Consumer Product): 1692.0 (#2)
Claude Opus 5.5 on WebDev Arena (Content Creation Tools): 1867.0 (#1)
Claude Opus 5.5 on WebDev Arena (Data & Analytics): 1700.0 (#1)
Claude Opus 5.5 on WebDev Arena (Frontend): 1863.0 (#1)
Claude Opus 5.5 on WebDev Arena (Gaming): 1940.0 (#1)
Claude Opus 5.5 on WebDev Arena (React): 1869.0 (#1)
Claude Opus 5.5 on WebDev Arena (Reference-Based Design): 1854.0 (#1)
Claude Opus 5.5 on WebDev Arena (Simulations): 1942.0 (#1)
Claude Opus 5.5 on ZeroBench: 43.0 (#2)
GPT-6 on AI Coding Daily (Codex CLI) - Laravel Code Quality: 18.6 (#1)
GPT-6 on AI Coding Daily (Codex CLI) - React-TS Code Quality: 18.17 (#1)
GPT-6 on AI Coding Daily (Codex CLI) - Total: 56.77 (#1)
GPT-6 on Agents on Rails: 53.3 (#1)
GPT-6 on Agents' Last Exam: 34.2 (#2)
GPT-6 on Benchmarks.bio - BioSecBench-Surveillance: 48.5 (#2)
GPT-6 on Chartography: 71.0 (#1)
GPT-6 on Chatbot Arena (Text - Chinese): 1516.0 (#33)
GPT-6 on Chatbot Arena (Text - French): 1510.0 (#9)
GPT-6 on Chatbot Arena (Text - Math): 1486.0 (#28)
GPT-6 on Chatbot Arena (Text - Mathematical): 1486.0 (#34)
GPT-6 on Chatbot Arena (Text - Spanish): 1441.0 (#78)
GPT-6 on ChessBench - Coherence: 85.3 (#3)
GPT-6 on ComplexConstraints: 57.7 (#1)
GPT-6 on DataBench: 55.0 (#7)
GPT-6 on DuelLab Overall: 94.8 (#1)
GPT-6 on Epoch AI - Dtbench: 97.33 (#8)
GPT-6 on Epoch AI - Frontierswe: 65.51 (#1)
GPT-6 on Epoch AI - Lmca: 64.37 (#6)
GPT-6 on Kaggle ExtractBench: 90.28 (#2)
GPT-6 on Kaggle Math Puzzles (Lewis Carroll Sorites): 87.5 (#5)
GPT-6 on Kaggle SimpleQA: 73.06 (#2)
GPT-6 on Kaggle VideoQA: 75.25 (#10)
GPT-6 on LLM Chess (Saplin): 1579.3 (#2)
GPT-6 on MLS-Bench Lite: 50.0 (#3)
GPT-6 on OpenRouter Tau2-Bench Airline: 68.0 (#76)
GPT-6 on ParseBench: 70.67 (#23)
GPT-6 on Riemann-bench: 72.0 (#2)
GPT-6 on Sonar LLM Leaderboard - Java - Bug Density: 0.56 (#5)
GPT-6 on Sonar LLM Leaderboard - Java - Code Smell Density: 18.47 (#34)
GPT-6 on Sonar LLM Leaderboard - Java - Pass Rate: 85.85 (#3)
GPT-6 on Sonar LLM Leaderboard - Java - Vulnerability Density: 0.18 (#14)
GPT-6 on Vals AI BioMysteryBench: 79.26 (#1)
GPT-6 on Vals AI CyberBench: 41.07 (#9)
GPT-6 on Vals AI IOI: 100.0 (#1)
GPT-6 on WebDev Arena (Fullstack): 1755.0 (#1)
GPT-6 on ZeroBench: 52.0 (#1)
GPT-6 Sol on AA CritPt: 30.86 (#8)
GPT-6 Sol on AA GDP.pdf: 28.0 (#8)
GPT-6 Sol on AA GDP.pdf - Construction - Criterion Pass Rate: 77.27 (#7)
GPT-6 Sol on AA GDP.pdf - Criterion Pass Rate: 77.45 (#18)
GPT-6 Sol on AA GDP.pdf - Engineering - Criterion Pass Rate: 78.48 (#11)
GPT-6 Sol on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 87.4 (#12)
GPT-6 Sol on AA GDP.pdf - HR - Criterion Pass Rate: 70.07 (#74)
GPT-6 Sol on AA GDP.pdf - Healthcare - Criterion Pass Rate: 84.45 (#32)
GPT-6 Sol on AA GDP.pdf - Insurance - Criterion Pass Rate: 69.65 (#75)
GPT-6 Sol on AA GDP.pdf - Legal - Criterion Pass Rate: 79.29 (#48)
GPT-6 Sol on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 76.28 (#19)
GPT-6 Sol on AA GDP.pdf - Real Estate - Criterion Pass Rate: 79.84 (#1)
GPT-6 Sol on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 78.79 (#11)
GPT-6 Sol on AA GDPval: 1486.93 (#35)
GPT-6 Sol on AA Humanity's Last Exam: 47.91 (#25)
GPT-6 Sol on AA Long Context Reasoning: 83.67 (#15)
GPT-6 Sol on AA MMMU-Pro: 83.29 (#22)
GPT-6 Sol on AA Omniscience: 27.12 (#34)
GPT-6 Sol on AA SciCode: 57.64 (#21)
GPT-6 Sol on AA-Briefcase: 46.87 (#35)
GPT-6 Sol on AA-Briefcase - Analytical Quality Elo: 1652.11 (#29)
GPT-6 Sol on AA-Briefcase - DOCX - Rubric Pass Rate: 43.2 (#46)
GPT-6 Sol on AA-Briefcase - PDF - Rubric Pass Rate: 42.24 (#47)
GPT-6 Sol on AA-Briefcase - PPTX - Rubric Pass Rate: 45.31 (#30)
GPT-6 Sol on AA-Briefcase - Presentation Elo: 1473.3 (#23)
GPT-6 Sol on AA-Briefcase - XLSX - Rubric Pass Rate: 49.09 (#31)
GPT-6 Sol on AA-Omniscience Accuracy: 54.48 (#35)
GPT-6 Sol on AA-Omniscience Hallucination Rate: 50.74 (#102)
GPT-6 Sol on AA-Omniscience Index - Business: 22.8 (#26)
GPT-6 Sol on AA-Omniscience Index - Health: 11.3 (#30)
GPT-6 Sol on AA-Omniscience Index - Humanities & Social Sciences: 25.0 (#37)
GPT-6 Sol on AA-Omniscience Index - Law: 25.9 (#41)
GPT-6 Sol on AA-Omniscience Index - Science, Engineering & Mathematics: 16.4 (#45)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE): 66.0 (#32)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - C: 83.0 (#21)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - Dart: 56.0 (#30)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - Go: 70.0 (#30)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - HTML: 84.0 (#11)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - Java: 43.0 (#24)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 76.36 (#22)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - Julia: 44.0 (#43)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 68.0 (#30)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - PHP: 68.0 (#28)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - Python: 74.0 (#29)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - R: 44.0 (#37)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - Rust: 84.0 (#8)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - Swift: 60.0 (#38)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 88.89 (#13)
GPT-6 Sol on AI Chess Leaderboard (Continuation): 1281.0 (#31)
GPT-6 Sol on AI Chess Leaderboard (Reasoning): 1314.0 (#37)
GPT-6 Sol on AI Coding Daily (Codex CLI) - Laravel Code Quality: 17.67 (#6)
GPT-6 Sol on AI Coding Daily (Codex CLI) - React-TS Code Quality: 17.5 (#3)
GPT-6 Sol on AI Coding Daily (Codex CLI) - Total: 52.42 (#4)
GPT-6 Sol on AIMultiple - Agentic RAG Routing: 81.5 (#12)
GPT-6 Sol on AIMultiple - Text-to-SQL (Adjudicated): 77.7 (#11)
GPT-6 Sol on AIMultiple - Text-to-SQL (Strict Execution Match): 43.1 (#18)
GPT-6 Sol on Agent Arena: 7.68 (#6)
GPT-6 Sol on Agent Arena - Bash Recovery: 8.35 (#6)
GPT-6 Sol on Agent Arena - Confirmed Success: 11.36 (#4)
GPT-6 Sol on Agent Arena - Praise vs Complaint: 2.62 (#22)
GPT-6 Sol on Agent Arena - Steerability: 15.7 (#1)
GPT-6 Sol on Agent Arena - Tool Hallucination: -0.36 (#3)
GPT-6 Sol on Agents on Rails: 21.7 (#13)
GPT-6 Sol on Agents' Last Exam: 32.2 (#3)
GPT-6 Sol on Aiera Leaderboard - Depth: 80.6 (#9)
GPT-6 Sol on Aiera Leaderboard - Facts: 61.98 (#21)
GPT-6 Sol on Aiera Leaderboard - Grounding: 82.7 (#21)
GPT-6 Sol on Aiera Leaderboard - Research Score: 75.09 (#17)
GPT-6 Sol on Artificial Analysis Intelligence Index: 47.53 (#18)
GPT-6 Sol on AutomationBench: 33.2 (#7)
GPT-6 Sol on AutomationBench-AA: 61.69 (#16)
GPT-6 Sol on AutomationBench-AA - Finance - Raw Objective Completion: 84.57 (#27)
GPT-6 Sol on AutomationBench-AA - HR - Raw Objective Completion: 75.71 (#9)
GPT-6 Sol on AutomationBench-AA - Marketing - Raw Objective Completion: 86.29 (#20)
GPT-6 Sol on AutomationBench-AA - Operations - Raw Objective Completion: 96.51 (#4)
GPT-6 Sol on AutomationBench-AA - Sales - Raw Objective Completion: 90.07 (#1)
GPT-6 Sol on AutomationBench-AA - Strict: 32.57 (#13)
GPT-6 Sol on AutomationBench-AA - Support - Raw Objective Completion: 89.76 (#10)
GPT-6 Sol on BenchLM: 81.3 (#4)
GPT-6 Sol on Blueprint-Bench 2: 0.338 (#7)
GPT-6 Sol on Bug Hunt Bench: 29.3 (#18)
GPT-6 Sol on Bug Hunt Bench - LMS: 15.0 (#26)
GPT-6 Sol on Bug Hunt Bench - VS Code Extension: 15.3 (#14)
GPT-6 Sol on Chartography: 53.6 (#5)
GPT-6 Sol on Chatbot Arena (Code): 1681.0 (#5)
GPT-6 Sol on Chatbot Arena (Text - Business, Management & Financial Ops): 1465.0 (#47)
GPT-6 Sol on Chatbot Arena (Text - Chinese): 1495.0 (#67)
GPT-6 Sol on Chatbot Arena (Text - Coding): 1524.0 (#26)
GPT-6 Sol on Chatbot Arena (Text - Creative Writing): 1451.0 (#34)
GPT-6 Sol on Chatbot Arena (Text - English): 1462.0 (#72)
GPT-6 Sol on Chatbot Arena (Text - Entertainment, Sports & Media): 1445.0 (#42)
GPT-6 Sol on Chatbot Arena (Text - Expert): 1507.0 (#29)
GPT-6 Sol on Chatbot Arena (Text - Hard Prompts (English)): 1494.0 (#40)
GPT-6 Sol on Chatbot Arena (Text - Hard Prompts): 1485.0 (#54)
GPT-6 Sol on Chatbot Arena (Text - Instruction Following): 1447.0 (#65)
GPT-6 Sol on Chatbot Arena (Text - Legal & Government): 1464.0 (#67)
GPT-6 Sol on Chatbot Arena (Text - Life, Physical & Social Science): 1470.0 (#77)
GPT-6 Sol on Chatbot Arena (Text - Longer Query): 1467.0 (#59)
GPT-6 Sol on Chatbot Arena (Text - Math): 1450.0 (#69)
GPT-6 Sol on Chatbot Arena (Text - Mathematical): 1455.0 (#71)
GPT-6 Sol on Chatbot Arena (Text - Medicine & Healthcare): 1460.0 (#89)
GPT-6 Sol on Chatbot Arena (Text - Multi-Turn): 1461.0 (#71)
GPT-6 Sol on Chatbot Arena (Text - Non-English): 1448.0 (#55)
GPT-6 Sol on Chatbot Arena (Text - Russian): 1466.0 (#49)
GPT-6 Sol on Chatbot Arena (Text - Software & IT Services): 1507.0 (#38)
GPT-6 Sol on Chatbot Arena (Text - Spanish): 1430.0 (#93)
GPT-6 Sol on Chatbot Arena (Text - Writing, Literature & Language): 1440.0 (#68)
GPT-6 Sol on Chatbot Arena (Text): 1457.0 (#60)
GPT-6 Sol on ChessBench: 62.9 (#4)
GPT-6 Sol on ChessBench - Accuracy: 74.2 (#7)
GPT-6 Sol on ChessBench - Coherence: 81.6 (#7)
GPT-6 Sol on ChessBench - Elo: 1205.0 (#4)
GPT-6 Sol on ChessBench GitHub - Elo: 2490.0 (#8)
GPT-6 Sol on ComplexConstraints: 50.8 (#5)
GPT-6 Sol on Conceptual Reasoning Index: 71.04 (#21)
GPT-6 Sol on Conceptual Reasoning Index - Argument Evaluation (LMCA): 59.08 (#23)
GPT-6 Sol on Conceptual Reasoning Index - Consistency (ACCoRD): 82.4 (#6)
GPT-6 Sol on Conceptual Reasoning Index - Decision Theory (DTBench): 95.55 (#10)
GPT-6 Sol on DataBench: 61.3 (#5)
GPT-6 Sol on DeepsecBench: 40.91 (#1)
GPT-6 Sol on Design Arena (Agents - Fullstack): 1242.0 (#15)
GPT-6 Sol on Design Arena (Data Viz): 1323.0 (#8)
GPT-6 Sol on Design Arena (Game Dev): 1319.0 (#16)
GPT-6 Sol on Design Arena (UI Components): 1286.0 (#28)
GPT-6 Sol on Design Arena (Website): 1276.0 (#36)
GPT-6 Sol on DuelLab Overall: 62.8 (#14)
GPT-6 Sol on EQ-Bench Longform Writing: 82.8 (#8)
GPT-6 Sol on Enkrypt AI - Bias Risk: 71.58 (#68)
GPT-6 Sol on Enkrypt AI - Insecure Code Risk: 18.22 (#102)
GPT-6 Sol on Enkrypt AI - Jailbreak Risk: 12.23 (#198)
GPT-6 Sol on Enkrypt AI - Risk Score: 26.07 (#80)
GPT-6 Sol on Enkrypt AI - Safety Risk: 37.85 (#245)
GPT-6 Sol on Enkrypt AI - Toxicity Risk: 8.73 (#244)
GPT-6 Sol on Epoch AI - Ebr Bench: 53.33 (#4)
GPT-6 Sol on Featherbench: 82.0 (#23)
GPT-6 Sol on Featherbench - Rubric Quality: 8.3 (#23)
GPT-6 Sol on GDP.pdf: 26.4 (#11)
GPT-6 Sol on Gert Labs Rankings: 73.82 (#2)
GPT-6 Sol on HieroglyphBench: 26.0 (#9)
GPT-6 Sol on Icelandic LLM - Case Government: 94.38 (#15)
GPT-6 Sol on Icelandic LLM - Famous Phrases: 26.06 (#9)
GPT-6 Sol on Icelandic LLM - Flora & Fauna: 39.04 (#15)
GPT-6 Sol on Icelandic LLM - GEC: 74.77 (#7)
GPT-6 Sol on Icelandic LLM - Idiom Completion: 65.87 (#12)
GPT-6 Sol on Icelandic LLM - Idiom Meanings: 81.55 (#14)
GPT-6 Sol on Icelandic LLM - Song Lyrics: 12.5 (#31)
GPT-6 Sol on Icelandic LLM - Textile Terms: 70.1 (#5)
GPT-6 Sol on Icelandic LLM - WikiQA: 58.95 (#16)
GPT-6 Sol on Icelandic LLM Leaderboard - Average: 65.63 (#8)
GPT-6 Sol on InferenceBench: 7.64 (#8)
GPT-6 Sol on Kaggle FACTS (Google): 67.95 (#4)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Arabic: 93.5 (#12)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Bengali: 94.25 (#8)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Burmese: 92.5 (#12)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Chinese: 94.0 (#12)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Culturally Agnostic: 94.41 (#10)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Culturally Sensitive: 93.78 (#10)
GPT-6 Sol on Kaggle Global-MMLU-Lite - English: 95.75 (#9)
GPT-6 Sol on Kaggle Global-MMLU-Lite - French: 95.0 (#7)
GPT-6 Sol on Kaggle Global-MMLU-Lite - German: 94.0 (#13)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Hindi: 95.25 (#4)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Indonesian: 93.25 (#12)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Italian: 96.0 (#4)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Japanese: 95.75 (#3)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Korean: 94.5 (#10)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Portuguese: 94.75 (#9)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Spanish: 93.75 (#15)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Swahili: 92.5 (#12)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Yoruba: 90.75 (#12)
GPT-6 Sol on Kaggle Open Benchmarks - AIME 2025: 100.0 (#7)
GPT-6 Sol on KernelBench Hub - CUDA: 64.47 (#5)
GPT-6 Sol on KernelBench Hub - Mega: 0.96 (#14)
GPT-6 Sol on LLM Stats (Agents' Last Exam): 56.4 (#2)
GPT-6 Sol on LLM Stats (DeepSWE 1.1): 68.8 (#12)
GPT-6 Sol on LLM Stats (FrontierCode 1.1): 49.3 (#5)
GPT-6 Sol on LLM Stats (OSWorld 2.0): 64.4 (#6)
GPT-6 Sol on LLM Stats Score: 49.39 (#21)
GPT-6 Sol on LLM2014 Logic 2026-09: 61.94 (#8)
GPT-6 Sol on LM Market Cap LMC Score: 40.0 (#255)
GPT-6 Sol on LOL Arena: 47.9 (#19)
GPT-6 Sol on LiveBench AMPS Hard: 98.0 (#39)
GPT-6 Sol on LiveBench Code Completion: 80.44 (#25)
GPT-6 Sol on LiveBench Code Generation: 83.1 (#9)
GPT-6 Sol on LiveBench Connections: 100.0 (#16)
GPT-6 Sol on LiveBench Consecutive Events: 89.55 (#13)
GPT-6 Sol on LiveBench Integrals With Game: 99.0 (#7)
GPT-6 Sol on LiveBench JavaScript: 63.64 (#43)
GPT-6 Sol on LiveBench Logic With Navigation: 74.0 (#30)
GPT-6 Sol on LiveBench Math Comp: 97.06 (#19)
GPT-6 Sol on LiveBench Olympiad: 91.36 (#22)
GPT-6 Sol on LiveBench Paraphrase: 70.08 (#25)
GPT-6 Sol on LiveBench Plot Unscrambling: 77.9 (#5)
GPT-6 Sol on LiveBench Python: 55.0 (#34)
GPT-6 Sol on LiveBench Simplify: 69.28 (#16)
GPT-6 Sol on LiveBench Spatial: 96.0 (#50)
GPT-6 Sol on LiveBench Story Generation: 66.93 (#48)
GPT-6 Sol on LiveBench Summarize: 68.0 (#42)
GPT-6 Sol on LiveBench Table Join: 54.02 (#5)
GPT-6 Sol on LiveBench Table Reformat: 100.0 (#22)
GPT-6 Sol on LiveBench Theory of Mind: 84.61 (#15)
GPT-6 Sol on LiveBench TypeScript: 40.0 (#34)
GPT-6 Sol on LiveBench Typos: 78.0 (#46)
GPT-6 Sol on LiveBench Zebra Puzzle: 100.0 (#15)
GPT-6 Sol on MLCR-AA: 16.11 (#40)
GPT-6 Sol on MLCR-AA - Accuracy: 96.67 (#2)
GPT-6 Sol on MLCR-AA - Completeness: 16.11 (#61)
GPT-6 Sol on MLCR-AA - Conciseness: 100.0 (#1)
GPT-6 Sol on MathArena - ARXIV August: 92.98 (#1)
GPT-6 Sol on MathArena - ARXIV June: 90.97 (#2)
GPT-6 Sol on MathArena - ARXIV_FALSE August: 78.57 (#3)
GPT-6 Sol on MathArena - ARXIV_FALSE June: 95.37 (#4)
GPT-6 Sol on MathArena Arxiv: 90.0 (#2)
GPT-6 Sol on MathArena Arxiv False: 87.0 (#4)
GPT-6 Sol on Maze-Bench: 58.01 (#4)
GPT-6 Sol on Mercor APEX: 54.3 (#18)
GPT-6 Sol on NYT Connections Extended: 90.1 (#22)
GPT-6 Sol on Next.js Agent Evals (Codex) - Success Rate: 97.0 (#1)
GPT-6 Sol on Next.js Agent Evals (Codex) - Success Rate with AGENTS.md: 97.0 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Adversarial User Simulations - Emotional Reliance: 96.9 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Adversarial User Simulations - Mental Health: 99.7 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Adversarial User Simulations - Self-Harm: 97.3 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Age-Restricted Actions: 75.5 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Non-Violent Wrongdoing: 99.0 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Self-Harm: 92.0 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Violent Wrongdoing: 88.9 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Human Red-Teaming Chat Plugins: 93.6 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Human Red-Teaming Codex: 93.5 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Biology Refusal - Benign: 96.4 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Biology Refusal - Severe: 99.8 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Challenging Prompts - Extremism: 93.6 (#5)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Challenging Prompts - Hate: 98.2 (#6)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Challenging Prompts - Non-Violent Illicit Behavior: 99.3 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Challenging Prompts - Self-Harm: 98.6 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Challenging Prompts - Violent Illicit Behavior: 98.8 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Cyber Safety - Production Chat: 0.957 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Cyber Safety - Semi-Synthetic Agentic Environments: 0.984 (#4)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Cyber Safety - Synthetic Agentic Environments: 0.998 (#1)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Image Input - Erotic: 99.8 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Image Input - Extremism: 97.5 (#6)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Image Input - Hate: 99.8 (#4)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Image Input - Self-Harm: 98.2 (#6)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - SHP2 Protein Function Prediction: 0.273 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Static Jailbreaks - Bio High Risk: 85.8 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Static Jailbreaks - Bio Severe: 81.3 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Static Jailbreaks - Cyber: 79.8 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Static Jailbreaks - Violence Moderate: 89.5 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Static Jailbreaks - Violence Severe: 90.5 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - U18 - Age-Restricted Goods and Dangerous Activities: 86.1 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - U18 - Eating Disorders: 85.3 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - U18 - Self-Harm: 99.0 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - U18 - Sexual Content: 97.5 (#2)
GPT-6 Sol on OpenRouter GPQA Diamond: 92.4 (#11)
GPT-6 Sol on OpenRouter Tau2-Bench Airline: 68.7 (#72)
GPT-6 Sol on PLCC - Art & Entertainment: 94.0 (#7)
GPT-6 Sol on PLCC - Culture & Tradition: 93.0 (#26)
GPT-6 Sol on PLCC - Geography: 98.0 (#14)
GPT-6 Sol on PLCC - Grammar: 93.0 (#13)
GPT-6 Sol on PLCC - History: 97.0 (#7)
GPT-6 Sol on PLCC - Overall: 93.83 (#13)
GPT-6 Sol on PLCC - Vocabulary: 88.0 (#27)
GPT-6 Sol on PRISM (1C:Enterprise) - Algorithmic Tasks (A): 93.0 (#9)
GPT-6 Sol on PRISM (1C:Enterprise) - Platform Tasks (B): 95.0 (#4)
GPT-6 Sol on ParseBench: 69.69 (#28)
GPT-6 Sol on ReLE - Agents - BFCL-V3: 48.8 (#113)
GPT-6 Sol on ReLE - Agents and Tool Use: 55.6 (#101)
GPT-6 Sol on ReLE - Coding: 72.7 (#11)
GPT-6 Sol on ReLE - Education: 61.9 (#30)
GPT-6 Sol on ReLE - Education - Gaokao: 59.8 (#34)
GPT-6 Sol on ReLE - Education - High School Subjects: 70.3 (#29)
GPT-6 Sol on ReLE - Education - Middle School Subjects: 53.4 (#33)
GPT-6 Sol on ReLE - Education - Primary School Subjects: 64.0 (#50)
GPT-6 Sol on ReLE - Finance: 84.8 (#20)
GPT-6 Sol on ReLE - Language and Instruction Following: 71.3 (#37)
GPT-6 Sol on ReLE - Law - Lawyer Qualification Exam: 74.0 (#63)
GPT-6 Sol on ReLE - Law and Civil Service: 82.0 (#45)
GPT-6 Sol on ReLE - Medicine - Basic Medical Knowledge: 84.1 (#26)
GPT-6 Sol on ReLE - Medicine - Physician Exams: 85.8 (#40)
GPT-6 Sol on ReLE - Medicine and Mental Health: 86.9 (#30)
GPT-6 Sol on ReLE - Overall: 71.7 (#30)
GPT-6 Sol on ReLE - Reasoning - BBH: 44.1 (#180)
GPT-6 Sol on ReLE - Reasoning and Mathematics: 75.1 (#67)
GPT-6 Sol on Riemann-bench: 64.8 (#7)
GPT-6 Sol on RuneBench: 13372.0 (#3)
GPT-6 Sol on SEAL - Fortress: 13.46 (#60)
GPT-6 Sol on SEAL - Professional Reasoning Benchmark - Legal: 38.6 (#30)
GPT-6 Sol on SecIT Bench (Pydantic AI): 59.26 (#21)
GPT-6 Sol on SimpleBench: 73.1 (#15)
GPT-6 Sol on Sonar LLM Leaderboard - Java - Bug Density: 0.59 (#8)
GPT-6 Sol on Sonar LLM Leaderboard - Java - Code Smell Density: 19.28 (#42)
GPT-6 Sol on Sonar LLM Leaderboard - Java - Pass Rate: 83.64 (#5)
GPT-6 Sol on Sonar LLM Leaderboard - Java - Vulnerability Density: 0.28 (#38)
GPT-6 Sol on SpeechMap Compliance: 64.0 (#185)
GPT-6 Sol on SvelteBench: 100.0 (#16)
GPT-6 Sol on TaxCalcBench: 28.0 (#16)
GPT-6 Sol on Tinybird AI SQL Benchmark - First-Attempt Success Rate: 94.0 (#167)
GPT-6 Sol on Tinybird AI SQL Benchmark - Success Rate: 96.0 (#263)
GPT-6 Sol on Vals AI (Vals Index): 62.57 (#8)
GPT-6 Sol on Vals AI BioMysteryBench: 74.81 (#4)
GPT-6 Sol on Vals AI Code Migration: 57.2 (#4)
GPT-6 Sol on Vals AI Excel Modeling: 71.53 (#8)
GPT-6 Sol on Vals AI Finance Agent v2: 49.05 (#37)
GPT-6 Sol on Vals AI Harvey Legal Agent Bench: 1.67 (#41)
GPT-6 Sol on Vals AI IOI: 82.61 (#7)
GPT-6 Sol on Vals AI Legal Research Bench: 28.85 (#38)
GPT-6 Sol on Vals AI MedCode: 47.07 (#31)
GPT-6 Sol on Vals AI MedScribe: 82.03 (#45)
GPT-6 Sol on Vals AI ProgramBench: 81.91 (#5)
GPT-6 Sol on Vals AI ProofBench: 83.0 (#9)
GPT-6 Sol on Vals AI Public Benefits Bench: 56.63 (#32)
GPT-6 Sol on Vals AI SAGE: 44.79 (#39)
GPT-6 Sol on Vals AI Tax Agent Bench: 53.05 (#24)
GPT-6 Sol on Vals AI Terminal-Bench 2.1: 83.15 (#6)
GPT-6 Sol on Vals AI Vibe Code Bench: 87.82 (#6)
GPT-6 Sol on Vending-Bench 2: 14427.85 (#2)
GPT-6 Sol on VoxelBench: 2470.0 (#3)
GPT-6 Sol on WebDev Arena (Brand & Marketing): 1654.0 (#7)
GPT-6 Sol on WebDev Arena (Consumer Product): 1627.0 (#6)
GPT-6 Sol on WebDev Arena (Content Creation Tools): 1674.0 (#4)
GPT-6 Sol on WebDev Arena (Data & Analytics): 1611.0 (#5)
GPT-6 Sol on WebDev Arena (Frontend): 1692.0 (#5)
GPT-6 Sol on WebDev Arena (Gaming): 1768.0 (#4)
GPT-6 Sol on WebDev Arena (HTML): 1666.0 (#5)
GPT-6 Sol on WebDev Arena (React): 1696.0 (#5)
GPT-6 Sol on WebDev Arena (Reference-Based Design): 1720.0 (#4)
GPT-6 Sol on WebDev Arena (Simulations): 1768.0 (#4)
GPT-6 Sol on ZeroBench: 41.0 (#3)

New #1 Leaders (269)

Epoch AI - Vending Bench 2: GPT-6 (15514.7) beat Claude Opus 5 (Unknown) by 4332.83
StereoTales: GPT-5 Nano (67.0) beat Grok 4 by 237.0
AA-Briefcase - Analytical Quality Elo: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (2206.92) beat Qwen 3.8 Max (0902) by 171.45
WebDev Arena (Content Creation Tools): Claude Opus 5.5 (Max) (1867.0) beat GPT-6 (Max) by 121.0
AA GDPval: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (1846.17) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 111.5
XL-SafetyBench: Claude Sonnet 4.5 (2.8) beat Mistral Large 3 by 96.0
CULTURE-MT: Gemini 3 (9.2) beat Qwen 3 0.6B by 86.21
AgentCIBench: Claude Opus 4.7 (13.7) beat Gemini 3.1 Pro (Preview) by 84.6
MT-JailBench: Claude Haiku 4.5 (18.24) beat Ministral 3 8B IT by 81.13
WebDev Arena (Simulations): Claude Opus 5.5 (Max) (1942.0) beat Claude Fable 5.1 (Max) by 75.0
Boiling the Frog: Claude Haiku 4.5 (20.5) beat Gemini 3.1 Flash Lite by 72.4
ActiveVision: GPT-6 (76.5) beat GPT-5.5 by 65.9
WebDev Arena (Fullstack): GPT-6 (Max) (1755.0) beat Claude Fable 5.1 (Max) by 64.0
WebDev Arena (Gaming): Claude Opus 5.5 (Max) (1940.0) beat GPT-6 (Max) by 60.0
PhysicsFinals: O3 Mini (59.9) beat THUDM-chatglm3-6B by 58.7
AA-Briefcase - Presentation Elo: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (1709.91) beat GPT-5.6 Sol (Max) by 57.38
LLM Emergent Collusion: Grok 3 (19.0) beat Grok 4 by 56.0
WebDev Arena (Brand & Marketing): Claude Opus 5.5 (Max) (1799.0) beat GPT-6 (Max) by 52.0
WebDev Arena (Frontend): Claude Opus 5.5 (Max) (1863.0) beat GPT-6 (Max) by 51.0
FrontierCode: Claude Fable 5 (xHigh) (64.94) beat Gemini 3.1 Flash Lite by 50.34
Inverse Turing Bench: GPTZero-witness-only (89.41) beat GPT-3.5 Turbo (1106) by 49.55
WebDev Arena (React): Claude Opus 5.5 (Max) (1869.0) beat GPT-6 (Max) by 49.0
AgentTrap: Claude Sonnet 4.6 (33.0) beat Kimi K2.6 by 42.0
ActiveVision (Agents): GPT-6 Codex (91.8) beat Claude Fable 5 by 41.2
EnactToM: Gemini-Flash (42.5) beat DeepSeek V3.2 by 40.0
OJBench: Gemini 2.5 Pro (Preview 03-25) (38.91) beat DeepSeek-Coder-6.7B by 36.92
TukaBench: Gemini 3.1 Pro (Preview) (2.4) beat GPT-3.5 by 35.7
When Multiple Scripts Matter: Gemini 2.5 Pro (24.23) beat Qwen 3 0.6B by 34.86
Design Arena (Game Dev): Claude Opus 5.5 (1481.0) beat GPT-6 by 32.0
Chatbot Arena (Text - Japanese): Claude Fable 5.1 (Max) (1555.0) beat Claude Fable 5 (High) by 32.0
LITMUS: Claude Sonnet 4.6 (40.64) beat DeepSeek V3.2 by 30.87
The Compliance Trap: Claude Sonnet 4.6 (1.3) beat DeepSeek V4 Pro by 28.9
Chatbot Arena (Code): Claude Opus 5.5 (Max) (1827.0) beat GPT-6 (Max) by 27.0
WebDev Arena (Reference-Based Design): Claude Opus 5.5 (Max) (1854.0) beat GPT-6 (Max) by 27.0
ITBench-AA: Claude Opus 4.7 (Adaptive Reasoning, Max Effort) (46.7) beat Qwen 3.5 35B A3B (Reasoning) by 25.2
Chartography: GPT-6 (Max) (71.0) beat Claude Fable 5.1 (Max) by 24.8
BehaviorBench: Be.FM-1.5-4B (Ours) (7.0) beat GPT-5.4 by 24.4
VoxelBench: Claude Opus 5.5 (Max) (2662.0) beat GPT-6 (Max) by 23.0
SEA-HELM (English) - AA-LCR: Muse-Glimmer-30B (Reasoning) (81.42) beat Qwen 3.5 27B by 22.06
ZeroBench: GPT-6 (Max) (52.0) beat GPT-5.6 Sol (Max) by 22.0
MathArena - ARXIVLEAN June: AlephProver (86.96) beat GPT-6 (Max) by 21.74
ChessBench - Elo: GPT-6 (1448.0) beat Gemini 3.8 Flash by 20.0
SEA-HELM (Thai) - Thai Exam: Gemma 4 31B (IT) (Reasoning) (79.67) beat Qwen 3.5 27B by 19.84
AIMultiple - Text-to-SQL (Strict Execution Match): Claude Opus 5.5 (74.9) beat Gemini 3 Flash (Preview) by 19.8
TextQuests (No Clues): GPT-6 (74.24) beat Claude Fable 5 by 19.76
SEA-HELM (English) - MathArena: Gemma 4 31B (IT) (Reasoning) (56.21) beat Gemma 4 31B (IT) by 18.45
Agents on Rails: GPT-6 (Max) (53.3) beat GPT-6 (Medium) by 18.3
AppWorld Challenge: Claude Opus 4.7 (85.6) beat Qwen 3 14B by 18.0
MIRAGE: Claude Opus 4 (8.0) beat DeepSeek V3 by 18.0
LLM Stats (Humanity's Last Exam (no tools, text-only)): Claude Opus 5.5 (64.4) beat Hy3 by 17.4
Chatbot Arena (Text - Creative Writing): Claude Opus 5.5 (High) (1521.0) beat Claude Fable 5 (High) by 17.0
SEA-HELM (Burmese) - Global MMLU Lite: Qwen3.8-Flash-Next (Reasoning) (81.25) beat Gemma 4 31B (IT) by 15.52
SEA-HELM (Tamil) - Pragmatics: Qwen 3.6 35B A3B (Reasoning) (80.83) beat Qwen-SEA-LION-v4.5-27B-IT by 14.77
LLM Stats (Program Bench): Claude Opus 5.5 (91.2) beat Kimi K3 by 13.4
Legal Benchmarks - Data Extraction - Task Pass Rate: Claude Opus 5.5 (76.7) beat Claude Fable 5.1 by 13.4
SEA-HELM (Indonesian) - Syntax (LLM Judge): Qwen3.8-Flash-Next (Reasoning) (88.17) beat Gemma 4 31B (IT) by 11.51
MixRea: Gemini 2.5 Pro (9.4) beat Llama 3.1 8B by 10.9
KernelBench Hub - Mega: Claude Opus 5.5 (35.46) beat GPT-6 Pro by 10.66
SEA-HELM (Burmese) - Translations: Llama-SEA-LION-v3.5-8B-R (Reasoning) (99.13) beat Gemma 4 31B (IT) by 10.64
SEA-HELM (Filipino) - Global MMLU Lite: Qwen3.8-Flash-Next (Reasoning) (86.16) beat Gemma 4 31B (IT) by 10.42
SEA-HELM (Tamil) - Translations: Llama-SEA-LION-v3.5-8B-R (Reasoning) (99.63) beat Gemma 4 31B (IT) by 9.94
SEA-HELM (Malay) - Global MMLU Lite: Qwen3.8-Flash-Next (Reasoning) (86.96) beat Qwen 3.5 27B by 9.62
Kaggle Game Arena (Unified): Claude Opus 5 (354.47) beat GPT-5.5 by 9.56
SEA-HELM (Vietnamese) - Global MMLU Lite: Qwen3.8-Flash-Next (Reasoning) (85.71) beat Qwen 3.5 122B A10B by 9.45
Epoch AI - Frontierswe: GPT-6 (Max) (65.51) beat Claude Fable 5.1 (Max) by 9.22
SEA-HELM (Indonesian) - Syntax: Qwen3.8-Flash-Next (Reasoning) (76.39) beat Gemma 4 31B (IT) by 9.06
Chatbot Arena (Text - French): Muse Spark 1.3 (Max) (1541.0) beat Claude Fable 5.1 (Max) by 9.0
Chatbot Arena (Text - Writing, Literature & Language): Claude Opus 5.5 (High) (1520.0) beat Claude Fable 5 (High) by 9.0
OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Sensitive Personal Data: GPT-6 Sol (85.4) beat GPT-5.6 Sol by 8.9
AIMultiple - Text-to-SQL (Adjudicated): Claude Opus 5.5 (91.1) beat Claude Opus 5 by 8.7
SEA-HELM (Indonesian) - Global MMLU Lite: Qwen3.8-Flash-Next (Reasoning) (87.46) beat Qwen 3.5 27B by 8.55
SEA-HELM (Filipino) - Translations: Llama-SEA-LION-v3.5-8B-R (Reasoning) (99.79) beat Gemma 3 27B (IT) by 8.46
SuperCLUE-VLM - Overall: GPT-6 (83.9) beat Gemini 3.1 Pro (Preview) by 8.09
Chatbot Arena (Text - Hard Prompts): Claude Opus 5.5 (High) (1541.0) beat Claude Opus 4.6 (High) by 8.0
Chatbot Arena (Text - Expert): Claude Opus 5.5 (High) (1556.0) beat Claude Fable 5 (High) by 8.0
WebDev Arena (Data & Analytics): Claude Opus 5.5 (Max) (1700.0) beat GPT-6 (Max) by 8.0
LLM Stats (Humanity's Last Exam (with tools, text-only)): Claude Opus 5.5 (67.7) beat DeepSeek V4 Pro (0813) by 7.7
SEA-HELM (Tamil) - Linguistic Diagnostics: Qwen 3.6 35B A3B (Reasoning) (85.18) beat Qwen-SEA-LION-v4.5-27B-IT by 7.66
Kaggle Open Benchmarks - SciCode Subproblems: Claude Opus 5.5 (52.23) beat Gemini 3.1 Pro (Preview) by 7.33
TextQuests (With Clues): GPT-6 (94.0) beat GPT-5.6 Sol by 7.16
SEA-HELM (Malay) - Translations: Llama-SEA-LION-v3.5-8B-R (Reasoning) (99.83) beat Gemma 4 31B (IT) by 7.15
SEA-HELM (Burmese) - Causal Reasoning: Qwen3.8-Flash-Next (Reasoning) (83.9) beat Gemma 4 31B (IT) by 7.0
Chatbot Arena (Text - Life, Physical & Social Science): Claude Opus 5.5 (High) (1535.0) beat Claude Fable 5 (High) by 7.0
LLM Stats (CyberGym): MiMo-V2.6-Flash (95.1) beat DeepSeek V4.1 Flash by 7.0
Next.js Agent Evals (Codex) - Success Rate: GPT-6 Sol (High) (97.0) beat GPT-6 (High) by 7.0
LLM Chess (Saplin): GPT-6 (High) (1613.8) beat Gemini 3.8 Flash (High) by 6.7
SEA-HELM (Burmese) - SEA-IFEval: Qwen 3.5 35B A3B (Reasoning) (86.01) beat Gemma 4 31B (IT) by 6.64
Agents' Last Exam: Claude Opus 5.5 (38.2) beat Claude Opus 5 by 6.6
LLM Stats (SWE-bench Multilingual): Claude Opus 5.5 (93.9) beat Claude Mythos Preview by 6.6
SEA-HELM (Tamil) - SEA-IFEval: Qwen 3.5 27B (Reasoning) (95.7) beat Gemma 4 31B (IT) by 6.54
SEA-HELM (Indonesian) - Metaphor Understanding: Muse-Glimmer-30B (Reasoning) (89.85) beat Gemma 4 31B (IT) by 6.52
SEA-HELM (English) - LiveCodeBench v6: Gemma 4 31B (IT) (Reasoning) (80.09) beat Gemma 4 31B (IT) by 6.32
SEA-HELM (Vietnamese) - Translations: Llama-SEA-LION-v3.5-8B-R (Reasoning) (99.86) beat Gemma 4 31B (IT) by 6.15
SEA-HELM (Indonesian) - Translations: Llama-SEA-LION-v3.5-8B-R (Reasoning) (99.86) beat Gemma 4 31B (IT) by 6.11
APEX v1: Muse Spark 1.2 (xHigh) (75.6) beat GPT-5.6 Terra (Max) by 6.1
CursorBench 4.0: Claude Opus 5.5 (57.8) beat Claude Fable 5.1 (Max) by 6.0
Chatbot Arena (Text - Russian): Claude Opus 5.5 (High) (1525.0) beat Claude Fable 5 (High) by 6.0
Chatbot Arena (Text - Longer Query): Claude Opus 5.5 (High) (1530.0) beat Claude Opus 4.6 (High) by 6.0
AA-Omniscience Index - Software Engineering (SWE) - Rust: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (92.0) beat GPT-6 (Max) by 6.0
SEA-HELM (Thai) - Translations: Llama-SEA-LION-v3.5-8B-R (Reasoning) (99.83) beat Gemma 4 31B (IT) by 5.95
SEA-HELM (English) - SuperGPQA: Qwen 3.5 122B A10B (Reasoning) (63.33) beat Qwen 3.5 122B A10B by 5.87
ComplexConstraints: GPT-6 (Max) (57.7) beat Muse Spark 1.3 (xHigh) by 5.8
LLM Stats (Terminal-Bench 4.0): Claude Opus 5.5 (66.4) beat Claude Mythos 5.1 by 5.5
AIMultiple - Agentic RAG Routing: Claude Opus 5.5 (90.2) beat Claude Opus 5 by 5.4
SEA-HELM (Indonesian) - Linguistic Diagnostics: Qwen3.8-Flash-Next (Reasoning) (79.7) beat Gemma 4 31B (IT) by 5.34
SEA-HELM (Tamil) - SEA-MT-Bench (LLM Judge): Qwen3.8-Flash-Next (Reasoning) (85.23) beat Qwen 3.5 27B by 5.15
Chatbot Arena (Text - Non-English): Claude Opus 5.5 (High) (1501.0) beat Claude Fable 5 (High) by 5.0
AA-Omniscience Index - Software Engineering (SWE) - Java: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (80.0) beat GPT-6 (Max) by 5.0
PRISM (1C:Enterprise) - Platform Tasks (B): Claude Opus 5.5 (100.0) beat GPT-5.6 Sol by 5.0
Mercor APEX: Claude Opus 5.5 (73.5) beat Claude Fable 5.1 by 4.9
SEA-HELM (Burmese) - NLU: Muse-Glimmer-30B (Reasoning) (67.54) beat Gemma 4 31B (IT) by 4.82
SEA-HELM (Filipino) - Belebele QA: Gemma 4 31B (IT) (Reasoning) (88.5) beat Mistral Medium 3.5 by 4.81
Epoch AI - Rli: GPT-6 (20.83) beat Claude Fable 5 by 4.73
GDP.pdf: GPT-6 (35.4) beat GPT-5.6 Sol (Max) by 4.7
Agent Arena - Steerability: GPT-6 Sol (Max) (15.7) beat Claude Opus 4.8 (High) by 4.64
SEA-HELM (English) - MuSR: Qwen3.8-Flash-Next (Reasoning) (84.16) beat Gemma 4 31B (IT) by 4.63
OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Non-Violent Wrongdoing: GPT-6 Luna (100.0) beat GPT-6 by 4.6
Vals AI Terminal-Bench 4.0: Claude Opus 5.5 (61.62) beat GPT-6 by 4.55
Kaggle FACTS (Google): GPT-6 (75.68) beat Gemini 3.7 Flash by 4.43
MathArena - ARXIV August: GPT-6 Sol (Max) (92.98) beat GPT-6 (Max) by 4.38
Artificial Analysis Intelligence Index: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (57.62) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 4.27
AA-Omniscience Index - Science, Engineering & Mathematics: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (37.8) beat Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) by 4.2
AppWorld Normal: Claude Opus 4.6 (91.1) beat Qwen 3 14B by 4.2
SEA-HELM (Thai) - SEA-IFEval: Gemma 4 31B (IT) (Reasoning) (97.36) beat Gemma 4 31B (IT) by 4.11
AI Coding Daily (Claude Code) - Total: Claude Opus 5.5 (High) (57.83) beat Claude Opus 5 (High) by 4.11
OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Self-Harm: GPT-6 Luna (96.0) beat GPT-6 by 4.0
MathArena Arxiv False: Claude Opus 5.5 (High) (95.0) beat GPT-6 (Max) by 4.0
AA-Omniscience Index - Software Engineering (SWE) - C: Claude Opus 5.5 (Adaptive Reasoning, Low Effort, Default Fallback) (96.0) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 4.0
AA-Omniscience Index - Software Engineering (SWE) - Kotlin: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (96.0) beat GPT-5.6 Sol (xHigh) by 4.0
AA-Omniscience Index - Software Engineering (SWE) - PHP: Claude Opus 5.5 (Adaptive Reasoning, High Effort, Default Fallback) (96.0) beat GPT-5.6 Sol (Max) by 4.0
TaxCalcBench: GPT-6 Sol Web Search (66.0) beat GPT-5.6 Sol Web Search by 4.0
BenchLM: GPT-6 (88.7) beat Claude Fable 5.1 by 4.0
Next.js Agent Evals (OpenCode) - Success Rate: Grok 4.7 (94.0) beat Gemini 3.8 Flash by 4.0
SEA-HELM (English) - Knowledge: Qwen3.8-Flash-Next (Reasoning) (77.84) beat Qwen 3.5 122B A10B by 3.96
LLM Stats (OSWorld 2.0): Claude Opus 5.5 (81.8) beat Claude Fable 5.1 by 3.9
AA SciCode: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (66.9) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 3.82
SEA-HELM (Tamil) - NLG: Llama-SEA-LION-v3.5-8B-R (Reasoning) (55.2) beat Gemma 4 31B (IT) by 3.78
SEA-HELM (Burmese) - Belebele QA: Muse-Glimmer-30B (Reasoning) (85.88) beat Gemma 4 31B (IT) by 3.61
LLM Stats (SWE-Marathon): Grok 4.7 (46.0) beat GLM-5.3 by 3.5
SEA-HELM (Tamil) - Syntax: Gemma 4 31B (IT) (Reasoning) (94.93) beat Gemma 4 31B (IT) by 3.48
Sonar LLM Leaderboard - Java - Code Smell Density: GLM-5 (10.4) beat Claude Opus 4.5 (High) by 3.47
NRT-Bench: Claude Haiku 4.5 (8.7) beat Gemma 4 26B A4B (IT) by 3.4
LiveBench TypeScript: Claude Opus 5.5 (Max) (63.33) beat Claude Fable 5.1 (Max) by 3.33
Architectural RedlineBench: Claude Opus 5.5 (68.8) beat GPT-6 by 3.3
SEA-HELM (English) - IFEval: Gemma 4 31B (IT) (Reasoning) (94.92) beat Gemma 4 31B (IT) by 3.23
DeepsecBench: GPT-6 Sol (xHigh) (40.91) beat GPT-6 (xHigh) by 3.12
ChessBench - Accuracy: GPT-6 (81.2) beat Gemini 3.8 Flash by 3.1
SEA-HELM (Filipino) - Kalahi (LLM Judge): Gemma 4 31B (IT) (Reasoning) (81.21) beat Gemma 4 31B (IT) by 3.02
Chatbot Arena (Text): Claude Opus 5.5 (High) (1509.0) beat Claude Fable 5 (High) by 3.0
Chatbot Arena (Text - Instruction Following): Claude Opus 5.5 (High) (1516.0) beat Claude Opus 4.6 (High) by 3.0
Chatbot Arena (Text - Polish): Muse Spark 1.1 (1513.0) beat Gemini 3.5 Flash (High) by 3.0
Conceptual Reasoning Index: Claude Opus 5.5 (77.86) beat Claude Fable 5.1 (Medium) by 2.91
Conceptual Reasoning Index - Argument Evaluation (LMCA): Claude Opus 5.5 (68.23) beat Claude Fable 5.1 by 2.77
AA Omniscience: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (46.42) beat GPT-6 (High) by 2.69
AA-Omniscience Index - Software Engineering (SWE): Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (88.8) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 2.6
SEA-HELM (Thai) - SEA-MT-Bench (LLM Judge): Qwen3.8-Flash-Next (Reasoning) (86.55) beat Qwen 3.5 27B by 2.57
Icelandic LLM - Idiom Completion: Claude Opus 5.5 (75.03) beat GPT-6 by 2.52
ChessBench: GPT-6 (78.4) beat Gemini 3.8 Flash by 2.5
AutomationBench-AA - Sales - Raw Objective Completion: GPT-6 Sol (xHigh) (90.07) beat Grok 4.6 (xHigh) by 2.45
SEA-HELM (Vietnamese) - Causal Reasoning: Qwen 3.6 27B (Reasoning) (95.24) beat Qwen-SEA-LION-v4.5-27B-IT by 2.44
SEA-HELM (Burmese) - NLR: Gemma 4 31B (IT) (Reasoning) (70.4) beat Gemma 4 31B (IT) by 2.42
DuelLab Overall: GPT-6 (xHigh) (94.8) beat Claude Fable 5.1 (xHigh) by 2.4
SEA-HELM (Indonesian) - SEA-Safeguard: Muse-Glimmer-30B (Reasoning) (78.42) beat Gemma 3 12B (IT) by 2.39
SEA-HELM (Filipino) - Causal Reasoning: Gemma 4 31B (IT) (Reasoning) (97.06) beat Gemma 4 31B (IT) by 2.39
InferenceBench: Claude Opus 5.5 (Max) (12.08) beat Claude Fable 5.1 by 2.25
AA Humanity's Last Exam: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (61.35) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 2.22
Epoch AI - Lmca: Claude Fable 5.1 (Unknown) (65.46) beat Claude Opus 5 (Max) by 2.16
SEA-HELM (Indonesian) - SEA-MT-Bench (LLM Judge): Qwen 3.5 122B A10B (Reasoning) (88.03) beat Qwen 3.6 27B by 2.11
SEA-HELM (Tamil) - Causal Reasoning: Qwen 3.6 27B (Reasoning) (95.87) beat Qwen-SEA-LION-v4.5-27B-IT by 2.08
AA GDP.pdf - Finance/Investing - Criterion Pass Rate: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (91.25) beat GPT-6 (Max) by 2.06
SEA-HELM (Indonesian) - NLU: Muse-Glimmer-30B (Reasoning) (83.99) beat Gemma 4 31B (IT) by 2.01
When the Manual Lies: Qwen 3 32B (98.0) beat GPT-4o by 2.0
AI Coding Daily (Claude Code) - React-TS Code Quality: Claude Opus 5.5 (Medium) (19.67) beat Claude Opus 5 (High) by 2.0
Legal Benchmarks - Contract Workflows - Form: Claude Opus 5.5 (93.9) beat GPT-6 by 2.0
SEA-HELM (Filipino) - SEA-MT-Bench (LLM Judge): Qwen3.8-Flash-Next (Reasoning) (84.81) beat Gemma 4 31B (IT) by 1.98
TrackingAI IQ Test (Vision): GPT-5.6 Sol (Ultra) (98.04) beat GPT-Luna-Max by 1.96
SEA-HELM (Indonesian) - Causal Reasoning: Gemma 4 31B (IT) (Reasoning) (97.54) beat Gemma 4 31B (IT) by 1.91
SEA-HELM (Filipino) - SEA-IFEval: Gemma 4 31B (IT) (Reasoning) (94.02) beat Gemma 4 31B (IT) by 1.89
SEA-HELM (Indonesian) - NLG: Llama-SEA-LION-v3.5-8B-R (Reasoning) (57.98) beat Gemma 4 31B (IT) by 1.88
SEA-HELM (Thai) - NLG: Llama-SEA-LION-v3.5-8B-R (Reasoning) (61.29) beat Gemma 4 31B (IT) by 1.84
SEA-HELM (Filipino) - Kalahi: Gemma 4 31B (IT) (Reasoning) (92.37) beat Gemma 4 31B (IT) by 1.83
Vals AI Vibe Code Bench 1-100: Claude Opus 5.5 (30.36) beat Claude Opus 5 by 1.83
SimpleBench: Claude Opus 5.5 (88.4) beat Claude Fable 5.1 by 1.8
SEA-HELM (Filipino) - Natural Language Inference: Qwen-SEA-LION-v4.5-27B-IT (Reasoning) (69.98) beat Gemma 4 31B (IT) by 1.78
AI Coding Daily (OpenCode) - Total: Space Bunny (Max) (52.61) beat Kimi K3 (High) by 1.76
SEA-HELM (Vietnamese) - SEA-Safeguard: Gemma 4 31B (IT) (Reasoning) (85.68) beat Qwen-SEA-LION-v4.5-27B-IT by 1.75
OpenAI GPT-6 Astra System Card - Challenging Prompts - Sexual/Minors: GPT-6 Sol (99.1) beat GPT-6 by 1.7
SEA-HELM (Burmese) - Natural Language Inference: Qwen 3.6 27B (Reasoning) (60.72) beat Gemma 4 31B (IT) by 1.65
AA-Omniscience Index - Humanities & Social Sciences: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (44.7) beat GPT-6 (Max) by 1.6
SEA-HELM (Filipino) - NLU: Gemma 4 31B (IT) (Reasoning) (76.85) beat Gemma 4 31B (IT) by 1.59
SEA-HELM (Malay) - Belebele QA: Gemma 4 31B (IT) (Reasoning) (94.85) beat Gemma 4 31B (IT) by 1.58
AA GDP.pdf - STEM/Research - Criterion Pass Rate: Claude Opus 5.5 (Adaptive Reasoning, Low Effort, Default Fallback) (81.47) beat GPT-6 (Medium) by 1.58
Vals AI ProgramBench: Claude Opus 5.5 (Max) (86.99) beat GPT-6 (Max) by 1.57
Icelandic LLM - Generative Quality: GPT-6 Sol (83.75) beat GPT-6 by 1.53
SEA-HELM (Tamil) - NLR: Qwen 3.6 27B (Reasoning) (76.81) beat Gemma 4 31B (IT) by 1.51
AA GDP.pdf - Engineering - Criterion Pass Rate: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (84.35) beat GPT-6 (xHigh) by 1.49
Conceptual Reasoning Index - Consistency (ACCoRD): Claude Opus 5.5 (86.41) beat Claude Fable 5 by 1.46
SEA-HELM (Filipino) - NLR: Gemma 4 31B (IT) (Reasoning) (82.85) beat Gemma 4 31B (IT) by 1.41
SEA-HELM (Indonesian) - NLR: Gemma 4 31B (IT) (Reasoning) (90.35) beat Gemma 4 31B (IT) by 1.34
AutomationBench-AA - Finance - Raw Objective Completion: Grok 4.7 (xHigh) (93.49) beat Grok 4.6 (xHigh) by 1.33
SEA-HELM (English) - MMLU Redux: Qwen3.8-Flash-Next (Reasoning) (92.44) beat Gemma 4 31B (IT) by 1.27
SEA-HELM (Vietnamese) - NLG: Llama-SEA-LION-v3.5-8B-R (Reasoning) (57.54) beat Llama-SEA-LION-v3-70B-IT by 1.27
SEA-HELM (Thai) - Causal Reasoning: Gemma 4 26B A4B (IT) (Reasoning) (98.06) beat Gemma 4 31B (IT) by 1.23
Legal Benchmarks - Data Extraction - Criteria Pass Rate: Claude Opus 5.5 (91.1) beat Claude Fable 5.1 by 1.2
SEA-HELM (Indonesian) - Natural Language Inference: Qwen 3.6 27B (Reasoning) (83.57) beat Gemma 4 31B (IT) by 1.18
SEA-HELM (Indonesian) - SEA-IFEval: Gemma 4 31B (IT) (Reasoning) (97.75) beat Gemma 4 31B (IT) by 1.13
AutomationBench: Claude Opus 5.5 (default fallbacks, Max) (42.5) beat GPT-6 (Max) by 1.1
AA GDP.pdf - HR - Criterion Pass Rate: Claude Opus 5.5 (Adaptive Reasoning, Low Effort, Default Fallback) (90.16) beat Claude Opus 5 (Adaptive Reasoning, High Effort) by 1.09
OpenAI GPT-6 Astra System Card - Biology Refusal - Dual Use: GPT-6 Sol (98.0) beat GPT-6 by 1.0
Legal Benchmarks - Contract Workflows - Criteria Pass Rate: Claude Opus 5.5 (91.7) beat Claude Fable 5.1 by 1.0
SEA-HELM (Malay) - SEA-MT-Bench (LLM Judge): Qwen 3.5 122B A10B (Reasoning) (86.35) beat Qwen 3.5 122B A10B by 0.96
MathArena - ARXIV_FALSE June: Claude Opus 5.5 (High) (100.0) beat GPT-6 (Max) by 0.93
ParseBench: Claude Opus 5.5 (Effort High) (79.85) beat Claude Fable 5.1 by 0.93
SEA-HELM (Malay) - NLU: Gemma 4 31B (IT) (Reasoning) (74.15) beat Gemma 4 31B (IT) by 0.92
LLM Stats (FrontierCode 1.1): Claude Opus 5.5 (54.4) beat Claude Fable 5 by 0.9
Conceptual Reasoning Index - Decision Theory (DTBench): Claude Opus 5.5 (98.22) beat Claude Fable 5 by 0.89
SEA-HELM (Tamil) - Sentiment Analysis: Muse-Glimmer-30B (Reasoning) (98.19) beat Gemma 3 27B (IT) by 0.88
Icelandic LLM - Flora & Fauna: Claude Opus 5.5 (57.42) beat GPT-6 by 0.88
SEA-HELM (Vietnamese) - SEA-IFEval: Gemma 4 31B (IT) (Reasoning) (96.21) beat Gemma 4 31B (IT) by 0.87
SEA-HELM (Malay) - SEA-IFEval: Gemma 4 31B (IT) (Reasoning) (90.72) beat Gemma 4 31B (IT) by 0.86
Vals AI (Vals Index): Claude Opus 5.5 (69.69) beat Claude Fable 5.1 by 0.86
AIMultiple - FinanceReasoning (Hard): Claude Opus 5.5 (91.6) beat GPT-5.6 Pro Sol by 0.84
SWE-Together: Claude Opus 4.6 (80.89) beat Claude Opus 4.8 by 0.83
AA MMMU-Pro: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (87.69) beat GPT-6 (Max) by 0.81
SEA-HELM (Filipino) - NLG: Llama-SEA-LION-v3.5-8B-R (Reasoning) (59.48) beat Gemma 4 31B (IT) by 0.76
Kaggle Global-MMLU-Lite - Swahili: Claude Opus 5.5 (95.5) beat Gemini 3.8 Flash by 0.75
SEA-HELM (Thai) - NLU: Qwen 3.6 27B (Reasoning) (64.67) beat Qwen-SEA-LION-v4.5-27B-IT by 0.74
AI Coding Daily (Codex CLI) - Total: GPT-6 (High) (56.77) beat GPT-6 (Medium) by 0.74
Harvey LAB-AA - Immigration: Claude Opus 5.5 (Adaptive Reasoning, Xhigh Effort, Default Fallback) (94.91) beat Muse Spark 1.3 (xHigh) by 0.73
AA GDP.pdf - Real Estate - Criterion Pass Rate: GPT-6 Sol (High) (79.84) beat GPT-6 (Medium) by 0.69
AA-Briefcase - XLSX - Rubric Pass Rate: Claude Opus 5.5 (Adaptive Reasoning, Xhigh Effort, Default Fallback) (61.14) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 0.69
AutomationBench-AA: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (69.54) beat DeepSeek V4.1 Flash (Reasoning, Max Effort) by 0.65
Icelandic LLM Leaderboard - Average: Claude Opus 5.5 (70.87) beat GPT-6 by 0.62
AA-Briefcase - DOCX - Rubric Pass Rate: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (60.36) beat Claude Opus 5 (Adaptive Reasoning, Xhigh Effort) by 0.6
AA GDP.pdf - Healthcare - Criterion Pass Rate: Claude Opus 5.5 (Adaptive Reasoning, High Effort, Default Fallback) (93.1) beat GPT-6 (Medium) by 0.59
Open Japanese LLM v2 - M-IFEval-Ja: llm-jp-4-kappa-32B-a3B-v0.1 (Thinking) (47.67) beat Llama 3.3 70B Instruct by 0.58
AA-Omniscience Index - Software Engineering (SWE) - Python: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (91.5) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 0.5
SEA-HELM (Burmese) - Toxicity Detection: Gemma 4 31B (IT) (Reasoning) (69.99) beat Gemma 4 31B (IT) by 0.47
SEA-HELM (Indonesian) - Question Answering: Qwen3.8-Flash-Next (Reasoning) (85.9) beat Qwen-SEA-LION-v4.5-27B-IT by 0.44
SEA-HELM (Vietnamese) - SEA-MT-Bench (LLM Judge): Qwen 3.5 35B A3B (Reasoning) (87.3) beat Qwen 3.6 27B by 0.43
Kaggle AssetOpsBench: GPT-6 (69.55) beat Gemini 3 Pro (Preview) by 0.41
OpenAI GPT-6 Astra System Card - Challenging Prompts - Sexual: GPT-6 Sol (98.4) beat GPT-6 by 0.4
OpenAI GPT-6 Astra System Card - U18 - Emotional Reliance: GPT-6 Sol (94.8) beat GPT-6 by 0.4
LLM Stats (HealthBench): Claude Opus 5.5 (60.6) beat Qwen 3.8 Max by 0.4
AI Coding Daily (Claude Code) - Laravel Code Quality: Claude Opus 5.5 (Medium) (18.95) beat Claude Opus 5 (High) by 0.4
AI Coding Daily (Codex CLI) - Laravel Code Quality: GPT-6 (High) (18.6) beat GPT-6 (Medium) by 0.4
ReLE - Language and Instruction Following: Hy4 preview (78.8) beat doubao-seed-2-1-pro-260628 by 0.4
Icelandic LLM - WikiQA: Claude Opus 5.5 (74.32) beat Claude Fable 5.1 by 0.37
Vals AI Terminal-Bench 2.1: Claude Opus 5.5 (87.64) beat GPT-6 by 0.37
AutomationBench-AA - Marketing - Raw Objective Completion: MiMo-V2.6-Pro (91.11) beat Claude Opus 5 (Adaptive Reasoning, Max Effort) by 0.35
AI Coding Daily (Codex CLI) - React-TS Code Quality: GPT-6 (High) (18.17) beat GPT-6 (Medium) by 0.34
SEA-HELM (Vietnamese) - NLR: Qwen 3.6 27B (Reasoning) (79.22) beat Gemma 4 31B (IT) by 0.33
LiveBench Olympiad: Claude Opus 5.5 (Max) (93.25) beat Claude Fable 5.1 (Max) by 0.28
SEA-HELM (Malay) - Sentiment Analysis: Gemma 4 31B (IT) (Reasoning) (53.45) beat Gemma 4 31B (IT) by 0.27
SEA-HELM (Vietnamese) - NLU: Qwen3.8-Flash-Next (Reasoning) (72.0) beat Qwen 3 VL 8B Instruct by 0.26
Kaggle Global-MMLU-Lite - English: Claude Opus 5.5 (96.75) beat Claude Opus 5 by 0.25
SEA-HELM (Vietnamese) - Safety: Gemma 4 31B (IT) (Reasoning) (72.42) beat Gemma 4 31B (IT) by 0.24
OpenAI GPT-6 Astra System Card - Challenging Prompts - Gore: GPT-6 Sol (90.0) beat GPT-6 by 0.2
OpenAI GPT-6 Astra System Card - Image Input - Self-Harm: GPT-6 Luna (99.9) beat GPT-6 by 0.2
AA GDP.pdf - Criterion Pass Rate: Claude Opus 5.5 (Adaptive Reasoning, High Effort, Default Fallback) (82.25) beat GPT-6 (High) by 0.2
AutomationBench-AA - HR - Raw Objective Completion: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (77.27) beat GPT-6 (High) by 0.19
Vals AI MedScribe: Claude Opus 5.5 (Max) (91.43) beat Claude Fable 5.1 (Max) by 0.14
LLM Stats Score: Claude Opus 5.5 (59.7) beat GPT-6 by 0.12
Epoch AI - Blueprint Bench 2: GPT-6 (0.497) beat Claude Fable 5.1 (Unknown) by 0.08
Icelandic LLM - Idiom Meanings: GPT-6 (84.61) beat Claude Opus 5 by 0.06
SEA-HELM (Burmese) - NLG: Gemma 4 31B (IT) (Reasoning) (57.95) beat Gemma 4 31B (IT) by 0.05
HalluHard: GPT-6 (0.3652) beat GPT-5.4 (Thinking) by 0.05
Blueprint-Bench 2: Claude Opus 5.5 (0.512) beat GPT-6 by 0.02
Vals AI RSI Index: Claude Opus 5.5 (Max) (0.371) beat Claude Fable 5.1 (Max) by 0.02

The Aggregate Digest — 2026-09-26

2026-09-26T06:34:55.337280+00:00

Daily

New Benchmarks (156)

PhysAI-Bench (Decision accuracy (%; four-option choice of the next action at decision points extracted from autonomous UAV mission traces, with mission context, physical and sensor state, MCP tool responses, A2A messages and 6G network conditions available before the decision; 500 human-verified held-out instances from episodes disjoint from the development set; strict option-label match, parse and inference failures scored wrong; mean of three runs with each model's prompting (0-, 3- or 5-shot) and temperature selected on a 35-instance development set)): leader GPT-5.3 (52.0), 30 models
PhysAI-Bench (Ferrag et al., 2026) tests agentic decision-making for autonomous drone missions: each item asks for the next action at a decision point extracted from a conversational mission trace, as a four-option choice given only what was known before that decision, including mission and sensor state, tool and agent-to-agent messages and 6G network conditions. Accuracy in percent on 500 human-verified held-out items from missions not used in development, the mean of three runs with each model using the prompting and temperature it did best with on a small development set; higher is better.
MuLA-Bench (Accuracy (%; all 5,038 questions over 1,769 recordings in 16 languages and eight domains, semantic and acoustic evidence tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (73.4), 10 models
MuLA-Bench (Yang et al., 2026) tests long-form audio understanding with open-ended questions over in-the-wild recordings, often tens of minutes long, in 16 languages and eight content domains, about both spoken content and natural non-speech sounds. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent over all 5,038 questions, higher is better.
MuLA-Bench - Semantic Evidence (Accuracy (%; the 3,840 semantic-track questions about spoken facts and relations, 30 per language-domain cell; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (79.06), 10 models
The semantic-evidence track of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone: 3,840 questions about spoken facts and relations, 30 for every language and domain. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Acoustic Evidence (Accuracy (%; the 1,198 acoustic-track questions about naturally occurring non-speech or paralinguistic events such as laughter, applause and music; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (55.26), 10 models
The acoustic-evidence track of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone: 1,198 questions about naturally occurring non-speech or paralinguistic sounds such as laughter, applause and music. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Factual (Accuracy (%; the 1,891 factual and completeness questions (single facts, exhaustive lists, multi-slot answers) over both tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (71.29), 10 models
Factual and completeness questions in MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, asking for single facts, full lists or several required details from spoken content or natural sounds. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Reasoning (Accuracy (%; the 1,146 reasoning questions requiring inference over the recording, both tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (72.51), 10 models
Reasoning questions in MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, which require an inference from what the recording says or contains rather than general knowledge. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Temporal Localization (Accuracy (%; the 1,008 temporal questions on event order and clock-time localization, both tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (66.67), 10 models
Temporal questions in MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, asking when an event happens on the clock, within 3 seconds, or in what order events occur. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Long-Range Retrieval (Accuracy (%; the 993 long-range questions whose evidence starts late or spans at least ten minutes, both tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (85.3), 10 models
Long-range retrieval questions in MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, whose evidence appears late in the recording or is spread over at least ten minutes. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Chinese (Accuracy (%; the 337 Chinese-language questions over both evidence tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.7 Flash (74.8), 8 models
The Chinese questions of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, covering spoken content and natural sounds in original Chinese recordings rather than translations. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - English (Accuracy (%; the 347 English-language questions over both evidence tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (75.8), 8 models
The English questions of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, covering spoken content and natural sounds in original English recordings rather than translations. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Spanish (Accuracy (%; the 316 Spanish-language questions over both evidence tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (71.5), 8 models
The Spanish questions of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, covering spoken content and natural sounds in original Spanish recordings rather than translations. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - French (Accuracy (%; the 338 French-language questions over both evidence tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (76.3), 8 models
The French questions of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, covering spoken content and natural sounds in original French recordings rather than translations. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Japanese (Accuracy (%; the 320 Japanese-language questions over both evidence tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.7 Flash (78.8), 8 models
The Japanese questions of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, covering spoken content and natural sounds in original Japanese recordings rather than translations. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Korean (Accuracy (%; the 333 Korean-language questions over both evidence tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.7 Flash (68.5), 8 models
The Korean questions of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, covering spoken content and natural sounds in original Korean recordings rather than translations. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Indonesian (Accuracy (%; the 324 Indonesian-language questions over both evidence tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.7 Flash (74.4), 8 models
The Indonesian questions of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, covering spoken content and natural sounds in original Indonesian recordings rather than translations. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Russian (Accuracy (%; the 308 Russian-language questions over both evidence tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (72.1), 8 models
The Russian questions of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, covering spoken content and natural sounds in original Russian recordings rather than translations. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Arabic (Accuracy (%; the 313 Arabic-language questions over both evidence tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.7 Flash (72.5), 8 models
The Arabic questions of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, covering spoken content and natural sounds in original Arabic recordings rather than translations. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Thai (Accuracy (%; the 309 Thai-language questions over both evidence tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (72.5), 8 models
The Thai questions of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, covering spoken content and natural sounds in original Thai recordings rather than translations. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Vietnamese (Accuracy (%; the 288 Vietnamese-language questions over both evidence tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.7 Flash (78.1), 8 models
The Vietnamese questions of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, covering spoken content and natural sounds in original Vietnamese recordings rather than translations. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - German (Accuracy (%; the 304 German-language questions over both evidence tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (73.7), 8 models
The German questions of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, covering spoken content and natural sounds in original German recordings rather than translations. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Italian (Accuracy (%; the 304 Italian-language questions over both evidence tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (71.1), 8 models
The Italian questions of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, covering spoken content and natural sounds in original Italian recordings rather than translations. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Portuguese (Accuracy (%; the 306 Portuguese-language questions over both evidence tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.7 Flash (73.5), 8 models
The Portuguese questions of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, covering spoken content and natural sounds in original Portuguese recordings rather than translations. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Hindi (Accuracy (%; the 290 Hindi-language questions over both evidence tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (73.4), 8 models
The Hindi questions of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, covering spoken content and natural sounds in original Hindi recordings rather than translations. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Turkish (Accuracy (%; the 301 Turkish-language questions over both evidence tracks; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (77.4), 8 models
The Turkish questions of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone, covering spoken content and natural sounds in original Turkish recordings rather than translations. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Knowledge Domain (Accuracy (%; the 480 semantic-track questions from knowledge recordings, 30 in each of the 16 languages; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (80.0), 8 models
The knowledge domain of the semantic track of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone: 480 questions about spoken content in knowledge and educational recordings, 30 in each of 16 languages. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - How-to Domain (Accuracy (%; the 480 semantic-track questions from how-to recordings, 30 in each of the 16 languages; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.7 Flash (77.7), 8 models
The how-to domain of the semantic track of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone: 480 questions about spoken content in how-to and instructional recordings, 30 in each of 16 languages. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Narrative Domain (Accuracy (%; the 480 semantic-track questions from narrative recordings, 30 in each of the 16 languages; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (78.1), 8 models
The narrative domain of the semantic track of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone: 480 questions about spoken content in narrative and storytelling recordings, 30 in each of 16 languages. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Entertainment Domain (Accuracy (%; the 480 semantic-track questions from entertainment recordings, 30 in each of the 16 languages; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.7 Flash (82.5), 8 models
The entertainment domain of the semantic track of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone: 480 questions about spoken content in entertainment recordings, 30 in each of 16 languages. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Lifestyle Domain (Accuracy (%; the 480 semantic-track questions from lifestyle recordings, 30 in each of the 16 languages; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (80.8), 8 models
The lifestyle domain of the semantic track of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone: 480 questions about spoken content in lifestyle recordings, 30 in each of 16 languages. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - News Domain (Accuracy (%; the 480 semantic-track questions from news recordings, 30 in each of the 16 languages; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.7 Flash (77.5), 8 models
The news domain of the semantic track of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone: 480 questions about spoken content in news recordings, 30 in each of 16 languages. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Product Domain (Accuracy (%; the 480 semantic-track questions from product recordings, 30 in each of the 16 languages; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.7 Flash (78.8), 8 models
The product domain of the semantic track of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone: 480 questions about spoken content in product and review recordings, 30 in each of 16 languages. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
MuLA-Bench - Belief Domain (Accuracy (%; the 480 semantic-track questions from belief and spiritual culture recordings, 30 in each of the 16 languages; open-ended questions over in-the-wild long-form recordings (median 36.8 minutes, capped at three hours) in the source language, answered from the full recording without transcripts, audio clipped only to a local model's native input limit; each answer judged correct or not by GPT-5.6 Sol on a route-specific binary rubric with programmatic structural and 3-second timing checks; missing or invalid answers scored wrong)): leader Gemini 3.8 Flash (80.8), 8 models
The belief domain of the semantic track of MuLA-Bench (Yang et al., 2026), a benchmark of open-ended questions over long in-the-wild recordings, answered from the audio alone: 480 questions about spoken content in belief and spiritual recordings, 30 in each of 16 languages. Each answer is judged correct or wrong by GPT-5.6 Sol against the reference with rubric and timing checks; accuracy in percent, higher is better.
CESBench (Composite score (%; unweighted mean of the four task-type means: multiple-choice accuracy (209), judgment verdict-gated rubric score (67), scenario rubric score (63) and code task success on 572 hidden tests (41); 380 expert-written items on cryptographic engineering security for IoT devices; zero-shot, temperature 0, no tools, provider default reasoning mode; judgment and scenario answers scored by a Qwen3.5-397B judge from outside the evaluated set)): leader GLM-5.2 (83.6), 11 models
CESBench (Zhou et al., 2026) tests large language models on cryptographic engineering security for IoT devices with 380 expert-written items on side-channel and fault attacks, implementation, countermeasures, evaluation and integration. The composite score is the unweighted mean in percent of four task-type scores (multiple-choice accuracy, verdict-gated judgment rubric, scenario rubric and code task success on hidden tests), with open answers scored by a Qwen3.5-397B judge; higher is better.
CESBench - Multiple Choice (Accuracy (%; the 209 four-option multiple-choice items whose distractors encode practitioner misconceptions; 380 expert-written items on cryptographic engineering security for IoT devices; zero-shot, temperature 0, no tools, provider default reasoning mode; judgment and scenario answers scored by a Qwen3.5-397B judge from outside the evaluated set)): leader GLM-5.2 (98.6), 11 models
The 209 multiple-choice items of CESBench (Zhou et al., 2026) on cryptographic engineering security for IoT devices, with distractors that encode documented practitioner misconceptions. Accuracy in percent, zero-shot at temperature 0; higher is better.
CESBench - Judgment (Judgment score (%; the 67 true-or-false security claims: a wrong verdict scores 0, a right one earns the mean of four justification rubric points judged 0, 0.5 or 1; 380 expert-written items on cryptographic engineering security for IoT devices; zero-shot, temperature 0, no tools, provider default reasoning mode; judgment and scenario answers scored by a Qwen3.5-397B judge from outside the evaluated set)): leader GLM-5.2 (58.8), 11 models
The 67 judgment items of CESBench (Zhou et al., 2026): the model labels a technical security claim true or false and justifies it. A wrong verdict scores zero and a right one earns the mean of four expert rubric points judged by Qwen3.5-397B, reported in percent; higher is better.
CESBench - Scenario (Scenario score (%; the 63 open-ended engineering diagnosis items, mean of 3 to 5 expert rubric dimensions judged 0, 0.5 or 1; 380 expert-written items on cryptographic engineering security for IoT devices; zero-shot, temperature 0, no tools, provider default reasoning mode; judgment and scenario answers scored by a Qwen3.5-397B judge from outside the evaluated set)): leader GPT-5.6 Luna (88.4), 11 models
The 63 scenario items of CESBench (Zhou et al., 2026): open-ended diagnosis of a concrete cryptographic-engineering situation with platform, measurement and trace details. Scored in percent as the mean of three to five expert rubric dimensions judged by Qwen3.5-397B; higher is better.
CESBench - Code (Task success (%; the 41 Python and embedded C code tasks, success when every functional test passes and no process check (import allowlist, secret-independent control flow, Cortex-M0+ build and memory budget) fails; 380 expert-written items on cryptographic engineering security for IoT devices; zero-shot, temperature 0, no tools, provider default reasoning mode; judgment and scenario answers scored by a Qwen3.5-397B judge from outside the evaluated set)): leader DeepSeek V4 Pro (95.1), 11 models
The 41 code tasks of CESBench (Zhou et al., 2026) in Python and embedded C, such as masked or constant-time cryptographic routines. A task succeeds when every hidden test passes and no process check fails (import allowlist, secret-independent control flow, Cortex-M0+ build and memory budget); percent of tasks, higher is better.
CESBench - Side-Channel (Composite score (%; mean over the four task types of the item-score means within the D1 side-channel sub-domain (80 items); 380 expert-written items on cryptographic engineering security for IoT devices; zero-shot, temperature 0, no tools, provider default reasoning mode; judgment and scenario answers scored by a Qwen3.5-397B judge from outside the evaluated set)): leader Gemini 3.7 Flash (85.5), 11 models
The side-channel sub-domain of CESBench (Zhou et al., 2026): 80 items on leakage models, power and electromagnetic analysis, profiled attacks and timing channels, across multiple-choice, judgment, scenario and code tasks. Score in percent, the mean of the four task-type scores within the sub-domain; higher is better.
CESBench - Fault Injection (Composite score (%; mean over the four task types of the item-score means within the D2 fault-injection sub-domain (51 items); 380 expert-written items on cryptographic engineering security for IoT devices; zero-shot, temperature 0, no tools, provider default reasoning mode; judgment and scenario answers scored by a Qwen3.5-397B judge from outside the evaluated set)): leader GLM-5.2 (81.7), 11 models
The fault-injection sub-domain of CESBench (Zhou et al., 2026): 51 items on injection methods, fault models and classical fault attacks such as differential fault analysis, across multiple-choice, judgment, scenario and code tasks. Score in percent, the mean of the four task-type scores within the sub-domain; higher is better.
CESBench - Implementation (Composite score (%; mean over the four task types of the item-score means within the D3 implementation sub-domain (83 items); 380 expert-written items on cryptographic engineering security for IoT devices; zero-shot, temperature 0, no tools, provider default reasoning mode; judgment and scenario answers scored by a Qwen3.5-397B judge from outside the evaluated set)): leader GLM-5.2 (85.4), 11 models
The implementation sub-domain of CESBench (Zhou et al., 2026): 83 items on public-key, constant-time, post-quantum, AES and hardware implementation, across multiple-choice, judgment, scenario and code tasks. Score in percent, the mean of the four task-type scores within the sub-domain; higher is better.
CESBench - Countermeasures (Composite score (%; mean over the four task types of the item-score means within the D4 countermeasures sub-domain (62 items); 380 expert-written items on cryptographic engineering security for IoT devices; zero-shot, temperature 0, no tools, provider default reasoning mode; judgment and scenario answers scored by a Qwen3.5-397B judge from outside the evaluated set)): leader Kimi K2.6 (85.5), 11 models
The countermeasures sub-domain of CESBench (Zhou et al., 2026): 62 items on fault-injection countermeasures, hiding, masking and threshold implementations, across multiple-choice, judgment, scenario and code tasks. Score in percent, the mean of the four task-type scores within the sub-domain; higher is better.
CESBench - Evaluation (Composite score (%; mean over the four task types of the item-score means within the D5 evaluation and certification sub-domain (51 items); 380 expert-written items on cryptographic engineering security for IoT devices; zero-shot, temperature 0, no tools, provider default reasoning mode; judgment and scenario answers scored by a Qwen3.5-397B judge from outside the evaluated set)): leader Kimi K2.6 (80.5), 11 models
The evaluation sub-domain of CESBench (Zhou et al., 2026): 51 items on module standards and security levels, leakage assessment and random number generator testing, across multiple-choice, judgment, scenario and code tasks. Score in percent, the mean of the four task-type scores within the sub-domain; higher is better.
CESBench - Integration (Composite score (%; mean over the four task types of the item-score means within the D6 integration and key management sub-domain (53 items); 380 expert-written items on cryptographic engineering security for IoT devices; zero-shot, temperature 0, no tools, provider default reasoning mode; judgment and scenario answers scored by a Qwen3.5-397B judge from outside the evaluated set)): leader GLM-5.2 (90.7), 11 models
The integration sub-domain of CESBench (Zhou et al., 2026): 53 items on key management, trusted execution, secure boot, firmware update and public-key infrastructure, across multiple-choice, judgment, scenario and code tasks. Score in percent, the mean of the four task-type scores within the sub-domain; higher is better.
IndicBankBench (Strict pass^3 (%; cases passed out of all 799 cases (twelve behavioral axes and eight capability and refusal axes); each case run three times at temperature 0.7 with the stated reasoning modes; a case passes only when all three runs pass the deterministic safety and tool-use gates and the GLM-5.2 response judge; scripted Indian retail-banking conversations over deterministic mock tools)): leader DeepSeek V4 Pro (0813) (Max) (58.197747), 11 models
IndicBankBench (Paul et al., 2026) tests language-model assistants on 799 multi-turn Indian retail-banking conversations with stateful mock tools, covering five operational domains and a capability and refusal domain. Each case runs three times and passes only if every run clears deterministic safety and tool-use checks and a GLM-5.2 response judge; the score is the percentage of cases passed (strict pass^3), higher is better.
IndicBankBench - Happy Path (Strict pass^3 (%; 39 cases passed out of the 39 cases of the happy-path axis; each case run three times at temperature 0.7 with the stated reasoning modes; a case passes only when all three runs pass the deterministic safety and tool-use gates and the GLM-5.2 response judge; scripted Indian retail-banking conversations over deterministic mock tools)): leader DeepSeek V4 Pro (0813) (Max) (87.179487), 11 models
The happy-path axis of IndicBankBench (Paul et al., 2026): 39 straightforward retail-banking requests the assistant must resolve with the right tool calls. Percentage of cases passed in all three runs (strict pass^3), under deterministic safety and tool-use checks and a GLM-5.2 response judge; higher is better.
IndicBankBench - Bad Tool Response (Strict pass^3 (%; 36 cases passed out of the 36 cases of the bad-tool-response axis (misleading or failed tool results); each case run three times at temperature 0.7 with the stated reasoning modes; a case passes only when all three runs pass the deterministic safety and tool-use gates and the GLM-5.2 response judge; scripted Indian retail-banking conversations over deterministic mock tools)): leader Grok 4.6 (High) (69.444444), 11 models
The bad-tool-response axis of IndicBankBench (Paul et al., 2026): 36 retail-banking cases in which a tool returns a misleading or failed result. Percentage of cases passed in all three runs (strict pass^3), under deterministic safety and tool-use checks and a GLM-5.2 response judge; higher is better.
IndicBankBench - Confusing Intent (Strict pass^3 (%; 53 cases passed out of the 53 cases of the confusing-intent axis; each case run three times at temperature 0.7 with the stated reasoning modes; a case passes only when all three runs pass the deterministic safety and tool-use gates and the GLM-5.2 response judge; scripted Indian retail-banking conversations over deterministic mock tools)): leader GLM-5.3 Flash (High) (71.698113), 11 models
The confusing-intent axis of IndicBankBench (Paul et al., 2026): 53 retail-banking cases built around a confusing customer request. Percentage of cases passed in all three runs (strict pass^3), under deterministic safety and tool-use checks and a GLM-5.2 response judge; higher is better.
IndicBankBench - Not Enough Info (Strict pass^3 (%; 61 cases passed out of the 61 cases of the not-enough-information axis; each case run three times at temperature 0.7 with the stated reasoning modes; a case passes only when all three runs pass the deterministic safety and tool-use gates and the GLM-5.2 response judge; scripted Indian retail-banking conversations over deterministic mock tools)): leader Grok 4.6 (High) (62.295082), 11 models
The not-enough-information axis of IndicBankBench (Paul et al., 2026): 61 retail-banking cases in which the information needed to act is incomplete. Percentage of cases passed in all three runs (strict pass^3), under deterministic safety and tool-use checks and a GLM-5.2 response judge; higher is better.
IndicBankBench - Contradicting Info (Strict pass^3 (%; 77 cases passed out of the 77 cases of the contradicting-information axis; each case run three times at temperature 0.7 with the stated reasoning modes; a case passes only when all three runs pass the deterministic safety and tool-use gates and the GLM-5.2 response judge; scripted Indian retail-banking conversations over deterministic mock tools)): leader Claude Opus 5 (Max) (68.831169), 11 models
The contradicting-information axis of IndicBankBench (Paul et al., 2026): 77 retail-banking cases with contradictory information in the conversation or records. Percentage of cases passed in all three runs (strict pass^3), under deterministic safety and tool-use checks and a GLM-5.2 response judge; higher is better.
IndicBankBench - Context Switching (Strict pass^3 (%; 44 cases passed out of the 44 cases of the context-switching axis; each case run three times at temperature 0.7 with the stated reasoning modes; a case passes only when all three runs pass the deterministic safety and tool-use gates and the GLM-5.2 response judge; scripted Indian retail-banking conversations over deterministic mock tools)): leader GLM-5.3 Flash (High) (84.090909), 11 models
The context-switching axis of IndicBankBench (Paul et al., 2026): 44 retail-banking conversations that switch context midway. Percentage of cases passed in all three runs (strict pass^3), under deterministic safety and tool-use checks and a GLM-5.2 response judge; higher is better.
IndicBankBench - Wrong Info (Strict pass^3 (%; 142 cases passed out of the 142 cases of the wrong-information axis (plausible but incorrect customer claims); each case run three times at temperature 0.7 with the stated reasoning modes; a case passes only when all three runs pass the deterministic safety and tool-use gates and the GLM-5.2 response judge; scripted Indian retail-banking conversations over deterministic mock tools)): leader Claude Opus 5 (Max) (40.140845), 11 models
The wrong-information axis of IndicBankBench (Paul et al., 2026): 142 retail-banking cases in which the customer states a plausible but incorrect claim that the assistant must check against the records. Percentage of cases passed in all three runs (strict pass^3), under deterministic safety and tool-use checks and a GLM-5.2 response judge; higher is better.
IndicBankBench - Long Context (Strict pass^3 (%; 89 cases passed out of the 89 cases of the long-context axis; each case run three times at temperature 0.7 with the stated reasoning modes; a case passes only when all three runs pass the deterministic safety and tool-use gates and the GLM-5.2 response judge; scripted Indian retail-banking conversations over deterministic mock tools)): leader Gemma 4 31B (IT) (Thinking) (60.674157), 11 models
The long-context axis of IndicBankBench (Paul et al., 2026): 89 retail-banking cases with long conversation context. Percentage of cases passed in all three runs (strict pass^3), under deterministic safety and tool-use checks and a GLM-5.2 response judge; higher is better.
IndicBankBench - Irrelevant RAG (Strict pass^3 (%; 24 cases passed out of the 24 cases of the irrelevant-retrieval axis; each case run three times at temperature 0.7 with the stated reasoning modes; a case passes only when all three runs pass the deterministic safety and tool-use gates and the GLM-5.2 response judge; scripted Indian retail-banking conversations over deterministic mock tools)): leader Grok 4.6 (High) (75.0), 11 models
The irrelevant-retrieval axis of IndicBankBench (Paul et al., 2026): 24 retail-banking cases that supply irrelevant retrieved context. Percentage of cases passed in all three runs (strict pass^3), under deterministic safety and tool-use checks and a GLM-5.2 response judge; higher is better.
IndicBankBench - Out-of-Scope Refusal (Strict pass^3 (%; 40 cases passed out of the 40 cases of the out-of-scope refusal axis; each case run three times at temperature 0.7 with the stated reasoning modes; a case passes only when all three runs pass the deterministic safety and tool-use gates and the GLM-5.2 response judge; scripted Indian retail-banking conversations over deterministic mock tools)): leader Grok 4.6 (High) (77.5), 11 models
The out-of-scope refusal axis of IndicBankBench (Paul et al., 2026): 40 requests outside retail-banking self-service that the assistant should decline for the right reason. Percentage of cases passed in all three runs (strict pass^3), under deterministic safety and tool-use checks and a GLM-5.2 response judge; higher is better.
IndicBankBench - Multi-Tool Agentic (Strict pass^3 (%; 112 cases passed out of the 112 cases of the general multi-tool agentic axis; each case run three times at temperature 0.7 with the stated reasoning modes; a case passes only when all three runs pass the deterministic safety and tool-use gates and the GLM-5.2 response judge; scripted Indian retail-banking conversations over deterministic mock tools)): leader Claude Opus 5 (Max) (42.857143), 11 models
The general multi-tool agentic axis of IndicBankBench (Paul et al., 2026): 112 retail-banking cases that need several tool calls. Percentage of cases passed in all three runs (strict pass^3), under deterministic safety and tool-use checks and a GLM-5.2 response judge; higher is better.
IndicBankBench - Social Engineering Refusal (Strict pass^3 (%; 19 cases passed out of the 19 cases of the social-engineering capability and refusal axis; each case run three times at temperature 0.7 with the stated reasoning modes; a case passes only when all three runs pass the deterministic safety and tool-use gates and the GLM-5.2 response judge; scripted Indian retail-banking conversations over deterministic mock tools)): leader Gemini 3.8 Flash (89.473684), 11 models
The social-engineering axis of the capability and refusal domain of IndicBankBench (Paul et al., 2026): 19 requests the assistant should decline for the right reason. Percentage of cases passed in all three runs (strict pass^3), under deterministic safety and tool-use checks and a GLM-5.2 response judge; higher is better.
IndicBankBench - Third-Party Request Refusal (Strict pass^3 (%; 13 cases passed out of the 13 cases of the third-party capability and refusal axis (requests touching another customer); each case run three times at temperature 0.7 with the stated reasoning modes; a case passes only when all three runs pass the deterministic safety and tool-use gates and the GLM-5.2 response judge; scripted Indian retail-banking conversations over deterministic mock tools)): leader DeepSeek V4 Pro (0813) (Max) (100.0), 11 models
The third-party axis of the capability and refusal domain of IndicBankBench (Paul et al., 2026): 13 requests involving another party that the assistant should decline for the right reason. Percentage of cases passed in all three runs (strict pass^3), under deterministic safety and tool-use checks and a GLM-5.2 response judge; higher is better.
LegendBench (QA accuracy (%; proportion of all individual questions answered correctly; LegendBench snapshot of 26,689 legend-centric chart questions generated from 900 base charts with counterfactual legend variants; the same prompt template and hyperparameters for every model)): leader GPT-5.6 Sol (83.81), 12 models
LegendBench (Zhang et al., 2026) tests how vision-language models use chart legends, with generated charts, controlled legend interventions and questions on legend parsing, grounding, reasoning and abstention. QA accuracy is the percentage of all individual questions answered correctly; higher is better.
LegendBench - Guide Reading (Guide-reading accuracy (%; legend-parsing questions: extract the mapping the legend makes explicit; LegendBench snapshot of 26,689 legend-centric chart questions generated from 900 base charts with counterfactual legend variants; the same prompt template and hyperparameters for every model)): leader GPT-5.6 Sol (91.12), 12 models
Legend parsing in LegendBench (Zhang et al., 2026): questions that ask the model to read the mapping a legend states, such as which color or marker stands for which series or what a color bar encodes. Accuracy in percent; higher is better.
LegendBench - Binding (Binding accuracy (%; legend-grounding questions: associate a legend entry with its chart marks; LegendBench snapshot of 26,689 legend-centric chart questions generated from 900 base charts with counterfactual legend variants; the same prompt template and hyperparameters for every model)): leader GPT-5.6 Sol (84.39), 12 models
Legend grounding in LegendBench (Zhang et al., 2026): questions that require binding a legend entry to the corresponding marks in the chart. Accuracy in percent; higher is better.
LegendBench - Value Extraction (Value-extraction accuracy (%; legend-grounding questions: recover the quantitative value of the bound mark, cell or colour-scale position; LegendBench snapshot of 26,689 legend-centric chart questions generated from 900 base charts with counterfactual legend variants; the same prompt template and hyperparameters for every model)): leader GPT-5.6 Sol (78.31), 12 models
Value extraction in LegendBench (Zhang et al., 2026): questions that require recovering the quantitative value of a legend-bound mark, cell or color-scale position. Accuracy in percent; higher is better.
LegendBench - Reasoning (Reasoning accuracy (%; raw accuracy on legend-conditioned questions that require inference; LegendBench snapshot of 26,689 legend-centric chart questions generated from 900 base charts with counterfactual legend variants; the same prompt template and hyperparameters for every model)): leader GPT-5.6 Sol (74.16), 12 models
Legend-conditioned reasoning in LegendBench (Zhang et al., 2026): questions that require inference over legend-bound values. Raw accuracy in percent; higher is better.
LegendBench - Abstention F1 (Abstention F1 (%; harmonic mean of abstention precision and recall, abstaining when the legend evidence needed is masked or missing; LegendBench snapshot of 26,689 legend-centric chart questions generated from 900 base charts with counterfactual legend variants; the same prompt template and hyperparameters for every model)): leader GPT-5.6 Sol (97.46), 12 models
Legend-aware abstention in LegendBench (Zhang et al., 2026): charts whose legend evidence is masked or missing, where the model should decline rather than guess. Abstention F1 in percent, the harmonic mean of the precision and recall of its refusals; higher is better.
VA-Bench (Task success (%; all 14 base task families (11 single-arm, 3 dual-arm); 20 physically verified seeds per task family in RoboTwin simulation; the model watches an RGB-only expert video, writes its own task summary, chooses camera viewpoints and issues metric Cartesian arm commands that a fixed model-agnostic controller executes, with execution feedback; terminal success from the environment checker; task-family macro-average, mean of three runs)): leader Qwen 3.8 Max (53.93), 12 models
VA-Bench (Zhang et al., 2026) tests whether general-purpose multimodal models can carry out simulated robot manipulation from visual demonstrations: the model watches an RGB-only expert video, chooses camera viewpoints, issues metric Cartesian arm commands for a fixed controller and revises them from execution feedback. Terminal task success in percent, macro-averaged over 14 task families (11 single-arm, 3 dual-arm) with 20 seeds each and averaged over three runs; higher is better.
VA-Bench - Single-Arm (Task success (%; the 11 single-arm base task families; 20 physically verified seeds per task family in RoboTwin simulation; the model watches an RGB-only expert video, writes its own task summary, chooses camera viewpoints and issues metric Cartesian arm commands that a fixed model-agnostic controller executes, with execution feedback; terminal success from the environment checker; task-family macro-average, mean of three runs)): leader Qwen 3.8 Max (65.61), 12 models
The 11 single-arm task families of VA-Bench (Zhang et al., 2026), in which a general-purpose multimodal model learns a manipulation task from an RGB-only expert video, chooses camera viewpoints and issues metric arm commands for a fixed controller. Terminal task success in percent, macro-averaged over the families with 20 seeds each and averaged over three runs; higher is better.
FinInteract (Frontier Pilot) (Accuracy (%; the stratified 50-instance pilot subset run for the cross-vendor panel; ReAct agent in the standard full-interaction mode (A+S+I): it may search the filing corpus, ask the simulated user (GPT-5, answering yes, no or I don't know from the hidden disambiguating context) yes-or-no questions, and answer; a final answer counts as correct only if it matches the intended interpretation, graded by GPT-4o-mini with a 1% numeric tolerance)): leader Claude Sonnet 5 (44.0), 7 models
The FinInteract protocol (Wang et al., 2026) run as a preliminary pilot on a stratified 50-instance subset: the agent may search the filings and ask a simulated user yes-or-no questions, and must answer under the intended interpretation. Accuracy in percent on the pilot subset, which is not comparable with the full 173-instance board; higher is better.
FinFIRST (Weighted rubric score (%; Loose Pass, share of the 12,300 expert-weighted rubric points satisfied; 123 expert-authored bilingual financial research tasks, each scored against expert atomic rubric criteria (701 in all, weights summing to 100 per task) by a GLM-5.1 judge validated against finance experts (Cohen kappa 0.816); all models run in the same ReAct harness with web search, page visit and Python tools, temperature 1.0, one run; missing outputs count as failed)): leader Claude Opus 5 (87.61), 15 models
FinFIRST (Wang et al., 2026) evaluates financial search agents on 123 expert-authored Chinese and English research questions that need time-valid facts from authoritative sources plus calculations, scored with expert-weighted atomic rubric criteria. The score is the percentage of rubric points satisfied. Every model runs in the same ReAct harness with web search, page visit and Python tools, and a GLM-5.1 judge checks each criterion; higher is better.
FinFIRST - Raw-Information Acquisition (Weighted rubric score (%; the 7,322 rubric points for retrieving the correct values for the required entity, period, unit, definition and data version; 123 expert-authored bilingual financial research tasks, each scored against expert atomic rubric criteria (701 in all, weights summing to 100 per task) by a GLM-5.1 judge validated against finance experts (Cohen kappa 0.816); all models run in the same ReAct harness with web search, page visit and Python tools, temperature 1.0, one run; missing outputs count as failed)): leader Claude Opus 5 (88.98), 15 models
The raw-information acquisition rubric of FinFIRST (Wang et al., 2026): the percentage of rubric points for retrieving the correct values for the required entity, period, unit, definition and data version in expert-authored financial research questions. Every model runs in the same ReAct harness with web search, page visit and Python tools, and a GLM-5.1 judge checks each criterion; higher is better.
FinFIRST - Source Verification (Weighted rubric score (%; the 2,085 rubric points for using authoritative, relevant and time-valid sources; 123 expert-authored bilingual financial research tasks, each scored against expert atomic rubric criteria (701 in all, weights summing to 100 per task) by a GLM-5.1 judge validated against finance experts (Cohen kappa 0.816); all models run in the same ReAct harness with web search, page visit and Python tools, temperature 1.0, one run; missing outputs count as failed)): leader Claude Opus 5 (89.59), 15 models
The source verification rubric of FinFIRST (Wang et al., 2026): the percentage of rubric points for grounding answers to expert-authored financial research questions in authoritative, relevant and time-valid sources. Every model runs in the same ReAct harness with web search, page visit and Python tools, and a GLM-5.1 judge checks each criterion; higher is better.
FinFIRST - Computation and Answer Formation (Weighted rubric score (%; the 2,893 rubric points for calculation, reasoning and an evidence-supported final answer; 123 expert-authored bilingual financial research tasks, each scored against expert atomic rubric criteria (701 in all, weights summing to 100 per task) by a GLM-5.1 judge validated against finance experts (Cohen kappa 0.816); all models run in the same ReAct harness with web search, page visit and Python tools, temperature 1.0, one run; missing outputs count as failed)): leader Claude Opus 5 (82.72), 15 models
The computation and answer formation rubric of FinFIRST (Wang et al., 2026): the percentage of rubric points for calculation, reasoning and an evidence-supported final answer in expert-authored financial research questions. Every model runs in the same ReAct harness with web search, page visit and Python tools, and a GLM-5.1 judge checks each criterion; higher is better.
RxScribe Bench - Correctness (Field-level F score (%; harmonic mean of credit-weighted precision and recall per run, averaged over the three runs and then over images; all fields; 200 handwritten Indian outpatient prescription images, three independent cold runs per image, image and JSON schema only, deterministic field-by-field scoring against human ground truth with no model judge)): leader GPT-5.6 Sol (68.1), 4 models
RxScribe Bench (2026) asks vision-language models to transcribe handwritten Indian outpatient prescriptions into a structured record given only the image and a JSON schema. Correctness is a field-level F score in percent that balances precision and recall of the filled fields, averaged over three cold runs on each of 200 prescriptions and scored deterministically against human ground truth; higher is better.
RxScribe Bench - Correctness (Medicine Table) (Field-level F score (%; harmonic mean of credit-weighted precision and recall per run, averaged over the three runs and then over images; the medicine table only: drug name, corrected name, inferred active ingredient, frequency, duration, dosage form and relation to meals, each cell pass or fail; 200 handwritten Indian outpatient prescription images, three independent cold runs per image, image and JSON schema only, deterministic field-by-field scoring against human ground truth with no model judge)): leader GPT-5.6 Sol (57.83), 4 models
RxScribe Bench (2026) correctness on the medicine table alone: drug name, corrected name, inferred active ingredient, frequency, duration, dosage form and relation to meals, each cell graded pass or fail. Field-level F score in percent over 200 handwritten prescriptions with three runs each; higher is better.
RxScribe Bench - Correctness (Clinical and Non-Clinical Fields) (Field-level F score (%; harmonic mean of credit-weighted precision and recall per run, averaged over the three runs and then over images; diagnoses, symptoms, vitals, history, allergies and tests plus prescriber, facility and date fields, scored by ANLS string similarity; 200 handwritten Indian outpatient prescription images, three independent cold runs per image, image and JSON schema only, deterministic field-by-field scoring against human ground truth with no model judge)): leader Gemini 3.1 Pro (Preview) (78.94), 4 models
RxScribe Bench (2026) correctness outside the medicine table: diagnoses, symptoms, vitals, history, allergies and recommended tests plus prescriber, facility and date fields, scored by normalized edit-distance similarity. Field-level F score in percent over 200 handwritten prescriptions with three runs each; higher is better.
RxScribe Bench - Fabrications per Run (Fabrication events per run (lower is better; invented medicine rows or wrongly inferred active ingredients, clinical values and non-clinical fields asserted without support in the image, summed over 600 runs and divided by the run count; 200 handwritten Indian outpatient prescription images, three independent cold runs per image, image and JSON schema only, deterministic field-by-field scoring against human ground truth with no model judge)): leader Muse Spark 1.2 (2.28), 4 models
RxScribe Bench (2026) hallucination axis: fabrication events per transcription run, counting invented medicine rows or wrongly inferred active ingredients, clinical values and administrative fields that the prescription image does not support. Mean over 600 runs on 200 handwritten prescriptions; lower is better.
RxScribe Bench - Hard Fields (Delivered) (Hard-field score (%; field credit on the 1,740 fields annotated as partially legible or blurred but recoverable, with a skipped field scored as zero; 200 handwritten Indian outpatient prescription images, three independent cold runs per image, image and JSON schema only, deterministic field-by-field scoring against human ground truth with no model judge)): leader Gemini 3.1 Pro (Preview) (51.03), 4 models
RxScribe Bench (2026) robustness axis: field credit on the 1,740 fields that annotators marked as partly legible or blurred but still recoverable, with a skipped field counted as zero. Percent over three runs on each of 200 handwritten prescriptions; higher is better.
Omni2Web - Direct Editing (Edit Fidelity Score (%; Track A: 0.10 target grounding + 0.50 edit fulfillment + 0.20 no-damage + 0.20 revision handling per instance, averaged over 918 bilingual screen recordings of weakly referential webpage edit requests (speech plus cursor), 13,907 edit steps; the model edits the source HTML directly from the recording under its native input interface (omni: audio-visual video; vision-language: video or 1 fps frames plus manual transcript); rubric items judged by Gemini 3.1 Pro; unusable outputs score zero)): leader Gemini 3.5 Flash (51.17), 15 models
Omni2Web (Han et al., 2026) asks multimodal models to edit a source webpage directly from 918 bilingual screen recordings of weakly referential web-editing requests, where speech, cursor movement and edit history identify the targets. Edit Fidelity Score in percent combines target grounding, edit fulfilment, preservation of unrelated content and revision handling, judged against step-level rubrics by Gemini 3.1 Pro; higher is better.
Omni2Web - Direct Editing Grounding (Target-grounding rate (%; Track A rubric items: the edited HTML changes the element each active step refers to; 918 bilingual screen recordings of weakly referential webpage edit requests (speech plus cursor), 13,907 edit steps; the model edits the source HTML directly from the recording under its native input interface (omni: audio-visual video; vision-language: video or 1 fps frames plus manual transcript); rubric items judged by Gemini 3.1 Pro; unusable outputs score zero)): leader Gemini 3.5 Flash (59.2), 15 models
The target-grounding component of Omni2Web direct editing (Han et al., 2026): the share of rubric items for which the edited page changes the element each step refers to, over 918 bilingual screen recordings of weakly referential web-editing requests, where speech, cursor movement and edit history identify the targets. Percent, judged by Gemini 3.1 Pro; higher is better.
Omni2Web - Direct Editing Fulfillment (Edit-fulfillment rate (%; Track A rubric items: each active step's requested change is realised in the edited HTML; 918 bilingual screen recordings of weakly referential webpage edit requests (speech plus cursor), 13,907 edit steps; the model edits the source HTML directly from the recording under its native input interface (omni: audio-visual video; vision-language: video or 1 fps frames plus manual transcript); rubric items judged by Gemini 3.1 Pro; unusable outputs score zero)): leader Gemini 3.5 Flash (40.52), 15 models
The edit-fulfilment component of Omni2Web direct editing (Han et al., 2026): the share of rubric items whose requested change is actually realised in the edited HTML, over 918 bilingual screen recordings of weakly referential web-editing requests, where speech, cursor movement and edit history identify the targets. Percent, judged by Gemini 3.1 Pro; higher is better.
Omni2Web - Direct Editing Revision Handling (Revision-handling rate (%; Track A rubric items: revision and undo steps are honoured in the final page; 918 bilingual screen recordings of weakly referential webpage edit requests (speech plus cursor), 13,907 edit steps; the model edits the source HTML directly from the recording under its native input interface (omni: audio-visual video; vision-language: video or 1 fps frames plus manual transcript); rubric items judged by Gemini 3.1 Pro; unusable outputs score zero)): leader Gemini 3.5 Flash (71.26), 15 models
The revision-handling component of Omni2Web direct editing (Han et al., 2026): the share of revision and undo steps honoured in the final page, over 918 bilingual screen recordings of weakly referential web-editing requests, where speech, cursor movement and edit history identify the targets. Percent, judged by Gemini 3.1 Pro; higher is better.
Omni2Web - Instruction Recovery (Instruction Recovery Score (%; Track B: mean of target F1 and action F1 per instance, macro-averaged over 918 bilingual screen recordings of weakly referential webpage edit requests; the model rewrites the recording into an ordered list of explicit target and action instructions without the source HTML; one-to-one matching against reference instructions, semantic equivalence judged by Gemini 3.1 Pro; unusable outputs score zero)): leader Qwen 3.5 Omni Plus (49.14), 17 models
Omni2Web (Han et al., 2026) asks multimodal models to rewrite 918 bilingual screen recordings of weakly referential web-editing requests, where speech, cursor movement and edit history identify the targets into an ordered list of explicit targets and editing actions, without the source HTML. Instruction Recovery Score in percent is the mean of target F1 and action F1 after one-to-one matching with the reference instructions; higher is better.
Omni2Web - Instruction Recovery Target F1 (Target F1 (%; Track B component: recovered edit targets matched one-to-one to the reference targets; 918 bilingual screen recordings of weakly referential webpage edit requests; the model rewrites the recording into an ordered list of explicit target and action instructions without the source HTML; one-to-one matching against reference instructions, semantic equivalence judged by Gemini 3.1 Pro; unusable outputs score zero)): leader Qwen 3.5 Omni Plus (49.99), 17 models
The target component of Omni2Web instruction recovery (Han et al., 2026): F1 in percent of the recovered edit targets against the reference targets, from 918 bilingual screen recordings of weakly referential web-editing requests, where speech, cursor movement and edit history identify the targets. Extra or duplicate predictions lower the score; higher is better.
Omni2Web - Instruction Recovery Action F1 (Action F1 (%; Track B component: recovered edit actions matched one-to-one to the reference actions; 918 bilingual screen recordings of weakly referential webpage edit requests; the model rewrites the recording into an ordered list of explicit target and action instructions without the source HTML; one-to-one matching against reference instructions, semantic equivalence judged by Gemini 3.1 Pro; unusable outputs score zero)): leader Claude Opus 5 (49.76), 17 models
The action component of Omni2Web instruction recovery (Han et al., 2026): F1 in percent of the recovered editing actions against the reference actions, from 918 bilingual screen recordings of weakly referential web-editing requests, where speech, cursor movement and edit history identify the targets. Extra or duplicate predictions lower the score; higher is better.
Omni2Web - Instruction Utility (Edit Fidelity Score (%; Track C: the model's Track B instructions are executed on the source HTML by a fixed DeepSeek-v4-pro coding model, then scored with the Track A rubric: 0.10 grounding + 0.50 edit fulfillment + 0.20 no-damage + 0.20 revision handling; 918 instances; judged by Gemini 3.1 Pro)): leader Qwen 3.5 Omni Plus (51.08), 17 models
Omni2Web (Han et al., 2026) measures how useful the instructions recovered by each model are: a fixed DeepSeek-v4-pro coding model executes them on the source page, and the result is scored with the direct-editing rubric. Edit Fidelity Score in percent over the 918 recordings; higher is better.
ODU-Bench (Audio-Visual) (Weighted demand-understanding score (%; 0.15 demand detection + 0.60 key-point coverage + 0.10 temporal localization + 0.10 transcription + 0.05 user profile, on 1,347 audio-visual scenes (1,061 with a demand, 286 without; 1,070 synthetic and 277 human-recorded), user turns as video with audio, earlier assistant replies as text; key points judged by Qwen3.6-Flash with thinking)): leader Gemini 3.1 Pro (Preview) (72.6), 12 models
ODU-Bench (Chen et al., 2026) tests Omni Demand Understanding: whether a model infers whether the final spoken user turn in a multimodal interaction carries a demand for the assistant and recovers it from visual, acoustic and dialogue context. Weighted score in percent over 1,347 audio-visual scenes (1,070 synthetic and 277 human-recorded; user turns as video with audio, earlier assistant replies as text), combining demand detection, key-point coverage, temporal localization, transcription and user profile; higher is better.
ODU-Bench (Audio-Visual) - Demand Detection (Macro-F1 (%; whether the final user turn carries a valid demand for the assistant, over demand and no-demand scenes; 1,347 audio-visual scenes (1,061 with a demand, 286 without; 1,070 synthetic and 277 human-recorded), user turns as video with audio, earlier assistant replies as text)): leader Gemini 3.7 Flash (83.9), 12 models
Demand detection in ODU-Bench (Chen et al., 2026): whether the final user turn carries a valid demand for the assistant, on 1,347 audio-visual scenes (1,070 synthetic and 277 human-recorded; user turns as video with audio, earlier assistant replies as text). Macro-F1 in percent over demand and no-demand scenes; higher is better.
ODU-Bench (Audio-Visual) - Key-Point Coverage (Key-point hit rate (%; share of reference key points on structured intent and required context covered by the predicted intent and context, demand scenes only; 1,347 audio-visual scenes (1,061 with a demand, 286 without; 1,070 synthetic and 277 human-recorded), user turns as video with audio, earlier assistant replies as text; key points judged by Qwen3.6-Flash with thinking)): leader Qwen 3.5 Omni Plus (65.8), 12 models
The primary semantic dimension of ODU-Bench (Chen et al., 2026): the share of reference key points on the intended outcome and required context that the predicted demand covers, on the demand scenes of 1,347 audio-visual scenes (1,070 synthetic and 277 human-recorded; user turns as video with audio, earlier assistant replies as text). Percent, judged by Qwen3.6-Flash; higher is better.
ODU-Bench (Audio-Visual) - Demand Localization (Temporal IoU (%; overlap of the predicted and reference demand spans in the final user clip, demand scenes only; 1,347 audio-visual scenes (1,061 with a demand, 286 without; 1,070 synthetic and 277 human-recorded), user turns as video with audio, earlier assistant replies as text)): leader Seed 2.0 Lite (82.8), 12 models
Temporal localization in ODU-Bench (Chen et al., 2026): overlap between the predicted and reference time span of the demand in the final user clip, on the demand scenes of 1,347 audio-visual scenes (1,070 synthetic and 277 human-recorded; user turns as video with audio, earlier assistant replies as text). Temporal IoU in percent; higher is better.
ODU-Bench (Audio-Visual) - Transcription (Transcript accuracy (%; max(0, 1 - pooled CER or WER) of the demand transcript, demand scenes only; 1,347 audio-visual scenes (1,061 with a demand, 286 without; 1,070 synthetic and 277 human-recorded), user turns as video with audio, earlier assistant replies as text)): leader Gemini 3.1 Pro (Preview) (91.3), 12 models
Transcription in ODU-Bench (Chen et al., 2026): accuracy of the transcript of the spoken demand, one minus the pooled character or word error rate, on the demand scenes of 1,347 audio-visual scenes (1,070 synthetic and 277 human-recorded; user turns as video with audio, earlier assistant replies as text). Percent; higher is better.
ODU-Bench (Audio-Visual) - User Profile (Profile accuracy (%; mean per-field accuracy of predicted user-in-frame, gender and age group (gender and age group without video), demand scenes only; 1,347 audio-visual scenes (1,061 with a demand, 286 without; 1,070 synthetic and 277 human-recorded), user turns as video with audio, earlier assistant replies as text)): leader Gemini 3.1 Pro (Preview) (95.7), 12 models
User profiling in ODU-Bench (Chen et al., 2026): accuracy of predicted speaker attributes (gender, age group and, with video, whether the user is in frame) for the demand, on the demand scenes of 1,347 audio-visual scenes (1,070 synthetic and 277 human-recorded; user turns as video with audio, earlier assistant replies as text). Percent; higher is better.
ODU-Bench (Audio-Only) (Weighted demand-understanding score (%; 0.15 demand detection + 0.60 key-point coverage + 0.10 temporal localization + 0.10 transcription + 0.05 user profile, on 731 synthetic audio-only scenes (570 with a demand, 161 without), user turns as audio, earlier assistant replies as text; key points judged by Qwen3.6-Flash with thinking)): leader Seed 2.0 Lite (77.3), 14 models
ODU-Bench (Chen et al., 2026) tests Omni Demand Understanding: whether a model infers whether the final spoken user turn in a multimodal interaction carries a demand for the assistant and recovers it from visual, acoustic and dialogue context. Weighted score in percent over 731 synthetic audio-only scenes (user turns as audio, earlier assistant replies as text), combining demand detection, key-point coverage, temporal localization, transcription and user profile; higher is better.
ODU-Bench (Audio-Only) - Demand Detection (Macro-F1 (%; whether the final user turn carries a valid demand for the assistant, over demand and no-demand scenes; 731 synthetic audio-only scenes (570 with a demand, 161 without), user turns as audio, earlier assistant replies as text)): leader Gemini 3.7 Flash (89.6), 14 models
Demand detection in ODU-Bench (Chen et al., 2026): whether the final user turn carries a valid demand for the assistant, on 731 synthetic audio-only scenes (user turns as audio, earlier assistant replies as text). Macro-F1 in percent over demand and no-demand scenes; higher is better.
ODU-Bench (Audio-Only) - Key-Point Coverage (Key-point hit rate (%; share of reference key points on structured intent and required context covered by the predicted intent and context, demand scenes only; 731 synthetic audio-only scenes (570 with a demand, 161 without), user turns as audio, earlier assistant replies as text; key points judged by Qwen3.6-Flash with thinking)): leader Seed 2.0 Lite (78.0), 14 models
The primary semantic dimension of ODU-Bench (Chen et al., 2026): the share of reference key points on the intended outcome and required context that the predicted demand covers, on the demand scenes of 731 synthetic audio-only scenes (user turns as audio, earlier assistant replies as text). Percent, judged by Qwen3.6-Flash; higher is better.
ODU-Bench (Audio-Only) - Demand Localization (Temporal IoU (%; overlap of the predicted and reference demand spans in the final user clip, demand scenes only; 731 synthetic audio-only scenes (570 with a demand, 161 without), user turns as audio, earlier assistant replies as text)): leader Seed 2.0 Lite (71.4), 14 models
Temporal localization in ODU-Bench (Chen et al., 2026): overlap between the predicted and reference time span of the demand in the final user clip, on the demand scenes of 731 synthetic audio-only scenes (user turns as audio, earlier assistant replies as text). Temporal IoU in percent; higher is better.
ODU-Bench (Audio-Only) - Transcription (Transcript accuracy (%; max(0, 1 - pooled CER or WER) of the demand transcript, demand scenes only; 731 synthetic audio-only scenes (570 with a demand, 161 without), user turns as audio, earlier assistant replies as text)): leader Seed 2.0 Lite (93.2), 14 models
Transcription in ODU-Bench (Chen et al., 2026): accuracy of the transcript of the spoken demand, one minus the pooled character or word error rate, on the demand scenes of 731 synthetic audio-only scenes (user turns as audio, earlier assistant replies as text). Percent; higher is better.
ODU-Bench (Audio-Only) - User Profile (Profile accuracy (%; mean per-field accuracy of predicted user-in-frame, gender and age group (gender and age group without video), demand scenes only; 731 synthetic audio-only scenes (570 with a demand, 161 without), user turns as audio, earlier assistant replies as text)): leader Gemini 3.1 Pro (Preview) (97.6), 14 models
User profiling in ODU-Bench (Chen et al., 2026): accuracy of predicted speaker attributes (gender, age group and, with video, whether the user is in frame) for the demand, on the demand scenes of 731 synthetic audio-only scenes (user turns as audio, earlier assistant replies as text). Percent; higher is better.
CraftBench-UE (C++, File-Shell) (Task completion (%; C++ tasks completed within 40 minutes over 33 tasks, file and shell tools without editor MCP, deterministic build and runtime checks, Claude Code driver)): leader GPT-5.6 Sol (87.878788), 7 models
CraftBench-UE (2026) evaluates coding agents on bounded gameplay and editor tasks in Unreal Engine 5.8, reconstructing each saved submission in a fresh project and applying deterministic build, runtime and saved-asset checks with no LLM judge. This board is C++ source tasks completed within a 40-minute budget under a file-and-shell agent without an editor MCP server, over 33 C++ tasks, driven by the Claude Code CLI; the metric is task completion in percent, higher is better.
CraftBench-UE (C++, Unreal MCP) (Task completion (%; C++ tasks completed within 40 minutes over 33 tasks, Unreal MCP editor tools, deterministic build and runtime checks, Claude Code driver)): leader Claude Sonnet 5 (90.909091), 7 models
CraftBench-UE (2026) evaluates coding agents on bounded gameplay and editor tasks in Unreal Engine 5.8, reconstructing each saved submission in a fresh project and applying deterministic build, runtime and saved-asset checks with no LLM judge. This board is C++ source tasks completed within a 40-minute budget under the Unreal MCP editor tool server, over 33 C++ tasks, driven by the Claude Code CLI; the metric is task completion in percent, higher is better.
CraftBench-UE (C++, Aura MCP) (Task completion (%; C++ tasks completed within 40 minutes over 33 tasks, Aura MCP editor tools, deterministic build and runtime checks, Claude Code driver)): leader Gemini 3.7 Flash (93.939394), 7 models
CraftBench-UE (2026) evaluates coding agents on bounded gameplay and editor tasks in Unreal Engine 5.8, reconstructing each saved submission in a fresh project and applying deterministic build, runtime and saved-asset checks with no LLM judge. This board is C++ source tasks completed within a 40-minute budget under the Aura MCP editor tool server, over 33 C++ tasks, driven by the Claude Code CLI; the metric is task completion in percent, higher is better.
CraftBench-UE (Blueprint, Unreal MCP) (Task completion (%; Blueprint tasks completed within 40 minutes over 25 tasks, Unreal MCP editor tools, build, runtime and saved-asset checks, Claude Code driver)): leader Claude Sonnet 5 (76.0), 7 models
CraftBench-UE (2026) evaluates coding agents on bounded gameplay and editor tasks in Unreal Engine 5.8, reconstructing each saved submission in a fresh project and applying deterministic build, runtime and saved-asset checks with no LLM judge. This board is Blueprint asset tasks completed within a 40-minute budget under the Unreal MCP editor tool server, over 25 Blueprint tasks, driven by the Claude Code CLI; the metric is task completion in percent, higher is better.
CraftBench-UE (Blueprint, Aura MCP) (Task completion (%; Blueprint tasks completed within 40 minutes over 25 tasks, Aura MCP editor tools, build, runtime and saved-asset checks, Claude Code driver)): leader GPT-5.6 Sol (76.0), 7 models
CraftBench-UE (2026) evaluates coding agents on bounded gameplay and editor tasks in Unreal Engine 5.8, reconstructing each saved submission in a fresh project and applying deterministic build, runtime and saved-asset checks with no LLM judge. This board is Blueprint asset tasks completed within a 40-minute budget under the Aura MCP editor tool server, over 25 Blueprint tasks, driven by the Claude Code CLI; the metric is task completion in percent, higher is better.
CraftBench-UE (Python, Unreal MCP) (Task completion (%; editor-scripting tasks completed within 40 minutes over 12 tasks, Unreal MCP editor tools, saved-project-state checks, Claude Code driver)): leader Claude Sonnet 5 (91.666667), 7 models
CraftBench-UE (2026) evaluates coding agents on bounded gameplay and editor tasks in Unreal Engine 5.8, reconstructing each saved submission in a fresh project and applying deterministic build, runtime and saved-asset checks with no LLM judge. This board is editor-scripting tasks completed within a 40-minute budget under the Unreal MCP editor tool server, over 12 editor-scripting tasks, driven by the Claude Code CLI; the metric is task completion in percent, higher is better.
CraftBench-UE (Python, Aura MCP) (Task completion (%; editor-scripting tasks completed within 40 minutes over 12 tasks, Aura MCP editor tools, saved-project-state checks, Claude Code driver)): leader DeepSeek V4 Pro (0813) (100.0), 7 models
CraftBench-UE (2026) evaluates coding agents on bounded gameplay and editor tasks in Unreal Engine 5.8, reconstructing each saved submission in a fresh project and applying deterministic build, runtime and saved-asset checks with no LLM judge. This board is editor-scripting tasks completed within a 40-minute budget under the Aura MCP editor tool server, over 12 editor-scripting tasks, driven by the Claude Code CLI; the metric is task completion in percent, higher is better.
SafeSceneReason (Accuracy (%; micro average over all 1,000 questions; single-answer multiple choice on the 1,000-question test set balanced across answer positions, built from workplace scene graphs and NIOSH FACE and CSB accident-report figures; temperature 0, unparseable answers counted wrong)): leader GPT-5.6 Sol (88.7), 8 models
SafeSceneReason (Shanghai Jiao Tong University and collaborators, 2026) is a multimodal industrial-safety reasoning benchmark linking workplace scenes and accident-report figures with multiple-choice questions. Micro accuracy over all 1,000 test questions, in percent; higher is better.
SafeSceneReason - Quantitative Reasoning (Accuracy (%; the 279 quantitative reasoning questions; single-answer multiple choice on the 1,000-question test set balanced across answer positions, built from workplace scene graphs and NIOSH FACE and CSB accident-report figures; temperature 0, unparseable answers counted wrong)): leader GPT-5.6 Sol (87.81), 8 models
Accuracy on the quantitative questions of SafeSceneReason (2026), an industrial-safety multiple-choice benchmark. Percent, higher is better.
SafeSceneReason - Spatial and Structural Reasoning (Accuracy (%; the 164 spatial/structural reasoning questions; single-answer multiple choice on the 1,000-question test set balanced across answer positions, built from workplace scene graphs and NIOSH FACE and CSB accident-report figures; temperature 0, unparseable answers counted wrong)): leader GPT-5.6 Sol (86.59), 8 models
Accuracy on the spatial and structural questions of SafeSceneReason (2026), an industrial-safety multiple-choice benchmark. Percent, higher is better.
SafeSceneReason - Causal Reasoning (Accuracy (%; the 163 causal reasoning questions; single-answer multiple choice on the 1,000-question test set balanced across answer positions, built from workplace scene graphs and NIOSH FACE and CSB accident-report figures; temperature 0, unparseable answers counted wrong)): leader GPT-5.6 Sol (93.25), 8 models
Accuracy on the causal questions of SafeSceneReason (2026), an industrial-safety multiple-choice benchmark. Percent, higher is better.
SafeSceneReason - Comparative Reasoning (Accuracy (%; the 108 comparative reasoning questions; single-answer multiple choice on the 1,000-question test set balanced across answer positions, built from workplace scene graphs and NIOSH FACE and CSB accident-report figures; temperature 0, unparseable answers counted wrong)): leader GPT-5.5 (86.11), 8 models
Accuracy on the comparative questions of SafeSceneReason (2026), an industrial-safety multiple-choice benchmark. Percent, higher is better.
SafeSceneReason - Evidence Synthesis (Accuracy (%; the 101 evidence synthesis questions; single-answer multiple choice on the 1,000-question test set balanced across answer positions, built from workplace scene graphs and NIOSH FACE and CSB accident-report figures; temperature 0, unparseable answers counted wrong)): leader GPT-5.5 (93.07), 8 models
Accuracy on the evidence-synthesis questions of SafeSceneReason (2026), an industrial-safety multiple-choice benchmark. Percent, higher is better.
SafeSceneReason - Hazard Identification (Accuracy (%; the 84 hazard identification questions; single-answer multiple choice on the 1,000-question test set balanced across answer positions, built from workplace scene graphs and NIOSH FACE and CSB accident-report figures; temperature 0, unparseable answers counted wrong)): leader GPT-5.6 Sol (89.29), 8 models
Accuracy on the hazard-identification questions of SafeSceneReason (2026), an industrial-safety multiple-choice benchmark. Percent, higher is better.
SafeSceneReason - Compliance Assessment (Accuracy (%; the 74 compliance assessment questions; single-answer multiple choice on the 1,000-question test set balanced across answer positions, built from workplace scene graphs and NIOSH FACE and CSB accident-report figures; temperature 0, unparseable answers counted wrong)): leader GPT-5.5 (94.59), 8 models
Accuracy on the compliance-assessment questions of SafeSceneReason (2026), an industrial-safety multiple-choice benchmark. Percent, higher is better.
SafeSceneReason - Mitigation and Decision Making (Accuracy (%; the 27 mitigation/decision making questions; single-answer multiple choice on the 1,000-question test set balanced across answer positions, built from workplace scene graphs and NIOSH FACE and CSB accident-report figures; temperature 0, unparseable answers counted wrong)): leader GPT-5.5 (96.3), 8 models
Accuracy on the mitigation and decision-making questions of SafeSceneReason (2026), an industrial-safety multiple-choice benchmark. Percent, higher is better.
PLSQLBench (Mean test pass@1 (%; mean of the three held-out test sets: MBPP+-PLSQL (308 tasks), Spider2 single-turn test (103) and Spider2 multi-turn test (63 conversations); direct generation of Oracle PL/SQL program units at temperature 0, executed on Oracle Autonomous Database 23ai; per-task fraction of unit tests passed by one solution, averaged over tasks)): leader GPT-5.4 (64.96), 8 models
PLSQLBench (Oracle, 2026) evaluates whether LLMs can write executable Oracle PL/SQL programs, scored by execution-based unit tests at temperature 0. Mean test pass at one, averaged over the three held-out test sets (MBPP+, Spider2 single-turn and multi-turn), in percent; higher is better.
PLSQLBench - MBPP+ (Test) (Mean test pass@1 (%; the 308 MBPP+ problems translated to PL/SQL with withheld expanded unit tests; direct generation of Oracle PL/SQL program units at temperature 0, executed on Oracle Autonomous Database 23ai; per-task fraction of unit tests passed by one solution, averaged over tasks)): leader Claude Opus 4.8 (47.73), 8 models
Mean test pass at one on 308 MBPP+ problems translated to PL/SQL in PLSQLBench (2026), execution-based, in percent; higher is better.
PLSQLBench - Spider2 Single-Turn (Test) (Mean test pass@1 (%; the 103 held-out, database-disjoint single-turn tasks over enterprise-style Spider 2 schemas; direct generation of Oracle PL/SQL program units at temperature 0, executed on Oracle Autonomous Database 23ai; per-task fraction of unit tests passed by one solution, averaged over tasks)): leader GPT-5.4 (79.47), 8 models
Mean test pass at one on 103 held-out single-turn schema-grounded Spider 2 tasks in PLSQLBench (2026), execution-based, in percent; higher is better.
PLSQLBench - Spider2 Multi-Turn (Test) (Mean test pass@1 (%; the turns of the 63 held-out, database-disjoint multi-turn Spider 2 conversations, each turn conditioned on the model own earlier answers; direct generation of Oracle PL/SQL program units at temperature 0, executed on Oracle Autonomous Database 23ai; per-task fraction of unit tests passed by one solution, averaged over tasks)): leader GPT-5.4 (73.19), 8 models
Mean test pass at one across the turns of 63 held-out multi-turn Spider 2 conversations in PLSQLBench (2026), execution-based, in percent; higher is better.
FormalTCS - Theorem Elicitation (LLM-rubric score (0-100) from a Qwen3.8-Max judge in QoderCLI, a model outside the evaluated families, weighted 0.4 logical validity, 0.3 completeness, 0.2 correctness and 0.1 clarity; one deterministic generation; natural-language theorem written from the core claim; 143 instances from theoretical computer science papers accepted at STOC, FOCS, SODA, COLT and similar venues in 2025-2026, each stage given the human-annotated input; each model runs in its own vendor harness): leader GPT-5.6 Sol (Codex) (67.4), 8 models
FormalTCS (2026) benchmarks LLMs on frontier theoretical computer science research over 143 instances from 2025-2026 conference papers, each annotated in natural language and in Lean 4. LLM-rubric score for writing a natural-language theorem from the core claim, in percent; higher is better.
FormalTCS - Autoformalization (BEq+ equivalence rate (%; natural-language theorem translated to a Lean 4 statement, counted when bidirectional proof search shows it equivalent to the reference statement; 8 sampled candidates per instance at temperature 0.6; 143 instances from theoretical computer science papers accepted at STOC, FOCS, SODA, COLT and similar venues in 2025-2026, each stage given the human-annotated input; each model runs in its own vendor harness)): leader Claude Opus 5 (Claude Code) (11.2), 8 models
FormalTCS (2026) benchmarks LLMs on frontier theoretical computer science research over 143 instances from 2025-2026 conference papers, each annotated in natural language and in Lean 4. BEq+ equivalence rate for translating a natural-language theorem into a Lean 4 statement, verified by bidirectional proof search, in percent; higher is better.
FormalTCS - Proof Elicitation (LLM-rubric score (0-100) from a Qwen3.8-Max judge in QoderCLI, a model outside the evaluated families, weighted 0.4 logical validity, 0.3 completeness, 0.2 correctness and 0.1 clarity; one deterministic generation; natural-language proof written from the natural and formal theorem; 143 instances from theoretical computer science papers accepted at STOC, FOCS, SODA, COLT and similar venues in 2025-2026, each stage given the human-annotated input; each model runs in its own vendor harness): leader Claude Opus 5 (Claude Code) (68.7), 8 models
FormalTCS (2026) benchmarks LLMs on frontier theoretical computer science research over 143 instances from 2025-2026 conference papers, each annotated in natural language and in Lean 4. LLM-rubric score for writing a natural-language proof from the theorem, in percent; higher is better.
FormalTCS - Theorem Proving (Pass@8 (%; at least one of 8 sampled Lean 4 proofs of the human-annotated formal theorem accepted by the Lean compiler with warnings as errors and an axiom whitelist; 143 instances from theoretical computer science papers accepted at STOC, FOCS, SODA, COLT and similar venues in 2025-2026, each stage given the human-annotated input; each model runs in its own vendor harness)): leader Claude Opus 5 (Claude Code) (28.7), 8 models
FormalTCS (2026) benchmarks LLMs on frontier theoretical computer science research over 143 instances from 2025-2026 conference papers, each annotated in natural language and in Lean 4. Pass at eight for producing a Lean 4 proof of the annotated formal theorem accepted by the compiler, in percent; higher is better.
AesCanvas - Contextual Suitability (Exact-match accuracy (%; one option label per ContextCanvas case on whether an image is an appropriate visual choice for a stated use scenario, audience and context; all 301 expert-reviewed cases (291 two-option, 10 three-option); images resized to 448 x 448, deterministic decoding, no retrieval or tools, unparsable answers counted wrong)): leader Claude Opus 5 (91.69), 17 models
AesCanvas (Huazhong University of Science and Technology and Nanyang Technological University, 2026) tests contextual aesthetic suitability: judging whether an image is an appropriate visual choice for a stated purpose, audience and cultural context, over 301 expert-reviewed ContextCanvas cases. Exact-match accuracy in percent, higher is better.
GenIaC-SecBench - Simple (Median) (Median vulnerability density (vulnerabilities per declared cloud resource in model-generated Infrastructure-as-Code across 100 deployment scenarios, scanned by Checkov, Trivy and KICS; lower is better (0 to unbounded; human-authored baseline median 1.00); simple single-service scenarios)): leader GPT-5 (High) (3.0), 11 models
GenIaC-SecBench (2026) measures the security of LLM-generated Infrastructure-as-Code across 100 deployment scenarios scanned by three policy engines, reporting vulnerability density against a size-matched human baseline (median 1.00). Median vulnerabilities per declared resource on simple single-service scenarios; lower is better, from 0 upward.
GenIaC-SecBench - Complex (Median) (Median vulnerability density (vulnerabilities per declared cloud resource in model-generated Infrastructure-as-Code across 100 deployment scenarios, scanned by Checkov, Trivy and KICS; lower is better (0 to unbounded; human-authored baseline median 1.00); complex multi-resource scenarios)): leader Claude Sonnet 4.6 (0.49), 12 models
GenIaC-SecBench (2026) measures the security of LLM-generated Infrastructure-as-Code across 100 deployment scenarios scanned by three policy engines, reporting vulnerability density against a size-matched human baseline (median 1.00). Median vulnerabilities per declared resource on complex multi-resource scenarios; lower is better, from 0 upward.
TelemetrySuffBench - Origin-Step Localization (Origin-step Top-1 accuracy (%; the model names the exact event that introduced the fault; over TelemetrySuffBench's 216 delayed-binding fault traces; full-telemetry view; closed candidate sets, invalid outputs scored zero)): leader GPT-5.6 Sol (97.2), 5 models
TelemetrySuffBench (East China Normal University, 2026) tests whether a language model can pinpoint the origin of a fault in an agent execution trace given full telemetry, separating fault detection from origin localization over 216 delayed-binding fault traces. Origin-step Top-1 accuracy is the share of fault traces for which the model names the exact event that introduced the fault, in percent, higher is better.
TelemetrySuffBench - Fault-Type Classification (Fault-type macro-F1 (%; two-class taxonomy of wrong-reference-binding vs stale-reference-state; over TelemetrySuffBench's 216 delayed-binding fault traces; full-telemetry view; closed candidate sets, invalid outputs scored zero)): leader GPT-5.5 (79.9), 5 models
TelemetrySuffBench (East China Normal University, 2026) tests whether a language model can pinpoint the origin of a fault in an agent execution trace given full telemetry, separating fault detection from origin localization over 216 delayed-binding fault traces. Fault-type macro-F1 scores the two-class taxonomy of wrong-reference-binding versus stale-reference-state, in percent, higher is better.
TelemetrySuffBench - Component Localization (Origin-component accuracy (%; the model names the component that introduced the fault; over TelemetrySuffBench's 216 delayed-binding fault traces; full-telemetry view; closed candidate sets, invalid outputs scored zero)): leader GPT-5.6 Sol (97.7), 5 models
TelemetrySuffBench (East China Normal University, 2026) tests whether a language model can pinpoint the origin of a fault in an agent execution trace given full telemetry, separating fault detection from origin localization over 216 delayed-binding fault traces. Component accuracy is the share of fault traces for which the model names the component that introduced the fault, in percent, higher is better.
Apodex Discovery - AAV Viability Prediction (Out-of-distribution AUROC (0 to 1) for predicting whether adeno-associated virus capsid variants remain viable, separating viable from non-viable variants with the same number of mutations, averaged over a multi-mutation and a single-mutation instance; solver runs in the authors' domain-specific environment): leader Kimi K3 (0.936), 10 models
Apodex Discovery (2026) puts agentic solvers in a domain-specific environment for adeno-associated virus capsid design with real screening data and hidden verifiers. This task asks the solver to predict whether capsid variants stay viable, scored by out-of-distribution AUROC on a 0 to 1 scale; higher is better.
Apodex Discovery - AAV Tropism Prediction (Correlation (-1 to 1) between predicted and measured tissue enrichment of AAV9 peptide-insertion variants (Fit4Function screens), averaged over multi-organ mouse, cross-species macaque and human in-vitro instances; solver runs in the authors' domain-specific environment): leader Claude Opus 5 (0.653), 10 models
Apodex Discovery (2026) puts agentic solvers in a domain-specific environment for adeno-associated virus capsid design with real screening data and hidden verifiers. This task asks the solver to predict the tissue enrichment of AAV9 peptide-insertion variants across organs, species and human assays, scored by correlation with the measured enrichment (-1 to 1); higher is better.
CoCoBench (Task success rate (%; share of episodes whose goal predicates all hold at the end; 897 oracle-validated executable household multi-agent task instances with two to four agents, centralized planner with image observations, task-specific step budgets)): leader GPT-5.6 Sol (84.8), 11 models
CoCoBench (2026) evaluates multimodal LLMs as centralized planners for two to four cooperating embodied agents on 897 executable household tasks. Task success rate is the share of episodes whose goal predicates all hold at the end, in percent; higher is better.
CoCoBench - Construct Score (Construct score (0-100): construct-specific score of how well the target coordination construct is handled, printed 0-1 and scaled by 100; 897 oracle-validated executable household multi-agent task instances with two to four agents, centralized planner with image observations, task-specific step budgets): leader GPT-5.6 Sol (90.0), 11 models
CoCoBench (2026) evaluates multimodal LLMs as centralized planners for two to four cooperating embodied agents on 897 executable household tasks. The construct score rates how well the target coordination construct of each task is handled, on a 0-100 scale; higher is better.
CoCoBench - Task Allocation (Task success rate (%; task-allocation instances, dividing independent subtasks without duplicated work; 897 oracle-validated executable household multi-agent task instances with two to four agents, centralized planner with image observations, task-specific step budgets)): leader Qwen 3 VL 8B Instruct (95.0), 11 models
CoCoBench (2026) evaluates multimodal LLMs as centralized planners for two to four cooperating embodied agents on 897 executable household tasks. Task success rate in percent, higher is better, on the task-allocation instances, where agents must divide independent subtasks without repeated work.
CoCoBench - Sequential Ordering (Task success rate (%; sequential-ordering instances, respecting ordering constraints between agents; 897 oracle-validated executable household multi-agent task instances with two to four agents, centralized planner with image observations, task-specific step budgets)): leader Claude Opus 4.8 (94.7), 11 models
CoCoBench (2026) evaluates multimodal LLMs as centralized planners for two to four cooperating embodied agents on 897 executable household tasks. Task success rate in percent, higher is better, on the sequential-ordering instances, where a final action must wait for the other agents.
CoCoBench - Mutual Exclusion (Task success rate (%; mutual-exclusion instances, sharing a tool or resource without contention; 897 oracle-validated executable household multi-agent task instances with two to four agents, centralized planner with image observations, task-specific step budgets)): leader GPT-5.6 Sol (96.5), 11 models
CoCoBench (2026) evaluates multimodal LLMs as centralized planners for two to four cooperating embodied agents on 897 executable household tasks. Task success rate in percent, higher is better, on the mutual-exclusion instances, where agents share a single tool without blocking each other.
CoCoBench - Handoff Coordination (Task success rate (%; handoff-coordination instances, relaying objects through intermediate points; 897 oracle-validated executable household multi-agent task instances with two to four agents, centralized planner with image observations, task-specific step budgets)): leader GPT-5.6 Sol (94.2), 11 models
CoCoBench (2026) evaluates multimodal LLMs as centralized planners for two to four cooperating embodied agents on 897 executable household tasks. Task success rate in percent, higher is better, on the handoff-coordination instances, where objects are relayed through intermediate points.
CircuitReason-1k (Accuracy (%; exact problem-level correctness on all 1,000 of CircuitReason-1k's authentic textbook circuit problems; typed deterministic scoring for 585 problems and identity-blinded, reference-assisted multi-model judge consensus (strict majority of at least three votes) for the other 415; every requested output must be correct; missing, failed or unresolved answers count as wrong; image and question only, no output-token cap, open models in their native thinking mode)): leader GPT-5.6 Sol (84.8), 9 models
CircuitReason-1k (2026) pairs 1,000 authentic textbook circuit-analysis problems with their diagrams and reference solutions to test long-horizon visual-to-symbolic reasoning: reading the diagram, recovering topology, formulating equations and carrying units, signs and phases to the final answer. Accuracy in percent with every problem in the denominator, scored by typed deterministic rules where possible and by an identity-blinded multi-model judge majority against the gold answer otherwise; higher is better.
CircuitReason-1k - DC Networks (Accuracy (%; exact problem-level correctness on the 206 DC network problems of CircuitReason-1k's authentic textbook circuit problems; typed deterministic scoring for 585 problems and identity-blinded, reference-assisted multi-model judge consensus (strict majority of at least three votes) for the other 415; every requested output must be correct; missing, failed or unresolved answers count as wrong; image and question only, no output-token cap, open models in their native thinking mode)): leader GPT-5.6 Sol (86.89), 9 models
The 206 DC network problems of CircuitReason-1k (2026), a benchmark of authentic textbook circuit-analysis problems with diagrams. Accuracy in percent, every requested output must be correct; higher is better.
CircuitReason-1k - AC and Phasor (Accuracy (%; exact problem-level correctness on the 308 AC and phasor problems of CircuitReason-1k's authentic textbook circuit problems; typed deterministic scoring for 585 problems and identity-blinded, reference-assisted multi-model judge consensus (strict majority of at least three votes) for the other 415; every requested output must be correct; missing, failed or unresolved answers count as wrong; image and question only, no output-token cap, open models in their native thinking mode)): leader Kimi K2.6 (84.09), 9 models
The 308 AC and phasor problems of CircuitReason-1k (2026), a benchmark of authentic textbook circuit-analysis problems with diagrams. Accuracy in percent, every requested output must be correct; higher is better.
CircuitReason-1k - Transient and Frequency (Accuracy (%; exact problem-level correctness on the 173 transient and frequency-response problems of CircuitReason-1k's authentic textbook circuit problems; typed deterministic scoring for 585 problems and identity-blinded, reference-assisted multi-model judge consensus (strict majority of at least three votes) for the other 415; every requested output must be correct; missing, failed or unresolved answers count as wrong; image and question only, no output-token cap, open models in their native thinking mode)): leader Kimi K2.6 (87.86), 9 models
The 173 transient and frequency-response problems of CircuitReason-1k (2026), a benchmark of authentic textbook circuit-analysis problems with diagrams. Accuracy in percent, every requested output must be correct; higher is better.
CircuitReason-1k - Symbolic and Signal (Accuracy (%; exact problem-level correctness on the 55 symbolic and signal problems of CircuitReason-1k's authentic textbook circuit problems; typed deterministic scoring for 585 problems and identity-blinded, reference-assisted multi-model judge consensus (strict majority of at least three votes) for the other 415; every requested output must be correct; missing, failed or unresolved answers count as wrong; image and question only, no output-token cap, open models in their native thinking mode)): leader GPT-5.6 Sol (90.91), 9 models
The 55 symbolic and signal problems of CircuitReason-1k (2026), a benchmark of authentic textbook circuit-analysis problems with diagrams. Accuracy in percent, every requested output must be correct; higher is better.
CircuitReason-1k - Mixed and General (Accuracy (%; exact problem-level correctness on the 258 mixed and general problems of CircuitReason-1k's authentic textbook circuit problems; typed deterministic scoring for 585 problems and identity-blinded, reference-assisted multi-model judge consensus (strict majority of at least three votes) for the other 415; every requested output must be correct; missing, failed or unresolved answers count as wrong; image and question only, no output-token cap, open models in their native thinking mode)): leader Qwen 3.5 122B A10B (84.5), 9 models
The 258 mixed and general problems of CircuitReason-1k (2026), a benchmark of authentic textbook circuit-analysis problems with diagrams. Accuracy in percent, every requested output must be correct; higher is better.
CircuitReason-1k - Direct Reasoning (Accuracy (%; exact problem-level correctness on the 239 direct (single-step) problems of CircuitReason-1k's authentic textbook circuit problems; typed deterministic scoring for 585 problems and identity-blinded, reference-assisted multi-model judge consensus (strict majority of at least three votes) for the other 415; every requested output must be correct; missing, failed or unresolved answers count as wrong; image and question only, no output-token cap, open models in their native thinking mode)): leader GPT-5.6 Sol (85.36), 9 models
The 239 direct, single-step problems of CircuitReason-1k (2026), grouped by the length of the dependent chain from diagram evidence to the final physical answer. Accuracy in percent, every requested output must be correct; higher is better.
CircuitReason-1k - Multi-Step Reasoning (Accuracy (%; exact problem-level correctness on the 429 multi-step problems of CircuitReason-1k's authentic textbook circuit problems; typed deterministic scoring for 585 problems and identity-blinded, reference-assisted multi-model judge consensus (strict majority of at least three votes) for the other 415; every requested output must be correct; missing, failed or unresolved answers count as wrong; image and question only, no output-token cap, open models in their native thinking mode)): leader Qwen 3.5 122B A10B (89.51), 9 models
The 429 multi-step problems of CircuitReason-1k (2026), grouped by the length of the dependent chain from diagram evidence to the final physical answer. Accuracy in percent, every requested output must be correct; higher is better.
CircuitReason-1k - Long-Horizon Reasoning (Accuracy (%; exact problem-level correctness on the 332 long-horizon problems of CircuitReason-1k's authentic textbook circuit problems; typed deterministic scoring for 585 problems and identity-blinded, reference-assisted multi-model judge consensus (strict majority of at least three votes) for the other 415; every requested output must be correct; missing, failed or unresolved answers count as wrong; image and question only, no output-token cap, open models in their native thinking mode)): leader Kimi K2.6 (83.73), 9 models
The 332 long-horizon problems of CircuitReason-1k (2026), grouped by the length of the dependent chain from diagram evidence to the final physical answer. Accuracy in percent, every requested output must be correct; higher is better.
THPT-Ladder (Share of available marks (%; marks earned under the official Decision 764 marking scheme as a percentage of the 220.00 marks the 22 exam variants are worth: 0.25 per correct four-option item, the convex 0/0.10/0.25/0.50/1.00 ladder per four-statement true/false question, the subject value per correct short answer; 632 items from 21 official 2025-2026 Vietnamese National High School Graduation Examination papers in 11 subjects, presented in Vietnamese one item at a time with text and figures, scored against the ministry keys; open-weight models at their publishers' recommended sampling (16,384-token cap), closed models at their default settings)): leader Claude Opus 5 (97.0), 8 models
THPT-Ladder (2026) scores models on 632 items from 21 official papers of the 2025-2026 Vietnamese National High School Graduation Examination in 11 subjects, graded with the ministry keys and marking scheme. Share of the 220.00 available marks earned, in percent, including the convex partial-credit ladder of the true/false part; higher is better.
THPT-Ladder - Part I (Multiple Choice) (Accuracy (%; x100 of the fraction of Part I four-option items answered correctly; 632 items from 21 official 2025-2026 Vietnamese National High School Graduation Examination papers in 11 subjects, presented in Vietnamese one item at a time with text and figures, scored against the ministry keys; open-weight models at their publishers' recommended sampling (16,384-token cap), closed models at their default settings)): leader Claude Opus 5 (98.8), 8 models
THPT-Ladder (2026) scores models on 632 items from 21 official papers of the 2025-2026 Vietnamese National High School Graduation Examination in 11 subjects, graded with the ministry keys and marking scheme. This board is Part I, the four-option multiple-choice items: accuracy in percent; higher is better.
THPT-Ladder - Part II (Official Ladder) (Points per question (%; x100 of the mean points the official convex ladder pays per Part II question of four true/false statements, out of 1.00: 0 to 4 correct statements earn 0, 0.10, 0.25, 0.50 or 1.00; 632 items from 21 official 2025-2026 Vietnamese National High School Graduation Examination papers in 11 subjects, presented in Vietnamese one item at a time with text and figures, scored against the ministry keys; open-weight models at their publishers' recommended sampling (16,384-token cap), closed models at their default settings)): leader Claude Opus 5 (94.9), 8 models
THPT-Ladder (2026) scores models on 632 items from 21 official papers of the 2025-2026 Vietnamese National High School Graduation Examination in 11 subjects, graded with the ministry keys and marking scheme. This board is Part II, four true/false statements per question paid on the official convex ladder (0, 0.10, 0.25, 0.50 or 1.00 for 0 to 4 correct): mean points per question in percent of the maximum; higher is better.
THPT-Ladder - Part III (Short Answer) (Accuracy (%; x100 of the fraction of Part III short-answer items answered correctly; 632 items from 21 official 2025-2026 Vietnamese National High School Graduation Examination papers in 11 subjects, presented in Vietnamese one item at a time with text and figures, scored against the ministry keys; open-weight models at their publishers' recommended sampling (16,384-token cap), closed models at their default settings)): leader Claude Opus 5 (95.0), 8 models
THPT-Ladder (2026) scores models on 632 items from 21 official papers of the 2025-2026 Vietnamese National High School Graduation Examination in 11 subjects, graded with the ministry keys and marking scheme. This board is Part III, the short-answer items: accuracy in percent; higher is better.
Reconstruction (Research Idea Recovery) (Match rate (%; unweighted mean of the six domain scores over 643 seed papers; share of a model's five hypotheses, proposed in one call from a published paper's pre-publication bibliography only (titles and abstracts under anonymous IDs, the paper itself and later literature withheld), that an LLM judge matches to the held-out paper's idea; mean over the six other evaluated models as leave-one-out judges)): leader Claude Opus 4.8 (13.3), 7 models
Reconstruction (AI-Professor Project, 2026) is a blind idea-recovery benchmark: from only the pre-publication bibliography of a published paper, a model proposes five hypotheses, and an LLM judge decides which match the withheld paper idea. Match rate in percent, the unweighted mean of six domain scores over 643 papers, each scored by the six other evaluated models as leave-one-out judges; higher is better.
Reconstruction (Research Idea Recovery) - Machine Learning (Match rate (%; the 120 machine learning seed papers; share of a model's five hypotheses, proposed in one call from a published paper's pre-publication bibliography only (titles and abstracts under anonymous IDs, the paper itself and later literature withheld), that an LLM judge matches to the held-out paper's idea; mean over the six other evaluated models as leave-one-out judges)): leader Claude Opus 4.8 (8.2), 7 models
The 120 machine learning papers of Reconstruction (AI-Professor Project, 2026), where a model sees only the pre-publication bibliography of a paper and proposes five hypotheses judged against its withheld idea. Match rate in percent, averaged over leave-one-out peer judges; higher is better.
Reconstruction (Research Idea Recovery) - Astronomy (Match rate (%; the 85 astronomy seed papers; share of a model's five hypotheses, proposed in one call from a published paper's pre-publication bibliography only (titles and abstracts under anonymous IDs, the paper itself and later literature withheld), that an LLM judge matches to the held-out paper's idea; mean over the six other evaluated models as leave-one-out judges)): leader Claude Opus 4.8 (14.7), 7 models
The 85 astronomy papers of Reconstruction (AI-Professor Project, 2026), where a model sees only the pre-publication bibliography of a paper and proposes five hypotheses judged against its withheld idea. Match rate in percent, averaged over leave-one-out peer judges; higher is better.
Reconstruction (Research Idea Recovery) - Chemistry (Match rate (%; the 105 chemistry seed papers; share of a model's five hypotheses, proposed in one call from a published paper's pre-publication bibliography only (titles and abstracts under anonymous IDs, the paper itself and later literature withheld), that an LLM judge matches to the held-out paper's idea; mean over the six other evaluated models as leave-one-out judges)): leader GPT-5.6 Pro Sol (15.0), 7 models
The 105 chemistry papers of Reconstruction (AI-Professor Project, 2026), where a model sees only the pre-publication bibliography of a paper and proposes five hypotheses judged against its withheld idea. Match rate in percent, averaged over leave-one-out peer judges; higher is better.
Reconstruction (Research Idea Recovery) - Materials (Match rate (%; the 117 materials seed papers; share of a model's five hypotheses, proposed in one call from a published paper's pre-publication bibliography only (titles and abstracts under anonymous IDs, the paper itself and later literature withheld), that an LLM judge matches to the held-out paper's idea; mean over the six other evaluated models as leave-one-out judges)): leader Claude Opus 4.8 (14.0), 7 models
The 117 materials papers of Reconstruction (AI-Professor Project, 2026), where a model sees only the pre-publication bibliography of a paper and proposes five hypotheses judged against its withheld idea. Match rate in percent, averaged over leave-one-out peer judges; higher is better.
Reconstruction (Research Idea Recovery) - Medicine (Match rate (%; the 78 medicine seed papers; share of a model's five hypotheses, proposed in one call from a published paper's pre-publication bibliography only (titles and abstracts under anonymous IDs, the paper itself and later literature withheld), that an LLM judge matches to the held-out paper's idea; mean over the six other evaluated models as leave-one-out judges)): leader GPT-5.6 Pro Sol (14.6), 7 models
The 78 medicine papers of Reconstruction (AI-Professor Project, 2026), where a model sees only the pre-publication bibliography of a paper and proposes five hypotheses judged against its withheld idea. Match rate in percent, averaged over leave-one-out peer judges; higher is better.
Reconstruction (Research Idea Recovery) - Physics (Match rate (%; the 138 physics seed papers; share of a model's five hypotheses, proposed in one call from a published paper's pre-publication bibliography only (titles and abstracts under anonymous IDs, the paper itself and later literature withheld), that an LLM judge matches to the held-out paper's idea; mean over the six other evaluated models as leave-one-out judges)): leader Claude Opus 4.8 (15.0), 7 models
The 138 physics papers of Reconstruction (AI-Professor Project, 2026), where a model sees only the pre-publication bibliography of a paper and proposes five hypotheses judged against its withheld idea. Match rate in percent, averaged over leave-one-out peer judges; higher is better.
OdinEval (Resolved rate (%; share of 168 filtered Odin issues whose single generated unified patch applies, builds and passes the issue-specific regression test and the declared non-target checks; patch-only protocol with a fixed supplied source scope, no feedback, one attempt)): leader Kimi K3 (66.7), 6 models
OdinEval (Xie et al., 2026) collects documented defects from public Odin repositories, each bound to base and fix commits, a historical toolchain and an issue-specific regression test. The model sees the issue and a fixed set of source files and returns one unified patch without feedback. Resolved rate in percent over 168 instances; higher is better.
MedSafe-Dx (Triage Success Rate (%)): leader GPT-5.6 Terra (73.2), 23 models
Medical diagnostic safety benchmark testing whether models identify likely conditions, avoid harmful misses, and calibrate escalation in clinical scenarios.
MedSafe-Dx - Safety Pass Rate (Safety Pass Rate (%)): leader GPT-5.2 (97.6), 23 models
MedSafe-Dx Safety Pass Rate (Cortico Health, 2026): the percentage of the 250 synthetic DDXPlus cases in the eval-250-v0 set where the model commits no hard safety failure, meaning it does not miss a needed escalation, give a confident differential that misses the diagnosis, or confidently send an ambiguous case to routine care. Unnecessary escalation is not penalized here; higher is better.

Top-10 New Scores (69)

Claude Fable 5.1 on AIMultiple - Agentic RAG Routing: 83.7 (#6)
Claude Fable 5.1 on AIMultiple - Text-to-SQL (Adjudicated): 82.8 (#2)
Claude Fable 5.1 on AIMultiple - Text-to-SQL (Strict Execution Match): 54.7 (#5)
Claude Opus 5 on DeepSearchQA: 90.4 (#7)
Claude Opus 5.5 on Agent Security League - Functional Correctness: 68.7 (#16)
Claude Opus 5.5 on Agent Security League - Security Correctness: 33.5 (#3)
Claude Opus 5.5 on Chess Bench LLM: 1197.0 (#30)
Claude Opus 5.5 on ClockBench: 66.7 (#2)
Claude Opus 5.5 on DuelLab Overall: 81.0 (#4)
Claude Opus 5.5 on ErdosBench: 3.24 (#2)
Claude Opus 5.5 on InferenceBench: 12.08 (#1)
Claude Opus 5.5 on Kaggle FACTS (Google): 65.3 (#8)
Claude Opus 5.5 on Kaggle FACTS Multimodal: 48.37 (#3)
Claude Opus 5.5 on MineBench: 2030.0 (#5)
Claude Opus 5.5 on Multi-turn Debate (Lechmazur): 1701.0 (#4)
Claude Opus 5.5 on SEAL - Fortress: 13.18 (#61)
Claude Opus 5.5 on SEAL - Professional Reasoning Benchmark - Legal: 47.58 (#16)
Claude Opus 5.5 on WebDev Arena (Content Creation Tools): 1867.0 (#1)
GPT-6 on ChessBench - Coherence: 85.3 (#3)
GPT-6 on DuelLab Overall: 94.3 (#1)
GPT-6 on MLS-Bench Lite: 50.0 (#3)
GPT-6 on ParseBench: 70.67 (#23)
GPT-6 on ParseBench - Charts: 35.08 (#54)
GPT-6 on ParseBench - Content Faithfulness: 89.6 (#17)
GPT-6 on ParseBench - Semantic Formatting: 75.52 (#8)
GPT-6 on ParseBench - Tables: 93.17 (#3)
GPT-6 on ParseBench - Visual Grounding: 59.99 (#39)
GPT-6 Sol on AIMultiple - Agentic RAG Routing: 81.5 (#12)
GPT-6 Sol on AIMultiple - Text-to-SQL (Adjudicated): 77.7 (#11)
GPT-6 Sol on AIMultiple - Text-to-SQL (Strict Execution Match): 43.1 (#18)
GPT-6 Sol on Agent Arena: 7.68 (#6)
GPT-6 Sol on Agent Arena - Bash Recovery: 8.35 (#6)
GPT-6 Sol on Agent Arena - Confirmed Success: 11.36 (#4)
GPT-6 Sol on Agent Arena - Praise vs Complaint: 2.62 (#22)
GPT-6 Sol on Agent Arena - Steerability: 15.7 (#1)
GPT-6 Sol on Agent Arena - Tool Hallucination: -0.36 (#3)
GPT-6 Sol on ChessBench: 62.9 (#4)
GPT-6 Sol on ChessBench - Accuracy: 74.2 (#7)
GPT-6 Sol on ChessBench - Coherence: 81.6 (#7)
GPT-6 Sol on ChessBench - Elo: 1205.0 (#4)
GPT-6 Sol on DuelLab Overall: 61.6 (#14)
GPT-6 Sol on InferenceBench: 7.64 (#8)
GPT-6 Sol on Kaggle FACTS (Google): 67.95 (#4)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Arabic: 93.5 (#12)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Bengali: 94.25 (#8)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Burmese: 92.5 (#12)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Chinese: 94.0 (#12)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Culturally Agnostic: 94.41 (#10)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Culturally Sensitive: 93.78 (#10)
GPT-6 Sol on Kaggle Global-MMLU-Lite - English: 95.75 (#9)
GPT-6 Sol on Kaggle Global-MMLU-Lite - French: 95.0 (#7)
GPT-6 Sol on Kaggle Global-MMLU-Lite - German: 94.0 (#13)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Hindi: 95.25 (#4)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Indonesian: 93.25 (#12)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Italian: 96.0 (#4)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Japanese: 95.75 (#3)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Korean: 94.5 (#10)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Portuguese: 94.75 (#9)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Spanish: 93.75 (#15)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Swahili: 92.5 (#12)
GPT-6 Sol on Kaggle Global-MMLU-Lite - Yoruba: 90.75 (#12)
GPT-6 Sol on ParseBench: 69.69 (#28)
GPT-6 Sol on ParseBench - Charts: 48.05 (#36)
GPT-6 Sol on ParseBench - Content Faithfulness: 87.81 (#38)
GPT-6 Sol on ParseBench - Semantic Formatting: 67.91 (#27)
GPT-6 Sol on ParseBench - Tables: 92.63 (#5)
GPT-6 Sol on ParseBench - Visual Grounding: 52.06 (#50)
GPT-6 Sol on SEAL - Fortress: 13.46 (#60)
GPT-6 Sol on SEAL - Professional Reasoning Benchmark - Legal: 38.6 (#30)

New #1 Leaders (17)

WebDev Arena (Content Creation Tools): Claude Opus 5.5 (Max) (1867.0) beat GPT-6 (Max) by 98.0
Design Arena (Game Dev): Claude Opus 5.5 (1473.0) beat GPT-6 by 33.0
ChessBench - Elo: GPT-6 (1448.0) beat Gemini 3.8 Flash by 27.0
AIMultiple - Text-to-SQL (Strict Execution Match): Claude Opus 5.5 (74.9) beat Gemini 3 Flash (Preview) by 19.8
Design Arena (Website): Claude Opus 5.5 (1371.0) beat Muse Spark 1.3 by 9.0
AIMultiple - Text-to-SQL (Adjudicated): Claude Opus 5.5 (91.1) beat Claude Opus 5 by 8.7
AIMultiple - Agentic RAG Routing: Claude Opus 5.5 (90.2) beat Claude Opus 5 by 5.4
Agent Arena - Steerability: GPT-6 Sol (Max) (15.7) beat Claude Opus 4.8 (High) by 4.64
Kaggle FACTS (Google): GPT-6 (75.68) beat Gemini 3.7 Flash by 4.43
ChessBench: GPT-6 (78.4) beat Gemini 3.8 Flash by 3.0
ChessBench - Accuracy: GPT-6 (81.2) beat Gemini 3.8 Flash by 2.5
InferenceBench: Claude Opus 5.5 (Max) (12.08) beat Claude Fable 5.1 by 2.25
AI Coding Daily (OpenCode) - Total: Space Bunny (Max) (52.61) beat Kimi K3 (High) by 1.76
ParseBench - Charts: Claude Opus 5.5 (Effort High) (70.89) beat Qwen 3.8 27B (Thinking) by 1.72
ParseBench: Claude Opus 5.5 (Effort High) (79.85) beat Claude Fable 5.1 by 0.93
ParseBench - Tables: Claude Opus 5.5 (Effort High) (94.25) beat Claude Opus 5.5 by 0.39
Blueprint-Bench 2: Claude Opus 5.5 (0.512) beat GPT-6 by 0.02

The Aggregate Digest — 2026-09-25

2026-09-25T05:25:46.078127+00:00

Frontier

Best available model: Claude Opus 5.5 (2082) takes the crown from GPT-6 (2072), which had trailed for 3 days

Daily

New Benchmarks (15)

LongDS v1.1 Lite (Accuracy (%; judged turn accuracy macro-averaged over 24 tasks), each model under one agentic coding harness): leader GPT-6 (78.17), 8 models
LongDS (Xu et al., Zhejiang University and Ant Group, EMNLP 2026) tests long-horizon agentic data analysis on multi-turn tasks built from Kaggle notebooks and datasets. This board is the authors' audited v1.1 release on the 24-task Lite selection, where each model runs under one vendor's agentic coding harness (Claude Code, Codex, Kimi Code or Qoder). Later turns update, perturb, roll back or combine analytical states set up earlier; a judge scores each turn against executable reference code, and the score is turn accuracy macro-averaged over tasks, in percent.
LongDS v1.1 Full (Accuracy (%; judged turn accuracy macro-averaged over 68 tasks), each model under one agentic coding harness): leader GPT-5.6 Sol (64.42), 6 models
LongDS (Xu et al., Zhejiang University and Ant Group, EMNLP 2026) tests long-horizon agentic data analysis on multi-turn tasks built from Kaggle notebooks and datasets. This board is the authors' audited v1.1 release on all 68 tasks (2,225 turns), where each model runs under one vendor's agentic coding harness (Claude Code, Codex, Kimi Code or Qoder). Later turns update, perturb, roll back or combine analytical states set up earlier; a judge scores each turn against executable reference code, and the score is turn accuracy macro-averaged over tasks, in percent.
LongDS v1.1 Full - Education (Accuracy (%; judged turn accuracy macro-averaged over 8 tasks), each model under one agentic coding harness): leader GPT-5.6 Sol (93.87), 6 models
The Education domain of LongDS v1.1 Full: 8 long-horizon data-analysis tasks built from Kaggle notebooks and datasets, where each model runs under one vendor's agentic coding harness (Claude Code, Codex, Kimi Code or Qoder). Later turns update, perturb, roll back or combine analytical states set up earlier; a judge scores each turn against executable reference code, and the score is turn accuracy macro-averaged over tasks, in percent.
LongDS v1.1 Full - Community (Accuracy (%; judged turn accuracy macro-averaged over 16 tasks), each model under one agentic coding harness): leader GPT-5.6 Sol (77.06), 6 models
The Community domain of LongDS v1.1 Full: 16 long-horizon data-analysis tasks built from Kaggle notebooks and datasets, where each model runs under one vendor's agentic coding harness (Claude Code, Codex, Kimi Code or Qoder). Later turns update, perturb, roll back or combine analytical states set up earlier; a judge scores each turn against executable reference code, and the score is turn accuracy macro-averaged over tasks, in percent.
LongDS v1.1 Full - Social Good (Accuracy (%; judged turn accuracy macro-averaged over 10 tasks), each model under one agentic coding harness): leader Kimi K3 (56.12), 6 models
The Social Good domain of LongDS v1.1 Full: 10 long-horizon data-analysis tasks built from Kaggle notebooks and datasets, where each model runs under one vendor's agentic coding harness (Claude Code, Codex, Kimi Code or Qoder). Later turns update, perturb, roll back or combine analytical states set up earlier; a judge scores each turn against executable reference code, and the score is turn accuracy macro-averaged over tasks, in percent.
LongDS v1.1 Full - Business (Accuracy (%; judged turn accuracy macro-averaged over 12 tasks), each model under one agentic coding harness): leader GPT-5.6 Sol (56.99), 6 models
The Business domain of LongDS v1.1 Full: 12 long-horizon data-analysis tasks built from Kaggle notebooks and datasets, where each model runs under one vendor's agentic coding harness (Claude Code, Codex, Kimi Code or Qoder). Later turns update, perturb, roll back or combine analytical states set up earlier; a judge scores each turn against executable reference code, and the score is turn accuracy macro-averaged over tasks, in percent.
LongDS v1.1 Full - Geoscience (Accuracy (%; judged turn accuracy macro-averaged over 19 tasks), each model under one agentic coding harness): leader GPT-5.6 Sol (56.18), 6 models
The Geoscience domain of LongDS v1.1 Full: 19 long-horizon data-analysis tasks built from Kaggle notebooks and datasets, where each model runs under one vendor's agentic coding harness (Claude Code, Codex, Kimi Code or Qoder). Later turns update, perturb, roll back or combine analytical states set up earlier; a judge scores each turn against executable reference code, and the score is turn accuracy macro-averaged over tasks, in percent.
V-ICAL Bench (Normalized score (0-100; environment-weighted)): leader Seed 2.1 Pro (54.4), 19 models
V-ICAL Bench (Fan et al., 2026) tests video in-context learning for multimodal agents: in 342 tasks across 37 game and control environments from the Gym ecosystem (MiniGrid, Atari, Procgen, classic control, HighwayEnv and others), the agent watches one to four human demonstration videos, then plays from a fixed start state one action per turn from raw frames, with rewards hidden. The score is a normalized 0-100 progress score per task against expert-calibrated targets, averaged within each environment and then equally across environments, higher is better. Each model runs once per task through its provider's multimodal API, so provider context limits count against it.
V-ICAL Bench - Pass Rate (Pass rate (%)): leader Seed 2.1 Pro (58.0), 19 models
V-ICAL Bench's second metric: the percentage of its 342 video-demonstration game and control tasks in which the agent meets the task's predefined success condition (reaching a goal, triggering a required event, hitting a target score or surviving the horizon), higher is better. The condition is judged independently of the normalized progress score; agents learn each task from one to four human demonstration videos and act from raw frames with rewards hidden (Fan et al., 2026).
Vending-Bench Arena - Round 13 (Final money balance ($, mean over runs)): leader GPT-6 Sol (10529.63), 3 models
Vending-Bench Arena round 13 (Andon Labs, September 2026, after the Claude Opus 5.5 and GPT-6 Sol releases; the page gives no date for this round): GPT-6 Sol, Claude Opus 5.5 and Grok 4.7 each run a vending machine at one location in the Vending-Bench 2 simulation and can email, pay and trade with each other. Final money balance in US dollars (start $500), averaged over 4 runs; higher is better. Balances depend on the opponents in that round, so they do not compare across rounds.
ProgramDistill (Binary score (%)): leader GPT-6 (84.33), 9 models
Microsoft Research's benchmark of reference-guided web development: a coding agent gets a web application with one behavior masked out and a running copy of the original, which it can use but whose source it cannot read, and must restore the behavior in code. Each fix is checked by replaying the recorded browser traces. The board is the public 300-task subset across 26 applications, at restoration depths 1 to 8, all run in one modified R2E-Gym harness. Binary score: the share of tasks whose every replayed behavior passes.
PFArena - Measurement-Free Multi-Mutant Ranking (Spearman): leader Claude Opus 5 (0.2841), 6 models
Task T2 of AMix-Bio's PFArena protein-mutation benchmark (September 2026): given a protein and a pool of multi-point mutants with no measured data for that protein, rank the candidates by predicted fitness, stability or binding. Ground truth is deep mutational scanning data from ProteinGym, MaveDB, MegaScale, FLAb and other assays. Spearman correlation between the model's ranking and the measured scores (Random -0.02); higher is better. Prompted LLMs only.
PFArena - Anchor-Informed Multi-Mutant Ranking (Spearman): leader Claude Opus 5 (0.354), 6 models
Task T3 of AMix-Bio's PFArena protein-mutation benchmark (September 2026): given one experimentally measured anchor mutant and its score, rank candidate successors that add further mutations to it. Ground truth is deep mutational scanning data. Spearman correlation between the model's ranking and the measured scores (Random 0.04); higher is better. Prompted LLMs only.
PFArena - Single-Mutant-Informed Multi-Mutant Ranking (Spearman): leader GPT-6 (0.5988), 6 models
Task T4 of AMix-Bio's PFArena protein-mutation benchmark (September 2026): given the measured effects of individual point mutations, rank combinations of them, which tests whether a model can reason about epistasis rather than add effects. Ground truth is deep mutational scanning data. Spearman correlation between the model's ranking and the measured combination scores (Random -0.02); higher is better. Prompted LLMs only.
Alibaba CTF (Pass rate (%)): leader GPT-5.6 Sol (62.1), 6 models
Alibaba AAIG's capture-the-flag benchmark: 87 challenges from the 2023 to 2026 AlibabaCTF competitions (25 web, 19 pwn, 16 reverse, 14 misc, 13 crypto), each a Docker environment in the Harbor framework with an oracle solution and an automatic flag check. Agents get up to 6.6 hours and 200 turns per task. Pass rate is the share of the 87 flags recovered, as charted on the dataset card.

Top-10 New Scores (100)

Claude Opus 5.5 on APEX v1: 74.9 (#2)
Claude Opus 5.5 on Chatbot Arena (Code): 1818.0 (#1)
Claude Opus 5.5 on Enkrypt AI - Bias Risk: 25.58 (#5)
Claude Opus 5.5 on Enkrypt AI - Insecure Code Risk: 0.44 (#4)
Claude Opus 5.5 on Enkrypt AI - Jailbreak Risk: 4.46 (#62)
Claude Opus 5.5 on Enkrypt AI - Risk Score: 8.19 (#2)
Claude Opus 5.5 on Enkrypt AI - Safety Risk: 12.03 (#5)
Claude Opus 5.5 on Enkrypt AI - Toxicity Risk: 3.09 (#136)
Claude Opus 5.5 on Epoch AI - Ebr Bench: 71.43 (#2)
Claude Opus 5.5 on Kaggle FACTS Grounding: 68.46 (#20)
Claude Opus 5.5 on Kaggle FACTS Parametric: 57.87 (#11)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Arabic: 95.5 (#3)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Bengali: 94.75 (#4)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Burmese: 92.5 (#10)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Chinese: 95.5 (#5)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Culturally Agnostic: 95.53 (#4)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Culturally Sensitive: 95.19 (#2)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - French: 96.25 (#2)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - German: 96.0 (#2)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Hindi: 94.75 (#6)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Indonesian: 95.25 (#5)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Italian: 96.5 (#2)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Japanese: 95.0 (#4)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Korean: 95.25 (#4)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Portuguese: 96.5 (#2)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Spanish: 96.5 (#4)
Claude Opus 5.5 on Kaggle Global-MMLU-Lite - Yoruba: 93.25 (#5)
Claude Opus 5.5 on LOL Arena: 49.3 (#5)
Claude Opus 5.5 on Legal Benchmarks - Contract Workflows - Task Pass Rate: 48.4 (#1)
Claude Opus 5.5 on Legal Benchmarks - Data Extraction - Form: 82.2 (#2)
Claude Opus 5.5 on MathArena - ARXIV August: 85.96 (#4)
Claude Opus 5.5 on MathArena - ARXIV June: 83.33 (#8)
Claude Opus 5.5 on MathArena - ARXIV_FALSE August: 77.38 (#5)
Claude Opus 5.5 on MathArena - ARXIV_FALSE June: 100.0 (#1)
Claude Opus 5.5 on MathArena Arxiv: 81.67 (#6)
Claude Opus 5.5 on MathArena Arxiv False: 95.0 (#1)
Claude Opus 5.5 on Maze-Bench: 85.17 (#2)
Claude Opus 5.5 on PRISM (1C:Enterprise) - Algorithmic Tasks (A): 93.0 (#7)
Claude Opus 5.5 on Vending-Bench 2: 9235.25 (#7)
Claude Opus 5.5 on WebDev Arena (Brand & Marketing): 1791.0 (#1)
Claude Opus 5.5 on WebDev Arena (Consumer Product): 1690.0 (#2)
Claude Opus 5.5 on WebDev Arena (Data & Analytics): 1727.0 (#1)
Claude Opus 5.5 on WebDev Arena (Frontend): 1848.0 (#1)
Claude Opus 5.5 on WebDev Arena (Gaming): 1908.0 (#1)
Claude Opus 5.5 on WebDev Arena (React): 1851.0 (#1)
Claude Opus 5.5 on WebDev Arena (Reference-Based Design): 1842.0 (#1)
Claude Opus 5.5 on WebDev Arena (Simulations): 1951.0 (#1)
GPT-6 on Android Bench: 28.0 (#33)
GPT-6 Sol on AI Coding Daily (Codex CLI) - Laravel Code Quality: 17.67 (#5)
GPT-6 Sol on AI Coding Daily (Codex CLI) - React-TS Code Quality: 17.5 (#2)
GPT-6 Sol on AI Coding Daily (Codex CLI) - Total: 52.42 (#3)
GPT-6 Sol on Agents' Last Exam: 32.2 (#3)
GPT-6 Sol on Aiera Leaderboard - Depth: 80.6 (#9)
GPT-6 Sol on Aiera Leaderboard - Facts: 61.98 (#21)
GPT-6 Sol on Aiera Leaderboard - Grounding: 82.7 (#21)
GPT-6 Sol on Aiera Leaderboard - Research Score: 75.09 (#17)
GPT-6 Sol on Blueprint-Bench 2: 0.338 (#6)
GPT-6 Sol on Conceptual Reasoning Index: 71.04 (#21)
GPT-6 Sol on Conceptual Reasoning Index - Argument Evaluation (LMCA): 59.08 (#23)
GPT-6 Sol on Conceptual Reasoning Index - Consistency (ACCoRD): 82.4 (#6)
GPT-6 Sol on Conceptual Reasoning Index - Decision Theory (DTBench): 95.55 (#10)
GPT-6 Sol on Design Arena (Agents - Fullstack): 1252.0 (#11)
GPT-6 Sol on Design Arena (UI Components): 1286.0 (#30)
GPT-6 Sol on Design Arena (Website): 1267.0 (#41)
GPT-6 Sol on Enkrypt AI - Bias Risk: 71.58 (#68)
GPT-6 Sol on Enkrypt AI - Insecure Code Risk: 18.22 (#102)
GPT-6 Sol on Enkrypt AI - Jailbreak Risk: 12.23 (#198)
GPT-6 Sol on Enkrypt AI - Risk Score: 26.07 (#80)
GPT-6 Sol on Enkrypt AI - Safety Risk: 37.85 (#245)
GPT-6 Sol on Enkrypt AI - Toxicity Risk: 8.73 (#244)
GPT-6 Sol on KernelBench Hub - CUDA: 64.47 (#5)
GPT-6 Sol on KernelBench Hub - Mega: 0.96 (#14)
GPT-6 Sol on LOL Arena: 47.9 (#19)
GPT-6 Sol on MathArena - ARXIV August: 92.98 (#1)
GPT-6 Sol on MathArena - ARXIV June: 90.97 (#2)
GPT-6 Sol on MathArena - ARXIV_FALSE August: 78.57 (#3)
GPT-6 Sol on MathArena - ARXIV_FALSE June: 95.37 (#4)
GPT-6 Sol on MathArena Arxiv: 90.0 (#2)
GPT-6 Sol on MathArena Arxiv False: 87.0 (#4)
GPT-6 Sol on PRISM (1C:Enterprise) - Algorithmic Tasks (A): 93.0 (#9)
GPT-6 Sol on PRISM (1C:Enterprise) - Platform Tasks (B): 95.0 (#4)
GPT-6 Sol on ReLE - Coding: 72.7 (#9)
GPT-6 Sol on ReLE - Education: 61.9 (#30)
GPT-6 Sol on ReLE - Education - Gaokao: 59.8 (#34)
GPT-6 Sol on ReLE - Education - High School Subjects: 70.3 (#29)
GPT-6 Sol on ReLE - Education - Middle School Subjects: 53.4 (#33)
GPT-6 Sol on ReLE - Education - Primary School Subjects: 64.0 (#50)
GPT-6 Sol on ReLE - Finance: 84.8 (#20)
GPT-6 Sol on ReLE - Language and Instruction Following: 71.3 (#36)
GPT-6 Sol on ReLE - Law - Lawyer Qualification Exam: 74.0 (#63)
GPT-6 Sol on ReLE - Law and Civil Service: 82.0 (#45)
GPT-6 Sol on ReLE - Medicine - Basic Medical Knowledge: 84.1 (#25)
GPT-6 Sol on ReLE - Medicine - Physician Exams: 85.8 (#39)
GPT-6 Sol on ReLE - Medicine and Mental Health: 86.9 (#30)
GPT-6 Sol on ReLE - Overall: 71.7 (#30)
GPT-6 Sol on ReLE - Reasoning - BBH: 44.1 (#179)
GPT-6 Sol on ReLE - Reasoning and Mathematics: 75.1 (#66)
GPT-6 Sol on SecIT Bench (Pydantic AI): 59.26 (#21)
GPT-6 Sol on SimpleBench: 73.1 (#15)
GPT-6 Sol on Vending-Bench 2: 14427.85 (#2)

New #1 Leaders (29)

WebDev Arena (Simulations): Claude Opus 5.5 (Max) (1951.0) beat GPT-6 (Max) by 80.0
WebDev Arena (Frontend): Claude Opus 5.5 (Max) (1848.0) beat GPT-6 (Max) by 38.0
WebDev Arena (Data & Analytics): Claude Opus 5.5 (Max) (1727.0) beat GPT-6 (Max) by 38.0
WebDev Arena (React): Claude Opus 5.5 (Max) (1851.0) beat GPT-6 (Max) by 37.0
WebDev Arena (Gaming): Claude Opus 5.5 (Max) (1908.0) beat Claude Fable 5.1 (Max) by 33.0
WebDev Arena (Brand & Marketing): Claude Opus 5.5 (Max) (1791.0) beat GPT-6 (Max) by 27.0
Chatbot Arena (Code): Claude Opus 5.5 (Max) (1818.0) beat GPT-6 (Max) by 25.0
MathArena - ARXIVLEAN June: AlephProver (86.96) beat GPT-6 (Max) by 21.74
WebDev Arena (Reference-Based Design): Claude Opus 5.5 (Max) (1842.0) beat GPT-6 (Max) by 20.0
Legal Benchmarks - Data Extraction - Task Pass Rate: Claude Opus 5.5 (76.7) beat Claude Fable 5.1 by 13.4
Kaggle Open Benchmarks - SciCode Subproblems: Claude Opus 5.5 (52.23) beat Gemini 3.1 Pro (Preview) by 7.33
PRISM (1C:Enterprise) - Platform Tasks (B): Claude Opus 5.5 (100.0) beat GPT-5.6 Sol by 5.0
MathArena - ARXIV August: GPT-6 Sol (Max) (92.98) beat GPT-6 (Max) by 4.38
MathArena Arxiv False: Claude Opus 5.5 (High) (95.0) beat GPT-6 (Max) by 4.0
Agents' Last Exam: Claude Opus 5.5 (38.2) beat GPT-6 by 4.0
Conceptual Reasoning Index: Claude Opus 5.5 (77.86) beat Claude Fable 5.1 (Medium) by 2.91
Conceptual Reasoning Index - Argument Evaluation (LMCA): Claude Opus 5.5 (68.23) beat Claude Fable 5.1 by 2.77
Legal Benchmarks - Contract Workflows - Form: Claude Opus 5.5 (93.9) beat GPT-6 by 2.0
SimpleBench: Claude Opus 5.5 (88.4) beat Claude Fable 5.1 by 1.8
Vals AI CyberBench: MiMo-V2.6-Flash (75.36) beat DeepSeek V4.1 Flash by 1.67
APEX v1: Muse Spark 1.2 (xHigh) (75.6) beat Muse Spark 1.3 (xHigh) by 1.6
Conceptual Reasoning Index - Consistency (ACCoRD): Claude Opus 5.5 (86.41) beat Claude Fable 5 by 1.46
Legal Benchmarks - Data Extraction - Criteria Pass Rate: Claude Opus 5.5 (91.1) beat Claude Fable 5.1 by 1.2
Legal Benchmarks - Contract Workflows - Criteria Pass Rate: Claude Opus 5.5 (91.7) beat Claude Fable 5.1 by 1.0
MathArena - ARXIV_FALSE June: Claude Opus 5.5 (High) (100.0) beat GPT-6 (Max) by 0.93
Conceptual Reasoning Index - Decision Theory (DTBench): Claude Opus 5.5 (98.22) beat Claude Fable 5 by 0.89
AIMultiple - FinanceReasoning (Hard): Claude Opus 5.5 (91.6) beat GPT-5.6 Pro Sol by 0.84
Kaggle Global-MMLU-Lite - Swahili: Claude Opus 5.5 (95.5) beat Gemini 3.8 Flash by 0.75
Kaggle Global-MMLU-Lite - English: Claude Opus 5.5 (96.75) beat Claude Opus 5 by 0.25

The Aggregate Digest — 2026-09-24

2026-09-24T04:42:52.282911+00:00

Daily

New Benchmarks (64)

Anthropic ECI - September 2026 Fit (Opus 5.5 System Card) (AECI Score): leader Claude Opus 5.5 (Max) (169.36), 5 models
Anthropic's fork of the Epoch Capabilities Index (AECI): an IRT fit over public and internal benchmark results, refit on September 18, 2026 on 374 benchmarks and 732 models, with Claude Sonnet 3.5 pinned at 130 and GPT-5 at 150 (higher is better). Values from this fit are not comparable to earlier AECI releases. From Table 2.3.5.3.A of the Claude Opus 5.5 System Card (September 2026); Opus 5, Mythos 5.1 and Opus 5.5 at max effort, earlier models at their best configuration.
OpenAI GPT-6 Sol & Luna Launch - Unauthorized Agent Interaction (Misaligned action rate (%)): leader GPT-6 (0.0), 4 models
Alignment chart from OpenAI's GPT-6 Sol and GPT-6 Luna launch post (2026-09-22): share of browsing-task samples in which the agent performs an unauthorized action instructed by a simulated message board it can reach, such as disclosing private information or approving illicit access. Percent, lower is better; no effort stated. GPT-5.6 Luna was not measured.
OpenAI GPT-6 Astra System Card - Multimodal Troubleshooting Virology (pass@1 (%)): leader GPT-6 Sol (50.6), 2 models
Table 33 (biological capability, High threshold) of OpenAI's GPT-6 Astra System Card, section 11 (GPT-6 Sol and GPT-6 Luna, added 2026-09-22): multiple-choice virology questions on troubleshooting wet-lab protocols, some with images. pass@1, percent, higher is better; the card's High threshold is 31%. GPT-6 Sol and GPT-6 Luna only; the Astra section prints this evaluation as a figure.
OpenAI GPT-6 Astra System Card - ProtocolQA Open-Ended (pass@1 (%)): leader GPT-6 Sol (44.4), 2 models
Table 33 (biological capability, High threshold) of OpenAI's GPT-6 Astra System Card, section 11 (GPT-6 Sol and GPT-6 Luna, added 2026-09-22): open-ended questions on troubleshooting flawed wet-lab protocols. pass@1, percent, higher is better; the card's High threshold is 54%. GPT-6 Sol and GPT-6 Luna only; the Astra section prints this evaluation as a figure.
OpenAI GPT-6 Astra System Card - Tacit Knowledge and Troubleshooting (cons@32 (%)): leader GPT-6 Sol (79.2), 2 models
Table 33 (biological capability, High threshold) of OpenAI's GPT-6 Astra System Card, section 11 (GPT-6 Sol and GPT-6 Luna, added 2026-09-22): multiple-choice questions on tacit wet-lab knowledge and troubleshooting that experts answer from experience. cons@32, percent, higher is better; the card's High threshold is 80%. GPT-6 Sol and GPT-6 Luna only; the Astra section prints this evaluation as a figure.
OpenAI GPT-6 Astra System Card - TroubleshootingBench (pass@1 (%)): leader GPT-6 Sol (45.3), 2 models
Table 33 (biological capability, High threshold) of OpenAI's GPT-6 Astra System Card, section 11 (GPT-6 Sol and GPT-6 Luna, added 2026-09-22): identifying and fixing real errors in expert-written lab protocols that rely on tacit knowledge. pass@1, percent, higher is better; the card's High threshold is 36.4%. GPT-6 Sol and GPT-6 Luna only; the Astra section prints this evaluation as a figure.
OpenAI GPT-6 Astra System Card - HealthBench Professional (Length-Adjusted Score (0-100)): leader GPT-6 (64.7), 3 models
Table 29 of OpenAI's GPT-6 Astra System Card, section 11 (GPT-6 Sol and GPT-6 Luna, added 2026-09-22): HealthBench professional subset for medically challenging, expert-oriented healthcare questions, on the length-adjusted scale (0-100), higher is better. GPT-6 Astra, GPT-6 Sol and GPT-6 Luna only; the table's GPT-5.5 and GPT-5.6 columns repeat the GPT-5.6 card. Astra's value is the one the card corrected on 2026-09-22 for a misconfigured earlier run.
OpenAI GPT-6 Astra System Card - HealthBench (Length-Adjusted Score (0-100)): leader GPT-6 (58.3), 3 models
Table 29 of OpenAI's GPT-6 Astra System Card, section 11 (GPT-6 Sol and GPT-6 Luna, added 2026-09-22): HealthBench health conversations graded against physician-written rubrics, on the length-adjusted scale (0-100), higher is better. GPT-6 Astra, GPT-6 Sol and GPT-6 Luna only; the table's GPT-5.5 and GPT-5.6 columns repeat the GPT-5.6 card. Astra's value is the one the card corrected on 2026-09-22 for a misconfigured earlier run.
OpenAI GPT-6 Astra System Card - HealthBench Hard (Length-Adjusted Score (0-100)): leader GPT-6 (36.6), 3 models
Table 29 of OpenAI's GPT-6 Astra System Card, section 11 (GPT-6 Sol and GPT-6 Luna, added 2026-09-22): the hard subset of HealthBench, the conversations frontier models still fail, on the length-adjusted scale (0-100), higher is better. GPT-6 Astra, GPT-6 Sol and GPT-6 Luna only; the table's GPT-5.5 and GPT-5.6 columns repeat the GPT-5.6 card. Astra's value is the one the card corrected on 2026-09-22 for a misconfigured earlier run.
OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Chat Plugins (Safe completion rate (%)): leader GPT-5.6 Sol (100.0), 4 models
Table 25 (agentic safe completions) of OpenAI's GPT-6 Astra System Card, section 11 (GPT-6 Sol and GPT-6 Luna, added 2026-09-22): safe completion rate judging the model's actions as well as its final response, on challenging ChatGPT production conversations with Plugin services connected. Percent (the card prints 0-1), higher is better. GPT-5.6 Sol and GPT-6 Astra score 1.000, as in the Astra section's Table 3.
OpenAI MentalHealthBench (Task-clipped rubric score (%)): leader GPT-6 (57.33), 17 models
OpenAI benchmark of 1,215 synthetic mental health conversations, from everyday stress to emergencies, with adult, teen, caregiver and clinician users in 11 languages. GPT-5.6 Sol grades the reply to the last message against rubrics written by licensed psychologists and psychiatrists; score is the mean task-clipped share of positive rubric points.
OpenAI MentalHealthBench - Acuity - Non-acute (Task-clipped rubric score (%)): leader GPT-6 (56.64), 17 models
Acuity split of OpenAI's MentalHealthBench (launch post and paper, 2026-09-23): the 650 non-acute conversations (everyday concerns with an emotional component). Mean task-clipped rubric score, 0-100, higher is better: GPT-5.6 Sol grades each reply against expert-written signed rubrics, and models run at their API default reasoning effort. GPT-5 Thinking and Gemini 2.5 Flash come from the paper's Figure 7.
OpenAI MentalHealthBench - Acuity - High Acuity (Task-clipped rubric score (%)): leader GPT-6 (57.92), 17 models
Acuity split of OpenAI's MentalHealthBench (launch post and paper, 2026-09-23): the 221 high-acuity conversations (serious concerns or significant distress, not an immediate emergency). Mean task-clipped rubric score, 0-100, higher is better: GPT-5.6 Sol grades each reply against expert-written signed rubrics, and models run at their API default reasoning effort. GPT-5 Thinking and Gemini 2.5 Flash come from the paper's Figure 7.
OpenAI MentalHealthBench - Acuity - Emergent (Task-clipped rubric score (%)): leader GPT-6 (58.26), 17 models
Acuity split of OpenAI's MentalHealthBench (launch post and paper, 2026-09-23): the 344 emergent conversations (signs of a mental health emergency or immediate safety risk). Mean task-clipped rubric score, 0-100, higher is better: GPT-5.6 Sol grades each reply against expert-written signed rubrics, and models run at their API default reasoning effort. GPT-5 Thinking and Gemini 2.5 Flash come from the paper's Figure 7.
OpenAI MentalHealthBench - User Profile - Adult (Task-clipped rubric score (%)): leader GPT-6 (56.48), 17 models
User-profile split of OpenAI's MentalHealthBench (launch post and paper, 2026-09-23): the 828 conversations with an adult user. Mean task-clipped rubric score, 0-100, higher is better: GPT-5.6 Sol grades each reply against expert-written signed rubrics, and models run at their API default reasoning effort. GPT-5 Thinking and Gemini 2.5 Flash come from the paper's Figure 10.
OpenAI MentalHealthBench - User Profile - Teen (Task-clipped rubric score (%)): leader Claude Opus 5.5 (56.99), 17 models
User-profile split of OpenAI's MentalHealthBench (launch post and paper, 2026-09-23): the 257 conversations with a teen user, whose age (13-17) is stated in a system message. Mean task-clipped rubric score, 0-100, higher is better: GPT-5.6 Sol grades each reply against expert-written signed rubrics, and models run at their API default reasoning effort. GPT-5 Thinking and Gemini 2.5 Flash come from the paper's Figure 10.
OpenAI MentalHealthBench - User Profile - Clinician (Task-clipped rubric score (%)): leader GPT-6 (65.15), 17 models
User-profile split of OpenAI's MentalHealthBench (launch post and paper, 2026-09-23): the 70 conversations with a clinician user. Mean task-clipped rubric score, 0-100, higher is better: GPT-5.6 Sol grades each reply against expert-written signed rubrics, and models run at their API default reasoning effort. GPT-5 Thinking and Gemini 2.5 Flash come from the paper's Figure 10.
OpenAI MentalHealthBench - User Profile - Caregiver (Task-clipped rubric score (%)): leader GPT-6 (66.3), 17 models
User-profile split of OpenAI's MentalHealthBench (launch post and paper, 2026-09-23): the 60 conversations with a caregiver user. Mean task-clipped rubric score, 0-100, higher is better: GPT-5.6 Sol grades each reply against expert-written signed rubrics, and models run at their API default reasoning effort. GPT-5 Thinking and Gemini 2.5 Flash come from the paper's Figure 10.
OpenAI MentalHealthBench - Behavior - Context Seeking (Axis score, normalized to its possible range (%)): leader Claude Opus 5.5 (56.88), 15 models
Behavior axis of OpenAI's MentalHealthBench (launch post and paper, 2026-09-23): identifying and using the context needed, asking follow-up questions when necessary and not when context is already sufficient, scored on the 1,124 rubric criteria tagged with this axis. The post's axis score is the signed rubric score on this axis, normalized to its possible range (0-100, higher is better); GPT-5.6 Sol grades, models run at API default reasoning effort. The post's 15 models only: the paper's behavior figure uses a different scale.
OpenAI MentalHealthBench - Behavior - Actionable Guidance (Axis score, normalized to its possible range (%)): leader Claude Haiku 4.5 (72.43), 15 models
Behavior axis of OpenAI's MentalHealthBench (launch post and paper, 2026-09-23): practical next steps and coping strategies that fit the moment, scored on the 933 rubric criteria tagged with this axis. The post's axis score is the signed rubric score on this axis, normalized to its possible range (0-100, higher is better); GPT-5.6 Sol grades, models run at API default reasoning effort. The post's 15 models only: the paper's behavior figure uses a different scale.
OpenAI MentalHealthBench - Behavior - Clinical Accuracy (Axis score, normalized to its possible range (%)): leader GPT-6 (82.6), 15 models
Behavior axis of OpenAI's MentalHealthBench (launch post and paper, 2026-09-23): reliable, appropriately qualified mental health information, scored on the 853 rubric criteria tagged with this axis. The post's axis score is the signed rubric score on this axis, normalized to its possible range (0-100, higher is better); GPT-5.6 Sol grades, models run at API default reasoning effort. The post's 15 models only: the paper's behavior figure uses a different scale.
OpenAI MentalHealthBench - Behavior - Interpretation (Axis score, normalized to its possible range (%)): leader GPT-6 Luna (79.62), 15 models
Behavior axis of OpenAI's MentalHealthBench (launch post and paper, 2026-09-23): balanced interpretation and cognitive reframing of ambiguous experiences without over-claiming, scored on the 659 rubric criteria tagged with this axis. The post's axis score is the signed rubric score on this axis, normalized to its possible range (0-100, higher is better); GPT-5.6 Sol grades, models run at API default reasoning effort. The post's 15 models only: the paper's behavior figure uses a different scale.
OpenAI MentalHealthBench - Behavior - Empathy and Support (Axis score, normalized to its possible range (%)): leader Claude Opus 5.5 (84.25), 15 models
Behavior axis of OpenAI's MentalHealthBench (launch post and paper, 2026-09-23): warmth, validation and emotional attunement without minimizing, judging or exaggerated mirroring, scored on the 596 rubric criteria tagged with this axis. The post's axis score is the signed rubric score on this axis, normalized to its possible range (0-100, higher is better); GPT-5.6 Sol grades, models run at API default reasoning effort. The post's 15 models only: the paper's behavior figure uses a different scale.
OpenAI MentalHealthBench - Behavior - Reality Testing (Axis score, normalized to its possible range (%)): leader GPT-6 (91.65), 15 models
Behavior axis of OpenAI's MentalHealthBench (launch post and paper, 2026-09-23): not affirming implausible, paranoid or delusional beliefs as fact while acknowledging distress, scored on the 363 rubric criteria tagged with this axis. The post's axis score is the signed rubric score on this axis, normalized to its possible range (0-100, higher is better); GPT-5.6 Sol grades, models run at API default reasoning effort. The post's 15 models only: the paper's behavior figure uses a different scale.
OpenAI MentalHealthBench - Behavior - Urgency Calibration (Axis score, normalized to its possible range (%)): leader Gemini 3.8 Flash (81.44), 15 models
Behavior axis of OpenAI's MentalHealthBench (launch post and paper, 2026-09-23): matching concern, escalation and referral to the severity of the situation, avoiding both over-alarm and under-response, scored on the 285 rubric criteria tagged with this axis. The post's axis score is the signed rubric score on this axis, normalized to its possible range (0-100, higher is better); GPT-5.6 Sol grades, models run at API default reasoning effort. The post's 15 models only: the paper's behavior figure uses a different scale.
OpenAI MentalHealthBench - Behavior - Harm Avoidance (Axis score, normalized to its possible range (%)): leader Muse Spark 1.3 (82.29), 15 models
Behavior axis of OpenAI's MentalHealthBench (launch post and paper, 2026-09-23): not enabling harmful or coercive behavior while staying helpful through proportionate refusal and redirection, scored on the 230 rubric criteria tagged with this axis. The post's axis score is the signed rubric score on this axis, normalized to its possible range (0-100, higher is better); GPT-5.6 Sol grades, models run at API default reasoning effort. The post's 15 models only: the paper's behavior figure uses a different scale.
OpenAI MentalHealthBench - Behavior - User Agency (Axis score, normalized to its possible range (%)): leader Claude Sonnet 5 (67.09), 15 models
Behavior axis of OpenAI's MentalHealthBench (launch post and paper, 2026-09-23): respecting autonomy and supporting the user's own informed decisions, scored on the 170 rubric criteria tagged with this axis. The post's axis score is the signed rubric score on this axis, normalized to its possible range (0-100, higher is better); GPT-5.6 Sol grades, models run at API default reasoning effort. The post's 15 models only: the paper's behavior figure uses a different scale.
OpenAI MentalHealthBench - Behavior - Communication (Axis score, normalized to its possible range (%)): leader GPT-6 Sol (89.28), 15 models
Behavior axis of OpenAI's MentalHealthBench (launch post and paper, 2026-09-23): clear, direct, accessible writing, scored on the 49 rubric criteria tagged with this axis. The post's axis score is the signed rubric score on this axis, normalized to its possible range (0-100, higher is better); GPT-5.6 Sol grades, models run at API default reasoning effort. The post's 15 models only: the paper's behavior figure uses a different scale.
IOL-AI Challenge 2026 (Geometric Mean of chrF and Exact Match (%; IOL 2026 Individual Contest, automatic)): leader Claude Opus 4.8 (75.1), 18 models
The five unseen problems of the International Linguistics Olympiad 2026 Individual Contest, where the model works out an unfamiliar language's grammar and vocabulary from a small set of worked examples. Automatic score after the paper's answer parsing: the geometric mean of chrF and exact match. Higher is better. Source: arxiv.org/abs/2608.18011.
IOL-AI Challenge 2026 - Exact Match (Exact Match (%; IOL 2026 Individual Contest)): leader Claude Opus 4.8 (68.1), 18 models
Exact-match share of answers on the IOL 2026 Individual Contest problems, the strict half of the IOL-AI Challenge score with no credit for near-miss spellings. Higher is better. Source: arxiv.org/abs/2608.18011.
Linguini - IOL-AI Harness (Geometric Mean of chrF and Exact Match (%; after custom answer parsing)): leader GPT-5.6 Sol (51.5), 18 models
The Linguini olympiad-linguistics set re-run with the IOL-AI Challenge harness on a 2026 model panel: geometric mean of chrF and exact match after answer parsing. Higher is better. Source: arxiv.org/abs/2608.18011, Table 12.
Phi-Bench - Training (Infrastructure Score (%)): leader Claude Opus 5 (56.4), 8 models
Phi-Bench evaluates agents implementing and optimizing AI infrastructure, including kernels, serving, compression and system engineering. Scores combine correctness and performance under the paper's shared harness; task formats stay separate. This series reports Phi-Bench - Training from 2609.10226v1, Sx3.T2: Infrastructure Score (%). Higher is better.
Phi-Bench - Inference and Serving (Infrastructure Score (%)): leader Kimi K3 (30.5), 8 models
Phi-Bench evaluates agents implementing and optimizing AI infrastructure, including kernels, serving, compression and system engineering. Scores combine correctness and performance under the paper's shared harness; task formats stay separate. This series reports Phi-Bench - Inference and Serving from 2609.10226v1, Sx3.T2: Infrastructure Score (%). Higher is better.
Phi-Bench - Compression (Infrastructure Score (%)): leader Claude Opus 5 (15.7), 8 models
Phi-Bench evaluates agents implementing and optimizing AI infrastructure, including kernels, serving, compression and system engineering. Scores combine correctness and performance under the paper's shared harness; task formats stay separate. This series reports Phi-Bench - Compression from 2609.10226v1, Sx3.T2: Infrastructure Score (%). Higher is better.
Phi-Bench - Kernel (Infrastructure Score (%)): leader Claude Opus 5 (30.0), 8 models
Phi-Bench evaluates agents implementing and optimizing AI infrastructure, including kernels, serving, compression and system engineering. Scores combine correctness and performance under the paper's shared harness; task formats stay separate. This series reports Phi-Bench - Kernel from 2609.10226v1, Sx3.T2: Infrastructure Score (%). Higher is better.
Phi-Bench - I/O (Infrastructure Score (%)): leader Claude Opus 5 (57.1), 8 models
Phi-Bench evaluates agents implementing and optimizing AI infrastructure, including kernels, serving, compression and system engineering. Scores combine correctness and performance under the paper's shared harness; task formats stay separate. This series reports Phi-Bench - I/O from 2609.10226v1, Sx3.T2: Infrastructure Score (%). Higher is better.
Phi-Bench - Hardware and Edge (Infrastructure Score (%)): leader Qwen 3.7 Max (5.4), 8 models
Phi-Bench evaluates agents implementing and optimizing AI infrastructure, including kernels, serving, compression and system engineering. Scores combine correctness and performance under the paper's shared harness; task formats stay separate. This series reports Phi-Bench - Hardware and Edge from 2609.10226v1, Sx3.T2: Infrastructure Score (%). Higher is better.
Phi-Bench - Data Infrastructure (Infrastructure Score (%)): leader Claude Opus 5 (49.4), 8 models
Phi-Bench evaluates agents implementing and optimizing AI infrastructure, including kernels, serving, compression and system engineering. Scores combine correctness and performance under the paper's shared harness; task formats stay separate. This series reports Phi-Bench - Data Infrastructure from 2609.10226v1, Sx3.T2: Infrastructure Score (%). Higher is better.
Phi-Bench - System Optimization (Infrastructure Score (%)): leader Kimi K3 (41.1), 8 models
Phi-Bench evaluates agents implementing and optimizing AI infrastructure, including kernels, serving, compression and system engineering. Scores combine correctness and performance under the paper's shared harness; task formats stay separate. This series reports Phi-Bench - System Optimization from 2609.10226v1, Sx3.T2: Infrastructure Score (%). Higher is better.
Phi-Bench - System Assurance (Infrastructure Score (%)): leader GLM-5.2 (57.4), 8 models
Phi-Bench evaluates agents implementing and optimizing AI infrastructure, including kernels, serving, compression and system engineering. Scores combine correctness and performance under the paper's shared harness; task formats stay separate. This series reports Phi-Bench - System Assurance from 2609.10226v1, Sx3.T2: Infrastructure Score (%). Higher is better.
Phi-Bench - Overall (Infrastructure Score (%)): leader Claude Opus 5 (36.53), 8 models
Phi-Bench evaluates agents implementing and optimizing AI infrastructure, including kernels, serving, compression and system engineering. Scores combine correctness and performance under the paper's shared harness; task formats stay separate. This series reports Phi-Bench - Overall from 2609.10226v1, Sx3.T2: Infrastructure Score (%). Higher is better.
Phi-Bench - Kernel Function Completion (Infrastructure Score (%)): leader Claude Opus 5 (37.16), 8 models
Phi-Bench evaluates agents implementing and optimizing AI infrastructure, including kernels, serving, compression and system engineering. Scores combine correctness and performance under the paper's shared harness; task formats stay separate. This series reports Phi-Bench - Kernel Function Completion from 2609.10226v1, Sx5.T3: Infrastructure Score (%). Higher is better.
Phi-Bench - Long-Horizon Implementation (Infrastructure Score (%)): leader Claude Opus 5 (21.6), 8 models
Phi-Bench evaluates agents implementing and optimizing AI infrastructure, including kernels, serving, compression and system engineering. Scores combine correctness and performance under the paper's shared harness; task formats stay separate. This series reports Phi-Bench - Long-Horizon Implementation from 2609.10226v1, Sx5.T3: Infrastructure Score (%). Higher is better.
Phi-Bench - End-to-End Optimization (Infrastructure Score (%)): leader Claude Opus 5 (62.94), 8 models
Phi-Bench evaluates agents implementing and optimizing AI infrastructure, including kernels, serving, compression and system engineering. Scores combine correctness and performance under the paper's shared harness; task formats stay separate. This series reports Phi-Bench - End-to-End Optimization from 2609.10226v1, Sx5.T3: Infrastructure Score (%). Higher is better.
MTVA-Bench - Actions (Score (0-100)): leader electron (70.8), 7 models
MTVA-Bench evaluates the language-model component of cascaded voice agents on 490 authored scenarios for 49 agents in seven languages. Each scenario uses one rollout; caller-invalid episodes are excluded. Language subsets contain different agents and tasks, rather than matched translations. This series reports deterministic required/prohibited tool-action checks from 2609.20152v1, S7.T5: Score (0-100). Higher is better.
MTVA-Bench - Criteria (Score (0-100)): leader GPT-5.6 Sol (82.7), 7 models
MTVA-Bench evaluates the language-model component of cascaded voice agents on 490 authored scenarios for 49 agents in seven languages. Each scenario uses one rollout; caller-invalid episodes are excluded. Language subsets contain different agents and tasks, rather than matched translations. This series reports judged task criteria from 2609.20152v1, S7.T5: Score (0-100). Higher is better.
MTVA-Bench - Task (Score (0-100)): leader GPT-5.6 Sol (73.6), 7 models
MTVA-Bench evaluates the language-model component of cascaded voice agents on 490 authored scenarios for 49 agents in seven languages. Each scenario uses one rollout; caller-invalid episodes are excluded. Language subsets contain different agents and tasks, rather than matched translations. This series reports combined action and criteria task score from 2609.20152v1, S7.T5: Score (0-100). Higher is better.
MTVA-Bench - Conversation (Score (0-100)): leader GPT-5.6 Sol (74.1), 7 models
MTVA-Bench evaluates the language-model component of cascaded voice agents on 490 authored scenarios for 49 agents in seven languages. Each scenario uses one rollout; caller-invalid episodes are excluded. Language subsets contain different agents and tasks, rather than matched translations. This series reports anchored conversational quality scaled to 0-100 from 2609.20152v1, S7.T5: Score (0-100). Higher is better.
MTVA-Bench - Overall (Score (0-100)): leader GPT-5.6 Sol (73.9), 7 models
MTVA-Bench evaluates the language-model component of cascaded voice agents on 490 authored scenarios for 49 agents in seven languages. Each scenario uses one rollout; caller-invalid episodes are excluded. Language subsets contain different agents and tasks, rather than matched translations. This series reports combined task and conversation score from 2609.20152v1, S7.T5: Score (0-100). Higher is better.
MTVA-Bench - Pass Rate (Pass Rate (%)): leader GPT-5.6 Sol (65.0), 7 models
MTVA-Bench evaluates the language-model component of cascaded voice agents on 490 authored scenarios for 49 agents in seven languages. Each scenario uses one rollout; caller-invalid episodes are excluded. Language subsets contain different agents and tasks, rather than matched translations. This series reports fraction meeting the provisional overall cutoff of 70 from 2609.20152v1, S7.T5: Pass Rate (%). Higher is better.
MTVA-Bench - English (Overall Score (0-100)): leader GPT-5.6 Sol (79.5), 7 models
MTVA-Bench evaluates the language-model component of cascaded voice agents on 490 authored scenarios for 49 agents in seven languages. Each scenario uses one rollout; caller-invalid episodes are excluded. Language subsets contain different agents and tasks, rather than matched translations. This series reports MTVA-Bench - English from 2609.20152v1, S7.T6: Overall Score (0-100). Higher is better.
MTVA-Bench - Gujarati (Overall Score (0-100)): leader GPT-5.6 Sol (76.9), 7 models
MTVA-Bench evaluates the language-model component of cascaded voice agents on 490 authored scenarios for 49 agents in seven languages. Each scenario uses one rollout; caller-invalid episodes are excluded. Language subsets contain different agents and tasks, rather than matched translations. This series reports MTVA-Bench - Gujarati from 2609.20152v1, S7.T6: Overall Score (0-100). Higher is better.
MTVA-Bench - Hindi (Overall Score (0-100)): leader electron (67.7), 7 models
MTVA-Bench evaluates the language-model component of cascaded voice agents on 490 authored scenarios for 49 agents in seven languages. Each scenario uses one rollout; caller-invalid episodes are excluded. Language subsets contain different agents and tasks, rather than matched translations. This series reports MTVA-Bench - Hindi from 2609.20152v1, S7.T6: Overall Score (0-100). Higher is better.
MTVA-Bench - Marathi (Overall Score (0-100)): leader GPT-5.6 Sol (74.7), 7 models
MTVA-Bench evaluates the language-model component of cascaded voice agents on 490 authored scenarios for 49 agents in seven languages. Each scenario uses one rollout; caller-invalid episodes are excluded. Language subsets contain different agents and tasks, rather than matched translations. This series reports MTVA-Bench - Marathi from 2609.20152v1, S7.T6: Overall Score (0-100). Higher is better.
MTVA-Bench - Spanish (Overall Score (0-100)): leader GPT-5.6 Sol (70.9), 7 models
MTVA-Bench evaluates the language-model component of cascaded voice agents on 490 authored scenarios for 49 agents in seven languages. Each scenario uses one rollout; caller-invalid episodes are excluded. Language subsets contain different agents and tasks, rather than matched translations. This series reports MTVA-Bench - Spanish from 2609.20152v1, S7.T6: Overall Score (0-100). Higher is better.
MTVA-Bench - Tamil (Overall Score (0-100)): leader GPT-5.6 Sol (70.6), 7 models
MTVA-Bench evaluates the language-model component of cascaded voice agents on 490 authored scenarios for 49 agents in seven languages. Each scenario uses one rollout; caller-invalid episodes are excluded. Language subsets contain different agents and tasks, rather than matched translations. This series reports MTVA-Bench - Tamil from 2609.20152v1, S7.T6: Overall Score (0-100). Higher is better.
MTVA-Bench - Telugu (Overall Score (0-100)): leader GPT-5.6 Sol (75.5), 7 models
MTVA-Bench evaluates the language-model component of cascaded voice agents on 490 authored scenarios for 49 agents in seven languages. Each scenario uses one rollout; caller-invalid episodes are excluded. Language subsets contain different agents and tasks, rather than matched translations. This series reports MTVA-Bench - Telugu from 2609.20152v1, S7.T6: Overall Score (0-100). Higher is better.
MTVA-Bench - Medium (Overall Score (0-100)): leader GPT-5.6 Sol (75.3), 7 models
MTVA-Bench evaluates the language-model component of cascaded voice agents on 490 authored scenarios for 49 agents in seven languages. Each scenario uses one rollout; caller-invalid episodes are excluded. Language subsets contain different agents and tasks, rather than matched translations. This series reports MTVA-Bench - Medium from 2609.20152v1, S7.T6: Overall Score (0-100). Higher is better.
MTVA-Bench - Hard (Overall Score (0-100)): leader GPT-5.6 Sol (72.5), 7 models
MTVA-Bench evaluates the language-model component of cascaded voice agents on 490 authored scenarios for 49 agents in seven languages. Each scenario uses one rollout; caller-invalid episodes are excluded. Language subsets contain different agents and tasks, rather than matched translations. This series reports MTVA-Bench - Hard from 2609.20152v1, S7.T6: Overall Score (0-100). Higher is better.
ParseBench - Tables (GTRM Score (GriTS + TableRecordMatch)): leader Claude Opus 5.5 (93.86), 78 models
LlamaIndex's ParseBench scores document parsing of human-verified enterprise pages from insurance, finance and government documents with rule-based checks; VLM rows only, parser products excluded. This board reports the Tables dimension: structural fidelity of tables with merged cells and hierarchical headers (GTRM, combining GriTS and TableRecordMatch).
ParseBench - Charts (ChartDataPointMatch Score): leader Qwen 3.8 27B (Thinking) (69.17), 78 models
LlamaIndex's ParseBench scores document parsing of human-verified enterprise pages from insurance, finance and government documents with rule-based checks; VLM rows only, parser products excluded. This board reports the Charts dimension: exact data-point extraction with correct series and axis labels from bar, line, pie and compound charts.
ParseBench - Content Faithfulness (Content Faithfulness Score): leader Claude Opus 5.5 (91.81), 78 models
LlamaIndex's ParseBench scores document parsing of human-verified enterprise pages from insurance, finance and government documents with rule-based checks; VLM rows only, parser products excluded. This board reports the Content Faithfulness dimension: penalises omitted text, hallucinated text and reading-order violations.
ParseBench - Semantic Formatting (Semantic Formatting Score): leader GLM-5.3 Flash (79.49), 78 models
LlamaIndex's ParseBench scores document parsing of human-verified enterprise pages from insurance, finance and government documents with rule-based checks; VLM rows only, parser products excluded. This board reports the Semantic Formatting dimension: preservation of meaning-bearing strikethrough, superscript/subscript, bold and title hierarchy.
ParseBench - Visual Grounding (Element Pass Rate): leader Infinity-Parser2-Flash (80.61), 69 models
LlamaIndex's ParseBench scores document parsing of human-verified enterprise pages from insurance, finance and government documents with rule-based checks; VLM rows only, parser products excluded. This board reports the Visual Grounding dimension: element pass rate for tracing extracted elements to their location on the page. Models that emit no layout boxes (upstream score exactly 0) are omitted.

Top-10 New Scores (171)

Claude Fable 5 on ForecastBench: 65.9 (#64)
Claude Fable 5 on ForecastBench Tournament - Market: 72.0 (#44)
Claude Opus 5 on ForecastBench: 66.1 (#53)
Claude Opus 5 on ForecastBench Tournament - Market: 72.2 (#38)
Claude Opus 5.5 on AI Coding Daily (Claude Code) - Laravel Code Quality: 18.95 (#1)
Claude Opus 5.5 on AI Coding Daily (Claude Code) - React-TS Code Quality: 19.67 (#1)
Claude Opus 5.5 on AI Coding Daily (Claude Code) - Total: 57.83 (#1)
Claude Opus 5.5 on ActiveVision: 41.2 (#2)
Claude Opus 5.5 on ActiveVision (Agents): 82.4 (#2)
Claude Opus 5.5 on Agents on Rails: 33.3 (#3)
Claude Opus 5.5 on Aiera Leaderboard - Depth: 71.1 (#15)
Claude Opus 5.5 on Aiera Leaderboard - Facts: 61.08 (#22)
Claude Opus 5.5 on Aiera Leaderboard - Grounding: 76.2 (#25)
Claude Opus 5.5 on Aiera Leaderboard - Research Score: 69.46 (#21)
Claude Opus 5.5 on BridgeBench Overall: 760.0 (#1)
Claude Opus 5.5 on Bug Hunt Bench: 22.3 (#39)
Claude Opus 5.5 on Bug Hunt Bench - LMS: 11.0 (#46)
Claude Opus 5.5 on Bug Hunt Bench - VS Code Extension: 11.3 (#32)
Claude Opus 5.5 on Chartography: 66.3 (#3)
Claude Opus 5.5 on ChessBench: 53.9 (#7)
Claude Opus 5.5 on ChessBench - Accuracy: 76.3 (#5)
Claude Opus 5.5 on ChessBench - Coherence: 64.4 (#17)
Claude Opus 5.5 on ChessBench - Elo: 1167.0 (#4)
Claude Opus 5.5 on ComplexConstraints: 49.9 (#7)
Claude Opus 5.5 on DeepsecBench: 26.68 (#9)
Claude Opus 5.5 on EQ-Bench Longform Writing: 84.6 (#3)
Claude Opus 5.5 on EnterpriseBench (CoreCraft Agents): 74.4 (#2)
Claude Opus 5.5 on GDP.pdf: 31.4 (#3)
Claude Opus 5.5 on Gert Labs Rankings: 73.78 (#2)
Claude Opus 5.5 on Icelandic LLM - Case Government: 97.5 (#5)
Claude Opus 5.5 on Icelandic LLM - Famous Phrases: 27.62 (#6)
Claude Opus 5.5 on Icelandic LLM - GEC: 80.73 (#4)
Claude Opus 5.5 on Icelandic LLM - Generative Quality: 74.54 (#7)
Claude Opus 5.5 on Icelandic LLM - Idiom Meanings: 84.04 (#2)
Claude Opus 5.5 on Icelandic LLM - Song Lyrics: 20.53 (#2)
Claude Opus 5.5 on Icelandic LLM - Textile Terms: 73.3 (#2)
Claude Opus 5.5 on Kaggle Open Benchmarks - AIME 2025: 100.0 (#6)
Claude Opus 5.5 on KernelBench Hub - CUDA: 79.38 (#2)
Claude Opus 5.5 on MLCR-AA: 66.67 (#2)
Claude Opus 5.5 on MLCR-AA - Accuracy: 93.15 (#6)
Claude Opus 5.5 on MLCR-AA - Completeness: 88.36 (#2)
Claude Opus 5.5 on MLCR-AA - Conciseness: 81.11 (#86)
Claude Opus 5.5 on OpenRouter GPQA Diamond: 89.6 (#20)
Claude Opus 5.5 on OpenRouter Tau2-Bench Airline: 73.8 (#43)
Claude Opus 5.5 on PLCC - Art & Entertainment: 86.0 (#29)
Claude Opus 5.5 on PLCC - Culture & Tradition: 93.0 (#27)
Claude Opus 5.5 on PLCC - Geography: 94.0 (#46)
Claude Opus 5.5 on PLCC - Grammar: 94.0 (#10)
Claude Opus 5.5 on PLCC - History: 97.0 (#8)
Claude Opus 5.5 on PLCC - Overall: 92.5 (#14)
Claude Opus 5.5 on PLCC - Vocabulary: 91.0 (#16)
Claude Opus 5.5 on Riemann-bench: 69.6 (#3)
Claude Opus 5.5 on Tinybird AI SQL Benchmark - Exactness: 61.33 (#4)
Claude Opus 5.5 on Tinybird AI SQL Benchmark - First-Attempt Success Rate: 90.0 (#216)
Claude Opus 5.5 on Tinybird AI SQL Benchmark - Success Rate: 100.0 (#40)
Claude Opus 5.5 on VoxelBench: 2554.0 (#1)
GPT-6 on Chartography: 71.0 (#1)
GPT-6 on ComplexConstraints: 57.7 (#1)
GPT-6 on Kaggle ExtractBench: 90.28 (#2)
GPT-6 on Kaggle SimpleQA: 73.06 (#2)
GPT-6 on Kaggle VideoQA: 75.25 (#10)
GPT-6 on Riemann-bench: 72.0 (#2)
GPT-6 Sol on Agents on Rails: 21.7 (#13)
GPT-6 Sol on BridgeBench Overall: 643.0 (#4)
GPT-6 Sol on Bug Hunt Bench: 25.0 (#31)
GPT-6 Sol on Bug Hunt Bench - LMS: 15.0 (#26)
GPT-6 Sol on Bug Hunt Bench - VS Code Extension: 10.0 (#38)
GPT-6 Sol on Chartography: 53.6 (#5)
GPT-6 Sol on Chatbot Arena (Code): 1689.0 (#4)
GPT-6 Sol on ChessBench GitHub - Elo: 2490.0 (#8)
GPT-6 Sol on ComplexConstraints: 50.8 (#5)
GPT-6 Sol on DeepsecBench: 40.91 (#1)
GPT-6 Sol on Design Arena (Data Viz): 1200.0 (#70)
GPT-6 Sol on Design Arena (Game Dev): 1331.0 (#15)
GPT-6 Sol on EQ-Bench Longform Writing: 82.8 (#8)
GPT-6 Sol on Epoch AI - Ebr Bench: 53.33 (#3)
GPT-6 Sol on GDP.pdf: 26.4 (#11)
GPT-6 Sol on Gert Labs Rankings: 73.75 (#3)
GPT-6 Sol on HieroglyphBench: 26.0 (#9)
GPT-6 Sol on Icelandic LLM - Case Government: 94.38 (#11)
GPT-6 Sol on Icelandic LLM - Famous Phrases: 26.06 (#7)
GPT-6 Sol on Icelandic LLM - Flora & Fauna: 39.04 (#10)
GPT-6 Sol on Icelandic LLM - GEC: 74.77 (#5)
GPT-6 Sol on Icelandic LLM - Idiom Completion: 65.87 (#10)
GPT-6 Sol on Icelandic LLM - Idiom Meanings: 81.55 (#9)
GPT-6 Sol on Icelandic LLM - Song Lyrics: 12.5 (#17)
GPT-6 Sol on Icelandic LLM - Textile Terms: 70.1 (#4)
GPT-6 Sol on Icelandic LLM - WikiQA: 58.95 (#11)
GPT-6 Sol on Icelandic LLM Leaderboard - Average: 65.63 (#6)
GPT-6 Sol on Kaggle Open Benchmarks - AIME 2025: 100.0 (#7)
GPT-6 Sol on LLM2014 Logic 2026-09: 61.94 (#8)
GPT-6 Sol on Mercor APEX: 54.3 (#18)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Adversarial User Simulations - Emotional Reliance: 96.9 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Adversarial User Simulations - Mental Health: 99.7 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Adversarial User Simulations - Self-Harm: 97.3 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Age-Restricted Actions: 75.5 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Non-Violent Wrongdoing: 99.0 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Self-Harm: 92.0 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Violent Wrongdoing: 88.9 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Human Red-Teaming Chat Plugins: 93.6 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Human Red-Teaming Codex: 93.5 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Biology Refusal - Benign: 96.4 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Biology Refusal - Severe: 99.8 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Challenging Prompts - Extremism: 93.6 (#5)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Challenging Prompts - Hate: 98.2 (#6)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Challenging Prompts - Non-Violent Illicit Behavior: 99.3 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Challenging Prompts - Self-Harm: 98.6 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Challenging Prompts - Violent Illicit Behavior: 98.8 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Cyber Safety - Production Chat: 0.957 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Cyber Safety - Semi-Synthetic Agentic Environments: 0.984 (#4)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Cyber Safety - Synthetic Agentic Environments: 0.998 (#1)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Image Input - Erotic: 99.8 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Image Input - Extremism: 97.5 (#6)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Image Input - Hate: 99.8 (#4)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Image Input - Self-Harm: 98.2 (#6)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - SHP2 Protein Function Prediction: 0.273 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Static Jailbreaks - Bio High Risk: 85.8 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Static Jailbreaks - Bio Severe: 81.3 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Static Jailbreaks - Cyber: 79.8 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Static Jailbreaks - Violence Moderate: 89.5 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - Static Jailbreaks - Violence Severe: 90.5 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - U18 - Age-Restricted Goods and Dangerous Activities: 86.1 (#2)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - U18 - Eating Disorders: 85.3 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - U18 - Self-Harm: 99.0 (#3)
GPT-6 Sol on OpenAI GPT-6 Astra System Card - U18 - Sexual Content: 97.5 (#2)
GPT-6 Sol on OpenRouter GPQA Diamond: 92.4 (#11)
GPT-6 Sol on OpenRouter Tau2-Bench Airline: 68.7 (#70)
GPT-6 Sol on PLCC - Art & Entertainment: 94.0 (#7)
GPT-6 Sol on PLCC - Culture & Tradition: 93.0 (#26)
GPT-6 Sol on PLCC - Geography: 98.0 (#14)
GPT-6 Sol on PLCC - Grammar: 93.0 (#13)
GPT-6 Sol on PLCC - History: 97.0 (#7)
GPT-6 Sol on PLCC - Overall: 93.83 (#13)
GPT-6 Sol on PLCC - Vocabulary: 88.0 (#27)
GPT-6 Sol on ParseBench: 68.19 (#19)
GPT-6 Sol on ReLE - Agents - BFCL-V3: 46.5 (#116)
GPT-6 Sol on ReLE - Agents and Tool Use: 50.7 (#111)
GPT-6 Sol on Riemann-bench: 64.8 (#7)
GPT-6 Sol on Sonar LLM Leaderboard - Java - Bug Density: 0.59 (#8)
GPT-6 Sol on Sonar LLM Leaderboard - Java - Code Smell Density: 19.28 (#44)
GPT-6 Sol on Sonar LLM Leaderboard - Java - Pass Rate: 83.64 (#5)
GPT-6 Sol on Sonar LLM Leaderboard - Java - Vulnerability Density: 0.28 (#40)
GPT-6 Sol on Tinybird AI SQL Benchmark - First-Attempt Success Rate: 94.0 (#167)
GPT-6 Sol on Tinybird AI SQL Benchmark - Success Rate: 96.0 (#263)
GPT-6 Sol on Vals AI (Vals Index): 62.57 (#8)
GPT-6 Sol on Vals AI BioMysteryBench: 74.81 (#4)
GPT-6 Sol on Vals AI Code Migration: 57.2 (#4)
GPT-6 Sol on Vals AI Excel Modeling: 71.53 (#8)
GPT-6 Sol on Vals AI Finance Agent v2: 49.05 (#37)
GPT-6 Sol on Vals AI Harvey Legal Agent Bench: 1.67 (#41)
GPT-6 Sol on Vals AI IOI: 82.61 (#7)
GPT-6 Sol on Vals AI Legal Research Bench: 28.85 (#38)
GPT-6 Sol on Vals AI MedCode: 47.07 (#31)
GPT-6 Sol on Vals AI MedScribe: 82.03 (#45)
GPT-6 Sol on Vals AI ProgramBench: 81.91 (#5)
GPT-6 Sol on Vals AI ProofBench: 83.0 (#9)
GPT-6 Sol on Vals AI Public Benefits Bench: 56.63 (#30)
GPT-6 Sol on Vals AI SAGE: 44.79 (#39)
GPT-6 Sol on Vals AI Tax Agent Bench: 53.05 (#22)
GPT-6 Sol on Vals AI Terminal-Bench 2.1: 83.15 (#6)
GPT-6 Sol on Vals AI Vibe Code Bench: 87.82 (#6)
GPT-6 Sol on WebDev Arena (Brand & Marketing): 1666.0 (#5)
GPT-6 Sol on WebDev Arena (Consumer Product): 1634.0 (#4)
GPT-6 Sol on WebDev Arena (Content Creation Tools): 1703.0 (#3)
GPT-6 Sol on WebDev Arena (Data & Analytics): 1600.0 (#6)
GPT-6 Sol on WebDev Arena (Frontend): 1702.0 (#4)
GPT-6 Sol on WebDev Arena (Gaming): 1775.0 (#3)
GPT-6 Sol on WebDev Arena (HTML): 1688.0 (#3)
GPT-6 Sol on WebDev Arena (React): 1699.0 (#4)
GPT-6 Sol on WebDev Arena (Reference-Based Design): 1686.0 (#4)
GPT-6 Sol on WebDev Arena (Simulations): 1768.0 (#3)

New #1 Leaders (26)

Chartography: GPT-6 (Max) (71.0) beat Claude Fable 5.1 (Max) by 24.8
OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Sensitive Personal Data: GPT-6 Sol (85.4) beat GPT-5.6 Sol by 8.9
ComplexConstraints: GPT-6 (Max) (57.7) beat Muse Spark 1.3 (xHigh) by 5.8
GDP.pdf: GPT-6 (35.4) beat GPT-5.6 Sol (Max) by 4.7
OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Non-Violent Wrongdoing: GPT-6 Luna (100.0) beat GPT-6 by 4.6
AI Coding Daily (Claude Code) - Total: Claude Opus 5.5 (High) (57.83) beat Claude Opus 5 (High) by 4.11
OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Self-Harm: GPT-6 Luna (96.0) beat GPT-6 by 4.0
Architectural RedlineBench: Claude Opus 5.5 (68.8) beat GPT-6 by 3.3
DeepsecBench: GPT-6 Sol (xHigh) (40.91) beat GPT-6 (xHigh) by 3.12
Icelandic LLM - Idiom Completion: Claude Opus 5.5 (75.03) beat GPT-6 by 2.52
TaxCalcBench: GPT-6 Sol Web Search (64.0) beat GPT-5.6 Sol Web Search by 2.0
AI Coding Daily (Claude Code) - React-TS Code Quality: Claude Opus 5.5 (Medium) (19.67) beat Claude Opus 5 (High) by 2.0
OpenAI GPT-6 Astra System Card - Challenging Prompts - Sexual/Minors: GPT-6 Sol (99.1) beat GPT-6 by 1.7
Icelandic LLM - Generative Quality: GPT-6 Sol (83.75) beat GPT-6 by 1.53
AutomationBench: Claude Opus 5.5 (default fallbacks, Max) (42.5) beat GPT-6 (Max) by 1.1
OpenAI GPT-6 Astra System Card - Biology Refusal - Dual Use: GPT-6 Sol (98.0) beat GPT-6 by 1.0
Icelandic LLM - Flora & Fauna: Claude Opus 5.5 (57.42) beat GPT-6 by 0.88
Icelandic LLM Leaderboard - Average: Claude Opus 5.5 (70.87) beat GPT-6 by 0.62
Kaggle AssetOpsBench: GPT-6 (69.55) beat Gemini 3 Pro (Preview) by 0.41
OpenAI GPT-6 Astra System Card - Challenging Prompts - Sexual: GPT-6 Sol (98.4) beat GPT-6 by 0.4
OpenAI GPT-6 Astra System Card - U18 - Emotional Reliance: GPT-6 Sol (94.8) beat GPT-6 by 0.4
AI Coding Daily (Claude Code) - Laravel Code Quality: Claude Opus 5.5 (Medium) (18.95) beat Claude Opus 5 (High) by 0.4
ReLE - Language and Instruction Following: Hy4 preview (78.8) beat doubao-seed-2-1-pro-260628 by 0.4
Icelandic LLM - WikiQA: Claude Opus 5.5 (74.32) beat Claude Fable 5.1 by 0.37
OpenAI GPT-6 Astra System Card - Challenging Prompts - Gore: GPT-6 Sol (90.0) beat GPT-6 by 0.2
OpenAI GPT-6 Astra System Card - Image Input - Self-Harm: GPT-6 Luna (99.9) beat GPT-6 by 0.2

The Aggregate Digest — 2026-09-23

2026-09-23T06:04:41.923216+00:00

Frontier

Best available model: Claude Opus 5.5 enters at #2 (Aggregate Rating 2158) on 173 benchmarks
Best available model: GPT-6 Sol enters at #9 (Aggregate Rating 2088) on 109 benchmarks

Daily

New Benchmarks (4)

LLM Stats (BenchCAD) (Score (%)): leader Claude Opus 5.5 (73.0), 5 models
LLM Stats compilation of vendor-reported BenchCAD results without tools: BenchCAD is built from 17,900 execution-verified CadQuery programs across 106 industrial part families, and its Vision2Code task asks the model to write CadQuery code that reproduces a part from multi-view renders, scored by voxel IoU against the reference geometry (0-1, shown as %). Higher is better. All rows are self-reported, and vendors may report different task subsets.
LLM Stats (BenchCAD (with Python tool)) (Score (%)): leader Claude Opus 5.5 (96.2), 5 models
LLM Stats compilation of vendor-reported BenchCAD Vision2Code results with a Python tool available: the model writes CadQuery code reproducing an industrial part from multi-view renders and can execute code while working, scored by voxel IoU against the reference geometry (0-1, shown as %). Higher is better. Tool access lifts scores well above the no-tool BenchCAD board; all rows are self-reported.
LLM Stats (SWE-Bench Multimodal) (Score (%)): leader Claude Opus 5.5 (61.4), 5 models
LLM Stats compilation of vendor-reported SWE-bench Multimodal results: GitHub issues from JavaScript front-end and visualisation libraries whose reports carry screenshots, mockups or diagrams (517-task test split), resolved when the model's patch passes the repository tests. Percent resolved, higher is better. All rows are self-reported and harnesses differ by vendor (Anthropic reports its internal harness), so this is not the swebench.com board.
BridgeBench Overall (Overall (mean of nine axes, 1-10)): leader GPT-6 (7.2), 12 models
BridgeBench V3 coding leaderboard: each model is rated 1-10 on nine axes (reasoning, front end, back end, one-shot, security, trust, laziness, speed, cost) and ranked by Overall, the mean of the nine. Laziness and cost are inverted so higher is better. Speed and cost are not capability measures, so cheap, fast models gain on Overall. It replaced the head-to-head arena ladder in September 2026.

New Models (3)

Claude Opus 5.5 — Aggregate Rating 2158, #2
- CursorBench 4.0: 57.8 (#1/44)
- KernelBench Hub - Mega: 35.46 (#1/12)
- LiveBench Olympiad: 93.25 (#1/63)
- LiveBench TypeScript: 63.33 (#1/63)
- Artificial Analysis Intelligence Index: 57.62 (#1/664)
- AA Humanity's Last Exam: 61.35 (#1/636)
- AA SciCode: 66.9 (#1/200)
- AA Omniscience: 46.42 (#1/548)
- AA GDPval: 1846.17 (#1/254)
- AA MMMU-Pro: 87.69 (#1/278)
- ...and 160 more
GPT-6 Sol — Aggregate Rating 2088, #9
- MLCR-AA - Conciseness: 100.0 (#1/90)
- AA GDP.pdf - Real Estate - Criterion Pass Rate: 79.84 (#1/179)
- AutomationBench-AA - Sales - Raw Objective Completion: 90.07 (#1/189)
- Next.js Agent Evals (Codex) - Success Rate: 97.0 (#1/8)
- MLCR-AA - Accuracy: 96.67 (#2/90)
- VoxelBench: 2440.0 (#2/51)
- LLM Stats (Agents' Last Exam): 56.4 (#2/21)
- Next.js Agent Evals (Codex) - Success Rate with AGENTS.md: 97.0 (#2/8)
- RuneBench: 13372.0 (#3/58)
- Maze-Bench: 58.01 (#3/36)
- ...and 99 more
GPT-6 Luna — Aggregate Rating 1935, #47
- Next.js Agent Evals (Codex) - Success Rate with AGENTS.md: 87.0 (#3/8)
- MLCR-AA - Accuracy: 93.75 (#4/90)
- Next.js Agent Evals (Codex) - Success Rate: 77.0 (#4/8)
- LLM Stats (Agents' Last Exam): 50.9 (#7/21)
- LLM Stats (OSWorld 2.0): 52.7 (#9/14)
- Featherbench: 96.0 (#10/25)
- LLM Stats (FrontierCode 1.1): 42.4 (#11/20)
- DataBench: 52.3 (#13/64)
- SvelteBench: 100.0 (#15/226)
- LiveBench Connections: 100.0 (#17/63)
- ...and 97 more

Top-10 New Scores (275)

Claude Fable 5.1 on SuperCLUE-VLM - Overall: 75.13 (#4)
Claude Fable 5.1 on Vals AI CyberBench: 70.42 (#3)
Claude Opus 5 on Vals AI CyberBench: 65.36 (#5)
Claude Opus 5.5 on AA CritPt: 31.71 (#2)
Claude Opus 5.5 on AA GDP.pdf: 28.8 (#6)
Claude Opus 5.5 on AA GDP.pdf - Construction - Criterion Pass Rate: 75.95 (#10)
Claude Opus 5.5 on AA GDP.pdf - Criterion Pass Rate: 82.25 (#1)
Claude Opus 5.5 on AA GDP.pdf - Engineering - Criterion Pass Rate: 84.35 (#1)
Claude Opus 5.5 on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 91.25 (#1)
Claude Opus 5.5 on AA GDP.pdf - HR - Criterion Pass Rate: 90.16 (#1)
Claude Opus 5.5 on AA GDP.pdf - Healthcare - Criterion Pass Rate: 93.1 (#1)
Claude Opus 5.5 on AA GDP.pdf - Insurance - Criterion Pass Rate: 88.05 (#4)
Claude Opus 5.5 on AA GDP.pdf - Legal - Criterion Pass Rate: 84.79 (#10)
Claude Opus 5.5 on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 79.36 (#7)
Claude Opus 5.5 on AA GDP.pdf - Real Estate - Criterion Pass Rate: 72.66 (#21)
Claude Opus 5.5 on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 81.47 (#1)
Claude Opus 5.5 on AA GDPval: 1846.17 (#1)
Claude Opus 5.5 on AA Humanity's Last Exam: 61.35 (#1)
Claude Opus 5.5 on AA Long Context Reasoning: 84.67 (#5)
Claude Opus 5.5 on AA MMMU-Pro: 87.69 (#1)
Claude Opus 5.5 on AA Omniscience: 46.42 (#1)
Claude Opus 5.5 on AA SciCode: 66.9 (#1)
Claude Opus 5.5 on AA-Briefcase: 61.21 (#2)
Claude Opus 5.5 on AA-Briefcase - Analytical Quality Elo: 2206.92 (#1)
Claude Opus 5.5 on AA-Briefcase - DOCX - Rubric Pass Rate: 60.36 (#1)
Claude Opus 5.5 on AA-Briefcase - PDF - Rubric Pass Rate: 64.66 (#2)
Claude Opus 5.5 on AA-Briefcase - PPTX - Rubric Pass Rate: 62.04 (#2)
Claude Opus 5.5 on AA-Briefcase - Presentation Elo: 1709.91 (#1)
Claude Opus 5.5 on AA-Briefcase - XLSX - Rubric Pass Rate: 61.14 (#1)
Claude Opus 5.5 on AA-Omniscience Accuracy: 66.22 (#2)
Claude Opus 5.5 on AA-Omniscience Hallucination Rate: 58.61 (#131)
Claude Opus 5.5 on AA-Omniscience Index - Business: 32.7 (#7)
Claude Opus 5.5 on AA-Omniscience Index - Health: 26.4 (#2)
Claude Opus 5.5 on AA-Omniscience Index - Humanities & Social Sciences: 44.7 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Law: 48.1 (#3)
Claude Opus 5.5 on AA-Omniscience Index - Science, Engineering & Mathematics: 37.8 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE): 88.8 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - C: 96.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Dart: 90.0 (#2)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Go: 88.0 (#7)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - HTML: 88.0 (#4)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Java: 80.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 88.18 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Julia: 84.0 (#8)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 96.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - PHP: 96.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Python: 91.5 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - R: 72.0 (#2)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Rust: 92.0 (#1)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - Swift: 92.0 (#4)
Claude Opus 5.5 on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 93.33 (#3)
Claude Opus 5.5 on AI Chess Leaderboard (Continuation): 1797.0 (#3)
Claude Opus 5.5 on ARC-AGI-1: 98.5 (#5)
Claude Opus 5.5 on ARC-AGI-2: 93.33 (#4)
Claude Opus 5.5 on Artificial Analysis Intelligence Index: 57.62 (#1)
Claude Opus 5.5 on AutomationBench: 40.0 (#2)
Claude Opus 5.5 on AutomationBench-AA: 69.54 (#1)
Claude Opus 5.5 on AutomationBench-AA - Finance - Raw Objective Completion: 88.25 (#11)
Claude Opus 5.5 on AutomationBench-AA - HR - Raw Objective Completion: 77.27 (#1)
Claude Opus 5.5 on AutomationBench-AA - Marketing - Raw Objective Completion: 91.07 (#2)
Claude Opus 5.5 on AutomationBench-AA - Operations - Raw Objective Completion: 96.74 (#2)
Claude Opus 5.5 on AutomationBench-AA - Sales - Raw Objective Completion: 89.49 (#2)
Claude Opus 5.5 on AutomationBench-AA - Strict: 41.25 (#2)
Claude Opus 5.5 on AutomationBench-AA - Support - Raw Objective Completion: 91.11 (#5)
Claude Opus 5.5 on BenchLM: 81.0 (#4)
Claude Opus 5.5 on Bug Hunt Bench: 41.0 (#5)
Claude Opus 5.5 on Bug Hunt Bench - LMS: 23.5 (#3)
Claude Opus 5.5 on Bug Hunt Bench - VS Code Extension: 17.5 (#10)
Claude Opus 5.5 on Coarena: 1037.8 (#3)
Claude Opus 5.5 on Coarena - Task Completion: 38.5 (#18)
Claude Opus 5.5 on DataBench: 70.5 (#1)
Claude Opus 5.5 on Featherbench: 100.0 (#3)
Claude Opus 5.5 on Featherbench - Rubric Quality: 9.5 (#1)
Claude Opus 5.5 on FrontierSWE V2: 62.3 (#2)
Claude Opus 5.5 on Harvey LAB-AA: 91.24 (#12)
Claude Opus 5.5 on Harvey LAB-AA - All Pass: 8.33 (#12)
Claude Opus 5.5 on Harvey LAB-AA - Antitrust & Competition: 92.03 (#10)
Claude Opus 5.5 on Harvey LAB-AA - Arbitration & International Dispute Resolution: 94.62 (#8)
Claude Opus 5.5 on Harvey LAB-AA - Banking & Finance: 94.42 (#6)
Claude Opus 5.5 on Harvey LAB-AA - Bankruptcy & Restructuring: 92.04 (#8)
Claude Opus 5.5 on Harvey LAB-AA - Capital Markets: 94.9 (#8)
Claude Opus 5.5 on Harvey LAB-AA - Corporate Governance: 90.41 (#13)
Claude Opus 5.5 on Harvey LAB-AA - Corporate M&A: 94.87 (#9)
Claude Opus 5.5 on Harvey LAB-AA - Data Privacy & Cybersecurity: 98.59 (#2)
Claude Opus 5.5 on Harvey LAB-AA - Emerging Companies & Venture Capital: 94.88 (#4)
Claude Opus 5.5 on Harvey LAB-AA - Employment & Labor: 88.39 (#9)
Claude Opus 5.5 on Harvey LAB-AA - Energy & Natural Resources: 96.04 (#14)
Claude Opus 5.5 on Harvey LAB-AA - Environmental & ESG: 86.74 (#16)
Claude Opus 5.5 on Harvey LAB-AA - Funds & Asset Management: 89.8 (#11)
Claude Opus 5.5 on Harvey LAB-AA - Healthcare & Life Sciences: 93.07 (#8)
Claude Opus 5.5 on Harvey LAB-AA - Immigration: 94.91 (#1)
Claude Opus 5.5 on Harvey LAB-AA - Insurance: 94.14 (#5)
Claude Opus 5.5 on Harvey LAB-AA - Intellectual Property: 96.11 (#7)
Claude Opus 5.5 on Harvey LAB-AA - International Trade & Sanctions: 91.27 (#7)
Claude Opus 5.5 on Harvey LAB-AA - Litigation & Dispute Resolution: 94.38 (#6)
Claude Opus 5.5 on Harvey LAB-AA - Real Estate: 94.46 (#12)
Claude Opus 5.5 on Harvey LAB-AA - Structured Finance & Securitization: 88.89 (#8)
Claude Opus 5.5 on Harvey LAB-AA - Tax: 85.28 (#17)
Claude Opus 5.5 on Harvey LAB-AA - Trusts, Estates & Private Client: 93.51 (#9)
Claude Opus 5.5 on Harvey LAB-AA - White Collar Defense & Investigations: 88.89 (#15)
Claude Opus 5.5 on HieroglyphBench: 37.4 (#7)
Claude Opus 5.5 on LLM Stats (AutomationBench): 40.0 (#7)
Claude Opus 5.5 on LLM Stats (BioMysteryBench): 89.3 (#2)
Claude Opus 5.5 on LLM Stats (DeepSWE 1.1): 74.2 (#2)
Claude Opus 5.5 on LLM Stats (HealthBench Professional): 65.6 (#2)
Claude Opus 5.5 on LLM Stats (Legal Agent Benchmark): 8.3 (#3)
Claude Opus 5.5 on LLM Stats (OfficeQA Pro): 67.7 (#3)
Claude Opus 5.5 on LM Market Cap LMC Score: 95.1 (#5)
Claude Opus 5.5 on LiveBench AMPS Hard: 99.0 (#11)
Claude Opus 5.5 on LiveBench Code Completion: 86.96 (#3)
Claude Opus 5.5 on LiveBench Code Generation: 91.55 (#2)
Claude Opus 5.5 on LiveBench Connections: 99.33 (#27)
Claude Opus 5.5 on LiveBench Consecutive Events: 90.56 (#5)
Claude Opus 5.5 on LiveBench Integrals With Game: 100.0 (#3)
Claude Opus 5.5 on LiveBench JavaScript: 81.82 (#2)
Claude Opus 5.5 on LiveBench Logic With Navigation: 86.0 (#6)
Claude Opus 5.5 on LiveBench Math Comp: 97.06 (#17)
Claude Opus 5.5 on LiveBench Olympiad: 93.25 (#1)
Claude Opus 5.5 on LiveBench Paraphrase: 64.68 (#39)
Claude Opus 5.5 on LiveBench Plot Unscrambling: 77.48 (#6)
Claude Opus 5.5 on LiveBench Python: 70.0 (#8)
Claude Opus 5.5 on LiveBench Simplify: 63.0 (#33)
Claude Opus 5.5 on LiveBench Spatial: 98.0 (#34)
Claude Opus 5.5 on LiveBench Story Generation: 71.18 (#32)
Claude Opus 5.5 on LiveBench Summarize: 69.33 (#35)
Claude Opus 5.5 on LiveBench Table Join: 52.33 (#8)
Claude Opus 5.5 on LiveBench Table Reformat: 98.04 (#47)
Claude Opus 5.5 on LiveBench Theory of Mind: 84.61 (#13)
Claude Opus 5.5 on LiveBench TypeScript: 63.33 (#1)
Claude Opus 5.5 on LiveBench Typos: 82.0 (#31)
Claude Opus 5.5 on LiveBench Zebra Puzzle: 100.0 (#13)
Claude Opus 5.5 on NYT Connections Extended: 88.5 (#28)
Claude Opus 5.5 on Next.js Agent Evals (Claude Code) - Success Rate: 97.0 (#2)
Claude Opus 5.5 on Next.js Agent Evals (Claude Code) - Success Rate with AGENTS.md: 97.0 (#3)
Claude Opus 5.5 on ParseBench: 78.01 (#2)
Claude Opus 5.5 on RuneBench: 18333.0 (#2)
Claude Opus 5.5 on SvelteBench: 93.3 (#62)
Claude Opus 5.5 on TaxCalcBench: 38.0 (#7)
Claude Opus 5.5 on Vals AI BioMysteryBench: 79.26 (#2)
Claude Opus 5.5 on Vals AI Code Migration: 66.65 (#2)
Claude Opus 5.5 on Vals AI Excel Modeling: 75.94 (#2)
Claude Opus 5.5 on Vals AI Finance Agent v2: 58.59 (#8)
Claude Opus 5.5 on Vals AI Harvey Legal Agent Bench: 3.75 (#33)
Claude Opus 5.5 on Vals AI IOI: 95.06 (#2)
Claude Opus 5.5 on Vals AI Legal Research Bench: 50.48 (#4)
Claude Opus 5.5 on Vals AI MedCode: 49.8 (#15)
Claude Opus 5.5 on Vals AI MedScribe: 91.43 (#1)
Claude Opus 5.5 on Vals AI MysteryMechanism: 49.55 (#2)
Claude Opus 5.5 on Vals AI ProgramBench: 86.99 (#1)
Claude Opus 5.5 on Vals AI ProofBench: 100.0 (#1)
Claude Opus 5.5 on Vals AI Public Benefits Bench: 70.64 (#3)
Claude Opus 5.5 on Vals AI RSI Index: 0.371 (#1)
Claude Opus 5.5 on Vals AI ReverseEngBench: 33.59 (#2)
Claude Opus 5.5 on Vals AI SAGE: 45.83 (#33)
Claude Opus 5.5 on Vals AI Tax Agent Bench: 70.5 (#6)
Claude Opus 5.5 on Vals AI Terminal-Bench Science: 48.57 (#2)
Claude Opus 5.5 on Vals AI Vibe Code Bench: 90.29 (#2)
GPT-6 on Kaggle Math Puzzles (Lewis Carroll Sorites): 87.5 (#5)
GPT-6 on LLM Chess (Saplin): 1579.3 (#2)
GPT-6 on Sonar LLM Leaderboard - Java - Bug Density: 0.56 (#5)
GPT-6 on Sonar LLM Leaderboard - Java - Code Smell Density: 18.47 (#33)
GPT-6 on Sonar LLM Leaderboard - Java - Pass Rate: 85.85 (#3)
GPT-6 on Sonar LLM Leaderboard - Java - Vulnerability Density: 0.18 (#19)
GPT-6 on Vals AI BioMysteryBench: 79.26 (#1)
GPT-6 on Vals AI CyberBench: 41.07 (#7)
GPT-6 on WebDev Arena (Fullstack): 1758.0 (#1)
GPT-6 Sol on AA CritPt: 30.86 (#8)
GPT-6 Sol on AA GDP.pdf: 28.0 (#8)
GPT-6 Sol on AA GDP.pdf - Construction - Criterion Pass Rate: 77.27 (#7)
GPT-6 Sol on AA GDP.pdf - Criterion Pass Rate: 77.45 (#18)
GPT-6 Sol on AA GDP.pdf - Engineering - Criterion Pass Rate: 78.48 (#11)
GPT-6 Sol on AA GDP.pdf - Finance/Investing - Criterion Pass Rate: 87.4 (#12)
GPT-6 Sol on AA GDP.pdf - HR - Criterion Pass Rate: 70.07 (#73)
GPT-6 Sol on AA GDP.pdf - Healthcare - Criterion Pass Rate: 84.45 (#32)
GPT-6 Sol on AA GDP.pdf - Insurance - Criterion Pass Rate: 69.65 (#74)
GPT-6 Sol on AA GDP.pdf - Legal - Criterion Pass Rate: 79.29 (#48)
GPT-6 Sol on AA GDP.pdf - Manufacturing/Supply Chains - Criterion Pass Rate: 76.28 (#19)
GPT-6 Sol on AA GDP.pdf - Real Estate - Criterion Pass Rate: 79.84 (#1)
GPT-6 Sol on AA GDP.pdf - STEM/Research - Criterion Pass Rate: 78.79 (#11)
GPT-6 Sol on AA GDPval: 1486.93 (#36)
GPT-6 Sol on AA Humanity's Last Exam: 47.91 (#25)
GPT-6 Sol on AA Long Context Reasoning: 83.67 (#15)
GPT-6 Sol on AA MMMU-Pro: 83.29 (#22)
GPT-6 Sol on AA Omniscience: 27.12 (#34)
GPT-6 Sol on AA SciCode: 57.64 (#21)
GPT-6 Sol on AA-Briefcase: 46.87 (#35)
GPT-6 Sol on AA-Briefcase - Analytical Quality Elo: 1652.11 (#31)
GPT-6 Sol on AA-Briefcase - DOCX - Rubric Pass Rate: 43.2 (#46)
GPT-6 Sol on AA-Briefcase - PDF - Rubric Pass Rate: 42.24 (#47)
GPT-6 Sol on AA-Briefcase - PPTX - Rubric Pass Rate: 45.31 (#30)
GPT-6 Sol on AA-Briefcase - Presentation Elo: 1473.3 (#23)
GPT-6 Sol on AA-Briefcase - XLSX - Rubric Pass Rate: 49.09 (#31)
GPT-6 Sol on AA-Omniscience Accuracy: 54.48 (#35)
GPT-6 Sol on AA-Omniscience Hallucination Rate: 50.74 (#104)
GPT-6 Sol on AA-Omniscience Index - Business: 22.8 (#26)
GPT-6 Sol on AA-Omniscience Index - Health: 11.3 (#30)
GPT-6 Sol on AA-Omniscience Index - Humanities & Social Sciences: 25.0 (#37)
GPT-6 Sol on AA-Omniscience Index - Law: 25.9 (#41)
GPT-6 Sol on AA-Omniscience Index - Science, Engineering & Mathematics: 16.4 (#45)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE): 66.0 (#32)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - C: 83.0 (#21)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - Dart: 56.0 (#30)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - Go: 70.0 (#30)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - HTML: 84.0 (#11)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - Java: 43.0 (#24)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - JavaScript: 76.36 (#22)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - Julia: 44.0 (#43)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - Kotlin: 68.0 (#30)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - PHP: 68.0 (#28)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - Python: 74.0 (#29)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - R: 44.0 (#37)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - Rust: 84.0 (#8)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - Swift: 60.0 (#38)
GPT-6 Sol on AA-Omniscience Index - Software Engineering (SWE) - TypeScript: 88.89 (#13)
GPT-6 Sol on AI Chess Leaderboard (Continuation): 1195.0 (#40)
GPT-6 Sol on AI Chess Leaderboard (Reasoning): 1072.0 (#65)
GPT-6 Sol on Artificial Analysis Intelligence Index: 47.53 (#18)
GPT-6 Sol on AutomationBench: 33.2 (#7)
GPT-6 Sol on AutomationBench-AA: 61.69 (#16)
GPT-6 Sol on AutomationBench-AA - Finance - Raw Objective Completion: 84.57 (#28)
GPT-6 Sol on AutomationBench-AA - HR - Raw Objective Completion: 75.71 (#9)
GPT-6 Sol on AutomationBench-AA - Marketing - Raw Objective Completion: 86.29 (#20)
GPT-6 Sol on AutomationBench-AA - Operations - Raw Objective Completion: 96.51 (#4)
GPT-6 Sol on AutomationBench-AA - Sales - Raw Objective Completion: 90.07 (#1)
GPT-6 Sol on AutomationBench-AA - Strict: 32.57 (#13)
GPT-6 Sol on AutomationBench-AA - Support - Raw Objective Completion: 89.76 (#10)
GPT-6 Sol on BenchLM: 80.5 (#6)
GPT-6 Sol on Bug Hunt Bench: 32.0 (#13)
GPT-6 Sol on Bug Hunt Bench - LMS: 14.0 (#29)
GPT-6 Sol on Bug Hunt Bench - VS Code Extension: 18.0 (#8)
GPT-6 Sol on DataBench: 61.3 (#5)
GPT-6 Sol on Featherbench: 82.0 (#23)
GPT-6 Sol on Featherbench - Rubric Quality: 8.3 (#23)
GPT-6 Sol on LLM Stats (Agents' Last Exam): 56.4 (#2)
GPT-6 Sol on LLM Stats (DeepSWE 1.1): 68.8 (#12)
GPT-6 Sol on LLM Stats (FrontierCode 1.1): 49.3 (#5)
GPT-6 Sol on LLM Stats (OSWorld 2.0): 64.4 (#6)
GPT-6 Sol on LLM Stats Score: 49.39 (#21)
GPT-6 Sol on LM Market Cap LMC Score: 40.0 (#245)
GPT-6 Sol on LiveBench AMPS Hard: 98.0 (#39)
GPT-6 Sol on LiveBench Code Completion: 80.44 (#25)
GPT-6 Sol on LiveBench Code Generation: 83.1 (#9)
GPT-6 Sol on LiveBench Connections: 100.0 (#16)
GPT-6 Sol on LiveBench Consecutive Events: 89.55 (#13)
GPT-6 Sol on LiveBench Integrals With Game: 99.0 (#7)
GPT-6 Sol on LiveBench JavaScript: 63.64 (#43)
GPT-6 Sol on LiveBench Logic With Navigation: 74.0 (#30)
GPT-6 Sol on LiveBench Math Comp: 97.06 (#19)
GPT-6 Sol on LiveBench Olympiad: 91.36 (#22)
GPT-6 Sol on LiveBench Paraphrase: 70.08 (#25)
GPT-6 Sol on LiveBench Plot Unscrambling: 77.9 (#5)
GPT-6 Sol on LiveBench Python: 55.0 (#34)
GPT-6 Sol on LiveBench Simplify: 69.28 (#16)
GPT-6 Sol on LiveBench Spatial: 96.0 (#50)
GPT-6 Sol on LiveBench Story Generation: 66.93 (#48)
GPT-6 Sol on LiveBench Summarize: 68.0 (#42)
GPT-6 Sol on LiveBench Table Join: 54.02 (#5)
GPT-6 Sol on LiveBench Table Reformat: 100.0 (#22)
GPT-6 Sol on LiveBench Theory of Mind: 84.61 (#15)
GPT-6 Sol on LiveBench TypeScript: 40.0 (#34)
GPT-6 Sol on LiveBench Typos: 78.0 (#46)
GPT-6 Sol on LiveBench Zebra Puzzle: 100.0 (#15)
GPT-6 Sol on MLCR-AA: 16.11 (#38)
GPT-6 Sol on MLCR-AA - Accuracy: 96.67 (#2)
GPT-6 Sol on MLCR-AA - Completeness: 16.11 (#59)
GPT-6 Sol on MLCR-AA - Conciseness: 100.0 (#1)
GPT-6 Sol on Maze-Bench: 58.01 (#3)
GPT-6 Sol on NYT Connections Extended: 90.1 (#22)
GPT-6 Sol on Next.js Agent Evals (Codex) - Success Rate: 97.0 (#1)
GPT-6 Sol on Next.js Agent Evals (Codex) - Success Rate with AGENTS.md: 97.0 (#2)
GPT-6 Sol on ParseBench: 68.19 (#19)
GPT-6 Sol on RuneBench: 13372.0 (#3)
GPT-6 Sol on SvelteBench: 100.0 (#16)
GPT-6 Sol on TaxCalcBench: 26.0 (#14)
GPT-6 Sol on VoxelBench: 2440.0 (#2)

New #1 Leaders (135)

AA-Briefcase - Analytical Quality Elo: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (2206.92) beat Qwen 3.8 Max (0902) by 173.22
AA GDPval: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (1846.17) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 111.5
WebDev Arena (Fullstack): GPT-6 (Max) (1758.0) beat Claude Fable 5.1 (Max) by 67.0
AA-Briefcase - Presentation Elo: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (1709.91) beat GPT-5.6 Sol (Max) by 57.38
SEA-HELM (English) - AA-LCR: Muse-Glimmer-30B (Reasoning) (81.42) beat Qwen 3.5 27B by 22.06
SEA-HELM (Thai) - Thai Exam: Gemma 4 31B (IT) (Reasoning) (79.67) beat Qwen 3.5 27B by 19.84
SEA-HELM (English) - MathArena: Gemma 4 31B (IT) (Reasoning) (56.21) beat Gemma 4 31B (IT) by 18.45
LLM Stats (Humanity's Last Exam (no tools, text-only)): Claude Opus 5.5 (64.4) beat Hy3 by 17.4
SEA-HELM (Burmese) - Global MMLU Lite: Qwen3.8-Flash-Next (Reasoning) (81.25) beat Gemma 4 31B (IT) by 15.52
DataBench: Claude Opus 5.5 (Max) (70.5) beat GPT-6 (Low) by 15.5
SEA-HELM (Tamil) - Pragmatics: Qwen 3.6 35B A3B (Reasoning) (80.83) beat Qwen-SEA-LION-v4.5-27B-IT by 14.77
LLM Stats (Program Bench): Claude Opus 5.5 (91.2) beat Kimi K3 by 13.4
SEA-HELM (Indonesian) - Syntax (LLM Judge): Qwen3.8-Flash-Next (Reasoning) (88.17) beat Gemma 4 31B (IT) by 11.51
KernelBench Hub - Mega: Claude Opus 5.5 (35.46) beat GPT-6 Pro by 10.66
SEA-HELM (Burmese) - Translations: Llama-SEA-LION-v3.5-8B-R (Reasoning) (99.13) beat Gemma 4 31B (IT) by 10.64
SEA-HELM (Filipino) - Global MMLU Lite: Qwen3.8-Flash-Next (Reasoning) (86.16) beat Gemma 4 31B (IT) by 10.42
SEA-HELM (Tamil) - Translations: Llama-SEA-LION-v3.5-8B-R (Reasoning) (99.63) beat Gemma 4 31B (IT) by 9.94
SEA-HELM (Malay) - Global MMLU Lite: Qwen3.8-Flash-Next (Reasoning) (86.96) beat Qwen 3.5 27B by 9.62
SEA-HELM (Vietnamese) - Global MMLU Lite: Qwen3.8-Flash-Next (Reasoning) (85.71) beat Qwen 3.5 122B A10B by 9.45
SEA-HELM (Indonesian) - Syntax: Qwen3.8-Flash-Next (Reasoning) (76.39) beat Gemma 4 31B (IT) by 9.06
SEA-HELM (Indonesian) - Global MMLU Lite: Qwen3.8-Flash-Next (Reasoning) (87.46) beat Qwen 3.5 27B by 8.55
SEA-HELM (Filipino) - Translations: Llama-SEA-LION-v3.5-8B-R (Reasoning) (99.79) beat Gemma 3 27B (IT) by 8.46
SuperCLUE-VLM - Overall: GPT-6 (83.9) beat Gemini 3.1 Pro (Preview) by 8.09
LLM Stats (Humanity's Last Exam (with tools, text-only)): Claude Opus 5.5 (67.7) beat DeepSeek V4 Pro (0813) by 7.7
SEA-HELM (Tamil) - Linguistic Diagnostics: Qwen 3.6 35B A3B (Reasoning) (85.18) beat Qwen-SEA-LION-v4.5-27B-IT by 7.66
SEA-HELM (Malay) - Translations: Llama-SEA-LION-v3.5-8B-R (Reasoning) (99.83) beat Gemma 4 31B (IT) by 7.15
SEA-HELM (Burmese) - Causal Reasoning: Qwen3.8-Flash-Next (Reasoning) (83.9) beat Gemma 4 31B (IT) by 7.0
Next.js Agent Evals (Codex) - Success Rate: GPT-6 Sol (High) (97.0) beat GPT-6 (High) by 7.0
SEA-HELM (Burmese) - SEA-IFEval: Qwen 3.5 35B A3B (Reasoning) (86.01) beat Gemma 4 31B (IT) by 6.64
LLM Stats (SWE-bench Multilingual): Claude Opus 5.5 (93.9) beat Claude Mythos Preview by 6.6
SEA-HELM (Tamil) - SEA-IFEval: Qwen 3.5 27B (Reasoning) (95.7) beat Gemma 4 31B (IT) by 6.54
SEA-HELM (Indonesian) - Metaphor Understanding: Muse-Glimmer-30B (Reasoning) (89.85) beat Gemma 4 31B (IT) by 6.52
SEA-HELM (English) - LiveCodeBench v6: Gemma 4 31B (IT) (Reasoning) (80.09) beat Gemma 4 31B (IT) by 6.32
SEA-HELM (Vietnamese) - Translations: Llama-SEA-LION-v3.5-8B-R (Reasoning) (99.86) beat Gemma 4 31B (IT) by 6.15
SEA-HELM (Indonesian) - Translations: Llama-SEA-LION-v3.5-8B-R (Reasoning) (99.86) beat Gemma 4 31B (IT) by 6.11
CursorBench 4.0: Claude Opus 5.5 (57.8) beat Claude Fable 5.1 (Max) by 6.0
AA-Omniscience Index - Software Engineering (SWE) - Rust: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (92.0) beat GPT-6 (Max) by 6.0
SEA-HELM (Thai) - Translations: Llama-SEA-LION-v3.5-8B-R (Reasoning) (99.83) beat Gemma 4 31B (IT) by 5.95
SEA-HELM (English) - SuperGPQA: Qwen 3.5 122B A10B (Reasoning) (63.33) beat Qwen 3.5 122B A10B by 5.87
LLM Stats (Terminal-Bench 4.0): Claude Opus 5.5 (66.4) beat Claude Mythos 5.1 by 5.5
SEA-HELM (Indonesian) - Linguistic Diagnostics: Qwen3.8-Flash-Next (Reasoning) (79.7) beat Gemma 4 31B (IT) by 5.34
SEA-HELM (Tamil) - SEA-MT-Bench (LLM Judge): Qwen3.8-Flash-Next (Reasoning) (85.23) beat Qwen 3.5 27B by 5.15
AA-Omniscience Index - Software Engineering (SWE) - Java: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (80.0) beat GPT-6 (Max) by 5.0
Mercor APEX: Claude Opus 5.5 (73.5) beat Claude Fable 5.1 by 4.9
SEA-HELM (Burmese) - NLU: Muse-Glimmer-30B (Reasoning) (67.54) beat Gemma 4 31B (IT) by 4.82
SEA-HELM (Filipino) - Belebele QA: Gemma 4 31B (IT) (Reasoning) (88.5) beat Mistral Medium 3.5 by 4.81
SEA-HELM (English) - MuSR: Qwen3.8-Flash-Next (Reasoning) (84.16) beat Gemma 4 31B (IT) by 4.63
Vals AI Terminal-Bench 4.0: Claude Opus 5.5 (61.62) beat GPT-6 by 4.55
Artificial Analysis Intelligence Index: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (57.62) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 4.27
AA-Omniscience Index - Science, Engineering & Mathematics: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (37.8) beat Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) by 4.2
SEA-HELM (Thai) - SEA-IFEval: Gemma 4 31B (IT) (Reasoning) (97.36) beat Gemma 4 31B (IT) by 4.11
AA-Omniscience Index - Software Engineering (SWE) - C: Claude Opus 5.5 (Adaptive Reasoning, Low Effort, Default Fallback) (96.0) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 4.0
AA-Omniscience Index - Software Engineering (SWE) - Kotlin: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (96.0) beat GPT-5.6 Sol (xHigh) by 4.0
AA-Omniscience Index - Software Engineering (SWE) - PHP: Claude Opus 5.5 (Adaptive Reasoning, High Effort, Default Fallback) (96.0) beat GPT-5.6 Sol (Max) by 4.0
SEA-HELM (English) - Knowledge: Qwen3.8-Flash-Next (Reasoning) (77.84) beat Qwen 3.5 122B A10B by 3.96
LLM Stats (OSWorld 2.0): Claude Opus 5.5 (81.8) beat Claude Fable 5.1 by 3.9
AA SciCode: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (66.9) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 3.82
SEA-HELM (Tamil) - NLG: Llama-SEA-LION-v3.5-8B-R (Reasoning) (55.2) beat Gemma 4 31B (IT) by 3.78
SEA-HELM (Burmese) - Belebele QA: Muse-Glimmer-30B (Reasoning) (85.88) beat Gemma 4 31B (IT) by 3.61
SEA-HELM (Tamil) - Syntax: Gemma 4 31B (IT) (Reasoning) (94.93) beat Gemma 4 31B (IT) by 3.48
LiveBench TypeScript: Claude Opus 5.5 (Max) (63.33) beat Claude Fable 5.1 (Max) by 3.33
SEA-HELM (English) - IFEval: Gemma 4 31B (IT) (Reasoning) (94.92) beat Gemma 4 31B (IT) by 3.23
SEA-HELM (Filipino) - Kalahi (LLM Judge): Gemma 4 31B (IT) (Reasoning) (81.21) beat Gemma 4 31B (IT) by 3.02
AA Omniscience: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (46.42) beat GPT-6 (High) by 2.69
AA-Omniscience Index - Software Engineering (SWE): Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (88.8) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 2.6
SEA-HELM (Thai) - SEA-MT-Bench (LLM Judge): Qwen3.8-Flash-Next (Reasoning) (86.55) beat Qwen 3.5 27B by 2.57
AutomationBench-AA - Sales - Raw Objective Completion: GPT-6 Sol (xHigh) (90.07) beat Grok 4.6 (xHigh) by 2.45
SEA-HELM (Vietnamese) - Causal Reasoning: Qwen 3.6 27B (Reasoning) (95.24) beat Qwen-SEA-LION-v4.5-27B-IT by 2.44
SEA-HELM (Burmese) - NLR: Gemma 4 31B (IT) (Reasoning) (70.4) beat Gemma 4 31B (IT) by 2.42
SEA-HELM (Indonesian) - SEA-Safeguard: Muse-Glimmer-30B (Reasoning) (78.42) beat Gemma 3 12B (IT) by 2.39
SEA-HELM (Filipino) - Causal Reasoning: Gemma 4 31B (IT) (Reasoning) (97.06) beat Gemma 4 31B (IT) by 2.39
AA Humanity's Last Exam: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (61.35) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 2.22
SEA-HELM (Indonesian) - SEA-MT-Bench (LLM Judge): Qwen 3.5 122B A10B (Reasoning) (88.03) beat Qwen 3.6 27B by 2.11
SEA-HELM (Tamil) - Causal Reasoning: Qwen 3.6 27B (Reasoning) (95.87) beat Qwen-SEA-LION-v4.5-27B-IT by 2.08
AA GDP.pdf - Finance/Investing - Criterion Pass Rate: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (91.25) beat GPT-6 (Max) by 2.06
SEA-HELM (Indonesian) - NLU: Muse-Glimmer-30B (Reasoning) (83.99) beat Gemma 4 31B (IT) by 2.01
SEA-HELM (Filipino) - SEA-MT-Bench (LLM Judge): Qwen3.8-Flash-Next (Reasoning) (84.81) beat Gemma 4 31B (IT) by 1.98
SEA-HELM (Indonesian) - Causal Reasoning: Gemma 4 31B (IT) (Reasoning) (97.54) beat Gemma 4 31B (IT) by 1.91
SEA-HELM (Filipino) - SEA-IFEval: Gemma 4 31B (IT) (Reasoning) (94.02) beat Gemma 4 31B (IT) by 1.89
SEA-HELM (Indonesian) - NLG: Llama-SEA-LION-v3.5-8B-R (Reasoning) (57.98) beat Gemma 4 31B (IT) by 1.88
SEA-HELM (Thai) - NLG: Llama-SEA-LION-v3.5-8B-R (Reasoning) (61.29) beat Gemma 4 31B (IT) by 1.84
SEA-HELM (Filipino) - Kalahi: Gemma 4 31B (IT) (Reasoning) (92.37) beat Gemma 4 31B (IT) by 1.83
Vals AI Vibe Code Bench 1-100: Claude Opus 5.5 (30.36) beat Claude Opus 5 by 1.83
SEA-HELM (Filipino) - Natural Language Inference: Qwen-SEA-LION-v4.5-27B-IT (Reasoning) (69.98) beat Gemma 4 31B (IT) by 1.78
SEA-HELM (Vietnamese) - SEA-Safeguard: Gemma 4 31B (IT) (Reasoning) (85.68) beat Qwen-SEA-LION-v4.5-27B-IT by 1.75
SEA-HELM (Burmese) - Natural Language Inference: Qwen 3.6 27B (Reasoning) (60.72) beat Gemma 4 31B (IT) by 1.65
AA-Omniscience Index - Humanities & Social Sciences: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (44.7) beat GPT-6 (Max) by 1.6
SEA-HELM (Filipino) - NLU: Gemma 4 31B (IT) (Reasoning) (76.85) beat Gemma 4 31B (IT) by 1.59
AA GDP.pdf - STEM/Research - Criterion Pass Rate: Claude Opus 5.5 (Adaptive Reasoning, Low Effort, Default Fallback) (81.47) beat GPT-6 (Medium) by 1.58
SEA-HELM (Malay) - Belebele QA: Gemma 4 31B (IT) (Reasoning) (94.85) beat Gemma 4 31B (IT) by 1.58
Vals AI ProgramBench: Claude Opus 5.5 (Max) (86.99) beat GPT-6 (Max) by 1.57
SEA-HELM (Tamil) - NLR: Qwen 3.6 27B (Reasoning) (76.81) beat Gemma 4 31B (IT) by 1.51
AA GDP.pdf - Engineering - Criterion Pass Rate: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (84.35) beat GPT-6 (xHigh) by 1.49
SEA-HELM (Filipino) - NLR: Gemma 4 31B (IT) (Reasoning) (82.85) beat Gemma 4 31B (IT) by 1.41
SEA-HELM (Indonesian) - NLR: Gemma 4 31B (IT) (Reasoning) (90.35) beat Gemma 4 31B (IT) by 1.34
SEA-HELM (English) - MMLU Redux: Qwen3.8-Flash-Next (Reasoning) (92.44) beat Gemma 4 31B (IT) by 1.27
SEA-HELM (Vietnamese) - NLG: Llama-SEA-LION-v3.5-8B-R (Reasoning) (57.54) beat Llama-SEA-LION-v3-70B-IT by 1.27
SEA-HELM (Thai) - Causal Reasoning: Gemma 4 26B A4B (IT) (Reasoning) (98.06) beat Gemma 4 31B (IT) by 1.23
SEA-HELM (Indonesian) - Natural Language Inference: Qwen 3.6 27B (Reasoning) (83.57) beat Gemma 4 31B (IT) by 1.18
SEA-HELM (Indonesian) - SEA-IFEval: Gemma 4 31B (IT) (Reasoning) (97.75) beat Gemma 4 31B (IT) by 1.13
SEA-HELM (Malay) - SEA-MT-Bench (LLM Judge): Qwen 3.5 122B A10B (Reasoning) (86.35) beat Qwen 3.5 122B A10B by 0.96
SEA-HELM (Malay) - NLU: Gemma 4 31B (IT) (Reasoning) (74.15) beat Gemma 4 31B (IT) by 0.92
LLM Stats (FrontierCode 1.1): Claude Opus 5.5 (54.4) beat Claude Fable 5 by 0.9
SEA-HELM (Tamil) - Sentiment Analysis: Muse-Glimmer-30B (Reasoning) (98.19) beat Gemma 3 27B (IT) by 0.88
SEA-HELM (Vietnamese) - SEA-IFEval: Gemma 4 31B (IT) (Reasoning) (96.21) beat Gemma 4 31B (IT) by 0.87
SEA-HELM (Malay) - SEA-IFEval: Gemma 4 31B (IT) (Reasoning) (90.72) beat Gemma 4 31B (IT) by 0.86
Vals AI (Vals Index): Claude Opus 5.5 (69.69) beat Claude Fable 5.1 by 0.86
AA MMMU-Pro: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (87.69) beat GPT-6 (Max) by 0.81
SEA-HELM (Filipino) - NLG: Llama-SEA-LION-v3.5-8B-R (Reasoning) (59.48) beat Gemma 4 31B (IT) by 0.76
SEA-HELM (Thai) - NLU: Qwen 3.6 27B (Reasoning) (64.67) beat Qwen-SEA-LION-v4.5-27B-IT by 0.74
Harvey LAB-AA - Immigration: Claude Opus 5.5 (Adaptive Reasoning, Xhigh Effort, Default Fallback) (94.91) beat Muse Spark 1.3 (xHigh) by 0.73
AA GDP.pdf - Real Estate - Criterion Pass Rate: GPT-6 Sol (High) (79.84) beat GPT-6 (Medium) by 0.69
AA-Briefcase - XLSX - Rubric Pass Rate: Claude Opus 5.5 (Adaptive Reasoning, Xhigh Effort, Default Fallback) (61.14) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 0.69
AutomationBench-AA: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (69.54) beat DeepSeek V4.1 Flash (Reasoning, Max Effort) by 0.65
AA-Briefcase - DOCX - Rubric Pass Rate: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (60.36) beat Claude Opus 5 (Adaptive Reasoning, Xhigh Effort) by 0.6
AA GDP.pdf - Healthcare - Criterion Pass Rate: Claude Opus 5.5 (Adaptive Reasoning, High Effort, Default Fallback) (93.1) beat GPT-6 (Medium) by 0.59
Open Japanese LLM v2 - M-IFEval-Ja: llm-jp-4-kappa-32B-a3B-v0.1 (Thinking) (47.67) beat Llama 3.3 70B Instruct by 0.58
AA-Omniscience Index - Software Engineering (SWE) - Python: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (91.5) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 0.5
SEA-HELM (Burmese) - Toxicity Detection: Gemma 4 31B (IT) (Reasoning) (69.99) beat Gemma 4 31B (IT) by 0.47
SEA-HELM (Indonesian) - Question Answering: Qwen3.8-Flash-Next (Reasoning) (85.9) beat Qwen-SEA-LION-v4.5-27B-IT by 0.44
SEA-HELM (Vietnamese) - SEA-MT-Bench (LLM Judge): Qwen 3.5 35B A3B (Reasoning) (87.3) beat Qwen 3.6 27B by 0.43
LLM Stats (HealthBench): Claude Opus 5.5 (60.6) beat Qwen 3.8 Max by 0.4
Vals AI Terminal-Bench 2.1: Claude Opus 5.5 (87.64) beat GPT-6 by 0.37
SEA-HELM (Vietnamese) - NLR: Qwen 3.6 27B (Reasoning) (79.22) beat Gemma 4 31B (IT) by 0.33
LiveBench Olympiad: Claude Opus 5.5 (Max) (93.25) beat Claude Fable 5.1 (Max) by 0.28
SEA-HELM (Malay) - Sentiment Analysis: Gemma 4 31B (IT) (Reasoning) (53.45) beat Gemma 4 31B (IT) by 0.27
SEA-HELM (Vietnamese) - NLU: Qwen3.8-Flash-Next (Reasoning) (72.0) beat Qwen 3 VL 8B Instruct by 0.26
SEA-HELM (Vietnamese) - Safety: Gemma 4 31B (IT) (Reasoning) (72.42) beat Gemma 4 31B (IT) by 0.24
AA GDP.pdf - Criterion Pass Rate: Claude Opus 5.5 (Adaptive Reasoning, High Effort, Default Fallback) (82.25) beat GPT-6 (High) by 0.2
AutomationBench-AA - HR - Raw Objective Completion: Claude Opus 5.5 (Adaptive Reasoning, Max Effort, Default Fallback) (77.27) beat GPT-6 (High) by 0.19
Vals AI MedScribe: Claude Opus 5.5 (Max) (91.43) beat Claude Fable 5.1 (Max) by 0.14
LLM Stats Score: Claude Opus 5.5 (59.7) beat GPT-6 by 0.12
SEA-HELM (Burmese) - NLG: Gemma 4 31B (IT) (Reasoning) (57.95) beat Gemma 4 31B (IT) by 0.05
AA GDP.pdf - HR - Criterion Pass Rate: Claude Opus 5.5 (Adaptive Reasoning, Low Effort, Default Fallback) (90.16) beat MiMo-V2.6-Pro by 0.03
Vals AI RSI Index: Claude Opus 5.5 (Max) (0.371) beat Claude Fable 5.1 (Max) by 0.02

The Aggregate Digest — 2026-09-18

2026-09-18T14:35:56.586908+00:00

Daily

New Benchmarks (2)

Vals AI Terminal-Bench 4.0 (Accuracy (%)): leader GPT-6 (57.07), 28 models
Terminal-Bench 4.0 from Vals AI: long-horizon terminal work across seven domains (software engineering, science, machine learning, operations, hardware design, security and media production), scored pass@1 on the share of tasks completed.
Vals AI MysteryMechanism (Accuracy (%)): leader GPT-6 (53.15), 9 models
MysteryMechanism (Vals AI) — agents rediscover a hidden mathematical relationship by choosing experiments against a noisy oracle under a limited query budget; accuracy over the private task set.

Top-10 New Scores (61)

Claude Fable 5 on Chatbot Arena (Code): 1628.0 (#10)
Claude Fable 5 on Chatbot Arena (Document): 1496.0 (#5)
Claude Fable 5 on Chatbot Arena (Image-to-WebDev): 1623.0 (#6)
Claude Fable 5 on Chatbot Arena (Text - Business, Management & Financial Ops): 1502.0 (#3)
Claude Fable 5 on Chatbot Arena (Text - Chinese): 1553.0 (#5)
Claude Fable 5 on Chatbot Arena (Text - Coding): 1552.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - Creative Writing): 1504.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - English): 1513.0 (#2)
Claude Fable 5 on Chatbot Arena (Text - Entertainment, Sports & Media): 1496.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - Expert): 1548.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - French): 1523.0 (#3)
Claude Fable 5 on Chatbot Arena (Text - German): 1497.0 (#10)
Claude Fable 5 on Chatbot Arena (Text - Hard Prompts (English)): 1533.0 (#2)
Claude Fable 5 on Chatbot Arena (Text - Hard Prompts): 1532.0 (#2)
Claude Fable 5 on Chatbot Arena (Text - Instruction Following): 1511.0 (#2)
Claude Fable 5 on Chatbot Arena (Text - Japanese): 1523.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - Korean): 1500.0 (#2)
Claude Fable 5 on Chatbot Arena (Text - Legal & Government): 1512.0 (#2)
Claude Fable 5 on Chatbot Arena (Text - Life, Physical & Social Science): 1528.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - Longer Query): 1521.0 (#2)
Claude Fable 5 on Chatbot Arena (Text - Math): 1526.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - Mathematical): 1516.0 (#4)
Claude Fable 5 on Chatbot Arena (Text - Medicine & Healthcare): 1505.0 (#10)
Claude Fable 5 on Chatbot Arena (Text - Multi-Turn): 1518.0 (#2)
Claude Fable 5 on Chatbot Arena (Text - Non-English): 1496.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - Polish): 1505.0 (#3)
Claude Fable 5 on Chatbot Arena (Text - Russian): 1519.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - Software & IT Services): 1540.0 (#3)
Claude Fable 5 on Chatbot Arena (Text - Spanish): 1514.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - Writing, Literature & Language): 1511.0 (#1)
Claude Fable 5 on Chatbot Arena (Text): 1506.0 (#1)
Claude Fable 5 on Chatbot Arena (Vision - Chinese): 1379.0 (#1)
Claude Fable 5 on Chatbot Arena (Vision - Creative Writing): 1328.0 (#1)
Claude Fable 5 on Chatbot Arena (Vision - Diagram): 1355.0 (#1)
Claude Fable 5 on Chatbot Arena (Vision - English): 1305.0 (#1)
Claude Fable 5 on Chatbot Arena (Vision - Homework): 1343.0 (#2)
Claude Fable 5 on Chatbot Arena (Vision - Humor): 1332.0 (#1)
Claude Fable 5 on Chatbot Arena (Vision - OCR): 1325.0 (#1)
Claude Fable 5 on Chatbot Arena (Vision): 1310.0 (#1)
Claude Fable 5 on SEAL - Remote Labor Index (RLI): 15.8 (#3)
Claude Fable 5 on WebDev Arena (Brand & Marketing): 1625.0 (#14)
Claude Fable 5 on WebDev Arena (Consumer Product): 1567.0 (#17)
Claude Fable 5 on WebDev Arena (Content Creation Tools): 1605.0 (#10)
Claude Fable 5 on WebDev Arena (Data & Analytics): 1552.0 (#19)
Claude Fable 5 on WebDev Arena (Frontend): 1641.0 (#10)
Claude Fable 5 on WebDev Arena (Fullstack): 1616.0 (#12)
Claude Fable 5 on WebDev Arena (Gaming): 1701.0 (#10)
Claude Fable 5 on WebDev Arena (HTML): 1663.0 (#4)
Claude Fable 5 on WebDev Arena (React): 1633.0 (#11)
Claude Fable 5 on WebDev Arena (Reference-Based Design): 1652.0 (#9)
Claude Fable 5 on WebDev Arena (Simulations): 1677.0 (#9)
Claude Fable 5.1 on SEAL - MCP Atlas: 87.2 (#2)
Claude Fable 5.1 on SEAL - Remote Labor Index (RLI): 17.92 (#2)
GPT-6 on FutureSearch BTF-3: 0.1346 (#3)
GPT-6 on FutureSearch BTF-3 Binary: 0.1372 (#4)
GPT-6 on FutureSearch BTF-3 Numeric: 0.13 (#2)
GPT-6 on Kaggle Game Arena (Unified): 301.89 (#4)
GPT-6 on Kaggle Game Arena Chess: 1236.86 (#8)
GPT-6 on Kaggle Game Arena Four in a Row: 553.09 (#10)
GPT-6 on Kaggle Game Arena Poker (Heads Up): 11.31 (#5)
GPT-6 on SpeechMap Compliance: 68.2 (#158)

New #1 Leaders (3)

FrontierSWE V2: GPT-6 (65.5) beat Claude Fable 5.1 by 9.2
SEAL - Remote Labor Index (RLI): GPT-6 (20.83) beat Claude Fable 5 by 5.03
MLS-Bench Lite: Claude Fable 5.1 (50.3) beat Qwen 3.8 Max (0902) by 0.2

The Aggregate Digest — 2026-09-17

2026-09-17T07:58:40.133492+00:00

Daily

New Benchmarks (6)

MathArena - ARXIV_FALSE August (Accuracy (%)): leader GPT-6 (Max) (81.94), 6 models
MathArena BrokenArXiv (August): plausible but false proof statements altered from recent August arXiv papers, scoring whether a model refuses to prove the claim and explicitly identifies it as false rather than fabricating a proof.
MathArena - ARXIV August (Accuracy (%)): leader GPT-6 (Max) (88.6), 6 models
Research-level math problems extracted from recent August arXiv papers with verifiable answers, testing LLMs on authentic mathematical research rather than competition problems.
Benchouse Analytics Agent Benchmark (Accuracy (%)): leader Claude Opus 4.8 (Harness: cortex-agents) (83.67), 5 models
Benchouse accuracy over all 300 questions of a season: analytics agents query a deliberately messy simulated e-commerce warehouse through their own semantic layer, and answers are graded against a withheld deterministic truth ledger.
Benchouse Analytics Agent Benchmark - Descriptive (Accuracy (%)): leader Claude Opus 4.8 (Harness: cortex-agents) (98.94), 5 models
Benchouse accuracy on the 100 descriptive questions of a season, which ask what happened in the warehouse data and are answerable by straightforward aggregation.
Benchouse Analytics Agent Benchmark - Diagnostic (Accuracy (%)): leader Claude Opus 4.8 (Harness: cortex-agents) (81.0), 5 models
Benchouse accuracy on the 100 diagnostic questions of a season, which ask why a metric moved and require the agent to decompose it across dimensions and avoid the dataset's planted bad joins.
Benchouse Analytics Agent Benchmark - Prescriptive (Accuracy (%)): leader Claude Opus 4.8 (Harness: cortex-agents) (72.0), 5 models
Benchouse accuracy on the 100 prescriptive questions of a season, which ask what should be done next and require the agent to reason from the warehouse data to a defensible recommendation.

Top-10 New Scores (2)

GPT-6 on Kaggle Open Benchmarks - GPQA Diamond (Zero-Shot): 91.92 (#3)
GPT-6 on Kaggle SimpleQA Verified: 75.81 (#2)

New #1 Leaders (3)

Design Arena (Agents - Web Apps): Claude Fable 5.1 (1336.0) beat GPT-6 by 5.0
SecIT Bench (Pydantic AI): DeepSeek V4.1 Flash (82.9) beat GLM-5.3 by 1.46
OpenRouter GPQA Diamond: Fugu Ultra (94.6) beat Gemini 3.1 Pro (Preview) by 0.2

The Aggregate Digest — 2026-09-16

2026-09-16T07:36:48.556513+00:00

Daily

New Benchmarks (3)

GBA Eval - Gameplay Replays (Frame-by-frame SSIM on replayed gameplay inputs (0-1)): leader Claude Opus 5 (0.9289), 21 models
GBA Eval section score for Game Boy Advance emulators that models write in Rust/WebAssembly: recorded gameplay inputs are replayed and frames compared with the Mesen2 reference by SSIM, 0-1, higher is better. Carries 60% of the overall score; ungraded emulators are omitted.
GBA Eval - Procedural Tests (Hardware test ROM score: CPU, memory, timers, interrupts, DMA (0-1)): leader GPT-5.6 Sol (0.559), 21 models
GBA Eval section score from hardware test ROMs covering CPU, memory, timers, interrupts and DMA, graded against the Mesen2 reference emulator on a 0-1 scale, higher is better. Carries 20% of the overall GBA Eval score; ungraded emulators are omitted.
GBA Eval - Audio (PSG and FIFO audio score from log-mel spectrogram distance (0-1)): leader Claude Opus 4.8 (0.6906), 21 models
GBA Eval section score for emulated PSG and FIFO audio, graded by log-mel spectrogram distance from the Mesen2 reference on a 0-1 scale, higher is better. Carries 20% of the overall GBA Eval score; ungraded emulators are omitted.

New Models (1)

Median Expert — Aggregate Rating 2315
- OpenAI GPT-5.5 System Card - Tacit Knowledge and Troubleshooting: 80.0 (#2/2)

Top-10 New Scores (3)

Claude Opus 5 on SEAL - Fortress: 16.84 (#51)
GPT-6 on Chess Bench LLM: 1838.0 (#9)
GPT-6 on Kaggle Open Benchmarks - AIME 2025: 100.0 (#5)

New #1 Leaders (9)

AA-Briefcase - Analytical Quality Elo: Qwen 3.8 Max (0902) (1984.78) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 26.23
Design Arena (Video Editing): fig (1365.0) beat minimax-h3 by 10.0
HELM Capabilities - GPQA: Gemini 3 Pro (Preview) (80.27) beat O3 (2025-04-16) by 4.93
HELM Capabilities - MMLU-Pro: Gemini 3 Pro (Preview) (90.3) beat Claude Opus 4 (20250514) (Thinking) by 2.8
LLM Stats (BrowseComp): Atria Dawn Preview (92.5) beat GPT-6 by 1.0
LLM Stats (DeepSearchQA): Atria Dawn Preview (96.0) beat Kimi K3 by 1.0
HELM Capabilities - WildBench: Qwen3 235B A22B Instruct 2507 FP8 (86.63) beat Kimi K2 by 0.44
LLM Stats (Tau3 Banking): Atria Dawn Preview (41.2) beat Ling 3.0 Flash Fin by 0.2
HELM Capabilities - Omni-MATH: GPT-5 Mini (2025-08-07) (72.2) beat O4 Mini (2025-04-16) by 0.17