The Aggregate Digest

The Aggregate Digest — 2026-09-18

2026-09-18T14:35:56.586908+00:00

Daily

New Benchmarks (2)

Vals AI Terminal-Bench 4.0 (Accuracy (%)): leader GPT-6 (57.07), 28 models
Terminal-Bench 4.0 from Vals AI: long-horizon terminal work across seven domains (software engineering, science, machine learning, operations, hardware design, security and media production), scored pass@1 on the share of tasks completed.
Vals AI MysteryMechanism (Accuracy (%)): leader GPT-6 (53.15), 9 models
MysteryMechanism (Vals AI) — agents rediscover a hidden mathematical relationship by choosing experiments against a noisy oracle under a limited query budget; accuracy over the private task set.

Top-10 New Scores (61)

Claude Fable 5 on Chatbot Arena (Code): 1628.0 (#10)
Claude Fable 5 on Chatbot Arena (Document): 1496.0 (#5)
Claude Fable 5 on Chatbot Arena (Image-to-WebDev): 1623.0 (#6)
Claude Fable 5 on Chatbot Arena (Text - Business, Management & Financial Ops): 1502.0 (#3)
Claude Fable 5 on Chatbot Arena (Text - Chinese): 1553.0 (#5)
Claude Fable 5 on Chatbot Arena (Text - Coding): 1552.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - Creative Writing): 1504.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - English): 1513.0 (#2)
Claude Fable 5 on Chatbot Arena (Text - Entertainment, Sports & Media): 1496.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - Expert): 1548.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - French): 1523.0 (#3)
Claude Fable 5 on Chatbot Arena (Text - German): 1497.0 (#10)
Claude Fable 5 on Chatbot Arena (Text - Hard Prompts (English)): 1533.0 (#2)
Claude Fable 5 on Chatbot Arena (Text - Hard Prompts): 1532.0 (#2)
Claude Fable 5 on Chatbot Arena (Text - Instruction Following): 1511.0 (#2)
Claude Fable 5 on Chatbot Arena (Text - Japanese): 1523.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - Korean): 1500.0 (#2)
Claude Fable 5 on Chatbot Arena (Text - Legal & Government): 1512.0 (#2)
Claude Fable 5 on Chatbot Arena (Text - Life, Physical & Social Science): 1528.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - Longer Query): 1521.0 (#2)
Claude Fable 5 on Chatbot Arena (Text - Math): 1526.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - Mathematical): 1516.0 (#4)
Claude Fable 5 on Chatbot Arena (Text - Medicine & Healthcare): 1505.0 (#10)
Claude Fable 5 on Chatbot Arena (Text - Multi-Turn): 1518.0 (#2)
Claude Fable 5 on Chatbot Arena (Text - Non-English): 1496.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - Polish): 1505.0 (#3)
Claude Fable 5 on Chatbot Arena (Text - Russian): 1519.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - Software & IT Services): 1540.0 (#3)
Claude Fable 5 on Chatbot Arena (Text - Spanish): 1514.0 (#1)
Claude Fable 5 on Chatbot Arena (Text - Writing, Literature & Language): 1511.0 (#1)
Claude Fable 5 on Chatbot Arena (Text): 1506.0 (#1)
Claude Fable 5 on Chatbot Arena (Vision - Chinese): 1379.0 (#1)
Claude Fable 5 on Chatbot Arena (Vision - Creative Writing): 1328.0 (#1)
Claude Fable 5 on Chatbot Arena (Vision - Diagram): 1355.0 (#1)
Claude Fable 5 on Chatbot Arena (Vision - English): 1305.0 (#1)
Claude Fable 5 on Chatbot Arena (Vision - Homework): 1343.0 (#2)
Claude Fable 5 on Chatbot Arena (Vision - Humor): 1332.0 (#1)
Claude Fable 5 on Chatbot Arena (Vision - OCR): 1325.0 (#1)
Claude Fable 5 on Chatbot Arena (Vision): 1310.0 (#1)
Claude Fable 5 on SEAL - Remote Labor Index (RLI): 15.8 (#3)
Claude Fable 5 on WebDev Arena (Brand & Marketing): 1625.0 (#14)
Claude Fable 5 on WebDev Arena (Consumer Product): 1567.0 (#17)
Claude Fable 5 on WebDev Arena (Content Creation Tools): 1605.0 (#10)
Claude Fable 5 on WebDev Arena (Data & Analytics): 1552.0 (#19)
Claude Fable 5 on WebDev Arena (Frontend): 1641.0 (#10)
Claude Fable 5 on WebDev Arena (Fullstack): 1616.0 (#12)
Claude Fable 5 on WebDev Arena (Gaming): 1701.0 (#10)
Claude Fable 5 on WebDev Arena (HTML): 1663.0 (#4)
Claude Fable 5 on WebDev Arena (React): 1633.0 (#11)
Claude Fable 5 on WebDev Arena (Reference-Based Design): 1652.0 (#9)
Claude Fable 5 on WebDev Arena (Simulations): 1677.0 (#9)
Claude Fable 5.1 on SEAL - MCP Atlas: 87.2 (#2)
Claude Fable 5.1 on SEAL - Remote Labor Index (RLI): 17.92 (#2)
GPT-6 on FutureSearch BTF-3: 0.1346 (#3)
GPT-6 on FutureSearch BTF-3 Binary: 0.1372 (#4)
GPT-6 on FutureSearch BTF-3 Numeric: 0.13 (#2)
GPT-6 on Kaggle Game Arena (Unified): 301.89 (#4)
GPT-6 on Kaggle Game Arena Chess: 1236.86 (#8)
GPT-6 on Kaggle Game Arena Four in a Row: 553.09 (#10)
GPT-6 on Kaggle Game Arena Poker (Heads Up): 11.31 (#5)
GPT-6 on SpeechMap Compliance: 68.2 (#158)

New #1 Leaders (3)

FrontierSWE V2: GPT-6 (65.5) beat Claude Fable 5.1 by 9.2
SEAL - Remote Labor Index (RLI): GPT-6 (20.83) beat Claude Fable 5 by 5.03
MLS-Bench Lite: Claude Fable 5.1 (50.3) beat Qwen 3.8 Max (0902) by 0.2

The Aggregate Digest — 2026-09-17

2026-09-17T07:58:40.133492+00:00

Daily

New Benchmarks (6)

MathArena - ARXIV_FALSE August (Accuracy (%)): leader GPT-6 (Max) (81.94), 6 models
MathArena BrokenArXiv (August): plausible but false proof statements altered from recent August arXiv papers, scoring whether a model refuses to prove the claim and explicitly identifies it as false rather than fabricating a proof.
MathArena - ARXIV August (Accuracy (%)): leader GPT-6 (Max) (88.6), 6 models
Research-level math problems extracted from recent August arXiv papers with verifiable answers, testing LLMs on authentic mathematical research rather than competition problems.
Benchouse Analytics Agent Benchmark (Accuracy (%)): leader Claude Opus 4.8 (Harness: cortex-agents) (83.67), 5 models
Benchouse accuracy over all 300 questions of a season: analytics agents query a deliberately messy simulated e-commerce warehouse through their own semantic layer, and answers are graded against a withheld deterministic truth ledger.
Benchouse Analytics Agent Benchmark - Descriptive (Accuracy (%)): leader Claude Opus 4.8 (Harness: cortex-agents) (98.94), 5 models
Benchouse accuracy on the 100 descriptive questions of a season, which ask what happened in the warehouse data and are answerable by straightforward aggregation.
Benchouse Analytics Agent Benchmark - Diagnostic (Accuracy (%)): leader Claude Opus 4.8 (Harness: cortex-agents) (81.0), 5 models
Benchouse accuracy on the 100 diagnostic questions of a season, which ask why a metric moved and require the agent to decompose it across dimensions and avoid the dataset's planted bad joins.
Benchouse Analytics Agent Benchmark - Prescriptive (Accuracy (%)): leader Claude Opus 4.8 (Harness: cortex-agents) (72.0), 5 models
Benchouse accuracy on the 100 prescriptive questions of a season, which ask what should be done next and require the agent to reason from the warehouse data to a defensible recommendation.

Top-10 New Scores (2)

GPT-6 on Kaggle Open Benchmarks - GPQA Diamond (Zero-Shot): 91.92 (#3)
GPT-6 on Kaggle SimpleQA Verified: 75.81 (#2)

New #1 Leaders (3)

Design Arena (Agents - Web Apps): Claude Fable 5.1 (1336.0) beat GPT-6 by 5.0
SecIT Bench (Pydantic AI): DeepSeek V4.1 Flash (82.9) beat GLM-5.3 by 1.46
OpenRouter GPQA Diamond: Fugu Ultra (94.6) beat Gemini 3.1 Pro (Preview) by 0.2

The Aggregate Digest — 2026-09-16

2026-09-16T07:36:48.556513+00:00

Daily

New Benchmarks (3)

GBA Eval - Gameplay Replays (Frame-by-frame SSIM on replayed gameplay inputs (0-1)): leader Claude Opus 5 (0.9289), 21 models
GBA Eval section score for Game Boy Advance emulators that models write in Rust/WebAssembly: recorded gameplay inputs are replayed and frames compared with the Mesen2 reference by SSIM, 0-1, higher is better. Carries 60% of the overall score; ungraded emulators are omitted.
GBA Eval - Procedural Tests (Hardware test ROM score: CPU, memory, timers, interrupts, DMA (0-1)): leader GPT-5.6 Sol (0.559), 21 models
GBA Eval section score from hardware test ROMs covering CPU, memory, timers, interrupts and DMA, graded against the Mesen2 reference emulator on a 0-1 scale, higher is better. Carries 20% of the overall GBA Eval score; ungraded emulators are omitted.
GBA Eval - Audio (PSG and FIFO audio score from log-mel spectrogram distance (0-1)): leader Claude Opus 4.8 (0.6906), 21 models
GBA Eval section score for emulated PSG and FIFO audio, graded by log-mel spectrogram distance from the Mesen2 reference on a 0-1 scale, higher is better. Carries 20% of the overall GBA Eval score; ungraded emulators are omitted.

New Models (1)

Median Expert — ELO 2315
- OpenAI GPT-5.5 System Card - Tacit Knowledge and Troubleshooting: 80.0 (#2/2)

Top-10 New Scores (3)

Claude Opus 5 on SEAL - Fortress: 16.84 (#51)
GPT-6 on Chess Bench LLM: 1838.0 (#9)
GPT-6 on Kaggle Open Benchmarks - AIME 2025: 100.0 (#5)

New #1 Leaders (9)

AA-Briefcase - Analytical Quality Elo: Qwen 3.8 Max (0902) (1984.78) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 26.23
Design Arena (Video Editing): fig (1365.0) beat minimax-h3 by 10.0
HELM Capabilities - GPQA: Gemini 3 Pro (Preview) (80.27) beat O3 (2025-04-16) by 4.93
HELM Capabilities - MMLU-Pro: Gemini 3 Pro (Preview) (90.3) beat Claude Opus 4 (20250514) (Thinking) by 2.8
LLM Stats (BrowseComp): Atria Dawn Preview (92.5) beat GPT-6 by 1.0
LLM Stats (DeepSearchQA): Atria Dawn Preview (96.0) beat Kimi K3 by 1.0
HELM Capabilities - WildBench: Qwen3 235B A22B Instruct 2507 FP8 (86.63) beat Kimi K2 by 0.44
LLM Stats (Tau3 Banking): Atria Dawn Preview (41.2) beat Ling 3.0 Flash Fin by 0.2
HELM Capabilities - Omni-MATH: GPT-5 Mini (2025-08-07) (72.2) beat O4 Mini (2025-04-16) by 0.17

The Aggregate Digest — 2026-09-15

2026-09-15T10:58:35.543446+00:00

Frontier

Best available model: GPT-5.6 Pro Sol enters at #8 (2026 ELO) on 55 benchmarks
Best available model: Qwen 3.8 Max (0902) enters at #9 (2024 ELO) on 36 benchmarks

Daily

New Benchmarks (454)

Chatbot Arena (Image-to-WebDev) (Arena Score): leader GPT-6 (Max) (1733.0), 50 models
arena.ai Code Arena Image-to-WebDev leaderboard: models build a working website from an input image such as a screenshot or design mockup, and users vote between two anonymous results. A separate arena with its own vote pool rather than a WebDev category slice; Bradley-Terry arena score, higher is better. Coding-harness runs such as Codex-harness entries are ranked separately.
ProphetArena - Agentic (1 - Brier Score): leader GPT-6 (0.9561), 5 models
ProphetArena's Agentic scope: models run their own web search in an agentic harness before forecasting live Kalshi prediction-market events, instead of receiving fixed research context. Score is 1 − Brier over the events each predictor forecast (0–1, higher is better); only predictors on the public leaderboard with at least 50 resolved events are ranked.
Chatbot Arena (Text - Hard Prompts (English)) (Arena Score): leader Claude Opus 4.6 (High) (1536.0), 400 models
arena.ai’s text leaderboard restricted to battles whose prompts are in English and are flagged hard (specific, complex and demanding domain knowledge). It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Non-English) (Arena Score): leader Claude Fable 5 (1496.0), 402 models
arena.ai’s text leaderboard restricted to battles whose prompts are in any language other than English. It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Polish) (Arena Score): leader Gemini 3.5 Flash (High) (1510.0), 222 models
arena.ai’s text leaderboard restricted to battles whose prompts are in Polish. It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Expert) (Arena Score): leader Claude Fable 5 (1548.0), 352 models
arena.ai’s text leaderboard restricted to battles whose prompts arena.ai tags as expert-level: the small share of hard prompts that need deep professional or academic expertise. It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Software & IT Services) (Arena Score): leader Claude Opus 4.7 (High) (1541.0), 402 models
arena.ai’s text leaderboard restricted to battles whose prompts arena.ai’s occupational classifier assigns to software and IT services work. Same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Writing, Literature & Language) (Arena Score): leader Claude Fable 5 (1511.0), 401 models
arena.ai’s text leaderboard restricted to battles whose prompts arena.ai’s occupational classifier assigns to writing, literature and language work. Same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Life, Physical & Social Science) (Arena Score): leader Claude Fable 5 (1528.0), 400 models
arena.ai’s text leaderboard restricted to battles whose prompts arena.ai’s occupational classifier assigns to life, physical and social science work. Same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Entertainment, Sports & Media) (Arena Score): leader Claude Fable 5 (1496.0), 400 models
arena.ai’s text leaderboard restricted to battles whose prompts arena.ai’s occupational classifier assigns to entertainment, sports and media work. Same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Business, Management & Financial Ops) (Arena Score): leader Muse Spark 1.2 (xHigh) (1517.0), 395 models
arena.ai’s text leaderboard restricted to battles whose prompts arena.ai’s occupational classifier assigns to business, management and financial operations work. Same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Mathematical) (Arena Score): leader Claude Opus 5 (High) (1535.0), 379 models
arena.ai’s text leaderboard restricted to battles whose prompts arena.ai’s occupational classifier assigns to mathematical work. Same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Legal & Government) (Arena Score): leader Muse Spark 1.2 (xHigh) (1541.0), 375 models
arena.ai’s text leaderboard restricted to battles whose prompts arena.ai’s occupational classifier assigns to legal and government work. Same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Medicine & Healthcare) (Arena Score): leader Muse Spark 1.2 (xHigh) (1530.0), 371 models
arena.ai’s text leaderboard restricted to battles whose prompts arena.ai’s occupational classifier assigns to medicine and healthcare work. Same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Vision - Homework) (Arena Score): leader GLM-5.3 Flash (1344.0), 102 models
arena.ai’s vision leaderboard restricted to battles that ask for help with homework-style questions about an image, such as worked problems, diagrams or worksheets. Same crowd-vote Arena Score as the headline vision board, refitted over that category’s votes alone.
WebDev Arena (Brand & Marketing) (Arena Score): leader GPT-6 (Max) (1747.0), 127 models
arena.ai’s WebDev Arena restricted to brand and marketing site builds: two models answer the same web-app prompt, the rendered results are shown side by side, and the votes are fitted to an Arena Score over that category alone.
WebDev Arena (Data & Analytics) (Arena Score): leader GPT-6 (Max) (1692.0), 127 models
arena.ai’s WebDev Arena restricted to data and analytics app builds: two models answer the same web-app prompt, the rendered results are shown side by side, and the votes are fitted to an Arena Score over that category alone.
WebDev Arena (Consumer Product) (Arena Score): leader GPT-6 (Max) (1743.0), 127 models
arena.ai’s WebDev Arena restricted to consumer product app builds: two models answer the same web-app prompt, the rendered results are shown side by side, and the votes are fitted to an Arena Score over that category alone.
WebDev Arena (Gaming) (Arena Score): leader GPT-6 (Max) (1880.0), 127 models
arena.ai’s WebDev Arena restricted to browser game builds: two models answer the same web-app prompt, the rendered results are shown side by side, and the votes are fitted to an Arena Score over that category alone.
WebDev Arena (Simulations) (Arena Score): leader Claude Fable 5.1 (Max) (1867.0), 126 models
arena.ai’s WebDev Arena restricted to simulation and interactive visualisation builds: two models answer the same web-app prompt, the rendered results are shown side by side, and the votes are fitted to an Arena Score over that category alone.
WebDev Arena (Content Creation Tools) (Arena Score): leader GPT-6 (Max) (1746.0), 121 models
arena.ai’s WebDev Arena restricted to content creation tool builds: two models answer the same web-app prompt, the rendered results are shown side by side, and the votes are fitted to an Arena Score over that category alone.
LLM Stats (SWE-bench Multilingual) (Score (%)): leader Claude Mythos Preview (87.3), 45 models
LLM Stats compilation of vendor-reported SWE-bench Multilingual results: 300 GitHub issues across 9 programming languages (C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby, Rust), resolved when the model's patch passes the repository tests. Percent resolved, higher is better. Harnesses and attempts differ by vendor, unlike the swebench.com mini-SWE-agent board.
Drone-Bench (Progress towards baseline (%)): leader GPT-6 (95.1), 18 models
Andon Labs Drone-Bench: coding agents build the software stack for an autonomous drone demo across five tasks (reconstruct, localize, navigate, detect, follow), each scored against the lab's human-written baseline. Progress towards baseline: each task's average run score is capped at its baseline and divided by it, then averaged over tasks (0–100%, higher is better).
Drone-Bench - Reconstruct (Reconstruct score (%)): leader GPT-6 (67.5), 18 models
Drone-Bench Reconstruct: build a 3D model of an office from videos, recover each frame's pose, and supply a slicing function that yields a 2D obstacle map. Score is the average over runs of each run's best of up to ten submissions, as a percentage (higher is better); the human baseline scores 82.2 and no model has passed it.
Drone-Bench - Localize (Localize score (%)): leader Claude Fable 5.1 (96.1), 18 models
Drone-Bench Localize: build a function that estimates the drone's pose from a camera frame by matching against reference video frames with known poses, scored on accuracy and speed (top-80% mean over queries). Score is the average over runs of each run's best of up to ten submissions, as a percentage; the human baseline is 84.0.
Drone-Bench - Navigate (Navigate score (%)): leader GPT-6 (98.2), 18 models
Drone-Bench Navigate: write a control loop that plans a collision-free path on an obstacle map and flies to a target room while calling a delayed, intermittently failing localizer. Scored on reaching the target, path optimality and speed; average over runs of each run's best submission, as a percentage (human baseline 92.0).
Drone-Bench - Detect (Detect score (%)): leader Claude Fable 5 (87.8), 18 models
Drone-Bench Detect: build a stateful detector that returns a whole-body bounding box for a person given one reference photo, and reports when the target leaves the frame; scored against labeled drone clips with a hidden test set. Average over runs of each run's best submission, as a percentage (human baseline 72.1).
Drone-Bench - Follow (Follow score (%)): leader GPT-6 (88.6), 18 models
Drone-Bench Follow: write a control loop that uses detector boxes to keep a moving person in the center third of the frame at close range; a collision zeroes the remaining frames. Score is the average over runs of each run's best of up to ten submissions, as a percentage (human baseline 67.4); higher is better.
DiG-bench - Composite Score (Expected composite score (%)): leader Claude Opus 5 (68.49), 9 models
DiG-bench composite: expected score across the 70 hidden-rule text games, weighting a win at 0.8 and in-game progress at 0.2, from the paper's agent-strength fit (0–100, humans 100). Rows are model and harness runs, such as the benchmark agent or Claude Code, kept separate; higher is better.
Kaggle Game Arena (Unified) (Unified Score): leader GPT-5.5 (349.7), 30 models
Kaggle Game Arena's unified leaderboard: one rating per model pooled from its matches across the Game Arena games, which include text chess, Four in a Row, heads-up poker and Werewolf; each model covered 5 to 16 games on 2026-09-13. Higher is better on an unbounded scale where the lowest-rated model sits at 0, not comparable to the per-game Elo boards.
Kaggle Global-MMLU-Lite - Culturally Agnostic (Accuracy (%)): leader Gemini 3.5 Flash (96.31), 37 models
Accuracy on the culturally agnostic questions of Cohere Labs' Global-MMLU-Lite leaderboard on Kaggle, pooled over its 16 languages. Global-MMLU-Lite is an MMLU-derived multilingual multiple-choice set whose questions are annotated as culturally sensitive or agnostic. Shown as 0-100, higher is better; four answer options, so guessing scores about 25.
Kaggle Global-MMLU-Lite - Culturally Sensitive (Accuracy (%)): leader Claude Opus 5 (95.44), 37 models
Accuracy on the culturally sensitive questions of Cohere Labs' Global-MMLU-Lite leaderboard on Kaggle, pooled over its 16 languages; these questions need region- or culture-specific knowledge. Global-MMLU-Lite is an MMLU-derived multilingual multiple-choice set. Shown as 0-100, higher is better; four answer options, so guessing scores about 25.
Kaggle Global-MMLU-Lite - Arabic (Accuracy (%)): leader Gemini 3.5 Flash (96.0), 37 models
Accuracy on the Arabic questions (400 on this board) of Cohere Labs' Global-MMLU-Lite leaderboard on Kaggle, an MMLU-derived multilingual multiple-choice set with culturally sensitive and culturally agnostic questions. Shown as 0-100, higher is better; four answer options, so guessing scores about 25.
Kaggle Global-MMLU-Lite - Bengali (Accuracy (%)): leader Claude Opus 4.8 (95.75), 37 models
Accuracy on the Bengali questions (400 on this board) of Cohere Labs' Global-MMLU-Lite leaderboard on Kaggle, an MMLU-derived multilingual multiple-choice set with culturally sensitive and culturally agnostic questions. Shown as 0-100, higher is better; four answer options, so guessing scores about 25.
Kaggle Global-MMLU-Lite - Burmese (Accuracy (%)): leader Gemini 3.5 Flash (95.5), 37 models
Accuracy on the Burmese questions (400 on this board) of Cohere Labs' Global-MMLU-Lite leaderboard on Kaggle, an MMLU-derived multilingual multiple-choice set with culturally sensitive and culturally agnostic questions. Shown as 0-100, higher is better; four answer options, so guessing scores about 25.
Kaggle Global-MMLU-Lite - Chinese (Accuracy (%)): leader Claude Opus 5 (97.25), 37 models
Accuracy on the Chinese questions (400 on this board) of Cohere Labs' Global-MMLU-Lite leaderboard on Kaggle, an MMLU-derived multilingual multiple-choice set with culturally sensitive and culturally agnostic questions. Shown as 0-100, higher is better; four answer options, so guessing scores about 25.
Kaggle Global-MMLU-Lite - English (Accuracy (%)): leader Claude Opus 5 (96.5), 37 models
Accuracy on the English questions (400 on this board) of Cohere Labs' Global-MMLU-Lite leaderboard on Kaggle, an MMLU-derived multilingual multiple-choice set with culturally sensitive and culturally agnostic questions. Shown as 0-100, higher is better; four answer options, so guessing scores about 25.
Kaggle Global-MMLU-Lite - French (Accuracy (%)): leader Claude Opus 5 (96.25), 37 models
Accuracy on the French questions (400 on this board) of Cohere Labs' Global-MMLU-Lite leaderboard on Kaggle, an MMLU-derived multilingual multiple-choice set with culturally sensitive and culturally agnostic questions. Shown as 0-100, higher is better; four answer options, so guessing scores about 25.
Kaggle Global-MMLU-Lite - German (Accuracy (%)): leader Claude Opus 5 (97.0), 37 models
Accuracy on the German questions (400 on this board) of Cohere Labs' Global-MMLU-Lite leaderboard on Kaggle, an MMLU-derived multilingual multiple-choice set with culturally sensitive and culturally agnostic questions. Shown as 0-100, higher is better; four answer options, so guessing scores about 25.
Kaggle Global-MMLU-Lite - Hindi (Accuracy (%)): leader Claude Opus 5 (95.5), 37 models
Accuracy on the Hindi questions (400 on this board) of Cohere Labs' Global-MMLU-Lite leaderboard on Kaggle, an MMLU-derived multilingual multiple-choice set with culturally sensitive and culturally agnostic questions. Shown as 0-100, higher is better; four answer options, so guessing scores about 25.
Kaggle Global-MMLU-Lite - Indonesian (Accuracy (%)): leader Claude Opus 5 (96.75), 37 models
Accuracy on the Indonesian questions (400 on this board) of Cohere Labs' Global-MMLU-Lite leaderboard on Kaggle, an MMLU-derived multilingual multiple-choice set with culturally sensitive and culturally agnostic questions. Shown as 0-100, higher is better; four answer options, so guessing scores about 25.
Kaggle Global-MMLU-Lite - Italian (Accuracy (%)): leader Claude Opus 5 (97.0), 37 models
Accuracy on the Italian questions (400 on this board) of Cohere Labs' Global-MMLU-Lite leaderboard on Kaggle, an MMLU-derived multilingual multiple-choice set with culturally sensitive and culturally agnostic questions. Shown as 0-100, higher is better; four answer options, so guessing scores about 25.
Kaggle Global-MMLU-Lite - Japanese (Accuracy (%)): leader Claude Opus 5 (96.0), 37 models
Accuracy on the Japanese questions (400 on this board) of Cohere Labs' Global-MMLU-Lite leaderboard on Kaggle, an MMLU-derived multilingual multiple-choice set with culturally sensitive and culturally agnostic questions. Shown as 0-100, higher is better; four answer options, so guessing scores about 25.
Kaggle Global-MMLU-Lite - Korean (Accuracy (%)): leader Claude Opus 5 (95.5), 37 models
Accuracy on the Korean questions (400 on this board) of Cohere Labs' Global-MMLU-Lite leaderboard on Kaggle, an MMLU-derived multilingual multiple-choice set with culturally sensitive and culturally agnostic questions. Shown as 0-100, higher is better; four answer options, so guessing scores about 25.
Kaggle Global-MMLU-Lite - Portuguese (Accuracy (%)): leader Claude Opus 5 (96.5), 37 models
Accuracy on the Portuguese questions (400 on this board) of Cohere Labs' Global-MMLU-Lite leaderboard on Kaggle, an MMLU-derived multilingual multiple-choice set with culturally sensitive and culturally agnostic questions. Shown as 0-100, higher is better; four answer options, so guessing scores about 25.
Kaggle Global-MMLU-Lite - Spanish (Accuracy (%)): leader Claude Opus 5 (97.0), 37 models
Accuracy on the Spanish questions (400 on this board) of Cohere Labs' Global-MMLU-Lite leaderboard on Kaggle, an MMLU-derived multilingual multiple-choice set with culturally sensitive and culturally agnostic questions. Shown as 0-100, higher is better; four answer options, so guessing scores about 25.
Kaggle Global-MMLU-Lite - Swahili (Accuracy (%)): leader Gemini 3.8 Flash (94.75), 37 models
Accuracy on the Swahili questions (400 on this board) of Cohere Labs' Global-MMLU-Lite leaderboard on Kaggle, an MMLU-derived multilingual multiple-choice set with culturally sensitive and culturally agnostic questions. Shown as 0-100, higher is better; four answer options, so guessing scores about 25.
Kaggle Global-MMLU-Lite - Yoruba (Accuracy (%)): leader Gemini 3.5 Flash (96.0), 37 models
Accuracy on the Yoruba questions (400 on this board) of Cohere Labs' Global-MMLU-Lite leaderboard on Kaggle, an MMLU-derived multilingual multiple-choice set with culturally sensitive and culturally agnostic questions. Shown as 0-100, higher is better; four answer options, so guessing scores about 25.
Kaggle SimpleQA (Score (%)): leader Gemini 3.1 Pro (Preview) (74.76), 38 models
OpenAI's original SimpleQA short-form factuality benchmark of fact-seeking questions with single verifiable answers, run by Kaggle on its own harness under OpenAI's Kaggle organization. Kaggle's task score is shown as a percentage (0-100, higher is better). This is the original question set, not SimpleQA Verified.
Kaggle SimpleQA Verified (Score (%)): leader Gemini 3.1 Pro (Preview) (77.52), 45 models
Google DeepMind's SimpleQA Verified, a 1,000-prompt revision of OpenAI's SimpleQA that removes noisy or incorrect labels, redundant questions and topical bias, from its official Kaggle leaderboard. Kaggle's task score is shown as a percentage (0-100, higher is better); the paper's headline metric is an F1 over correct and attempted answers.
Kaggle VideoQA (Accuracy (%)): leader Gemini 3.7 Flash (89.11), 13 models
Google DeepMind's VideoQA leaderboard on Kaggle: question answering over video inputs, scored as accuracy and shown as 0-100 (higher is better). The roster is small and Gemini-heavy (9 of 13 models on 2026-09-13), so treat rankings outside that family with care.
Anthropic AI R&D - Kernel Task Best Speedup (Best Speedup (x)): leader Claude Opus 5 (449.46), 4 models
Kernel optimization task from Anthropic's internal AI R&D rule-out suite: best speedup (x) achieved on the hard task with the standard scaffold; unbounded, higher is better. From Table 2.3.5.A of the Claude Opus 5 System Card (July 2026); the Sonnet 5 row is from Table 2.3.3.A of the Sonnet 5 card.
Anthropic AI R&D - Time Series Forecasting MSE (Mean Squared Error (Hard Variant)): leader Claude Mythos 5 (4.51), 4 models
Time-series forecasting task from Anthropic's internal AI R&D rule-out suite: mean squared error on the hard variant, where the card equates below 5.3 with 40 hours of human effort; lower is better. From Table 2.3.5.A of the Claude Opus 5 System Card (July 2026); the Sonnet 5 row is from Table 2.3.3.A of the Sonnet 5 card.
Anthropic AI R&D - LLM Training Average Speedup (Average Speedup (x)): leader Claude Mythos 5 (69.61), 4 models
LLM training task (easy variant) from Anthropic's internal AI R&D rule-out suite: average speedup (x) from optimizing a small language-model training implementation; unbounded, higher is better. From Table 2.3.5.A of the Claude Opus 5 System Card (July 2026); the Sonnet 5 row is from Table 2.3.3.A of the Sonnet 5 card.
Anthropic AI R&D - LLM Training Hard Variant Average Speedup (Average Speedup (x)): leader Claude Opus 5 (14.19), 2 models
Harder variant of the LLM training task in Anthropic's internal AI R&D rule-out suite, starting from already-optimized reference code with a neutral prompt; average speedup (x), unbounded, higher is better. Introduced in Table 2.3.5.A of the Claude Opus 5 System Card (July 2026); only Mythos 5 and Opus 5 were run.
Anthropic AI R&D - Quadruped RL Score (Highest Score): leader Claude Opus 5 (31.3), 4 models
Quadruped reinforcement-learning task from Anthropic's internal AI R&D rule-out suite: highest score reached training a quadruped agent without supplied hyperparameters; unbounded, higher is better. From Table 2.3.5.A of the Claude Opus 5 System Card (July 2026); the Sonnet 5 row is from Table 2.3.3.A of the Sonnet 5 card.
Anthropic AI R&D - Novel Compiler Pass Rate (Pass Rate on Complex Tests (%)): leader Claude Mythos 5 (85.3), 4 models
Novel compiler task from Anthropic's internal AI R&D rule-out suite, the bounded task the Opus 5 card still reports as unsaturated: pass rate on complex tests (percent, higher is better; 90% equals 40 hours of human effort). From Table 2.3.5.A of the Claude Opus 5 System Card (July 2026); the Sonnet 5 row is from Table 2.3.3.A of the Sonnet 5 card.
ExploitBench AutoNudge Mean Flags (Sonnet 5 & Opus 5 System Cards) (Mean Capability Flags per Trial (of 16)): leader Claude Mythos 5 (10.8), 5 models
ExploitBench (Lee and Brumley, 2026): exploit development for 41 recent V8 vulnerabilities, scored by 16 mechanically verified capability flags, five trials each, 300-turn budget. Mean flags captured per trial (0-16, higher is better) in the AutoNudge arm, which re-prompts a model that stops early. Anthropic's runs from the Claude Opus 5 (p. 37) and Sonnet 5 (p. 31) system cards.
ExploitBench AutoNudge Flag Capture Rate (Sonnet 5 & Opus 5 System Cards) (Flags Captured per Environment (%)): leader Claude Mythos 5 (78.0), 5 models
ExploitBench (Lee and Brumley, 2026) on 41 V8 vulnerabilities with 16 capability flags. Cap% is the percentage of available flags captured per environment over a random subset of three AutoNudge-arm trials, averaged across environments (higher is better). Anthropic's runs from the Claude Opus 5 (p. 37) and Sonnet 5 (p. 31) system cards.
ExploitBench Full ACE Exploit Runs (Sonnet 5 & Opus 5 System Cards) (Runs with Full Arbitrary Code Execution (of 410)): leader Claude Mythos 5 (132.0), 5 models
ExploitBench (Lee and Brumley, 2026) on 41 V8 vulnerabilities: number of complete arbitrary-code-execution exploits produced across the plain and AutoNudge arms, five trials per environment in each (0-410, higher is better). Anthropic's runs from the Claude Opus 5 (p. 37) and Sonnet 5 (p. 31) system cards.
BBQ Disambiguated Accuracy (Sonnet 5 & Opus 5 System Cards) (Accuracy (%)): leader Claude Sonnet 4.6 (Non-reasoning) (88.1), 5 models
Bias Benchmark for Question Answering (BBQ) accuracy on disambiguated questions, where the context supplies the answer, as run by Anthropic (percent, higher is better; three answer options; thinking settings as labelled). From Table 4.4.2.A of the Claude Opus 5 System Card (July 2026), with Mythos Preview and Sonnet 4.6 rows from the Sonnet 5 card.
BBQ Ambiguous Accuracy (Sonnet 5 & Opus 5 System Cards) (Accuracy (%)): leader Claude Opus 5 (Non-reasoning) (100.0), 6 models
Bias Benchmark for Question Answering (BBQ) accuracy on ambiguous questions, where the correct answer is that the context is insufficient, as run by Anthropic (percent, higher is better; near the ceiling for every model). From Table 4.4.2.A of the Claude Opus 5 System Card (July 2026), with Mythos Preview and Sonnet 4.6 rows from the Sonnet 5 card.
Anthropic Claude Code Malicious Requests - Refusal Rate (Sonnet 5 & Opus 5 System Cards) (Refusal Rate (%)): leader Claude Mythos Preview (96.21), 6 models
Anthropic's Claude Code agentic safety evaluation: refusal rate on 61 malicious cyber prompts that violate the Usage Policy, such as malware or DDoS code (percent, higher is better). From Table 5.1.1.A of the Claude Opus 5 System Card (July 2026), with Mythos Preview and Sonnet 4.6 rows from the Sonnet 5 card; the September re-run is a separate board.
Anthropic Claude Code Dual-Use Requests - Success Rate (Sonnet 5 & Opus 5 System Cards) (Success Rate (%)): leader Claude Opus 5 (99.82), 6 models
Anthropic's Claude Code agentic safety evaluation: success rate on 61 dual-use and benign cyber prompts Claude should assist with, such as running network reconnaissance tools (percent, higher is better). From Table 5.1.1.A of the Claude Opus 5 System Card (July 2026), with Mythos Preview and Sonnet 4.6 rows from the Sonnet 5 card.
Humanity's Last Exam No Tools (Opus 5 System Card) (Accuracy (%)): leader Claude Fable 5 (56.5), 2 models
Humanity's Last Exam (HLE), a 2,500-question multimodal benchmark of expert-level questions, answered without tools; accuracy in percent, higher is better. Anthropic's runs from Table 8.1.A of the Claude Opus 5 System Card (July 2026); the September card's HLE re-run reports different values and is a separate board.
Anthropic Single-Turn Harmful Requests - Harmless Rate (API) (Harmless Response Rate (%)): leader Claude Sonnet 4.6 (97.71), 8 models
Anthropic's single-turn harmful-request evaluation across 16 Usage Policy areas in seven languages: percentage of prompts whose response did not facilitate the requested harm, via the API without a system prompt (higher is better). From Table 4.1.1.A of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026); older-model rows come from the Opus 5 and Sonnet 5 cards.
Anthropic Single-Turn Benign Requests - Over-Refusal Rate (API) (Over-Refusal Rate (%)): leader Claude Fable 5.1 (0.0), 8 models
Anthropic's single-turn benign-request evaluation across seven languages: percentage of sensitive but appropriate prompts the model refuses, via the API without a system prompt (lower is better). From Table 4.1.2.A of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026); older-model rows come from the Opus 5 and Sonnet 5 cards.
Anthropic Single-Turn Harmful Requests - Harmless Rate (Claude.ai) (Harmless Response Rate (%)): leader Claude Fable 5 (99.54), 6 models
Anthropic's single-turn harmful-request evaluation across 16 Usage Policy areas in seven languages: percentage of prompts whose response did not facilitate the requested harm, with the claude.ai system prompt (higher is better). From Table 4.1.1.A of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026); older-model rows come from the Opus 5 and Sonnet 5 cards.
Anthropic Single-Turn Benign Requests - Over-Refusal Rate (Claude.ai) (Over-Refusal Rate (%)): leader Claude Fable 5.1 (0.34), 6 models
Anthropic's single-turn benign-request evaluation across seven languages: percentage of sensitive but appropriate prompts the model refuses, with the claude.ai system prompt (lower is better). From Table 4.1.2.A of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026); older-model rows come from the Opus 5 and Sonnet 5 cards.
Anthropic Suicide and Self-Harm Single-Turn Risk Requests - Harmless Rate (API) (Harmless Response Rate (%)): leader Claude Mythos 5 (99.67), 8 models
Anthropic's suicide and self-harm evaluation: percentage of single-turn requests posing potential risk that received a harmless response, via the API without a system prompt (higher is better). From Section 4.3.1 of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026); older-model rows come from the Opus 5 and Sonnet 5 cards.
Anthropic Disordered Eating Single-Turn Risk Requests - Harmless Rate (API) (Harmless Response Rate (%)): leader Claude Fable 5 (97.88), 8 models
Anthropic's disordered eating evaluation: percentage of single-turn requests that could reinforce disordered eating that received a harmless response, via the API without a system prompt (higher is better). From Table 4.3.2.A of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026); older-model rows come from the Opus 5 and Sonnet 5 cards.
Anthropic Child Safety Single-Turn Benign Requests - Over-Refusal Rate (Claude.ai) (Over-Refusal Rate (%)): leader Claude Fable 5.1 (0.17), 6 models
Anthropic's child safety evaluation: percentage of benign single-turn child-safety-related requests the model refuses with the claude.ai system prompt (lower is better). From Section 4.2 of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026); older-model rows come from the Opus 5 and Sonnet 5 cards.
Anthropic Disordered Eating Single-Turn Risk Requests - Harmless Rate (Claude.ai) (Harmless Response Rate (%)): leader Claude Opus 4.8 (99.7), 6 models
Anthropic's disordered eating evaluation: percentage of single-turn requests that could reinforce disordered eating that received a harmless response, with the claude.ai system prompt (higher is better). From Table 4.3.2.A of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026); older-model rows come from the Opus 5 and Sonnet 5 cards.
Anthropic Child Safety Multi-Turn - Appropriate Response Rate (API) (Appropriate Response Rate (%)): leader Claude Sonnet 4.6 (92.0), 8 models
Anthropic's multi-turn child safety evaluation: a Claude model plays a synthetic user from an expert-written persona specification, scored as the percentage of conversations handled appropriately throughout, via the API without a system prompt (higher is better). From Section 4.2 of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026); older-model rows come from the Opus 5 and Sonnet 5 cards.
Anthropic Suicide and Self-Harm Multi-Turn - Appropriate Response Rate (API) (Appropriate Response Rate (%)): leader Claude Sonnet 4.6 (74.0), 8 models
Anthropic's multi-turn suicide and self-harm evaluation: a Claude model plays a synthetic user from an expert-written persona specification, scored as the percentage of conversations handled appropriately throughout, via the API without a system prompt (higher is better). From Section 4.3.1 of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026); older-model rows come from the Opus 5 and Sonnet 5 cards.
Anthropic Election Integrity Multi-Turn - Appropriate Response Rate (API) (Appropriate Response Rate (%)): leader Claude Mythos 5 (93.0), 6 models
Anthropic's multi-turn election integrity evaluation: a Claude model plays a synthetic user from an expert-written persona specification, scored as the percentage of conversations handled appropriately throughout, via the API without a system prompt (higher is better). From Section 4.4.3 of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026); the Opus 4.8 row comes from the Opus 5 card.
Anthropic Child Safety Multi-Turn - Appropriate Response Rate (Claude.ai) (Appropriate Response Rate (%)): leader Claude Fable 5.1 (100.0), 6 models
Anthropic's multi-turn child safety evaluation: a Claude model plays a synthetic user from an expert-written persona specification, scored as the percentage of conversations handled appropriately throughout, with the claude.ai system prompt (higher is better). From Section 4.2 of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026); older-model rows come from the Opus 5 and Sonnet 5 cards.
Anthropic Suicide and Self-Harm Multi-Turn - Appropriate Response Rate (Claude.ai) (Appropriate Response Rate (%)): leader Claude Fable 5 (100.0), 6 models
Anthropic's multi-turn suicide and self-harm evaluation: a Claude model plays a synthetic user from an expert-written persona specification, scored as the percentage of conversations handled appropriately throughout, with the claude.ai system prompt (higher is better). From Section 4.3.1 of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026); older-model rows come from the Opus 5 and Sonnet 5 cards.
Anthropic Election Integrity Multi-Turn - Appropriate Response Rate (Claude.ai) (Appropriate Response Rate (%)): leader Claude Fable 5 (93.0), 5 models
Anthropic's multi-turn election integrity evaluation: a Claude model plays a synthetic user from an expert-written persona specification, scored as the percentage of conversations handled appropriately throughout, with the claude.ai system prompt (higher is better). From Table 4.4.3.B of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026), whose Mythos 5.1 claude.ai cell was run as Fable 5.1; Opus 4.8 from the Opus 5 card.
Anthropic Malicious Computer Use - Refusal Rate (Refusal Rate (%)): leader Claude Opus 5 (93.75), 7 models
Anthropic's malicious computer-use evaluation: refusal rate on 112 harmful tasks (surveillance and data collection, harmful content, scaled abuse) given GUI and CLI tools in a sandbox (percent, higher is better). From Section 5.1.2 of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026), task set unchanged since the Opus 5 card; older-model rows come from the Opus 5 and Sonnet 5 cards.
Anthropic Claude Code Malicious Requests - Refusal Rate (Mythos 5.1 System Card) (Refusal Rate (%)): leader Claude Sonnet 5 (90.7), 4 models
Anthropic's Claude Code agentic safety evaluation as re-run for the September card: refusal rate on 61 malicious cyber prompts that violate the Usage Policy (percent, higher is better). From Table 5.1.1.A of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026); values differ from the Opus 5 card's version.
Anthropic Claude Code Dual-Use Requests - Success Rate (Mythos 5.1 System Card) (Success Rate (%)): leader Claude Opus 5 (99.7), 4 models
Anthropic's Claude Code agentic safety evaluation as re-run for the September card: success rate on 61 dual-use and benign cyber prompts Claude should assist with (percent, higher is better). From Table 5.1.1.A of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026); values differ from the Opus 5 card's version.
SWE-bench Multilingual (Opus 5 & Mythos 5.1 System Cards) (Resolved (%)): leader Claude Opus 5 (Max) (89.5), 3 models
SWE-bench Multilingual, real GitHub issues from repositories in programming languages beyond Python, in Anthropic's own runs; percentage resolved (higher is better). From Table 8.1.A of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026), matching the Opus 5 card; the Opus 4.8 cell already on SWE-bench Multilingual (Anthropic Scaffold) is omitted.
SWE-bench Multimodal (Opus 5 & Mythos 5.1 System Cards) (Resolved (%)): leader Claude Opus 5 (Max) (59.4), 3 models
SWE-bench Multimodal, GitHub issues from JavaScript repositories whose problem statements include images, in Anthropic's own runs; percentage resolved (higher is better). From Table 8.1.A of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026), matching the Opus 5 card; the Opus 4.8 cell already on the Opus 4.8 card's board is omitted.
Humanity's Last Exam No Tools (Mythos 5.1 System Card) (Accuracy (%)): leader Claude Fable 5.1 (Max) (60.9), 3 models
Humanity's Last Exam (HLE), a 2,500-question multimodal benchmark of expert-level questions, answered without tools; accuracy in percent, higher is better. Anthropic's re-run from Table 8.1.A of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026), with values that differ from the Opus 5 card's version.
Humanity's Last Exam With Tools (Mythos 5.1 System Card) (Accuracy (%)): leader Claude Fable 5 (63.8), 2 models
Humanity's Last Exam (HLE), a 2,500-question multimodal benchmark of expert-level questions, answered with tools; accuracy in percent, higher is better. Anthropic's runs from Table 8.1.A of the Claude Fable 5.1 & Claude Mythos 5.1 System Card (September 2026); the Fable 5.1 cell, already mirrored by Vellum, is omitted.
Kaggle ExtractBench (Score (%)): leader GPT-5.6 Sol (91.04), 16 models
LlamaIndex's ExtractBench leaderboard on Kaggle: schema-guided extraction of structured data from enterprise documents (370 documents, 4,869 pages and 8 business domains in the paper). This board is Kaggle's score on the full set, extractbench-full, converted from a 0-1 fraction to 0-100, higher is better.
Kaggle Open Benchmarks - GPQA Diamond (Zero-Shot) (Accuracy (%)): leader Grok 4.5 (94.95), 52 models
Zero-shot accuracy on the 198-question GPQA Diamond subset of graduate-level, Google-proof multiple-choice science questions (biology, chemistry, physics). Kaggle's standardized re-run on its own harness under the open-benchmarks organization, not the original authors' leaderboard; Kaggle's fraction is shown as a percentage (0-100, higher is better). Four answer options, so guessing scores about 25.
Kaggle Open Benchmarks - SciCode Subproblems (Subproblem Pass Rate (%)): leader Gemini 3.1 Pro (Preview) (44.9), 55 models
SciCode subproblem pass rate: the share of scientific-computing subproblems (drawn from 65 research coding problems across physics, chemistry, biology, mathematics and materials science) whose generated code passes the tests, in the standard setting without added background text. Kaggle's standardized re-run on its own harness under the open-benchmarks organization, not the original authors' leaderboard; Kaggle's fraction is shown as a percentage (0-100, higher is better).
Kaggle Open Benchmarks - AIME 2025 (Accuracy (%)): leader GPT-5.5 (100.0), 55 models
Accuracy on the 30 problems of the 2025 American Invitational Mathematics Examination (AIME I and II), integer answers from 0 to 999 graded by exact match. Kaggle's standardized re-run on its own harness under the open-benchmarks organization, not the original authors' leaderboard; Kaggle's fraction is shown as a percentage (0-100, higher is better). With 30 problems, each question is worth about 3.3 points.
Kaggle Open Benchmarks - MMLU-Pro (Accuracy (%)): leader Claude Opus 5 (91.68), 55 models
Accuracy on MMLU-Pro, a harder ten-option successor to MMLU with reasoning-focused questions across 14 subject areas. Kaggle's standardized re-run on its own harness under the open-benchmarks organization, not the original authors' leaderboard; Kaggle's fraction is shown as a percentage (0-100, higher is better). Ten answer options, so guessing scores about 10.
Kaggle Open Benchmarks - MMLU (Accuracy (%)): leader Grok 4.5 (94.54), 51 models
Accuracy on MMLU, four-option multiple-choice questions across 57 subjects from elementary mathematics to law and medicine. Kaggle's standardized re-run on its own harness under the open-benchmarks organization, not the original authors' leaderboard; Kaggle's fraction is shown as a percentage (0-100, higher is better). Guessing scores about 25; frontier models are near the ceiling.
Kaggle Open Benchmarks - MATH-500 (Accuracy (%)): leader Gemini 3.8 Flash (99.11), 16 models
Accuracy on MATH-500, a 500-problem subset of the MATH competition-mathematics dataset spanning seven subjects and five difficulty levels, graded by final-answer equivalence. Kaggle's standardized re-run on its own harness under the open-benchmarks organization, not the original authors' leaderboard; Kaggle's fraction is shown as a percentage (0-100, higher is better).
OpenAI GPT-5.6 Launch - Management Consulting Tasks (Internal) (Score (%)): leader GPT-5.6 Sol (43.2), 7 models
OpenAI's internal management-consulting task set, professional table of OpenAI's GPT-5.6 launch post (2026-07-09). The tasks and grading are not published. Score in percent, higher is better. Compares GPT-5.6 Sol, Terra and Luna, GPT-5.5, Claude Fable 5, Claude Opus 4.8 and Gemini 3.1 Pro Preview.
OpenAI GPT-5.6 Launch - GeneBench Pro (Score (%)): leader GPT-5.6 Sol (28.7), 7 models
GeneBench Pro row of the science and health table in OpenAI's GPT-5.6 launch post (2026-07-09). Score in percent, higher is better; seven models including Claude Opus 4.8, Gemini 3.1 Pro Preview and Gemini 3.5 Flash. The GPT-6 Astra post used the later v13 (GPT-5.6 Sol 32.3 there vs 28.7 here), so each launch keeps its own board.
OpenAI GPT-5.6 Launch - LifeSciBench (Score (%)): leader GPT-5.6 Sol (59.9), 5 models
LifeSciBench row of the science and health table in OpenAI's GPT-5.6 launch post (2026-07-09). Score in percent, higher is better; GPT-5.6 Sol, Terra, Luna, GPT-5.5 and Claude Opus 4.8. Separate from the GPT-Rosalind LifeSciBench exact-pass-rate board and the GPT-6 Astra launch board.
OpenAI GPT-5.6 Launch - MedChemBench (Internal) (Score (%)): leader GPT-5.6 Sol (48.3), 4 models
OpenAI's internal medicinal-chemistry benchmark (MedChemBench), science and health table of OpenAI's GPT-5.6 launch post (2026-07-09). Score in percent, higher is better; only GPT-5.6 Sol, Terra, Luna and GPT-5.5 have values.
OpenAI GPT-5.6 Launch - BenchCAD (Score (%)): leader GPT-5.6 Sol (70.6), 7 models
BenchCAD row of the computer use table in OpenAI's GPT-5.6 launch post (2026-07-09): models reconstruct 3D objects from multi-view renders by writing CAD code, without the python tool reported on a separate row. Score in percent, higher is better; includes Claude Mythos 5, Mythos Preview and Opus 4.8.
OpenAI GPT-5.6 Launch - BenchCAD (python tool) (Score (%)): leader GPT-5.6 Sol (83.4), 7 models
BenchCAD (python tool) row of the computer use table in OpenAI's GPT-5.6 launch post (2026-07-09): the CAD-code reconstruction task from multi-view renders with a python tool available. Score in percent, higher is better; includes Claude Mythos 5, Mythos Preview and Opus 4.8.
OpenAI GPT-5.6 Launch - Capture-the-Flag Challenges (Score (%)): leader GPT-5.6 Sol (96.7), 4 models
Capture-the-Flag Challenges row of the cybersecurity table in OpenAI's GPT-5.6 launch post (2026-07-09). Score in percent, higher is better; GPT-5.6 Sol, Terra, Luna and GPT-5.5. Not the pass@12 'OpenAI CTF (Professional)' board from the GPT-5.5 system card, whose values differ.
OpenAI GPT-5.6 Launch - Internal Research Debugging Evaluation (Score (%)): leader GPT-5.6 Sol (68.3), 4 models
Self-improvement table of OpenAI's GPT-5.6 launch post (2026-07-09): models must find and fix real bugs in internal OpenAI research experiments that took researchers hours to days to resolve (per the GPT-5.6 system card). Score in percent, higher is better; OpenAI models only.
OpenAI GPT-5.6 Launch - KernelGen 1P (Score (%)): leader GPT-5.6 Sol (61.1), 4 models
KernelGen 1P row of the self-improvement table in OpenAI's GPT-5.6 launch post (2026-07-09), an OpenAI first-party kernel-generation evaluation whose tasks and grading are not published. Score in percent, higher is better; GPT-5.6 Sol, Terra, Luna and GPT-5.5 only.
OpenAI GPT-5.6 Launch - NanoGPT (Normalized reward (%)): leader GPT-5.6 Terra (14.5), 4 models
NanoGPT row of the self-improvement table in OpenAI's GPT-5.6 launch post (2026-07-09), reported as a normalized reward in percent; the post does not define the normalization. Higher is better; GPT-5.6 Sol, Terra, Luna and GPT-5.5 only.
OpenAI GPT-5.6 Launch - PostTrainBench Lite (Score (%)): leader GPT-5.6 Terra (51.5), 4 models
PostTrainBench Lite row of the self-improvement table in OpenAI's GPT-5.6 launch post (2026-07-09), a reduced version of PostTrainBench, in which agents post-train language models. Score in percent, higher is better; GPT-5.6 Sol, Terra, Luna and GPT-5.5 only.
OpenAI GPT-5.6 Launch - RSI Index (Index score (%)): leader GPT-5.6 Sol (57.9), 4 models
RSI Index row of the self-improvement table in OpenAI's GPT-5.6 launch post (2026-07-09), an aggregate index score printed in percent; the post does not publish its components or weighting. Higher is better; GPT-5.6 Sol, Terra, Luna and GPT-5.5 only.
OpenAI GPT-5.6 Launch - OpenAI MRCR v2 8-needle 256K-512K (Score (%)): leader GPT-5.6 Sol (91.5), 4 models
OpenAI MRCR v2 long-context multi-round coreference test with 8 needles and prompts of 256K-512K tokens, long context table of OpenAI's GPT-5.6 launch post (2026-07-09). Score in percent, higher is better; GPT-5.6 Sol, Terra, Luna and GPT-5.5.
OpenAI GPT-5.6 Launch - OpenAI MRCR v2 8-needle 512K-1M (Score (%)): leader GPT-5.5 (74.0), 4 models
OpenAI MRCR v2 long-context multi-round coreference test with 8 needles and prompts of 512K-1M tokens, long context table of OpenAI's GPT-5.6 launch post (2026-07-09). Score in percent, higher is better; GPT-5.6 Sol, Terra, Luna and GPT-5.5.
OpenAI GPT-6 Astra System Card - Misaligned Outcomes (No Confirmation Policy) - Overall (Misaligned outcome rate (%)): leader GPT-6 (3.4), 4 models
Table 8 (misaligned outcomes in realistic work environments) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of adversarial computer- and browser-use workplace tasks ending in any misaligned outcome, for the base model without the confirmation policy. Percent, lower is better; GPT-5.6 Luna, Terra, Sol and GPT-6 Astra.
OpenAI GPT-6 Astra System Card - Misaligned Outcomes (No Confirmation Policy) - Unauthorized Transactions (Misaligned outcome rate (%)): leader GPT-6 (6.8), 4 models
Table 8 (misaligned outcomes in realistic work environments) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of adversarial computer- and browser-use workplace tasks ending in an unauthorized transaction, for the base model without the confirmation policy. Percent, lower is better; GPT-5.6 Luna, Terra, Sol and GPT-6 Astra.
OpenAI GPT-6 Astra System Card - Misaligned Outcomes (No Confirmation Policy) - Data Exfiltration (Misaligned outcome rate (%)): leader GPT-6 (4.3), 4 models
Table 8 (misaligned outcomes in realistic work environments) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of adversarial computer- and browser-use workplace tasks ending in data exfiltration, for the base model without the confirmation policy. Percent, lower is better; GPT-5.6 Luna, Terra, Sol and GPT-6 Astra.
OpenAI GPT-6 Astra System Card - Misaligned Outcomes (No Confirmation Policy) - Destructive Action (Misaligned outcome rate (%)): leader GPT-6 (0.0), 4 models
Table 8 (misaligned outcomes in realistic work environments) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of adversarial computer- and browser-use workplace tasks ending in a destructive action, for the base model without the confirmation policy. Percent, lower is better; GPT-5.6 Luna, Terra, Sol and GPT-6 Astra.
OpenAI GPT-6 Astra System Card - Misaligned Outcomes (No Confirmation Policy) - Service Disruption (Misaligned outcome rate (%)): leader GPT-6 (0.0), 4 models
Table 8 (misaligned outcomes in realistic work environments) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of adversarial computer- and browser-use workplace tasks ending in service disruption, for the base model without the confirmation policy. Percent, lower is better; GPT-5.6 Luna, Terra, Sol and GPT-6 Astra.
OpenAI GPT-6 Astra System Card - Misaligned Outcomes (No Confirmation Policy) - Security Weakening (Misaligned outcome rate (%)): leader GPT-6 (0.0), 4 models
Table 8 (misaligned outcomes in realistic work environments) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of adversarial computer- and browser-use workplace tasks ending in weakened security, for the base model without the confirmation policy. Percent, lower is better; GPT-5.6 Luna, Terra, Sol and GPT-6 Astra.
OpenAI GPT-6 Astra System Card - Misaligned Outcomes (No Confirmation Policy) - Unauthorized External Communication (Misaligned outcome rate (%)): leader GPT-6 (1.7), 4 models
Table 8 (misaligned outcomes in realistic work environments) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of adversarial computer- and browser-use workplace tasks ending in unauthorized external communication, for the base model without the confirmation policy. Percent, lower is better; GPT-5.6 Luna, Terra, Sol and GPT-6 Astra.
OpenAI GPT-6 Astra System Card - Misaligned Outcomes (No Confirmation Policy) - Circumventing Restrictions (Misaligned outcome rate (%)): leader GPT-6 (0.0), 4 models
Table 8 (misaligned outcomes in realistic work environments) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of adversarial computer- and browser-use workplace tasks ending in circumvented restrictions, for the base model without the confirmation policy. Percent, lower is better; GPT-5.6 Luna, Terra, Sol and GPT-6 Astra.
OpenAI GPT-6 Astra System Card - Misaligned Outcomes (Confirmation Policy) - Overall (Misaligned outcome rate (%)): leader GPT-6 (3.0), 4 models
Table 8 (misaligned outcomes in realistic work environments) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of adversarial computer- and browser-use workplace tasks ending in any misaligned outcome, with the confirmation policy that is the default for users. Percent, lower is better; GPT-5.6 Luna, Terra, Sol and GPT-6 Astra.
OpenAI GPT-6 Astra System Card - Misaligned Outcomes (Confirmation Policy) - Unauthorized Transactions (Misaligned outcome rate (%)): leader GPT-6 (4.3), 4 models
Table 8 (misaligned outcomes in realistic work environments) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of adversarial computer- and browser-use workplace tasks ending in an unauthorized transaction, with the confirmation policy that is the default for users. Percent, lower is better; GPT-5.6 Luna, Terra, Sol and GPT-6 Astra.
OpenAI GPT-6 Astra System Card - Misaligned Outcomes (Confirmation Policy) - Data Exfiltration (Misaligned outcome rate (%)): leader GPT-6 (4.5), 4 models
Table 8 (misaligned outcomes in realistic work environments) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of adversarial computer- and browser-use workplace tasks ending in data exfiltration, with the confirmation policy that is the default for users. Percent, lower is better; GPT-5.6 Luna, Terra, Sol and GPT-6 Astra.
OpenAI GPT-6 Astra System Card - Misaligned Outcomes (Confirmation Policy) - Service Disruption (Misaligned outcome rate (%)): leader GPT-5.6 Terra (0.3), 4 models
Table 8 (misaligned outcomes in realistic work environments) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of adversarial computer- and browser-use workplace tasks ending in service disruption, with the confirmation policy that is the default for users. Percent, lower is better; GPT-5.6 Luna, Terra, Sol and GPT-6 Astra.
OpenAI GPT-6 Astra System Card - Misaligned Outcomes (Confirmation Policy) - Security Weakening (Misaligned outcome rate (%)): leader GPT-6 (0.0), 4 models
Table 8 (misaligned outcomes in realistic work environments) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of adversarial computer- and browser-use workplace tasks ending in weakened security, with the confirmation policy that is the default for users. Percent, lower is better; GPT-5.6 Luna, Terra, Sol and GPT-6 Astra.
OpenAI GPT-6 Astra System Card - Misaligned Outcomes (Confirmation Policy) - Unauthorized External Communication (Misaligned outcome rate (%)): leader GPT-6 (1.4), 4 models
Table 8 (misaligned outcomes in realistic work environments) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of adversarial computer- and browser-use workplace tasks ending in unauthorized external communication, with the confirmation policy that is the default for users. Percent, lower is better; GPT-5.6 Luna, Terra, Sol and GPT-6 Astra.
OpenAI GPT-6 Astra System Card - Misaligned Outcomes (Confirmation Policy) - Circumventing Restrictions (Misaligned outcome rate (%)): leader GPT-5.6 Luna (0.0), 4 models
Table 8 (misaligned outcomes in realistic work environments) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of adversarial computer- and browser-use workplace tasks ending in circumvented restrictions, with the confirmation policy that is the default for users. Percent, lower is better; GPT-5.6 Luna, Terra, Sol and GPT-6 Astra.
OpenAI GPT-6 Astra System Card - Biology Refusal - Severe (Safe rate (%)): leader GPT-6 (99.8), 3 models
Table 19 (Biology Model Refusal Evaluation) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of safe responses to severe-risk biology prompts, model responses only without monitors. Percent (the card prints 0-1), higher is better. The GPT-5.5 Thinking and GPT-5.6 Sol cells repeat the GPT-5.6 card's values.
OpenAI GPT-6 Astra System Card - Biology Refusal - Dual Use (Safe rate (%)): leader GPT-6 (97.0), 3 models
Table 19 (Biology Model Refusal Evaluation) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of safe responses to dual-use biology prompts, model responses only without monitors. Percent (the card prints 0-1), higher is better. The GPT-5.5 Thinking and GPT-5.6 Sol cells repeat the GPT-5.6 card's values.
OpenAI GPT-6 Astra System Card - Biology Refusal - Benign (Not overrefuse rate (%)): leader GPT-5.6 Sol (98.9), 3 models
Table 19 (Biology Model Refusal Evaluation) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of benign biology prompts answered without over-refusal, model responses only without monitors. Percent (the card prints 0-1), higher is better. The GPT-5.5 Thinking and GPT-5.6 Sol cells repeat the GPT-5.6 card's values.
OpenAI GPT-6 Astra System Card - Challenging Prompts - Violent Illicit Behavior (Safe completion rate (%)): leader GPT-6 (99.0), 4 models
Table 1 (production benchmarks with challenging prompts) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate, i.e. responses and actions not disallowed by policy, on deliberately difficult violent illicit behavior prompts, run without system-level safeguards. Percent (the card prints 0-1), higher is better.
OpenAI GPT-6 Astra System Card - Challenging Prompts - Non-Violent Illicit Behavior (Safe completion rate (%)): leader GPT-6 (99.7), 4 models
Table 1 (production benchmarks with challenging prompts) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate, i.e. responses and actions not disallowed by policy, on deliberately difficult non-violent illicit behavior prompts, run without system-level safeguards. Percent (the card prints 0-1), higher is better.
OpenAI GPT-6 Astra System Card - Challenging Prompts - Extremism (Safe completion rate (%)): leader GPT-6 (98.1), 4 models
Table 1 (production benchmarks with challenging prompts) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate, i.e. responses and actions not disallowed by policy, on deliberately difficult extremism prompts, run without system-level safeguards. Percent (the card prints 0-1), higher is better.
OpenAI GPT-6 Astra System Card - Challenging Prompts - Hate (Safe completion rate (%)): leader GPT-5.4 (Thinking) (100.0), 4 models
Table 1 (production benchmarks with challenging prompts) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate, i.e. responses and actions not disallowed by policy, on deliberately difficult hate prompts, run without system-level safeguards. Percent (the card prints 0-1), higher is better.
OpenAI GPT-6 Astra System Card - Challenging Prompts - Self-Harm (Safe completion rate (%)): leader GPT-6 (99.2), 4 models
Table 1 (production benchmarks with challenging prompts) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate, i.e. responses and actions not disallowed by policy, on deliberately difficult self-harm prompts, run without system-level safeguards. Percent (the card prints 0-1), higher is better.
OpenAI GPT-6 Astra System Card - Challenging Prompts - Gore (Safe completion rate (%)): leader GPT-6 (89.8), 4 models
Table 1 (production benchmarks with challenging prompts) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate, i.e. responses and actions not disallowed by policy, on deliberately difficult gore prompts, run without system-level safeguards. Percent (the card prints 0-1), higher is better.
OpenAI GPT-6 Astra System Card - Challenging Prompts - Sexual (Safe completion rate (%)): leader GPT-6 (98.0), 4 models
Table 1 (production benchmarks with challenging prompts) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate, i.e. responses and actions not disallowed by policy, on deliberately difficult sexual prompts, run without system-level safeguards. Percent (the card prints 0-1), higher is better.
OpenAI GPT-6 Astra System Card - Challenging Prompts - Sexual/Minors (Safe completion rate (%)): leader GPT-6 (97.4), 4 models
Table 1 (production benchmarks with challenging prompts) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate, i.e. responses and actions not disallowed by policy, on deliberately difficult sexual/minors prompts, run without system-level safeguards. Percent (the card prints 0-1), higher is better.
OpenAI GPT-6 Astra System Card - U18 - Age-Restricted Goods and Dangerous Activities (Score (%)): leader GPT-6 (91.8), 4 models
Table 2 (U18 evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): adherence to teen-specific safety standards on difficult age-restricted goods, services, and dangerous challenges or activities cases, assessed with U18 protections in place. Percent (the card prints 0-1), higher is better. Long-tail cases, not estimates of production frequency.
OpenAI GPT-6 Astra System Card - U18 - Sexual Content (Score (%)): leader GPT-6 (99.1), 4 models
Table 2 (U18 evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): adherence to teen-specific safety standards on difficult sexual content cases, assessed with U18 protections in place. Percent (the card prints 0-1), higher is better. Long-tail cases, not estimates of production frequency.
OpenAI GPT-6 Astra System Card - U18 - Eating Disorders (Score (%)): leader GPT-6 (92.1), 4 models
Table 2 (U18 evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): adherence to teen-specific safety standards on difficult eating disorders cases, assessed with U18 protections in place. Percent (the card prints 0-1), higher is better. Long-tail cases, not estimates of production frequency.
OpenAI GPT-6 Astra System Card - U18 - Emotional Reliance (Score (%)): leader GPT-6 (94.4), 4 models
Table 2 (U18 evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): adherence to teen-specific safety standards on difficult emotional reliance cases, assessed with U18 protections in place. Percent (the card prints 0-1), higher is better. Long-tail cases, not estimates of production frequency.
OpenAI GPT-6 Astra System Card - U18 - Self-Harm (Score (%)): leader GPT-6 (99.5), 4 models
Table 2 (U18 evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): adherence to teen-specific safety standards on difficult self-harm cases, assessed with U18 protections in place. Percent (the card prints 0-1), higher is better. Long-tail cases, not estimates of production frequency.
OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Age-Restricted Actions (Safe completion rate (%)): leader GPT-6 (81.1), 2 models
Table 3 (agentic safe completion evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate judging the model's actions as well as its final response, on challenging Codex production conversations about age-restricted actions. Percent (the card prints 0-1), higher is better. GPT-5.6 Sol and GPT-6 Astra only.
OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Non-Violent Wrongdoing (Safe completion rate (%)): leader GPT-6 (95.4), 2 models
Table 3 (agentic safe completion evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate judging the model's actions as well as its final response, on challenging Codex production conversations about non-violent wrongdoing. Percent (the card prints 0-1), higher is better. GPT-5.6 Sol and GPT-6 Astra only.
OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Violent Wrongdoing (Safe completion rate (%)): leader GPT-6 (90.7), 2 models
Table 3 (agentic safe completion evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate judging the model's actions as well as its final response, on challenging Codex production conversations about violent wrongdoing. Percent (the card prints 0-1), higher is better. GPT-5.6 Sol and GPT-6 Astra only.
OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Sensitive Personal Data (Safe completion rate (%)): leader GPT-5.6 Sol (76.5), 2 models
Table 3 (agentic safe completion evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate judging the model's actions as well as its final response, on challenging Codex production conversations involving sensitive personal data. Percent (the card prints 0-1), higher is better. GPT-5.6 Sol and GPT-6 Astra only.
OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Codex Self-Harm (Safe completion rate (%)): leader GPT-6 (92.0), 2 models
Table 3 (agentic safe completion evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate judging the model's actions as well as its final response, on challenging Codex production conversations about self-harm. Percent (the card prints 0-1), higher is better. GPT-5.6 Sol and GPT-6 Astra only.
OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Human Red-Teaming Codex (Safe completion rate (%)): leader GPT-6 (97.7), 2 models
Table 3 (agentic safe completion evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate judging the model's actions as well as its final response, on human red-teaming conversations in Codex. Percent (the card prints 0-1), higher is better. GPT-5.6 Sol and GPT-6 Astra only.
OpenAI GPT-6 Astra System Card - Agentic Safe Completions - Human Red-Teaming Chat Plugins (Safe completion rate (%)): leader GPT-6 (100.0), 2 models
Table 3 (agentic safe completion evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate judging the model's actions as well as its final response, on human red-teaming ChatGPT conversations with Plugin services connected. Percent (the card prints 0-1), higher is better. GPT-5.6 Sol and GPT-6 Astra only.
OpenAI GPT-6 Astra System Card - Image Input - Hate (Safe completion rate (%)): leader GPT-5.5 (99.9), 4 models
Table 4 (image input evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate on disallowed hate requests that combine text and images, the suite introduced with ChatGPT Agent. Percent (the card prints 0-1), higher is better. Scores sit near the ceiling.
OpenAI GPT-6 Astra System Card - Image Input - Extremism (Safe completion rate (%)): leader GPT-6 (99.1), 4 models
Table 4 (image input evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate on disallowed extremism requests that combine text and images, the suite introduced with ChatGPT Agent. Percent (the card prints 0-1), higher is better. Scores sit near the ceiling.
OpenAI GPT-6 Astra System Card - Image Input - Self-Harm (Safe completion rate (%)): leader GPT-6 (99.7), 4 models
Table 4 (image input evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate on disallowed self-harm requests that combine text and images, the suite introduced with ChatGPT Agent. Percent (the card prints 0-1), higher is better. Scores sit near the ceiling.
OpenAI GPT-6 Astra System Card - Image Input - Erotic (Safe completion rate (%)): leader GPT-6 (100.0), 4 models
Table 4 (image input evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): safe completion rate on disallowed erotic requests that combine text and images, the suite introduced with ChatGPT Agent. Percent (the card prints 0-1), higher is better. Scores sit near the ceiling.
OpenAI GPT-6 Astra System Card - Static Jailbreaks - Bio High Risk (Jailbreak refusal rate (%)): leader GPT-6 (97.3), 3 models
Table 5 (static jailbreak evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of fixed adversarial high-risk biology jailbreak prompts the model refuses, without production safeguards. Percent (the card prints 0-1), higher is better. Attacker models generated the prompts against earlier GPT models, so GPT-5.5 Thinking and GPT-5.6 Sol scores are depressed by construction.
OpenAI GPT-6 Astra System Card - Static Jailbreaks - Bio Severe (Jailbreak refusal rate (%)): leader GPT-6 (98.2), 3 models
Table 5 (static jailbreak evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of fixed adversarial severe biology jailbreak prompts the model refuses, without production safeguards. Percent (the card prints 0-1), higher is better. Attacker models generated the prompts against earlier GPT models, so GPT-5.5 Thinking and GPT-5.6 Sol scores are depressed by construction.
OpenAI GPT-6 Astra System Card - Static Jailbreaks - Violence Moderate (Jailbreak refusal rate (%)): leader GPT-6 (94.7), 3 models
Table 5 (static jailbreak evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of fixed adversarial moderate violence jailbreak prompts the model refuses, without production safeguards. Percent (the card prints 0-1), higher is better. Attacker models generated the prompts against earlier GPT models, so GPT-5.5 Thinking and GPT-5.6 Sol scores are depressed by construction.
OpenAI GPT-6 Astra System Card - Static Jailbreaks - Violence Severe (Jailbreak refusal rate (%)): leader GPT-6 (98.3), 3 models
Table 5 (static jailbreak evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of fixed adversarial severe violence jailbreak prompts the model refuses, without production safeguards. Percent (the card prints 0-1), higher is better. Attacker models generated the prompts against earlier GPT models, so GPT-5.5 Thinking and GPT-5.6 Sol scores are depressed by construction.
OpenAI GPT-6 Astra System Card - Static Jailbreaks - Cyber (Jailbreak refusal rate (%)): leader GPT-6 (91.5), 3 models
Table 5 (static jailbreak evaluations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of fixed adversarial cybersecurity jailbreak prompts the model refuses, without production safeguards. Percent (the card prints 0-1), higher is better. Attacker models generated the prompts against earlier GPT models, so GPT-5.5 Thinking and GPT-5.6 Sol scores are depressed by construction.
OpenAI GPT-6 Astra System Card - Adversarial User Simulations - Mental Health (Safe message rate (%)): leader GPT-6 (100.0), 4 models
Table 7 (dynamic benchmarks with adversarial user simulations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of assistant messages that do not violate safety policy across evolving simulated mental health conversations with adversarial users. Percent (the card prints 0-1), higher is better.
OpenAI GPT-6 Astra System Card - Adversarial User Simulations - Emotional Reliance (Safe message rate (%)): leader GPT-6 (99.3), 4 models
Table 7 (dynamic benchmarks with adversarial user simulations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of assistant messages that do not violate safety policy across evolving simulated emotional reliance conversations with adversarial users. Percent (the card prints 0-1), higher is better.
OpenAI GPT-6 Astra System Card - Adversarial User Simulations - Self-Harm (Safe message rate (%)): leader GPT-6 (98.9), 4 models
Table 7 (dynamic benchmarks with adversarial user simulations) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): share of assistant messages that do not violate safety policy across evolving simulated self-harm conversations with adversarial users. Percent (the card prints 0-1), higher is better.
OpenAI GPT-6 Astra System Card - SHP2 Protein Function Prediction (Mean R² (three assay datasets)): leader GPT-6 (0.4), 2 models
Table 13 of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): models predict held-out SHP2 mutant assay results from structures and partial assay data. Mean R-squared across three assay datasets, higher is better (indicative threshold 0.60). GPT-6 Astra's 0.40 is its production score; a helpful-only variant scored 0.35.
OpenAI GPT-6 Astra System Card - Cyber Safety - Production Chat (Safety score (0-1)): leader GPT-5.6 Sol (0.983), 3 models
Table 20 (cybersecurity safety evaluation) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): average safety score from 0 (clearly disallowed rollout) to 1 (safe and allowed), macro-averaged over three production chat sources. Higher is better; the card's 0-1 scale is kept.
OpenAI GPT-6 Astra System Card - Cyber Safety - Synthetic Agentic Environments (Safety score (0-1)): leader GPT-5.6 Sol (0.998), 3 models
Table 20 (cybersecurity safety evaluation) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): average safety score from 0 (clearly disallowed rollout) to 1 (safe and allowed), macro-averaged over two synthetic agentic environment sources. Higher is better; the card's 0-1 scale is kept.
OpenAI GPT-6 Astra System Card - Cyber Safety - Semi-Synthetic Agentic Environments (Safety score (0-1)): leader GPT-6 (0.997), 3 models
Table 20 (cybersecurity safety evaluation) of OpenAI's GPT-6 Astra System Card (Deployment Safety Hub, 2026-09-03): average safety score from 0 (clearly disallowed rollout) to 1 (safe and allowed), macro-averaged over three semi-synthetic agentic environments seeded from cyber-relevant production traces. Higher is better; the card's 0-1 scale is kept.
OpenAI GPT-5.6 System Card - Prompt Injection - Connectors (Prompt-injection robustness rate (%)): leader GPT-5.5 (100.0), 7 models
Table 5 (prompt injection) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): robustness to known prompt-injection attacks embedded in connector tool outputs that try to override system, developer or user instructions. Percent (the card prints 0-1), higher is better. Seven GPT models from GPT-5.1 Thinking to GPT-5.6 Luna.
OpenAI GPT-5.6 System Card - Prompt Injection - Search and Function Calling (Prompt-injection robustness rate (%)): leader GPT-5.6 Terra (94.6), 6 models
Table 5 (prompt injection) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): robustness to stronger prompt-injection attacks embedded in search and function-calling tool outputs. Percent (the card prints 0-1), higher is better. GPT-5.5 has no value on this row.
OpenAI GPT-5.6 System Card - Biology Refusal - Severe (Not unsafe rate (%)): leader GPT-5.4 (Thinking) (96.1), 6 models
Biology Model Refusal Evaluation table of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): share of responses to severe-risk biology prompts graded not unsafe, model responses only without monitors. Percent (the card prints 0-1), higher is better. GPT-5.2, 5.4 and 5.5 Thinking and GPT-5.6 Sol, Terra and Luna.
OpenAI GPT-5.6 System Card - Biology Refusal - Dual Use (Not unsafe rate (%)): leader GPT-5.4 (Thinking) (95.5), 6 models
Biology Model Refusal Evaluation table of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): share of responses to dual-use biology prompts graded not unsafe, model responses only without monitors. Percent (the card prints 0-1), higher is better. GPT-5.2, 5.4 and 5.5 Thinking and GPT-5.6 Sol, Terra and Luna.
OpenAI GPT-5.6 System Card - Biology Refusal - Benign (Not overrefuse rate (%)): leader GPT-5.6 Sol (98.9), 6 models
Biology Model Refusal Evaluation table of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): share of responses to benign biology prompts graded not over-refused, model responses only without monitors. Percent (the card prints 0-1), higher is better. GPT-5.2, 5.4 and 5.5 Thinking and GPT-5.6 Sol, Terra and Luna.
OpenAI GPT-5.6 System Card - Challenging Prompts - Violent Illicit Behavior (not_unsafe rate (%)): leader GPT-5.2 (Thinking) (97.5), 7 models
Table 1 (production benchmarks with challenging prompts) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): not_unsafe rate on deliberately difficult violent illicit behavior prompts drawn from production-like traffic. Percent (the card prints 0-1), higher is better. Seven GPT models from GPT-5.1 Thinking to GPT-5.6 Luna.
OpenAI GPT-5.6 System Card - Challenging Prompts - Non-Violent Illicit Behavior (not_unsafe rate (%)): leader GPT-5.4 (Thinking) (100.0), 7 models
Table 1 (production benchmarks with challenging prompts) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): not_unsafe rate on deliberately difficult non-violent illicit behavior prompts drawn from production-like traffic. Percent (the card prints 0-1), higher is better. Seven GPT models from GPT-5.1 Thinking to GPT-5.6 Luna.
OpenAI GPT-5.6 System Card - Challenging Prompts - Extremism (not_unsafe rate (%)): leader GPT-5.1 (Thinking) (100.0), 7 models
Table 1 (production benchmarks with challenging prompts) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): not_unsafe rate on deliberately difficult extremism prompts drawn from production-like traffic. Percent (the card prints 0-1), higher is better. Seven GPT models from GPT-5.1 Thinking to GPT-5.6 Luna.
OpenAI GPT-5.6 System Card - Challenging Prompts - Hate (not_unsafe rate (%)): leader GPT-5.5 (100.0), 7 models
Table 1 (production benchmarks with challenging prompts) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): not_unsafe rate on deliberately difficult hate prompts drawn from production-like traffic. Percent (the card prints 0-1), higher is better. Seven GPT models from GPT-5.1 Thinking to GPT-5.6 Luna.
OpenAI GPT-5.6 System Card - Challenging Prompts - Self-Harm (not_unsafe rate (%)): leader GPT-5.4 (Thinking) (98.7), 7 models
Table 1 (production benchmarks with challenging prompts) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): not_unsafe rate on deliberately difficult self-harm prompts drawn from production-like traffic. Percent (the card prints 0-1), higher is better. Seven GPT models from GPT-5.1 Thinking to GPT-5.6 Luna.
OpenAI GPT-5.6 System Card - Challenging Prompts - Gore (not_unsafe rate (%)): leader GPT-5.2 (Thinking) (87.7), 7 models
Table 1 (production benchmarks with challenging prompts) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): not_unsafe rate on deliberately difficult gore prompts drawn from production-like traffic. Percent (the card prints 0-1), higher is better. Seven GPT models from GPT-5.1 Thinking to GPT-5.6 Luna.
OpenAI GPT-5.6 System Card - Challenging Prompts - Sexual (not_unsafe rate (%)): leader GPT-5.6 Terra (96.6), 7 models
Table 1 (production benchmarks with challenging prompts) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): not_unsafe rate on deliberately difficult sexual prompts drawn from production-like traffic. Percent (the card prints 0-1), higher is better. Seven GPT models from GPT-5.1 Thinking to GPT-5.6 Luna.
OpenAI GPT-5.6 System Card - Challenging Prompts - Sexual/Minors (not_unsafe rate (%)): leader GPT-5.6 Terra (97.4), 7 models
Table 1 (production benchmarks with challenging prompts) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): not_unsafe rate on deliberately difficult sexual/minors prompts drawn from production-like traffic. Percent (the card prints 0-1), higher is better. Seven GPT models from GPT-5.1 Thinking to GPT-5.6 Luna.
OpenAI GPT-5.6 System Card - Image Input - Hate (not_unsafe rate (%)): leader GPT-5.5 (99.9), 7 models
Table 2 (image input evaluations) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): not_unsafe rate on disallowed hate requests that combine text and images. Percent (the card prints 0-1), higher is better. Seven GPT models; scores sit near the ceiling.
OpenAI GPT-5.6 System Card - Image Input - Extremism (not_unsafe rate (%)): leader GPT-5.4 (Thinking) (99.5), 7 models
Table 2 (image input evaluations) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): not_unsafe rate on disallowed extremism requests that combine text and images. Percent (the card prints 0-1), higher is better. Seven GPT models; scores sit near the ceiling.
OpenAI GPT-5.6 System Card - Image Input - Self-Harm (not_unsafe rate (%)): leader GPT-5.4 (Thinking) (99.9), 7 models
Table 2 (image input evaluations) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): not_unsafe rate on disallowed self-harm requests that combine text and images. Percent (the card prints 0-1), higher is better. Seven GPT models; scores sit near the ceiling.
OpenAI GPT-5.6 System Card - Image Input - Erotic (not_unsafe rate (%)): leader GPT-5.1 (Thinking) (99.9), 7 models
Table 2 (image input evaluations) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): not_unsafe rate on disallowed erotic requests that combine text and images. Percent (the card prints 0-1), higher is better. Seven GPT models; scores sit near the ceiling.
OpenAI GPT-5.6 System Card - Destructive Actions - Avoidance Only (Overwrite-avoidance rate (%)): leader GPT-5.5 (88.0), 4 models
Table 3 (avoiding accidental data-destructive actions) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): share of tasks in which the model avoids overwriting user changes and data adversarially injected into the task environment. Percent (the card prints 0-1), higher is better. GPT-5.5 and GPT-5.6 Sol, Terra and Luna.
OpenAI GPT-5.6 System Card - Destructive Actions - Avoidance and Correctness (Avoidance and task-correctness rate (%)): leader GPT-5.5 (44.0), 4 models
Table 3 (avoiding accidental data-destructive actions) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): combined metric, the share of challenging tasks completed correctly without overwriting undesired data. Percent (the card prints 0-1), higher is better. GPT-5.5 and GPT-5.6 Sol, Terra and Luna.
OpenAI GPT-5.6 System Card - GPT-Red Prompt Injection - Direct (Attack success rate (%)): leader GPT-5.6 Sol (0.051), 3 models
GPT-Red table of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09), added 2026-08-03: average attack-attempt success rate of GPT-Red, OpenAI's RL-trained automated red-teamer, in direct (chat-based) instruction-hierarchy environments. Percent as printed (e.g. 0.051%), lower is better; GPT-5.6 Sol, Terra and Luna only.
OpenAI GPT-5.6 System Card - GPT-Red Prompt Injection - Indirect (Attack success rate (%)): leader GPT-5.6 Luna (2.94), 3 models
GPT-Red table of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09), added 2026-08-03: average attack-attempt success rate of GPT-Red, OpenAI's RL-trained automated red-teamer, for indirect (agentic) injections through third-party tools. Percent as printed, lower is better; GPT-5.6 Sol, Terra and Luna only.
OpenAI GPT-5.6 System Card - Adversarial User Simulations - Mental Health (not_unsafe message rate (%)): leader GPT-5.6 Sol (99.1), 7 models
Table 7 (dynamic benchmarks with adversarial user simulations) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): share of assistant messages that are not unsafe across evolving simulated mental health conversations with adversarial users. Percent (the card prints 0-1), higher is better.
OpenAI GPT-5.6 System Card - Adversarial User Simulations - Emotional Reliance (not_unsafe message rate (%)): leader GPT-5.4 (Thinking) (98.5), 7 models
Table 7 (dynamic benchmarks with adversarial user simulations) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): share of assistant messages that are not unsafe across evolving simulated emotional reliance conversations with adversarial users. Percent (the card prints 0-1), higher is better.
OpenAI GPT-5.6 System Card - Adversarial User Simulations - Self-Harm (not_unsafe message rate (%)): leader GPT-5.4 (Thinking) (97.7), 7 models
Table 7 (dynamic benchmarks with adversarial user simulations) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): share of assistant messages that are not unsafe across evolving simulated self-harm conversations with adversarial users. Percent (the card prints 0-1), higher is better.
OpenAI GPT-5.6 System Card - Cyber Safety - Production Data (Safety score (0-1)): leader GPT-5.6 Terra (0.987), 6 models
Table 15 (cybersecurity safety evaluation) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): average cyber safety score on production data, from 0 (clearly disallowed rollout) to 1 (safe and allowed). Higher is better; the card's 0-1 scale is kept. GPT-5.3 Codex, GPT-5.4 Thinking, GPT-5.5 and GPT-5.6 Sol, Terra and Luna.
OpenAI GPT-5.6 System Card - Cyber Safety - Synthetic Data (Safety score (0-1)): leader GPT-5.6 Luna (1.0), 6 models
Table 15 (cybersecurity safety evaluation) of OpenAI's GPT-5.6 System Card (Deployment Safety Hub, 2026-07-09): average cyber safety score on synthetic data, from 0 (clearly disallowed rollout) to 1 (safe and allowed). Higher is better; the card's 0-1 scale is kept. GPT-5.3 Codex, GPT-5.4 Thinking, GPT-5.5 and GPT-5.6 Sol, Terra and Luna.
OpenAI GPT-6 Astra Launch - BenchCAD (with tools) (Geometric-overlap score (%)): leader GPT-6 (95.9), 2 models
BenchCAD row of the professional table in OpenAI's GPT-6 Astra launch post (2026-09-03): models reconstruct 3D objects from multi-view renders by writing CAD code, run with tools. Geometric-overlap score in percent, higher is better. Only OpenAI's runs (GPT-6 Astra, GPT-5.6 Sol) are kept; footnote 5 marks the Claude values as Anthropic-reported under a modified protocol.
OpenAI GPT-6 Astra Launch - OpenScore String Quartets (1 - OMR-NED) (1 - OMR-NED (0-1)): leader GPT-6 (0.84), 2 models
OpenScore String Quartets row of the professional table in OpenAI's GPT-6 Astra launch post (2026-09-03): optical music recognition of string-quartet scores, reported as 1 minus the normalized edit distance to the reference (OMR-NED). 0-1 scale, higher is better; only GPT-6 Astra and GPT-5.6 Sol have values.
OpenAI GPT-6 Astra Launch - Internal Design Tasks (Score (%)): leader GPT-6 (50.0), 3 models
OpenAI's internal design task set, professional table of OpenAI's GPT-6 Astra launch post (2026-09-03). The tasks and grading are not published. Score in percent, higher is better. Scores are the maximum at any reasoning effort. Non-OpenAI models appear only where the post prints a value.
OpenAI GPT-6 Astra Launch - Internal Data Science Tasks (Score (%)): leader GPT-6 (40.9), 3 models
OpenAI's internal data science task set, professional table of OpenAI's GPT-6 Astra launch post (2026-09-03). The tasks and grading are not published. Score in percent, higher is better. Scores are the maximum at any reasoning effort. Non-OpenAI models appear only where the post prints a value.
OpenAI GPT-6 Astra Launch - Internal Database Migration Tasks (Score (%)): leader GPT-6 (63.9), 4 models
OpenAI's internal database migration task set, coding table of OpenAI's GPT-6 Astra launch post (2026-09-03). The tasks and grading are not published. Score in percent, higher is better. Scores are the maximum at any reasoning effort. Non-OpenAI models appear only where the post prints a value.
OpenAI GPT-6 Astra Launch - GeneBench Pro (Score (%)): leader GPT-6 (37.1), 2 models
GeneBench Pro v13 row of the science and health table in OpenAI's GPT-6 Astra launch post (2026-09-03). Score in percent, higher is better. Claude Fable 5 and 5.1 are omitted because they refuse most questions (footnote 12). The GPT-5.6 launch post used an earlier version (GPT-5.6 Sol 28.7 there vs 32.3 here), so each launch keeps its own board.
OpenAI GPT-6 Astra Launch - MedChemBench (Internal) (Score (%)): leader GPT-6 (49.3), 2 models
OpenAI's internal medicinal-chemistry benchmark (MedChemBench), science and health table of OpenAI's GPT-6 Astra launch post (2026-09-03). Score in percent, higher is better. Claude Fable 5 and 5.1 are omitted because they refuse most questions (footnote 12); only GPT-6 Astra and GPT-5.6 Sol have values.
OpenAI GPT-6 Astra Launch - LifeSciBench (Score (%)): leader GPT-6 (60.3), 2 models
LifeSciBench Gold v1 row of the science and health table in OpenAI's GPT-6 Astra launch post (2026-09-03). Score in percent, higher is better; only GPT-6 Astra and GPT-5.6 Sol have values. Separate from the GPT-5.6 launch and GPT-Rosalind LifeSciBench boards, whose version and metric differ.
OpenAI GPT-6 Astra Launch - SRE-Bench (pass@1 (%)): leader GPT-6 (88.0), 3 models
SRE-Bench row of the cybersecurity table in OpenAI's GPT-6 Astra launch post (2026-09-03): models reverse engineer software binaries to recover their core logic without source code. pass@1 in percent (tasks solved in a single attempt), higher is better. Compares GPT-6 Astra, GPT-5.6 Sol and Claude Opus 5.
OpenAI GPT-6 Astra Launch - Internal Computer Use Safety Benchmark (Unintended-consequence rate (%)): leader GPT-6 (2.4), 5 models
Alignment table of OpenAI's GPT-6 Astra launch post (2026-09-03): rate of unintended consequences on computer-use tasks adversarially selected to elicit misbehavior. Percent, lower is better. Footnote 16: third-party models run in a simpler research setup than Codex, and the no-confirmation scenario is an internal research configuration.
OpenAI GPT-6 Astra Launch - Internal Computer Use Safety Benchmark (with AutoReview) (Unintended-consequence rate (%)): leader GPT-6 (1.8), 2 models
Alignment table of OpenAI's GPT-6 Astra launch post (2026-09-03): the internal computer-use safety benchmark rerun with Codex Auto-Review enabled, reporting the rate of unintended consequences. Percent, lower is better; only GPT-6 Astra and GPT-5.6 Sol have values.
OpenAI GPT-6 Astra Launch - Internal Circumvention Benchmark (Circumvention attempt rate (%)): leader GPT-6 (0.0), 2 models
Alignment table of OpenAI's GPT-6 Astra launch post (2026-09-03): how often a model tries to circumvent a Codex Auto-Review denial in knowledge-work tasks, even when Auto-Review is configured to be evadable and the task is otherwise impossible. Percent, lower is better.
OpenAI GPT-6 Astra Launch - ExploitGym Honeypot (Honeypot attack rate (%)): leader GPT-6 (0.0), 2 models
Alignment table of OpenAI's GPT-6 Astra launch post (2026-09-03): share of difficult ExploitGym problems in which the model exploits a planted honeypot instead of declining (system card section on declining honeypots). Percent, lower is better; only GPT-6 Astra and GPT-5.6 Sol have values.
OpenAI GPT-6 Astra Launch - Internal Capability Hallucination Benchmark (Capability-hallucination rate (%)): leader GPT-6 (4.2), 2 models
Alignment table of OpenAI's GPT-6 Astra launch post (2026-09-03): OpenAI's internal capability-hallucination evaluation, the rate of inaccurate or misleading claims a model makes about its own capabilities and affordances. Percent, lower is better; only GPT-6 Astra and GPT-5.6 Sol have values.
OpenAI GPT-6 Astra Launch - OpenAI MRCR v2 8-needle 256K-512K (Score (%)): leader GPT-6 (100.0), 2 models
OpenAI MRCR v2 long-context multi-round coreference test with 8 needles and prompts of 256K-512K tokens, long context table of OpenAI's GPT-6 Astra launch post (2026-09-03). Score in percent, higher is better; only GPT-6 Astra and GPT-5.6 Sol have values. Kept per launch, separate from the GPT-5.6 launch board.
OpenAI GPT-6 Astra Launch - OpenAI MRCR v2 8-needle 512K-1M (Score (%)): leader GPT-6 (96.3), 2 models
OpenAI MRCR v2 long-context multi-round coreference test with 8 needles and prompts of 512K-1M tokens, long context table of OpenAI's GPT-6 Astra launch post (2026-09-03). Score in percent, higher is better; only GPT-6 Astra and GPT-5.6 Sol have values. Kept per launch, separate from the GPT-5.6 launch board.
M-GATE - Grammatical Error Detection (Mean MCC over 30 languages (-1 to 1; temperature 0)): leader Gemini 3.1 Pro (Preview) (0.36), 82 models
Binary grammatical-error detection on linguist-written, adversarially selected sentences in 30 languages, from M-GATE (arXiv 2608.03803v1, Appendix Table J.1, frozen July 2026 snapshot). Score is the mean Matthews correlation coefficient across languages at temperature 0 (-1 to 1, 0 = chance; higher is better); each reasoning configuration is a separate row.
M-GATE - Round-Trip Translation (Hard Subset) (Mean normalized judge score over 29 target languages (0-1; hard subset)): leader GPT-5.5 (High) (0.82), 82 models
Round-trip English-to-target-to-English translation of M-GATE's fixed 50-sentence hard subset across 29 target languages, scored for meaning preservation by a three-provider small-model judge panel (arXiv 2608.03803v1, Appendix Table J.4). Mean normalized judge score from 0 to 1; higher is better. The evaluated model performs both translation directions.
FinRiskAtlas - Domain Knowledge - Overall (Macro-Averaged Accuracy (%)): leader Gemini 3 Flash (Preview) (80.69), 33 models
Unweighted macro-average over 42 financial-risk domain-knowledge task families (4,679 instances) covering concepts, regulations, obligations and risk mechanisms behind financial review, from FinRiskAtlas (arXiv 2608.25325v1, Appendix Table 8). Percent; higher is better; 33 archived model configurations answering zero-shot.
FinRiskAtlas - Domain Knowledge - Risk & Compliance (Macro-Averaged Accuracy (%)): leader Kimi K3 (78.11), 33 models
Macro-averaged accuracy on the Risk & Compliance group of FinRiskAtlas domain-knowledge task families, a subset of the 42-family knowledge layer used for coarse screening of financial-review models (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 8). Percent; higher is better; zero-shot direct answers from 33 archived configurations.
FinRiskAtlas - Domain Knowledge - Trade (Macro-Averaged Accuracy (%)): leader Gemini 3 Flash (Preview) (83.8), 33 models
Macro-averaged accuracy on the Trade group of FinRiskAtlas domain-knowledge task families, a subset of the 42-family knowledge layer used for coarse screening of financial-review models (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 8). Percent; higher is better; zero-shot direct answers from 33 archived configurations.
FinRiskAtlas - Domain Knowledge - FinTech & Security (Macro-Averaged Accuracy (%)): leader Claude Opus 4.7 (86.92), 33 models
Macro-averaged accuracy on the FinTech & Security group of FinRiskAtlas domain-knowledge task families, a subset of the 42-family knowledge layer used for coarse screening of financial-review models (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 8). Percent; higher is better; zero-shot direct answers from 33 archived configurations.
FinRiskAtlas - Domain Knowledge - Commerce & Payment (Macro-Averaged Accuracy (%)): leader Claude Opus 4.7 (86.81), 33 models
Macro-averaged accuracy on the Commerce & Payment group of FinRiskAtlas domain-knowledge task families, a subset of the 42-family knowledge layer used for coarse screening of financial-review models (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 8). Percent; higher is better; zero-shot direct answers from 33 archived configurations.
FinRiskAtlas - Domain Knowledge - Finance & Law (Macro-Averaged Accuracy (%)): leader GPT-5.4 (85.34), 33 models
Macro-averaged accuracy on the Finance & Law group of FinRiskAtlas domain-knowledge task families, a subset of the 42-family knowledge layer used for coarse screening of financial-review models (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 8). Percent; higher is better; zero-shot direct answers from 33 archived configurations.
FinRiskAtlas - Evidence-Grounded Processing - Case Classification (Accuracy (%)): leader Kimi K2.5 (88.09), 33 models
FinRiskAtlas fixed-evidence review operation: routing a full case record to its procedure category from a fixed candidate set, scored by accuracy (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 9). Percent; higher is better. Reported for 33 archived model configurations under zero-shot direct-answer inference.
FinRiskAtlas - Evidence-Grounded Processing - Information Extraction (Field-Level Score (%)): leader GPT-5.6 Luna (85.97), 33 models
FinRiskAtlas fixed-evidence review operation: extracting an ordered list of requested evidence fields from a source record, scored by normalized field-level comparison (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 9). Percent; higher is better. Reported for 33 archived model configurations under zero-shot direct-answer inference.
FinRiskAtlas - Evidence-Grounded Processing - Institution Matching (Accuracy (%)): leader Gemini 3 Flash (Preview) (93.6), 33 models
FinRiskAtlas fixed-evidence review operation: deciding whether two institution descriptions, possibly from different records or languages, denote the same institution, scored by accuracy (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 9). Percent; higher is better. Reported for 33 archived model configurations under zero-shot direct-answer inference.
FinRiskAtlas - Evidence-Grounded Processing - Person Matching (Accuracy (%)): leader Kimi K3 (87.0), 33 models
FinRiskAtlas fixed-evidence review operation: deciding whether two partially overlapping person records denote the same natural person, scored by accuracy (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 9). Percent; higher is better. Reported for 33 archived model configurations under zero-shot direct-answer inference.
FinRiskAtlas - Evidence-Grounded Processing - Quantitative Reasoning (Numerical Accuracy (%)): leader Ling-2.6-1T (86.21), 33 models
FinRiskAtlas fixed-evidence review operation: computing a requested quantity from case figures and the applicable definition or formula, scored by numerical correctness (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 9). Percent; higher is better. Reported for 33 archived model configurations under zero-shot direct-answer inference.
FinRiskAtlas - Applied Review - Risk Classification (Accuracy (%)): leader Gemini 3 Flash (Preview) (76.78), 33 models
FinRiskAtlas fixed-evidence review operation: assigning a case-level risk grade from evidence and the applicable risk-category set, scored by accuracy (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 9). Percent; higher is better. Reported for 33 archived model configurations under zero-shot direct-answer inference.
FinRiskAtlas - Applied Review - Legal-Outcome Prediction (Accuracy (%)): leader Kimi K3 (68.15), 33 models
FinRiskAtlas fixed-evidence review operation: predicting a case disposition from facts and procedural record with the outcome withheld, scored by accuracy (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 9). Percent; higher is better. Reported for 33 archived model configurations under zero-shot direct-answer inference.
FinRiskAtlas - Applied Review - Applicable-Provision Selection (Set-Based Accuracy (%)): leader GPT-5.6 Sol (67.55), 33 models
FinRiskAtlas fixed-evidence review operation: selecting the governing provisions from a candidate set, scored by a set-based accuracy-style measure (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 9). Percent; higher is better. Reported for 33 archived model configurations under zero-shot direct-answer inference.
FinRiskAtlas - Applied Review - Legal-Judgment Generation (Rubric Score (%; DeepSeek-V4-Flash judge)): leader GPT-5.4 (46.36), 32 models
Open-ended FinRiskAtlas review generation: writing a reasoned legal analysis supporting a disposition, scored by a fixed semantic rubric for correctness, coverage and evidence use with DeepSeek-V4-Flash as evaluator (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 10). Percent; higher is better. The evaluator's own self-scored row is excluded, as in the paper's comparisons.
FinRiskAtlas - Applied Review - Decision-View Generation (Rubric Score (%; DeepSeek-V4-Flash judge)): leader DeepSeek V3 (0324) (71.31), 32 models
Open-ended FinRiskAtlas review generation: writing a recommended review opinion for the question presented to the reviewer, scored by a fixed semantic rubric for correctness, coverage and evidence use with DeepSeek-V4-Flash as evaluator (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 10). Percent; higher is better. The evaluator's own self-scored row is excluded, as in the paper's comparisons.
FinRiskAtlas - Applied Review - Disputed-Issue Generation (Rubric Score (%; DeepSeek-V4-Flash judge)): leader DeepSeek V4 Pro (77.12), 32 models
Open-ended FinRiskAtlas review generation: writing a structured list of contested issues in a case with overlapping legal and contractual relationships, scored by a fixed semantic rubric for correctness, coverage and evidence use with DeepSeek-V4-Flash as evaluator (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 10). Percent; higher is better. The evaluator's own self-scored row is excluded, as in the paper's comparisons.
FinRiskAtlas - FinRisk-Ask - Evidence-Request Alignment (ERA (%; DeepSeek-V4-Flash request-alignment judge)): leader Ling-2.6-1T (79.75), 32 models
End-to-end evidence-request alignment (ERA) on FinRisk-Ask's replayed professional review states: credit only when a model chooses to ask and its request matches an expert-verified unresolved evidence need, judged on a three-level scale by DeepSeek-V4-Flash (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 11). Percent; higher is better; the evaluator's self-scored row is excluded.
FinRiskAtlas - FinRisk-Ask - Balanced Action Agreement (BAcc (%)): leader Kimi K3 (59.46), 33 models
Balanced agreement (BAcc) with recorded reviewer Ask-or-Proceed decisions on 680 replayed FinRisk-Ask review states, the mean of Ask recall and Proceed recall (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 11). Percent; higher is better; constant always-ask or always-proceed policies score 50. It measures agreement with one recorded workflow, not a unique optimal policy.
FinRiskAtlas - FinRisk-Ask - Conditional Request Alignment (CRA (%; DeepSeek-V4-Flash request-alignment judge)): leader Qwen 3.7 Max (87.67), 32 models
Conditional request alignment (CRA) in FinRisk-Ask: mean three-level alignment of evidence requests with expert-verified needs, computed only over recorded Ask states where the model chose to ask, judged by DeepSeek-V4-Flash (FinRiskAtlas, arXiv 2608.25325v1, Appendix Table 11). Percent; higher is better; the evaluator's self-scored row is excluded.
MobilePA-Bench - Overall (Weighted success (%; 50/10/20/20)): leader Claude Opus 5 (75.52), 13 models
Weighted overall success of mobile planning agents in an executable sandbox with live app databases across 13 domains and 212 tools: 50% basic tool use, 10% sub-agent collaboration, 20% memory usage and 20% skill usage (MobilePA-Bench, arXiv 2608.23035v2, Table 3). Percent; higher is better.
MobilePA-Bench - Basic Tool Use (Aggregate accuracy (%)): leader Claude Opus 5 (83.85), 13 models
Aggregate accuracy of a mobile planning agent calling tools in MobilePA-Bench's stateful sandbox with strict ordering, permission limits and runtime errors (MobilePA-Bench, arXiv 2608.23035v2, Table 3). Percent; higher is better.
MobilePA-Bench - Sub-agent Collaboration (Routing-and-handoff joint success (%)): leader Gemini 3.1 Pro (Preview) (77.53), 13 models
Joint routing-and-handoff success when a mobile planning agent decomposes a complex task and delegates work to specialized sub-agents in MobilePA-Bench (MobilePA-Bench, arXiv 2608.23035v2, Table 3). Percent; higher is better.
MobilePA-Bench - Memory Usage (End-to-end success (%)): leader Qwen 3.8 Max (64.63), 13 models
End-to-end success on MobilePA-Bench tasks that require recalling stored memories, user profiles and past preferences to resolve implicit requests, verified by database state or behavior (MobilePA-Bench, arXiv 2608.23035v2, Table 3). Percent; higher is better.
MobilePA-Bench - Skill Usage (Joint success (%; SOR and MTSR)): leader Claude Opus 5 (78.0), 13 models
Joint success when a mobile planning agent invokes pre-packaged composite skills instead of planning each step, pooled over the SOR and MTSR settings of MobilePA-Bench (MobilePA-Bench, arXiv 2608.23035v2, Table 3). Percent; higher is better.
Tinybird AI SQL Benchmark - Exactness (Result exactness vs human reference queries (0-100)): leader Claude Opus 4.7 (65.48), 327 models
Tinybird's LLM SQL benchmark has each model write SQL for 50 natural-language analytical questions over a 200M-row GitHub Archive table in Tinybird. Exactness is a 0-100 correctness score from comparing each query's output with human-written reference queries; higher is better. Runs the site still shows with an unvalidated exactness of 0 are omitted.
Tinybird AI SQL Benchmark - Success Rate (Questions answered with a valid query within 3 retries (%)): leader GPT-5.2 Codex (100.0), 332 models
Tinybird's LLM SQL benchmark has each model write SQL for 50 natural-language analytical questions over a 200M-row GitHub Archive table in Tinybird. Success rate is the percentage of questions answered with SQL that executes, allowing up to three retries that feed the SQL API error back to the model; higher is better.
Tinybird AI SQL Benchmark - First-Attempt Success Rate (Questions answered with a valid query on the first attempt (%)): leader GPT-5.2 Codex (100.0), 332 models
Tinybird's LLM SQL benchmark has each model write SQL for 50 natural-language analytical questions over a 200M-row GitHub Archive table in Tinybird. This is the percentage of questions whose first generated query executes without error, before any error-feedback retry; higher is better.
THOR Finding Triage - CW% (Confidence-weighted classification score (%)): leader Gemini 3.7 Flash (72.5), 81 models
Nextron's THOR Finding Triage Benchmark asks models to triage 189 expert-labelled findings from THOR forensic scanner reports. CW% is the confidence-weighted classification score in percent; higher is better. The authors advise reading it with Balanced OTS and Critical Miss Rate; naive always-one-class baselines are excluded.
THOR Finding Triage - Balanced OTS (Class-balanced operational triage score (%)): leader Gemini 3.7 Flash (72.5), 81 models
Nextron's THOR Finding Triage Benchmark asks models to triage 189 expert-labelled findings from THOR forensic scanner reports. Balanced OTS is the class-balanced operational triage score in percent; higher is better. Naive always-one-class baselines are excluded.
THOR Finding Triage - Critical Miss Rate (True positives suppressed as false positives (%, lower is better)): leader Gemini 3.7 Flash (0.0), 81 models
Nextron's THOR Finding Triage Benchmark asks models to triage 189 expert-labelled findings from THOR forensic scanner reports. Critical Miss Rate is the percentage of true-positive findings the model suppresses as false positives; lower is better.
THOR Finding Triage - False Review Load (False positives not suppressed (%, lower is better)): leader Qwen 3.6 Max (22.7), 81 models
Nextron's THOR Finding Triage Benchmark asks models to triage 189 expert-labelled findings from THOR forensic scanner reports. False Review Load is the percentage of false-positive findings the model fails to suppress, leaving review work for analysts; lower is better.
KiloBench - Terminal Bench 2.0 (Task completion in Kilo's agent harness, mean of 5 attempts (%)): leader GPT-6 (79.33), 37 models
Kilo reruns all 89 Terminal-Bench 2.0 tasks in its own production agent harness and averages task completion over five attempts per model; score is completion in percent, higher is better. Same tasks as other Terminal-Bench 2.0 boards under a different harness; Kilo's auto router is excluded.
Next.js Agent Evals (Claude Code) - Success Rate (Evals passed, pass@4 (%)): leader Claude Fable 5.1 (High) (97.0), 9 models
Vercel's Next.js Agent Evals run coding agents on 31 real Next.js tasks; success is pass@4, infrastructure failures are rerun and evals an agent never ran count as failures. This board is the percentage of evals passed by models in the Claude Code harness without bundled docs; higher is better.
Next.js Agent Evals (Claude Code) - Success Rate with AGENTS.md (Evals passed with bundled Next.js docs in AGENTS.md, pass@4 (%)): leader Claude Fable 5.1 (High) (97.0), 9 models
Vercel's Next.js Agent Evals run coding agents on 31 real Next.js tasks; success is pass@4, infrastructure failures are rerun and evals an agent never ran count as failures. This board is the percentage of evals passed by models in the Claude Code harness with Next.js documentation bundled in AGENTS.md; higher is better.
Next.js Agent Evals (Codex) - Success Rate (Evals passed, pass@4 (%)): leader GPT-6 (High) (90.0), 6 models
Vercel's Next.js Agent Evals run coding agents on 31 real Next.js tasks; success is pass@4, infrastructure failures are rerun and evals an agent never ran count as failures. This board is the percentage of evals passed by models in the Codex harness without bundled docs; higher is better.
Next.js Agent Evals (Codex) - Success Rate with AGENTS.md (Evals passed with bundled Next.js docs in AGENTS.md, pass@4 (%)): leader GPT-6 (High) (97.0), 6 models
Vercel's Next.js Agent Evals run coding agents on 31 real Next.js tasks; success is pass@4, infrastructure failures are rerun and evals an agent never ran count as failures. This board is the percentage of evals passed by models in the Codex harness with Next.js documentation bundled in AGENTS.md; higher is better.
Next.js Agent Evals (OpenCode) - Success Rate (Evals passed, pass@4 (%)): leader Gemini 3.8 Flash (90.0), 11 models
Vercel's Next.js Agent Evals run coding agents on 31 real Next.js tasks; success is pass@4, infrastructure failures are rerun and evals an agent never ran count as failures. This board is the percentage of evals passed by models in the OpenCode harness without bundled docs; higher is better.
Next.js Agent Evals (OpenCode) - Success Rate with AGENTS.md (Evals passed with bundled Next.js docs in AGENTS.md, pass@4 (%)): leader Gemini 3.8 Flash (97.0), 11 models
Vercel's Next.js Agent Evals run coding agents on 31 real Next.js tasks; success is pass@4, infrastructure failures are rerun and evals an agent never ran count as failures. This board is the percentage of evals passed by models in the OpenCode harness with Next.js documentation bundled in AGENTS.md; higher is better.
PRISM (1C:Enterprise) - Algorithmic Tasks (A) (Algorithmic BSL tasks fully solved, run in OneScript (%)): leader GPT-5.6 Sol (100.0), 51 models
PRISM (genlab-1c) benchmarks code generation in 1C:Enterprise's BSL language over 35 tasks. Category A is algorithmic tasks in pure BSL executed in OneScript; the score is the percentage of tasks fully solved, higher is better, in coarse steps.
PRISM (1C:Enterprise) - Platform Tasks (B) (1C platform tasks fully solved, run in headless 1C (%)): leader GPT-5.6 Sol (95.0), 51 models
PRISM (genlab-1c) benchmarks code generation in 1C:Enterprise's BSL language over 35 tasks. Category B is platform tasks (FIFO, stock balances, prices, costing, currencies) executed in headless 1C against a synthetic database; the score is the percentage fully solved, higher is better.
Wiz Cyber Model Arena (ADK React) - Overall (Macro average of the four category pass@1 rates (%)): leader Gemini 3.8 Flash Cyber (74.9), 14 models
Wiz Cyber Model Arena scores AI agents on offensive security challenges in four categories (code vulnerabilities, API security, Websec.fr CTFs, cloud security). This board is Overall pass@1, the mean of the four category rates, for models run in the ADK React harness; higher is better.
Wiz Cyber Model Arena (ADK React) - Code Vulnerabilities (Code Vulnerabilities challenges solved, pass@1 (%)): leader Gemini 3.8 Flash Cyber (57.0), 14 models
Wiz Cyber Model Arena scores AI agents on offensive security challenges in four categories (code vulnerabilities, API security, Websec.fr CTFs, cloud security). This board is the percentage of code vulnerability challenges solved on the first attempt (pass@1) by models run in the ADK React harness; higher is better.
Wiz Cyber Model Arena (ADK React) - API Security (API Security challenges solved, pass@1 (%)): leader Gemini 3.8 Flash Cyber (85.2), 14 models
Wiz Cyber Model Arena scores AI agents on offensive security challenges in four categories (code vulnerabilities, API security, Websec.fr CTFs, cloud security). This board is the percentage of API security challenges solved on the first attempt (pass@1) by models run in the ADK React harness; higher is better.
Wiz Cyber Model Arena (ADK React) - Websec.fr CTFs (Websec.fr CTFs challenges solved, pass@1 (%)): leader Gemini 3.8 Flash Cyber (90.8), 14 models
Wiz Cyber Model Arena scores AI agents on offensive security challenges in four categories (code vulnerabilities, API security, Websec.fr CTFs, cloud security). This board is the percentage of Websec.fr web capture-the-flag challenges solved on the first attempt (pass@1) by models run in the ADK React harness; higher is better.
Wiz Cyber Model Arena (ADK React) - Cloud Security (Cloud Security challenges solved, pass@1 (%)): leader Gemini 3.8 Flash Cyber (66.7), 14 models
Wiz Cyber Model Arena scores AI agents on offensive security challenges in four categories (code vulnerabilities, API security, Websec.fr CTFs, cloud security). This board is the percentage of cloud security challenges solved on the first attempt (pass@1) by models run in the ADK React harness; higher is better.
Wiz Cyber Model Arena (Claude Code) - Overall (Macro average of the four category pass@1 rates (%)): leader Claude Opus 5 (71.2), 14 models
Wiz Cyber Model Arena scores AI agents on offensive security challenges in four categories (code vulnerabilities, API security, Websec.fr CTFs, cloud security). This board is Overall pass@1, the mean of the four category rates, for models run in the Claude Code harness; higher is better.
Wiz Cyber Model Arena (Claude Code) - Code Vulnerabilities (Code Vulnerabilities challenges solved, pass@1 (%)): leader Claude Opus 5 (55.2), 14 models
Wiz Cyber Model Arena scores AI agents on offensive security challenges in four categories (code vulnerabilities, API security, Websec.fr CTFs, cloud security). This board is the percentage of code vulnerability challenges solved on the first attempt (pass@1) by models run in the Claude Code harness; higher is better.
Wiz Cyber Model Arena (Claude Code) - API Security (API Security challenges solved, pass@1 (%)): leader Gemini 3.8 Flash Cyber (84.6), 14 models
Wiz Cyber Model Arena scores AI agents on offensive security challenges in four categories (code vulnerabilities, API security, Websec.fr CTFs, cloud security). This board is the percentage of API security challenges solved on the first attempt (pass@1) by models run in the Claude Code harness; higher is better.
Wiz Cyber Model Arena (Claude Code) - Websec.fr CTFs (Websec.fr CTFs challenges solved, pass@1 (%)): leader Claude Opus 5 (81.6), 14 models
Wiz Cyber Model Arena scores AI agents on offensive security challenges in four categories (code vulnerabilities, API security, Websec.fr CTFs, cloud security). This board is the percentage of Websec.fr web capture-the-flag challenges solved on the first attempt (pass@1) by models run in the Claude Code harness; higher is better.
Wiz Cyber Model Arena (Claude Code) - Cloud Security (Cloud Security challenges solved, pass@1 (%)): leader Claude Opus 5 (63.8), 14 models
Wiz Cyber Model Arena scores AI agents on offensive security challenges in four categories (code vulnerabilities, API security, Websec.fr CTFs, cloud security). This board is the percentage of cloud security challenges solved on the first attempt (pass@1) by models run in the Claude Code harness; higher is better.
Legal Benchmarks - Contract Workflows - Task Pass Rate (Task pass rate (%, every applicable criterion met in both attempts)): leader Claude Fable 5.1 (48.4), 19 models
Legal Benchmarks (legalbenchmarks.ai) model leaderboard, Contract Workflows: lawyer-authored drafting and review tasks completed in a sealed agentic harness, two attempts each, high reasoning effort where offered. Share of tasks meeting every applicable criterion in both attempts, judged per criterion by GLM 5.3 Flash; %, higher is better.
Legal Benchmarks - Contract Workflows - Criteria Pass Rate (Criteria pass rate (%, criteria met in both attempts)): leader Claude Fable 5.1 (90.7), 19 models
Legal Benchmarks Contract Workflows: share of applicable lawyer-authored pass/fail criteria satisfied in both of a model's two attempts, graded one criterion at a time by a calibrated GLM 5.3 Flash judge. Shows coverage of the legal requirements even when a whole task fails; %, higher is better. Tables are replaced each monthly cycle.
Legal Benchmarks - Contract Workflows - Form (Form (%, binary form checks passed in both attempts)): leader GPT-6 (91.9), 19 models
Legal Benchmarks Contract Workflows: Form is the share of applicable binary presentation checks (output usable as delivered) passed in both attempts, judged by GLM 5.3 Flash with a separate checklist that has no lawyer-labelled accuracy measurement. It does not affect the site's ranking; %, higher is better.
Legal Benchmarks - Data Extraction - Task Pass Rate (Task pass rate (%, every applicable criterion met in both attempts)): leader Claude Fable 5.1 (63.3), 19 models
Legal Benchmarks (legalbenchmarks.ai) model leaderboard, Data Extraction: lawyer-authored extraction tasks over native legal documents in a sealed agentic harness, two attempts each, high reasoning effort where offered. Share of tasks meeting every applicable criterion in both attempts, judged per criterion by GLM 5.3 Flash; %, higher is better.
Legal Benchmarks - Data Extraction - Criteria Pass Rate (Criteria pass rate (%, criteria met in both attempts)): leader Claude Fable 5.1 (89.9), 19 models
Legal Benchmarks Data Extraction: share of applicable lawyer-authored pass/fail criteria satisfied in both of a model's two attempts, graded one criterion at a time by a calibrated GLM 5.3 Flash judge. Shows coverage of the requested terms even when a whole task fails; %, higher is better. Tables are replaced each monthly cycle.
Legal Benchmarks - Data Extraction - Form (Form (%, binary form checks passed in both attempts)): leader GPT-6 (83.5), 19 models
Legal Benchmarks Data Extraction: Form is the share of applicable binary presentation checks (output usable as delivered) passed in both attempts, judged by GLM 5.3 Flash with a separate checklist that has no lawyer-labelled accuracy measurement. It does not affect the site's ranking; %, higher is better.
AIMultiple - FinanceReasoning (Hard) (Accuracy (%, 238 hard FinanceReasoning questions)): leader GPT-5.6 Pro Sol (90.76), 55 models
AIMultiple's run of the FinanceReasoning benchmark (Tang et al.) hard subset: 238 multi-step quantitative finance questions with free-form chain-of-thought answers, extracted by Claude Sonnet 4.5 and scored against ground truth with a 0.2% relative tolerance. Models run through provider APIs; accuracy %, higher is better.
AIMultiple - Agentic RAG Routing (Routing accuracy (%, 184 hardest of 759 BIRD questions, 11 databases)): leader Claude Opus 5 (84.8), 37 models
AIMultiple Agentic RAG benchmark: each model chooses which of 11 anonymised, deliberately confusable BIRD-SQL databases holds the answer, using per-database schema and query tools (at most 10 API calls, temperature 0). Accuracy of the explicitly declared database on the 184 hardest of 759 questions; %, higher is better, chance 9.1%.
AIMultiple - Text-to-SQL (Strict Execution Match) (Execution match against BIRD gold (%, over correctly routed questions)): leader Gemini 3 Flash (Preview) (55.1), 37 models
AIMultiple Text-to-SQL, from the same BIRD multi-database routing runs: the model's final SQL is executed and its result set compared with BIRD's gold query, without the domain hint, over the questions the model routed correctly, so denominators differ by model. Strict execution match; %, higher is better.
AIMultiple - Text-to-SQL (Adjudicated) (Execution match with jury-credited equivalents and broken golds (%, over correctly routed questions)): leader Claude Opus 5 (82.4), 37 models
AIMultiple Text-to-SQL adjudicated score: misses under strict execution match are re-read by a blind three-model jury from other model families, crediting equivalent answers and questions whose BIRD gold query is broken. Measured over each model's correctly routed questions; an optimistic LLM-jury estimate, %, higher is better.
Arabic Broad Leaderboard - MMLU (Average Score (0-10)): leader Gemini 3.1 Flash Lite (10.0), 111 models
MMLU-style Arabic multiple-choice knowledge questions: the 121-question MMLU category of SILMA's Arabic Broad Benchmark, whose answers are scored 0-10 by rules or an LLM judge. Category average score (0-10), higher is better; each model's latest result file is used.
Arabic Broad Leaderboard - General Knowledge (Average Score (0-10)): leader Gemini 3 Pro (Preview) (9.651), 111 models
Arabic general-knowledge questions: the 63-question General Knowledge category of SILMA's Arabic Broad Benchmark, whose answers are scored 0-10 by rules or an LLM judge. Category average score (0-10), higher is better; each model's latest result file is used.
Arabic Broad Leaderboard - Reasoning & Math (Average Score (0-10)): leader Gemini 3.1 Pro (Preview) (9.767), 111 models
Arabic reasoning and mathematics questions: the 43-question Reasoning & Math category of SILMA's Arabic Broad Benchmark, whose answers are scored 0-10 by rules or an LLM judge. Category average score (0-10), higher is better; each model's latest result file is used.
Arabic Broad Leaderboard - RAG QA (Average Score (0-10)): leader GLM-5.3 (9.732), 111 models
Arabic question answering over supplied context (retrieval-augmented generation): the 41-question RAG QA category of SILMA's Arabic Broad Benchmark, whose answers are scored 0-10 by rules or an LLM judge. Category average score (0-10), higher is better; each model's latest result file is used.
Arabic Broad Leaderboard - Translation (incl Dialects) (Average Score (0-10)): leader GPT-4o (8.389), 111 models
Translation involving Arabic, including dialects: the 36-question Translation (incl Dialects) category of SILMA's Arabic Broad Benchmark, whose answers are scored 0-10 by rules or an LLM judge. Category average score (0-10), higher is better; each model's latest result file is used.
Arabic Broad Leaderboard - Trust & Safety (Average Score (0-10)): leader Claude Sonnet 4 (20250514) (10.0), 111 models
Arabic trust and safety prompts: the 30-question Trust & Safety category of SILMA's Arabic Broad Benchmark, whose answers are scored 0-10 by rules or an LLM judge. Category average score (0-10), higher is better; each model's latest result file is used.
Arabic Broad Leaderboard - Writing (incl Dialects) (Average Score (0-10)): leader GPT-4o (8.545), 111 models
Arabic writing tasks, including dialects: the 22-question Writing (incl Dialects) category of SILMA's Arabic Broad Benchmark, whose answers are scored 0-10 by rules or an LLM judge. Category average score (0-10), higher is better; each model's latest result file is used.
Aiera Leaderboard - Research Score (Aiera Research Score (0-100)): leader Claude Fable 5.1 (88.74), 34 models
Aiera Leaderboard: models answer proprietary institutional-finance research questions through a client-side tool loop connected to Aiera's MCP server (filings, transcripts, research), best of three. The Aiera Research Score is the equal-weight mean of the Facts, Grounding and Depth axes; 0-100, higher is better.
Aiera Leaderboard - Facts (Facts axis (0-100)): leader Claude Opus 5 (78.5), 34 models
Aiera Leaderboard Facts axis: whether a financial research answer gets the substance right and complete, combining key-facts accuracy on proprietary research questions with rubric coverage on complex analyst prompts, answered through Aiera's MCP server. 0-100, higher is better.
Aiera Leaderboard - Grounding (Grounding axis (0-100)): leader Claude Fable 5 (97.1), 34 models
Aiera Leaderboard Grounding axis: whether the claims in a model's financial research answer are tied to explicit sources such as filings, transcripts or research retrieved through Aiera's MCP server, rather than asserted bare. 0-100, higher is better.
Aiera Leaderboard - Depth (Depth axis (0-100)): leader Claude Opus 5 (96.4), 34 models
Aiera Leaderboard Depth axis: given a complete, grounded answer, whether the analysis reaches the caliber an institutional financial professional would produce (prioritized, specific, insightful) rather than thorough but generic. Proprietary questions answered through Aiera's MCP server; 0-100, higher is better.
ChemGraph - Single-Agent (Accuracy (%, 40 computational-chemistry queries)): leader Claude Haiku 4.5 (100.0), 32 models
ChemGraph Leaderboard (Argonne): 40 computational-chemistry queries in 12 categories, including SMILES lookup, geometry optimization, vibrational frequencies, thermochemistry, dipole moments and reaction Gibbs energies, solved with ASE-based tools by a single agent. A structured judge accepts answers within 5% relative tolerance; accuracy %, higher is better.
ChemGraph - Multi-Agent (Accuracy (%, 40 computational-chemistry queries)): leader Claude Sonnet 4.5 (100.0), 31 models
ChemGraph Leaderboard (Argonne) multi-agent workflow: several specialised agents coordinate chemistry tool calls to answer the same 40 computational-chemistry queries in 12 categories. A structured judge marks each answer correct within 5% relative tolerance; accuracy % from the latest published evaluation run, higher is better.
MindBench - Alignment (Alignment with human consensus (0-1, 1 - MAE / 3, higher is better)): leader O3 (0.826), 18 models
MindBench.ai: models rate 500 fixed chatbot replies from mental-health conversations on a -3 to +3 appropriateness scale, compared with a consensus of 167 clinicians and people with lived experience. Alignment is 1 - MAE/3 against that consensus, the site's headline; 0-1, higher is better. It measures judging replies, not writing them.
MindBench - Safety Miss Rate (Harmful replies rated acceptable (%, lower is better)): leader Gemini 3.5 Flash (0.0), 18 models
MindBench.ai safety miss rate: among chatbot replies that community raters judged clearly harmful (consensus at or below -1), the share a model rated as appropriate (+0.5 or higher). Based on about 213 eligible replies per model, so several models tie at zero; %, lower is better.
Nutrient Chart Parsing (Mean per-chart cell F1 at 3% tolerance (%, 2,500 charts)): leader Nutrient Chart Parsing (85.67), 13 models
Nutrient Chart Parsing Leaderboard: models extract the underlying data from 2,500 charts. Headline mean per-chart cell F1 at 3% tolerance (exact when the chart prints data labels, 3 percentage points for share charts, otherwise 3% of the value range); %, higher is better. The publisher's own commercial chart-parsing model is ranked alongside provider APIs and open-weight VLMs.
AGI-Eval Community - Objective Accuracy (Accuracy (%)): leader GPT-5.5 (86.14), 141 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is overall objective-evaluation accuracy across all ability dimensions (the board's ranking column). Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - Objective Accuracy (Chinese) (Accuracy (%)): leader GPT-5.5 (84.95), 139 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the Chinese-language item split (_中) of the objective accuracy column; the parent column combines both splits. Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - Objective Accuracy (English) (Accuracy (%)): leader Claude Opus 4.8 (91.58), 139 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the English-language item split (_英) of the objective accuracy column; the parent column combines both splits. Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - Algorithmic Reasoning (Accuracy (%)): leader GPT-5.5 (87.77), 141 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the algorithmic (code) reasoning dimension; upstream's Chinese split is identical, so only this column is kept. Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - Cognition (Accuracy (%)): leader Claude Fable 5 (88.43), 141 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the cognition dimension, combining Chinese and English items. Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - Cognition (Chinese) (Accuracy (%)): leader Claude Fable 5 (89.64), 139 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the Chinese-language item split (_中) of the cognition column; the parent column combines both splits. Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - Cognition (English) (Accuracy (%)): leader Doubao-Seed-1.6-Auto (91.35), 139 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the English-language item split (_英) of the cognition column; the parent column combines both splits. Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - General Reasoning (Accuracy (%)): leader Claude Fable 5 (97.82), 141 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the general reasoning dimension; its English split carries about 93% of the weight and its small Chinese split is saturated, so only this column is kept. Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - Interaction (Accuracy (%)): leader Claude Opus 4.8 (91.2), 141 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the interaction dimension, combining Chinese and English items. Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - Interaction (Chinese) (Accuracy (%)): leader Claude Opus 4.8 (89.93), 139 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the Chinese-language item split (_中) of the interaction column; the parent column combines both splits. Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - Interaction (English) (Accuracy (%)): leader Gemini 3.1 Pro (Preview) (95.45), 139 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the English-language item split (_英) of the interaction column; the parent column combines both splits. Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - Learning (Accuracy (%)): leader Gemini 3.1 Pro (Preview) (94.78), 141 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the learning dimension, combining Chinese and English items. Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - Learning (Chinese) (Accuracy (%)): leader Gemini 3.1 Pro (Preview) (91.15), 139 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the Chinese-language item split (_中) of the learning column; the parent column combines both splits. Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - Learning (English) (Accuracy (%)): leader Gemini 3.1 Pro (Preview) (97.52), 139 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the English-language item split (_英) of the learning column; the parent column combines both splits. Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - Mathematical Reasoning (Accuracy (%)): leader Claude Fable 5 (89.37), 141 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the mathematical reasoning dimension; upstream's Chinese split is identical, so only this column is kept. Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - Subject Knowledge (Accuracy (%)): leader Seed 2.0 Pro (91.1), 141 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the subject knowledge dimension; upstream's Chinese split is identical, so only this column is kept. Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - Instruction Following (Accuracy (%)): leader GPT-5.5 (71.29), 141 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the instruction following dimension; upstream's Chinese split is identical, so only this column is kept. Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - Subject Reasoning (Accuracy (%)): leader Claude Opus 4.8 (90.01), 141 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the subject reasoning dimension, combining Chinese and English items. Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - Subject Reasoning (Chinese) (Accuracy (%)): leader Seed 2.1 Pro (94.54), 139 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the Chinese-language item split (_中) of the subject reasoning column; the parent column combines both splits. Upstream accuracy 0-1, shown as 0-100; higher is better.
AGI-Eval Community - Subject Reasoning (English) (Accuracy (%)): leader Claude Opus 4.8 (88.79), 139 models
AGI-Eval community leaderboard (agi-eval.cn), the private-set LLM board built from the platform's non-public items; latest monthly snapshot. Unrelated to the AGIEval exam dataset. This board is the English-language item split (_英) of the subject reasoning column; the parent column combines both splits. Upstream accuracy 0-1, shown as 0-100; higher is better.
ReLE - Overall (Accuracy (%)): leader Qwen 3.7 Max (76.9), 162 models
ReLE Chinese LLM benchmark (NoneLinear, formerly CLiB), a continuously updated Chinese-language evaluation run through each model's API. Overall score, which ReLE computes as 0.7 x general ability (mean of reasoning and math, language and instruction following, agents and tool use, coding) plus 0.3 x professional ability (mean of education, medicine and mental health, finance, law and civil service). Accuracy 0-100, higher is better.
ReLE - Reasoning and Mathematics (Accuracy (%)): leader Claude Opus 5 (92.0), 174 models
ReLE Chinese LLM benchmark (NoneLinear, formerly CLiB), a continuously updated Chinese-language evaluation run through each model's API. Reasoning and mathematical calculation domain: deductive and commonsense reasoning, BBH, arithmetic, table QA and summarization, olympiad math and sudoku items. Accuracy 0-100, higher is better.
ReLE - Language and Instruction Following (Accuracy (%)): leader doubao-seed-2-1-pro-260628 (78.4), 179 models
ReLE Chinese LLM benchmark (NoneLinear, formerly CLiB), a continuously updated Chinese-language evaluation run through each model's API. Language and instruction-following domain: idioms, sentiment, entailment, classification, information extraction, reading comprehension, pronoun resolution, poetry matching, Chinese instruction following and character glyphs. Accuracy 0-100, higher is better.
ReLE - Agents and Tool Use (Accuracy (%)): leader MiniMax-M2.7 (75.8), 152 models
ReLE Chinese LLM benchmark (NoneLinear, formerly CLiB), a continuously updated Chinese-language evaluation run through each model's API. Agents and tool-use domain, ReLE's own runs of BFCL-V3 and TAU-bench style tasks. Accuracy 0-100, higher is better.
ReLE - Education (Accuracy (%)): leader Gemini 3.1 Pro (Preview) (68.6), 179 models
ReLE Chinese LLM benchmark (NoneLinear, formerly CLiB), a continuously updated Chinese-language evaluation run through each model's API. Education domain: gaokao questions and primary, middle and high school subject exams. Accuracy 0-100, higher is better.
ReLE - Medicine and Mental Health (Accuracy (%)): leader doubao-seed-2-1-pro-260628 (91.6), 178 models
ReLE Chinese LLM benchmark (NoneLinear, formerly CLiB), a continuously updated Chinese-language evaluation run through each model's API. Medicine and mental health domain: Chinese physician, nursing, pharmacist and medical technician licensing exams, basic medical knowledge, medical postgraduate entrance exams and mental health. Accuracy 0-100, higher is better.
ReLE - Finance (Accuracy (%)): leader doubao-seed-evolving (91.7), 178 models
ReLE Chinese LLM benchmark (NoneLinear, formerly CLiB), a continuously updated Chinese-language evaluation run through each model's API. Finance domain: accounting, banking, insurance, securities and other financial qualification exams plus financial knowledge and applications. Accuracy 0-100, higher is better.
ReLE - Law and Civil Service (Accuracy (%)): leader GPT-5.5 (89.0), 178 models
ReLE Chinese LLM benchmark (NoneLinear, formerly CLiB), a continuously updated Chinese-language evaluation run through each model's API. Law and civil service domain: the national lawyer qualification exam (including JEC-QA items) and civil service exams. Accuracy 0-100, higher is better.
ReLE - Coding (Accuracy (%)): leader Qwen 3.6 Max Preview (75.7), 98 models
ReLE Chinese LLM benchmark (NoneLinear, formerly CLiB), a continuously updated Chinese-language evaluation run through each model's API. Coding domain, the mean of ReLE's own LiveCodeBench and Terminal-Bench 2.0 runs; read from the repository's generated coding board because the site API omits this domain. Accuracy 0-100, higher is better.
ReLE - Education - Gaokao (Accuracy (%)): leader doubao-seed-evolving (69.3), 179 models
ReLE Chinese LLM benchmark (NoneLinear, formerly CLiB), a continuously updated Chinese-language evaluation run through each model's API. Education sub-dimension: questions from China's national college entrance examination (gaokao) across subjects. Accuracy 0-100, higher is better.
ReLE - Education - High School Subjects (Accuracy (%)): leader Gemini 3.1 Pro (Preview) (78.1), 179 models
ReLE Chinese LLM benchmark (NoneLinear, formerly CLiB), a continuously updated Chinese-language evaluation run through each model's API. Education sub-dimension: high school subject exam questions (Chinese, mathematics, sciences, humanities, English). Accuracy 0-100, higher is better.
ReLE - Education - Middle School Subjects (Accuracy (%)): leader Seed-OSS-36B-Instruct (62.5), 179 models
ReLE Chinese LLM benchmark (NoneLinear, formerly CLiB), a continuously updated Chinese-language evaluation run through each model's API. Education sub-dimension: middle school subject exam questions. Accuracy 0-100, higher is better.
ReLE - Education - Primary School Subjects (Accuracy (%)): leader Gemini 3.1 Pro (Preview) (72.0), 179 models
ReLE Chinese LLM benchmark (NoneLinear, formerly CLiB), a continuously updated Chinese-language evaluation run through each model's API. Education sub-dimension: primary school subject exam questions. Accuracy 0-100, higher is better.
ReLE - Medicine - Physician Exams (Accuracy (%)): leader ERNIE-4.5-Turbo-32K (92.8), 178 models
ReLE Chinese LLM benchmark (NoneLinear, formerly CLiB), a continuously updated Chinese-language evaluation run through each model's API. Medicine sub-dimension: Chinese physician qualification and attending-physician exam questions across specialties. Accuracy 0-100, higher is better.
ReLE - Medicine - Basic Medical Knowledge (Accuracy (%)): leader GPT-6 (89.2), 178 models
ReLE Chinese LLM benchmark (NoneLinear, formerly CLiB), a continuously updated Chinese-language evaluation run through each model's API. Medicine sub-dimension: basic medical science knowledge questions. Accuracy 0-100, higher is better.
ReLE - Law - Lawyer Qualification Exam (Accuracy (%)): leader Doubao Seed 2.0 Mini (89.3), 178 models
ReLE Chinese LLM benchmark (NoneLinear, formerly CLiB), a continuously updated Chinese-language evaluation run through each model's API. Law sub-dimension: China's national legal professional qualification (lawyer) exam, including JEC-QA knowledge and case-analysis items. Accuracy 0-100, higher is better.
ReLE - Reasoning - BBH (Accuracy (%)): leader Qwen 3 32B (89.9), 179 models
ReLE Chinese LLM benchmark (NoneLinear, formerly CLiB), a continuously updated Chinese-language evaluation run through each model's API. Reasoning sub-dimension: ReLE's own run of BIG-Bench Hard symbolic reasoning tasks; not a copy of other BBH leaderboards. Accuracy 0-100, higher is better.
ReLE - Agents - BFCL-V3 (Accuracy (%)): leader MiniMax-M2.7 (76.5), 151 models
ReLE Chinese LLM benchmark (NoneLinear, formerly CLiB), a continuously updated Chinese-language evaluation run through each model's API. Agents sub-dimension: ReLE's own run of the Berkeley Function Calling Leaderboard V3 tasks; not a mirror of the Gorilla leaderboard. Accuracy 0-100, higher is better.
Dnotitia Korean LLM Leaderboard (Mean Item Score (%)): leader Gemini 3.7 Flash (100.0), 226 models
Dnotitia's Korean LLM leaderboard: 30 private Korean items covering understanding, generation, reasoning, RAG and tool calling, each scored 0-1; the board is the published mean over repeated runs, shown as 0-100 (higher is better). The operator also builds DNA models, and the top is saturated (16 models at 98 or above).
KorT - Korean-English Translation (Elo (pairwise LLM-judged)): leader GPT-5.6 Luna (1201.8), 15 models
KorT (kort.worldsw.dev): about 100 private bidirectional Korean-English items across 28 phenomenon types. Every pair of system outputs is compared in both orders by GPT-5.6 Sol and Gemini 3.1 Pro judges with a reference, fitted to Bradley-Terry Elo. Higher is better; Elo is relative within this record. Commercial and research MT engines are excluded.
PLCC - Overall (Mean category accuracy (%)): leader Gemini 3.1 Pro (Preview) (97.0), 227 models
PLCC (Polish Linguistic and Cultural Competency benchmark, Dadas et al. 2025): 600 open-ended questions about Polish language and culture, 100 in each of six categories. Score is the mean of the six category accuracies (0-100, higher is better), reported only for models evaluated on all six, as on the site.
PLCC - Art & Entertainment (Accuracy (%)): leader Gemini 3.0 Pro (Preview) (95.0), 227 models
PLCC (Polish Linguistic and Cultural Competency benchmark, Dadas et al. 2025) category of 100 open-ended questions on Polish art (literature, painting, sculpture, theatre, music, dance, film) and entertainment (sports, popular music, television, radio, show business). Each answer is graded by rule-based acceptance conditions with binary credit; accuracy 0-100, higher is better.
PLCC - Culture & Tradition (Accuracy (%)): leader Gemini 3.1 Pro (Preview) (100.0), 227 models
PLCC (Polish Linguistic and Cultural Competency benchmark, Dadas et al. 2025) category of 100 open-ended questions on Polish beliefs and religion (Christian and folk traditions, Slavic mythology), pop culture, and everyday customs, cuisine and clothing. Each answer is graded by rule-based acceptance conditions with binary credit; accuracy 0-100, higher is better.
PLCC - Geography (Accuracy (%)): leader Gemini 3.0 Pro (Preview) (100.0), 227 models
PLCC (Polish Linguistic and Cultural Competency benchmark, Dadas et al. 2025) category of 100 open-ended questions on Polish natural and man-made geography, socio-political geography (population, borders, administrative units) and native fauna and flora. Each answer is graded by rule-based acceptance conditions with binary credit; accuracy 0-100, higher is better.
PLCC - Grammar (Accuracy (%)): leader DeepSeek V4.1 Flash (98.0), 227 models
PLCC (Polish Linguistic and Cultural Competency benchmark, Dadas et al. 2025) category of 100 open-ended questions on Polish grammar and orthography, including morphology, parts of speech and of the sentence, phonetics, word formation and rhetorical figures. Each answer is graded by rule-based acceptance conditions with binary credit; accuracy 0-100, higher is better.
PLCC - History (Accuracy (%)): leader GPT-6 (99.0), 227 models
PLCC (Polish Linguistic and Cultural Competency benchmark, Dadas et al. 2025) category of 100 open-ended questions on Polish history from Mieszko I to the present, covering key events and figures in politics, science and the humanities. Each answer is graded by rule-based acceptance conditions with binary credit; accuracy 0-100, higher is better.
PLCC - Vocabulary (Accuracy (%)): leader Gemini 3.1 Pro (Preview) (96.0), 227 models
PLCC (Polish Linguistic and Cultural Competency benchmark, Dadas et al. 2025) category of 100 open-ended questions on the meaning of less frequent Polish words, idioms, sayings and proverbs, including slang, regionalisms, dialects, youth language and archaisms. Each answer is graded by rule-based acceptance conditions with binary credit; accuracy 0-100, higher is better.
EKI Estonian LLM - Overall (Mean benchmark score (0-100)): leader GPT-6 (74.32), 77 models
Keelemudelite mõõdupuu, the Institute of the Estonian Language's independent Estonian leaderboard: unweighted mean of its benchmark scores (seven at launch of this board), 0-100, higher is better. As on the site, only models evaluated on every published benchmark are included; models run without web search or tools.
EKI Estonian LLM - Book Recommendations (Recommendations matching a real book (%)): leader GPT-5.6 Sol (90.62), 85 models
EKI book-recommendation benchmark: 40 prompts each asking for four Estonian-language books across 8 genres and 5 criteria; every title-author pair is checked against the Estonian National Bibliography. Score is the share of recommendations matching a real book (0-100, higher is better).
EKI Estonian LLM - Language Advice (Accuracy (%)): leader GPT-6 (76.04), 85 models
EKI language-advice benchmark: 240 questions on Estonian orthography, inflected forms and word choice based on real queries to EKI's language advice service, mixing true/false, multiple-choice, short-answer and judge-graded open questions. Accuracy 0-100, higher is better.
EKI Estonian LLM - Trivia (Accuracy (%)): leader Gemini 3.1 Pro (Preview) (85.0), 85 models
EKI multiple-choice trivia benchmark (Eesti mäng): 1,000 questions on Estonian history, culture, sports, nature and geography from a published quiz book; random guessing yields about 26%. Accuracy 0-100, higher is better; no web search or tools.
EKI Estonian LLM - Idioms (Accuracy (%)): leader GPT-6 (74.0), 85 models
EKI idioms benchmark: 300 fill-in-the-blank tasks restoring Estonian-specific, universal and ambiguous figurative expressions in exactly the right grammatical form, with accepted variants vetted by EKI linguists. Accuracy 0-100, higher is better.
EKI Estonian LLM - Terminology (Accuracy (%)): leader GPT-6 (61.81), 85 models
EKI terminology benchmark: 1,380 definition-to-term questions, 30 terms in each of 46 specialist fields from the Esterm term base and field term collections; an answer must give the term or a registered synonym. Accuracy 0-100, higher is better.
EKI Estonian LLM - Propaganda Resistance (Geometric mean of 1-5 judge ratings, rescaled to 0-100): leader Claude Opus 5 (97.59), 85 models
EKI propaganda-resistance benchmark: 75 questions on Kremlin strategic narratives, asked neutrally, with a biased premise or maliciously, each in Estonian, English and Russian; a Claude Opus 4.5 judge rates answers 1-5. Geometric mean rescaled to 0-100, higher means more resistant.
EKI Estonian LLM - Trivia Hard (Accuracy (%)): leader Gemini 3.1 Pro (Preview) (50.75), 77 models
EKI open-answer trivia benchmark (Mälumäng): 800 harder questions about Estonia, 80 in each of ten topics, graded by exact match with a GPT-5.4-nano judge for non-matching answers. Accuracy 0-100, higher is better; retired models were never run on it.
AgentIdeaBench - Active (Weighted Hypothesis Score (1-10)): leader Claude Opus 5 (7.135), 63 models
Research-hypothesis ideation in AgentIdeaBench's Active track, where the model gets a subfield name and a literature search tool and decides what to look up (arXiv 2609.07611v1). Weighted LLM-critic total on a 1-10 scale (originality 2, impact 1.5, feasibility 1, clarity 0.5, specificity 0.5; highest of three critics dropped), from reports/leaderboard_full.json; higher is better.
AgentIdeaBench - Active - Originality (Originality Critic Score (1-10)): leader Claude Opus 5 (7.3), 63 models
Originality dimension of AgentIdeaBench's Active track, where the model gets a subfield name and a literature search tool and decides what to look up (arXiv 2609.07611v1). Mean LLM-critic score on a 1-10 scale, dropping the highest of three critics and averaging over ideas and subfields, from the dims field of reports/leaderboard_full.json; higher is better.
AgentIdeaBench - Active - Feasibility (Feasibility Critic Score (1-10)): leader GPT-5.4 (7.004), 63 models
Feasibility dimension of AgentIdeaBench's Active track, where the model gets a subfield name and a literature search tool and decides what to look up (arXiv 2609.07611v1). Mean LLM-critic score on a 1-10 scale, dropping the highest of three critics and averaging over ideas and subfields, from the dims field of reports/leaderboard_full.json; higher is better.
AgentIdeaBench - Active - Clarity (Clarity Critic Score (1-10)): leader Claude Opus 5 (7.873), 63 models
Clarity dimension of AgentIdeaBench's Active track, where the model gets a subfield name and a literature search tool and decides what to look up (arXiv 2609.07611v1). Mean LLM-critic score on a 1-10 scale, dropping the highest of three critics and averaging over ideas and subfields, from the dims field of reports/leaderboard_full.json; higher is better.
AgentIdeaBench - Active - Impact (Impact Critic Score (1-10)): leader Claude Opus 5 (6.748), 63 models
Impact dimension of AgentIdeaBench's Active track, where the model gets a subfield name and a literature search tool and decides what to look up (arXiv 2609.07611v1). Mean LLM-critic score on a 1-10 scale, dropping the highest of three critics and averaging over ideas and subfields, from the dims field of reports/leaderboard_full.json; higher is better.
AgentIdeaBench - Active - Specificity (Specificity Critic Score (1-10)): leader Claude Opus 5 (7.631), 63 models
Specificity dimension of AgentIdeaBench's Active track, where the model gets a subfield name and a literature search tool and decides what to look up (arXiv 2609.07611v1). Mean LLM-critic score on a 1-10 scale, dropping the highest of three critics and averaging over ideas and subfields, from the dims field of reports/leaderboard_full.json; higher is better.
AgentIdeaBench - Static (Weighted Hypothesis Score (1-10)): leader GPT-5.6 Sol (6.9), 63 models
Research-hypothesis ideation in AgentIdeaBench's Static track, where the model is handed the references for a subfield (arXiv 2609.07611v1). Weighted LLM-critic total on a 1-10 scale (originality 2, impact 1.5, feasibility 1, clarity 0.5, specificity 0.5; highest of three critics dropped), from reports/leaderboard_full.json; higher is better.
AgentIdeaBench - Static - Originality (Originality Critic Score (1-10)): leader Claude Opus 5 (7.254), 63 models
Originality dimension of AgentIdeaBench's Static track, where the model is handed the references for a subfield (arXiv 2609.07611v1). Mean LLM-critic score on a 1-10 scale, dropping the highest of three critics and averaging over ideas and subfields, from the static_dims field of reports/leaderboard_full.json; higher is better.
AgentIdeaBench - Static - Feasibility (Feasibility Critic Score (1-10)): leader GPT-5.6 Terra (6.525), 63 models
Feasibility dimension of AgentIdeaBench's Static track, where the model is handed the references for a subfield (arXiv 2609.07611v1). Mean LLM-critic score on a 1-10 scale, dropping the highest of three critics and averaging over ideas and subfields, from the static_dims field of reports/leaderboard_full.json; higher is better.
AgentIdeaBench - Static - Clarity (Clarity Critic Score (1-10)): leader GPT-5.6 Sol (7.787), 63 models
Clarity dimension of AgentIdeaBench's Static track, where the model is handed the references for a subfield (arXiv 2609.07611v1). Mean LLM-critic score on a 1-10 scale, dropping the highest of three critics and averaging over ideas and subfields, from the static_dims field of reports/leaderboard_full.json; higher is better.
AgentIdeaBench - Static - Impact (Impact Critic Score (1-10)): leader Claude Opus 5 (6.667), 63 models
Impact dimension of AgentIdeaBench's Static track, where the model is handed the references for a subfield (arXiv 2609.07611v1). Mean LLM-critic score on a 1-10 scale, dropping the highest of three critics and averaging over ideas and subfields, from the static_dims field of reports/leaderboard_full.json; higher is better.
AgentIdeaBench - Static - Specificity (Specificity Critic Score (1-10)): leader GPT-5.6 Sol (7.454), 63 models
Specificity dimension of AgentIdeaBench's Static track, where the model is handed the references for a subfield (arXiv 2609.07611v1). Mean LLM-critic score on a 1-10 scale, dropping the highest of three critics and averaging over ideas and subfields, from the static_dims field of reports/leaderboard_full.json; higher is better.
PUMA Polish Multimodal - History and Culture (Strict Accuracy (%)): leader Gemini 3.7 Flash (85.0), 65 models
History and culture category of PUMA (Polish Unified Multimodal Assessment), a culturally grounded Polish benchmark for vision-language models (Images group; arXiv 2608.21853v1). Strict score, the percentage of answers meeting every verification rule, 0-100, higher is better, from the OPI-PIB/puma Space data.json; audio-pipeline rows and the auxiliary Soft score are excluded.
PUMA Polish Multimodal - Contemporary Life (Strict Accuracy (%)): leader Gemini 3.1 Pro (Preview) (83.0), 65 models
Contemporary life category of PUMA (Polish Unified Multimodal Assessment), a culturally grounded Polish benchmark for vision-language models (Images group; arXiv 2608.21853v1). Strict score, the percentage of answers meeting every verification rule, 0-100, higher is better, from the OPI-PIB/puma Space data.json; audio-pipeline rows and the auxiliary Soft score are excluded.
PUMA Polish Multimodal - Geography and Environment (Strict Accuracy (%)): leader Gemini 3.7 Flash (89.0), 65 models
Geography and environment category of PUMA (Polish Unified Multimodal Assessment), a culturally grounded Polish benchmark for vision-language models (Images group; arXiv 2608.21853v1). Strict score, the percentage of answers meeting every verification rule, 0-100, higher is better, from the OPI-PIB/puma Space data.json; audio-pipeline rows and the auxiliary Soft score are excluded.
PUMA Polish Multimodal - OCR (Strict Accuracy (%)): leader Claude Fable 5 (90.0), 65 models
OCR category of PUMA (Polish Unified Multimodal Assessment), a culturally grounded Polish benchmark for vision-language models (Documents group; arXiv 2608.21853v1). Strict score, the percentage of answers meeting every verification rule, 0-100, higher is better, from the OPI-PIB/puma Space data.json; audio-pipeline rows and the auxiliary Soft score are excluded.
PUMA Polish Multimodal - Document QA (Strict Accuracy (%)): leader Gemini 3.5 Flash (88.0), 65 models
Document QA category of PUMA (Polish Unified Multimodal Assessment), a culturally grounded Polish benchmark for vision-language models (Documents group; arXiv 2608.21853v1). Strict score, the percentage of answers meeting every verification rule, 0-100, higher is better, from the OPI-PIB/puma Space data.json; audio-pipeline rows and the auxiliary Soft score are excluded.
PUMA Polish Multimodal - Structured Extraction (Strict Accuracy (%)): leader Gemini 3.1 Pro (Preview) (79.0), 65 models
Structured extraction category of PUMA (Polish Unified Multimodal Assessment), a culturally grounded Polish benchmark for vision-language models (Documents group; arXiv 2608.21853v1). Strict score, the percentage of answers meeting every verification rule, 0-100, higher is better, from the OPI-PIB/puma Space data.json; audio-pipeline rows and the auxiliary Soft score are excluded.
PUMA Polish Multimodal - Vision Mean (Mean Strict Accuracy over 6 Vision Categories (%)): leader Gemini 3.7 Flash (84.0), 65 models
Unweighted mean of PUMA's six vision categories (History and culture, Contemporary life, Geography and environment, OCR, Document QA, Structured extraction), the Mean score of the paper's Table 5 (arXiv 2608.21853v1). Strict score 0-100, higher is better, from the OPI-PIB/puma Space data.json; only models scored on all six are listed.
HealthBench-Psych (Three-Judge Panel Mean Rubric Score (%)): leader Kimi K2.6 (62.695), 21 models
Rubric-graded answers on 610 mental-health conversations selected from OpenAI's HealthBench with clinical expert review (arXiv 2608.25071v1). Mean of three judges' clipped rubric scores (GPT-4.1, Claude Haiku 4.5 and Gemini 2.5 Flash) as a percentage, higher is better, from eval/runs/matrix_healthbench-psych-v1.csv; unpinned -latest aliases are omitted.
HealthBench-Psych - Hard (Three-Judge Panel Mean Rubric Score (%)): leader Claude Opus 5 (41.513), 20 models
The 119 HealthBench-Psych conversations that also belong to OpenAI's HealthBench Hard (arXiv 2608.25071v1). Mean of three judges' clipped rubric scores (GPT-4.1, Claude Haiku 4.5 and Gemini 2.5 Flash) as a percentage, higher is better, from eval/runs/matrix_healthbench-psych-hard-v1.csv; rows with a judge score on a clip bound are omitted.
APIFlow-Bench 1.0 - 20-Subtask Chain (Pass Rate (%)): leader GPT-5.5 (72.73), 24 models
Postman's APIFlow-Bench 1.0 measures whether agents complete long, dependent REST API workflows (arXiv 2608.29128v1). This board covers the leaderboard's default 20-subtask chain set (11 tasks); pass rate is passed over all trials, 0-100, higher is better, and refusals and abstentions count as failures. From the live leaderboard's overview page.
APIFlow-Bench 1.0 - 15-Subtask Chain (Pass Rate (%)): leader GLM-5.2 (74.55), 24 models
Postman's APIFlow-Bench 1.0 measures whether agents complete long, dependent REST API workflows (arXiv 2608.29128v1). This board covers the 15-subtask chain set (11 tasks); pass rate is passed over all trials, 0-100, higher is better, and refusals and abstentions count as failures. From the live leaderboard's overview-chain15 page.
APIFlow-Bench 1.0 - 10-Subtask Chain (Pass Rate (%)): leader MiniMax-M2.7 (75.0), 24 models
Postman's APIFlow-Bench 1.0 measures whether agents complete long, dependent REST API workflows (arXiv 2608.29128v1). This board covers the 10-subtask chain set (12 tasks); pass rate is passed over all trials, 0-100, higher is better, and refusals and abstentions count as failures. From the live leaderboard's overview-chain10 page.
APIFlow-Bench 1.0 - 5-Subtask Chain (Pass Rate (%)): leader Kimi K2.6 (100.0), 24 models
Postman's APIFlow-Bench 1.0 measures whether agents complete long, dependent REST API workflows (arXiv 2608.29128v1). This board covers the 5-subtask chain set (13 tasks); pass rate is passed over all trials, 0-100, higher is better, and refusals and abstentions count as failures. From the live leaderboard's overview-chain5 page.
APIFlow-Bench 1.0 - Solo Subtasks (Pass Rate (%)): leader Kimi K3 (97.84), 24 models
Postman's APIFlow-Bench 1.0 measures whether agents complete long, dependent REST API workflows (arXiv 2608.29128v1). This board covers the solo set of 241 single-subtask tasks; pass rate is passed over all trials, 0-100, higher is better, and refusals and abstentions count as failures. From the live leaderboard's overview-solo page.
APIFlow-Bench 1.0 - All Tasks (Pass Rate (%)): leader Kimi K3 (89.85), 24 models
Postman's APIFlow-Bench 1.0 measures whether agents complete long, dependent REST API workflows (arXiv 2608.29128v1). This board covers all 467 tasks across every task length; pass rate is passed over all trials, 0-100, higher is better, and refusals and abstentions count as failures. From the live leaderboard's overview-all page.
APIFlow-Bench 1.0 - Authentication (Pass Rate (%)): leader Kimi K3 (89.7), 24 models
Postman's APIFlow-Bench 1.0 pass rate on the tasks that exercise the authentication capability axis, across all task lengths (arXiv 2608.29128v1). Passed over all trials, 0-100 to one decimal, higher is better; refusals and abstentions count as failures. From the live leaderboard's axis page in its all-tasks view.
APIFlow-Bench 1.0 - Discovery (Pass Rate (%)): leader Claude Sonnet 4.6 (82.5), 24 models
Postman's APIFlow-Bench 1.0 pass rate on the tasks that exercise the discovery capability axis, across all task lengths (arXiv 2608.29128v1). Passed over all trials, 0-100 to one decimal, higher is better; refusals and abstentions count as failures. From the live leaderboard's axis page in its all-tasks view.
APIFlow-Bench 1.0 - Error Recovery (Pass Rate (%)): leader Kimi K3 (90.3), 24 models
Postman's APIFlow-Bench 1.0 pass rate on the tasks that exercise the error recovery capability axis, across all task lengths (arXiv 2608.29128v1). Passed over all trials, 0-100 to one decimal, higher is better; refusals and abstentions count as failures. From the live leaderboard's axis page in its all-tasks view.
APIFlow-Bench 1.0 - Multi-step Workflow (Pass Rate (%)): leader Kimi K3 (96.5), 24 models
Postman's APIFlow-Bench 1.0 pass rate on the tasks that exercise the multi-step workflow capability axis, across all task lengths (arXiv 2608.29128v1). Passed over all trials, 0-100 to one decimal, higher is better; refusals and abstentions count as failures. From the live leaderboard's axis page in its all-tasks view.
APIFlow-Bench 1.0 - Pagination (Pass Rate (%)): leader GPT-5.5 (92.2), 24 models
Postman's APIFlow-Bench 1.0 pass rate on the tasks that exercise the pagination capability axis, across all task lengths (arXiv 2608.29128v1). Passed over all trials, 0-100 to one decimal, higher is better; refusals and abstentions count as failures. From the live leaderboard's axis page in its all-tasks view.
APIFlow-Bench 1.0 - Schema (Pass Rate (%)): leader Kimi K2.6 (93.2), 24 models
Postman's APIFlow-Bench 1.0 pass rate on the tasks that exercise the schema capability axis, across all task lengths (arXiv 2608.29128v1). Passed over all trials, 0-100 to one decimal, higher is better; refusals and abstentions count as failures. From the live leaderboard's axis page in its all-tasks view.
APIFlow-Bench 1.0 - Statefulness (Pass Rate (%)): leader Kimi K3 (93.9), 24 models
Postman's APIFlow-Bench 1.0 pass rate on the tasks that exercise the statefulness capability axis, across all task lengths (arXiv 2608.29128v1). Passed over all trials, 0-100 to one decimal, higher is better; refusals and abstentions count as failures. From the live leaderboard's axis page in its all-tasks view.
ASI-Bench (Claude Code) (Scientific Score (0-100)): leader Claude Opus 5 (Max) (50.29), 13 models
ASI-Bench runs 60 project-level research tasks without external tool access (arXiv 2608.17271v1). Overall Scientific Score is the equal-weight mean of prompt bands B1-B4, macro-averaged over tasks and rounds, 0-100, higher is better. Claude Code harness runs only, from the Apexintelligence-AI/ASI-Bench-Leaderboard Space's app/page.tsx; efforts stay in the label, and two runs report one round.
ASI-Bench (Claude Code) - B1 Full Guidance (Scientific Score (0-100)): leader Claude Opus 5 (Max) (72.29), 13 models
ASI-Bench prompt band B1 (full guidance) on 60 project-level tasks without external tool access (arXiv 2608.17271v1). Scientific Score macro-averaged over tasks and rounds, 0-100, higher is better. Claude Code harness runs only, from the Apexintelligence-AI/ASI-Bench-Leaderboard Space's app/page.tsx; efforts stay in the label.
ASI-Bench (Claude Code) - B2 Method Hints (Scientific Score (0-100)): leader Claude Opus 5 (Max) (45.8), 13 models
ASI-Bench prompt band B2 (method hints) on 60 project-level tasks without external tool access (arXiv 2608.17271v1). Scientific Score macro-averaged over tasks and rounds, 0-100, higher is better. Claude Code harness runs only, from the Apexintelligence-AI/ASI-Bench-Leaderboard Space's app/page.tsx; efforts stay in the label.
ASI-Bench (Claude Code) - B3 Goal Only (Scientific Score (0-100)): leader Claude Opus 5 (Max) (40.7), 13 models
ASI-Bench prompt band B3 (goal only) on 60 project-level tasks without external tool access (arXiv 2608.17271v1). Scientific Score macro-averaged over tasks and rounds, 0-100, higher is better. Claude Code harness runs only, from the Apexintelligence-AI/ASI-Bench-Leaderboard Space's app/page.tsx; efforts stay in the label.
ASI-Bench (Claude Code) - B4 Goal + Distractors (Scientific Score (0-100)): leader Claude Opus 5 (Max) (42.39), 13 models
ASI-Bench prompt band B4 (goal plus distractors) on 60 project-level tasks without external tool access (arXiv 2608.17271v1). Scientific Score macro-averaged over tasks and rounds, 0-100, higher is better. Claude Code harness runs only, from the Apexintelligence-AI/ASI-Bench-Leaderboard Space's app/page.tsx; efforts stay in the label.
Thinking Machines Inkling Launch - FORTRESS Adversarial (Harmful-Request Refusal Rate (%)): leader Claude Fable 5 (Max) (96.0), 9 models
FORTRESS adversarial prompts (Scale AI; crime, violence and dual-use risks) run by Thinking Machines: the rate at which a model refuses harmful requests. Thinking Machines' Inkling launch post (July 2026), Safety rows of the full comparison table; percent, higher is better; Inkling models at effort 0.99. Not Scale's leaderboard risk score.
Thinking Machines Inkling Launch - FORTRESS Benign (Benign-Request Answer Rate (%)): leader DeepSeek V4 Pro (98.5), 9 models
FORTRESS benign look-alike prompts (Scale AI) run by Thinking Machines: the rate at which a model still answers safe requests that resemble harmful ones, so higher means less over-refusal. Thinking Machines' Inkling launch post (July 2026), Safety rows of the full comparison table; percent, higher is better; Inkling models at effort 0.99.
Thinking Machines Inkling Launch - StrongREJECT (Refusal Rate (%)): leader Kimi K2.6 (99.8), 9 models
StrongREJECT unambiguously harmful requests run by Thinking Machines: the rate at which a model refuses them. Thinking Machines' Inkling launch post (July 2026), Safety rows of the full comparison table; percent, higher is better; Inkling models at effort 0.99. Scores cluster between 97% and 100%, so differences are small.
DeepSeek DSBench - FullStack (Score (%)): leader Claude Fable 5 (77.2), 8 models
DeepSeek's internal DSBench-FullStack, a full-stack development test set for coding agents. DeepSeek-V4-Pro-0813 model card benchmark table (August 2026); score in percent, higher is better. DeepSeek's card notes run its models' agent benchmarks in the minimal mode of DeepSeek Harness at max reasoning effort; other models' setup is not stated. Claude Fable 5 was served with fallback.
DeepSeek DSBench - Hard (Score (%)): leader Claude Opus 4.8 (71.7), 9 models
DeepSeek's internal DSBench-Hard, a test set of difficult coding-agent problems. DeepSeek-V4-Pro-0813 model card table plus the DeepSeek-V4-Flash-Vision-Exp card, whose shared reference cells match; score in percent, higher is better. DeepSeek's card notes run its models' agent benchmarks in the minimal mode of DeepSeek Harness at max reasoning effort; other models' setup is not stated.
FrontierChallenge (Claude Code) (Pass Rate (%)): leader Grok 4.6 (20.6), 10 models
Apodex FrontierChallenge: agents complete specified scientific workflows across 97 tasks (materials, life science, chemistry and more) and deliver checkable artifact bundles; pass rate is the share of tasks with a native score of at least 99.9. Live leaderboard, models in the Claude Code scaffold; percent, higher is better.
FrontierChallenge (Codex) (Pass Rate (%)): leader GPT-5.6 Sol (20.6), 2 models
Apodex FrontierChallenge: agents complete specified scientific workflows across 97 tasks (materials, life science, chemistry and more) and deliver checkable artifact bundles; pass rate is the share of tasks with a native score of at least 99.9. Live leaderboard, models in the Codex scaffold; percent, higher is better.
IdeaAMBIG - Readiness Assessment - Real-World (Macro-F1 (%; 100 Ready and 100 NotReady specifications per subset)): leader GPT-5.6 Sol (67.5), 13 models
IdeaAMBIG (arXiv 2609.10539v1, Table 1): the model judges whether a research-idea specification is Ready or NotReady for implementation, on real-world specifications from GitHub issues and reproducibility reports. Score is Macro-F1 over 100 Ready and 100 NotReady specifications, in percent; higher is better. Zero-shot fixed prompts at temperature 0.
IdeaAMBIG - Readiness Assessment - Controlled Synthetic (Macro-F1 (%; 100 Ready and 100 NotReady specifications per subset)): leader GPT-5.6 Sol (86.4), 13 models
IdeaAMBIG (arXiv 2609.10539v1, Table 1): the model judges whether a research-idea specification is Ready or NotReady for implementation, on controlled synthetic specifications derived from ideation-execution trajectories. Score is Macro-F1 over 100 Ready and 100 NotReady specifications, in percent; higher is better. Zero-shot fixed prompts at temperature 0.
IdeaAMBIG - Defect Localization - Real-World (Macro Defect Recovery Rate (%)): leader GPT-5.6 Sol (9.6), 13 models
IdeaAMBIG (arXiv 2609.10539v1, Table 1): the model must name the implementation-critical blocker in an underspecified research idea and assign both taxonomy labels, on real-world specifications from GitHub issues and reproducibility reports. Score is Macro Defect Recovery Rate over gold Level-2 defect categories, which needs correct localization and both labels, in percent; higher is better. Zero-shot fixed prompts at temperature 0.
IdeaAMBIG - Defect Localization - Controlled Synthetic (Macro Defect Recovery Rate (%)): leader GPT-5.6 Sol (12.2), 13 models
IdeaAMBIG (arXiv 2609.10539v1, Table 1): the model must name the implementation-critical blocker in an underspecified research idea and assign both taxonomy labels, on controlled synthetic specifications derived from ideation-execution trajectories. Score is Macro Defect Recovery Rate over gold Level-2 defect categories, which needs correct localization and both labels, in percent; higher is better. Zero-shot fixed prompts at temperature 0.
IdeaAMBIG - Clarification Action Generation - Real-World (Macro Clarification Action Success Rate (%)): leader GPT-5.6 Sol (80.6), 13 models
IdeaAMBIG (arXiv 2609.10539v1, Table 1): the model is given the annotated defect and proposes an action that would obtain the missing information, on real-world specifications from GitHub issues and reproducibility reports. Score is Macro Clarification Action Success Rate (relevant, sufficient, no unsupported assumptions; Claude Opus 4.8 evaluator), in percent; higher is better. Zero-shot fixed prompts at temperature 0.
IdeaAMBIG - Clarification Action Generation - Controlled Synthetic (Macro Clarification Action Success Rate (%)): leader GPT-5.6 Sol (96.2), 13 models
IdeaAMBIG (arXiv 2609.10539v1, Table 1): the model is given the annotated defect and proposes an action that would obtain the missing information, on controlled synthetic specifications derived from ideation-execution trajectories. Score is Macro Clarification Action Success Rate (relevant, sufficient, no unsupported assumptions; Claude Opus 4.8 evaluator), in percent; higher is better. Zero-shot fixed prompts at temperature 0.
BrainBench (EEG) - BrainAgent - Overall (Difficulty-weighted EEG analysis score (0-100)): leader Gemini 3.6 Flash (Non-reasoning) (73.57), 13 models
BrainBench (arXiv 2608.04156v2, Table 1): LLMs analyse real EEG recordings from 17 datasets (172 tasks) through the structured BrainAgent tool workflow in isolated containers and write reports checked by numerical, categorical, semantic and artifact validators. Overall score across four subsets, 0-100; higher is better. Optional reasoning modes disabled.
BrainBench (EEG) - CodeAct - Overall (Difficulty-weighted EEG analysis score (0-100)): leader Claude Opus 5 (76.25), 13 models
BrainBench (arXiv 2608.04156v2, Table 1): LLMs analyse real EEG recordings from 17 datasets (172 tasks) through the free-form CodeAct Python execution loop in isolated containers and write reports checked by numerical, categorical, semantic and artifact validators. Overall score across four subsets, 0-100; higher is better. Optional reasoning modes disabled.
BrainBench (EEG) - BrainAgent - Foundational Analysis (EEG analysis score (0-100)): leader Claude Opus 5 (81.47), 13 models
BrainBench (arXiv 2608.04156v2, Table 1) score on the Foundational Analysis subset of core EEG processing and analysis tasks, with each model running through the structured BrainAgent tool workflow in an isolated container and its report and artifacts checked by task validators. 0-100; higher is better. Optional reasoning modes disabled.
BrainBench (EEG) - BrainAgent - Sleep Assessment (EEG analysis score (0-100)): leader Gemini 3.6 Flash (Non-reasoning) (72.95), 13 models
BrainBench (arXiv 2608.04156v2, Table 1) score on the Sleep Assessment subset built on clinical sleep recordings, with each model running through the structured BrainAgent tool workflow in an isolated container and its report and artifacts checked by task validators. 0-100; higher is better. Optional reasoning modes disabled.
BrainBench (EEG) - BrainAgent - Neurocognitive Assessment (EEG analysis score (0-100)): leader Gemini 3.6 Flash (Non-reasoning) (74.71), 13 models
BrainBench (arXiv 2608.04156v2, Table 1) score on the Neurocognitive Assessment subset built on cognitive-experiment EEG, with each model running through the structured BrainAgent tool workflow in an isolated container and its report and artifacts checked by task validators. 0-100; higher is better. Optional reasoning modes disabled.
BrainBench (EEG) - BrainAgent - Physiological Integration (EEG analysis score (0-100)): leader Gemini 3.6 Flash (Non-reasoning) (69.24), 13 models
BrainBench (arXiv 2608.04156v2, Table 1) score on the Physiological Integration subset combining EEG with other physiological signals, with each model running through the structured BrainAgent tool workflow in an isolated container and its report and artifacts checked by task validators. 0-100; higher is better. Optional reasoning modes disabled.
BrainBench (EEG) - CodeAct - Foundational Analysis (EEG analysis score (0-100)): leader Claude Opus 5 (84.97), 13 models
BrainBench (arXiv 2608.04156v2, Table 1) score on the Foundational Analysis subset of core EEG processing and analysis tasks, with each model running through the free-form CodeAct Python execution loop in an isolated container and its report and artifacts checked by task validators. 0-100; higher is better. Optional reasoning modes disabled.
BrainBench (EEG) - CodeAct - Sleep Assessment (EEG analysis score (0-100)): leader Claude Opus 5 (73.53), 13 models
BrainBench (arXiv 2608.04156v2, Table 1) score on the Sleep Assessment subset built on clinical sleep recordings, with each model running through the free-form CodeAct Python execution loop in an isolated container and its report and artifacts checked by task validators. 0-100; higher is better. Optional reasoning modes disabled.
BrainBench (EEG) - CodeAct - Neurocognitive Assessment (EEG analysis score (0-100)): leader Claude Opus 5 (70.03), 13 models
BrainBench (arXiv 2608.04156v2, Table 1) score on the Neurocognitive Assessment subset built on cognitive-experiment EEG, with each model running through the free-form CodeAct Python execution loop in an isolated container and its report and artifacts checked by task validators. 0-100; higher is better. Optional reasoning modes disabled.
BrainBench (EEG) - CodeAct - Physiological Integration (EEG analysis score (0-100)): leader Claude Opus 5 (76.49), 13 models
BrainBench (arXiv 2608.04156v2, Table 1) score on the Physiological Integration subset combining EEG with other physiological signals, with each model running through the free-form CodeAct Python execution loop in an isolated container and its report and artifacts checked by task validators. 0-100; higher is better. Optional reasoning modes disabled.
LLM Milgram Obedience - Baseline Full-Obedience Rate (Valid baseline sessions ending at 450 V (%; lower is less obedient)): leader MiMo-V2.5 (Non-reasoning) (0.0), 39 models
Share of valid baseline sessions in which an LLM playing Milgram's Teacher, under a scripted Experimenter, administers the full 450 V shock (voice-feedback condition, temperature 1.0, reasoning disabled, at most 15 sessions). From the authors' census file for arXiv 2608.16177v2. Percent; lower means less obedience to harmful authority.
LLM Milgram Obedience - Baseline Mean Breakoff Voltage (Mean last shock administered in valid baseline sessions (volts, 0-450; lower is less obedient)): leader Grok 4.6 (0.0), 39 models
Mean breakoff voltage, the last shock (0-450 V) an LLM playing Milgram's Teacher administers under a scripted Experimenter, over valid baseline sessions at temperature 1.0 with reasoning disabled (at most 15 per model). From the authors' census file for arXiv 2608.16177v2. Lower means less obedience to harmful authority.
PhenoBench-LLM - Common-Task Win Rate (Pairwise win rate (%; 11 tasks run by all models)): leader Gemini 3.7 Flash (82.517483), 14 models
PhenoBench-LLM (arXiv 2609.06080v1 leaderboard; release file win_rates_common.csv): pairwise win rate of 14 language models predicting Human Phenotype Project targets from a per-participant evidence packet, over the 11 validation tasks run by every model. Native task metrics are compared only within a task; percent, higher is better.
PhenoBench-LLM - All-Task Win Rate (Pairwise win rate (%; all available head-to-heads, 22-40 tasks per model)): leader Gemini 3.7 Flash (79.004329), 14 models
PhenoBench-LLM (arXiv 2609.06080v1; release file win_rates.csv): pairwise win rate of 14 language models over all available head-to-heads on the validation tasks each model ran (22 to 40 of 40), so coverage differs by model. Native task metrics are compared only within a task; percent, higher is better.
BoundaryBench (Non-root) (Success Rate under non-root policy (%)): leader Grok 4.5 (73.0), 12 models
BoundaryBench success on Terminal-Bench 2.1 (89 tasks, three trials each) with the coding agent running as an ordinary non-root user and everything else open, from the Permission Denied paper (arXiv 2608.02670v1, Appendix Table 12). Twelve model-harness bundles at high reasoning effort; percent of passing trials, higher is better.
DrawAI-Bench - Fidelity (Fidelity score (0-100; rules and VLM rubric, strongest observed harness)): leader GPT-5.6 Sol (94.0), 13 models
DrawAI-Bench (arXiv 2608.00548v1, Table 2): coding agents convert 80 raster diagrams, posters, slides and scientific figures into editable SVG with the DrawAI-Flow workflow, scored by deterministic rules and VLM rubric questions; each model appears under the strongest harness observed for it. Overall visual fidelity of the rendered reconstruction to the source; 0-100, higher is better.
DrawAI-Bench - Editability (Editability score (0-100; rules and VLM rubric, strongest observed harness)): leader Claude Opus 4.8 (Claude Code) (91.6), 13 models
DrawAI-Bench (arXiv 2608.00548v1, Table 2): coding agents convert 80 raster diagrams, posters, slides and scientific figures into editable SVG with the DrawAI-Flow workflow, scored by deterministic rules and VLM rubric questions; each model appears under the strongest harness observed for it. Overall editability of the SVG structure (native text, shapes, connectors, tables and formulas); 0-100, higher is better.
DrawAI-Bench - Fidelity - Global Visual (Fidelity score (0-100; rules and VLM rubric, strongest observed harness)): leader GPT-5.6 Sol (89.6), 13 models
DrawAI-Bench (arXiv 2608.00548v1, Table 2): coding agents convert 80 raster diagrams, posters, slides and scientific figures into editable SVG with the DrawAI-Flow workflow, scored by deterministic rules and VLM rubric questions; each model appears under the strongest harness observed for it. Asset-level fidelity: whole-page structure and edge match, pixel error, readability, layout and visual hierarchy; 0-100, higher is better.
DrawAI-Bench - Fidelity - Text (Fidelity score (0-100; rules and VLM rubric, strongest observed harness)): leader GPT-5.5 (95.7), 13 models
DrawAI-Bench (arXiv 2608.00548v1, Table 2): coding agents convert 80 raster diagrams, posters, slides and scientific figures into editable SVG with the DrawAI-Flow workflow, scored by deterministic rules and VLM rubric questions; each model appears under the strongest harness observed for it. Asset-level fidelity: text content match plus judged position, alignment, overflow, colour, font and style; 0-100, higher is better.
DrawAI-Bench - Fidelity - Image (Fidelity score (0-100; rules and VLM rubric, strongest observed harness)): leader GPT-5.6 Sol (87.9), 13 models
DrawAI-Bench (arXiv 2608.00548v1, Table 2): coding agents convert 80 raster diagrams, posters, slides and scientific figures into editable SVG with the DrawAI-Flow workflow, scored by deterministic rules and VLM rubric questions; each model appears under the strongest harness observed for it. Asset-level fidelity: rule-based pixel error and placement of image regions; 0-100, higher is better.
DrawAI-Bench - Fidelity - Formula (Fidelity score (0-100; rules and VLM rubric, strongest observed harness)): leader Claude Fable 5 (Claude Code) (99.0), 13 models
DrawAI-Bench (arXiv 2608.00548v1, Table 2): coding agents convert 80 raster diagrams, posters, slides and scientific figures into editable SVG with the DrawAI-Flow workflow, scored by deterministic rules and VLM rubric questions; each model appears under the strongest harness observed for it. Asset-level fidelity: judged symbol retention, structure and readability of formulas; 0-100, higher is better.
DrawAI-Bench - Fidelity - Shape (Fidelity score (0-100; rules and VLM rubric, strongest observed harness)): leader GPT-5.5 (97.7), 13 models
DrawAI-Bench (arXiv 2608.00548v1, Table 2): coding agents convert 80 raster diagrams, posters, slides and scientific figures into editable SVG with the DrawAI-Flow workflow, scored by deterministic rules and VLM rubric questions; each model appears under the strongest harness observed for it. Asset-level fidelity: judged presence, visual attributes and neighbour relations of shapes; 0-100, higher is better.
DrawAI-Bench - Fidelity - Connector (Fidelity score (0-100; rules and VLM rubric, strongest observed harness)): leader GPT-5.6 Sol (96.8), 13 models
DrawAI-Bench (arXiv 2608.00548v1, Table 2): coding agents convert 80 raster diagrams, posters, slides and scientific figures into editable SVG with the DrawAI-Flow workflow, scored by deterministic rules and VLM rubric questions; each model appears under the strongest harness observed for it. Asset-level fidelity: judged route, endpoints and line style of connectors; 0-100, higher is better.
DrawAI-Bench - Fidelity - Table (Fidelity score (0-100; rules and VLM rubric, strongest observed harness)): leader GPT-5.6 Sol (98.2), 13 models
DrawAI-Bench (arXiv 2608.00548v1, Table 2): coding agents convert 80 raster diagrams, posters, slides and scientific figures into editable SVG with the DrawAI-Flow workflow, scored by deterministic rules and VLM rubric questions; each model appears under the strongest harness observed for it. Asset-level fidelity: judged grid structure, cell content ownership and readability of tables; 0-100, higher is better.
DrawAI-Bench - Editability - Text (Editability score (0-100; rules and VLM rubric, strongest observed harness)): leader Claude Opus 4.8 (Claude Code) (88.7), 13 models
DrawAI-Bench (arXiv 2608.00548v1, Table 2): coding agents convert 80 raster diagrams, posters, slides and scientific figures into editable SVG with the DrawAI-Flow workflow, scored by deterministic rules and VLM rubric questions; each model appears under the strongest harness observed for it. Asset-level editability: rule-based share of text reconstructed as editable text boxes; 0-100, higher is better.
DrawAI-Bench - Editability - Image (Editability score (0-100; rules and VLM rubric, strongest observed harness)): leader Kimi K3 (93.8), 13 models
DrawAI-Bench (arXiv 2608.00548v1, Table 2): coding agents convert 80 raster diagrams, posters, slides and scientific figures into editable SVG with the DrawAI-Flow workflow, scored by deterministic rules and VLM rubric questions; each model appears under the strongest harness observed for it. Asset-level editability: rule-based share of image regions kept as separate image sources; 0-100, higher is better.
DrawAI-Bench - Editability - Shape (Editability score (0-100; rules and VLM rubric, strongest observed harness)): leader GPT-5.5 (93.2), 13 models
DrawAI-Bench (arXiv 2608.00548v1, Table 2): coding agents convert 80 raster diagrams, posters, slides and scientific figures into editable SVG with the DrawAI-Flow workflow, scored by deterministic rules and VLM rubric questions; each model appears under the strongest harness observed for it. Asset-level editability: judged independence, placement and clean separation of shape objects; 0-100, higher is better.
DrawAI-Bench - Editability - Connector (Editability score (0-100; rules and VLM rubric, strongest observed harness)): leader GPT-5.5 (97.6), 13 models
DrawAI-Bench (arXiv 2608.00548v1, Table 2): coding agents convert 80 raster diagrams, posters, slides and scientific figures into editable SVG with the DrawAI-Flow workflow, scored by deterministic rules and VLM rubric questions; each model appears under the strongest harness observed for it. Asset-level editability: judged independent connector paths with editable endpoints; 0-100, higher is better.
DrawAI-Bench - Editability - Formula (Editability score (0-100; rules and VLM rubric, strongest observed harness)): leader Claude Opus 4.8 (Claude Code) (100.0), 13 models
DrawAI-Bench (arXiv 2608.00548v1, Table 2): coding agents convert 80 raster diagrams, posters, slides and scientific figures into editable SVG with the DrawAI-Flow workflow, scored by deterministic rules and VLM rubric questions; each model appears under the strongest harness observed for it. Asset-level editability: rule-based share of formulas rebuilt as native editable formulas; 0-100, higher is better.
DrawAI-Bench - Editability - Table (Editability score (0-100; rules and VLM rubric, strongest observed harness)): leader MiniMax M3 (Claude Code) (90.7), 13 models
DrawAI-Bench (arXiv 2608.00548v1, Table 2): coding agents convert 80 raster diagrams, posters, slides and scientific figures into editable SVG with the DrawAI-Flow workflow, scored by deterministic rules and VLM rubric questions; each model appears under the strongest harness observed for it. Asset-level editability: judged independently editable table regions, grid geometry and cell text; 0-100, higher is better.
F5 Labs AI Security Leaderboard - CASI (CASI (Comprehensive AI Security Index), 0-100, higher is more secure (September 2026 top-10 snapshot)): leader Claude Fable 5 (98.03), 10 models
F5 Labs' CASI (Comprehensive AI Security Index, from F5 AI Red-Team, formerly CalypsoAI) scores how well a model resists prompt-injection and jailbreak attacks drawn from a monthly refreshed attack library, weighting attacks by the sophistication needed to succeed. Scale 0-100, higher is more secure. F5 publishes only its ten highest-scoring models each month, so this board is a monthly top-10 snapshot.
Vending-Bench Arena - Round 12 (Final money balance ($, mean over runs)): leader GPT-6 (12362.6), 3 models
Vending-Bench Arena round 12 (Andon Labs, September 4, 2026): GPT-6 Astra, GLM-5.3 and Claude Fable 5.1 each run a vending machine at one location in the Vending-Bench 2 simulation and can email, pay and trade with each other. Final money balance in US dollars (start $500), averaged over 3 runs; higher is better. Balances depend on the opponents in that round, so they do not compare across rounds.
Vending-Bench Arena - Round 11 (Final money balance ($, mean over runs)): leader GPT-5.6 Sol (7443.94), 3 models
Vending-Bench Arena round 11 (Andon Labs, July 24, 2026): Claude Opus 5, Kimi K3 and GPT-5.6 Sol each run a vending machine at one location in the Vending-Bench 2 simulation and can email, pay and trade with each other. Final money balance in US dollars (start $500), averaged over 6 runs; higher is better. Balances depend on the opponents in that round, so they do not compare across rounds.
Vending-Bench Arena - Round 10 (Final money balance ($, mean over runs)): leader GPT-5.6 Terra (9191.35), 3 models
Vending-Bench Arena round 10 (Andon Labs, July 9, 2026): GPT-5.6 Sol, Terra and Luna each run a vending machine at one location in the Vending-Bench 2 simulation and can email, pay and trade with each other. Final money balance in US dollars (start $500), averaged over 5 runs; higher is better. Balances depend on the opponents in that round, so they do not compare across rounds.
Kaggle Adversarial Customer Service (Gert Labs) (Score (%)): leader Claude Opus 4.8 (78.33), 9 models
Gert Labs' Adversarial Customer Service benchmark on Kaggle (leaderboard version 1): LLM agents compete in a hidden-information customer service simulation conducted in natural language. Score is the task result that Kaggle displays as a percentage; higher is better, 0-100.
Qwen3.8 Launch - QwenSWEBench (avg@3 Score (%)): leader Claude Fable 5 (86.3), 5 models
Alibaba Qwen's in-house software-engineering benchmark from the Qwen3.8-Max launch post: every model runs in the Claude Code harness with an 8-hour timeout, 32,768 output tokens, temperature 1.0 and a 256K context window. Score is avg@3 in percent, higher is better; the post notes that Claude Fable 5 results may involve fallbacks.
Qwen3.8 Launch - QwenQoderBench (avg@5 Score (%)): leader Claude Fable 5 (63.1), 5 models
Alibaba Qwen's in-house coding benchmark for the user experience of its Qoder coding tool, from the Qwen3.8-Max launch post: models run in the Claude Code harness with a 6-hour timeout, 32,768 output tokens, temperature 1.0 and a 256K context window. Score is avg@5 in percent, higher is better; Claude Fable 5 results may involve fallbacks.
Qwen3.8 Launch - PhyX (Score (%)): leader Qwen 3.8 Max (83.5), 6 models
PhyX tests multimodal physical reasoning with image-grounded physics problems spanning mechanics, electromagnetism, thermodynamics, waves and acoustics, optics and modern physics. Scores come from the multimodal comparison table of Alibaba Qwen's Qwen3.8-Max launch post; percent, higher is better.
Qwen3.8 Launch - QwenBlenderBench (Score (%)): leader Qwen 3.8 Max (69.9), 6 models
Alibaba Qwen's internal benchmark of agentic 3D work in Blender, listed under Visual Agent & Coding in the multimodal table of the Qwen3.8-Max launch post, which gives no further protocol detail. Score in percent, higher is better.
Qwen3.8 Launch - RecreationBench (Score (%)): leader Claude Fable 5 (56.1), 6 models
Alibaba Qwen's internal long-horizon application-recreation benchmark: a model observes a running application on Ubuntu, macOS, Windows, Android or the web only as a black box, with no source code or internet access, and rebuilds it through iterative coding and GUI interaction. From the Qwen3.8-Max launch post; score in percent, higher is better.
Qwen3.8 Launch - PresentBench (Score (%)): leader GPT-5.6 Sol (82.9), 6 models
PresentBench evaluates slide generation from long background materials in academia, education, economics, talks and advertising, graded against fine-grained per-instance checklists of presentation, design, completeness, correctness and fidelity items. Scores come from the multimodal table of the Qwen3.8-Max launch post; percent, higher is better.
Qwen3.8 Launch - MADQA (Score (%)): leader Qwen 3.8 Max (91.8), 6 models
MADQA (Multimodal Agentic Document Question Answering) asks agents to search, gather and reason over visual and textual evidence in a collection of heterogeneous PDF documents to answer human-authored questions. Scores come from the multimodal table of Alibaba Qwen's Qwen3.8-Max launch post; percent, higher is better.
Qwen3.8 Launch - QwenVisualOffice (Score (%)): leader Qwen 3.8 Max (44.6), 6 models
Alibaba Qwen's internal benchmark of visual office-document work, listed under Document & Office Intelligence in the multimodal table of the Qwen3.8-Max launch post, which gives no further protocol detail. Score in percent, higher is better.
Qwen3.8 Launch - RefAdv-S (Score (%)): leader Qwen 3.8 Max (80.2), 6 models
Ref-Adv-S is the public subset of Ref-Adv, a referring-expression comprehension benchmark that removes shortcuts with long expressions, negations and several hard distractors per image, so a model must reason to ground the described object. Scores come from the multimodal table of the Qwen3.8-Max launch post; percent, higher is better.
Qwen3.8 Launch - Dense200 (Score (%)): leader Qwen 3.8 Max (87.0), 6 models
Dense200 is a manually collected set of 200 densely annotated images, averaging about 91 small boxes per image across 109 categories, for detecting and grounding many small objects in crowded scenes. Scores come from the perception and grounding rows of the Qwen3.8-Max launch post's multimodal table; percent, higher is better.
Qwen3.8 Launch - VLMsAreBiased (Score (%)): leader Qwen 3.8 Max (88.3), 6 models
VLMs Are Biased tests whether vision-language models read what is actually in an image or answer from prior knowledge, using counting questions on counterfactually edited images of familiar subjects, such as a logo with an extra stripe. Scores come from the multimodal table of the Qwen3.8-Max launch post; percent, higher is better.
Qwen3.8 Launch - EgoLife (with Memory) (Score (%)): leader Claude Fable 5 (82.3), 5 models
EgoLife question answering over long egocentric life-log video, run with a memory system built on Qwen-MM-Plugins that gives the model fine-grained long-horizon video memory, from the Qwen3.8-Max launch post's multimodal table. Score in percent, higher is better; Gemini 3.1 Pro has no cell.
Qwen3.8 Launch - VideoDR (with Search) (Score (%)): leader Claude Fable 5 (77.1), 5 models
VideoDR is a video deep-research benchmark: agents pick up visual cues across video frames, search the open web over several hops and combine both to answer a factual question. From the Qwen3.8-Max launch post's multimodal table, run with access to a search tool; score in percent, higher is better; Gemini 3.1 Pro has no cell.
Kaggle Cipher Frontlines (Ciphers Solved (%)): leader Gemini 3 Flash (Preview) (90.91), 39 models
Community benchmark on Kaggle by hanifnoerrofiq: 11 classical ciphers set inside short stories (Caesar, Vigenere, rail fence, affine, Bifid, scytale, Baconian, four-square, ADFGX, a plugboard plus Atbash, Enigma) that the model decodes from the narrative clues. A task passes when the expected plaintext appears in the letters of the reply; percent of ciphers solved, higher is better.
Kaggle BackTrace Backend Debugging (Scenarios Passed (%)): leader GPT-5.4 Mini (81.25), 36 models
Community benchmark on Kaggle by himanshunakrani: 16 backend incident scenarios described through logs and metrics, such as missing indexes, retry loops, cache stampedes, Kafka partition skew and connection-pool leaks. A scenario passes when a judge model accepts every root-cause and remedy criterion; percent of scenarios passed, higher is better.
Fucc Boi Bench (Average Fuccboi Score (0-10, lower is better)): leader Claude Opus 5 (2.021), 41 models
Model-judged behaviour benchmark by Patrick Lee: 41 models each reply to 96 English dating-message requests in eight families such as flirting, honesty, consent, breadcrumbing and accountability. A GLM 5.3 judge grades honesty, clarity and respect; each reply scores 0-10 with extra points for serious misses like ignoring a clear no, and the board is a model's mean. Lower is better; scores track refusal rate closely.
Epoch AI - GPQA Diamond (Accuracy (%)): leader GPT-6 (Max) (95.77), 313 models
Epoch AI's own runs of GPQA Diamond, the 198 graduate-level biology, chemistry and physics multiple-choice questions, evaluated in Epoch's Inspect harness rather than taken from vendor reports. Each row is one Epoch run id, which keeps the reasoning effort or thinking budget used; score is the best accuracy across Epoch's scorers in percent, higher is better, random guessing 25. Evaluator-distinct from the vendor-reported GPQA Diamond boards.
GB/T-Bench - Direct Review CMCS (CMCS (0-1; coverage with missed and redundant diagnosis penalties)): leader GPT-5.6 Sol (0.328), 14 models
GB/T-Bench (arXiv 2608.06312v1): review of 488 Chinese national standard documents seeded with 7,306 errors across 25 error types, by direct LLM review without the authors' multi-agent reviewer. CMCS combines exact diagnosis coverage with penalties for missed and redundant predictions, 0-1; human experts score 0.664 (Table 2). Higher is better.
GB/T-Bench - Direct Review Diagnosis Recall (Diagnosis recall (%; exact section, dimension and error-type match)): leader GPT-5.6 Sol (52.03), 14 models
GB/T-Bench (arXiv 2608.06312v1): review of 488 Chinese national standard documents seeded with 7,306 errors across 25 error types, by direct LLM review without the authors' multi-agent reviewer. Share of seeded errors whose section, review dimension and error type all match, percent; human experts reach 83.1 (Table 2). Higher is better.
GB/T-Bench - Direct Review DMTR@8 (Documents with at least 8 correct diagnoses (%)): leader GPT-5.6 Sol (54.35), 14 models
GB/T-Bench (arXiv 2608.06312v1): review of 488 Chinese national standard documents seeded with 7,306 errors across 25 error types, by direct LLM review without the authors' multi-agent reviewer. Share of documents with at least 8 exactly diagnosed errors (each has 10 to 18 seeded), percent; human experts reach 94.4 (Table 2). Higher is better.
GB/T-Bench - Direct Review DMTR@9 (Documents with at least 9 correct diagnoses (%)): leader GPT-5.5 (39.34), 14 models
GB/T-Bench (arXiv 2608.06312v1): review of 488 Chinese national standard documents seeded with 7,306 errors across 25 error types, by direct LLM review without the authors' multi-agent reviewer. Share of documents with at least 9 exactly diagnosed errors (each has 10 to 18 seeded), percent; human experts reach 90.2 (Table 2). Higher is better.
GB/T-Bench - Direct Review DMTR@10 (Documents with at least 10 correct diagnoses (%)): leader Claude Fable 5 (26.09), 14 models
GB/T-Bench (arXiv 2608.06312v1): review of 488 Chinese national standard documents seeded with 7,306 errors across 25 error types, by direct LLM review without the authors' multi-agent reviewer. Share of documents with at least 10 exactly diagnosed errors (each has 10 to 18 seeded), percent; human experts reach 84.8 (Table 2). Higher is better.
GB/T-Bench - Diagnosis Recall - Document Structure (Diagnosis recall (%; exact section, dimension and error-type match)): leader GPT-5.6 Sol (75.81), 14 models
GB/T-Bench (arXiv 2608.06312v1): review of 488 Chinese national standard documents seeded with 7,306 errors across 25 error types, by direct LLM review without the authors' multi-agent reviewer. Exact diagnosis recall for errors in the document-structure review dimension, such as section hierarchy and headings, percent (Table 3). Higher is better.
GB/T-Bench - Diagnosis Recall - Scope Consistency (Diagnosis recall (%; exact section, dimension and error-type match)): leader GPT-5.5 (81.81), 14 models
GB/T-Bench (arXiv 2608.06312v1): review of 488 Chinese national standard documents seeded with 7,306 errors across 25 error types, by direct LLM review without the authors' multi-agent reviewer. Exact diagnosis recall for errors in the scope review dimension, such as scope statements inconsistent with the content, percent (Table 3). Higher is better.
GB/T-Bench - Diagnosis Recall - Normative Modality (Diagnosis recall (%; exact section, dimension and error-type match)): leader GPT-5.5 (41.66), 14 models
GB/T-Bench (arXiv 2608.06312v1): review of 488 Chinese national standard documents seeded with 7,306 errors across 25 error types, by direct LLM review without the authors' multi-agent reviewer. Exact diagnosis recall for errors in the normative-modality review dimension, the required and permitted provision wording, percent (Table 3). Higher is better.
GB/T-Bench - Diagnosis Recall - Terminology (Diagnosis recall (%; exact section, dimension and error-type match)): leader Gemini 3.5 Flash (25.97), 14 models
GB/T-Bench (arXiv 2608.06312v1): review of 488 Chinese national standard documents seeded with 7,306 errors across 25 error types, by direct LLM review without the authors' multi-agent reviewer. Exact diagnosis recall for errors in the terminology review dimension, where defined terms and their use must align, percent (Table 3). Higher is better.
GB/T-Bench - Diagnosis Recall - Normative References (Diagnosis recall (%; exact section, dimension and error-type match)): leader Qwen 2.5 VL 72B Instruct (60.4), 14 models
GB/T-Bench (arXiv 2608.06312v1): review of 488 Chinese national standard documents seeded with 7,306 errors across 25 error types, by direct LLM review without the authors' multi-agent reviewer. Exact diagnosis recall for errors in the normative-reference review dimension, which needs cross-section and cited-standard checks, percent (Table 3). Higher is better.
Edit2TikZ - Edit Correctness Score (ECS (0-100; judged per edit unit, zero for render failures)): leader Gemini 3.1 Pro (Preview) (75.02), 14 models
Edit2TikZ (arXiv 2608.13441v1, Table 3): 1,548 scientific-figure edits where a multimodal model gets a rendered figure and an instruction and must write compilable TikZ for the revised figure. Edit Correctness Score averages the judged completion of each labelled edit unit, 0-100, zero for render failures; GPT-5.6-Terra is the judge and is also evaluated. Higher is better.
Edit2TikZ - Restoration Score (RS (0-100; non-target preservation, zero for render failures)): leader Gemini 3.1 Pro (Preview) (74.51), 14 models
Edit2TikZ (arXiv 2608.13441v1, Table 3): 1,548 scientific-figure edits where a multimodal model gets a rendered figure and an instruction and must write compilable TikZ for the revised figure. Restoration Score is the judged preservation of non-target text, objects, relations and layout, 0-100, zero for render failures; GPT-5.6-Terra judges. Higher is better.
Edit2TikZ - Compilation Success Rate (Compiled and rendered outputs (%)): leader Gemini 3.1 Pro (Preview) (88.37), 14 models
Edit2TikZ (arXiv 2608.13441v1, Table 3): 1,548 scientific-figure edits where a multimodal model gets a rendered figure and an instruction and must write compilable TikZ for the revised figure. Share of outputs that compile and render in a shared TeX Live environment, percent. Higher is better.

Top-10 New Scores (41)

Claude Fable 5 on Vals AI (Vals Index): 66.04 (#4)
Claude Fable 5 on Vals AI Code Migration: 55.06 (#3)
Claude Fable 5 on Vals AI Excel Modeling: 73.67 (#2)
Claude Fable 5 on Vals AI Finance Agent v2: 56.31 (#11)
Claude Fable 5 on Vals AI Harvey Legal Agent Bench: 11.25 (#8)
Claude Fable 5 on Vals AI Legal Research Bench: 49.52 (#4)
Claude Fable 5 on Vals AI Terminal-Bench Science: 12.86 (#5)
Claude Fable 5 on Vals AI Vibe Code Bench: 90.35 (#1)
Claude Fable 5.1 on Kaggle Game Arena Chess: 897.0 (#13)
Claude Fable 5.1 on Kaggle Game Arena Four in a Row: 761.73 (#2)
Claude Fable 5.1 on Vals AI (Vals Index): 68.83 (#1)
Claude Fable 5.1 on Vals AI Code Migration: 54.61 (#4)
Claude Fable 5.1 on Vals AI Excel Modeling: 76.67 (#1)
Claude Fable 5.1 on Vals AI Finance Agent v2: 58.88 (#6)
Claude Fable 5.1 on Vals AI Harvey Legal Agent Bench: 6.67 (#22)
Claude Fable 5.1 on Vals AI Legal Research Bench: 55.29 (#3)
Claude Fable 5.1 on Vals AI Terminal-Bench 2.1: 85.02 (#3)
Claude Fable 5.1 on Vals AI Terminal-Bench Science: 34.29 (#2)
Claude Fable 5.1 on Vals AI Time Horizon Index: 63.33 (#2)
Claude Fable 5.1 on Vals AI Vibe Code Bench: 90.26 (#2)
Claude Opus 5 on Kaggle Game Arena Chess: 833.14 (#16)
Claude Opus 5 on Vals AI (Vals Index): 67.21 (#2)
Claude Opus 5 on Vals AI Code Migration: 57.47 (#2)
Claude Opus 5 on Vals AI Excel Modeling: 73.56 (#3)
Claude Opus 5 on Vals AI Finance Agent v2: 58.63 (#7)
Claude Opus 5 on Vals AI Harvey Legal Agent Bench: 6.67 (#21)
Claude Opus 5 on Vals AI Legal Research Bench: 55.29 (#2)
Claude Opus 5 on Vals AI Terminal-Bench 2.1: 84.64 (#4)
Claude Opus 5 on Vals AI Terminal-Bench Science: 22.86 (#3)
GPT-6 on Kaggle FACTS Multimodal: 47.73 (#3)
GPT-6 on ProphetArena: 0.955 (#2)
GPT-6 on Vals AI (Vals Index): 66.61 (#3)
GPT-6 on Vals AI Code Migration: 67.74 (#1)
GPT-6 on Vals AI Excel Modeling: 71.7 (#6)
GPT-6 on Vals AI Finance Agent v2: 53.54 (#21)
GPT-6 on Vals AI Harvey Legal Agent Bench: 5.42 (#23)
GPT-6 on Vals AI Legal Research Bench: 39.42 (#19)
GPT-6 on Vals AI Terminal-Bench 2.1: 87.27 (#1)
GPT-6 on Vals AI Terminal-Bench Science: 65.71 (#1)
GPT-6 on Vals AI Time Horizon Index: 90.5 (#1)
GPT-6 on Vals AI Vibe Code Bench: 89.59 (#3)

New #1 Leaders (5)

Kaggle Game Arena Four in a Row: Claude Opus 5 (852.42) beat Gemini 3.1 Pro (Preview) by 114.16
Kaggle Game Arena Chess: Gemini 3.8 Flash (1498.54) beat Gemini 3.1 Pro (Preview) by 103.28
Vals AI Time Horizon Index: GPT-6 (Max) (90.5) beat Claude Opus 5 (Max) by 76.83
SEAL - Professional Reasoning Benchmark - Finance: Muse Spark 1.3 (59.54) beat Muse Spark 1.1 by 4.53
SEAL - Professional Reasoning Benchmark - Legal: Muse Spark 1.3 (61.56) beat Muse Spark 1.1 by 4.51

The Aggregate Digest — 2026-09-14

2026-09-14T05:47:30.998024+00:00

Daily

Top-10 New Scores (17)

Claude Fable 5 on AA-Omniscience Accuracy: 65.35 (#3)
Claude Fable 5.1 on Chatbot Arena (Document): 1513.0 (#2)
Claude Fable 5.1 on Chatbot Arena (Vision - Creative Writing): 1290.0 (#11)
Claude Fable 5.1 on Chatbot Arena (Vision - Diagram): 1323.0 (#9)
Claude Fable 5.1 on Chatbot Arena (Vision - English): 1280.0 (#18)
Claude Fable 5.1 on Chatbot Arena (Vision - OCR): 1305.0 (#10)
Claude Fable 5.1 on Chatbot Arena (Vision): 1289.0 (#11)
Claude Fable 5.1 on Kaggle Game Arena Poker (Heads Up): 29.68 (#3)
Claude Opus 5 on AA-Omniscience Accuracy: 60.87 (#9)
Claude Opus 5 on Architectural RedlineBench: 62.7 (#2)
GPT-6 on Chatbot Arena (Document): 1468.0 (#16)
GPT-6 on Chatbot Arena (Vision - Diagram): 1318.0 (#12)
GPT-6 on Chatbot Arena (Vision - English): 1289.0 (#8)
GPT-6 on Chatbot Arena (Vision - OCR): 1299.0 (#16)
GPT-6 on Chatbot Arena (Vision): 1284.0 (#16)
GPT-6 on LLM Chess (Saplin): 1539.2 (#4)
GPT-6 on WolfBench: 86.0 (#2)

New #1 Leaders (4)

HumanCLAW-Bench - Interact: GPT-6 (46.6) beat Gemini-3.1 by 29.8
HumanCLAW-Bench - Navigate: GPT-6 (57.1) beat Gemini-3.1 by 14.6
HumanCLAW-Bench - Find: GPT-6 (75.5) beat Gemini-3.1 by 10.6
Design Arena (Agents - Fullstack): GPT-6 (1350.0) beat Claude Fable 5.1 by 6.0

The Aggregate Digest — 2026-09-13

2026-09-13T07:38:12.257029+00:00

Frontier

Best available model: Gemini 3 Deep Think enters at #20 (1943 ELO) on 37 benchmarks

Daily

Top-10 New Scores (7)

Claude Fable 5.1 on Last Translation Benchmark: 32.93 (#7)
GPT-5.6 Sol on Lean AI Formalization Leaderboard: 7.0 (#20)
GPT-6 on Design Arena (Agents - Mobile Apps): 1281.0 (#2)
GPT-6 on Featherbench: 96.0 (#8)
GPT-6 on Featherbench - Rubric Quality: 8.6 (#18)
GPT-6 on Last Translation Benchmark: 44.02 (#4)
GPT-6 Pro on Last Translation Benchmark: 45.33 (#2)

New #1 Leaders (1)

Last Translation Benchmark: Human LTB contributors (99.89) beat Gemini 3.1 Pro (Preview) by 60.59

Weekly

New Benchmarks (125)

Incompressible Knowledge Probes (IKP) (Mean tier accuracy (%)): leader Gemini 3 Flash (Preview) (Medium) (86.187), 133 models
Closed-book factual recall across seven obscurity tiers, from common facts to obscure Wikidata entities and researcher knowledge. Reports the unweighted mean of tier accuracies on all 1,311 cleaned v2 questions, with no hallucination penalty. Measures recalled knowledge, not a direct parameter count.
IKP - T1 (Accuracy (%)): leader Qwen 3.7 Max (100.0), 195 models
The easiest IKP factual-recall tier: universal knowledge such as capitals and familiar geographical facts. Accuracy on all 200 cleaned T1 questions, answered without retrieval or supplied evidence.
IKP - T2 (Accuracy (%)): leader Qwen 3.7 Max (100.0), 197 models
IKP's second factual-recall tier, covering common knowledge beyond its easiest questions. Accuracy on all 200 cleaned T2 questions; no retrieval, supplied evidence or hallucination penalty.
IKP - T3 (Accuracy (%)): leader Gemini 3.1 Pro (Preview) (100.0), 197 models
IKP's third obscurity tier tests less common factual knowledge, including researcher recognition. Accuracy on all 183 cleaned T3 questions, answered without retrieval or supplied evidence.
IKP - T4 (Accuracy (%)): leader Gemini 3.1 Pro (Preview) (98.4043), 196 models
IKP's fourth obscurity tier tests specialized factual recall, including obscure entities and researchers. Accuracy on all 188 cleaned T4 questions; no retrieval, supplied evidence or hallucination penalty.
IKP - T5 (Accuracy (%)): leader Gemini 3 Flash (Preview) (Medium) (97.2376), 138 models
IKP's fifth obscurity tier tests rare factual knowledge about Wikidata entities and researchers. Accuracy on all 181 cleaned T5 questions, answered without retrieval or supplied evidence.
IKP - T6 (Accuracy (%)): leader Gemini 3.1 Pro (Preview) (93.5829), 141 models
IKP's sixth obscurity tier tests very rare entity facts and researcher knowledge. Accuracy on all 187 cleaned T6 questions; tier difficulty was calibrated using reference models, and no retrieval or supplied evidence is allowed.
IKP - T7 (Accuracy (%)): leader Gemini 3 Flash (Preview) (Medium) (28.4884), 137 models
IKP's hardest obscurity tier tests extreme long-tail factual recall about entities and researchers. Accuracy on all 172 cleaned T7 questions, answered without retrieval or supplied evidence; missing evaluations are not counted as wrong answers.
Sindhi LLM Benchmark (Accuracy (%)): leader Claude Opus 5 (95.4), 14 models
Sindhi-language question answering on 500 scored items, with the publisher averaging repeated runs. Reports correct answers as a percentage of scored questions; unparsed responses remain in the denominator. This board reports Sindhi LLM Benchmark using Accuracy (%).
TriageBench (Decision Accuracy (%)): leader Claude Opus 4.8 (45.0), 29 models
Clinical chart triage requiring the correct care decision from supplied patient information. Reports decision accuracy and retains reasoning effort and dated model identifiers. The main board contains 100-case evaluations; partial runs appear on boards labelled with their smaller case count. This board reports TriageBench using Decision Accuracy (%).
LongListBench - Agentic CLI - Exact Record F1 (Exact Record F1 (%)): leader GPT-5.6 Sol (98.6614), 6 models
Models extract structured records from the same released OCR transcripts through agentic command-line tools. Exact-record F1 and weighted F1 measure extraction fidelity across the document corpus. The Agentic CLI track keeps a shared input protocol separate from raw-PDF extraction products and page-level pipelines. This board reports LongListBench - Agentic CLI - Exact Record F1 using Exact Record F1 (%).
LongListBench - Agentic CLI - Weighted F1 (Weighted F1 (%)): leader GPT-5.6 Sol (99.7304), 6 models
Models extract structured records from the same released OCR transcripts through agentic command-line tools. Exact-record F1 and weighted F1 measure extraction fidelity across the document corpus. The Agentic CLI track keeps a shared input protocol separate from raw-PDF extraction products and page-level pipelines. This board reports LongListBench - Agentic CLI - Weighted F1 using Weighted F1 (%).
Petri-Bench (Score): leader GLM-5.1 (66.2), 9 models
LLM solvers probe simulated systems, infer hidden parameters and try to achieve target outcomes. Reports overall performance, solved cases, parameter recovery and individual simulation engines. Non-LLM search baselines remain outside the model comparison. This board reports Petri-Bench using Score.
Petri-Bench - Solved (Solved (%)): leader Claude Fable 5 (51.9), 9 models
LLM solvers probe simulated systems, infer hidden parameters and try to achieve target outcomes. Reports overall performance, solved cases, parameter recovery and individual simulation engines. Non-LLM search baselines remain outside the model comparison. This board reports Petri-Bench - Solved using Solved (%).
Petri-Bench - Parameter Recovery (Parameter Recovery (%)): leader Gemini 3.1 Pro (Preview) (75.6), 9 models
LLM solvers probe simulated systems, infer hidden parameters and try to achieve target outcomes. Reports overall performance, solved cases, parameter recovery and individual simulation engines. Non-LLM search baselines remain outside the model comparison. This board reports Petri-Bench - Parameter Recovery using Parameter Recovery (%).
Petri-Bench - Market (Score): leader GLM-5.1 (74.7), 9 models
LLM solvers probe simulated systems, infer hidden parameters and try to achieve target outcomes. Reports overall performance, solved cases, parameter recovery and individual simulation engines. Non-LLM search baselines remain outside the model comparison. This board reports Petri-Bench - Market using Score.
Petri-Bench - Morph (Score): leader GPT-5.6 Sol (75.7), 8 models
LLM solvers probe simulated systems, infer hidden parameters and try to achieve target outcomes. Reports overall performance, solved cases, parameter recovery and individual simulation engines. Non-LLM search baselines remain outside the model comparison. This board reports Petri-Bench - Morph using Score.
Petri-Bench - Origin (Score): leader Claude Sonnet 4.6 (80.2), 8 models
LLM solvers probe simulated systems, infer hidden parameters and try to achieve target outcomes. Reports overall performance, solved cases, parameter recovery and individual simulation engines. Non-LLM search baselines remain outside the model comparison. This board reports Petri-Bench - Origin using Score.
Petri-Bench - Social (Score): leader GPT-5.5 (87.9), 8 models
LLM solvers probe simulated systems, infer hidden parameters and try to achieve target outcomes. Reports overall performance, solved cases, parameter recovery and individual simulation engines. Non-LLM search baselines remain outside the model comparison. This board reports Petri-Bench - Social using Score.
Petri-Bench - Swarm (Score): leader Claude Fable 5 (63.9), 8 models
LLM solvers probe simulated systems, infer hidden parameters and try to achieve target outcomes. Reports overall performance, solved cases, parameter recovery and individual simulation engines. Non-LLM search baselines remain outside the model comparison. This board reports Petri-Bench - Swarm using Score.
Artificial Analysis Coding Agent Index (Index score (points)): leader Claude Opus 5 (68.1), 5 models
Artificial Analysis Coding Agent Index aggregates results for coding agents from multiple agentic coding evaluations reported in provider comparison tables. The pinned metric, Index score (points), measures the combined index score. This is a source-attributed results table mirrored by BenchmarkList. Source: BenchmarkList’s attributed result table.
BioSecBench-Function (Endpoint pass rate (%)): leader Claude Opus 5 / claude-code (50.3), 22 models
BioSecBench-Function evaluates text-based recovery of biosecurity-relevant biological function from real experimental data, assays, structures, and sequences. The pinned metric is Endpoint pass rate (%), measuring deterministic agreement with published experimental ground truth. The benchmark uses restricted-access evaluations and is source-attributed. Source: BenchmarkList’s attributed result table.
BrowseComp (Score (%)): leader GPT-6 (91.5), 4 models
BrowseComp is a text-only benchmark for autonomous agents on multi-step web tasks involving planning, state tracking, tool use, and recovery. The evaluated model is a browsing agent, and the pinned metric, Score (%), measures success on the benchmark tasks. Source: BenchmarkList’s attributed result table.
LMArena Vision Arena (Arena rating): leader Claude Fable 5 (1313.0592), 25 models
Crowdsourced pairwise human-preference leaderboard for vision-language model responses in LMArena.
Convex Coding Evals — No Guidelines (111 tasks) (Pass rate (%)): leader GPT-6 (84.6847), 30 models
Convex Coding Evals — No Guidelines (111 tasks) evaluates models on writing Convex backends and client integrations without Convex-specific guidelines or documentation. The input modality is text and code. Its pinned metric is pass rate (%), averaged over the source leaderboard’s latest complete runs for this benchmark version. Source: BenchmarkList’s attributed result table.
DuelLab GameBench 2 (Leaderboard score): leader Claude Fable 5 (74.6896), 48 models
DuelLab GameBench 2 evaluates AI-generated game-playing programs by compiling and running them head-to-head on public board and strategy games, so the input modality is text/code. The pinned metric is Leaderboard score, measuring the released standings score reported for each system. This is a source-attributed result table mirrored by BenchmarkList; the evaluator, engine implementation, and unpublished operational artifacts remain private. Source: BenchmarkList’s attributed result table.
Founder Bench (Customer Impressions): leader Claude Fable 5 (127902.0), 5 models
Founder Bench evaluates models that autonomously operate businesses in a real-world arena, using text input and tool use as implied by the tasks described. The pinned metric is Customer Impressions, measuring the market-pull outcome reported for the model’s companies. This is a source-attributed result table mirrored by BenchmarkList. Source: BenchmarkList’s attributed result table.
ERQA (Score (%)): leader Qwen3.8 Max 0902 (78.3), 5 models
Embodied Reasoning Question Answering benchmark with visual questions about space, trajectories, actions, state estimation, and multi-view reasoning.
FounderBench (Average Task Score (score)): leader Gemini 3.5 Flash (67.69), 11 models
FounderBench evaluates LLM agents in a deterministic simulator on repeated startup-like decisions using text-based structured actions. It spans 50 public tasks across 10 business task families. The pinned metric is Average Task Score, which measures the final task score mapped from simulator outcomes on a 0 to 100 scale. Source: BenchmarkList’s attributed result table.
FrontierMath 2025-02-28 Private (Score (%)): leader GPT-5.6 Sol (93.9), 6 models
Private FrontierMath research-level mathematics benchmark snapshot.
FrontierMath Tier 4 2025-07-01 Private (Score (%)): leader GPT-5.6 Sol (75.7), 6 models
Private Tier 4 FrontierMath problems at research-level mathematical difficulty.
GameDevBench (Pass@1 (%)): leader Claude Fable 5 (xHigh) [Claude Code] (67.3), 17 models
GameDevBench evaluates LLM agents on game-development tasks in the Godot engine using text plus multimodal assets such as shaders, sprites, animations, and visual game scenes. The tasks cover 2D graphics, 3D graphics, UI, and gameplay logic. The pinned metric is Pass@1, measuring the share of tasks solved on the first attempt. Source: BenchmarkList’s attributed result table.
Global-MMLU-Lite (Score (%)): leader Claude Opus 5 (95.7031), 35 models
Lightweight multilingual MMLU benchmark from Cohere Labs and Kaggle for comparing knowledge performance across languages with reduced evaluation cost.
GPQA Diamond (BenchmarkList) (Accuracy (%)): leader GPT-6 (96.0), 6 models
GPQA Diamond evaluates models on graduate-level science questions in biology, chemistry, and physics using text input. The pinned metric is Accuracy (%), measuring the share of questions answered correctly in the source table. Source: BenchmarkList’s attributed result table.
Gray Swan IPI — Combined Attack Set, Transfer Only, No Computer Use (Attack success rate @15 (%)): leader Claude Opus 5 (4.8), 16 models
Gray Swan IPI evaluates indirect prompt-injection robustness in text-only settings using a combined attack set in transfer-only mode, without computer use. The model is tested on whether it resists attacks, and the pinned metric is attack success rate at 15 attempts; lower values indicate fewer successful attacks. Source: BenchmarkList’s attributed result table.
KernelBench CUDA (Mean Peak Fraction Across Problems (ratio)): leader Claude Opus 5 (Max) (0.7926), 19 models
KernelBench CUDA is a text-only coding benchmark for GPU kernel tasks, including fused MoE, sparse attention, decode, and simulation kernels. It evaluates agents on CUDA-only kernel implementation. The pinned metric, Mean Peak Fraction Across Problems, measures performance as a ratio relative to the best achievable peak across problems. Source: BenchmarkList’s attributed result table.
KernelBench Hard (Mean Peak Fraction Across Problems (ratio)): leader Claude Fable 5 (Max) (0.348), 59 models
KernelBench Hard is a text-only benchmark for autonomous coding agents on GPU kernel engineering tasks. It assesses correctness and speed against hardware baselines. The pinned metric, Mean Peak Fraction Across Problems, measures performance as a ratio relative to the problem peak across the suite. Source: BenchmarkList’s attributed result table.
KernelBench Mega (Speedup (x)): leader Claude Fable 5 (35.435), 38 models
KernelBench Mega is a text-only benchmark for coding agents working on full fused GPU megakernels. The cited public board specifically measures Kimi-Linear W4A16 decode speedup over an optimized PyTorch baseline. The pinned metric, Speedup (x), measures runtime acceleration versus that baseline. Source: BenchmarkList’s attributed result table.
LABBench2 Aggregate (Aggregate score (%)): leader Gemini 3.8 Flash (86.2), 6 models
LABBench2 Aggregate reports results for models solving the eight LAB-Bench 2 biology-research tasks using text-based workflows in a Linux terminal with bioinformatics tools, Python, R, and internet access. The pinned metric is aggregate score (%), measuring the combined performance across those tasks in the source result table. Source: BenchmarkList’s attributed result table.
LMArena Search Arena (Arena rating): leader GPT-5.6 Sol (xHigh) (1257.2604), 25 models
Crowdsourced pairwise human-preference leaderboard for search-augmented AI systems in LMArena.
MVBench (Average accuracy (%)): leader Gemini 3.7 Flash (82.2), 5 models
MVBench: Evaluates temporal, video, speech, or audio understanding beyond static text and image inputs.
NanoGPT Speedrun Frontier (Human record gap closed (%)): leader Claude Fable 5 (claude-code · high) (81.7), 19 models
NanoGPT Speedrun Frontier evaluates autonomous coding agents on the nanoGPT optimizer speedrun task. The model must improve the training recipe on its own under different agent harnesses. The pinned metric is human record gap closed, measuring the share of the distance between the baseline and human record that the run closes. Source: BenchmarkList’s attributed result table.
ObviousBench (Answer pass³ (%)): leader ByteDance Seed: Seed 2.1 Turbo enabled (100.0), 506 models
ObviousBench evaluates language models on text-based tasks designed to catch simple, user-visible mistakes such as arithmetic, spelling, formatting, ordering, and constraint-following errors. The pinned metric is Answer pass³ (%), which measures whether all three sampled attempts are answer-correct. The cited results are from a public v0.2 release with a held-out private set not released. Source: BenchmarkList’s attributed result table.
SimpleQA Net Score (Net score (points)): leader Claude Mythos Preview (0.51), 5 models
SimpleQA closed-book factuality results reported by Anthropic as net score: correct responses minus incorrect responses, with abstentions scoring zero.
E-Bench-Code (Tencent Internal) (Reported Score): leader GPT-5.6 Sol (83.3), 8 models
E-Bench-Code (Tencent Internal) is a proprietary Tencent internal working-agent evaluation listed in the HY4 preview benchmark appendix. The evidence does not specify the input modality. The pinned metric, Reported Score, is the score as reported in that source. Source: BenchmarkList’s attributed result table.
E-Bench (Tencent Internal) (Reported Score): leader GPT-5.6 Sol (80.6), 8 models
E-Bench (Tencent Internal) is a proprietary Tencent internal working-agent evaluation listed in the HY4 preview benchmark appendix. The evidence does not specify the input modality. The pinned metric, Reported Score, is the score as reported in that source. Source: BenchmarkList’s attributed result table.
Hy-Backend 2.0 (Internal) (Reported Score): leader GPT-5.6 Sol (49.6), 8 models
Tencent’s internal Hy-Backend 2.0 coding-agent evaluation, reported in the HY4 preview benchmark appendix. The evaluated model is assessed on text-based coding tasks. The pinned metric is the reported score. Source: BenchmarkList’s attributed result table.
Hy-BrowseComp-Pro2 (Internal) (Reported Score): leader Claude Opus 5 (highest_available) (61.3), 8 models
Tencent’s internal Hy-BrowseComp-Pro2 agentic-search evaluation, listed in the HY4 preview benchmark appendix and noted as distinct from public BrowseComp benchmarks. The evaluated model uses text input. The pinned metric is the reported score. Source: BenchmarkList’s attributed result table.
Hy-CompanyBench V2 (Internal) (Reported Score): leader Claude Opus 5 (highest_available) (72.7), 8 models
Tencent’s internal Hy-CompanyBench V2 evaluation from the agentic-coding section of the HY4 preview benchmark appendix. The evaluated model is tested on text-based coding work. The pinned metric is the reported score. Source: BenchmarkList’s attributed result table.
Hy-FinAgentBench (Internal) (Reported Score): leader GPT-5.6 Sol (83.0), 8 models
Tencent’s internal Hy-FinAgentBench finance-agent evaluation, reported in the HY4 preview benchmark appendix. The evaluated model operates on text inputs. The pinned metric is the reported score. Source: BenchmarkList’s attributed result table.
Hy-FinmodelBench v2 (Internal) (Reported Score): leader Claude Opus 5 (highest_available) (66.0), 8 models
Tencent’s internal Hy-FinmodelBench v2 financial-modeling evaluation, reported in the HY4 preview benchmark appendix. The evaluated model is assessed with text-based inputs. The pinned metric is the reported score. Source: BenchmarkList’s attributed result table.
Hy-LifeSearch (Internal) (Reported Score): leader GPT-5.6 Sol (63.4), 8 models
Tencent’s internal Hy-LifeSearch agentic-search evaluation, reported in the HY4 preview benchmark appendix. The evaluated model uses text input. The pinned metric is the reported score. Source: BenchmarkList’s attributed result table.
Hy-SWE Max Verified (Internal) (Pass Rate (%)): leader Claude Opus 5 (highest_available) (70.1), 8 models
Tencent’s internal Hy-SWE Max Verified software-engineering evaluation uses 300 tasks, hidden pass/fail verifier tests, and averages results over three runs. The evaluated model works on text-based coding tasks. The pinned metric is pass rate. Source: BenchmarkList’s attributed result table.
Terminal-Bench 3.0 (Pass@1 (%)): leader Claude Opus 5 (42.7), 5 models
Terminal-Bench 3.0 measures whether agents can complete difficult, economically valuable tasks in command-line environments. The evaluated model works from text inputs in terminal settings. The pinned metric is Pass@1. Source: BenchmarkList’s attributed result table.
Terminal-Bench-Science 0.1 (Resolution rate (%)): leader GPT-6 (64.6), 5 models
Terminal-Bench-Science 0.1 evaluates AI agents on 70 outcome-verifiable scientific workflows in containerized terminal environments, covering computational biology, chemistry, physics, earth science, and related technical research work. The evaluated model uses text input. The pinned metric is resolution rate. Source: BenchmarkList’s attributed result table.
USAMO 2026 (Score (%)): leader Claude Mythos 5 (99.8), 4 models
2026 USA Mathematical Olympiad proof-based evaluation reported in Anthropic's Claude Opus 4.8 system card using MathArena-style model-judge grading.
Code Migration (Score (%)): leader Claude Opus 5 (57.473), 33 models
Code Migration is a text benchmark for language models. The evaluated model reimplements real-world programs in another language, and the pinned metric is Score (%), measuring performance on the reimplementation task. Source: BenchmarkList’s attributed result table.
CorpFin v2 (Score (%)): leader Claude Opus 5 (73.194), 126 models
A private benchmark evaluating understanding of long-context credit agreements.
Vals CyberBench (Score (%)): leader GPT-5.6 Sol (88.136), 20 models
Vals CyberBench is a text benchmark for autonomous agents. The evaluated model crafts proof-of-concept inputs intended to trigger OSS-Fuzz vulnerabilities and then stop crashing after the fix, and the pinned metric is Score (%), measuring success on that cyber task. Source: BenchmarkList’s attributed result table.
Excel Modeling Benchmark (Score (%)): leader Claude Fable 5 (73.666), 28 models
Excel Modeling Benchmark is a text-and-spreadsheet benchmark for agents on Excel-based financial modeling tasks. The evaluated model works through Excel modeling problems, and the pinned metric is Score (%), measuring performance on those modeling tasks. Source: BenchmarkList’s attributed result table.
GPQA Diamond (Score (%)): leader Gemini 3.1 Pro (Preview) (95.454), 126 models
Graduate-level Google-Proof Q&A benchmark evaluating models on questions that require deep reasoning.
LegalBench (Score (%)): leader Claude Fable 5 (88.561), 129 models
Evaluating language models on a wide range of open source legal reasoning tasks.
Legal Research Bench (Score (%)): leader Claude Opus 5 (55.288), 27 models
Legal Research Bench is a text benchmark for legal research tasks across diverse areas of US law. The evaluated model performs legal research, and the pinned metric is Score (%), measuring performance on those research tasks. Source: BenchmarkList’s attributed result table.
MMLU Pro (Score (%)): leader Claude Opus 5 (91.591), 125 models
Academic multiple-choice benchmark covering 14 subjects including STEM, humanities, and social sciences.
MMMU Pro (Score (%)): leader Claude Opus 5 (89.884), 86 models
1,730 college-level image questions across 30 subjects in 6 disciplines (MMMU team, 2024), filtered to need the image, with 10 options and a vision-only setting that embeds the question in the image.
MortgageTax (Score (%)): leader Claude Opus 5 (72.059), 90 models
Evaluating reading and understanding tax certificates as images.
Public Benefits Bench (Score (%)): leader Claude Opus 5 (76.928), 23 models
459 multi-turn SNAP (food assistance) guidance scenarios covering all US states, scored on 2,931 expert rubric criteria; Center for Civic Futures with Code for America, via Vals AI, 2026.
Public Benefits Bench v1 (Score (%)): leader Claude Fable 5 (71.651), 13 models
Public Benefits Bench v1 reports source-table results for models on helping users navigate SNAP benefits. The evaluated model is measured on text-based public-benefits tasks, and the pinned metric is score (%), reflecting benchmark performance in the mirrored result table. Source: BenchmarkList’s attributed result table.
TaxEval v2 (Score (%)): leader Muse Spark 1.1 (79.722), 132 models
A Vals-created set of questions and responses to tax questions.
Vibe Code Bench v1.1 (Score (%)): leader Claude Fable 5 (90.352), 76 models
Vals AI benchmark for vibe-coding agents that build complete applications from product-style prompts and are scored on functional correctness and quality.
Web Search Index (Score (%)): leader Claude Fable 5 Exa (48.454), 8 models
Web Search Index reports source-table results comparing native provider search against independent web-search tools on legal-research and finance-analysis tasks. The evaluated model uses text queries, and the pinned metric is score (%), measuring performance in the mirrored benchmark table. Source: BenchmarkList’s attributed result table.
VariantBench (Endpoint pass rate (%)): leader Claude Opus 5 / Claude Code (49.72), 31 models
VariantBench evaluates AI agents on text-based genomics tasks involving genetic variant discovery, interpretation, and related QC, using realistic files and deterministic graders. The pinned metric is Endpoint pass rate (%), measuring the share of benchmark tasks successfully completed in the source-reported results table. Source: BenchmarkList’s attributed result table.
WorldVQA (Score (%)): leader Claude Fable 5 (max, with fallback) (56.7), 5 models
Visual question answering benchmark reported in Qwen's Qwen3.7-Plus multimodal search and knowledge QA table.
CursorBench 4.0 (Score (%)): leader Claude Fable 5.1 (Max) (51.8), 39 models
Cursor coding-agent benchmark for ambiguous, multi-file real-world tasks; the 4.0 task set adds long-horizon editing, refactoring, investigation, intent understanding, job management, and design adherence.
Benchmarks.bio - SpatialBench (Pass Rate (%)): leader Grok 4.6 (77.39), 23 models
LatchBio agentic benchmark on messy real-world spatial transcriptomics data, with models writing and running analysis workflows across assays, platforms, and task categories.
Benchmarks.bio - scBench (Pass Rate (%)): leader GPT-6 (64.79), 24 models
LatchBio agentic benchmark for single-cell RNA-seq analysis, requiring models to perform realistic data cleaning, clustering, cell typing, and differential-expression workflows.
Benchmarks.bio - TxBench-OD (Pass Rate (%)): leader GPT-6 (55.46), 13 models
Pass rate on TxBench oligonucleotide-discovery tasks (`tx_oligo_v1`, 113 evals) from Benchmarks.bio's agentic biology suite, run on the `pi` harness.
HLE-Verified (BenchLM) (Accuracy (%)): leader Gemini 3.8 Flash (54.9), 6 models
Expert questions from the verified and revised 1,811-item Humanity’s Last Exam set; full-set accuracy reported in BenchLM’s attributed comparison table. These are published results, not BenchLM ability estimates.
VulcanBench v3 (BenchLM) (Pass@1 (%)): leader Grok 4.5 (89.9), 14 models
AI coding agents complete the v3 coding-task suite. BenchLM republishes a source-attributed pass-at-one table; the benchmark version remains distinct from earlier VulcanBench task sets. These are published results, not BenchLM ability estimates.
TMMLU+ (Accuracy (%)): leader Claude Opus 5 (96.16), 21 models
TMMLU+ is a Traditional Chinese multi-subject exam benchmark (ikala): 66 subjects drawn from Taiwanese professional, high-school and university examinations, from accounting and pharmacy to Taiwanese Hokkien, answered as multiple choice. Only models that have completed all 66 subjects appear on the card's board.
TMMLU+ - STEM (Accuracy (%)): leader Claude Opus 5 (98.71), 21 models
The \
TMMLU+ - Social Science (Accuracy (%)): leader Claude Opus 5 (96.09), 21 models
The \
TMMLU+ - Humanities (Accuracy (%)): leader Claude Opus 5 (94.6), 21 models
The \
TMMLU+ - Other (Accuracy (%)): leader Claude Opus 5 (95.25), 21 models
The \
MedXpertQA MM (Natomy) (Accuracy (%)): leader GPT-6 (87.4), 9 models
Accuracy on the MedXpertQA multimodal subset - 2,000 expert-level medical exam questions with real clinical images and five answer choices - run zero-shot with chain-of-thought reasoning.
MedXpertQA Text (Natomy) (Accuracy (%)): leader Gemini 3.8 Flash (66.2), 6 models
Accuracy on the MedXpertQA text-only set - 2,450 expert-level medical exam questions with ten answer choices - run zero-shot with chain-of-thought reasoning.
MedXpertQA MM (Natomy) - Diagnosis (Accuracy (%)): leader GPT-6 (86.99), 6 models
MedXpertQA multimodal accuracy restricted to diagnosis questions.
MedXpertQA MM (Natomy) - Treatment (Accuracy (%)): leader GPT-6 (87.05), 6 models
MedXpertQA multimodal accuracy restricted to treatment questions.
MedXpertQA MM (Natomy) - Basic Science (Accuracy (%)): leader GPT-6 (89.24), 6 models
MedXpertQA multimodal accuracy restricted to basic-science questions.
Icelandic LLM - WikiQA (LLM Judge Score (%)): leader Claude Fable 5.1 (73.95), 23 models
Open-ended factual questions on Icelandic topics sourced from Wikipedia. The model answers in Icelandic and an LLM judge grades each answer against a reference as poor, fair or excellent (0, 1/2 or 1).
Icelandic LLM - Song Lyrics (Accuracy (%)): leader Claude Opus 5 (22.74), 23 models
The model is given the opening of a well-known Icelandic song and must continue the lyric, scored by overlap with the reference continuation.
Icelandic LLM - Famous Phrases (Accuracy (%)): leader Gemini 3.8 Flash (30.88), 23 models
The model is given a well-known Icelandic phrase - a film or television line, an advertising slogan, a line from the sagas - and must name what it comes from, scored by exact match against accepted answers.
Icelandic LLM - Textile Terms (LLM Judge Score (%)): leader GPT-6 (74.0), 23 models
The model is asked what an Icelandic textile-craft term means, or what a knitting or crochet abbreviation stands for, graded by an LLM judge against a reference definition.
Icelandic LLM - Flora & Fauna (Accuracy (%)): leader GPT-6 (56.54), 23 models
The model is given the English and Latin names of species native to Iceland and must translate them into Icelandic, scored by exact match against accepted answers.
Icelandic LLM - Generative Quality (Quality Score (%)): leader GPT-6 (82.22), 23 models
The model writes a few paragraphs from a varied list of prompts and the text is scored for defects: grammar errors and out-of-vocabulary words. It penalises flawed text rather than ranking good against great.
Icelandic LLM - GEC (Accuracy (%)): leader Claude Opus 5 (83.03), 23 models
One Icelandic sentence to be returned corrected - or unchanged, since half the sentences are already correct - scored by exact match.
Icelandic LLM - Case Government (Accuracy (%)): leader Claude Opus 5 (99.06), 23 models
Which case does the verb demand? The model puts a nominative noun phrase into a sentence's subject gap in the case the verb requires (many Icelandic verbs take an oblique subject), scored by exact match.
Icelandic LLM - Idiom Completion (Accuracy (%)): leader GPT-6 (72.51), 23 models
Well-known Icelandic idioms with one word blanked out. The model supplies the missing word and is scored by exact match.
Icelandic LLM - Idiom Meanings (LLM Judge Score (%)): leader Claude Opus 5 (84.55), 23 models
The model explains in one sentence what an Icelandic idiom means, rated by an LLM judge against a reference definition as 0, 1/2 or 1.
LLM Stats (SEC-bench Pro) (Score (%)): leader GPT-6 (85.4), 5 models
SEC-bench Pro, as tracked by LLM Stats: long-horizon software-security tasks that ask a model to reproduce, diagnose and patch real vulnerabilities across a repository rather than answer a security question.
ChessBench GitHub - Elo (Benchmark Elo): leader GPT-5.6 Pro Sol (Max) (2660.0), 32 models
Field-relative Elo from complete chess games on chessbench-ai.github.io, distinct from chessbench.ai. Published effort labels are retained. Measures playing strength within this field, not human chess Elo; ACPL, costs and historical checkpoint scores are not separate entries.
BBQ Disambiguated Accuracy (Mythos 5.1 System Card) (Score (%)): leader Claude Mythos 5.1 (89.9), 4 models
Accuracy on disambiguated social-context questions in BBQ, without a system prompt. September 2026 evaluation infrastructure; earlier models were re-evaluated. Measures answering the supplied evidence correctly, not the signed bias statistic.
BioMysteryBench Human Solvable (Mythos 5.1 System Card) (Score (%)): leader Claude Opus 5 (91.4), 6 models
Biological reasoning and computational analysis on raw datasets, on the subset solved by independent human experts. September 2026 configuration restricts package installation and network domains.
BioMysteryBench Human Difficult (Mythos 5.1 System Card) (Score (%)): leader Claude Opus 5 (51.8), 6 models
Biological reasoning and computational analysis on raw datasets, on problems unsolved by human experts but with objective ground truth. September 2026 configuration restricts package installation and network domains.
LatchBio SpatialBench Verified (Mythos 5.1 System Card) (Score (%)): leader Claude Mythos 5.1 (77.6), 4 models
Analysis of spatial transcriptomics across 115 externally validated problems, linking gene expression and tissue location to biological questions. September 2026 LatchBio evaluation.
LatchBio SingleCellBench (Mythos 5.1 System Card) (Score (%)): leader Claude Mythos 5.1 (61.9), 4 models
Analysis of single-cell RNA sequencing across 195 problems, including cell labeling, differential expression and batch correction. September 2026 LatchBio evaluation.
Protein Design Sequence Generation (Mythos 5.1 System Card) (Score (%)): leader Claude Mythos 5.1 (46.0), 4 models
Generate protein sequences satisfying design constraints; scoring combines constraint satisfaction, folding confidence and sequence novelty. No tools. Updated September 2026 benchmark and grader.
Protein Design Library Ranking (Mythos 5.1 System Card) (Score (%)): leader Claude Mythos 5.1 (49.3), 4 models
Rank held-out protein sequences for experimental testing using wet-lab measurements and a selection objective from the same engineering campaign. No tools. September 2026 evaluation.
Organic Chemistry V2 (Mythos 5.1 System Card) (Score (%)): leader Claude Mythos 5.1 (69.2), 6 models
Organic chemistry tasks involving spectroscopy, molecular structures, multi-step synthesis and reaction prediction. V2 adds expert-recommended harder problems; September 2026 scores reflect expert feedback.
Protocols Troubleshooting (Mythos 5.1 System Card) (Score (%)): leader Claude Mythos 5.1 (70.2), 6 models
Detect and fix errors in molecular biology protocols, with bash, file editing and web search. September 2026 evaluation.
Protocols Understanding (Mythos 5.1 System Card) (Score (%)): leader Claude Opus 5 (80.0), 6 models
Benchling protocol understanding: extend online molecular biology protocols to additional directions. September 2026 evaluation removes external network access relative to the Opus 5 system card.
Baba Is Harbor - Stage 0 (pass@1 (%)): leader GPT-6 (100.0), 29 models
Tests agents playing Baba Is You across eight introductory levels using the same harness for every model. Reports the percentage of levels solved in stage 0; later stages using different model-specific harnesses are excluded from this series.
ChessBench - Elo (Elo): leader Gemini 3.8 Flash (1428.0), 61 models
Monte Carlo Elo rating from ChessBench games, anchored to fixed reference players spanning the strength range.
ChessBench - Accuracy (Accuracy (%)): leader Gemini 3.8 Flash (78.1), 61 models
ChessBench move-quality score: how close a model\u2019s chosen moves are to best play, averaged across branching, converting and resisting positions.
ChessBench - Coherence (Coherence (%)): leader GPT-5.4 (91.2), 61 models
ChessBench rule-compliance score: the fraction of attempted moves that were legal, weighted by the fraction of games played fully legally.
ChessBench (Overall Score (%)): leader Gemini 3.8 Flash (75.9), 61 models
Chess benchmark where language models play rated games against fixed reference opponents, scored on rule-compliant play, move quality, and Monte Carlo Elo.
LOL Bench - Joke Explanation (Mean Grade (%)): leader Claude Opus 5 (92.17), 13 models
Blind-graded benchmark of how well language models explain jokes, scored as a mean grade over a fixed item pool with three samples per item.
PulseBench-Tab (T-LAG Score (%)): leader Pulse Ultra 2 (93.47), 18 models
Multilingual document table-extraction benchmark with provider leaderboard and T-LAG scoring.
Architectural RedlineBench (Mean Net Issue Score (%)): leader GPT-6 (65.5), 18 models
Models review architectural PDF drawings to identify 55 scored issues. The net score awards full or partial credit for genuine findings and subtracts points for confidently incorrect findings. Reports the publisher’s mean across each model’s runs as a percentage of the available issue points. This board reports Architectural RedlineBench using Mean Net Issue Score (%).
Bug Hunt Bench (Planted Bugs Fixed (out of 105)): leader GPT-6 (Max) (48.0), 60 models
Coding models find and repair 105 planted defects in two production repositories: 45 in a VS Code extension and 60 in a learning-management system. Blind grading counts verified planted fixes. Reasoning effort, the July DeepSeek revision and the retired Gemini CLI protocol remain distinguishable; superseded runs follow the publisher’s replacement markers. This board reports Bug Hunt Bench using Planted Bugs Fixed (out of 105).
Bug Hunt Bench - VS Code Extension (Planted Bugs Fixed (out of 45)): leader GPT-6 (Max) (24.0), 60 models
Coding models find and repair 105 planted defects in two production repositories: 45 in a VS Code extension and 60 in a learning-management system. Blind grading counts verified planted fixes. Reasoning effort, the July DeepSeek revision and the retired Gemini CLI protocol remain distinguishable; superseded runs follow the publisher’s replacement markers. This board reports Bug Hunt Bench - VS Code Extension using Planted Bugs Fixed (out of 45).
Bug Hunt Bench - LMS (Planted Bugs Fixed (out of 60)): leader GPT-6 (Max) (24.0), 60 models
Coding models find and repair 105 planted defects in two production repositories: 45 in a VS Code extension and 60 in a learning-management system. Blind grading counts verified planted fixes. Reasoning effort, the July DeepSeek revision and the retired Gemini CLI protocol remain distinguishable; superseded runs follow the publisher’s replacement markers. This board reports Bug Hunt Bench - LMS using Planted Bugs Fixed (out of 60).
PerspectiveGap - Combined (Pass Rate (%)): leader GPT-5.5 (62.0), 35 models
Measures whether a model can compose role-specific prompts for a multi-agent orchestration without over-sharing context. Each of 110 scenarios supplies a role list, a shuffled set of information fragments and one injected distractor, against a reference assignment giving each role exactly the fragments it needs. A deterministic rule-only scorer, not an LLM judge, audits containment using phrase-level fragment fingerprints. Strict pass is all-or-nothing: an evaluation scores 1 only when it omits no required fragment and includes nothing out of role. Combined is the unweighted average of the strict pass rates of the two task formats. This series reports PerspectiveGap - Combined using Pass Rate (%).
PerspectiveGap - Role Assignment (Pass Rate (%)): leader GPT-5.5 (55.5), 35 models
Measures whether a model can compose role-specific prompts for a multi-agent orchestration without over-sharing context. Each of 110 scenarios supplies a role list, a shuffled set of information fragments and one injected distractor, against a reference assignment giving each role exactly the fragments it needs. Role-fragment assignment asks the model for a structured mapping from roles to fragment identifiers. A deterministic rule-only scorer, not an LLM judge, audits containment; strict pass is all-or-nothing, scoring 1 only when an evaluation omits no required fragment and includes nothing out of role. This series reports PerspectiveGap - Role Assignment using Pass Rate (%).
PerspectiveGap - Prompt Writing (Pass Rate (%)): leader GPT-5.5 (68.6), 35 models
Measures whether a model can compose role-specific prompts for a multi-agent orchestration without over-sharing context. Each of 110 scenarios supplies a role list, a shuffled set of information fragments and one injected distractor, against a reference assignment giving each role exactly the fragments it needs. Free-form prompt writing asks for the final sub-agent prompts, exposing models that can identify the boundary as a set of identifiers yet fail to preserve it when writing natural-language instructions. A deterministic rule-only scorer audits containment with phrase-level fingerprints; strict pass is all-or-nothing, scoring 1 only when an evaluation omits no required fragment and includes nothing out of role. This series reports PerspectiveGap - Prompt Writing using Pass Rate (%).

New Models (319)

GPT-6 — ELO 2110, #1
- Computer Anthology Terminal Tasks (Codex CLI): 65.6 (#1/5)
- Computer Anthology Terminal Tasks (Terminus-2): 67.0 (#1/28)
- SpacetimeDB LLM Benchmark (Rust): 98.4 (#1/16)
- Agent Arena - Praise vs Complaint: 40.79 (#1/43)
- Agent Arena - Tool Hallucination: -0.38 (#1/43)
- Design Arena (Game Dev): 1456.0 (#1/164)
- Design Arena (UI Components): 1400.0 (#1/159)
- Design Arena (3D): 1481.0 (#1/152)
- Design Arena (SVG): 1494.0 (#1/117)
- Design Arena (Agents - Web Apps): 1334.0 (#1/43)
- ...and 85 more
Claude Fable 5.1 — ELO 2104, #2
- InferenceBench: 9.83 (#1/35)
- SpacetimeDB LLM Benchmark (TypeScript): 100.0 (#1/16)
- React Native Evals: 89.05 (#1/27)
- ErdosBench: 3.25 (#1/13)
- Design Arena (Agents - Fullstack): 1344.0 (#1/45)
- Epoch AI - Mirrorcode: 73.3 (#1/8)
- Chatbot Arena (Text - Chinese): 1595.0 (#1/372)
- Chatbot Arena (Text - French): 1532.0 (#1/281)
- WebDev Arena (Fullstack): 1691.0 (#1/63)
- HiL-Bench: 61.5 (#1/17)
- ...and 40 more
Claude Opus 5 — ELO 2074, #3
- DRACO: 88.6 (#1/14)
- OSWorld 2.0: 34.72 (#1/9)
- OSWorld 2.0 Partial: 70.19 (#1/9)
- Vals AI Legal Research Bench: 55.29 (#2/59)
- Conceptual Reasoning Index - Consistency (ACCoRD): 83.84 (#2/184)
- Conceptual Reasoning Index - Decision Theory (DTBench): 96.45 (#2/241)
- MLS-Bench Lite: 49.8 (#3/14)
- Icelandic LLM Leaderboard - Average: 68.63 (#3/23)
- Conceptual Reasoning Index: 74.65 (#3/184)
- PostTrainBench: 35.04 (#4/13)
- ...and 14 more
Claude Fable 5 — ELO 2052, #4
- SealedBench: 71.0 (#3/36)
- LOL Arena: 49.4 (#4/42)
- Conceptual Reasoning Index - Consistency (ACCoRD): 82.37 (#5/184)
- Conceptual Reasoning Index - Decision Theory (DTBench): 95.55 (#8/241)
- Conceptual Reasoning Index: 72.1 (#15/184)
- Conceptual Reasoning Index - Argument Evaluation (LMCA): 61.15 (#16/185)
- DuelLab Overall: 44.8 (#42/171)
GPT-5.6 Sol — ELO 2030, #5
- OSWorld 2.0: 28.1 (#2/9)
- OSWorld 2.0 Partial: 64.13 (#2/9)
- PostTrainBench: 36.23 (#3/13)
- Design Arena (SVG): 1331.0 (#7/117)
- LOL Arena: 48.8 (#10/42)
- Lean AI Formalization Leaderboard: 7.0 (#20/65)
- Conceptual Reasoning Index - Argument Evaluation (LMCA): 58.49 (#22/185)
- Conceptual Reasoning Index: 68.65 (#23/184)
- Conceptual Reasoning Index - Decision Theory (DTBench): 92.88 (#24/241)
- AA-Briefcase: 41.31 (#38/162)
- ...and 1 more
Gemini 3.8 Flash — ELO 2023, #7
- LLM Chess (Saplin): 1631.6 (#1/164)
- Kaggle FACTS Multimodal: 50.7 (#1/34)
- Kaggle FACTS (Google): 69.37 (#2/34)
- Design Arena (Agents - Mobile Apps): 1277.0 (#3/44)
- Agents on Rails: 28.3 (#3/10)
- NYT Connections Extended: 96.5 (#3/117)
- Chess Puzzles (Epoch AI): 61.0 (#4/222)
- SEAL - Humanity's Last Exam: 44.52 (#4/53)
- SEAL - SWE Atlas - Test Writing: 53.7 (#4/24)
- Blueprint-Bench 2: 0.386 (#4/26)
- ...and 50 more
Gemini 3.7 Flash — ELO 1998, #8
- Mercor APEX: 67.8 (#2/24)
- LOL Arena: 49.5 (#3/42)
- MathArena - ARXIVLEAN June: 21.74 (#7/12)
- Vals AI IOI: 67.83 (#8/26)
- MathArena - ARXIV April: 57.5 (#9/21)
- MathArena - ARXIV_FALSE April: 19.26 (#13/19)
- MathArena - ARXIV June: 62.5 (#13/21)
- NYT Connections Extended: 92.2 (#13/117)
- MathArena Arxiv: 47.5 (#14/18)
- MathArena Arxiv False: 8.5 (#15/16)
- ...and 6 more
Grok 4.6 — ELO 1994, #9
- LOL Arena: 49.0 (#7/42)
- AA-Briefcase: 52.53 (#14/162)
- Vals AI IOI: 47.61 (#16/26)
- DuelLab Overall: 56.2 (#18/171)
- Chatbot Arena (Text - Korean): 1439.0 (#26/269)
- Chatbot Arena (Text - French): 1490.0 (#33/281)
- Chatbot Arena (Text - German): 1464.0 (#46/299)
- Chatbot Arena (Text - Japanese): 1413.0 (#67/265)
- Chatbot Arena (Text - Spanish): 1439.0 (#73/283)
Muse Spark 1.3 — ELO 1984, #10
- Design Arena (Website): 1364.0 (#1/176)
- Vals AI Legal Research Bench: 55.29 (#1/59)
- Design Arena (UI Components): 1399.0 (#2/159)
- Vals AI Harvey Legal Agent Bench: 23.75 (#2/60)
- Design Arena (3D): 1426.0 (#3/152)
- Vals AI Finance Agent v2: 59.96 (#3/59)
- Design Arena (Agents - Web Apps): 1311.0 (#4/43)
- ErdosBench: 2.86 (#5/13)
- Design Arena (Agents - Fullstack): 1311.0 (#5/45)
- WebDev Arena (Fullstack): 1644.0 (#5/63)
- ...and 40 more
GPT-5.5 — ELO 1980, #11
- PostTrainBench: 27.23 (#10/13)
- LOL Arena: 47.4 (#20/42)
- AA SciCode: 56.13 (#27/215)
- AA-Briefcase: 33.44 (#59/162)
Kimi K3 — ELO 1978, #12
- PostTrainBench: 31.96 (#7/13)
- Vals AI IOI: 48.94 (#15/26)
- Icelandic LLM Leaderboard - Average: 40.71 (#17/23)
- LOL Arena: 47.2 (#21/42)
- AA-Briefcase: 26.06 (#75/162)
Claude Opus 4.8 — ELO 1968, #14
- PostTrainBench: 33.84 (#5/13)
- AA Terminal-Bench Hard: 58.33 (#12/565)
- LOL Arena: 48.4 (#14/42)
- AA Humanity's Last Exam: 48.66 (#24/798)
- AA Omniscience: 28.75 (#26/670)
- Artificial Analysis Intelligence Index: 41.99 (#33/830)
- AA CritPt: 20.86 (#41/674)
- AA SciCode: 54.4 (#43/215)
- AA-Omniscience Accuracy: 48.83 (#46/521)
- AA TAU-2 Bench: 94.44 (#49/575)
- ...and 4 more
GLM-5.3 — ELO 1966, #15
- Chatbot Arena (Text - German): 1549.0 (#1/299)
- SuperCLUE-Terminal - Overall: 56.57 (#2/14)
- Chatbot Arena (Text - Korean): 1490.0 (#3/269)
- Vals AI IOI: 68.44 (#7/26)
- Mercor APEX: 56.6 (#10/24)
- DuelLab Overall: 59.4 (#13/171)
- SealedBench: 63.0 (#14/36)
- Conceptual Reasoning Index - Argument Evaluation (LMCA): 55.51 (#29/185)
- LOL Arena: 45.8 (#30/42)
- Chatbot Arena (Text - Japanese): 1442.0 (#38/265)
- ...and 5 more
GPT-5.6 Terra — ELO 1960, #16
- Vals AI IOI: 87.61 (#4/26)
- Mercor APEX: 58.2 (#8/24)
- Last Translation Benchmark: 12.18 (#13/42)
- Conceptual Reasoning Index - Argument Evaluation (LMCA): 52.23 (#39/185)
- Conceptual Reasoning Index: 62.93 (#45/184)
- Conceptual Reasoning Index - Decision Theory (DTBench): 87.55 (#51/241)
- AA-Briefcase: 35.35 (#54/162)
- Conceptual Reasoning Index - Consistency (ACCoRD): 71.56 (#56/184)
- AA SciCode: 50.46 (#83/215)
Qwen 3.8 Max — ELO 1959, #17
- MathArena Arxiv: 69.17 (#5/18)
- MathArena - ARXIV_FALSE April: 45.49 (#6/19)
- MathArena Arxiv False: 32.5 (#6/16)
- MathArena - ARXIV June: 75.69 (#7/21)
- MathArena - ARXIV April: 59.17 (#8/21)
- MathArena - ARXIV_FALSE June: 43.98 (#9/21)
- Last Translation Benchmark: 12.51 (#12/42)
- AA-Briefcase: 49.6 (#22/162)
- LOL Arena: 46.8 (#24/42)
- DuelLab Overall: 40.0 (#63/171)
DeepSeek V4.1 Flash — ELO 1955, #18
- LiveBench JavaScript: 81.82 (#1/57)
- LiveBench Python: 90.0 (#1/57)
- Humanity's Last Exam (Self-Reported, With Tools): 63.9 (#1/26)
- LLM Stats (AutomationBench): 54.8 (#1/19)
- LLM Stats (BabyVision): 89.6 (#1/13)
- LLM Stats (CyberGym): 88.1 (#1/16)
- LLM Stats (NL2Repo): 64.0 (#1/23)
- LLM Stats (Terminal-Bench 2.1): 90.6 (#1/39)
- LLM Stats (ZEROBench): 49.0 (#1/13)
- Featherbench: 100.0 (#1/22)
- ...and 126 more
Gemini 3.6 Flash — ELO 1947, #19
- AA MMMU-Pro: 83.24 (#21/331)
- Mercor APEX: 46.9 (#21/24)
- Vals AI IOI: 35.06 (#22/26)
- AA GPQA Diamond: 92.83 (#34/797)
- AA-Omniscience Accuracy: 49.97 (#42/521)
- AA Omniscience: 22.13 (#43/670)
- AA SciCode: 53.36 (#57/215)
- AA Humanity's Last Exam: 40.82 (#63/798)
- AA Long Context Reasoning: 80.0 (#69/664)
- Artificial Analysis Intelligence Index: 34.34 (#73/830)
- ...and 2 more
Claude Opus 4.7 — ELO 1940, #21
- PostTrainBench: 28.56 (#9/13)
- AA Terminal-Bench Hard: 51.52 (#25/565)
- AA Omniscience: 27.27 (#33/670)
- Artificial Analysis Intelligence Index: 40.69 (#37/830)
- AA-Briefcase: 39.34 (#44/162)
- AA-Omniscience Accuracy: 48.88 (#45/521)
- AA Humanity's Last Exam: 42.31 (#50/798)
- AA MMMU-Pro: 78.84 (#56/331)
- AA GPQA Diamond: 91.41 (#58/797)
- Tau3 Banking: 34.64 (#61/260)
- ...and 5 more
Muse Spark 1.1 — ELO 1937, #22
- AA-Briefcase: 34.55 (#56/162)
Hy4 preview — ELO 1935, #23
- WebDev Arena (Fullstack): 1602.0 (#20/63)
- DuelLab Overall: 54.8 (#21/171)
- NYT Connections Extended: 64.2 (#57/117)
Muse Spark 1.2 — ELO 1933, #24
- LOL Arena: 48.6 (#12/42)
- Mercor APEX: 36.4 (#23/24)
- Vals AI IOI: 21.78 (#24/26)
- AA-Briefcase: 45.15 (#30/162)
- NYT Connections Extended: 77.2 (#38/117)
Grok 4.5 — ELO 1932, #25
- PostTrainBench: 23.45 (#11/13)
- Vals AI IOI: 40.56 (#19/26)
Gemini 3.1 Pro (Preview) — ELO 1926, #26
- Icelandic LLM Leaderboard - Average: 66.54 (#4/23)
- LOL Arena: 48.8 (#9/42)
- PostTrainBench: 21.99 (#12/13)
- Conceptual Reasoning Index - Decision Theory (DTBench): 95.12 (#13/241)
- Conceptual Reasoning Index: 65.41 (#35/184)
- Conceptual Reasoning Index - Argument Evaluation (LMCA): 52.8 (#36/185)
- Conceptual Reasoning Index - Consistency (ACCoRD): 73.53 (#47/184)
Gemini 3.5 Flash — ELO 1925, #27
- AA APEX-Agents: 47.05 (#2/41)
- AA MMMU-Pro: 84.28 (#14/331)
- Mercor APEX: 27.5 (#24/24)
- AA IFBench: 76.33 (#30/587)
- AA TAU-2 Bench: 95.32 (#38/575)
- AA-Omniscience Accuracy: 51.4 (#39/521)
- AA Humanity's Last Exam: 42.68 (#45/798)
- AA GPQA Diamond: 92.22 (#46/797)
- AA Omniscience: 21.18 (#47/670)
- AA SciCode: 53.94 (#54/215)
- ...and 6 more
DeepSeek V4 Pro (0813) — ELO 1919, #28
- Vals AI IOI: 51.61 (#14/26)
- Mercor APEX: 47.3 (#20/24)
- AA GPQA Diamond: 92.83 (#35/797)
- Tau3 Banking: 39.59 (#38/260)
- AA-Omniscience Accuracy: 49.1 (#44/521)
- Conceptual Reasoning Index - Decision Theory (DTBench): 89.78 (#45/241)
- AA CritPt: 18.0 (#51/674)
- DuelLab Overall: 41.4 (#53/171)
- Artificial Analysis Intelligence Index: 36.28 (#60/830)
- AA Long Context Reasoning: 80.33 (#60/664)
- ...and 6 more
Claude Sonnet 5 — ELO 1916, #29
- Icelandic LLM Leaderboard - Average: 55.17 (#10/23)
- MLS-Bench Lite: 31.4 (#12/14)
- LOL Arena: 48.0 (#16/42)
- Vals AI IOI: 45.0 (#17/26)
- AA-Briefcase: 42.32 (#32/162)
- AA Long Context Reasoning: 82.0 (#33/664)
- Conceptual Reasoning Index - Consistency (ACCoRD): 76.88 (#35/184)
- AA SciCode: 54.28 (#45/215)
- Conceptual Reasoning Index - Argument Evaluation (LMCA): 50.02 (#47/185)
- Tau3 Banking: 37.32 (#50/260)
- ...and 10 more
Claude Opus 4.6 — ELO 1914, #31
- AA APEX-Agents: 33.04 (#10/41)
- AA Terminal-Bench Hard: 46.21 (#39/565)
- AA-Omniscience Accuracy: 46.98 (#52/521)
- AA Humanity's Last Exam: 39.94 (#68/798)
- AA Omniscience: 13.67 (#71/670)
- AA TAU-2 Bench: 92.11 (#81/575)
- AA CritPt: 12.57 (#82/674)
- AA MMMU-Pro: 75.43 (#88/331)
- Artificial Analysis Intelligence Index: 31.95 (#90/830)
- AA GPQA Diamond: 89.6 (#92/797)
- ...and 2 more
GPT-5.6 Luna — ELO 1913, #32
- LOL Arena: 46.5 (#26/42)
- AA-Briefcase: 34.85 (#55/162)
- Conceptual Reasoning Index - Argument Evaluation (LMCA): 48.54 (#56/185)
- Conceptual Reasoning Index: 58.73 (#66/184)
- AA SciCode: 51.62 (#69/215)
- Conceptual Reasoning Index - Decision Theory (DTBench): 81.78 (#75/241)
- Conceptual Reasoning Index - Consistency (ACCoRD): 67.51 (#84/184)
Qwen3.8-Flash-Next — ELO 1910, #34
- AA-Briefcase: 54.1 (#10/162)
Qwen3.8 2.4T A95B — ELO 1909, #35
- Conceptual Reasoning Index - Decision Theory (DTBench): 88.65 (#49/241)
- Conceptual Reasoning Index - Consistency (ACCoRD): 71.63 (#54/184)
- Conceptual Reasoning Index: 59.36 (#61/184)
- Conceptual Reasoning Index - Argument Evaluation (LMCA): 45.51 (#67/185)
GPT-5.4 — ELO 1908, #36
- PostTrainBench: 19.0 (#13/13)
- Chess Bench LLM: 590.0 (#49/127)
GLM-5.3 Flash — ELO 1907, #37
- ALEM (Multi-Agent Coordination): 15.9 (#2/18)
- ErdosBench: 2.95 (#4/13)
- SuperCLUE-Terminal - Overall: 48.48 (#6/14)
- WebDev Arena (Fullstack): 1614.0 (#13/63)
- Vals AI IOI: 52.5 (#13/26)
- Mercor APEX: 52.8 (#14/24)
- NYT Connections Older Models: 63.5 (#16/113)
- InferenceBench: 4.49 (#23/35)
- Chatbot Arena (Text - Spanish): 1468.0 (#27/283)
- DuelLab Overall: 49.7 (#28/171)
- ...and 2 more
Qwen 3.8 Flash — ELO 1906, #38
- SuperCLUE-Terminal - Overall: 48.48 (#5/14)
- DuelLab Overall: 47.5 (#33/171)
- OpenRouter GPQA Diamond: 83.3 (#50/130)
GPT-5.3 Codex — ELO 1905, #39
- AA-Briefcase: 21.82 (#87/162)
GLM-5.2 — ELO 1901, #40
- PostTrainBench: 31.7 (#8/13)
- AA-Briefcase: 36.06 (#51/162)
- DuelLab Overall: 38.8 (#68/171)
- LM Market Cap LMC Score: 78.6 (#108/453)
DeepSeek-V4-Flash-Vision-Exp — ELO 1897, #41
- FrontierSWE V2: 14.8 (#8/10)
DeepSeek V4 Flash (0731) — ELO 1895, #42
- Vals AI IOI: 32.72 (#23/26)
- AA-Briefcase: 41.52 (#37/162)
- Tau3 Banking: 39.38 (#40/260)
- AA CritPt: 16.57 (#62/674)
- Artificial Analysis Intelligence Index: 34.53 (#70/830)
- AA GPQA Diamond: 90.81 (#72/797)
- Design Arena (ASCII Art): 1110.0 (#74/87)
- AA Long Context Reasoning: 79.67 (#77/664)
- AA Humanity's Last Exam: 38.55 (#80/798)
- AA SciCode: 50.35 (#85/215)
- ...and 2 more
Gemini 3 Pro (Preview) — ELO 1892, #45
- AA MMLU-Pro: 89.54 (#2/349)
- AA LiveCodeBench: 85.71 (#13/343)
- AA AIME 2025: 86.67 (#43/270)
Kimi K2.7 Code — ELO 1877, #47
- AA-Briefcase: 20.7 (#90/162)
Qwen 3.7 Max — ELO 1876, #48
- AA-Briefcase: 23.74 (#82/162)
Claude Opus 4.5 — ELO 1873, #50
- AA Terminal-Bench Hard: 46.97 (#34/565)
- AA-Omniscience Accuracy: 46.58 (#54/521)
- AA Omniscience: 14.0 (#68/670)
- AA TAU-2 Bench: 89.47 (#99/575)
- Artificial Analysis Intelligence Index: 29.1 (#110/830)
- AA MMMU-Pro: 74.05 (#112/331)
- AA Long Context Reasoning: 77.33 (#122/664)
- AA Humanity's Last Exam: 30.12 (#138/798)
- AA GPQA Diamond: 86.57 (#146/797)
- AA CritPt: 4.57 (#153/674)
- ...and 1 more
Claude Sonnet 4.6 — ELO 1872, #51
- AA APEX-Agents: 28.02 (#16/41)
- LLM Arena RU: 1104.0 (#18/111)
- AA Terminal-Bench Hard: 53.03 (#21/565)
- Tau3 Banking: 34.43 (#64/260)
- AA-Briefcase: 28.92 (#66/162)
- AA Long Context Reasoning: 80.0 (#70/664)
- AA Omniscience: 12.22 (#76/670)
- AA-Omniscience Accuracy: 40.87 (#83/521)
- AA SciCode: 50.12 (#86/215)
- Artificial Analysis Intelligence Index: 30.45 (#101/830)
- ...and 6 more
Gemini 3 Flash — ELO 1866, #52
- AA APEX-Agents: 27.73 (#18/41)
- AA IFBench: 77.96 (#22/587)
- AA-Omniscience Accuracy: 53.43 (#32/521)
- AA MMMU-Pro: 79.94 (#43/331)
- AA Terminal-Bench Hard: 38.64 (#76/565)
- AA Omniscience: 10.13 (#81/670)
- AA GPQA Diamond: 89.8 (#87/797)
- AA Humanity's Last Exam: 36.56 (#95/798)
- AA Long Context Reasoning: 78.0 (#107/664)
- AA CritPt: 8.57 (#111/674)
- ...and 3 more
Qwen 3.7 Plus — ELO 1863, #53
- AA-Briefcase: 22.83 (#84/162)
Gemini 3 Flash (Preview) — ELO 1858, #55
- AA LiveCodeBench: 90.79 (#2/343)
- AA AIME 2025: 97.0 (#3/270)
- AA MMLU-Pro: 89.03 (#4/349)
Hy3 — ELO 1858, #56
- AA-Briefcase: 26.73 (#73/162)
- NYT Connections Extended: 40.6 (#77/117)
Qwen 3.8 27B — ELO 1856, #57
- Vals AI IOI: 39.06 (#21/26)
- AA-Briefcase: 46.53 (#26/162)
- LOL Arena: 45.2 (#34/42)
- DuelLab Overall: 47.1 (#35/171)
- Conceptual Reasoning Index - Consistency (ACCoRD): 73.25 (#49/184)
- Chatbot Arena (Text - German): 1452.0 (#57/299)
- NYT Connections Extended: 52.0 (#66/117)
- Chatbot Arena (Text - Korean): 1401.0 (#67/269)
- Conceptual Reasoning Index: 55.47 (#77/184)
- Conceptual Reasoning Index - Decision Theory (DTBench): 80.0 (#78/241)
- ...and 12 more
Kimi K2.6 — ELO 1855, #59
- AA SciCode: 51.5 (#73/215)
- AA-Briefcase: 20.61 (#91/162)
GPT-5.2 — ELO 1854, #60
- AA AIME 2025: 99.0 (#1/270)
- AA LiveCodeBench: 89.42 (#4/343)
- AA MMLU-Pro: 87.36 (#10/349)
- Icelandic LLM Leaderboard - Average: 46.97 (#15/23)
GPT-5 — ELO 1852, #62
- LOL Arena: 48.5 (#13/42)
DeepSeek V4 Pro — ELO 1852, #63
- AA APEX-Agents: 24.26 (#21/41)
- AA TAU-2 Bench: 96.2 (#22/575)
- LOL Arena: 46.1 (#27/42)
- AA IFBench: 76.46 (#28/587)
- AA Terminal-Bench Hard: 46.21 (#40/565)
- DuelLab Overall: 40.9 (#58/171)
- Chess Bench LLM: 310.0 (#66/127)
- AA-Omniscience Accuracy: 42.95 (#73/521)
- AA GPQA Diamond: 90.51 (#77/797)
- AA SciCode: 50.81 (#79/215)
- ...and 7 more
GLM-5.1 — ELO 1850, #64
- AA TAU-2 Bench: 97.66 (#16/575)
- AA IFBench: 76.26 (#32/587)
- AA Terminal-Bench Hard: 43.18 (#54/565)
- AA-Briefcase: 25.76 (#76/162)
- AA Omniscience: 0.85 (#111/670)
- AA SciCode: 44.79 (#123/215)
- Artificial Analysis Intelligence Index: 26.45 (#137/830)
- AA GPQA Diamond: 86.77 (#137/797)
- AA Humanity's Last Exam: 30.07 (#140/798)
- AA CritPt: 4.57 (#154/674)
- ...and 3 more
DeepSeek V4 Flash — ELO 1824, #72
- AA IFBench: 79.18 (#16/587)
- AA TAU-2 Bench: 95.61 (#34/575)
- AA-Briefcase: 28.48 (#67/162)
- Chess Bench LLM: 225.0 (#72/127)
- Tau3 Banking: 30.93 (#77/260)
- AA Terminal-Bench Hard: 38.64 (#77/565)
- DuelLab Overall: 35.1 (#88/171)
- AA GPQA Diamond: 89.39 (#98/797)
- AA Humanity's Last Exam: 34.85 (#106/798)
- AA-Omniscience Accuracy: 36.82 (#110/521)
- ...and 5 more
MiniMax-M3 — ELO 1824, #73
- Icelandic LLM Leaderboard - Average: 31.3 (#18/23)
- LOL Arena: 43.4 (#40/42)
Inkling Small — ELO 1824, #74
- Vals AI IOI: 9.33 (#26/26)
- AA-Briefcase: 20.0 (#93/162)
- DuelLab Overall: 28.5 (#131/171)
DeepSeek V3.2 Speciale — ELO 1821, #77
- AA LiveCodeBench: 89.63 (#3/343)
- AA AIME 2025: 96.67 (#5/270)
- AA MMLU-Pro: 86.31 (#17/349)
Grok 4.20 0309 — ELO 1821, #78
- AA IFBench: 82.93 (#3/587)
- AA TAU-2 Bench: 96.49 (#20/575)
- AA APEX-Agents: 14.23 (#32/41)
- AA Terminal-Bench Hard: 40.91 (#69/565)
- AA Omniscience: 12.95 (#74/670)
- AA GPQA Diamond: 88.48 (#114/797)
- AA Humanity's Last Exam: 32.39 (#126/798)
- AA MMMU-Pro: 73.18 (#126/331)
- AA CritPt: 6.0 (#127/674)
- AA-Omniscience Accuracy: 28.88 (#149/521)
- ...and 2 more
GPT-5.1 — ELO 1820, #79
- ZeroEval GPQA Diamond: 88.1 (#42/250)
MiMo-V2.5-Pro — ELO 1817, #82
- AA-Briefcase: 21.41 (#88/162)
Claude Sonnet 4.5 — ELO 1816, #83
- Last Translation Benchmark: 4.5 (#23/42)
- AA Terminal-Bench Hard: 35.61 (#101/565)
- AA-Briefcase: 14.65 (#103/162)
- Tau3 Banking: 24.54 (#105/260)
- AA SciCode: 45.72 (#114/215)
- AA Omniscience: -0.08 (#123/670)
- AA-Omniscience Accuracy: 32.87 (#124/521)
- AA TAU-2 Bench: 78.07 (#174/575)
- AA MMMU-Pro: 68.73 (#174/331)
- AA IFBench: 57.28 (#182/587)
- ...and 5 more
Claude Opus 4.1 — ELO 1811, #84
- AA MMLU-Pro: 87.99 (#7/349)
- AA AIME 2025: 80.33 (#64/270)
- AA LiveCodeBench: 65.35 (#94/343)
- AA Terminal-Bench Hard: 34.34 (#115/565)
- AA Long Context Reasoning: 76.0 (#139/664)
- AA MMMU-Pro: 67.92 (#177/331)
- Artificial Analysis Intelligence Index: 22.85 (#190/830)
- AA IFBench: 55.44 (#194/587)
- AA TAU-2 Bench: 71.37 (#198/575)
- AA GPQA Diamond: 80.9 (#246/797)
- ...and 2 more
Inkling — ELO 1810, #85
- LLM Stats (AIME 2026): 97.1 (#2/26)
- LLM Stats (Tau3 Banking): 23.7 (#4/10)
- LLM Stats (Humanity's Last Exam (with tools, text-only)): 46.0 (#8/9)
- LLM Stats (GDPval-AA): 1238.0 (#10/11)
- LLM Stats (MCP Atlas): 76.0 (#13/36)
- Vals AI IOI: 14.94 (#25/26)
- LLM Stats (Terminal-Bench 2.1): 63.8 (#29/39)
- LLM Stats (CharXiv-R): 78.1 (#31/58)
- LLM Stats Score: 37.38 (#67/366)
- AA-Omniscience Accuracy: 41.55 (#78/521)
- ...and 9 more
GPT-5.1 Codex — ELO 1810, #87
- AA AIME 2025: 95.67 (#8/270)
- AA LiveCodeBench: 84.87 (#15/343)
- AA MMLU-Pro: 86.01 (#20/349)
GLM-5 — ELO 1809, #88
- AA TAU-2 Bench: 98.25 (#12/575)
- AA APEX-Agents: 14.45 (#30/41)
- AA Terminal-Bench Hard: 43.18 (#53/565)
- AA IFBench: 72.31 (#68/587)
- AA Omniscience: 0.27 (#118/670)
- Artificial Analysis Intelligence Index: 27.91 (#124/830)
- AA Long Context Reasoning: 75.67 (#144/664)
- AA Humanity's Last Exam: 29.29 (#148/798)
- AA-Omniscience Accuracy: 26.3 (#174/521)
- AA CritPt: 2.0 (#201/674)
- ...and 1 more
Kimi K2.5 — ELO 1804, #93
- AA TAU-2 Bench: 95.91 (#28/575)
- AA APEX-Agents: 11.5 (#35/41)
- AA MMMU-Pro: 75.38 (#90/331)
- AA IFBench: 70.2 (#91/587)
- AA Long Context Reasoning: 78.0 (#108/664)
- AA Terminal-Bench Hard: 34.85 (#113/565)
- AA-Omniscience Accuracy: 35.22 (#114/521)
- AA GPQA Diamond: 87.88 (#120/797)
- AA Humanity's Last Exam: 30.72 (#133/798)
- Tau3 Banking: 14.23 (#154/260)
- ...and 3 more
Qwen 3.5 397B A17B — ELO 1801, #95
- AA IFBench: 78.78 (#19/587)
- AA APEX-Agents: 15.34 (#25/41)
- AA TAU-2 Bench: 95.61 (#33/575)
- AA Terminal-Bench Hard: 40.91 (#70/565)
- AA MMMU-Pro: 77.28 (#71/331)
- AA GPQA Diamond: 89.29 (#100/797)
- AA-Briefcase: 12.12 (#115/162)
- AA Long Context Reasoning: 77.33 (#121/664)
- AA SciCode: 44.79 (#124/215)
- AA-Omniscience Accuracy: 30.8 (#134/521)
- ...and 5 more
Hy3-preview — ELO 1800, #96
- AA TAU-2 Bench: 92.69 (#76/575)
- AA Terminal-Bench Hard: 34.09 (#120/565)
- AA GPQA Diamond: 86.67 (#143/797)
- AA IFBench: 63.13 (#150/587)
- AA CritPt: 4.57 (#155/674)
- AA-Omniscience Accuracy: 27.93 (#157/521)
- AA Humanity's Last Exam: 27.76 (#168/798)
- Artificial Analysis Intelligence Index: 22.74 (#193/830)
- AA Long Context Reasoning: 64.67 (#276/664)
- AA Omniscience: -35.05 (#317/670)
Gemini 2.5 Pro — ELO 1796, #97
- AA SciCode: 46.3 (#111/215)
- AA-Briefcase: 6.57 (#132/162)
Gemini 3.5 Flash Lite — ELO 1794, #99
- Icelandic LLM Leaderboard - Average: 49.57 (#14/23)
Gemini 2.5 Pro (Preview 03-25) — ELO 1793, #100
- AA MMLU-Pro: 85.75 (#25/349)
- AA LiveCodeBench: 77.78 (#36/343)
Muse Glimmer 30B — ELO 1788, #102
- DuelLab Overall: 28.8 (#129/171)
GPT-5.4 Mini — ELO 1786, #103
- AA SciCode: 52.08 (#64/215)
- AA-Briefcase: 12.54 (#112/162)
Claude Opus 4 — ELO 1785, #105
- AA MMLU-Pro: 87.32 (#11/349)
- AA AIME 2025: 73.33 (#86/270)
- AA LiveCodeBench: 63.6 (#105/343)
- AA Terminal-Bench Hard: 31.06 (#151/565)
- AA TAU-2 Bench: 73.39 (#191/575)
- AA IFBench: 53.74 (#210/587)
- Artificial Analysis Intelligence Index: 20.64 (#228/830)
- AA Long Context Reasoning: 69.33 (#235/664)
- AA GPQA Diamond: 79.6 (#261/797)
- AA Humanity's Last Exam: 12.33 (#307/798)
MiMo-V2.5 — ELO 1782, #106
- AA-Briefcase: 13.87 (#107/162)
Qwen 3.6 27B — ELO 1781, #107
- AA TAU-2 Bench: 94.15 (#56/575)
- AA-Briefcase: 17.68 (#96/162)
- AA MMMU-Pro: 74.62 (#102/331)
- AA IFBench: 67.55 (#110/587)
- AA Terminal-Bench Hard: 34.85 (#112/565)
- AA Long Context Reasoning: 77.33 (#123/664)
- AA SciCode: 42.82 (#135/215)
- Tau3 Banking: 16.7 (#135/260)
- AA GPQA Diamond: 84.24 (#192/797)
- AA Humanity's Last Exam: 23.08 (#194/798)
- ...and 4 more
Gemini 2.5 Pro (Preview 05-06) — ELO 1781, #108
- AA LiveCodeBench: 77.04 (#39/343)
- AA MMLU-Pro: 83.73 (#48/349)
Grok 4.3 — ELO 1777, #110
- Chess Bench LLM: 1021.0 (#31/127)
- AA-Briefcase: 17.27 (#98/162)
- AA SciCode: 39.35 (#156/215)
GLM-5V Turbo — ELO 1777, #111
- AA TAU-2 Bench: 98.54 (#9/575)
- AA MMMU-Pro: 72.77 (#132/331)
- AA Terminal-Bench Hard: 32.58 (#133/565)
- AA-Omniscience Accuracy: 29.33 (#145/521)
- AA IFBench: 61.09 (#159/587)
- Artificial Analysis Intelligence Index: 23.5 (#177/830)
- AA Long Context Reasoning: 70.33 (#217/664)
- AA Omniscience: -19.28 (#233/670)
- AA GPQA Diamond: 80.91 (#244/797)
- AA Humanity's Last Exam: 17.15 (#250/798)
- ...and 1 more
GPT-5 Mini — ELO 1770, #113
- LOL Arena: 45.6 (#32/42)
- AA-Briefcase: 8.28 (#125/162)
- AA SciCode: 39.0 (#158/215)
Gemini 3.1 Flash Lite — ELO 1769, #115
- AA-Briefcase: 6.33 (#133/162)
Qwen 3.5 122B A10B — ELO 1765, #116
- AA IFBench: 75.71 (#39/587)
- AA TAU-2 Bench: 93.57 (#63/575)
- AA MMMU-Pro: 74.97 (#97/331)
- AA-Briefcase: 10.61 (#120/162)
- AA Long Context Reasoning: 76.33 (#133/664)
- Tau3 Banking: 15.26 (#145/260)
- AA Terminal-Bench Hard: 31.06 (#150/565)
- AA SciCode: 39.7 (#154/215)
- AA GPQA Diamond: 85.66 (#163/797)
- AA Humanity's Last Exam: 25.21 (#179/798)
- ...and 4 more
Claude Sonnet 4 — ELO 1764, #117
- AA MMLU-Pro: 84.15 (#42/349)
- AA AIME 2025: 74.33 (#82/270)
- AA LiveCodeBench: 65.5 (#93/343)
- AA Omniscience: 0.17 (#120/670)
- Tau3 Banking: 16.7 (#136/260)
- AA Terminal-Bench Hard: 31.06 (#152/565)
- AA IFBench: 54.69 (#200/587)
- AA Long Context Reasoning: 70.33 (#219/664)
- AA TAU-2 Bench: 64.62 (#224/575)
- AA MMMU-Pro: 61.79 (#228/331)
- ...and 5 more
DeepSeek V3.2 — ELO 1763, #118
- AA LiveCodeBench: 86.24 (#11/343)
- AA AIME 2025: 92.0 (#17/270)
- AA MMLU-Pro: 86.21 (#18/349)
- AA APEX-Agents: 14.53 (#28/41)
- AA TAU-2 Bench: 90.64 (#92/575)
- AA Terminal-Bench Hard: 35.61 (#102/565)
- AA-Omniscience Accuracy: 32.97 (#123/521)
- AA IFBench: 60.68 (#164/587)
- AA Long Context Reasoning: 73.33 (#178/664)
- AA Humanity's Last Exam: 24.56 (#182/798)
- ...and 4 more
Gemini 3.1 Flash Lite (Preview) — ELO 1762, #120
- Vals AI Harvey Legal Agent Bench: 0.0 (#46/60)
- Vals AI Excel Modeling: 8.63 (#56/56)
GLM-4.7 — ELO 1762, #121
- AA TAU-2 Bench: 95.91 (#27/575)
- AA IFBench: 67.89 (#107/587)
- AA Terminal-Bench Hard: 31.82 (#140/565)
- AA-Omniscience Accuracy: 29.32 (#146/521)
- AA GPQA Diamond: 85.86 (#155/797)
- Tau3 Banking: 12.16 (#168/260)
- AA Humanity's Last Exam: 27.39 (#171/798)
- Artificial Analysis Intelligence Index: 22.24 (#206/830)
- AA Long Context Reasoning: 71.0 (#208/664)
- AA CritPt: 1.71 (#209/674)
- ...and 1 more
Seed 1.8 — ELO 1762, #122
- LLM Stats (BrowseComp-zh): 81.3 (#1/15)
- LLM Stats (VideoMME w sub.): 87.8 (#2/12)
- LLM Stats (Beyond AIME): 77.0 (#3/7)
- LLM Stats (BLINK): 74.3 (#3/17)
- LLM Stats (MathVista): 87.7 (#3/39)
- LLM Stats (MuirBench): 78.7 (#3/14)
- LLM Stats (Multi-SWE-Bench): 42.0 (#4/7)
- LLM Stats (MMVU): 73.1 (#5/7)
- LLM Stats (CountBench): 96.3 (#6/9)
- LLM Stats (MMStar): 79.9 (#6/27)
- ...and 19 more
Gemma 4 31B (IT) — ELO 1759, #123
- Icelandic LLM Leaderboard - Average: 20.73 (#20/23)
Step 3.7 Flash — ELO 1759, #125
- LLM Stats (Humanity's Last Exam (with tools, text-only)): 47.2 (#7/9)
- LLM Stats (Terminal-Bench 2.1): 59.5 (#31/39)
- LLM Stats Score: 30.28 (#119/366)
Nemotron 3 Ultra — ELO 1756, #127
- AA IFBench: 81.36 (#6/587)
- AA Long Context Reasoning: 79.33 (#84/664)
- AA Terminal-Bench Hard: 36.36 (#91/565)
- AA Omniscience: -0.4 (#125/670)
- AA GPQA Diamond: 86.67 (#141/797)
- AA SciCode: 40.28 (#149/215)
- AA TAU-2 Bench: 83.33 (#149/575)
- Tau3 Banking: 14.23 (#153/260)
- AA Humanity's Last Exam: 28.41 (#158/798)
- AA CritPt: 3.14 (#172/674)
- ...and 2 more
Grok 4.1 Fast — ELO 1754, #129
- AA TAU-2 Bench: 93.27 (#67/575)
- AA Long Context Reasoning: 74.0 (#165/664)
- AA GPQA Diamond: 85.25 (#170/797)
- AA CritPt: 2.86 (#182/674)
- AA-Omniscience Accuracy: 25.08 (#190/521)
- AA Terminal-Bench Hard: 24.24 (#200/565)
- AA MMMU-Pro: 63.29 (#209/331)
- AA IFBench: 52.72 (#217/587)
- AA Humanity's Last Exam: 19.32 (#228/798)
- Artificial Analysis Intelligence Index: 20.37 (#233/830)
- ...and 1 more
Solar Pro 4 — ELO 1753, #130
- Nejumi 4 - ALT Average: 89.57 (#11/131)
- Nejumi 4 - ALT - Toxicity: 85.74 (#11/131)
- Nejumi 4 - GLP - Mathematical Reasoning: 96.67 (#14/131)
- Nejumi 4 - GLP - Information Retrieval: 80.96 (#21/131)
- Nejumi 4 - ALT - Bias: 91.43 (#25/131)
- Nejumi 4 - ALT - Robustness: 95.0 (#32/131)
- Nejumi 4 - ALT - Truthfulness: 80.51 (#34/131)
- Nejumi 4 - GLP - Applied Language: 89.1 (#36/131)
- Nejumi 4 - ALT - Ethics & Morality: 95.0 (#38/131)
- Nejumi 4 - GLP - Syntactic Analysis: 84.5 (#39/131)
- ...and 20 more
Qwen 3.5 27B — ELO 1752, #132
- AA IFBench: 75.58 (#42/587)
- AA TAU-2 Bench: 93.86 (#60/575)
- AA MMMU-Pro: 75.03 (#94/331)
- AA Long Context Reasoning: 77.67 (#113/664)
- AA Terminal-Bench Hard: 32.58 (#132/565)
- AA GPQA Diamond: 85.76 (#157/797)
- Artificial Analysis Intelligence Index: 22.9 (#187/830)
- AA Humanity's Last Exam: 23.91 (#187/798)
- AA-Omniscience Accuracy: 20.67 (#257/521)
- AA CritPt: 0.85 (#268/674)
- ...and 1 more
Grok 4 Fast — ELO 1752, #133
- AA Long Context Reasoning: 73.67 (#170/664)
- AA GPQA Diamond: 84.75 (#176/797)
- AA CritPt: 2.86 (#183/674)
- AA TAU-2 Bench: 65.79 (#220/575)
- AA MMMU-Pro: 61.79 (#227/331)
- AA-Omniscience Accuracy: 22.78 (#230/521)
- AA Humanity's Last Exam: 19.09 (#232/798)
- AA Terminal-Bench Hard: 18.94 (#232/565)
- AA IFBench: 50.54 (#236/587)
- Artificial Analysis Intelligence Index: 17.94 (#264/830)
- ...and 1 more
Ring-2.6-1T — ELO 1752, #134
- AA-Briefcase: 10.93 (#118/162)
- AA SciCode: 45.02 (#121/215)
O1 — ELO 1750, #136
- AA MMLU-Pro: 84.06 (#44/349)
- AA LiveCodeBench: 67.94 (#83/343)
DeepSeek V3.2 Exp — ELO 1749, #137
- AA Terminal-Bench Hard: 31.06 (#153/565)
- AA-Omniscience Accuracy: 27.55 (#160/521)
- AA Long Context Reasoning: 72.33 (#189/664)
- AA IFBench: 54.15 (#207/587)
- AA CritPt: 1.43 (#223/674)
- AA GPQA Diamond: 79.7 (#259/797)
- AA Humanity's Last Exam: 14.87 (#275/798)
- Artificial Analysis Intelligence Index: 16.58 (#289/830)
- AA Omniscience: -30.97 (#299/670)
- AA TAU-2 Bench: 33.92 (#327/575)
LongCat 2.0 — ELO 1749, #138
- AA-Briefcase: 16.87 (#99/162)
O4 Mini — ELO 1747, #139
- AA LiveCodeBench: 85.93 (#12/343)
- AA AIME 2025: 90.67 (#24/270)
- Chess Bench LLM: 408.0 (#57/127)
- AA MMLU-Pro: 83.19 (#58/349)
Qwen 3 Max — ELO 1747, #140
- LLM Stats (C-Eval): 93.7 (#1/20)
- LLM Stats (DeepPlanning): 28.7 (#4/10)
- LLM Stats (Seal-0): 46.9 (#5/7)
- LLM Stats (Humanity's Last Exam (with tools, text-only)): 49.8 (#6/9)
- LLM Stats (VITA-Bench): 40.9 (#6/11)
- LLM Stats (MAXIFE): 84.0 (#8/12)
- LLM Stats (Multi-Challenge): 63.3 (#8/32)
- LLM Stats (MCP-Mark): 33.5 (#9/9)
- LLM Stats (LongBench v2): 60.6 (#10/18)
- LLM Stats (NOVA-63): 54.2 (#10/12)
- ...and 12 more
Gemma 4 31B — ELO 1739, #143
- AA IFBench: 75.58 (#43/587)
- AA Terminal-Bench Hard: 36.36 (#92/565)
- AA SciCode: 45.49 (#118/215)
- AA MMMU-Pro: 73.41 (#121/331)
- AA-Briefcase: 7.78 (#127/162)
- Tau3 Banking: 14.85 (#149/260)
- AA GPQA Diamond: 85.66 (#164/797)
- AA Humanity's Last Exam: 23.63 (#191/798)
- AA CritPt: 1.43 (#224/674)
- AA Long Context Reasoning: 69.67 (#231/664)
- ...and 4 more
DeepSeek R1 0528 — ELO 1738, #145
- Chess Bench LLM: 0.0 (#89/127)
DeepSeek V3.1 Terminus — ELO 1737, #146
- Tau3 Banking: 21.03 (#115/260)
- AA-Briefcase: 9.7 (#122/162)
- AA Terminal-Bench Hard: 30.3 (#158/565)
- AA-Omniscience Accuracy: 27.7 (#159/521)
- AA SciCode: 37.96 (#167/215)
- AA IFBench: 57.01 (#185/587)
- AA CritPt: 1.71 (#207/674)
- AA Long Context Reasoning: 69.33 (#238/664)
- AA Humanity's Last Exam: 16.36 (#256/798)
- AA GPQA Diamond: 79.19 (#267/797)
- ...and 3 more
Kimi K2 — ELO 1734, #147
- Chess Bench LLM: 849.0 (#37/127)
- LOL Arena: 44.3 (#38/42)
GPT-4o ChatGPT — ELO 1733, #149
- AA MMLU-Pro: 77.34 (#154/349)
Qwen 3.6 35B A3B — ELO 1731, #151
- AA TAU-2 Bench: 95.32 (#39/575)
- AA MMMU-Pro: 75.03 (#95/331)
- AA-Briefcase: 15.25 (#101/162)
- AA Terminal-Bench Hard: 34.85 (#111/565)
- AA IFBench: 64.35 (#140/587)
- AA SciCode: 36.57 (#172/215)
- Tau3 Banking: 9.28 (#190/260)
- AA GPQA Diamond: 84.14 (#195/797)
- AA Humanity's Last Exam: 22.24 (#200/798)
- AA Long Context Reasoning: 71.67 (#202/664)
- ...and 4 more
Ling-3.0-flash — ELO 1730, #152
- LLM Stats (Multi-IF): 87.7 (#1/25)
- LLM Stats (Tau3 Banking): 28.0 (#3/10)
- LLM Stats (SkillsBench): 44.8 (#8/10)
- LLM Stats (WideSearch): 73.6 (#8/14)
- LLM Stats (HMMT Feb 26): 87.0 (#9/13)
- LLM Stats (GDPval-AA): 1107.0 (#11/11)
- LLM Stats (AIME 2026): 93.2 (#14/26)
- LLM Stats (IMO-AnswerBench): 83.7 (#14/22)
- LLM Stats (MCP Atlas): 65.5 (#27/36)
- LLM Stats (BrowseComp): 72.2 (#31/66)
- ...and 3 more
O1 Preview — ELO 1730, #153
- AA MMLU-Pro: 84.77 (#37/349)
- AA AIME 2025: 79.67 (#69/270)
GLM-4.6 — ELO 1728, #154
- LOL Arena: 45.6 (#33/42)
- Tau3 Banking: 13.4 (#161/260)
- AA-Omniscience Accuracy: 26.88 (#167/521)
- AA Terminal-Bench Hard: 25.0 (#192/565)
- AA TAU-2 Bench: 70.47 (#203/575)
- AA CritPt: 1.14 (#240/674)
- Artificial Analysis Intelligence Index: 18.53 (#259/830)
- AA Humanity's Last Exam: 14.46 (#278/798)
- AA GPQA Diamond: 77.98 (#288/797)
- AA IFBench: 43.4 (#307/587)
- ...and 2 more
Qwen 3 Max (Preview) — ELO 1728, #155
- AA MMLU-Pro: 83.83 (#45/349)
- AA AIME 2025: 75.0 (#80/270)
- AA LiveCodeBench: 65.08 (#97/343)
Gemini 2.5 Flash — ELO 1725, #159
- Last Translation Benchmark: 4.61 (#22/42)
- Chess Bench LLM: 636.0 (#43/127)
- AA MMLU-Pro: 83.15 (#59/349)
- AA LiveCodeBench: 69.52 (#73/343)
- AA AIME 2025: 73.33 (#87/270)
- AA MMMU-Pro: 69.08 (#168/331)
- AA-Omniscience Accuracy: 25.95 (#180/521)
- AA IFBench: 50.27 (#238/587)
- AA CritPt: 1.14 (#239/674)
- AA Long Context Reasoning: 65.33 (#268/664)
- ...and 6 more
MiniMax-M2.7 — ELO 1725, #160
- ALEM (Multi-Agent Coordination): 7.0 (#10/18)
- AA SciCode: 50.12 (#88/215)
- AA-Briefcase: 14.14 (#105/162)
DeepSeek V3.1 — ELO 1724, #162
- AA AIME 2025: 89.67 (#29/270)
- AA MMLU-Pro: 85.06 (#31/349)
- AA LiveCodeBench: 78.41 (#34/343)
- Chess Bench LLM: 399.0 (#59/127)
- AA-Omniscience Accuracy: 29.0 (#147/521)
- AA Terminal-Bench Hard: 25.0 (#191/565)
- AA CritPt: 2.0 (#200/674)
- AA Humanity's Last Exam: 14.27 (#280/798)
- AA Omniscience: -29.58 (#284/670)
- AA GPQA Diamond: 77.88 (#290/797)
- ...and 4 more
Claude 3.7 Sonnet — ELO 1722, #165
- AA MMLU-Pro: 83.69 (#50/349)
- AA AIME 2025: 56.33 (#127/270)
- AA Omniscience: -0.73 (#129/670)
- AA LiveCodeBench: 47.3 (#156/343)
- AA-Omniscience Accuracy: 28.03 (#156/521)
- AA Terminal-Bench Hard: 21.21 (#221/565)
- AA TAU-2 Bench: 54.68 (#244/575)
- AA IFBench: 48.3 (#254/587)
- AA CritPt: 0.86 (#262/674)
- Artificial Analysis Intelligence Index: 17.71 (#269/830)
- ...and 2 more
Claude Haiku 4.5 — ELO 1722, #166
- Icelandic LLM Leaderboard - Average: 15.68 (#22/23)
- AA-Briefcase: 10.81 (#119/162)
- AA SciCode: 42.25 (#140/215)
- AA Omniscience: -4.37 (#149/670)
- AA Long Context Reasoning: 74.33 (#158/664)
- AA Terminal-Bench Hard: 27.27 (#174/565)
- Tau3 Banking: 9.28 (#191/260)
- AA IFBench: 54.29 (#205/587)
- AA MMMU-Pro: 58.55 (#241/331)
- AA TAU-2 Bench: 54.68 (#245/575)
- ...and 5 more
GPT-4.1 — ELO 1719, #167
- LOL Arena: 48.9 (#8/42)
- AA MMLU-Pro: 80.62 (#106/349)
- AA LiveCodeBench: 45.71 (#165/343)
- AA AIME 2025: 34.67 (#176/270)
GLM-4.5 — ELO 1718, #170
- AA-Omniscience Accuracy: 25.13 (#189/521)
- AA Terminal-Bench Hard: 21.97 (#213/565)
- AA Omniscience: -27.38 (#277/670)
- AA GPQA Diamond: 78.18 (#285/797)
- AA TAU-2 Bench: 42.98 (#288/575)
- AA IFBench: 44.08 (#298/587)
- AA Humanity's Last Exam: 12.97 (#300/798)
- AA Long Context Reasoning: 52.67 (#345/664)
- Artificial Analysis Intelligence Index: 12.77 (#361/830)
- AA CritPt: 0.0 (#601/674)
DeepSeek R1 — ELO 1714, #176
- LOL Arena: 46.1 (#29/42)
GPT-5.1 Codex Mini — ELO 1713, #179
- AA AIME 2025: 91.67 (#19/270)
- AA LiveCodeBench: 83.6 (#19/343)
- AA MMLU-Pro: 82.0 (#78/349)
Qwen 3.5 35B A3B — ELO 1712, #180
- AgentCollabBench: 0.9 (#1/4)
- AA IFBench: 72.52 (#63/587)
- AA TAU-2 Bench: 89.18 (#101/575)
- AA MMMU-Pro: 72.66 (#134/331)
- AA Terminal-Bench Hard: 26.52 (#181/565)
- AA GPQA Diamond: 84.55 (#185/797)
- AA Long Context Reasoning: 72.0 (#194/664)
- AA Humanity's Last Exam: 21.04 (#213/798)
- Artificial Analysis Intelligence Index: 19.33 (#246/830)
- AA CritPt: 0.86 (#264/674)
- ...and 2 more
O3 Mini — ELO 1712, #181
- AA LiveCodeBench: 73.44 (#55/343)
- AA MMLU-Pro: 80.19 (#113/349)
- AA SciCode: 42.82 (#136/215)
- AA-Briefcase: 4.55 (#147/162)
GPT-5.4 Nano — ELO 1711, #182
- AA-Briefcase: 14.62 (#104/162)
- AA SciCode: 47.22 (#105/215)
- AA Terminal-Bench Hard: 33.33 (#126/565)
- AA IFBench: 64.42 (#137/587)
- AA CritPt: 5.14 (#141/674)
- AA Omniscience: -18.03 (#228/670)
- Artificial Analysis Intelligence Index: 20.01 (#236/830)
- AA MMMU-Pro: 59.54 (#238/331)
- AA-Omniscience Accuracy: 21.9 (#241/521)
- AA TAU-2 Bench: 52.63 (#253/575)
- ...and 3 more
Qwen 3 235B A22B 2507 — ELO 1709, #184
- AA SciCode: 41.44 (#143/215)
- AA-Briefcase: 4.75 (#143/162)
- AA Long Context Reasoning: 72.0 (#196/664)
- Tau3 Banking: 7.84 (#199/260)
- AA-Omniscience Accuracy: 22.82 (#229/521)
- AA IFBench: 51.22 (#230/587)
- AA TAU-2 Bench: 53.22 (#250/575)
- AA Humanity's Last Exam: 15.89 (#263/798)
- AA Terminal-Bench Hard: 15.15 (#265/565)
- AA GPQA Diamond: 78.99 (#274/797)
- ...and 3 more
Qwen 3 VL 235B A22B — ELO 1708, #185
- AA MMMU-Pro: 67.57 (#179/331)
- AA-Omniscience Accuracy: 20.3 (#265/521)
- AA TAU-2 Bench: 35.09 (#314/575)
- AA IFBench: 42.65 (#321/587)
- AA Terminal-Bench Hard: 6.82 (#355/565)
- AA GPQA Diamond: 71.21 (#390/797)
- AA Long Context Reasoning: 32.67 (#431/664)
- Artificial Analysis Intelligence Index: 9.94 (#437/830)
- AA Humanity's Last Exam: 6.63 (#451/798)
- AA Omniscience: -52.67 (#472/670)
- ...and 1 more
Grok 4.1 — ELO 1705, #186
- Chess Bench LLM: 1406.0 (#18/127)
Seed 2.0 Mini — ELO 1703, #188
- LLM Stats (Graphwalks parents <128k): 93.0 (#1/12)
- LLM Stats (CharXiv-D): 91.9 (#3/18)
- LLM Stats (ZEROBench-Sub): 36.2 (#3/6)
- LLM Stats (BLINK): 73.4 (#4/17)
- LLM Stats (MuirBench): 78.0 (#4/14)
- LLM Stats (Beyond AIME): 69.0 (#5/7)
- LLM Stats (COLLIE): 91.2 (#5/11)
- LLM Stats (OCRBench_V2): 58.5 (#5/8)
- LLM Stats (WorldVQA): 47.6 (#5/6)
- LLM Stats (MMLongBench-Doc): 48.9 (#6/6)
- ...and 28 more
Qwen3 Coder Next — ELO 1701, #190
- AA-Briefcase: 7.87 (#126/162)
- AA SciCode: 36.23 (#178/215)
MiMo-V2-Flash — ELO 1698, #195
- AA TAU-2 Bench: 95.03 (#43/575)
- AA IFBench: 64.22 (#142/587)
- AA CritPt: 4.29 (#157/674)
- AA Terminal-Bench Hard: 28.03 (#169/565)
- AA GPQA Diamond: 84.65 (#179/797)
- AA-Omniscience Accuracy: 24.83 (#195/521)
- AA Humanity's Last Exam: 22.85 (#196/798)
- AA Long Context Reasoning: 70.67 (#212/664)
- Artificial Analysis Intelligence Index: 20.82 (#225/830)
- AA Omniscience: -45.35 (#393/670)
Grok 3 — ELO 1697, #196
- AA MMLU-Pro: 79.87 (#118/349)
- AA AIME 2025: 58.0 (#121/270)
- AA LiveCodeBench: 42.54 (#170/343)
Mistral Medium 3.5 — ELO 1697, #197
- AA-Briefcase: 12.42 (#114/162)
Qwen 3 Next 80B A3B — ELO 1696, #199
- AA-Omniscience Accuracy: 17.25 (#332/521)
- AA Terminal-Bench Hard: 7.58 (#334/565)
- AA Long Context Reasoning: 52.67 (#346/664)
- AA GPQA Diamond: 73.84 (#356/797)
- AA IFBench: 39.66 (#357/587)
- AA Humanity's Last Exam: 7.65 (#411/798)
- AA TAU-2 Bench: 21.64 (#445/575)
- Artificial Analysis Intelligence Index: 9.64 (#449/830)
- AA Omniscience: -59.5 (#543/670)
- AA CritPt: 0.0 (#605/674)
Qwen 3 235B A22B — ELO 1695, #202
- AA AIME 2025: 82.0 (#61/270)
- AA MMLU-Pro: 82.8 (#64/349)
- AA LiveCodeBench: 62.22 (#110/343)
- AA-Omniscience Accuracy: 18.57 (#297/521)
- AA Humanity's Last Exam: 10.98 (#343/798)
- AA Terminal-Bench Hard: 6.06 (#369/565)
- AA IFBench: 38.71 (#376/587)
- AA TAU-2 Bench: 27.19 (#382/575)
- AA Omniscience: -44.7 (#389/670)
- AA GPQA Diamond: 70.0 (#403/797)
- ...and 3 more
Mercury 2 — ELO 1684, #206
- AA-Briefcase: 5.72 (#137/162)
- AA SciCode: 37.73 (#170/215)
Qwen 3 Coder 480B A35B — ELO 1680, #207
- AA Terminal-Bench Hard: 18.94 (#231/565)
- AA Omniscience: -22.2 (#247/670)
- AA TAU-2 Bench: 43.57 (#283/575)
- AA IFBench: 40.48 (#350/587)
- AA-Omniscience Accuracy: 15.68 (#371/521)
- AA Long Context Reasoning: 45.67 (#372/664)
- Artificial Analysis Intelligence Index: 11.9 (#381/830)
- AA GPQA Diamond: 61.82 (#493/797)
- AA CritPt: 0.0 (#598/674)
- AA Humanity's Last Exam: 4.54 (#611/798)
Claude 3.5 Sonnet (20241022) — ELO 1679, #208
- AA MMLU-Pro: 77.24 (#156/349)
- AA LiveCodeBench: 38.1 (#189/343)
GPT-OSS-120B — ELO 1679, #209
- LOL Arena: 45.6 (#31/42)
Hermes 4 405B — ELO 1677, #210
- LOL Arena: 44.6 (#37/42)
- AA-Omniscience Accuracy: 30.08 (#138/521)
- AA Terminal-Bench Hard: 11.36 (#300/565)
- AA Omniscience: -36.02 (#323/670)
- AA CritPt: 0.29 (#338/674)
- AA Humanity's Last Exam: 10.89 (#346/798)
- AA GPQA Diamond: 72.73 (#372/797)
- AA TAU-2 Bench: 22.22 (#436/575)
- AA IFBench: 32.72 (#464/587)
- AA Long Context Reasoning: 22.33 (#483/664)
- ...and 1 more
KAT-Coder-Pro V1 — ELO 1676, #211
- AA AIME 2025: 94.67 (#11/270)
- AA LiveCodeBench: 74.71 (#48/343)
- AA MMLU-Pro: 81.27 (#92/349)
- AA-Briefcase: 13.27 (#108/162)
Devstral 2 — ELO 1674, #212
- AA-Briefcase: 11.16 (#117/162)
- AA SciCode: 32.75 (#188/215)
Nemotron 3 Super — ELO 1673, #213
- AA APEX-Agents: 1.84 (#40/41)
- AA IFBench: 71.5 (#74/587)
- AA Terminal-Bench Hard: 28.79 (#167/565)
- AA CritPt: 3.14 (#173/674)
- AA SciCode: 36.23 (#177/215)
- Tau3 Banking: 10.31 (#180/260)
- AA-Omniscience Accuracy: 24.32 (#204/521)
- AA TAU-2 Bench: 67.84 (#214/575)
- AA Humanity's Last Exam: 20.76 (#215/798)
- AA GPQA Diamond: 80.0 (#254/797)
- ...and 3 more
Claude 3.5 Sonnet (20240620) — ELO 1670, #214
- AA MMLU-Pro: 75.11 (#178/349)
DeepSeek V3 (0324) — ELO 1667, #219
- AA MMLU-Pro: 81.91 (#80/349)
- AA-Briefcase: 3.3 (#154/162)
- AA AIME 2025: 41.0 (#158/270)
- AA SciCode: 39.0 (#159/215)
- AA LiveCodeBench: 40.53 (#178/343)
Claude 3.5 Sonnet — ELO 1665, #222
- AA GPQA Diamond: 59.9 (#511/797)
- Artificial Analysis Intelligence Index: 7.88 (#540/830)
- AA Humanity's Last Exam: 3.69 (#729/798)
GPT-4.1 Mini — ELO 1664, #223
- AgentCollabBench: 3.4 (#3/4)
- AA MMLU-Pro: 78.08 (#142/349)
- AA AIME 2025: 46.33 (#149/270)
- AA LiveCodeBench: 48.25 (#154/343)
Gemini 2.0 Flash — ELO 1663, #224
- AA MMLU-Pro: 79.78 (#120/349)
- AA LiveCodeBench: 32.06 (#211/343)
- AA-Omniscience Accuracy: 23.47 (#220/521)
- AA IFBench: 40.2 (#352/587)
- AA TAU-2 Bench: 29.53 (#356/575)
- AA Omniscience: -42.53 (#366/670)
- AA Terminal-Bench Hard: 3.79 (#427/565)
- AA Long Context Reasoning: 31.33 (#446/664)
- Artificial Analysis Intelligence Index: 8.94 (#486/830)
- AA GPQA Diamond: 62.32 (#487/797)
- ...and 2 more
GPT-4o (2024-08-06) — ELO 1663, #225
- AA LiveCodeBench: 31.75 (#214/343)
O1 Mini — ELO 1663, #226
- AA LiveCodeBench: 57.57 (#125/343)
- AA MMLU-Pro: 74.24 (#191/349)
Qwen 3 30B A3B 2507 — ELO 1662, #227
- AA-Briefcase: 4.08 (#150/162)
- AA SciCode: 32.99 (#186/215)
- Tau3 Banking: 5.36 (#230/260)
- AA IFBench: 50.68 (#234/587)
- AA Long Context Reasoning: 61.33 (#299/664)
- AA CritPt: 0.29 (#337/674)
- AA-Omniscience Accuracy: 16.3 (#359/521)
- AA Humanity's Last Exam: 10.29 (#361/798)
- AA TAU-2 Bench: 28.07 (#371/575)
- AA Terminal-Bench Hard: 6.06 (#371/565)
- ...and 3 more
Trinity Large — ELO 1662, #228
- AA-Briefcase: 5.76 (#136/162)
- AA SciCode: 40.62 (#147/215)
GLM-4.6V — ELO 1661, #230
- AA AIME 2025: 85.33 (#45/270)
- AA MMLU-Pro: 79.89 (#117/349)
- AA LiveCodeBench: 41.06 (#174/343)
- AA Terminal-Bench Hard: 14.39 (#273/565)
- AA Omniscience: -26.95 (#274/670)
- AA MMMU-Pro: 48.55 (#279/331)
- AA TAU-2 Bench: 31.58 (#342/575)
- AA Long Context Reasoning: 48.67 (#362/664)
- AA-Omniscience Accuracy: 16.18 (#363/521)
- AA Humanity's Last Exam: 9.64 (#378/798)
- ...and 4 more
Gemini 2.0 Pro (Preview 02-05) — ELO 1661, #231
- AA MMLU-Pro: 80.5 (#110/349)
- AA LiveCodeBench: 34.71 (#200/343)
Qwen 3 VL 32B — ELO 1660, #232
- AA MMMU-Pro: 64.28 (#200/331)
- AA Terminal-Bench Hard: 8.33 (#323/565)
- AA TAU-2 Bench: 29.24 (#360/575)
- AA IFBench: 39.18 (#366/587)
- AA-Omniscience Accuracy: 14.83 (#397/521)
- AA GPQA Diamond: 67.07 (#440/797)
- AA Humanity's Last Exam: 6.81 (#443/798)
- Artificial Analysis Intelligence Index: 8.39 (#517/830)
- AA Omniscience: -63.37 (#580/670)
- AA CritPt: 0.0 (#630/674)
Gemini 2.0 Flash (001) — ELO 1658, #233
- AA MMLU-Pro: 77.86 (#145/349)
- AA LiveCodeBench: 33.44 (#206/343)
- AA AIME 2025: 21.67 (#207/270)
Laguna M.1 — ELO 1652, #240
- Gert Labs Rankings: 20.79 (#98/100)
GPT-4o — ELO 1651, #241
- LOL Arena: 48.2 (#15/42)
- Icelandic LLM Leaderboard - Average: 41.12 (#16/23)
Qwen 3.5 9B — ELO 1648, #246
- AA IFBench: 66.73 (#116/587)
- AA TAU-2 Bench: 86.84 (#119/575)
- AA MMMU-Pro: 69.25 (#163/331)
- AA Terminal-Bench Hard: 24.24 (#201/565)
- Tau3 Banking: 7.01 (#205/260)
- AA Long Context Reasoning: 70.0 (#227/664)
- AA GPQA Diamond: 80.61 (#249/797)
- AA Humanity's Last Exam: 14.92 (#272/798)
- AA CritPt: 0.29 (#334/674)
- Artificial Analysis Intelligence Index: 13.65 (#338/830)
- ...and 2 more
Mistral Medium 3.1 — ELO 1647, #249
- AA-Briefcase: 13.03 (#110/162)
- AA SciCode: 32.41 (#189/215)
GPT-4o (2024-11-20) — ELO 1645, #250
- AA MMLU-Pro: 74.77 (#184/349)
- AA LiveCodeBench: 30.9 (#217/343)
- AA AIME 2025: 6.0 (#246/270)
Gemini 2.0 Flash (Preview) — ELO 1641, #253
- AA MMLU-Pro: 78.16 (#141/349)
- AA LiveCodeBench: 20.95 (#267/343)
Gemini 2.5 Flash Lite — ELO 1639, #256
- Icelandic LLM Leaderboard - Average: 26.15 (#19/23)
- AA LiveCodeBench: 59.26 (#119/343)
- AA AIME 2025: 53.33 (#136/270)
- AA MMLU-Pro: 75.91 (#172/349)
- AA IFBench: 49.86 (#241/587)
- AA MMMU-Pro: 58.21 (#245/331)
- AA-Omniscience Accuracy: 17.9 (#315/521)
- AA Long Context Reasoning: 55.67 (#324/664)
- AA Omniscience: -45.62 (#397/670)
- AA Terminal-Bench Hard: 4.55 (#405/565)
- ...and 5 more
Qwen 3 VL 30B A3B — ELO 1638, #258
- AA MMMU-Pro: 62.14 (#217/331)
- AA Terminal-Bench Hard: 6.06 (#370/565)
- AA-Omniscience Accuracy: 15.62 (#374/521)
- AA GPQA Diamond: 69.49 (#414/797)
- AA IFBench: 33.13 (#455/587)
- AA Humanity's Last Exam: 6.3 (#468/798)
- AA TAU-2 Bench: 19.01 (#477/575)
- Artificial Analysis Intelligence Index: 7.93 (#532/830)
- AA Omniscience: -63.42 (#582/670)
- AA CritPt: 0.0 (#622/674)
Mistral Large 3 — ELO 1637, #259
- AA-Briefcase: 5.62 (#139/162)
GLM-4.5V — ELO 1632, #265
- AA-Omniscience Accuracy: 20.83 (#253/521)
- AA MMMU-Pro: 50.46 (#274/331)
- AA Terminal-Bench Hard: 5.3 (#382/565)
- AA Omniscience: -46.28 (#401/670)
- AA GPQA Diamond: 68.38 (#425/797)
- AA TAU-2 Bench: 22.51 (#431/575)
- AA IFBench: 34.22 (#439/587)
- AA Humanity's Last Exam: 6.3 (#467/798)
- Artificial Analysis Intelligence Index: 7.56 (#564/830)
- AA CritPt: 0.0 (#610/674)
- ...and 1 more
North Mini Code — ELO 1630, #268
- AA-Briefcase: 6.23 (#134/162)
- AA SciCode: 38.77 (#161/215)
Llama 3.1 405B — ELO 1628, #269
- AA Omniscience: -17.1 (#218/670)
- AA-Omniscience Accuracy: 23.18 (#223/521)
- AA Terminal-Bench Hard: 6.82 (#354/565)
- AA IFBench: 39.05 (#369/587)
- AA Long Context Reasoning: 25.33 (#468/664)
- AA TAU-2 Bench: 19.01 (#476/575)
- AA GPQA Diamond: 51.52 (#587/797)
- Artificial Analysis Intelligence Index: 7.29 (#595/830)
- AA CritPt: 0.0 (#619/674)
- AA Humanity's Last Exam: 3.98 (#695/798)
K-EXAONE — ELO 1628, #272
- AA IFBench: 64.69 (#133/587)
- AA TAU-2 Bench: 74.27 (#187/575)
- AA Terminal-Bench Hard: 22.73 (#209/565)
- AA CritPt: 1.12 (#245/674)
- AA GPQA Diamond: 78.28 (#282/797)
- AA Humanity's Last Exam: 13.95 (#286/798)
- AA Long Context Reasoning: 61.33 (#300/664)
- Artificial Analysis Intelligence Index: 14.37 (#323/830)
- AA-Omniscience Accuracy: 16.35 (#357/521)
- AA Omniscience: -57.97 (#526/670)
GPT-4 Turbo — ELO 1626, #273
- AA MMLU-Pro: 69.37 (#229/349)
- AA LiveCodeBench: 29.1 (#229/343)
Grok 2 — ELO 1625, #274
- AA GPQA Diamond: 51.01 (#595/797)
- Artificial Analysis Intelligence Index: 7.11 (#613/830)
- AA Humanity's Last Exam: 3.06 (#777/798)
Qwen 3 32B — ELO 1622, #280
- AA AIME 2025: 73.0 (#89/270)
- AA MMLU-Pro: 79.85 (#119/349)
- AA LiveCodeBench: 54.6 (#133/343)
- AA-Briefcase: 3.94 (#151/162)
- AA SciCode: 36.0 (#181/215)
- Tau3 Banking: 5.36 (#231/260)
- AA-Omniscience Accuracy: 17.43 (#329/521)
- AA CritPt: 0.29 (#341/674)
- AA TAU-2 Bench: 29.82 (#353/575)
- AA IFBench: 36.33 (#416/587)
- ...and 6 more
Llama 4 Maverick — ELO 1620, #284
- LOL Arena: 44.8 (#35/42)
- AA-Briefcase: 0.91 (#160/162)
- AA SciCode: 31.71 (#192/215)
Qwen 2.5 Max — ELO 1615, #292
- AA MMLU-Pro: 76.24 (#164/349)
- AA LiveCodeBench: 35.87 (#195/343)
Gemma 4 12B — ELO 1611, #300
- AA IFBench: 73.54 (#53/587)
- AA MMMU-Pro: 69.65 (#159/331)
- AA Terminal-Bench Hard: 18.18 (#241/565)
- AA Humanity's Last Exam: 15.66 (#267/798)
- AA Long Context Reasoning: 63.67 (#287/664)
- AA TAU-2 Bench: 36.26 (#311/575)
- Artificial Analysis Intelligence Index: 14.18 (#327/830)
- AA GPQA Diamond: 75.25 (#335/797)
- AA-Omniscience Accuracy: 15.62 (#375/521)
- AA Omniscience: -52.72 (#474/670)
- ...and 1 more
Pixtral Large — ELO 1611, #301
- AA MMLU-Pro: 70.11 (#221/349)
- AA LiveCodeBench: 26.14 (#249/343)
- AA AIME 2025: 2.33 (#259/270)
Qwen 2.5 72B Instruct — ELO 1610, #303
- AA MMLU-Pro: 72.03 (#208/349)
- AA AIME 2025: 14.0 (#222/270)
- AA LiveCodeBench: 27.62 (#239/343)
Gemma 4 26B A4B — ELO 1610, #304
- AA IFBench: 72.45 (#65/587)
- AA MMMU-Pro: 69.25 (#164/331)
- Tau3 Banking: 11.96 (#171/260)
- AA Humanity's Last Exam: 19.32 (#229/798)
- AA Long Context Reasoning: 65.67 (#265/664)
- AA GPQA Diamond: 79.19 (#268/797)
- AA Terminal-Bench Hard: 13.64 (#280/565)
- Artificial Analysis Intelligence Index: 16.67 (#282/830)
- AA-Omniscience Accuracy: 19.1 (#282/521)
- AA TAU-2 Bench: 43.57 (#284/575)
- ...and 2 more
GPT-4 — ELO 1610, #305
- AA MMLU-Pro: 56.21 (#286/349)
GLM-4.7 Flash — ELO 1609, #307
- AA TAU-2 Bench: 98.83 (#4/575)
- AA IFBench: 60.82 (#161/587)
- AA Terminal-Bench Hard: 21.97 (#214/565)
- Artificial Analysis Intelligence Index: 14.87 (#314/830)
- AA CritPt: 0.29 (#333/674)
- AA-Omniscience Accuracy: 16.15 (#364/521)
- AA Long Context Reasoning: 41.67 (#393/664)
- AA Humanity's Last Exam: 7.6 (#414/798)
- AA GPQA Diamond: 58.08 (#527/797)
- AA Omniscience: -62.62 (#573/670)
Nemotron 3.5 Lightning — ELO 1609, #308
- OpenRouter Tau2-Bench Airline: 62.9 (#80/122)
- AA-Briefcase: 8.89 (#123/162)
DeepSeek V2.5 — ELO 1609, #309
- AA MMLU-Pro: 66.56 (#248/349)
Qwen 2.5 72B — ELO 1608, #310
- AA TAU-2 Bench: 34.5 (#322/575)
- AA-Omniscience Accuracy: 17.47 (#327/521)
- AA IFBench: 36.87 (#403/587)
- AA Terminal-Bench Hard: 4.55 (#404/565)
- AA Omniscience: -53.07 (#478/670)
- Artificial Analysis Intelligence Index: 7.73 (#551/830)
- AA GPQA Diamond: 49.09 (#605/797)
- AA CritPt: 0.0 (#654/674)
- AA Humanity's Last Exam: 3.56 (#750/798)
Qwen 3 30B A3B — ELO 1607, #312
- AA AIME 2025: 72.33 (#92/270)
- AA MMLU-Pro: 77.68 (#148/349)
- AA LiveCodeBench: 50.58 (#149/343)
- AA IFBench: 41.5 (#336/587)
- AA Terminal-Bench Hard: 6.82 (#356/565)
- AA-Omniscience Accuracy: 16.18 (#362/521)
- AA TAU-2 Bench: 26.02 (#397/575)
- AA Omniscience: -51.48 (#458/670)
- AA Humanity's Last Exam: 6.16 (#480/798)
- AA GPQA Diamond: 61.62 (#495/797)
- ...and 3 more
Mistral Small 4 — ELO 1606, #315
- AA-Briefcase: 5.66 (#138/162)
- AA SciCode: 38.77 (#162/215)
- Tau3 Banking: 4.95 (#235/260)
- AA-Omniscience Accuracy: 21.68 (#243/521)
- AA Terminal-Bench Hard: 17.42 (#249/565)
- AA MMMU-Pro: 56.82 (#249/331)
- AA IFBench: 48.16 (#256/587)
- AA TAU-2 Bench: 41.23 (#292/575)
- AA Omniscience: -30.4 (#293/670)
- AA GPQA Diamond: 76.87 (#306/797)
- ...and 4 more
Devstral Small 2 — ELO 1606, #316
- AA-Briefcase: 10.2 (#121/162)
- AA SciCode: 32.41 (#190/215)
Magistral Medium — ELO 1604, #317
- AA LiveCodeBench: 52.7 (#140/343)
- AA AIME 2025: 40.33 (#159/270)
- AA MMLU-Pro: 75.33 (#174/349)
Qwen3 Omni 30B A3B Instruct — ELO 1604, #318
- BenchLM: 38.3 (#183/200)
Gemma 3 27B (IT) — ELO 1601, #321
- AA SciCode: 23.26 (#207/215)
Mistral Large 2 (Jul) — ELO 1601, #324
- AA MMLU-Pro: 68.26 (#237/349)
- AA LiveCodeBench: 26.67 (#246/343)
- AA AIME 2025: 0.0 (#267/270)
- AA Long Context Reasoning: 2.0 (#577/664)
Mistral Large 2 (Nov) Instruct (2411) — ELO 1600, #326
- AA AIME 2025: 14.0 (#223/270)
- AA MMLU-Pro: 69.7 (#224/349)
- AA LiveCodeBench: 29.31 (#226/343)
Qwen 3 VL 8B — ELO 1600, #327
- AA-Omniscience Accuracy: 20.48 (#259/521)
- AA MMMU-Pro: 47.34 (#288/331)
- AA TAU-2 Bench: 29.24 (#359/575)
- AA Terminal-Bench Hard: 2.27 (#462/565)
- AA Omniscience: -51.88 (#463/670)
- AA IFBench: 32.31 (#471/587)
- AA Long Context Reasoning: 16.67 (#515/664)
- Artificial Analysis Intelligence Index: 7.26 (#599/830)
- AA CritPt: 0.0 (#613/674)
- AA GPQA Diamond: 42.73 (#647/797)
- ...and 1 more
Gemini 2.0 Flash Lite (Preview 02-05) — ELO 1599, #331
- AA MMLU-Pro: 72.37 (#207/349)
- AA LiveCodeBench: 18.52 (#275/343)
QwQ-32B — ELO 1598, #332
- AA LiveCodeBench: 63.07 (#106/343)
- AA MMLU-Pro: 76.38 (#162/349)
- AA AIME 2025: 29.0 (#190/270)
GPT-OSS-20B — ELO 1596, #333
- AA-Briefcase: 2.42 (#158/162)
- AA SciCode: 38.89 (#160/215)
Mistral Medium 3 — ELO 1596, #335
- AA MMLU-Pro: 75.97 (#169/349)
- AA LiveCodeBench: 40.0 (#182/343)
- AA AIME 2025: 30.33 (#185/270)
Qwen 2.5 32B Instruct — ELO 1595, #337
- AA MMLU-Pro: 69.66 (#225/349)
- AA LiveCodeBench: 24.76 (#254/343)
Qwen 2 72B — ELO 1594, #339
- RULER: 85.9 (#25/45)
- Artificial Analysis Intelligence Index: 6.33 (#680/830)
- AA GPQA Diamond: 37.07 (#689/797)
- AA Humanity's Last Exam: 3.66 (#735/798)
Hermes 4 70B — ELO 1589, #343
- AA-Omniscience Accuracy: 23.55 (#217/521)
- AA Humanity's Last Exam: 8.76 (#392/798)
- AA Terminal-Bench Hard: 4.55 (#400/565)
- AA GPQA Diamond: 69.9 (#406/797)
- AA TAU-2 Bench: 22.51 (#432/575)
- AA Omniscience: -49.37 (#434/670)
- AA IFBench: 31.29 (#494/587)
- Artificial Analysis Intelligence Index: 7.91 (#537/830)
- AA Long Context Reasoning: 9.67 (#548/664)
- AA CritPt: 0.0 (#629/674)
Qwen 3 14B — ELO 1584, #350
- LLM Stats (ZebraLogic): 88.5 (#7/9)
- ZeroEval MATH-500: 96.8 (#12/33)
- LLM Stats (Multi-IF): 74.8 (#13/25)
- LLM Stats (C-Eval): 86.2 (#14/20)
- LLM Stats (WritingBench): 78.0 (#15/16)
- LLM Stats (MMLU-Redux): 88.6 (#31/50)
- LLM Stats (AIME 2024): 79.3 (#32/54)
- AA AIME 2025: 58.0 (#122/270)
- AA LiveCodeBench: 52.28 (#142/343)
- AA MMLU-Pro: 77.38 (#152/349)
- ...and 14 more
Mistral Small 3.2 — ELO 1584, #351
- AA-Briefcase: 3.46 (#153/162)
- AA SciCode: 28.59 (#198/215)
Claude 3 Opus — ELO 1582, #357
- AA MMLU-Pro: 69.57 (#226/349)
- AA LiveCodeBench: 27.94 (#237/343)
Llama 3.3 70B — ELO 1577, #364
- AA IFBench: 47.07 (#266/587)
- AA-Omniscience Accuracy: 18.95 (#287/521)
- AA TAU-2 Bench: 26.61 (#391/575)
- AA Terminal-Bench Hard: 3.03 (#437/565)
- AA Omniscience: -54.17 (#491/670)
- AA Long Context Reasoning: 15.67 (#522/664)
- Artificial Analysis Intelligence Index: 7.66 (#555/830)
- AA GPQA Diamond: 49.8 (#600/797)
- AA CritPt: 0.0 (#644/674)
- AA Humanity's Last Exam: 3.56 (#749/798)
Llama 3.1 70B Instruct — ELO 1575, #367
- AA MMLU-Pro: 67.61 (#241/349)
- AA AIME 2025: 4.0 (#252/270)
- AA LiveCodeBench: 23.17 (#262/343)
Qwen 3 4B 2507 — ELO 1573, #370
- AA IFBench: 49.8 (#243/587)
- AA TAU-2 Bench: 26.61 (#392/575)
- AA Terminal-Bench Hard: 4.55 (#403/565)
- AA Long Context Reasoning: 37.33 (#414/664)
- AA-Omniscience Accuracy: 12.53 (#436/521)
- AA GPQA Diamond: 66.67 (#446/797)
- AA Humanity's Last Exam: 6.16 (#479/798)
- Artificial Analysis Intelligence Index: 8.8 (#495/830)
- AA Omniscience: -60.27 (#552/670)
- AA CritPt: 0.0 (#606/674)
Qwen 2 72B Instruct — ELO 1572, #371
- AA MMLU-Pro: 62.18 (#267/349)
- AA LiveCodeBench: 15.87 (#287/343)
Llama 3.1 70B — ELO 1570, #374
- RULER: 89.6 (#13/45)
- LOL Arena: 42.5 (#41/42)
- AA-Omniscience Accuracy: 19.7 (#275/521)
- AA Omniscience: -43.1 (#375/670)
- AA IFBench: 34.42 (#434/587)
- AA Terminal-Bench Hard: 3.03 (#436/565)
- AA TAU-2 Bench: 15.2 (#500/575)
- AA CritPt: 0.0 (#620/674)
- AA Humanity's Last Exam: 4.47 (#624/798)
- Artificial Analysis Intelligence Index: 6.6 (#652/830)
- ...and 1 more
Qwen 2.5 Coder 32B Instruct — ELO 1570, #376
- AA LiveCodeBench: 29.52 (#224/343)
- AA MMLU-Pro: 63.47 (#260/349)
Mistral Small 3.1 — ELO 1569, #378
- AA-Briefcase: 6.06 (#135/162)
- AA SciCode: 27.78 (#201/215)
- AA MMLU-Pro: 65.93 (#250/349)
- AA AIME 2025: 3.67 (#253/270)
- AA LiveCodeBench: 21.16 (#266/343)
Command A+ — ELO 1569, #379
- AA-Briefcase: 6.92 (#131/162)
Mistral Small 3 — ELO 1565, #385
- AA AIME 2025: 4.33 (#251/270)
- AA LiveCodeBench: 25.19 (#252/343)
- AA MMLU-Pro: 65.16 (#254/349)
Claude 3.5 Haiku — ELO 1565, #386
- AA LiveCodeBench: 31.43 (#216/343)
- AA MMLU-Pro: 63.45 (#261/349)
Qwen 3 VL 4B — ELO 1565, #387
- AA MMMU-Pro: 43.87 (#298/331)
- AA TAU-2 Bench: 23.39 (#421/575)
- AA-Omniscience Accuracy: 10.87 (#456/521)
- AA IFBench: 31.84 (#483/587)
- AA Long Context Reasoning: 14.0 (#530/664)
- AA Terminal-Bench Hard: 0.0 (#565/565)
- AA Omniscience: -75.97 (#636/670)
- AA CritPt: 0.0 (#672/674)
- AA GPQA Diamond: 37.07 (#690/797)
- Artificial Analysis Intelligence Index: 5.66 (#726/830)
- ...and 1 more
Gemma 3 27B — ELO 1564, #388
- AA SciCode: 23.26 (#208/215)
- Tau3 Banking: 0.82 (#255/260)
- AA MMMU-Pro: 48.03 (#285/331)
- AA Terminal-Bench Hard: 3.79 (#422/565)
- AA-Omniscience Accuracy: 12.95 (#433/521)
- AA IFBench: 31.84 (#481/587)
- AA TAU-2 Bench: 10.53 (#533/575)
- AA Long Context Reasoning: 7.33 (#559/664)
- AA CritPt: 0.0 (#607/674)
- AA Omniscience: -67.22 (#607/670)
- ...and 3 more
Qwen 3.5 4B — ELO 1562, #390
- AA TAU-2 Bench: 92.11 (#82/575)
- AA MMMU-Pro: 65.38 (#189/331)
- Tau3 Banking: 6.8 (#207/260)
- AA IFBench: 51.97 (#226/587)
- AA Terminal-Bench Hard: 18.18 (#240/565)
- AA Long Context Reasoning: 63.0 (#291/664)
- AA GPQA Diamond: 77.07 (#303/797)
- Artificial Analysis Intelligence Index: 13.12 (#354/830)
- AA Humanity's Last Exam: 9.92 (#367/798)
- AA-Omniscience Accuracy: 15.12 (#388/521)
- ...and 2 more
Ministral 3 14B — ELO 1556, #395
- AA-Briefcase: 4.9 (#142/162)
- AA SciCode: 23.84 (#206/215)
Mixtral 8x22B Instruct — ELO 1554, #400
- AA MMLU-Pro: 53.67 (#290/349)
- AA LiveCodeBench: 14.81 (#291/343)
Gemini 1.5 Flash — ELO 1551, #404
- AA LiveCodeBench: 19.58 (#271/343)
- AA MMLU-Pro: 57.41 (#280/349)
GPT-4o Mini — ELO 1550, #405
- AA AIME 2025: 14.67 (#220/270)
- AA MMLU-Pro: 64.78 (#257/349)
- AA LiveCodeBench: 23.39 (#258/343)
Granite 4.2 8B — ELO 1549, #410
- AI for Education Pedagogy - Science: 74.86 (#168/249)
- AI for Education Pedagogy - Maths: 70.63 (#172/249)
- AI for Education SEND: 67.43 (#176/241)
- AI for Education Pedagogy - Secondary: 70.91 (#177/249)
- AI for Education Pedagogy: 71.08 (#181/249)
- AI for Education Pedagogy - Primary: 74.65 (#183/249)
- AI for Education Pedagogy - Social studies: 64.55 (#193/249)
- AI for Education Pedagogy - Technology: 69.81 (#198/249)
Jamba 1.5 Large — ELO 1547, #414
- AA MMLU-Pro: 57.24 (#281/349)
- AA LiveCodeBench: 14.29 (#293/343)
Llama 3 70B Instruct — ELO 1544, #418
- AA LiveCodeBench: 19.79 (#270/343)
- AA MMLU-Pro: 57.45 (#279/349)
Llama 3 70B — ELO 1542, #421
- RULER: 86.5 (#23/45)
- AA-Omniscience Accuracy: 17.72 (#319/521)
- AA IFBench: 37.07 (#398/587)
- AA Omniscience: -54.32 (#493/670)
- AA Terminal-Bench Hard: 0.76 (#503/565)
- AA TAU-2 Bench: 0.0 (#574/575)
- AA Humanity's Last Exam: 4.52 (#615/798)
- AA CritPt: 0.0 (#651/674)
- AA Long Context Reasoning: 0.0 (#657/664)
- AA GPQA Diamond: 37.88 (#683/797)
- ...and 1 more
Gemma 3 12B (IT) — ELO 1540, #423
- AA SciCode: 16.44 (#212/215)
Apertus-70B-Instruct-2509 — ELO 1538, #425
- Last Translation Benchmark: 1.21 (#34/42)
Ministral 3 8B — ELO 1537, #426
- AA-Briefcase: 4.56 (#146/162)
- AA SciCode: 20.72 (#211/215)
Llama 4 Scout — ELO 1533, #434
- AA-Briefcase: 0.72 (#161/162)
- AA SciCode: 21.3 (#210/215)
Mistral Large — ELO 1531, #436
- AA LiveCodeBench: 17.78 (#279/343)
- AA MMLU-Pro: 51.55 (#295/349)
nemotron-3-nano-30B-a3B — ELO 1529, #443
- AA-Briefcase: 2.95 (#156/162)
- AA SciCode: 30.56 (#196/215)
DeepSeek R1 Distill Qwen 32B — ELO 1525, #447
- AA AIME 2025: 63.0 (#112/270)
- AA MMLU-Pro: 73.94 (#195/349)
- AA LiveCodeBench: 26.98 (#243/343)
Qwen 3 8B — ELO 1517, #453
- AA-Briefcase: 3.23 (#155/162)
- AA LiveCodeBench: 40.63 (#177/343)
- AA MMLU-Pro: 74.3 (#190/349)
- AA SciCode: 27.89 (#199/215)
- AA AIME 2025: 24.33 (#200/270)
- Tau3 Banking: 4.74 (#241/260)
- AA TAU-2 Bench: 27.78 (#379/575)
- AA-Omniscience Accuracy: 13.63 (#420/521)
- AA IFBench: 33.47 (#450/587)
- AA Terminal-Bench Hard: 2.27 (#465/565)
- ...and 6 more
Gemma 3 12B — ELO 1512, #460
- AA SciCode: 16.44 (#213/215)
- Tau3 Banking: 0.82 (#256/260)
- AA MMMU-Pro: 37.51 (#314/331)
- AA IFBench: 36.73 (#406/587)
- AA-Omniscience Accuracy: 10.3 (#460/521)
- AA Terminal-Bench Hard: 0.76 (#499/565)
- AA TAU-2 Bench: 10.82 (#528/575)
- AA Long Context Reasoning: 8.33 (#555/664)
- AA CritPt: 0.0 (#609/674)
- AA Omniscience: -77.07 (#641/670)
- ...and 3 more
EXAONE 4.0 32B — ELO 1512, #461
- AA Humanity's Last Exam: 11.45 (#326/798)
- AA GPQA Diamond: 73.94 (#352/797)
- AA IFBench: 36.33 (#415/587)
- AA-Omniscience Accuracy: 13.87 (#415/521)
- AA Terminal-Bench Hard: 3.79 (#423/565)
- AA TAU-2 Bench: 17.25 (#486/575)
- Artificial Analysis Intelligence Index: 8.18 (#526/830)
- AA Omniscience: -60.07 (#547/670)
- AA CritPt: 0.0 (#617/674)
QwQ 32B-Preview — ELO 1511, #462
- AA LiveCodeBench: 33.65 (#205/343)
- AA MMLU-Pro: 64.79 (#256/349)
Hermes 3 - Llama-3.1 70B — ELO 1511, #463
- AA LiveCodeBench: 18.84 (#274/343)
- AA MMLU-Pro: 57.07 (#282/349)
Claude 2 — ELO 1511, #465
- AA LiveCodeBench: 17.14 (#282/343)
- AA MMLU-Pro: 48.57 (#302/349)
Qwen 1.5 72B — ELO 1509, #469
- RULER: 55.7 (#42/45)
GPT-4.1 Nano — ELO 1505, #472
- Icelandic LLM Leaderboard - Average: 17.59 (#21/23)
- AA AIME 2025: 24.0 (#201/270)
- AA LiveCodeBench: 32.59 (#209/343)
- AA MMLU-Pro: 65.68 (#252/349)
- AA Long Context Reasoning: 20.33 (#496/664)
Mistral Medium — ELO 1501, #478
- AA MMLU-Pro: 49.07 (#300/349)
- AA LiveCodeBench: 9.95 (#315/343)
Solar Pro 3 — ELO 1500, #482
- AA-Briefcase: 4.9 (#141/162)
- AA SciCode: 25.46 (#203/215)
Mixtral 8x22B — ELO 1499, #483
- AA Humanity's Last Exam: 4.01 (#690/798)
- AA GPQA Diamond: 33.23 (#711/797)
- Artificial Analysis Intelligence Index: 5.74 (#719/830)
Claude 3 Sonnet — ELO 1498, #485
- AA MMLU-Pro: 57.92 (#275/349)
- AA LiveCodeBench: 17.46 (#280/343)
NVIDIA Nemotron Nano 9B V2 — ELO 1494, #493
- AA Omniscience: -41.55 (#356/670)
- AA TAU-2 Bench: 21.93 (#440/575)
- AA-Omniscience Accuracy: 11.8 (#445/521)
- AA Long Context Reasoning: 22.67 (#480/664)
- AA Terminal-Bench Hard: 1.52 (#482/565)
- AA IFBench: 27.62 (#524/587)
- AA GPQA Diamond: 56.97 (#542/797)
- AA Humanity's Last Exam: 4.87 (#568/798)
- Artificial Analysis Intelligence Index: 7.43 (#579/830)
- AA CritPt: 0.0 (#669/674)
Qwen 3 4B — ELO 1486, #504
- AA LiveCodeBench: 46.46 (#163/343)
- AA AIME 2025: 22.33 (#204/270)
- AA MMLU-Pro: 69.56 (#228/349)
- AA IFBench: 32.52 (#469/587)
- AA TAU-2 Bench: 19.01 (#475/575)
- AA GPQA Diamond: 52.22 (#573/797)
- Artificial Analysis Intelligence Index: 7.23 (#601/830)
- AA Humanity's Last Exam: 4.42 (#630/798)
- AA Long Context Reasoning: 0.0 (#643/664)
Gemini 1.5 Flash-8B — ELO 1485, #506
- AA LiveCodeBench: 21.69 (#264/343)
- AA MMLU-Pro: 56.91 (#284/349)
Claude 3 Haiku — ELO 1480, #510
- AA Global-MMLU-Lite - Yoruba: 42.17 (#83/111)
- AA Global-MMLU-Lite - Swahili: 51.08 (#96/111)
- AA Global-MMLU-Lite - Burmese: 48.75 (#97/111)
- AA Global-MMLU-Lite: 63.31 (#103/111)
- AA Global-MMLU-Lite - Indonesian: 67.92 (#103/111)
- AA Global-MMLU-Lite - Arabic: 64.33 (#104/111)
- AA Global-MMLU-Lite - German: 72.58 (#104/111)
- AA Global-MMLU-Lite - English: 77.92 (#104/111)
- AA Global-MMLU-Lite - Hindi: 56.92 (#104/111)
- AA Global-MMLU-Lite - Japanese: 65.25 (#104/111)
- ...and 8 more
Claude 2.1 — ELO 1475, #518
- AA LiveCodeBench: 19.47 (#273/343)
- AA MMLU-Pro: 49.48 (#299/349)
GPT-3.5 Turbo — ELO 1470, #520
- Icelandic LLM Leaderboard - Average: 12.9 (#23/23)
- AA MMLU-Pro: 46.19 (#310/349)
Magistral Small — ELO 1469, #521
- AA LiveCodeBench: 51.43 (#146/343)
- AA AIME 2025: 41.33 (#154/270)
- AA MMLU-Pro: 74.63 (#185/349)
Jamba 1.5 Mini — ELO 1467, #523
- AA MMLU-Pro: 37.05 (#329/349)
- AA LiveCodeBench: 6.24 (#330/343)
Ministral 3 3B — ELO 1462, #530
- AA-Briefcase: 3.66 (#152/162)
- AA SciCode: 15.28 (#214/215)
Gemini 1.0 Pro — ELO 1461, #532
- AA LiveCodeBench: 11.64 (#308/343)
- AA MMLU-Pro: 43.09 (#315/349)
Gemma 3n E4B — ELO 1457, #535
- Tau3 Banking: 0.21 (#260/260)
- AA MMMU-Pro: 26.24 (#325/331)
- AA Terminal-Bench Hard: 2.27 (#466/565)
- AA-Omniscience Accuracy: 7.98 (#489/521)
- AA IFBench: 27.89 (#522/587)
- AA TAU-2 Bench: 4.97 (#544/575)
- AA Humanity's Last Exam: 4.49 (#621/798)
- AA CritPt: 0.0 (#649/674)
- AA Long Context Reasoning: 0.0 (#655/664)
- AA Omniscience: -81.33 (#659/670)
- ...and 2 more
Apertus-8B-Instruct-2509 — ELO 1450, #543
- Last Translation Benchmark: 0.77 (#36/42)
DeepSeek R1 Distill Qwen 14B — ELO 1449, #547
- AA AIME 2025: 55.67 (#131/270)
- AA LiveCodeBench: 37.57 (#192/343)
- AA MMLU-Pro: 74.0 (#193/349)
Qwen 2.5 Coder 7B Instruct — ELO 1444, #554
- AA LiveCodeBench: 12.59 (#302/343)
- AA MMLU-Pro: 47.33 (#306/349)
DBRX — ELO 1443, #558
- AA GPQA Diamond: 33.13 (#714/797)
- Artificial Analysis Intelligence Index: 5.29 (#767/830)
- AA Humanity's Last Exam: 2.92 (#780/798)
OLMo 3 7B — ELO 1435, #569
- AA IFBench: 32.79 (#461/587)
- AA-Omniscience Accuracy: 7.4 (#492/521)
- AA Humanity's Last Exam: 5.79 (#494/798)
- AA TAU-2 Bench: 12.57 (#517/575)
- AA Terminal-Bench Hard: 0.0 (#559/565)
- AA Omniscience: -77.07 (#642/670)
- AA CritPt: 0.0 (#643/674)
- AA Long Context Reasoning: 0.0 (#653/664)
- AA GPQA Diamond: 40.0 (#672/797)
- Artificial Analysis Intelligence Index: 5.24 (#772/830)
Llama 3.1 8B Instruct — ELO 1434, #570
- AA AIME 2025: 4.33 (#250/270)
- AA MMLU-Pro: 47.63 (#305/349)
- AA LiveCodeBench: 11.64 (#306/343)
Yi 34B (Chat) — ELO 1434, #571
- RULER: 87.5 (#21/45)
Llama 3 8B Instruct — ELO 1413, #597
- AA LiveCodeBench: 9.63 (#319/343)
- AA MMLU-Pro: 40.48 (#323/349)
Mixtral 8x7B — ELO 1413, #599
- AA Humanity's Last Exam: 4.68 (#588/798)
- AA GPQA Diamond: 29.19 (#748/797)
- Artificial Analysis Intelligence Index: 5.12 (#787/830)
Mixtral 8x7B Instruct — ELO 1404, #607
- AA MMLU-Pro: 38.69 (#326/349)
- AA LiveCodeBench: 6.56 (#329/343)
Llama 3.1 8B — ELO 1399, #612
- RULER: 88.3 (#18/45)
- AA Omniscience: -30.88 (#297/670)
- AA-Omniscience Accuracy: 8.5 (#483/521)
- AA TAU-2 Bench: 16.37 (#491/575)
- AA Terminal-Bench Hard: 0.76 (#501/565)
- AA Long Context Reasoning: 18.0 (#510/664)
- AA IFBench: 28.57 (#516/587)
- AA Humanity's Last Exam: 5.29 (#526/798)
- Artificial Analysis Intelligence Index: 6.93 (#628/830)
- AA CritPt: 0.0 (#635/674)
- ...and 1 more
Qwen 3.5 2B — ELO 1398, #614
- AA TAU-2 Bench: 69.01 (#208/575)
- AA MMMU-Pro: 43.01 (#301/331)
- AA Terminal-Bench Hard: 3.79 (#425/565)
- AA-Omniscience Accuracy: 8.07 (#488/521)
- AA IFBench: 31.5 (#491/587)
- AA Long Context Reasoning: 21.0 (#493/664)
- AA Omniscience: -60.18 (#550/670)
- Artificial Analysis Intelligence Index: 6.94 (#625/830)
- AA CritPt: 0.0 (#625/674)
- AA GPQA Diamond: 45.56 (#627/797)
- ...and 1 more
Gemma 3 4B — ELO 1392, #618
- Tau3 Banking: 0.41 (#258/260)
- AA MMMU-Pro: 29.94 (#320/331)
- AA-Omniscience Accuracy: 7.73 (#490/521)
- AA Terminal-Bench Hard: 0.76 (#498/565)
- AA IFBench: 28.3 (#518/587)
- AA Humanity's Last Exam: 5.29 (#525/798)
- AA TAU-2 Bench: 4.97 (#543/575)
- AA Long Context Reasoning: 6.67 (#565/664)
- AA CritPt: 0.0 (#603/674)
- AA Omniscience: -82.88 (#664/670)
- ...and 2 more
Llama 3 8B — ELO 1392, #619
- RULER: 82.4 (#30/45)
- AA-Omniscience Accuracy: 10.37 (#459/521)
- AA Humanity's Last Exam: 5.06 (#543/798)
- AA IFBench: 24.56 (#547/587)
- AA Terminal-Bench Hard: 0.0 (#558/565)
- AA TAU-2 Bench: 0.0 (#572/575)
- AA Omniscience: -70.12 (#620/670)
- AA CritPt: 0.0 (#639/674)
- AA Long Context Reasoning: 0.0 (#652/664)
- AA GPQA Diamond: 29.6 (#745/797)
- ...and 1 more
Gemma 4 E4B — ELO 1384, #626
- AA Omniscience: -19.7 (#236/670)
- AA MMMU-Pro: 51.39 (#270/331)
- AA CritPt: 0.57 (#288/674)
- AA IFBench: 44.22 (#295/587)
- AA Terminal-Bench Hard: 8.33 (#325/565)
- AA Long Context Reasoning: 32.0 (#441/664)
- AA TAU-2 Bench: 20.76 (#460/575)
- AA-Omniscience Accuracy: 8.58 (#481/521)
- Artificial Analysis Intelligence Index: 8.91 (#490/830)
- AA GPQA Diamond: 57.58 (#534/797)
- ...and 1 more
Llama 2 70B Chat — ELO 1382, #627
- AA LiveCodeBench: 9.84 (#316/343)
- AA MMLU-Pro: 40.58 (#322/349)
Qwen 3 1.7B — ELO 1364, #646
- AA AIME 2025: 38.67 (#163/270)
- AA LiveCodeBench: 30.79 (#218/343)
- AA MMLU-Pro: 56.97 (#283/349)
- AA TAU-2 Bench: 26.02 (#398/575)
- AA-Omniscience Accuracy: 8.87 (#479/521)
- AA IFBench: 26.87 (#529/587)
- AA Terminal-Bench Hard: 0.0 (#555/565)
- AA Humanity's Last Exam: 4.64 (#592/798)
- AA CritPt: 0.0 (#623/674)
- AA Omniscience: -77.53 (#648/670)
- ...and 3 more
DeepSeek R1 Distill Llama 8B — ELO 1345, #667
- AA AIME 2025: 41.33 (#156/270)
- AA LiveCodeBench: 23.28 (#261/343)
- AA MMLU-Pro: 54.26 (#289/349)
Llama 3.2 3B — ELO 1323, #682
- AA TAU-2 Bench: 21.05 (#454/575)
- AA Humanity's Last Exam: 5.35 (#518/798)
- AA IFBench: 26.19 (#537/587)
- AA Long Context Reasoning: 4.33 (#576/664)
- Artificial Analysis Intelligence Index: 5.7 (#722/830)
- AA GPQA Diamond: 25.45 (#774/797)
AFM-4.5B — ELO 1319, #688
- LOL Arena: 40.5 (#42/42)
Qwen 3.5 0.8B — ELO 1311, #693
- AA TAU-2 Bench: 47.66 (#265/575)
- AA MMMU-Pro: 25.84 (#327/331)
- AA Omniscience: -54.52 (#498/670)
- AA-Omniscience Accuracy: 4.23 (#515/521)
- AA Long Context Reasoning: 9.0 (#551/664)
- AA Terminal-Bench Hard: 0.0 (#560/565)
- AA IFBench: 21.5 (#571/587)
- AA CritPt: 0.0 (#648/674)
- Artificial Analysis Intelligence Index: 6.13 (#684/830)
- AA GPQA Diamond: 11.11 (#796/797)
- ...and 1 more
Mistral 7B — ELO 1301, #704
- RULER: 68.4 (#39/45)
- AA-Omniscience Accuracy: 9.02 (#475/521)
- AA TAU-2 Bench: 0.0 (#569/575)
- AA IFBench: 19.93 (#578/587)
- AA Omniscience: -64.92 (#594/670)
- AA CritPt: 0.0 (#602/674)
- AA Humanity's Last Exam: 4.55 (#607/798)
- AA Long Context Reasoning: 0.0 (#644/664)
- AA GPQA Diamond: 17.68 (#794/797)
- Artificial Analysis Intelligence Index: 5.03 (#797/830)
Gemma 4 E2B — ELO 1290, #711
- AA Omniscience: -23.58 (#258/670)
- AA MMMU-Pro: 44.57 (#294/331)
- AA IFBench: 38.03 (#387/587)
- AA Terminal-Bench Hard: 3.03 (#439/565)
- AA TAU-2 Bench: 20.76 (#459/575)
- AA-Omniscience Accuracy: 6.63 (#501/521)
- AA Long Context Reasoning: 16.33 (#518/664)
- Artificial Analysis Intelligence Index: 7.77 (#547/830)
- AA Humanity's Last Exam: 4.77 (#581/798)
- AA GPQA Diamond: 43.33 (#639/797)
- ...and 1 more
Mistral 7B Instruct — ELO 1269, #725
- AA LiveCodeBench: 4.55 (#334/343)
- AA MMLU-Pro: 24.53 (#342/349)
Qwen 3 0.6B — ELO 1261, #729
- AA AIME 2025: 18.0 (#216/270)
- AA LiveCodeBench: 12.06 (#305/343)
- AA MMLU-Pro: 34.68 (#333/349)
- AA TAU-2 Bench: 21.05 (#453/575)
- AA Humanity's Last Exam: 5.64 (#498/798)
- AA-Omniscience Accuracy: 6.33 (#504/521)
- AA Terminal-Bench Hard: 0.0 (#554/565)
- AA IFBench: 23.33 (#555/587)
- AA CritPt: 0.0 (#621/674)
- AA Long Context Reasoning: 0.0 (#650/664)
- ...and 3 more
Llama 2 7B — ELO 1256, #730
- RULER: 85.6 (#26/45)
MiniCPM5-1B — ELO 1222, #751
- AA TAU-2 Bench: 80.99 (#159/575)
- AA Omniscience: -14.62 (#207/670)
- AA IFBench: 49.32 (#248/587)
- AA Humanity's Last Exam: 6.49 (#454/798)
- Artificial Analysis Intelligence Index: 8.8 (#496/830)
- AA-Omniscience Accuracy: 2.03 (#519/521)
- AA Terminal-Bench Hard: 0.0 (#556/565)
- AA Long Context Reasoning: 6.0 (#571/664)
- AA CritPt: 0.0 (#636/674)
- AA GPQA Diamond: 27.78 (#760/797)
Llama 3.2 1B — ELO 1217, #755
- AA-Omniscience Accuracy: 7.0 (#495/521)
- AA Omniscience: -54.7 (#502/670)
- AA Humanity's Last Exam: 5.46 (#512/798)
- AA Terminal-Bench Hard: 0.0 (#557/565)
- AA IFBench: 22.79 (#560/587)
- AA Long Context Reasoning: 6.67 (#566/664)
- AA TAU-2 Bench: 0.0 (#571/575)
- AA CritPt: 0.0 (#638/674)
- AA GPQA Diamond: 19.6 (#792/797)
- Artificial Analysis Intelligence Index: 4.83 (#824/830)
Gemma 3 1B — ELO 1213, #759
- AA-Omniscience Accuracy: 3.75 (#518/521)
- AA Humanity's Last Exam: 5.33 (#521/798)
- AA TAU-2 Bench: 10.53 (#535/575)
- AA Terminal-Bench Hard: 0.0 (#562/565)
- AA IFBench: 19.93 (#579/587)
- AA Omniscience: -75.45 (#634/670)
- AA CritPt: 0.0 (#661/674)
- AA Long Context Reasoning: 0.0 (#661/664)
- AA GPQA Diamond: 23.74 (#779/797)
- Artificial Analysis Intelligence Index: 4.83 (#828/830)
DeepSeek R1 Distill Qwen 1.5B — ELO 1167, #774
- AA AIME 2025: 22.0 (#205/270)
- AA LiveCodeBench: 6.98 (#327/343)
- AA MMLU-Pro: 26.92 (#340/349)
Celeris-1 — ELO 1522
- AA-Briefcase: 5.45 (#140/162)
- AA MMLU-Pro: 77.99 (#144/349)
Claude Fable 5 (with fallback) — ELO 1973
- AA SciCode: 61.0 (#3/215)
- AA-Omniscience Accuracy: 65.35 (#3/521)
- AA Humanity's Last Exam: 55.47 (#5/798)
- AA Terminal-Bench Hard: 62.88 (#5/565)
- AA Omniscience: 43.3 (#6/670)
- Artificial Analysis Intelligence Index: 49.7 (#10/830)
- AA TAU-2 Bench: 98.54 (#10/575)
- AA CritPt: 28.57 (#17/674)
- AA Long Context Reasoning: 82.33 (#27/664)
- AA GPQA Diamond: 92.63 (#40/797)
- ...and 2 more
Claude Instant — ELO 1485
- AA MMLU-Pro: 43.44 (#313/349)
- AA LiveCodeBench: 10.9 (#314/343)
Claude Mythos 5.1 — ELO 2155
- LLM Stats (Terminal-Bench 4.0): 60.9 (#1/15)
- LLM Stats Score: 45.22 (#30/366)
Cogito v2.1 — ELO 1779
- AA-Omniscience Accuracy: 30.18 (#137/521)
- AA Terminal-Bench Hard: 16.67 (#257/565)
- AA Omniscience: -25.78 (#266/670)
- AA IFBench: 46.26 (#271/587)
- AA GPQA Diamond: 76.77 (#310/797)
- AA Humanity's Last Exam: 12.0 (#318/798)
- AA Long Context Reasoning: 22.67 (#479/664)
- AA CritPt: 0.0 (#667/674)
DeepSeek R1 (Jan '25) — ELO 1652
- AA MMLU-Pro: 84.45 (#38/349)
- AA AIME 2025: 68.0 (#102/270)
- AA LiveCodeBench: 61.69 (#111/343)
- AA-Briefcase: 4.17 (#148/162)
- AA SciCode: 38.31 (#165/215)
DeepSeek V2 — ELO 1546
- Artificial Analysis Intelligence Index: 5.51 (#738/830)
DeepSeek V3 (Dec '24) — ELO 1563
- AA-Briefcase: 4.75 (#145/162)
- AA MMLU-Pro: 75.22 (#175/349)
- AA SciCode: 35.76 (#183/215)
- AA LiveCodeBench: 35.87 (#194/343)
- AA AIME 2025: 26.0 (#196/270)
DeepSeek V4 Pro Preview — ELO 1831
- MLS-Bench Lite: 24.4 (#14/14)
ERNIE 5.0 Thinking Preview — ELO 1627
- AA LiveCodeBench: 81.16 (#24/343)
- AA AIME 2025: 85.0 (#47/270)
- AA MMLU-Pro: 83.03 (#60/349)
Exaone 4.0 1.2B — ELO 1383
- AA Humanity's Last Exam: 5.98 (#486/798)
- AA TAU-2 Bench: 16.37 (#492/575)
- AA-Omniscience Accuracy: 6.83 (#498/521)
- AA IFBench: 22.99 (#557/587)
- AA Terminal-Bench Hard: 0.0 (#563/565)
- AA GPQA Diamond: 51.52 (#588/797)
- AA Omniscience: -79.82 (#653/670)
- AA Long Context Reasoning: 0.0 (#662/664)
- AA CritPt: 0.0 (#664/674)
- Artificial Analysis Intelligence Index: 5.27 (#770/830)
GPT-4o (Mar 2025) — ELO 1656
- AA MMLU-Pro: 80.29 (#112/349)
- AA LiveCodeBench: 42.54 (#171/343)
- AA AIME 2025: 25.67 (#197/270)
GPT-5.5 Instant (June 2026) — ELO 1909
- AA SciCode: 52.55 (#61/215)
- AA-Briefcase: 26.97 (#72/162)
GPT-6 Pro — ELO 2123
- KernelBench Hub - Mega: 24.8 (#1/10)
- WeirdML: 93.57 (#1/161)
- SimpleBench: 86.5 (#2/101)
- Last Translation Benchmark: 45.33 (#2/42)
- KernelBench Hub - CUDA: 68.37 (#4/13)
- Arabic Broad Leaderboard: 9.196 (#6/111)
Gemini 1.5 Pro (0514) — ELO 1586
- AA MMLU-Pro: 65.74 (#251/349)
- AA LiveCodeBench: 24.44 (#256/343)
Gemini 1.5 Pro (Sept) — ELO 1627
- AA MMLU-Pro: 75.02 (#181/349)
- AA LiveCodeBench: 31.64 (#215/343)
Gemini 2.5 Flash (Preview) — ELO 1702
- AA MMLU-Pro: 79.98 (#116/349)
- AA LiveCodeBench: 50.48 (#150/343)
Gemma 3n E2B — ELO 1425
- AA-Omniscience Accuracy: 6.83 (#497/521)
- AA Terminal-Bench Hard: 0.76 (#502/565)
- AA IFBench: 22.04 (#566/587)
- AA TAU-2 Bench: 0.0 (#573/575)
- AA CritPt: 0.0 (#650/674)
- AA Long Context Reasoning: 0.0 (#656/664)
- AA Omniscience: -80.17 (#656/670)
- AA Humanity's Last Exam: 4.17 (#669/798)
- AA GPQA Diamond: 22.93 (#785/797)
- Artificial Analysis Intelligence Index: 4.83 (#827/830)
Granite 4.2 3B — ELO 1520
- AA-Briefcase: 4.75 (#144/162)
Grok Beta — ELO 1571
- AA MMLU-Pro: 70.3 (#220/349)
- AA LiveCodeBench: 24.13 (#257/343)
Jamba 1.6 Large — ELO 1452
- AA LiveCodeBench: 17.25 (#281/343)
- AA MMLU-Pro: 56.46 (#285/349)
Jamba 1.6 Mini — ELO 1377
- AA LiveCodeBench: 7.09 (#326/343)
- AA MMLU-Pro: 36.68 (#330/349)
Laguna S 2.1 — ELO 1709
- ALEM (Multi-Agent Coordination): 7.4 (#9/18)
MiniMax M1 40k — ELO 1464
- AA LiveCodeBench: 65.71 (#91/343)
- AA MMLU-Pro: 80.83 (#100/349)
- AA AIME 2025: 13.67 (#225/270)
MiniMax M1 80k — ELO 1631
- AA LiveCodeBench: 71.11 (#65/343)
- AA MMLU-Pro: 81.62 (#85/349)
- AA AIME 2025: 61.0 (#117/270)
Muse Spark — ELO 1885
- AA-Briefcase: 16.77 (#100/162)
Nemotron 3 Nano — ELO 1608
- AA IFBench: 71.09 (#82/587)
- AA SciCode: 30.56 (#197/215)
- Tau3 Banking: 5.98 (#216/260)
- AA CritPt: 0.86 (#265/674)
- AA Terminal-Bench Hard: 13.64 (#281/565)
- AA TAU-2 Bench: 40.94 (#294/575)
- AA Humanity's Last Exam: 11.4 (#329/798)
- AA GPQA Diamond: 75.66 (#331/797)
- AA-Omniscience Accuracy: 17.3 (#331/521)
- AA Long Context Reasoning: 38.0 (#411/664)
- ...and 2 more
Nemotron 3 Nano Omni 30B A3B — ELO 1728
- AA IFBench: 63.2 (#147/587)
- AA MMMU-Pro: 53.18 (#262/331)
- AA TAU-2 Bench: 45.32 (#278/575)
- AA Terminal-Bench Hard: 8.33 (#324/565)
- AA-Omniscience Accuracy: 15.2 (#385/521)
- AA Long Context Reasoning: 39.67 (#404/664)
- Artificial Analysis Intelligence Index: 10.25 (#429/830)
- AA Omniscience: -57.43 (#520/670)
- AA Humanity's Last Exam: 4.82 (#573/798)
- AA GPQA Diamond: 46.87 (#619/797)
- ...and 1 more
Qwen 2.5 Coder 32B — ELO 1561
- Artificial Analysis Intelligence Index: 6.74 (#637/830)
- AA GPQA Diamond: 41.72 (#654/797)
- AA Humanity's Last Exam: 3.51 (#754/798)
Qwen 2.5 Coder 7B — ELO 1351
- AA Humanity's Last Exam: 4.89 (#564/798)
- AA GPQA Diamond: 33.94 (#705/797)
- Artificial Analysis Intelligence Index: 5.79 (#716/830)
Qwen 3 Coder 30B A3B — ELO 1630
- AA Terminal-Bench Hard: 15.15 (#266/565)
- AA TAU-2 Bench: 34.5 (#323/575)
- AA-Omniscience Accuracy: 16.4 (#355/521)
- AA Long Context Reasoning: 32.67 (#432/664)
- AA Omniscience: -50.73 (#446/670)
- Artificial Analysis Intelligence Index: 9.59 (#452/830)
- AA IFBench: 32.65 (#466/587)
- AA GPQA Diamond: 51.62 (#582/797)
- AA CritPt: 0.0 (#665/674)
- AA Humanity's Last Exam: 3.85 (#706/798)
Qwen 3.8 Max (Preview) — ELO 1927
- Design Arena (ASCII Art): 1212.0 (#28/87)
Solar Open 100B — ELO 1653
- AA IFBench: 57.69 (#178/587)
- AA TAU-2 Bench: 48.25 (#263/575)
- AA-Omniscience Accuracy: 18.53 (#298/521)
- AA Humanity's Last Exam: 10.43 (#355/798)
- AA Long Context Reasoning: 41.0 (#398/664)
- Artificial Analysis Intelligence Index: 10.36 (#425/830)
- AA GPQA Diamond: 65.66 (#459/797)
- AA Terminal-Bench Hard: 2.27 (#464/565)
- AA Omniscience: -54.5 (#496/670)
- AA CritPt: 0.0 (#624/674)
Sonar — ELO 1693
- AA LiveCodeBench: 29.52 (#223/343)
- AA MMLU-Pro: 68.92 (#234/349)
Sonar Pro — ELO 1695
- AA MMLU-Pro: 75.52 (#173/349)
- AA LiveCodeBench: 27.51 (#240/343)
glm-4-9B — ELO 1474
- RULER: 89.9 (#12/45)
internlm2.5-7B — ELO 1434
- RULER: 80.9 (#33/45)
openchat-3.5-1210 — ELO 1420
- AA LiveCodeBench: 11.53 (#309/343)
- AA MMLU-Pro: 31.02 (#336/349)
openchat_3.5 — ELO 1422
- AA Humanity's Last Exam: 4.79 (#578/798)
- Artificial Analysis Intelligence Index: 5.29 (#768/830)
- AA GPQA Diamond: 23.03 (#783/797)
qwen3.8-max-0902 — ELO 1876
- OTIS Mock AIME 2024-25: 100.0 (#1/290)
- Chess Puzzles (Epoch AI): 40.0 (#23/222)
- Epoch AI - Frontiermath Tiers 1 3 V2: 65.61 (#23/106)
- Epoch AI - Frontiermath Tier 4 V2: 34.15 (#24/62)
- SimpleQA Verified: 47.3 (#28/80)
- SvelteBench: 87.8 (#102/185)

Top-10 New Scores (167)

Claude Fable 5 on Conceptual Reasoning Index: 72.1 (#15)
Claude Fable 5 on Conceptual Reasoning Index - Argument Evaluation (LMCA): 61.15 (#16)
Claude Fable 5 on Conceptual Reasoning Index - Consistency (ACCoRD): 82.37 (#5)
Claude Fable 5 on Conceptual Reasoning Index - Decision Theory (DTBench): 95.55 (#8)
Claude Fable 5 on DuelLab Overall: 44.8 (#42)
Claude Fable 5 on LOL Arena: 49.4 (#4)
Claude Fable 5 on SealedBench: 71.0 (#3)
Claude Fable 5.1 on AA-Briefcase: 55.35 (#9)
Claude Fable 5.1 on AI for Education Pedagogy: 92.21 (#3)
Claude Fable 5.1 on AI for Education Pedagogy - Primary: 95.77 (#5)
Claude Fable 5.1 on AI for Education Pedagogy - Science: 93.99 (#11)
Claude Fable 5.1 on AI for Education Pedagogy - Social studies: 88.18 (#20)
Claude Fable 5.1 on AI for Education Pedagogy - Technology: 90.57 (#2)
Claude Fable 5.1 on Bullshit Benchmark: 69.1 (#14)
Claude Fable 5.1 on Chatbot Arena (Text - Chinese): 1595.0 (#1)
Claude Fable 5.1 on Chatbot Arena (Text - French): 1532.0 (#1)
Claude Fable 5.1 on Chatbot Arena (Text - Math): 1521.0 (#5)
Claude Fable 5.1 on Chess Bench LLM: 1118.0 (#29)
Claude Fable 5.1 on Conceptual Reasoning Index: 74.95 (#1)
Claude Fable 5.1 on Conceptual Reasoning Index - Consistency (ACCoRD): 83.67 (#3)
Claude Fable 5.1 on Conceptual Reasoning Index - Decision Theory (DTBench): 96.0 (#4)
Claude Fable 5.1 on Design Arena (ASCII Art): 1324.0 (#5)
Claude Fable 5.1 on Design Arena (Agents - Fullstack): 1344.0 (#1)
Claude Fable 5.1 on DuelLab Overall: 92.4 (#1)
Claude Fable 5.1 on EQ-Bench Longform Writing: 85.3 (#2)
Claude Fable 5.1 on Epoch AI - Mirrorcode: 73.3 (#1)
Claude Fable 5.1 on ErdosBench: 3.25 (#1)
Claude Fable 5.1 on FoodTruckBench: 62732.0 (#2)
Claude Fable 5.1 on Icelandic LLM Leaderboard - Average: 69.42 (#2)
Claude Fable 5.1 on LOL Arena: 50.0 (#2)
Claude Fable 5.1 on Last Translation Benchmark: 32.93 (#7)
Claude Fable 5.1 on MathArena - ARXIV April: 77.5 (#2)
Claude Fable 5.1 on MathArena - ARXIV June: 90.97 (#2)
Claude Fable 5.1 on MathArena - ARXIV_FALSE April: 80.33 (#2)
Claude Fable 5.1 on MathArena - ARXIV_FALSE June: 84.72 (#3)
Claude Fable 5.1 on MathArena Arxiv: 87.5 (#3)
Claude Fable 5.1 on MathArena Arxiv False: 67.0 (#2)
Claude Fable 5.1 on NYT Connections Extended: 88.5 (#20)
Claude Fable 5.1 on ProphetArena: 0.9534 (#5)
Claude Fable 5.1 on SEAL - Professional Reasoning Benchmark - Finance: 50.8 (#7)
Claude Fable 5.1 on SEAL - Professional Reasoning Benchmark - Legal: 51.6 (#5)
Claude Fable 5.1 on SealedBench: 72.0 (#2)
Claude Fable 5.1 on SpacetimeDB LLM Benchmark (C#): 98.9 (#2)
Claude Fable 5.1 on SpacetimeDB LLM Benchmark (Rust): 96.6 (#6)
Claude Fable 5.1 on Vals AI IOI: 90.78 (#3)
Claude Fable 5.1 on WebDev Arena (Fullstack): 1691.0 (#1)
Claude Fable 5.1 on You're Absolutely Right!: 3.625 (#7)
Claude Mythos 5.1 on LLM Stats Score: 45.22 (#30)
Claude Opus 5 on AA CritPt: 29.14 (#13)
Claude Opus 5 on AA GPQA Diamond: 93.74 (#14)
Claude Opus 5 on AA Humanity's Last Exam: 54.87 (#7)
Claude Opus 5 on AA Long Context Reasoning: 82.0 (#34)
Claude Opus 5 on AA MMMU-Pro: 84.74 (#10)
Claude Opus 5 on AA Omniscience: 37.07 (#13)
Claude Opus 5 on AA SciCode: 56.37 (#26)
Claude Opus 5 on AA-Briefcase: 48.48 (#23)
Claude Opus 5 on AA-Omniscience Accuracy: 60.87 (#9)
Claude Opus 5 on Artificial Analysis Intelligence Index: 50.7 (#8)
Claude Opus 5 on Conceptual Reasoning Index: 74.65 (#3)
Claude Opus 5 on Conceptual Reasoning Index - Argument Evaluation (LMCA): 64.45 (#5)
Claude Opus 5 on Conceptual Reasoning Index - Consistency (ACCoRD): 83.84 (#2)
Claude Opus 5 on Conceptual Reasoning Index - Decision Theory (DTBench): 96.45 (#2)
Claude Opus 5 on Icelandic LLM Leaderboard - Average: 68.63 (#3)
Claude Opus 5 on LOL Arena: 48.7 (#11)
Claude Opus 5 on MLS-Bench Lite: 49.8 (#3)
Claude Opus 5 on PostTrainBench: 35.04 (#4)
Claude Opus 5 on Tau3 Banking: 44.74 (#18)
Claude Opus 5 on Vals AI Harvey Legal Agent Bench: 6.67 (#21)
Claude Opus 5 on Vals AI Legal Research Bench: 55.29 (#2)
GPT-5.6 Sol on AA-Briefcase: 41.31 (#38)
GPT-5.6 Sol on Conceptual Reasoning Index: 68.65 (#23)
GPT-5.6 Sol on Conceptual Reasoning Index - Argument Evaluation (LMCA): 58.49 (#22)
GPT-5.6 Sol on Conceptual Reasoning Index - Consistency (ACCoRD): 75.05 (#40)
GPT-5.6 Sol on Conceptual Reasoning Index - Decision Theory (DTBench): 92.88 (#24)
GPT-5.6 Sol on Design Arena (SVG): 1331.0 (#7)
GPT-5.6 Sol on LOL Arena: 48.8 (#10)
GPT-5.6 Sol on Lean AI Formalization Leaderboard: 7.0 (#20)
GPT-5.6 Sol on OSWorld 2.0: 28.1 (#2)
GPT-5.6 Sol on OSWorld 2.0 Partial: 64.13 (#2)
GPT-5.6 Sol on PostTrainBench: 36.23 (#3)
GPT-6 on AA-Briefcase: 40.2 (#41)
GPT-6 on AI for Education Pedagogy: 92.66 (#1)
GPT-6 on AI for Education Pedagogy - Maths: 95.24 (#4)
GPT-6 on AI for Education Pedagogy - Primary: 96.71 (#2)
GPT-6 on AI for Education Pedagogy - Science: 95.08 (#3)
GPT-6 on AI for Education Pedagogy - Secondary: 91.51 (#2)
GPT-6 on AI for Education Pedagogy - Social studies: 90.91 (#5)
GPT-6 on AI for Education Pedagogy - Technology: 86.79 (#24)
GPT-6 on AI for Education SEND: 88.07 (#7)
GPT-6 on APEX v1: 69.3 (#2)
GPT-6 on Agent Arena: 12.39 (#2)
GPT-6 on Agent Arena - Bash Recovery: 5.71 (#15)
GPT-6 on Agent Arena - Confirmed Success: 18.82 (#2)
GPT-6 on Agent Arena - Praise vs Complaint: 40.79 (#1)
GPT-6 on Agent Arena - Steerability: -3.75 (#31)
GPT-6 on Agent Arena - Tool Hallucination: -0.38 (#1)
GPT-6 on Agent Security League - Functional Correctness: 82.1 (#6)
GPT-6 on Agent Security League - Security Correctness: 34.1 (#2)
GPT-6 on Agents on Rails: 35.0 (#1)
GPT-6 on Benchmarks.bio - BioSecBench-Refusal: 44.6 (#4)
GPT-6 on Bullshit Benchmark: 47.3 (#46)
GPT-6 on Chatbot Arena (Text - Coding): 1537.0 (#7)
GPT-6 on Chatbot Arena (Text - Creative Writing): 1464.0 (#22)
GPT-6 on Chatbot Arena (Text - English): 1495.0 (#11)
GPT-6 on Chatbot Arena (Text - Hard Prompts): 1495.0 (#33)
GPT-6 on Chatbot Arena (Text - Instruction Following): 1459.0 (#43)
GPT-6 on Chatbot Arena (Text - Longer Query): 1478.0 (#40)
GPT-6 on Chatbot Arena (Text - Multi-Turn): 1498.0 (#9)
GPT-6 on Chatbot Arena (Text - Russian): 1452.0 (#64)
GPT-6 on Chatbot Arena (Text): 1478.0 (#24)
GPT-6 on Chess Bench LLM: 1666.0 (#11)
GPT-6 on ClockBench: 65.6 (#2)
GPT-6 on Computer Anthology Terminal Tasks (Codex CLI): 65.6 (#1)
GPT-6 on Computer Anthology Terminal Tasks (Terminus-2): 67.0 (#1)
GPT-6 on Conceptual Reasoning Index: 73.89 (#6)
GPT-6 on Conceptual Reasoning Index - Argument Evaluation (LMCA): 64.37 (#6)
GPT-6 on Conceptual Reasoning Index - Consistency (ACCoRD): 81.69 (#10)
GPT-6 on Conceptual Reasoning Index - Decision Theory (DTBench): 95.55 (#7)
GPT-6 on Design Arena (ASCII Art): 1368.0 (#2)
GPT-6 on Design Arena (Agents - Mobile Apps): 1281.0 (#2)
GPT-6 on Design Arena (Agents - Web Apps): 1334.0 (#1)
GPT-6 on Design Arena (Data Viz): 1337.0 (#5)
GPT-6 on DuelLab Overall: 91.0 (#2)
GPT-6 on EQ-Bench Longform Writing: 82.8 (#4)
GPT-6 on Epoch AI - Apex Agents: 46.7 (#2)
GPT-6 on Epoch AI - Critpt: 31.71 (#2)
GPT-6 on Epoch AI - Proofbench: 99.0 (#3)
GPT-6 on Epoch AI - Scicode: 56.48 (#15)
GPT-6 on ErdosBench: 3.23 (#2)
GPT-6 on Featherbench: 96.0 (#8)
GPT-6 on Featherbench - Rubric Quality: 8.6 (#18)
GPT-6 on FoodTruckBench: 57761.0 (#5)
GPT-6 on Icelandic LLM Leaderboard - Average: 70.25 (#1)
GPT-6 on InferenceBench: 7.9 (#6)
GPT-6 on Last Translation Benchmark: 44.02 (#4)
GPT-6 on MathArena - ARXIV April: 90.83 (#1)
GPT-6 on MathArena - ARXIV June: 94.44 (#1)
GPT-6 on MathArena - ARXIVLEAN June: 65.22 (#2)
GPT-6 on MathArena - ARXIV_FALSE April: 97.54 (#1)
GPT-6 on MathArena - ARXIV_FALSE June: 99.07 (#1)
GPT-6 on MathArena Arxiv: 95.0 (#1)
GPT-6 on MathArena Arxiv False: 91.0 (#1)
GPT-6 on NYT Connections Extended: 98.7 (#1)
GPT-6 on OpenRouter GPQA Diamond: 94.4 (#2)
GPT-6 on RSI-Exam: 0.513 (#1)
GPT-6 on React Native Evals: 86.37 (#5)
GPT-6 on SEAL - Professional Reasoning Benchmark - Finance: 47.54 (#13)
GPT-6 on SEAL - Professional Reasoning Benchmark - Legal: 48.36 (#13)
GPT-6 on SEAL - SWE Atlas - Codebase QnA: 59.14 (#3)
GPT-6 on SEAL - SWE Atlas - Refactoring: 59.05 (#1)
GPT-6 on SEAL - SWE Atlas - Test Writing: 50.74 (#5)
GPT-6 on SecIT Bench (Pydantic AI): 81.01 (#2)
GPT-6 on SimpleBench: 83.6 (#3)
GPT-6 on SpacetimeDB LLM Benchmark (C#): 98.4 (#4)
GPT-6 on SpacetimeDB LLM Benchmark (TypeScript): 99.5 (#3)
GPT-6 on SvelteBench: 92.2 (#57)
GPT-6 on TaxCalcBench: 36.0 (#8)
GPT-6 on Vals AI IOI: 100.0 (#1)
GPT-6 on Vals AI ReverseEngBench: 56.87 (#1)
GPT-6 on WebDev Arena: 1797.04 (#1)
GPT-6 on WeirdML: 93.28 (#2)
GPT-6 on You're Absolutely Right!: 3.0 (#11)
GPT-6 Pro on Arabic Broad Leaderboard: 9.196 (#6)
GPT-6 Pro on KernelBench Hub - CUDA: 68.37 (#4)
GPT-6 Pro on Last Translation Benchmark: 45.33 (#2)
GPT-6 Pro on SimpleBench: 86.5 (#2)
GPT-6 Pro on WeirdML: 93.57 (#1)

New #1 Leaders (90)

Vending-Bench 2: GPT-6 (15514.7) beat Claude Opus 5 by 4332.83
Design Arena (SVG): GPT-6 (1494.0) beat Claude Fable 5.1 by 133.0
LLM Chess (Saplin): Gemini 3.8 Flash (High) (1631.6) beat GPT-5.6 Sol (xHigh) by 81.9
Last Translation Benchmark: Human LTB contributors (99.89) beat Gemini 3.1 Pro (Preview) by 60.59
Chatbot Arena (Image Edit): gpt-image-2.5-sunburst (1520.0) beat gpt-image-2 (Medium) by 59.0
MathArena - ARXIVLEAN June: Mistral Prover (Leanstral 1.5 + K3) (82.61) beat GPT-5.6 Sol by 45.11
Design Arena (3D): GPT-6 (1481.0) beat Kimi K3 by 45.0
MathArena Arxiv False: GPT-6 (Max) (91.0) beat GPT-5.5 (xHigh) by 41.0
Chatbot Arena (Text-to-Image): gpt-image-2.5-sunburst (1421.0) beat gpt-image-2 (Medium) by 39.0
WebDev Arena: GPT-6 (Max) (1797.04) beat Claude Fable 5.1 (Max) by 31.67
Chatbot Arena (Text - German): GLM-5.3 (Max) (1549.0) beat Gemini 3 Pro by 27.0
Vals AI ReverseEngBench: GPT-6 (Max) (56.87) beat GPT-5.6 Sol (Max) by 26.34
Chatbot Arena (Text - Chinese): Claude Fable 5.1 (Max) (1595.0) beat Claude Opus 5 (Max) by 26.0
MathArena - ARXIV_FALSE April: GPT-6 (Max) (97.54) beat GPT-5.5 (xHigh) by 25.41
Design Arena (Game Dev): GPT-6 (1456.0) beat Claude Fable 5.1 by 25.0
Open LLM Leaderboard - MuSR: calme-3.2-instruct-78B (60.24) beat T3Q-Qwen2.5-14B-Instruct-1M-e3 by 21.55
Design Arena (UI Components): GPT-6 (1400.0) beat Kimi K3 by 21.0
MyPCBench: Navigator n2 (82.6) beat Claude Opus 4.8 by 20.6
MathArena - ARXIV April: GPT-6 (Max) (90.83) beat Claude Fable 5 (Max) by 20.1
OSWorld 2.0 Partial: Claude Opus 5 (70.19) beat Claude Opus 4.8 by 15.39
Design Arena (Image Editing): gpt-image-2.5-sunburst (1386.0) beat riverflow-2.5-pro-edit by 15.0
LiveBench Python: DeepSeek V4.1 Flash (Max) (90.0) beat Claude Opus 5 (Max) by 15.0
OSWorld 2.0: Claude Opus 5 (34.72) beat Claude Opus 4.8 by 14.12
WebDev Arena (Reference-Based Design): GPT-6 (Max) (1827.0) beat Claude Fable 5.1 (Max) by 12.0
LLM Stats (BrowseComp-zh): Seed 1.8 (81.3) beat Qwen 3.5 397B A17B by 11.0
Epoch AI - Mirrorcode: Claude Fable 5.1 (High) (73.3) beat Claude Fable 5 (High) by 9.41
MathArena - ARXIV_FALSE June: GPT-6 (Max) (99.07) beat Claude Opus 5 (Max) by 8.33
Vals AI IOI: GPT-6 (Max) (100.0) beat Claude Opus 5 (Max) by 8.33
SEAL - Humanity's Last Exam: GPT-6 (54.8) beat Claude Fable 5.1 (xHigh) by 8.3
Chatbot Arena (Text - Korean): Claude Opus 5 (Max) (1509.0) beat Claude Fable 5 by 8.0
LLM Stats (Tau3 Banking): Ling 3.0 Flash Fin (41.0) beat Grok 4.5 by 8.0
LLM Stats (ZEROBench): DeepSeek V4.1 Flash (49.0) beat Kimi K3 by 8.0
MathArena - ARXIV June: GPT-6 (Max) (94.44) beat GPT-5.6 Sol (Max) by 7.71
Computer Anthology Terminal Tasks (Codex CLI): GPT-6 (High) (65.6) beat GPT-5.6 Sol (High) by 7.6
MathArena Arxiv: GPT-6 (Max) (95.0) beat Claude Fable 5 (Max) by 7.5
LLM Stats (Multi-IF): Ling-3.0-flash (87.7) beat Qwen 3 235B A22B 2507 (Thinking) by 7.1
Design Arena (Website): Muse Spark 1.3 (Max) (1364.0) beat Kimi K3 by 7.0
SEAL - Humanity's Last Exam (Text Only): GPT-6 (54.17) beat Gemini 3.1 Pro (Preview) (High) by 6.86
Coding Agent Leaderboard - Shellbench: GLM-5.2-FP8 + OpenClaw (18.7) beat Qwen3.6-27B-FP8 + OpenClaw by 5.9
LLM Stats (AutomationBench): DeepSeek V4.1 Flash (54.8) beat Muse Spark 1.3 by 5.4
DuelLab Overall: Claude Fable 5.1 (xHigh) (92.4) beat GPT-5.6 Sol (xHigh) by 5.1
Chatbot Arena (Text - French): Claude Fable 5.1 (Max) (1532.0) beat Claude Fable 5 by 5.0
Agent Security League - Security Correctness: Claude Fable 5.1 (37.4) beat Claude Opus 5 by 5.0
Computer Anthology Terminal Tasks (Terminus-2): GPT-6 (Medium) (67.0) beat Claude Fable 5.1 (High) by 4.6
LiveBench JavaScript: DeepSeek V4.1 Flash (Max) (81.82) beat Muse Spark 1.1 (xHigh) by 4.55
HiL-Bench: Claude Fable 5.1 (61.5) beat Claude Opus 5 by 4.5
Kaggle FACTS Parametric: GPT-6 (83.08) beat Gemini 3.1 Pro (Preview) by 4.12
WebDev Arena (Fullstack): Claude Fable 5.1 (Max) (1691.0) beat Qwen 3.8 Max by 4.0
SealedBench: GPT-6 (75.0) beat Claude Fable 5 by 4.0
LLM Stats (BabyVision): DeepSeek V4.1 Flash (89.6) beat Kimi K3 by 3.9
PostTrainBench: locus (45.58) beat Claude Fable 5 (Max) by 3.79
Vals AI SkillsBench: DeepSeek V4.1 Flash (High) (69.8) beat Grok 4.5 (High) by 3.77
NatureBench: Claude Opus 5 (HELIX) (26.67) beat Claude Opus 5 by 3.34
LLM Stats (Graphwalks parents <128k): Seed 2.0 Mini (93.0) beat GPT-5.4 by 3.2
LLM Stats (Terminal-Bench 4.0): Claude Mythos 5.1 (60.9) beat GPT-6 by 3.2
NYT Connections Extended: GPT-6 (Thinking, xHigh) (98.7) beat Gemini 3.1 Pro (Preview) by 2.6
LLM Stats (NL2Repo): DeepSeek V4.1 Flash (64.0) beat DeepSeek V4 Pro (0813) by 2.5
SEAL - SWE Atlas - Refactoring: GPT-6 (xHigh) (59.05) beat Claude Fable 5.1 (Claude Code) xHigh by 2.38
Agent Security League - Functional Correctness: Claude Fable 5.1 (87.2) beat GPT-5.5 by 2.3
DRACO: Claude Opus 5 (88.6) beat Claude Mythos 5 by 2.2
Conceptual Reasoning Index - Argument Evaluation (LMCA): Claude Fable 5.1 (65.46) beat Claude Opus 5 by 2.16
LLM Stats (Finance Agent v2): Ling 3.0 Flash Fin (59.81) beat Gemini 3.5 Flash by 1.95
LLM Stats (CyberGym): DeepSeek V4.1 Flash (88.1) beat Gemini 3.8 Flash Cyber by 1.9
Humanity's Last Exam (Self-Reported, With Tools): DeepSeek V4.1 Flash (63.9) beat GLM-5.3 by 1.4
Conceptual Reasoning Index: Claude Fable 5.1 (Medium) (74.95) beat Claude Opus 5 by 1.38
LOL Arena: GPT-6 (50.3) beat Gemini 3 Pro by 1.25
LLM Stats (Terminal-Bench 2.1): DeepSeek V4.1 Flash (90.6) beat Gemini 3.8 Flash by 1.2
Kaggle FACTS Grounding: GPT-6 (81.98) beat Gemma 4 26B A4B by 1.11
React Native Evals: Claude Fable 5.1 (89.05) beat Claude Opus 5 by 0.94
InferenceBench: Claude Fable 5.1 (9.83) beat Claude Opus 5 by 0.93
Kaggle FACTS Multimodal: Gemini 3.8 Flash (50.7) beat Gemini 3.7 Flash by 0.87
AI for Education Pedagogy - Maths: Claude Fable 5.1 (96.03) beat Gemini 3.6 Flash by 0.79
MMEB Image: Ovis-VL-Embedding-9B (81.69) beat WeMM-Embedding-9B by 0.7
WeirdML: GPT-6 Pro (Max) (93.57) beat Claude Fable 5.1 (Max) by 0.67
SpacetimeDB LLM Benchmark (Rust): GPT-6 (98.4) beat Grok 4.3 by 0.6
AI for Education Pedagogy: GPT-6 (High) (92.66) beat GPT-5.5 by 0.56
ForecastBench: fire-hedgehog (69.3) beat ceramic-kettle by 0.5
SpacetimeDB LLM Benchmark (TypeScript): Claude Fable 5.1 (100.0) beat GPT-5.6 Sol by 0.5
AI for Education Pedagogy - Secondary: Claude Fable 5.1 (91.67) beat Grok 4.6 by 0.48
AI for Education SEND: Claude Fable 5.1 (89.45) beat Claude Opus 5 by 0.46
LLM Stats (C-Eval): Qwen 3 Max (Thinking) (93.7) beat Qwen 3.6 Plus by 0.4
AA AIME 2025: GPT-5.2 (xHigh) (99.0) beat GPT-5 Codex (High) by 0.33
MLS-Bench Lite: Qwen3.8-Max-0902 (50.1) beat Claude Fable 5 by 0.2
KernelBench Hub - Mega: GPT-6 Pro (24.8) beat Claude Fable 5 by 0.19
ErdosBench: Claude Fable 5.1 (xHigh) (3.25) beat GPT-5.6 Sol (xHigh) by 0.13
Blueprint-Bench 2: GPT-6 (0.497) beat Claude Fable 5.1 by 0.08
RSI-Exam: GPT-6 (Max) (0.513) beat Claude Opus 5 (Max) by 0.05
FrontierMath - Tier 4 (v2): GPT-6 (High) (97.6) beat GPT-6 (Medium) by 0.04
Epoch AI - Frontiermath Tier 4 V2: GPT-6 (High) (97.6) beat GPT-6 (Medium) by 0.04
Epoch AI - Blueprint Bench 2: Claude Fable 5.1 (Unknown) (0.419) beat Claude Fable 5 (Unknown) by 0.03

The Aggregate Digest — 2026-09-05

2026-09-05T08:57:04.372946+00:00

Daily

New Benchmarks (114)

Design Arena (ASCII Art) (Elo): leader Claude Opus 5 (1382.0), 83 models
Design Arena’s crowdsourced design comparison on ASCII-art prompts, where the whole artefact is text. Visitors pick the better of two anonymous outputs and the votes become an Elo; models with too few votes are withheld by the site rather than ranked.
Design Arena (Agents - Fullstack) (Elo): leader Claude Opus 5 (1352.0), 42 models
Design Arena’s crowdsourced design comparison on agent runs that have to deliver a working full-stack app. Visitors pick the better of two anonymous outputs and the votes become an Elo; models with too few votes are withheld by the site rather than ranked.
Design Arena (Agents - Mobile Apps) (Elo): leader Claude Opus 5 (1348.0), 44 models
Design Arena’s crowdsourced design comparison on agent runs that have to deliver a mobile app. Visitors pick the better of two anonymous outputs and the votes become an Elo; models with too few votes are withheld by the site rather than ranked.
Design Arena (Agents - Web Apps) (Elo): leader Kimi K3 (1335.0), 40 models
Design Arena’s crowdsourced design comparison on agent runs that have to deliver a web app. Visitors pick the better of two anonymous outputs and the votes become an Elo; models with too few votes are withheld by the site rather than ranked.
LiveBench Code Completion (Score): leader GPT-5.2 Codex (86.96), 54 models
Partial code snippets requiring correct completion, measuring intent inference, pattern following, and syntactically valid continuations.
LiveBench Code Generation (Score): leader Claude Fable 5 (Max) (91.55), 54 models
Programming problems from LeetCode and AtCoder requiring complete solutions from natural language specs, measuring algorithmic problem-solving.
LiveBench Consecutive Events (Score): leader Claude Sonnet 4.6 (Thinking, Medium) (92.68), 54 models
Data analysis requiring detection and reasoning about sequences of events within structured datasets, measuring temporal pattern recognition.
LiveBench Integrals With Game (Score): leader GPT-5.6 Sol (Max) (100.0), 54 models
Calculus integration problems combined with game theory scenarios, measuring interdisciplinary mathematical reasoning.
LiveBench JavaScript (Score): leader Muse Spark 1.1 (xHigh) (77.27), 54 models
JavaScript programming tasks requiring correct, executable code, measuring JS syntax, standard library usage, and web-oriented programming.
LiveBench Logic With Navigation (Score): leader Muse Spark 1.2 (xHigh) (88.0), 54 models
Logic problems applied to 2D grid navigation, measuring combined deductive reasoning and spatial planning.
LiveBench Python (Score): leader Claude Opus 5 (Max) (75.0), 54 models
Python programming tasks requiring correct and idiomatic solutions, measuring syntax knowledge and algorithmic implementation.
LiveBench Spatial (Score): leader Claude Opus 4.7 (xHigh) (100.0), 54 models
Reasoning about 2D and 3D shapes and spatial relationships, measuring geometric visualization and mental rotation.
LiveBench Theory of Mind (Score): leader GPT-5.4 (xHigh) (88.46), 54 models
Reasoning about beliefs, intentions, and mental states of individuals in scenarios, measuring social cognition.
LiveBench TypeScript (Score): leader Claude Fable 5.1 (Max) (60.0), 54 models
TypeScript programming tasks requiring type-safe implementations, measuring knowledge of TypeScript's type system and patterns.
Chatbot Arena (Text - English) (Arena Score): leader Claude Opus 4.6 (High) (1514.0), 400 models
arena.ai’s text leaderboard restricted to battles whose prompts are in English. It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Chinese) (Arena Score): leader Claude Opus 5 (Max) (1569.0), 371 models
arena.ai’s text leaderboard restricted to battles whose prompts are in Chinese. It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - German) (Arena Score): leader Gemini 3 Pro (1522.0), 296 models
arena.ai’s text leaderboard restricted to battles whose prompts are in German. It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - French) (Arena Score): leader Claude Fable 5 (1527.0), 278 models
arena.ai’s text leaderboard restricted to battles whose prompts are in French. It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Spanish) (Arena Score): leader Claude Fable 5 (1518.0), 279 models
arena.ai’s text leaderboard restricted to battles whose prompts are in Spanish. It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Russian) (Arena Score): leader Claude Fable 5 (1521.0), 364 models
arena.ai’s text leaderboard restricted to battles whose prompts are in Russian. It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Japanese) (Arena Score): leader Claude Fable 5 (1526.0), 261 models
arena.ai’s text leaderboard restricted to battles whose prompts are in Japanese. It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Korean) (Arena Score): leader Claude Fable 5 (1501.0), 265 models
arena.ai’s text leaderboard restricted to battles whose prompts are in Korean. It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Coding) (Arena Score): leader Claude Opus 4.7 (High) (1552.0), 395 models
arena.ai’s text leaderboard restricted to battles whose prompts ask for code. It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Math) (Arena Score): leader Claude Fable 5 (1529.0), 383 models
arena.ai’s text leaderboard restricted to battles whose prompts are mathematical. It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Creative Writing) (Arena Score): leader Claude Fable 5 (1504.0), 398 models
arena.ai’s text leaderboard restricted to battles whose prompts ask for creative writing. It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Instruction Following) (Arena Score): leader Claude Opus 4.6 (High) (1514.0), 400 models
arena.ai’s text leaderboard restricted to battles whose prompts carry explicit instructions to obey. It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Longer Query) (Arena Score): leader Claude Opus 4.6 (High) (1524.0), 378 models
arena.ai’s text leaderboard restricted to the longest-prompt subset of battles. It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Multi-Turn) (Arena Score): leader Muse Spark 1.2 (xHigh) (1518.0), 398 models
arena.ai’s text leaderboard restricted to battles that ran past a single turn. It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Text - Hard Prompts) (Arena Score): leader Claude Opus 4.6 (High) (1533.0), 400 models
arena.ai’s text leaderboard restricted to the harder subset of battles, picked out by the site’s own difficulty classifier. It is the same crowd-vote Arena Score as the headline board, refitted over that category’s votes alone, so the ordering can differ from the overall table and the error bars are wider.
Chatbot Arena (Vision - English) (Arena Score): leader Claude Fable 5 (1309.0), 148 models
arena.ai’s vision leaderboard restricted to battles whose prompts are in English. Same crowd-vote Arena Score as the headline vision board, refitted over that category’s votes alone.
Chatbot Arena (Vision - Chinese) (Arena Score): leader Claude Opus 5 (High) (1371.0), 115 models
arena.ai’s vision leaderboard restricted to battles whose prompts are in Chinese. Same crowd-vote Arena Score as the headline vision board, refitted over that category’s votes alone.
Chatbot Arena (Vision - Creative Writing) (Arena Score): leader Gemini 3 Pro (1324.0), 87 models
arena.ai’s vision leaderboard restricted to battles that ask for creative writing about an image. Same crowd-vote Arena Score as the headline vision board, refitted over that category’s votes alone.
Chatbot Arena (Vision - Diagram) (Arena Score): leader Claude Fable 5 (1362.0), 104 models
arena.ai’s vision leaderboard restricted to battles whose images are diagrams and charts. Same crowd-vote Arena Score as the headline vision board, refitted over that category’s votes alone.
Chatbot Arena (Vision - Humor) (Arena Score): leader Claude Fable 5 (1331.0), 82 models
arena.ai’s vision leaderboard restricted to battles that ask the model to read or make humour about an image. Same crowd-vote Arena Score as the headline vision board, refitted over that category’s votes alone.
Chatbot Arena (Vision - OCR) (Arena Score): leader Claude Fable 5 (1329.0), 104 models
arena.ai’s vision leaderboard restricted to battles that ask the model to read text out of an image. Same crowd-vote Arena Score as the headline vision board, refitted over that category’s votes alone.
WebDev Arena (Frontend) (Arena Score): leader Claude Fable 5.1 (Max) (1797.0), 125 models
arena.ai’s WebDev Arena restricted to front-end builds: two models answer the same web-app prompt, the rendered results are shown side by side, and the votes are fitted to an Arena Score over that category alone.
WebDev Arena (Fullstack) (Arena Score): leader Qwen 3.8 Max (1687.0), 58 models
arena.ai’s WebDev Arena restricted to full-stack builds that need a backend as well as a UI: two models answer the same web-app prompt, the rendered results are shown side by side, and the votes are fitted to an Arena Score over that category alone.
WebDev Arena (HTML) (Arena Score): leader Claude Opus 5 (Max) (1683.0), 124 models
arena.ai’s WebDev Arena restricted to single-file HTML builds: two models answer the same web-app prompt, the rendered results are shown side by side, and the votes are fitted to an Arena Score over that category alone.
WebDev Arena (React) (Arena Score): leader Claude Fable 5.1 (Max) (1811.0), 109 models
arena.ai’s WebDev Arena restricted to React builds: two models answer the same web-app prompt, the rendered results are shown side by side, and the votes are fitted to an Arena Score over that category alone.
WebDev Arena (Reference-Based Design) (Arena Score): leader Claude Fable 5.1 (Max) (1819.0), 124 models
arena.ai’s WebDev Arena restricted to builds that have to reproduce a supplied reference design: two models answer the same web-app prompt, the rendered results are shown side by side, and the votes are fitted to an Arena Score over that category alone.
E-Commerce Bench (Final Assets (¥k, mean of 5 episodes)): leader GPT-5.6 Sol (Max) (1431.0), 18 models
Eighteen agents are handed ¥100,000 each and told to run up to four simulated online stores for 365 days against real market data — sourcing, negotiating with suppliers, pricing, managing inventory and staying solvent. The score is end-of-year total assets averaged over five independent episodes, which is why the board spans three orders of magnitude and why two models went bankrupt in 2 of their 5 runs. Qwen, 2026.
Terminal-Bench 4.0 (Accuracy (%)): leader GPT-6 (58.2), 14 models
Sixty-six containerized terminal tasks weighted toward science-adjacent and frontier engineering work, each agent run five times. Relative to 2.1 it lengthens timeouts and raises RAM and CPU on selected tasks, so a failure is more likely the agent's than the harness's.
Terminal-Bench 4.0 (Claude Code) (Accuracy (%)): leader Claude Fable 5.1 (57.9), 6 models
Terminal-Bench 4.0 restricted to the Claude Code harness, so the column compares models rather than scaffolds.
MERA v2 - SAGE (Score (%)): leader Gemini 3.7 Flash (78.6), 37 models
MERA v2 is the second generation of the Russian MERA suite, rebuilt after the v1 tasks saturated; this board is SAGE, spelling and grammatical error correction for Russian.
MERA v2 - Humor (Score (%)): leader Gemini 3.7 Flash (58.6), 37 models
MERA v2 is the second generation of the Russian MERA suite, rebuilt after the v1 tasks saturated; this board is Russian humour: reading a joke and picking or producing the funny reading.
MERA v2 - LIMUR (Score (%)): leader Claude Opus 5 (83.7), 37 models
MERA v2 is the second generation of the Russian MERA suite, rebuilt after the v1 tasks saturated; this board is the LIMUR subtest.
MERA v2 - RUBIN (Score (%)): leader Gemini 3.7 Flash (88.4), 37 models
MERA v2 is the second generation of the Russian MERA suite, rebuilt after the v1 tasks saturated; this board is the RUBIN subtest.
MERA v2 - Riddles (Score (%)): leader Gemini 3.7 Flash (71.9), 37 models
MERA v2 is the second generation of the Russian MERA suite, rebuilt after the v1 tasks saturated; this board is Russian riddles, which resist retrieval because the answer is a pun rather than a fact.
MERA v2 - SOB-Hard (Score (%)): leader GPT-5.6 Sol (93.9), 37 models
MERA v2 is the second generation of the Russian MERA suite, rebuilt after the v1 tasks saturated; this board is the hard SOB split.
MERA v2 - Characters (Score (%)): leader Gemini 3.7 Flash (64.3), 37 models
MERA v2 is the second generation of the Russian MERA suite, rebuilt after the v1 tasks saturated; this board is character-level manipulation, where tokenisation rather than knowledge is the obstacle.
MERA v2 - Enantiosemy (Score (%)): leader Grok 4.6 (70.8), 37 models
MERA v2 is the second generation of the Russian MERA suite, rebuilt after the v1 tasks saturated; this board is enantiosemy, the Russian words that carry two opposite senses, disambiguated from context.
MERA v2 - GorillaHard (Score (%)): leader Grok 4.6 (76.2), 37 models
MERA v2 is the second generation of the Russian MERA suite, rebuilt after the v1 tasks saturated; this board is the hard split of API and function-calling selection.
MERA v2 - IFHardBench (Score (%)): leader Gemini 3.7 Flash (95.8), 37 models
MERA v2 is the second generation of the Russian MERA suite, rebuilt after the v1 tasks saturated; this board is hard instruction following, where the constraints are the difficulty.
MERA v2 - NewReasoning (Score (%)): leader Gemini 3.7 Flash (82.6), 37 models
MERA v2 is the second generation of the Russian MERA suite, rebuilt after the v1 tasks saturated; this board is the v2 reasoning refresh, built after the v1 reasoning tasks saturated.
MERA v2 - RussianRegions (Score (%)): leader Gemini 3.7 Flash (20.6), 37 models
MERA v2 is the second generation of the Russian MERA suite, rebuilt after the v1 tasks saturated; this board is regional knowledge of Russia, where models trained on Moscow-centric text lose ground.
MERA v2 - Reasoning (Score (%)): leader Claude Opus 5 (76.2), 37 models
MERA v2 is the second generation of the Russian MERA suite, rebuilt after the v1 tasks saturated; this board is the general reasoning aggregate.
MERA v2 - Human Centric (Score (%)): leader Gemini 3.7 Flash (64.7), 37 models
MERA v2 is the second generation of the Russian MERA suite, rebuilt after the v1 tasks saturated; this board is the human-centric subset, judged against human preferences rather than a key.
MERA v2 - Agentic (Score (%)): leader Grok 4.6 (85.5), 37 models
MERA v2 is the second generation of the Russian MERA suite, rebuilt after the v1 tasks saturated; this board is agentic tool use and multi-step task completion in Russian.
MERA v2 - Culture Specific (Score (%)): leader Gemini 3.7 Flash (52.3), 37 models
MERA v2 is the second generation of the Russian MERA suite, rebuilt after the v1 tasks saturated; this board is questions that need Russian cultural context to answer.
Nejumi 4 - Total Score (Score (%)): leader Claude Opus 5 (87.2), 126 models
The headline score of Nejumi 4, the Japanese LLM leaderboard run by Weights & Biases: a weighted blend of the general language performance (GLP) and alignment (ALT) halves.
Nejumi 4 - GLP Average (Score (%)): leader Claude Opus 5 (86.68), 126 models
Nejumi 4’s general language performance half, averaged: the capability side of the Japanese leaderboard, spanning language, knowledge, reasoning, mathematics, coding, translation and function calling.
Nejumi 4 - GLP - Applied Language (Score (%)): leader Gemini 3.6 Flash (90.97), 126 models
Nejumi 4 (Japanese, Weights & Biases), general language performance: the applied Japanese language use category.
Nejumi 4 - GLP - Reasoning (Score (%)): leader Claude Opus 5 (96.56), 126 models
Nejumi 4 (Japanese, Weights & Biases), general language performance: the the reasoning aggregate category.
Nejumi 4 - GLP - Knowledge & QA (Score (%)): leader GPT-5.6 Sol (81.54), 126 models
Nejumi 4 (Japanese, Weights & Biases), general language performance: the knowledge-grounded question answering category.
Nejumi 4 - GLP - Basic Language (Score (%)): leader GPT-5.5 (87.8), 126 models
Nejumi 4 (Japanese, Weights & Biases), general language performance: the basic Japanese language competence category.
Nejumi 4 - GLP - Application Development (Score (%)): leader Claude Opus 5 (80.38), 126 models
Nejumi 4 (Japanese, Weights & Biases), general language performance: the application-level development tasks rather than single functions category.
Nejumi 4 - GLP - Expression (Score (%)): leader Qwen 3.6 Max Preview (99.5), 126 models
Nejumi 4 (Japanese, Weights & Biases), general language performance: the expressive Japanese writing category.
Nejumi 4 - GLP - Translation (Score (%)): leader Claude Fable 5 (91.65), 126 models
Nejumi 4 (Japanese, Weights & Biases), general language performance: the translation into and out of Japanese category.
Nejumi 4 - GLP - Information Retrieval (Score (%)): leader Qwen 3.6 27B (82.46), 126 models
Nejumi 4 (Japanese, Weights & Biases), general language performance: the retrieving the right fact from supplied material category.
Nejumi 4 - GLP - Abstract Reasoning (Score (%)): leader GPT-5.6 Terra (98.0), 126 models
Nejumi 4 (Japanese, Weights & Biases), general language performance: the abstract-pattern reasoning category.
Nejumi 4 - GLP - Logical Reasoning (Score (%)): leader Inkling Small (98.5), 126 models
Nejumi 4 (Japanese, Weights & Biases), general language performance: the logical reasoning category.
Nejumi 4 - GLP - Mathematical Reasoning (Score (%)): leader Claude Opus 5 (98.17), 126 models
Nejumi 4 (Japanese, Weights & Biases), general language performance: the mathematical reasoning category.
Nejumi 4 - GLP - General Knowledge (Score (%)): leader Gemini 3.1 Pro (Preview) (85.56), 126 models
Nejumi 4 (Japanese, Weights & Biases), general language performance: the general knowledge category.
Nejumi 4 - GLP - Specialized Knowledge (Score (%)): leader Claude Fable 5 (81.39), 126 models
Nejumi 4 (Japanese, Weights & Biases), general language performance: the specialised and professional knowledge category.
Nejumi 4 - GLP - Semantic Analysis (Score (%)): leader Claude Opus 5 (87.1), 126 models
Nejumi 4 (Japanese, Weights & Biases), general language performance: the semantic analysis category.
Nejumi 4 - GLP - Syntactic Analysis (Score (%)): leader GPT-5.5 (90.0), 126 models
Nejumi 4 (Japanese, Weights & Biases), general language performance: the syntactic analysis category.
Nejumi 4 - GLP - Coding (Score (%)): leader Claude Opus 5 (90.81), 126 models
Nejumi 4 (Japanese, Weights & Biases), general language performance: the code generation category.
Nejumi 4 - GLP - Function Calling (Score (%)): leader GPT-5.6 Sol (72.53), 126 models
Nejumi 4 (Japanese, Weights & Biases), general language performance: the tool and function calling category.
Nejumi 4 - ALT Average (Score (%)): leader Qwen 3.6 Max Preview (91.81), 126 models
Nejumi 4’s alignment half, averaged: controllability, ethics, toxicity, bias, truthfulness and robustness. It is scored separately from the capability half because the two rank models differently.
Nejumi 4 - ALT - Controllability (Score (%)): leader Grok 4.5 (94.73), 126 models
Nejumi 4 (Japanese, Weights & Biases), alignment half: the controllability, meaning whether the model obeys format and constraint instructions category.
Nejumi 4 - ALT - Ethics & Morality (Score (%)): leader Gemini 3.5 Flash (99.0), 126 models
Nejumi 4 (Japanese, Weights & Biases), alignment half: the ethics and morality category.
Nejumi 4 - ALT - Toxicity (Score (%)): leader Claude Haiku 4.5 (88.21), 126 models
Nejumi 4 (Japanese, Weights & Biases), alignment half: the toxicity category.
Nejumi 4 - ALT - Bias (Score (%)): leader Qwen 3.6 35B A3B (100.0), 126 models
Nejumi 4 (Japanese, Weights & Biases), alignment half: the bias category.
Nejumi 4 - ALT - Truthfulness (Score (%)): leader Claude Opus 4.1 (88.5), 126 models
Nejumi 4 (Japanese, Weights & Biases), alignment half: the truthfulness category.
Nejumi 4 - ALT - Robustness (Score (%)): leader Gemini 3.1 Pro (Preview) (100.0), 126 models
Nejumi 4 (Japanese, Weights & Biases), alignment half: the robustness to prompt perturbation category.
Nejumi 4 - jaster (0-shot) (Score (%)): leader GPT-5.5 (88.02), 126 models
Nejumi 4’s jaster battery run zero-shot: a large collection of Japanese NLP datasets converted to a single evaluation harness.
Nejumi 4 - jaster (2-shot) (Score (%)): leader Claude Fable 5 (88.63), 126 models
Nejumi 4’s jaster battery run with two in-context examples. The gap against the 0-shot board is the suite’s own measure of how much a model depends on demonstrations.
Nejumi 4 - MT-Bench (Japanese) (Score (%)): leader Claude Opus 5 (98.81), 126 models
Nejumi 4’s Japanese MT-Bench: multi-turn open-ended prompts graded by an LLM judge on the MT-Bench rubric, translated and re-authored for Japanese rather than machine translated.
Nejumi 4 - SWE-Bench (Japanese) (Score (%)): leader Claude Fable 5 (80.0), 126 models
Nejumi 4’s Japanese SWE-Bench: real repository issues resolved end to end, with the issue text in Japanese. It is the hardest board in the suite and the one where the Japanese-specialised models fall furthest behind the frontier.
E-commerce Last Exam (Mean Reward (%)): leader Kimi K3 (56.74), 29 models
FlyaiLab’s agentic shopping and travel benchmark: the agent has to complete a multi-step purchase or booking against a simulated storefront, and the score is mean reward over the episode rather than a pass or fail. This board is the overall configuration; the travel and e-commerce boards split it.
E-commerce Last Exam - Travel (Mean Reward (%)): leader Kimi K3 (48.11), 29 models
E-commerce Last Exam restricted to travel episodes: itinerary search and booking against a simulated provider, scored by mean reward over the episode.
E-commerce Last Exam - E-commerce (Mean Reward (%)): leader Grok 4.5 (78.65), 29 models
E-commerce Last Exam restricted to shopping episodes: search, compare, cart and checkout against a simulated storefront, scored by mean reward over the episode.
Phare - Average Safety (Score (%)): leader Claude Haiku 4.5 (83.16), 67 models
Giskard’s Phare multilingual safety probe (English, French, Spanish), Average Safety module: the headline average over the four resistance modules below. The metric is a resistance score, so a higher number means the model held the line more often.
Phare - Hallucination Resistance (Score (%)): leader Claude Opus 4.5 (88.23), 71 models
Giskard’s Phare multilingual safety probe (English, French, Spanish), Hallucination Resistance module: whether the model corrects a misleading premise instead of elaborating on it. The metric is a resistance score, so a higher number means the model held the line more often.
Phare - Harm Resistance (Score (%)): leader Claude Opus 4.6 (100.0), 71 models
Giskard’s Phare multilingual safety probe (English, French, Spanish), Harm Resistance module: whether it declines requests for harmful content. The metric is a resistance score, so a higher number means the model held the line more often.
Phare - Bias Resistance (Score (%)): leader GPT-4.1 Mini (88.12), 67 models
Giskard’s Phare multilingual safety probe (English, French, Spanish), Bias Resistance module: whether it reproduces stereotypes when the prompt invites one. The metric is a resistance score, so a higher number means the model held the line more often.
Phare - Jailbreak Resistance (Score (%)): leader Claude Sonnet 5 (86.84), 68 models
Giskard’s Phare multilingual safety probe (English, French, Spanish), Jailbreak Resistance module: whether policy holds under jailbreak framing. The metric is a resistance score, so a higher number means the model held the line more often.
BenchX - Pass Rate (Pass Rate (%)): leader GPT-5.6 Sol (82.94), 14 models
Sawt-ML’s BenchX tests consent and boundary handling in Saudi dialect Arabic: the share of scenarios where the model respects the stated boundary instead of talking past it. Dialect matters here, because the polite-refusal conventions the score keys on are not the Modern Standard Arabic ones.
Kivira k-bench (Overall Score): leader GPT-5.5 (98.96), 33 models
Kivira Health’s k-bench scores mental-health safety: how a model handles disclosure, crisis language and requests for clinical advice. The upstream board is one row per variant — system prompt crossed with reasoning level, 125 of them across 33 distinct models — and this board keeps each model’s best variant.
RAI-Bench - Refusal Rate (General) (Rate (%)): leader Qwen3 235B A22B Thinking (2507) (86.0), 79 models
GovTech Singapore’s RAI-Bench, localised undesired content (RabakBench): the share of general localised harmful prompts the model refuses. Written for Singapore deployment contexts, so the content is locally phrased rather than translated; higher is better.
RAI-Bench - Refusal Rate (Physics) (Rate (%)): leader Claude Opus 4 (100.0), 78 models
GovTech Singapore’s RAI-Bench, localised undesired content (RabakBench): the share of the physical-harm category the model refuses. Written for Singapore deployment contexts, so the content is locally phrased rather than translated; higher is better.
RAI-Bench - Refusal Rate (Career) (Rate (%)): leader Claude Sonnet 4 (100.0), 78 models
GovTech Singapore’s RAI-Bench, localised undesired content (RabakBench): the share of the career and employment category the model refuses. Written for Singapore deployment contexts, so the content is locally phrased rather than translated; higher is better.
RAI-Bench - Refusal Rate (JD) (Rate (%)): leader Claude Opus 4 (100.0), 79 models
GovTech Singapore’s RAI-Bench, localised undesired content (RabakBench): the share of the job-description category the model refuses. Written for Singapore deployment contexts, so the content is locally phrased rather than translated; higher is better.
RAI-Bench - RAG Robustness (LC Abstention) (Rate (%)): leader Claude Sonnet 4 (97.0), 70 models
GovTech Singapore’s RAI-Bench, out-of-knowledge-base robustness in the LC retrieval configuration: how often it abstains when the answer is not in the retrieved context. The pair is read together, because a model can score well on one by failing the other.
RAI-Bench - RAG Robustness (LC Factuality) (Rate (%)): leader Claude Sonnet 4 (56.0), 70 models
GovTech Singapore’s RAI-Bench, out-of-knowledge-base robustness in the LC retrieval configuration: how factual its answers are when it does answer. The pair is read together, because a model can score well on one by failing the other.
RAI-Bench - RAG Robustness (HY Abstention) (Rate (%)): leader Gemini 2.5 Flash Lite (93.0), 70 models
GovTech Singapore’s RAI-Bench, out-of-knowledge-base robustness in the HY retrieval configuration: how often it abstains when the answer is not in the retrieved context. The pair is read together, because a model can score well on one by failing the other.
RAI-Bench - RAG Robustness (HY Factuality) (Rate (%)): leader O3 (49.0), 70 models
GovTech Singapore’s RAI-Bench, out-of-knowledge-base robustness in the HY retrieval configuration: how factual its answers are when it does answer. The pair is read together, because a model can score well on one by failing the other.
Enkrypt AI - Risk Score (Risk Score): leader Claude Fable 5 (0.0), 267 models
Enkrypt AI’s LLM safety leaderboard: the composite risk score over all of its red-team suites. It is reported as a risk, so lower is better here and the leader is the least risky model rather than the most capable one.
Enkrypt AI - Safety Risk (Risk Score): leader Claude Fable 5 (0.0), 265 models
Enkrypt AI’s LLM safety leaderboard: how often the model produces unsafe content on red-team prompts. It is reported as a risk, so lower is better here and the leader is the least risky model rather than the most capable one.
Enkrypt AI - Bias Risk (Risk Score): leader Claude Fable 5 (0.0), 267 models
Enkrypt AI’s LLM safety leaderboard: measured bias in answers to demographically loaded prompts. It is reported as a risk, so lower is better here and the leader is the least risky model rather than the most capable one.
Enkrypt AI - Toxicity Risk (Risk Score): leader Claude Fable 5 (0.0), 265 models
Enkrypt AI’s LLM safety leaderboard: the rate of toxic output under provocation. It is reported as a risk, so lower is better here and the leader is the least risky model rather than the most capable one.
Enkrypt AI - Jailbreak Risk (Risk Score): leader Claude Fable 5 (0.0), 265 models
Enkrypt AI’s LLM safety leaderboard: how often a jailbreak attack gets the model past its own policy. It is reported as a risk, so lower is better here and the leader is the least risky model rather than the most capable one.
Enkrypt AI - Insecure Code Risk (Risk Score): leader Claude Fable 5 (0.0), 267 models
Enkrypt AI’s LLM safety leaderboard: the share of generated code carrying an exploitable weakness. It is reported as a risk, so lower is better here and the leader is the least risky model rather than the most capable one.
SuperCLUE-Law - Overall (Score): leader Kimi K3 (92.07), 11 models
SuperCLUE is the Chinese-language evaluation programme run by CLUE; its specialised boards each cover one professional domain and are graded by the organisers rather than self-reported. This board covers law: Chinese legal knowledge and reasoning.

Top-10 New Scores (164)

Claude Fable 5 on LiveBench AMPS Hard: 99.0 (#4)
Claude Fable 5 on LiveBench Connections: 99.33 (#18)
Claude Fable 5 on LiveBench Math Comp: 95.1 (#35)
Claude Fable 5 on LiveBench Olympiad: 92.84 (#2)
Claude Fable 5 on LiveBench Paraphrase: 77.65 (#7)
Claude Fable 5 on LiveBench Plot Unscrambling: 78.72 (#3)
Claude Fable 5 on LiveBench Simplify: 72.03 (#6)
Claude Fable 5 on LiveBench Story Generation: 71.7 (#28)
Claude Fable 5 on LiveBench Summarize: 81.7 (#7)
Claude Fable 5 on LiveBench Table Join: 56.13 (#3)
Claude Fable 5 on LiveBench Table Reformat: 94.12 (#50)
Claude Fable 5 on LiveBench Typos: 94.0 (#1)
Claude Fable 5 on LiveBench Zebra Puzzle: 100.0 (#4)
Claude Fable 5.1 on Chartography: 46.2 (#1)
Claude Fable 5.1 on ClockBench: 53.3 (#4)
Claude Fable 5.1 on ComplexConstraints: 45.1 (#10)
Claude Fable 5.1 on Design Arena (SVG): 1359.0 (#2)
Claude Fable 5.1 on EnterpriseBench (CoreCraft Agents): 77.4 (#1)
Claude Fable 5.1 on Epoch AI - Ebr Bench: 47.62 (#3)
Claude Fable 5.1 on GDP.pdf: 29.6 (#3)
Claude Fable 5.1 on KernelBench Hub - CUDA: 106.27 (#2)
Claude Fable 5.1 on LiveBench AMPS Hard: 99.0 (#8)
Claude Fable 5.1 on LiveBench Connections: 99.33 (#22)
Claude Fable 5.1 on LiveBench Math Comp: 97.06 (#11)
Claude Fable 5.1 on LiveBench Olympiad: 92.97 (#1)
Claude Fable 5.1 on LiveBench Paraphrase: 71.52 (#20)
Claude Fable 5.1 on LiveBench Plot Unscrambling: 75.17 (#4)
Claude Fable 5.1 on LiveBench Simplify: 70.17 (#12)
Claude Fable 5.1 on LiveBench Story Generation: 78.1 (#7)
Claude Fable 5.1 on LiveBench Summarize: 72.17 (#22)
Claude Fable 5.1 on LiveBench Table Join: 56.15 (#2)
Claude Fable 5.1 on LiveBench Table Reformat: 94.12 (#53)
Claude Fable 5.1 on LiveBench Typos: 94.0 (#2)
Claude Fable 5.1 on LiveBench Zebra Puzzle: 100.0 (#10)
Claude Fable 5.1 on Riemann-bench: 65.6 (#3)
Claude Fable 5.1 on SEAL - MCP Atlas: 87.2 (#2)
Claude Fable 5.1 on Vals AI Code Migration: 54.61 (#4)
Claude Fable 5.1 on Vals AI ProgramBench: 82.69 (#2)
Claude Fable 5.1 on WeirdML: 92.9 (#1)
Claude Opus 5 on FutureEval: 15.41 (#1)
Claude Opus 5 on LiveBench AMPS Hard: 99.01 (#1)
Claude Opus 5 on LiveBench Connections: 99.33 (#19)
Claude Opus 5 on LiveBench Math Comp: 94.12 (#44)
Claude Opus 5 on LiveBench Olympiad: 92.8 (#3)
Claude Opus 5 on LiveBench Paraphrase: 65.73 (#30)
Claude Opus 5 on LiveBench Plot Unscrambling: 74.73 (#5)
Claude Opus 5 on LiveBench Simplify: 61.58 (#32)
Claude Opus 5 on LiveBench Story Generation: 61.32 (#51)
Claude Opus 5 on LiveBench Summarize: 66.43 (#38)
Claude Opus 5 on LiveBench Table Join: 51.96 (#6)
Claude Opus 5 on LiveBench Table Reformat: 94.12 (#51)
Claude Opus 5 on LiveBench Typos: 92.0 (#3)
Claude Opus 5 on LiveBench Zebra Puzzle: 100.0 (#6)
GPT-6 on AutomationBench: 41.4 (#1)
GPT-6 on Benchmarks.bio - BioSecBench-Function: 35.1 (#10)
GPT-6 on Benchmarks.bio - VariantBench: 48.6 (#2)
GPT-6 on Coarena: 1000.0 (#11)
GPT-6 on Coarena - Task Completion: 66.7 (#12)
GPT-6 on DeepsecBench: 37.79 (#1)
GPT-6 on LM Market Cap LMC Score: 40.0 (#236)
GPT-6 on LiveBench AMPS Hard: 98.0 (#33)
GPT-6 on LiveBench Connections: 100.0 (#14)
GPT-6 on LiveBench Math Comp: 97.06 (#13)
GPT-6 on LiveBench Olympiad: 92.17 (#6)
GPT-6 on LiveBench Paraphrase: 78.25 (#5)
GPT-6 on LiveBench Plot Unscrambling: 86.29 (#1)
GPT-6 on LiveBench Simplify: 70.53 (#8)
GPT-6 on LiveBench Story Generation: 73.98 (#20)
GPT-6 on LiveBench Summarize: 79.55 (#9)
GPT-6 on LiveBench Table Join: 58.37 (#1)
GPT-6 on LiveBench Table Reformat: 100.0 (#19)
GPT-6 on LiveBench Typos: 82.0 (#28)
GPT-6 on LiveBench Zebra Puzzle: 100.0 (#11)
GPT-6 on Vals AI (Vals Index): 66.61 (#3)
GPT-6 on Vals AI BioMysteryBench: 79.26 (#1)
GPT-6 on Vals AI Code Migration: 67.74 (#1)
GPT-6 on Vals AI Excel Modeling: 71.7 (#6)
GPT-6 on Vals AI Finance Agent v2: 53.54 (#20)
GPT-6 on Vals AI Harvey Legal Agent Bench: 5.42 (#21)
GPT-6 on Vals AI Legal Research Bench: 39.42 (#17)
GPT-6 on Vals AI MedCode: 48.49 (#22)
GPT-6 on Vals AI MedScribe: 87.91 (#10)
GPT-6 on Vals AI ProgramBench: 85.42 (#1)
GPT-6 on Vals AI ProofBench: 99.0 (#3)
GPT-6 on Vals AI SAGE: 46.37 (#31)
GPT-6 on Vals AI Terminal-Bench 2.1: 87.27 (#1)
GPT-6 on Vals AI Vibe Code Bench: 89.59 (#3)
GPT-6 on VoxelBench: 2656.0 (#1)
Gemini 3.7 Flash on LiveBench AMPS Hard: 98.0 (#30)
Gemini 3.7 Flash on LiveBench Connections: 100.0 (#12)
Gemini 3.7 Flash on LiveBench Math Comp: 96.08 (#21)
Gemini 3.7 Flash on LiveBench Olympiad: 91.79 (#10)
Gemini 3.7 Flash on LiveBench Paraphrase: 78.07 (#6)
Gemini 3.7 Flash on LiveBench Plot Unscrambling: 66.36 (#18)
Gemini 3.7 Flash on LiveBench Simplify: 74.62 (#3)
Gemini 3.7 Flash on LiveBench Story Generation: 82.25 (#2)
Gemini 3.7 Flash on LiveBench Summarize: 84.77 (#3)
Gemini 3.7 Flash on LiveBench Table Join: 45.48 (#43)
Gemini 3.7 Flash on LiveBench Table Reformat: 98.04 (#39)
Gemini 3.7 Flash on LiveBench Typos: 90.0 (#4)
Gemini 3.7 Flash on LiveBench Zebra Puzzle: 94.5 (#27)
Gemini 3.7 Flash on UGI - Natural Intelligence: 76.64 (#6)
Gemini 3.7 Flash on UGI - Willingness (W/10): 2.2 (#1134)
Gemini 3.7 Flash on UGI - Writing: 77.53 (#2)
Gemini 3.7 Flash on UGI Leaderboard: 47.02 (#209)
Gemini 3.8 Flash on AutomationBench: 29.7 (#8)
Gemini 3.8 Flash on Chartography: 42.5 (#5)
Gemini 3.8 Flash on ClockBench: 62.8 (#2)
Gemini 3.8 Flash on ComplexConstraints: 48.4 (#6)
Gemini 3.8 Flash on Computer Anthology Terminal Tasks (Terminus-2): 51.8 (#3)
Gemini 3.8 Flash on Deep20Bench: 13.14 (#5)
Gemini 3.8 Flash on Design Arena (Data Viz): 1304.0 (#13)
Gemini 3.8 Flash on Design Arena (Game Dev): 1344.0 (#9)
Gemini 3.8 Flash on Design Arena (UI Components): 1368.0 (#3)
Gemini 3.8 Flash on GDP.pdf: 23.4 (#12)
Gemini 3.8 Flash on HANDBOOK.md Agents: 11.2 (#27)
Gemini 3.8 Flash on LiveBench AMPS Hard: 99.0 (#9)
Gemini 3.8 Flash on LiveBench Connections: 100.0 (#13)
Gemini 3.8 Flash on LiveBench Math Comp: 95.1 (#38)
Gemini 3.8 Flash on LiveBench Olympiad: 92.16 (#7)
Gemini 3.8 Flash on LiveBench Paraphrase: 80.0 (#1)
Gemini 3.8 Flash on LiveBench Plot Unscrambling: 73.38 (#8)
Gemini 3.8 Flash on LiveBench Simplify: 75.4 (#2)
Gemini 3.8 Flash on LiveBench Story Generation: 82.75 (#1)
Gemini 3.8 Flash on LiveBench Summarize: 87.5 (#2)
Gemini 3.8 Flash on LiveBench Table Join: 45.33 (#44)
Gemini 3.8 Flash on LiveBench Table Reformat: 96.08 (#49)
Gemini 3.8 Flash on LiveBench Typos: 90.0 (#5)
Gemini 3.8 Flash on LiveBench Zebra Puzzle: 98.25 (#15)
Gemini 3.8 Flash on ParseBench: 72.08 (#9)
Gemini 3.8 Flash on Riemann-bench: 51.2 (#6)
Gemini 3.8 Flash on UGI - Natural Intelligence: 78.41 (#2)
Gemini 3.8 Flash on UGI - Willingness (W/10): 2.2 (#1137)
Gemini 3.8 Flash on UGI - Writing: 78.55 (#1)
Gemini 3.8 Flash on UGI Leaderboard: 46.71 (#213)
Muse Spark 1.3 on AA-Briefcase: 54.87 (#8)
Muse Spark 1.3 on Chartography: 27.6 (#21)
Muse Spark 1.3 on Chatbot Arena (Code): 1618.0 (#11)
Muse Spark 1.3 on ClockBench: 42.2 (#8)
Muse Spark 1.3 on ComplexConstraints: 51.9 (#1)
Muse Spark 1.3 on GDP.pdf: 27.6 (#4)
Muse Spark 1.3 on HANDBOOK.md Agents: 15.4 (#19)
Muse Spark 1.3 on LiveBench AMPS Hard: 99.0 (#10)
Muse Spark 1.3 on LiveBench Connections: 94.0 (#40)
Muse Spark 1.3 on LiveBench Math Comp: 97.06 (#12)
Muse Spark 1.3 on LiveBench Olympiad: 91.74 (#11)
Muse Spark 1.3 on LiveBench Paraphrase: 78.62 (#4)
Muse Spark 1.3 on LiveBench Plot Unscrambling: 68.38 (#12)
Muse Spark 1.3 on LiveBench Simplify: 72.52 (#4)
Muse Spark 1.3 on LiveBench Story Generation: 77.48 (#8)
Muse Spark 1.3 on LiveBench Summarize: 83.4 (#5)
Muse Spark 1.3 on LiveBench Table Join: 48.19 (#25)
Muse Spark 1.3 on LiveBench Table Reformat: 100.0 (#18)
Muse Spark 1.3 on LiveBench Typos: 86.0 (#11)
Muse Spark 1.3 on LiveBench Zebra Puzzle: 98.0 (#18)
Muse Spark 1.3 on Riemann-bench: 28.0 (#18)
Muse Spark 1.3 on RuneBench: 2653.0 (#26)
Muse Spark 1.3 on Vals AI (Vals Index): 60.31 (#8)
Muse Spark 1.3 on Vals AI Excel Modeling: 62.71 (#18)
Muse Spark 1.3 on Vals AI Finance Agent v2: 58.9 (#4)
Muse Spark 1.3 on Vals AI Harvey Legal Agent Bench: 22.08 (#2)
Muse Spark 1.3 on Vals AI Legal Research Bench: 40.87 (#14)
Muse Spark 1.3 on Vals AI Terminal-Bench 2.1: 72.28 (#16)
Muse Spark 1.3 on Vals AI Vibe Code Bench: 82.86 (#6)

New #1 Leaders (25)

RuneBench: GPT-6 (34063.0) beat Grok 4.6 by 21306.0
LiveBench Zebra Puzzle: GPT-5.5 (xHigh) (100.0) beat Claude 3.5 Sonnet (20240620) by 52.0
LiveBench AMPS Hard: Claude Opus 5 (Max) (99.01) beat Claude 3.5 Sonnet (20240620) by 50.01
LiveBench Math Comp: Claude Opus 4.7 (xHigh) (98.04) beat Gemini 1.5 Pro (Preview 0827) by 43.87
LiveBench Connections: Gemini 3.1 Pro (Preview) (High) (100.0) beat GPT-4o (2024-08-06) by 42.0
LiveBench Plot Unscrambling: GPT-6 (Max) (86.29) beat Claude 3.5 Sonnet (20240620) by 31.15
LiveBench Table Reformat: DeepSeek V4 Pro (100.0) beat GPT-4 Preview (0125) by 30.0
LiveBench Typos: Claude Fable 5 (Max) (94.0) beat Claude 3 Opus (20240229) by 26.0
LiveBench Olympiad: Claude Fable 5.1 (Max) (92.97) beat Gemini 1.5 Pro (Preview 0827) by 24.31
LiveBench Table Join: GPT-6 (Max) (58.37) beat GPT-4o ChatGPT by 13.07
Vals AI Code Migration: GPT-6 (Max) (67.74) beat Claude Opus 5 (Max) by 10.27
AutomationBench: GPT-6 (Max) (41.4) beat Claude Fable 5.1 with Opus 5 Fallback (Max) by 10.0
EnterpriseBench (CoreCraft Agents): Claude Fable 5.1 (Adaptive Reasoning, Max Effort) (77.4) beat Claude Fable 5 (Adaptive Reasoning, Max Effort) by 7.1
Design Arena (Game Dev): Claude Fable 5.1 (1430.0) beat Kimi K3 by 6.0
LiveBench Summarize: Gemini 3.1 Pro (Preview) (High) (88.23) beat Llama 3.1 70B Instruct by 4.38
UGI - Writing: Gemini 3.8 Flash (Thinking, Medium) (78.55) beat Claude Fable 5 (Adaptive Reasoning, High Effort) by 4.32
Benchmarks.bio - TxBench-AD: GPT-6 (56.9) beat Claude Opus 5 by 3.94
Vals AI ProgramBench: GPT-6 (Max) (85.42) beat Claude Opus 5 (Max) by 3.15
DeepsecBench: GPT-6 (xHigh) (37.79) beat GPT-5.6 Sol (xHigh) by 2.21
FutureEval: Claude Opus 5 (High) (15.41) beat Claude Fable 5 (High) by 2.18
Vals AI Terminal-Bench 2.1: GPT-6 (Max) (87.27) beat GPT-5.6 Sol (Max) by 1.5
LiveBench Story Generation: Gemini 3.8 Flash (High) (82.75) beat Gemini 1.5 Flash (Preview 0827) by 1.42
ComplexConstraints: Muse Spark 1.3 (xHigh) (51.9) beat GPT-5.6 Sol (Max) by 1.4
Chartography: Claude Fable 5.1 (Adaptive Reasoning, Max Effort) (46.2) beat GPT-5.6 Sol (Max) by 1.2
WeirdML: Claude Fable 5.1 (Max) (92.9) beat Claude Fable 5.1 (High) by 0.6

The Aggregate Digest — 2026-09-04

2026-09-04T05:07:34.915933+00:00

Frontier

Best available model: GPT-6 (1815) takes the crown from Claude Fable 5.1 (1806)
Best measured model: GPT-6 (1815) takes the crown from Claude Fable 5.1 (1806)
Best available model: GPT-6 enters at #1 (1815 ELO) on 55 benchmarks
Best available model: Muse Spark 1.3 enters at #9 (1766 ELO) on 41 benchmarks

Daily

New Benchmarks (5)

Benchmarks.bio - TxBench-AD (Pass Rate (%)): leader Claude Opus 5 (52.96), 12 models
Antibody discovery run end to end: 100 agentic evaluations spanning ten stages from target hypothesis and antigen design through binder generation, epitope and escape analysis, to preclinical candidate de-risking. Graded deterministically, and the whole board sits near 50 percent pass rate.
LLM Stats (ARC-AGI-3) (Score (%)): leader GPT-6 (99.9), 5 models
LLM Stats mirror of ARC Prize’s interactive-reasoning benchmark: an agent is dropped into a novel game environment with no instructions, no stated goal and no rules, and has to infer all three by playing. Six environments of eight to ten levels each, scored on levels completed with total actions as the tiebreaker — humans clear every one of them.
LLM Stats (ExploitGym) (Score (%)): leader GPT-6 (42.4), 5 models
LLM Stats mirror of ExploitGym, 898 exploit-development tasks built from real vulnerabilities in C/C++ userspace projects, Google’s V8 engine and the Linux kernel. The agent works a containerised target over multiple rounds under configurable mitigations, and a run only counts when it captures the flag and a judge confirms the exploit used the intended vulnerability rather than some other way in. UC Berkeley, MPI-SP, UC Santa Barbara and Arizona State, 2026.
LLM Stats (Humanity's Last Exam (with tools, text-only)) (Score (%)): leader DeepSeek V4 Pro (0813) (60.0), 5 models
LLM Stats mirror of Humanity’s Last Exam run two ways at once: tools enabled, and the multimodal questions dropped so the set is text alone. Both choices lift the numbers well above the bare no-tools board, so read this column against other tool-enabled runs rather than against plain HLE.
FrontierSWE V2 (Mean@5 Score (%)): leader Claude Fable 5.1 (56.3), 10 models
The second FrontierSWE round, September 2026: 34 ultra-long-horizon tasks across implementation, performance engineering, scientific computing, visual reasoning and AI research — reimplementing git in Zig, decoding speech from MEG recordings, training a medium-range weather model. Each task gets a 20-hour budget and five runs, and the board reports the mean of those five rather than V1’s pairwise dominance.

New Models (1)

GPT-6 — ELO 1815, #1
- DeepSWE: 74.1 (#1/31)
- FrontierMath - Tier 4 (v2): 97.56 (#1/60)
- FrontierMath - Tiers 1-3 (v2): 93.68 (#1/101)
- ZeroEval GPQA Diamond: 96.0 (#1/247)
- Epoch AI - ECI: 169.23 (#1/585)
- SimpleQA Verified: 75.6 (#1/78)
- Chess Puzzles (Epoch AI): 72.0 (#1/220)
- Epoch AI - Frontiermath Tiers 1 3 V2: 93.68 (#1/104)
- Epoch AI - Frontiermath Tier 4 V2: 97.56 (#1/60)
- Epoch AI - Ebr Bench: 76.19 (#1/20)
- ...and 44 more

Top-10 New Scores (72)

Claude Fable 5 on FrontierCode: 64.9 (#54)
Claude Fable 5 on SEAL - Professional Reasoning Benchmark - Finance: 53.86 (#2)
Claude Fable 5 on SEAL - Professional Reasoning Benchmark - Legal: 52.56 (#2)
Claude Fable 5.1 on FrontierCode: 63.6 (#49)
Claude Fable 5.1 on OpenRouter GPQA Diamond: 82.7 (#52)
Claude Fable 5.1 on OpenRouter Tau2-Bench Airline: 76.7 (#16)
Claude Fable 5.1 on SEAL - Fortress: 13.72 (#50)
Claude Fable 5.1 on SEAL - Humanity's Last Exam: 46.5 (#1)
Claude Fable 5.1 on SEAL - Humanity's Last Exam (Text Only): 46.8 (#2)
Claude Fable 5.1 on SEAL - Professional Reasoning Benchmark - Finance: 50.8 (#7)
Claude Fable 5.1 on SEAL - Professional Reasoning Benchmark - Legal: 51.6 (#5)
Claude Fable 5.1 on SpeechMap Compliance: 72.4 (#128)
Claude Fable 5.1 on Terminal-Bench 2.1: 57.9 (#2)
Claude Fable 5.1 on Vending-Bench 2: 5421.56 (#20)
Claude Fable 5.1 on VoxelBench: 1000.0 (#49)
Claude Fable 5.1 on WeirdML: 92.3 (#1)
GPT-6 on AA CritPt: 31.71 (#2)
GPT-6 on AA GDPval: 1629.31 (#23)
GPT-6 on AA GPQA Diamond: 96.26 (#1)
GPT-6 on AA Humanity's Last Exam: 54.68 (#6)
GPT-6 on AA Long Context Reasoning: 76.33 (#51)
GPT-6 on AA MMMU-Pro: 86.88 (#1)
GPT-6 on AA Omniscience: 43.73 (#1)
GPT-6 on AA Omniscience - Business: 52.0 (#7)
GPT-6 on AA Omniscience - Health: 55.9 (#2)
GPT-6 on AA Omniscience - Humanities & Social Sciences: 61.8 (#6)
GPT-6 on AA Omniscience - Law: 61.4 (#10)
GPT-6 on AA Omniscience - Science, Engineering & Mathematics: 54.3 (#12)
GPT-6 on AA Omniscience - Software Engineering (SWE): 91.0 (#3)
GPT-6 on AA SciCode: 54.05 (#33)
GPT-6 on AA-Briefcase: 52.32 (#12)
GPT-6 on AA-Omniscience Accuracy: 62.6 (#6)
GPT-6 on ARC-AGI-1: 98.5 (#3)
GPT-6 on ARC-AGI-2: 95.0 (#2)
GPT-6 on Artificial Analysis Intelligence Index: 61.22 (#9)
GPT-6 on BenchLM: 81.9 (#4)
GPT-6 on BenchmarkList ECI: 158.28 (#4)
GPT-6 on Chess Puzzles (Epoch AI): 72.0 (#1)
GPT-6 on DeepSWE: 74.1 (#1)
GPT-6 on Epoch AI - ECI: 169.23 (#1)
GPT-6 on Epoch AI - Ebr Bench: 76.19 (#1)
GPT-6 on Epoch AI - Frontiermath Tier 4 V2: 97.56 (#1)
GPT-6 on Epoch AI - Frontiermath Tiers 1 3 V2: 93.68 (#1)
GPT-6 on Epoch AI - Mirrorcode: 46.7 (#2)
GPT-6 on Epoch AI - Mystery Game Puzzles: 84.0 (#1)
GPT-6 on FrontierCode: 64.5 (#52)
GPT-6 on FrontierMath - Tier 4 (v2): 97.56 (#1)
GPT-6 on FrontierMath - Tiers 1-3 (v2): 93.68 (#1)
GPT-6 on LLM Stats (AutomationBench): 41.4 (#4)
GPT-6 on LLM Stats (DeepSWE 1.1): 74.1 (#2)
GPT-6 on LLM Stats (FrontierCode 1.1): 53.3 (#3)
GPT-6 on LLM Stats (HealthBench Professional): 63.4 (#2)
GPT-6 on LLM Stats (OSWorld 2.0): 72.6 (#2)
GPT-6 on Mercor APEX: 46.7 (#2)
GPT-6 on OTIS Mock AIME 2024-25: 100.0 (#2)
GPT-6 on SimpleQA Verified: 75.6 (#1)
GPT-6 on Tau3 Banking: 43.09 (#22)
GPT-6 on Vellum - GPQA: 96.0 (#2)
GPT-6 on Vellum - Humanity's Last Exam: 57.2 (#7)
GPT-6 on VoxelBench: 1000.0 (#48)
Gemini 3.7 Flash on LLM Chess (Saplin): 1107.1 (#13)
Gemini 3.7 Flash on Terminal-Bench 2.1: 11.2 (#14)
Gemini 3.8 Flash on Design Arena (3D): 1340.0 (#12)
Gemini 3.8 Flash on FrontierCode: 56.3 (#35)
Gemini 3.8 Flash on Kaggle FACTS Grounding: 73.09 (#8)
Gemini 3.8 Flash on LLM2014 Logic 2026-09: 62.05 (#6)
Gemini 3.8 Flash on SecIT Bench (Pydantic AI): 69.0 (#13)
Gemini 3.8 Flash on SimpleBench: 82.4 (#2)
Gemini 3.8 Flash on Surface Evolver Bench: 76.88 (#8)
Gemini 3.8 Flash on Surface Evolver Bench Pass Rate: 56.25 (#8)
Gemini 3.8 Flash on TaxCalcBench: 4.0 (#20)
Gemini 3.8 Flash on Terminal-Bench 2.1: 19.1 (#10)

New #1 Leaders (49)

ARC-AGI-3: GPT-6 (Provider Adapter, High) (99.95) beat Claude Opus 5 (High) by 69.79
Epoch AI - Ebr Bench: GPT-6 (Max) (76.19) beat Claude Opus 5 (Max) by 26.19
Epoch AI - Mystery Game Puzzles: GPT-6 (Max) (84.0) beat Claude Opus 5 (Max) by 25.0
LLM Stats (ExploitBench): GPT-6 (100.0) beat Claude Fable 5 by 22.0
TemporalBench - FreshRetailNet T3: TimeSeries Scientist (deepseek-chat) (34.09) beat AgentScope (gpt-4o) by 20.45
TemporalBench - MIMIC T3: TimeClaw (deepseek-v4-pro) (56.86) beat Single LLM (gpt-4o) by 20.25
TemporalBench - CausalChambers T4: TimeClaw (deepseek-v4-pro) (51.33) beat AgentScope (gpt-4o) by 19.33
TemporalBench - T4: TimeClaw (deepseek-v4-pro) (43.63) beat CAMEL (gpt-4o) by 15.71
TemporalBench - PSML T4: TimeClaw (deepseek-v4-pro) (50.0) beat Single LLM (gpt-4o) by 14.0
TemporalBench - T3: TimeClaw (deepseek-v4-pro) (42.18) beat Single LLM (gpt-4o) by 12.94
TemporalBench - MIMIC T2: TimeClaw (deepseek-v4-pro) (33.33) beat TimeSeries Scientist (gpt-4o) by 9.93
FrontierMath - Tier 4 (v2): GPT-6 (Medium) (97.56) beat Claude Fable 5.1 (Max) by 9.76
Epoch AI - Frontiermath Tier 4 V2: GPT-6 (Medium) (97.56) beat Claude Fable 5.1 (Max) by 9.76
TemporalBench - T2: TimeClaw (deepseek-v4-pro) (38.22) beat Single LLM (gpt-4o) by 8.38
Chess Puzzles (Epoch AI): GPT-6 (Max) (72.0) beat GPT-5.6 Pro Sol (Max) by 8.0
TemporalBench: TimeClaw (deepseek-v4-pro) (41.8) beat Single LLM (gpt-4o) by 7.69
LLM Stats (Agents' Last Exam): GPT-6 (59.3) beat GPT-5.6 Sol by 6.6
Terminal-Bench 2.1: GPT-6 (58.2) beat Claude Opus 5 by 6.4
TemporalBench - FreshRetailNet T1: AgentScope (deepseek-chat) (70.45) beat CAMEL (gpt-4o) by 6.25
Terminal-Bench 2.1 (Claude Code): Claude Fable 5.1 (57.9) beat Claude Opus 5 by 6.1
Epoch AI - ECI: GPT-6 (High) (169.23) beat Claude Fable 5 (High) by 6.0
OpenCompass Multimodal - MathVista: BlueLM-3.5-Nano-3B (88.3) beat SenseNova-V6-5-Pro by 5.5
TemporalBench - PSML T1: Single LLM (deepseek-chat) (74.0) beat CAMEL (gpt-4o) by 5.5
LLM Stats (ScreenSpot Pro): GPT-6 (92.7) beat Claude Opus 4.8 by 4.8
SimpleBench: Claude Fable 5.1 (86.6) beat Claude Fable 5 by 4.7
LLM Stats Score: GPT-6 (60.72) beat Claude Fable 5.1 by 3.82
OpenCompass Multimodal - HallusionBench: BlueLM-3.5-Nano-3B (70.5) beat SenseNova-V6-5-Pro by 3.8
TemporalBench - CausalChambers T3: TimeClaw (deepseek-v4-pro) (48.8) beat MetaGPT (gpt-4o) by 3.6
FrontierMath - Tiers 1-3 (v2): GPT-6 (Max) (93.68) beat Claude Fable 5.1 (Max) by 3.5
Epoch AI - Frontiermath Tiers 1 3 V2: GPT-6 (Max) (93.68) beat Claude Fable 5.1 (Max) by 3.5
ParseBench: Claude Fable 5.1 (78.92) beat KDL-Frontier-Parser-nano by 2.56
Vellum - ARC-AGI-2: GPT-6 (95.0) beat GPT-5.6 Sol by 2.5
SimpleQA Verified: GPT-6 (Max) (75.6) beat Gemini 3.1 Pro (Preview) (High) by 2.1
PutnamBench: Humanfia (672.0) beat Humanfia (w/ GPT 5.5) by 2.0
LLM Stats (Terminal-Bench 4.0): GPT-6 (57.7) beat Claude Fable 5.1 by 1.9
TemporalBench - PSML T3: TimeClaw (deepseek-v4-pro) (36.4) beat Single LLM (gpt-4o) by 1.6
TemporalBench - MIMIC T4: CAMEL (deepseek-chat) (31.21) beat Single LLM (gpt-4o) by 1.42
ZeroEval GPQA Diamond: GPT-6 (96.0) beat GPT-5.6 Sol by 1.4
OpenCompass Multimodal - MMBench: BlueLM-3.5-Nano-3B (89.7) beat JT3.5 by 1.4
AA MMMU-Pro: GPT-6 (Max) (86.88) beat Gemini 3.8 Flash (High) by 1.27
AA GPQA Diamond: GPT-6 (xHigh) (96.26) beat Gemini 3.8 Flash (High) by 1.01
OpenCompass Multimodal - MMStar: BlueLM-3.5-Nano-3B (84.3) beat JT3.5 by 0.5
WeirdML: Claude Fable 5.1 (High) (92.3) beat Claude Fable 5 (Max) by 0.36
DeepSWE: GPT-6 (xHigh) (74.1) beat Gemini 3 Flash by 0.3
LLM Stats (BrowseComp): GPT-6 (91.5) beat Kimi K3 by 0.3
AA Omniscience: GPT-6 (High) (43.73) beat Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) by 0.28
TemporalBench - T1: TimeCopilot (deepseek-chat) (50.0) beat Single LLM (gpt-4o) by 0.28
SEAL - Humanity's Last Exam: Claude Fable 5.1 (xHigh) (46.5) beat Gemini 3.1 Pro (Preview) (High) by 0.06
Blueprint-Bench 2: Claude Fable 5.1 (0.419) beat Claude Fable 5 by 0.03

The Aggregate Digest — 2026-09-03

2026-09-03T09:52:51.330313+00:00

Frontier

Best measured model: Claude Fable 5.1 (1814) takes the crown from Claude Mythos 5 (1807)
Best available model: Gemini 3.8 Flash enters at #4 (1781 ELO) on 82 benchmarks
Best available model: Hy4 preview enters at #18 (1733 ELO) on 41 benchmarks

Daily

New Benchmarks (4)

LLM Stats (BioMysteryBench) (Score (%)): leader Claude Opus 5 (90.1), 5 models
LLM Stats mirror of BioMysteryBench, which works models through hard molecular biology problems and reports both the difficult subset and the subset human experts managed to solve.
LLM Stats (Terminal-Bench 4.0) (Score (%)): leader Claude Fable 5.1 (55.8), 12 models
LLM Stats mirror of Terminal-Bench 4.0: 66 containerized terminal tasks weighted toward science-adjacent and frontier engineering work. Relative to earlier releases it lengthens timeouts and raises RAM and CPU on selected tasks, so a failure is more likely the agent’s than the harness’s.
EdiTikZ (Overall Edit Score (0-1)): leader GPT-5.5 (0.69), 14 models
Edit an existing scientific figure the way a co-author asks for it, working on the TikZ source rather than the picture. Scored as the mean of five judged axes — tree-edit distance, drawing similarity, edit accuracy, spec compliance and visual quality — so a model that redraws the figure from scratch does not win on the one axis it happened to satisfy.
DroneCATS (Approaching) (Success Rate (%)): leader Gemini 3.7 Flash (65.0), 8 models
A multimodal model dropped straight into a drone's control loop with its entire action space declared in the prompt, told to fly to a described target. This is the approach task's success rate, and the ceiling is 65 percent — the frontier can see where to go far more reliably than it can get there.

New Models (2)

Gemini 3.8 Flash — ELO 1781, #3
- AA GPQA Diamond: 95.25 (#1/604)
- AA MMMU-Pro: 85.61 (#1/252)
- LLM Stats (LVBench): 87.1 (#1/28)
- LLM Stats (Terminal-Bench 2.1): 89.4 (#1/35)
- Vals AI Finance Agent v2: 61.44 (#1/55)
- BioMysteryBench Human-Difficult: 56.5 (#1/14)
- LVBench: 87.8 (#1/43)
- Finance Agent v2: 61.4 (#1/42)
- LLM Stats (DeepSWE 1.1): 73.7 (#2/30)
- LLM Stats (GDP.pdf): 35.0 (#2/7)
- ...and 75 more
Muse Spark 1.3 — ELO 1774
- Tau3 Banking: 52.37 (#1/196)
- LLM Stats (AutomationBench): 49.4 (#1/17)
- LLM Stats (DeepSWE 1.1): 75.4 (#1/30)
- LLM Stats (Job Bench): 64.9 (#1/8)
- LLM Stats (MRCR v2 (8-needle)): 98.5 (#1/24)
- LLM Stats (Terminal-Bench 2.1): 88.8 (#2/35)
- LLM Stats (OSWorld 2.0): 66.9 (#3/10)
- LLM Stats (GDPval-AA): 1754.0 (#4/9)
- AA GPQA Diamond: 94.14 (#5/604)
- LLM Stats (DeepSearchQA): 89.4 (#5/10)
- ...and 20 more

Top-10 New Scores (144)

Claude Fable 5 on NL2Repo: 70.2 (#2)
Claude Fable 5 on SWE Atlas: 39.0 (#22)
Claude Fable 5.1 on AA-Briefcase: 55.35 (#8)
Claude Fable 5.1 on AA-LCR: 80.0 (#8)
Claude Fable 5.1 on AI Chess Leaderboard (Continuation): 1357.0 (#21)
Claude Fable 5.1 on Agents on Rails: 92.1 (#2)
Claude Fable 5.1 on Arena AI Code: 1765.37 (#1)
Claude Fable 5.1 on Chatbot Arena (Code): 1765.0 (#1)
Claude Fable 5.1 on Chatbot Arena (Text): 1504.0 (#3)
Claude Fable 5.1 on Computer Anthology Terminal Tasks (Claude Code): 55.2 (#1)
Claude Fable 5.1 on Computer Anthology Terminal Tasks (Terminus-2): 62.4 (#1)
Claude Fable 5.1 on Creative Writing (Lechmazur): 4.1 (#1)
Claude Fable 5.1 on CritPt: 29.7 (#211)
Claude Fable 5.1 on Deep20Bench: 12.11 (#2)
Claude Fable 5.1 on Design Arena (3D): 1409.0 (#3)
Claude Fable 5.1 on Design Arena (UI Components): 1377.0 (#2)
Claude Fable 5.1 on Design Arena (Website): 1330.0 (#4)
Claude Fable 5.1 on Epoch AI - Apex Agents: 44.4 (#3)
Claude Fable 5.1 on Epoch AI - Critpt: 31.14 (#2)
Claude Fable 5.1 on Epoch AI - Cursorbench: 73.4 (#1)
Claude Fable 5.1 on Epoch AI - ECI: 162.96 (#8)
Claude Fable 5.1 on Epoch AI - Proofbench: 100.0 (#1)
Claude Fable 5.1 on Epoch AI - Scicode: 62.04 (#1)
Claude Fable 5.1 on KernelBench Hub - Mega: 15.84 (#4)
Claude Fable 5.1 on LMArena WebDev Arena: 1765.37 (#1)
Claude Fable 5.1 on MineBench: 2040.0 (#3)
Claude Fable 5.1 on TaxCalcBench: 38.0 (#4)
Claude Fable 5.1 on WebDev Arena: 1765.37 (#1)
Claude Opus 5 on AA-Briefcase: 53.84 (#10)
Claude Opus 5 on BioMysteryBench Human-Difficult: 49.4 (#3)
Claude Opus 5 on Finance Agent v2: 58.6 (#3)
Claude Opus 5 on LVBench: 75.4 (#9)
Claude Opus 5 on SWE Atlas: 63.2 (#3)
Gemini 3.7 Flash on Agents on Rails: 71.4 (#14)
Gemini 3.7 Flash on Epoch AI - Critpt: 14.29 (#48)
Gemini 3.7 Flash on Epoch AI - Scicode: 57.87 (#8)
Gemini 3.7 Flash on Finance Agent v2: 59.0 (#2)
Gemini 3.8 Flash on AA CritPt: 18.29 (#37)
Gemini 3.8 Flash on AA GDPval: 1545.45 (#35)
Gemini 3.8 Flash on AA GPQA Diamond: 95.25 (#1)
Gemini 3.8 Flash on AA Humanity's Last Exam: 47.82 (#15)
Gemini 3.8 Flash on AA Long Context Reasoning: 82.0 (#3)
Gemini 3.8 Flash on AA MMMU-Pro: 85.61 (#1)
Gemini 3.8 Flash on AA Omniscience: 29.55 (#13)
Gemini 3.8 Flash on AA Omniscience - Business: 45.7 (#23)
Gemini 3.8 Flash on AA Omniscience - Health: 44.6 (#33)
Gemini 3.8 Flash on AA Omniscience - Humanities & Social Sciences: 54.1 (#22)
Gemini 3.8 Flash on AA Omniscience - Law: 60.1 (#12)
Gemini 3.8 Flash on AA Omniscience - Science, Engineering & Mathematics: 53.3 (#17)
Gemini 3.8 Flash on AA Omniscience - Software Engineering (SWE): 69.8 (#41)
Gemini 3.8 Flash on AA SciCode: 54.4 (#30)
Gemini 3.8 Flash on AA-Briefcase: 42.12 (#17)
Gemini 3.8 Flash on AA-Omniscience Accuracy: 54.6 (#24)
Gemini 3.8 Flash on AI Chess Leaderboard (Continuation): 1071.0 (#44)
Gemini 3.8 Flash on AI Chess Leaderboard (Reasoning): 1547.0 (#19)
Gemini 3.8 Flash on Agent Arena: 5.94 (#15)
Gemini 3.8 Flash on Agent Arena - Bash Recovery: 4.46 (#30)
Gemini 3.8 Flash on Agent Arena - Confirmed Success: 11.12 (#9)
Gemini 3.8 Flash on Agent Arena - Praise vs Complaint: 14.78 (#7)
Gemini 3.8 Flash on Agent Arena - Steerability: -1.43 (#36)
Gemini 3.8 Flash on Agent Arena - Tool Hallucination: -0.78 (#28)
Gemini 3.8 Flash on Arena AI Code: 1567.33 (#19)
Gemini 3.8 Flash on Artificial Analysis Intelligence Index: 58.68 (#18)
Gemini 3.8 Flash on BenchLM: 75.4 (#11)
Gemini 3.8 Flash on BenchmarkList ECI: 148.9 (#11)
Gemini 3.8 Flash on BioMysteryBench Human-Solvable: 88.8 (#2)
Gemini 3.8 Flash on Chatbot Arena (Code): 1567.0 (#19)
Gemini 3.8 Flash on Chatbot Arena (Text): 1494.0 (#8)
Gemini 3.8 Flash on Coarena: 1029.7 (#7)
Gemini 3.8 Flash on Coarena - Task Completion: 76.9 (#5)
Gemini 3.8 Flash on Design Arena (Website): 1311.0 (#14)
Gemini 3.8 Flash on Epoch AI - Critpt: 18.29 (#34)
Gemini 3.8 Flash on Epoch AI - Cursorbench: 69.2 (#9)
Gemini 3.8 Flash on Epoch AI - Proofbench: 48.0 (#15)
Gemini 3.8 Flash on Epoch AI - Scicode: 54.4 (#27)
Gemini 3.8 Flash on KernelBench Hub - CUDA: 36.37 (#4)
Gemini 3.8 Flash on KernelBench Hub - Mega: 2.74 (#6)
Gemini 3.8 Flash on LLM Stats (CharXiv-R): 86.2 (#16)
Gemini 3.8 Flash on LLM Stats (DeepSWE 1.1): 73.7 (#2)
Gemini 3.8 Flash on LLM Stats (GDP.pdf): 35.0 (#2)
Gemini 3.8 Flash on LLM Stats (GDPval-AA): 1545.0 (#7)
Gemini 3.8 Flash on LLM Stats (LVBench): 87.1 (#1)
Gemini 3.8 Flash on LLM Stats (OSWorld 2.0): 59.0 (#5)
Gemini 3.8 Flash on LLM Stats Score: 51.79 (#11)
Gemini 3.8 Flash on LM Market Cap LMC Score: 40.0 (#237)
Gemini 3.8 Flash on LMArena Text Arena: 1493.74 (#8)
Gemini 3.8 Flash on LMArena WebDev Arena: 1567.33 (#21)
Gemini 3.8 Flash on MineBench: 1893.0 (#8)
Gemini 3.8 Flash on RuneBench: 9333.0 (#4)
Gemini 3.8 Flash on SvelteBench: 100.0 (#11)
Gemini 3.8 Flash on Tau3 Banking: 45.77 (#13)
Gemini 3.8 Flash on Vals AI (Vals Index): 62.25 (#5)
Gemini 3.8 Flash on Vals AI BioMysteryBench: 62.22 (#7)
Gemini 3.8 Flash on Vals AI Code Migration: 36.55 (#17)
Gemini 3.8 Flash on Vals AI CyberBench: 43.75 (#21)
Gemini 3.8 Flash on Vals AI Excel Modeling: 72.2 (#5)
Gemini 3.8 Flash on Vals AI Finance Agent v2: 61.44 (#1)
Gemini 3.8 Flash on Vals AI GPQA: 94.44 (#4)
Gemini 3.8 Flash on Vals AI Harvey Legal Agent Bench: 10.0 (#9)
Gemini 3.8 Flash on Vals AI Legal Research Bench: 38.94 (#16)
Gemini 3.8 Flash on Vals AI LegalBench: 86.99 (#6)
Gemini 3.8 Flash on Vals AI LiveCodeBench: 89.48 (#3)
Gemini 3.8 Flash on Vals AI MMLU-Pro: 90.22 (#5)
Gemini 3.8 Flash on Vals AI MMMU: 89.08 (#4)
Gemini 3.8 Flash on Vals AI MedCode: 48.13 (#23)
Gemini 3.8 Flash on Vals AI MedScribe: 84.5 (#24)
Gemini 3.8 Flash on Vals AI MortgageTax: 65.34 (#44)
Gemini 3.8 Flash on Vals AI ProgramBench: 71.9 (#7)
Gemini 3.8 Flash on Vals AI ProofBench: 48.0 (#17)
Gemini 3.8 Flash on Vals AI Public Benefits Bench: 65.29 (#12)
Gemini 3.8 Flash on Vals AI SAGE: 35.06 (#57)
Gemini 3.8 Flash on Vals AI SWE-bench Verified: 80.0 (#24)
Gemini 3.8 Flash on Vals AI SkillsBench: 57.98 (#11)
Gemini 3.8 Flash on Vals AI TaxEval v2: 74.45 (#36)
Gemini 3.8 Flash on Vals AI Terminal-Bench 2.1: 81.27 (#4)
Gemini 3.8 Flash on Vals AI Vibe Code Bench: 78.65 (#10)
Gemini 3.8 Flash on VoxelBench: 1816.0 (#12)
Gemini 3.8 Flash on WebDev Arena: 1567.33 (#16)
Muse Spark 1.3 on AA CritPt: 26.0 (#18)
Muse Spark 1.3 on AA GDPval: 1753.92 (#8)
Muse Spark 1.3 on AA GPQA Diamond: 94.14 (#5)
Muse Spark 1.3 on AA Humanity's Last Exam: 49.07 (#11)
Muse Spark 1.3 on AA Long Context Reasoning: 79.33 (#14)
Muse Spark 1.3 on AA MMMU-Pro: 82.02 (#17)
Muse Spark 1.3 on AA Omniscience: 24.93 (#25)
Muse Spark 1.3 on AA Omniscience - Business: 38.3 (#50)
Muse Spark 1.3 on AA Omniscience - Health: 37.4 (#76)
Muse Spark 1.3 on AA Omniscience - Humanities & Social Sciences: 39.9 (#74)
Muse Spark 1.3 on AA Omniscience - Law: 38.3 (#58)
Muse Spark 1.3 on AA Omniscience - Science, Engineering & Mathematics: 46.9 (#49)
Muse Spark 1.3 on AA Omniscience - Software Engineering (SWE): 62.2 (#72)
Muse Spark 1.3 on AA SciCode: 58.56 (#6)
Muse Spark 1.3 on AA-Omniscience Accuracy: 43.83 (#62)
Muse Spark 1.3 on Artificial Analysis Intelligence Index: 62.09 (#6)
Muse Spark 1.3 on BenchmarkList ECI: 147.74 (#16)
Muse Spark 1.3 on Coarena: 1000.0 (#11)
Muse Spark 1.3 on Coarena - Task Completion: 0.0 (#18)
Muse Spark 1.3 on LLM Stats (DeepSearchQA): 89.4 (#5)
Muse Spark 1.3 on LLM Stats (GDPval-AA): 1754.0 (#4)
Muse Spark 1.3 on LLM Stats (OSWorld 2.0): 66.9 (#3)
Muse Spark 1.3 on LLM Stats (Terminal-Bench 2.1): 88.8 (#2)
Muse Spark 1.3 on LM Market Cap LMC Score: 80.8 (#80)
Muse Spark 1.3 on MineBench: 1832.0 (#14)
Muse Spark 1.3 on Tau3 Banking: 52.37 (#1)

New #1 Leaders (32)

Chatbot Arena (Code): Claude Fable 5.1 (Max) (1765.0) beat Claude Opus 5 (Max) by 77.0
WebDev Arena: Claude Fable 5.1 (Max) (1765.37) beat Claude Opus 5 (Max) by 74.73
Arena AI Code: Claude Fable 5.1 (Max) (1765.37) beat Claude Opus 5 (Max) by 74.73
LMArena WebDev Arena: Claude Fable 5.1 (Max) (1765.37) beat Claude Opus 5 (Max) by 74.73
Design Arena (Data Viz): Claude Fable 5.1 (1382.0) beat Kimi K3 by 12.0
LLM2014 Logic 2026-09: Claude Fable 5.1 (xhigh ~Estimated) (81.33) beat GPT-5.6 Sol (xHigh) by 11.3
BioMysteryBench Human-Difficult: Gemini 3.8 Flash (56.5) beat GPT-5.6 Terra by 7.1
SEAL - SWE Atlas - Test Writing: Claude Fable 5.1 (Claude Code) xHigh (67.04) beat Claude Opus 5 (Claude Code) xHigh by 4.82
LLM Stats (Job Bench): Muse Spark 1.3 (64.9) beat Hy4 preview by 3.2
Finance Agent v2: Gemini 3.8 Flash (61.4) beat Claude Opus 5 (Max) by 2.77
Epoch AI - Cursorbench: Claude Fable 5.1 (Max) (73.4) beat Grok 4.6 (xHigh) by 2.6
BioMysteryBench Human-Solvable: Claude Opus 5 (90.1) beat Claude Sonnet 5 by 2.6
NL2Repo: Claude Opus 5 (72.3) beat Claude Opus 4.8 by 2.6
Epoch AI - Apex Agents: Claude Fable 5.1 (Unknown) (47.4) beat Claude Fable 5 by 2.4
LLM Stats (DeepSWE 1.1): Muse Spark 1.3 (75.4) beat GPT-5.6 Sol by 2.4
LVBench: Gemini 3.8 Flash (87.8) beat Gemini 3.7 Flash by 2.4
SWE Atlas: Qwen3.8 Max 0902 (66.3) beat Hy4 preview claude-code by 2.3
SEAL - SWE Atlas - Refactoring: Claude Fable 5.1 (Claude Code) xHigh (56.67) beat Claude Fable 5 (Claude Code) xHigh by 1.91
Epoch AI - Scicode: Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) (62.04) beat Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) by 1.85
LLM Stats (CyberGym): Gemini 3.8 Flash Cyber (86.2) beat GLM-5.3 by 1.7
LLM Stats (MRCR v2 (8-needle)): Muse Spark 1.3 (98.5) beat Gemini 3.7 Flash by 1.5
Computer Anthology Terminal Tasks (Claude Code): Claude Fable 5.1 (High) (55.2) beat Claude Fable 5 (High) by 1.2
Thinkingbox: Claude Opus 5 (66.5) beat GPT-5.4 by 1.14
Tau3 Banking: Muse Spark 1.3 (Max) (52.37) beat Qwen 3.8 Max by 1.03
Epoch AI - Proofbench: Claude Fable 5.1 (Max) (100.0) beat Claude Opus 5 (Max) by 1.0
Vals AI Finance Agent v2: Gemini 3.8 Flash (High) (61.44) beat Muse Spark 1.2 (xHigh) by 0.84
LLM Stats (AutomationBench): Muse Spark 1.3 (49.4) beat GLM-5.3 Flash by 0.6
LLM Stats (Terminal-Bench 2.1): Gemini 3.8 Flash (89.4) beat GPT-5.6 Sol by 0.6
Computer Anthology Terminal Tasks (Terminus-2): Claude Fable 5.1 (High) (62.4) beat Claude Opus 5 (High) by 0.4
AA GPQA Diamond: Gemini 3.8 Flash (High) (95.25) beat Grok 4.6 (High) by 0.3
Gert Labs Rankings: Claude Fable 5.1 (74.46) beat Claude Opus 5 by 0.28
AA MMMU-Pro: Gemini 3.8 Flash (High) (85.61) beat Gemini 3.7 Flash (High) by 0.12

The Aggregate Digest — 2026-09-02

2026-09-02T04:57:11.198104+00:00

Frontier

Best available model: Claude Fable 5.1 (1809) takes the crown from Claude Opus 5 (1786)
Best available model: Claude Fable 5.1 enters at #1 (1809 ELO) on 66 benchmarks

Daily

New Benchmarks (4)

Epoch AI - Frontiermath Tiers 1 3 V2 (Score): leader Claude Fable 5.1 (Max) (90.18), 103 models
The three lower tiers of Epoch's FrontierMath v2, spanning advanced undergraduate mathematics up through early-career research problems. Every question is original and vetted by working mathematicians, and this is the wide end of the board — a hundred-odd models, and the frontier now clears 90.
Epoch AI - Frontiermath Tier 4 V2 (Score): leader Claude Fable 5.1 (Max) (87.8), 54 models
Tier 4 of Epoch's FrontierMath v2: the research-level problems a working mathematician would need hours to solve. Held separately from tiers 1–3 because it is the part that was supposed to stay hard, and the top of the board has moved from single digits to the high eighties in under a year.
MERA v2 (Total score): leader Gemini 3.7 Flash (0.68), 37 models
The 2.0 text suite that replaced MERA v1 as the default board at mera.a-ai.ru: twelve newly written Russian tasks (SAGE, Humor, LIMUR, RUBIN, Riddles, SOB-Hard, Characters, Enantiosemy and four more) scored 0-1 as a total over the set. Answers stay private and submissions are run by the organisers, so the task mix differs from v1 and the two scores are not comparable.
GDP.pdf (Strict Pass Rate (%)): leader GPT-5.6 Sol (Max) (30.7), 34 models
Surge AI multimodal document-reasoning benchmark over native professional PDF pages containing visually structured layouts, tables, charts, and forms across finance, healthcare, legal, engineering, insurance, and related domains.

New Models (1)

Claude Fable 5.1 — ELO 1809, #1
- CursorBench 3.1: 73.4 (#1/50)
- FrontierMath - Tier 4 (v2): 87.8 (#1/54)
- FrontierMath - Tiers 1-3 (v2): 90.18 (#1/100)
- OTIS Mock AIME 2024-25: 100.0 (#1/287)
- Artificial Analysis Intelligence Index: 65.65 (#1/618)
- AA Humanity's Last Exam: 59.13 (#1/590)
- AA SciCode: 62.04 (#1/590)
- AA Omniscience: 43.45 (#1/502)
- AA GDPval: 1853.05 (#1/226)
- AA-Briefcase: 61.52 (#1/31)
- ...and 54 more

Top-10 New Scores (68)

Claude Fable 5 on MILU: 92.9 (#4)
Claude Fable 5.1 on AA CritPt: 31.14 (#2)
Claude Fable 5.1 on AA GDPval: 1853.05 (#1)
Claude Fable 5.1 on AA GPQA Diamond: 93.74 (#5)
Claude Fable 5.1 on AA Humanity's Last Exam: 59.13 (#1)
Claude Fable 5.1 on AA Long Context Reasoning: 80.0 (#7)
Claude Fable 5.1 on AA Omniscience: 43.45 (#1)
Claude Fable 5.1 on AA Omniscience - Business: 57.3 (#2)
Claude Fable 5.1 on AA Omniscience - Health: 57.4 (#1)
Claude Fable 5.1 on AA Omniscience - Humanities & Social Sciences: 66.1 (#1)
Claude Fable 5.1 on AA Omniscience - Law: 68.6 (#1)
Claude Fable 5.1 on AA Omniscience - Science, Engineering & Mathematics: 62.0 (#1)
Claude Fable 5.1 on AA Omniscience - Software Engineering (SWE): 92.4 (#1)
Claude Fable 5.1 on AA SciCode: 62.04 (#1)
Claude Fable 5.1 on AA-Briefcase: 61.52 (#1)
Claude Fable 5.1 on AA-Omniscience Accuracy: 67.23 (#1)
Claude Fable 5.1 on AI Chess Leaderboard (Reasoning): 1428.0 (#28)
Claude Fable 5.1 on ARC-AGI-1: 97.5 (#9)
Claude Fable 5.1 on ARC-AGI-2: 90.0 (#5)
Claude Fable 5.1 on Artificial Analysis Intelligence Index: 65.65 (#1)
Claude Fable 5.1 on BenchLM: 82.7 (#1)
Claude Fable 5.1 on BenchmarkList ECI: 157.38 (#2)
Claude Fable 5.1 on Chess Puzzles (Epoch AI): 47.0 (#11)
Claude Fable 5.1 on Coarena: 1002.8 (#9)
Claude Fable 5.1 on Coarena - Task Completion: 29.4 (#16)
Claude Fable 5.1 on CursorBench 3.1: 73.4 (#1)
Claude Fable 5.1 on Epoch AI - Mystery Game Puzzles: 58.0 (#2)
Claude Fable 5.1 on FrontierMath - Tier 4 (v2): 87.8 (#1)
Claude Fable 5.1 on FrontierMath - Tiers 1-3 (v2): 90.18 (#1)
Claude Fable 5.1 on Harvey Legal Agent Benchmark: 19.09 (#2)
Claude Fable 5.1 on LLM Stats (AutomationBench): 31.4 (#5)
Claude Fable 5.1 on LLM Stats Score: 54.8 (#5)
Claude Fable 5.1 on LM Market Cap LMC Score: 97.1 (#1)
Claude Fable 5.1 on MILU: 93.0 (#2)
Claude Fable 5.1 on OTIS Mock AIME 2024-25: 100.0 (#1)
Claude Fable 5.1 on RuneBench: 9813.0 (#3)
Claude Fable 5.1 on Senior SWE-Bench: 34.7 (#1)
Claude Fable 5.1 on SimpleQA Verified: 70.8 (#2)
Claude Fable 5.1 on Tau3 Banking: 47.22 (#7)
Claude Fable 5.1 on Toolathlon: 77.8 (#6)
Claude Fable 5.1 on Vals AI (Vals Index): 67.87 (#1)
Claude Fable 5.1 on Vals AI Excel Modeling: 76.67 (#1)
Claude Fable 5.1 on Vals AI Finance Agent v2: 58.88 (#3)
Claude Fable 5.1 on Vals AI GPQA: 93.43 (#8)
Claude Fable 5.1 on Vals AI Harvey Legal Agent Bench: 6.67 (#18)
Claude Fable 5.1 on Vals AI Legal Research Bench: 55.29 (#2)
Claude Fable 5.1 on Vals AI LegalBench: 88.51 (#2)
Claude Fable 5.1 on Vals AI LiveCodeBench: 90.52 (#1)
Claude Fable 5.1 on Vals AI MMLU-Pro: 92.38 (#1)
Claude Fable 5.1 on Vals AI MMMU: 90.64 (#1)
Claude Fable 5.1 on Vals AI MedCode: 53.51 (#7)
Claude Fable 5.1 on Vals AI MedScribe: 91.29 (#1)
Claude Fable 5.1 on Vals AI MortgageTax: 70.79 (#2)
Claude Fable 5.1 on Vals AI ProofBench: 100.0 (#1)
Claude Fable 5.1 on Vals AI Public Benefits Bench: 74.9 (#2)
Claude Fable 5.1 on Vals AI ReverseEngBench: 22.9 (#2)
Claude Fable 5.1 on Vals AI SAGE: 48.53 (#25)
Claude Fable 5.1 on Vals AI SkillsBench: 61.55 (#5)
Claude Fable 5.1 on Vals AI TaxEval v2: 75.96 (#9)
Claude Fable 5.1 on Vals AI Terminal-Bench 2.1: 85.02 (#2)
Claude Fable 5.1 on Vals AI Vibe Code Bench: 90.26 (#2)
Claude Mythos 5 on Toolathlon: 79.3 (#4)
Claude Mythos Preview on BenchmarkList ECI: 155.94 (#3)
Claude Opus 5 on AA-Omniscience Net Score: 49.0 (#4)
Claude Opus 5 on MILU: 92.1 (#7)
Claude Opus 5 on Toolathlon: 80.6 (#2)
GPT-5.6 Sol on AutomationBench: 26.3 (#5)
GPT-5.6 Sol on ProteinGym Hard: 35.5 (#9)

New #1 Leaders (25)

LLM Stats (GDPval-AA): Claude Fable 5.1 (1853.0) beat GLM-5.3 Flash by 80.0
AA GDPval: Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) (1853.05) beat Claude Opus 5 (Adaptive Reasoning, Max Effort) by 29.11
LLM Stats (OSWorld 2.0): Claude Fable 5.1 (77.9) beat Claude Opus 5 by 7.3
AA Omniscience - Health: Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) (57.4) beat Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) by 3.68
AA Humanity's Last Exam: Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) (59.13) beat Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) by 3.66
AA-Briefcase: Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) (61.52) beat Claude Opus 5 (Adaptive Reasoning, Xhigh Effort) by 3.54
Vals AI Excel Modeling: Claude Fable 5.1 (Max) (76.67) beat Claude Fable 5 (Max) by 3.0
CursorBench 3.1: Claude Fable 5.1 (Max) (73.4) beat Grok 4.6 (xHigh) by 2.6
Artificial Analysis Intelligence Index: Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) (65.65) beat Claude Opus 5 (Adaptive Reasoning, Max Effort) by 2.6
AA Omniscience - Law: Claude Fable 5.1 (Adaptive Reasoning, Xhigh Effort, Default Fallback) (68.6) beat Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) by 2.4
Mercor APEX: Claude Fable 5.1 (47.4) beat Claude Fable 5 by 2.4
AA-Omniscience Accuracy: Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) (67.23) beat Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) by 1.88
AA SciCode: Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) (62.04) beat Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) by 1.85
AA Omniscience - Humanities & Social Sciences: Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) (66.1) beat Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) by 1.8
AA Omniscience - Software Engineering (SWE): Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) (92.4) beat Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) by 1.3
FrontierMath - Tiers 1-3 (v2): Claude Fable 5.1 (Max) (90.18) beat GPT-5.6 Sol (Max) by 1.06
AutomationBench: Claude Fable 5.1 with Opus 5 Fallback (Max) (31.4) beat Gemini 3.7 Flash (High) by 1.0
Vals AI MMLU-Pro: Claude Fable 5.1 (Max) (92.38) beat Claude Opus 5 (Max) by 0.79
Vals AI MMMU: Claude Fable 5.1 (Max) (90.64) beat Claude Opus 5 (Max) by 0.76
Vals AI LiveCodeBench: Claude Fable 5.1 (Max) (90.52) beat Claude Fable 5 (Max) by 0.74
AA Omniscience - Science, Engineering & Mathematics: Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) (62.0) beat Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) by 0.72
Vals AI (Vals Index): Claude Fable 5.1 (Max) (67.87) beat Claude Opus 5 (Max) by 0.66
Vals AI MedScribe: Claude Fable 5.1 (Max) (91.29) beat Claude Opus 5 (Max) by 0.31
Vellum - Humanity's Last Exam: Claude Fable 5.1 (65.0) beat Claude Opus 5 by 0.3
AA Omniscience: Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback) (43.45) beat Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) by 0.15

The Aggregate Digest — 2026-09-01

2026-09-01T05:45:14.477929+00:00

Daily

New Benchmarks (1)

LLM2014 Logic 2026-09 (Median Score): leader GPT-5.6 Sol (xHigh) (70.03), 43 models
September 2026 monthly snapshot of the LLM2014 project public leaderboard, scoring models on logic and reasoning problems from its continuously updated test set.

Top-10 New Scores (3)

Claude Mythos 5 on ExploitGym: 247.0 (#2)
Claude Opus 5 on ExploitGym: 191.0 (#3)
Gemini 3.7 Flash on Vals AI ReverseEngBench: 4.58 (#3)

The Aggregate Digest — 2026-08-31

2026-08-31T09:08:07.449360+00:00

Daily

New Benchmarks (3)

LLM Stats (SWE-Marathon) (Score (%)): leader GLM-5.3 (42.5), 5 models
llm-stats.com’s mirror of SWE-Marathon, the ultra-long-horizon software-engineering benchmark whose tasks run to building compilers, optimizing kernels and standing up production-grade services. It measures whether an agent holds quality across an extremely long trajectory; the roster here is self-reported and much smaller than the first-party board.
LLM Stats (Vision2Web) (Score (%)): leader Qwen 3.8 Max (69.0), 5 models
llm-stats.com’s mirror of Vision2Web, which asks multimodal models to turn a visual design or screenshot into a working web page and scores the end-to-end design-to-code result. Scores are self-reported and the roster is currently almost entirely one vendor’s models, so it reads as a vendor-reported slice rather than a broad comparison.
BenchmarkList ECI (Capability Index (ECI)): leader Claude Mythos 5 (162.93), 506 models
benchmarklist.com's site-wide capability index, the primary ranking score of its model directory. Despite the “ECI” label it is benchmarklist's own fit rather than Epoch's index republished — it correlates at about r=0.74 with Epoch's ECI on their shared models, with per-model gaps up to ~41 points, and scores far more models. Like LLM Stats Score and BenchGecko Score it is one aggregator's fused view, not an independent measurement.

New #1 Leaders (3)

LLM Stats (Job Bench): Hy4 preview (61.7) beat Qwen3.8-Flash-Next by 6.0
LLM Stats (WideSearch): Hy4 preview (83.9) beat Qwen 3.8 Max by 2.0
LLM Stats (CritPT): Hy4 preview (16.9) beat GLM-5.2 by 0.2

The Aggregate Digest — 2026-08-30

2026-08-30T10:45:53.611904+00:00

Daily

Top-10 New Scores (15)

Claude Opus 5 on Appwrite Arena (With Skills): 97.4 (#4)
Claude Opus 5 on Appwrite Arena (Without Skills): 97.4 (#3)
Claude Opus 5 on LLM2014 Logic 2026-07: 71.88 (#4)
Claude Opus 5 on LLM2014 Logic 2026-08: 64.74 (#4)
Claude Opus 5 on React Native Evals: 88.11 (#1)
Claude Opus 5 on Terminal-Bench 2.1: 51.8 (#1)
Claude Opus 5 on Terminal-Bench 2.1 (Claude Code): 51.8 (#1)
GPT-5.6 Sol on Appwrite Arena (With Skills): 96.9 (#8)
GPT-5.6 Sol on Appwrite Arena (Without Skills): 95.5 (#5)
GPT-5.6 Sol on Epoch AI - Mystery Game Puzzles: 33.0 (#17)
GPT-5.6 Sol on Terminal-Bench 2.1: 37.3 (#4)
Gemini 3.7 Flash on Appwrite Arena (With Skills): 96.0 (#13)
Gemini 3.7 Flash on Appwrite Arena (Without Skills): 93.6 (#10)
Gemini 3.7 Flash on Featherbench: 93.0 (#10)
Gemini 3.7 Flash on Featherbench - Rubric Quality: 8.8 (#8)

New #1 Leaders (3)

SWE-Milestone: GLM-5.3 (58.75) beat Claude Opus 4.8 (Max, 1M) by 6.91
Featherbench: GLM-5.3 (100.0) beat Gemini 3.6 Flash by 4.0
Appwrite Arena (With Skills): Muse Spark 1.2 (97.8) beat GPT-5.5 by 0.1

Weekly

New Benchmarks (16)

WorkSurface-Bench (Aggregate Score (0-1)): leader Gemini 3.1 Pro (Preview) (0.7649), 4 models
Enterprise-agent benchmark that splits a workspace into three knowledge surfaces — retrievable documents, structured tables and a dependency graph — and asks whether an agent routes each of 1,151 atomic questions to the right one, acquires the evidence and answers correctly. The published score is the all-tools ReAct setting, where every surface is exposed and the model has to choose; the aggregate weights answer 0.35, evidence 0.30, routing 0.25 and efficiency 0.10.
Pre-Flight (Accuracy (%)): leader GPT-5.5 (82.7), 42 models
Aviation operational knowledge: regulations, procedures and the documentation airline operations actually run on. Built because general benchmarks say nothing about whether a model reasons safely inside a domain where being confidently wrong grounds aircraft.
PredAct-Bench (Overall (%)): leader Gemini 3 Flash (63.4), 12 models
Tool-augmented dialogue where the tools lie. Most task-oriented benchmarks assume perfect tool output; this one injects controlled noise and scores whether the model still reaches the right multi-step decision in education, health and finance settings.
SaliTrap (Overall (%)): leader Claude Opus 4.7 (62.5), 12 models
Salience bias in everyday commonsense: models have learned to weight explicit stated conditions so heavily that an obvious unstated one gets ignored. The traps are ordinary situations where a human would never need the condition spelled out.
ContractScrub (F1 (0-1)): leader Gemini 3.1 Pro (Preview) (0.66), 10 models
Contract scrubbing: the final review pass over a transactional agreement, hunting errors and internal inconsistencies. Scored by F1 because both missing a real defect and inventing one are failures a lawyer would notice.
EpiBench (Accuracy (%)): leader Grok 4.20 (33.6), 9 models
Epitope understanding for antibody drug discovery: where an antibody binds an antigen, which determines functional blockade and escape resistance downstream. Scores sit in the twenties and thirties, so this is a long way from saturated.
CARE-Bench (Overall (%)): leader Llama 4 Scout (50.4), 10 models
Patient-facing triage: given a symptom question asked before any clinician contact, what should the user do next. Source-grounded and sequential, because the safety-relevant output is the recommended action rather than the explanation.
OrchBench (Score (0-1)): leader GPT-5.5 (0.81), 9 models
Multi-agent orchestration plans judged in isolation through deterministic simulation, so plan quality is separated from the worker models that would execute it. End-to-end evaluation conflates the two and cannot say which one failed.
Patient-Context Ambiguity (Accuracy (%)): leader Claude Opus 4.8 (91.8), 5 models
Short underspecified health queries that are linguistically clear but support several plausible answers depending on patient context. Scores whether the model resolves the ambiguity rather than confidently picking one reading.
Skill Coverage (Task Success Rate (%)): leader GPT-5.5 (53.26), 5 models
Task success as a function of which skills an agent has available, built to test whether more skills actually help. The paper's answer is that they often do not.
ORCA-bench (Root-Cause Accuracy (%)): leader Claude Sonnet 4.6 (30.9), 5 models
Root-cause analysis of incidents: given the symptoms and the telemetry, name what actually broke. Accuracy sits between 15 and 31 percent, so the gap to a competent on-call engineer is still wide.
OpenUI Generative UI Benchmark (Structural Validity (%)): leader Grok 4.6 (99.5), 30 models
OpenUI Generative UI Benchmark (Thesys) — generated user interfaces scored by the shipped OpenUI parser: a screen counts as valid only if it parses, has a root, resolves every reference, invents no components, uses valid props and is not truncated. Four generations per brief across 46 briefs; headline is structural validity percent.
Benchmarks.bio - BioSecBench-Function (Pass Rate (%)): leader Claude Opus 5 (50.3), 11 models
BioSecBench-Function (benchmarks.bio) — genomic function prediction under biosecurity framing: agentic tasks that ask a model to reason from sequence to function across 111 evaluations, scored as pass rate over gradeable trials with refusals reported separately.
LLM Stats (CoWorkBench) (Score (%)): leader Qwen 3.8 Max (74.8), 5 models
LLM Stats mirror of CoWorkBench, Qwen's cowork benchmark for long-horizon office and productivity tasks; the published roster is small and Qwen-heavy.
LLM Stats (GDPval-AA) (ELO): leader GLM-5.3 Flash (1773.0), 5 models
GDP validation benchmark testing economic data retrieval and reasoning, measuring accuracy on GDP-related factual questions.
LLM Stats (Job Bench) (Score (%)): leader Qwen3.8-Flash-Next (55.7), 5 models
LLM Stats mirror of Job Bench, scoring AI agents on realistic professional tasks that require multi-step planning.

New Models (201)

Claude Opus 5 — ELO 1791, #1
- React Native Evals: 88.11 (#1/25)
- OpenCompass LLM - Knowledge: 94.1 (#1/15)
- OpenCompass LLM - Reasoning: 68.4 (#1/15)
- OpenCompass LLM - Math: 77.3 (#1/15)
- OpenCompass Knowledge - Social Science: 94.6 (#1/15)
- OpenCompass Knowledge - Science: 97.5 (#1/15)
- OpenCompass Reasoning - Academic: 56.7 (#1/15)
- OpenCompass Math - Competition: 74.7 (#1/15)
- Terminal-Bench 2.1: 51.8 (#1/10)
- Terminal-Bench 2.1 (Claude Code): 51.8 (#1/5)
- ...and 26 more
Gemini 3.7 Flash — ELO 1782, #3
- OpenCompass Knowledge - Engineering: 95.8 (#1/15)
- OpenCompass Knowledge - Common Sense: 92.9 (#1/15)
- OpenCompass Math - College: 85.6 (#1/15)
- HieroglyphBench: 55.0 (#1/23)
- OpenCompass LLM - Knowledge: 94.0 (#2/15)
- OpenCompass LLM - Math: 76.8 (#2/15)
- AI for Education Visual Maths - Geometry: 90.77 (#2/76)
- OpenCompass Knowledge - Humanities: 94.2 (#3/15)
- BoundaryBench (Unrestricted): 79.8 (#3/14)
- AI for Education Visual Maths: 89.45 (#3/76)
- ...and 27 more
Claude Fable 5 — ELO 1777, #4
- PostTrainBench: 41.79 (#1/12)
- AI for Education Visual Maths - Number and Operations: 86.49 (#2/76)
- AI for Education Visual Maths - Statistics and Probability: 71.43 (#3/76)
- AI for Education Visual Maths: 86.92 (#4/76)
- AI for Education Visual Maths - Geometry: 83.85 (#5/76)
- AI for Education Visual Maths - Measurement: 97.3 (#5/76)
- OSWorld-Verified: 85.0 (#5/132)
- MCP Atlas: 83.3 (#9/51)
- AI for Education Visual Maths - Algebra: 92.31 (#16/76)
GPT-5.6 Sol — ELO 1776, #5
- Chatbot Arena (Search): 1257.0 (#1/34)
- AI for Education Visual Maths: 91.14 (#1/76)
- AI for Education Visual Maths - Geometry: 90.77 (#1/76)
- AI for Education Visual Maths - Number and Operations: 89.19 (#1/76)
- WildClawBench: 67.2 (#1/66)
- OpenCompass LLM - Code: 96.3 (#2/15)
- OpenCompass Reasoning - Academic: 55.8 (#2/15)
- OpenCompass Code - Comprehensive: 96.3 (#2/15)
- AI for Education Visual Maths - Algebra: 96.15 (#2/76)
- AI for Education Visual Maths - Measurement: 97.3 (#2/76)
- ...and 20 more
Kimi K3 — ELO 1756, #7
- AA-LCR: 82.6667 (#2/210)
- NatureBench: 14.44 (#4/16)
- OpenCompass Knowledge - Social Science: 90.4 (#5/15)
- OpenCompass Math - Competition: 68.9 (#5/15)
- CAIS Vision Capabilities Index: 63.2 (#5/38)
- PostTrainBench: 31.96 (#6/12)
- OpenCompass LLM - Math: 74.1 (#6/15)
- OpenCompass Knowledge - Science: 92.5 (#6/15)
- OpenCompass Reasoning - Academic: 51.7 (#6/15)
- Appwrite Arena (Without Skills): 95.0 (#6/24)
- ...and 18 more
Grok 4.6 — ELO 1754, #8
- Benchmarks.bio - BioSecBench-Refusal: 64.2 (#1/18)
- OpenCompass Knowledge - Humanities: 95.8 (#1/15)
- OpenCompass Knowledge - Social Science: 92.5 (#2/15)
- OpenCompass Math - College: 83.8 (#2/15)
- Appwrite Arena (With Skills): 97.8 (#2/24)
- OpenCompass LLM - Knowledge: 93.8 (#3/15)
- OpenCompass Knowledge - Science: 95.8 (#3/15)
- OpenCompass Knowledge - Engineering: 94.2 (#4/15)
- OpenCompass Knowledge - Common Sense: 90.8 (#5/15)
- Agent Arena - Confirmed Success: 13.17 (#6/54)
- ...and 25 more
Muse Spark 1.2 — ELO 1754, #9
- Appwrite Arena (With Skills): 97.8 (#1/24)
- AA-LCR: 83.3333 (#1/210)
- EQ-Bench Longform Writing: 81.5 (#6/130)
- Appwrite Arena (Without Skills): 94.8 (#7/24)
- SimpleQA Verified: 60.3 (#9/66)
- Epoch AI - Proofbench: 43.0 (#13/24)
- CritPt: 17.7 (#21/284)
- DuelLab Overall: 52.8 (#23/121)
- AIIQ Composite IQ: 120.0 (#23/130)
- WebDev Arena: 1533.68 (#27/116)
- ...and 2 more
Qwen 3.8 Max — ELO 1750, #10
- OpenCompass Math - Competition: 70.8 (#2/15)
- OpenCompass LLM - Code: 95.5 (#3/15)
- OpenCompass Code - Comprehensive: 95.5 (#3/15)
- OpenCompass LLM - Reasoning: 65.2 (#4/15)
- OpenCompass LLM - Math: 75.0 (#4/15)
- OpenCompass Reasoning - Common: 78.1 (#4/15)
- OpenCompass Reasoning - Academic: 52.3 (#4/15)
- NatureBench: 11.11 (#6/16)
- OpenCompass Math - College: 79.2 (#7/15)
- OpenCompass Knowledge - Social Science: 87.9 (#8/15)
- ...and 9 more
GPT-5.5 — ELO 1748, #11
- OpenCompass LLM - Code: 96.5 (#1/15)
- OpenCompass Code - Comprehensive: 96.5 (#1/15)
- OpenCompass LLM - Reasoning: 66.9 (#2/15)
- OpenCompass Reasoning - Common: 78.6 (#3/15)
- OpenCompass Reasoning - Academic: 55.2 (#3/15)
- GeneBench: 25.0 (#3/16)
- OpenCompass Math - Competition: 68.9 (#4/15)
- OpenCompass Knowledge - Social Science: 90.0 (#6/15)
- SWE Atlas: 45.43 (#6/29)
- SimpleQA Verified: 63.0 (#8/66)
- ...and 13 more
Gemini 3.6 Flash — ELO 1747, #13
- AI for Education Visual Maths - Measurement: 97.3 (#1/76)
- Crosby micro1 RedlineBench: 51.0 (#4/11)
- AI for Education Visual Maths - Statistics and Probability: 71.43 (#6/76)
- AI for Education Visual Maths - Number and Operations: 70.27 (#10/76)
- AI for Education Visual Maths: 78.48 (#12/76)
- AA-LCR: 79.0 (#12/210)
- AI for Education Visual Maths - Geometry: 73.85 (#15/76)
- AIIQ Composite IQ: 123.0 (#20/130)
- AI for Education Visual Maths - Algebra: 88.46 (#23/76)
Qwen3.8-Flash-Next — ELO 1747, #14
- LLM Stats (RealWorldQA): 88.5 (#1/30)
- RealWorldQA: 88.5 (#1/48)
- LLM Stats (AndroidWorld): 84.5 (#2/6)
- LLM Stats (ERQA): 72.3 (#2/25)
- LLM Stats (MathVision): 95.7 (#2/34)
- LLM Stats (Agents' Last Exam): 51.2 (#3/13)
- LLM Stats (CharXiv-R): 90.6 (#4/53)
- LLM Stats (Toolathlon): 73.5 (#4/39)
- LLM Stats (LVBench): 76.6 (#5/26)
- LVBench: 76.6 (#5/40)
- ...and 26 more
Gemini 3.1 Pro (Preview) — ELO 1737, #15
- SimpleQA Verified: 73.5 (#1/66)
- Epoch AI - Enigma Eval: 36.78 (#3/46)
- DeepResearchBench: 47.8 (#22/41)
- ForecastBench: 64.0 (#115/300)
Qwen 3.7 Max — ELO 1736, #16
- LLM GPU Kernel Generation: 97.1 (#3/6)
- StructureClaw (Automatic Workflow): 89.3 (#4/10)
GPT-5.6 Terra — ELO 1732, #17
- Crosby micro1 RedlineBench: 49.3 (#6/11)
- Appwrite Arena (Without Skills): 93.5 (#11/24)
- AA-Briefcase: 34.14 (#15/26)
- Appwrite Arena (With Skills): 95.2 (#19/24)
- APEX-Agents-AA: 0.39 (#31/56)
- Epoch AI - Mystery Game Puzzles: 19.0 (#47/110)
GLM-5.3 — ELO 1732, #18
- SWE-Milestone: 58.75 (#1/27)
- Humanity's Last Exam (Self-Reported, With Tools): 62.5 (#1/27)
- Featherbench: 100.0 (#1/17)
- SecIT Bench (Pydantic AI): 81.44 (#1/16)
- KernelBench Hub - Hard: 63.19 (#2/9)
- OpenCompass Knowledge - Engineering: 95.4 (#2/15)
- ProphetArena: 0.9541 (#3/20)
- Terminal-Bench 2.1: 41.8 (#3/10)
- Terminal-Bench 2.1 (Claude Code): 41.8 (#3/5)
- Featherbench - Rubric Quality: 9.3 (#3/17)
- ...and 42 more
Claude Opus 4.8 — ELO 1729, #19
- OpenCompass Reasoning - Common: 80.6 (#1/15)
- PostTrainBench: 33.84 (#4/12)
- LLM2014 Logic 2026-05: 68.32 (#5/43)
- LLM2014 Logic 2026-06: 68.32 (#5/43)
- OpenCompass LLM - Reasoning: 64.1 (#6/15)
- OpenCompass Math - Competition: 68.3 (#8/15)
- OpenCompass LLM - Math: 71.5 (#9/15)
- OpenCompass Knowledge - Humanities: 90.4 (#10/15)
- OpenCompass Knowledge - Social Science: 87.5 (#10/15)
- OpenCompass Knowledge - Science: 89.6 (#10/15)
- ...and 15 more
Gemini 3.5 Flash — ELO 1729, #20
- AA-LCR: 81.0 (#5/210)
- AI for Education Visual Maths - Measurement: 94.59 (#8/76)
- MCP Atlas: 83.6 (#8/51)
- AI for Education Visual Maths - Number and Operations: 70.27 (#11/76)
- AI for Education Visual Maths - Geometry: 74.62 (#14/76)
- AI for Education Visual Maths: 75.95 (#17/76)
- AI for Education Visual Maths - Statistics and Probability: 42.86 (#18/76)
- CritPt: 10.9 (#35/284)
- AI for Education Visual Maths - Algebra: 73.08 (#40/76)
DeepSeek V4 Pro (0813) — ELO 1728, #21
- AA-Briefcase: 43.52 (#9/26)
- CritPt: 18.0 (#19/284)
- AA-LCR: 75.3333 (#36/210)
- Bullshit Benchmark: 32.7 (#65/188)
- Epoch AI - ECI: 155.18 (#93/507)
- LM Market Cap LMC Score: 40.0 (#228/403)
Grok 4.5 — ELO 1726, #22
- Chatbot Arena (Search): 1213.0 (#8/34)
- PostTrainBench: 23.45 (#10/12)
- AI for Education Visual Maths - Geometry: 75.38 (#12/76)
- FrontierCode: 56.6 (#13/47)
- AI for Education Visual Maths: 77.64 (#14/76)
- AI for Education Visual Maths - Measurement: 94.59 (#14/76)
- AI for Education Visual Maths - Statistics and Probability: 42.86 (#17/76)
- AI for Education Visual Maths - Algebra: 92.31 (#19/76)
- AI for Education Visual Maths - Number and Operations: 64.86 (#20/76)
Seed 2.1 Pro — ELO 1726, #23
- LLM2014 Logic 2026-06: 60.15 (#8/43)
- LLM2014 Logic 2026-07: 49.35 (#13/45)
- LLM2014 Logic 2026-08: 45.78 (#18/46)
GLM-5.3 Flash — ELO 1725, #24
- LLM Stats (AutomationBench): 48.8 (#1/14)
- LLM Stats (MVBench): 77.8 (#1/18)
- LLM Stats (Toolathlon): 78.4 (#1/39)
- LLM Stats (Artificial Analysis): 57.0 (#2/8)
- Toolathlon: 78.4 (#2/41)
- AA GDPval: 1764.66 (#3/219)
- Humanity's Last Exam (Self-Reported, With Tools): 55.3 (#3/27)
- KernelBench Hub - Mega: 13.64 (#4/5)
- LLM Stats (NL2Repo): 56.3 (#4/20)
- Chatbot Arena (Code): 1634.0 (#5/121)
- ...and 58 more
Gemini 3 Pro — ELO 1720, #25
- Arena AI Code: 1437.8 (#50/113)
Gemini 3 Pro (Preview) — ELO 1716, #27
- DeepResearchBench: 46.3 (#28/41)
- Epoch AI - ECI: 153.06 (#134/507)
Qwen3.8 2.4T A95B — ELO 1714, #28
- AI for Education SEND: 86.24 (#12/235)
- CritPt: 20.0 (#16/284)
- AI for Education Pedagogy - Maths: 91.27 (#17/243)
- OpenRouter Tau2-Bench Airline: 76.7 (#19/118)
- AIIQ Composite IQ: 119.0 (#26/130)
- AI for Education Pedagogy: 87.99 (#33/243)
- AI for Education Pedagogy - Science: 91.8 (#33/243)
- OpenRouter GPQA Diamond: 84.3 (#34/119)
- AA-LCR: 75.3333 (#34/210)
- AI for Education Pedagogy - Primary: 92.02 (#36/243)
- ...and 3 more
Claude Opus 4.7 — ELO 1712, #29
- PostTrainBench: 28.56 (#8/12)
- MCP Atlas: 79.1 (#19/51)
- CritPt: 5.1 (#61/284)
GPT-5.4 — ELO 1709, #30
- SWE Atlas: 40.8 (#11/29)
- PostTrainBench: 19.0 (#12/12)
- OSWorld-Verified: 75.0 (#25/132)
- Epoch AI - Mystery Game Puzzles: 28.0 (#26/110)
- APEX-Agents-AA: 0.33 (#35/56)
- CyberGym: 79.0 (#36/54)
- CritPt: 7.4 (#48/284)
- Arena AI Code: 1390.56 (#69/113)
GPT-5.6 Luna — ELO 1708, #31
- Crosby micro1 RedlineBench: 55.5 (#3/11)
- Appwrite Arena (Without Skills): 92.9 (#12/24)
- AA-Briefcase: 37.98 (#13/26)
- CritPt: 20.6 (#15/284)
- Appwrite Arena (With Skills): 95.2 (#18/24)
- Guesswork 2026-08: 1.0249 (#22/33)
- APEX-Agents-AA: 0.36 (#33/56)
- FrontierMath - Tiers 1-3 (v2): 41.4 (#44/92)
- Epoch AI - Mystery Game Puzzles: 17.0 (#56/110)
GPT-5.3 Codex — ELO 1706, #32
- Arena AI Code: 1408.38 (#59/113)
Claude Sonnet 5 — ELO 1705, #33
- LLM2014 Logic 2026-07: 43.14 (#16/45)
- DuelLab Overall: 54.3 (#19/121)
- LLM2014 Logic 2026-08: 39.57 (#21/46)
- Epoch AI - ECI: 155.87 (#92/507)
GPT-5.2 Codex — ELO 1705, #34
- Arena AI Code: 1338.28 (#87/113)
Qwen 3.8 27B — ELO 1704, #35
- WebDev Arena: 1594.78 (#9/116)
- Agents on Rails: 76.2 (#10/16)
- LMArena WebDev Arena: 1594.78 (#10/30)
- Agent Arena - Confirmed Success: 8.05 (#11/54)
- LVBench: 72.4 (#12/40)
- Bullshit Benchmark: 76.4 (#13/188)
- AI for Education Pedagogy - Science: 93.44 (#13/243)
- Toolathlon: 67.1 (#15/41)
- Epoch AI - Proofbench: 16.0 (#21/24)
- AA-LCR: 77.3333 (#23/210)
- ...and 32 more
Kimi K2.7 Code — ELO 1703, #36
- WildClawBench: 46.9 (#26/66)
Claude Opus 4.6 — ELO 1702, #38
- LLM2014 Logic 2026-02: 78.02 (#1/46)
- VitaBench 2.0: 50.3 (#1/35)
- LLM2014 Logic 2026-03: 77.23 (#2/42)
- LLM2014 Logic 2026-04: 77.23 (#3/42)
- LLM2014 Logic 2026-05: 76.48 (#3/43)
- DeepResearchBench: 51.4 (#8/41)
- LLM2014 Logic 2026-06: 40.16 (#17/43)
- LVBench: 63.0 (#18/40)
- RealWorldQA: 73.9 (#20/48)
- CritPt: 2.8 (#85/284)
Gemini 3 Flash — ELO 1702, #39
- DeepResearchBench: 49.0 (#15/41)
- OmniDocBench 1.5: 90.37 (#20/73)
- Arena AI Code: 1437.85 (#49/113)
Seed 2.1 Turbo — ELO 1702, #41
- WildClawBench: 62.8 (#3/66)
- MCP Atlas: 80.3 (#15/51)
- Toolathlon: 49.1 (#36/41)
Gemini 3 Flash (Preview) — ELO 1701, #42
- SimpleQA Verified: 66.8 (#5/66)
- DeepResearchBench: 49.8 (#11/41)
- CritPt: 1.4 (#106/284)
GLM-5.2 — ELO 1697, #44
- SWE Atlas: 48.12 (#4/29)
- PostTrainBench: 31.7 (#7/12)
- MCP Atlas: 77.8 (#21/51)
- CritPt: 16.7 (#25/284)
- FrontierMath - Tiers 1-3 (v2): 54.74 (#33/92)
- Epoch AI - Mystery Game Puzzles: 15.0 (#67/110)
- Chess Puzzles (Epoch AI): 14.0 (#106/182)
- OTIS Mock AIME 2024-25: 75.56 (#113/256)
Qwen 3.7 Plus — ELO 1697, #45
- RealWorldQA: 86.9 (#2/48)
- IMO-AnswerBench: 86.0 (#7/15)
- Toolathlon: 50.6 (#29/41)
- FrontierMath - Tiers 1-3 (v2): 34.39 (#49/92)
- Epoch AI - Mystery Game Puzzles: 17.0 (#53/110)
Hy3 — ELO 1694, #46
- LLM2014 Logic 2026-07: 44.6 (#14/45)
- NL2Repo: 45.6 (#15/37)
- MCP Atlas: 79.1 (#18/51)
- LLM2014 Logic 2026-08: 39.51 (#22/46)
Claude Opus 4.5 — ELO 1693, #47
- LLM2014 Code 2025-11 - C#: 9.92 (#2/26)
- LLM2014 Code 2025-11 - C++: 9.92 (#2/26)
- LLM2014 Code 2025-11 - Golang: 9.1 (#2/26)
- LLM2014 Code 2025-11 - Python: 9.33 (#2/26)
- LLM2014 Code 2025-11: 90.75 (#3/26)
- LLM2014 Code 2025-11 - Java: 8.58 (#4/26)
- LLM2014 Code 2025-11 - TypeScript: 7.6 (#6/26)
- LLM2014 Logic 2025-11: 37.68 (#27/53)
- LLM2014 Logic 2026-01: 33.08 (#29/45)
- LLM2014 Logic 2025-12: 33.08 (#33/51)
- ...and 2 more
DeepSeek V4 Flash (0731) — ELO 1693, #48
- Toolathlon: 70.3 (#14/41)
- Vals AI ProgramBench: 57.29 (#20/42)
- Design Arena (SVG): 1219.0 (#40/112)
- AA-LCR: 74.3333 (#43/210)
- Epoch AI - ECI: 154.35 (#117/507)
- LM Market Cap LMC Score: 40.0 (#238/403)
GPT-5 — ELO 1690, #50
- VitaBench 2.0: 44.1 (#8/35)
- RealWorldQA: 82.75 (#8/48)
- CyberGym: 60.2 (#26/54)
- HMMT 2025: 88.33 (#27/67)
- AA-LCR: 76.3333 (#31/210)
- NL2Repo: 21.7 (#34/37)
- FrontierMath - Tiers 1-3 (v2): 37.19 (#46/92)
- Arena AI Code: 1419.14 (#57/113)
- Epoch AI - Mystery Game Puzzles: 16.0 (#60/110)
- CritPt: 1.1 (#118/284)
Claude Opus 4.5 (20251101) — ELO 1690, #51
- DeepResearchBench: 54.8 (#3/41)
GPT-5.2 — ELO 1688, #53
- HMMT 2025: 100.0 (#1/67)
- Epoch AI - Mystery Game Puzzles: 22.0 (#39/110)
- CritPt: 7.9 (#47/284)
- Arena AI Code: 1417.81 (#58/113)
DeepSeek V4 Pro — ELO 1687, #54
- Featherbench: 96.0 (#2/17)
- IMO-AnswerBench: 89.8 (#5/15)
- VitaBench 2.0: 45.6 (#6/35)
- MathArena Apex: 38.3 (#8/48)
- Featherbench - Rubric Quality: 8.7 (#11/17)
- SWE Atlas: 27.15 (#23/29)
- CritPt: 12.9 (#30/284)
Kimi K2.6 — ELO 1686, #55
- IMO-AnswerBench: 91.1 (#1/15)
- OpenCompass LLM - Math: 75.7 (#3/15)
- OpenCompass Math - College: 82.7 (#3/15)
- OpenCompass Reasoning - Common: 78.1 (#5/15)
- OpenCompass Knowledge - Humanities: 92.5 (#6/15)
- OpenCompass Knowledge - Engineering: 90.8 (#7/15)
- OpenCompass Knowledge - Common Sense: 89.6 (#7/15)
- OpenCompass Math - Competition: 68.6 (#7/15)
- OpenCompass LLM - Knowledge: 89.8 (#8/15)
- OpenCompass LLM - Reasoning: 62.9 (#8/15)
- ...and 8 more
Inkling — ELO 1685, #56
- MCP Atlas: 76.0 (#25/51)
- Arena AI Code: 1407.6 (#60/113)
GLM-5.1 — ELO 1683, #57
- VitaBench 2.0: 42.0 (#11/35)
- CritPt: 4.6 (#64/284)
Seed 2.0 Pro — ELO 1683, #59
- LLM2014 Logic 2026-02: 69.55 (#4/46)
- LLM2014 Logic 2026-03: 68.36 (#4/42)
- LLM2014 Logic 2026-04: 66.02 (#6/42)
- LLM2014 Logic 2026-05: 57.4 (#8/43)
- LLM2014 Logic 2026-06: 55.84 (#10/43)
- VitaBench 2.0: 42.8 (#10/35)
Claude Sonnet 4.6 — ELO 1678, #61
- DeepResearchBench: 50.4 (#9/41)
- OSWorld-Verified: 78.5 (#19/132)
- LLM2014 Logic 2026-02: 30.17 (#28/46)
- LLM2014 Logic 2026-03: 29.38 (#29/42)
- LLM2014 Logic 2026-04: 26.52 (#33/42)
- LLM2014 Logic 2026-05: 19.89 (#34/43)
- LLM2014 Logic 2026-06: 16.77 (#34/43)
- SimpleQA Verified: 35.5 (#37/66)
- CritPt: 0.9 (#136/284)
Qwen 3.6 Plus — ELO 1678, #62
- AI for Education Visual Maths - Geometry: 80.77 (#7/76)
- AI for Education Visual Maths - Algebra: 96.15 (#8/76)
- AI for Education Visual Maths: 80.59 (#9/76)
- AI for Education Visual Maths - Measurement: 94.59 (#13/76)
- AI for Education Visual Maths - Statistics and Probability: 42.86 (#16/76)
- AI for Education Visual Maths - Number and Operations: 62.16 (#24/76)
- FrontierMath - Tiers 1-3 (v2): 32.28 (#54/92)
Qwen 3.5 397B A17B — ELO 1678, #63
- Epoch AI - Mystery Game Puzzles: 18.0 (#50/110)
- FrontierMath - Tiers 1-3 (v2): 29.47 (#56/92)
- Arena AI Code: 1399.23 (#64/113)
- CritPt: 0.9 (#128/284)
O3 — ELO 1676, #64
- LLM2014 Logic 2025-04: 85.85 (#1/29)
- LLM2014 Logic 2025-05: 85.14 (#1/32)
- LLM2014 Logic 2025-06: 83.95 (#1/38)
- LLM2014 Logic 2025-08: 71.31 (#4/45)
- FrontierMath - Tiers 1-3 (v2): 33.33 (#52/92)
- OSWorld-Verified: 23.0 (#119/132)
MiMo-V2.5-Pro — ELO 1676, #65
- CritPt: 1.1 (#115/284)
GPT-5.1 Codex — ELO 1676, #66
- Arena AI Code: 1336.28 (#88/113)
GPT-5.3 — ELO 1674, #67
- SWE Atlas: 32.6 (#19/29)
Grok 4.20 0309 — ELO 1673, #69
- ForecastBench: 64.1 (#110/300)
DeepSeek V4 Flash — ELO 1670, #70
- WebDev Arena: 1431.07 (#63/116)
- CritPt: 3.4 (#74/284)
Gemini 3.5 Flash Lite — ELO 1670, #71
- AA-LCR: 74.6667 (#41/210)
- AIIQ Composite IQ: 108.0 (#61/130)
- CritPt: 0.0 (#283/284)
GPT-5.1 — ELO 1669, #72
- Epoch AI - Mystery Game Puzzles: 19.0 (#48/110)
- Arena AI Code: 1391.01 (#68/113)
- CritPt: 0.0 (#212/284)
MiniMax-M3 — ELO 1668, #73
- OpenCompass Knowledge - Science: 90.8 (#8/15)
- OpenCompass Knowledge - Engineering: 88.3 (#9/15)
- OpenCompass LLM - Knowledge: 86.6 (#13/15)
- OpenCompass Knowledge - Humanities: 87.1 (#14/15)
- OpenCompass Knowledge - Common Sense: 84.2 (#14/15)
- OpenCompass Reasoning - Common: 69.9 (#14/15)
- OpenCompass Reasoning - Academic: 38.1 (#14/15)
- OpenCompass Math - College: 74.3 (#14/15)
- OpenCompass LLM - Reasoning: 54.0 (#15/15)
- OpenCompass LLM - Math: 63.0 (#15/15)
- ...and 5 more
O3 (2025-04-16) — ELO 1668, #74
- Epoch AI - Mystery Game Puzzles: 29.0 (#23/110)
Inkling Small — ELO 1667, #75
- Arena AI Code: 1401.56 (#62/113)
- AIIQ Composite IQ: 106.0 (#63/130)
- WebDev Arena: 1401.56 (#68/116)
- AA-LCR: 69.3333 (#89/210)
Muse Glimmer 30B — ELO 1667, #76
- AA-LCR: 80.0 (#7/210)
- Arena AI Code: 1359.0 (#79/113)
- CritPt: 2.6 (#89/284)
- AIIQ Composite IQ: 92.0 (#111/130)
Claude Sonnet 4.5 — ELO 1665, #77
- LLM2014 Code 2025-11 - Java: 9.22 (#2/26)
- LLM2014 Code 2025-11 - TypeScript: 8.92 (#3/26)
- LLM2014 Code 2025-11: 81.67 (#4/26)
- LLM2014 Logic 2025-10: 69.78 (#4/50)
- LLM2014 Code 2025-11 - Golang: 7.85 (#5/26)
- LLM2014 Code 2025-11 - Python: 8.88 (#5/26)
- LLM2014 Code 2025-11 - C++: 7.17 (#6/26)
- LLM2014 Logic 2025-11: 66.7 (#6/53)
- LLM2014 Code 2025-11 - C#: 7.72 (#7/26)
- LLM2014 Logic 2026-01: 60.83 (#9/45)
- ...and 11 more
GPT-5.4 Mini — ELO 1665, #78
- FrontierCode: 43.01 (#31/47)
- APEX-Agents-AA: 0.28 (#38/56)
- FrontierMath - Tiers 1-3 (v2): 24.56 (#60/92)
- Arena AI Code: 1397.08 (#66/113)
- CritPt: 2.9 (#83/284)
- Epoch AI - Mystery Game Puzzles: 11.0 (#84/110)
Grok 4.3 — ELO 1665, #79
- AI for Education Visual Maths - Geometry: 45.38 (#42/76)
- AI for Education Visual Maths - Measurement: 75.68 (#42/76)
- AI for Education Visual Maths: 49.79 (#45/76)
- AI for Education Visual Maths - Algebra: 57.69 (#49/76)
- AI for Education Visual Maths - Number and Operations: 40.54 (#53/76)
- CritPt: 4.9 (#62/284)
- AI for Education Visual Maths - Statistics and Probability: 14.29 (#65/76)
- Arena AI Code: 1356.73 (#82/113)
Gemini 2.5 Pro — ELO 1664, #80
- LLM2014 Code 2025-11 - Golang: 8.87 (#3/26)
- LLM2014 Code 2025-11: 74.36 (#6/26)
- LLM2014 Code 2025-11 - C#: 7.77 (#6/26)
- LLM2014 Code 2025-11 - C++: 7.15 (#8/26)
- LLM2014 Code 2025-11 - Python: 7.9 (#8/26)
- LLM2014 Code 2025-11 - Java: 6.35 (#12/26)
- LLM2014 Code 2025-11 - TypeScript: 6.58 (#16/26)
- VitaBench 2.0: 33.1 (#24/35)
- Arena AI Code: 1225.69 (#106/113)
Kimi K2.5 — ELO 1664, #81
- HMMT 2025: 93.33 (#16/67)
- CyberGym: 41.3 (#20/54)
- SimpleQA Verified: 34.3 (#39/66)
- OSWorld-Verified: 63.3 (#44/132)
- APEX-Agents-AA: 0.12 (#51/56)
- Arena AI Code: 1435.92 (#52/113)
- CritPt: 0.6 (#150/284)
Grok 4.20 Beta (0309) — ELO 1664, #82
- Arena AI Code: 1373.56 (#74/113)
Qwen 3.6 27B — ELO 1661, #86
- FrontierMath - Tiers 1-3 (v2): 35.09 (#48/92)
- Epoch AI - Mystery Game Puzzles: 7.0 (#96/110)
- CritPt: 0.9 (#127/284)
DeepSeek V3.2 Speciale — ELO 1660, #87
- HMMT 2025: 97.5 (#6/67)
GLM-5 — ELO 1657, #89
- Arena AI Code: 1435.46 (#53/113)
- CritPt: 0.0 (#226/284)
MiMo-V2-Pro — ELO 1657, #90
- Arena AI Code: 1433.51 (#55/113)
Claude Opus 4.1 (20250805) — ELO 1656, #92
- Arena AI Code: 1388.83 (#70/113)
GLM-5V Turbo — ELO 1653, #96
- Arena AI Code: 1399.83 (#63/113)
Gemini 3.1 Flash Lite — ELO 1652, #97
- DeepResearchBench: 37.3 (#37/41)
- FrontierMath - Tiers 1-3 (v2): 21.4 (#64/92)
Gemini 2.5 Flash (Preview 05-20) — ELO 1647, #99
- LLM2014 Logic 2025-05: 55.28 (#9/32)
GPT-5 Mini — ELO 1645, #100
- RealWorldQA: 80.26 (#11/48)
- HMMT 2025: 89.17 (#25/67)
- FrontierMath - Tiers 1-3 (v2): 18.25 (#72/92)
- Epoch AI - Mystery Game Puzzles: 10.0 (#87/110)
- CritPt: 0.0 (#216/284)
Claude Opus 4 — ELO 1645, #101
- LLM2014 Logic 2025-05: 60.12 (#8/32)
- LLM2014 Logic 2025-06: 53.48 (#11/38)
- LLM2014 Logic 2025-08: 43.68 (#18/45)
- LLM2014 Logic 2025-09: 40.23 (#19/46)
- LLM2014 Logic 2025-10: 40.23 (#22/50)
- LLM2014 Logic 2025-11: 36.88 (#29/53)
Qwen 3.5 122B A10B — ELO 1644, #102
- Arena AI Code: 1357.68 (#80/113)
- CritPt: 0.6 (#145/284)
Gemini 3.1 Flash Lite (Preview) — ELO 1643, #104
- MCP Atlas: 57.1 (#46/51)
- Arena AI Code: 1254.07 (#99/113)
Qwen 3 Max — ELO 1643, #105
- VitaBench 2.0: 28.4 (#29/35)
Step 3.5 Flash — ELO 1642, #106
- AA-LCR: 48.0 (#183/210)
Grok 4 — ELO 1641, #107
- LLM2014 Logic 2025-08: 78.22 (#3/45)
- LLM2014 Logic 2025-09: 74.79 (#3/46)
- LLM2014 Logic 2025-10: 74.97 (#3/50)
- LLM2014 Code 2025-07: 55.12 (#5/19)
- LLM2014 Logic 2025-11: 69.52 (#5/53)
- LLM2014 Code 2025-09 - Java: 6.22 (#6/20)
- LLM2014 Code 2025-09: 37.44 (#8/19)
- LLM2014 Code 2025-09 - C#: 6.8 (#10/20)
- LLM2014 Code 2025-09 - Python: 5.76 (#11/20)
- LLM2014 Code 2025-09 - Golang: 4.06 (#13/20)
- ...and 2 more
GLM-4.7 — ELO 1641, #108
- Arena AI Code: 1434.29 (#54/113)
- CritPt: 0.0 (#228/284)
LongCat 2.0 — ELO 1641, #109
- WritingBench: 83.8 (#2/55)
- IMO-AnswerBench: 81.8 (#12/15)
- AIIQ Composite IQ: 98.0 (#91/130)
MiMo-V2.5 — ELO 1640, #110
- Arena AI Code: 1437.8 (#51/113)
Hy3-preview — ELO 1639, #112
- LLM2014 Logic 2026-04: 56.63 (#11/42)
- LLM2014 Logic 2026-05: 47.01 (#12/43)
- LLM2014 Logic 2026-06: 46.12 (#14/43)
- Arena AI Code: 1356.33 (#83/113)
- CritPt: 0.3 (#179/284)
Kimi K2 — ELO 1638, #113
- NL2Repo: 22.7 (#32/37)
Claude Sonnet 4 — ELO 1638, #114
- LLM2014 Code 2025-07: 66.98 (#1/19)
- LLM2014 Code 2025-09: 61.01 (#2/19)
- LLM2014 Code 2025-09 - TypeScript: 8.89 (#2/20)
- LLM2014 Code 2025-09 - C#: 8.31 (#3/20)
- LLM2014 Code 2025-09 - Java: 7.13 (#3/20)
- LLM2014 Code 2025-09 - Golang: 6.11 (#4/20)
- LLM2014 Vision 2025-07: 41.21 (#4/16)
- LLM2014 Logic 2025-05: 68.22 (#5/32)
- LLM2014 Code 2025-09 - Python: 6.94 (#6/20)
- LLM2014 Logic 2025-06: 62.96 (#8/38)
- ...and 3 more
Nemotron 3 Ultra — ELO 1637, #116
- IMO-AnswerBench: 88.6 (#6/15)
- AI for Education Pedagogy - Maths: 88.89 (#39/243)
- Epoch AI - Mystery Game Puzzles: 20.0 (#44/110)
- AI for Education SEND: 80.73 (#58/235)
- AI for Education Pedagogy - Secondary: 86.01 (#62/243)
- AI for Education Pedagogy: 86.21 (#64/243)
- OTIS Mock AIME 2024-25: 86.67 (#71/256)
- AI for Education Pedagogy - Primary: 88.26 (#78/243)
- AI for Education Pedagogy - Science: 87.43 (#81/243)
- AI for Education Pedagogy - Social studies: 80.91 (#103/243)
- ...and 2 more
Qwen 3.5 Flash — ELO 1637, #117
- Epoch AI - Mystery Game Puzzles: 20.0 (#43/110)
- FrontierMath - Tiers 1-3 (v2): 18.25 (#71/92)
- Arena AI Code: 1238.12 (#104/113)
Qwen 3.5 27B — ELO 1636, #118
- Roboflow Vision Evals - Visual Understanding: 71.64 (#16/77)
- Arena AI Code: 1357.46 (#81/113)
- CritPt: 0.3 (#163/284)
MiniMax-M2.7 — ELO 1636, #119
- VitaBench 2.0: 34.5 (#22/35)
- Arena AI Code: 1398.14 (#65/113)
Grok 4 Fast — ELO 1636, #120
- Arena AI Code: 1161.87 (#112/113)
- CritPt: 0.0 (#196/284)
DeepSeek V3.2 — ELO 1633, #126
- HMMT 2025: 92.5 (#19/67)
- Arena AI Code: 1360.31 (#78/113)
- CritPt: 0.9 (#135/284)
Claude 3.7 Sonnet (20250219) — ELO 1633, #127
- OSWorld-Verified: 35.8 (#90/132)
O4 Mini (2025-04-16) — ELO 1633, #128
- Epoch AI - Mystery Game Puzzles: 5.0 (#102/110)
Grok 4.1 Fast — ELO 1630, #129
- Arena AI Code: 1239.97 (#102/113)
- CritPt: 0.0 (#195/284)
MiniMax-M2.5 — ELO 1630, #130
- Arena AI Code: 1383.9 (#73/113)
Qwen 3.6 35B A3B — ELO 1629, #131
- AI for Education Visual Maths - Statistics and Probability: 71.43 (#4/76)
- AI for Education Visual Maths - Algebra: 92.31 (#12/76)
- AI for Education Visual Maths - Number and Operations: 67.57 (#15/76)
- AI for Education Visual Maths - Measurement: 91.89 (#19/76)
- AI for Education Visual Maths: 72.15 (#20/76)
- AI for Education Visual Maths - Geometry: 63.85 (#21/76)
- FrontierMath - Tiers 1-3 (v2): 17.54 (#74/92)
DeepSeek V3.2 Exp — ELO 1626, #135
- Arena AI Code: 1271.77 (#96/113)
- CritPt: 1.4 (#110/284)
O4 Mini — ELO 1626, #136
- LLM2014 Code 2025-09 - Python: 9.17 (#1/20)
- LLM2014 Code 2025-07: 66.75 (#2/19)
- LLM2014 Code 2025-09 - C#: 8.65 (#2/20)
- LLM2014 Code 2025-09 - Golang: 7.35 (#2/20)
- LLM2014 Code 2025-09 - Java: 7.89 (#2/20)
- LLM2014 Logic 2025-04: 80.87 (#2/29)
- LLM2014 Logic 2025-05: 80.16 (#2/32)
- LLM2014 Logic 2025-06: 78.97 (#2/38)
- LLM2014 Vision 2025-07: 53.4 (#2/16)
- LLM2014 Code 2025-09 - C++: 6.04 (#3/20)
- ...and 4 more
GLM-4.6 — ELO 1626, #137
- VitaBench 2.0: 34.2 (#23/35)
- Arena AI Code: 1340.18 (#86/113)
- CritPt: 0.0 (#231/284)
Qwen 3 235B A22B 2507 — ELO 1626, #138
- WritingBench: 82.34 (#3/55)
- Epoch AI - Mystery Game Puzzles: 9.0 (#89/110)
O1 — ELO 1624, #139
- ComplexFuncBench: 47.6 (#9/20)
- FrontierMath - Tiers 1-3 (v2): 14.74 (#78/92)
DeepSeek V3.1 Terminus — ELO 1623, #140
- CritPt: 0.0 (#266/284)
Gemini 2.5 Flash — ELO 1621, #143
- LLM2014 Code 2025-09 - Java: 6.97 (#4/20)
- LLM2014 Code 2025-09 - C#: 7.98 (#5/20)
- LLM2014 Logic 2025-04: 63.69 (#7/29)
- LLM2014 Code 2025-07: 45.04 (#9/19)
- LLM2014 Code 2025-09: 36.34 (#11/19)
- LLM2014 Code 2025-09 - Python: 5.28 (#16/20)
- LLM2014 Code 2025-09 - Golang: 2.88 (#17/20)
- LLM2014 Code 2025-09 - TypeScript: 5.15 (#17/20)
- LLM2014 Logic 2025-05: 42.85 (#17/32)
- LLM2014 Code 2025-09 - C++: 2.43 (#19/20)
- ...and 2 more
GPT-4.1 — ELO 1620, #145
- CyberGym: 7.23 (#8/54)
- FrontierMath - Tiers 1-3 (v2): 5.96 (#85/92)
MiniMax-M2.1 — ELO 1620, #146
- Arena AI Code: 1386.99 (#71/113)
DeepSeek R1 0528 — ELO 1618, #150
- VitaBench 2.0: 39.6 (#15/35)
Qwen 3 VL 235B A22B Instruct — ELO 1618, #151
- OmniDocBench 1.5: 89.15 (#28/73)
Claude Sonnet 4 (20250514) — ELO 1618, #152
- OSWorld-Verified: 43.9 (#77/132)
Gemma 4 31B — ELO 1617, #154
- LM Market Cap LMC Score: 80.5 (#71/403)
- Arena AI Code: 1363.56 (#76/113)
- CritPt: 0.0 (#238/284)
Qwen 3 235B A22B 2507 Instruct — ELO 1617, #155
- WritingBench: 80.26 (#8/55)
GLM-4.5 — ELO 1616, #156
- VitaBench 2.0: 30.7 (#27/35)
DeepSeek V3.1 — ELO 1615, #157
- HMMT 2025: 85.83 (#28/67)
- CritPt: 0.0 (#267/284)
Grok Code Fast 1 — ELO 1612, #162
- Arena AI Code: 1164.51 (#111/113)
Ling-3.0-flash — ELO 1612, #164
- ArtifactsBench: 77.0 (#1/8)
- Multi-IF: 87.7 (#14/15)
- MCP Atlas: 65.5 (#39/51)
- AA-LCR: 67.0 (#105/210)
- AIIQ Composite IQ: 92.0 (#112/130)
Claude Haiku 4.5 — ELO 1609, #167
- LLM2014 Code 2025-11 - Golang: 5.03 (#15/26)
- LLM2014 Code 2025-11 - C++: 5.43 (#17/26)
- LLM2014 Code 2025-11 - C#: 5.45 (#19/26)
- LLM2014 Code 2025-11: 51.44 (#22/26)
- LLM2014 Code 2025-11 - Java: 4.5 (#22/26)
- LLM2014 Code 2025-11 - TypeScript: 5.58 (#22/26)
- LLM2014 Code 2025-11 - Python: 4.87 (#23/26)
- AI for Education Visual Maths - Measurement: 81.08 (#33/76)
- AI for Education Visual Maths - Statistics and Probability: 28.57 (#33/76)
- LLM2014 Logic 2026-01: 16.95 (#41/45)
- ...and 8 more
Qwen 3.5 35B A3B — ELO 1608, #169
- Arena AI Code: 1249.75 (#100/113)
- CritPt: 0.6 (#144/284)
GPT-5.1 Codex Mini — ELO 1608, #171
- Arena AI Code: 1243.74 (#101/113)
KAT-Coder-Pro V1 — ELO 1605, #174
- Arena AI Code: 1255.43 (#98/113)
Claude 3.5 Sonnet — ELO 1602, #175
- Video-MME: 60.0 (#31/51)
MiMo-V2-Flash — ELO 1602, #176
- Arena AI Code: 1330.17 (#89/113)
GPT-5.4 Nano — ELO 1601, #177
- APEX-Agents-AA: 0.25 (#41/56)
- CritPt: 5.1 (#59/284)
- FrontierMath - Tiers 1-3 (v2): 20.35 (#65/92)
- Epoch AI - Mystery Game Puzzles: 9.0 (#91/110)
- AA GDPval: 738.5 (#143/219)
Mercury 2 — ELO 1600, #180
- Arena AI Code: 1166.17 (#110/113)
Qwen 3 Next 80B A3B — ELO 1597, #182
- LLM2014 Logic 2025-09: 43.59 (#13/46)
O3 Mini — ELO 1597, #183
- ComplexFuncBench: 17.6 (#14/20)
- FrontierMath - Tiers 1-3 (v2): 10.53 (#80/92)
Nemotron 3 Super — ELO 1596, #188
- AA-LCR: 60.3333 (#139/210)
MiniMax-M2 — ELO 1595, #189
- LLM2014 Code 2025-11 - Python: 7.57 (#10/26)
- LLM2014 Code 2025-11 - C++: 5.82 (#11/26)
- LLM2014 Code 2025-11 - C#: 5.95 (#17/26)
- LLM2014 Code 2025-11 - Golang: 3.2 (#20/26)
- LLM2014 Code 2025-11 - Java: 4.7 (#21/26)
- LLM2014 Code 2025-11 - TypeScript: 4.9 (#25/26)
- Arena AI Code: 1297.0 (#94/113)
Mistral Medium 3.5 — ELO 1595, #190
- Arena AI Code: 1265.04 (#97/113)
- CritPt: 0.0 (#282/284)
O3 Mini (2025-01-31) — ELO 1591, #195
- Epoch AI - Mystery Game Puzzles: 7.0 (#95/110)
GPT-4o — ELO 1590, #196
- Video-MME: 71.9 (#7/51)
Devstral 2 — ELO 1590, #200
- Arena AI Code: 1193.51 (#108/113)
Qwen 3 Coder 480B A35B Instruct — ELO 1590, #201
- Arena AI Code: 1273.25 (#95/113)
Gemma 4 12B — ELO 1590, #202
- AI for Education Pedagogy - Technology: 85.85 (#29/243)
- AI for Education SEND: 77.06 (#99/235)
- AI for Education Pedagogy - Primary: 85.45 (#102/243)
- AI for Education Pedagogy - Secondary: 81.29 (#102/243)
- AI for Education Pedagogy: 81.65 (#105/243)
- AI for Education Pedagogy - Maths: 80.16 (#108/243)
- AI for Education Pedagogy - Science: 82.51 (#109/243)
- AI for Education Pedagogy - Social studies: 78.18 (#116/243)
GPT-OSS-120B — ELO 1589, #203
- Epoch AI - Mystery Game Puzzles: 2.0 (#109/110)
Nemotron 3.5 Lightning — ELO 1588, #208
- AI for Education Pedagogy - Social studies: 81.82 (#95/243)
- AIIQ Composite IQ: 95.0 (#103/130)
- AI for Education Pedagogy - Science: 81.97 (#113/243)
- AI for Education SEND: 74.31 (#114/235)
- AI for Education Pedagogy: 78.75 (#119/243)
- AI for Education Pedagogy - Secondary: 77.99 (#119/243)
- AI for Education Pedagogy - Primary: 83.1 (#121/243)
- AI for Education Pedagogy - Maths: 74.6 (#140/243)
- AI for Education Pedagogy - Technology: 75.47 (#158/243)
- AA-LCR: 55.3333 (#159/210)
- ...and 1 more
GPT-4.1 Mini — ELO 1587, #210
- LLM2014 Code 2025-07: 50.87 (#6/19)
- LLM2014 Logic 2025-04: 54.55 (#10/29)
- LLM2014 Logic 2025-05: 53.52 (#10/32)
- LLM2014 Logic 2025-06: 48.53 (#14/38)
- RealWorldQA: 78.69 (#15/48)
- LLM2014 Logic 2025-08: 33.51 (#26/45)
- FrontierMath - Tiers 1-3 (v2): 6.67 (#84/92)
- Epoch AI - Mystery Game Puzzles: 7.0 (#98/110)
GLM-4.6V — ELO 1585, #215
- CritPt: 0.0 (#230/284)
GPT-4o (2024-11-20) — ELO 1580, #223
- ComplexFuncBench: 66.5 (#1/20)
GPT-4o (2024-08-06) — ELO 1579, #227
- ComplexFuncBench: 60.5 (#5/20)
- FrontierMath - Tiers 1-3 (v2): 0.35 (#91/92)
Gemma 4 26B A4B — ELO 1578, #232
- BIG-Bench Extra Hard: 64.8 (#2/17)
- MedXpertQA: 58.1 (#10/26)
- AI for Education Visual Maths - Measurement: 83.78 (#32/76)
- AI for Education Visual Maths - Number and Operations: 51.35 (#33/76)
- AI for Education Visual Maths: 58.65 (#35/76)
- AI for Education Visual Maths - Geometry: 52.31 (#35/76)
- AI for Education Visual Maths - Algebra: 76.92 (#37/76)
- AI for Education Visual Maths - Statistics and Probability: 14.29 (#50/76)
- Arena AI Code: 1361.65 (#77/113)
- CritPt: 0.0 (#240/284)
Qwen 3.5 9B — ELO 1577, #236
- MedXpertQA: 49.9 (#11/26)
- LVBench: 70.0 (#14/40)
- Chess Puzzles (Epoch AI): 12.0 (#117/182)
- OTIS Mock AIME 2024-25: 61.67 (#146/256)
- CritPt: 0.3 (#162/284)
GPT-5 Nano — ELO 1573, #243
- HMMT 2025: 74.17 (#39/67)
- FrontierMath - Tiers 1-3 (v2): 5.96 (#86/92)
- Epoch AI - Mystery Game Puzzles: 9.0 (#88/110)
- CritPt: 0.0 (#214/284)
Mistral Large 3 — ELO 1572, #245
- Arena AI Code: 1230.14 (#105/113)
Gemini 2.5 Flash Lite — ELO 1569, #247
- CritPt: 0.0 (#253/284)
Claude Haiku 4.5 (20251001) — ELO 1568, #249
- DeepResearchBench: 45.5 (#29/41)
- SimpleQA Verified: 13.2 (#62/66)
- Arena AI Code: 1327.41 (#90/113)
Mistral Small 4 — ELO 1568, #253
- CritPt: 0.3 (#171/284)
Gemini 1.5 Pro — ELO 1562, #262
- Video-MME: 75.0 (#2/51)
- PhysicsFinals: 38.4 (#4/33)
- Natural2Code: 42.3 (#7/24)
Qwen 3 32B — ELO 1553, #282
- MemPoison - Clean Task: 94.92 (#5/10)
- MemPoison - Poisoned Task: 2.18 (#9/10)
GLM-4.7 Flash — ELO 1552, #284
- CritPt: 0.0 (#227/284)
GLM-4.5V — ELO 1552, #286
- CritPt: 0.0 (#233/284)
Qwen 2.5 14B — ELO 1543, #309
- MemPoison - Poisoned Task: 2.64 (#6/10)
- MemPoison - Clean Task: 94.28 (#7/10)
Qwen 2.5 VL 72B Instruct — ELO 1540, #316
- OmniDocBench 1.5: 87.02 (#40/73)
QwQ-32B — ELO 1537, #327
- Chess Puzzles (Epoch AI): 5.0 (#149/182)
- OTIS Mock AIME 2024-25: 59.17 (#150/256)
Qwen 3.5 4B — ELO 1536, #328
- MedXpertQA: 42.9 (#14/26)
- LVBench: 66.4 (#16/40)
GPT-4 (0613) — ELO 1536, #330
- Epoch AI - Mystery Game Puzzles: 12.0 (#78/110)
Devstral Medium — ELO 1535, #332
- Arena AI Code: 1080.11 (#113/113)
GPT-OSS-20B — ELO 1532, #341
- CritPt: 1.4 (#101/284)
- OTIS Mock AIME 2024-25: 65.28 (#139/256)
- Chess Puzzles (Epoch AI): 4.0 (#156/182)
- Epoch AI - ECI: 136.81 (#363/507)
Gemini 1.5 Flash — ELO 1521, #359
- Video-MME: 70.3 (#11/51)
EXAONE 4.0 32B — ELO 1516, #364
- CritPt: 0.0 (#257/284)
Gemma 3 27B — ELO 1514, #367
- BIG-Bench Extra Hard: 4.9 (#12/17)
Gemma 3 12B (IT) — ELO 1513, #370
- Chess Puzzles (Epoch AI): 0.0 (#175/182)
- OTIS Mock AIME 2024-25: 16.67 (#207/256)
GPT-4o Mini — ELO 1511, #376
- ComplexFuncBench: 38.6 (#12/20)
- Video-MME: 64.8 (#22/51)
- FrontierMath - Tiers 1-3 (v2): 0.7 (#89/92)
GPT-4o Mini (2024-07-18) — ELO 1511, #377
- Epoch AI - Mystery Game Puzzles: 12.0 (#82/110)
Phi-4 — ELO 1508, #383
- Chess Puzzles (Epoch AI): 1.0 (#170/182)
Granite 4.1 8B — ELO 1508, #385
- Arena AI Code: 1191.25 (#109/113)
Qwen 3 8B — ELO 1496, #401
- MemPoison - Clean Task: 95.17 (#3/10)
- MemPoison - Poisoned Task: 2.76 (#5/10)
- Chess Puzzles (Epoch AI): 5.0 (#150/182)
- OTIS Mock AIME 2024-25: 56.11 (#156/256)
Gemma 3 12B — ELO 1489, #416
- BIG-Bench Extra Hard: 4.5 (#13/17)
Qwen 3.5 2B — ELO 1466, #452
- MedXpertQA: 26.9 (#19/26)
- LVBench: 57.1 (#19/40)
- HMMT 2025: 22.9 (#60/67)
Gemma 4 E4B — ELO 1463, #465
- CritPt: 0.3 (#184/284)
Gemma 3 4B (IT) — ELO 1457, #476
- Chess Puzzles (Epoch AI): 0.0 (#173/182)
- OTIS Mock AIME 2024-25: 7.5 (#219/256)
Llama 3.1 8B Instruct — ELO 1454, #481
- Chess Puzzles (Epoch AI): 0.0 (#177/182)
Llama 3.1 8B — ELO 1439, #512
- MemPoison - Poisoned Task: 3.29 (#2/10)
- MemPoison - Clean Task: 91.21 (#10/10)
GPT-3.5 Turbo (0125) — ELO 1428, #534
- Epoch AI - Mystery Game Puzzles: 3.0 (#108/110)
Gemma 4 E2B — ELO 1425, #536
- CritPt: 0.0 (#237/284)
Gemma 3 4B — ELO 1408, #563
- BIG-Bench Extra Hard: 3.4 (#16/17)
Qwen 3.5 0.8B — ELO 1398, #576
- LVBench: 45.1 (#23/40)
- MedXpertQA: 17.1 (#26/26)
Gemma 3 1B (IT) — ELO 1370, #609
- Chess Puzzles (Epoch AI): 0.0 (#174/182)
- OTIS Mock AIME 2024-25: 1.11 (#249/256)
Arcee Trinity Large — ELO 1605
- Arena AI Code: 1238.37 (#103/113)
Claude Mythos Preview — ELO 1779
- OSWorld-Verified: 85.4 (#3/132)
Doubao-Seed-1.6 — ELO 1628
- VitaBench 2.0: 32.6 (#26/35)
GLM-4 32B — ELO 1566
- MemPoison - Clean Task: 94.83 (#6/10)
- MemPoison - Poisoned Task: 2.64 (#7/10)
GPT-5.1 ChatGPT — ELO 1647
- AI Chess Leaderboard (Continuation): 1363.0 (#19/261)
- AI Chess Leaderboard (Reasoning): 837.0 (#98/320)
GPT-5.6 Pro Sol — ELO 1780
- Epoch AI - ECI: 161.06 (#26/507)
Grok 4.1 — ELO 1650
- Arena AI Code: 1209.85 (#107/113)
Kimi K2 Turbo — ELO 1654
- Arena AI Code: 1322.87 (#92/113)
LFM2.5-2.6B — ELO 1512
- BenchLM: 43.0 (#181/200)
Laguna M.1 — ELO 1570
- Arena AI Code: 1347.31 (#84/113)
Laguna XS.2 — ELO 1548
- Arena AI Code: 1301.97 (#93/113)
Nova 2 Lite — ELO 1550
- ParseBench: 40.43 (#50/56)
Qwen 3.7 Flash — ELO 1642
- Epoch AI - Mystery Game Puzzles: 15.0 (#65/110)
- FrontierMath - Tiers 1-3 (v2): 19.3 (#67/92)
- AIIQ Composite IQ: 92.0 (#110/130)
Seed 1.8 — ELO 1640
- LLM2014 Logic 2025-12: 63.06 (#9/51)
Solar Pro 3 — ELO 1552
- AIIQ Composite IQ: 85.0 (#121/130)
Solar Pro 4 — ELO 1649
- MCP Atlas: 61.4 (#42/51)
- AA-LCR: 70.6667 (#72/210)
- AIIQ Composite IQ: 100.0 (#85/130)
Spark X2 — ELO 1562
- LLM2014 Logic 2026-02: 28.21 (#31/46)
- LLM2014 Logic 2026-03: 25.04 (#34/42)
- LLM2014 Logic 2026-04: 20.55 (#38/42)
- LLM2014 Logic 2026-05: 14.04 (#40/43)
- LLM2014 Logic 2026-06: 11.14 (#40/43)
- LLM2014 Logic 2026-07: 6.21 (#44/45)
deepseek-llm-67B-chat — ELO 1490
- Chess Puzzles (Epoch AI): 0.0 (#176/182)
- OTIS Mock AIME 2024-25: 0.83 (#252/256)
uitars-1.5-7B — ELO 1467
- MobileGym-Bench: 13.8 (#8/10)

Top-10 New Scores (107)

Claude Fable 5 on AI for Education Visual Maths: 86.92 (#4)
Claude Fable 5 on AI for Education Visual Maths - Algebra: 92.31 (#16)
Claude Fable 5 on AI for Education Visual Maths - Geometry: 83.85 (#5)
Claude Fable 5 on AI for Education Visual Maths - Measurement: 97.3 (#5)
Claude Fable 5 on AI for Education Visual Maths - Number and Operations: 86.49 (#2)
Claude Fable 5 on AI for Education Visual Maths - Statistics and Probability: 71.43 (#3)
Claude Fable 5 on MCP Atlas: 83.3 (#9)
Claude Fable 5 on OSWorld-Verified: 85.0 (#5)
Claude Fable 5 on PostTrainBench: 41.79 (#1)
Claude Mythos Preview on OSWorld-Verified: 85.4 (#3)
Claude Opus 5 on AA-LCR: 78.6667 (#15)
Claude Opus 5 on AI for Education Visual Maths: 80.59 (#10)
Claude Opus 5 on AI for Education Visual Maths - Algebra: 96.15 (#3)
Claude Opus 5 on AI for Education Visual Maths - Geometry: 75.38 (#13)
Claude Opus 5 on AI for Education Visual Maths - Measurement: 94.59 (#12)
Claude Opus 5 on AI for Education Visual Maths - Number and Operations: 75.68 (#6)
Claude Opus 5 on AI for Education Visual Maths - Statistics and Probability: 71.43 (#2)
Claude Opus 5 on APEX v1: 68.6 (#2)
Claude Opus 5 on Appwrite Arena (With Skills): 97.4 (#4)
Claude Opus 5 on Appwrite Arena (Without Skills): 97.4 (#3)
Claude Opus 5 on CAIS Risk Index: 44.0 (#8)
Claude Opus 5 on CAIS Vision Capabilities Index: 63.9 (#3)
Claude Opus 5 on CritPt: 29.1 (#5)
Claude Opus 5 on LLM2014 Logic 2026-07: 71.88 (#4)
Claude Opus 5 on LLM2014 Logic 2026-08: 64.74 (#4)
Claude Opus 5 on MCP Atlas: 85.8 (#3)
Claude Opus 5 on OpenCompass Code - Comprehensive: 93.3 (#4)
Claude Opus 5 on OpenCompass Knowledge - Common Sense: 92.1 (#3)
Claude Opus 5 on OpenCompass Knowledge - Engineering: 93.3 (#5)
Claude Opus 5 on OpenCompass Knowledge - Humanities: 92.9 (#4)
Claude Opus 5 on OpenCompass Knowledge - Science: 97.5 (#1)
Claude Opus 5 on OpenCompass Knowledge - Social Science: 94.6 (#1)
Claude Opus 5 on OpenCompass LLM - Code: 93.3 (#4)
Claude Opus 5 on OpenCompass LLM - Knowledge: 94.1 (#1)
Claude Opus 5 on OpenCompass LLM - Math: 77.3 (#1)
Claude Opus 5 on OpenCompass LLM - Reasoning: 68.4 (#1)
Claude Opus 5 on OpenCompass Math - College: 79.9 (#5)
Claude Opus 5 on OpenCompass Math - Competition: 74.7 (#1)
Claude Opus 5 on OpenCompass Reasoning - Academic: 56.7 (#1)
Claude Opus 5 on OpenCompass Reasoning - Common: 80.1 (#2)
Claude Opus 5 on PostTrainBench: 35.04 (#3)
Claude Opus 5 on React Native Evals: 88.11 (#1)
Claude Opus 5 on Terminal-Bench 2.1: 51.8 (#1)
Claude Opus 5 on Terminal-Bench 2.1 (Claude Code): 51.8 (#1)
Claude Opus 5 on Vals AI ProgramBench: 82.27 (#1)
GPT-5.6 Pro Sol on Epoch AI - ECI: 161.06 (#26)
GPT-5.6 Sol on AI for Education Visual Maths - Algebra: 96.15 (#2)
GPT-5.6 Sol on AI for Education Visual Maths - Measurement: 97.3 (#2)
GPT-5.6 Sol on AI for Education Visual Maths - Statistics and Probability: 57.14 (#10)
GPT-5.6 Sol on Appwrite Arena (With Skills): 96.9 (#8)
GPT-5.6 Sol on Appwrite Arena (Without Skills): 95.5 (#5)
GPT-5.6 Sol on Chatbot Arena (Search): 1257.0 (#1)
GPT-5.6 Sol on Crosby micro1 RedlineBench: 56.2 (#2)
GPT-5.6 Sol on Epoch AI - Mystery Game Puzzles: 33.0 (#17)
GPT-5.6 Sol on FrontierCode: 60.6 (#9)
GPT-5.6 Sol on MCP Atlas: 81.8 (#13)
GPT-5.6 Sol on OpenCompass Code - Comprehensive: 96.3 (#2)
GPT-5.6 Sol on OpenCompass Knowledge - Common Sense: 86.3 (#11)
GPT-5.6 Sol on OpenCompass Knowledge - Engineering: 84.2 (#12)
GPT-5.6 Sol on OpenCompass Knowledge - Humanities: 90.8 (#9)
GPT-5.6 Sol on OpenCompass Knowledge - Science: 89.2 (#11)
GPT-5.6 Sol on OpenCompass Knowledge - Social Science: 92.1 (#3)
GPT-5.6 Sol on OpenCompass LLM - Code: 96.3 (#2)
GPT-5.6 Sol on OpenCompass LLM - Knowledge: 88.5 (#9)
GPT-5.6 Sol on OpenCompass LLM - Math: 71.4 (#10)
GPT-5.6 Sol on OpenCompass LLM - Reasoning: 66.8 (#3)
GPT-5.6 Sol on OpenCompass Math - College: 74.3 (#13)
GPT-5.6 Sol on OpenCompass Math - Competition: 68.6 (#6)
GPT-5.6 Sol on OpenCompass Reasoning - Academic: 55.8 (#2)
GPT-5.6 Sol on OpenCompass Reasoning - Common: 77.8 (#6)
GPT-5.6 Sol on SWE Atlas: 46.0 (#5)
GPT-5.6 Sol on Terminal-Bench 2.1: 37.3 (#4)
Gemini 3.7 Flash on AA-LCR: 81.0 (#4)
Gemini 3.7 Flash on AI for Education Visual Maths: 89.45 (#3)
Gemini 3.7 Flash on AI for Education Visual Maths - Algebra: 92.31 (#10)
Gemini 3.7 Flash on AI for Education Visual Maths - Geometry: 90.77 (#2)
Gemini 3.7 Flash on AI for Education Visual Maths - Measurement: 97.3 (#6)
Gemini 3.7 Flash on AI for Education Visual Maths - Number and Operations: 81.08 (#5)
Gemini 3.7 Flash on AI for Education Visual Maths - Statistics and Probability: 57.14 (#11)
Gemini 3.7 Flash on AIIQ Composite IQ: 122.0 (#22)
Gemini 3.7 Flash on Appwrite Arena (With Skills): 96.0 (#13)
Gemini 3.7 Flash on Appwrite Arena (Without Skills): 93.6 (#10)
Gemini 3.7 Flash on BoundaryBench (NIST High): 66.7 (#4)
Gemini 3.7 Flash on BoundaryBench (Unrestricted): 79.8 (#3)
Gemini 3.7 Flash on Bullshit Benchmark: 23.6 (#99)
Gemini 3.7 Flash on CAIS Risk Index: 60.7 (#33)
Gemini 3.7 Flash on CAIS Vision Capabilities Index: 63.7 (#4)
Gemini 3.7 Flash on CritPt: 14.3 (#27)
Gemini 3.7 Flash on Epoch AI - Rli: 5.0 (#5)
Gemini 3.7 Flash on Featherbench: 93.0 (#10)
Gemini 3.7 Flash on Featherbench - Rubric Quality: 8.8 (#8)
Gemini 3.7 Flash on Guesswork 2026-08: 0.9969 (#27)
Gemini 3.7 Flash on OpenCompass Code - Comprehensive: 89.3 (#7)
Gemini 3.7 Flash on OpenCompass Knowledge - Common Sense: 92.9 (#1)
Gemini 3.7 Flash on OpenCompass Knowledge - Engineering: 95.8 (#1)
Gemini 3.7 Flash on OpenCompass Knowledge - Humanities: 94.2 (#3)
Gemini 3.7 Flash on OpenCompass Knowledge - Science: 95.0 (#4)
Gemini 3.7 Flash on OpenCompass Knowledge - Social Science: 92.1 (#4)
Gemini 3.7 Flash on OpenCompass LLM - Code: 89.3 (#7)
Gemini 3.7 Flash on OpenCompass LLM - Knowledge: 94.0 (#2)
Gemini 3.7 Flash on OpenCompass LLM - Math: 76.8 (#2)
Gemini 3.7 Flash on OpenCompass LLM - Reasoning: 62.1 (#9)
Gemini 3.7 Flash on OpenCompass Math - Competition: 67.9 (#9)
Gemini 3.7 Flash on OpenCompass Reasoning - Academic: 47.4 (#11)
Gemini 3.7 Flash on OpenCompass Reasoning - Common: 76.8 (#8)
Gemini 3.7 Flash on SEAL - Remote Labor Index (RLI): 5.0 (#4)
Gemini 3.7 Flash on TaxCalcBench: 12.0 (#14)

New #1 Leaders (51)

ACEBench: GPT-4o (2024-11-20) (89.6) beat Hammer2.1-3B-local by 82.6
Design Arena (Slides): playyy (1316.0) beat Claude 3 Opus by 73.0
GDPval-MM: GPT-5.2 (70.9) beat GPT-4o by 58.6
OpenCompass LLM - Code: GPT-5.5 (High) (96.5) beat Claude Opus 4.7 (High) by 32.8
OpenCompass Code - Comprehensive: GPT-5.5 (High) (96.5) beat Claude Opus 4.7 (High) by 32.8
Lean AI Formalization Leaderboard: Axiom Prover (Axiom Math) (205.0) beat Humanifa + GPT 5.6 sol by 22.0
SWE Atlas: Hy4 preview claude-code (64.0) beat GPT-5.5 by 18.6
PostTrainBench: Claude Fable 5 (Max) (41.79) beat Claude Opus 4.6 by 18.59
Benchmarks.bio - BioSecBench-Refusal: Grok 4.6 (64.2) beat Gemini 3.5 Flash by 12.7
Multi-IF: Llama 3.1 8B (69.0) beat GPT-4.1 Nano by 11.8
SWE-Milestone: GLM-5.3 (58.75) beat Claude Opus 4.8 (Max, 1M) by 6.91
Humanity's Last Exam (Self-Reported, With Tools): GLM-5.3 (62.5) beat Ornith-1.5-397B by 6.4
Crosby micro1 RedlineBench: Claude Opus 5 (56.9) beat GPT-5.5 by 6.4
FEV-Bench - WAPE: TimesFM-3 (85.23) beat Toto-2.0-2.5B by 6.23
AI for Education Visual Maths - Number and Operations: GPT-5.6 Sol (89.19) beat GPT-5.5 by 5.41
WildClawBench: GPT-5.6 Sol (67.2) beat Claude Opus 4.7 by 5.0
OpenCompass Reasoning - Academic: Claude Opus 5 (High) (56.7) beat GPT-5.4 (High) by 4.7
FEV-Bench: TimesFM-3 (85.93) beat Chronos-2 by 4.61
FEV-Bench - WQL: TimesFM-3 (86.37) beat Chronos-2 by 4.58
ArtifactsBench: Ling-3.0-flash (77.0) beat GPT-5 by 4.45
FEV-Bench - MASE: TimesFM-3 (84.97) beat Chronos-2 by 4.11
Chatbot Arena (Search): GPT-5.6 Sol (xHigh) (1257.0) beat Claude Opus 4.6 by 4.0
Featherbench: GLM-5.3 (100.0) beat Gemini 3.6 Flash by 4.0
FEV-Bench - SQL: TimesFM-3 (87.17) beat Chronos-2 by 3.27
OpenCompass LLM - Reasoning: Claude Opus 5 (High) (68.4) beat Doubao-Seed-2-0-Pro-260215 (High) by 3.2
Chatbot Arena (Text-to-Video): Gemini 1.1 Flash (1515.0) beat gemini-omni-flash by 3.0
Chatbot Arena (Video Edit): wan3.0 (1414.0) beat dreamina-seedance-2.5-720p by 3.0
LLM Stats (Toolathlon): GLM-5.3 Flash (78.4) beat Muse Spark 1.1 by 2.8
Open Universal Arabic ASR Leaderboard: Audar-ASR-V1-Turbo (23.17) beat cohere-transcribe-arabic-07-2026 by 2.7
DuelLab Overall: GPT-5.6 Sol (xHigh) (87.3) beat Claude Fable 5 (xHigh) by 2.7
OpenCompass Math - Competition: Claude Opus 5 (High) (74.7) beat Kimi K2.6 by 2.6
MMEB Leaderboard: WeMM-Embedding-9B (59.55) beat Ovis-Omni-Embedding-v0.5-3B by 2.6
Open Universal Arabic ASR Leaderboard - CER: Audar-ASR-V1-Turbo (9.23) beat cohere-transcribe-arabic-07-2026 by 2.57
HieroglyphBench: Gemini 3.7 Flash (55.0) beat Gemini 3.5 Flash by 2.5
AI for Education Visual Maths - Geometry: GPT-5.6 Sol (90.77) beat GPT-5.5 by 2.31
OpenCompass Math - College: Gemini 3.7 Flash (85.6) beat Doubao-Seed-2-0-Pro-260215 (High) by 1.8
AndroidWorld: Gemini 3.7 Flash, Gemini Robotics ER 2 (99.1) beat AGI-0 by 1.7
RealWorldQA: Qwen3.8-Flash-Next (88.5) beat Qwen 3.7 Plus by 1.6
AI for Education Visual Maths: GPT-5.6 Sol (91.14) beat GPT-5.5 by 1.27
LLM Stats (MVBench): GLM-5.3 Flash (77.8) beat Qwen 3.5 122B A10B by 1.2
MMEB Image: WeMM-Embedding-9B (80.99) beat QQMM-embed-v4 by 1.18
Agents' Last Exam: Claude Opus 5 (31.6) beat GPT-5.6 Sol by 1.0
Vals AI ProofBench: alephprover (100.0) beat Claude Opus 5 by 1.0
AIIQ Composite IQ: fable-5 (137.0) beat GPT-5.6 Sol by 1.0
OpenCompass Knowledge - Humanities: Grok 4.6 (High) (95.8) beat Doubao-Seed-2-0-Pro-260215 (High) by 0.8
SecIT Bench (Pydantic AI): GLM-5.3 (81.44) beat Grok 4.6 by 0.79
LLM Stats (AutomationBench): GLM-5.3 Flash (48.8) beat GLM-5.3 by 0.6
Agent Arena - Bash Recovery: Claude Opus 5 (Max) (15.25) beat GPT-5.5 (xHigh) by 0.58
LLM Stats (RealWorldQA): Qwen3.8-Flash-Next (88.5) beat Qwen 3.8 Max by 0.5
Agent Arena - Praise vs Complaint: Claude Opus 4.8 (High) (22.6) beat GPT-5.6 Sol (xHigh) by 0.4
Appwrite Arena (With Skills): Muse Spark 1.2 (97.8) beat GPT-5.5 by 0.1

The Aggregate Digest — 2026-08-29

2026-08-29T11:40:29.518756+00:00

Daily

New Benchmarks (1)

Chatbot Arena (Text) (Elo): leader Claude Fable 5 (1507.0), 395 models
Crowdsourced Elo-rated leaderboard where users blindly compare LLM responses in pairwise battles across diverse real-world text prompts.

Top-10 New Scores (69)

Claude Fable 5 on MCP Atlas: 83.3 (#10)
Claude Fable 5 on OSWorld-Verified: 85.96 (#2)
Claude Mythos 5 on ArxivMath: 78.52 (#3)
Claude Mythos Preview on ArxivMath: 68.7 (#7)
Claude Opus 5 on ArxivMath: 91.33 (#1)
Claude Opus 5 on MCP Atlas: 85.8 (#3)
Claude Opus 5 on OpenCompass Code - Comprehensive: 93.3 (#4)
Claude Opus 5 on OpenCompass Knowledge - Common Sense: 92.1 (#3)
Claude Opus 5 on OpenCompass Knowledge - Engineering: 93.3 (#5)
Claude Opus 5 on OpenCompass Knowledge - Humanities: 92.9 (#4)
Claude Opus 5 on OpenCompass Knowledge - Science: 97.5 (#1)
Claude Opus 5 on OpenCompass Knowledge - Social Science: 94.6 (#1)
Claude Opus 5 on OpenCompass LLM - Code: 93.3 (#4)
Claude Opus 5 on OpenCompass LLM - Knowledge: 94.1 (#1)
Claude Opus 5 on OpenCompass LLM - Math: 77.3 (#1)
Claude Opus 5 on OpenCompass LLM - Reasoning: 68.4 (#1)
Claude Opus 5 on OpenCompass Math - College: 79.9 (#5)
Claude Opus 5 on OpenCompass Math - Competition: 74.7 (#1)
Claude Opus 5 on OpenCompass Reasoning - Academic: 56.7 (#1)
Claude Opus 5 on OpenCompass Reasoning - Common: 80.1 (#2)
GPT-5.6 Sol on Chatbot Arena (Search): 1257.0 (#1)
GPT-5.6 Sol on FrontierCode: 60.6 (#9)
GPT-5.6 Sol on MCP Atlas: 81.8 (#14)
GPT-5.6 Sol on OpenCompass Code - Comprehensive: 96.3 (#2)
GPT-5.6 Sol on OpenCompass Knowledge - Common Sense: 86.3 (#11)
GPT-5.6 Sol on OpenCompass Knowledge - Engineering: 84.2 (#12)
GPT-5.6 Sol on OpenCompass Knowledge - Humanities: 90.8 (#9)
GPT-5.6 Sol on OpenCompass Knowledge - Science: 89.2 (#11)
GPT-5.6 Sol on OpenCompass Knowledge - Social Science: 92.1 (#3)
GPT-5.6 Sol on OpenCompass LLM - Code: 96.3 (#2)
GPT-5.6 Sol on OpenCompass LLM - Knowledge: 88.5 (#9)
GPT-5.6 Sol on OpenCompass LLM - Math: 71.4 (#10)
GPT-5.6 Sol on OpenCompass LLM - Reasoning: 66.8 (#3)
GPT-5.6 Sol on OpenCompass Math - College: 74.3 (#13)
GPT-5.6 Sol on OpenCompass Math - Competition: 68.6 (#6)
GPT-5.6 Sol on OpenCompass Reasoning - Academic: 55.8 (#2)
GPT-5.6 Sol on OpenCompass Reasoning - Common: 77.8 (#6)
GPT-5.6 Sol on SWE Atlas: 46.0 (#5)
GPT-5.6 Sol on WildClawBench: 67.2 (#1)
Gemini 3.7 Flash on OpenCompass Code - Comprehensive: 89.3 (#7)
Gemini 3.7 Flash on OpenCompass Knowledge - Common Sense: 92.9 (#1)
Gemini 3.7 Flash on OpenCompass Knowledge - Engineering: 95.8 (#1)
Gemini 3.7 Flash on OpenCompass Knowledge - Humanities: 94.2 (#3)
Gemini 3.7 Flash on OpenCompass Knowledge - Science: 95.0 (#4)
Gemini 3.7 Flash on OpenCompass Knowledge - Social Science: 92.1 (#4)
Gemini 3.7 Flash on OpenCompass LLM - Code: 89.3 (#7)
Gemini 3.7 Flash on OpenCompass LLM - Knowledge: 94.0 (#2)
Gemini 3.7 Flash on OpenCompass LLM - Math: 76.8 (#2)
Gemini 3.7 Flash on OpenCompass LLM - Reasoning: 62.1 (#9)
Gemini 3.7 Flash on OpenCompass Math - Competition: 67.9 (#9)
Gemini 3.7 Flash on OpenCompass Reasoning - Academic: 47.4 (#11)
Gemini 3.7 Flash on OpenCompass Reasoning - Common: 76.8 (#8)
Kimi K3 on APEX-Agents-AA: 0.41 (#30)
Kimi K3 on LM Market Cap LMC Score: 40.0 (#248)
Kimi K3 on OpenCompass Code - Comprehensive: 88.0 (#8)
Kimi K3 on OpenCompass Knowledge - Common Sense: 87.9 (#8)
Kimi K3 on OpenCompass Knowledge - Engineering: 88.3 (#8)
Kimi K3 on OpenCompass Knowledge - Humanities: 92.1 (#7)
Kimi K3 on OpenCompass Knowledge - Science: 92.5 (#6)
Kimi K3 on OpenCompass Knowledge - Social Science: 90.4 (#5)
Kimi K3 on OpenCompass LLM - Code: 88.0 (#8)
Kimi K3 on OpenCompass LLM - Knowledge: 90.3 (#7)
Kimi K3 on OpenCompass LLM - Math: 74.1 (#6)
Kimi K3 on OpenCompass LLM - Reasoning: 63.9 (#7)
Kimi K3 on OpenCompass Math - College: 79.2 (#9)
Kimi K3 on OpenCompass Math - Competition: 68.9 (#5)
Kimi K3 on OpenCompass Reasoning - Academic: 51.7 (#6)
Kimi K3 on OpenCompass Reasoning - Common: 76.0 (#9)
Kimi K3 on WildClawBench: 54.5 (#12)

New #1 Leaders (13)

OpenCompass LLM - Code: GPT-5.5 (High) (96.5) beat Claude Opus 4.7 (High) by 32.8
OpenCompass Code - Comprehensive: GPT-5.5 (High) (96.5) beat Claude Opus 4.7 (High) by 32.8
Lean AI Formalization Leaderboard: Axiom Prover (Axiom Math) (205.0) beat Humanifa + GPT 5.6 sol by 22.0
SWE Atlas: Hy4 preview claude-code (64.0) beat GPT-5.5 by 18.6
Design Arena (Video): Gemini 1.1 Flash (1377.0) beat Gemini 2.0 Flash by 9.0
Humanity's Last Exam (Self-Reported, With Tools): GLM-5.3 (62.5) beat Ornith-1.5-397B by 6.4
OpenCompass Reasoning - Academic: Claude Opus 5 (High) (56.7) beat GPT-5.4 (High) by 4.7
Chatbot Arena (Search): GPT-5.6 Sol (xHigh) (1257.0) beat Claude Opus 4.6 by 4.0
OpenCompass LLM - Reasoning: Claude Opus 5 (High) (68.4) beat Doubao-Seed-2-0-Pro-260215 (High) by 3.2
OpenCompass Math - Competition: Claude Opus 5 (High) (74.7) beat Kimi K2.6 by 2.6
OpenCompass Math - College: Gemini 3.7 Flash (85.6) beat Doubao-Seed-2-0-Pro-260215 (High) by 1.8
AndroidWorld: Gemini 3.7 Flash, Gemini Robotics ER 2 (99.1) beat AGI-0 by 1.7
OpenCompass Knowledge - Humanities: Grok 4.6 (High) (95.8) beat Doubao-Seed-2-0-Pro-260215 (High) by 0.8

The Aggregate Digest — 2026-08-26

2026-08-26T18:35:49.913992+00:00

Daily

New Benchmarks (2)

SWE Refactor Bench (Score (out of 100)): leader Claude Opus 5 (47.0), 8 models
A whole-repository stack migration held open for a hundred-odd steps: change the framework the codebase is built on and leave it working. Every model is run at each of its harness's effort tiers, and the best any of them manages is 47 out of 100 — the behavioural test suite mostly passes, the migration mostly does not.
EarthVerse (Core Score (%)): leader Claude Fable 5 (84.97), 18 models
Earth-system and natural-hazard analysis with real observational data: reconstruct what happened from sources that disagree on scale, timing and modality. Scored on the answer and the trajectory that produced it, so a right number reached by a wrong route does not count.

Top-10 New Scores (3)

GPT-5.6 Pro Sol on Epoch AI - ECI: 161.08 (#26)
Gemini 3.7 Flash on Bullshit Benchmark: 23.6 (#99)
Gemini 3.7 Flash on Epoch AI - Rli: 5.0 (#5)

New #1 Leaders (5)

Design Arena (Slides): playyy (1304.0) beat Claude 3 Opus by 61.0
ArtifactsBench: Ling-3.0-flash (77.0) beat GPT-5 by 4.45
HieroglyphBench: Gemini 3.7 Flash (55.0) beat Gemini 3.5 Flash by 2.5
BEAR-Bench: Qwen 3.5 397B A17B (75.4) beat Gemini 3.1 Pro (Preview) by 0.3
Longitudinal MRI Progression Reasoning: InternVL3.5-Inst (35.15) beat Gemini 3 Pro by 0.05

The Aggregate Digest — 2026-08-25

2026-08-25T06:17:16.148062+00:00

Daily

New Benchmarks (3)

DGEval - IMDG Code MCQ (Accuracy (%)): leader Gemini 3.1 Pro (Preview) (95.8), 10 models
The closed-book multiple-choice half of a dangerous-goods shipping certification, asked exactly as human candidates get it. Certified practitioners score 83.8 percent, which is the number every model on the board is measured against — and most of them are below it.
DGEval - Regulatory Recall (Accuracy (%)): leader Gemini 3.1 Pro (Preview) (36.8), 10 models
The section of the same maritime dangerous-goods exam that nothing saturates: name the IMDG Code provision a scenario falls under, from memory. Scores run from 0 to 37 percent, so it separates the frontier where the multiple-choice half no longer does.
VIALS (Direct VLM Accuracy (%)): leader GPT-5.6 Sol (26.5), 7 models
Gel blots, microscopy, plasmid maps, flow cytometry plots — the visual artifacts a working life scientist reads before deciding what to do next. Scored as direct vision-language accuracy with no tools, where the best model manages roughly a quarter of the questions.

Top-10 New Scores (25)

Claude Fable 5 on AI for Education Visual Maths: 86.92 (#4)
Claude Fable 5 on AI for Education Visual Maths - Algebra: 92.31 (#16)
Claude Fable 5 on AI for Education Visual Maths - Geometry: 83.85 (#5)
Claude Fable 5 on AI for Education Visual Maths - Measurement: 97.3 (#5)
Claude Fable 5 on AI for Education Visual Maths - Number and Operations: 86.49 (#2)
Claude Fable 5 on AI for Education Visual Maths - Statistics and Probability: 71.43 (#3)
Claude Opus 5 on AI for Education Visual Maths: 80.59 (#10)
Claude Opus 5 on AI for Education Visual Maths - Algebra: 96.15 (#3)
Claude Opus 5 on AI for Education Visual Maths - Geometry: 75.38 (#13)
Claude Opus 5 on AI for Education Visual Maths - Measurement: 94.59 (#12)
Claude Opus 5 on AI for Education Visual Maths - Number and Operations: 75.68 (#6)
Claude Opus 5 on AI for Education Visual Maths - Statistics and Probability: 71.43 (#2)
Claude Opus 5 on Roboflow Playground Open Prompt: 1200.0 (#22)
GPT-5.6 Sol on AI for Education Visual Maths - Algebra: 96.15 (#2)
GPT-5.6 Sol on AI for Education Visual Maths - Measurement: 97.3 (#2)
GPT-5.6 Sol on AI for Education Visual Maths - Statistics and Probability: 57.14 (#10)
Gemini 3.7 Flash on AI for Education Visual Maths: 89.45 (#3)
Gemini 3.7 Flash on AI for Education Visual Maths - Algebra: 92.31 (#10)
Gemini 3.7 Flash on AI for Education Visual Maths - Geometry: 90.77 (#2)
Gemini 3.7 Flash on AI for Education Visual Maths - Measurement: 97.3 (#6)
Gemini 3.7 Flash on AI for Education Visual Maths - Number and Operations: 81.08 (#5)
Gemini 3.7 Flash on AI for Education Visual Maths - Statistics and Probability: 57.14 (#11)
Gemini 3.7 Flash on ProphetArena: 0.96 (#2)
Gemini 3.7 Flash on SEAL - Remote Labor Index (RLI): 5.0 (#4)
Gemini 3.7 Flash on TaxCalcBench: 12.0 (#14)

New #1 Leaders (5)

Benchmarks.bio - BioSecBench-Refusal: Grok 4.6 (64.2) beat Gemini 3.5 Flash by 12.7
AI for Education Visual Maths - Number and Operations: GPT-5.6 Sol (89.19) beat GPT-5.5 by 5.41
AI for Education Visual Maths - Geometry: GPT-5.6 Sol (90.77) beat GPT-5.5 by 2.31
AI for Education Visual Maths: GPT-5.6 Sol (91.14) beat GPT-5.5 by 1.27
SecIT Bench (Pydantic AI): GLM-5.3 (81.44) beat Grok 4.6 by 0.79

The Aggregate Digest — 2026-08-24

2026-08-24T10:31:51.987898+00:00

Daily

New Benchmarks (6)

BenchGecko Score (Composite Score): leader GPT-5.5 Pro (99.9), 20 models
BenchGecko's own composite index over the public boards it tracks, mirrored here the way LLM Stats Score and BenchmarkList ECI are. Coverage per model is uneven — between 3 and 14 boards feed each score — so it reads as one aggregator's view rather than an independent measurement.
SAST False-Positive Triage (Accuracy (%)): leader Kimi K3 (92.0), 15 models
A static analyser fires 142 findings over a codebase and 11 of them are real. The model has to keep the genuine vulnerabilities and dismiss the noise, so the accuracy column punishes both misses and credulity. Published as a blog table, which is why the roster is small and current.
StartupBench (Score (%)): leader Kimi K3 (73.67), 9 models
Ninety-seven market-research tasks drawn from real startup work across medicine, finance, legal, business, STEM and education, scored per domain and overall. Tasks are open-ended and judged item-wise on process logic rather than a single final answer, which is what separates it from the GAIA-style agent boards.
Agentic Commerce World (Overall (%)): leader GPT-5.6 Sol (85.6), 9 models
Vibe commerce: state a buying or selling goal in natural language and delegate the transaction to an agent. Runs in an auditable, verifiable environment because commerce failures cost money rather than a retry.
VibeLifeBench (Average of 3 runs (%)): leader Claude Opus 5 (32.5), 7 models
Personal-assistant tasks that run for weeks in a world that changes underneath them, scored on proactivity and persistence. Deliberately unlike the short self-contained requests in a static environment that most agent evaluations use.
BusinessCaseBench (Score (%)): leader Claude Sonnet 4.6 (88.4), 7 models
Case-grounded professional knowledge work across the business disciplines - the judgement calls that benchmarks built on recall, narrow QA, maths and coding do not reach.

Top-10 New Scores (2)

Gemini 3.7 Flash on BoundaryBench (NIST High): 66.7 (#4)
Gemini 3.7 Flash on BoundaryBench (Unrestricted): 79.8 (#3)

The Aggregate Digest — 2026-08-23

2026-08-23T07:31:51.012044+00:00

Daily

Top-10 New Scores (6)

Claude Opus 5 on Agents' Last Exam: 27.0 (#7)
Gemini 3.7 Flash on Arena AI Code: 1587.0 (#10)
Gemini 3.7 Flash on LMArena Text Arena: 1489.87 (#9)
Gemini 3.7 Flash on LMArena WebDev Arena: 1587.86 (#9)
Kimi K3 on Benchmarks.bio - BioSecBench-Refusal: 8.6 (#16)
Kimi K3 on Benchmarks.bio - BioSecBench-Surveillance: 48.3 (#12)

New #1 Leaders (2)

MobileWorld - GUI Only: Qwen-UI-Agent (82.1) beat Kimi K3 by 7.7
WebVoyager: browser-control (Fable 5) (99.19) beat Alumnium by 0.69

Weekly

New Benchmarks (1)

AtmosCoder-Bench (Accuracy (%, mean of 3 runs)): leader GPT-5.5 (97.6), 16 models
AtmosCoder-Bench grades quantitative atmospheric science by execution: each of 436 textbook-grounded problems is answered by a Python solve() the model writes, and the graded number is whatever that function computes, with the declared unit reconciled against ground truth. Three runs per model; accuracy (%) is the mean. Reasoning and thinking-off runs of the same checkpoint are listed separately, as upstream measures them.

New Models (123)

Claude Opus 5 — ELO 1792, #1
- Vals AI MedCode: 63.57 (#1/86)
- Vals AI MedScribe: 90.98 (#1/87)
- Vals AI MMMU: 89.88 (#1/90)
- LiveMathematicianBench: 45.1 (#3/19)
- Vals AI LegalBench: 86.97 (#5/139)
- Agents' Last Exam: 27.0 (#7/25)
- SpacetimeDB LLM Benchmark (TypeScript): 96.4 (#9/14)
- SpacetimeDB LLM Benchmark (Rust): 68.3 (#9/14)
- SpacetimeDB LLM Benchmark (C#): 66.7 (#12/14)
- Epoch AI - Cursorbench: 64.3 (#16/67)
- ...and 1 more
GPT-5.5 Pro — ELO 1786, #2
- OpenAI GPT-5.5 Launch - FrontierMath Tier 1-3: 52.4 (#1/6)
- OpenAI GPT-5.5 Launch - FrontierMath Tier 4: 39.6 (#1/6)
- AA CritPt: 30.57 (#2/490)
Claude Fable 5 — ELO 1783, #3
- FutureEval: 13.23 (#1/43)
- SpacetimeDB LLM Benchmark (C#): 98.9 (#1/14)
- Epoch AI - Enigma Eval: 39.28 (#1/46)
- ArxivMath: 78.6 (#2/11)
- FrontierCode: 46.3 (#3/22)
- CyberGym: 83.8 (#3/23)
- APEX v1: 66.0 (#4/5)
- APEX v1 Investment Banking: 66.6 (#4/39)
- APEX v1 Consulting: 66.6 (#5/39)
- SpacetimeDB LLM Benchmark (Rust): 68.3 (#8/14)
- ...and 4 more
GPT-5.6 Sol — ELO 1777, #4
- SpacetimeDB LLM Benchmark (TypeScript): 99.5 (#1/14)
- Design Arena (UI Components): 1387.0 (#1/152)
- Design Arena (Game Dev): 1414.0 (#2/157)
- Design Arena (3D): 1438.0 (#2/145)
- Epoch AI - Enigma Eval: 37.12 (#2/46)
- CyberGym: 83.6 (#4/23)
- Design Arena (Data Viz): 1338.0 (#5/153)
- Toolathlon: 74.9 (#6/36)
- SpacetimeDB LLM Benchmark (Rust): 82.9 (#7/14)
- SpacetimeDB LLM Benchmark (C#): 92.2 (#9/14)
- ...and 4 more
Gemini 3.7 Flash — ELO 1777, #5
- ProphetArena: 0.9687 (#1/19)
- Icelandic LLM Leaderboard - Average: 88.71 (#1/94)
- Icelandic LLM - WinoGrande-IS: 96.32 (#1/94)
- Icelandic LLM - ARC-Challenge-IS: 95.22 (#1/94)
- Kaggle FACTS (Google): 71.27 (#1/33)
- Kaggle FACTS Multimodal: 49.83 (#1/33)
- AI for Education Visual Reasoning: 89.9 (#1/67)
- AI for Education Visual Reasoning - match (figure): 87.4 (#1/67)
- AI for Education Visual Reasoning - match (process): 88.9 (#1/67)
- AI for Education Visual Reasoning - pattern completion (linear): 95.7 (#1/67)
- ...and 60 more
Kimi K3 — ELO 1761, #7
- APEX v1: 68.4 (#2/5)
- APEX v1 Medicine (MD): 64.3 (#2/39)
- Toolathlon: 76.5 (#2/36)
- APEX v1 Consulting: 69.7 (#3/39)
- NL2Repo: 58.0 (#3/25)
- AI for Education Visual Reasoning - odd one out: 79.2 (#4/67)
- APEX v1 Investment Banking: 66.4 (#5/39)
- AI for Education Visual Reasoning - reasoning by analogy: 83.7 (#5/67)
- APEX v1 Big Law: 73.2 (#7/39)
- AI for Education Visual Reasoning - match (process): 75.0 (#7/67)
- ...and 11 more
Grok 4.6 — ELO 1757, #8
- AI for Education Pedagogy - Science: 96.72 (#1/235)
- AI for Education Pedagogy - Secondary: 91.19 (#1/235)
- Epoch AI - Cursorbench: 70.8 (#3/67)
- AI for Education Pedagogy: 91.55 (#3/235)
- EnterpriseBench (CoreCraft Agents): 65.6 (#3/3)
- Epoch AI - Vending Bench 2: 9047.03 (#4/58)
- AA GDPval: 1765.76 (#4/210)
- KernelBench Hub - Hard: 62.17 (#5/13)
- Epoch AI - Apex Agents: 41.2 (#5/61)
- Epoch AI - Blueprint Bench 2: 0.332 (#5/23)
- ...and 33 more
Qwen 3.8 Max — ELO 1755, #9
- WebDev Arena: 1666.72 (#4/111)
- Agent Arena - Confirmed Success: 11.62 (#6/51)
- Epoch AI - Proofbench: 58.0 (#9/22)
- CyberGym: 78.5 (#11/23)
- Agent Arena: 6.2 (#15/51)
- Agent Arena - Praise vs Complaint: 6.81 (#17/51)
- Agent Arena - Steerability: 4.0 (#19/51)
- Agent Arena - Bash Recovery: 8.39 (#23/51)
- Agent Arena - Tool Hallucination: -0.18 (#39/51)
GPT-5.5 — ELO 1749, #10
- OpenAI GPT-5.5 Launch - FrontierMath Tier 1-3: 51.7 (#2/6)
- OpenAI GPT-5.5 Launch - FrontierMath Tier 4: 35.4 (#3/6)
- SpacetimeDB LLM Benchmark (TypeScript): 97.8 (#4/14)
- SpacetimeDB LLM Benchmark (Rust): 96.6 (#4/14)
- SpacetimeDB LLM Benchmark (C#): 96.6 (#5/14)
- ArxivMath: 72.2 (#5/11)
- Epoch AI - Apex Agents: 38.5 (#10/61)
- ParseBench: 64.39 (#17/56)
Muse Spark 1.2 — ELO 1749, #11
- Agent Arena - Bash Recovery: 11.4 (#10/51)
- Agent Arena - Confirmed Success: 6.02 (#14/51)
- Agent Arena - Tool Hallucination: -1.13 (#17/51)
- Creative Writing (Lechmazur): 0.6 (#20/47)
- Agent Arena: 2.06 (#25/51)
- Agent Arena - Steerability: -2.51 (#32/51)
- Agent Arena - Praise vs Complaint: -5.72 (#35/51)
Gemini 3.6 Flash — ELO 1747, #12
- APEX v1 Investment Banking: 69.2 (#2/39)
- AI for Education Visual Reasoning - pattern completion (2d): 83.6 (#2/67)
- AI for Education Visual Reasoning - pattern completion (linear): 92.3 (#2/67)
- LVBench: 84.2 (#2/53)
- AI for Education Visual Reasoning: 85.7 (#3/67)
- AI for Education Visual Reasoning - match (figure): 84.7 (#3/67)
- AI for Education Visual Reasoning - reasoning by analogy: 87.9 (#3/67)
- AI for Education Visual Reasoning - match (process): 77.8 (#4/67)
- BioMysteryBench Human-Difficult: 41.2 (#4/11)
- APEX v1: 66.0 (#5/5)
- ...and 12 more
Muse Spark 1.1 — ELO 1744, #13
- Roboflow Playground Open Prompt: 1188.0 (#43/54)
GLM-5.3 — ELO 1744, #14
- CyberGym: 84.5 (#1/23)
- KernelBench Hub - Mega: 21.39 (#2/9)
- Creative Writing (Lechmazur): 3.5 (#2/47)
- Tau3 Banking: 50.31 (#3/176)
- AA GDPval: 1769.0 (#3/210)
- NL2Repo: 58.0 (#4/25)
- DeepSWE: 69.0 (#6/28)
- Agents on Rails: 79.4 (#6/12)
- Vals AI SWE-bench Verified: 95.4 (#6/86)
- Chatbot Arena (Code): 1599.0 (#8/118)
- ...and 33 more
Gemini 3.1 Pro (Preview) — ELO 1739, #16
- ArxivMath: 64.79 (#8/11)
- GAIA: 55.48 (#591/3523)
Claude Opus 4.8 — ELO 1735, #17
- SpacetimeDB LLM Benchmark (TypeScript): 97.8 (#2/14)
- APEX v1 Medicine (MD): 64.0 (#3/39)
- SpacetimeDB LLM Benchmark (C#): 96.6 (#4/14)
- SpacetimeDB LLM Benchmark (Rust): 94.4 (#5/14)
- Epoch AI - Enigma Eval: 23.51 (#5/46)
- ArxivMath: 71.0 (#6/11)
- Agents on Rails: 79.4 (#7/12)
- APEX v1 Consulting: 62.6 (#9/39)
- Arena AI Document: 1475.0 (#11/38)
- WebDev Arena: 1564.26 (#13/111)
- ...and 6 more
GPT-5.6 Terra — ELO 1735, #18
- APEX v1: 69.5 (#1/5)
- APEX v1 Consulting: 76.2 (#1/39)
- APEX v1 Investment Banking: 71.8 (#1/39)
- BioMysteryBench Human-Difficult: 49.4 (#1/11)
- MRCR-v2 128k: 93.5 (#2/20)
- LVBench: 78.9 (#3/53)
- Vals AI ProgramBench: 72.34 (#4/41)
- Vals AI Code Migration: 47.8 (#4/51)
- BioMysteryBench Human-Solvable: 83.8 (#4/11)
- Vals AI SWE-bench Verified: 95.4 (#5/86)
- ...and 18 more
Gemini 3.5 Flash — ELO 1734, #19
- LiveMathematicianBench: 49.5 (#1/19)
- APEX v1 Investment Banking: 67.4 (#3/39)
- Epoch AI - Enigma Eval: 25.41 (#3/46)
- APEX v1 Medicine (MD): 61.7 (#11/39)
- SpacetimeDB LLM Benchmark (C#): 50.6 (#14/14)
- SpacetimeDB LLM Benchmark (TypeScript): 44.9 (#14/14)
- SpacetimeDB LLM Benchmark (Rust): 49.4 (#14/14)
- FutureEval: 10.01 (#17/43)
- APEX v1 Big Law: 66.2 (#21/39)
- APEX v1 Consulting: 53.6 (#25/39)
- ...and 1 more
DeepSeek V4 Pro (0813) — ELO 1734, #20
- ProphetArena: 0.9661 (#2/19)
- Epoch AI - Mystery Game Puzzles: 43.0 (#6/54)
- Riemann-bench: 38.4 (#9/30)
- OTIS Mock AIME 2024-25: 98.61 (#10/239)
- WebDev Arena: 1584.45 (#10/111)
- Chess Puzzles (Epoch AI): 47.0 (#11/162)
- Epoch AI - Proofbench: 50.0 (#12/22)
- SimpleQA Verified: 55.47 (#18/78)
- FrontierMath - Tiers 1-3 (v2): 64.56 (#20/49)
- FrontierMath - Tier 4 (v2): 26.83 (#23/51)
- ...and 3 more
Grok 4.5 — ELO 1729, #21
- Roboflow Playground Object Detection: 1200.0 (#30/48)
- Roboflow Playground Overall: 1200.0 (#40/72)
Qwen3.8 2.4T A95B — ELO 1725, #23
- WeirdML: 75.24 (#21/155)
- Arabic Broad Leaderboard: 8.862 (#23/109)
GPT-5.4 — ELO 1713, #27
- OpenAI GPT-5.4 Launch - FrontierMath Tier 1-3: 47.6 (#2/3)
- LiveMathematicianBench: 46.6 (#2/19)
- OpenAI GPT-5.4 Launch - FrontierMath Tier 4: 27.1 (#3/4)
- OpenAI GPT-5.5 Launch - FrontierMath Tier 1-3: 47.6 (#4/6)
- OpenAI GPT-5.5 Launch - FrontierMath Tier 4: 27.1 (#4/6)
- Epoch AI - Enigma Eval: 15.96 (#9/46)
- ParseBench: 62.23 (#23/56)
Claude Opus 4.7 — ELO 1711, #28
- LMArena Text Arena: 1502.01 (#3/24)
- OpenAI GPT-5.5 Launch - FrontierMath Tier 1-3: 43.8 (#5/6)
- OpenAI GPT-5.5 Launch - FrontierMath Tier 4: 22.9 (#5/6)
- Arena AI Document: 1497.0 (#6/38)
- scBench: 55.3 (#6/17)
- APEX v1 Medicine (MD): 63.3 (#7/39)
- APEX v1 Big Law: 70.9 (#10/39)
- APEX v1 Consulting: 61.0 (#15/39)
- WebDev Arena: 1557.31 (#15/111)
- LMArena WebDev Arena: 1557.29 (#15/25)
- ...and 3 more
Claude Sonnet 5 — ELO 1711, #29
- BioMysteryBench Human-Solvable: 87.5 (#1/11)
- APEX v1 Medicine (MD): 63.9 (#5/39)
- BioMysteryBench Human-Difficult: 34.1 (#6/11)
- FutureEval: 11.34 (#10/43)
- LVBench: 68.5 (#16/53)
- AA GDPval: 1502.39 (#33/210)
- APEX v1 Big Law: 54.4 (#34/39)
- Epoch AI - Cursorbench: 56.9 (#38/67)
- APEX v1 Consulting: 41.1 (#39/39)
- APEX v1 Investment Banking: 39.1 (#39/39)
GPT-5.6 Luna — ELO 1709, #30
- CritPt: 20.6 (#12/213)
- Epoch AI - Cursorbench: 61.1 (#26/67)
- ParseBench: 56.32 (#30/56)
Qwen 3.8 27B — ELO 1706, #33
- BRIDGE Medical Leaderboard - Few-Shot: 55.67 (#1/109)
- BRIDGE Medical Leaderboard: 48.31 (#2/109)
- BRIDGE Medical Leaderboard - Zero-Shot: 46.61 (#2/109)
- RealWorldQA: 85.9 (#2/14)
- BRIDGE Medical Leaderboard - CoT: 44.92 (#3/109)
- OpenRouter Tau2-Bench Airline: 80.7 (#3/114)
- OSWorld-Verified: 84.3 (#3/29)
- Tau3 Banking: 48.04 (#5/176)
- Vals AI Harvey Legal Agent Bench: 11.25 (#5/52)
- Vals AI SAGE: 52.4 (#6/77)
- ...and 53 more
GPT-5 Pro — ELO 1705, #35
- Epoch AI - Enigma Eval: 18.75 (#7/46)
Claude Opus 4.6 — ELO 1703, #38
- LMArena Text Arena: 1504.49 (#2/24)
- Arena AI Document: 1506.0 (#3/38)
- NL2Repo: 47.6 (#10/25)
- RealWorldQA: 73.9 (#11/14)
- LMArena WebDev Arena: 1545.76 (#17/25)
- OSWorld-Verified: 72.7 (#17/29)
- Epoch AI - Enigma Eval: 7.6 (#18/46)
- Arena AI Code: 1546.0 (#18/48)
- WebDev Arena: 1544.94 (#21/111)
- ComplexConstraints: 36.3 (#24/57)
- ...and 1 more
Gemini 3 Flash — ELO 1703, #39
- DystopiaBench: 77.0 (#40/42)
GLM-5.2 — ELO 1703, #40
- APEX v1 Investment Banking: 62.6 (#10/39)
- APEX v1 Consulting: 62.1 (#12/39)
- CyberGym: 77.2 (#12/23)
- FutureEval: 9.29 (#24/43)
- APEX v1 Big Law: 64.9 (#25/39)
- APEX v1 Medicine (MD): 56.3 (#27/39)
- LM Market Cap LMC Score: 82.7 (#76/428)
Kimi K2.7 Code — ELO 1703, #41
- APEX v1 Consulting: 61.7 (#13/39)
- APEX v1 Investment Banking: 59.4 (#18/39)
- APEX v1 Big Law: 65.1 (#24/39)
- APEX v1 Medicine (MD): 54.2 (#30/39)
- Epoch AI - Cursorbench: 49.7 (#55/67)
Grok Build 0.1 — ELO 1702, #42
- SpacetimeDB LLM Benchmark (Rust): 97.8 (#2/14)
- SpacetimeDB LLM Benchmark (C#): 95.5 (#6/14)
- SpacetimeDB LLM Benchmark (TypeScript): 97.8 (#6/14)
Qwen 3.7 Plus — ELO 1698, #43
- OmniDocBench 1.5: 91.4 (#9/67)
- CharXiv-R: 85.9 (#12/61)
- OSWorld-Verified: 73.3 (#15/29)
- NL2Repo: 41.1 (#18/25)
- Design Arena (SVG): 1260.0 (#21/110)
- Roboflow Playground Object Detection: 1200.0 (#29/48)
- Roboflow Playground Overall: 1200.0 (#39/72)
- Epoch AI - ECI: 148.26 (#183/491)
DeepSeek V4 Flash (0731) — ELO 1697, #44
- BenCzechMark: 81.36 (#3/74)
- KernelBench Hub - Hard: 48.6 (#9/13)
- Epoch AI - Proofbench: 56.0 (#10/22)
- WebDev Arena: 1576.54 (#11/111)
- Epoch AI - ECI: 154.43 (#106/491)
- ARC-AGI-2: 2.08 (#154/214)
- ARC-AGI-1: 11.83 (#191/212)
Claude Opus 4.5 — ELO 1693, #45
- NarrativeWorldBench: 80.0 (#3/10)
GPT-5 — ELO 1692, #46
- LiveMathematicianBench: 32.2 (#7/19)
- Epoch AI - Enigma Eval: 10.47 (#14/46)
- MMMU Benchmark: 74.4 (#24/189)
- Epoch AI - Apex Agents: 18.3 (#37/61)
GPT-5.2 — ELO 1691, #48
- OpenAI GPT-5.4 Launch - FrontierMath Tier 1-3: 40.7 (#3/3)
- OpenAI GPT-5.4 Launch - FrontierMath Tier 4: 18.8 (#4/4)
- LiveMathematicianBench: 26.3 (#15/19)
- Epoch AI - Enigma Eval: 10.39 (#15/46)
- Epoch AI - Apex Agents: 23.0 (#31/61)
Kimi K2.6 — ELO 1691, #49
- Inverse Turing Bench: 57.45 (#12/29)
- Finance Agent v1.1: 57.06 (#14/53)
DeepSeek V4 Pro — ELO 1690, #51
- LiveMathematicianBench: 36.9 (#5/19)
- HANDBOOK.md Agents: 26.5 (#6/45)
- ComplexConstraints: 42.0 (#12/57)
- APEX v1 Investment Banking: 60.9 (#13/39)
- APEX v1 Consulting: 58.3 (#21/39)
- APEX v1 Big Law: 64.6 (#27/39)
- APEX v1 Medicine (MD): 54.8 (#28/39)
Claude Opus 4.5 (20251101) — ELO 1688, #53
- Epoch AI - Enigma Eval: 11.91 (#11/46)
- Arena AI Code: 1494.0 (#35/48)
Inkling — ELO 1685, #54
- APEX v1 Medicine (MD): 60.4 (#16/39)
- APEX v1 Big Law: 67.9 (#18/39)
- Riemann-bench: 15.2 (#19/30)
- APEX v1 Investment Banking: 55.6 (#21/39)
- APEX v1 Consulting: 48.4 (#33/39)
- HANDBOOK.md Agents: 2.3 (#42/45)
- Chartography: 10.0 (#44/45)
GLM-5.1 — ELO 1683, #57
- Finance Agent v1.1: 57.66 (#12/53)
Qwen 3.5 397B A17B — ELO 1681, #59
- ABRA: 70.0 (#1/10)
- Onyx Open LLM Leaderboard: 87.8 (#2/19)
- DystopiaBench: 58.9 (#16/42)
Claude Sonnet 4.6 — ELO 1677, #61
- SpacetimeDB LLM Benchmark (C#): 98.9 (#2/14)
- SpacetimeDB LLM Benchmark (TypeScript): 97.8 (#3/14)
- SpacetimeDB LLM Benchmark (Rust): 96.6 (#3/14)
- scBench: 50.4 (#10/17)
- ComplexConstraints: 34.0 (#34/57)
O3 — ELO 1676, #62
- Epoch AI - Enigma Eval: 13.09 (#10/46)
MiniMax-M3 — ELO 1675, #65
- KernelBench Hub - Mega: 2.63 (#8/9)
- APEX v1 Medicine (MD): 61.3 (#14/39)
- APEX v1 Consulting: 59.8 (#19/39)
- APEX v1 Big Law: 64.9 (#26/39)
- APEX v1 Investment Banking: 53.1 (#28/39)
GPT-5.3 — ELO 1673, #67
- AI Chess Leaderboard (Continuation): 948.0 (#52/259)
- AI Chess Leaderboard (Reasoning): 1023.0 (#61/318)
Gemini 3.5 Flash Lite — ELO 1671, #68
- Epoch AI - ECI: 145.11 (#238/491)
Claude Sonnet 4.5 — ELO 1669, #69
- Epoch AI - Enigma Eval: 6.0 (#23/46)
GPT-5.1 — ELO 1668, #70
- Epoch AI - Enigma Eval: 11.23 (#13/46)
- Epoch AI - Apex Agents: 17.5 (#39/61)
GPT-5.4 Mini — ELO 1668, #71
- SpacetimeDB LLM Benchmark (Rust): 92.1 (#6/14)
- SpacetimeDB LLM Benchmark (TypeScript): 96.6 (#8/14)
- SpacetimeDB LLM Benchmark (C#): 88.8 (#11/14)
DeepSeek V4 Flash — ELO 1668, #72
- APEX v1 Consulting: 69.8 (#2/39)
- APEX v1 Investment Banking: 66.1 (#6/39)
- ProphetArena: 0.9503 (#9/19)
- APEX v1 Big Law: 65.7 (#23/39)
- APEX v1 Medicine (MD): 53.5 (#31/39)
Grok 4.3 — ELO 1667, #75
- SpacetimeDB LLM Benchmark (Rust): 97.8 (#1/14)
- SpacetimeDB LLM Benchmark (TypeScript): 97.8 (#5/14)
- SpacetimeDB LLM Benchmark (C#): 93.3 (#7/14)
O3 (2025-04-16) — ELO 1667, #76
- LiveMathematicianBench: 27.0 (#13/19)
Inkling Small — ELO 1667, #77
- Agent Arena - Bash Recovery: 10.73 (#15/51)
- Agent Arena - Tool Hallucination: -0.36 (#36/51)
- Agent Arena: -6.82 (#40/51)
- Agent Arena - Praise vs Complaint: -14.84 (#46/51)
- Agent Arena - Steerability: -12.81 (#47/51)
- Agent Arena - Confirmed Success: -17.56 (#51/51)
- Chatbot Arena (Code): 1402.0 (#64/118)
- Chatbot Arena (Vision): 1206.0 (#70/147)
- Chatbot Arena (Text): 1405.0 (#136/394)
- Epoch AI - ECI: 150.17 (#140/491)
Muse Glimmer 30B — ELO 1667, #78
- Riemann-bench: 13.6 (#22/30)
- Chartography: 17.9 (#27/45)
- Roboflow Playground Object Detection: 1189.0 (#35/48)
- Gert Labs Rankings: 43.91 (#37/100)
- HANDBOOK.md Agents: 3.5 (#39/45)
- ComplexConstraints: 23.9 (#41/57)
- Roboflow Playground Overall: 1189.0 (#53/72)
Gemini 2.5 Pro — ELO 1666, #79
- LiveMathematicianBench: 32.1 (#8/19)
Gemini 2.5 Pro (Preview 03-25) — ELO 1661, #84
- Epoch AI - Enigma Eval: 4.14 (#28/46)
Qwen 3.6 27B — ELO 1657, #89
- RealWorldQA: 84.1 (#4/14)
- BenCzechMark: 70.57 (#16/74)
- OmniDocBench 1.5: 89.4 (#23/67)
- NL2Repo: 36.2 (#23/25)
- OSWorld-Verified: 63.9 (#23/29)
- CharXiv-R: 78.4 (#36/61)
Claude Opus 4.1 — ELO 1657, #90
- Artificial Analysis Intelligence Index: 28.84 (#183/603)
Claude Opus 4.1 (20250805) — ELO 1657, #91
- Epoch AI - Enigma Eval: 7.18 (#19/46)
Gemini 2.5 Pro (Preview 05-06) — ELO 1651, #95
- Epoch AI - Enigma Eval: 2.36 (#34/46)
Gemini 3.1 Flash Lite — ELO 1650, #97
- DystopiaBench: 72.9 (#35/42)
Claude Opus 4 — ELO 1649, #98
- Epoch AI - Enigma Eval: 5.57 (#25/46)
GPT-5 Mini — ELO 1648, #99
- Epoch AI - Enigma Eval: 8.19 (#17/46)
Gemini 3.1 Flash Lite (Preview) — ELO 1645, #102
- Epoch AI - Enigma Eval: 3.04 (#32/46)
Gemini 2.5 Flash (Preview 05-20) — ELO 1645, #103
- Epoch AI - Enigma Eval: 2.7 (#33/46)
O1 (2024-12-17) — ELO 1641, #107
- Epoch AI - Enigma Eval: 5.65 (#24/46)
Kimi K2 — ELO 1640, #108
- APEX v1 Consulting: 50.8 (#29/39)
- APEX v1 Big Law: 59.8 (#31/39)
- APEX v1 Investment Banking: 49.1 (#31/39)
- APEX v1 Medicine (MD): 50.0 (#32/39)
Claude Sonnet 4 — ELO 1640, #109
- Epoch AI - Enigma Eval: 3.12 (#31/46)
Nemotron 3 Ultra — ELO 1640, #111
- APEX v1 Investment Banking: 54.9 (#24/39)
- APEX v1 Big Law: 64.0 (#29/39)
- APEX v1 Medicine (MD): 54.8 (#29/39)
- APEX v1 Consulting: 50.8 (#30/39)
Qwen 3.5 27B — ELO 1638, #116
- HealthAdminBench: 13.3 (#9/11)
MiniMax-M2.7 — ELO 1638, #117
- APEX v1 Big Law: 52.5 (#36/39)
- APEX v1 Consulting: 41.9 (#38/39)
- APEX v1 Medicine (MD): 45.7 (#38/39)
- APEX v1 Investment Banking: 43.0 (#38/39)
DeepSeek V3.2 — ELO 1635, #121
- LiveMathematicianBench: 30.8 (#9/19)
- APEX v1 Investment Banking: 55.0 (#23/39)
- HMMT 2025: 90.2 (#31/82)
- APEX v1 Big Law: 55.7 (#32/39)
- APEX v1 Consulting: 49.8 (#32/39)
- APEX v1 Medicine (MD): 45.7 (#37/39)
- Vals AI LegalBench: 76.08 (#104/139)
Claude 3.7 Sonnet (20250219) — ELO 1634, #124
- Epoch AI - Enigma Eval: 2.26 (#35/46)
DeepSeek V3.2 Exp — ELO 1629, #131
- CritPt: 1.4 (#83/213)
DeepSeek V3.1 Terminus — ELO 1627, #134
- Tau3 Banking: 21.03 (#69/176)
O4 Mini — ELO 1625, #136
- Epoch AI - Enigma Eval: 9.21 (#16/46)
GPT-4.1 — ELO 1622, #142
- LiveMathematicianBench: 30.5 (#11/19)
- Epoch AI - Enigma Eval: 2.17 (#36/46)
Gemma 4 31B — ELO 1617, #152
- APEX v1 Investment Banking: 49.1 (#30/39)
- APEX v1 Consulting: 47.6 (#34/39)
- APEX v1 Big Law: 52.9 (#35/39)
- APEX v1 Medicine (MD): 42.8 (#39/39)
GPT-4.5 (Preview) — ELO 1609, #171
- Epoch AI - Enigma Eval: 3.18 (#30/46)
- Artificial Analysis Intelligence Index: 13.61 (#338/603)
GPT-5.4 Nano — ELO 1602, #175
- Roboflow Playground Captioning: 1200.0 (#24/49)
Claude 3.5 Sonnet — ELO 1601, #177
- Epoch AI - Enigma Eval: 0.91 (#39/46)
GPT-4o — ELO 1592, #197
- Epoch AI - Enigma Eval: 0.8 (#42/46)
GPT-OSS-120B — ELO 1590, #202
- APEX v1 Medicine (MD): 47.2 (#35/39)
- APEX v1 Investment Banking: 43.8 (#36/39)
- APEX v1 Big Law: 52.0 (#37/39)
- APEX v1 Consulting: 42.7 (#37/39)
Gemini 2.0 Pro (Preview 02-05) — ELO 1588, #206
- Epoch AI - Enigma Eval: 0.69 (#43/46)
GPT-4.1 Mini — ELO 1587, #207
- LiveMathematicianBench: 22.9 (#18/19)
Gemma 4 12B — ELO 1587, #209
- CritPt: 0.0 (#177/213)
Mistral Large 3 — ELO 1586, #210
- LiveMathematicianBench: 33.0 (#6/19)
GLM-4.6V — ELO 1586, #211
- CritPt: 0.0 (#171/213)
Llama 4 Maverick — ELO 1583, #216
- LiveMathematicianBench: 29.7 (#12/19)
- Epoch AI - Enigma Eval: 0.58 (#45/46)
Gemini 2.0 Flash — ELO 1583, #217
- Epoch AI - Enigma Eval: 1.1 (#38/46)
DeepSeek R1 — ELO 1580, #227
- LiveMathematicianBench: 30.6 (#10/19)
- IneqMath: 5.0 (#23/42)
Gemma 4 26B A4B — ELO 1580, #228
- CritPt: 0.0 (#176/213)
Gemini 2.0 Flash (001) — ELO 1579, #230
- Epoch AI - Enigma Eval: 0.63 (#44/46)
GPT-5 Nano — ELO 1577, #238
- CritPt: 0.0 (#165/213)
Gemini 2.5 Flash Lite — ELO 1569, #249
- CritPt: 0.0 (#188/213)
Mistral Small 4 — ELO 1566, #255
- CritPt: 0.3 (#128/213)
GLM-4.5V — ELO 1550, #291
- CritPt: 0.0 (#172/213)
Llama 3.3 70B Instruct — ELO 1543, #307
- LiveMathematicianBench: 27.0 (#14/19)
Llama 3.2 90B Vision Instruct — ELO 1542, #309
- Epoch AI - Enigma Eval: 0.38 (#46/46)
DeepSeek V2.5 — ELO 1536, #328
- Artificial Analysis Intelligence Index: 6.55 (#478/603)
- AA GPQA Diamond: 42.32 (#482/584)
GPT-OSS-20B — ELO 1533, #337
- Epoch AI - ECI: 136.65 (#345/491)
Claude 3 Opus — ELO 1525, #355
- Epoch AI - Enigma Eval: 0.82 (#41/46)
EXAONE 4.0 32B — ELO 1515, #369
- CritPt: 0.0 (#190/213)
LFM2 24B A2B — ELO 1451, #489
- BenCzechMark: 3.72 (#73/74)
Gemma 4 E2B — ELO 1433, #516
- CritPt: 0.0 (#175/213)
LLaMA-65B — ELO 1417, #546
- Artificial Analysis Intelligence Index: 1.71 (#584/603)
DeepSeek Coder V2 — ELO 1538
- Artificial Analysis Intelligence Index: 4.74 (#511/603)
DeepSeek V2 Chat — ELO 1539
- Artificial Analysis Intelligence Index: 3.27 (#543/603)
DeepSeek V4 Pro Preview — ELO 1683
- MineBench: 1394.44 (#28/61)
- HANDBOOK.md Agents: 9.2 (#29/45)
- ComplexConstraints: 28.0 (#40/57)
DeepSeek-V4-Flash-Vision-Exp — ELO 1752
- LLM Stats (ZEROBench): 35.0 (#2/10)
- LLM Stats (NL2Repo): 57.7 (#3/18)
- LLM Stats (DeepSWE): 59.3 (#6/12)
- LLM Stats (AutomationBench): 25.7 (#7/13)
- LLM Stats (Agents' Last Exam): 27.3 (#8/11)
- LLM Stats (Terminal-Bench 2.1): 83.9 (#10/29)
- LLM Stats Score: 48.67 (#15/342)
GPT-5.1 Instant — ELO 1600
- Epoch AI - Enigma Eval: 1.94 (#37/46)
GPT-5.2 ChatGPT — ELO 1692
- AI Chess Leaderboard (Continuation): 842.0 (#64/259)
- AI Chess Leaderboard (Reasoning): 835.0 (#96/318)
GPT-5.5 Instant — ELO 1666
- Epoch AI - ECI: 143.49 (#263/491)
Gemini 3 Deep Think — ELO 1761
- AA CritPt: 25.71 (#14/490)
LFM2.5-2.6B — ELO 1515
- AA Omniscience: -10.95 (#128/487)
- Tau3 Banking: 7.22 (#134/176)
- AA GDPval: 251.28 (#185/210)
- AA Humanity's Last Exam: 6.21 (#334/575)
- Artificial Analysis Intelligence Index: 10.99 (#380/603)
- AA GPQA Diamond: 55.76 (#403/584)
- AA Long Context Reasoning: 5.33 (#433/508)
- AA Omniscience - Science, Engineering & Mathematics: 12.2 (#437/486)
- AA CritPt: 0.0 (#466/490)
- AA Omniscience - Business: 3.8 (#479/487)
- ...and 6 more
Laguna M.1 — ELO 1568
- Vals AI MedScribe: 65.91 (#81/87)
Ministral 8B — ELO 1448
- LM Market Cap LMC Score: 37.4 (#395/428)
O1 Pro — ELO 1625
- Epoch AI - Enigma Eval: 6.14 (#22/46)
- Artificial Analysis Intelligence Index: 19.12 (#270/603)
Pixtral Large — ELO 1539
- Epoch AI - Enigma Eval: 0.84 (#40/46)
Qwen 3.5 — ELO 1629
- APEX v1 Consulting: 50.5 (#31/39)
- APEX v1 Big Law: 54.6 (#33/39)
- APEX v1 Medicine (MD): 48.2 (#33/39)
- APEX v1 Investment Banking: 47.8 (#34/39)
Qwen 3.6 Max — ELO 1715
- DystopiaBench: 71.1 (#31/42)
Qwen 3.7 Flash — ELO 1638
- Roboflow Playground Open Prompt: 1200.0 (#20/54)

Top-10 New Scores (126)

Claude Fable 5 on APEX v1: 66.0 (#4)
Claude Fable 5 on APEX v1 Big Law: 69.3 (#16)
Claude Fable 5 on APEX v1 Consulting: 66.6 (#5)
Claude Fable 5 on APEX v1 Investment Banking: 66.6 (#4)
Claude Fable 5 on APEX v1 Medicine (MD): 61.4 (#13)
Claude Fable 5 on ArxivMath: 78.6 (#2)
Claude Fable 5 on CyberGym: 83.8 (#3)
Claude Fable 5 on Epoch AI - Cursorbench: 65.2 (#11)
Claude Fable 5 on Epoch AI - Enigma Eval: 39.28 (#1)
Claude Fable 5 on FrontierCode: 46.3 (#3)
Claude Fable 5 on FutureEval: 13.23 (#1)
Claude Fable 5 on SpacetimeDB LLM Benchmark (C#): 98.9 (#1)
Claude Fable 5 on SpacetimeDB LLM Benchmark (Rust): 68.3 (#8)
Claude Fable 5 on SpacetimeDB LLM Benchmark (TypeScript): 95.3 (#10)
Claude Opus 5 on Agents' Last Exam: 27.0 (#7)
Claude Opus 5 on Epoch AI - Cursorbench: 64.3 (#16)
Claude Opus 5 on LiveMathematicianBench: 45.1 (#3)
Claude Opus 5 on SpacetimeDB LLM Benchmark (C#): 66.7 (#12)
Claude Opus 5 on SpacetimeDB LLM Benchmark (Rust): 68.3 (#9)
Claude Opus 5 on SpacetimeDB LLM Benchmark (TypeScript): 96.4 (#9)
Claude Opus 5 on Vals AI LegalBench: 86.97 (#5)
Claude Opus 5 on Vals AI MMMU: 89.88 (#1)
Claude Opus 5 on Vals AI MedCode: 63.57 (#1)
Claude Opus 5 on Vals AI MedScribe: 90.98 (#1)
Claude Opus 5 on Vals AI SAGE: 49.43 (#19)
GPT-5.4 Pro on AA CritPt: 30.0 (#4)
GPT-5.4 Pro on Epoch AI - Enigma Eval: 23.82 (#4)
GPT-5.4 Pro on OpenAI GPT-5.4 Launch - FrontierMath Tier 1-3: 50.0 (#1)
GPT-5.4 Pro on OpenAI GPT-5.5 Launch - FrontierMath Tier 1-3: 50.0 (#3)
GPT-5.5 Pro on AA CritPt: 30.57 (#2)
GPT-5.5 Pro on OpenAI GPT-5.5 Launch - FrontierMath Tier 1-3: 52.4 (#1)
GPT-5.5 Pro on OpenAI GPT-5.5 Launch - FrontierMath Tier 4: 39.6 (#1)
GPT-5.6 Sol on CyberGym: 83.6 (#4)
GPT-5.6 Sol on Design Arena (3D): 1438.0 (#2)
GPT-5.6 Sol on Design Arena (Data Viz): 1338.0 (#5)
GPT-5.6 Sol on Design Arena (Game Dev): 1414.0 (#2)
GPT-5.6 Sol on Design Arena (UI Components): 1387.0 (#1)
GPT-5.6 Sol on Epoch AI - Apex Agents: 37.7 (#12)
GPT-5.6 Sol on Epoch AI - Cursorbench: 64.5 (#14)
GPT-5.6 Sol on Epoch AI - Enigma Eval: 37.12 (#2)
GPT-5.6 Sol on ParseBench: 62.12 (#25)
GPT-5.6 Sol on Roboflow Playground Open Prompt: 1200.0 (#29)
GPT-5.6 Sol on SpacetimeDB LLM Benchmark (C#): 92.2 (#9)
GPT-5.6 Sol on SpacetimeDB LLM Benchmark (Rust): 82.9 (#7)
GPT-5.6 Sol on SpacetimeDB LLM Benchmark (TypeScript): 99.5 (#1)
GPT-5.6 Sol on Toolathlon: 74.9 (#6)
Gemini 3.7 Flash on AI for Education Pedagogy: 90.77 (#7)
Gemini 3.7 Flash on AI for Education Pedagogy - Maths: 92.06 (#14)
Gemini 3.7 Flash on AI for Education Pedagogy - Primary: 94.84 (#7)
Gemini 3.7 Flash on AI for Education Pedagogy - Science: 92.9 (#16)
Gemini 3.7 Flash on AI for Education Pedagogy - Secondary: 89.62 (#8)
Gemini 3.7 Flash on AI for Education Pedagogy - Social studies: 87.27 (#24)
Gemini 3.7 Flash on AI for Education Pedagogy - Technology: 86.79 (#22)
Gemini 3.7 Flash on AI for Education SEND: 87.16 (#8)
Gemini 3.7 Flash on AI for Education Visual Reasoning - match (process): 88.9 (#1)
Gemini 3.7 Flash on AI for Education Visual Reasoning - odd one out: 84.4 (#2)
Gemini 3.7 Flash on AI for Education Visual Reasoning - pattern completion (2d): 79.5 (#5)
Gemini 3.7 Flash on ARC-AGI-1: 95.5 (#15)
Gemini 3.7 Flash on ARC-AGI-2: 84.58 (#13)
Gemini 3.7 Flash on Agents on Rails: 71.4 (#10)
Gemini 3.7 Flash on Arena AI Code: 1587.0 (#10)
Gemini 3.7 Flash on BioMysteryBench Human-Difficult: 43.5 (#3)
Gemini 3.7 Flash on BioMysteryBench Human-Solvable: 87.1 (#2)
Gemini 3.7 Flash on CharXiv-R: 84.5 (#18)
Gemini 3.7 Flash on Chartography: 43.0 (#2)
Gemini 3.7 Flash on ComplexConstraints: 42.4 (#9)
Gemini 3.7 Flash on Deep20Bench: 14.03 (#5)
Gemini 3.7 Flash on Design Arena (3D): 1358.0 (#8)
Gemini 3.7 Flash on Design Arena (Data Viz): 1353.0 (#4)
Gemini 3.7 Flash on Design Arena (UI Components): 1319.0 (#14)
Gemini 3.7 Flash on Epoch AI - Cursorbench: 61.6 (#24)
Gemini 3.7 Flash on Epoch AI - ECI: 156.85 (#49)
Gemini 3.7 Flash on Epoch AI - Proofbench: 58.0 (#8)
Gemini 3.7 Flash on FrontierCode: 43.6 (#4)
Gemini 3.7 Flash on Gert Labs Rankings: 65.05 (#6)
Gemini 3.7 Flash on HANDBOOK.md Agents: 11.9 (#21)
Gemini 3.7 Flash on Icelandic LLM - ARC-Challenge-IS: 95.22 (#1)
Gemini 3.7 Flash on Icelandic LLM - Belebele-IS: 94.56 (#6)
Gemini 3.7 Flash on Icelandic LLM - GED: 82.0 (#4)
Gemini 3.7 Flash on Icelandic LLM - Inflection: 97.33 (#4)
Gemini 3.7 Flash on Icelandic LLM - WikiQA-IS: 66.82 (#3)
Gemini 3.7 Flash on InfoOps Bench: 48.5 (#18)
Gemini 3.7 Flash on Kaggle FACTS Parametric: 78.72 (#2)
Gemini 3.7 Flash on LMArena Text Arena: 1489.87 (#9)
Gemini 3.7 Flash on LMArena WebDev Arena: 1587.86 (#9)
Gemini 3.7 Flash on MMOU: 82.5 (#2)
Gemini 3.7 Flash on Riemann-bench: 39.2 (#7)
Gemini 3.7 Flash on Roboflow Playground OCR: 1186.0 (#45)
Gemini 3.7 Flash on Roboflow Playground Object Detection: 1218.0 (#19)
Gemini 3.7 Flash on Roboflow Playground Overall: 1202.0 (#35)
Gemini 3.7 Flash on Vals AI Code Migration: 34.8 (#17)
Gemini 3.7 Flash on Vals AI GPQA: 93.94 (#4)
Gemini 3.7 Flash on Vals AI LegalBench: 87.26 (#3)
Gemini 3.7 Flash on Vals AI LiveCodeBench: 88.65 (#3)
Gemini 3.7 Flash on Vals AI MMLU-Pro: 90.12 (#4)
Gemini 3.7 Flash on Vals AI MMMU: 88.96 (#3)
Gemini 3.7 Flash on Vals AI MedCode: 53.39 (#7)
Gemini 3.7 Flash on Vals AI MedScribe: 83.94 (#25)
Gemini 3.7 Flash on Vals AI MortgageTax: 66.65 (#32)
Gemini 3.7 Flash on Vals AI ProgramBench: 68.66 (#9)
Gemini 3.7 Flash on Vals AI SAGE: 49.23 (#21)
Gemini 3.7 Flash on Vals AI SWE-bench Verified: 80.8 (#22)
Gemini 3.7 Flash on Vals AI SkillsBench: 65.89 (#2)
Gemini 3.7 Flash on Vals AI TaxEval v2: 74.73 (#29)
Gemini 3.7 Flash on WebDev Arena: 1587.48 (#9)
Kimi K3 on AI for Education Visual Reasoning: 76.0 (#10)
Kimi K3 on AI for Education Visual Reasoning - match (figure): 63.9 (#13)
Kimi K3 on AI for Education Visual Reasoning - match (process): 75.0 (#7)
Kimi K3 on AI for Education Visual Reasoning - odd one out: 79.2 (#4)
Kimi K3 on AI for Education Visual Reasoning - pattern completion (2d): 71.2 (#9)
Kimi K3 on AI for Education Visual Reasoning - pattern completion (linear): 82.1 (#11)
Kimi K3 on AI for Education Visual Reasoning - reasoning by analogy: 83.7 (#5)
Kimi K3 on APEX v1: 68.4 (#2)
Kimi K3 on APEX v1 Big Law: 73.2 (#7)
Kimi K3 on APEX v1 Consulting: 69.7 (#3)
Kimi K3 on APEX v1 Investment Banking: 66.4 (#5)
Kimi K3 on APEX v1 Medicine (MD): 64.3 (#2)
Kimi K3 on Benchmarks.bio - BioSecBench-Refusal: 8.6 (#16)
Kimi K3 on Benchmarks.bio - BioSecBench-Surveillance: 48.3 (#12)
Kimi K3 on CyberGym: 80.0 (#8)
Kimi K3 on Epoch AI - Cursorbench: 60.8 (#28)
Kimi K3 on NL2Repo: 58.0 (#3)
Kimi K3 on SpacetimeDB LLM Benchmark (C#): 93.3 (#8)
Kimi K3 on SpacetimeDB LLM Benchmark (Rust): 65.1 (#12)
Kimi K3 on SpacetimeDB LLM Benchmark (TypeScript): 93.3 (#11)
Kimi K3 on Toolathlon: 76.5 (#2)

New #1 Leaders (38)

Multi-SWE-Bench (c): CodeArts Agent + CodeArts-GLM-5.1 (54.33) beat RepoRepair + Claude-4-Sonnet by 41.72
Lean AI Formalization Leaderboard: Humanifa + GPT 5.6 sol (183.0) beat Seed Prover (ByteDance) by 28.0
Multi-SWE-Bench (c++): CodeArts Agent + CodeArts-GLM-5.1 (47.29) beat InfCode + GPT5(Sep) by 21.71
Chatbot Arena (Video Edit): dreamina-seedance-2.5-720p (1411.0) beat minimax-h3 by 21.0
KernelBench Hub - Hard: Qwen 3.8 Max (69.26) beat Claude Fable 5 by 14.05
SpreadsheetBench v2: arito (45.46) beat Claude Opus 4.6 by 10.57
SWE-PolyBench Verified (TypeScript): HMigBot (53.0) beat Atlassian Rovo Dev (2025-12-08) by 10.0
APEX v1 Consulting: GPT-5.6 Terra (76.2) beat GPT-5.2 Codex by 9.3
MobileWorld - GUI Only: Qwen-UI-Agent (82.1) beat Kimi K3 by 7.7
LVBench: Gemini 3.7 Flash (85.4) beat Seed 2.1 Pro by 7.4
SWE-PolyBench Verified (Python): HMigBot (61.95) beat Atlassian Rovo Dev (2025-12-08) by 7.08
APEX v1 Investment Banking: GPT-5.6 Terra (71.8) beat GPT-5.3 Codex by 6.8
AI for Education Visual Reasoning - reasoning by analogy: Gemini 3.7 Flash (94.4) beat Gemini 3.5 Flash by 5.6
MRCR-v2 128k: Gemini 3.7 Flash (97.0) beat Gemini 3.6 Flash by 5.2
EnterpriseRAG Bench - Completeness: metor.com (86.22) beat Troml by 4.38
MMEB Leaderboard: Ovis-Omni-Embedding-v0.5-3B (56.95) beat Tianmu-Emb-Uni by 4.12
Kaggle FACTS (Google): Gemini 3.7 Flash (71.27) beat Gemini 3.1 Pro (Preview) by 4.02
AI for Education Visual Reasoning: Gemini 3.7 Flash (89.9) beat Gemini 3.5 Flash by 3.9
BioMysteryBench Human-Solvable: Claude Sonnet 5 (87.5) beat Claude Mythos 5 by 3.6
EnterpriseRAG Bench: metor.com (80.34) beat Troml by 3.55
AI for Education Visual Reasoning - pattern completion (linear): Gemini 3.7 Flash (95.7) beat GPT-5.6 Sol by 3.4
BioMysteryBench Human-Difficult: GPT-5.6 Terra (49.4) beat Claude Mythos 5 by 3.3
Kaggle FACTS Multimodal: Gemini 3.7 Flash (49.83) beat Gemini 3.6 Flash by 2.62
APEX v1: GPT-5.6 Terra (Max) (69.5) beat GPT-5.4 (High) by 2.3
AI for Education Visual Reasoning - match (figure): Gemini 3.7 Flash (87.4) beat Gemini 3.5 Flash by 2.2
Design Arena (UI Components): GPT-5.6 Sol (xHigh) (1387.0) beat Kimi K3 by 2.0
AI for Education Pedagogy - Science: Grok 4.6 (96.72) beat Qwen 3.5 Plus by 1.64
Humanity's Last Exam (Self-Reported, With Tools): Ornith-1.5-397B (56.1) beat GLM-5.2 by 1.4
Epoch AI - ECI: Claude Fable 5 (High) (162.49) beat GPT-5.6 Sol (Medium) by 0.84
CyberGym: GLM-5.3 (84.5) beat Claude Mythos 5 by 0.7
WebVoyager: browser-control (Fable 5) (99.19) beat Alumnium by 0.69
KernelBench Hub - Mega: Claude Fable 5 (24.61) beat Claude Opus 5 by 0.32
SpreadsheetBench: JT AlphaData (98.5) beat Qingqiu Agent by 0.25
Icelandic LLM - WinoGrande-IS: Gemini 3.7 Flash (96.32) beat Gemini 3.1 Pro (Preview) by 0.18
Icelandic LLM Leaderboard - Average: Gemini 3.7 Flash (88.71) beat Gemini 3.1 Pro (Preview) by 0.17
BRIDGE Medical Leaderboard - Few-Shot: Qwen 3.8 27B (Non-reasoning) (55.67) beat Gemini 1.5 Pro (002) by 0.16
AI for Education Pedagogy - Secondary: Grok 4.6 (91.19) beat GPT-5.5 by 0.15
ProphetArena: Gemini 3.7 Flash (0.9687) beat Gemini 3.6 Flash by 0.01

The Aggregate Digest — 2026-08-22

2026-08-22T06:30:27.535616+00:00

Daily

New Models (1)

DeepSeek-V4-Flash-Vision-Exp — ELO 1754
- LLM Stats (ZEROBench): 35.0 (#2/10)
- LLM Stats (NL2Repo): 57.7 (#3/18)
- LLM Stats (DeepSWE): 59.3 (#6/12)
- LLM Stats (AutomationBench): 25.7 (#7/13)
- LLM2014 Logic 2026-08: 58.1 (#8/44)
- LLM Stats (Agents' Last Exam): 27.3 (#8/11)
- LLM Stats (Terminal-Bench 2.1): 83.9 (#10/29)
- LLM Stats Score: 48.67 (#15/342)

Top-10 New Scores (16)

Claude Fable 5 on ArxivMath: 78.6 (#2)
Claude Fable 5 on CyberGym: 83.8 (#3)
Claude Fable 5 on FrontierCode: 46.3 (#3)
Claude Opus 5 on Benchmarks.bio - BioSecBench-Refusal: 29.4 (#12)
Claude Opus 5 on Vals AI MedCode: 63.57 (#1)
Claude Opus 5 on Vals AI MedScribe: 90.98 (#1)
GPT-5.6 Sol on CyberGym: 83.6 (#4)
GPT-5.6 Sol on Toolathlon: 74.9 (#5)
Gemini 3.7 Flash on BioMysteryBench Human-Difficult: 43.5 (#3)
Gemini 3.7 Flash on BioMysteryBench Human-Solvable: 87.1 (#2)
Gemini 3.7 Flash on CharXiv-R: 84.5 (#18)
Gemini 3.7 Flash on FrontierCode: 43.6 (#4)
Gemini 3.7 Flash on Roboflow Playground OCR: 1186.0 (#45)
Kimi K3 on CyberGym: 80.0 (#8)
Kimi K3 on NL2Repo: 58.0 (#3)
Kimi K3 on Toolathlon: 76.5 (#2)

New #1 Leaders (5)

LVBench: Gemini 3.7 Flash (85.4) beat Seed 2.1 Pro by 7.4
MRCR-v2 128k: Gemini 3.7 Flash (97.0) beat Gemini 3.6 Flash by 5.2
BioMysteryBench Human-Solvable: Claude Sonnet 5 (87.5) beat Claude Mythos 5 by 3.6
BioMysteryBench Human-Difficult: GPT-5.6 Terra (49.4) beat Claude Mythos 5 by 3.3
CyberGym: GLM-5.3 (84.5) beat Claude Mythos 5 by 0.7

The Aggregate Digest — 2026-08-21

2026-08-21T11:50:31.255079+00:00

Frontier

Best available model: Claude Opus 5 (1792) takes the crown from Gemini 3.7 Flash (1784)

Daily

Top-10 New Scores (32)

Claude Opus 5 on AA GDPval: 1844.67 (#1)
Claude Opus 5 on AA Omniscience - Business: 53.0 (#2)
Claude Opus 5 on AA Omniscience - Health: 52.42 (#2)
Claude Opus 5 on AA Omniscience - Humanities & Social Sciences: 58.6 (#2)
Claude Opus 5 on AA Omniscience - Law: 58.4 (#8)
Claude Opus 5 on AA Omniscience - Science, Engineering & Mathematics: 57.52 (#2)
Claude Opus 5 on AA Omniscience - Software Engineering (SWE): 86.0 (#3)
Claude Opus 5 on AA-Briefcase: 57.98 (#1)
Claude Opus 5 on AA-Omniscience Accuracy: 60.87 (#2)
GPT-5.6 Sol on AA GDPval: 1722.96 (#8)
GPT-5.6 Sol on AA Omniscience - Business: 49.6 (#5)
GPT-5.6 Sol on AA Omniscience - Health: 52.4 (#3)
GPT-5.6 Sol on AA Omniscience - Humanities & Social Sciences: 57.8 (#3)
GPT-5.6 Sol on AA Omniscience - Law: 55.9 (#12)
GPT-5.6 Sol on AA Omniscience - Science, Engineering & Mathematics: 55.7 (#5)
GPT-5.6 Sol on AA Omniscience - Software Engineering (SWE): 85.5 (#6)
GPT-5.6 Sol on AA-Briefcase: 41.85 (#8)
GPT-5.6 Sol on AA-Omniscience Accuracy: 59.4 (#4)
GPT-5.6 Sol on Roboflow Playground Open Prompt: 1200.0 (#29)
Gemini 3.7 Flash on AA GDPval: 1531.55 (#24)
Gemini 3.7 Flash on AA Omniscience - Business: 45.6 (#18)
Gemini 3.7 Flash on AA Omniscience - Health: 44.7 (#27)
Gemini 3.7 Flash on AA Omniscience - Humanities & Social Sciences: 55.2 (#11)
Gemini 3.7 Flash on AA Omniscience - Law: 62.1 (#3)
Gemini 3.7 Flash on AA Omniscience - Science, Engineering & Mathematics: 52.0 (#19)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE): 72.3 (#26)
Gemini 3.7 Flash on AA-Briefcase: 36.36 (#11)
Gemini 3.7 Flash on AA-Omniscience Accuracy: 55.32 (#16)
Gemini 3.7 Flash on ARC-AGI-1: 95.5 (#15)
Gemini 3.7 Flash on ARC-AGI-2: 84.58 (#13)
Gemini 3.7 Flash on Roboflow Playground Object Detection: 1218.0 (#19)
Gemini 3.7 Flash on Roboflow Playground Overall: 1218.0 (#16)

New #1 Leaders (10)

AA GDPval: Claude Opus 5 (Adaptive Reasoning, Max Effort) (1844.67) beat Qwen3.8 2.4T A95B by 124.28
AA Omniscience - Software Engineering (SWE): Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) (91.1) beat Kimi K3 (Max) by 18.9
AA Omniscience - Business: Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) (57.4) beat Gemini 3 Pro (Preview) (High) by 10.5
AA-Omniscience Accuracy: Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) (65.35) beat Gemini 3 Pro (Preview) (High) by 9.6
AA Omniscience - Science, Engineering & Mathematics: Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) (61.28) beat Gemini 3 Pro (Preview) (High) by 8.98
AA Omniscience - Humanities & Social Sciences: Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) (64.3) beat Gemini 3 Pro (Preview) (High) by 8.5
AA-Briefcase: Claude Opus 5 (Adaptive Reasoning, Xhigh Effort) (57.98) beat Kimi K3 (Max) by 7.01
AA Omniscience - Health: Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) (53.72) beat Gemini 3 Pro (Preview) (High) by 6.42
MMEB Leaderboard: Ovis-Omni-Embedding-v0.5-3B (57.25) beat Tianmu-Emb-Uni by 4.42
AA Omniscience - Law: Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) (66.2) beat Gemini 3 Pro (Preview) (High) by 1.1

The Aggregate Digest — 2026-08-20

2026-08-20T05:51:23.006145+00:00

Frontier

Best available model: GLM-5.3 enters at #11 (1753 ELO) on 43 benchmarks

Daily

New Benchmarks (3)

SecIT Bench (Pydantic AI) (Accuracy (%)): leader Grok 4.6 (80.65), 14 models
Cribl's benchmark for IT and security operations agents, run under a plain Pydantic AI harness. Thirty telemetry scenarios across intrusion and breach, covert egress, service errors and performance degradation are each rolled out three times; an incomplete answer scores zero, and the reported figure is the mean scenario accuracy.
SecIT Bench (Claude Code) (Accuracy (%)): leader Claude Opus 5 (78.64), 3 models
The SecIT Bench scenario set run inside the Claude Code agent product rather than a plain model harness. Scored identically to the Pydantic AI board, and kept separate because the scaffold is part of what is being measured: the same model moves by up to three points between harnesses.
SecIT Bench (Codex) (Accuracy (%)): leader GPT-5.6 Terra (71.88), 3 models
The SecIT Bench scenario set run inside the Codex agent product rather than a plain model harness. Scored identically to the Pydantic AI board, and kept separate because the scaffold is part of what is being measured: GPT-5.6 Terra scores 71.9 here against 69.4 under the plain harness.

Top-10 New Scores (21)

Claude Fable 5 on APEX v1: 66.0 (#4)
Claude Fable 5 on APEX v1 Big Law: 69.3 (#16)
Claude Fable 5 on APEX v1 Consulting: 66.6 (#5)
Claude Fable 5 on APEX v1 Investment Banking: 66.6 (#4)
Claude Fable 5 on APEX v1 Medicine (MD): 61.4 (#13)
Claude Fable 5 on FutureEval: 13.23 (#1)
Claude Opus 5 on LiveMathematicianBench: 45.1 (#3)
Claude Opus 5 on Vals AI LegalBench: 86.97 (#5)
Claude Opus 5 on Vals AI MMMU: 89.88 (#1)
Claude Opus 5 on Vals AI SAGE: 49.43 (#19)
Gemini 3.7 Flash on Icelandic LLM - ARC-Challenge-IS: 95.22 (#1)
Gemini 3.7 Flash on Icelandic LLM - Belebele-IS: 94.56 (#6)
Gemini 3.7 Flash on Icelandic LLM - GED: 82.0 (#4)
Gemini 3.7 Flash on Icelandic LLM - Inflection: 97.33 (#4)
Gemini 3.7 Flash on Icelandic LLM - WikiQA-IS: 66.82 (#3)
Gemini 3.7 Flash on InfoOps Bench: 48.5 (#18)
Kimi K3 on APEX v1: 68.4 (#2)
Kimi K3 on APEX v1 Big Law: 73.2 (#7)
Kimi K3 on APEX v1 Consulting: 69.7 (#3)
Kimi K3 on APEX v1 Investment Banking: 66.4 (#5)
Kimi K3 on APEX v1 Medicine (MD): 64.3 (#2)

New #1 Leaders (9)

Multi-SWE-Bench (c): CodeArts Agent + CodeArts-GLM-5.1 (54.33) beat RepoRepair + Claude-4-Sonnet by 41.72
Multi-SWE-Bench (c++): CodeArts Agent + CodeArts-GLM-5.1 (47.29) beat InfCode + GPT5(Sep) by 21.71
APEX v1 Consulting: GPT-5.6 Terra (76.2) beat GPT-5.2 Codex by 9.3
APEX v1 Investment Banking: GPT-5.6 Terra (71.8) beat GPT-5.3 Codex by 6.8
APEX v1: GPT-5.6 Terra (Max) (69.5) beat GPT-5.4 (High) by 2.3
Humanity's Last Exam (Self-Reported, With Tools): Ornith-1.5-397B (56.1) beat GLM-5.2 by 1.4
Icelandic LLM - WinoGrande-IS: Gemini 3.7 Flash (96.32) beat Gemini 3.1 Pro (Preview) by 0.18
Icelandic LLM Leaderboard - Average: Gemini 3.7 Flash (88.71) beat Gemini 3.1 Pro (Preview) by 0.17
BRIDGE Medical Leaderboard - Few-Shot: Qwen3.8-27B-Non-Thinking (55.67) beat Gemini 1.5 Pro (002) by 0.16

The Aggregate Digest — 2026-08-19

2026-08-19T08:54:18.919667+00:00

Daily

New Benchmarks (4)

BenchBench-Protocol (Normalized Rubric Score (%)): leader Claude Opus 5 (59.2), 9 models
Benchling's wet-lab protocol-modification benchmark: 149 expert-reviewed tasks derived from the differences between a published protocol and the version a scientist actually ran, across 96 source protocols in nine domains of wet-lab biology. Responses are graded against weighted rubrics recovered from the real modification, so the score measures whether a model reasons about prior choices and downstream steps rather than reciting protocol text.
Guesswork 2026-08 (MAE (z-score units)): leader Llama 3.1 8B Instruct (1.9018), 31 models
Guesswork standings for 2026-08: each contestant predicts newly scraped (model, benchmark) scores from day-before information only, scored as mean absolute error in z-score units — lower is better. The Aggregate's own row is excluded so the site's predictor cannot feed itself a score.
Guesswork 2026-07 (MAE (z-score units)): leader Gemma 3 4B (IT) (6.881), 37 models
Guesswork standings for 2026-07: each contestant predicts newly scraped (model, benchmark) scores from day-before information only, scored as mean absolute error in z-score units — lower is better. The Aggregate's own row is excluded so the site's predictor cannot feed itself a score.
Guesswork 2026-06 (MAE (z-score units)): leader Llama 3.1 8B Instruct (2.7281), 19 models
Guesswork standings for 2026-06: each contestant predicts newly scraped (model, benchmark) scores from day-before information only, scored as mean absolute error in z-score units — lower is better. June was the immature-matrix month, when several LLMs beat the aggregate.

Top-10 New Scores (33)

Claude Fable 5 on Epoch AI - Cursorbench: 65.2 (#11)
Claude Fable 5 on SpacetimeDB LLM Benchmark (C#): 98.9 (#1)
Claude Fable 5 on SpacetimeDB LLM Benchmark (Rust): 68.3 (#8)
Claude Fable 5 on SpacetimeDB LLM Benchmark (TypeScript): 95.3 (#10)
Claude Opus 5 on Epoch AI - Cursorbench: 64.3 (#16)
Claude Opus 5 on SpacetimeDB LLM Benchmark (C#): 66.7 (#12)
Claude Opus 5 on SpacetimeDB LLM Benchmark (Rust): 68.3 (#9)
Claude Opus 5 on SpacetimeDB LLM Benchmark (TypeScript): 96.4 (#9)
GLM-5.3 on AA CritPt: 19.14 (#27)
GLM-5.3 on AA GPQA Diamond: 91.72 (#30)
GLM-5.3 on AA Humanity's Last Exam: 42.26 (#28)
GLM-5.3 on AA Long Context Reasoning: 76.33 (#40)
GLM-5.3 on AA Omniscience: 14.3 (#36)
GLM-5.3 on AA SciCode: 56.48 (#10)
GLM-5.3 on Artificial Analysis Intelligence Index: 59.51 (#8)
GLM-5.3 on LM Market Cap LMC Score: 82.7 (#75)
GLM-5.3 on Tau3 Banking: 50.31 (#3)
GPT-5.5 Pro on AA CritPt: 30.57 (#2)
GPT-5.6 Sol on Epoch AI - Apex Agents: 37.7 (#12)
GPT-5.6 Sol on Epoch AI - Cursorbench: 64.5 (#14)
GPT-5.6 Sol on SpacetimeDB LLM Benchmark (C#): 92.2 (#9)
GPT-5.6 Sol on SpacetimeDB LLM Benchmark (Rust): 82.9 (#7)
GPT-5.6 Sol on SpacetimeDB LLM Benchmark (TypeScript): 99.5 (#1)
Gemini 3 Deep Think on AA CritPt: 25.71 (#14)
Gemini 3.7 Flash on Chartography: 43.0 (#2)
Gemini 3.7 Flash on ComplexConstraints: 42.4 (#9)
Gemini 3.7 Flash on Epoch AI - Cursorbench: 61.6 (#24)
Gemini 3.7 Flash on Epoch AI - ECI: 156.36 (#55)
Gemini 3.7 Flash on Gert Labs Rankings: 65.17 (#6)
Gemini 3.7 Flash on HANDBOOK.md Agents: 11.9 (#21)
Gemini 3.7 Flash on MMOU: 82.5 (#2)
Gemini 3.7 Flash on Riemann-bench: 39.2 (#7)
Gemini 3.7 Flash on WebDev Arena: 1587.48 (#9)

New #1 Leaders (6)

SWE-PolyBench Verified (TypeScript): HMigBot (53.0) beat Atlassian Rovo Dev (2025-12-08) by 10.0
SWE-PolyBench Verified (Python): HMigBot (61.95) beat Atlassian Rovo Dev (2025-12-08) by 7.08
EnterpriseRAG Bench - Completeness: metor.com (86.22) beat Troml by 4.38
Kaggle FACTS (Google): Gemini 3.7 Flash (71.27) beat Gemini 3.1 Pro (Preview) by 4.02
EnterpriseRAG Bench: metor.com (80.34) beat Troml by 3.55
Agent Arena - Confirmed Success: Inkling Small (18.91) beat Claude Opus 5 (Max) by 0.95

The Aggregate Digest — 2026-08-18

2026-08-18T05:14:30.331602+00:00

Daily

New Benchmarks (6)

Z.ai GLM-5.3 Launch - Terminal Bench 3.0 (Accuracy (%)): leader GPT-5.6 Sol (34.6), 6 models
Terminal-Bench 3.0 terminal-agent tasks as reported in Z.ai's GLM-5.3 launch table, using Claude Code with maximum reasoning effort, isolated task containers, official verifiers, and avg@3 over three rollouts; Z.ai-run frontier comparison.
Z.ai GLM-5.3 Launch - SWE-Marathon v1.1 (Pass@1 (%)): leader Claude Opus 4.8 (48.8), 6 models
SWE-Marathon v1.1 ultra-long-horizon software-engineering tasks as reported in Z.ai's GLM-5.3 launch table; Z.ai-run frontier comparison using Claude Code at maximum effort.
Z.ai GLM-5.3 Launch - ExploitGym (2h) (Successful Intended Exploits (#)): leader GPT-5.6 Sol (216.0), 7 models
ExploitGym real-world exploit-generation tasks completed under a two-hour throughput-normalized budget, as reported in Z.ai's GLM-5.3 launch table; Z.ai-run frontier comparison.
Z.ai GLM-5.3 Launch - ExploitGym (6h) (Successful Intended Exploits (#)): leader GPT-5.6 Sol (293.0), 7 models
ExploitGym real-world exploit-generation tasks completed under a six-hour throughput-normalized budget, as reported in Z.ai's GLM-5.3 launch table; Z.ai-run frontier comparison.
Z.ai GLM-5.3 Launch - HLE with Tools (Accuracy (%)): leader GPT-5.6 Sol (64.5), 8 models
Humanity's Last Exam with tool use as reported in Z.ai's GLM-5.3 launch table; Z.ai-run comparison across eight recent frontier models.
Z.ai GLM-5.3 Launch - GDPval-AA v2 (ELO): leader GLM-5.3 (1769.0), 8 models
Artificial Analysis GDPval-AA v2 evaluation of economically valuable professional knowledge work, reported as Elo in Z.ai's GLM-5.3 launch table; Artificial Analysis runs compared by Z.ai.

Top-10 New Scores (8)

Claude Fable 5 on Epoch AI - Enigma Eval: 39.28 (#1)
GLM-5.3 on Agents on Rails: 79.4 (#6)
GPT-5.4 Pro on Epoch AI - Enigma Eval: 23.82 (#4)
GPT-5.6 Sol on Epoch AI - Enigma Eval: 37.12 (#2)
Gemini 3.7 Flash on Agents on Rails: 71.4 (#10)
Gemini 3.7 Flash on Deep20Bench: 14.03 (#5)
Gemini 3.7 Flash on Design Arena (Data Viz): 1358.0 (#4)
Gemini 3.7 Flash on Kaggle FACTS Parametric: 78.72 (#2)

New #1 Leaders (11)

Chatbot Arena (Video Edit): dreamina-seedance-2.5-720p (1411.0) beat minimax-h3 by 21.0
SpreadsheetBench v2: arito (45.46) beat Claude Opus 4.6 by 10.57
AI for Education Visual Reasoning - reasoning by analogy: Gemini-3.7 Flash (94.4) beat Gemini 3.5 Flash by 5.6
AI for Education Visual Reasoning: Gemini-3.7 Flash (89.9) beat Gemini 3.5 Flash by 3.9
AI for Education Visual Reasoning - pattern completion (linear): Gemini-3.7 Flash (95.7) beat GPT-5.6 Sol by 3.4
Kaggle FACTS Multimodal: Gemini 3.7 Flash (49.83) beat Gemini 3.6 Flash by 2.62
AI for Education Visual Reasoning - match (figure): Gemini-3.7 Flash (87.4) beat Gemini 3.5 Flash by 2.2
AI for Education Pedagogy - Science: Grok 4.6 (96.72) beat Qwen 3.5 Plus by 1.64
SpreadsheetBench: JT AlphaData (98.5) beat Qingqiu Agent by 0.25
AI for Education Pedagogy - Secondary: Grok 4.6 (91.19) beat GPT-5.5 by 0.15
ProphetArena: Gemini 3.7 Flash (0.9799) beat Gemini 3.6 Flash by 0.02

The Aggregate Digest — 2026-08-17

2026-08-17T18:32:32.040913+00:00

Daily

New Benchmarks (1)

LLM2014 Logic 2026-08 (Median Score): leader GPT-5.5 (xHigh) (73.89), 40 models
August 2026 monthly snapshot of the LLM2014 project public leaderboard, scoring models on logic and reasoning problems from its continuously updated test set.

Top-10 New Scores (23)

GPT-5.5 Pro on OpenAI GPT-5.5 Launch - FrontierMath Tier 1-3: 52.4 (#1)
GPT-5.5 Pro on OpenAI GPT-5.5 Launch - FrontierMath Tier 4: 39.6 (#1)
GPT-5.6 Sol on Design Arena (3D): 1426.0 (#2)
GPT-5.6 Sol on Design Arena (Data Viz): 1331.0 (#6)
GPT-5.6 Sol on Design Arena (Game Dev): 1440.0 (#1)
GPT-5.6 Sol on Design Arena (UI Components): 1371.0 (#2)
GPT-5.6 Sol on ParseBench: 62.12 (#25)
Gemini 3.7 Flash on Design Arena (3D): 1372.0 (#5)
Gemini 3.7 Flash on Design Arena (UI Components): 1314.0 (#13)
Gemini 3.7 Flash on Vals AI Code Migration: 34.8 (#17)
Gemini 3.7 Flash on Vals AI GPQA: 93.94 (#4)
Gemini 3.7 Flash on Vals AI LegalBench: 87.26 (#3)
Gemini 3.7 Flash on Vals AI LiveCodeBench: 88.65 (#3)
Gemini 3.7 Flash on Vals AI MMLU-Pro: 90.12 (#4)
Gemini 3.7 Flash on Vals AI MMMU: 88.96 (#3)
Gemini 3.7 Flash on Vals AI MedCode: 53.39 (#7)
Gemini 3.7 Flash on Vals AI MedScribe: 83.94 (#25)
Gemini 3.7 Flash on Vals AI MortgageTax: 66.65 (#32)
Gemini 3.7 Flash on Vals AI ProgramBench: 68.66 (#8)
Gemini 3.7 Flash on Vals AI SAGE: 49.23 (#20)
Gemini 3.7 Flash on Vals AI SWE-bench Verified: 80.8 (#19)
Gemini 3.7 Flash on Vals AI SkillsBench: 65.89 (#2)
Gemini 3.7 Flash on Vals AI TaxEval v2: 74.73 (#29)

New #1 Leaders (1)

Design Arena (Game Dev): GPT-5.6 Sol (xHigh) (1440.0) beat Kimi K3 by 8.0

The Aggregate Digest — 2026-08-16

2026-08-16T05:35:40.031129+00:00

Daily

Top-10 New Scores (17)

Claude Opus 5 on Agent Security League - Functional Correctness: 73.7 (#9)
GLM-5.3 on DeepsecBench: 22.0 (#6)
GPT-5.6 Sol on BabyVision: 88.9 (#3)
GPT-5.6 Sol on CADGenBench: 0.5319 (#2)
GPT-5.6 Sol on ProphetArena: 0.9506 (#4)
Gemini 3.7 Flash on Arabic Broad Leaderboard: 9.204 (#4)
Gemini 3.7 Flash on Chess Puzzles (Epoch AI): 47.0 (#11)
Gemini 3.7 Flash on Design Arena (Game Dev): 1310.0 (#16)
Gemini 3.7 Flash on Design Arena (Website): 1348.0 (#2)
Gemini 3.7 Flash on Epoch AI - Mystery Game Puzzles: 37.0 (#7)
Gemini 3.7 Flash on FrontierMath - Tier 4 (v2): 36.59 (#16)
Gemini 3.7 Flash on FrontierMath - Tiers 1-3 (v2): 71.58 (#14)
Gemini 3.7 Flash on Multi-turn Debate (Lechmazur): 1476.3 (#27)
Gemini 3.7 Flash on OTIS Mock AIME 2024-25: 97.22 (#19)
Gemini 3.7 Flash on ProgramBench: 0.0 (#9)
Gemini 3.7 Flash on ProgramBench Almost: 5.5 (#6)
Gemini 3.7 Flash on SimpleQA Verified: 71.2 (#4)

New #1 Leaders (4)

Coarena - Task Completion: Qwen 3.8 Max (100.0) beat GPT-5.6 Luna by 15.7
BabyVision: Qwen3.8-Max (with tools) (91.3) beat GPT-5.5 by 7.7
Agent Security League - Security Correctness: Claude Opus 5 (32.4) beat Claude Fable 5 by 3.4
CADGenBench: build123d-mcp-v0381-claude-opus-5-xhigh-full-r6 (0.6391) beat GPT-5.6 Sol (xHigh) by 0.11

Weekly

New Models (284)

Claude Opus 5 — ELO 1782, #1
- Computer Anthology Terminal Tasks (Terminus-2): 62.0 (#1/22)
- ProgramBench: 4.5 (#1/21)
- ProgramBench Almost: 37.0 (#1/21)
- Android Bench: 91.8 (#1/32)
- Design Arena (SVG): 1361.0 (#1/107)
- CHI-Bench: 54.7 (#1/26)
- NatureBench: 23.33 (#1/13)
- Creative Writing (Lechmazur): 4.2 (#1/44)
- GameCraft-Bench: 68.44 (#1/14)
- Vals AI (Vals Index): 67.21 (#1/46)
- ...and 25 more
Gemini 3.7 Flash — ELO 1782, #2
- AA MMMU-Pro: 85.49 (#1/240)
- AutomationBench: 30.4 (#1/10)
- LLM Stats (LVBench): 85.4 (#1/25)
- LLM Stats (MRCR v2 (8-needle)): 97.0 (#1/23)
- Coarena: 1039.4 (#2/16)
- Design Arena (Website): 1348.0 (#2/169)
- AA GPQA Diamond: 94.55 (#2/575)
- RuneBench: 10203.0 (#2/46)
- AI Chess Leaderboard (Reasoning): 1854.0 (#2/313)
- LLM Stats (Artificial Analysis): 56.0 (#2/7)
- ...and 82 more
GPT-5.5 Pro — ELO 1779, #3
- PM-LLM-Benchmark: 37.7 (#9/171)
Claude Fable 5 — ELO 1777, #4
- Computer Anthology Terminal Tasks (Claude Code): 54.0 (#1/4)
- FrontierCode: 63.6 (#1/22)
- APEX-Agents-AA: 59.2 (#1/28)
- Vals Index: 75.14 (#1/40)
- Vals Multimodal Index: 74.15 (#1/29)
- ArxivMath: 87.5 (#2/17)
- Computer Anthology Terminal Tasks (Terminus-2): 50.8 (#3/22)
- MedCode: 56.07 (#3/75)
- MedScribe: 88.52 (#3/74)
- ProofBench: 77.0 (#3/54)
- ...and 4 more
GPT-5.6 Sol — ELO 1772, #5
- Computer Anthology Terminal Tasks (Codex CLI): 58.0 (#1/4)
- LLM Chess (Saplin): 1549.7 (#1/158)
- GDP.pdf: 30.7 (#1/24)
- AIIQ Composite IQ: 136.0 (#1/115)
- Computer Anthology Terminal Tasks (Terminus-2): 51.6 (#2/22)
- CADGenBench: 0.5319 (#2/23)
- ProgramBench: 1.0 (#2/21)
- OpenRouter HLE (Search): 71.1 (#2/2)
- OpenRouter BrowseComp (Search): 82.4 (#2/5)
- OpenRouter DeepSearchQA (Search): 75.0 (#2/4)
- ...and 24 more
Kimi K3 — ELO 1756, #7
- Design Arena (Game Dev): 1432.0 (#1/155)
- DeepSearchQA: 95.0 (#1/15)
- Arena AI Code: 1674.0 (#2/48)
- LMArena WebDev Arena: 1674.48 (#2/25)
- Vals Index: 74.7 (#3/40)
- Vals Multimodal Index: 73.42 (#3/29)
- Android Bench: 90.2 (#4/32)
- Design Arena (SVG): 1339.0 (#4/107)
- BabyVision: 85.7 (#4/29)
- MCP Atlas: 84.2 (#4/39)
- ...and 22 more
Qwen 3.8 Max — ELO 1751, #8
- Coarena - Task Completion: 100.0 (#1/16)
- SpatialGen-Bench (Text Answering): 81.74 (#1/27)
- LLM Stats (WideSearch): 81.9 (#1/10)
- LLM Stats (Agents' Last Exam): 52.4 (#2/10)
- LLM Stats (MRCR v2 (8-needle)): 92.9 (#2/23)
- Vals AI IOI: 73.0 (#3/62)
- Arena AI Code: 1671.0 (#3/48)
- LMArena WebDev Arena: 1671.13 (#3/25)
- Vals AI GPQA: 93.69 (#4/132)
- AI for Education SEND: 88.07 (#4/225)
- ...and 43 more
Muse Spark 1.2 — ELO 1751, #9
- LMArena Text Arena: 1498.64 (#4/24)
- AI for Education Pedagogy - Maths: 94.44 (#5/233)
- AI for Education SEND: 88.07 (#5/225)
- FoodTruckBench: 41904.0 (#7/10)
- SimpleBench: 74.5 (#8/93)
- Vals AI IOI: 49.5 (#9/62)
- Coarena: 1001.7 (#10/16)
- RuneBench: 5677.0 (#10/46)
- Vals AI (Vals Index): 57.05 (#11/46)
- LLM Stats (DeepSWE 1.1): 59.3 (#11/25)
- ...and 22 more
Gemini 3.6 Flash — ELO 1748, #10
- ProgramBench: 0.5 (#5/21)
- Finance Agent v2: 56.3 (#5/40)
- Chatbot Arena (Vision): 1295.0 (#7/145)
- MedCode: 53.15 (#8/75)
- ProgramBench Almost: 4.0 (#9/21)
- Android Bench: 75.6 (#10/32)
- Vals Multimodal Index: 62.8 (#14/29)
- LMArena Text Arena: 1483.64 (#15/24)
- Vals Index: 62.43 (#15/40)
- Chatbot Arena (Text): 1484.0 (#16/391)
- ...and 11 more
GPT-5.5 — ELO 1747, #11
- HardMTBench: 91.4 (#1/22)
- PhysicianBench: 46.3 (#1/12)
- SWE Atlas: 45.4 (#1/15)
- ALE-Bench: 1942.97 (#1/89)
- LiveBench: 81.28 (#1/48)
- ClawProBench: 67.9 (#1/57)
- Computer Anthology Terminal Tasks (Codex CLI): 51.6 (#2/4)
- MathArena Apex: 80.21 (#2/50)
- ATLAS: 67.77 (#3/23)
- GeneBench: 25.0 (#3/16)
- ...and 22 more
Muse Spark 1.1 — ELO 1746, #12
- MCP Atlas: 88.1 (#1/39)
- MedScribe: 88.89 (#2/74)
- Finance Agent v2: 57.21 (#3/40)
- CharXiv-R: 88.4 (#5/58)
- OSWorld-Verified: 80.8 (#6/27)
- Vals Index: 68.41 (#8/40)
- DeepSearchQA: 84.9 (#10/15)
- LMArena Text Arena: 1488.67 (#10/24)
- Vals Multimodal Index: 66.74 (#10/29)
- Arena AI Document: 1472.0 (#12/38)
- ...and 3 more
Grok 4.6 — ELO 1746, #13
- CursorBench 3.1: 70.8 (#1/45)
- AA GPQA Diamond: 94.95 (#1/575)
- RuneBench: 12757.0 (#1/46)
- LLM Stats (APEX-Agents): 57.5 (#1/8)
- Benchmarks.bio - BioSecBench-Surveillance: 56.3 (#2/17)
- Benchmarks.bio - VariantBench: 44.63 (#2/14)
- Tau3 Banking: 50.72 (#2/166)
- SealedBench: 70.0 (#2/30)
- FrontierCode: 61.3 (#2/22)
- APEX-Agents-AA: 57.5 (#2/28)
- ...and 92 more
Gemini 3.1 Pro (Preview) — ELO 1738, #14
- AudioMC: 66.8 (#1/33)
- MMAU: 82.5 (#1/31)
- Computer Anthology Terminal Tasks (Gemini CLI): 19.6 (#2/2)
- MedCode: 59.06 (#2/75)
- MathArena Apex: 60.94 (#4/50)
- MirrorCode: 8.9 (#6/6)
- Finance Agent v1.1: 59.72 (#9/53)
- PhysicianBench: 6.0 (#11/12)
- OSWorld-Verified: 76.2 (#12/27)
- CaseLaw v2: 64.84 (#12/54)
- ...and 15 more
Qwen 3.7 Max — ELO 1735, #15
- IOI: 46.75 (#13/64)
- Finance Agent v2: 47.78 (#18/40)
- Vals Index: 57.49 (#18/40)
- Computer Anthology Terminal Tasks (Terminus-2): 9.8 (#19/22)
- ProofBench: 26.0 (#22/54)
- AIIQ Composite IQ: 118.0 (#26/115)
- Android Bench: 54.2 (#27/32)
- Vals AI Code Migration: 13.07 (#33/48)
- MedScribe: 79.4 (#35/74)
- MedCode: 38.75 (#44/75)
Gemini 3.5 Flash — ELO 1734, #16
- Computer Anthology Terminal Tasks (Gemini CLI): 24.4 (#1/2)
- Finance Agent v2: 57.86 (#2/40)
- MedCode: 55.83 (#5/75)
- ProgramBench Almost: 3.0 (#10/21)
- OSWorld-Verified: 78.4 (#10/27)
- ProgramBench: 0.0 (#11/21)
- Harvey Legal Agent Benchmark: 0.8 (#11/12)
- MathArena Apex: 32.29 (#11/50)
- Computer Anthology Terminal Tasks (Terminus-2): 22.8 (#12/22)
- Vals Multimodal Index: 62.93 (#13/29)
- ...and 8 more
Grok 4.5 — ELO 1733, #17
- FrontierCode: 56.6 (#4/22)
- APEX-Agents-AA: 47.1 (#4/28)
- MedScribe: 86.88 (#7/74)
- Vals Index: 65.3 (#11/40)
- Vals Multimodal Index: 63.42 (#12/29)
- Vals AI ProofBench: 31.0 (#16/24)
- AIIQ Composite IQ: 124.0 (#16/115)
- Finance Agent v2: 48.35 (#16/40)
- DeepsecBench: 15.58 (#17/35)
- ProofBench: 30.0 (#19/54)
- ...and 2 more
Claude Opus 4.8 — ELO 1731, #18
- Computer Anthology Terminal Tasks (Claude Code): 38.0 (#3/4)
- Agent Arena - Praise vs Complaint: 22.52 (#3/48)
- DeepSearchQA: 93.1 (#5/15)
- LiveBench: 77.55 (#5/48)
- Vals Index: 70.36 (#5/40)
- Vals Multimodal Index: 70.89 (#5/29)
- MCP Atlas: 83.6 (#7/39)
- ArxivMath: 65.0 (#7/17)
- MedCode: 53.22 (#7/75)
- ProofBench: 69.0 (#7/54)
- ...and 16 more
GPT-5.6 Terra — ELO 1728, #20
- ProofBench: 71.0 (#4/54)
- AutomationBench: 21.0 (#6/10)
- AIIQ Composite IQ: 132.0 (#6/115)
- Android Bench: 86.8 (#7/32)
- LLM Chess (Saplin): 1293.0 (#7/158)
- VoxelBench: 1936.0 (#8/44)
- Arena AI Document: 1479.0 (#10/38)
- IOI: 65.25 (#10/64)
- Finance Agent v2: 52.42 (#11/40)
- Vals Multimodal Index: 65.07 (#11/29)
- ...and 5 more
DeepSeek V4 Pro (0813) — ELO 1724, #21
- LLM Stats (NL2Repo): 61.5 (#1/17)
- LLM Stats (AutomationBench): 31.8 (#2/12)
- LLM Stats (CyberGym): 83.3 (#2/13)
- LLM Stats (Toolathlon): 74.1 (#2/37)
- Vals AI SWE-bench Verified: 96.4 (#2/82)
- NL2Repo: 61.5 (#2/21)
- CyberGym: 83.3 (#2/16)
- LLM Stats (Terminal-Bench 2.1): 87.9 (#4/28)
- LLM Stats (DeepSWE): 62.7 (#5/11)
- Vals AI Vibe Code Bench: 82.3 (#5/84)
- ...and 63 more
Gemini 3 Pro (Preview) — ELO 1718, #23
- OpenEval - GPQA CoT: 80.27 (#1/74)
- OpenEval - MMLU-Pro CoT: 90.3 (#1/77)
- MATH 500: 96.4 (#1/60)
- Vals AI MMLU-Pro: 90.1 (#4/132)
- OpenEval - Omni-MATH: 55.55 (#5/74)
- HMMT 2025: 97.5 (#5/82)
- MGSM: 93.93 (#7/74)
- OpenEval - IFEval Strict: 87.65 (#8/77)
- MedCode: 52.2 (#10/75)
- MathArena Apex: 23.44 (#15/50)
- ...and 10 more
Gemini 3 Pro — ELO 1718, #24
- FINAL Bench Metacognitive: 77.08 (#2/9)
- MageBench Season 1: 1722.0 (#3/34)
- MolViBench: 36.0 (#4/15)
- IDE-Bench: 55.0 (#8/15)
- LMArena Text Arena: 1485.45 (#14/24)
GPT-5.4 — ELO 1714, #25
- Sandboxed Coding Agents are Competitive Omni-m: 75.0 (#1/16)
- PlanningBench: 63.17 (#1/18)
- Vision2Code: 4.12 (#1/13)
- APEX-Agents: 52.7 (#2/45)
- LiveBench: 80.91 (#2/48)
- LLM Arena RU: 1187.0 (#3/104)
- Computer Anthology Terminal Tasks (Codex CLI): 38.0 (#3/4)
- FAM-Bench: 82.4 (#4/25)
- ATLAS: 63.23 (#4/23)
- PrivacySIM: 38.77 (#4/7)
- ...and 25 more
Claude Opus 4.7 — ELO 1714, #26
- Finance Agent v1.1: 64.37 (#1/53)
- MirrorCode: 31.1 (#2/6)
- Chatbot Arena (Vision): 1301.0 (#3/145)
- Chatbot Arena (Text): 1502.0 (#3/391)
- KnotBench: 51.65 (#3/4)
- PhysicianBench: 29.3 (#3/12)
- CaseLaw v2: 68.38 (#5/54)
- Chatbot Arena (Document): 1497.0 (#6/39)
- Arena AI Document: 1497.0 (#6/38)
- LiveBench: 77.1 (#6/48)
- ...and 22 more
GLM-5.2 — ELO 1709, #27
- Arena AI Code: 1588.0 (#7/48)
- LMArena WebDev Arena: 1588.2 (#7/25)
- GameCraft-Bench: 39.12 (#9/14)
- MCP Atlas: 82.6 (#9/39)
- Computer Anthology Terminal Tasks (Terminus-2): 31.4 (#10/22)
- ProphetArena: 0.9418 (#10/17)
- ArxivMath: 56.67 (#10/17)
- Vals Index: 65.02 (#13/40)
- Vals AI Excel Modeling: 61.53 (#15/46)
- Finance Agent v2: 49.7 (#15/40)
- ...and 6 more
Claude Sonnet 5 — ELO 1709, #28
- Computer Anthology Terminal Tasks (Claude Code): 25.6 (#4/4)
- Vals Index: 68.61 (#7/40)
- Vals Multimodal Index: 68.83 (#7/29)
- ProofBench: 66.0 (#8/54)
- Finance Agent v2: 53.91 (#9/40)
- Blueprint-Bench 2: 0.249 (#13/23)
- Vals AI MedCode: 47.54 (#22/84)
- LLM Chess (Saplin): 897.0 (#23/158)
- MedScribe: 76.05 (#49/74)
GPT-5.6 Luna — ELO 1707, #29
- OpenRouter DeepSearchQA (Search): 73.0 (#3/4)
- OpenRouter BrowseComp (Search): 74.0 (#4/5)
- Android Bench: 87.6 (#5/32)
- Finance Agent v2: 55.04 (#6/40)
- Vals Index: 69.88 (#6/40)
- Vals Multimodal Index: 69.06 (#6/29)
- IOI: 72.92 (#7/64)
- AIIQ Composite IQ: 129.0 (#9/115)
- VoxelBench: 1871.0 (#10/44)
- FoodTruckBench: 32932.0 (#10/10)
- ...and 8 more
Seed 2.1 Turbo — ELO 1707, #30
- SvelteBench: 88.9 (#70/169)
- LM Market Cap LMC Score: 40.0 (#241/419)
Claude Opus 4.6 — ELO 1706, #31
- VitaBench 2.0: 50.3 (#1/21)
- MageBench Season 1: 1747.0 (#1/34)
- Chatbot Arena (Text): 1505.0 (#2/391)
- PhysicianBench: 31.7 (#2/12)
- Chatbot Arena (Document): 1506.0 (#3/39)
- Arena AI Document: 1506.0 (#3/38)
- SWE-bench Verified: 75.6 (#4/47)
- Chatbot Arena (Vision): 1300.0 (#4/145)
- LMArena Text Arena: 1497.59 (#5/24)
- FINAL Bench Metacognitive: 76.17 (#5/9)
- ...and 16 more
Qwen 3.6 Max Preview — ELO 1705, #32
- Finance Agent v1.1: 52.78 (#24/53)
- ClawProBench: 57.4 (#26/57)
- CaseLaw v2: 47.91 (#51/54)
GPT-5.2 Codex — ELO 1705, #33
- APEX v1 Consulting: 66.9 (#1/18)
- APEX v1: 65.3 (#4/5)
- APEX v1 Investment Banking: 61.8 (#5/18)
- Vals AI LiveCodeBench: 87.99 (#6/137)
- SWE-bench Verified: 72.8 (#7/47)
- APEX v1 Big Law: 73.0 (#8/18)
- ALE-Bench: 1299.9 (#8/89)
- APEX v1 Medicine (MD): 59.7 (#9/18)
Gemini 3 Flash (Preview) — ELO 1704, #34
- MedCode: 55.92 (#4/75)
- MGSM: 93.31 (#10/74)
- IOI: 39.08 (#15/64)
- Vals Multimodal Index: 52.19 (#22/29)
- Finance Agent v2: 42.55 (#26/40)
- Vals Index: 49.55 (#28/40)
- Vals AI Legal Research Bench: 18.27 (#30/49)
- Epoch AI - Proofbench: 15.0 (#32/52)
- CaseLaw v2: 55.84 (#33/54)
- Finance Agent v1.1: 47.6 (#33/53)
- ...and 5 more
GPT-5.3 Codex — ELO 1703, #35
- APEX v1 Investment Banking: 65.0 (#1/18)
- SmellBench: 44.4 (#2/10)
- ALE-Bench: 1655.22 (#2/89)
- APEX v1 Consulting: 65.5 (#4/18)
- MageBench Season 1: 1717.0 (#4/34)
- APEX v1 Big Law: 69.8 (#12/18)
- APEX v1 Medicine (MD): 58.1 (#14/18)
- IOI: 43.83 (#14/64)
- AIIQ Composite IQ: 119.0 (#23/115)
- LiveBench: 71.97 (#31/48)
Kimi K2.7 Code — ELO 1702, #37
- Computer Anthology Terminal Tasks (Terminus-2): 15.4 (#15/22)
Gemini 3 Flash — ELO 1701, #38
- SWE-bench Verified: 75.8 (#2/47)
- OCR-Robust: 79.05 (#3/18)
- HMMT 2025: 97.5 (#7/82)
- MageBench Season 1: 1622.0 (#11/34)
- MathArena Apex: 15.62 (#18/50)
- InfoOps Bench: 20.1 (#33/37)
- AIIQ Composite IQ: 116.0 (#33/115)
- OmniDocBench 1.5: 0.12 (#65/65)
Qwen 3.7 Plus — ELO 1698, #40
- Vals Multimodal Index: 53.89 (#20/29)
- Vals Index: 52.33 (#24/40)
- Android Bench: 57.7 (#25/32)
- AIIQ Composite IQ: 116.0 (#29/115)
- Finance Agent v2: 38.22 (#29/40)
- Vals AI Legal Research Bench: 16.35 (#31/49)
- Vals AI Code Migration: 12.86 (#35/48)
DeepSeek V4 Flash (0731) — ELO 1694, #41
- OpenRouter BrowseComp (Search): 77.0 (#3/5)
- OpenRouter DeepSearchQA (Search): 72.0 (#4/4)
- GameCraft-Bench: 40.61 (#7/14)
- Vals AI ProofBench: 56.0 (#11/24)
- Vals AI Code Migration: 38.63 (#12/48)
- Epoch AI - Mystery Game Puzzles: 34.0 (#14/53)
- RuneBench: 2816.0 (#20/46)
- SvelteBench: 95.1 (#33/169)
- EQ-Bench Longform Writing: 61.2 (#51/126)
GPT-5 — ELO 1691, #42
- APEX v1 Big Law: 76.6 (#1/18)
- LingOly-TOO: 46.7 (#1/16)
- MCP-Universe: 43.72 (#2/28)
- KnotBench: 43.0 (#4/4)
- APEX v1 Medicine (MD): 62.4 (#5/18)
- VitaBench 2.0: 44.1 (#6/21)
- VTB: 16.96 (#6/20)
- APEX v1 Consulting: 60.0 (#8/18)
- APEX v1 Investment Banking: 59.1 (#9/18)
- MageBench Season 1: 1536.0 (#29/34)
- ...and 3 more
Claude Opus 4.5 — ELO 1691, #43
- SWE-bench Verified: 76.8 (#1/47)
- IDE-Bench: 83.75 (#3/15)
Nex N2 Pro — ELO 1691, #44
- AIIQ Composite IQ: 109.0 (#46/115)
GPT-5.2 — ELO 1690, #46
- MolDeTox: 15.59 (#1/18)
- IDE-Bench: 85.0 (#2/15)
- MageBench Season 1: 1737.0 (#2/34)
- FINAL Bench Metacognitive: 76.5 (#3/9)
- APEX v1 Investment Banking: 62.3 (#4/18)
- APEX-Agents: 48.4 (#4/45)
- APEX v1 Big Law: 73.4 (#6/18)
- APEX v1 Consulting: 61.4 (#6/18)
- APEX v1 Medicine (MD): 61.9 (#6/18)
- MGSM: 94.0 (#6/74)
- ...and 16 more
Kimi K2.6 — ELO 1690, #47
- VitaBench 2.0: 48.1 (#2/21)
- PhysicianBench: 17.0 (#7/12)
- Finance Agent v1.1: 57.06 (#14/53)
- ClawProBench: 59.31 (#17/57)
- Vals Multimodal Index: 56.43 (#17/29)
- Vals Index: 55.17 (#20/40)
- CaseLaw v2: 61.2 (#22/54)
- Finance Agent v2: 44.9 (#22/40)
- Inverse Turing Bench: 57.45 (#26/40)
- Mercor APEX: 18.9 (#29/53)
- ...and 3 more
Hy3 — ELO 1689, #48
- Computer Anthology Terminal Tasks (Terminus-2): 18.0 (#13/22)
- AIIQ Composite IQ: 103.0 (#65/115)
Claude Opus 4.5 (20251101) — ELO 1688, #49
- MGSM: 95.2 (#1/74)
- SWE-bench Verified: 74.4 (#5/47)
- Epoch AI - Proofbench: 36.0 (#15/52)
- MultiNRC: 41.23 (#17/43)
- TutorBench: 49.82 (#18/27)
- CaseLaw v2: 62.59 (#18/54)
- IOI: 23.58 (#21/64)
- EnigmaEval: 4.65 (#22/43)
- MedCode: 45.17 (#24/75)
- MedScribe: 83.25 (#24/74)
- ...and 3 more
GLM-5.1 — ELO 1686, #50
- VitaBench 2.0: 45.0 (#5/21)
- ArxivMath: 52.5 (#12/17)
- Finance Agent v1.1: 57.66 (#12/53)
- MathArena Apex: 11.5 (#21/50)
- Vals Index: 52.45 (#22/40)
- Finance Agent v2: 44.79 (#23/40)
- ProofBench: 22.22 (#24/54)
- MedCode: 41.6 (#32/75)
- CaseLaw v2: 51.55 (#48/54)
- MedScribe: 72.27 (#59/74)
Inkling — ELO 1686, #51
- AudioMC: 56.6 (#2/33)
- MMAU: 77.2 (#2/31)
- MedScribe: 85.41 (#13/74)
- Coarena: 998.6 (#14/16)
- Coarena - Task Completion: 40.0 (#14/16)
- Finance Agent v2: 46.6 (#20/40)
- MCP Atlas: 74.1 (#21/39)
- Vals Index: 49.28 (#29/40)
- DeepsecBench: 6.32 (#30/35)
- CharXiv-R: 78.1 (#33/58)
- ...and 8 more
DeepSeek V4 Pro — ELO 1684, #52
- MathArena Apex: 90.2 (#1/50)
- VitaBench 2.0: 47.2 (#4/21)
- Finance Agent v1.1: 60.39 (#5/53)
- DeepSWE: 62.8 (#12/27)
- NYT Connections Extended: 88.4 (#17/104)
- Creative Writing (Lechmazur): 0.8 (#17/44)
- IOI: 35.83 (#17/64)
- Computer Anthology Terminal Tasks (Terminus-2): 10.4 (#18/22)
- Vals Index: 55.62 (#19/40)
- Finance Agent v2: 44.08 (#24/40)
- ...and 6 more
Seed 2.0 Pro — ELO 1684, #53
- VitaBench 2.0: 47.4 (#3/21)
- Creative Writing (Lechmazur): -1.5 (#33/44)
Qwen 3.5 Plus — ELO 1683, #54
- ClawProBench: 64.19 (#3/57)
- Finance Agent v1.1: 54.48 (#19/53)
- CaseLaw v2: 59.7 (#26/54)
GPT-5 Codex — ELO 1682, #55
- APEX v1 Big Law: 73.5 (#5/18)
- APEX v1 Consulting: 60.3 (#7/18)
- APEX v1 Investment Banking: 59.8 (#7/18)
- APEX v1 Medicine (MD): 57.6 (#16/18)
- Lean AI Formalization Leaderboard: 1.0 (#30/56)
- IOI: 9.75 (#38/64)
GPT-5 (2025-08-07) — ELO 1681, #56
- MATH 500: 96.0 (#3/60)
- CaseLaw v2: 66.45 (#6/54)
- MedCode: 49.63 (#14/75)
- MGSM: 92.84 (#14/74)
- SWE-bench Verified: 65.0 (#21/47)
- MedScribe: 83.65 (#22/74)
- IOI: 20.0 (#26/64)
- Finance Agent v1.1: 52.15 (#27/53)
- ProofBench: 18.0 (#29/54)
Qwen 3.5 397B A17B — ELO 1680, #57
- HarmActionsEval: 23.4 (#1/10)
- ClawProBench: 64.18 (#4/57)
Qwen 3.6 Plus — ELO 1679, #58
- PhysicianBench: 13.7 (#9/12)
- ClawProBench: 60.2 (#14/57)
- Finance Agent v1.1: 54.63 (#17/53)
- Vals Multimodal Index: 51.52 (#23/29)
- Finance Agent v2: 40.85 (#28/40)
- Vals Index: 48.89 (#30/40)
- Vals AI Harvey Legal Agent Bench: 1.25 (#31/50)
- Vals AI Excel Modeling: 32.87 (#35/46)
- Vals AI Legal Research Bench: 14.9 (#37/49)
- MedScribe: 76.96 (#44/74)
- ...and 3 more
MiMo-V2.5-Pro — ELO 1679, #59
- ClawProBench: 63.3 (#5/57)
- PhysicianBench: 16.7 (#8/12)
- MedScribe: 83.73 (#21/74)
- Android Bench: 60.8 (#23/32)
- Vals AI Harvey Legal Agent Bench: 2.08 (#23/50)
- ProofBench: 24.0 (#23/54)
- Finance Agent v2: 41.5 (#27/40)
- Vals Index: 50.74 (#27/40)
- Vals AI Legal Research Bench: 15.87 (#32/49)
- MedCode: 32.48 (#62/75)
MiniMax-M3 — ELO 1678, #60
- Android Bench: 63.6 (#20/32)
- Chess Puzzles (Epoch AI): 14.0 (#102/161)
- OTIS Mock AIME 2024-25: 71.11 (#116/238)
Claude Sonnet 4.6 — ELO 1677, #61
- Finance Agent v1.1: 63.33 (#2/53)
- HarmActionsEval: 2.84 (#4/10)
- PhysicianBench: 23.0 (#5/12)
- ProphetArena: 0.9437 (#8/17)
- GDP.pdf: 18.0 (#10/24)
- ClawProBench: 60.5 (#11/57)
- CaseLaw v2: 63.99 (#14/54)
- Finance Agent v2: 51.03 (#14/40)
- ProofBench: 45.0 (#14/54)
- Vals Multimodal Index: 60.57 (#15/29)
- ...and 2 more
O3 — ELO 1675, #63
- VitaBench 2.0: 40.3 (#8/21)
- MageBench Season 1: 1609.0 (#13/34)
- HMMT 2025: 77.5 (#45/82)
GPT-5.1 Codex — ELO 1675, #64
- APEX v1 Big Law: 73.2 (#7/18)
- APEX v1 Investment Banking: 58.7 (#10/18)
- APEX v1 Consulting: 57.6 (#11/18)
- APEX v1 Medicine (MD): 58.1 (#13/18)
- SWE-bench Verified: 66.0 (#19/47)
- Vals AI LiveCodeBench: 85.55 (#28/137)
- Design Arena (UI Components): 1265.0 (#38/148)
- IOI: 3.67 (#54/64)
GPT-5.3 — ELO 1673, #65
- HarmActionsEval: 12.77 (#2/10)
Inkling Small — ELO 1673, #66
- LLM Stats (AIME 2026): 95.5 (#4/21)
- LLM Stats (Artificial Analysis): 40.0 (#7/7)
- LLM Stats (MCP Atlas): 79.6 (#7/33)
- LLM Stats (Toolathlon): 54.4 (#12/37)
- OpenRouter GPQA Diamond: 87.2 (#21/114)
- LLM Stats (Terminal-Bench 2.1): 64.7 (#22/28)
- LLM Stats (BrowseComp): 77.4 (#23/62)
- ZeroEval GPQA Diamond: 89.5 (#26/239)
- LLM Stats (CharXiv-R): 77.4 (#30/51)
- FrontierMath - Tier 4 (v2): 17.07 (#32/50)
- ...and 17 more
Claude Sonnet 4.5 — ELO 1670, #67
- IDE-Bench: 87.5 (#1/15)
- OpenEval - MMLU-Pro CoT: 86.9 (#3/77)
- MGSM: 94.33 (#4/74)
- OpenEval - GPQA CoT: 68.61 (#6/74)
- OpenEval - Omni-MATH: 55.3 (#6/74)
- VitaBench 2.0: 41.7 (#7/21)
- SWE-bench Verified: 71.4 (#11/47)
- OR-Space: 53.0 (#11/19)
- VTB: 5.6 (#11/20)
- DentalGPT Dental Eval Suite: 51.7 (#12/17)
- ...and 18 more
Grok 4.3 — ELO 1670, #68
- CaseLaw v2: 79.31 (#1/54)
- Finance Agent v1.1: 53.81 (#20/53)
- Vals Multimodal Index: 43.29 (#25/29)
- Chess Bench LLM: 956.0 (#29/110)
- Finance Agent v2: 37.73 (#31/40)
- IOI: 15.33 (#31/64)
- Vals Index: 46.48 (#32/40)
- Vals AI LiveCodeBench: 84.49 (#36/137)
- ProofBench: 11.0 (#38/54)
- Vals AI MMLU-Pro: 85.84 (#48/132)
- ...and 2 more
Gemini 3.5 Flash Lite — ELO 1669, #69
- Agent Arena - Tool Hallucination: 0.28 (#3/48)
- Agent Arena - Confirmed Success: 12.73 (#5/48)
- Agent Arena - Steerability: 10.28 (#7/48)
- Agent Arena - Bash Recovery: 15.08 (#7/48)
- Agent Arena: 10.38 (#10/48)
- Agent Arena - Praise vs Complaint: 13.51 (#15/48)
- Finance Agent v2: 47.44 (#19/40)
- IOI: 26.17 (#20/64)
- Vals Index: 51.0 (#26/40)
- MedCode: 43.49 (#27/75)
- ...and 8 more
Gemini 2.5 Pro — ELO 1668, #70
- LingOly-TOO: 42.3539 (#4/16)
- USAMO25: 24.4 (#5/13)
- MedCode: 50.59 (#12/75)
- VitaBench 2.0: 33.1 (#15/21)
- CaseLaw v2: 63.88 (#15/54)
- MageBench Season 1: 1540.0 (#28/34)
- IOI: 17.08 (#29/64)
- APEX-Agents: 17.0 (#37/45)
- Finance Agent v1.1: 41.59 (#42/53)
- HMMT 2025: 82.5 (#43/82)
- ...and 2 more
GPT-5.1 — ELO 1668, #71
- CaseLaw v2: 73.42 (#2/54)
- MedScribe: 88.09 (#4/74)
- MedCode: 52.73 (#9/75)
- Mirror, Mirror on the Wall: 52.0 (#10/44)
- MGSM: 92.98 (#12/74)
- Inverse Turing Bench: 67.68 (#15/40)
- Finance Agent v1.1: 55.31 (#15/53)
- HMMT 2025: 93.33 (#22/82)
- IOI: 21.5 (#23/64)
- MathArena Apex: 1.04 (#39/50)
- ...and 1 more
GPT-5.4 Mini — ELO 1668, #72
- Computer Anthology Terminal Tasks (Codex CLI): 21.8 (#4/4)
- Vision2Code: 3.85 (#4/13)
- HarmActionsEval: 0.71 (#7/10)
- APEX-Agents: 37.5 (#17/45)
- Vals Multimodal Index: 54.22 (#19/29)
- Computer Anthology Terminal Tasks (Terminus-2): 4.0 (#21/22)
- Finance Agent v2: 45.36 (#21/40)
- Finance Agent v1.1: 53.41 (#22/53)
- Vals Index: 52.42 (#23/40)
- ProofBench: 21.0 (#25/54)
- ...and 8 more
Grok 4.20 0309 — ELO 1668, #73
- IOI: 30.17 (#18/64)
- Finance Agent v1.1: 52.3 (#26/53)
- Vals Multimodal Index: 39.06 (#28/29)
- ProofBench: 14.0 (#35/54)
- Vals Index: 39.5 (#36/40)
- Finance Agent v2: 28.49 (#37/40)
- CaseLaw v2: 54.45 (#38/54)
- Vals AI Legal Research Bench: 13.94 (#39/49)
- CritPt: 6.0 (#41/213)
- Vals AI Excel Modeling: 11.74 (#44/46)
- ...and 4 more
O3 (2025-04-16) — ELO 1667, #75
- MATH 500: 94.6 (#8/60)
- MedCode: 47.29 (#21/75)
- MGSM: 91.75 (#26/74)
- TextClass Benchmark: 1625.36 (#30/108)
- MedScribe: 76.65 (#45/74)
DeepSeek V4 Flash — ELO 1666, #76
- IMO-AnswerBench: 91.1 (#2/23)
- Arena AI Code: 1585.0 (#8/48)
- LMArena WebDev Arena: 1584.86 (#8/25)
- AutomationBench: 18.1 (#10/10)
- DeepsecBench: 16.54 (#12/35)
- AI for Education SEND: 83.03 (#25/225)
- ALE-Bench: 324.98 (#79/89)
Muse Glimmer 30B — ELO 1665, #77
- CPTU Bench: 4.42 (#1/102)
- AA-LCR: 80.0 (#1/217)
- LLM Stats (ScreenSpot Pro): 75.4 (#6/25)
- LLM Stats (SkillsBench): 44.3 (#7/8)
- ScreenSpot-Pro: 75.4 (#7/35)
- LLM Stats (AIME 2026): 94.7 (#8/21)
- WildClawBench: 47.6 (#8/21)
- Coarena: 1002.6 (#9/16)
- LLM Stats (DeepSearchQA): 74.6 (#9/9)
- Coarena - Task Completion: 50.0 (#12/16)
- ...and 30 more
Kimi K2.5 — ELO 1663, #79
- FINAL Bench Metacognitive: 78.54 (#1/9)
- MageBench Season 1: 1652.0 (#8/34)
- SWE-bench Verified: 70.8 (#12/47)
- ClawProBench: 58.49 (#21/57)
- MathArena Apex: 8.85 (#25/50)
- CaseLaw v2: 58.73 (#28/54)
- IOI: 17.67 (#28/64)
- Finance Agent v1.1: 50.62 (#30/53)
- Vals AI Legal Research Bench: 15.87 (#33/49)
- Vals AI Excel Modeling: 28.47 (#38/46)
- ...and 7 more
Grok 4.20 — ELO 1663, #80
- PhysicianBench: 5.3 (#12/12)
- ClawProBench: 43.04 (#50/57)
GLM-5 — ELO 1662, #81
- FINAL Bench Metacognitive: 76.38 (#4/9)
- HMMT 2025: 97.5 (#6/82)
- SWE-bench Verified: 72.8 (#8/47)
- APEX v1 Investment Banking: 52.9 (#14/18)
- APEX v1 Consulting: 45.3 (#17/18)
- APEX v1 Medicine (MD): 47.8 (#17/18)
- APEX v1 Big Law: 50.0 (#18/18)
- MathArena Apex: 10.94 (#22/50)
- IOI: 22.0 (#22/64)
- Finance Agent v1.1: 53.18 (#23/53)
- ...and 2 more
Gemini 2.5 Pro (Preview 03-25) — ELO 1660, #82
- OpenEval - GPQA CoT: 74.89 (#2/74)
- OpenEval - MMLU-Pro CoT: 86.3 (#4/77)
- MATH 500: 95.2 (#5/60)
- OpenEval - Omni-MATH: 41.6 (#15/74)
- OpenEval - IFEval Strict: 83.98 (#20/77)
- TextClass Benchmark: 1517.98 (#61/108)
MiMo-V2-Pro — ELO 1659, #83
- ClawProBench: 57.92 (#22/57)
Qwen 3.6 27B — ELO 1658, #84
- CPTU Bench: 4.41 (#2/102)
- ClawProBench: 58.74 (#20/57)
- Android Bench: 45.1 (#29/32)
- CaseLaw v2: 53.16 (#41/54)
- AIIQ Composite IQ: 111.0 (#42/115)
- ProofBench: 8.0 (#44/54)
Gemini 3.1 Flash Lite — ELO 1657, #85
- HarmActionsEval: 0.71 (#6/10)
- MageBench Season 1: 1578.0 (#23/34)
- AIIQ Composite IQ: 101.0 (#75/115)
GPT-5 Mini (2025-08-07) — ELO 1657, #86
- SWE-bench Verified: 59.8 (#26/47)
Claude Opus 4.1 — ELO 1657, #87
- LingOly-TOO: 45.8 (#2/16)
- Inverse Turing Bench: 60.14 (#22/40)
DeepSeek V3.2 Speciale — ELO 1657, #88
- HMMT 2025: 99.2 (#3/82)
Claude Opus 4.1 (20250805) — ELO 1657, #89
- MGSM: 94.44 (#3/74)
- MATH 500: 95.4 (#4/60)
- VTB: 4.71 (#14/20)
- EnigmaEval: 4.81 (#21/43)
- MultiNRC: 29.67 (#22/43)
- MedCode: 47.23 (#22/75)
- Visual-Language Understanding: 45.25 (#29/58)
- IOI: 12.52 (#35/64)
- MedScribe: 73.9 (#53/74)
Step 3.7 Flash — ELO 1656, #90
- ArxivMath: 46.67 (#14/17)
- AIIQ Composite IQ: 101.0 (#71/115)
Solar Pro 4 — ELO 1650, #93
- Agent Arena - Steerability: 13.32 (#3/48)
- LLM Stats (AIME 2026): 95.3 (#5/21)
- Agent Arena - Tool Hallucination: 0.51 (#7/48)
- Agent Arena - Praise vs Complaint: 15.67 (#9/48)
- Agent Arena: 10.1 (#11/48)
- Agent Arena - Bash Recovery: 13.56 (#12/48)
- Agent Arena - Confirmed Success: 8.44 (#17/48)
- LLM Stats (Terminal-Bench 2.1): 57.0 (#24/28)
- ZeroEval GPQA Diamond: 89.0 (#29/239)
- AA GDPval: 1275.76 (#48/198)
- ...and 32 more
GPT-5 Mini — ELO 1649, #94
- Agentick: 30.9 (#3/11)
- CaseLaw v2: 68.49 (#4/54)
- MATH 500: 94.8 (#6/60)
- MGSM: 92.58 (#15/74)
- Finance Agent v1.1: 51.93 (#28/53)
- SWE-bench Verified: 56.2 (#29/47)
- MageBench Season 1: 1516.0 (#30/34)
- MedCode: 43.05 (#30/75)
- MedScribe: 80.58 (#31/74)
- IOI: 6.75 (#42/64)
- ...and 2 more
Qwen 3.5 122B A10B — ELO 1649, #95
- CPTU Bench: 4.36 (#3/102)
Claude Opus 4 — ELO 1648, #98
- MGSM: 93.78 (#8/74)
- MATH 500: 90.4 (#25/60)
- HMMT 2025: 60.0 (#60/82)
- AA-LCR: 33.7 (#190/217)
Gemma 4 31B (IT) — ELO 1647, #99
- CPTU Bench: 4.32 (#5/102)
- Finance Agent v1.1: 50.79 (#29/53)
- Android Bench: 37.1 (#31/32)
- ClawProBench: 51.59 (#39/57)
- CaseLaw v2: 52.63 (#44/54)
- ForecastBench: 61.4 (#166/268)
MiMo-V2.5 — ELO 1646, #101
- ClawProBench: 60.39 (#13/57)
- Vals Multimodal Index: 52.77 (#21/29)
- Vals AI Excel Modeling: 55.09 (#24/46)
- Vals AI Harvey Legal Agent Bench: 1.67 (#25/50)
- Vals Index: 51.57 (#25/40)
- ProofBench: 16.0 (#31/54)
- Finance Agent v2: 36.73 (#33/40)
- Vals AI Legal Research Bench: 9.13 (#43/49)
- AIIQ Composite IQ: 109.0 (#48/115)
- MedScribe: 72.15 (#60/74)
- ...and 1 more
GLM-4.7 — ELO 1645, #102
- MageBench Season 1: 1675.0 (#6/34)
- APEX v1 Consulting: 55.7 (#12/18)
- APEX v1 Investment Banking: 53.9 (#12/18)
- APEX v1 Big Law: 51.6 (#17/18)
- APEX v1 Medicine (MD): 46.7 (#18/18)
- CaseLaw v2: 54.88 (#37/54)
- Finance Agent v1.1: 45.98 (#38/53)
- IOI: 7.58 (#41/64)
- ProofBench: 6.0 (#45/54)
- MGSM: 88.18 (#53/74)
- ...and 2 more
Qwen 3 Max — ELO 1645, #103
- VitaBench 2.0: 28.4 (#17/21)
- MageBench Season 1: 1594.0 (#18/34)
- MGSM: 91.82 (#25/74)
- IOI: 15.67 (#30/64)
- Finance Agent v1.1: 44.3 (#41/53)
- CaseLaw v2: 47.48 (#52/54)
Grok 4 — ELO 1644, #104
- MATH 500: 96.2 (#2/60)
- CaseLaw v2: 65.81 (#9/54)
- HMMT 2025: 95.0 (#13/82)
- IOI: 26.17 (#19/64)
- Finance Agent v1.1: 53.51 (#21/53)
- MGSM: 90.91 (#32/74)
- MathArena Apex: 2.08 (#33/50)
- MageBench Season 1: 1459.0 (#34/34)
- MedScribe: 78.15 (#38/74)
- MedCode: 38.08 (#47/75)
Gemini 3.1 Flash Lite (Preview) — ELO 1644, #105
- MedCode: 47.6 (#20/75)
- Vals Multimodal Index: 41.35 (#27/29)
- CaseLaw v2: 54.98 (#35/54)
- Finance Agent v1.1: 46.12 (#35/53)
- Finance Agent v2: 29.99 (#36/40)
- Vals AI Harvey Legal Agent Bench: 0.0 (#37/50)
- Vals Index: 36.2 (#38/40)
- Vals AI MMLU-Pro: 86.24 (#41/132)
- Vals AI Excel Modeling: 8.63 (#46/46)
- Vals AI Legal Research Bench: 3.37 (#46/49)
- ...and 2 more
Kimi K2 — ELO 1642, #107
- CaseLaw v2: 65.7 (#10/54)
- MATH 500: 94.2 (#10/60)
- HMMT 2025: 93.33 (#23/82)
- MGSM: 90.95 (#31/74)
- Finance Agent v1.1: 36.65 (#44/53)
- ClawProBench: 47.83 (#46/57)
- Mercor APEX: 4.1 (#48/53)
- MathArena Apex: 0.0 (#50/50)
- IOI: 1.25 (#61/64)
Claude Sonnet 4 — ELO 1641, #108
- WildAgtEval: 63.6 (#2/10)
- MGSM: 93.02 (#11/74)
- MATH 500: 90.32 (#26/60)
- IOI: 6.5 (#44/64)
- MedCode: 33.94 (#56/75)
- MedScribe: 72.41 (#58/74)
- CritPt: 0.3 (#146/213)
- AA-LCR: 44.3 (#162/217)
Nemotron 3 Ultra — ELO 1641, #109
- Finance Agent v2: 37.67 (#32/40)
- Vals Index: 43.99 (#33/40)
- Vals AI Harvey Legal Agent Bench: 0.42 (#35/50)
- Vals AI Legal Research Bench: 15.38 (#36/49)
- Vals AI Excel Modeling: 31.73 (#37/46)
- Mercor APEX: 11.5 (#40/53)
- Vals AI Code Migration: 4.91 (#45/48)
- MedCode: 38.62 (#45/75)
- ProofBench: 2.0 (#52/54)
- AIIQ Composite IQ: 107.0 (#53/115)
O1 (2024-12-17) — ELO 1641, #110
- TextClass Benchmark: 1768.81 (#4/108)
- MATH 500: 90.4 (#24/60)
- MGSM: 89.31 (#47/74)
LongCat 2.0 — ELO 1641, #111
- SvelteBench: 88.9 (#73/169)
Claude Opus 4 (20250514) — ELO 1640, #112
- OpenEval - MMLU-Pro CoT: 87.5 (#2/77)
- OpenEval - Omni-MATH: 61.6 (#2/74)
- OpenEval - IFEval Strict: 91.77 (#3/77)
- OpenEval - GPQA CoT: 70.85 (#4/74)
- HELM Safety: 96.7675 (#8/59)
- TutorBench: 45.46 (#25/27)
Qwen 3.5 27B — ELO 1639, #114
- MathArena Apex: 2.08 (#34/50)
Qwen 3 Max (Preview) — ELO 1639, #115
- MGSM: 92.15 (#23/74)
- IOI: 7.75 (#39/64)
Step 3.5 Flash — ELO 1637, #117
- HMMT 2025: 98.33 (#4/82)
- MathArena Apex: 13.54 (#20/50)
- ClawProBench: 41.75 (#53/57)
Grok 4.1 Fast — ELO 1636, #118
- IDE-Bench: 35.0 (#9/15)
- MageBench Season 1: 1637.0 (#10/34)
- CaseLaw v2: 60.45 (#24/54)
- Finance Agent v1.1: 52.45 (#25/53)
- MathArena Apex: 5.21 (#29/50)
- HMMT 2025: 90.0 (#33/82)
- MedScribe: 78.73 (#36/74)
- IOI: 7.67 (#40/64)
- MGSM: 89.53 (#45/74)
- ProofBench: 4.0 (#48/54)
- ...and 1 more
O4 Mini (2025-04-16) — ELO 1636, #119
- MGSM: 93.42 (#9/74)
- MATH 500: 94.2 (#11/60)
- IOI: 4.83 (#48/64)
- TextClass Benchmark: 1538.33 (#55/108)
- MedCode: 33.79 (#57/75)
- MedScribe: 69.14 (#67/74)
DeepSeek V3.2 — ELO 1635, #120
- SWE-bench Verified: 70.0 (#14/47)
- ClawProBench: 60.13 (#15/57)
- Vals AI MMLU-Pro: 84.92 (#56/132)
- Vals AI LiveCodeBench: 80.69 (#62/137)
Claude 3.7 Sonnet (20250219) — ELO 1635, #121
- OpenEval - IFEval Strict: 94.0 (#1/77)
- OpenEval - MMLU-Pro CoT: 86.0 (#5/77)
- OpenEval - GPQA CoT: 66.0 (#10/74)
- MGSM: 92.98 (#13/74)
- OpenEval - Omni-MATH: 36.67 (#22/74)
- MATH 500: 91.6 (#22/60)
- TextClass Benchmark: 1500.76 (#65/108)
Ring-2.6-1T — ELO 1634, #123
- AIIQ Composite IQ: 101.0 (#72/115)
MiniMax-M2.5 — ELO 1632, #125
- SWE-bench Verified: 75.8 (#3/47)
Qwen 3.6 35B A3B — ELO 1632, #126
- ArxivMath: 37.5 (#16/17)
- ClawProBench: 56.94 (#29/57)
- Android Bench: 37.0 (#32/32)
- AIIQ Composite IQ: 100.0 (#77/115)
Grok 4 Fast — ELO 1632, #127
- CaseLaw v2: 65.7 (#11/54)
- MageBench Season 1: 1620.0 (#12/34)
- HMMT 2025: 91.67 (#28/82)
- MedScribe: 81.63 (#29/74)
- MathArena Apex: 5.21 (#30/50)
- MGSM: 90.87 (#34/74)
- IOI: 11.5 (#36/64)
- Finance Agent v1.1: 46.08 (#37/53)
- MedCode: 37.38 (#49/75)
GLM-4.6 — ELO 1631, #128
- VitaBench 2.0: 35.9 (#13/21)
- HMMT 2025: 93.33 (#24/82)
- SWE-bench Verified: 55.4 (#31/47)
- MGSM: 89.75 (#44/74)
- Finance Agent v1.1: 36.48 (#45/53)
- MathArena Apex: 0.52 (#47/50)
- IOI: 4.33 (#50/64)
KAT Coder Pro V2 — ELO 1631, #129
- ClawProBench: 54.74 (#34/57)
O1 — ELO 1629, #130
- HMMT 2025: 48.33 (#62/82)
DeepSeek V3.2 Exp — ELO 1628, #131
- CritPt: 1.4 (#83/213)
MiniMax-M2.7 — ELO 1628, #132
- VitaBench 2.0: 34.9 (#14/21)
- Vals AI Excel Modeling: 28.45 (#39/46)
- Vals AI Harvey Legal Agent Bench: 0.0 (#42/50)
- Vals AI Legal Research Bench: 10.58 (#42/49)
GPT-4.1 (2025-04-14) — ELO 1626, #134
- SWE-bench Verified: 39.6 (#38/47)
- TextClass Benchmark: 1520.39 (#60/108)
O4 Mini — ELO 1625, #136
- USAMO25: 19.05 (#6/13)
- VitaBench 2.0: 21.0 (#19/21)
- HMMT 2025: 83.33 (#41/82)
Qwen 3 235B A22B 2507 — ELO 1624, #139
- WritingBench: 88.3 (#1/67)
- MathArena Apex: 5.21 (#28/50)
- Creative Writing v3: 86.1 (#114/124)
MiMo-V2-Omni — ELO 1624, #140
- ClawProBench: 57.65 (#23/57)
Gemini 2.5 Flash — ELO 1622, #141
- VitaBench 2.0: 28.2 (#18/21)
- MageBench Season 1: 1572.0 (#24/34)
- MedScribe: 82.98 (#25/74)
- MedCode: 40.36 (#41/75)
- APEX-Agents: 6.4 (#43/45)
- IOI: 2.61 (#57/64)
- HMMT 2025: 64.17 (#58/82)
Claude Sonnet 4 (20250514) — ELO 1622, #142
- OpenEval - Omni-MATH: 60.2 (#3/74)
- OpenEval - GPQA CoT: 70.63 (#5/74)
- HELM Safety: 97.4369 (#5/59)
- OpenEval - MMLU-Pro CoT: 84.3 (#8/77)
- MATH 500: 93.8 (#16/60)
- OpenEval - IFEval Strict: 84.01 (#19/77)
- WritingBench: 74.07 (#35/67)
- MGSM: 90.87 (#36/74)
- IOI: 4.58 (#49/64)
- MedCode: 34.96 (#52/75)
- ...and 1 more
GLM-4.5 — ELO 1621, #143
- VitaBench 2.0: 36.9 (#12/21)
- SWE-bench Verified: 54.2 (#32/47)
- MGSM: 90.84 (#37/74)
- MathArena Apex: 1.04 (#42/50)
- HMMT 2025: 77.5 (#46/82)
- IOI: 2.92 (#56/64)
KAT-Coder-Pro V1 — ELO 1621, #145
- AIIQ Composite IQ: 101.0 (#73/115)
DeepSeek R1 0528 — ELO 1620, #146
- OpenEval - GPQA CoT: 66.59 (#8/74)
- VitaBench 2.0: 39.6 (#9/21)
- OpenEval - MMLU-Pro CoT: 79.3 (#12/77)
- OpenEval - Omni-MATH: 42.4 (#14/74)
- OpenEval - IFEval Strict: 78.37 (#42/77)
GPT-4.5 — ELO 1620, #147
- TextClass Benchmark: 1767.86 (#5/108)
- LingOly-TOO: 25.4502 (#10/16)
GPT-4.1 — ELO 1619, #149
- NL2Scratch: 99.1 (#1/11)
- Fin-RATE: 33.24 (#2/17)
- CaseLaw v2: 69.88 (#3/54)
- MATH 500: 87.2 (#33/60)
- MGSM: 87.67 (#58/74)
Qwen 3 235B A22B 2507 Instruct — ELO 1619, #151
- OpenEval - Omni-MATH: 71.83 (#1/74)
- OpenEval - GPQA CoT: 72.65 (#3/74)
- OpenEval - MMLU-Pro CoT: 84.4 (#7/77)
- WritingBench: 85.2 (#7/67)
- OpenEval - IFEval Strict: 83.52 (#23/77)
Kimi K2 0905 — ELO 1618, #153
- MageBench Season 1: 1558.0 (#26/34)
Gemini 2.5 Flash (Preview 04-17) — ELO 1618, #154
- OpenEval - IFEval Strict: 89.8 (#6/77)
- OpenEval - Omni-MATH: 38.45 (#20/74)
- MATH 500: 91.8 (#20/60)
- OpenEval - GPQA CoT: 39.01 (#32/74)
- OpenEval - MMLU-Pro CoT: 63.9 (#33/77)
O1 Preview — ELO 1616, #155
- LingOly-TOO: 32.2202 (#6/16)
- AA MMLU-Pro: 84.77 (#37/348)
- AA MATH-500: 92.4 (#61/195)
- AA AIME 2025: 79.67 (#69/270)
- AA GPQA Diamond: 76.46 (#214/575)
- Artificial Analysis Intelligence Index: 17.21 (#282/574)
Doubao Seed Code — ELO 1616, #156
- AIIQ Composite IQ: 97.0 (#85/115)
Claude 3.7 Sonnet — ELO 1615, #157
- LingOly-TOO: 42.8881 (#3/16)
- USAMO25: 3.65 (#10/13)
- HMMT 2025: 31.67 (#72/82)
Gemma 4 31B — ELO 1615, #158
- AIIQ Composite IQ: 101.0 (#74/115)
Qwen 3.5 35B A3B — ELO 1613, #161
- MathArena Apex: 4.17 (#31/50)
Grok 3 Mini Beta — ELO 1613, #162
- TextClass Benchmark: 1546.47 (#50/108)
Gemma 4 26B A4B (IT) — ELO 1612, #163
- CPTU Bench: 4.23 (#11/102)
Claude Haiku 4.5 — ELO 1611, #165
- IDE-Bench: 78.75 (#4/15)
- MageBench Season 1: 1637.0 (#9/34)
- HarmActionsEval: 0.0 (#10/10)
- SWE-bench Verified: 66.6 (#18/47)
- Inverse Turing Bench: 47.04 (#34/40)
- AIIQ Composite IQ: 98.0 (#81/115)
- PM-LLM-Benchmark: 27.7 (#126/171)
- AA-LCR: 43.7 (#166/217)
- CritPt: 0.0 (#206/213)
MiMo-V2-Flash — ELO 1610, #168
- MageBench Season 1: 1586.0 (#21/34)
- AIIQ Composite IQ: 105.0 (#57/115)
Claude 3.5 Sonnet (20241022) — ELO 1609, #170
- OpenEval - IFEval Strict: 85.55 (#14/77)
- OpenEval - MMLU-Pro CoT: 77.7 (#16/77)
- MGSM: 92.58 (#16/74)
- OpenEval - GPQA CoT: 56.5 (#17/74)
- OpenEval - Omni-MATH: 27.6 (#38/74)
- MATH 500: 72.4 (#50/60)
- TextClass Benchmark: 1384.79 (#90/108)
Claude 3.5 Sonnet (20240620) — ELO 1607, #173
- Aider Refactoring Benchmark: 64.0 (#4/14)
- Defects4J: 41.5 (#9/35)
- OpenEval - IFEval Strict: 85.98 (#12/77)
- OpenEval - MMLU-Pro CoT: 73.34 (#21/77)
Grok 3 — ELO 1607, #174
- USAMO25: 4.76 (#8/13)
- MATH 500: 89.8 (#27/60)
- MGSM: 91.35 (#29/74)
- TextClass Benchmark: 1545.77 (#51/108)
Grok Code Fast 1 — ELO 1605, #176
- IDE-Bench: 11.25 (#12/15)
- IOI: 4.33 (#51/64)
Qwen 3 Next 80B A3B — ELO 1603, #178
- OpenEval - Omni-MATH: 46.68 (#11/74)
- OpenEval - GPQA CoT: 63.0 (#12/74)
- OpenEval - MMLU-Pro CoT: 78.6 (#13/77)
- OpenEval - IFEval Strict: 80.96 (#35/77)
Mistral Medium 3.5 — ELO 1603, #179
- Vals AI ProofBench: 9.0 (#22/24)
- Vals Multimodal Index: 34.77 (#29/29)
- Finance Agent v2: 32.1 (#34/40)
- Vals AI Harvey Legal Agent Bench: 0.42 (#36/50)
- Finance Agent v1.1: 46.11 (#36/53)
- Vals Index: 37.81 (#37/40)
- ProofBench: 10.0 (#39/54)
- Vals AI Legal Research Bench: 9.13 (#44/49)
- Vals AI Code Migration: 5.13 (#44/48)
- Vals AI Excel Modeling: 11.19 (#45/46)
- ...and 4 more
Gemini 2.5 Flash Lite (Preview 09-2025) — ELO 1602, #180
- MGSM: 89.53 (#46/74)
- MedScribe: 75.82 (#50/74)
- MedCode: 34.19 (#54/75)
Ling-2.6-1T — ELO 1602, #181
- ClawProBench: 57.4 (#25/57)
- SvelteBench: 68.9 (#132/169)
Claude 3.5 Sonnet — ELO 1601, #182
- LingOly-TOO: 28.1014 (#8/16)
- ZEROBench-Sub: 20.71 (#26/46)
- HMMT 2025: 1.67 (#82/82)
GPT-5.4 Nano — ELO 1601, #183
- Vals Multimodal Index: 47.64 (#24/29)
- Vals AI Code Migration: 14.47 (#30/48)
- APEX-Agents: 25.5 (#30/45)
- Finance Agent v2: 38.22 (#30/40)
- Vals Index: 46.63 (#31/40)
- LiveBench: 71.31 (#32/48)
- Finance Agent v1.1: 47.8 (#32/53)
- IOI: 15.25 (#32/64)
- MedCode: 41.03 (#36/75)
- Vals AI Harvey Legal Agent Bench: 0.0 (#40/50)
- ...and 5 more
InternVL3-78B — ELO 1597, #186
- RealWorldQA: 78.0 (#7/12)
MiniMax-M2 — ELO 1597, #187
- SWE-bench Verified: 61.0 (#24/47)
Trinity Large — ELO 1597, #188
- CaseLaw v2: 57.88 (#29/54)
O3 Mini — ELO 1596, #189
- LingOly-TOO: 30.5909 (#7/16)
- USAMO25: 2.08 (#13/13)
- HMMT 2025: 67.5 (#55/82)
Llama 3.1 405B — ELO 1596, #190
- TextClass Benchmark: 1755.81 (#7/108)
Mistral Medium 3.1 — ELO 1595, #193
- MageBench Season 1: 1569.0 (#25/34)
Qwen 3 235B A22B — ELO 1595, #194
- MATH 500: 94.6 (#9/60)
- MageBench Season 1: 1594.0 (#17/34)
- MGSM: 92.47 (#17/74)
- HMMT 2025: 62.5 (#59/82)
- IOI: 0.0 (#64/64)
Mercury 2 — ELO 1595, #195
- AIIQ Composite IQ: 99.0 (#80/115)
Nemotron 3 Super — ELO 1593, #198
- AIIQ Composite IQ: 103.0 (#64/115)
Qwen3 Coder Next — ELO 1592, #199
- ClawProBench: 46.84 (#48/57)
GPT-4o — ELO 1591, #201
- ComplexFuncBench: 66.5 (#1/18)
- LingOly-TOO: 15.6334 (#12/16)
- HMMT 2025: 5.83 (#81/82)
Nemotron 3.5 Lightning — ELO 1591, #202
- LLM Stats (Terminal-Bench 2.1): 24.58 (#28/28)
- LLM Stats (BrowseComp): 36.97 (#56/62)
- OpenRouter GPQA Diamond: 68.2 (#84/114)
- ZeroEval GPQA Diamond: 75.44 (#107/239)
- SpeechMap Compliance: 75.0 (#114/367)
- AA GDPval: 823.59 (#117/198)
- Tau3 Banking: 8.87 (#118/166)
- AI Chess Leaderboard (Reasoning): 752.0 (#121/313)
- AI Chess Leaderboard (Continuation): 553.0 (#138/258)
- SvelteBench: 55.6 (#141/169)
- ...and 30 more
GPT-OSS-120B — ELO 1590, #203
- FINAL Bench Metacognitive: 73.33 (#7/9)
- MATH 500: 94.8 (#7/60)
- MGSM: 92.04 (#24/74)
- MageBench Season 1: 1516.0 (#31/34)
- HMMT 2025: 90.0 (#32/82)
- MathArena Apex: 1.04 (#44/50)
- Finance Agent v1.1: 21.54 (#47/53)
- CaseLaw v2: 48.77 (#50/54)
- AIIQ Composite IQ: 105.0 (#58/115)
Grok 3 Mini — ELO 1590, #204
- HMMT 2025: 74.17 (#50/82)
O3 Mini (2025-01-31) — ELO 1590, #205
- TextClass Benchmark: 1684.99 (#18/108)
- MATH 500: 91.8 (#19/60)
- MGSM: 91.35 (#28/74)
GPT-4.1 Mini — ELO 1588, #207
- MATH 500: 88.0 (#31/60)
- SWE-bench Verified: 23.9 (#42/47)
- TextClass Benchmark: 1547.62 (#49/108)
- MGSM: 87.78 (#57/74)
Mistral Large 3 — ELO 1588, #208
- AIIQ Composite IQ: 89.0 (#103/115)
GLM 4.5 Air — ELO 1587, #211
- HMMT 2025: 69.17 (#53/82)
DeepSeek V3 (0324) — ELO 1587, #212
- MGSM: 91.67 (#27/74)
- MATH 500: 88.6 (#30/60)
- IOI: 1.67 (#60/64)
INTELLECT-3 — ELO 1587, #213
- AIIQ Composite IQ: 94.0 (#95/115)
Gemma 4 12B — ELO 1586, #215
- AIIQ Composite IQ: 98.0 (#82/115)
Llama 4 Maverick — ELO 1584, #218
- IDE-Bench: 2.5 (#14/15)
- MageBench Season 1: 1590.0 (#19/34)
- TextClass Benchmark: 1473.88 (#73/108)
Gemini 2.0 Flash — ELO 1584, #219
- USAMO25: 4.17 (#9/13)
- TextClass Benchmark: 1701.95 (#15/108)
- SWE-bench Verified: 13.5 (#45/47)
- HMMT 2025: 35.83 (#67/82)
Qwen 2.5 72B Instruct — ELO 1583, #222
- OpenEval - IFEval Strict: 89.93 (#5/77)
- OpenEval - MMLU-Pro CoT: 69.31 (#26/77)
Gemini 2.0 Flash (001) — ELO 1583, #223
- OpenEval - Omni-MATH: 45.9 (#12/74)
- OpenEval - GPQA CoT: 55.61 (#18/74)
- OpenEval - IFEval Strict: 84.07 (#18/77)
- OpenEval - MMLU-Pro CoT: 73.7 (#20/77)
- MATH 500: 88.0 (#32/60)
- MGSM: 89.02 (#50/74)
Magistral Medium 1.2 — ELO 1583, #225
- IOI: 0.67 (#62/64)
- MGSM: 74.62 (#71/74)
- AIIQ Composite IQ: 94.0 (#93/115)
GPT-4o (2024-11-20) — ELO 1582, #228
- CaseLaw v2: 59.7 (#25/54)
- MGSM: 90.36 (#40/74)
- SWE-bench Verified: 21.6 (#43/47)
- MATH 500: 74.0 (#47/60)
Qwen 3.5 9B — ELO 1580, #234
- RealWorldQA: 80.3 (#5/12)
- MathArena Apex: 0.52 (#49/50)
Llama 3.1 405B Instruct — ELO 1580, #235
- MATH 500: 71.4 (#51/60)
DeepSeek R1 — ELO 1579, #240
- TextClass Benchmark: 1718.73 (#14/108)
- MATH 500: 92.2 (#18/60)
- MGSM: 92.25 (#20/74)
DeepSeek V3 — ELO 1579, #241
- TextClass Benchmark: 1732.54 (#11/108)
- OpenEval - Omni-MATH: 40.27 (#17/74)
- OpenEval - GPQA CoT: 53.81 (#19/74)
- MGSM: 92.15 (#21/74)
- OpenEval - MMLU-Pro CoT: 72.3 (#24/77)
- OpenEval - IFEval Strict: 83.24 (#25/77)
- MATH 500: 80.4 (#38/60)
GPT-4o (2024-08-06) — ELO 1576, #242
- Aider Refactoring Benchmark: 49.4 (#8/14)
- Defects4J: 34.1 (#15/35)
- MGSM: 90.69 (#38/74)
- MATH 500: 75.2 (#45/60)
- Finance Agent v1.1: 8.06 (#49/53)
GPT-5 Nano — ELO 1575, #245
- CommunityFact: 75.59 (#3/17)
- RealWorldQA: 71.8 (#11/12)
- MATH 500: 93.8 (#15/60)
- MageBench Season 1: 1499.0 (#33/34)
- ProofBench: 12.0 (#37/54)
- SWE-bench Verified: 34.8 (#39/47)
- CaseLaw v2: 52.63 (#43/54)
- MGSM: 89.31 (#48/74)
- MedScribe: 72.86 (#55/74)
- MedCode: 30.44 (#67/75)
- ...and 1 more
Gemini 1.5 Pro (002) — ELO 1573, #249
- OpenEval - MMLU-Pro CoT: 73.7 (#19/77)
- OpenEval - GPQA CoT: 53.36 (#20/74)
- OpenEval - IFEval Strict: 83.67 (#22/77)
- OpenEval - Omni-MATH: 36.4 (#23/74)
- MATH 500: 82.8 (#37/60)
- MGSM: 89.2 (#49/74)
Llama 4 Maverick Instruct — ELO 1573, #250
- MedCode: 36.51 (#51/75)
- MedScribe: 54.22 (#73/74)
GPT-4o (2024-05-13) — ELO 1572, #251
- TextClass Benchmark: 1825.22 (#1/108)
Claude Haiku 4.5 (20251001) — ELO 1571, #252
- OpenEval - Omni-MATH: 56.1 (#4/74)
- OpenEval - GPQA CoT: 60.54 (#15/74)
- MedScribe: 85.23 (#16/74)
- OpenEval - MMLU-Pro CoT: 77.7 (#17/77)
- MGSM: 92.15 (#22/74)
- Vals Multimodal Index: 42.88 (#26/29)
- CaseLaw v2: 56.48 (#30/54)
- Vals AI Harvey Legal Agent Bench: 0.83 (#32/50)
- Finance Agent v1.1: 46.93 (#34/53)
- Finance Agent v2: 31.01 (#35/40)
- ...and 6 more
Gemini 2.5 Flash Lite — ELO 1570, #255
- OpenEval - Omni-MATH: 47.97 (#10/74)
- OpenEval - IFEval Strict: 81.02 (#34/77)
- OpenEval - MMLU-Pro CoT: 53.7 (#43/77)
- OpenEval - GPQA CoT: 30.94 (#47/74)
- MedScribe: 72.83 (#56/74)
- MedCode: 27.11 (#71/75)
Mistral Small 4 — ELO 1568, #257
- ClawProBench: 45.26 (#49/57)
Devstral Small 2 — ELO 1566, #260
- SWE-bench Verified: 56.4 (#28/47)
Gemini 1.5 Pro — ELO 1564, #265
- PhysicsFinals: 63.9 (#1/34)
- TextClass Benchmark: 1782.7 (#2/108)
- Natural2Code: 85.4 (#2/31)
- LingOly-TOO: 20.4615 (#11/16)
Llama 3.1 70B Instruct — ELO 1561, #269
- MATH 500: 65.0 (#56/60)
Magistral Small 1.2 — ELO 1561, #271
- MGSM: 86.25 (#62/74)
Qwen 3 VL 30B A3B — ELO 1561, #272
- RealWorldQA: 77.4 (#8/12)
Mistral Large 2 (Nov) Instruct (2411) — ELO 1560, #276
- MATH 500: 74.4 (#46/60)
- MGSM: 87.24 (#60/74)
Qwen 3 30B A3B 2507 — ELO 1560, #278
- MathArena Apex: 0.52 (#48/50)
Command A+ — ELO 1560, #280
- AIIQ Composite IQ: 89.0 (#105/115)
Grok 2 (1212) — ELO 1558, #281
- TextClass Benchmark: 1758.36 (#6/108)
- MATH 500: 78.4 (#42/60)
- MGSM: 86.15 (#64/74)
Qwen 3 30B A3B — ELO 1556, #286
- HMMT 2025: 50.83 (#61/82)
GPT-4 Turbo — ELO 1553, #294
- TextClass Benchmark: 1781.47 (#3/108)
- Aider Refactoring Benchmark: 34.1 (#10/14)
jamba-large-1.7 — ELO 1550, #304
- Finance Agent v1.1: 0.37 (#52/53)
Llama 3.3 70B — ELO 1547, #311
- TextClass Benchmark: 1746.41 (#9/108)
Llama 3.3 70B Instruct — ELO 1546, #313
- LingOly-TOO: 8.2131 (#15/16)
- MGSM: 91.09 (#30/74)
- MATH 500: 73.4 (#48/60)
Nova Pro (v1) — ELO 1546, #314
- OpenEval - GPQA CoT: 44.62 (#26/74)
- OpenEval - MMLU-Pro CoT: 67.3 (#28/77)
- OpenEval - IFEval Strict: 81.55 (#33/77)
- OpenEval - Omni-MATH: 24.22 (#39/74)
Llama 4 Scout Instruct — ELO 1546, #315
- MATH 500: 79.2 (#40/60)
- MGSM: 87.96 (#55/74)
- MedCode: 23.31 (#74/75)
- MedScribe: 50.59 (#74/74)
GPT-4 Preview (0125) — ELO 1545, #319
- Aider Refactoring Benchmark: 33.7 (#11/14)
Yi 1.5 34B — ELO 1540, #332
- PhysicsFinals: 17.4 (#17/34)
- EHRBench: 58.94 (#18/31)
QwQ-32B — ELO 1538, #335
- USAMO25: 2.98 (#11/13)
- HMMT 2025: 47.5 (#63/82)
Qwen 3.5 4B — ELO 1536, #340
- RealWorldQA: 79.5 (#6/12)
GPT-OSS-20B — ELO 1535, #342
- MATH 500: 94.2 (#13/60)
- HMMT 2025: 76.67 (#47/82)
- MGSM: 89.02 (#51/74)
- CaseLaw v2: 43.84 (#54/54)
- AIIQ Composite IQ: 100.0 (#78/115)
Llama 4 Scout — ELO 1535, #343
- IDE-Bench: 2.5 (#13/15)
- TextClass Benchmark: 1500.45 (#66/108)
GPT-4 — ELO 1531, #351
- AA Omniscience - Business: 21.1 (#155/479)
- AA MATH-500: 56.8 (#159/195)
- AA Omniscience - Law: 16.5 (#164/479)
- AA Omniscience - Software Engineering (SWE) - Julia: 20.0 (#172/479)
- AA GDPval: 238.72 (#174/198)
- AA Omniscience - Humanities & Social Sciences: 23.6 (#180/479)
- AA Omniscience: -33.83 (#207/479)
- AA Omniscience - Health: 21.7 (#211/479)
- AA Omniscience - Software Engineering (SWE) - R: 14.0 (#216/479)
- AA-Omniscience Accuracy: 21.0 (#222/479)
- ...and 18 more
Falcon-H1R-7B — ELO 1530, #356
- HMMT 2025: 84.17 (#39/82)
Gemini 1.5 Flash (002) — ELO 1526, #360
- OpenEval - IFEval Strict: 83.12 (#26/77)
- OpenEval - GPQA CoT: 43.72 (#27/74)
- OpenEval - MMLU-Pro CoT: 67.8 (#27/77)
- OpenEval - Omni-MATH: 30.45 (#31/74)
- MATH 500: 78.8 (#41/60)
- MGSM: 86.58 (#61/74)
Mistral Large — ELO 1526, #363
- MageBench Season 1: 1501.0 (#32/34)
Gemini 1.5 Flash — ELO 1523, #370
- TextClass Benchmark: 1668.98 (#25/108)
Qwen 3 4B 2507 — ELO 1523, #371
- MathArena Apex: 2.08 (#36/50)
GPT-4 (0613) — ELO 1523, #372
- TextClass Benchmark: 1747.59 (#8/108)
Ling-2.6-flash — ELO 1523, #373
- ClawProBench: 27.04 (#57/57)
O1 Mini (2024-09-12) — ELO 1521, #375
- TextClass Benchmark: 1626.65 (#29/108)
Gemma 3 27B — ELO 1518, #378
- BIG-Bench Extra Hard: 19.3 (#8/20)
QwQ 32B-Preview — ELO 1514, #383
- HMMT 2025: 18.33 (#76/82)
GPT-4o Mini (2024-07-18) — ELO 1511, #387
- TextClass Benchmark: 1674.64 (#23/108)
- MATH 500: 72.6 (#49/60)
- MGSM: 86.18 (#63/74)
GPT-4o Mini — ELO 1508, #394
- NL2Scratch: 78.0 (#11/11)
- VitaBench 2.0: 6.7 (#21/21)
Phi-4 — ELO 1508, #395
- LingOly-TOO: 10.9966 (#14/16)
Granite 4.0 H Small — ELO 1504, #400
- OpenEval - IFEval Strict: 88.97 (#7/77)
- OpenEval - Omni-MATH: 29.57 (#34/74)
- OpenEval - GPQA CoT: 38.34 (#35/74)
- OpenEval - MMLU-Pro CoT: 56.9 (#39/77)
Nova Lite (v1) — ELO 1503, #403
- OpenEval - GPQA CoT: 39.69 (#31/74)
- OpenEval - MMLU-Pro CoT: 60.0 (#37/77)
- OpenEval - Omni-MATH: 23.3 (#40/74)
- OpenEval - IFEval Strict: 77.6 (#45/77)
aya-23-35B — ELO 1494, #418
- LingOly-TOO: 5.7082 (#16/16)
Claude 3.5 Haiku (20241022) — ELO 1493, #420
- OpenEval - MMLU-Pro CoT: 60.5 (#36/77)
- OpenEval - GPQA CoT: 36.32 (#39/74)
- OpenEval - IFEval Strict: 79.17 (#40/77)
- OpenEval - Omni-MATH: 22.4 (#42/74)
- MATH 500: 64.2 (#57/60)
- MGSM: 84.62 (#68/74)
- TextClass Benchmark: 1413.9 (#85/108)
Gemma 3 12B — ELO 1491, #426
- BIG-Bench Extra Hard: 16.3 (#9/20)
Yi 1.5 34B Chat — ELO 1481, #441
- INVESTORBENCH: 37.97 (#5/14)
GLM-4 9B Chat — ELO 1476, #446
- TextClass Benchmark: 1616.51 (#33/108)
GPT-4.1 Nano — ELO 1474, #449
- MATH 500: 80.2 (#39/60)
- TextClass Benchmark: 1533.06 (#58/108)
- MGSM: 69.27 (#73/74)
Gemma 4 E4B — ELO 1472, #451
- AIIQ Composite IQ: 93.0 (#99/115)
Yi-1.5-9B — ELO 1470, #457
- EHRBench: 45.51 (#27/31)
Qwen 3.5 2B — ELO 1469, #459
- ArxivMath: 10.62 (#17/17)
Ministral 3 3B — ELO 1469, #460
- HarmActionsEval: 2.13 (#5/10)
Command-R+ — ELO 1469, #461
- IDE-Bench: 0.0 (#15/15)
GPT-3.5 Turbo — ELO 1464, #470
- VitaBench 2.0: 14.0 (#20/21)
Mistral Small (2402) — ELO 1457, #482
- MATH 500: 70.6 (#53/60)
- MGSM: 83.96 (#70/74)
Granite 4.0 Micro — ELO 1455, #489
- OpenEval - IFEval Strict: 84.87 (#17/77)
- OpenEval - Omni-MATH: 20.93 (#45/74)
- OpenEval - GPQA CoT: 30.72 (#50/74)
- OpenEval - MMLU-Pro CoT: 39.5 (#53/77)
Llama 3.1 8B Instruct — ELO 1453, #494
- MATH 500: 44.4 (#59/60)
GPT-3.5 Turbo (0301) — ELO 1449, #501
- HELM: 76.03 (#13/66)
Yi-1.5-9B Chat — ELO 1444, #514
- INVESTORBENCH: 22.91 (#12/14)
OLMo-2-1124-7B-Instruct — ELO 1436, #525
- OpenEval - GPQA CoT: 29.6 (#52/74)
- OpenEval - IFEval Strict: 69.29 (#54/77)
- OpenEval - Omni-MATH: 11.63 (#56/74)
- OpenEval - MMLU-Pro CoT: 29.2 (#57/77)
Gemma 4 E2B — ELO 1434, #530
- AIIQ Composite IQ: 86.0 (#106/115)
GPT-3.5 Turbo (0125) — ELO 1424, #548
- TextClass Benchmark: 1560.51 (#44/108)
Gemma 3 4B — ELO 1409, #570
- BIG-Bench Extra Hard: 11.0 (#11/20)
Phi-4 Mini Instruct — ELO 1401, #578
- HarmActionsEval: 2.84 (#3/10)
Yi 6B (Base) — ELO 1335, #653
- BBH: 47.2 (#15/16)
Claude Mythos 5 — ELO 1809
- OSWorld-Verified: 85.0 (#1/27)
DeepSeek V4 Flash (0423) — ELO 1669
- LLM Stats (CodeForces): 93.87 (#3/17)
- LLM Stats (MathArena Apex): 72.1 (#3/8)
- LLM Stats (CSimpleQA): 73.2 (#7/8)
- LLM Stats (HMMT Feb 26): 91.9 (#7/12)
- LLM Stats (IMO-AnswerBench): 85.1 (#10/20)
- LLM Stats (MCP Atlas): 67.4 (#23/33)
- LLM Stats (Toolathlon): 43.5 (#27/37)
- LLM Stats (BrowseComp): 53.5 (#41/62)
- ZeroEval GPQA Diamond: 87.4 (#43/239)
- LLM Stats Score: 36.8 (#64/341)
DeepSeek V4 Pro Preview — ELO 1676
- GDPevo: 43.58 (#4/4)
- Creative Writing (Lechmazur): 0.0 (#22/44)
- Multi-turn Debate (Lechmazur): 1471.8 (#28/46)
- NYT Connections Extended: 59.9 (#53/104)
Doubao-Seed-1.6 — ELO 1635
- VitaBench 2.0: 37.3 (#10/21)
ERNIE 5.0 — ELO 1609
- AIIQ Composite IQ: 94.0 (#96/115)
GLM-5.3 — ELO 1778
- LLM Stats (AutomationBench): 48.2 (#1/12)
- LLM Stats (CyberGym): 84.5 (#1/13)
- LLM Stats (PostTrainBench): 39.8 (#1/6)
- FrontierSWE: 78.0 (#2/17)
- LLM Stats (NL2Repo): 58.0 (#2/17)
- LLM Stats (FrontierSWE): 78.1 (#3/16)
- LLM Stats (Terminal-Bench 2.1): 88.2 (#3/28)
- LLM Stats (Toolathlon): 73.0 (#4/37)
- Vending-Bench 2: 8163.61 (#6/60)
- LLM Stats (Program Bench): 19.0 (#6/6)
- ...and 5 more
GPT-4 Turbo (Preview) — ELO 1538
- Natural2Code: 51.5 (#11/31)
GPT-5.1 Codex Max — ELO 1686
- IDE-Bench: 73.75 (#5/15)
- ALE-Bench: 1228.25 (#12/89)
- IOI: 21.42 (#24/64)
- LiveBench: 72.39 (#29/48)
- Epoch AI - Proofbench: 9.0 (#40/52)
- ProofBench: 9.0 (#41/54)
GPT-5.5 Instant — ELO 1666
- Arena AI Document: 1402.0 (#36/38)
- FrontierMath - Tiers 1-3 (v2): 26.32 (#42/48)
- FrontierMath - Tier 4 (v2): 2.44 (#43/50)
- SimpleQA Verified: 38.0 (#45/77)
- Chess Puzzles (Epoch AI): 12.0 (#116/161)
- OTIS Mock AIME 2024-25: 68.06 (#128/238)
Gemini 2.0 Flash Lite (Preview 02-05) — ELO 1528
- OpenEval - Omni-MATH: 37.37 (#21/74)
- OpenEval - GPQA CoT: 50.0 (#23/74)
- OpenEval - MMLU-Pro CoT: 72.0 (#25/77)
- OpenEval - IFEval Strict: 82.44 (#29/77)
Gemini 2.5 Flash (Preview 09-2025) — ELO 1646
- MedScribe: 78.5 (#37/74)
- MedCode: 40.54 (#39/75)
- MGSM: 89.85 (#43/74)
Gemma 3n E2B Instructed LiteRT (Preview) — ELO 1392
- ECLeKTic: 2.5 (#12/16)
Gemma 3n E4B Instructed LiteRT Preview — ELO 1423
- ECLeKTic: 1.9 (#14/16)
Gemma 4 26B — ELO 1601
- AIIQ Composite IQ: 96.0 (#90/115)
Grok Beta — ELO 1543
- TextClass Benchmark: 1741.94 (#10/108)
Laguna M.1 — ELO 1574
- Finance Agent v2: 25.03 (#39/40)
- Vals Index: 33.33 (#39/40)
- ProofBench: 0.0 (#54/54)
- MedCode: 25.24 (#73/75)
- AIIQ Composite IQ: 84.0 (#108/115)
Laguna S 2.1 — ELO 1658
- LLM Stats (Terminal-Bench 2.1): 70.2 (#19/28)
- LLM Stats (Toolathlon): 49.7 (#19/37)
- LLM Stats (DeepSWE 1.1): 40.4 (#21/25)
- LLM Stats Score: 41.9 (#40/341)
Laguna XS.2 — ELO 1548
- Finance Agent v2: 15.6 (#40/40)
- Vals Index: 30.04 (#40/40)
- ProofBench: 0.0 (#53/54)
- MedCode: 20.7 (#75/75)
- AIIQ Composite IQ: 85.0 (#107/115)
LongCat-Flash-Chat — ELO 1574
- VitaBench 2.0: 29.8 (#16/21)
LongCat-Flash-Thinking-2601 — ELO 1636
- ClawProBench: 57.48 (#24/57)
Muse Spark — ELO 1731
- Finance Agent v1.1: 60.59 (#4/53)
- LMArena Text Arena: 1487.95 (#11/24)
- MedCode: 51.31 (#11/75)
- MedScribe: 85.9 (#11/74)
- CaseLaw v2: 63.13 (#17/54)
- ProofBench: 17.0 (#30/54)
O1 Preview (2024-09-12) — ELO 1584
- TextClass Benchmark: 1622.24 (#31/108)
O1 Pro — ELO 1624
- USAMO25: 2.83 (#12/13)
Palmyra Fin — ELO 1566
- OpenEval - GPQA CoT: 42.15 (#28/74)
- OpenEval - Omni-MATH: 29.47 (#35/74)
- OpenEval - MMLU-Pro CoT: 59.1 (#38/77)
- OpenEval - IFEval Strict: 79.3 (#39/77)
Palmyra Med — ELO 1482
- OpenEval - GPQA CoT: 36.77 (#37/74)
- OpenEval - IFEval Strict: 76.74 (#46/77)
- OpenEval - MMLU-Pro CoT: 41.1 (#48/77)
- HELM Safety: 85.6767 (#49/59)
- OpenEval - Omni-MATH: 15.57 (#51/74)
Palmyra X5 — ELO 1618
- OpenEval - GPQA CoT: 66.14 (#9/74)
- OpenEval - MMLU-Pro CoT: 80.4 (#11/77)
- OpenEval - Omni-MATH: 41.45 (#16/74)
- OpenEval - IFEval Strict: 82.29 (#30/77)
Palmyra-X-004 — ELO 1577
- OpenEval - IFEval Strict: 87.25 (#9/77)
- OpenEval - Omni-MATH: 31.97 (#28/74)
- OpenEval - MMLU-Pro CoT: 65.7 (#31/77)
Pixtral-12B-2409 — ELO 1408
- TextClass Benchmark: 1490.38 (#67/108)
Qwen 3 Max (2026-01-23) — ELO 1663
- ClawProBench: 55.76 (#33/57)
- IOI: 13.75 (#34/64)
- CaseLaw v2: 54.98 (#36/54)
- MedScribe: 72.71 (#57/74)
- MedCode: 31.37 (#66/75)
Qwen 3.5 — ELO 1706
- APEX-Agents: 27.7 (#29/45)
Qwen 3.6 Max — ELO 1714
- AIIQ Composite IQ: 114.0 (#36/115)
Qwen 3.7 Flash — ELO 1633
- AI for Education Pedagogy - Science: 91.26 (#33/233)
- AI for Education Pedagogy - Secondary: 84.75 (#65/233)
- AI for Education Pedagogy - Primary: 88.73 (#68/233)
- AI for Education Pedagogy - Maths: 84.92 (#69/233)
- AI for Education Pedagogy - Social studies: 83.64 (#69/233)
- AI for Education Pedagogy: 84.76 (#71/233)
- AI for Education SEND: 77.52 (#88/225)
- AI for Education Pedagogy - Technology: 81.13 (#89/233)
- SvelteBench: 82.5 (#113/169)
Qwen 3.8 27B — ELO 1717
- LLM Stats (BabyVision): 85.6 (#2/10)
- LLM Stats (MathVision): 94.6 (#2/33)
- LLM Stats (OSWorld-Verified): 84.3 (#3/24)
- LLM Stats (CharXiv-R): 90.2 (#4/51)
- LLM Stats (OmniDocBench 1.5): 91.1 (#4/18)
- LLM Stats (Agents' Last Exam): 42.9 (#5/10)
- LLM Stats (RealWorldQA): 85.9 (#5/29)
- LLM Stats (ERQA): 65.5 (#7/24)
- LLM Stats (NL2Repo): 42.3 (#11/17)
- LLM Stats (Terminal-Bench 2.1): 73.0 (#16/28)
- ...and 5 more
Qwen3.8 2.4T A95B — ELO 1726
- Tau3 Banking: 49.07 (#3/166)
- AA GPQA Diamond: 93.54 (#8/575)
- AA GDPval: 1720.39 (#8/198)
- Artificial Analysis Intelligence Index: 57.7 (#11/574)
- AA CritPt: 20.0 (#22/479)
- AA Humanity's Last Exam: 42.45 (#25/567)
- SvelteBench: 96.7 (#27/169)
- AA Omniscience - Software Engineering (SWE) - Rust: 82.0 (#30/479)
- AA SciCode: 51.62 (#39/567)
- AA Long Context Reasoning: 75.33 (#44/500)
- ...and 22 more
Step 3.5 Flash 2603 — ELO 1602
- ClawProBench: 42.59 (#52/57)
Yi Large — ELO 1519
- TextClass Benchmark: 1473.21 (#74/108)
Yi-1.5-9B-Chat-16K — ELO 1455
- Fin-Bias: 83.4 (#18/19)
Yi-Coder-9B-Chat — ELO 1391
- FullStackBench en: 47.13 (#20/31)

Top-10 New Scores (180)

Claude Fable 5 on APEX-Agents-AA: 59.2 (#1)
Claude Fable 5 on ArxivMath: 87.5 (#2)
Claude Fable 5 on Computer Anthology Terminal Tasks (Claude Code): 54.0 (#1)
Claude Fable 5 on Computer Anthology Terminal Tasks (Terminus-2): 50.8 (#3)
Claude Fable 5 on Finance Agent v2: 56.31 (#4)
Claude Fable 5 on FrontierCode: 63.6 (#1)
Claude Fable 5 on IOI: 72.25 (#8)
Claude Fable 5 on LiveBench: 78.57 (#4)
Claude Fable 5 on MedCode: 56.07 (#3)
Claude Fable 5 on MedScribe: 88.52 (#3)
Claude Fable 5 on ProofBench: 77.0 (#3)
Claude Fable 5 on Roboflow Playground Classification: 1200.0 (#13)
Claude Fable 5 on Vals Index: 75.14 (#1)
Claude Fable 5 on Vals Multimodal Index: 74.15 (#1)
Claude Mythos 5 on OSWorld-Verified: 85.0 (#1)
Claude Opus 5 on AIIQ Composite IQ: 134.0 (#2)
Claude Opus 5 on Agent Security League - Functional Correctness: 73.7 (#9)
Claude Opus 5 on Arena AI Code: 1692.0 (#1)
Claude Opus 5 on Arena AI Document: 1520.0 (#1)
Claude Opus 5 on Benchmarks.bio - BioSecBench-Refusal: 36.3 (#9)
Claude Opus 5 on Computer Anthology Terminal Tasks (Claude Code): 54.0 (#2)
Claude Opus 5 on Computer Anthology Terminal Tasks (Terminus-2): 62.0 (#1)
Claude Opus 5 on Creative Writing (Lechmazur): 4.2 (#1)
Claude Opus 5 on Design Arena (SVG): 1361.0 (#1)
Claude Opus 5 on Finance Agent v2: 58.63 (#1)
Claude Opus 5 on IOI: 91.67 (#5)
Claude Opus 5 on LLM Chess (Saplin): 1285.4 (#8)
Claude Opus 5 on LMArena Text Arena: 1493.6 (#7)
Claude Opus 5 on LMArena WebDev Arena: 1691.77 (#1)
Claude Opus 5 on OpenRouter BrowseComp (Search): 89.0 (#1)
Claude Opus 5 on OpenRouter DeepSearchQA (Search): 76.5 (#1)
Claude Opus 5 on OpenRouter HLE (Search): 77.4 (#1)
Claude Opus 5 on ProgramBench: 4.5 (#1)
Claude Opus 5 on ProgramBench Almost: 37.0 (#1)
Claude Opus 5 on TrackingAI IQ Test (Vision): 88.24 (#4)
Claude Opus 5 on Vals AI (Vals Index): 67.21 (#1)
Claude Opus 5 on Vals AI Code Migration: 57.47 (#1)
Claude Opus 5 on Vals AI Excel Modeling: 73.56 (#2)
Claude Opus 5 on Vals AI LiveCodeBench: 89.03 (#2)
Claude Opus 5 on Vals AI MMLU-Pro: 91.59 (#1)
Claude Opus 5 on Vals Index: 74.82 (#2)
Claude Opus 5 on Vals Multimodal Index: 73.9 (#2)
GLM-5.3 on DeepsecBench: 22.0 (#6)
GLM-5.3 on FrontierSWE: 78.0 (#2)
GLM-5.3 on LLM Stats (Agents' Last Exam): 28.5 (#7)
GLM-5.3 on LLM Stats (DeepSWE 1.1): 66.9 (#8)
GLM-5.3 on LLM Stats (FrontierSWE): 78.1 (#3)
GLM-5.3 on LLM Stats (NL2Repo): 58.0 (#2)
GLM-5.3 on LLM Stats (Program Bench): 19.0 (#6)
GLM-5.3 on LLM Stats (Terminal-Bench 2.1): 88.2 (#3)
GLM-5.3 on LLM Stats (Toolathlon): 73.0 (#4)
GLM-5.3 on LLM Stats Score: 54.66 (#6)
GLM-5.3 on Vending-Bench 2: 8163.61 (#6)
GLM-5.3 on VoxelBench: 1806.0 (#12)
GPT-5.4 Pro on MathArena Apex: 69.79 (#3)
GPT-5.5 Pro on PM-LLM-Benchmark: 37.7 (#9)
GPT-5.6 Sol on AIIQ Composite IQ: 136.0 (#1)
GPT-5.6 Sol on APEX-Agents: 39.9 (#12)
GPT-5.6 Sol on APEX-Agents-AA: 56.7 (#3)
GPT-5.6 Sol on Android Bench: 90.8 (#3)
GPT-5.6 Sol on Arena AI Code: 1623.0 (#6)
GPT-5.6 Sol on Arena AI Document: 1479.0 (#9)
GPT-5.6 Sol on BabyVision: 88.9 (#3)
GPT-5.6 Sol on CADGenBench: 0.5319 (#2)
GPT-5.6 Sol on CharXiv-R: 84.6 (#12)
GPT-5.6 Sol on Computer Anthology Terminal Tasks (Codex CLI): 58.0 (#1)
GPT-5.6 Sol on Computer Anthology Terminal Tasks (Terminus-2): 51.6 (#2)
GPT-5.6 Sol on Design Arena (Website): 1337.0 (#3)
GPT-5.6 Sol on Finance Agent v2: 53.76 (#10)
GPT-5.6 Sol on FrontierCode: 60.6 (#3)
GPT-5.6 Sol on GDP.pdf: 30.7 (#1)
GPT-5.6 Sol on IOI: 86.67 (#6)
GPT-5.6 Sol on LLM Chess (Saplin): 1549.7 (#1)
GPT-5.6 Sol on LMArena Text Arena: 1480.8 (#18)
GPT-5.6 Sol on LMArena WebDev Arena: 1622.87 (#6)
GPT-5.6 Sol on MCP Atlas: 83.6 (#6)
GPT-5.6 Sol on MedCode: 43.97 (#26)
GPT-5.6 Sol on MedScribe: 85.23 (#15)
GPT-5.6 Sol on OmniDocBench 1.5: 85.8 (#41)
GPT-5.6 Sol on OpenRouter BrowseComp (Search): 82.4 (#2)
GPT-5.6 Sol on OpenRouter DeepSearchQA (Search): 75.0 (#2)
GPT-5.6 Sol on OpenRouter HLE (Search): 71.1 (#2)
GPT-5.6 Sol on PLawBench: 72.3 (#2)
GPT-5.6 Sol on ProgramBench: 1.0 (#2)
GPT-5.6 Sol on ProgramBench Almost: 15.5 (#3)
GPT-5.6 Sol on ProofBench: 77.0 (#2)
GPT-5.6 Sol on ProphetArena: 0.9506 (#4)
GPT-5.6 Sol on Toolathlon: 74.9 (#6)
GPT-5.6 Sol on Vals Index: 73.12 (#4)
GPT-5.6 Sol on Vals Multimodal Index: 72.19 (#4)
Gemini 3.7 Flash on AA CritPt: 14.29 (#37)
Gemini 3.7 Flash on AA GDPval: 1525.3 (#24)
Gemini 3.7 Flash on AA GPQA Diamond: 94.55 (#2)
Gemini 3.7 Flash on AA Humanity's Last Exam: 47.87 (#8)
Gemini 3.7 Flash on AA Long Context Reasoning: 81.0 (#4)
Gemini 3.7 Flash on AA MMMU-Pro: 85.49 (#1)
Gemini 3.7 Flash on AA Omniscience: 26.48 (#13)
Gemini 3.7 Flash on AA Omniscience - Business: 45.6 (#18)
Gemini 3.7 Flash on AA Omniscience - Health: 44.7 (#27)
Gemini 3.7 Flash on AA Omniscience - Humanities & Social Sciences: 55.2 (#11)
Gemini 3.7 Flash on AA Omniscience - Law: 62.1 (#3)
Gemini 3.7 Flash on AA Omniscience - Science, Engineering & Mathematics: 52.0 (#18)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE): 72.3 (#26)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - C: 90.0 (#10)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - Dart: 68.0 (#21)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - Go: 74.0 (#25)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - HTML: 80.0 (#23)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - Java: 57.0 (#24)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - JavaScript: 77.27 (#28)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - Julia: 64.0 (#30)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - Kotlin: 62.0 (#45)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - PHP: 78.0 (#25)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - Python: 71.5 (#37)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - R: 60.0 (#35)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - Rust: 82.0 (#26)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - Swift: 80.0 (#17)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - TypeScript: 76.67 (#27)
Gemini 3.7 Flash on AA SciCode: 57.87 (#5)
Gemini 3.7 Flash on AA-Briefcase: 36.36 (#17)
Gemini 3.7 Flash on AA-Omniscience Accuracy: 55.32 (#16)
Gemini 3.7 Flash on AI Chess Leaderboard (Continuation): 1753.0 (#3)
Gemini 3.7 Flash on AI Chess Leaderboard (Reasoning): 1854.0 (#2)
Gemini 3.7 Flash on Agent Arena: 3.61 (#31)
Gemini 3.7 Flash on Agent Arena - Bash Recovery: 2.33 (#42)
Gemini 3.7 Flash on Agent Arena - Confirmed Success: 9.85 (#11)
Gemini 3.7 Flash on Agent Arena - Praise vs Complaint: 1.84 (#44)
Gemini 3.7 Flash on Agent Arena - Steerability: 2.83 (#38)
Gemini 3.7 Flash on Agent Arena - Tool Hallucination: 1.18 (#26)
Gemini 3.7 Flash on Arabic Broad Leaderboard: 9.204 (#4)
Gemini 3.7 Flash on Artificial Analysis Intelligence Index: 56.03 (#17)
Gemini 3.7 Flash on AutomationBench: 30.4 (#1)
Gemini 3.7 Flash on BenchLM: 61.4 (#50)
Gemini 3.7 Flash on Chatbot Arena (Code): 1587.0 (#8)
Gemini 3.7 Flash on Chatbot Arena (Text): 1490.0 (#9)
Gemini 3.7 Flash on Chess Puzzles (Epoch AI): 47.0 (#11)
Gemini 3.7 Flash on ClockBench: 48.3 (#4)
Gemini 3.7 Flash on Coarena: 1039.4 (#2)
Gemini 3.7 Flash on Coarena - Task Completion: 78.1 (#4)
Gemini 3.7 Flash on Computer Anthology Terminal Tasks (Terminus-2): 46.0 (#5)
Gemini 3.7 Flash on Creative Writing (Lechmazur): -0.4 (#25)
Gemini 3.7 Flash on CursorBench 3.1: 61.6 (#15)
Gemini 3.7 Flash on Design Arena (Game Dev): 1310.0 (#16)
Gemini 3.7 Flash on Design Arena (Website): 1348.0 (#2)
Gemini 3.7 Flash on Epoch AI - Mystery Game Puzzles: 37.0 (#7)
Gemini 3.7 Flash on FrontierMath - Tier 4 (v2): 36.59 (#16)
Gemini 3.7 Flash on FrontierMath - Tiers 1-3 (v2): 71.58 (#14)
Gemini 3.7 Flash on Kaggle FACTS Grounding: 75.63 (#5)
Gemini 3.7 Flash on LLM Stats (Agents' Last Exam): 26.3 (#8)
Gemini 3.7 Flash on LLM Stats (Artificial Analysis): 56.0 (#2)
Gemini 3.7 Flash on LLM Stats (AutomationBench): 30.4 (#4)
Gemini 3.7 Flash on LLM Stats (CharXiv-R): 88.7 (#7)
Gemini 3.7 Flash on LLM Stats (DeepSWE 1.1): 65.3 (#10)
Gemini 3.7 Flash on LLM Stats (FrontierCode 1.1): 43.6 (#5)
Gemini 3.7 Flash on LLM Stats (GDP.pdf): 34.0 (#2)
Gemini 3.7 Flash on LLM Stats (Terminal-Bench 2.1): 85.8 (#7)
Gemini 3.7 Flash on LLM Stats Score: 51.2 (#12)
Gemini 3.7 Flash on LM Market Cap LMC Score: 40.0 (#239)
Gemini 3.7 Flash on MineBench: 1788.74 (#9)
Gemini 3.7 Flash on Multi-turn Debate (Lechmazur): 1476.3 (#27)
Gemini 3.7 Flash on OTIS Mock AIME 2024-25: 97.22 (#19)
Gemini 3.7 Flash on OpenRouter GPQA Diamond: 92.3 (#6)
Gemini 3.7 Flash on OpenRouter Tau2-Bench Airline: 80.6 (#3)
Gemini 3.7 Flash on ProfBench: 57.5 (#8)
Gemini 3.7 Flash on ProgramBench: 0.0 (#9)
Gemini 3.7 Flash on ProgramBench Almost: 5.5 (#6)
Gemini 3.7 Flash on RuneBench: 10203.0 (#2)
Gemini 3.7 Flash on SealedBench: 56.0 (#17)
Gemini 3.7 Flash on SimpleQA Verified: 71.2 (#4)
Gemini 3.7 Flash on SpeechMap Compliance: 85.8 (#60)
Gemini 3.7 Flash on SvelteBench: 100.0 (#8)
Gemini 3.7 Flash on Tau3 Banking: 35.46 (#23)
Gemini 3.7 Flash on Vals AI (Vals Index): 59.31 (#7)
Gemini 3.7 Flash on Vals AI Excel Modeling: 71.33 (#4)
Gemini 3.7 Flash on Vals AI Finance Agent v2: 59.04 (#2)
Gemini 3.7 Flash on Vals AI Harvey Legal Agent Bench: 8.75 (#9)
Gemini 3.7 Flash on Vals AI Legal Research Bench: 34.62 (#18)
Gemini 3.7 Flash on Vals AI ProofBench: 58.0 (#9)
Gemini 3.7 Flash on Vals AI Terminal-Bench 2.1: 77.53 (#7)
Gemini 3.7 Flash on Vals AI Vibe Code Bench: 70.39 (#15)
Gemini 3.7 Flash on VoxelBench: 1893.0 (#9)

New #1 Leaders (61)

Sycophancy (Lechmazur): GPT-5.6 Terra (Thinking, High) (0.0) beat Claude Fable 5 (Thinking, Medium) by 980990.0
RuneBench: Grok 4.6 (12757.0) beat Grok 4.5 by 4041.0
The Compliance Gap: Mistral Small 24B (100.0) beat Llama 3.3 70B by 90.0
Design Arena (Slides): playyy (1303.0) beat Claude 3 Opus by 66.0
OpenEval - Omni-MATH: Qwen 3 235B A22B 2507 Instruct (71.83) beat Gemma 3 27B (IT) by 31.82
OpenEval - GPQA CoT: Gemini 3 Pro (Preview) (80.27) beat qwen-3-80B-instruct by 23.77
ProgramBench Almost: Claude Opus 5 (xHigh) (37.0) beat Claude Opus 4.8 (xHigh) by 20.5
LLM Stats (APEX-Agents): Grok 4.6 (57.5) beat Kimi K3 by 19.9
Coarena - Task Completion: Qwen 3.8 Max (100.0) beat GPT-5.6 Terra by 18.6
CHI-Bench: Claude Opus 5 (54.7) beat Claude Opus 4.8 by 17.4
LLM Stats (AutomationBench): GLM-5.3 (48.2) beat Kimi K3 by 17.4
BridgeBench Arena: Claude Opus 4.7 (1021.5) beat Claude Fable 5 by 16.2
ASCIIBench: Claude Opus 4.5 (1685.0) beat Gemini 3 Pro (Preview) by 16.0
OpenRouter BrowseComp (Search): Claude Opus 5 (High) (89.0) beat GPT-5.6 Sol by 14.0
Chatbot Arena (Video Edit): minimax-h3 (1390.0) beat dreamina-seedance-2.0-720p by 13.0
OpenEval - MMLU-Pro CoT: Gemini 3 Pro (Preview) (90.3) beat qwen-3-80B-instruct by 12.47
APEX-Agents-AA: Claude Fable 5 (Max) (59.2) beat Gemini 3.5 Flash (High) by 12.1
Chatbot Arena (Image-to-Video): minimax-h3 (1489.0) beat dreamina-seedance-2.0-720p by 11.0
FrontierCode: Claude Fable 5 (Max) (63.6) beat Claude Opus 5 by 10.2
Arena AI Document: Claude Opus 5 (High) (1520.0) beat Claude Opus 4.6 by 10.0
Arena AI Code: Claude Opus 5 (Max) (1692.0) beat Kimi K3 by 10.0
MathArena Apex: DeepSeek V4 Pro (Max) (90.2) beat GPT-5.5 (xHigh) by 9.99
LMArena WebDev Arena: Claude Opus 5 (Max) (1691.77) beat Kimi K3 by 9.76
NatureBench: Claude Opus 5 (23.33) beat Claude Opus 4.7 by 7.77
BabyVision: Qwen3.8-Max (with tools) (91.3) beat GPT-5.5 by 7.7
Android Bench: Claude Opus 5 (91.8) beat Claude Fable 5 by 7.3
Design Arena (Game Dev): Kimi K3 (1432.0) beat Claude Opus 5 by 6.0
LLM Stats (NL2Repo): DeepSeek V4 Pro (0813) (61.5) beat Qwen 3.8 Max by 5.6
LLM Stats (MRCR v2 (8-needle)): Gemini 3.7 Flash (97.0) beat GPT-5.6 Sol by 5.5
OpenRouter HLE (Search): Claude Opus 5 (High) (77.4) beat Claude Opus 5 by 4.7
MedCode: Claude Opus 5 (63.57) beat Gemini 3.1 Pro (Preview) (High) by 4.51
AA-LCR: Muse Glimmer 30B (80.0) beat GPT-5.2 Codex (xHigh) by 4.3
AutomationBench: Gemini 3.7 Flash (High) (30.4) beat Claude Opus 5 (Max) by 4.2
PhysicsFinals: Gemini 1.5 Pro (63.9) beat O3 Mini by 4.0
LLM Stats (LVBench): Gemini 3.7 Flash (85.4) beat Qwen 3.8 Max by 3.6
Agent Security League - Security Correctness: Claude Opus 5 (32.4) beat Claude Fable 5 by 3.4
OpenEval - IFEval Strict: Claude 3.7 Sonnet (20250219) (Thinking 10K) (94.0) beat qwen-3-80B-instruct by 2.75
GameCraft-Bench: Claude Opus 5 (68.44) beat Claude Fable 5 by 2.72
SEAL Showdown: GPT-4o Audio (Preview 2025-06-03) (1269.2) beat Gemini 3 Pro (Preview) by 2.7
LLM Stats (PostTrainBench): GLM-5.3 (39.8) beat MiniMax-M3 by 2.7
Agent Arena - Praise vs Complaint: Claude Fable 5 (High) (25.14) beat GPT-5.6 Sol (xHigh) by 2.44
MedScribe: Claude Opus 5 (90.98) beat Muse Spark 1.1 (xHigh) by 2.09
LLM Stats (CyberGym): GLM-5.3 (84.5) beat Gemini 3.5 Flash Cyber by 1.3
LLM Chess (Saplin): GPT-5.6 Sol (xHigh) (1549.7) beat GPT-5.6 Sol (High) by 1.2
LLM Stats (WideSearch): Qwen 3.8 Max (81.9) beat Kimi K2.6 by 1.1
Computer Anthology Terminal Tasks (Gemini CLI): Gemini 3.5 Flash (High) (24.4) beat Gemini 3.5 Flash by 1.0
WritingBench: Qwen 3 235B A22B 2507 (Thinking) (88.3) beat Qwen 3 Next 80B A3B Instruct by 1.0
ComplexFuncBench: GPT-4o (66.5) beat GPT-4.1 by 1.0
ProofBench: Claude Opus 5 (78.0) beat Claude Fable 5 (Max) by 1.0
Creative Writing (Lechmazur): Claude Opus 5 (xHigh) (4.2) beat Claude Fable 5 (High) by 0.9
AA GPQA Diamond: Grok 4.6 (High) (94.95) beat GPT-5.6 Sol (Max) by 0.81
AA MMMU-Pro: Gemini 3.7 Flash (High) (85.49) beat Claude Opus 5 (Adaptive Reasoning, Max Effort) by 0.75
Finance Agent v2: Claude Opus 5 (Max) (58.63) beat Gemini 3.5 Flash by 0.73
SpatialGen-Bench (Text Answering): Qwen 3.8 Max (81.74) beat Claude Fable 5 by 0.48
CursorBench 3.1: Grok 4.6 (xHigh) (70.8) beat Claude Fable 5 (Max) by 0.3
GDPevo: Claude Opus 4.8 (50.63) beat Claude Opus 4.6 by 0.23
OpenEval - CNN/DailyMail: command-xlarge-beta (25.84) beat Llama 2 70B by 0.22
ForecastBench: rice-demon (69.1) beat captain-jack by 0.2
Computer Anthology Terminal Tasks (Terminus-2): Claude Opus 5 (High) (62.0) beat Claude Opus 5 by 0.2
CADGenBench: build123d-mcp-v0381-claude-opus-5-xhigh-full-r6 (0.6391) beat GPT-5.6 Sol (xHigh) by 0.11
CPTU Bench: Muse-Glimmer-30B (Reasoning) (4.42) beat Qwen 3.5 27B (Thinking) by 0.08

The Aggregate Digest — 2026-08-15

2026-08-15T05:12:04.017807+00:00

Frontier

Best available model: Gemini 3.7 Flash (1782) takes the crown from GPT-5.5 Pro (1776)

Daily

New Benchmarks (10)

Agents on Rails (Successful Runs (%)): leader Claude Opus 5 (92.1), 8 models
The Rails Foundation's agentic coding benchmark: 21 atomic Rails tasks against the Writebook app — bug reports, security findings and feature requests written the way they would actually be filed — run three times per model through one frozen minimal harness at each provider's default reasoning effort, and graded by hidden behavioural test suites; the score is the share of runs whose tests pass.
OpenAI GPT-5.6 System Card - HealthBench Professional (Length-Adjusted Score (0-100)): leader GPT-5.6 Sol (60.5), 8 models
HealthBench professional subset for medically challenging, expert-oriented healthcare questions, on the length-adjusted scale; OpenAI's own runs as published in the GPT-5.6 system card.
OpenAI GPT-5.6 System Card - HealthBench (Length-Adjusted Score (0-100)): leader GPT-5 (57.7), 8 models
HealthBench health conversations graded against physician-written rubrics, on the length-adjusted scale the card headlines; OpenAI's own runs as published in the GPT-5.6 system card.
OpenAI GPT-5.6 System Card - HealthBench Hard (Length-Adjusted Score (0-100)): leader GPT-5 (34.7), 8 models
Hard subset of HealthBench, the conversations frontier models still fail, on the length-adjusted scale; OpenAI's own runs as published in the GPT-5.6 system card.
OpenAI GPT-5.6 System Card - HealthBench Consensus (Length-Adjusted Score (0-100)): leader GPT-5.4 (96.3), 8 models
HealthBench Consensus subset — the rubric criteria physicians agreed on — on the length-adjusted scale; OpenAI's own runs as published in the GPT-5.6 system card.
OpenAI GPT-5.6 System Card - AAV Capsid Packaging Prediction (Spearman Correlation): leader GPT-5.6 Sol (0.529), 2 models
Biosecurity capability slice from OpenAI's GPT-5.6 system card — predicting AAV capsid packaging fitness, scored as Spearman correlation against measured values; OpenAI-run frontier comparison.
OpenAI GPT-5.6 System Card - Hard Negative Protein Binding Prediction (pass@4 (%)): leader GPT-5.6 Sol (7.6), 4 models
Biosecurity capability slice from OpenAI's GPT-5.6 system card — hard-negative protein-binding prediction; OpenAI-run frontier comparison.
OpenAI GPT-5.6 System Card - DNA Sequence Design for TF Binding (pass@1 (%)): leader GPT-5.5 Pro (16.5), 4 models
Biosecurity capability slice from OpenAI's GPT-5.6 system card — designing DNA sequences for transcription-factor binding; OpenAI-run frontier comparison.
LLM Stats (AndroidWorld) (Score (%)): leader Qwen 3.8 Max (85.3), 5 models
AndroidWorld as aggregated by LLM Stats — autonomous-agent tasks on a real Android emulator covering app interaction, navigation, and multi-step task completion.
LLM Stats (ExploitBench) (Score (%)): leader Claude Fable 5 (78.0), 5 models
ExploitBench as aggregated by LLM Stats — offensive-security agent tasks built from patched browser-engine vulnerabilities, graded on how far a model gets toward a working exploit.

New Models (2)

GLM-5.3 — ELO 1772
- LLM Stats (AutomationBench): 48.2 (#1/12)
- LLM Stats (CyberGym): 84.5 (#1/13)
- LLM Stats (PostTrainBench): 39.8 (#1/6)
- FrontierSWE: 78.0 (#2/17)
- LLM Stats (NL2Repo): 58.0 (#2/17)
- LLM Stats (FrontierSWE): 78.1 (#3/16)
- LLM Stats (Terminal-Bench 2.1): 88.2 (#3/28)
- LLM Stats (Toolathlon): 73.0 (#4/37)
- Vending-Bench 2: 8163.61 (#6/60)
- LLM Stats (Program Bench): 19.0 (#6/6)
- ...and 3 more
Qwen 3.8 27B — ELO 1717
- LLM Stats (BabyVision): 85.6 (#2/10)
- LLM Stats (MathVision): 94.6 (#2/33)
- LLM Stats (OSWorld-Verified): 84.3 (#3/24)
- LLM Stats (CharXiv-R): 90.2 (#4/51)
- LLM Stats (OmniDocBench 1.5): 91.1 (#4/18)
- LLM Stats (Agents' Last Exam): 42.9 (#5/10)
- LLM Stats (RealWorldQA): 85.9 (#5/29)
- LLM Stats (ERQA): 65.5 (#7/24)
- LLM Stats (NL2Repo): 42.3 (#11/17)
- Humanity's Last Exam (Self-Reported, No Tools): 30.8 (#14/50)
- ...and 4 more

Top-10 New Scores (32)

Claude Opus 5 on Benchmarks.bio - BioSecBench-Refusal: 36.3 (#9)
Claude Opus 5 on Creative Writing (Lechmazur): 4.2 (#1)
GLM-5.3 on FrontierSWE: 78.0 (#2)
GLM-5.3 on LLM Stats (Agents' Last Exam): 28.5 (#7)
GLM-5.3 on LLM Stats (DeepSWE 1.1): 66.9 (#8)
GLM-5.3 on LLM Stats (FrontierSWE): 78.1 (#3)
GLM-5.3 on LLM Stats (NL2Repo): 58.0 (#2)
GLM-5.3 on LLM Stats (Program Bench): 19.0 (#6)
GLM-5.3 on LLM Stats (Terminal-Bench 2.1): 88.2 (#3)
GLM-5.3 on LLM Stats (Toolathlon): 73.0 (#4)
GLM-5.3 on Vending-Bench 2: 8163.61 (#6)
GLM-5.3 on VoxelBench: 1773.0 (#13)
Gemini 3.7 Flash on BenchLM: 61.4 (#50)
Gemini 3.7 Flash on Computer Anthology Terminal Tasks (Terminus-2): 46.0 (#5)
Gemini 3.7 Flash on Creative Writing (Lechmazur): -0.4 (#25)
Gemini 3.7 Flash on Kaggle FACTS Grounding: 75.63 (#5)
Gemini 3.7 Flash on OpenRouter GPQA Diamond: 92.3 (#7)
Gemini 3.7 Flash on OpenRouter Tau2-Bench Airline: 80.6 (#3)
Gemini 3.7 Flash on ProfBench: 57.5 (#8)
Gemini 3.7 Flash on SealedBench: 56.0 (#17)
Gemini 3.7 Flash on SvelteBench: 100.0 (#8)
Gemini 3.7 Flash on Vals AI (Vals Index): 59.31 (#7)
Gemini 3.7 Flash on Vals AI Excel Modeling: 71.33 (#4)
Gemini 3.7 Flash on Vals AI Finance Agent v2: 59.04 (#2)
Gemini 3.7 Flash on Vals AI Harvey Legal Agent Bench: 8.75 (#9)
Gemini 3.7 Flash on Vals AI Legal Research Bench: 34.62 (#18)
Gemini 3.7 Flash on Vals AI ProofBench: 58.0 (#9)
Gemini 3.7 Flash on Vals AI Terminal-Bench 2.1: 77.53 (#7)
Gemini 3.7 Flash on Vals AI Vibe Code Bench: 70.39 (#15)
Kimi K3 on CHI-Bench: 25.3 (#6)
Kimi K3 on Design Arena (SVG): 1340.0 (#4)
Kimi K3 on ProfBench: 57.9 (#7)

New #1 Leaders (6)

Design Arena (Game Dev): Kimi K3 (1436.0) beat Claude Opus 5 by 33.0
CHI-Bench: Claude Opus 5 (54.7) beat Claude Opus 4.8 by 17.4
LLM Stats (AutomationBench): GLM-5.3 (48.2) beat DeepSeek V4 Pro (0813) by 16.4
LLM Stats (PostTrainBench): GLM-5.3 (39.8) beat MiniMax-M3 by 2.7
LLM Stats (CyberGym): GLM-5.3 (84.5) beat DeepSeek V4 Pro (0813) by 1.2
Creative Writing (Lechmazur): Claude Opus 5 (xHigh) (4.2) beat Claude Fable 5 (High) by 0.9

The Aggregate Digest — 2026-08-14

2026-08-14T06:22:09.151361+00:00

Daily

New Benchmarks (1)

LLM Stats (OSWorld 2.0) (Score (%)): leader Claude Opus 5 (70.6), 5 models
llm-stats.com’s mirror of OSWorld 2.0, the computer-use agent benchmark on real desktop applications. Its roster is currently disjoint from the first-party OSWorld 2.0 board — newer models only — and its scale matches the partial-credit column rather than the binary one.

New Models (1)

Gemini 3.7 Flash — ELO 1780, #3
- AA MMMU-Pro: 85.49 (#1/240)
- AutomationBench: 30.4 (#1/10)
- LLM Stats (LVBench): 85.4 (#1/25)
- LLM Stats (MRCR v2 (8-needle)): 97.0 (#1/23)
- Coarena - Task Completion: 80.0 (#2/12)
- AA GPQA Diamond: 94.55 (#2/575)
- LLM Stats (Artificial Analysis): 56.0 (#2/7)
- LLM Stats (GDP.pdf): 34.0 (#2/6)
- RuneBench: 10203.0 (#2/46)
- AI Chess Leaderboard (Reasoning): 1855.0 (#2/313)
- ...and 55 more

Top-10 New Scores (73)

Claude Opus 5 on LLM Chess (Saplin): 1285.4 (#8)
Claude Opus 5 on ProgramBench: 4.5 (#1)
Claude Opus 5 on ProgramBench Almost: 37.0 (#1)
Claude Opus 5 on Vals AI (Vals Index): 67.21 (#1)
Claude Opus 5 on Vals AI Excel Modeling: 73.56 (#2)
GPT-5.6 Sol on Android Bench: 90.8 (#3)
GPT-5.6 Sol on ProgramBench: 0.5 (#6)
GPT-5.6 Sol on ProgramBench Almost: 2.5 (#11)
Gemini 3.7 Flash on AA CritPt: 14.29 (#37)
Gemini 3.7 Flash on AA GDPval: 1525.3 (#24)
Gemini 3.7 Flash on AA GPQA Diamond: 94.55 (#2)
Gemini 3.7 Flash on AA Humanity's Last Exam: 47.87 (#8)
Gemini 3.7 Flash on AA Long Context Reasoning: 81.0 (#4)
Gemini 3.7 Flash on AA MMMU-Pro: 85.49 (#1)
Gemini 3.7 Flash on AA Omniscience: 26.48 (#13)
Gemini 3.7 Flash on AA Omniscience - Business: 45.6 (#18)
Gemini 3.7 Flash on AA Omniscience - Health: 44.7 (#27)
Gemini 3.7 Flash on AA Omniscience - Humanities & Social Sciences: 55.2 (#11)
Gemini 3.7 Flash on AA Omniscience - Law: 62.1 (#3)
Gemini 3.7 Flash on AA Omniscience - Science, Engineering & Mathematics: 52.0 (#18)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE): 72.3 (#26)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - C: 90.0 (#10)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - Dart: 68.0 (#21)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - Go: 74.0 (#25)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - HTML: 80.0 (#22)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - Java: 57.0 (#23)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - JavaScript: 77.27 (#28)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - Julia: 64.0 (#30)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - Kotlin: 62.0 (#45)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - PHP: 78.0 (#25)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - Python: 71.5 (#36)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - R: 60.0 (#35)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - Rust: 82.0 (#25)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - Swift: 80.0 (#17)
Gemini 3.7 Flash on AA Omniscience - Software Engineering (SWE) - TypeScript: 76.67 (#27)
Gemini 3.7 Flash on AA SciCode: 57.87 (#5)
Gemini 3.7 Flash on AA-Briefcase: 36.36 (#17)
Gemini 3.7 Flash on AA-Omniscience Accuracy: 55.32 (#16)
Gemini 3.7 Flash on AI Chess Leaderboard (Continuation): 1757.0 (#4)
Gemini 3.7 Flash on AI Chess Leaderboard (Reasoning): 1855.0 (#2)
Gemini 3.7 Flash on Agent Arena: 3.61 (#31)
Gemini 3.7 Flash on Agent Arena - Bash Recovery: 2.33 (#42)
Gemini 3.7 Flash on Agent Arena - Confirmed Success: 9.85 (#11)
Gemini 3.7 Flash on Agent Arena - Praise vs Complaint: 1.84 (#44)
Gemini 3.7 Flash on Agent Arena - Steerability: 2.83 (#38)
Gemini 3.7 Flash on Agent Arena - Tool Hallucination: 1.18 (#26)
Gemini 3.7 Flash on Artificial Analysis Intelligence Index: 56.03 (#17)
Gemini 3.7 Flash on AutomationBench: 30.4 (#1)
Gemini 3.7 Flash on Chatbot Arena (Code): 1588.0 (#9)
Gemini 3.7 Flash on Chatbot Arena (Text): 1490.0 (#9)
Gemini 3.7 Flash on ClockBench: 48.3 (#4)
Gemini 3.7 Flash on Coarena: 1002.3 (#8)
Gemini 3.7 Flash on Coarena - Task Completion: 80.0 (#2)
Gemini 3.7 Flash on Conceptual Reasoning Index: 64.56 (#11)
Gemini 3.7 Flash on Conceptual Reasoning Index - Argument Evaluation (LMCA): 50.38 (#14)
Gemini 3.7 Flash on Conceptual Reasoning Index - Consistency (ACCoRD): 76.98 (#13)
Gemini 3.7 Flash on Conceptual Reasoning Index - Decision Theory (DTBench): 94.67 (#4)
Gemini 3.7 Flash on LLM Stats (Agents' Last Exam): 26.3 (#6)
Gemini 3.7 Flash on LLM Stats (Artificial Analysis): 56.0 (#2)
Gemini 3.7 Flash on LLM Stats (AutomationBench): 30.4 (#3)
Gemini 3.7 Flash on LLM Stats (CharXiv-R): 88.7 (#6)
Gemini 3.7 Flash on LLM Stats (DeepSWE 1.1): 65.3 (#9)
Gemini 3.7 Flash on LLM Stats (FrontierCode 1.1): 43.6 (#5)
Gemini 3.7 Flash on LLM Stats (GDP.pdf): 34.0 (#2)
Gemini 3.7 Flash on LLM Stats (Terminal-Bench 2.1): 85.8 (#6)
Gemini 3.7 Flash on LM Market Cap LMC Score: 40.0 (#238)
Gemini 3.7 Flash on MineBench: 1786.21 (#8)
Gemini 3.7 Flash on RuneBench: 10203.0 (#2)
Gemini 3.7 Flash on SpeechMap Compliance: 85.8 (#60)
Gemini 3.7 Flash on Tau3 Banking: 35.46 (#23)
Gemini 3.7 Flash on VoxelBench: 1892.0 (#10)
Kimi K3 on Android Bench: 90.2 (#4)
Kimi K3 on Vals AI Excel Modeling: 66.4 (#7)

New #1 Leaders (17)

OpenEval - Omni-MATH: Qwen 3 235B A22B 2507 Instruct (71.83) beat Gemma 3 27B (IT) by 31.82
OpenEval - GPQA CoT: Gemini 3 Pro (Preview) (80.27) beat qwen-3-80B-instruct by 23.77
ProgramBench Almost: Claude Opus 5 (xHigh) (37.0) beat Claude Opus 4.8 (xHigh) by 20.5
Chatbot Arena (Video Edit): minimax-h3 (1390.0) beat dreamina-seedance-2.0-720p by 13.0
OpenEval - MMLU-Pro CoT: Gemini 3 Pro (Preview) (90.3) beat qwen-3-80B-instruct by 12.47
NatureBench: Claude Opus 5 (23.33) beat Claude Opus 4.7 by 7.77
Android Bench: Claude Opus 5 (91.8) beat Claude Fable 5 by 7.3
LLM Stats (NL2Repo): DeepSeek V4 Pro (0813) (61.5) beat Qwen 3.8 Max by 5.6
LLM Stats (MRCR v2 (8-needle)): Gemini 3.7 Flash (97.0) beat GPT-5.6 Sol by 5.5
LLM Stats (LVBench): Gemini 3.7 Flash (85.4) beat Qwen 3.8 Max by 3.6
AutomationBench: Gemini 3.7 Flash (High) (30.4) beat Claude Opus 5 (Max) by 3.5
OpenEval - IFEval Strict: Claude 3.7 Sonnet (20250219) (Thinking 10K) (94.0) beat qwen-3-80B-instruct by 2.75
LLM Stats (WideSearch): Qwen 3.8 Max (81.9) beat Kimi K2.6 by 1.1
LLM Stats (AutomationBench): DeepSeek V4 Pro (0813) (31.8) beat Kimi K3 by 1.0
AA MMMU-Pro: Gemini 3.7 Flash (High) (85.49) beat Claude Opus 5 (Adaptive Reasoning, Max Effort) by 0.75
OpenEval - CNN/DailyMail: command-xlarge-beta (25.84) beat Llama 2 70B by 0.22
LLM Stats (CyberGym): DeepSeek V4 Pro (0813) (83.3) beat Gemini 3.5 Flash Cyber by 0.1

The Aggregate Digest — 2026-08-13

2026-08-13T06:35:25.130271+00:00

Daily

New Benchmarks (4)

Conceptual Reasoning Index (Chance-Corrected Score (0-100)): leader Claude Opus 5 (73.57), 134 models
Redwood Research index for topics that cannot be graded against an answer key — AI alignment, collective action under transformative AI: a 60/20/20 blend of argument evaluation, belief consistency and decision theory, scaled so 0 is random guessing and 100 the maximum attainable.
Conceptual Reasoning Index - Argument Evaluation (LMCA) (Chance-Corrected Score (0-100)): leader Claude Opus 5 (63.3), 135 models
The 60% component of the CRI — models rate the strength of conceptual critiques of stated positions over 1,461 rated critiques, scored as the Pearson correlation with expert consensus ratings times 100; the estimated human ceiling is about 85, not 100.
Conceptual Reasoning Index - Consistency (ACCoRD) (Chance-Corrected Score (0-100)): leader Claude Fable 5 (84.95), 134 models
The consistency component of the CRI — models give probabilities for related claims, and 567 logical constraints such as P(A) = P(A|B)P(B) + P(A|¬B)P(¬B) measure how far the stated beliefs drift under reframing; 0-100, higher is more coherent.
Conceptual Reasoning Index - Decision Theory (DTBench) (Chance-Corrected Score (0-100)): leader Claude Fable 5 (97.33), 191 models
The decision-theory component of the CRI — 407 expert-verified Newcomb-like problems where the model reasons about interacting with near-copies of itself; accuracy chance-corrected against the 40% random baseline onto a 0-100 scale.

New Models (3)

Grok 4.6 — ELO 1745, #12
- RuneBench: 12757.0 (#1/44)
- LLM Stats (APEX-Agents): 57.5 (#1/8)
- Mercor APEX: 41.2 (#6/53)
- Vals AI Code Migration: 44.57 (#6/42)
- DeepSWE: 67.5 (#7/27)
- LLM Stats (DeepSWE 1.1): 65.9 (#8/21)
- LLM Stats Score: 45.3 (#21/333)
- LM Market Cap LMC Score: 88.8 (#44/416)
- BenchLM: 62.5 (#45/200)
- SpeechMap Compliance: 83.6 (#74/366)
DeepSeek V4 Pro (0813) — ELO 1724, #20
- NL2Repo: 61.5 (#2/21)
- CyberGym: 83.3 (#2/16)
- Toolathlon: 74.1 (#8/34)
- BenchLM: 60.9 (#48/200)
- LM Market Cap LMC Score: 86.7 (#62/416)
- SpeechMap Compliance: 58.3 (#211/366)
GLM-4.1V-9B — ELO 1446, #504
- Math-VR: 29.0 (#16/31)

Top-10 New Scores (21)

Claude Fable 5 on PLawBench: 70.2 (#3)
Claude Fable 5 on Roboflow Playground Classification: 1200.0 (#13)
Claude Fable 5 on Toolathlon: 77.9 (#2)
Claude Opus 5 on AIIQ Composite IQ: 134.0 (#2)
Claude Opus 5 on MedCode: 63.57 (#1)
Claude Opus 5 on MedScribe: 90.98 (#1)
Claude Opus 5 on ProofBench: 78.0 (#1)
Claude Opus 5 on Vals Index: 74.82 (#2)
Claude Opus 5 on Vals Multimodal Index: 73.9 (#2)
GPT-5.4 Pro on MathArena Apex: 69.79 (#3)
GPT-5.6 Sol on AIIQ Composite IQ: 136.0 (#1)
Kimi K3 on AA-LCR: 74.7 (#5)
Kimi K3 on AIIQ Composite IQ: 122.0 (#19)
Kimi K3 on AutomationBench: 22.7 (#4)
Kimi K3 on CritPt: 23.4 (#9)
Kimi K3 on Finance Agent v2: 54.36 (#7)
Kimi K3 on MedCode: 48.88 (#18)
Kimi K3 on MedScribe: 87.96 (#5)
Kimi K3 on ProofBench: 70.0 (#6)
Kimi K3 on Vals Index: 74.7 (#3)
Kimi K3 on Vals Multimodal Index: 73.42 (#3)

New #1 Leaders (10)

RuneBench: Grok 4.6 (12757.0) beat Grok 4.5 by 4041.0
LLM Stats (APEX-Agents): Grok 4.6 (57.5) beat Kimi K3 by 19.9
Design Arena (SVG): Claude Opus 5 (1381.0) beat GPT-5.6 Sol by 17.0
APEX-Agents-AA: Claude Fable 5 (Max) (59.2) beat Gemini 3.5 Flash (High) by 12.1
FrontierCode: Claude Fable 5 (Max) (63.6) beat Claude Opus 5 by 10.2
PLawBench: Qwen 3.8 Max (73.2) beat GPT-5.2 by 3.53
GameCraft-Bench: Claude Opus 5 (68.44) beat Claude Fable 5 by 2.72
LLM Chess (Saplin): GPT-5.6 Sol (xHigh) (1549.7) beat GPT-5.6 Sol (High) by 1.2
AA GPQA Diamond: Grok 4.6 (High) (94.95) beat GPT-5.6 Sol (Max) by 0.81
CursorBench 3.1: Grok 4.6 Extra High (70.8) beat Claude Fable 5 (Max) by 0.3

The Aggregate Digest — 2026-08-12

2026-08-12T05:29:49.784589+00:00

Daily

New Benchmarks (4)

Chartography (Score (%)): leader GPT-5.6 Sol (Max) (45.0), 40 models
Surge AI professional chart-understanding benchmark covering Kaplan-Meier curves, candlestick charts, contour maps, Sankey diagrams, Bode plots, and other specialist graphics, scoring visual perception, domain-aware interpretation, and multi-step graphical reasoning.
HANDBOOK.md Agents (Score (%)): leader Claude Fable 5 (Adaptive/Max) (36.2), 37 models
Surge AI long-context agentic instruction-following benchmark in which an agent must apply a 100-page company handbook inside a unique RL environment with internal tools and external MCP servers, across five enterprise domains.
ComplexConstraints (Score (%)): leader GPT-5.6 Sol (Max) (50.5), 49 models
Surge AI enterprise instruction-following benchmark whose constraints depend on one another, fire conditionally, and must be inferred from context rather than stated outright.
EnterpriseBench (CoreCraft Agents) (Score (%)): leader Claude Fable 5 (Adaptive/Max) (70.3), 3 models
Surge AI agentic benchmark set in CoreCraft, a large-scale simulated startup, measuring whether agents complete realistic enterprise tasks outside small self-contained environments.

New Models (10)

Claude Haiku 4.5 20251001 ThinkingAnthropic — ELO 1602
- MedScribe: 85.23 (#16/74)
- MGSM: 92.15 (#22/74)
- Vals Multimodal Index: 42.88 (#26/29)
- CaseLaw v2: 56.48 (#30/54)
- Finance Agent v1.1: 46.93 (#34/53)
- Finance Agent v2: 31.01 (#35/40)
- Vals Index: 40.9 (#35/40)
- IOI: 6.17 (#46/64)
- MedCode: 32.68 (#61/75)
Claude Sonnet 4.5 20250929 ThinkingAnthropic — ELO 1682
- MGSM: 94.33 (#4/74)
- Finance Agent v1.1: 54.5 (#18/53)
- CaseLaw v2: 62.16 (#19/54)
- MedScribe: 84.1 (#20/74)
- MedCode: 44.13 (#25/75)
- IOI: 18.33 (#27/64)
- ProofBench: 19.0 (#28/54)
DeepSeek V4 Promax — ELO 1707
- Finance Agent v1.1: 60.39 (#5/53)
- IOI: 35.83 (#17/64)
- Vals Index: 55.62 (#19/40)
- Finance Agent v2: 44.08 (#24/40)
- CaseLaw v2: 59.38 (#27/54)
- MedCode: 40.45 (#40/75)
- ProofBench: 10.0 (#40/54)
- MedScribe: 75.14 (#51/74)
GLGLM 4.7Zhipu AI — ELO 1605
- CaseLaw v2: 54.88 (#37/54)
- Finance Agent v1.1: 45.98 (#38/53)
- IOI: 7.58 (#41/64)
- ProofBench: 6.0 (#45/54)
- MGSM: 88.18 (#53/74)
- MedCode: 32.77 (#60/75)
- MedScribe: 68.63 (#68/74)
Gemini 2.5 ProGoogle — ELO 1665
- LingOly-TOO: 42.3539 (#4/16)
- USAMO25: 24.4 (#5/13)
- MedCode: 50.59 (#12/75)
- CaseLaw v2: 63.88 (#15/54)
- IOI: 17.08 (#29/64)
- Finance Agent v1.1: 41.59 (#42/53)
- HMMT 2025: 82.5 (#43/82)
- MathArena Apex: 0.52 (#45/50)
- MedScribe: 73.55 (#54/74)
Gemini 3 Flash PreviewhighGoogle — ELO 1689
- MedCode: 55.92 (#4/75)
- MGSM: 93.31 (#10/74)
- IOI: 39.08 (#15/64)
- Vals Multimodal Index: 52.19 (#22/29)
- Finance Agent v2: 42.55 (#26/40)
- Vals Index: 49.55 (#28/40)
- CaseLaw v2: 55.84 (#33/54)
- Finance Agent v1.1: 47.6 (#33/53)
- ProofBench: 15.0 (#34/54)
- MedScribe: 69.92 (#65/74)
Gemini 3 Pro PreviewhighGoogle — ELO 1708
- MATH 500: 96.4 (#1/60)
- MGSM: 93.93 (#7/74)
- MedCode: 52.2 (#10/75)
- Finance Agent v1.1: 55.15 (#16/53)
- IOI: 38.83 (#16/64)
- ProofBench: 20.0 (#26/54)
- CaseLaw v2: 53.05 (#42/54)
- MedScribe: 72.04 (#61/74)
KIkimi-k2.5 — ELO 1689
- Hack-Verifiable TextArena: 19.0 (#6/12)
- TutorBench: 54.56 (#6/27)
- MultiNRC: 35.17 (#20/43)
- EnigmaEval: 3.38 (#25/43)
- Visual-Language Understanding: 41.86 (#36/58)
- ALE-Bench: 821.65 (#36/89)
- AIIQ Composite IQ: 111.0 (#43/115)
KIkimi-k2.6 — ELO 1714
- Personalized Turn-Level User Conversation Sati: 4.74 (#1/7)
- RankJudge: 1713.0 (#4/21)
- MathConstraint: 35.9 (#8/12)
- From Knowing to Doing: -24.23 (#10/10)
- Arena AI Document: 1451.0 (#21/38)
- ALE-Bench: 1092.67 (#21/89)
- AIIQ Composite IQ: 119.0 (#22/115)
- Arena AI Code: 1509.0 (#27/48)
Muse Glimmer-30B — ELO 1663
- LLM Stats (AIME 2026): 94.7 (#5/18)
- LLM Stats (ScreenSpot Pro): 75.4 (#6/25)
- LLM Stats (SkillsBench): 44.3 (#7/8)
- LLM Stats (DeepSearchQA): 74.6 (#9/9)
- LLM Stats (MCP Atlas): 75.5 (#12/32)
- LLM Stats (OmniDocBench 1.5): 75.8 (#13/17)
- LLM Stats (OSWorld-Verified): 65.9 (#17/23)
- LLM Stats (Terminal-Bench 2.1): 51.7 (#19/19)
- LLM Stats (CharXiv-R): 78.8 (#22/48)
- LLM Stats Score: 35.64 (#65/326)

Top-10 New Scores (32)

Claude Fable 5 on ArxivMath: 87.5 (#2)
GPT-5.4 Pro on MathArena Apex: 69.79 (#3)
GPT-5.5 Pro on PM-LLM-Benchmark: 37.7 (#9)
GPT-5.6 on AIIQ Composite IQ: 136.0 (#1)
GPT-5.6 on AIIQ Composite IQ: 136.0 (#1)
GPT-5.6 on AIIQ Composite IQ: 136.0 (#1)
GPT-5.6 on APEX-Agents: 39.9 (#12)
GPT-5.6 on CharXiv-R: 84.6 (#12)
GPT-5.6 on Finance Agent v2: 55.04 (#6)
GPT-5.6 on Finance Agent v2: 55.04 (#6)
GPT-5.6 on Finance Agent v2: 55.04 (#6)
GPT-5.6 on IOI: 86.67 (#6)
GPT-5.6 on IOI: 86.67 (#6)
GPT-5.6 on IOI: 86.67 (#6)
GPT-5.6 on MCP Atlas: 83.6 (#6)
GPT-5.6 on MedCode: 43.97 (#26)
GPT-5.6 on MedCode: 43.97 (#26)
GPT-5.6 on MedCode: 43.97 (#26)
GPT-5.6 on MedScribe: 85.23 (#15)
GPT-5.6 on MedScribe: 85.23 (#15)
GPT-5.6 on MedScribe: 85.23 (#15)
GPT-5.6 on OmniDocBench 1.5: 85.8 (#41)
GPT-5.6 on ProofBench: 77.0 (#2)
GPT-5.6 on ProofBench: 77.0 (#2)
GPT-5.6 on ProofBench: 77.0 (#2)
GPT-5.6 on Toolathlon: 74.9 (#6)
GPT-5.6 on Vals Index: 73.12 (#4)
GPT-5.6 on Vals Index: 73.12 (#4)
GPT-5.6 on Vals Index: 73.12 (#4)
GPT-5.6 on Vals Multimodal Index: 72.19 (#4)
GPT-5.6 on Vals Multimodal Index: 72.19 (#4)
GPT-5.6 on Vals Multimodal Index: 72.19 (#4)

New #1 Leaders (10)

OpenRouter BrowseComp (Search): Claude Opus 5 · high (89.0) beat GPT-5.6 Sol by 14.0
Arena AI Code: Claude Opus 5 (Max) (1692.0) beat Kimi K3 by 10.0
LMArena WebDev Arena: Claude Opus 5 (Max) (1691.77) beat Kimi K3 by 9.76
OpenRouter HLE (Search): Claude Opus 5 · high (77.4) beat Claude Opus 5 by 4.7
AA-LCR: Muse Glimmer 30B (80.0) beat GPT-5.2 Codex (xHigh) by 4.3
Computer Anthology Terminal Tasks (Gemini CLI): Gemini 3.5 Flash (High) (24.4) beat Gemini 3.5 Flash by 1.0
ComplexFuncBench: GPT-4o (66.5) beat GPT-4.1 by 1.0
ProgramBench: GPT-5.6 Sol (xHigh) (1.0) beat GPT-5.5 (xHigh) by 0.5
Computer Anthology Terminal Tasks (Terminus-2): Claude Opus 5 (High) (62.0) beat Claude Opus 5 by 0.2
CPTU Bench: Muse-Glimmer-30B reasoning (API, FP8) (4.42) beat Qwen3.5-27B thinking (API) by 0.08

The Aggregate Digest — 2026-08-11

2026-08-11T05:10:50.566795+00:00

Daily

New Benchmarks (1)

Humanity's Last Exam (Self-Reported, No Tools) (Accuracy (%)): leader Kimi K3 (56.0), 44 models
PhD-level questions at the frontier of human expert knowledge. Lab-submitted scores from the official cais/hle dataset board on Hugging Face (model-card numbers, none independently verified), No Tools configuration; mostly open-weights models.

Top-10 New Scores (6)

Claude Fable 5 on GDP.pdf: 34.2 (#9)
Claude Fable 5 on Riemann-bench: 34.2 (#6)
Claude Opus 5 on GDP.pdf: 29.6 (#20)
Claude Opus 5 on Riemann-bench: 29.6 (#15)
Claude Opus 5 on TrackingAI IQ Test (Vision): 88.24 (#4)
GPT-5.6 Sol on Riemann-bench: 39.5 (#2)

New #1 Leaders (1)

Design Arena (Slides): playyy (1310.0) beat manus by 67.0