BEAKER - Affordance Grounding: leaderboard
Offline multimodal chemical-laboratory execution QA, with 1,000 expert-authored items across eleven tasks. Published task score (0–100). BEAKER v1, single-turn standardized QA; original-resolution images; short/mid videos 8/32 frames, long videos 128 frames in32grids; temperature0.7 when configurable. MCQ accuracy, affordance IoU, exp(-mean discrete Frechet distance/0.40) trajectory score, expert-semantic fill-in correctness, each scaled0–100. Overall uses expert weights0.25/0.15/0.35/0.25; experimentation image/video weights2:5. Source checkpoint/effort ambiguity is retained.
Metric: Published task score (0–100). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 28 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 4.7 (Benchmark Sweep Run: source=BEAKER; single-turn; temperature=0.7 when configurable; image=original; video=8/32/128frames; checkpoint and effort as published) | 34.5 | |
| 2 | GPT-5.5 (Benchmark Sweep Run: source=BEAKER; single-turn; temperature=0.7 when configurable; image=original; video=8/32/128frames; checkpoint and effort as published) | 32.7 | |
| 3 | RynnBrain-CoP-8B (Benchmark Sweep Run: source=BEAKER; single-turn; temperature=0.7 when configurable; image=original; video=8/32/128frames; checkpoint and effort as published) | 28.6 | |
| 4 | GPT-5.4 (Benchmark Sweep Run: source=BEAKER; single-turn; temperature=0.7 when configurable; image=original; video=8/32/128frames; checkpoint and effort as published) | 26.8 | |
| 5 | Claude Sonnet 4.6 (Benchmark Sweep Run: source=BEAKER; single-turn; temperature=0.7 when configurable; image=original; video=8/32/128frames; checkpoint and effort as published) | 18.3 | |
| 6 | MiMo-V2.5 (Benchmark Sweep Run: source=BEAKER; single-turn; temperature=0.7 when configurable; image=original; video=8/32/128frames; checkpoint and effort as published) | 11.3 | |
| 7 | Pelican-VL-1.0-7B (Benchmark Sweep Run: source=BEAKER; single-turn; temperature=0.7 when configurable; image=original; video=8/32/128frames; checkpoint and effort as published) | 10.9 | |
| 8 | MiMo-Embodied-7B (Benchmark Sweep Run: source=BEAKER; single-turn; temperature=0.7 when configurable; image=original; video=8/32/128frames; checkpoint and effort as published) | 9.8 | |
| 9 | Grok 4.20 (Reasoning) (Benchmark Sweep Run: source=BEAKER; single-turn; temperature=0.7 when configurable; image=original; video=8/32/128frames; checkpoint and effort as published) | 9.7 | |
| 10 | MiMo-V2-Omni (Benchmark Sweep Run: source=BEAKER; single-turn; temperature=0.7 when configurable; image=original; video=8/32/128frames; checkpoint and effort as published) | 9.2 |
Interactive version: theaggregate.ai/benchmark?slug=beaker-affordance-grounding · How It Works · Data refreshed daily, snapshot 2026-11-06.