V-ICAL Bench: leaderboard
V-ICAL Bench (Fan et al., 2026) tests video in-context learning for multimodal agents: in 342 tasks across 37 game and control environments from the Gym ecosystem (MiniGrid, Atari, Procgen, classic control, HighwayEnv and others), the agent watches one to four human demonstration videos, then plays from a fixed start state one action per turn from raw frames, with rewards hidden. The score is a normalized 0-100 progress score per task against expert-calibrated targets, averaged within each environment and then equally across environments, higher is better. Each model runs once per task through its provider's multimodal API, so provider context limits count against it.
Metric: Normalized score (0-100; environment-weighted). Source: arxiv.org. Saturation forecast: Around February 2027. 19 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Seed 2.1 Pro | 54.4 | #133 |
| 2 | Gemini 3.6 Flash | 49.5 | #72 |
| 3 | Gemini 3.1 Pro (Preview) | 48 | #91 |
| 4 | GPT-5.6 Sol | 46.7 | #29 |
| 5 | Gemini 3 Flash | 39.6 | #159 |
| 6 | Seed 2.0 Lite | 37 | #286 |
| 7 | DeepSeek V4.1 Flash | 34.8 | #58 |
| 8 | Qwen 3.5 397B A17B | 32.5 | #242 |
| 9 | Kimi K3 | 32.5 | #53 |
| 10 | Gemma 4 31B (IT) | 21.9 | #343 |
| 11 | MiMo-V2.5 | 20.4 | #353 |
| 12 | Qwen 2.5 VL 72B Instruct | 17.7 | #845 |
| 13 | Qwen 3.6 27B | 16.9 | #339 |
| 14 | GLM-4.5V | 16.6 | #739 |
| 15 | Qwen 3.5 27B | 16.4 | #358 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=v-ical-bench · How It Works · Data refreshed daily, snapshot 2026-11-12.