MORSE-500: leaderboard

Each model answers all 500 programmatically generated videos, in which the question itself appears inside the video (API models such as o3 and Gemini 2.5 get frames at 2 fps, at most 32), covering abstract, math, physical, planning, spatial and temporal reasoning; the score is overall accuracy. Unlike video QA sets with text questions, the model gets only the minimal prompt "Answer the question in this video".

Metric: Accuracy (%). Source: arxiv.org. Saturation forecast: Estimated already saturated. 19 models tracked. Updated 17 Nov 2026.

Top models

#ModelScoreOverall rank
1O3 (2025-04-16)23.6#229
2O4 Mini (2025-04-16)22.2#376
3Gemini 2.5 Pro (Preview 05-06)21.8#271
4O1 (2024-12-17)19.8#314
5Gemini 2.5 Flash (Preview 04-17)19.2#455
6Qwen 2.5 VL 72B Instruct17.8#849
7GPT-4o (2024-11-20)17.4#864
8Qwen2.5-VL-32B-Instruct-AWQ16.8
9Qwen2.5-VL-72B-Instruct-AWQ16.4#704
10Gemini 2.0 Flash (001)16#727
11Qwen 2.5 VL 32B Instruct15.6#1074
12Gemma 3 27B (IT)14.6#1339
13Gemini 2.0 Flash Lite (001)14.2#1100
14MiniCPM-o-2.611.6#2417
15Qwen2.5-Omni-7B11.4#1883

No result here: #1 Claude Fable 5.2, #2 GPT-6.2 Astra, #3 Gemini 4 Carbon.

Interactive version: theaggregate.ai/benchmark?slug=morse-500 · How It Works · Data refreshed daily, snapshot 2026-11-17.