SocialReasonBench - Counterfactual Reasoning: leaderboard
SocialReasonBench (2026) is a six-option video question-answering benchmark on counterfactual narrative clips from the game Detroit: Become Human, whose branching storylines let answers be checked against the game script and recorded branches. This board covers the 50 counterfactual reasoning questions, asking what would happen under an alternative choice. Accuracy in percent; higher is better.
Metric: Accuracy (%; the 50 counterfactual reasoning questions of SocialReasonBench; six-option multiple-choice questions on counterfactual narrative gameplay clips of Detroit: Become Human, labels grounded in the game's script, flowchart and recorded branches; zero-shot, temperature 0 or 0.01; Gemini, Qwen3-Omni, MiniCPM-o and ARC-Hunyuan-Video read native video with audio, GPT-4o, Claude 3.5 Sonnet and GLM-4.6V-Flash 32 evenly sampled keyframes without audio). Source: arxiv.org. Saturation forecast: Around February 2027. 8 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3 Pro | 62 | #135 |
| 2 | Claude 3.5 Sonnet (20240620) | 56 | #684 |
| 3 | GPT-4o (2024-08-06) | 50 | #731 |
| 4 | Gemini 3 Flash | 46 | #159 |
| 5 | Qwen3 Omni 30B A3B Instruct | 36 | #902 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=socialreasonbench-counterfactual-reasoning · How It Works · Data refreshed daily, snapshot 2026-11-12.