Social Caption - Directed Social Analysis: leaderboard
Given a one-minute Social-IQ 2.0 video at one frame per second, its WhisperX transcript and a question about the interaction, a multimodal model describes the scene details, individuals, interactions and context needed to answer it; Social Caption (Mathur et al., 2026). Two human annotators rate each description on six 1-5 sub-dimensions, and the score is the summed mean rating out of 30 over a 50-video subset of the validation set; higher is better.
Metric: Human rating of a description of the social information needed to answer a Social-IQ 2.0 question about the video (sum of six 1-5 Likert sub-dimensions: relevant scene details, key individuals, relevant interactions, relevant context, answer detail, prompt adherence; 6-30; mean of two annotators per video; Social-IQ 2.0 validation videos (one minute each), a 50-video subset matched to the full set; the model gets the video at 1 frame per second and a timestamped WhisperX transcript, default temperature). Source: arxiv.org. Saturation forecast: Estimated already saturated. 10 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 2.5 Pro | 25.94 | #262 |
| 2 | GPT-4o | 24.42 | #741 |
| 3 | MiniCPM-V-2.6 | 23.51 | #3095 |
| 4 | Qwen 2.5 VL 7B | 23.49 | #1796 |
| 5 | Gemini 1.5 Pro | 23.42 | #1053 |
| 6 | InternVL3-8B | 23.01 | #1695 |
| 7 | InternVL2-8B | 19.88 | #3025 |
| 8 | Qwen 2 VL 7B | 18.9 | #2273 |
| 9 | Qwen2.5-Omni-7B | 8.91 | #1797 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=social-caption-directed-social-analysis · How It Works · Data refreshed daily, snapshot 2026-11-12.