Video Representation Validity - News verbal constructs (Native video): leaderboard
What Does the Model See? (arXiv 2611.05181v1) scores six open omni models on coding social-science constructs in videos against human labels, as macro-F1 averaged over the constructs in a group (Appendix Table 9). All models run "using bfloat16 precision and greedy decoding" with reasoning disabled, and "missing or invalid model outputs" count as incorrect; the authors state their focus is "rather than on a model ranking". Seven verbally defined constructs on 603 German-language TikTok news videos. Native video route: one prepared 120-s video file that the model's own video pathway samples to 16 frames. Metric: Macro-F1 (0–1).
Metric: Macro-F1 (0-1; mean of construct-level macro-F1 against human labels; seven verbally defined constructs on 603 German-language TikTok news videos (News on TikTok); Native video route: one prepared 120-s, 448-px video file sampled to 16 frames by the model's own video pathway; open checkpoints at pinned revisions, vLLM 0.30, bfloat16, greedy decoding, reasoning disabled (enable_thinking=False), JSON-schema-constrained codes; missing or invalid outputs count as incorrect). Source: arxiv.org. Saturation forecast: Around February 2027. 6 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemma 4 E4B (IT) (Non-reasoning) | 0.73 | |
| 2 | Qwen2.5-Omni-7B | 0.7 | #633 |
| 3 | Qwen3 Omni 30B A3B Instruct | 0.7 | #407 |
No result here: #3 Claude Opus 5.5, #5 GPT-6 Astra, #6 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=video-representation-validity-news-verbal-constructs-native-video · How It Works · Data refreshed daily, snapshot 2026-11-06.