Video Representation Validity - HateMM target Black (Native video): leaderboard
What Does the Model See? (arXiv 2611.05181v1) scores six open omni models on coding social-science constructs in videos against human labels, as macro-F1 averaged over the constructs in a group (Appendix Table 9). All models run "using bfloat16 precision and greedy decoding" with reasoning disabled, and "missing or invalid model outputs" count as incorrect; the authors state their focus is "rather than on a model ranking". HateMM's Black-target variable on the same 400 videos. Native video route: one prepared 120-s video file that the model's own video pathway samples to 16 frames. Metric: Macro-F1 (0–1).
Metric: Macro-F1 (0-1; mean of construct-level macro-F1 against human labels; HateMM Black-target variable on the same 400 videos; Native video route: one prepared 120-s, 448-px video file sampled to 16 frames by the model's own video pathway; open checkpoints at pinned revisions, vLLM 0.30, bfloat16, greedy decoding, reasoning disabled (enable_thinking=False), JSON-schema-constrained codes; missing or invalid outputs count as incorrect). Source: arxiv.org. Saturation forecast: Estimated already saturated. 6 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen3 Omni 30B A3B Instruct | 0.8 | #407 |
| 2 | Gemma 4 E4B (IT) (Non-reasoning) | 0.75 | |
| 3 | Qwen2.5-Omni-7B | 0.71 | #633 |
No result here: #3 Claude Opus 5.5, #5 GPT-6 Astra, #6 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=video-representation-validity-hatemm-target-black-native-video · How It Works · Data refreshed daily, snapshot 2026-11-06.