Video Representation Validity - HateMM hate (Frames + audio): leaderboard
What Does the Model See? (arXiv 2611.05181v1) scores six open omni models on coding social-science constructs in videos against human labels, as macro-F1 averaged over the constructs in a group (Appendix Table 9). All models run "using bfloat16 precision and greedy decoding" with reasoning disabled, and "missing or invalid model outputs" count as incorrect; the authors state their focus is "rather than on a model ranking". The HateMM hate label on 400 videos. Frames + audio route: 16 extracted frames over the first 120 s and separate 30-s audio clips, "identical pixels and audio for every model". Metric: Macro-F1 (0–1).
Metric: Macro-F1 (0-1; mean of construct-level macro-F1 against human labels; HateMM hate label on 400 videos; Frames + audio route: 16 extracted 448-px frames over the first 120 s plus separate 30-s audio clips, identical for every model; open checkpoints at pinned revisions, vLLM 0.30, bfloat16, greedy decoding, reasoning disabled (enable_thinking=False), JSON-schema-constrained codes; missing or invalid outputs count as incorrect). Source: arxiv.org. Saturation forecast: Estimated already saturated. 6 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemma 4 E4B (IT) (Non-reasoning) | 0.78 | |
| 2 | Qwen2.5-Omni-7B | 0.77 | #633 |
| 3 | Qwen3 Omni 30B A3B Instruct | 0.74 | #407 |
No result here: #3 Claude Opus 5.5, #5 GPT-6 Astra, #6 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=video-representation-validity-hatemm-hate-frames-plus-audio · How It Works · Data refreshed daily, snapshot 2026-11-06.