M-Drama - Summary: leaderboard
The summary task of M-Drama (Qin et al., EMNLP 2026): a model watches a micro-drama episode and writes a structured summary, which DeepSeek-V3.2 rates from 1 to 10 against the human-verified reference description, penalizing hallucinated or wrong facts most. Mean score over 1,036 Chinese and English episodes on the 10-point scale; higher is better.
Metric: Summary judge score (out of 10): DeepSeek-V3.2 rating on a 1-10 rubric of accuracy, completeness and fluency against the human-verified reference description, penalizing hallucinated and wrong facts most; 1,036 episode summaries; test split of 1,036 Chinese and English micro-drama episodes; zero-shot, 192 frames sampled at 2 fps, temperature 0.7, top-p 0.8. Source: arxiv.org. Saturation forecast: Around February 2027. 11 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5 | 7.54 | #163 |
| 2 | Gemini 3 Flash | 6.21 | #159 |
| 3 | Gemini 2.5 Flash | 5.78 | #452 |
| 4 | Qwen 2.5 VL 72B Instruct | 5.03 | #845 |
| 5 | Qwen 3 VL 32B (Thinking) | 4.93 | #557 |
| 6 | Qwen 3 VL 8B (Thinking) | 4.18 | #887 |
| 7 | Qwen 2.5 VL 7B Instruct | 3.67 | #1842 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=m-drama-summary · How It Works · Data refreshed daily, snapshot 2026-11-12.