PushupBench - Mean Absolute Error: leaderboard

Repetition counting error on PushupBench (2026), 446 long-form exercise videos of 22 to 117 seconds, sampled at 5 frames per second: the mean absolute difference between predicted and annotated repetition counts, leaving out predictions off by more than 50. Repetitions; lower is better.

Metric: Mean absolute error of the predicted repetition count (repetitions; predictions off by more than 50 are excluded as parsing or hallucination outliers), on PushupBench's 446 long-form exercise clips (22-117 s, 11 creators in 10 countries; 27 clips accept several valid counts) sampled at 5 fps up to 112 frames (Claude models capped at 100 frames by the API), 360p frames, greedy decoding, 10 prompt templates; lower is better. Source: arxiv.org. Saturation forecast: Around June 2027. 20 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 3 Flash2.9#175
2Gemini 3 Pro3.6#145
3Gemini 2.5 Pro5.7#276
4Qwen 3 VL 235B A22B Instruct7.1#532
5GPT-57.6#178
6Gemini 2.5 Flash7.7#463
7Qwen 3 VL 32B Instruct7.7#606
8Gemini 2.0 Flash7.8#723
9Qwen 2.5 VL 7B7.8#1862
10Qwen 3 VL 8B Instruct7.9#1029
11Qwen 3 VL 4B Instruct8.2#1332
12Qwen 3 VL 30B A3B (Thinking)8.8#719
13Qwen 3 VL 4B (Thinking)8.9#1334
14Qwen 2.5 VL 3B8.9#2639
15Claude Sonnet 4.59#261

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=pushupbench-mean-absolute-error · How It Works · Data refreshed daily, snapshot 2026-11-21.