PaperAudit-Bench (Fast Mode) - Gemini 2.5 Pro Errors: leaderboard
PaperAudit-Bench (Tu et al., 2026) tests whether a model finds errors injected into accepted papers. This board uses the 46 NeurIPS 2025 oral papers as corrupted by Gemini 2.5 Pro (10 to 20 injected errors of eight types each), reviewed by each model in the Fast mode's single pass over the whole paper; text-only models see the papers without figures. Macro-F1 in percent over papers, the harmonic mean of error coverage and finding precision with a GPT-5.1 judge matching findings to injected errors, higher is better; the synthesis model is itself one of the detectors.
Metric: Macro-F1 (%; the per-paper harmonic mean of error coverage and finding precision, averaged over the 46 NeurIPS 2025 oral papers of the benchmark's NeurIPS branch as corrupted by gemini-2.5-pro with 10-20 injected errors of eight types; Fast mode, one global review pass over the whole paper; findings matched to the injected errors by a GPT-5.1 judge; the three text-only detectors read the papers with figures removed). Source: arxiv.org. Saturation forecast: Around February 2027. 10 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 2.5 Pro | 39.7 | #266 |
| 2 | GPT-5 | 37.4 | #167 |
| 3 | Grok 4 | 31.6 | #307 |
| 4 | Claude Sonnet 4.5 | 29.9 | #250 |
| 5 | O4 Mini | 27.8 | #353 |
| 6 | Qwen 3 235B A22B 2507 Instruct | 27.6 | #619 |
| 7 | GLM-4.6 (Non-reasoning) | 26.1 | #486 (GLM-4.6) |
| 8 | Kimi K2 0905 | 18.2 | #558 |
| 9 | DeepSeek V3.1 (Non-reasoning) | 15.9 | #497 (DeepSeek V3.1) |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=paperaudit-bench-fast-mode-gemini-2-5-pro-errors · How It Works · Data refreshed daily, snapshot 2026-11-13.