NovBench - Relevance: leaderboard
NovBench (2026) tests whether LLMs can write the novelty assessment of a research paper from the novelty claims in its introduction, on 1,684 EMNLP 2023 papers with expert peer reviews. Relevance is how well the generated assessment covers each novelty claim, as an information-matching score from 0 to 5, higher is better.
Metric: Relevance (AvgIMS, 0-5; the paper states a 1-5 scale but prints values below 1): for each novelty-description sentence the best information-matching score against any sentence of the generated evaluation, averaged; zero-shot prompt; 1,684 EMNLP 2023 papers: from the novelty-description sentences of the paper introduction (selected by GPT-5) the model writes a novelty evaluation as positive, neutral and negative statements, compared with the expert reviewers novelty comments (extracted by GPT-4o-mini, then deduplicated, consolidated and sorted by sentiment by GPT-4o); greedy decoding, at most 4,096 tokens; higher is better. Source: arxiv.org. Saturation forecast: Not forecast. 19 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-4o | 3.7 | #751 |
| 2 | SEA-S | 3.63 | |
| 3 | Qwen 3 8B | 3.58 | #1873 |
| 4 | DeepSeek R1 Distill Llama 70B | 3.49 | #1669 |
| 5 | Qwen 3 32B | 3.47 | #972 |
| 6 | Gemini 2.5 Flash | 3.47 | #463 |
| 7 | CycleReviewer-70B | 3.46 | #1880 |
| 8 | SEA-E | 3.43 | |
| 9 | DeepSeek R1 Distill Qwen 14B | 3.41 | #2771 |
| 10 | Qwen 3 14B | 3.39 | #1269 |
| 11 | GPT-OSS-20B | 3.32 | #1183 |
| 12 | GPT-5 | 3.28 | #178 |
| 13 | GPT-OSS-120B | 3.26 | #704 |
| 14 | CycleReviewer-8B | 3.07 | |
| 15 | DeepSeek-R1-8B (NovBench checkpoint unspecified) | 2.95 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=novbench-relevance · How It Works · Data refreshed daily, snapshot 2026-11-21.