NovBench - Relevance: leaderboard

NovBench (2026) tests whether LLMs can write the novelty assessment of a research paper from the novelty claims in its introduction, on 1,684 EMNLP 2023 papers with expert peer reviews. Relevance is how well the generated assessment covers each novelty claim, as an information-matching score from 0 to 5, higher is better.

Metric: Relevance (AvgIMS, 0-5; the paper states a 1-5 scale but prints values below 1): for each novelty-description sentence the best information-matching score against any sentence of the generated evaluation, averaged; zero-shot prompt; 1,684 EMNLP 2023 papers: from the novelty-description sentences of the paper introduction (selected by GPT-5) the model writes a novelty evaluation as positive, neutral and negative statements, compared with the expert reviewers novelty comments (extracted by GPT-4o-mini, then deduplicated, consolidated and sorted by sentiment by GPT-4o); greedy decoding, at most 4,096 tokens; higher is better. Source: arxiv.org. Saturation forecast: Not forecast. 19 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1GPT-4o3.7#751
2SEA-S3.63
3Qwen 3 8B3.58#1873
4DeepSeek R1 Distill Llama 70B3.49#1669
5Qwen 3 32B3.47#972
6Gemini 2.5 Flash3.47#463
7CycleReviewer-70B3.46#1880
8SEA-E3.43
9DeepSeek R1 Distill Qwen 14B3.41#2771
10Qwen 3 14B3.39#1269
11GPT-OSS-20B3.32#1183
12GPT-53.28#178
13GPT-OSS-120B3.26#704
14CycleReviewer-8B3.07
15DeepSeek-R1-8B (NovBench checkpoint unspecified)2.95

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=novbench-relevance · How It Works · Data refreshed daily, snapshot 2026-11-21.