Long-tail book recall
Closed-book questions about details in 60 public-domain books, from the famous to the obscure.
Run Sep 2–9, 2026, 282 items.
Method
Each question asks for one specific detail from a Project Gutenberg book, such as a minor character's name or the place where a scene happens, and the answer key is a verbatim span of the book. The 60 books sit on five rungs by how often their Wikipedia article is read, from the canon down to books almost nobody opens, so the bank keeps separating models long after the famous titles are solved.
Answers are matched against the key; an answer that misses the exact wording goes to a second-pass judge that accepts equivalent answers. Saying “unknown” counts as wrong. Reasoning effort was set to high.
Results
| # | Model | Accuracy |
|---|---|---|
| 1 | Claude Fable 5.1 | 69.5% |
| 2 | Claude Fable 5 | 69.1% |
| 3 | Claude Opus 5 | 62.8% |
| 4 | Gemini 3.7 Flash | 56.7% |
| 5 | GPT-5.6 Sol | 55.3% |
| 6 | GPT-6 Astra | 50.7% |
| 7 | GPT-5.4 | 46.1% |
| 8 | DeepSeek V4 Pro | 44.0% |
| 9 | Claude Opus 4.8 | 42.2% |
| 10 | Muse Spark 1.3 | 39.4% |
| 11 | GPT-5.4 Mini | 38.3% |
| 12 | GPT-5.2 | 37.2% |
| 13 | Qwen 3.6 Plus | 31.9% |
| 14 | Kimi K2.6 | 30.5% |
| 15 | Kimi K2 0905 | 28.7% |
| 16 | Qwen 3.6 35B A3B | 27.7% |
| 17 | Claude Haiku 4.5 | 17.4% |
| 18 | GPT-OSS-120B | 17.4% |
- GPT-6 Astra, GPT-5.6 Sol and Gemini 3.7 Flash were added on September 9. Their answers that missed the exact wording were checked by a different reviewer than the original panel's.
- Muse Spark 1.3 answered “unknown” to 54% of the questions, so its score is a floor.
- Claude Fable 5 and its point release 5.1 land half a point apart. Asking the same model again flips about 8% of its correct answers, so gaps of a point or two are noise.
- Not ranked, every reply came back empty: LFM2.5-2.6B (0 of 282), Nemotron 3.5 Lightning (0 of 282).
In the main leaderboard
These results are collected for the main leaderboard alongside external benchmarks. New scores enter the rating through the regular audited data updates. Comparisons with that rating therefore share some of its inputs.
Interactive version: theaggregate.ai/our-benchmarks/book-recall · How It Works · Data refreshed daily, snapshot 2026-10-11.