German LLM Index - MuSR (564 questions): leaderboard
German Language LLM Index (dach.peerbench.ai, a PeerBench project, 2026), which reruns every model itself with lm-evaluation-harness: chat template, temperature 0, up to 8,192 generated tokens, thinking models set to minimal reasoning effort where the API allows it and other models run as served. Labels carry (Reasoning off) when the site classed the run as reasoning-off (at most ten reasoning tokens, or under 0.1% of its completion tokens); other rows reasoned at the provider default or at the effort the label names. German translation of MuSR, multistep soft reasoning over long narratives (murder mysteries, object placements, team allocation), 2 to 5 answer options. This board holds runs on all 564 translated questions, before the v1.1 correction removed 11 team-allocation questions whose translation was truncated; it is a separate board from MuSR v1.1 because the question set differs. Accuracy (%), higher is better. Source: https://dach.peerbench.ai/
Metric: Accuracy (%). Source: dach.peerbench.ai. Saturation forecast: Around February 2027. 19 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Fable 5 | 89.54 | #20 |
| 2 | Gemini 3.1 Pro (Preview) | 88.12 | #64 |
| 3 | GPT-5.5 (Low) | 86.35 | #33 (GPT-5.5) |
| 4 | Claude Opus 4.8 (Non-reasoning) | 86.35 | #39 (Claude Opus 4.8) |
| 5 | DeepSeek V4 Pro (Non-reasoning) | 85.28 | #112 (DeepSeek V4 Pro) |
| 6 | Gemini 3.1 Flash Lite (Non-reasoning) | 84.4 | #200 (Gemini 3.1 Flash Lite) |
| 7 | Gemma 4 26B A4B (IT) (Non-reasoning) | 83.69 | #293 (Gemma 4 26B A4B (IT)) |
| 8 | DeepSeek V4 Flash (Non-reasoning) | 83.51 | #155 (DeepSeek V4 Flash) |
| 9 | Gemma 4 31B (IT) (Non-reasoning) | 83.51 | #178 (Gemma 4 31B (IT)) |
| 10 | Gemini 2.5 Flash (Non-reasoning) | 83.33 | #255 (Gemini 2.5 Flash) |
| 11 | GLM-5.1 (Non-reasoning) | 81.38 | #109 (GLM-5.1) |
| 12 | Qwen 3.5 9B (Non-reasoning) | 80.85 | #391 (Qwen 3.5 9B) |
| 13 | MiMo-V2.5-Pro (Non-reasoning) | 80.85 | #167 (MiMo-V2.5-Pro) |
| 14 | Gemini 3 Flash (Preview) (Non-reasoning) | 79.26 | #90 (Gemini 3 Flash (Preview)) |
| 15 | Qwen 3 14B (Non-reasoning) | 69.68 | #542 (Qwen 3 14B) |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=german-llm-index-musr-564-questions · How It Works · Data refreshed daily, snapshot 2026-11-05.