MOEVE - Summarization: leaderboard
MOEVE (Modelle fuer die Oeffentliche Verwaltung Evaluieren, Bundesdruckerei, 2026) benchmarks language models for the German public sector with German prompts and German datasets, many of them from public administration. This board is the mean overall summarization score on a 0-100 scale over four German legal and administrative datasets, combining embedding similarity, LLM-judged factual correctness and the share of output in German; higher is better.
Metric: Mean overall summarization score x 100 (0-100) over four German datasets (Eur-Lex-Sum, Swiss Leading Decision Summarization, KIKC Summary, German Ministry Publications), combining BERTScore, SemScore, LLM-judged factual correctness and the German-output proportion; German prompts, default decoding, open-weight models on vLLM or Ollama at the quantization listed in the paper's model table, others by API; the paper prints only the top 10 of 39 evaluated models; higher is better. Source: arxiv.org. Saturation forecast: Not forecast. 10 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Mistral Large 3 | 78 | #817 |
| 2 | GPT-4o Mini | 77 | #1623 |
| 3 | Phi-4 | 77 | #1913 |
| 4 | Mistral Small 3.1 | 76.7 | #1689 |
| 5 | Llama 3.3 70B | 76.6 | #1498 |
| 6 | GPT-4o | 76.3 | #741 |
| 7 | DeepSeek R1 | 76.1 | #473 |
| 8 | GPT-OSS-120B | 75.8 | #700 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=moeve-summarization · How It Works · Data refreshed daily, snapshot 2026-11-13.