Werewolf Accusation Benchmark - Role Separation: leaderboard
Observer-side Werewolf test: from a village-side player's view of an LLM-played game, the model rates the accuser and the accused of an annotated accusation from definitely villager (-3) to definitely wolf (+3) before seeing the message. Scored as role separation, the mean rating of true werewolves minus that of true villagers (-6 to 6, higher is better); Yang et al., EMNLP 2026.
Metric: Belief separation (wolves minus villagers, -3..+3 scale). Source: github.com. Saturation forecast: Around May 2027. 40 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemma 4 31B (IT) (Thinking) | 1.69 | #343 (Gemma 4 31B (IT)) |
| 2 | Qwen 3.6 27B (Thinking) | 1.64 | #339 (Qwen 3.6 27B) |
| 3 | Qwen 3.6 27B (Non-reasoning) | 1.64 | #339 (Qwen 3.6 27B) |
| 4 | Gemma 4 31B (IT) (Non-reasoning) | 1.52 | #343 (Gemma 4 31B (IT)) |
| 5 | Qwen 3 14B (Reasoning) | 1.51 | #1235 (Qwen 3 14B) |
| 6 | Qwen 3 8B (Thinking) | 1.51 | #1838 (Qwen 3 8B) |
| 7 | Qwen 3.6 35B A3B (Non-reasoning) | 1.5 | #461 (Qwen 3.6 35B A3B) |
| 8 | gemma-4-26B-A4B-it (Thinking) | 1.48 | |
| 9 | Qwen 3.5 2B (Thinking) | 1.48 | #4849 (Qwen 3.5 2B) |
| 10 | Qwen 3.5 4B (Thinking) | 1.47 | #1180 (Qwen 3.5 4B) |
| 11 | Gemma 4 26B A4B (IT) (Non-reasoning) | 1.44 | #545 (Gemma 4 26B A4B (IT)) |
| 12 | OLMo-3.1 32B Think | 1.41 | #1452 |
| 13 | Phi-4 | 1.39 | #1920 |
| 14 | Qwen 3.5 9B (Thinking) | 1.37 | #803 (Qwen 3.5 9B) |
| 15 | OLMo 3.1 32B Instruct | 1.33 | #2403 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=werewolf-accusation-benchmark-role-separation · How It Works · Data refreshed daily, snapshot 2026-11-12.