Beyond the Sycophancy Score - Unknowable Quantities: leaderboard
The model is asked about a quantity nobody can know, then a simulated user insists on a specific value for three turns; the number is the share of eligible conversations (those whose first reply had not already taken that position) where the final reply endorses that value. Unlike answer-flip tests there is no correct answer to abandon: the failure is asserting an unknowable figure because the user wants it.
Metric: Final-turn adoption of a user-asserted value for an unknowable quantity (%). Source: arxiv.org. 8 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3.5 397B A17B (Non-reasoning) | 0 | #246 (Qwen 3.5 397B A17B) |
| 2 | Claude Sonnet 5 (Non-reasoning) | 0 | #111 (Claude Sonnet 5) |
| 3 | Grok 4.3 (Non-reasoning) | 0.3 | #333 (Grok 4.3) |
| 4 | GPT-5.6 Luna (Non-reasoning) | 0.7 | #121 (GPT-5.6 Luna) |
| 5 | GLM-5.2 (Non-reasoning) | 1.6 | #138 (GLM-5.2) |
| 6 | DeepSeek V4 Flash (0731) (Non-reasoning) | 24.6 | #139 (DeepSeek V4 Flash (0731)) |
Interactive version: theaggregate.ai/benchmark?slug=beyond-the-sycophancy-score-unknowable-quantities · How It Works · Data refreshed daily, snapshot 2026-11-13.