Lumen-bench - Mandarin: leaderboard
Lumen-bench harmful tool-action rate on the Mandarin versions of the 108 scenarios (both system prompts, valid trials from 1,296 scheduled per model). It isolates conduct in one language where the headline pools all five.
Metric: Harmful tool-action rate (%, lower is better). Source: arxiv.org. 30 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3.1 Pro (Preview) | 0 | #94 |
| 2 | GPT-5.4 | 0 | #123 |
| 3 | Claude Haiku 4.5 | 0 | #568 |
| 4 | GPT-5 Mini (2025-08-07) | 0 | #343 |
| 5 | DeepSeek V3.2 | 0.08 | #406 |
| 6 | MiniMax-M2.7 | 0.31 | #518 |
| 7 | Qwen 3.5 27B | 0.46 | #359 |
| 8 | Gemma 4 26B A4B (IT) | 0.46 | #549 |
| 9 | Gemma 4 31B (IT) | 0.54 | #342 |
| 10 | Qwen 3.5 397B A17B | 0.62 | #246 |
| 11 | GPT-OSS-120B | 0.77 | #700 |
| 12 | GPT-OSS-20B | 0.93 | #1147 |
| 13 | Qwen 3.5 35B A3B | 1 | #482 |
| 14 | GPT-4.1 | 1.62 | #495 |
| 15 | Gemini 3.1 Flash Lite (Preview) | 1.93 | #418 |
Interactive version: theaggregate.ai/benchmark?slug=lumen-bench-mandarin · How It Works · Data refreshed daily, snapshot 2026-11-13.