BenchX - Pass Rate: leaderboard
Sawt-ML’s BenchX tests consent and boundary handling in Saudi dialect Arabic: the share of scenarios where the model respects the stated boundary instead of talking past it. Dialect matters here, because the polite-refusal conventions the score keys on are not the Modern Standard Arabic ones.
Metric: Pass Rate (%). Source: huggingface.co. Saturation forecast: Rough model projection: around 2026. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | gemma-4-31b-it · T=1.0 | 90.14 | |
| 2 | gemma-4-31b-it · T=0 | 90.04 | |
| 3 | Q11 avg4 · T=0 | 89.94 | |
| 4 | gemma-4-31b-it · T=0.7 | 88.91 | |
| 5 | Q11 avg4 · T=1.0 | 88.11 | |
| 6 | Jais-3-32B · T=0.2 | 87.91 | |
| 7 | Q11 avg4 · T=0.7 | 87.91 | |
| 8 | Jais-3-MoVA-36B-A4B · T=0.2 | 80.37 | |
| 9 | sawt-v02-ep3 (production) | 75.1 |
Interactive version: theaggregate.ai/benchmark?slug=benchx-pass-rate · How It Works · Data refreshed daily, snapshot 2026-10-20.