BDBBQ Certified - Native Chat - ben_accuracy: leaderboard
BDBBQ official certified2026-10-30JSON,30656questions perlanguage,61312permodel,English/Bengali andoverall/ambiguous/disambiguated cohorts explicit. BF16,L40S,vLLM,context2048,temp0. Seven nativechatmodels16tokens;BanglaLLM ChatML is separate from paperplaincompletion5tokens and is not substituted. Ambiguous correctunknown;disambiguated factualgold,invalidoutputwrong,answerpermutationsseparaterecords,nomajorityvote. Accuracy only; signedbias and parsefailrates are diagnostics.
Metric: Score (%). Source: raw.githubusercontent.com. Saturation forecast: Estimated already saturated. 7 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemma 2 9B (IT) | 84.68 | #2275 |
| 2 | Mistral Nemo Instruct (2407) | 82.44 | #2978 |
| 3 | Qwen 2.5 7B Instruct | 66.62 | #2583 |
| 4 | Ministral-8B-Instruct-2410 | 65.51 | #4720 |
| 5 | Llama 3.1 8B Instruct | 62.74 | #4277 |
| 6 | Qwen 2.5 3B Instruct | 51.38 | #5206 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=bdbbq-certified-native-chat-ben-accuracy · How It Works · Data refreshed daily, snapshot 2026-11-12.