ImplicitBBQ - Disambiguated: leaderboard
ImplicitBBQ (2026) extends the BBQ question-answering bias test to implicit identity, signaled only through characteristic cues such as appearance or customs across age, gender, region, religion, caste and socioeconomic status. This board scores contexts that state which person the question is about, as accuracy in percent, higher is better.
Metric: Accuracy (%, times 100) in disambiguated implicit contexts, where the context names which person the question is about and the answer is that person: demographic identity is conveyed only by human-validated characteristic cues (kappa at least 0.60) instead of labels, about 8k implicit instances built from 760 BBQ and BharatBBQ contexts and 60 cues across age, gender, region, religion, caste and socioeconomic status, negative and non-negative questions, zero-shot single-word answers, model-wise average over demographics; open models on the full set, closed models on a balanced 600-sample subset; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 11 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5 Mini | 100 | #364 |
| 2 | Claude Haiku 4.5 | 99 | #583 |
| 3 | GPT-OSS-120B | 98 | #704 |
| 4 | Qwen 3 32B | 98 | #972 |
| 5 | Gemini 3.1 Flash Lite | 98 | #371 |
| 6 | Llama 3.3 70B Instruct | 97 | #1385 |
| 7 | GPT-OSS-20B | 97 | #1183 |
| 8 | Llama 3.1 8B Instruct | 96 | #4600 |
| 9 | Mistral 7B Instruct | 95 | #7568 |
| 10 | Qwen 2.5 7B Instruct | 94 | #2723 |
| 11 | Phi-4 Mini Instruct | 93 | #6631 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=implicitbbq-disambiguated · How It Works · Data refreshed daily, snapshot 2026-11-21.