ImplicitBBQ - Disambiguated: leaderboard

ImplicitBBQ (2026) extends the BBQ question-answering bias test to implicit identity, signaled only through characteristic cues such as appearance or customs across age, gender, region, religion, caste and socioeconomic status. This board scores contexts that state which person the question is about, as accuracy in percent, higher is better.

Metric: Accuracy (%, times 100) in disambiguated implicit contexts, where the context names which person the question is about and the answer is that person: demographic identity is conveyed only by human-validated characteristic cues (kappa at least 0.60) instead of labels, about 8k implicit instances built from 760 BBQ and BharatBBQ contexts and 60 cues across age, gender, region, religion, caste and socioeconomic status, negative and non-negative questions, zero-shot single-word answers, model-wise average over demographics; open models on the full set, closed models on a balanced 600-sample subset; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 11 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1GPT-5 Mini100#364
2Claude Haiku 4.599#583
3GPT-OSS-120B98#704
4Qwen 3 32B98#972
5Gemini 3.1 Flash Lite98#371
6Llama 3.3 70B Instruct97#1385
7GPT-OSS-20B97#1183
8Llama 3.1 8B Instruct96#4600
9Mistral 7B Instruct95#7568
10Qwen 2.5 7B Instruct94#2723
11Phi-4 Mini Instruct93#6631

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=implicitbbq-disambiguated · How It Works · Data refreshed daily, snapshot 2026-11-21.