JuICE - Span Detection: leaderboard

Span-level cultural error detection on JuICE (2026): an LLM judge reads a user query and a long-form response in a US-English, Korean, Indonesian or Bangladeshi setting and marks the spans that contain cultural or linguistic errors, matched to human-validated error spans by word overlap. Span F1 times 100; higher is better.

Metric: Span-level F1 (times 100, so 0-100) of erroneous-span detection: the judge reads a user query and a full long-form response and marks the spans with cultural or linguistic errors; a predicted span is a true positive when its word-level IoU with a human-validated JuICE error span exceeds 0.15, over 1,050 query-response pairs in seven country-language settings; higher is better. Source: arxiv.org. Saturation forecast: Around November 2027. 10 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 3.1 Pro (Preview)52.25#99
2GPT-5.551.12#54
3Gemini 3 Flash (Preview)48.39#152
4Claude Opus 4.745.03#85
5Gemma 4 31B (IT)44.5#350
6Claude Haiku 4.536.91#583
7GPT-OSS-120B36.57#704
8Qwen 3 30B A3B 2507 Instruct31.28#1100
9GPT-5.4 Mini30.79#406
10Llama 4 Scout Instruct28.05#1579

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=juice-span-detection · How It Works · Data refreshed daily, snapshot 2026-11-21.