JuICE - Span Detection: leaderboard
Span-level cultural error detection on JuICE (2026): an LLM judge reads a user query and a long-form response in a US-English, Korean, Indonesian or Bangladeshi setting and marks the spans that contain cultural or linguistic errors, matched to human-validated error spans by word overlap. Span F1 times 100; higher is better.
Metric: Span-level F1 (times 100, so 0-100) of erroneous-span detection: the judge reads a user query and a full long-form response and marks the spans with cultural or linguistic errors; a predicted span is a true positive when its word-level IoU with a human-validated JuICE error span exceeds 0.15, over 1,050 query-response pairs in seven country-language settings; higher is better. Source: arxiv.org. Saturation forecast: Around November 2027. 10 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3.1 Pro (Preview) | 52.25 | #99 |
| 2 | GPT-5.5 | 51.12 | #54 |
| 3 | Gemini 3 Flash (Preview) | 48.39 | #152 |
| 4 | Claude Opus 4.7 | 45.03 | #85 |
| 5 | Gemma 4 31B (IT) | 44.5 | #350 |
| 6 | Claude Haiku 4.5 | 36.91 | #583 |
| 7 | GPT-OSS-120B | 36.57 | #704 |
| 8 | Qwen 3 30B A3B 2507 Instruct | 31.28 | #1100 |
| 9 | GPT-5.4 Mini | 30.79 | #406 |
| 10 | Llama 4 Scout Instruct | 28.05 | #1579 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=juice-span-detection · How It Works · Data refreshed daily, snapshot 2026-11-21.