SG-LegalClarify Mandatory Closed book: leaderboard
SG-LegalClarify Mandatory Closed book — Separateclosedbookandwebconditions; helpers/gateclosedbook, max8modelturns. Web8OpenAI/Anthropic/Geminimodelsnativeprovidersearch,remaining11Serper–Jinaloop; sameclosedbooksampling. AppendixBdetailsread andretainedinscale-paper.txt. 19models,1860scenarios (1100 originalquestions,760underspecifiedpairedwithfullyspecified),FS+UScombined. Mandatoryrubricpercentage macroaveragedacrosstrials andtheneligible3runs; nativeexceptions1/2runs retained. ThreejudgepanelDeepSeekV4Flash,GPT5.6Sol,KimiK3. Gemini3.1Pro gemini/gemini-3.1-pro-preview;Mistral mistral/mistral-large-latest;Qwen3.6 fireworks_ai/deployed/t5cvm6h4;Llama4ScoutBedrockclosedbookGroqweb;Gemma3Bedrockclosedbooklocalweb. Do not inventdatedmodelIDs absentpaper. Metric: Mandatory rubric criteria satisfied (%). Source: https://cdn.sanity.io/files/50zba0eo/production/aad2df6575c85900d3f862ae56e4bed01af76e4f.pdf
Metric: Mandatory rubric criteria satisfied (%). Source: cdn.sanity.io. Saturation forecast: Around May 2027. 19 models tracked. Updated 16 Nov 2026.
Top models
| # | Model | Setting | Score | Overall rank |
|---|---|---|---|---|
| 1 | Claude Fable 5 | High | 69.8 | #27 |
| 2 | Claude Opus 4.8 | High | 66.6 | #58 |
| 3 | Kimi K3 | High | 65.9 | #56 |
| 4 | GPT-5.6 Sol | High | 64.6 | #30 |
| 5 | Gemini 3.5 Flash | High | 62.5 | #100 |
| 6 | Claude Sonnet 5 | High | 61.2 | #111 |
| 7 | GPT-5.6 Terra | High | 59.8 | #64 |
| 8 | GPT-5.6 Luna | High | 58.4 | #122 |
| 9 | Gemini 3.1 Pro (Preview) | High | 57.2 | #93 |
| 10 | Kimi K2.6 | High | 54.2 | #181 |
| 11 | DeepSeek V4 Pro | High | 52.4 | #191 |
| 12 | Mistral Large 3 | Scale run: reasoning=none | 51.8 | #827 |
| 13 | DeepSeek V4 Flash | High | 50.1 | #259 |
| 14 | Qwen 3.7 Plus | High | 49.9 | #208 |
| 15 | Qwen 3.6 27B | High | 47 | #344 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=sg-legalclarify-mandatory-closed-book · How It Works · Data refreshed daily, snapshot 2026-11-16.