SG-LegalClarify Mandatory Closed book: leaderboard

SG-LegalClarify Mandatory Closed book — Separateclosedbookandwebconditions; helpers/gateclosedbook, max8modelturns. Web8OpenAI/Anthropic/Geminimodelsnativeprovidersearch,remaining11Serper–Jinaloop; sameclosedbooksampling. AppendixBdetailsread andretainedinscale-paper.txt. 19models,1860scenarios (1100 originalquestions,760underspecifiedpairedwithfullyspecified),FS+UScombined. Mandatoryrubricpercentage macroaveragedacrosstrials andtheneligible3runs; nativeexceptions1/2runs retained. ThreejudgepanelDeepSeekV4Flash,GPT5.6Sol,KimiK3. Gemini3.1Pro gemini/gemini-3.1-pro-preview;Mistral mistral/mistral-large-latest;Qwen3.6 fireworks_ai/deployed/t5cvm6h4;Llama4ScoutBedrockclosedbookGroqweb;Gemma3Bedrockclosedbooklocalweb. Do not inventdatedmodelIDs absentpaper. Metric: Mandatory rubric criteria satisfied (%). Source: https://cdn.sanity.io/files/50zba0eo/production/aad2df6575c85900d3f862ae56e4bed01af76e4f.pdf

Metric: Mandatory rubric criteria satisfied (%). Source: cdn.sanity.io. Saturation forecast: Around May 2027. 19 models tracked. Updated 16 Nov 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Fable 5High69.8#27
2Claude Opus 4.8High66.6#58
3Kimi K3High65.9#56
4GPT-5.6 SolHigh64.6#30
5Gemini 3.5 FlashHigh62.5#100
6Claude Sonnet 5High61.2#111
7GPT-5.6 TerraHigh59.8#64
8GPT-5.6 LunaHigh58.4#122
9Gemini 3.1 Pro (Preview)High57.2#93
10Kimi K2.6High54.2#181
11DeepSeek V4 ProHigh52.4#191
12Mistral Large 3Scale run: reasoning=none51.8#827
13DeepSeek V4 FlashHigh50.1#259
14Qwen 3.7 PlusHigh49.9#208
15Qwen 3.6 27BHigh47#344

No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.

Interactive version: theaggregate.ai/benchmark?slug=sg-legalclarify-mandatory-closed-book · How It Works · Data refreshed daily, snapshot 2026-11-16.