TRACE Legislation - Test + Holdout Top 10 Lenient - TRACE v1: leaderboard
TRACE Legislation - Test + Holdout Top 10 Lenient - TRACE v1: Recall against automated NAZK findings (0-1). TRACE arXiv 2611.14244v1: agreement with model-extracted NAZK legal findings, strict Same and lenient Same+partial recall. Single runs. Claude Sonnet 5.5 low reasoning, Opus 5.5 same settings; monolithic prompt temperature 0 and max output 12000. Native parser/version, split, top-10 versus all-flags, blind/non-blind and Codex/OpenRouter judge routes are kept separate. Gemma local profile is held because source describes FP8 or higher precision, without identifying one released served artifact. This is automated agreement, not human-validated legal correctness.
Metric: Recall against automated NAZK findings (0-1). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 2 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Sonnet 5.5 (Benchmark Sweep Run: source=trace_legislation_paper; harness=TRACE v1; reasoning=low; judge route=Codex) | 0.43 | |
| 2 | Claude Opus 5.5 (Benchmark Sweep Run: source=trace_legislation_paper; harness=TRACE v1; reasoning=low; judge route=OpenRouter) | 0.34 |
Interactive version: theaggregate.ai/benchmark?slug=trace-legislation-test-plus-holdout-top-10-lenient-trace-v1 · How It Works · Data refreshed daily, snapshot 2026-11-12.