τ³-bench-LILT - German - Banking - Pass^1: leaderboard
τ³-bench-LILT - German - Banking - Pass^1: Pass^1 (fraction) in LILT Aurora's localized evaluation. Localized customer-support tool use under the official tau3 scaffold; 4 trials and domain-specific judges; banking uses a 20-task sample.
Metric: Pass^1 (fraction). Source: aurora.lilt.com. Saturation forecast: Rough model projection: around 2026. 8 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 5.5 (Aurora Run: τ³-bench-LILT; effort low; OpenRouter) | 0.34 | |
| 2 | Claude Opus 5 (Aurora Run: τ³-bench-LILT; effort low; OpenRouter) | 0.31 | |
| 3 | GPT-6 Sol (Aurora Run: τ³-bench-LILT; effort none; OpenRouter) | 0.26 | |
| 4 | GPT-5.6 Sol (Aurora Run: τ³-bench-LILT; effort none; OpenRouter) | 0.2 | |
| 5 | Gemini 3.8 Flash (Aurora Run: τ³-bench-LILT; effort low; OpenRouter) | 0.1 | |
| 6 | Mistral Medium 3.5 (Aurora Run: τ³-bench-LILT; effort off; local vLLM; mistralai/Mistral-Medium-3.5-128B; temperature 0.1) | 0.09 | |
| 7 | Nemotron 3 Ultra (Aurora Run: τ³-bench-LILT; settings unreported) | 0.05 | |
| 8 | Command A+ (Aurora Run: τ³-bench-LILT; effort no off switch; local vLLM; CohereLabs/command-a-plus-05-2026) | 0.01 |
Interactive version: theaggregate.ai/benchmark?slug=tau3-bench-lilt-german-banking-pass-pow-1 · How It Works · Data refreshed daily, snapshot 2026-11-05.