τ³-bench-LILT - English - Telecom - Pass^4: leaderboard
τ³-bench-LILT - English - Telecom - Pass^4: Pass^4 (fraction) in LILT Aurora's localized evaluation. Localized customer-support tool use under the official tau3 scaffold; 4 trials and domain-specific judges; banking uses a 20-task sample.
Metric: Pass^4 (fraction). Source: aurora.lilt.com. Saturation forecast: Rough model projection: around 2026. 8 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 5.5 (Aurora Run: τ³-bench-LILT; effort low; OpenRouter) | 0.96 | |
| 2 | GPT-6 Sol (Aurora Run: τ³-bench-LILT; effort none; OpenRouter) | 0.69 | |
| 3 | Command A+ (Aurora Run: τ³-bench-LILT; effort no off switch; local vLLM; CohereLabs/command-a-plus-05-2026) | 0.68 | |
| 4 | Mistral Medium 3.5 (Aurora Run: τ³-bench-LILT; effort off; local vLLM; mistralai/Mistral-Medium-3.5-128B; temperature 0.1) | 0.67 | |
| 5 | Claude Opus 5 (Aurora Run: τ³-bench-LILT; effort off; OpenRouter) | 0.48 | |
| 6 | Nemotron 3 Ultra (Aurora Run: τ³-bench-LILT; settings unreported) | 0.47 | |
| 7 | Gemini 3.8 Flash (Aurora Run: τ³-bench-LILT; effort low; OpenRouter) | 0.38 | |
| 8 | GPT-5.6 Sol (Aurora Run: τ³-bench-LILT; effort none; OpenRouter) | 0.37 |
Interactive version: theaggregate.ai/benchmark?slug=tau3-bench-lilt-english-telecom-pass-pow-4 · How It Works · Data refreshed daily, snapshot 2026-11-05.