TauIndianBankBench - Rare Tools: leaderboard
TauIndianBankBench (NPCI, arXiv 2610.30161) is a tau2-bench domain: the model is the chat assistant of a synthetic Indian retail bank with 33 servicing tools (accounts, deposits, loans, cards, mandates, cheques), and MiniMax-M2.7 plays the customer at temperature zero. Unlike tau-bench's final-state reward, a session passes only when the final records match and the required tool actions, lookups included, were made with no unrequested write ("We score every session twice: with the final-state check alone, as in τ-bench, and with both checks, as in the headline score"). One attempt per task at each model's most deterministic sampling setting; the paper does not state a thinking mode. Score: pass rate (%) on the rare-tool tasks, which "use infrequently needed operations". Higher is better. Source: https://arxiv.org/abs/2610.30161
Metric: Pass^1 (%). Source: arxiv.org. Saturation forecast: Around February 2027. 6 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3.8 27B | 75.7 | #123 |
| 2 | Gemma 4 31B | 55.9 | #212 |
| 3 | Gemma 4 12B | 53.3 | #503 |
| 4 | Gemma 4 26B A4B | 50 | #360 |
| 5 | Gemma 4 E4B | 48 | #762 |
| 6 | Gemma 4 E2B | 38.8 | #1062 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=tauindianbankbench-rare-tools · How It Works · Data refreshed daily, snapshot 2026-11-05.