Tau2-Bench Verified - Airline: leaderboard

The 50 airline tasks of tau2-Bench Verified, as corrected by Amazon AGI (bookings, changes, cancellations, compensation under the airline policy), with gpt-5.1 as the customer; the score is the share of task runs that pass (the README says averaged over 5 runs). Amazon's own Nova 2 rows were added in a later commit that does not state how they were run.

Metric: Airline pass^1 (%). Source: github.com. Saturation forecast: Around February 2027. 14 models tracked. Updated 17 Nov 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Opus 4.574.4#161
2GPT-5.2xHigh74.27#185
3GPT-5.1High72.4#235
4Grok 4.1 FastReasoning72#429
5GPT-5Medium72#177
6Gemini 3 Pro70.8#145
7GPT-5 MiniMedium68.8#359
8Nova 2 Omni68.8#472
9Claude Sonnet 4.566.8#258
10Nova 2 Pro65.2#516
11Nova 2.0 Lite64.8#1125
12Gemini 2.5 Pro60#274
13Claude Haiku 4.554#580
14Gemini 2.5 Flash44#467

No result here: #1 Claude Fable 5.2, #2 GPT-6.2 Astra, #3 Gemini 4 Carbon.

Interactive version: theaggregate.ai/benchmark?slug=tau2-bench-verified-airline · How It Works · Data refreshed daily, snapshot 2026-11-17.