SAGE (Service Agent) - Airline Refund: leaderboard

SAGE overall assessment score in the airline refund customer-service scenario: airline refunds under rigid time-sensitive cancellation-fee policies, combining procedure compliance and conversational quality on a 0-100 scale, higher is better. It is unrelated to the SAGE robustness benchmark.

Metric: Overall Assessment Score (0-100), 0.8 x logical compliance + 0.2 conversational quality, in the Airline Refund scenario (airline refunds under rigid time-sensitive cancellation-fee policies), over SAGE customer-service dialogues: standard operating procedures formalized as dialogue graphs, an adversarial simulated user with varied intents and personas, turns 1, 5, 10, 15 and the final turn scored; three unnamed judge LLMs (majority vote and mean) plus a deterministic rule engine produce the reference; closed models through their APIs, open models served with vLLM; 0-100 scale; higher is better. Source: arxiv.org. Saturation forecast: Around 2029. 27 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Claude Opus 4.570.12#165
2Claude Sonnet 4.566.52#267
3MiniMax-M2.166.37#582
4DeepSeek V3.265.84#425
5Kimi K2.564.45#272
6Gemini 3 Pro (Preview)64.45#123
7DeepSeek R164.03#484
8Qwen Max (SAGE checkpoint unspecified)63.85
9Qwen 3 14B63.78#1284
10GPT-4.163.68#506
11Qwen 2.5 14B Instruct62.96#1788
12Llama 3.3 70B Instruct62.93#1394
13Gemini 3 Flash (Preview)62.4#156
14Qwen 3 235B A22B62.37#633
15Seed 1.862.19#258

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=sage-service-agent-airline-refund · How It Works · Data refreshed daily, snapshot 2026-11-22.