SAGE (Service Agent) - Airline Refund: leaderboard

Metric: Overall Assessment Score (0-100), 0.8 x logical compliance + 0.2 conversational quality, in the Airline Refund scenario (airline refunds under rigid time-sensitive cancellation-fee policies), over SAGE customer-service dialogues: standard operating procedures formalized as dialogue graphs, an adversarial simulated user with varied intents and personas, turns 1, 5, 10, 15 and the final turn scored; three unnamed judge LLMs (majority vote and mean) plus a deterministic rule engine produce the reference; closed models through their APIs, open models served with vLLM; 0-100 scale; higher is better. Source: arxiv.org. Saturation forecast: Around 2030. 27 models tracked.

Top models

Interactive version: theaggregate.ai/benchmark?slug=sage-service-agent-airline-refund · How It Works · Data refreshed daily, snapshot 2026-10-07.