CompliBench - Airline: leaderboard

CompliBench (2026) evaluates LLMs as judges of guideline compliance in task-oriented dialogue: synthesized multi-turn conversations carry injected, adversarially optimized violations, and the judge must name the governing guideline and violation status of every assistant turn. This board is conversation-level accuracy in percent in the airline domain, the share of conversations judged correctly at every turn, averaged over four runs; higher is better.

Metric: Conversation-level accuracy (%): share of airline conversations in which the judge predicts both the governing guideline and the violation label correctly at every assistant turn, of an LLM judge on CompliBench's synthesized multi-turn dialogues with injected, adversarially optimized guideline violations, mean of four runs at default reasoning effort; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 15 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Gemini 3 Pro49.09#145
2GPT-547.26#178
3DeepSeek V3.2Thinking42.07#419
4Qwen 3.5 Plus41.16#242
5GLM-531.4#269
6Kimi K2.525.91#266
7Qwen 3 Max24.39#415
8Qwen 3 30B A3B21.95#1156
9Claude Sonnet 4.621.18#173
10GPT-4o17.07#751
11Qwen 3 32B8.54#972
12Qwen 3 4B3.66#2348
13GPT-4o Mini2.74#1667
14Qwen 3 14B2.13#1269
15Qwen 3 8B1.22#1873

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=complibench-airline · How It Works · Data refreshed daily, snapshot 2026-11-21.