CompliBench - Airline - Violation Detection: leaderboard

CompliBench (2026) evaluates LLMs as judges of guideline compliance in task-oriented dialogue: synthesized multi-turn conversations carry injected, adversarially optimized violations, and the judge must name the governing guideline and violation status of every assistant turn. This board is violation detection accuracy in percent in the airline domain, the share of violating turns the judge flags; higher is better.

Metric: Violation detection accuracy (%): share of violating assistant turns in the airline domain that the judge flags as violations, of an LLM judge on CompliBench's synthesized multi-turn dialogues with injected, adversarially optimized guideline violations, mean of four runs at default reasoning effort; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 15 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Gemini 3 Pro90.61#145
2Kimi K2.586.88#266
3GLM-586.46#269
4DeepSeek V3.2Thinking84.81#419
5Qwen 3.5 Plus82.32#242
6Claude Sonnet 4.681.76#173
7GPT-581.08#178
8Qwen 3 Max71.41#415
9Qwen 3 30B A3B69.75#1156
10GPT-4o50.55#751
11Qwen 3 32B41.44#972
12Qwen 3 14B32.04#1269
13Qwen 3 8B28.18#1873
14Qwen 3 4B24.86#2348
15GPT-4o Mini19.61#1667

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=complibench-airline-violation-detection · How It Works · Data refreshed daily, snapshot 2026-11-21.