CompliBench - Insurance: leaderboard

CompliBench (2026) evaluates LLMs as judges of guideline compliance in task-oriented dialogue: synthesized multi-turn conversations carry injected, adversarially optimized violations, and the judge must name the governing guideline and violation status of every assistant turn. This board is conversation-level accuracy in percent in the insurance domain, the share of conversations judged correctly at every turn, averaged over four runs; higher is better.

Metric: Conversation-level accuracy (%): share of insurance conversations in which the judge predicts both the governing guideline and the violation label correctly at every assistant turn, of an LLM judge on CompliBench's synthesized multi-turn dialogues with injected, adversarially optimized guideline violations, mean of four runs at default reasoning effort; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 15 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Gemini 3 Pro51.3#145
2Qwen 3.5 Plus40.43#242
3DeepSeek V3.2Thinking33.91#419
4GLM-531.09#269
5GPT-530#178
6Claude Sonnet 4.628.75#173
7Qwen 3 30B A3B26.52#1156
8Qwen 3 Max25.65#415
9Kimi K2.517.61#266
10GPT-4o11.3#751
11Qwen 3 32B8.91#972
12Qwen 3 14B3.26#1269
13GPT-4o Mini2.17#1667
14Qwen 3 8B1.74#1873
15Qwen 3 4B1.09#2348

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=complibench-insurance · How It Works · Data refreshed daily, snapshot 2026-11-21.