CompliBench - Healthcare: leaderboard

CompliBench (2026) evaluates LLMs as judges of guideline compliance in task-oriented dialogue: synthesized multi-turn conversations carry injected, adversarially optimized violations, and the judge must name the governing guideline and violation status of every assistant turn. This board is conversation-level accuracy in percent in the healthcare domain, the share of conversations judged correctly at every turn, averaged over four runs; higher is better.

Metric: Conversation-level accuracy (%): share of healthcare conversations in which the judge predicts both the governing guideline and the violation label correctly at every assistant turn, of an LLM judge on CompliBench's synthesized multi-turn dialogues with injected, adversarially optimized guideline violations, mean of four runs at default reasoning effort; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 15 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Gemini 3 Pro57.11#146
2DeepSeek V3.2Thinking39.68#425
3Claude Sonnet 4.639.41#179
4Qwen 3.5 Plus36.47#250
5GLM-533.26#276
6GPT-528.44#185
7Qwen 3 Max25.69#420
8Qwen 3 30B A3B25#1165
9Kimi K2.522.94#272
10GPT-4o17.66#754
11Qwen 3 32B8.49#983
12Qwen 3 14B5.5#1284
13Qwen 3 8B1.83#1890
14Qwen 3 4B1.83#2379
15GPT-4o Mini0.23#1680

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=complibench-healthcare · How It Works · Data refreshed daily, snapshot 2026-11-22.