Ishigaki-IDS-Bench - Content Compliance: leaderboard

Ishigaki-IDS-Bench (2026) asks a model to turn building information requirements, given in Japanese or English as CSV or prose, into a buildingSMART Information Delivery Specification (IDS 1.0) XML file, over 166 expert-authored examples. Share of examples whose output passes the official IDS audit tool checks of the IDS standard and IFC vocabulary, in percent; higher is better.

Metric: Content pass rate (%): share of examples whose output passes IDSAuditTool checks of the IDS standard and IFC vocabulary constraints, on Ishigaki-IDS-Bench, 166 expert-authored examples (83 practical BIM information-requirement scenarios in Japanese and English, CSV or natural-language input, single- or multi-turn) whose output must be an IDS 1.0 XML file matching a gold IDS, zero-shot; higher is better. Source: arxiv.org. Saturation forecast: Around 2030. 10 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Opus 4.533.1#165
2GPT-5.5xHigh27.7#56
3Gemini 3.1 Pro (Preview)25.9#102
4Kimi K2.622.3#197
5Qwen 3.5 397B A17B19.3#270
6DeepSeek V4 Pro16.3#201
7Qwen 3 8B15.7#1890
8Qwen 3 14B13.9#1284
9Ishigaki-IDS-Bench gpt-oss-120B xhigh (checkpoint unspecified)12
10Qwen 3 32B10.2#983

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=ishigaki-ids-bench-content-compliance · How It Works · Data refreshed daily, snapshot 2026-11-22.