Ishigaki-IDS-Bench - Structure Validity: leaderboard

Ishigaki-IDS-Bench (2026) asks a model to turn building information requirements, given in Japanese or English as CSV or prose, into a buildingSMART Information Delivery Specification (IDS 1.0) XML file, over 166 expert-authored examples. Share of examples whose output is XML that validates against the IDS schema in the official IDS audit tool, in percent; higher is better.

Metric: Structure pass rate (%): share of examples whose output parses as XML and validates against the IDS XSD in IDSAuditTool (ids-tool v1.0.96), on Ishigaki-IDS-Bench, 166 expert-authored examples (83 practical BIM information-requirement scenarios in Japanese and English, CSV or natural-language input, single- or multi-turn) whose output must be an IDS 1.0 XML file matching a gold IDS, zero-shot; higher is better. Source: arxiv.org. Saturation forecast: Around 2029. 10 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Opus 4.545.8#160
2Kimi K2.641#192
3Gemini 3.1 Pro (Preview)37.3#99
4GPT-5.5xHigh30.1#54
5Qwen 3 8B20.5#1873
6Qwen 3.5 397B A17B20.5#264
7DeepSeek V4 Pro19.3#196
8Qwen 3 14B18.1#1269
9Qwen 3 32B13.9#972
10Ishigaki-IDS-Bench gpt-oss-120B xhigh (checkpoint unspecified)13.9

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=ishigaki-ids-bench-structure-validity · How It Works · Data refreshed daily, snapshot 2026-11-21.