Ishigaki-IDS-Bench: leaderboard

Ishigaki-IDS-Bench (2026) asks a model to turn building information requirements, given in Japanese or English as CSV or prose, into a buildingSMART Information Delivery Specification (IDS 1.0) XML file, over 166 expert-authored examples. Facet F1 in percent: exact matches between generated and gold IDS facet blocks, macro-averaged over examples, with outputs that hold no extractable IDS scored as invalid; higher is better.

Metric: Facet F1 (%): F1 of exact matches between generated and gold IDS facet blocks (IFC version, entity, attribute and property facets with their applicability or requirement role and constraints), macro-averaged over examples, an output with no extractable IDS counting as invalid, on Ishigaki-IDS-Bench, 166 expert-authored examples (83 practical BIM information-requirement scenarios in Japanese and English, CSV or natural-language input, single- or multi-turn) whose output must be an IDS 1.0 XML file matching a gold IDS, zero-shot; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 8 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1GPT-5.5xHigh65.6#56
2Gemini 3.1 Pro (Preview)52.2#102
3Kimi K2.648.4#197
4Claude Opus 4.542.2#165
5DeepSeek V4 Pro40#201
6Qwen 3.5 397B A17B26.8#270
7Ishigaki-IDS-Bench gpt-oss-120B xhigh (checkpoint unspecified)21.6
8Qwen 3 14B21.3#1284

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Example tasks

IDS Version: IFC2X3
filename: row-0001.ids
建物全体情報についてチェックを行いたい。
- 建蔽率60%以下
- 容積率300%以下
- 最高高さ31m以下

Source

IDS Version: IFC4X3_ADD2
filename: row-0004.ids
I want to check whether the sidewalk attribute information, which is the width and curb height, satisfies the municipality barrier-free ordinance conditions of "sidewalk effective width 2.0 m or more / curb height 0.02 m or less".
I want to check whether the following are within the range for the sidewalk:
- effective width: 2.0 m or more
- curb…

Source

Interactive version: theaggregate.ai/benchmark?slug=ishigaki-ids-bench · How It Works · Data refreshed daily, snapshot 2026-11-22.