Ishigaki-IDS-Bench: leaderboard
Ishigaki-IDS-Bench (2026) asks a model to turn building information requirements, given in Japanese or English as CSV or prose, into a buildingSMART Information Delivery Specification (IDS 1.0) XML file, over 166 expert-authored examples. Facet F1 in percent: exact matches between generated and gold IDS facet blocks, macro-averaged over examples, with outputs that hold no extractable IDS scored as invalid; higher is better.
Metric: Facet F1 (%): F1 of exact matches between generated and gold IDS facet blocks (IFC version, entity, attribute and property facets with their applicability or requirement role and constraints), macro-averaged over examples, an output with no extractable IDS counting as invalid, on Ishigaki-IDS-Bench, 166 expert-authored examples (83 practical BIM information-requirement scenarios in Japanese and English, CSV or natural-language input, single- or multi-turn) whose output must be an IDS 1.0 XML file matching a gold IDS, zero-shot; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 8 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Setting | Score | Overall rank |
|---|---|---|---|---|
| 1 | GPT-5.5 | xHigh | 65.6 | #56 |
| 2 | Gemini 3.1 Pro (Preview) | 52.2 | #102 | |
| 3 | Kimi K2.6 | 48.4 | #197 | |
| 4 | Claude Opus 4.5 | 42.2 | #165 | |
| 5 | DeepSeek V4 Pro | 40 | #201 | |
| 6 | Qwen 3.5 397B A17B | 26.8 | #270 | |
| 7 | Ishigaki-IDS-Bench gpt-oss-120B xhigh (checkpoint unspecified) | 21.6 | ||
| 8 | Qwen 3 14B | 21.3 | #1284 |
No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Example tasks
IDS Version: IFC2X3
filename: row-0001.ids
建物全体情報についてチェックを行いたい。
- 建蔽率60%以下
- 容積率300%以下
- 最高高さ31m以下
IDS Version: IFC4X3_ADD2
filename: row-0004.ids
I want to check whether the sidewalk attribute information, which is the width and curb height, satisfies the municipality barrier-free ordinance conditions of "sidewalk effective width 2.0 m or more / curb height 0.02 m or less".
I want to check whether the following are within the range for the sidewalk:
- effective width: 2.0 m or more
- curb…
Interactive version: theaggregate.ai/benchmark?slug=ishigaki-ids-bench · How It Works · Data refreshed daily, snapshot 2026-11-22.