R2ABench (Mini-SWE-Agent) - Completeness: leaderboard

Requirements-to-architecture generation on R2ABench (2026): from a structured requirements specification of 68 educational and GitHub projects, the model driving the Mini-SWE-Agent framework writes a PlantUML architecture view, compared with a human-validated reference. LLM-judge rating of how fully the view covers key requirements, external systems, data stores and constraints, over the views that parse, from 1 to 5; higher is better.

Metric: Completeness score (1-5): judge rating (GPT-5.4-mini, calibrated against human review) of how fully the view covers key functional requirements, significant requirements, external systems, data stores and constraints (L2), over the 68 R2ABench projects (17 educational and 51 GitHub repositories), generating a PlantUML architecture view from the structured requirements specification with the Mini-SWE-Agent framework under default settings and greedy decoding; rated over the L2-evaluable views, those that pass the L0 render-and-parse gate (the paper samples its judge calibration from the views that pass L0); higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 4 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1GPT-53.42#178
2Claude Sonnet 4.63.24#173
3DeepSeek V3.23.08#419
4Qwen 3 Coder 480B A35B2.66#709

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=r2abench-mini-swe-agent-completeness · How It Works · Data refreshed daily, snapshot 2026-11-21.