RepoRef: leaderboard
Metric: Success rate (%; exact match of the submitted GitHub issue, pull request or commit URL against the hidden target over the 400 developer-chat segments from 92 repositories, each with one indirect reference to resolve; a ReAct agent with 22 read-only GitHub tools and a budget of 10 tool calls, provider-default model settings, or the Claude Code harness through a read-only GitHub MCP server). Source: arxiv.org. Saturation forecast: Around December 2026. 9 models tracked.
Top models
| # | Model | Score |
|---|---|---|
| 1 | Gemini 3 Flash (Preview) | 67 |
| 2 | DeepSeek V4 Pro | 60.25 |
| 3 | Grok 4.1 Fast (Reasoning) | 54 |
| 4 | GLM-5.1 | 52 |
| 5 | Claude Sonnet 4.6 | 51.25 |
| 6 | GPT-5 Mini | 49 |
| 7 | Gemini 2.5 Flash Lite | 4 |
Interactive version: theaggregate.ai/benchmark?slug=reporef · How It Works · Data refreshed daily, snapshot 2026-09-26.