RepoRef: leaderboard
RepoRef (2026) tests conversational reference grounding: an agent reads one of 400 real developer chat segments from Gitter and must use read-only GitHub tools to find the issue, pull request or commit that an indirect reference in the chat points to. Success rate in percent, an exact match of the submitted URL with the target under a budget of ten tool calls; higher is better.
Metric: Success rate (%; exact match of the submitted GitHub issue, pull request or commit URL against the hidden target over the 400 developer-chat segments from 92 repositories, each with one indirect reference to resolve; a ReAct agent with 22 read-only GitHub tools and a budget of 10 tool calls, provider-default model settings, or the Claude Code harness through a read-only GitHub MCP server). Source: arxiv.org. Saturation forecast: Around February 2027. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3 Flash (Preview) | 67 | #141 |
| 2 | Claude Opus 4.7 (Medium) | 63.25 | #75 (Claude Opus 4.7) |
| 3 | DeepSeek V4 Pro | 60.25 | #181 |
| 4 | Grok 4.1 Fast (Reasoning) | 54 | #426 (Grok 4.1 Fast) |
| 5 | GLM-5.1 | 52 | #175 |
| 6 | Claude Sonnet 4.6 | 51.25 | #158 |
| 7 | GPT-5 Mini | 49 | #349 |
| 8 | Gemini 2.5 Flash Lite | 4 | #958 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=reporef · How It Works · Data refreshed daily, snapshot 2026-11-12.