RuBench (23-Task Freshness Gate): leaderboard

Metric: Pass@1 (%; the 23 tasks whose fix commits postdate the model training cutoff of these configurations; raw rows as ranked by the paper; pass@1, mean share of tasks resolved over three independent runs, judged by the withheld upstream maintainer regression tests; 25 repository-level fix tasks from aiohttp, aiogram, Laravel, NestJS and Fastify with task statements natively written in Russian; deployed agent CLI, model and reasoning effort as one configuration, product defaults otherwise, network access allowed). Source: arxiv.org. Saturation forecast: Around December 2026. 4 models tracked.

Top models

Interactive version: theaggregate.ai/benchmark?slug=rubench-23-task-freshness-gate · How It Works · Data refreshed daily, snapshot 2026-09-29.