SUPER (NLP): leaderboard
Software Understanding and Performance Evaluation for Research: tests LLM agents on resolving real GitHub issues across NLP repositories.
Metric: Expert Accuracy (%). Source: huggingface.co. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | SWE-Agent + gpt-4o-2024-08-06 | 16.3 | |
| 2 | React-Super + gpt-4o-2024-08-06 | 14.4 | |
| 3 | React + gpt-4o-2024-08-06 | 12.2 | |
| 4 | React-Super + Llama 3.1 70B | 6.1 | |
| 5 | React-Super + gpt-4o-mini-2024-07-18 | 5.6 | |
| 6 | SWE-Agent + Llama 3.1 70B | 5.6 | |
| 7 | SWE-Agent + gpt-4o-mini-2024-07-18 | 3.3 | |
| 8 | React-Super + Mixtral-8x22B-Instruct-v0.1 | 3.3 | |
| 9 | SWE-Agent + Mixtral-8x22B-Instruct-v0.1 | 1.1 |
Interactive version: theaggregate.ai/benchmark?slug=super-nlp · How It Works · Data refreshed daily, snapshot 2026-10-20.