TeleResilienceBench - 3GPP TSG: leaderboard

TeleResilienceBench (2026) on its 3GPP TSG subset (classifying 3GPP documents by working group): the model sees the question, the options and the first half of a flawed reasoning trace that led a weak generator to a wrong answer, and must continue it to the correct option. Score is the percent of the 90 instances recovered (correct flip rate); higher is better.

Metric: Correct flip rate (CFR, %): share of the 90 3GPP TSG instances (classifying 3GPP documents by working group, GSMA Open-Telco ot-lite) where the model, given the question, the options and the first half of a flawed reasoning trace from qwen3.5:2b that ended in a wrong answer, continues the reasoning in its thinking stream and returns the correct option; one continuation prompt, Ollama builds on one RTX 4090; higher is better. Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 8 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1gemma4:31B (Ollama)16.7
2gemma4:26B (Ollama)16.7
3nemotron-3-nano:4B (Ollama)11.1
4gemma4:e2B (Ollama)5.6
5gemma4:e4B (Ollama)5.6
6qwen3.5:9B (Ollama)5.6
7qwen3.5:27B (Ollama)4.4
8qwen3.5:4B (Ollama)3.3

Interactive version: theaggregate.ai/benchmark?slug=teleresiliencebench-3gpp-tsg · How It Works · Data refreshed daily, snapshot 2026-11-21.