llmbenchmarks.io Local GPU - SciCode: leaderboard

SciCode on local hardware: 65 research coding problems from the natural sciences, split into 288 subproblems; the model writes a Python function for each step and hidden tests check it. The run follows the official SciCode harness (commit e3158ea) with Artificial Analysis's multi-step prompt template and the scientist-annotated background in the prompt, three epochs and a 300-second test timeout; the three steps the official harness pre-fills with reference code are pre-filled here too. Score: subproblems solved (%), pass@1 averaged over the three epochs; this is the subproblem rate, not the much lower main-problem rate. Runs before 2026-10-25, when the test image gained matplotlib, are not read. llmbenchmarks.io runs this on its own hardware (2026): open-weight models, mostly 4-bit and 8-bit quantised checkpoints (NVFP4, AWQ, QAT) pinned to a Hugging Face repository and revision, served by vLLM 0.29.0 on one consumer machine (2x RTX 5060 Ti, 32 GB VRAM, tensor parallel 2), each with thinking on or off. Unlike vendor or leaderboard scores for the same test, a row measures the quantised deployment a home user would run, so a quantised checkpoint is its own model, not the full-precision original. Labels carry the mode as (Thinking) or (Non-thinking); Llama 3.1 rows (no thinking mode) and Ministral 3 14B Reasoning rows (always reasoning) stay bare. A run on a subset of the test is not read. Higher is better. Source: https://llmbenchmarks.io/local-gpu-benchmarks/

Metric: Subproblems solved (%), pass@1 averaged over 3 epochs. Source: llmbenchmarks.io. Saturation forecast: Around 2028. 12 models tracked.

Top models

No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.

Interactive version: theaggregate.ai/benchmark?slug=llmbenchmarks-io-local-gpu-scicode · How It Works · Data refreshed daily, snapshot 2026-11-05.