CausalDS - Pass Rate: leaderboard
CausalDS (2026) generates narrated scenes from hidden structural causal models with observational data files and asks data-science agents associational, interventional and counterfactual questions that sometimes admit no identifiable answer; the 100-scene realistic exam is solved in the mini-swe-agent harness. This board is the pass rate on the discrete slice, which pools binary content answers and abstention decisions (34 to 38 tasks per model, since a wrongful abstention is graded as a binary), missing answers counted as failures, in percent; higher is better.
Metric: Pass rate (%; share correct on the discrete slice, which pools binary content answers and abstention decisions; a task is graded as an abstention binary when the truth is non-identifiable or the model abstains, so the slice holds 34 to 38 tasks per model; missing answers count as failures; on a 100-scene realistic exam of Pearl-ladder causal data-science tasks (28 associational, 51 interventional, 21 counterfactual) over SCM-generated data files, solved in the mini-swe-agent harness with at most 100 steps). Source: arxiv.org. Saturation forecast: Around February 2027. 6 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.5 (High) | 82.4 | #49 (GPT-5.5) |
| 2 | Claude Opus 4.8 (High) | 82.4 | #56 (Claude Opus 4.8) |
| 3 | Gemini 3.1 Pro (Preview) | 76.5 | #94 |
| 4 | Kimi K2.6 | 65.7 | #178 |
| 5 | Qwen 3.6 35B A3B | 63.2 | #462 |
| 6 | Gemma 4 26B | 55.9 | #612 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=causalds-pass-rate · How It Works · Data refreshed daily, snapshot 2026-11-13.