CausalDS: leaderboard
CausalDS (2026) generates narrated scenes from hidden structural causal models with observational data files and asks data-science agents associational, interventional and counterfactual questions that sometimes admit no identifiable answer; the 100-scene realistic exam is solved in the mini-swe-agent harness. The CausalDS score sums one minus the pass rate, a relative numeric error and a median F1 loss, weighted by task counts; 0 is perfect and lower is better.
Metric: CausalDS score (0 is perfect, lower is better, unbounded above; task-count-weighted sum of one minus the pass rate, the relative numeric error S_NR and the median F1 loss, on a 100-scene realistic exam of Pearl-ladder causal data-science tasks (28 associational, 51 interventional, 21 counterfactual) over SCM-generated data files, solved in the mini-swe-agent harness with at most 100 steps). Source: arxiv.org. Saturation forecast: Around August 2027. 6 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 4.8 (High) | 0.28 | #56 (Claude Opus 4.8) |
| 2 | Gemini 3.1 Pro (Preview) | 0.37 | #94 |
| 3 | Qwen 3.6 35B A3B | 0.45 | #462 |
| 4 | Kimi K2.6 | 0.48 | #178 |
| 5 | GPT-5.5 (High) | 0.56 | #49 (GPT-5.5) |
| 6 | Gemma 4 26B | 0.64 | #612 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=causalds · How It Works · Data refreshed daily, snapshot 2026-11-13.