Cached Trace Embeddings - Raw Trace - Correctness: leaderboard
Held-out linear-probe task and correctness prediction from trace embeddings. Cached Traces v1 Table 2 measures balanced linear-probe accuracy for predicting task identity and correctness from cached SWE-bench Verified trace embeddings. Evaluation holds out groups of systems. Seven encoders are compared separately for raw trace, generic and trubric representations. Raw trace embeds its beginning and end; generic and trubric use six fields with DeepSeek-V4-Flash as rubric generator and summarizer. Scores stay on the native 0–1 scale. System, family and harness leakage columns are nuisance-information diagnostics. Estimator MAEs do not measure model capability, and Appendix SWE-bench/Terminal-Bench scores are reused official observations rather than new experiments. Question access: Primary result reviewed; exact scored release availability not confirmed.
Metric: Balanced accuracy (fraction). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 7 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | text-embedding-3-small (Benchmark Sweep Run: source=CachedTraceEmbeddings; representation=Raw Trace; held-out system groups; trained linear probe) | 0.82 | |
| 2 | nomic-embed-text-v1.5 (Benchmark Sweep Run: source=CachedTraceEmbeddings; representation=Raw Trace; held-out system groups; trained linear probe) | 0.78 | |
| 3 | bge-large-en-v1.5 (Benchmark Sweep Run: source=CachedTraceEmbeddings; representation=Raw Trace; held-out system groups; trained linear probe) | 0.78 | |
| 4 | gte-large (Benchmark Sweep Run: source=CachedTraceEmbeddings; representation=Raw Trace; held-out system groups; trained linear probe) | 0.77 | |
| 5 | e5-large-v2 (Benchmark Sweep Run: source=CachedTraceEmbeddings; representation=Raw Trace; held-out system groups; trained linear probe) | 0.77 | |
| 6 | all-mpnet-base-v2 (Benchmark Sweep Run: source=CachedTraceEmbeddings; representation=Raw Trace; held-out system groups; trained linear probe) | 0.77 | |
| 7 | all-MiniLM-L6-v2 (Benchmark Sweep Run: source=CachedTraceEmbeddings; representation=Raw Trace; held-out system groups; trained linear probe) | 0.75 |
Interactive version: theaggregate.ai/benchmark?slug=cached-trace-embeddings-raw-trace-correctness · How It Works · Data refreshed daily, snapshot 2026-11-06.