PM-Bench (Optional Heartbeat): leaderboard
PM-Bench tests prospective memory in agents: over a simulated week the agent must remember intentions, notice the time or event that makes each one due, act then and ignore lures, while hidden state channels reveal cues only when queried. This board lets the agent switch on a periodic monitoring heartbeat itself. The score is the set-level F1 between the actions the agent selects and the actions actually due.
Metric: Set-F1 (%; F1 of the prospective actions the agent selects against the actions actually due, accumulated over the 80 steps and 81 scored tasks of the released simulated week, lures included; single agent that may switch on a 30- or 60-minute periodic monitoring heartbeat; one run per model). Source: arxiv.org. Saturation forecast: Around February 2027. 8 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.4 | 79.1 | #123 |
| 2 | GPT-5.3 Codex | 76.6 | #116 |
| 3 | Mistral Large 3 | 73.7 | #817 |
| 4 | Qwen 3 8B | 71.9 | #1830 |
| 5 | Llama 3.3 70B Instruct | 68.1 | #1357 |
| 6 | Mistral Small 3.2 | 53 | #1591 |
| 7 | Qwen 3 14B | 50 | #1233 |
| 8 | Qwen 3 32B | 48.5 | #960 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=pm-bench-optional-heartbeat · How It Works · Data refreshed daily, snapshot 2026-11-13.