PM-Bench (Auto Heartbeat 30m): leaderboard
PM-Bench tests prospective memory in agents: over a simulated week the agent must remember intentions, notice the time or event that makes each one due, act then and ignore lures, while hidden state channels reveal cues only when queried. This board switches a monitoring heartbeat on every thirty virtual minutes. The score is the set-level F1 between the actions the agent selects and the actions actually due.
Metric: Set-F1 (%; F1 of the prospective actions the agent selects against the actions actually due, accumulated over the 80 steps and 81 scored tasks of the released simulated week, lures included; single agent with a heartbeat switched on every 30 virtual minutes; one run per model). Source: arxiv.org. Saturation forecast: Around February 2027. 8 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Mistral Large 3 | 74.7 | #817 |
| 2 | GPT-5.4 | 74.1 | #123 |
| 3 | GPT-5.3 Codex | 71 | #116 |
| 4 | Llama 3.3 70B Instruct | 63.4 | #1357 |
| 5 | Qwen 3 32B | 57.5 | #960 |
| 6 | Mistral Small 3.2 | 51.6 | #1591 |
| 7 | Qwen 3 8B | 39.8 | #1830 |
| 8 | Qwen 3 14B | 30.3 | #1233 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=pm-bench-auto-heartbeat-30m · How It Works · Data refreshed daily, snapshot 2026-11-13.