MEGA-Bench Hanoi verification — The Aggregate, 2026-09-20

Source: https://huggingface.co/datasets/TIGER-Lab/MEGA-Bench
Task: core/test, move_pos_to_pos_hanoi_4_pole, items 2257–2270.
rows.json is the saved public dataset response. query_media filenames encode
disk positions largest-first; image bytes are available in the upstream data.zip.

After extracting this ZIP, run:
    python3 verify_answer_key.py
This uses only the Python standard library, performs no network calls, and prints
the 972-interpretation checks and correct-answer/key-copy scores.

To reproduce the saved Gemini measurement, install the response parser's regex
dependency in your Python environment, then run:
    python3 verify_measurement.py
This only reads saved answers; it does not call any model or provider.

responses.json contains the original answer text and model identifier only.
The Aggregate ran google/gemini-3.8-flash through OpenRouter on 2026-09-20,
temperature 0, provider-default thinking, maximum 32,000 output tokens, with
MEGA-Bench's one-shot OpenAI runner prompt structure (example and query images,
no system prompt). This independent evaluation is not a TIGER-Lab result.
Expected: task score 58.2483%; 14/14 legal, goal-reaching answers; 12/14 optimal.
Correct optimal answers score 71.9388%; copying the incorrect key scores 100%.
71.9388% is not a universal score cap. Hanoi remains included in our ratings.

Attribution: vendor/metrics/parsing/common/{parsers,utils}.py are unmodified
MEGA-Bench files, TIGER-AI-Lab/MEGA-Bench commit
90189587ec38576002bb3202c56975523b9df25c, under the included Apache 2.0 LICENSE.
scorer.py reimplements the relevant prefix and exact-match metrics; solver.py,
verify_answer_key.py and verify_measurement.py are The Aggregate's verification
code, provided here to reproduce the findings in this report.

Public report: https://theaggregate.ai/evidence/megabench-hanoi-2026-09-20.html
