MEGA-Bench Hanoi: verified references and an independent measurement

September 20, 2026 · Evaluation and verification by The Aggregate

Five of the 14 reference sequences are illegal. Correct optimal answers score 71.9388% against the published key; copying that key scores 100%. The first number is the score of correct behavior, not a cap on model scores. Hanoi remains included in our capability ratings and benchmark score measurements.

Download the verifier and saved answers (ZIP) · Saved model answers (JSON) · Benchmark page

Task and evidence

The MEGA-Bench dataset, configuration core, split test, contains items 2257–2270 for move_pos_to_pos_hanoi_4_pole. Despite its name, the pictures show three pegs and four disks. Filenames encode disk positions, largest disk first. The scorer averages exact move-count agreement and the length of the matching move-list prefix divided by the reference length.

Item 2270 starts with disks 0, 1 and 2 on peg 1 and the smallest disk, 3, on peg 2. Its first reference move, 2: 1->2, illegally places disk 2 on disk 3. Later moves also contradict earlier disk positions. A shortest legal solution is 2: 1->3, 3: 2->3, 1: 1->2, 3: 3->2.

The verifier tries 81 possible starts × two disk-index directions × six peg relabelings: 972 interpretations per reference. All five fail under every interpretation. The nine sound references pass under the plain reading. Trying every start already covers swapping the two images; swapping is not another factor.

Independent Gemini measurement

The saved run of google/gemini-3.8-flash scored 58.2483%: all 14 answers were legal and reached the goal, and 12 were shortest solutions. All five puzzles with invalid references were solved optimally. Items 2262 and 2264 used 13 moves where 12 suffice.

The Aggregate ran this evaluation through OpenRouter on September 20, 2026, at temperature 0 with provider-default thinking and a 32,000-token output limit. The prompt follows MEGA-Bench’s one-shot OpenAI runner structure, including the example and query images, with no system prompt. The response parser is vendored unchanged; the relevant prefix and exact-match metrics are reimplemented. This is one run, without a variance estimate.

This measurement was made by The Aggregate, not TIGER-Lab. The official board’s maximum was 10.7% among 44 models when checked; its roster includes 2025 models. The independent row enters our pooled board at the next audited data publication. A source’s presence here does not mean TIGER-Lab published that row.

Reproduce locally

Extract the ZIP, then run:

python3 verify_answer_key.py

This command uses only Python’s standard library and makes no network or model calls. It prints the legality checks, the 71.9388% correct-answer score and the 100% key-copy score. The bundle also includes the saved responses and a measurement verifier; see its README for the optional parser dependency.

The raw scores are preserved. Reference errors do not penalize every model equally and do not automatically justify excluding a benchmark. This evidence concerns this task only; sibling tasks were not checked. One modern run does not establish score compression across multiple frontier model families.

Sources: dataset and images; pinned MEGA-Bench code. The download includes attribution and the upstream Apache 2.0 license.