Did the model make working games?
Six new-game challenges test installation, controls, mechanics, game state, reset behavior, and stability.
Scoring without magic Build score Reproduce score Core score Experimental Official Human review
We test the rules. Humans judge the fun.
A game can be technically correct and still feel awful. It can also look great while breaking its own rules. GameBench keeps those judgments separate so every number means one understandable thing.
Could it remake an existing game?
Two reference-game challenges test behavioral fidelity and visual structure under the original license.
One balanced headline number
Build contributes 50%. Reproduce contributes 50%. Missing a whole track means no Core score—not a convenient zero-filled average.
From prompt to score
Five steps, all inspectable.
Same starting line
Each model gets the same starter project, task rules, time budget, fixed random condition (seed), and allowed network policy.
Real build
The submission must install, compile, start, and expose a valid game state. If that foundation fails, the task scores zero.
Browser plays
Automated tests use real keyboard and pointer input, advance time, reset the game, and inspect rule-relevant state.
Every game counts equally
No favorite game gets a secret bonus. Scores are averaged per game, then per track.
Clean rebuild
Before publication, private logs and credentials are removed. The clean source must rebuild and produce the same score.
People play separately
Blind human review can judge clarity, polish, controls, and feel, but it never changes the automatic mechanics score.
Official vs experimental
“Published” does not always mean “ranked.”
Fast, useful evidence
May use one reproducible sample, stop early, hit a provider limit, or lack independent operator verification. It is public but never mixed into the Official board.
The full championship run
Every game runs under all three fixed random conditions. Clean rebuilds, evidence, environment, and network isolation are verified.
A second opinion, not hidden weight
Human playtesting is shown beside machine scores, not folded into them. You can disagree with either.
Technical audit notes
The short version above is enough to understand a score. The details below exist for researchers and result auditors.
Version and identity rules
A Benchmark release freezes every task version and content hash, browser bridge protocol, scoring contract, seed policy, and aggregation rule. Scores from different releases are never ranked together.
Every physical execution has a unique run_id. Equivalent configurations and inputs have stable fingerprints, but reruns never overwrite history.
Deterministic browser contract
Tests combine native keyboard and pointer events with a public bridge that controls reset, time, actions, and state snapshots. The snapshot must report the active seed and match the task schema.
Dependency install, build, server start, bridge preflight, and the initial snapshot are hard gates. Any failure makes the complete task zero.
Official publication checklist
Official seeds are 104729, 130363, and 155921. The operator exports secret-free source, rebuilds in a digest-pinned evaluator image, recomputes the score, verifies the evidence manifest, and records network isolation.
A coding timeout is evaluated at the deadline snapshot. Agent and evaluator errors cannot fill an Official result cell.