The GameEngineBench paper tells readers that "the current repository already contains the key scripts required to reproduce benchmark runs" and points to github.com/Nitrode-Research/GameEngineBench. That address returns 404. Nitrode's GitHub organization shows only one public repository, "unrealbench" (14 Unreal tasks, 1 star), which doesn't mention the paper or nitrode.com. The research page itself links only to arXiv, with no code, download or submission route. Nitrode's YC launch promised to open-source a suite of benchmarks. The researchers and lab engineers most likely to become buyers are the people who click that link, and right now they hit a dead end.
First action
Decide today whether the cited repo should hold the full scripts or a pointer to unrealbench, and make that URL load.
How to measure it
Benchmark-sourced lab and studio conversations
- Now
- –
- Target
- 10 qualified conversations
- By
- 90 days
Steps
Fix the cited URL
Publish at github.com/Nitrode-Research/GameEngineBench, or create that repo with a README that points to the public release.
Member of Technical Staff, Evals 1 day
Write the README
What the benchmark measures, the public and held-out task split, hardware and Epic access needs, a quick start, the license and a citation.
Member of Technical Staff, Evals 2 days
Link both ways
Repo to /research and the paper; /research to the repo with a "Run GameEngineBench" block.
Full-Stack Engineer 1 day
Update arXiv
Post a new version with the working code link.
CTO 1 week
Add a scoring route
A "Get your model scored" link to the contact form, pre-set to Research / GameEngineBench.
Full-Stack Engineer 2 hours
Announce
One post on X and LinkedIn with the repo, the 31 unsolved tasks and how to try them.
CEO 1 day
Track
Repo traffic, clones, stars and tagged leads each week.
COO weekly
Why it works
Publish the reproduction scripts at the exact URL the paper cites, or, if unrealbench is the public release, rename it or put a README at the cited address that points to it. Add a "Run GameEngineBench" block to /research: repo link, what is public (tasks, starter projects, verifiers) and what stays held out, hardware needs, a BibTeX citation and a "get your model scored" link to the contact form. Update the arXiv listing with a new version that carries the working link, and link the repo README back to nitrode.com/research and the paper. Then track repo traffic, clones and contact-form leads tagged "Research / GameEngineBench" each week.
Benchmarks spread through people running them. A working repo turns each paper reader into a possible user, citer or contributor, and those users are the evals and post-training teams at labs who buy datasets and RL environments. It also keeps the open-source promise, which builds trust with researchers.
Ready to use
ChecklistBenchmark public-home checklist
[ ] github.com/Nitrode-Research/GameEngineBench loads [ ] README: what it measures, public vs held-out split, hardware, Epic access, quick start [ ] License and NOTICE [ ] BibTeX citation [ ] Repo links to nitrode.com/research and arXiv [ ] /research links to the repo [ ] "Get your model scored" link to /contact [ ] arXiv version with the working link [ ] Weekly repo and lead tracking
Risks and how to handle them
Releasing all tasks lets models train on the test set
Publish scripts and a public split, and keep a held-out split for official scores, as unrealbench already does with solutions.
Engine licensing limits what can be shared
Keep the NOTICE file and require users to bring their own Epic-linked access, as unrealbench does.
Evidence
- Research paperGameEngineBench paper (arXiv, submitted July 3, 2026, revised July 15, 2026; eight authors): tasks from "nine open-source Unreal projects" such as ActionRoguelike, GASShooter and Bomber; Appendix D points readers to github.com/Nitrode-Research/GameEngineBench and says "The current repository already contains the key scripts required to reproduce benchmark runs." It also compares cost and speed across configurations.Source
- GitHubgithub.com/Nitrode-Research/GameEngineBench returns 404 (checked twice on November 9, 2026). The organization's only public repository is "unrealbench": 14 Unreal Engine coding tasks for agents on Harbor, Apache-2.0, with instructions, starter projects and verifiers public; 1 star, 0 forks, and no link to nitrode.com or the paper.Source
- Your websiteGameEngineBench page: "110 UE5 C++ tasks", "9 source projects", "12 model configurations evaluated", "55.5% best pass rate"; every other configuration scores under 30% and 31 tasks are unsolved by all; failure analysis by category (save/persistence, replication, actor lifecycle). The leaderboard rows are not in the page text, only the top score, and the only outbound links are the arXiv paper, X and LinkedIn: no code, dataset download or submission route.Source
- Launch YCLaunch post "Nitrode - Advancing AI for Game Development": Nitrode "will open-source a suite of benchmarks" over the coming months; Mint connects game development experts (engine programmers, technical artists) and has "hundreds of contributors"; the ask is companies that need custom datasets or evals, and introductions to creative or gaming companies; contact [email protected]; model rankings track spatial reasoning benchmarks more closely than traditional agentic coding benchmarks.Source