{"id":"45a77135-b6d7-495e-88bd-30fb71797a1f","arxiv_id":"2607.03525","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Coding agents evaluated on 110 Unreal Engine C++ implementation tasks reach at most 55.5% pass@1, with 31 tasks unsolved by all 12 configurations.","lead":"GameEngineBench gives AI coding agents 110 real C++ implementation tasks inside Unreal Engine 5 game projects drawn from nine open-source repositories. The best of twelve agent configurations passes only 55.5% of tasks, and 31 tasks defeat all of them — a sign that runtime-integrated game-engine coding remains unsolved for frontier models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline pass@1 numbers rest on an unvalidated LLM judge; without judge-reliability evidence the leaderboard and failure-mode claims are provisional.","rationale":"The reader's weakest-assumption analysis correctly identifies the LLM judge as the load-bearing measurement premise. The paper's own Section 7 admits that judge review can override test failures, yet no evidence is provided that judge decisions are reliable or unbiased. This is especially important because the benchmark's contribution is not just a set of scores but a qualitative conclusion: failures are behavioral/integration errors, not syntax errors. That conclusion is derived from judge-labeled outcomes. The absence of judge-family reporting matters because family bias would not only inflate scores but also systematically distort cross-model comparisons. I do not see a more fundamental problem. The benchmark design is otherwise careful: withheld tests, preserved workspaces, PIE automation, task calibration, and a reproducible CLI are genuine strengths. The wrapper effect and contamination concerns are real but secondary; wrapper effects are explicitly acknowledged, and contamination would likely inflate rather than invalidate the \"struggle\" narrative. The correct verdict is CONDITIONAL: the benchmark's directional finding is plausible, but the precise leaderboard and failure taxonomy should be treated as provisional until judge reliability is quantified. I agree with the reader's identification and would not change the verdict; the concrete test above would either resolve the concern or require downgrading the headline claims.","tokens_in":14715,"tokens_out":1841,"duration_ms":22557,"concrete_test":"Select a stratified random sample of 30–50 runs spanning configurations, pass/fail outcomes, and task areas. Have two human Unreal Engine developers independently re-score each run as pass/fail using only the behavior spec, test results, workspace diff, and runtime logs, blinded to model identity and to the original judge decision. Compute inter-judge agreement and human-vs-LLM-judge agreement (e.g., Cohen's kappa). Then recompute the leaderboard order and the 31-task unsolved count from the human labels. If the LLM judge agrees with humans on the sampled runs and the leaderboard ordering is preserved, the measurement premise is supported; if kappa is low or the ordering changes materially, the headline results cannot be trusted as reported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that GameEngineBench produces a valid, non-saturated leaderboard and that failures are predominantly runtime-integration errors rather than syntax/compilation problems—depends on the accuracy and impartiality of the LLM judge. Section 5 defines success as \"the LLM judge determines that the generated implementation satisfies the requested behavior.\" The paper reports no judge-versus-human agreement statistics, no per-run judge family, and no cross-family agreement analysis. Section 7 concedes that LLM review can convert some test failures into passes, so the judge is not merely a filter on test outcomes; it is the final arbiter of correctness. If the judge is systematically lenient, or biased toward outputs from the same model family (the paper says cross-family judging is \"supported\" but does not say it was used), then the pass@1 ordering, the 55.5% headline, the 31-task unsolved set, and the Figure 5 failure decomposition all shift. This is a measurement-validity issue, not a mere implementation detail: the benchmark's main conclusions are expressed in the units of judge decisions. The paper also provides no error bars around pass@1 estimates, and the wrapper confound is self-admitted in §7, but the judge question is the most load-bearing because it affects every reported number and the qualitative failure taxonomy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GameEngineBench, a benchmark of 110 C++ implementation tasks drawn from nine open-source Unreal Engine 5 projects, and evaluates 12–13 frontier coding-agent configurations on them. Each task provides a buildable starting state, a list of editable C++ files, and a behavioral specification; a test suite is withheld during solving and injected afterward through Unreal's Play-in-Editor automation. Success is determined not by the tests alone but by an LLM judge that sees the specification, test outcomes, model edits, and reference solution. The headline result is that the strongest configuration reaches 55.5% pass@1, 31 tasks are unsolved by all configurations, and failures cluster around runtime-integration issues such as authority, replication, lifecycle, and initialization rather than compile errors.","tokens_in":14892,"tokens_out":3695,"duration_ms":41471,"significance":"If the measurement premises hold, GameEngineBench fills a genuine gap: existing coding benchmarks rarely require generated C++ to compile and run correctly inside a stateful, real-time engine with networking, replication, and lifecycle semantics. The protocol has real strengths: tests are withheld during solving, tasks are validated and judge-reviewed before release, workspaces/logs are preserved for auditing, and the repository ships reproducibility scripts. The task set is non-synthetic and spans a useful variety of engine subsystems. However, the load-bearing measurement assumption — that the LLM judge determines behavioral correctness accurately and without family bias — is unvalidated. Because every reported number, the unsolved-task set, and the failure taxonomy are expressed in units of judge decisions, the benchmark's central claims are currently provisional. The paper's self-admitted wrapper confound and lack of uncertainty quantification further limit the strength of the leaderboard claims. The contributions are valuable but need additional validation before the benchmark can serve as a reliable measurement axis.","major_comments":[{"comment":"The success metric is the LLM judge's determination, not the tests themselves. Section 5 says 'A run is counted as successful when the LLM judge determines that the generated implementation satisfies the requested behavior,' and Section 7 concedes that LLM review can convert some test failures into passes. The paper reports no judge-vs-human agreement statistics, no per-run judge model/family, no judge temperature/prompt details, and no cross-family agreement analysis, despite stating that cross-family judging is 'supported.' Because the same LLM-judge paradigm is also used in task calibration (§3), the benchmark's difficulty distribution and its pass labels are both partly products of LLM judgment. This is a measurement-validity issue: the 55.5% headline, the 31-task unsolved set, and the Figure 5 outcome decomposition all shift if the judge is systematically lenient or biased toward a","section":"§5, pass@1 definition; §7; §3"},{"comment":"The paper states 'twelve evaluated configurations' in the abstract and Section 6, but Figure 4, Figure 9, and the supplementary matrices show 13 bars/rows, including GLM 5.2, while Table 2 lists only 12 setups and omits GLM 5.2. This is more than a typo: the evaluated-configuration count and the corresponding per-config scores are part of the central results. Separately, pass@1 is reported as a point estimate with no confidence intervals or error bars. With 110 tasks and one attempt per task/config, differences such as 12.7% vs 10.0% or 8.2% vs 7.3% are within ordinary sampling noise, so the claim that 'the benchmark is sensitive to meaningful differences between model–harness setups' is not yet supported. Please reconcile the configuration count and provide uncertainty intervals, or use a paired/complementarity analysis that accounts for small counts.","section":"§6.1, Figure 4, Table 2"},{"comment":"The paper appropriately acknowledges that 'the observed pass@1 differences partially reflect wrapper effects rather than pure model capability,' but the abstract and Section 6.1 phrase the main result as a model result ('the strongest model reaches 55.5% pass@1'). Since configurations differ not only in the underlying model but also in the wrapper (codex, claude-code, qwen-code, kimi-code, etc.) and in reasoning-effort settings, the headline comparison is really a comparison of model–wrapper bundles. This does not undermine the basic claim that frontier agents struggle on these tasks, but it does undermine the leaderboard ordering and the complementarity analysis. The paper should either standardize the wrapper or explicitly report the model/wrapper decomposition and temper the 'model capability' language accordingly.","section":"§7, wrapper confound"}],"minor_comments":[{"comment":"Typo: 'the task's public start/ state' should be 'start state.' Also, the metric is called 'pass@1' in the text but 'Calibrated pass@1' in figures; clarify whether calibration refers to judge adjustment and define it in the text.","section":"§5, execution detail"},{"comment":"Axis label reads 'T ask (original benchmark order, 1110)' — the number should be 110.","section":"Figure 9"},{"comment":"Table 2 lists 12 setups but Figure 4 includes GLM 5.2; if GLM 5.2 is a real evaluated configuration, it should be in the table. If the figure inadvertently includes an extra configuration, remove it.","section":"Table 2"},{"comment":"The complementarity matrix in Figure 11 would benefit from a note about statistical stability: with single attempts, a small number of task-level flips can change off-diagonal entries substantially.","section":"§6.3 / Appendix B"},{"comment":"The limitations paragraph is candid about test-suite imperfection and judge calibration, but it does not mention the absence of human validation data. Please add a sentence stating that no human agreement study was performed and that this is a priority for future work.","section":"§7"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and protocol are promising, and the authors are unusually transparent about limitations. The major blockers are quantitative: judge reliability, uncertainty quantification, and the model-vs-wrapper confound. These are fixable within the manuscript's scope. I would not reject, but the paper should not be accepted until the judge-validity evidence is provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Benchmark papers usually bore me, but this one is worth your time. GameEngineBench gives you 110 scoped C++ implementation tasks in nine Unreal repos, with withheld tests, task calibration, and preserved workspaces. That protocol hygiene is real. The combination of existing game-engine repositories, native C++ edits, server/client correctness, and judge-audited runtime behavior is new relative to GameDevBench, AutoUE, and the SWE-bench family. Even the small task set shows a clear gap: the strongest configuration, claude-fable-5, sits at 55.5% pass@1, with 31 tasks untouched by all twelve setups. I believe the directional claim: frontier agents compile and then fail on authority, replication, lifecycle, and initialization issues. The examples in Appendix A make that concrete.\n\nThe soft spot is the measurement unit. Every headline number — the 55.5%, the 31-task unsolved set, the Figure 5 failure mix — is a decision made by an LLM judge, and the paper gives no judge-versus-human agreement data, no per-run judge family, and no cross-family agreement analysis. Section 5 says a run counts as solved when the LLM judge determines it satisfies the behavior; Section 7 concedes the judge can overturn failing tests. That means the judge is not a secondary filter; it is the final scorer. You cannot read the exact leaderboard as ground truth until that instrument is validated. The wrapper confound is also real and admitted. Pass@1 has no error bars despite single attempts per configuration. No contamination analysis for tasks mined from public repos. These are addressable gaps, not a broken design.\n\nThe paper flags some of this itself in Section 7, and the authors are not hiding the limits. The corrupted string in the author block looks like a LaTeX/encoding artifact, not a scientific issue.\n\nWho should read this: anyone building agent benchmarks for stateful C++ systems, and anyone who cares whether LLM code edits work in real runtime environments. It deserves a serious referee. I would send it out, with the explicit request that the reviewers pressure-test the judge methodology and demand error bars and cross-family judging results. The underlying difficulty finding will survive that scrutiny; the precise ranking may not.","headline":"Solid new benchmark with a real gap to fill, but the leaderboard is only as solid as the unvalidated LLM judge behind it.","tokens_in":15500,"tokens_out":2157,"would_cite":true,"duration_ms":24489,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Coding agents do not reliably integrate C++ into running game engines: the strongest evaluated configuration solves 55.5% of 110 real Unreal tasks, and 31 tasks defeat every configuration.","keywords":["coding agents","benchmark","C++","Unreal Engine","runtime integration","multiplayer","LLM-as-a-judge","pass@1"],"falsifier":"Take a random sample of runs across all twelve configurations and re-score them with an independent judge from a different model family, then have a human reviewer decide the disputed cases; compare with the original LLM-judge verdicts. If agreement is low, or a material share of judge-passed implementations fail human behavioral review, the benchmark's headline numbers are not stable.","tokens_in":14494,"feed_emoji":"🎮","tokens_out":8809,"duration_ms":87091,"temperature":0.7,"pith_summary":"The paper wants to establish that coding agents—models that edit codebases on their own—fail a large share of focused C++ programming tasks when the code must actually run inside a real-time game engine, not just compile. It introduces GameEngineBench, 110 tasks built from nine public Unreal Engine 5 repositories; each task gives an agent a buildable project, a list of editable C++ files, and a behavioral specification, with the test suite hidden until after the agent finishes. Under this protocol the strongest of twelve model setups solves 55.5% of tasks on the first attempt, and 31 tasks are not solved by any setup. The authors' failure analysis attributes most misses to runtime-integration errors—authority mistakes, state-synchronization failures, object-lifecycle bugs, initialization problems, and incomplete wiring into surrounding systems—rather than to syntax or compilation failures. If the measurement is sound, game-engine repositories offer a hard, non-saturated evaluation axis that complements existing software-engineering benchmarks.","feed_headline":"No evaluated agent passes 56% of game-engine C++ tasks","feed_subtitle":"Best model solves 55.5% of 110 Unreal tasks; 31 tasks resist all twelve configurations.","key_machinery":"The load-bearing mechanism is the task-and-judge protocol: agents receive a buildable Unreal Engine 5 project, scoped editable C++ files, and a behavioral spec; hidden tests are injected after solving and run in Play-in-Editor listen-server mode so authority and replication are real. Correctness is decided by an LLM judge that sees the spec, tests, results, edits, and reference solution; a task-calibration loop of validation, judge review, and revision keeps tests aligned with intended behavior. Difficulty is organized around four engine runtime behaviors: multiplayer authority, replication, object lifecycle, and subsystem architecture.","core_discovery":"On its own terms, the paper's central claim is that game-engine repositories form a hard, non-saturated testbed for coding agents: scoped native C++ edits inside real Unreal Engine 5 projects, where correctness means the code behaves correctly at runtime—authority, replication, object lifecycle, initialization—not merely that it compiles. The strongest of twelve evaluated configurations solves 55.5% of 110 tasks on the first attempt; 31 tasks are solved by none. Agent failures cluster in runtime integration—authority mistakes, state-synchronization failures, lifecycle bugs, initialization errors—rather than syntax or compile errors. The benchmark is presented as a complement to existing soft","pith_inferences":["Inference: because difficulty concentrates in tasks requiring coordination across several engine systems (save persistence, AI plus round state, procedural map generation with replication), the benchmark may be measuring a distinct 'cross-system integration' capability that existing issue-resolution benchmarks do not capture; an agent that excels at repo-level bug fixing could still score low here","Inference: the pairwise complementarity matrix implies an ensemble ceiling above any single config; routing each task to the configuration most likely to solve it could raise coverage beyond the leaderboard, and the paper's data already provide the information needed to estimate that gain.","Inference: a direct test of the general claim would be to build analogous tasks in another real-time framework not based on game engines; if the same failure patterns appear, the finding generalizes beyond Unreal.","Inference: the judge converts some test failures into passes; until judge-family assignments and human-agreement statistics are reported, the absolute 55.5% figure should be read as an upper-bound estimate, while the ordering across configurations is more likely to be robust."],"forward_implications":["GameEngineBench is not saturated: the strongest configuration solves only 55.5% of tasks, and 31 tasks remain unsolved across all twelve evaluated configurations.","The benchmark separates model setups by a wide margin (55.5% vs. 29.1% for the next best), and raising reasoning effort improves but does not close the gap, so the difficulty is not just about search budget.","Most failures are behavioral: authority mistakes, state-synchronization errors, object-lifecycle bugs, initialization errors, and incomplete integration, with compile failures playing a minor role.","The leaderboard is not a nested ordering: configurations that score higher can miss tasks that lower-scoring configurations solve, and no task is solved by all.","The benchmark broadens coding-agent evaluation from general repository work to runtime-integrated C++ in stateful, real-time interactive software."],"fun_headline_variants":["GameEngineBench: top coding agent hits 55.5%","Real Unreal C++ tasks beat all coding agents","31 Unreal C++ tasks unsolved by any agent","Coding agents cap at 55.5% on game-engine C++","GameEngineBench: toughest C++ test for agents?"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The measurement premise—that the LLM judge correctly decides whether a generated implementation satisfies the requested behavior—is load-bearing; the paper does not report judge-versus-human agreement or the judge families used, and it concedes that LLM review converts some test failures into passes. If the judge is lenient or biased toward solutions from its own model family, the 55.5% pass rate, the 31-task unsolved set, and the failure decomposition all shift.","fun_headline_variants_meta":{"raw":{"variants":["GameEngineBench: top coding agent hits 55.5%","Real Unreal C++ tasks beat all coding agents","31 Unreal C++ tasks unsolved by any agent","Coding agents cap at 55.5% on game-engine C++","GameEngineBench: toughest C++ test for agents?"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000154,"raw_usage":{"total_tokens":1057,"prompt_tokens":760,"completion_tokens":297,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":212}},"tokens_in":504,"tokens_out":297,"duration_ms":3912,"temperature":1.0,"reasoning_tokens":212,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T08:48:48.460682+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of runs across all twelve configurations and re-score them with an independent judge from a different model family, then have a human reviewer decide the disputed cases; compare with the original LLM-judge verdicts. If agreement is low, or a material share of judge-passed implementations fail human behavioral review, the benchmark's headline numbers are not stable.","supporting_citations":[],"review_version":2}