Pith. sign in

REVIEW 3 major objections 5 minor 20 references

GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Coding agents do not reliably integrate C++ into running game engines: the strongest evaluated configuration solves 55.5% of 110 real Unreal tasks, and 31 tasks defeat every configuration.

desk verdict Solid new benchmark with a real gap to fill, but the leaderboard is only as solid as the unvalidated LLM judge behind it. read the letter →

arxiv 2607.03525 v2 pith:BJ24ZPCC submitted 2026-07-03 cs.SE cs.CL

Brian La , Sejoon Chang , Ben Kim , Junyoung Bae , Aamish Ahmad Beg , Sei Chang , Gonzalo Gonzalez-Pumariega , Kanav Goyal This is my paper · ORCID
classification cs.SEcs.CL
keywords codingagentsbenchmarkC++UnrealEngineruntimeintegrationmultiplayerLLM-as-a-judgepass@1
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper wants to establish that coding agents—models that edit codebases on their own—fail a large share of focused C++ programming tasks when the code must actually run inside a real-time game engine, not just compile. It introduces GameEngineBench, 110 tasks built from nine public Unreal Engine 5 repositories; each task gives an agent a buildable project, a list of editable C++ files, and a behavioral specification, with the test suite hidden until after the agent finishes. Under this protocol the strongest of twelve model setups solves 55.5% of tasks on the first attempt, and 31 tasks are not solved by any setup. The authors' failure analysis attributes most misses to runtime-integration errors—authority mistakes, state-synchronization failures, object-lifecycle bugs, initialization problems, and incomplete wiring into surrounding systems—rather than to syntax or compilation failures. If the measurement is sound, game-engine repositories offer a hard, non-saturated evaluation axis that complements existing software-engineering benchmarks.

What carries the argument

The load-bearing mechanism is the task-and-judge protocol: agents receive a buildable Unreal Engine 5 project, scoped editable C++ files, and a behavioral spec; hidden tests are injected after solving and run in Play-in-Editor listen-server mode so authority and replication are real. Correctness is decided by an LLM judge that sees the spec, tests, results, edits, and reference solution; a task-calibration loop of validation, judge review, and revision keeps tests aligned with intended behavior. Difficulty is organized around four engine runtime behaviors: multiplayer authority, replication, object lifecycle, and subsystem architecture.

What would settle it

Take a random sample of runs across all twelve configurations and re-score them with an independent judge from a different model family, then have a human reviewer decide the disputed cases; compare with the original LLM-judge verdicts. If agreement is low, or a material share of judge-passed implementations fail human behavioral review, the benchmark's headline numbers are not stable.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that game-engine repositories form a hard, non-saturated testbed for coding agents: scoped native C++ edits inside real Unreal Engine 5 projects, where correctness means the code behaves correctly at runtime—authority, replication, object lifecycle, initialization—not merely that it compiles. The strongest of twelve evaluated configurations solves 55.5% of 110 tasks on the first attempt; 31 tasks are solved by none. Agent failures cluster in runtime integration—authority mistakes, state-synchronization failures, lifecycle bugs, initialization errors—rather than syntax or compile errors. The benchmark is presented as a complement to existing soft

Load-bearing premise

The measurement premise—that the LLM judge correctly decides whether a generated implementation satisfies the requested behavior—is load-bearing; the paper does not report judge-versus-human agreement or the judge families used, and it concedes that LLM review converts some test failures into passes. If the judge is lenient or biased toward solutions from its own model family, the 55.5% pass rate, the 31-task unsolved set, and the failure decomposition all shift.

Editorial extensions

If this is right

  • GameEngineBench is not saturated: the strongest configuration solves only 55.5% of tasks, and 31 tasks remain unsolved across all twelve evaluated configurations.
  • The benchmark separates model setups by a wide margin (55.5% vs. 29.1% for the next best), and raising reasoning effort improves but does not close the gap, so the difficulty is not just about search budget.
  • Most failures are behavioral: authority mistakes, state-synchronization errors, object-lifecycle bugs, initialization errors, and incomplete integration, with compile failures playing a minor role.
  • The leaderboard is not a nested ordering: configurations that score higher can miss tasks that lower-scoring configurations solve, and no task is solved by all.
  • The benchmark broadens coding-agent evaluation from general repository work to runtime-integrated C++ in stateful, real-time interactive software.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: because difficulty concentrates in tasks requiring coordination across several engine systems (save persistence, AI plus round state, procedural map generation with replication), the benchmark may be measuring a distinct 'cross-system integration' capability that existing issue-resolution benchmarks do not capture; an agent that excels at repo-level bug fixing could still score low here
  • Inference: the pairwise complementarity matrix implies an ensemble ceiling above any single config; routing each task to the configuration most likely to solve it could raise coverage beyond the leaderboard, and the paper's data already provide the information needed to estimate that gain.
  • Inference: a direct test of the general claim would be to build analogous tasks in another real-time framework not based on game engines; if the same failure patterns appear, the finding generalizes beyond Unreal.
  • Inference: the judge converts some test failures into passes; until judge-family assignments and human-agreement statistics are reported, the absolute 55.5% figure should be read as an upper-bound estimate, while the ordering across configurations is more likely to be robust.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces GameEngineBench, a benchmark of 110 C++ implementation tasks drawn from nine open-source Unreal Engine 5 projects, and evaluates 12–13 frontier coding-agent configurations on them. Each task provides a buildable starting state, a list of editable C++ files, and a behavioral specification; a test suite is withheld during solving and injected afterward through Unreal's Play-in-Editor automation. Success is determined not by the tests alone but by an LLM judge that sees the specification, test outcomes, model edits, and reference solution. The headline result is that the strongest configuration reaches 55.5% pass@1, 31 tasks are unsolved by all configurations, and failures cluster around runtime-integration issues such as authority, replication, lifecycle, and initialization rather than compile errors.

Significance. If the measurement premises hold, GameEngineBench fills a genuine gap: existing coding benchmarks rarely require generated C++ to compile and run correctly inside a stateful, real-time engine with networking, replication, and lifecycle semantics. The protocol has real strengths: tests are withheld during solving, tasks are validated and judge-reviewed before release, workspaces/logs are preserved for auditing, and the repository ships reproducibility scripts. The task set is non-synthetic and spans a useful variety of engine subsystems. However, the load-bearing measurement assumption — that the LLM judge determines behavioral correctness accurately and without family bias — is unvalidated. Because every reported number, the unsolved-task set, and the failure taxonomy are expressed in units of judge decisions, the benchmark's central claims are currently provisional. The paper's self-admitted wrapper confound and lack of uncertainty quantification further limit the strength of the leaderboard claims. The contributions are valuable but need additional validation before the benchmark can serve as a reliable measurement axis.

major comments (3)
  1. [§5, pass@1 definition; §7; §3] The success metric is the LLM judge's determination, not the tests themselves. Section 5 says 'A run is counted as successful when the LLM judge determines that the generated implementation satisfies the requested behavior,' and Section 7 concedes that LLM review can convert some test failures into passes. The paper reports no judge-vs-human agreement statistics, no per-run judge model/family, no judge temperature/prompt details, and no cross-family agreement analysis, despite stating that cross-family judging is 'supported.' Because the same LLM-judge paradigm is also used in task calibration (§3), the benchmark's difficulty distribution and its pass labels are both partly products of LLM judgment. This is a measurement-validity issue: the 55.5% headline, the 31-task unsolved set, and the Figure 5 outcome decomposition all shift if the judge is systematically lenient or biased toward a
  2. [§6.1, Figure 4, Table 2] The paper states 'twelve evaluated configurations' in the abstract and Section 6, but Figure 4, Figure 9, and the supplementary matrices show 13 bars/rows, including GLM 5.2, while Table 2 lists only 12 setups and omits GLM 5.2. This is more than a typo: the evaluated-configuration count and the corresponding per-config scores are part of the central results. Separately, pass@1 is reported as a point estimate with no confidence intervals or error bars. With 110 tasks and one attempt per task/config, differences such as 12.7% vs 10.0% or 8.2% vs 7.3% are within ordinary sampling noise, so the claim that 'the benchmark is sensitive to meaningful differences between model–harness setups' is not yet supported. Please reconcile the configuration count and provide uncertainty intervals, or use a paired/complementarity analysis that accounts for small counts.
  3. [§7, wrapper confound] The paper appropriately acknowledges that 'the observed pass@1 differences partially reflect wrapper effects rather than pure model capability,' but the abstract and Section 6.1 phrase the main result as a model result ('the strongest model reaches 55.5% pass@1'). Since configurations differ not only in the underlying model but also in the wrapper (codex, claude-code, qwen-code, kimi-code, etc.) and in reasoning-effort settings, the headline comparison is really a comparison of model–wrapper bundles. This does not undermine the basic claim that frontier agents struggle on these tasks, but it does undermine the leaderboard ordering and the complementarity analysis. The paper should either standardize the wrapper or explicitly report the model/wrapper decomposition and temper the 'model capability' language accordingly.
minor comments (5)
  1. [§5, execution detail] Typo: 'the task's public start/ state' should be 'start state.' Also, the metric is called 'pass@1' in the text but 'Calibrated pass@1' in figures; clarify whether calibration refers to judge adjustment and define it in the text.
  2. [Figure 9] Axis label reads 'T ask (original benchmark order, 1110)' — the number should be 110.
  3. [Table 2] Table 2 lists 12 setups but Figure 4 includes GLM 5.2; if GLM 5.2 is a real evaluated configuration, it should be in the table. If the figure inadvertently includes an extra configuration, remove it.
  4. [§6.3 / Appendix B] The complementarity matrix in Figure 11 would benefit from a note about statistical stability: with single attempts, a small number of task-level flips can change off-diagonal entries substantially.
  5. [§7] The limitations paragraph is candid about test-suite imperfection and judge calibration, but it does not mention the absence of human validation data. Please add a sentence stating that no human agreement study was performed and that this is a priority for future work.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GameEngineBench is an empirical evaluation whose judge-based metric is transparently defined, with no fitted parameter renamed as a prediction.

full rationale

GameEngineBench is an empirical benchmark paper, not a derivation chain. The success metric is explicitly defined in §5: 'A run is counted as successful when the LLM judge determines that the generated implementation satisfies the requested behavior.' The reported pass@1 values are therefore a transparent measurement of judge-accepted runs, not a hidden reduction of a prediction to an input. The paper fits no parameter to a subset of data and then predicts a closely related quantity; the LLM judge is the stated scoring rule. Judge use in task calibration (§3) is a quality-control step for task/test refinement, not a fitted input to the headline leaderboard. The LLM-as-a-judge paradigm is attributed to external prior work (reference [18]), and no load-bearing self-citations, imported uniqueness theorems, or ansatz-smuggling citations appear in the argument. The only substantive concern—absence of judge-vs-human agreement statistics and unspecified cross-family judging—is a measurement-validity issue: if the judge were biased, the reported numbers could be wrong, but they would not be true by construction. Per the reviewing rules, that concern belongs to correctness risk rather than circularity and does not raise the circularity score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on hand-set evaluation design choices rather than fitted equations: timeout budgets, the single-attempt protocol, the LLM judge configuration, and task-calibration criteria all shape the reported pass@1 and the 31-task unsolved set. Domain assumptions about PIE listen-server fidelity, judge-plus-tests coverage of correctness, representativeness of nine open-source repos, and absence of training-data contamination are load-bearing for the benchmark's validity. No new physical or theoretical entities are introduced.

free parameters (4)
  • Timeout budgets = solver 3600 s; compile/test 600 s each
    Hand-chosen limits in Table 2 that bound how much search and iteration agents can perform; different budgets would change pass@1.
  • Attempt protocol = 1 attempt per task/model pair
    Choice in Table 2 and §5: no retries or majority vote; pass@1 at n=1 carries no variance estimate, so leaderboard differences lack error bars.
  • LLM judge configuration = unspecified (cross-family 'whenever available')
    The judge's verdict is the success metric (§5); judge model, version, temperature, and family-pairing policy are not reported, and these choices affect every pass/fail label.
  • Task calibration acceptance criteria = 110 tasks after iterative revision
    §3's calibration loop (test validation + LLM-judge task review + revision) decides which tasks enter the set; the difficulty distribution and the 31-task no-solve core are partly outputs of this filter.
assumptions (4)
  • domain assumption Play-in-Editor listen-server automation validly exercises multiplayer authority, replication, and lifecycle correctness.
    §5 runs tests through Unreal's PIE listen-server mode; §7 concedes this does not fully reflect dedicated server fleets, console hardware, or long-running live operations, so some production correctness conditions are unmeasured.
  • domain assumption The combination of injected tests and the LLM judge correctly determines task success.
    §5 defines pass@1 by the LLM judge's determination and notes the test suite is inherently imperfect; no human-audit agreement or judge-reliability statistics are reported, so the error rate of this arbiter is unknown.
  • domain assumption Nine open-source Unreal repositories provide a representative sample of game-engine C++ work.
    §4 selects the nine repos for diversity; §7 acknowledges open-source projects expose gameplay/framework code more readily than proprietary engine, rendering, platform, performance, or security work, so coverage is a partial slice.
  • domain assumption Tasks extracted from public repositories are not memorized by the evaluated models.
    §4 mines tasks from nine public repos, yet no contamination/decontamination analysis is reported despite §2 citing SWE-rebench's decontamination emphasis; frontier training corpora plausibly include these codebases.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments." pith.science (2026). https://pith.science/paper/BJ24ZPCC

@misc{pith2026260703525,
  author       = {Pith},
  title        = {Pith review of: GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BJ24ZPCC}},
  note         = {Machine review of arXiv:2607.03525}
}
read the original abstract

Game engines provide real-time simulation, rendering, physics, interaction, networking, and asset pipelines, making them valuable not only for games but also for 3D applications in healthcare, robotics, architecture, manufacturing, and related domains. Because game development is where these systems are most mature and publicly available, it offers a practical testbed for evaluating coding agents that must modify C++ code within stateful, interactive, real-time systems. We present GameEngineBench, a benchmark for evaluating coding agents on scoped C++ implementation tasks inside Unreal Engine 5 projects, built from nine real-world game repositories. The evaluation set consists of 110 tasks spanning gameplay mechanics, multiplayer behavior, AI and world orchestration, animation and movement, UI and session code, loading behavior, online-service integration, persistence, data serialization, XR behavior, and rendering-oriented plugins. These tasks require models to make native C++ changes that compile and satisfy behavioral tests within executable Unreal Engine projects. Across twelve evaluated configurations, the strongest model reaches 55.5\% pass@1, while 31 tasks remain unsolved by every configuration. Our results demonstrate that frontier coding agents continue to struggle with deeply integrated C++ development for real-time interactive software, highlighting game-engine benchmarks as a valuable complement to existing software engineering evaluations.

Figures

Figures reproduced from arXiv: 2607.03525 by the authors.

Figure 1
Figure 1. Task authoring, test validation, LLM-as-a-Judge review, and task calibration process. subsystems that the engine creates, initializes, and exposes to gameplay code [9]. Correct implementations often depend on coordinating these behaviors to produce a correct solution to a development task. The emphasis on cross-system behavior is consistent with prior work showing that game-engine subsystems are tightly coupled and … view at source ↗
Figure 2
Figure 2. Example tasks from GameEngineBench. Screenshots are taken from reference solutions to illustrate the gameplay and engine-system behaviors evaluated by the benchmark. 4 Current C++ Task Set The current evaluated task set consists of 110 active tasks and is sourced from nine Unreal projects. The tasks focus on gameplay and systems programming within existing codebases, spanning gameplay mechanics, multiplayer behavior… view at source ↗
Figure 3
Figure 3. Primary game-development areas covered by the current task set. Categories are assigned once per task for readability; many tasks also exercise cross-cutting runtime requirements such as authority, state synchronization, object lifecycle handling, and subsystem initialization. 5 Evaluation Protocol The primary evaluation metric used in this work is pass@1, defined as the fraction of tasks solved by a single model at… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Success rate by model configuration over the active 110-task set. Each label includes the model and reasoning effort where applicable. be deployed more at scale effectively. The efficiency of these models should be considered in conjunction with the pass@1 leaderboard.…
Figure 5
Figure 5. Figure 5: Outcome decomposition per configuration. 6.5 Failure Mechanisms We uncover failure mechanisms from model-level and task-level results that explain the task-coverage gap [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Efficiency profile across evaluated configurations. Rows are sorted by pass@1 and compare success rate with median solve time, mean model cost, and mean output tokens per task [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Task coverage and model complementarity. Left: how many configurations solve each task. Right: each off-diagonal entry counts tasks solved by the row configuration but missed by the column configuration; diagonal entries are own solved counts. 0 3 6 9 12 15 18 21 Tasks…
Figure 8
Figure 8. Figure 8: Task-area coverage under pass@1. Green segments count tasks solved by at least one evaluated configuration; gray segments count tasks with no successful runs. Labels at the end of each bar show the number of solved tasks over the total number of tasks in each area. 14 …
Figure 9
Figure 9. Figure 9: Per-task outcome matrix over the active 110-task set. Green cells are passes; red cells are non-passes. This dense view supports the capability-frontier analysis in the main text. Claude Fable 5 (max) GPT-5.5 (xhigh) Claude Opus 4.8 (max) GPT-5.5 (high) Gemini 3.1 Pro …
Figure 10
Figure 10. Figure 10: Supplemental pass@1 breakdowns by task metadata. Left: pass@1 by game-system domain. Right: pass@1 by requirement category. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Supplemental task difficulty and complementarity views. Left: tasks ranked by fraction of configurations that solve them. Right: pairwise task complementarity between configurations. Claude Fable 5 (max) GPT-5.5 (xhigh) Claude Opus 4.8 (max) GPT-5.5 (high) Claude Opus…
Figure 12
Figure 12. Figure 12: Supplemental compile diagnostics from available trajectories. Left: mean number of Unreal compiler invocations per task. Right: mean number of compile errors encountered during solving. 2 4 6 8 10 12 14 16 18 Mean cost per task (USD) 10 20 30 40 50 Calibrated pass rat…
Figure 13
Figure 13. Figure 13: Supplemental resource-use diagnostics. Left: cost versus task-pass rate where cost data is available. Right: output tokens per task. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 9 linked inside Pith

  1. [1]

    Program synthesis with large language models, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URLhttps://arxiv.org/abs/2108.07732

    arXiv 2021

  2. [2]

    SWE-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents,

    Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. SWE-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents,

  3. [3]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  4. [4]

    GameDevBench: Evaluating agentic capabilities through game development, 2026

    Wayne Chi, Yixiong Fang, Arnav Yayavaram, Siddharth Yayavaram, Seth Karten, Qiuhong Anna Wei, Runkun Chen, Alexander Wang, Valerie Chen, Ameet Talwalkar, and Chris Donahue. GameDevBench: Evaluating agentic capabilities through game development, 2026. URLhttps://arxiv.org/abs/26 02.11103

    2026

  5. [5]

    Actor lifecycle in Unreal Engine

    Epic Games. Actor lifecycle in Unreal Engine. Unreal Engine 5.8 Documentation, 2026. URL https://dev.epicgames.com/documentation/en-us/unreal-engine/unreal-engine-act or-lifecycle. Accessed July 2, 2026

    2026

  6. [6]

    Networking and multiplayer in Unreal Engine

    Epic Games. Networking and multiplayer in Unreal Engine. Unreal Engine 5.8 Documentation, 2026. URL https://dev.epicgames.com/documentation/en-us/unreal-engine/networking-a nd-multiplayer-in-unreal-engine. Accessed July 2, 2026

    2026

  7. [7]

    Play in editor multiplayer options in Unreal Engine

    Epic Games. Play in editor multiplayer options in Unreal Engine. Unreal Engine 5.8 Documentation,

  8. [8]

    Replicate actor properties in Unreal Engine

    Epic Games. Replicate actor properties in Unreal Engine. Unreal Engine 5.8 Documentation, 2026. URL https://dev.epicgames.com/documentation/en-us/unreal-engine/replicate-act or-properties-in-unreal-engine. Accessed July 2, 2026

    2026

Show all 20 references
  1. [9]

    Programming subsystems in Unreal Engine

    Epic Games. Programming subsystems in Unreal Engine. Unreal Engine 5.8 Documentation, 2026. URL https://dev.epicgames.com/documentation/en-us/unreal-engine/programming-s ubsystems-in-unreal-engine. Accessed July 2, 2026

    2026

  2. [10]

    DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks

    Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks. Datacurve technical report, 2026. URLhttps: //deepswe.datacurve.ai/blog/deepswe. Published May 26, 2026

    2026

  3. [11]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues?, 2023. URL https://arxiv.org/abs/2310.06770

    2023 arXiv

  4. [12]

    Introducing FrontierCode

    Eric Lu, Ben Pan, Deniz Birlikci, Sam Lee, Ray Wang, Rohan Choudhury, Fermi Ma, TC Qin, Carlo Baronio, and Silas Alberti. Introducing FrontierCode. Cognition research blog, 2026. URL https://cognition.com/blog/frontier-code. Published June 8, 2026

    2026

  5. [13]

    Merrill, Alexander G

    Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, et al. Terminal- Bench: Benchmarking agents on hard, realistic tasks in command line interfaces, 2026. URL https://arxiv.org/abs/2601.11868

    2026 arXiv

  6. [14]

    Ullmann, Yann-Gaël Guéhéneuc, Fabio Petrillo, Nicolas Anquetil, and Cristiano Politowski

    Gabriel C. Ullmann, Yann-Gaël Guéhéneuc, Fabio Petrillo, Nicolas Anquetil, and Cristiano Politowski. Visualising game engine subsystem coupling, 2023. URLhttps://arxiv.org/abs/2309.06329

    2023 arXiv

  7. [15]

    ProgramBench: Can language models rebuild programs from scratch?, 2026

    John Yang, Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, Diyi Yang, and Ofir Press. ProgramBench: Can language models rebuild programs from scratch?, 2026. URLhttps://arxiv.org/abs/2605.03546. 11

    2026 arXiv

  8. [16]

    AutoUE: Automated generation of 3d games in Unreal Engine via multi-agent systems, 2026

    Lei Yin, Wentao Cheng, Zhida Qin, Tianyu Huang, Yidong Li, and Gangyi Ding. AutoUE: Automated generation of 3d games in Unreal Engine via multi-agent systems, 2026. URLhttps://arxiv.org/ abs/2603.07106

    2026 arXiv

  9. [17]

    SWE-bench goes live!, 2025

    LinghaoZhang,ShilinHe,ChaoyunZhang,YuKang,BowenLi,ChengxingXie,JunhaoWang,Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang. SWE-bench goes live!, 2025. URLhttps://arxiv.org/abs/2505.23419

    2025 arXiv

  10. [18]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM- as-a-Judge with MT-Bench and Chatbot Arena, 2023. URLhttps://arxiv.org/abs/2306.0568...

    2023 arXiv

  11. [2025]

    URLhttps://arxiv.org/abs/2505.20411

  12. [2026]

    Accessed July 2, 2026

    URL https://dev.epicgames.com/documentation/en-us/unreal-engine/play-i n-editor-multiplayer-options-in-unreal-engine. Accessed July 2, 2026

    2026

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.