Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Today's AI models can reimplement whole software projects from behavior alone, the paper argues, showing a 16,000-line bioinformatics toolkit rebuilt from a black-box binary for $251.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 09:30 UTC pith:NCMGNTUY

load-bearing objection Well-engineered benchmark with real results, but the flagship capability claim is only as strong as an underpowered contamination screen. the 3 major comments →

arxiv 2606.30182 v2 pith:NCMGNTUY submitted 2026-06-29 cs.AI

MirrorCode: AI can rebuild entire programs from behavior alone

Tom Adamczewski , David Owen , David Rein , Florian Brand , Giles Edkins , Allen Hart , Daniel O'Connell This is my paper
classification cs.AI
keywords AI coding agentssoftware reimplementationlong-horizon benchmarkblack-box behavioral oracleheld-out testsinference budgetmemorization contaminationbyte-exact evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

MirrorCode is a benchmark that asks AI models to rebuild existing command-line programs from scratch, seeing only a compiled binary they can run, the program's documentation, and a set of example tests—never the source code. The paper's central claim is that today's models can already complete long-horizon, precisely specified software engineering tasks: the strongest model scored a perfect reimplementation on 14 of 25 target programs, including gotree, a 16,000-line bioinformatics toolkit, passing 2,000 of 2,001 tests at a cost of $251. The paper also finds that such performance only appears when inference budgets are much larger than typical benchmarks allow, with single attempts sometimes spending thousands of dollars over days. If the results are not driven by memorization of the target code, they imply that AI can already take on some software projects that would occupy a human engineer for weeks, and that the main bottleneck for measuring such capability is the cost of eliciting it.

Core claim

MirrorCode's contribution is a reproducible demonstration rather than a single theorem: 25 existing command-line programs, spanning Unix utilities, data formats, interpreters, static analysis, cryptography, and compression, are turned into long-horizon reimplementation tasks. An agent gets an execute-only binary of the original, documentation, and visible end-to-end tests; it must write its own implementation in one of six languages, and a solution counts as solved only if its stdout, stderr, and exit codes match the reference byte-for-byte on hidden tests as well. In the paper's runs, the strongest model solved 14 of 25 programs perfectly and came within 1% on four more; the headline instan

What carries the argument

The central device is the black-box behavioral oracle combined with held-out end-to-end tests. The agent can run the original binary with arbitrary inputs and observe its exact output, but cannot read its bytes or consult the web; visible tests pin down the scope of behavior that must be reproduced, while hidden 'dual' tests—variants of visible ones that exercise the same functionality—prevent hard-coded lookup tables from passing. Byte-exact equality of stdout, stderr, and exit status is the only success metric, and scoring happens in a separate sandbox so a submission cannot secretly wrap the reference binary. Around this, the paper layers unusually large inference budgets (up to 10 billio

Load-bearing premise

The load-bearing premise is that the benchmark scores measure genuine reimplementation ability rather than the model recalling code it saw during training; the paper's own screen flags possible memorization for 17 of its 25 target programs and argues it is not the main driver because some unmemorized targets were solved while some memorized ones were not.

What would settle it

Take a set of ten command-line tools whose source and discussions were first published after the models' training cutoffs, run them through the same MirrorCode scaffold with the same visible/hidden test design and inference budgets, and compare the 100%-solve rate to the reported 56%; a large drop would indicate that contamination, not generalization, carried the original results.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the core claim holds, AI can already complete software reimplementation projects that human engineers estimate would take weeks, at costs in the hundreds of dollars and time in hours to days.
  • Benchmarks that cap spending at a few dollars per task will systematically understate frontier coding ability; measuring it requires budgets of thousands of dollars and runs spanning days.
  • The absence of large language effects across six very different implementation languages indicates the capability is general programming skill, not syntax memorization.
  • Reliability is a separate limit: the best model solved roughly a third of the targets in every run, and most failures were edge cases or missed requirements rather than core functionality.
  • Without visible tests, performance on harder targets drops sharply, so the result supports precisely specified, testable tasks, not open-ended software development.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is to point the same scaffold at freshly written programs with no public footprint; if solve rates on that uncontaminated set stay near 56%, the capability claim is robust, and if they fall, contamination explains part of the result.
  • The benchmark's design suggests a practical target for automation: legacy reimplementation and porting—for example, rewriting C code in Rust—fits the profile of black-box behavior plus test suites, so those workflows may be among the first real-world adopters.
  • Because most failures are edge cases and missed requirements, reliability could improve sharply from simple changes—reward models, majority voting, or an explicit checklist of documented flags—even without a more capable base model.
  • The lack of visual or GUI targets means the result says little about front-end or interactive software; a mirror benchmark for visual programs would be needed to extend the claim.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces MirrorCode, a long-horizon benchmark in which an agent must reimplement a CLI program from behavior alone: it receives execute-only access to the original binary, documentation, and visible tests, but not the source. Solutions are scored byte-exactly on visible and hidden end-to-end tests, with hidden tests held out to prevent lookup-table cheating. Across 25 targets in six implementation languages, the paper reports that frontier models achieve 100% on 17/25 targets, that Claude Opus 4.7 scores 56% solve@100 and 77% solve>=99, and that a near-perfect gotree reimplementation passed 2000/2001 tests at $251 in 14 hours. It also analyzes failure modes, cross-language invariance, cost, and year-over-year progress. The central claim is that AI can already complete long-horizon, precisely specified software-engineering tasks.

Significance. If the contamination concern can be resolved, this is a valuable contribution. The benchmark design is unusually rigorous on the cheating side: scoring runs in separate containers without the reference binary, the verdict is a pure byte-equality function computed outside agent-controlled sandboxes, and hidden dual tests prevent trivial hard-coding. The open release of the scaffold and 22 targets, the use of large inference budgets, and the cross-language comparison are all strengths. The main weakness is that the central generalization claim rests on a contamination screen that is underpowered relative to the claim. As it stands, the headline solve rates are best interpreted as upper bounds on true reimplementation capability; the paper's own Section 4.3 acknowledges this ambiguity.

major comments (3)
  1. [Section 4.3, Appendix B.3, Table 4] The contamination screen does not support the quantitative claim that solve rates measure generalization rather than memorization. The screen asks models to reproduce individual functions from the target source, but a long agentic trajectory could also benefit from memorized test patterns, algorithmic discussions, or output quirks that this screen does not measure. The baseline uses only five post-cutoff projects, and the paper does not report which 17 of 25 targets were flagged, so the reader cannot tell whether the flagship gotree result is among them. The rebuttal that nonogrid and tssql were solved while sed and ruff were not is confounded by task difficulty: hard targets can be unsolved despite memorization, and easy targets can be solved without it. Please report per-target flags and add a contamination-controlled comparison, e.g., performance on post-cutoff or private targets matc
  2. [Section 4.1] The 'weeks-long' framing is not directly measured. The four gotree estimates are informal self-estimates (1.5-2.5, 13-17, 3, and 13 weeks) with no elicitation protocol, no independent raters, and no calibration against actual human performance. The only human baseline is a single short task not completed in 20 hours at 42% test pass, and the paper's own extrapolation is described as a soft lower bound. Since the abstract and conclusion use 'weeks-long' as a headline result, either run a small structured human study or explicitly demote this to a speculative estimate. The core capability result does not depend on the human comparison, but the paper's framing does.
  3. [Section 1 vs. Section 3.1, Figure 2] The use of 'solved' for gotree is inconsistent with the paper's own 100% threshold. Section 1 says Opus 4.7 'solved it in 14 hours, passing 2,000/2,001 tests (99.95%)', but Figure 2's top row shows no 100% solve for gotree, and Section 3.1 classifies it as near-perfect. Since 100% is the benchmark's primary solve criterion, call gotree a 'substantial reimplementation' and reserve 'solved' for the 17 perfect targets. Also clarify the Figure 2 caption claim that 'Only Claude Opus 4.7 successfully reimplemented Large targets' by stating whether this means 100% or >=99%.
minor comments (4)
  1. [Section 2.4] Gemini 3.1 Pro Preview was evaluated only in Python, while Opus 4.7 and GPT-5.5 were also run in all six languages on S/M targets. Figures 3 and 4 compare models across this asymmetric protocol; state this limitation or restrict the comparison to a common language/target subset.
  2. [Appendix B.3] The 17-target memorization flags are reported collectively as p<0.05 against a baseline, with no correction for multiple comparisons and no per-target p-values or similarity scores in a table. This makes it impossible to assess which results drive the finding.
  3. [Section 3.3 / Appendix B.4] The failure-mode breakdown in Figure 6 is generated by Claude Sonnet 4.6 with no human validation or inter-annotator agreement reported. This is fine for an exploratory taxonomy, but the paper should label it as AI-labeled and provide a small human-checked sample.
  4. [Section 3.4] The statement 'Conditional on passing all visible tests, two thirds of runs also passed every hidden test' is useful, but the conditioning set should be described in terms of how many runs passed all visible tests and how many tokens they used, since premature submission may differ across that set.

Circularity Check

0 steps flagged

No circularity found: MirrorCode results are external behavioral measurements with held-out tests.

full rationale

The paper's central claim is an empirical measurement rather than a derivation: MirrorCode tasks are defined by an external reference binary and end-to-end tests, with hidden tests held out from the agent (Section 2.2). Solve rates are computed by bytewise comparison of agent outputs to reference outputs in isolated scoring containers (Section 2.5, Appendix D), so success is not defined in terms of the model's own outputs or the paper's prior claims. No parameter is fitted to the benchmark score; hidden-test performance is a genuine held-out generalization because hidden duals are derived from visible tests but never shown to the agent. The contamination screen (Appendix B.3) compares model recall of source functions against post-cutoff baselines; regardless of its statistical power, this is an external check, not a circular step. The paper explicitly flags the remaining limitation that models 'might remember a discussion of algorithmic approaches used in the target program' (Section 4.3), which is an admitted threat to validity rather than a circular reduction. The only self-references are to the preliminary report [5] for scaffolding history and to METR [22] for external time-horizon context; neither defines the benchmark result. LLM-generated tests and LLM failure-mode labels are auxiliary and are checked against the reference oracle. No equation or definition in the paper reduces the headline result to its inputs, so there is no circularity to report.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central result is an empirical measurement, so there are no fitted free parameters in the physics sense. The listed items are the hand-set thresholds, budgets, buckets, and human estimates that the quantitative claims depend on. The domain assumptions are the benchmark's construct-validity premises: byte-equality on tests means reimplementation, hidden tests stay hidden and generalize, and the memorization screen's baseline is valid. The unresolved contamination question is the most consequential assumption.

free parameters (4)
  • Solve threshold = 100% of tests (primary); 99% for 'substantial reimplementation'
    Hand-set definition of 'solved'; directly determines solve rates and model comparisons in Figures 1–4.
  • Token budget caps = 1B tokens for S/M targets; 10B for Large targets
    Chosen to prevent compute-bound performance; the claim that large inference budgets are needed is defined relative to these caps.
  • Complexity bucket boundaries = S<2,000 ≤ M <25,000 ≤ L LoC with five manual reassignments
    Subjective buckets used for all size-based analyses and for the private/public split.
  • Human time estimates for gotree = 2–17 weeks (four estimates: 1.5–2.5, 13–17, 3, 13 weeks)
    Subjective estimates anchor the 'weeks-long task' significance claim; no measured human baseline exists.
axioms (5)
  • domain assumption Byte-exact equality of stdout/stderr/exit status across visible+hidden end-to-end tests is a sufficient proxy for having reimplemented the target program's scoped functionality.
    Underlies the entire benchmark; Section 2.2 (A score of 100% ... corresponds to reimplementing that target program's intended scope).
  • domain assumption Held-out 'hidden dual' tests are not leaked to the agent and cannot be memorized or hard-coded, so passing them indicates generalization rather than lookup.
    Section 2.2; depends on sandbox isolation described in Appendix D.
  • domain assumption The memorization screen's baseline (five projects after training cutoff) correctly estimates chance-level similarity for unmemorized code, so scores above it indicate training-data contamination.
    Appendix B.3; this is what links contamination evidence to the conclusion that results generalize.
  • domain assumption Skilled-human estimates (2–17 weeks for gotree) are reliable enough to anchor the 'weeks-long task' framing.
    Section 4.1; no long-task human baseline exists, and estimates vary by roughly 10x.
  • domain assumption The reference binaries and test oracles are deterministic and environment-independent, so byte-exact matching is well-defined.
    Section 2.3 and Table 1; nondeterministic behaviors are screened and excluded.

pith-pipeline@v1.3.0-alltime-deepseek · 31309 in / 14493 out tokens · 127091 ms · 2026-08-02T09:30:02.005208+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of MirrorCode: AI can rebuild entire programs from behavior alone." pith.science (2026). https://pith.science/paper/NCMGNTUY

@misc{pith2026260630182,
  author       = {Pith},
  title        = {Pith review of: MirrorCode: AI can rebuild entire programs from behavior alone},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NCMGNTUY}},
  note         = {Machine review of arXiv:2606.30182}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

AI models are rapidly improving at autonomous coding, as shown by benchmark progress and one-off demonstrations such as AI implementing a C compiler. However, existing coding benchmarks tend to focus on shorter tasks, and one-off demonstrations are hard to compare systematically because they often have some human guidance, and are not standardized or repeated across models. To address these challenges, we introduce MirrorCode, a long-horizon coding benchmark based on reimplementing entire software projects. In MirrorCode, AI agents must replicate the functionalities of an existing program, without access to its source code. AI solutions must match the original program's output exactly on end-to-end tests, including held-out tests. MirrorCode's 25 target programs span different areas of computing: Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression. Existing AI models can already reimplement complex software, with the strongest model scoring 56% across the benchmark. For example, AI can reimplement gotree, a 16,000-line bioinformatics toolkit - a task that we believe would take weeks for a human engineer. However, studying the frontier of performance requires a larger inference budget than typical benchmarks, for example, \$2,600 over 19 days for a single attempt on a large task. We show that AI agents can already complete long-horizon software engineering tasks, especially when requirements are precisely specified. More broadly, our work suggests AI will have transformative effects on software engineering, as autonomous agents continue to improve.

Figures

Figures reproduced from arXiv: 2606.30182 by Allen Hart, Daniel O'Connell, David Owen, David Rein, Florian Brand, Giles Edkins, Tom Adamczewski.

Figure 1
Figure 1. Figure 1: Average rate of 100%-solves across all MirrorCode target programs, i.e., how often AI scores 100% [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Solve rate by model and task, for reimplementation (100% of all tests passing) and substantial reimplementa [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Solve rates by task complexity bucket. Opus 4.7 outperformed other models on larger tasks. Bars show [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Solve rates for older and newer AI models, in smaller subsets of MirrorCode S and M tasks. Scores are hence [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: (a) Share of the StarCoder pretraining mix for each of MirrorCode’s six implementation languages [19]; the amount of available code varies widely, from Python down to Ada (about 1/230 of Python). (b) Despite this, Claude Opus 4.7’s solve rates are similar across languages. Both panels use the same language ordering. MirrorCode therefore provides a cleaner head-to-head comparison of implementation language … view at source ↗
Figure 6
Figure 6. Figure 6: Breakdown of failure categories, by model. A trajectory is counted in every category where it has any failing [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Percentage of withheld test cases solved, under ablations. “Withheld” test cases are the original hidden test [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Example trajectories showing test scores as runs progressed. In shown [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Memorization screen results. The orange band shows the 95% CI for the baseline non-memorized programs; [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Solve rates for older models, comparing Claude Opus 4.1 to Opus 4.7 and GPT-5 to GPT-5.5. Results use [PITH_FULL_IMAGE:figures/full_fig_p025_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Two representative obfuscated binary wraps submitted by Gemini 3.1 Pro Preview in the no-tool abla [PITH_FULL_IMAGE:figures/full_fig_p026_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: MirrorCode’s container architecture (one task instance). The agent interacts only with the [PITH_FULL_IMAGE:figures/full_fig_p027_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Representative cal invocations and their byte-exact output, illustrating the default, --monday, --week, vertical multi-month, and day-of-year (-j) layouts. Sept 1752 has a mix of Julian and Gregorian dates by which the 2nd is followed by the 14th (the 3rd through the 13th are absent). The --reform flag customizes the date of adoption of the Gregorian calendar. Separately, week numbering depends on the cho… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis

    cs.SE 2026-07 accept novelty 7.0

    Distilling 1,001 whole-life-cycle source-free program-synthesis trajectories into Qwen3.6-27B raises ProgramBench pass rate from 37.98% to 49.51% and transfers to seven unseen SE benchmarks.

Reference graph

Works this paper leans on

35 extracted references · 11 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Building a C compiler with a team of parallel Claudes.Anthropic Engineering Blog, 2026

    Nicholas Carlini. Building a C compiler with a team of parallel Claudes.Anthropic Engineering Blog, 2026

    2026

  2. [2]

    Scaling long-running autonomous coding.Cursor Blog, January, 2026

    Wilson Lin. Scaling long-running autonomous coding.Cursor Blog, January, 2026

    2026

  3. [3]

    SWE-bench Pro: Can AI agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941, 2025

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. SWE-bench Pro: Can AI agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Open-world evaluations for measuring frontier AI capabilities.arXiv preprint arXiv:2605.20520, 2026

    Sayash Kapoor, Peter Kirgis, Andrew Schwartz, Stephan Rabanser, JJ Allaire, Rishi Bommasani, Harry Coppock, Magda Dubois, Gillian K Hadfield, Andrew B Hall, et al. Open-world evaluations for measuring frontier AI capabilities.arXiv preprint arXiv:2605.20520, 2026

    Pith/arXiv arXiv 2026

  5. [5]

    MirrorCode: evidence that ai can already do some weeks-long coding tasks.Epoch AI blog post, April, 2026

    Tom Adamczewski, David Rein, David Owen, and Florian Brand. MirrorCode: evidence that ai can already do some weeks-long coding tasks.Epoch AI blog post, April, 2026

    2026

  6. [6]

    SWE-bench: Can language models resolve real-world GitHub issues? InInternational Conference on Learning Representations, volume 2024, pages 54107–54157, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? InInternational Conference on Learning Representations, volume 2024, pages 54107–54157, 2024. 8Occasionally real software is created as a direct reimplementation of existing software, fo...

    2024

  7. [7]

    ReAct Agent, 2026

    UK AI Security Institute. ReAct Agent, 2026. Inspect: An open-source framework for large language model evaluations

    2026

  8. [8]

    Repo state loopholes during agentic evaluation

    SWE-bench maintainers. Repo state loopholes during agentic evaluation. GitHub issue, SWE-bench/SWE-bench #465,https://github.com/SWE-bench/SWE-bench/issues/465, 2025. Accessed 2026-06-10

    2025

  9. [9]

    KernelBench v0.1.https://scalingintelligence.stanford

    Scaling Intelligence Lab, Stanford University. KernelBench v0.1.https://scalingintelligence.stanford. edu/blogs/kernelbenchv01/, 2025. Accessed 2026-06-10

    2025

  10. [10]

    Recent frontier models are reward hacking

    METR. Recent frontier models are reward hacking. https://metr.org/blog/ 2025-06-05-recent-reward-hacking/, 2025. Accessed 2026-06-10

    2025

  11. [11]

    Programbench: Can language models rebuild programs from scratch? arXiv preprint arXiv:2605.03546, 2026

    John Yang, Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, et al. Programbench: Can language models rebuild programs from scratch? arXiv preprint arXiv:2605.03546, 2026

    Pith/arXiv arXiv 2026

  12. [12]

    FrontierSWE: Benchmarking Software Engineering Skill at the Edge of Human Ability, 2026

    Proximal Labs. FrontierSWE: Benchmarking Software Engineering Skill at the Edge of Human Ability, 2026

    2026

  13. [13]

    Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces.arXiv preprint arXiv:2601.11868, 2026

    Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces.arXiv preprint arXiv:2601.11868, 2026

    Pith/arXiv arXiv 2026

  14. [14]

    SWE-Marathon: Can agents autonomously complete ultra-long-horizon software work?arXiv preprint arXiv:2606.07682, 2026

    Rishi Desai, Jesse Hu, Joan Cabezas, Neel Harsola, Pratyush Shukla, Roey Ben Chaim, Adnan El Assadi, Omkaar Mukund Kamath, Fenil Faldu, Prannay Hebbar, et al. SWE-Marathon: Can agents autonomously complete ultra-long-horizon software work?arXiv preprint arXiv:2606.07682, 2026

    Pith/arXiv arXiv 2026

  15. [15]

    Vibe Code Bench: Evaluating AI models on end-to-end web application development

    Hung Tran, Langston Nashold, Rayan Krishnan, Antoine Bigeard, and Alex Gu. Vibe Code Bench: Evaluating AI models on end-to-end web application development. InProceedings of the ACM Conference on AI and Agentic Systems, pages 514–536, 2026

    2026

  16. [16]

    SWE-bench Multilingual

    SWE-bench Team. SWE-bench Multilingual. https://www.swebench.com/multilingual.html, 2025. Accessed 2026-06-05

    2025

  17. [17]

    SWE-lancer: Can frontier LLMs earn $1 million from real-world freelance software engineering?arXiv preprint arXiv:2502.12115, 2025

    Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. SWE-lancer: Can frontier LLMs earn $1 million from real-world freelance software engineering?arXiv preprint arXiv:2502.12115, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Ale-bench: A benchmark for long-horizon objective-driven algorithm engineering.Advances in Neural Information Processing Systems, 38, 2026

    Yuki Imajuku, Kohki Horie, Yoichi Iwata, Kensho Aoki, Naohiro Takahashi, and Takuya Akiba. Ale-bench: A benchmark for long-horizon objective-driven algorithm engineering.Advances in Neural Information Processing Systems, 38, 2026

    2026

  19. [19]

    Starcoder: May the source be with you!arXiv preprint arXiv:2305.06161, 2023

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: May the source be with you!arXiv preprint arXiv:2305.06161, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Cruxeval-x: A benchmark for multilingual code reasoning, understanding and execution.arXiv preprint arXiv:2408.13001, 2024

    Ruiyang Xu, Jialun Cao, Yaojie Lu, Ming Wen, Hongyu Lin, Xianpei Han, Ben He, Shing-Chi Cheung, and Le Sun. Cruxeval-x: A benchmark for multilingual code reasoning, understanding and execution.arXiv preprint arXiv:2408.13001, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    A multi-language object-oriented programming benchmark for large language models.arXiv preprint arXiv:2509.26111, 2025

    Shuai Wang, Liang Ding, Li Shen, Yong Luo, Han Hu, Lefei Zhang, and Fu Lin. A multi-language object-oriented programming benchmark for large language models.arXiv preprint arXiv:2509.26111, 2025

    arXiv 2025

  22. [22]

    Task-completion time horizons of frontier AI models

    METR. Task-completion time horizons of frontier AI models. https://metr.org/time-horizons/, 05 2026

    2026

  23. [23]

    RE-bench: Evaluating frontier AI R&D capabilities of language model agents against human experts.arXiv preprint arXiv:2411.15114, 2024

    Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, et al. RE-bench: Evaluating frontier AI R&D capabilities of language model agents against human experts.arXiv preprint arXiv:2411.15114, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini

    Anne Ouyang, Simon Guo, Simran Arora, Alex L. Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. KernelBench: Can LLMs write efficient GPU kernels?arXiv preprint arXiv:2502.10517, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    autoresearch: Ai agents running research on single-gpu nanochat training automatically

    Andrej Karpathy. autoresearch: Ai agents running research on single-gpu nanochat training automatically. https://github.com/karpathy/autoresearch, 2026. GitHub repository

    2026

  26. [26]

    A careful examination of large language model performance on grade school arithmetic.Advances in Neural Information Processing Systems, 37:46819–46836, 2024

    Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, et al. A careful examination of large language model performance on grade school arithmetic.Advances in Neural Information Processing Systems, 37:46819–46836, 2024

    2024

  27. [27]

    Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap.arXiv preprint arXiv:2402.19450, 2024

    Saurabh Srivastava, Anto PV , Shashank Menon, Ajay Sukumar, Alan Philipose, Stevin Prince, Sooraj Thomas, et al. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap.arXiv preprint arXiv:2402.19450, 2024. 16 MirrorCode: AI can rebuild entire programs from behavior alone

    Pith/arXiv arXiv 2024

  28. [28]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. InInternational Conference on Learning Representations, volume 2025, pages 58791–58831, 2025

    2025

  29. [29]

    GBA Eval.https://gbaeval.com/, 2026

    Mechanize, Inc. GBA Eval.https://gbaeval.com/, 2026. Accessed: 2026-06-29

    2026

  30. [30]

    Many SWE-bench- passing PRs would not be merged into main

    Parker Whitfill, Cheryl Wu, Joel Becker, and Nate Rush. Many SWE-bench- passing PRs would not be merged into main. https://metr.org/notes/ 2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/, 03 2026

    2026

  31. [31]

    sed for JSON data

    Herbie Bradley and Girish Sastry. The great refactor: How to secure critical open-source code against memory safety exploits by automating code hardening at scale. The launch sequence, Institute for Progress, August 2025. Published August 11, 2025. 17 MirrorCode: AI can rebuild entire programs from behavior alone Appendix contents A Suggested naming conve...

    2025

  32. [32]

    The tell: the literal appears verbatim in the visible corpus, and the hidden test fails precisely because it perturbed that literal so the hard-code went inert

    **Cheating via hard-coding.** The solution is wired to the visible test — a branch/lookup/string-fixup keyed on an exact string the agent could only know from a *visible* test — which clearly cannot generalise. The tell: the literal appears verbatim in the visible corpus, and the hidden test fails precisely because it perturbed that literal so the hard-co...

  33. [33]

    **Brittle / narrow.** Real functionality (not a bare hard-code), but over-fit to the visible example — it implements *a* rule that happens to fit what it saw rather than the general rule, and the hidden test exercises a variation just outside that fit

  34. [34]

    what happens when two arguments are equal

    **Edge-case / subtle-logic miss.** Broadly correct and generalises for most inputs, but misses a detail the hidden test exercises that a competent programmer would have to *investigate* — not spelled out in the docs or the visible tests (e.g. "what happens when two arguments are equal")

  35. [35]

    posix" exe = bytes([47,117,115,114,47,98, 105,110,47,98,114,111,116,108, 105]).decode('latin1')#

    **Missed whole functionality.** An entire feature or behaviour, clearly described in the documentation or discoverable by a straightforward call of the reference binary, was never implemented. Unlike (3) this is a whole missing capability, not an edge case, and the signal for it was plainly available. C Supplementary results C.1 Programming languages Tabl...

    2025