Pith. sign in

REVIEW 3 major objections 4 minor 53 references

This paper tries to establish that LLM-generated ('vibe-coded') agents cannot match graduate-student coders on a real-world logistics task requiring strategic bidding and route planning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 20:10 UTC pith:HMAAEHPW

load-bearing objection The tournament is well-run and the human-over-LLM result is credible, but the 'vibe coding' framing oversells a curated, human-debugged agent pool and the improvement experiment is a single run. the 3 major comments →

arxiv 2511.20613 v1 pith:HMAAEHPW submitted 2025-11-25 cs.LG cs.AIcs.MA

Can Vibe Coding Beat Graduate CS Students? An LLM vs. Human Coding Tournament on Market-driven Strategic Planning

Panayiotis Danassis , Naman Goel This is my paper
classification cs.LG cs.AIcs.MA
keywords LLM code generationvibe codingbenchmarkstrategic planningpickup and delivery problemauction biddinghuman vs AI codingtournament evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that state-of-the-art LLMs, used the way a typical user would use them (writing code from English prompts, 'vibe coding'), cannot match graduate-student programmers on a real-world, open-ended strategic task. The test problem is a logistics market where agents bid in an auction for delivery tasks and then must route a fleet under capacity constraints. Across twelve round-robin tournaments with roughly forty thousand matches, all top-five finishers were written by students before LLMs existed, and 33 of 40 LLM-written agents lost to simple baselines. A sympathetic reader would care because most coding benchmarks only check whether code passes unit tests, whereas this benchmark measures whether code survives competition against intelligent opponents — the kind of task that determines real-world usefulness.

Core claim

The paper's central claim is that LLM code generation, even from carefully engineered prompts and after manual debugging, is not competitive with human-coded agents on a task requiring planning, optimization, and strategic bidding. In the Auction, Pickup, and Delivery Problem (APDP), agents bid for tasks in a reverse first-price auction and then solve a capacity-constrained pickup-and-delivery routing problem; profit is the difference between winning bids and driving costs. The authors report that student-coded agents hold the top five positions in every tournament, that 33 of 40 LLM agents are beaten by simple baselines such as a fixed expected-cost bid, and that when the best LLM is given

What carries the argument

The central object is the APDP benchmark: a two-stage game that couples a reverse first-price sealed-bid auction with a fleet-routing optimization problem. Agents must (1) bid strategically under uncertainty, estimating marginal and opportunity costs of adding tasks to a route, sometimes underbidding to secure future synergies, and (2) solve an NP-hard pickup-and-delivery problem with capacity, pairing, and precedence constraints. The evaluation machinery is a double all-play-all tournament across four road-network topologies, with company-side swapped to ensure fairness, producing around 40k matches and a win-rate ranking. This design replaces unit-test pass/fail with head-to-head performan

Load-bearing premise

The result stands only if the 40 LLM agents that made it into the tournament are a fair sample of what vibe coding produces; the authors say they generated 'substantially more' agents and did significant manual debugging, but do not report how many were discarded or how much repair was done.

What would settle it

Take the same APDP task and run a tournament using the raw, unedited output of each LLM prompt — no manual bug fixing, no discarding of failing agents, with crashes counted as losses. If a majority of those raw agents finish above the simple baselines, or if any reach the top five, the paper's central claim fails. A second check: if re-running with the discarded agents included reverses the ranking, then the reported gap is an artifact of selection.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Existing unit-test benchmarks can pass LLM code that is not competitive in interactive or strategic settings; pass rate is not a proxy for real-world usefulness.
  • LLM-generated code cannot currently be trusted as a drop-in replacement for skilled programmers on complex optimization tasks; the paper's data suggest a need for human oversight.
  • Given a strong reference solution, at least one top LLM degraded rather than improved it, implying in-context exposure to a good solution does not transfer to better code.
  • Benchmark design for code synthesis should include multi-agent competition and open-ended objectives, not just deterministic tests.
  • The APDP testbed, with its pre-LLM human baselines, offers a reusable route for future models; improvements in LLM code synthesis should show up first as movements in this ranking.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the unreported discarded agents were much worse than the surviving 40, the paper understates the human-LLM gap; if the manual debugging substantially improved survivors, it overstates it. Future work should report discard rates and repair effort.
  • The tournament's emphasis on opponent modeling and loss-leading bids rewards strategic reasoning that current LLMs appear to lack; a testable extension is to give LLM agents explicit opponent models derived from past matches and see whether the gap narrows.
  • The failure to improve a winning solution suggests a ceiling on pure in-context repair; editing a whole working algorithm may require interleaved testing that prompt-based iteration does not provide.
  • Because the human agents were developed before LLMs, the comparison avoids contamination, but it also means the human side is a fixed target from 2020; as LLMs improve, the benchmark can track whether they approach that baseline.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper introduces the Auction, Pickup, and Delivery Problem (APDP) as a benchmark for LLM code generation. In APDP, agents bid strategically in a reverse first-price auction and then solve a pickup-and-delivery problem under capacity constraints. The authors compare 40 LLM-coded agents, produced by four LLMs under five prompting strategies, against 17 human-coded agents (12 student agents and 5 baselines) developed before LLMs existed. Using 12 double round-robin tournaments (~38k matches), they report that student agents occupy the top five ranks, that 33 of 40 LLM agents lose to simple baselines, and that a single attempt by the best-performing LLM to improve the winning student solution degraded it from 1st to 10th.

Significance. If the construct-validity concerns below are addressed, this is a valuable benchmark and a useful negative result. The tournament methodology is careful: double all-play-all with company swaps, multiple topologies, and a conservative treatment of human-agent crashes (credited as wins to the LLM). The pre-LLM human code is a natural external baseline, and the open-source release supports reproducibility. The paper is also honest about semantic bugs and manual debugging. Its main contribution is to shift code-generation evaluation from unit tests toward reasoning-driven, multi-agent tasks; even with the selection issue, the magnitude of the 33/40 result suggests the headline conclusion is likely robust.

major comments (3)
  1. [Section 4.2 (Debugging)] The paper states 'We had to generate substantially more agents to get the 40 bug-free ones we evaluated' and 'we observed the need for significant manual effort to achieve bug-free code,' but it does not report how many agents were generated, how many were discarded, or the number and type of repair cycles (prompt-based or human-edited). Because the title, abstract, and conclusions frame the result as an evaluation of 'vibe coding,' this missing operationalization is load-bearing: the 40 evaluated agents may be a best-of-many, expert-repaired sample rather than a direct sample of LLM output. The bias direction is ambiguous, so I am not claiming the headline result is wrong; I am claiming the construct measured is not yet defined. Please provide the full generation funnel (attempts per model/prompt, discard reasons, repair rounds, human debugging time) and a sensitivity analysis, e.g., in
  2. [Section 5.3 and Abstract] The improvement experiment is a single run with a single model, and the model name is inconsistent: 'GPT-5 Pro' here versus 'GPT-5 Thinking' in Section 4.2 and Table 1. The abstract promotes 'the best performing LLM makes the solution significantly worse' as a third headline result, but no variance, number of repetitions, or statistical test is reported. Since the tournament result is already strong, I would either remove this claim from the abstract or support it with multiple runs/models and confidence intervals.
  3. [Section 5.2 and Table 1] The claim that the top 5 spots are 'consistently' held by student agents is based only on aggregate win-rate means. The gap between 5th place (Student 5, 96.5 ± 2.9 wins/tournament) and 6th place (LLM(O, IR, 1), 95.4 ± 2.3) is small relative to the reported standard deviations. Please report per-tournament top-5 memberships or a paired test over the 12 tournaments to support the word 'consistently.'
minor comments (4)
  1. [Section 4.2] Please define 'bug-free' operationally. The current text says 'until all bugs we could identify got resolved,' which is vague and makes the later funnel discussion hard to interpret.
  2. [Section 5.2] The statement '33 out of 40 are beaten by very simple baseline agents' would be easier to verify if the baseline threshold and win/loss criterion were stated next to Table 1 (e.g., strictly lower average wins than ExpCostFixedBid).
  3. [Table 1] The rendering of the table is difficult to parse because agent labels and numbers are not visually separated (e.g., 'Student 1108.167'). A dedicated rank column or explicit spacing would improve readability.
  4. [Section 6] The limitations paragraph appropriately says the paper does not claim optimal LLM performance, but the abstract's third bullet on the improvement experiment is phrased more strongly. Aligning these statements would avoid overclaiming.

Circularity Check

0 steps flagged

No circularity: the comparison rests on an external pre-LLM human benchmark and tournament play, not on a fitted or self-referential derivation.

full rationale

The paper's central claim is an empirical comparison between LLM-generated agents and human-coded agents competing in a tournament. There is no fitted parameter, no quantity derived from the data and then re-predicted, and no prediction that is equivalent to its inputs by construction. The human-coded agents (12 student agents and 5 baselines) were developed before the advent of LLMs and serve as external benchmarks; the LLM agents were generated and evaluated in 12 double all-play-all tournaments with ~40k matches, and the outcome is not normalized or fitted to force the stated result. The improvement experiment in Section 5.3 is also not circular: it takes an external winning human solution as input and measures whether an LLM can improve it, an empirical question answered by tournament ranking. The authors' own Section 4.2 notes that 'We had to generate substantially more agents to get the 40 bug-free ones we evaluated' and mentions 'significant manual effort to achieve bug-free code.' This is a legitimate limitation regarding whether the 40 evaluated agents represent unsupervised 'vibe coding' output, but it is a construct-validity concern, not a circularity pattern: the selection/debugging process is not a fitted parameter renamed as a prediction, and the paper does not use its own results as evidence for the benchmark's construction. There are no load-bearing self-citations or imported uniqueness theorems; the one prior work by an author ([17], on ride-pooling) is cited only as background on pickup-and-delivery problems and plays no role in the derivation of the results. Accordingly, the score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The benchmark uses fixed simulator settings (capacities, costs, topologies, task distributions) rather than fitted free parameters. The main unquantified input is the human selection/debugging effort for LLM agents, which is a process assumption rather than a numeric free parameter. No new physical or conceptual entities are postulated.

axioms (5)
  • domain assumption The APDP problem and the Logist platform are a valid proxy for real-world logistics optimization.
    The entire benchmark's relevance depends on this; introduced in Section 3 and used throughout.
  • domain assumption The 2020 EPFL student agents and EPFL baseline agents fairly represent human graduate-student coding ability.
    The human comparison group is the ground truth for the central claim; described in Section 4.1.
  • domain assumption The LLM prompts contain the same information the human students received.
    Section 4.2 claims the A1 prompt aggregates the project description, slides, and pitfalls; this underpins a fair comparison.
  • standard math The pickup-and-delivery problem is NP-hard, so there is no known closed-form optimal solution and tournament win rate is a suitable metric.
    Section 3.2 argues that NP-hardness forces heuristic, time-bounded solutions, justifying the comparative evaluation.
  • ad hoc to paper The 40 evaluated LLM agents are representative of LLM 'vibe coding' despite being selected from a larger, unreported pool after manual debugging.
    Section 4.2 Debugging states 'We had to generate substantially more agents to get the 40 bug-free ones we evaluated,' but does not quantify the selection or human effort.

pith-pipeline@v1.3.0-alltime-deepseek · 17173 in / 9366 out tokens · 99984 ms · 2026-08-03T20:10:23.614789+00:00 · methodology

0 comments
read the original abstract

The rapid proliferation of Large Language Models (LLMs) has revolutionized AI-assisted code generation. This rapid development of LLMs has outpaced our ability to properly benchmark them. Prevailing benchmarks emphasize unit-test pass rates and syntactic correctness. Such metrics understate the difficulty of many real-world problems that require planning, optimization, and strategic interaction. We introduce a multi-agent reasoning-driven benchmark based on a real-world logistics optimization problem (Auction, Pickup, and Delivery Problem) that couples competitive auctions with capacity-constrained routing. The benchmark requires building agents that can (i) bid strategically under uncertainty and (ii) optimize planners that deliver tasks while maximizing profit. We evaluate 40 LLM-coded agents (by a wide range of state-of-the-art LLMs under multiple prompting methodologies, including vibe coding) against 17 human-coded agents developed before the advent of LLMs. Our results over 12 double all-play-all tournaments and $\sim 40$k matches demonstrate (i) a clear superiority of human(graduate students)-coded agents: the top 5 spots are consistently won by human-coded agents, (ii) the majority of LLM-coded agents (33 out of 40) are beaten by very simple baselines, and (iii) given the best human solution as an input and prompted to improve upon, the best performing LLM makes the solution significantly worse instead of improving it. Our results highlight a gap in LLMs' ability to produce code that works competitively in the real-world, and motivate new evaluations that emphasize reasoning-driven code synthesis in real-world scenarios.

Figures

Figures reproduced from arXiv: 2511.20613 by Naman Goel, Panayiotis Danassis.

Figure 1
Figure 1. Figure 1: Traditional benchmarks (top) focus on problems [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The Auction, Pickup, and Delivery Problem (APDP). [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Network Topologies. From top to bottom and left to [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 11 linked inside Pith

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

    Pith/arXiv arXiv 2023

  2. [2]

    Dario Amodei. 2024. Dario Amodei: Anthropic CEO on Claude, AGI & the Future of AI & Humanity | Lex Fridman Podcast. https://youtu.be/ugvHCXCOmm4?t=

    2024

  3. [3]

    Anthropic. [n.d.]. The Claude 3 Model Family: Opus, Sonnet, Haiku. https: //www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/ Model_Card_Claude_3.pdf. Accessed: 2025-09-12

    2025

  4. [4]

    Anthropic. 2025. Prompt engineering overview. https://docs.claude.com/en/ docs/build-with-claude/prompt-engineering/overview. Accessed: 2025-10-01

    2025

  5. [5]

    AtCoder. 2025. Group Commands and Wall Planning. https://atcoder.jp/contests/ awtf2025heuristic/tasks/awtf2025heuristic_a. Accessed: 2025-09-22

    2025

  6. [6]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models.arXiv preprint arXiv:2108.07732(2021)

    Pith/arXiv arXiv 2021

  7. [7]

    Joel Becker, Nate Rush, Elizabeth Barnes, and David Rein. 2025. Measuring the impact of early-2025 AI on experienced open-source developer productivity. arXiv preprint arXiv:2507.09089(2025)

    Pith/arXiv arXiv 2025

  8. [8]

    Gerardo Berbeglia, Jean-François Cordeau, Irina Gribkovskaia, and Gilbert La- porte. 2007. Static pickup and delivery problems: a classification scheme and survey.Top15, 1 (2007), 1–31

    2007

  9. [9]

    Bloomberg. 2025. AI Coding Assistant Cursor Draws a Million Users Without Even Trying. https://www.bloomberg.com/news/articles/2025-04-07/cursor- an-ai-coding-assistant-draws-a-million-users-without-even-trying. Accessed: 2025-09-23

    2025

  10. [10]

    Noam Brown. 2025. Twitter(X) Thread. https://x.com/polynoamial/status/ 1918746853866127700. Accessed: 2025-09-12

    2025

  11. [11]

    Junchuang Cai, Qingling Zhu, Qiuzhen Lin, Lijia Ma, Jianqiang Li, and Zhong Ming. 2023. A survey of dynamic pickup and delivery problems.Neurocomputing 554 (2023), 126631. https://doi.org/10.1016/j.neucom.2023.126631

    arXiv 2023

  12. [12]

    Satish Chandra and Maxim Tabachnyk. 2024. AI in software engineering at Google: Progress and the path ahead. https://research.google/blog/ai-in- software-engineering-at-google-progress-and-the-path-ahead/. Accessed: 2025- 09-23

    2024

  13. [13]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374(2021)

    Pith/arXiv arXiv 2021

  14. [14]

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. InForty-first International Conference on Machine Learning

    2024

  15. [15]

    Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Alek- sander Madry

    Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Car- los E. Jimenez, John Yang, Leyton Ho, Tejal Patwardhan, Kevin Liu, and Alek- sander Madry. 2025. Introducing SWE-bench Verified. https://openai.com/index/ introducing-swe-bench-verified/. Accessed: 2025-09-12

    2025

  16. [16]

    Zheyuan Kevin Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz. 2025. The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers.A vailable at SSRN 4945566(2025)

    2025

  17. [17]

    Panayiotis Danassis, Marija Sakota, Aris Filos-Ratsikas, and Boi Faltings. 2022. Putting ridesharing to the test: efficient and scalable solutions and the power of dynamic vehicle relocation.Artificial Intelligence Review55, 7 (01 Oct 2022), 5781–5844. https://doi.org/10.1007/s10462-022-10145-0

    doi:10.1007/s10462-022-10145-0 2022

  18. [18]

    Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. 2025. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?arXiv preprint arXiv:2509.16941(2025)

    Pith/arXiv arXiv 2025

  19. [19]

    Hannah Devlin. 2025. Competition shows humans are still better than AI at cod- ing. https://www.theguardian.com/technology/2025/jul/26/competition-shows- humans-are-still-better-than-ai-at-coding-just. Accessed: 2025-09-22

    2025

  20. [20]

    Yihong Dong, Xue Jiang, Jiaru Qian, Tian Wang, Kechi Zhang, Zhi Jin, and Ge Li. 2025. A Survey on Code Generation with LLM-based Agents.arXiv preprint arXiv:2508.00083(2025)

    Pith/arXiv arXiv 2025

  21. [21]

    Mingzhe Du, Anh Tuan Luu, Bin Ji, Qian Liu, and See-Kiong Ng. 2024. Mercury: A code efficiency benchmark for code large language models.Advances in Neural Information Processing Systems37 (2024), 16601–16622

    2024

  22. [22]

    Gauthier

    P. Gauthier. [n.d.]. Aider Polyglot Coding Leaderboard. https://aider.chat/docs/ leaderboards/. Accessed: 2025-09-12

    2025

  23. [23]

    Google. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Mul- timodality, Long Context, and Next Generation Agentic Capabilities. https: //storage.googleapis.com/deepmind-media/gemini/gemini_v2_5_report.pdf. Ac- cessed: 2025-09-12

    2025

  24. [24]

    Antoine Grosnit, Alexandre Maraval, Refinath S N, Zichao Zhao, James Do- ran, Giuseppe Paolo, Albert Thomas, Jonas Gonzalez, Abhineet Kumar, Khyati Khandelwal, Abdelhakim Benechehab, Hamza Cherkaoui, Youssef Attia El-Hili, Kun Shao, Jianye Hao, Jun Yao, Balázs Kégl, Haitham Bou-Ammar, and Jun Wang. 2025. Kolb-Based Experiential Learning for Generalist Age...

    arXiv 2025

  25. [25]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, J. Vanschoren and S. Yeung (Eds.), Vol. 1. https:/...

    2021

  26. [26]

    Nam Huynh and Beiyu Lin. 2025. Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applications. arXiv preprint arXiv:2503.01245(2025)

    Pith/arXiv arXiv 2025

  27. [27]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. Live- CodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. InThe Thirteenth International Conference on Learning Repre- sentations. https://openreview.net/forum?id=chfJJYC3iL

    2025

  28. [28]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VTF8yNQM66

    2024

  29. [29]

    Subbarao Kambhampati. 2024. Can large language models reason and plan? Annals of the New York Academy of Sciences1534, 1 (2024), 15–18

    2024

  30. [30]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode.Science378, 6624 (2022), 1092–1097

    2022

  31. [31]

    Fang Liu, Yang Liu, Lin Shi, Houkun Huang, Ruifeng Wang, Zhen Yang, Li Zhang, Zhongqi Li, and Yuchi Ma. 2024. Exploring and evaluating hallucinations in llm-powered code generation.arXiv preprint arXiv:2404.00971(2024)

    arXiv 2024

  32. [32]

    Johan Los, Frederik Schulte, Matthijs TJ Spaan, and Rudy R Negenborn. 2022. Strategic Bidding in Decentralized Collaborative Vehicle Routing. InInternational Conference on Dynamics in Logistics. Springer, 261–274

    2022

  33. [33]

    Iain Martin. 2025. Vibe Coding Turned This Swedish AI Unicorn Into The Fastest Growing Software Startup Ever. https://www.forbes.com/sites/iainmartin/2025/ 07/23/vibe-coding-turned-this-swedish-ai-unicorn-into-the-fastest-growing- software-startup-ever/. Accessed: 2025-09-14

    2025

  34. [34]

    Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke

  35. [35]

    Hussein Mozannar, Valerie Chen, Mohammed Alsobay, Subhro Das, Sebastian Zhao, Dennis Wei, Manish Nagireddy, Prasanna Sattigeri, Ameet Talwalkar, and David Sontag. 2025. The RealHumanEval: Evaluating Large Language Models’ Abilities to Support Programmers.Transactions on Machine Learning Research (2025). https://openreview.net/forum?id=hGaWq5Buj7

    2025

  36. [36]

    Alexander Novikov, Ngân Vu, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Ko- zlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al . 2025. Alphaevolve: A gemini-powered coding agent for designing advanced algorithms. https://deepmind.google/discover/blog/alphaevolve-a-gemini-powered- coding-agent-for-desig...

    2025

  37. [37]

    OpenAI. 2025. Prompting guidance. https://platform.openai.com/docs/guides/ latest-model#prompting-guidance. Accessed: 2025-10-01

    2025

  38. [38]

    Ruizhong Qiu, Weiliang Will Zeng, James Ezick, Christopher Lott, and Hanghang Tong. 2025. How efficient is LLM-generated code? A rigorous & high-standard benchmark. InThe Thirteenth International Conference on Learning Representa- tions. https://openreview.net/forum?id=suz4utPr9Y

    2025

  39. [39]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundare- san, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis.arXiv preprint arXiv:2009.10297(2020)

    Pith/arXiv arXiv 2020

  40. [40]

    Damian Reyes, Alan Erera, Martin Savelsbergh, Sagar Sahasrabudhe, and Ryan O’Neil. 2018. The meal delivery routing problem.Optimization Online6571, 2018 (2018), 2018

    2018

  41. [41]

    Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. 2024. To the Cutoff... and Beyond? A Longitudinal Perspective on LLM Data Contamination. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=m2NVG4Htxs

    2024

  42. [42]

    Runlopp. 2025. Assessing AI Code Quality: 10 Critical Dimensions for Evaluation. https://www.runloop.ai/blog/assessing-ai-code-quality-10-critical- dimensions-for-evaluation. Accessed: 2025-09-12

    2025

  43. [43]

    Ryan J. Salva. 2025. How are developers using AI? Inside our 2025 DORA report. https://blog.google/technology/developers/dora-report-2025/. Accessed: 2025- 09-25

    2025

  44. [44]

    LMArena Team. [n.d.]. Webdev Arena. https://web.lmarena.ai/leaderboard. Accessed: 2025-09-12

    2025

  45. [45]

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Sid- dhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Gold- blum. 2025. LiveBench: A Challenging, Contamination-Limited LLM Ben...

    2025

  46. [46]

    Sangyeop Yeo, Yu-Seung Ma, Sang Cheol Kim, Hyungkook Jun, and Taeho Kim

  47. [47]

    Doron Yeverechyahu, Raveesh Mayya, and Gal Oestreicher-Singer. 2024. The impact of large language models on open-source innovation: Evidence from github copilot.arXiv preprint arXiv:2409.08379(2024)

    Pith/arXiv arXiv 2024

  48. [48]

    Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, et al. 2025. SWE-bench Goes Live!arXiv preprint arXiv:2505.23419(2025)

    Pith/arXiv arXiv 2025

  49. [49]

    Li Zhong and Zilong Wang. 2024. Can llm replace stack overflow? a study on robustness and reliability of large language model code generation. InProceedings of the AAAI conference on artificial intelligence, Vol. 38. 21841–21849

    2024

  50. [50]

    Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen- Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Dav...

    2025

  51. [2024]

    Etri Journal46, 1 (2024), 106–117

    Framework for evaluating code generation ability of large language models. Etri Journal46, 1 (2024), 106–117

    2024

  52. [2025]

    SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?arXiv preprint arXiv:2502.12115(2025)

    Pith/arXiv arXiv 2025

  53. [8987]

    Accessed: 2025-09-12

    2025