{"id":"1407c7d3-b5dd-4982-aacb-8e025bbf8aff","arxiv_id":"2511.20613","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Student-coded agents from a 2020 logistics course beat 40 LLM-coded agents in a bidding-and-delivery tournament, with 33 of 40 AI agents losing to a simple baseline.","lead":"This paper creates a package-delivery bidding competition and pits 40 AI-generated agents against 17 human student-written agents built before LLMs. In almost 40,000 matches, the human agents dominate, and the best AI even makes a winning solution worse when asked to improve it.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 40 'vibe-coded' agents are a selected, human-debugged subset of an unreported larger pool; without the selection rate and repair effort, the central comparison's construct validity is unestablished.","rationale":"The reader's weakest assumption identifies exactly the concern that I consider most load-bearing: the connection between the 40 evaluated LLM agents and the paper's 'vibe coding' framing is not established by the reported methodology. The paper itself concedes that substantially more agents had to be generated to obtain 40 bug-free ones and that significant manual effort was required. Without reporting the denominator, the discard criteria, and the number of repair rounds, a reader cannot determine whether the evaluated agents are representative of a single vibe-coding session, a best-of-many selection, or an expert-guided debugging pipeline. This matters because the title and abstract make a general claim about vibe coding; if the actual protocol includes heavy human supervision, the central claim is narrower than advertised.\n\nI considered other possible concerns, including the lack of per-tournament rankings behind 'top 5 spots are consistently held by student agents' and the single-run nature of the improvement experiment. Both are real, but they affect specific secondary claims or the strength of the headline, whereas the operationalization gap affects the validity of the entire comparison. The tournament design is otherwise strong: the human agents are pre-LLM, crashes count as losses for humans, and the double all-play-all structure is a fair and demanding test. Those features make the paper's conservative bias credible, but they do not remove the need to define what 'vibe coding' means operationally. The requested data and ablation would settle whether the concern lands. The reader's CONDITIONAL verdict remains appropriate; I would not move it without first seeing the generation and debugging logs.","tokens_in":17571,"tokens_out":17569,"duration_ms":184290,"concrete_test":"Publish the full pipeline log and run a no-debugging ablation. For each of the 40 evaluated agents, report: (a) total agents generated for that (LLM, prompt) pair, (b) number of repair prompts needed to reach 'bug-free,' and (c) all repair prompt texts. Then re-run one tournament using only the first bug-free agent per (LLM, prompt) pair with no manual repair or best-of-many selection, and compare its win-rate distribution to Table 1. If the unselected first-pass agents are substantially worse, the reported result is conservative and the vibe-coding claim is supported; if they are comparable or better, the 'vibe coding' label is misleading.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The title and abstract frame the central claim as an evaluation of 'vibe coding' — LLM-generated code from natural-language prompts. Section 4.2 (Debugging) states: 'We had to generate substantially more agents to get the 40 bug-free ones we evaluated' and 'we observed the need for significant manual effort to achieve bug-free code.' The paper does not report how many agents were generated per final agent, how many were discarded, or how many prompt-based repair cycles were used. This makes the 40 evaluated agents a selected sample, not a direct sample of unsupervised LLM output. The bias direction is not obvious: if discarded agents were much worse, the reported gap is conservative; but if debugging and selection substantially improved survivors, the comparison is not purely 'AI-generated code' and may reflect an expert-in-the-loop pipeline. Because the manuscript's title, abstract, and conclusion all draw conclusions about vibe coding, the missing operationalization is load-bearing: the result may answer 'can LLMs plus substantial human supervision beat students?' rather than the stated question. This is an internal gap, not a disagreement with consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces the Auction, Pickup, and Delivery Problem (APDP) as a benchmark for LLM code generation. In APDP, agents bid strategically in a reverse first-price auction and then solve a pickup-and-delivery problem under capacity constraints. The authors compare 40 LLM-coded agents, produced by four LLMs under five prompting strategies, against 17 human-coded agents (12 student agents and 5 baselines) developed before LLMs existed. Using 12 double round-robin tournaments (~38k matches), they report that student agents occupy the top five ranks, that 33 of 40 LLM agents lose to simple baselines, and that a single attempt by the best-performing LLM to improve the winning student solution degraded it from 1st to 10th.","tokens_in":17809,"tokens_out":11032,"duration_ms":110617,"significance":"If the construct-validity concerns below are addressed, this is a valuable benchmark and a useful negative result. The tournament methodology is careful: double all-play-all with company swaps, multiple topologies, and a conservative treatment of human-agent crashes (credited as wins to the LLM). The pre-LLM human code is a natural external baseline, and the open-source release supports reproducibility. The paper is also honest about semantic bugs and manual debugging. Its main contribution is to shift code-generation evaluation from unit tests toward reasoning-driven, multi-agent tasks; even with the selection issue, the magnitude of the 33/40 result suggests the headline conclusion is likely robust.","major_comments":[{"comment":"The paper states 'We had to generate substantially more agents to get the 40 bug-free ones we evaluated' and 'we observed the need for significant manual effort to achieve bug-free code,' but it does not report how many agents were generated, how many were discarded, or the number and type of repair cycles (prompt-based or human-edited). Because the title, abstract, and conclusions frame the result as an evaluation of 'vibe coding,' this missing operationalization is load-bearing: the 40 evaluated agents may be a best-of-many, expert-repaired sample rather than a direct sample of LLM output. The bias direction is ambiguous, so I am not claiming the headline result is wrong; I am claiming the construct measured is not yet defined. Please provide the full generation funnel (attempts per model/prompt, discard reasons, repair rounds, human debugging time) and a sensitivity analysis, e.g., in","section":"Section 4.2 (Debugging)"},{"comment":"The improvement experiment is a single run with a single model, and the model name is inconsistent: 'GPT-5 Pro' here versus 'GPT-5 Thinking' in Section 4.2 and Table 1. The abstract promotes 'the best performing LLM makes the solution significantly worse' as a third headline result, but no variance, number of repetitions, or statistical test is reported. Since the tournament result is already strong, I would either remove this claim from the abstract or support it with multiple runs/models and confidence intervals.","section":"Section 5.3 and Abstract"},{"comment":"The claim that the top 5 spots are 'consistently' held by student agents is based only on aggregate win-rate means. The gap between 5th place (Student 5, 96.5 ± 2.9 wins/tournament) and 6th place (LLM(O, IR, 1), 95.4 ± 2.3) is small relative to the reported standard deviations. Please report per-tournament top-5 memberships or a paired test over the 12 tournaments to support the word 'consistently.'","section":"Section 5.2 and Table 1"}],"minor_comments":[{"comment":"Please define 'bug-free' operationally. The current text says 'until all bugs we could identify got resolved,' which is vague and makes the later funnel discussion hard to interpret.","section":"Section 4.2"},{"comment":"The statement '33 out of 40 are beaten by very simple baseline agents' would be easier to verify if the baseline threshold and win/loss criterion were stated next to Table 1 (e.g., strictly lower average wins than ExpCostFixedBid).","section":"Section 5.2"},{"comment":"The rendering of the table is difficult to parse because agent labels and numbers are not visually separated (e.g., 'Student 1108.167'). A dedicated rank column or explicit spacing would improve readability.","section":"Table 1"},{"comment":"The limitations paragraph appropriately says the paper does not claim optimal LLM performance, but the abstract's third bullet on the improvement experiment is phrased more strongly. Aligning these statements would avoid overclaiming.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"I would not reject: the missing funnel statistics and the single-run improvement claim can be fixed in revision. The benchmark is timely, the tournament design is solid, and the main 33/40 result is likely robust. Please ask the authors to report generation counts and repair cycles, and to soften or support the improvement claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what to know: this paper gives real evidence that current LLMs struggle on a complex, multi-agent logistics optimization task when asked to write complete agents. The tournament is 12 double all-play-all rounds with ~38k matches, 40 LLM-coded agents, and 17 human-coded agents from a pre-LLM EPFL course. The top five spots go to students, and 33/40 LLM agents lose to simple baselines like a fixed-bid agent. The design is careful: companies are swapped, student code is not debugged, and student crashes are scored as wins for the LLM. That last choice biases against the paper's conclusion, so I trust the direction of the effect.\n\nThe benchmark itself (APDP) is new and sensible: it combines reverse auctions with capacitated pickup-and-delivery, so an agent has to handle bidding under uncertainty and route optimization. Shipping the code and data is a plus.\n\nNow the soft spots. The title and abstract say 'vibe coding,' but the 40 LLM agents are not a direct sample of raw LLM output. Section 4.2 says they had to generate 'substantially more agents' to get 40 bug-free ones and observed 'significant manual effort' — the discard rate, number of repair cycles, and the amount of author debugging are not reported. So the result is really about LLMs plus expert selection and human fixing, not about what a user gets from a single vibe-coding session. That gap is fixable with transparency, and it does not flip the direction, but it should be fixed before publication.\n\nSecond, the 'LLM degrades the winning solution' experiment (Section 5.3) is a single run with no repeated trials. The paper reports it as a fact, and it is a nice anecdote, but it needs error bars or multiple seeds to be load-bearing.\n\nThird, the student sample is intentionally the top performers from a course tournament, so the comparison is 'good human students vs. a curated LLM pipeline.' That is a fair baseline if stated, but it strengthens the human side more than the paper acknowledges.\n\nOverall: the central claim — that these LLMs are not competitive with trained humans on this kind of strategic planning task — holds up. The paper is a useful counterweight to unit-test benchmarks and deserves serious refereeing. I would send it to a reputable ML/agent venue and ask for the missing funnel numbers, a repeated improvement experiment, and a more careful title.","headline":"The tournament is well-run and the human-over-LLM result is credible, but the 'vibe coding' framing oversells a curated, human-debugged agent pool and the improvement experiment is a single run.","tokens_in":18282,"tokens_out":2633,"would_cite":true,"duration_ms":28655,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper tries to establish that LLM-generated ('vibe-coded') agents cannot match graduate-student coders on a real-world logistics task requiring strategic bidding and route planning.","keywords":["LLM code generation","vibe coding","benchmark","strategic planning","pickup and delivery problem","auction bidding","human vs AI coding","tournament evaluation"],"falsifier":"Take the same APDP task and run a tournament using the raw, unedited output of each LLM prompt — no manual bug fixing, no discarding of failing agents, with crashes counted as losses. If a majority of those raw agents finish above the simple baselines, or if any reach the top five, the paper's central claim fails. A second check: if re-running with the discarded agents included reverses the ranking, then the reported gap is an artifact of selection.","tokens_in":17415,"feed_emoji":"🚚","tokens_out":5330,"duration_ms":53043,"temperature":0.7,"pith_summary":"This paper tries to establish that state-of-the-art LLMs, used the way a typical user would use them (writing code from English prompts, 'vibe coding'), cannot match graduate-student programmers on a real-world, open-ended strategic task. The test problem is a logistics market where agents bid in an auction for delivery tasks and then must route a fleet under capacity constraints. Across twelve round-robin tournaments with roughly forty thousand matches, all top-five finishers were written by students before LLMs existed, and 33 of 40 LLM-written agents lost to simple baselines. A sympathetic reader would care because most coding benchmarks only check whether code passes unit tests, whereas this benchmark measures whether code survives competition against intelligent opponents — the kind of task that determines real-world usefulness.","feed_headline":"Vibe-coded agents lose to grad students in 40k-match tournament","feed_subtitle":"On a logistics auction-and-routing benchmark, simple baselines beat most LLM code, and the best LLM worsens the winning human solution.","key_machinery":"The central object is the APDP benchmark: a two-stage game that couples a reverse first-price sealed-bid auction with a fleet-routing optimization problem. Agents must (1) bid strategically under uncertainty, estimating marginal and opportunity costs of adding tasks to a route, sometimes underbidding to secure future synergies, and (2) solve an NP-hard pickup-and-delivery problem with capacity, pairing, and precedence constraints. The evaluation machinery is a double all-play-all tournament across four road-network topologies, with company-side swapped to ensure fairness, producing around 40k matches and a win-rate ranking. This design replaces unit-test pass/fail with head-to-head performan","core_discovery":"The paper's central claim is that LLM code generation, even from carefully engineered prompts and after manual debugging, is not competitive with human-coded agents on a task requiring planning, optimization, and strategic bidding. In the Auction, Pickup, and Delivery Problem (APDP), agents bid for tasks in a reverse first-price auction and then solve a capacity-constrained pickup-and-delivery routing problem; profit is the difference between winning bids and driving costs. The authors report that student-coded agents hold the top five positions in every tournament, that 33 of 40 LLM agents are beaten by simple baselines such as a fixed expected-cost bid, and that when the best LLM is given","pith_inferences":["If the unreported discarded agents were much worse than the surviving 40, the paper understates the human-LLM gap; if the manual debugging substantially improved survivors, it overstates it. Future work should report discard rates and repair effort.","The tournament's emphasis on opponent modeling and loss-leading bids rewards strategic reasoning that current LLMs appear to lack; a testable extension is to give LLM agents explicit opponent models derived from past matches and see whether the gap narrows.","The failure to improve a winning solution suggests a ceiling on pure in-context repair; editing a whole working algorithm may require interleaved testing that prompt-based iteration does not provide.","Because the human agents were developed before LLMs, the comparison avoids contamination, but it also means the human side is a fixed target from 2020; as LLMs improve, the benchmark can track whether they approach that baseline."],"forward_implications":["Existing unit-test benchmarks can pass LLM code that is not competitive in interactive or strategic settings; pass rate is not a proxy for real-world usefulness.","LLM-generated code cannot currently be trusted as a drop-in replacement for skilled programmers on complex optimization tasks; the paper's data suggest a need for human oversight.","Given a strong reference solution, at least one top LLM degraded rather than improved it, implying in-context exposure to a good solution does not transfer to better code.","Benchmark design for code synthesis should include multi-agent competition and open-ended objectives, not just deterministic tests.","The APDP testbed, with its pre-LLM human baselines, offers a reusable route for future models; improvements in LLM code synthesis should show up first as movements in this ranking."],"fun_headline_variants":["Grad students outcode LLMs in 40k-match tournament","Simple baselines beat 33 of 40 LLM-coded agents","Best LLM worsens top human solution in logistics tournament","LLM-coded agents fail against grad students on strategic tasks","Human-coded agents sweep top 5 in LLM vs. human tournament"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The result stands only if the 40 LLM agents that made it into the tournament are a fair sample of what vibe coding produces; the authors say they generated 'substantially more' agents and did significant manual debugging, but do not report how many were discarded or how much repair was done.","fun_headline_variants_meta":{"raw":{"variants":["Grad students outcode LLMs in 40k-match tournament","Simple baselines beat 33 of 40 LLM-coded agents","Best LLM worsens top human solution in logistics tournament","LLM-coded agents fail against grad students on strategic tasks","Human-coded agents sweep top 5 in LLM vs. human tournament"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000625,"raw_usage":{"total_tokens":2777,"prompt_tokens":839,"completion_tokens":1938,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":1847}},"tokens_in":583,"tokens_out":1938,"duration_ms":14112,"temperature":1.0,"reasoning_tokens":1847,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T20:10:23.614789+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same APDP task and run a tournament using the raw, unedited output of each LLM prompt — no manual bug fixing, no discarding of failing agents, with crashes counted as losses. If a majority of those raw agents finish above the simple baselines, or if any reach the top five, the paper's central claim fails. A second check: if re-running with the discarded agents included reverses the ranking, then the reported gap is an artifact of selection.","supporting_citations":[],"review_version":1}