{"id":"dd1b0161-c969-4e53-a8c9-eceef5da635e","arxiv_id":"2607.26083","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Coding agents write correct parallel CLRS code with little prompting, but meaningful speedups are model-, language-, and algorithm-dependent, with Sonnet strongest and GPT producing none.","lead":"Three AI coding agents were tested on turning serial algorithms into parallel code in C++, Python, and Julia. Correctness came easily; real speedups did not, and depended heavily on the model, language, and algorithm.","discovery_kind":"extension","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The speedup scoreboard (Sonnet 7/15, GPT 0/15, \"no language dominates\") rests on single-sample generations and 3-run timing averages with no uncertainty, and many counted \"wins\" are ratios of 1.00–1.09 or microsecond-scale timings that plausibly flip sign under replication.","rationale":"The reader identified the iterative Cursor protocol (stopping rules, human hints, prompt counting, forbidden parallel cutoffs) as the weakest assumption. That is a legitimate concern about fairness of the agent comparison, but it is not the most load-bearing one, for two reasons. First, the paper is transparent about the protocol and applies the same prompt sequence to all agents; the asymmetry (human causal hints, user-prompt counting) is disclosed, and the \"modest prompting\" claim is robust to reasonable protocol variants since GPT and Sonnet hit correctness in one user prompt everywhere. Second, even under a perfectly fair protocol, the comparative speedup claims could not be trusted as reported, because they rest on n=1 generations and 3-run timings with no uncertainty, with many decisive cells at 1.00–1.09× or in the tens-of-microseconds range. That is where the central claim's support is thinnest: the scoreboard numbers, not the framing. Hence partial agreement — same neighborhood (experimental rigor of the speedup comparison), different mechanism. I recommend UNCHANGED rather than a downgrade: the reader already set CONDITIONAL with conditions covering variance and artifacts, and this concern lands squarely within those conditions. The correctness half of the claim is solid; the speedup half needs exactly the replication/timing-rigor fix described, which is cheap to run and fully determines whether the concern is real. If the recount holds, the paper's modest, non-overclaimed conclusions stand as a useful empirical snapshot.","tokens_in":10802,"tokens_out":2296,"duration_ms":45030,"concrete_test":"Repeat each (agent, algorithm, language) generation k≥5 times in fresh sessions, and re-time each resulting code with ≥30 interleaved serial/parallel runs on a fixed dataset, reporting medians with 95% bootstrap CIs; discard or flag cells where absolute runtime < 1 ms (e.g., all Julia search cells in Table VI). Recount a \"win\" only when the CI lower bound on the serial/parallel ratio exceeds 1. Then recompute Table III and the \"7 of 15\" scoreboard. If Sonnet's lead and GPT's zero survive, the central claim holds; if several 1.0x–1.1x cells flip, the \"heavily algorithm- and language-dependent\" pattern is partly measurement noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's correctness claims are well supported, but the interesting half of the central claim — that speedup is \"heavily algorithm- and language-dependent,\" that Sonnet wins 7 of 15 cells, and that GPT wins none — depends on classifying each cell as speedup>1 or not. Two compounding weaknesses make that classification fragile. (1) Timing noise: §II.C states timings are \"an average of three timed runs after one warm-up run,\" with each run on a *different* dataset, so the three runs are not even replicates — dataset variance and machine noise are conflated, and no spread is reported anywhere. Many cells that determine the win counts sit at the noise floor: Julia BFS 1.01×, Julia DFS 1.03×, Python DFS 1.07×, Python Minimum 1.08×, C++ Dijkstra 1.09× (Table IV/VI). The paper itself flags a hardware-oversubscription spike for C++/Sonnet at N=10^4 that appears at 32 threads but not 16 — direct evidence that single-configuration timings are unstable. (2) Microsecond-scale cells: the Julia search narrative (\"uneven,\" \"dramatically slower\") is built on timings like Binary Search serial 0.020 ms vs parallel 0.015 ms (a 5 µs difference claimed as 1.29×) and Minimum 0.014 vs 0.031 ms. At that scale, timer resolution, JIT state, and GC dominate; the 0.43× and 0.23× Julia \"regressions\" and the 1.29× \"win\" are not trustworthy measurements. (3) Generation sampling: each (agent, language, algorithm) cell is a single generation attempt from a stochastic model. The headline GPT-vs-Sonnet contrast is n=1 per cell; rerunning the same prompts could plausibly move a few cells across the speedup=1 line, which is all it takes to change \"7 of 15\" and \"no measurable speedups.\" None of this shows the conclusions are wrong — Sonnet's lead may well be real — but the specific scoreboard and the per-language win/loss pattern are not statistically supported as reported. This is distinct from, and more load-bearing than, the reader's protocol-fairness concern: even a perfectly symmetric prompting\n","agreement_with_reader":"partial"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The manuscript reports an empirical evaluation of three coding agents (Cursor Composer 2.0, GPT 5.4, Claude Sonnet 4.6) generating parallel multi-threaded implementations of twelve CLRS algorithms (sorting, graph traversal, search) in C++, Python, and Julia. Using an iterative prompting protocol in the Cursor IDE — an initial fixed prompt, error-log feedback, optional human causal hints, and at most two non-improving efficiency prompts — the authors measure iterations to first correctness and parallel speedup against custom serial baselines and (for Julia graphs) Graphs.jl. The headline findings are: (i) GPT 5.4 and Sonnet 4.6 reach correctness on the first user prompt in all languages, while Composer 2.0 needs more iterations, especially in Julia; (ii) correctness does not translate into speedup — GPT 5.4 achieves no parallel speedup in any language, while Sonnet 4.6 achieves speedup in 7 of 15 language–algorithm combinations; (iii) no language dominates: C++ is most consistent on graphs, Python and Julia do best on search, and both regress on some graph algorithms.","tokens_in":11278,"tokens_out":3209,"duration_ms":83438,"significance":"If the results hold, this is a useful addition to the growing parallel-code-generation benchmarking literature (ParEval, PCEBench), and its cross-language × cross-model design on non-embarrassingly-parallel algorithms (sorts, graph traversals with priority queues, selection) addresses a genuine gap: prior multi-language work (e.g., Valero-Lara et al.) covered massively parallel kernels without speedup measurement. The paper deserves credit for several honest practices: it reports negative and regressive results plainly (Table IV/VI regressions), it flags its own anomalous C++/Sonnet N=10^4 spike and attributes it to hardware oversubscription, it justifies the Polyester.jl choice with a direct comparison against Base.Threads (Table V), it includes a third-party library baseline (Graphs.jl, Fig. 4), and it is transparent about the ambiguity of counting user-level prompts versus internal tool calls. The central qualitative message — that correctness is cheap but parallel performance is not, and that runtime efficiency should be a first-class benchmark metric — is well motivated and likely to be of interest to the SE/HPC community. However, the quantitative scoreboard underlying the cr","major_comments":[{"comment":"Tables IV and VI report exactly one serial and one parallel runtime per (language, algorithm), yet the study evaluates three agents. It is never stated which agent's generated code produced these numbers, or whether the table shows the best agent, a fixed agent, or an aggregate. This is load-bearing: the abstract's claims that 'C++ is most consistently parallelizable for graph algorithms' and that 'Python and Julia achieve the largest speedups on search algorithms' rest on these tables, and the cross-model scoreboard (Sonnet 7/15, GPT 0/15) cannot be reconciled with tables that contain no per-agent breakdown. Please either report per-agent runtimes/speedups for graph and search as was done for sorting (Table III), or state explicitly what each row represents.","section":"§III.B–III.C, Tables IV and VI"},{"comment":"The timing protocol — one warm-up run plus the mean of three timed runs, each on a *different* dataset — conflates dataset variance with machine noise, and no spread (min/max or standard deviation) is reported anywhere. Several cells that determine the win counts sit at the noise floor: Julia BFS 1.01×, Julia DFS 1.03×, Python DFS 1.07×, Python Minimum 1.08×, C++ Dijkstra 1.09×, Python BFS 1.00×. The paper itself documents instability of single-configuration timings (the C++/Sonnet N=10^4 spike present at 32 threads but not 16). Worse, the Julia search narrative ('uneven', 'dramatically slower') is built on microsecond-scale cells: Binary Search 0.020 vs 0.015 ms (a 5 µs difference reported as 1.29×) and Minimum 0.014 vs 0.031 ms (0.43×). At that scale, timer resolution, JIT warm-up state, and GC dominate a 3-run mean. The speedup>1 classification of these cells, and hence part of the 7/","section":"§II.C, Tables IV and VI"},{"comment":"Each (agent, language, algorithm) cell appears to be a single generation attempt from a stochastic model, with a human in the loop providing error-cause hints 'if the model continues to fail' (§II.D). The headline cross-model comparison — Sonnet 7/15, GPT 0/15 — therefore rests on n=1 samples per cell, and the frequency and content of human hints per agent is not reported, so prompting effort is not held constant across agents. The adaptive efficiency-prompt stopping rule (continue while improving; stop after two non-improving prompts) further favors agents whose first efficiency prompt happens to help. The correctness claims (first-prompt success) are plausibly robust to this, but the speedup scoreboard is not. At minimum, report the number and nature of human interventions per cell and replicate each cell with a small number of independent sessions (e.g., 3–5 fresh conversations) to sh","section":"§II.B–II.D, §III"},{"comment":"The provenance of the serial baselines is unclear. Speedup is defined against 'custom serial baselines,' and §II.D says the agents were given 'serial implementations of the algorithms' — but it is not stated who wrote these, whether the identical serial code is used for all three agents and all timing runs, or whether the agents modified the serial code during the session. If the serial baseline differs across agents or was itself agent-touched, cross-agent speedup comparisons are confounded (an agent paired with a slow serial baseline wins by construction). Please state the baseline provenance explicitly and confirm a single fixed serial implementation per (language, algorithm) is used in all speedup ratios.","section":"§II, Tables III–VI"}],"minor_comments":[{"comment":"The '7 out of 15' Sonnet claim is stated 'at moderate array sizes of N=10^4', but the graph results (Table IV) are at n=10^5 and search results (Table VI) at n=10^6. Please clarify which sizes enter the 15-cell count, and note that 5 sorting + 4 graph + 3 search = 12 algorithms, so the composition of the 15 cells needs explaining.","section":"§III.C vs §III.B/Tables"},{"comment":"Typo in the initial prompt block: 'C+++' should presumably be 'C++'. Also 'Ensure the continued usePolyester.jl' is missing 'of'.","section":"§II.D"},{"comment":"Typo: 'mulitthreading' → 'multithreading'. Also 'Base.Threads.jl' is not the conventional name; 'Base.Threads' suffices.","section":"§III.C"},{"comment":"'fromIntroduction to Algorithms' is missing a space. More substantively: CLRS [27] is a benchmark of algorithmic *reasoning traces* for neural algorithmic reasoning, not a source of reference serial implementations. Please clarify exactly what was taken from CLRS (the algorithm list only, apparently) and where the serial code came from.","section":"§II.A"},{"comment":"Caption says 'at n=10^6 nodes' — these are arrays, so 'elements' is meant. Same issue in §III.C text ('array size n=10^6, Python achieves...').","section":"Table VI"},{"comment":"The y-axis label 'Iterations to first correctness' with values like '0 2 4 3 5 11 1 11 1 1' in the figure description is hard to parse; the figure would benefit from grouped bars per language with clear tick labels, and from stating whether '1 prompt' includes internal tool-call self-corrections (the text notes this ambiguity — consider reflecting it in the caption).","section":"Fig. 1"},{"comment":"Python uses ProcessPoolExecutor (processes, not threads) while Julia and C++ use threads; since the abstract and conclusions frame the study as 'parallel multi-threaded implementation', please discuss whether the process-based Python results are comparable to the thread-based ones, particularly given pickling overhead at these sizes.","section":"§II.E"},{"comment":"Reference [29] (LASSI) is incomplete ('IEEE.' with no venue/year). Reference [7] has a stray '&amp;' artifact. Several 2026-dated arXiv references should be checked for stability.","section":"References"},{"comment":"No artifact (generated code, prompts logs, timing scripts, raw data) is mentioned. Given that the central claims are empirical and the community standard in cs.SE is moving toward artifact availability, releasing the benchmark harness and generated implementations would substantially strengthen the paper.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The correctness results and the qualitative message (correctness ≠ parallel performance) are solid and worth publishing. The quantitative scoreboard that makes the paper interesting — Sonnet 7/15, GPT 0/15, language-by-category claims — currently depends on single generations, 3-run means over non-replicate datasets, and microsecond-scale timings, and Tables IV/VI do not even identify which agent produced the timed code. All of this is fixable within the paper's existing scope (replication, uncertainty reporting, per-agent tables), so major revision rather than rejection. The authors' own candor about the N=10^4 spike and prompt-counting ambiguity suggests they will engage constructively. Note the author list includes a prominent figure in the Julia community; the Polyester.jl mandate is justified empirically in Table V, which mitigates any concern about tool favoritism, but the editor may wish to ensure the per-agent Julia iteration counts are not colored by the mandated library choice."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing: this is a clean empirical snapshot that coding agents can parallelize CLRS-style sorts, graphs, and searches across C++/Python/Julia with modest prompting, yet real speedup is uneven—and the paper is honest that GPT can be correct and still never faster.\n\nWhat is actually new is the controlled cross of three current agents, three languages, and non-embarrassingly-parallel CLRS categories, with dual metrics (user prompts to correctness plus serial/parallel speedup, including some Graphs.jl baselines). Prior ParEval/PCEBench/kernel work already said parallel is harder; this adds a language×agent matrix on algorithms that need more reasoning than AXPY. Protocol is readable, tables mostly match the prose, and they do not pretend one language wins.\n\nSoft spots, in proportion. Correctness claims look solid. The interesting half—Sonnet 7/15, GPT 0/15, category-dependent language wins—rests on single generations per cell and three-run averages on different datasets with no spread. Several “wins” are 1.01–1.09×; Julia search cells are tens of microseconds where timer/JIT/GC dominate. Their own 32-vs-16-thread spike shows configuration sensitivity. Mandating Polyester, forbidding size cutoffs, optional human causal hints, and counting user prompts rather than tool calls all shape who looks good. No released code or model snapshots. None of that invents a fake result; it means the scoreboard is thinner than the abstract’s confidence.\n\nWho it is for: people building or buying coding agents for HPC/SE who need a reminder that pass@k is not enough. Serious referee material if they add variance, multi-sample generations, and either ablate human hints or fully script them. I would send it to review, not desk-reject. Engage if you care about agent evaluation practice; skim the tables if you only need the qualitative takeaway.","headline":"Useful bake-off showing correctness is easy and speedup is sparse, but the Sonnet-7/15 vs GPT-0 scoreboard sits on noisy single-shot timings.","tokens_in":12414,"tokens_out":503,"would_cite":false,"duration_ms":16277,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Coding agents write correct parallel code with little prompting, but real speedups depend on the algorithm, the language, and the model—and one strong model gets none.","keywords":["LLM","Agentic AI","HPC","Parallel Algorithms","Julia","code generation","runtime efficiency","CLRS"],"falsifier":"Rerun the same twelve algorithms and three languages with a stricter no-human-hint protocol, or with free choice of parallel cutoffs and backends, and check whether GPT still never speeds up and whether Sonnet still leads on the same language–algorithm pairs at matched problem sizes and thread counts.","tokens_in":11988,"feed_emoji":"⚡","tokens_out":919,"duration_ms":21151,"temperature":0.7,"pith_summary":"This paper asks whether today’s AI coding agents can turn ordinary serial algorithms into working parallel code, and whether that code actually runs faster. It tests three agents on twelve classical sorting, graph, and search algorithms in C++, Python, and Julia, counting how many user prompts it takes to get correct code and then measuring speedup against serial baselines. The agents usually reach correctness with modest effort, but meaningful speedup is uneven: one model produces the strongest gains, another produces none despite being correct, and no language wins every category. The point is practical—if agents are going to help with parallel software, accuracy alone is not enough; runtime efficiency has to be treated as a first-class measure of success.","feed_headline":"Agents write correct parallel code—speedups are another story","feed_subtitle":"Across C++, Python, and Julia, one model gains speed on many tasks; another never does despite passing tests.","key_machinery":"An iterative, agent-in-the-IDE protocol: start from a fixed serial baseline and a shared parallelization prompt, feed back benchmark or error output until correctness, then issue a bounded number of efficiency prompts, counting user-level prompts while measuring speedup versus custom serial code and third-party libraries on the same inputs.","core_discovery":"Modern coding agents can generate functionally correct multi-threaded versions of CLRS sorting, graph, and search algorithms across C++, Python, and Julia with only modest user-level prompting, yet turning that correctness into real parallel speedup is strongly algorithm- and language-dependent. Sonnet 4.6 delivers the strongest overall performance gains, GPT 5.4 yields no measurable speedups despite consistent correctness, C++ is most consistently helpful on graph algorithms, and Python and Julia show their largest gains on search—while sometimes regressing on graphs. Correctness is therefore an incomplete score for parallel code generation; runtime efficiency must sit beside it.","pith_inferences":["Training and eval sets that reward only compilable parallel patterns may entrench “correct but slow” agents like the GPT result here.","Forbidding serializing cutoffs in the prompt may hide the practical parallelizations engineers actually ship, so a follow-up with allowed thresholds would test ecological validity.","The recurring Julia type and threading-package failures suggest language tooling and docs quality shape agent success as much as model skill.","Extending the same protocol to GPUs and larger problem sizes would show whether the correctness–speed gap widens or narrows under heavier parallelism."],"forward_implications":["Benchmarks for coding agents on parallel work should score runtime speedup, not only pass/fail correctness.","Model choice matters for performance engineering even when all models can write correct parallel code.","Language choice should be matched to algorithm class: C++ for graph-style work, Python/Julia more often for search-style gains.","Agent-generated parallel code can still regress versus serial on some graph and search cases, so human performance review remains necessary.","Performance-aware prompting and automated profiling feedback are natural next levers if the correctness–speed gap is to close."],"fun_headline_variants":["Correct parallel code is easy; real speedups are not","Agents pass tests on parallel CLRS—few deliver speedups","Sonnet gains parallel speed; GPT stays correct but flat","C++ helps graphs; Python and Julia shine on search","Parallel accuracy is common; meaningful speedup is not"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That this particular prompting loop—fixed start, error feedback, occasional human hints when stuck, and at most two non-improving efficiency prompts—fairly reveals each agent’s true ability to produce fast parallel code.","fun_headline_variants_meta":{"raw":{"variants":["Correct parallel code is easy; real speedups are not","Agents pass tests on parallel CLRS—few deliver speedups","Sonnet gains parallel speed; GPT stays correct but flat","C++ helps graphs; Python and Julia shine on search","Parallel accuracy is common; meaningful speedup is not"]},"model":"grok-4.5","effort":"low","cost_usd":0.001991,"raw_usage":{"total_tokens":987,"prompt_tokens":874,"num_sources_used":0,"completion_tokens":65,"cost_in_usd_ticks":19908000,"prompt_tokens_details":{"text_tokens":874,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":48,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":874,"tokens_out":65,"duration_ms":2440,"temperature":1.0,"reasoning_tokens":48,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T11:29:57.233082+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Rerun the same twelve algorithms and three languages with a stricter no-human-hint protocol, or with free choice of parallel cutoffs and backends, and check whether GPT still never speeds up and whether Sonnet still leads on the same language–algorithm pairs at matched problem sizes and thread counts.","supporting_citations":[],"review_version":1}