{"id":"71071047-3f48-4451-aee6-e8320cf9c7e8","arxiv_id":"2505.12331","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"OSS-Bench automatically generates live coding benchmarks from real open-source functions and finds that LLM code edits degrade memory safety far more than compilation or test correctness.","lead":"OSS-Bench automatically builds live coding benchmarks by taking real functions from PHP and SQLite, asking LLMs to improve them, and scoring compilability, test pass rate, and sanitizer alerts. It profiles 17 models and finds that memory safety is the weakest dimension, with small models sometimes beating far larger ones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fuzzing-based '10x memory-safety violations' claim lacks per-alert attribution to the LLM edit.","rationale":"The central value proposition of OSS-Bench rests on the claim that natural signals can serve as ground truth for LLM edits. This is plausible for compilability, because a local edit that fails to compile in context is usually the cause. It becomes fragile for test failures and sanitizer alerts, since the project is a large interdependent system and Metric II replaces about 100 functions per iteration. The fuzzing extension provides the strongest evidence for the paper's security narrative, but the 10x figure is computed as a simple ratio of aggregate alert counts, not an attributable count. The pristine PHP baseline already produces 56 unique alerts, and the paper does not show that the additional alerts in GPT-O1 and Qwen3 runs are located in the replaced functions or absent from the baseline under the same fuzzing protocol. I agree with the reader's weakest assumption. A stack-trace-level differential experiment would settle the point; absent that, a conditional verdict is appropriate. I considered other issues (undefined dissimilarity threshold, hand-set scoring parameters, missing artifact links), but they affect ranking details rather than the existence of a valid profiling signal. The attribution gap directly threatens the central quantitative claim, so I recommend no change to the reader's verdict.","tokens_in":14386,"tokens_out":8168,"duration_ms":80357,"concrete_test":"Re-run the fuzzing experiment for the PHP baseline, GPT-O1, and Qwen3-A3B-30B using identical seeds and input corpora, one Metric-II iteration at a time. For each unique sanitizer alert, inspect the symbolized stack trace and record (a) whether the trace contains at least one LLM-replaced function from that iteration and (b) whether the identical alert is produced by the unmodified baseline under the same seed. Compute the ratio of (modified alerts attributable to replaced functions and absent from baseline) to (baseline alerts attributable to the same original functions). If this attributable ratio is materially below 10x, the headline claim should be revised. Separately recompute the totals excluding the 'Undefined' category to state the memory-safety ratio accurately.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's headline security result (Section 3.4) compares aggregate deduplicated sanitizer-alert counts from 500 fuzzing iterations over modified PHP (336 for GPT-O1, 736 for Qwen3-A3B-30B) to one baseline count for unmodified PHP (56), and concludes that LLM edits introduce roughly 10x more memory-safety violations. This inference requires that alerts in modified runs are caused by the LLM-replaced functions. The paper provides no such attribution. Metric III (Section 2) similarly counts sanitizer alerts in compilation/test logs without subtracting or matching alerts produced by the pristine code, even though the fuzzing table itself concedes the pristine baseline already yields 56 bugs. Because Metric II replaces about 100 functions at once per iteration, an alert can arise from any replaced function, from interactions between replacements, or from unmodified code whose paths become reachable after an edit or under a particular fuzz input. Without stack-trace-level matching against the baseline under identical seeds and inputs, the 'new violations' count is not established. The same attribution gap affects test failures in Metric II, but the load-bearing quantitative claim is the memory-safety ratio. Additionally, the totals aggregate a separate 'Undefined' category (e.g., integer overflows), which is not necessarily memory safety; excluding it changes the GPT-O1 ratio from about 6x to about 4.2x and the Qwen3 ratio from about 13x to about 9.4x.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents OSS-Bench, an automated benchmark generator that extracts functions from open-source projects (PHP and SQLite3), prompts LLMs to produce improved versions of those functions, and scores the results with three metrics: compilability, functional test pass rate, and memory-safety via sanitizer alerts, plus a dissimilarity bonus intended to discourage trivial edits. The evaluation profiles 17 LLMs, reports rankings on OSS-Bench(php) and OSS-Bench(sql), and presents an extended fuzzing experiment that claims LLM-generated edits introduce roughly ten times more memory-safety violations than the original code.","tokens_in":14754,"tokens_out":5715,"duration_ms":53118,"significance":"If the claims are supported, OSS-Bench is a valuable contribution: it uses external, natural ground truth (compiler results, official OSS test suites, and sanitizer tools), avoids LLM-generated oracles, and provides a live, continuously updatable benchmark that mitigates static-dataset overfitting. The contamination analysis and the focus on low-level memory safety are also useful and relatively novel for coding-LLM benchmarks. The main caveat is that the headline fuzzing result and several ranking conclusions currently depend on aggregate alert counts and hand-set scoring parameters, so the quantitative claims need stronger support.","major_comments":[{"comment":"The central claim that LLM edits introduce roughly ten times more memory-safety violations is not supported by the reported data. The fuzzing table compares aggregated deduplicated sanitizer-alert counts from 500 iterations over modified PHP (336 for GPT-O1, 736 for Qwen3-A3B-30B) to a single baseline count for unmodified PHP (56), without stack-trace-level matching or baseline subtraction. Because each iteration replaces about 100 functions at once, alerts in modified runs can originate from any replaced function, from interactions between replacements, or from unmodified code whose paths become reachable only after an edit; the baseline itself already exhibits 56 sanitizer bugs. The paper should provide per-alert attribution, such as stack-trace matching against the pristine baseline under identical seeds and inputs, or report baseline-subtracted counts before asserting the 10x ratio.","section":"Section 3.4 and Section 2 (Metric III)"},{"comment":"The final scores and rankings depend on hand-set parameters: 0.69 penalty points per sanitizer alert for PHP, 0.19 for SQLite, a 0.5 factor for memory-leak alerts, a dissimilarity weight w_d = 0.1, and an unspecified minimum dissimilarity threshold. No sensitivity analysis is reported, and these parameters materially affect the memory-safety subscore and thus the final ranking. The paper should demonstrate that model rankings are robust to reasonable variations of these parameters, or provide a principled justification for the chosen values.","section":"Section 2 (Scoring) and Section 3.2 (Table 1)"},{"comment":"The fuzzing table aggregates the 'Undefined' category (e.g., integer overflows) together with SEGV, spatial, and temporal memory-safety errors, but undefined behavior is not necessarily a memory-safety violation. Excluding the 'Undefined' category changes the GPT-O1 total from 336 to 173 and the Qwen3-A3B-30B total from 736 to 386, substantially reducing the reported ratio. The categories should be reported separately, and the wording should distinguish memory-safety bugs from undefined behavior in general.","section":"Section 3.4 (fuzzing table)"}],"minor_comments":[{"comment":"The 'Delta (10%)' column appears to report the weighted bonus w_d * d rather than the raw dissimilarity d defined in Section 2; the notation should be clarified to avoid confusion with the scoring formula.","section":"Section 3.2, Table 1"},{"comment":"There is a typo in the model list: 'Claude-3.7-Sonnect' should be 'Claude-3.7-Sonnet'. Model names such as 'Qwen3-A3B-30B-Instruct' and 'Qwen3-A3B' are also used inconsistently.","section":"Section 3.1"},{"comment":"The sentence 'even the original PHP code shows exhibits some safety bugs' contains a grammatical error and should be rewritten.","section":"Section 3.4"},{"comment":"Reference [16] for CodeArena points to a 2019 IEEE TechDebt paper about Minecraft, which does not match the description 'adaptive scoring based on collective model performance' in Section 5; this citation appears to be incorrect.","section":"References"},{"comment":"The use of 'pass@k' terminology with k=1 is unusual because the evaluation appears to record a single generated completion per function rather than sampling k completions; please clarify the relationship to the standard pass@k estimator.","section":"Section 2 (Metric II) and Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the scope of the journal and the central idea is promising. The main risk is the unattributed alert counting in the fuzzing claim and the arbitrary scoring parameters; if the authors provide per-alert attribution or baseline-subtracted counts and a sensitivity analysis, the paper could become acceptable. I would also encourage the editor to require release of the raw per-iteration data and analysis scripts as part of the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zeynep,\n\nQuick take on OSS-Bench (2505.12331). The core idea is genuinely good: instead of hand-curating tasks, automatically pull function-level snippets from real OSS, ask models to improve them, and score the results with compiler success, test-suite pass rates, and sanitizer alerts. No LLM-generated oracles. The live-update property is real — if the pipeline works, you get a benchmark that tracks repo evolution. The two instantiations (PHP, SQLite) are well chosen, and the evaluation across 17 models yields plausible, interesting patterns (e.g., model size not predicting performance, intra-family differences).\n\nWhat I like: the chained scoring (c1, c2, c3) is a clear way to handle the dependencies between compile, test, and memory-safety. The dissimilarity bonus is a sensible defense against trivial edits, even if the threshold is undefined in the text. The contamination probes in Section 3.3 are a nice attempt to address overfitting, and the case studies are illustrative.\n\nThe soft spots are real but not fatal. The biggest one is the fuzzing-based claim in Section 3.4 that LLM edits introduce ~10x more memory-safety violations. The paper compares total deduplicated alerts in modified runs (336 for GPT-O1, 736 for Qwen3) to the baseline (56) after fuzzing the pristine code. But the modified runs replace ~100 functions at once, and there is no per-alert attribution: no stack-trace matching against the baseline, no control for which replaced function caused the alert, no accounting for alerts in code untouched by the edit. Some of those alerts may be pre-existing bugs made reachable by a different input. Also, the totals include 'Undefined' behavior (e.g., integer overflow) that isn't strictly memory safety; excluding that drops the ratios to ~4x and ~9x. That doesn't invalidate the benchmark, but it does mean the headline number is an upper bound, not a clean causal estimate.\n\nMinor: the sanitizer penalty rates (0.69/0.19 per alert) and the dissimilarity weight (0.1) are hand-set, and the rankings are sensitive to them. No sensitivity analysis. The minimum dissimilarity threshold is never stated. Artifact links are missing (the paper mentions a leaderboard and GitHub but no URLs in the text). These are fixable.\n\nNet: the design is sound and the contribution is new. The fuzzing result needs to be reframed as a comparative indicator rather than a measured causal rate, and the parameter choices need transparency. I would send this to a serious referee — it deserves a review, not a desk reject — but I'd expect major revision on the fuzzing section.\n\nWould I cite it? Yes, as a benchmark methodology, not for the 10x number. Bring it to the reading group if you want a debate about what constitutes a fair attribution in LLM benchmarks.","headline":"Useful new live benchmark for coding LLMs; the 10x memory-safety claim outruns the evidence.","tokens_in":15250,"tokens_out":2623,"would_cite":true,"duration_ms":24297,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A benchmark generator, OSS-Bench, replaces functions in real open-source projects with LLM output and scores the result on compilability, tests, and memory safety.","keywords":["benchmark generator","code LLM evaluation","memory safety","sanitizer alerts","open-source software","function-level code editing","live benchmark","compilability"],"falsifier":"A control experiment in which each LLM's output is replaced by the original function verbatim, running the same compile, test, and fuzzing pipeline on that null edit: if the null-edit condition produces sanitizer alert counts comparable to those attributed to LLM edits, then the memory-safety metric is not measuring the edits themselves.","tokens_in":14196,"feed_emoji":"🐛","tokens_out":6899,"duration_ms":63829,"temperature":0.7,"pith_summary":"OSS-Bench is a method for building coding-LLM benchmarks automatically from real open-source projects. The paper claims that by deleting functions from large C codebases such as the PHP interpreter and SQLite3, asking an LLM to improve each function, and then rebuilding and testing the project, you get three natural ground-truth signals without human-written test cases or LLM-generated oracles: does it compile, do the existing tests still pass, and do sanitizers complain. The paper's central result is that this pipeline ranks 17 LLMs in a way that exposes behavior that simpler benchmarks miss: model size does not predict performance, memorizing OSS code can lead to counterproductive edits, and even the best models' edits introduce roughly ten times more memory-safety violations than the original code under fuzzing. A sympathetic reader would take away that evaluation of coding assistants should be anchored in real, evolving, low-level code rather than static hand-made tasks.","feed_headline":"New benchmark finds LLM edits add ~10x memory bugs","feed_subtitle":"A live benchmark replaces functions in PHP and SQLite with AI output, scoring compile, tests, and sanitizer alerts.","key_machinery":"The load-bearing mechanism is the replacement-and-rebuild loop: extract a function, swap in the LLM's output, recompile the whole project, run the project's own test suite, and collect sanitizer alerts from compilation and test logs. The scoring chain makes the dependencies explicit: $c_1=s_1$, $c_2=s_2\\times c_1/100$, and $c_3=s_3\\times c_2/100$, where $s_1$, $s_2$, and $s_3$ are raw compilability, test, and sanitizer scores, so a model can only earn later-metric credit on code that passed earlier stages. A dissimilarity bonus $d$, computed with difflib as the average number of changed lines per function, is added to discourage conservative no-edit responses.","core_discovery":"The central claim is that OSS-Bench can profile LLM coding ability by measuring the degradation caused when an LLM's rewrite of a single function is dropped into a real, unmodified project. Using libclang, it extracts 10,534 functions from the PHP interpreter and 7,321 from SQLite3; it prompts each model to optimize a snippet while preserving memory safety; then it scores the results with three chained metrics: compilability, the average test-pass rate over 1,000 sampled replacement batches, and the number of unique sanitizer alerts. The paper reports that closed instruction-tuned models lead, that Qwen3 models beat larger Qwen2.5-Coder models despite smaller size, and that extended fuzzing of 500 test iterations finds 336 unique sanitizer bugs for GPT-O1 and 736 for Qwen3-30B-A3B against 56 for unmodified PHP, supporting the claim that LLM edits introduce about ten times more memory-safety violations.","pith_inferences":["If OSS-Bench were extended to per-function differential attribution of sanitizer alerts, the 'ten times more violations' claim would become a statement about individual edits rather than whole-batch replacement, making the ranking more interpretable.","A live benchmark introduces version drift: scores will change as the underlying OSS evolves, so comparable leaderboard standings require pinning commits and re-running baselines on the same commit.","The same replacement-and-rebuild loop could be turned into a CI regression check for LLM code suggestions, flagging edits that compile and pass tests but introduce sanitizer alerts.","The dissimilarity bonus punishes conservative edits, but no metric directly verifies the prompt's second goal of improving efficiency; an oracle that checks whether the edit actually speeds up the code would sharpen the ranking."],"forward_implications":["Benchmarks built this way can be refreshed automatically whenever the upstream project changes, so leaderboard tasks no longer need to be frozen.","A model that passes compilation and tests can still degrade memory safety, so test-pass rate alone is not a sufficient safety signal; the sanitizer layer is necessary.","The chained scoring means a model with a low compilability score is capped in functional and memory-safety scores, so the overall ranking rewards edits that survive all downstream stages.","Model size is not a reliable predictor on these tasks; architecture and training matter more, as shown by Qwen3-8B outperforming the larger Qwen2.5-Coder-32B.","Memorized patterns from training data can actively cause compile errors when applied in a slightly different context, as seen in the variadic-function case where top models injected an extra argument."],"supporting_citations":[{"why":"Supplies the libclang-based function extraction that produces the benchmark tasks.","marker":"[29]"},{"why":"Defines AddressSanitizer, the tool set whose alerts serve as memory-safety ground truth.","marker":"[25]"},{"why":"Provides the OSS-Fuzz precedent for using sanitizer findings in continuous fuzzing.","marker":"[30]"},{"why":"Supplies the FlowFusion PHP fuzzer used in the extended memory-safety evaluation.","marker":"[61]"},{"why":"Serves as the PHP interpreter codebase for OSS-Bench php.","marker":"[26]"},{"why":"Serves as the SQLite3 engine codebase for OSS-Bench sql.","marker":"[27]"},{"why":"Computes the difflib-based dissimilarity delta used to penalize conservative no-op edits.","marker":"[31]"},{"why":"Defines the pass@k evaluation protocol used to sample model outputs.","marker":"[59, 60]"}],"fun_headline_variants":["LLM edits cause 10x more memory bugs in real code","New benchmark: LLM code rewrites add 10x memory flaws","OSS-Bench shows LLM edits spike memory-safety bugs","Real-world test: LLM patches raise memory bugs 10x","Benchmark generator finds LLM rewrites boost memory bugs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes that sanitizer alerts and test failures appearing after a function replacement are caused by the LLM's edit, rather than by pre-existing bugs in the unmodified project or by interactions with unmodified components.","fun_headline_variants_meta":{"raw":{"variants":["LLM edits cause 10x more memory bugs in real code","New benchmark: LLM code rewrites add 10x memory flaws","OSS-Bench shows LLM edits spike memory-safety bugs","Real-world test: LLM patches raise memory bugs 10x","Benchmark generator finds LLM rewrites boost memory bugs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00016,"raw_usage":{"total_tokens":1252,"prompt_tokens":982,"completion_tokens":270,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":181}},"tokens_in":598,"tokens_out":270,"duration_ms":2677,"temperature":1.0,"reasoning_tokens":181,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:36:21.177112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A control experiment in which each LLM's output is replaced by the original function verbatim, running the same compile, test, and fuzzing pipeline on that null edit: if the null-edit condition produces sanitizer alert counts comparable to those attributed to LLM edits, then the memory-safety metric is not measuring the edits themselves.","supporting_citations":[{"cited_title":"libclang: C interface to the clang library","cited_arxiv_id":null,"evidence_quote":"Supplies the libclang-based function extraction that produces the benchmark tasks."},{"cited_title":"Fuzzing the PHP Interpreter via Dataflow Fusion","cited_arxiv_id":"2410.21713","evidence_quote":"Supplies the FlowFusion PHP fuzzer used in the extended memory-safety evaluation."},{"cited_title":"php-src: The php interpreter.https://github.com/php/php-src, 2025","cited_arxiv_id":null,"evidence_quote":"Serves as the PHP interpreter codebase for OSS-Bench php."},{"cited_title":"Richard Hipp","cited_arxiv_id":null,"evidence_quote":"Serves as the SQLite3 engine codebase for OSS-Bench sql."},{"cited_title":"difflib — helpers for computing deltas between objects","cited_arxiv_id":null,"evidence_quote":"Computes the difflib-based dissimilarity delta used to penalize conservative no-op edits."}],"review_version":1}