{"id":"26fb2f1d-2178-4330-af77-14fe20922642","arxiv_id":"2608.09740","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Showing failing security tests to an LLM during repair fixes a subset of insecure candidates, but visible test success leaves a measurable hidden coverage gap.","lead":"This paper tests whether showing security tests to a code-generating AI, either before or during generation, makes the output both useful and secure. It found that executable feedback repairs some failed candidates in every setup tested, but visible test success alone is not enough to guarantee hidden safety.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing no-information retry control confounds feedback content with revision attempt; repair counts may be generic self-correction, not executable feedback.","rationale":"The reader's conditional verdict focuses on oracle validity and the absence of significant adjusted comparisons. Reading the paper in good faith, the shared-candidate design, the visible/hidden partition discipline, and the leak audits are careful and the repair counts are direct paired transitions. The most load-bearing unaddressed gap is not the construction of the hidden oracles, which is mitigated by the six-gate executable and human-review process, but the absence of a no-information retry control. Without such a control, the paper cannot separate 'executable feedback repairs' from 'a second attempt repairs.' This is a standard control in the self-debugging literature (e.g., Chen et al. 2023 compare execution feedback against retry without feedback) and is inexpensive to add to the existing grid. The oracle-validity concern is real but secondary: if the hidden oracles were imperfect, the absolute counts would be suspect, but the missing retry control threatens the mechanism-level attribution even under perfect oracles. Because this is an addressable design gap rather than a demonstrated error, the appropriate posture remains conditional acceptance; a single added control condition would settle whether the central attribution holds. I therefore keep the reader's CONDITIONAL verdict unchanged, while disagreeing with the reader's identification of the weakest assumption as oracle validity.","tokens_in":14328,"tokens_out":10037,"duration_ms":105223,"concrete_test":"Add a control condition R to the existing 465 task-repetition cells: load the byte-identical B0 response; whenever at least one visible test fails, prompt the model for a complete replacement module with the instruction 'Your solution is incorrect, please fix it' and no failing test identifiers, names, logs, or security hints. Use the same revision budget (at most two revisions, same token limits) and count hidden joint repairs and regressions exactly as in Table 4. If R repairs a comparable number (within roughly 20% of M's 80), the executable-feedback attribution collapses and the central claim must be reframed as 'revision attempts can repair,' with the value of test failure content unproven. If R repairs substantially fewer (for example, under 30), the feedback-content attribution survives. Run the same R condition against the B3 shared candidates for the B6−B3 comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that executable feedback repairs secure-code generation rests on paired repair counts in Table 4 (M−B0 repairs 80, B4−B0 repairs 83), but every feedback condition confounds the content of the failure message with the act of attempting a revision. B0 is a single generation; B4/B5/M load the byte-identical B0 response, and when a visible test fails the model is prompted to request a complete replacement module. There is no shared-initial-candidate control that requests a revision without revealing any failing test information (for example, 'your solution is incorrect, please fix it' with no case identifiers, logs, or security hints). The paper explicitly controls for differences in initial generation (Section 2.3), but it does not control for the generic effect of a second attempt. Consequently, the observed 80 repairs could be largely or entirely a self-correction effect: the model might repair its own defects when simply asked to try again, with the executable failure messages contributing little. This is distinct from the oracle-validity concern: even if every hidden oracle is correct and well-separated from visible tests, Table 4 cannot attribute the transitions to executable feedback without a no-information retry baseline. The B6−B3 comparison (18 repairs) suffers from the same confound. Because the paper's contribution is specifically about security tests as executable feedback, this missing control is load-bearing: it determines whether the mechanism is test-derived information or merely iterative prompting.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SecTDD, a scaffold that separates three factors in secure LLM code generation: whether visible tests are shown upfront, whether failed executions trigger revision, and how failures are selected and represented. Across 2,705 trajectories on 31 task instances from CWEval, SALLM, and CodeGuard+, and two model families, it reports that showing all visible tests upfront changes hidden functional-and-security joint success by +19.3 percentage points on average but helps only seven of nine benchmark-model conditions; that shared-initial-candidate feedback repairs 80 initially failing candidates with zero joint regressions for structured feedback and 83 with three joint regressions for fixed raw feedback; that structured and raw feedback are nearly indistinguishable head-to-head (six wins, six losses, 453 ties); and that 5.6% to 18.8% of visible-joint passes fail hidden evaluation (TFOR). The authors conclude with a mechanism-level claim: executable feedback can repair secure-code generation, but its opportunity is bounded by test coverage and its effect depends on the model, task, and feedback entry point.","tokens_in":14664,"tokens_out":6575,"duration_ms":63048,"significance":"If the result holds, the paper makes a valuable and unusually careful contribution to secure-code-generation evaluation. Its strengths include byte-identical initial candidates for repair comparisons, behavior-partitioned visible and hidden tests, paired repair/regression counts as direct observations, a clear distinction between internal and external benchmarks, and a detailed reproducibility package. The head-to-head M versus B4 comparison is particularly clean because both conditions share the same initial candidate and differ only in failure representation. The primary weakness is the absence of a no-information retry control, which prevents attributing the observed repairs to the content of executable feedback rather than to the act of a second attempt. The external-oracle validity concern is reduced by the six-gate qualification and human review, but the paper would be stronger with explicit agreement or sensitivity evidence. These issues are addressable and do not undermine the value of the head-to-head representation comparison or the TFOR diagnostic.","major_comments":[{"comment":"The central attribution of the repair counts to executable feedback is confounded with the generic effect of a second attempt. B0 is a single generation, while B4/B5/M and B6 are multi-round policies, and there is no condition in which the model is asked to revise its byte-identical initial candidate without receiving any failing-test information. Under this design, the 80/83 repairs in Table 4 could in principle be produced by self-correction alone (for example, a prompt such as 'your solution is incorrect, please fix it' with no case identifiers, logs, or security hints). Section 2.3 correctly states that shared initial candidates control for differences in initial generation, but that control does not separate feedback content from revision attempt. Please add a no-information retry baseline under the same revision and token budgets, and report whether the feedback conditions differ from it in repair counts; this is load-bearing for the paper's mechanism-level conclusion that executable feedback, not merely iterative revision, repairs secure-code generation.","section":"§5.2 (Table 4) and §3.2"},{"comment":"The external hidden oracles for SALLM and CodeGuard+ are constructed by a machine-assisted pipeline in which LLMs propose secure implementations and vulnerable mutants, and the same model families are under test; the manuscript acknowledges that correlated model errors remain possible. The six-gate qualification and the manual review by three doctoral students substantially reduce this risk, but the paper does not report inter-reviewer agreement or a sensitivity analysis that drops tasks judged least certain. Because the repair counts for SALLM and CodeGuard+ and the TFOR values in Table 5 depend entirely on these overlays, please provide agreement statistics (for example, the number of packets each reviewer accepted or rejected and any disagreement-resolution counts) or explicitly scope the external claims as conditional on overlay validity.","section":"§4.2 (gates 1–6) and §7 (Oracle and researcher bias)"},{"comment":"The confirmatory comparisons never reach an adjusted p below 0.05, and the paper is appropriately cautious in most of its prose. However, the RQ2 answer states that executable feedback 'can causally repair' failed shared candidates, and this causal language goes beyond what the inferential statistics support, especially without the no-information retry control requested above. Once such a control is added, please report a paired test or bootstrap confidence interval for the difference in repair counts between the feedback conditions and the retry-only condition. As it stands, the repair counts are descriptive transitions, and the paper's own statistical caveat (§7) should be carried into the RQ2 answer so readers do not read a causal claim into an unadjusted count.","section":"§4.5 and §7 (Statistical conclusion validity)"}],"minor_comments":[{"comment":"The note that the final B6 label supersedes an early living-protocol label is transparent, but the paper should state when the label change was frozen relative to the confirmatory analysis. If any confirmatory result was produced under the old B6 meaning before the supersession, that fact should be reported explicitly.","section":"§3.2"},{"comment":"Reporting the number of packets rejected at each of the six qualification gates would help readers gauge how difficult oracle construction was and how selective the final 11-task SALLM and 11-task CodeGuard+ sets are.","section":"§4.2"},{"comment":"The dashes in Table 3 denote methods not run, but the table does not indicate which methods are unavailable by design (B1/B2 only on CWEval, B5 not on SALLM) versus by grid choice; a footnote would clarify this for readers.","section":"Table 3"},{"comment":"TFOR is a ratio with a denominator that can be small, and the paper correctly reports numerator and denominator. Please state explicitly how zero denominators are handled in pooled counts, even if no zero denominator occurs in the reported data.","section":"Eq. (2)"},{"comment":"The sentence 'the robust effect belongs to receiving executable feedback' overstates the evidence because of the missing no-information retry control; consider rewording to 'the robust effect belongs to the revision attempt together with executable feedback' until that control is available.","section":"§6.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a software-engineering or security venue. The main risk is the missing no-information retry control, which is a fixable experimental gap rather than a fatal flaw. The oracle-validity concern is mitigated by the authors' unusually detailed six-gate qualification and human review, but the paper should report agreement or sensitivity evidence. The B6 label supersession should be checked for timing relative to the frozen analysis. I would not reject on the current evidence; the head-to-head representation comparison and the coverage diagnostic are valuable regardless of the retry-control outcome."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a serious, carefully scoped empirical study of security tests as feedback for LLM code generation, with real methodological strengths. Its main load-bearing claim — that executable feedback causes repair — is somewhat undercut by the absence of a no-information retry control.\n\nWhat's new: the three-way separation of upfront tests, feedback availability, and failure representation; byte-identical shared initial candidates for repair comparisons; the TFOR diagnostic for visible-to-hidden overfitting; and the outcome-unfiltered CodeGuard+ breadth wave showing that pilot selection inflates apparent treatment opportunity. The paper does well by being honest about its limits: it reports no adjusted p below 0.05, treats tasks as the inference unit, runs leak audits, discloses the machine-assisted oracle construction, and repeatedly warns against reading it as a leaderboard.\n\nThe soft spot that matters most is the missing retry-only control. B4/B5/M share the initial candidate with B0, but they also give the model a second attempt. Without a condition that says \"your solution is incorrect, please fix it\" while revealing no failing test information, the 80 repairs in Table 4 cannot be fully attributed to the content of the executable failure messages. Generic self-correction could account for a large share of the repairs, and the B6-B3 comparison has the same confound. This is distinct from the oracle-validity concern, which I think is adequately handled: the six-gate qualification, executable mutation gates, and human review give reasonable confidence that the hidden partitions are behaviorally distinct. Machine-assisted construction remains a minor worry, but not a demonstrated error.\n\nThe lack of statistical significance is disclosed and honestly discussed; with 31 task instances, it's expected. The protocol-label supersession is disclosed too, and doesn't affect the primary contrasts.\n\nThis paper deserves a serious referee, not a desk reject. A good reviewer should ask for a retry-only control to isolate the feedback content effect, and the authors should be able to add one. Even if that control weakens the causal claim, the methodology, the honest reporting, and the TFOR diagnostic are valuable on their own. I'd bring it to a reading group and would likely cite it for its design and diagnostic, while being careful not to cite the causal claim as settled.","headline":"A careful, well-scoped empirical study of security tests as feedback for LLM code generation, with real methodological strengths, but the central causal claim is weakened by the absence of a no-information retry control.","tokens_in":15124,"tokens_out":1837,"would_cite":true,"duration_ms":18673,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that executable security-test feedback can repair insecure LLM-generated code, but that the repair opportunity is bounded by test coverage and varies with the model, task, and feedback entry point.","keywords":["secure code generation","executable specifications","test feedback","LLM code repair","test coverage","joint functional-security evaluation","vulnerability repair","SecTDD"],"falsifier":"Take the frozen SALLM and CodeGuard+ qualification packets and have an independent team, without access to the original construction labels or LLM reviewers, manually execute the hidden cases against every visible-joint-pass but hidden-joint-fail candidate. If a meaningful number of those candidates pass the hidden cases under direct human execution, or if the hidden cases turn out to be near-duplicates of visible inputs, then the TFOR rates and repair transitions would not measure what the paper claims.","tokens_in":14089,"feed_emoji":"🛡️","tokens_out":8487,"duration_ms":67420,"temperature":0.7,"pith_summary":"The paper is trying to establish that executing security tests during LLM code generation and returning the failures to the model is a genuine repair mechanism, not just a prompt improvement. On shared initial candidates, structured feedback turned 80 hidden joint failures into successes with zero joint regressions, while fixed raw feedback produced 83 repairs and three regressions. Showing all visible tests upfront raised hidden joint success by 19.3 percentage points on average in seven of nine benchmark-model conditions but lowered it in two. The paper also finds that 5.6% to 18.8% of candidates passing all visible tests still fail hidden behavior families, so test coverage sets an upper bound on what any feedback loop can repair. A sympathetic reader would care because this separates the question of whether executable feedback works from which policy is best, and it gives tool builders measurable levers: trigger counts, repair transitions, and coverage gaps rather than a single leaderboard score.","feed_headline":"Failed security tests can fix LLM code—hidden tests still fail","feed_subtitle":"Repaired 80 hidden-failing candidates with zero regressions; 5.6–18.8% of visible passes still fail hidden attacks.","key_machinery":"The SecTDD scaffold is a controlled test-feedback workflow that separates three decisions usually conflated in prior work: whether tests are shown in the initial prompt, whether failed executions trigger a revision, and how failures are selected and represented. Its load-bearing feature is behavior-partitioned oracles, in which visible and hidden functional/security tests are split by attack mechanism rather than randomly, and the hidden partition is mounted only once after stopping. Repair comparisons reuse byte-identical initial candidates from a reference condition, isolating the feedback policy as the only variable. The test-feedback overfitting rate (TFOR) quantifies the coverage-limited stopping problem by measuring the fraction of visible-joint passes that fail hidden joint evaluation.","core_discovery":"The paper's central claim is that running security tests against generated code and feeding failures back to the model can causally repair a subset of failed candidates, with the effect visible across two model families and three benchmarks. The load-bearing evidence is paired: the feedback conditions reuse byte-identical round-0 responses from the requirement-only condition, so any final difference is attributable to the feedback policy. From those shared candidates, structured feedback repairs 80 hidden joint failures with no joint regressions, fixed raw feedback repairs 83 with three regressions, and the two strategies tie in 453 of 465 cells. The same mechanism appears when feedback is applied to an upfront-test candidate, repairing 18 of 47 triggered cells with no joint regression. The paper deliberately scopes its conclusion: no tested feedback representation wins everywhere, and when all visible tests pass, no feedback is generated even if a hidden attack family would fail.","pith_inferences":["A testable extension suggested by these results: compare feedback that names the weakness class against feedback that only shows the failing assertion, holding everything else fixed, to see whether the near-tie between structured and raw feedback survives when the message content is the only variable.","The paper's outcome-unfiltered breadth wave implies that pilot studies that preselect tasks with known failures will systematically overstate feedback gains; readers should expect effect sizes to shrink when tasks are chosen without looking at outcomes.","If coverage limits are as binding as the TFOR numbers suggest, the next generation of secure-code tools should treat test-suite diversity as a first-class resource, generating new tests online when the visible suite goes silent to extend the repair opportunity.","Because the external oracles were constructed with LLM assistance, an independent human-built oracle set on the same benchmarks would be the natural next test of whether the repair counts reflect true vulnerability repair or shared model bias."],"forward_implications":["Secure-code evaluation should report functionality and security on the same candidate, because a security-only score can reward broken or empty programs.","Iterative methods should report treatment opportunity—trigger counts, repairs per trigger, and four-state transitions—alongside final pass rates, or a strong base model can be mistaken for a good feedback policy.","When visible tests pass but hidden attacks fail, adding more revision rounds is inert; the useful investment is more behaviorally diverse visible tests.","Structured security-priority feedback and raw logs are nearly equivalent head-to-head (6 wins, 6 losses, 453 ties), so compact structured messages are an engineering convenience, not a proven accuracy gain.","A generic security reminder or all-tests prompt can hurt as well as help, so upfront specification is a model-dependent mechanism rather than a monotonic safety intervention."],"supporting_citations":[{"why":"CWEval supplies the outcome-driven functional/security oracles used for the internal confirmatory tasks.","marker":"[21]"},{"why":"SALLM provides the external security-centric Python task set on which the visible/hidden overlays are built.","marker":"[26]"},{"why":"CodeGuard+ supplies the joint correctness/security benchmark and motivates the joint metric; its tasks are adapted into the outcome-unfiltered breadth wave.","marker":"[8]"},{"why":"Dai et al. motivates evaluating functionality and security on the same candidate and supplies the SAFE metric used as sensitivity analysis.","marker":"[4]"},{"why":"Self-debugging demonstrates that execution results can drive LLM revision, the mechanism the feedback conditions build on.","marker":"[3]"},{"why":"VRpilot returns compiler, functional, and sanitizer feedback for vulnerability repair, grounding the structured-feedback design.","marker":"[13]"},{"why":"CodeRL's unit-test feedback loop supports the claim that execution feedback can improve functional generation.","marker":"[14]"},{"why":"EvalPlus shows that weak test suites can accept incorrect programs, the coverage-limitation concern that TFOR quantifies.","marker":"[15]"}],"fun_headline_variants":["Security tests as specs fix some LLM code, not all","Security tests as executable specs: repairs, coverage limits","LLM code repair via security tests: 80 fixes, 0 regressions","Security feedback improves LLM code, hidden tests persist","Executable specs: security tests guide LLM fixes, limits remain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire causal story depends on the hidden tests being valid and behaviorally distinct from the visible tests; these hidden sets were partly built by LLMs with human review afterward, so if that construction process encoded the same blind spots as the models under test, the repair counts and coverage rates would not measure real security repair.","fun_headline_variants_meta":{"raw":{"variants":["Security tests as specs fix some LLM code, not all","Security tests as executable specs: repairs, coverage limits","LLM code repair via security tests: 80 fixes, 0 regressions","Security feedback improves LLM code, hidden tests persist","Executable specs: security tests guide LLM fixes, limits remain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000647,"raw_usage":{"total_tokens":2984,"prompt_tokens":972,"completion_tokens":2012,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":1925}},"tokens_in":588,"tokens_out":2012,"duration_ms":13294,"temperature":1.0,"reasoning_tokens":1925,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:14:05.181781+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the frozen SALLM and CodeGuard+ qualification packets and have an independent team, without access to the original construction labels or LLM reviewers, manually execute the hidden cases against every visible-joint-pass but hidden-joint-fail candidate. If a meaningful number of those candidates pass the hidden cases under direct human execution, or if the hidden cases turn out to be near-duplicates of visible inputs, then the TFOR rates and repair transitions would not measure what the paper claims.","supporting_citations":[],"review_version":2}