{"id":"ccf99e4f-1106-49a1-bc0c-abb9c1f28cba","arxiv_id":"2501.15804","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Adapting program inputs with a layerwise validity score and genetic search improves deep code model accuracy by up to 8.78% without retraining.","lead":"CodeImprove is a method that checks whether a code snippet is outside a deep code model's handling ability, then rewrites the snippet with supposedly semantics-preserving transformations so the model can classify it correctly. It reports up to 8.78% absolute accuracy gains on vulnerability detection and defect prediction without retraining the models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central accuracy claim is only valid if all 15 transformations preserve semantics, but Table II operator 15 deletes printf statements and RQ5 checks a single example; this makes the reported gains uninterpretable until differential testing is done.","rationale":"I read the paper in good faith: the construction is coherent, the two-phase design is clearly described, and the public artifact is a real plus. The central claim, however, rests on a universal semantic-preservation guarantee for the transformation set. The paper's own Table II contradicts that guarantee: operator 15 deletes printf statements, which can alter observable program behavior and therefore change the ground-truth class in a behavior-defined task like CodeChef. RQ5 is a single-example demonstration, not a verification, and Section VII explicitly names the lack of ground truths as an internal threat. This is exactly the load-bearing assumption the reader identified, and it is the most direct way the headline accuracy and relative-improvement numbers could be inflated. A differential-testing pass over the artifact would settle the question: if transformations change behavior on a non-negligible fraction of adapted inputs, the accuracy gains are not attributable to adapting the same program; if they do not, the conditional concerns about semantic preservation are resolved. Since the reader already returned CONDITIONAL for this reason, my read does not change the verdict.","tokens_in":20146,"tokens_out":7903,"duration_ms":77188,"concrete_test":"Using the released artifact, reproduce Table III's adapted outputs and run differential testing on every transformed sample: compile each original and adapted C program with gcc -O0 and compare exit code, stdout, stderr, and a timeout bound on the same inputs used by the CodeChef or Devign harness. For transformations that are hard to execute, additionally run a lightweight equivalence checker or a second independent semantic-preservation test. If any adapted sample's observable behavior differs from its original, recompute Table III with that sample's original ground-truth label marked invalid. If the share of behavior-changing adaptations is above 1% of all adapted inputs, the accuracy and RI numbers must be revised downward; if it is zero on a meaningful sample, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In CodeImprove, an 'out-of-scope' input is transformed by AES into a variant that the model classifies correctly, and the accuracy gain is computed against the original ground-truth label. This inference is sound only if every transformation in Table II is semantics-preserving for every adapted input. Table II operator 15, 'changeDeleteComments,' explicitly includes deleting printf statements, not merely comments. For the CodeChef defect-prediction task, labels include 'wrong output,' 'timeout error,' and 'runtime error,' which are defined by program behavior; deleting a printf can change stdout, execution time, or side effects, so the transformed program can have a different ground-truth class than the original. The genetic search is free to select such an operator. RQ5 provides only one worked example (Figure 3) and refers to the website for more; it does not systematically verify semantics, and Section VII lists 'lack of ground truths' as an internal threat. Since a non-semantics-preserving transformation can flip a model's prediction for reasons unrelated to the original program's true label, the headline improvements (up to 8.78% accuracy, 51.28% RI) may be partly spurious.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CodeImprove, a two-phase inference-time method for improving deep code models without retraining. The first phase, Dropout-based Sub-Model Generation (DSMG), builds layerwise sub-models from the original model and computes a validity score (Equations 1-3) to flag out-of-scope inputs. The second phase, Adaptation by Evolutionary Search (AES), uses a genetic algorithm over a list of 15 program transformations guided by the DSMG validity score to convert out-of-scope inputs into in-scope ones. The evaluation covers CodeBERT, RoBERTa, and GraphCodeBERT on two tasks, Devign vulnerability detection and CodeChef defect prediction, and reports accuracy improvements up to 8.78% (relative improvement 51.28%) and input-validation AUC up to 0.924. The paper claims all transformations are semantic-preserving.","tokens_in":20398,"tokens_out":6020,"duration_ms":59369,"significance":"If the semantic-preservation claim were rigorously established, this would be a useful and relatively novel contribution: an inference-time alternative to retraining for code models, together with a layerwise validity score that clearly outperforms standard uncertainty metrics (AUC 0.781-0.924 vs. at most 0.624 in Table I). The public artifact and reproducible setup are positive aspects. However, the central adaptation result depends on the transformed programs retaining the original ground-truth labels, and the current evidence for that is a single worked example. The significance is therefore conditional on a systematic semantic-preservation verification.","major_comments":[{"comment":"The semantic-preservation assumption is load-bearing and is not established. Operator 15, 'changeDeleteComments,' is described as 'Deleting statements that print debugging hints and comment.(e.g., printf())'; deleting a printf statement can change stdout, execution time, or side effects. For the CodeChef defect-prediction task, the ground-truth labels include 'wrong output,' 'timeout error,' and 'runtime error,' all of which are behavior-dependent, so a transformed program can legitimately have a different label than the original. RQ5 verifies only one example (Figure 3) and does not execute either program; Section VII itself lists 'lack of ground truths' as an internal threat. This is not a presentation issue: if transformations change program behavior, the reported accuracy gains are not attributable to adapting the model's input space but to changing the labels. The authors should either remove or restrict operator 15, run differential testing on the actual test inputs for all transformed programs, and report per-operator correction statistics.","section":"Section IV-B, Table II, and Section VI-E (RQ5)"},{"comment":"The threshold values c=0.3 for vulnerability detection and c=0.2 for defect prediction are fixed per task with no reported calibration procedure. These thresholds determine which inputs are adapted and when AES terminates, and they affect the reported CVR, MVR, CSR, and accuracy numbers. In addition, all reported results are single point estimates without error bars, confidence intervals, or repeated runs. Given that the genetic search is stochastic, the headline improvement of 8.78% should be accompanied by a sensitivity analysis over thresholds and seeds, or at least multiple independent runs with variance reported.","section":"Section V, threshold settings and result reporting"},{"comment":"The validity score in Equation 1/2 is defined relative to whether the original model's predicted label l_x is correct or incorrect, and Section VII states that 'mispredictions and correct predictions' are used to simulate out-of-scope and in-scope inputs. The paper does not state which data split is used to train the sub-model dense layers or to determine these correctness labels, nor whether the thresholds are chosen on the test set. If the test-set correctness labels or the test-set predictions participate in training or threshold selection, the reported AUC (0.781-0.924), CVR, and MVR are optimistically biased. Please clarify the data flow and use a held-out validation set for all calibration steps.","section":"Section IV-A, Equations 1-3, and Section VII"}],"minor_comments":[{"comment":"For the CodeBERT vulnerability-detection configuration, Table III reports CSR/MCR as 39.9/2.6 while Table VI reports 39.9/4.5 for the same CodeImprove setting; these numbers should be reconciled.","section":"Table III vs. Table VI"},{"comment":"The entry for CodeImprove-rand with RoBERTa on vulnerability detection reads '6.11.1/1.5' and appears to be a formatting typo for '6.1/1.5'.","section":"Table VI"},{"comment":"There are typos and grammar issues that should be corrected: 'genetics algorithms' in the abstract, 'covert' in Section IV-B, 'upto' in the abstract, a duplicate 'the' in Section IV-A, and 'hiss guidance' in the acknowledgments.","section":"Throughout"},{"comment":"The notation for B(submodel_k) is confusing because the two cases in Equation 2 are not mutually exclusive in the way the surrounding text presents them; please restate the definition of B more clearly and define ls and lh before using them.","section":"Equations 1-2"},{"comment":"The RQ5 example shows several transformations (declaration splitting, loop rewriting, relational/incremental rewriting) but it does not show execution output or behavior of the original vs. transformed code; a systematic differential test would be much stronger.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is worth taking seriously: instead of retraining a code model, validate each input with layerwise sub-models and then use genetic search over program transformations to move out-of-scope inputs back in scope. That combination is new as far as I know, and the paper does several things well. The preliminary study showing standard uncertainty metrics sit near 0.5 AUC on code inputs is a useful negative result. Comparing DSMG against Dissector, against raw hidden states, and against random search and hill climbing gives a real sense of where the gains come from. The artifact is public, and the overhead numbers are honest.\n\nThe soft spots are concentrated where the stress-test note lands. The load-bearing assumption is that all 15 transformations preserve semantics, so the original ground-truth labels still apply after adaptation. Table II operator 15, \"changeDeleteComments,\" explicitly includes deleting printf statements, not just comments. For the CodeChef task, labels include wrong output, timeout error, and runtime error—all defined by observable behavior. Deleting a printf can change stdout, timing, or side effects, so the transformed program can have a different true label than the original. RQ5 verifies exactly one example. That is not enough to support accuracy gains of up to 8.78% and relative improvements of 51.28%; those gains may be partly spurious, since the search is free to select a semantics-changing operator and the model's new correct prediction may just reflect the new (different) program.\n\nOther issues are real but secondary. The validity score is trained on the original model's own hidden states and labels and used as the fitness function, which makes the search partially self-referential—though the final accuracy is checked against ground truth, so this is a bias, not a circle. Thresholds are hand-set per task with no calibration procedure. Results are point estimates with no error bars. The MCR for CodeBERT on vulnerability detection is 2.6 in Table III but 4.5 in Tables VI and VII for the same setting; that inconsistency needs explanation. CodeDenoise is discussed but not included as a direct baseline.\n\nWho gets value from this? Researchers working on inference-time adaptation, input validation, and robustness of code models. The paper deserves a serious referee, but it needs major revision before the claims can be accepted: differential testing (or at least a random sample of transformed programs) to verify semantics, removal or restriction of non-preserving operators, error bars, and threshold calibration. The idea is not invalidated, but the evidence as presented is not yet trustworthy.","headline":"Plausible inference-time adaptation idea with a solid evaluation skeleton, but the headline gains rest on a semantic-preservation assumption that one of the paper's own transformations violates, so the numbers are not yet interpretable.","tokens_in":20894,"tokens_out":1988,"would_cite":false,"duration_ms":21386,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CodeImprove claims that out-of-scope program inputs can be flagged by a layerwise sub-model validity score and rewritten by semantic-preserving transformations so that deep code models classify them correctly, with no retraining.","keywords":["input validation","input adaptation","program transformation","deep code models","uncertainty estimation","genetic algorithm","semantic-preserving transformation","out-of-scope detection"],"falsifier":"Run a differential test on the full set of before-and-after program pairs that CodeImprove reports as corrected: compile both versions with the same compiler, execute them on the same inputs, and compare observable outputs and exit codes. The paper's own semantic-preservation check (RQ5) inspects only a single transformed example; if any pair whose prediction was \"corrected\" differs in observable behavior — for instance, a program whose printf debugging output was deleted under operator 15 — the label no longer applies to the transformed code, and the accuracy gain is an artifact rather than a genuine model improvement.","tokens_in":19956,"feed_emoji":"🧬","tokens_out":7072,"duration_ms":56060,"temperature":0.7,"pith_summary":"The paper tries to establish that a deployed deep code model can be made more accurate without retraining by adapting its inputs. It proposes a two-stage pipeline: first, a validity score built from layerwise sub-models flags programs the model is likely to misclassify; second, a genetic algorithm applies semantic-preserving program transformations to rewrite those programs into ones the model handles correctly. On vulnerability detection and defect prediction across three pre-trained code models, the pipeline reports accuracy gains up to 8.78% absolute and 51.28% relative, with input validation reaching an AUC of 0.924. A sympathetic reader would care because frequent retraining of code models is expensive in labeling and deployment, and input adaptation offers a cheaper alternative that leaves the original model untouched.","feed_headline":"Rewriting code inputs lifts model accuracy 8.78%","feed_subtitle":"A validity score flags inputs a code model mishandles; genetic search rewrites them, no retraining.","key_machinery":"The load-bearing object is the validity score from DSMG. Each sub-model $k$ keeps the original model's weights up to layer $k$ and adds a newly trained dense layer; for an input predicted as label $l_x$, the sub-model score is either $\\mathrm{submodel}_k[l_x]$ plus the best-versus-second-best margin (if the prediction is correct) or $\\mathrm{submodel}_k[l_x]$ minus the difference between the highest and the chosen probability (if the prediction is wrong). The final validity score is the weighted average of sub-model scores using Dissector-style weight growth. This score does two jobs: it is the oracle that decides which inputs need adaptation, and it is the fitness function that drives AES's genetic search, so the whole pipeline depends on it being better at identifying misclassifications than conventional confidence and uncertainty metrics.","core_discovery":"The central claim is that out-of-scope program inputs — code snippets a trained model is prone to mishandle — can be reliably detected and then converted into in-scope inputs through syntax-level program transformation, all without modifying the model. Detection rests on a new validity score computed from Dropout-based Sub-Model Generation (DSMG): the original model is sliced at intermediate layers, each slice is capped with a newly trained dense classifier, and the agreement pattern of these sub-models' predictions (via a best-versus-second-best formula aggregated by weight growth) becomes the validity score. Adaptation rests on Adaptation by Evolutionary Search (AES), a genetic algorithm over fifteen listed transformations (renaming, loop rewrites, relational operator flips, and similar) guided by that validity score as fitness. The paper reports that this two-stage scheme improves three pre-trained models on two software-engineering tasks, correcting 23.1%–39.9% of mispredictions, and that the validity score distinguishes in-scope from out-of-scope inputs with AUC scores of 0.781–0.924, far above the best conventional uncertainty metric it tested (0.624).","pith_inferences":["The same machinery that repairs mispredictions could also be used adversarially: a search that finds validity-lowering transformations might generate semantic-preserving inputs that flip a model's prediction, so the paper's method doubles as a robustness probe.","The 0.924 AUC suggests sub-model disagreement encodes a general signal of model competence on code; a testable extension is to check whether the same score predicts performance on out-of-distribution projects or on code written by different developers.","The reported gains rest on an unproved semantics-preservation assumption; rerunning AES with differential testing as a constraint might reduce the correction rate but would make the numbers trustworthy.","The genetic search uses only three generations and a fixed operator list; larger budgets or learned transformation selection could push accuracy further if semantic equivalence is guaranteed."],"forward_implications":["Deployed code models can be improved on new data without retraining, so agile projects can keep models effective as the codebase shifts.","The validity score gives a principled way to pick which inputs to send to a more expensive fallback, such as human review or a larger model.","The same pipeline transfers across models and tasks: it worked on CodeBERT, RoBERTa, and GraphCodeBERT for both vulnerability detection and defect prediction, so it is not tied to one architecture.","Because corrections come from input changes, the approach composes with any model and any future retraining schedule."],"supporting_citations":[{"why":"Supplies the crossing-layer dissection method and the weight-growth aggregation formula that CodeImprove adapts for its validity score; also the main validation baseline.","marker":"[19]"},{"why":"Provides the dropout-as-Bayesian-approximation idea that DSMG extends and is one of the uncertainty baselines in the preliminary study.","marker":"[21]"},{"why":"The image-domain input-repair approach (InputReflector) that CodeImprove adapts as the primary adaptation baseline it must beat.","marker":"[17]"},{"why":"Devign dataset of C functions labeled vulnerable or clean; one of the two evaluation benchmarks.","marker":"[48]"},{"why":"CodeChef dataset of C/C++ functions for defect prediction; the other evaluation benchmark.","marker":"[49]"},{"why":"GraphCodeBERT, one of the three pre-trained code models on which CodeImprove is evaluated.","marker":"[13]"},{"why":"CodeBERT, another of the three pre-trained models evaluated.","marker":"[50]"},{"why":"RoBERTa, the third pre-trained model evaluated.","marker":"[51]"},{"why":"CodeDenoise, the closest prior attempt to improve deployed code models without retraining; used for numerical comparison.","marker":"[52]"}],"fun_headline_variants":["Fixing code inputs, not models, boosts accuracy 8.78%","Genetic rewrites fix out-of-scope code for deep models","Validity score flags bad code, then rewrites it: +8.78%","Adapting code inputs beats retraining: 8.78% gain","CodeImprove: transform code to fit model, no retraining needed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that all fifteen transformations preserve program semantics for every test input, so ground-truth labels remain valid after rewriting; the paper asserts this without formal proof or differential testing, and one listed operator — deleting comments and debugging print statements — is not generally semantics-preserving.","fun_headline_variants_meta":{"raw":{"variants":["Fixing code inputs, not models, boosts accuracy 8.78%","Genetic rewrites fix out-of-scope code for deep models","Validity score flags bad code, then rewrites it: +8.78%","Adapting code inputs beats retraining: 8.78% gain","CodeImprove: transform code to fit model, no retraining needed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000299,"raw_usage":{"total_tokens":1794,"prompt_tokens":1079,"completion_tokens":715,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":695,"completion_tokens_details":{"reasoning_tokens":627}},"tokens_in":695,"tokens_out":715,"duration_ms":6208,"temperature":1.0,"reasoning_tokens":627,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:56:08.427781+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a differential test on the full set of before-and-after program pairs that CodeImprove reports as corrected: compile both versions with the same compiler, execute them on the same inputs, and compare observable outputs and exit codes. The paper's own semantic-preservation check (RQ5) inspects only a single transformed example; if any pair whose prediction was \"corrected\" differs in observable behavior — for instance, a program whose printf debugging output was deleted under operator 15 — the label no longer applies to the transformed code, and the accuracy gain is an artifact rather than a genuine model improvement.","supporting_citations":[{"cited_title":"Devign: Effective vul- nerability identification by learning comprehensive program semantics via graph neural networks,","cited_arxiv_id":null,"evidence_quote":"Devign dataset of C functions labeled vulnerable or clean; one of the two evaluation benchmarks."},{"cited_title":"Code difference guided adversarial example generation for deep code models,","cited_arxiv_id":null,"evidence_quote":"CodeDenoise, the closest prior attempt to improve deployed code models without retraining; used for numerical comparison."}],"review_version":1}