{"id":"7f197b3d-29da-4ab1-bd70-71d21ae8edb0","arxiv_id":"2502.05456","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A proposal to reduce deep learning mispredictions at inference by validating, transforming, and searching over inputs, with preliminary evidence only for code classification.","lead":"This paper sketches a three-phase framework that tries to fix deep learning mispredictions at inference time by checking inputs, transforming them, and searching for better versions, instead of retraining the model. It reports preliminary gains on code classification tasks, but text and image parts are planned future work.","discovery_kind":"unclear","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on the untested assumption that the 15 code transformations preserve semantics; if any transformation changes program behavior, the reported 'corrections' are invalid because the ground-truth label no longer applies.","rationale":"The reader identified DSMG validity as the weakest assumption. I see that as important but secondary. The more fundamental, load-bearing assumption is semantic preservation of the code transformations. If the transformations are not semantic-preserving, the entire evaluation is invalid because the ground-truth labels of transformed inputs are no longer the original labels. This would make the reported correction rates and accuracy improvements spurious, regardless of how good DSMG is. Conversely, if semantic preservation is rigorously established, the DSMG concern becomes a question of search efficiency rather than correctness of the framework's premise. The manuscript provides no evidence for semantic preservation—no operator definitions, no equivalence proofs, no differential testing—only an assertion. The current verdict remains UNVERDICTED because the evidence is missing; the concern does not by itself prove the claims false, but it defines the condition that would need to be satisfied for the central claim to hold. Therefore I do not change the reader's verdict, but I disagree with the reader's identification of the weakest assumption and propose a concrete test that would settle the concern.","tokens_in":7213,"tokens_out":4621,"duration_ms":50346,"concrete_test":"Obtain the 15 transformation operators and the set of transformed inputs used in the evaluation (from the companion paper or the authors). For every transformation applied, compile/run the original and transformed code on a comprehensive test suite (or use a semantic-equivalence checker) and verify identical observable behavior. If any transformation changes behavior, re-run the evaluation after excluding that transformation. If the correction rate (23.1%–39.9%) or the accuracy improvement (up to 8.78%) drops materially, the central claim is an artifact of semantic non-preservation rather than true input refinement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section IV-A is that the framework genuinely corrects 23.1%–39.9% of mispredicted inputs. This is only meaningful if the transformed input is semantically equivalent to the original, so that the original ground-truth label remains valid. Section IV states the framework includes 15 semantic-preserving code transformation operators, and Section II/P2.1 describes restructuring, reordering, and simplifying code. However, no operator list, no equivalence proof, and no differential-testing evidence is provided. Many code rewrites—reordering statements, refactoring loops, simplifying redundant sections—can change behavior due to side effects, short-circuit evaluation, exception ordering, or data dependencies. If a transformation alters the program's semantics, then the label of the transformed input is no longer the original label, and a prediction flip from wrong to 'correct' is an artifact of label invalidation, not a genuine refinement. This assumption is more load-bearing than the DSMG validity concern: even a perfect validator and search cannot produce real corrections if the transformations themselves change the task's ground truth. The paper's assertion that 'the transformations consistently preserve original code functionality' (Section IV-D) is a conclusion, not supporting evidence. Until semantic preservation is demonstrated, the reported accuracy, precision, recall, and F1 improvements (up to 8.78%, 8.48%, 16.9%, and 13.5%) are unsubstantiated and could be inflated.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an adaptive, on-the-fly input refinement framework for deep learning models, consisting of three phases: input validation (P1), input transformation (P2), and an optimal search strategy (P3). The claim is that this framework reduces mispredictions at inference time without retraining, across code, NLP, and image domains. The current reported implementation covers code classification only, using CodeBERT, RoBERTa, and GraphCodeBERT on vulnerability detection and defect prediction. The paper reports improvements up to 8.78% in accuracy, 8.48% in precision, 16.9% in recall, and 13.5% in F1-score, as well as correction of 23.1%–39.9% of mispredicted inputs, but provides no experimental setup, dataset splits, or statistical detail. The text and image phases are explicitly future work in the timeline in Section V.","tokens_in":7515,"tokens_out":4938,"duration_ms":49301,"significance":"If the framework's claims are validated, it would offer a resource-efficient alternative to retraining for improving deep learning models at inference time, with potential value in software engineering, NLP, and computer vision. The decomposition into validation, transformation, and search is a sensible structure, and the authors have positioned the work against a concrete baseline (Input-Reflector) and uncertainty estimation baselines. The paper's significance is currently limited because the central quantitative results are not substantiated in this manuscript and are deferred to a companion paper [15]. The framework's key assumption—that the code transformations are semantic-preserving—is asserted but not demonstrated, and this assumption is load-bearing for the validity of the reported corrections. The paper also overclaims multi-domain applicability when only code classification is evaluated.","major_comments":[{"comment":"The central quantitative claim, that the framework achieved the best model improvement ranging up to 8.78% in accuracy, 8.48% in precision, 16.9% in recall, and 13.5% in F1-score on all subjects, is presented without any experimental detail. No dataset splits, subject descriptions, model checkpoints, evaluation protocol, standard deviations, or statistical tests are provided. The companion paper reference [15] does not make these results verifiable or reproducible from this manuscript. Since this is the main contribution, the claim is currently unsupported.","section":"IV-A"},{"comment":"The assertion that the framework includes 15 'semantic-preserving' code transformation operators is not substantiated. The paper does not list the operators, provide equivalence proofs, or show differential testing results. Many code rewrites—such as reordering statements, refactoring loops, or simplifying redundant sections—can alter behavior due to side effects, short-circuit evaluation, exception ordering, or data dependencies. If any transformation changes program semantics, the original ground-truth label no longer applies, and the reported 'corrections' of mispredicted inputs would be artifacts rather than genuine improvements. The statement in Section IV-D that 'the transformations consistently preserve original code functionality' is a conclusion, not supporting evidence, and this is load-bearing for the validity of the reported accuracy, precision, recall, and F1 improvements.","section":"IV-D / II (P2.1)"},{"comment":"The abstract claims that the framework 'reduces mispredictions across various domains' and the introduction hypothesizes improvements across code, NLP, and image domains. However, the reported experiments cover only code classification (vulnerability detection and defect prediction). Section V explicitly lists P1.2, P2.2, P2.3, and P1.3 as future work, including text and image transformations. The multi-domain claim is therefore not supported by any evidence presented in this manuscript, and the paper overstates its current contributions.","section":"Abstract / Section V"},{"comment":"The DSMG validity score is the central signal guiding both the validation phase (P1.1) and the AES search (P3). The paper reports DSMG AUC scores of 0.781–0.924 and compares them to uncertainty baselines, but it does not define what the AUC is computed over (e.g., whether the positive class is 'will mispredict' or 'out-of-scope'), nor does it specify the data split, thresholds, or the exact relationship between DSMG scores and true correctness on unseen inputs. The correction rates of 23.1%–39.9% are inherited from the companion paper [15] and are stated without enough methodological detail to rule out the possibility that the search optimizes an internal validity metric that does not track true mispredictions. The manuscript needs to establish, with concrete experimental evidence, that the DSMG-guided search yields genuine corrections rather than artifacts of the internal scoring function.","section":"IV-B and IV-C"}],"minor_comments":[{"comment":"The text first states 'I adapted Input-Reflector to code domain' and later says 'Input-Reflector cannot be applied to code data.' These statements are contradictory and should be reconciled: if it cannot be applied, how was the negative impact measured?","section":"IV-A"},{"comment":"The timeline in Section V contains completion dates that are already in the past relative to the manuscript version (e.g., 'by January 2025' and 'by March 2025'). For a journal submission, either update the timeline or remove this section, which reads as a research-progress plan rather than a completed technical paper.","section":"V"},{"comment":"The P3 description in Section II is repeated almost verbatim in the final paragraph of the same subsection; one of the two passages should be removed or merged to avoid redundancy.","section":"II (P3)"},{"comment":"The uncertainty baselines are listed but not described; in particular, the distinction between 'predictive entropy' and 'entropy' is unclear. Please provide definitions or references for each metric used.","section":"IV-B"},{"comment":"The paper repeatedly refers to companion paper [15] for essential details (transformation operators, experimental setup, results). Since the present manuscript is meant to stand alone, an appendix with the full operator list and a minimal experimental configuration would improve completeness.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"This manuscript reads as a research-progress report or thesis chapter rather than a complete journal paper. The central results are reported at a high level and are deferred to the author's own companion paper [15], which makes it difficult to evaluate novelty or correctness from this submission alone. The semantic-preservation assumption is the most critical risk; without evidence that the transformations do not change program behavior, the headline correction rates are not trustworthy. The paper would need to either incorporate the full experimental details and semantic-equivalence evidence or be reframed as a position/vision paper rather than a claim of validated results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this preprint is a research proposal, not a self-contained result. The only concrete numbers come from the author's own companion paper (CodeImprove, arXiv:2501.15804), and this text simply restructures those results into a three-phase framework with text and image phases still on the roadmap. The abstract claims multi-domain improvements, but the body explicitly says those domains are future work. That mismatch is the first thing to flag.\n\nWhat is genuinely reasonable here: the framework is a clean decomposition of input validation, transformation, and search. The DSMG and AES components are sensible extensions of what Input-Reflector and Dissector do, and the author cites those prior lines honestly. The paper is also unusually candid about its own status — Section IV says 'preliminary framework,' Section V is a timeline. That transparency deserves credit.\n\nThe soft spots are serious. No experimental setup is given: no dataset splits, no model details, no statistical tests, no artifacts. The reported improvements (up to 8.78% accuracy, 23.1–39.9% correction rate) are simply asserted, with no way to check them from this text. More importantly, the central claim rests on the assumption that the 15 code transformations preserve semantics. Section IV-D asserts they do, but no operator list, no equivalence proof, and no differential testing is provided. If a transformation changes program behavior, the original ground-truth label is invalid, and the correction numbers become meaningless. That is not a minor detail; it is load-bearing. The companion paper may well contain the missing evidence, but this preprint cannot be evaluated on its own.\n\nAlso note the circularity concern: AES search is guided by DSMG validity scores, and DSMG is the same validation metric being proposed. The final accuracy is measured externally, so it's not definitional circularity, but it does mean the search is optimizing a proxy that may not track true correctness. The reader's concern there is legitimate, though secondary to the semantic-preservation issue.\n\nBottom line: this is a proposal plus a pointer to a companion paper. If it crossed my desk as a full submission, I'd desk-reject it because the evidence is not in the text. The serious referee time should go to CodeImprove itself. For your own work, cite [15], not this arXiv posting. I would not bring it to the reading group — there's too little to chew on.","headline":"A progress report that inherits its only concrete results from the companion CodeImprove paper; the framework is plausible but this text contains no evidence to referee.","tokens_in":8007,"tokens_out":2780,"would_cite":false,"duration_ms":26732,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-phase input refinement framework improves code-classifier accuracy by up to 8.78% at inference time, without retraining the model.","keywords":["input refinement","input validation","uncertainty estimation","semantic-preserving transformation","evolutionary search","deep code models","vulnerability detection","defect prediction"],"falsifier":"A reader could take a held-out labelled set of code inputs, rank them by DSMG variance, and ask whether the model's true error rate rises monotonically with that rank; if the rank ordering is no better than chance, the validation signal is not carrying the reported correction. A stronger check: refine inputs selected by DSMG and the same number of randomly selected inputs; if random selection yields the same accuracy gain, the transformations alone, not the validity-guided search, explain the improvement.","tokens_in":7017,"feed_emoji":"🔧","tokens_out":5539,"duration_ms":51186,"temperature":0.7,"pith_summary":"The paper is trying to establish that mispredictions in deep learning models can be reduced at inference time by refining the input rather than retraining the model. It proposes a three-phase framework: validation flags inputs likely to be mispredicted, transformation applies meaning-preserving edits, and search selects the best edited candidate. On code classification tasks, namely vulnerability detection and defect prediction, with three encoder models, the framework reports accuracy gains up to 8.78 percentage points and corrects roughly 23 to 40 percent of previously mispredicted inputs. The sympathetic reading is that this gives a scalable, resource-efficient alternative to retraining for high-stakes software engineering applications.","feed_headline":"Fixing bad inputs at inference lifts code-model accuracy up to 8.78%","feed_subtitle":"A validation-plus-search layer corrects 23–40% of mispredicted code inputs without retraining.","key_machinery":"The two load-bearing pieces are DSMG and AES. DSMG applies dropout to create a set of slightly different submodels and measures how consistently an input's representation propagates across layers; high variance flags inputs likely to be mispredicted. AES then iteratively applies a set of 15 semantic-preserving code transformation operators, sampling candidate refinements guided by DSMG validity scores over multiple generations rather than stopping at the first improvement. The claim is that this combination makes the validation signal actionable: the same metric that detects bad inputs also steers the search toward inputs the model can classify correctly.","core_discovery":"The central claim is that layerwise variance across dropout-generated submodels (DSMG) identifies out-of-scope inputs reliably enough to guide an evolutionary search (AES) through semantic-preserving code transformations, converting mispredicted inputs into inputs the model handles correctly. The paper reports that this pipeline outperforms the Input-Reflector baseline, which cannot be applied to code and degrades CodeBERT performance, and outperforms random search and hill climbing, which stop at local minima. The best reported gains are up to 8.78% accuracy, 8.48% precision, 16.9% recall, and 13.5% F1-score, with 23.1% to 39.9% of mispredicted inputs corrected and at most 2.6% of previously correct inputs flipped to wrong.","pith_inferences":["This reader's inference: the same validation-plus-search loop could be tested on decoder and encoder-decoder models, which the paper lists as future phases P1.2 and P1.3 but does not evaluate.","This reader's inference: the reported correction rate could be compared against a control that applies the same number of random transformations to all inputs; if random transforms match the gain, DSMG's guidance, not the transformations, would be doing the work.","This reader's inference: if DSMG's variance signal tracks misprediction risk on code, similar layerwise consistency signals might identify adversarial or out-of-distribution inputs in NLP and vision, where the paper's P2.2 and P2.3 are proposed but untested."],"forward_implications":["If the reported gains hold, code models can be improved at deployment time without retraining, data labeling, or model updates.","The framework's at-most-2.6% flip rate bounds the cost of refinement: few correct predictions are damaged while a substantial share of mispredictions are fixed.","Evolutionary search matters: AES's multi-generation search gives up to 8.78% improvement while random search and hill climbing saturate near 2.13%.","DSMG's AUC of 0.781 to 0.924 suggests the validation signal transfers across model architectures such as CodeBERT, RoBERTa, and GraphCodeBERT and across two tasks."],"supporting_citations":[{"why":"Baseline technique adapted to code that this framework must beat; the paper reports it degrades CodeBERT performance.","marker":"[10]"},{"why":"Validation baseline; DSMG's AUC, correction validation rate, and mis-correction validation rate are compared against it.","marker":"[13]"},{"why":"Monte-Carlo dropout, one of the uncertainty baselines DSMG is compared with.","marker":"[18]"},{"why":"Deep ensembles, one of the uncertainty baselines DSMG is compared with.","marker":"[30]"},{"why":"One of the three encoder models on which the framework is evaluated.","marker":"[8]"},{"why":"One of the three encoder models on which the framework is evaluated.","marker":"[26]"},{"why":"One of the three encoder models on which the framework is evaluated.","marker":"[27]"},{"why":"Vulnerability detection dataset used to evaluate the framework.","marker":"[28]"},{"why":"Defect prediction dataset used to evaluate the framework.","marker":"[29]"}],"fun_headline_variants":["On-the-fly input fixes boost code accuracy up to 8.78%","Inference-time input refinement lifts code accuracy by 8.78%","No retraining needed: input tweaks fix code mispredictions","Adaptive input transformation improves code models without retraining","Refine inputs at inference to cut code errors by up to 8.78%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's success depends on the layerwise variance signal from dropout-generated submodels actually tracking whether a code input will be mispredicted; if that signal is unreliable on unseen code, the evolutionary search will refine inputs to please the metric rather than to fix the model's mistakes.","fun_headline_variants_meta":{"raw":{"variants":["On-the-fly input fixes boost code accuracy up to 8.78%","Inference-time input refinement lifts code accuracy by 8.78%","No retraining needed: input tweaks fix code mispredictions","Adaptive input transformation improves code models without retraining","Refine inputs at inference to cut code errors by up to 8.78%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00096,"raw_usage":{"total_tokens":4068,"prompt_tokens":905,"completion_tokens":3163,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":3067}},"tokens_in":521,"tokens_out":3163,"duration_ms":20034,"temperature":1.0,"reasoning_tokens":3067,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T19:14:16.770054+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could take a held-out labelled set of code inputs, rank them by DSMG variance, and ask whether the model's true error rate rises monotonically with that rank; if the rank ordering is no better than chance, the validation signal is not carrying the reported correction. A stronger check: refine inputs selected by DSMG and the same number of randomly selected inputs; if random selection yields the same accuracy gain, the transformations alone, not the validity-guided search, explain the improvement.","supporting_citations":[{"cited_title":"Repairing failure-inducing inputs with input reflection,","cited_arxiv_id":null,"evidence_quote":"Baseline technique adapted to code that this framework must beat; the paper reports it degrades CodeBERT performance."},{"cited_title":"Dissector: Input val- idation for deep learning applications by crossing-layer dissection,","cited_arxiv_id":null,"evidence_quote":"Validation baseline; DSMG's AUC, correction validation rate, and mis-correction validation rate are compared against it."},{"cited_title":"Dropout as a bayesian approximation: Representing model uncertainty in deep learning,","cited_arxiv_id":null,"evidence_quote":"Monte-Carlo dropout, one of the uncertainty baselines DSMG is compared with."},{"cited_title":"Graphcodebert: Pre- training code representations with data flow,","cited_arxiv_id":null,"evidence_quote":"One of the three encoder models on which the framework is evaluated."},{"cited_title":"CodeBERT: A Pre-Trained Model for Programming and Natural Languages,","cited_arxiv_id":null,"evidence_quote":"One of the three encoder models on which the framework is evaluated."},{"cited_title":"RoBERTa: A Robustly Optimized BERT Pretraining Approach,","cited_arxiv_id":null,"evidence_quote":"One of the three encoder models on which the framework is evaluated."},{"cited_title":"Convolutional neural networks on assembly code for predicting software defects,","cited_arxiv_id":null,"evidence_quote":"Defect prediction dataset used to evaluate the framework."}],"review_version":1}