{"id":"95d2a197-d3b2-4cc0-91ac-c7cff0e11e71","arxiv_id":"2501.00298","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Prom detects deployment-time data drift in ML models for code analysis and optimization by combining weighted conformal prediction with ensemble voting, and shows that retraining on fewer than 5% of flagged samples restores near-design-time performance.","lead":"Prom, a new open-source library, watches a deployed machine learning model for code analysis or optimization and flags the inputs where the model is likely to be wrong, catching 96% of mispredictions across 13 models and 5 tasks. If the results hold up, teams can restore lost accuracy at deployment time by relabeling a few flagged samples and retraining, instead of rebuilding the whole pipeline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Incremental-learning result lacks a random-relabeling control and a specified retraining protocol, so the 5%-relabeling recovery claim in the abstract and Table 2 is not yet established.","rationale":"The reader's CONDITIONAL verdict is appropriate. The strongest part of the paper is the misprediction-detection evaluation: real ground truth, 13 independently published models, 5 tasks, and a released artifact (Zenodo 10.5281/zenodo.14077780). The adaptive weighted p-values in Eq. (1)-(2) do lose the split-conformal exchangeability guarantee, and the Euclidean-distance premise is a real assumption, but those concerns are partially mitigated by the empirical coverage-deviation check (Fig. 13(d)) and by the breadth of the detection results. The recovery claim, by contrast, has no control arm and no protocol; it is the least secure load-bearing element of the headline. A random-relabeling arm would settle whether Prom's selection, rather than simply acquiring any deployment labels, drives the improvement. I therefore keep the verdict CONDITIONAL rather than moving to ACCEPT or REJECT: the flaw is a missing experiment and specification, not a demonstrated false result. My focus differs from the reader's weakest-assumption statement, so agreement is partial.","tokens_in":24359,"tokens_out":8027,"duration_ms":83807,"concrete_test":"Re-run the Sec. 7.3 incremental-learning experiments with a control arm that relabels 5% of test samples selected uniformly at random instead of by Prom, using the same retraining schedule and evaluation metric. If the random-arm performance-to-oracle ratio is within 5% of Prom's 0.807 average (or matches the 0.544-to-0.807 jump), the recovery claim is not attributable to Prom. Report the protocol (epochs, learning rate, reuse of original training set, exclusion of relabeled samples from evaluation) and per-case test set sizes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline has two parts; the misprediction-detection part is well supported, but the deployment-recovery part is not. Section 7.3 and Table 2 report that relabeling up to 5% of Prom-flagged samples raises the average deployment performance-to-oracle ratio from 0.544 to 0.807, yet no random-relabeling baseline is given. Without that control, the improvement could come from adding any labeled samples from the deployment distribution, not from Prom's selection. The retraining protocol is also underspecified: no epochs, learning rate, loss function, whether the original training set is reused, or whether relabeled samples are excluded from evaluation. Section 8 asserts that 'cross-validation shows that this retraining approach effectively improves generalization,' but no such cross-validation result appears anywhere in the paper. Case study 1 reports improvement after relabeling 'just one drifting sample,' suggesting very small test sets; per-case sample sizes are not reported, so the 5% claim is not interpretable. Because the abstract promises deployment-time recovery, this gap is load-bearing: the central practical benefit remains unverified even if detection recall is real.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Prom, a model-agnostic framework that applies conformal prediction at deployment time to flag test inputs likely to be mispredicted by an already-trained classifier or regressor. Prom selects a local subset of calibration examples, weights their nonconformity scores by feature-space distance (Eq. 1), computes class-conditional p-values (Eq. 2), and combines four nonconformity functions by majority vote. The authors evaluate Prom on 13 ML models across five code analysis and optimization tasks, reporting an average misprediction recall of 96% (up to 100%) with a false-positive rate below 14%, and show that relabeling up to 5% of flagged samples with incremental learning raises the average deployment-to-oracle performance ratio from 0.544 to 0.807 (Table 2).","tokens_in":24439,"tokens_out":4529,"duration_ms":43991,"significance":"If the detection results are taken at face value, the paper makes a useful empirical contribution: it is the first application of conformal-prediction-style credibility scoring to code optimization workloads, and it does so in a model-agnostic way that supports both classification and regression. The evaluation is grounded in external ground truth (oracle searches, profiling time, CVE labels), the method is specified in enough detail to re-implement, and the authors provide an artifact and comparisons against MAPIE, PUNCC, RISE, and TESSERACT. The main weakness is that the second half of the headline claim, namely deployment-time recovery through incremental learning, is not yet supported by a controlled experiment. In addition, the deviation from exchangeability in Eq. (2) means the method's statistical validity is empirical rather than guaranteed, and the paper should present the supporting coverage data more transparently. The core premise that Euclidean distance in the user-supplied feature space tracks misprediction risk is an empirical assumption; the case-study breadth provides support but not a guarantee.","major_comments":[{"comment":"The deployment-recovery claim is not yet established because the incremental-learning experiment has no random-relabeling control and no specified retraining protocol. As presented, the improvement from 0.544 to 0.807 could be obtained by adding any labeled samples from the deployment distribution, and the absence of epochs, learning rate, loss function, reuse of the original training set, and exclusion of relabeled samples from evaluation makes the result unreproducible. I request a random-relabeling baseline (and ideally an oracle-selection upper bound), a precise retraining specification, and per-case sample sizes and relabeling counts.","section":"Sec. 7.3, Table 2"},{"comment":"The statement that \"cross-validation shows that this retraining approach effectively improves generalization\" is unsupported: no cross-validation result for the retrained models appears anywhere in the paper. This claim should be either removed or substantiated with the actual cross-validation numbers.","section":"Sec. 8"},{"comment":"The adaptive weighting and local subset selection used in Eq. (2) break the exchangeability assumption that underlies split conformal prediction, so the significance level 1−ε cannot be claimed as a validity guarantee for Prom's p-values. The reported geomean coverage deviation of 2.5% (Fig. 13(d)) is useful empirical evidence, but the text should state plainly that Eq. (2) is a heuristic and should report per-model and per-case coverage deviations rather than only the geomean, especially for the 4.4% thread-coarsening deviation caused by the small calibration set.","section":"Sec. 5.1.2, Eq. (2)"},{"comment":"Some of the headline detection percentages rest on very small test sets: case study 1 uses 17 OpenCL kernels, and case study 4's \"recall of 1\" is reported without the number of mispredictions in the deployment test set. The paper should report the number of test samples and the number of mispredictions per case study so that the 96% average and up-to-100% recall are interpretable with confidence intervals.","section":"Sec. 7.4"}],"minor_comments":[{"comment":"The explanation of p-values is reversed: the text says a high p-value suggests the likelihood of the observed data under the null hypothesis is small, whereas a high p-value indicates that the data are likely under the null and provide weak evidence against it.","section":"Sec. 3.3"},{"comment":"The acceptance rule is stated ambiguously: the text says a sample is flagged if both scores \"fall below the significance level,\" while earlier the significance level is defined as 1−ε; please clarify the direction of the comparison and the exact threshold used.","section":"Sec. 5.3"},{"comment":"There is a typo in the caption: \"Prom performence\" should be \"Prom performance.\"","section":"Table 2 caption"},{"comment":"The introductory sentence contains an unresolved citation placeholder \"[64? ? ? ]\"; this should be fixed before publication.","section":"Supplemental Material, Nonconformity functions"},{"comment":"The notation for the number of selected calibration samples is inconsistent: the text refers to selecting \"K samples\" while Eq. (2) sums over i in {1,...,n}; please align the notation.","section":"Sec. 5.1.2"}],"recommendation":"major_revision","confidential_remarks":"The artifact and the breadth of case studies are strong, and the detection results appear well supported by external ground truth. The main blocker is the incremental-learning part of the headline claim, which needs a proper control and a reproducible protocol rather than a rewritten narrative. Once those are added, I would be supportive of acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline: the detection half of this paper is solid and worth engaging; the deployment-recovery half is not yet supported. The abstract relies on the claim that relabeling fewer than 5% of Prom-flagged samples restores near-training performance (Table 2: ratio-to-oracle from 0.544 to 0.807). That claim is load-bearing, and it currently rests on an underspecified retraining protocol with no random-relabeling control. The stress-test note is right on this point.\n\nWhat is actually new: Prom applies conformal-prediction-based drift detection to code analysis and optimization, with a locally weighted calibration subset (Eq. 1), ensemble voting over four nonconformity functions, and a clustering-plus-kNN extension for regression. The delta over Transcend, TESSERACT, RISE, MAPIE, and PUNCC is moderate but real. The evaluation is genuinely large: 13 independently published models across 5 tasks, with ground truth from oracles, profiling, and CVE labels. They ship code, a Docker image, and a Zenodo artifact. The detection claim—96% average recall, under 14% FPR—is measured against real ground truth and is the strongest part of the paper.\n\nSoft spots, in proportion. The recovery result lacks a random-relabeling baseline, so the improvement could come from adding any labeled deployment samples rather than from Prom's selection. The retraining schedule is underspecified: no epochs, no learning rate, no statement about whether the original training set is reused, and no statement about whether relabeled samples are excluded from evaluation. Section 8 asserts that cross-validation shows improved generalization, but no such cross-validation result appears anywhere. Case study 1 reports improvement after relabeling 'just one drifting sample,' which suggests very small test sets; per-case sample sizes are not reported. The statistical core is heuristic: Eq. (2) uses class-conditional p-values over heavily pruned calibration sets with distance weights, which abandons the exchangeability condition that gives split conformal prediction its validity guarantee. The paper substitutes an empirical coverage-deviation check (geomean 2.5%), which is reasonable evidence but not a guarantee. Also, the load-bearing premise—that Euclidean distance in the user-supplied feature space tracks misprediction risk—is a representation-task interaction assumption, not a theorem. The paper supports it empirically across several embeddings, but does not discuss failure modes.\n\nThe citation pattern looks fine; self-citation is not a burden here. The detection method is specified precisely enough to re-implement. I would send this to serious peer review: the detection contribution deserves referee time, and the recovery claim needs to be either fixed or explicitly downgraded. My own verdict would be a conditional accept requiring a random-relabeling control, a precise retraining protocol, and either a validity argument or per-task calibration diagnostics with error bars on the headline recall and FPR numbers.","headline":"The detection half of this paper is solid and worth engaging; the deployment-recovery claim is not yet supported and needs a random-relabeling control and a specified retraining protocol.","tokens_in":25152,"tokens_out":2220,"would_cite":true,"duration_ms":20412,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prom identifies 96% of test inputs on which a deployed code-analysis or optimization model will mispredict, and updating the model on 5% of flagged inputs restores near-training-time performance.","keywords":["data drift","conformal prediction","misprediction detection","deployment-time model robustness","incremental learning","code optimization","code analysis","machine learning for code"],"falsifier":"Take a trained model and a drifted test set with known ground truth, run Prom at the default significance level, and compare the flagged set with the set of actual mispredictions: the paper's claim predicts at least 96% of true mispredictions flagged and fewer than 14% of correct predictions flagged. A concrete stress test is to construct a drifted workload whose control-flow structure changes but whose feature embedding is nearly unchanged (for example, permuting loop structure or renaming variables in an IR-based embedding); if such inputs are mispredicted yet not flagged, the Euclidean-distance premise fails.","tokens_in":23947,"feed_emoji":"🤖","tokens_out":7591,"duration_ms":87141,"temperature":0.7,"pith_summary":"Prom is a deployment-time wrapper for already-trained machine-learning models used in code analysis and optimization. The paper's central claim is that Prom can flag, at inference time, the test inputs on which the deployed model will mispredict, catching on average 96% (up to 100%) of mispredictions with a false-positive rate below 14%. When a small fraction of flagged samples is relabeled and used to retrain the model, the paper reports that deployed performance returns to roughly the design-time level, with the performance-to-oracle ratio across 13 models and 5 tasks rising from 0.544 to 0.807. The practical payoff is that model developers need not anticipate every future hardware or workload change during training; the same trained model can be monitored and repaired in the field.","feed_headline":"ML code tools can catch 96% of their own mistakes","feed_subtitle":"Relabeling 5% of flagged samples restores deployed AI models to training-time accuracy.","key_machinery":"The load-bearing mechanism is the adaptive conformal-prediction p-value. For each test input, Prom selects the closest 50% of calibration samples in the user-supplied feature space, exponentiates negative squared Euclidean distance to form a weight $w_i = \\exp(-\\|v_i-v_{n+1}\\|^2/\\tau)$ for each selected sample, multiplies each sample's nonconformity score by that weight, and computes the p-value as the fraction of same-label calibration samples whose weighted score meets or exceeds the test score. Four nonconformity functions vote on whether to reject each prediction, and a Gaussian-transformed prediction-set size serves as the confidence score. This machinery converts 'is this input like what the model saw in training?' into a thresholdable number without touching the underlying model, and the same p-value is extended to regression through k-NN pseudo-labels and K-means clustering of the calibration set.","core_discovery":"Prom operates by computing two statistical scores for each prediction during deployment: a credibility score, which measures how similar the test input is to calibration samples held out from training, and a confidence score, which measures how concentrated the model's probability output is on the predicted class. Both are derived from conformal prediction p-values, and Prom flags a prediction when both scores fall below a user-set significance level. Unlike earlier conformal drift detectors, Prom does not compare the test input against the full calibration set: it adaptively selects the nearest calibration samples in the model's feature space and weights their nonconformity scores by distance, then combines four nonconformity functions into an ensemble that votes on whether to reject the prediction. For regression, Prom approximates the unknown true label with a k-nearest-neighbour average and clusters the calibration set so that the same p-value machinery applies. The paper evaluates this on 13 independently developed models spanning thread coarsening, loop vectorization, heterogeneous device mapping, vulnerability detection, and DNN code generation, reporting that Prom identifies 96.2% of mispredictions on average and, after incremental learning on up to 5% of flagged samples, lifts the mean performance-to-oracle ratio from 0.544 to 0.807.","pith_inferences":["The same adaptive-calibration machinery should transfer to deployment-time monitoring outside code, such as cloud resource prediction or medical image triage, since Prom only requires a probability output and a feature embedding; the paper does not test those domains.","Detection quality should depend on the embedding's sensitivity to drift: a testable extension is to measure Prom's recall on drift that changes program structure while leaving the feature vector nearly unchanged, which would stress the Euclidean-distance premise.","Because Eq. (2) weights calibration scores by distance, the exchangeability guarantee of split conformal prediction no longer applies; characterizing coverage under the adaptive weighting is an open question the paper addresses only empirically through coverage deviation.","The incremental-learning loop is currently a batch fine-tune; combining Prom with a fully online update of both the model and the calibration set is a natural next step the paper leaves for future work."],"forward_implications":["A deployed performance tuner can run an expensive fallback, such as autotuning or iterative compilation, only when Prom flags a prediction, so profiling effort is spent where the model is actually wrong.","A vulnerability detector can route flagged predictions to a human reviewer, turning expert attention into a targeted check rather than a full audit of every output.","Retraining on at most 5% of Prom-identified samples, sometimes a single sample, brings the deployed performance-to-oracle ratio from 0.544 back to 0.807 on average across the five tasks.","The same wrapper transfers across classical classifiers, LSTMs, transformers, and graph neural networks, because Prom only needs the model's probability output and a feature vector for each input.","Users can trade false positives against missed drift by adjusting the significance level, so a deployment with expensive rejection costs can choose a stricter threshold."],"supporting_citations":[{"why":"Supplies the conformal prediction theory from which Prom's p-value and prediction-set scores are derived.","marker":"[10]"},{"why":"Prior conformal-evaluation approach to malware classification that motivates using conformal prediction to flag drifted samples.","marker":"[11]"},{"why":"Earlier conformal drift detector for malware whose full-calibration, single-function design Prom replaces.","marker":"[37]"},{"why":"Prior conformal-prediction-based drift detection for wireless sensing that Prom extends to regression and code tasks.","marker":"[83]"},{"why":"Defines the TopK and RAPS nonconformity functions used in Prom's committee.","marker":"[6]"},{"why":"Defines the APS nonconformity function used in Prom's committee.","marker":"[57]"},{"why":"Defines the LAC nonconformity function used in Prom's committee.","marker":"[58]"},{"why":"Provides the split conformal prediction setup, including the default 10% calibration split.","marker":"[5]"},{"why":"Gap statistic method Prom uses to choose the cluster count for regression p-values.","marker":"[67]"},{"why":"Incremental learning background for retraining deployed models on flagged samples.","marker":"[30]"}],"fun_headline_variants":["Code AI flags 96% of its own mispredictions","ML code models catch 96% of mistakes at runtime","Deployment-time ML: detect 96% errors, relabel 5%","Prom: code ML self-detects 96% of mispredictions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a test input the model is about to mispredict will be far, in ordinary Euclidean distance, from the calibration samples in the numerical feature vectors the user supplies; if drift hides from that embedding, the flagging signal collapses.","fun_headline_variants_meta":{"raw":{"variants":["Code AI flags 96% of its own mispredictions","ML code models catch 96% of mistakes at runtime","Deployment-time ML: detect 96% errors, relabel 5%","Prom: code ML self-detects 96% of mispredictions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0003,"raw_usage":{"total_tokens":1738,"prompt_tokens":959,"completion_tokens":779,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":703}},"tokens_in":575,"tokens_out":779,"duration_ms":7527,"temperature":1.0,"reasoning_tokens":703,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:56:34.791592+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained model and a drifted test set with known ground truth, run Prom at the default significance level, and compare the flagged set with the set of actual mispredictions: the paper's claim predicts at least 96% of true mispredictions flagged and fewer than 14% of correct predictions flagged. A concrete stress test is to construct a drifted workload whose control-flow structure changes but whose feature embedding is nearly unchanged (for example, permuting loop structure or renaming variables in an IR-based embedding); if such inputs are mispredicted yet not flagged, the Euclidean-distance premise fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier conformal drift detector for malware whose full-calibration, single-function design Prom replaces."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the APS nonconformity function used in Prom's committee."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gap statistic method Prom uses to choose the cluster count for regression p-values."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Incremental learning background for retraining deployed models on flagged samples."}],"review_version":1}