{"id":"7f787e0a-6a31-4be9-9b32-042e0fd59412","arxiv_id":"2607.18573","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Across SCMS, DataCo, and Olist, ML exposure-weighted prioritization beat severity-only ranking but beat a no-model value-sorting baseline on only DataCo.","lead":"Machine-learning delay-risk ranking (predicted delay severity × shipment value) was tested against simply reviewing the highest-value shipments first across three public supply-chain datasets. It beat severity-only ranking everywhere but beat the no-model value-sorting baseline on only one dataset, so the paper argues value sorting should remain the default benchmark until ML clears a statistical gate.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pooled test-fold capture may not implement the per-cycle 10% budget: global top-k across rolling-origin folds can change the gate.","rationale":"I read the paper as making a deployment diagnostic claim: ML exposure-weighting should be adopted only when it beats a value-only baseline under temporally valid evaluation. The load-bearing result is Table 2. I agree with the reader that the d_i*v_i exposure proxy is a stated condition, but the paper explicitly concedes that in Section 8 and frames the conclusion as conditional on that cost model; it is a limitation rather than a latent defect. The more concerning issue is the capture computation. Section 5.2's 'pool outer-test scores' plus Eq. (3)'s global top-k definition means the evaluation is a global ranking over all test periods, not a per-period ranking of upcoming shipments. Given the paper's own emphasis on limited review capacity and the drift documented in Section 4.3, this can change which shipments are in the top 10% and therefore change the gate. The empirical claim is central and the fix is straightforward; until the check is run, I would not treat the headline numbers as validated. This differs from the reader's weakest assumption, so I mark disagreement.","tokens_in":10694,"tokens_out":8062,"duration_ms":100022,"concrete_test":"Recompute Capture@10% separately within each of the five outer rolling-origin test folds, then aggregate captured exposure across folds (sum or exposure-weighted mean) instead of pooling test scores; use a paired bootstrap that resamples folds/blocks to respect temporal dependence. Compare the resulting M1 − VALUE_ONLY CIs to Table 2. If DataCo's +10.10 pp CI or the SCMS/Olist negative CIs change sign or CI width materially, the Section 6.2 gate conclusion is not robust to the operational budget definition; if unchanged, the pooled protocol is benign. Releasing the code would make this check direct.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"Section 5.2 says to 'pool outer-test scores' before computing paired capture differences, and Eq. (3) defines T_k(r) as the top k% of scores over that set. Under the paper's own framing — a manager with capacity to review only a few upcoming shipments — the 10% budget applies within a decision cycle, not over a multi-period archive. Pooling the five rolling-origin test blocks and taking a global top 10% lets shipments from different periods compete in one ranking, so a fold with high value or delay drift (Section 4.3 documents such drift) can dominate the selected set. The Table 2 deltas therefore may not be the exposure a manager would actually capture by applying a fixed 10% review budget period by period. Because the deployment gate is a strictly positive CI on this pooled metric, the headline conclusion 'clears the gate only on DataCo' is not yet attached to the operational protocol the abstract promises. The concern is checkable and does not require new data.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether machine-learning-based exposure-weighted prioritization beats a no-model value-sorting baseline under a fixed review budget. On three public datasets (SCMS, DataCo, Olist), it uses five rolling-origin folds, training-fold-only preprocessing, and 1000-sample paired bootstrap confidence intervals to compare seven rankers, with the primary comparison M1 (predicted severity × known value) versus VALUE_ONLY. At k=10%, M1 beats severity-only ranking in all datasets but clears the value-only gate (strictly positive CI for M1−VALUE_ONLY) only on DataCo (+10.1 pp); it is not significantly different on SCMS and significantly worse on Olist. Nested-CV cost-sensitive retraining does not improve M1. The paper concludes that value sorting should remain a permanent benchmark and ML deployment should be gated by learnability/calibration audits and a strictly positive CI in rolling-origin evaluation.","tokens_in":10992,"tokens_out":8831,"duration_ms":112121,"significance":"This is a useful and appropriately modest conditional result. Its strengths are the explicit gate rule, leakage-controlled rolling-origin evaluation, training-fold-only preprocessing, paired bootstrap uncertainty, the SCMS selection-bias correction, and the use of three heterogeneous public datasets. The central message — that ML delay-risk prioritization is not automatically superior to sorting by known value — is practically important and clearly stated. The main reservation is that the gate is computed on a global top-k over pooled test folds, which may not correspond to the per-cycle budget described in the introduction; this needs to be resolved before the headline conclusion is fully supported. The paper also honestly limits its claims about causality and generalizability, and acknowledges that the multiplicative exposure proxy conditions the findings.","major_comments":[{"comment":"The gate is computed on pooled outer-test scores, not on a per-cycle basis. T_k(r) is defined as the top k% over the pooled set of all five rolling-origin test blocks, and the paired bootstrap CIs in Table 2 are computed on this pooled metric. The operational protocol in the introduction is a manager with a fixed review budget over upcoming/current shipments, i.e., a per-cycle decision. With the temporal drift documented in §4.3, a global top 10% over pooled folds allows one period to dominate the selected set, so the reported Δgate (e.g., +10.1 pp on DataCo) is not necessarily the capture a manager would achieve by applying a 10% budget period by period. Please recompute Capture@k per outer test fold and report per-fold deltas and a fold-aware bootstrap, or explicitly justify the pooled top-k as the intended operational target. This is central because the paper's headline conclusion is","section":"§5.2, Eq. (3), Table 2"},{"comment":"The target-transformation selection is described only as 'pooled rolling-origin validation RMSE' with a 'stored pipeline' that 'selects' raw or log(1+d). It is not stated whether this choice is made inside each outer training fold (e.g., by nested rolling-origin validation) or on the outer test folds. If the latter, the leakage-control claim is violated and M1's results could be biased upward. Please state explicitly that the raw/log choice is part of the inner training-loop selection, or, if it is not, re-run the analysis with the transformation chosen on training folds only.","section":"§5.2"}],"minor_comments":[{"comment":"R², Spearman ρ, and calibration bias are reported as point estimates without intervals. Since these are used to explain the gate pattern, report per-fold ranges or bootstrap intervals (at least for ρ and R²).","section":"Table 2, §6.3"},{"comment":"Code and scripts are promised for final publication but are not released with the manuscript. For a reproducibility-focused diagnostic, please release them before final acceptance.","section":"Data and Code Availability"},{"comment":"The full ranker comparison and the BINARY×VALUE versus M1 claims are given without CIs. Add paired intervals or at least indicate significance for all comparisons, not only for M1−VALUE_ONLY and M2b_tuned−M1.","section":"Table 4, §6.7"},{"comment":"The deployment checklist says to 'audit ρ and calibration' but gives no concrete thresholds. Specify a decision rule or state explicitly that thresholds must be context-specific and chosen before evaluation.","section":"§7.2"},{"comment":"The phrase 'stored pipeline' is unclear. Use a standard term such as 'inner-validation selection' and describe exactly which data are used to choose the target transformation.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the pooled test-fold issue is the key risk. If the authors can recompute per-fold capture and show the gate conclusion unchanged, this work is close to acceptable; otherwise the headline claim needs revision. The paper's multiplicative exposure proxy is acknowledged as a limitation, but it conditions all three-dataset conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is the result, not the method. On DataCo, M1 (predicted severity times value) beats value-only by 10pp with a tight CI; on SCMS and Olist it doesn't, and on Olist it's clearly worse. That is a real finding, and I don't think it is an artifact of leakage or overfitting. The rolling-origin design, fold-local preprocessing, paired bootstrap CIs, and the explicit gate rule are all done carefully. Credit also for reporting that cost-sensitive retraining does not help; the DataCo non-improvement CI is precise enough to be informative. The paper is honest about the multiplicative exposure proxy and the fact that three datasets don't establish a threshold.\n\nThe soft spots are mostly reporting. The one that could be load-bearing is the pooling of outer-test folds before computing top-k. The paper frames the decision as a manager with a fixed review budget, which is per cycle, but Eq. (3) and Section 5.2 pool scores from all folds and take a global top 10%. If value or delay prevalence drifts (Section 4.3 says it does), one period can dominate the selected set, and the gate CI no longer describes the per-cycle policy. This is checkable with a per-fold or blockwise analysis. I would not call it fatal, but it needs to be addressed; if the per-cycle result reverses the gate on any dataset, the headline changes. The fix is simple, so I expect it to be minor.\n\nAlso, code is promised but not released, so independent verification is limited. R², ρ, and calibration bias are reported without intervals, but these are descriptors for the mechanism story, not the main result.\n\nOverall, the central claim holds up as far as I can see. The paper is not trying to be a breakthrough; it is a protocol plus a cautionary empirical pattern, and it is clear about its limits. I'd send it to peer review. The pooling issue is a legitimate reviewer question, not a desk-reject. I'd also bring it to reading group as a good example of how to evaluate decision-focused ML.","headline":"Clear, well-scoped empirical paper: value sorting is a stronger baseline than most ML evaluations assume, and the three-dataset gate result is a useful caution; the main item for review is whether the pooled top-k matches the per-cycle deployment story.","tokens_in":11395,"tokens_out":3464,"would_cite":true,"duration_ms":40034,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that machine-learning shipment prioritization should be judged against no-model value sorting, and shows that across three datasets the exposure-weighted ML rule clears that gate only when delay severity is learnable and c","keywords":["delay prediction","exposure-weighted ranking","value sorting baseline","rolling-origin evaluation","paired bootstrap","calibration","cost-sensitive learning","shipment prioritization"],"falsifier":"Run the same leakage-controlled rolling-origin protocol on a fourth dataset whose out-of-sample severity R-squared is near zero but whose M1 minus VALUE_ONLY paired 95% confidence interval is strictly positive; that would falsify the claim that severity learnability is necessary to clear the value-only gate.","tokens_in":10624,"feed_emoji":"📦","tokens_out":4315,"duration_ms":47554,"temperature":0.7,"pith_summary":"The paper asks when a delay-prediction model is worth deploying for shipment review: with capacity to inspect only a fraction of shipments, should a manager use ML or just inspect the highest-value shipments first? It argues that predicted delay severity multiplied by known value is the right ranking rule, but that it must be compared with value-only sorting, not just severity-only ranking. Across three real supply-chain datasets, this ML rule beats severity-only ranking everywhere, but beats value sorting on only one of the three (DataCo, +10.1 percentage points at a 10% review budget), while losing by 5.5 and 4.9 points on the other two. The divide tracks severity learnability: DataCo severity is predictable and well calibrated, while the other two are near noise and miscalibrated. The paper concludes that value sorting should remain a permanent benchmark and that ML should be deployed only when paired bootstrap confidence intervals clear the gate.","feed_headline":"Machine learning beats value sorting in only one of three datasets","feed_subtitle":"At a 10% review budget, predicted delay × value clears the no-model gate only on DataCo; audit learnability before deploying.","key_machinery":"The exposure proxy e_i = d_i times v_i, where d_i is delay severity in days and v_i is known shipment value, defines the operational objective. Capture@k measures the share of realized exposure captured by the top k% of a scoring rule. The value-only gate is the deployment criterion: M1 clears it only if the paired bootstrap 95% confidence interval for Capture@10%(M1) minus Capture@10%(VALUE_ONLY) is strictly positive, computed over leakage-controlled rolling-origin test folds. The learnability/calibration audit (Spearman rho, R-squared, calibration bias) is the proposed pre-deployment diagnostic that explains and predicts the gate verdict.","core_discovery":"The central discovery is conditional: exposure-weighted ML ranking beats the no-model value-only baseline only when delay severity is genuinely learnable at planning time. In the three datasets, predicted severity times known value (M1) improves over severity-only ranking by 34.7, 11.1, and 27.1 percentage points, yet only DataCo shows a strictly positive paired 95% confidence interval for M1 minus value sorting at a 10% budget (+10.1 points, CI [9.67, 10.42]); SCMS and Olist have negative or crossing intervals. The same data show DataCo severity with Spearman rho 0.386, R-squared 0.270, calibration bias +0.01 days, versus near-zero rho and negative bias on the other two. A nested cross-vali","pith_inferences":["The same gate logic transfers to any budgeted review setting where exposure factorizes into a known magnitude and an uncertain risk, such as fraud screening, predictive maintenance, or debt collections; in each, sorting by known magnitude is the hard baseline to beat.","Because value enters exposure multiplicatively, the value-only baseline is strong by construction; under nonlinear or threshold delay penalties, such as contract breaches or production halts, the ranking could flip, so the paper's conclusion is conditional on the multiplicative cost model.","A cheap pre-deployment audit of rank correlation and calibration on rolling-origin folds may be enough to decide against ML without full bootstrap testing, since the gate verdict tracks these diagnostics in the three datasets.","If severity is not learnable, recalibration alone cannot rescue the model; the data suggest managers should check rank informativeness before investing in calibration improvements."],"forward_implications":["Value sorting must stay a permanent benchmark: a model that beats severity-only ranking can still forfeit exposure compared with a simple database sort.","Deploy ML only when a strictly positive paired bootstrap interval over rolling-origin folds supports the gain; point estimates alone do not pass the gate.","Cost-sensitive retraining via example weighting does not reliably beat plain predicted-severity-times-value, so simpler ranking rules should be preferred unless nested validation shows a stable advantage.","Known shipment value matters at ranking time: M1 beats severity-only ranking in all three datasets by 11 to 35 percentage points.","Gate verdicts are context-dependent; a model that helps in one setting can cost exposure in another, so cross-dataset transfer should not be assumed."],"fun_headline_variants":["ML beats value sorting only when severity is learnable","Value sorting beats ML on two of three datasets","Audit learnability before deploying ML for shipment checks","ML wins only on DataCo; value sorting rules elsewhere"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire verdict rests on equating business exposure with delay severity (in days) multiplied by shipment value; if real delay costs are nonlinear or threshold-based, the value-only baseline could be much weaker or stronger and the winner could change.","fun_headline_variants_meta":{"raw":{"variants":["ML beats value sorting only when severity is learnable","Value sorting beats ML on two of three datasets","Audit learnability before deploying ML for shipment checks","ML wins only on DataCo; value sorting rules elsewhere"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000648,"raw_usage":{"total_tokens":2858,"prompt_tokens":837,"completion_tokens":2021,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":1970}},"tokens_in":581,"tokens_out":2021,"duration_ms":13963,"temperature":1.0,"reasoning_tokens":1970,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T14:58:13.504515+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same leakage-controlled rolling-origin protocol on a fourth dataset whose out-of-sample severity R-squared is near zero but whose M1 minus VALUE_ONLY paired 95% confidence interval is strictly positive; that would falsify the claim that severity learnability is necessary to clear the value-only gate.","supporting_citations":[],"review_version":1}