{"id":"9c447887-558d-4c00-b35c-15113ffad952","arxiv_id":"2505.17340","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Conformal predictive distributions, including a new two-stage truncation method, outperform rule-based delivery time forecasts on a 5-million-order industrial dataset.","lead":"This paper applies conformal prediction methods to forecast e-commerce order fulfillment time, producing probabilistic delivery estimates with coverage guarantees. On a dataset of over five million orders, the machine learning models beat the company's rule-based system in accuracy and in flagging late deliveries.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The carrier-matching test filter and the two-stage truncation both break the exchangeability needed for the paper's coverage-guarantee claim; Table 2's below-nominal two-stage coverage is evidence, so the abstract overstates the guarantee.","rationale":"The paper's core value proposition is a conformal distributional forecasting framework with finite-sample guarantees and large empirical gains on an industrial dataset. The load-bearing condition is that the conformal machinery's exchangeability premise holds on the reported test population. The paper itself presents evidence against this: Section 5.1's carrier-matching filter selects test orders based on realized outcomes, and Table 2 shows the proposed two-stage methods undercover by 0.5 to 1.7 percentage points. This is not a minor calibration issue; it directly contradicts the abstract's promise of rigorous coverage or validity guarantees for the main proposed method. The reader's conditional verdict already identifies the exchangeability concern; my stress test makes it more concrete by tying the undercoverage to the two-stage truncation and by proposing a direct unfiltered-test check. I do not see a deeper internal inconsistency in the base CPS/CV AP machinery, and the reported industrial deployment interest is a positive external signal. However, the guarantee overstatement and the selection-sensitive evaluation are sufficient to keep the verdict conditional rather than accept the claims at face value.","tokens_in":15437,"tokens_out":6924,"duration_ms":90978,"concrete_test":"Repeat the Table 2 and Table 3 evaluations on a test set that is not filtered by realized carrier, using the same calibration splits and trained models, and report coverage and point metrics for all methods. If two-stage coverage returns to nominal and the late-detection gains shrink, the carrier-matching filter is the source of the reported behavior; if undercoverage persists on the unfiltered test set, the truncation method itself breaks the coverage guarantee.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the framework provides 'rigorous coverage or validity guarantees' depends on exchangeability of calibration and test scores. Section 5.1 breaks this premise in two ways. First, the test set is restricted to the last two months and then to orders whose actual carrier matches the carrier estimated at location selection; both are post-hoc selections made using information unavailable at calibration time, including the realized fulfillment outcome. The calibration folds are not shown to undergo the same filter, so the score distributions need not match. Second, the paper's own two-stage truncation procedure in Section 4.3 is not covered by the base SCPS/CV AP guarantees: scores are split at zero and renormalized inside status classes, while the mixture weights come from class probabilities calibrated by CV AP. The resulting CDF is not a standard conformal predictive distribution, so finite-sample coverage does not follow from the cited results. Table 2 is consistent with this: 2stg-SCPS achieves 79.5%, 87.9%, and 93.5% coverage for nominal 80%, 90%, and 95% intervals, and the text concedes a 'modest dip' in coverage. Thus the abstract's guarantee statement is unsupported for the main proposed method. The point-prediction gains (up to 14% accuracy, up to 75% late detection) are measured on this same carrier-filtered test set using a validation-tuned quantile, with no error bars, so their generalizability is also in question.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a distributional forecasting framework for e-commerce order fulfillment time deviations, combining Conformal Predictive Systems (SCPS and MCPS) for regression with Cross Venn-Abers Predictors (CV AP) for multi-class status classification. It adds a two-stage classify-then-regress method that truncates conformal scores by delivery status, a cost-sensitive quantile-selection rule for point predictions, and granular spatio-temporal features. The framework is evaluated on an industrial dataset of more than five million orders, with distributional metrics (CRPS, pinball loss, coverage) and point metrics (accuracy, RMSE, late-delivery detection) reported against rule-based and ML baselines. The paper claims that the proposed methods provide rigorous coverage or validity guarantees and that ML-based point predictions significantly outperform the existing rule-based system.","tokens_in":15713,"tokens_out":4870,"duration_ms":39252,"significance":"If the coverage and validity claims were fully supported, the paper would be a solid applied contribution: it demonstrates conformal predictive distributions on a large real logistics dataset, incorporates practically motivated features, introduces a plausible two-stage structure, and compares against several baselines using standard open-source conformal tools, which aids reproducibility. The cost-sensitive decision rule addresses a real operational need. However, the central advertised guarantee is not established for the main proposed methods: the test-set filtering breaks exchangeability, the two-stage truncation lacks a formal guarantee, and the headline point-prediction gains come from a validation-tuned rule without error bars or significance tests. These issues are load-bearing for the abstract's claims and must be resolved before publication.","major_comments":[{"comment":"The test set is restricted to the last two months and then to orders shipped via the carrier service that was estimated at location selection, but the same filter is not stated for the calibration folds. This post-hoc selection uses realized carrier information that is unavailable at prediction time and can change the score distribution relative to calibration, violating the exchangeability assumption on which the Section 4.2 guarantees rest. Table 2 is consistent with this concern: 2stg-SCPS achieves 79.5%, 87.9%, and 93.5% coverage for nominal 80%, 90%, and 95% intervals, and CV AP MC-CLF achieves only 65.8% at the nominal 80% level. The claim in Section 5.2 that CPS-based methods 'all reach nominal coverages' is true for the SCPS and MCPS rows, but it does not hold for the two-stage or CV AP methods, and the abstract's general guarantee statement is therefore unsupported for the main proposed framework. Please either apply the same selection rule to calibration and test data with a justification of exchangeability, or restrict the coverage-guarantee claims to the settings in which they are proven.","section":"Section 5.1, Table 2"},{"comment":"The two-stage truncation method constructs class-conditional predictive distributions by taking signed subsets of calibration scores and mixing them with CV AP class probabilities. This CDF is not a standard Split Conformal Predictive System or Mondrian CPS, and no theorem in the paper shows that its quantiles inherit finite-sample coverage from the cited CPS guarantees. The text in Section 4.3 states that the approach 'ensures that calibration guarantees are preserved across the full range of predictions,' but this assertion is not supported by a proof or by the reported coverage, which falls below nominal for 2stg-SCPS in Table 2. Please provide a formal guarantee for the two-stage mixture distribution or explicitly describe the coverage results as empirical rather than guaranteed.","section":"Section 4.3"},{"comment":"The cost-sensitive decision rule selects a single eta* by minimizing RMSE + beta*Late_RMSE + gamma*Early_RMSE on a validation set, with beta=0.5 and gamma=0 in the experiments. The headline late-detection improvements in Table 3 (for example, SCPS-Rule detecting 79.37% of late deliveries versus 3.82% for the current system) are produced by applying this validation-tuned rule to the same carrier-filtered test set, so the reported gains are partly fitted outcomes. No standard errors, confidence intervals, or significance tests are reported for any metric in Table 3, so the abstract's claim that ML-based predictions 'significantly outperform' the rule-based system is not statistically supported. Please report variability across folds or bootstrap resamples and clarify that eta* is a fitted parameter when presenting the decision rule.","section":"Section 4.4, Table 3"},{"comment":"The text says that CV AP provides 'distribution-free, finite-sample calibration guarantees' and the abstract extends this to the whole framework. However, Table 2 shows that CV AP MC-CLF coverage is 65.8%, 83.5%, and 93.5% for nominal 80%, 90%, and 95% intervals, which is substantially below nominal at the 80% level. The one-versus-rest extension to multi-class, followed by normalization, is not the standard binary Venn-Abers setting, and the paper does not prove that the normalized multi-class probabilities preserve the binary validity property. Please either provide a validity argument for the multi-class normalized estimator or soften the guarantee language to match the empirical evidence.","section":"Section 4.2, CV AP"}],"minor_comments":[{"comment":"The phrase 'up to 14% higher prediction accuracy' is ambiguous: Table 3 shows an increase from 49.53% to 64.41% accuracy, which is about 15 percentage points or about 30% relative. Please state the comparison base explicitly.","section":"Abstract and Table 3"},{"comment":"The phrase 'up to 75% improvement in identifying late deliveries' is an unusually large relative gain from a 3.82% baseline. Please report absolute percentage-point differences alongside relative changes.","section":"Abstract and Table 3"},{"comment":"The test-set restriction to orders shipped via the matching carrier service is described as enabling 'a direct and unbiased comparison between the proposed models and the existing system,' but the same restriction is not stated for the training and calibration data. Clarify how the rule-based system's historical decisions interact with this filter.","section":"Section 5.1"},{"comment":"The note says that coverage values achieving nominal rates are underlined, but the underlining does not appear in the text. Please ensure the formatting is visible or replace it with a marker.","section":"Table 2"},{"comment":"The objective function includes both beta and gamma as user-defined penalties, but the experiments only vary beta (with gamma=0). A sensitivity analysis for gamma would help justify the claim that the rule flexibly balances early and late penalties.","section":"Section 4.4"},{"comment":"The target Y is described as a discrete set {c1,...,cK}, but the two-stage method defines three status classes from the sign of Y. Please state explicitly how the K discrete deviation values map to the early, on-time, and late groups.","section":"Section 2 and Section 4.3"},{"comment":"The geometric-mean aggregation for CV AP is called 'minimax-style'; a derivation or citation showing why this aggregation preserves validity would be helpful for readers.","section":"Section 4.2, Equation (1)"},{"comment":"The text says that MCPS produces '0.1 to 0.3 days narrower intervals' than SCPS, but Table 2 shows this holds at the 90% level while at the 95% level SCPS has a smaller mean interval size. Please reconcile the claim with the table.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is mainly applicational: the conformal tools are from prior work, and the novel two-stage truncation method lacks a formal guarantee. The industrial dataset is valuable, but the presentation overstates the statistical guarantees, particularly in the abstract. If the journal's scope is applied machine learning or operations management, the paper can become acceptable after the claims are tempered and the empirical evaluation is augmented with uncertainty quantification. If the scope is statistical methodology, the missing proof for the two-stage method is a more serious barrier."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a serious applied conformal prediction paper with one genuinely new construction and a real industrial evaluation. It should get peer review, but the abstract overstates the coverage guarantee and the evaluation has a few selection issues that need to be addressed.\n\nWhat's new: first application of CPS and CV AP to fulfillment time forecasting; the two-stage truncation-based classify-then-regress (Section 4.3) is a new way to combine class probabilities with conformal residuals without training class-specific models, and it produces visibly sharper intervals. The cost-sensitive quantile selection rule (Section 4.4) is a sensible extension of known ideas. Feature engineering is thorough and the dataset is large (5M+ orders).\n\nWhat works: the point-prediction comparison against the rule-based system is the strongest part. Even after discounting for tuned quantiles, ML methods beating a static transit table by 14% accuracy and much better late detection is plausible and practically important. The distributional comparisons are suggestive: CPS methods hit nominal coverage on the test set (SCPS 81.7/90.7/95.3), and CRPS/pinball improvements are consistent.\n\nWhere it gets soft: the guarantee language. The abstract promises 'rigorous coverage or validity guarantees,' but the proposed two-stage method does not inherit standard conformal guarantees because scores are truncated at zero and renormalized within status classes, with mixture weights from CV AP. Table 2 confirms the cost: 2stg-SCPS gets 79.5%, 87.9%, 93.5% for nominal 80/90/95. That's a modest dip, but it means the main new method is not covered by the cited theory. On top of that, the test set is filtered to orders whose actual carrier matched the carrier estimated at location selection, and the last two months are used after validation tuning. Both break exchangeability for the standard methods too. The authors should either prove a modified guarantee for the truncation procedure or drop the word 'rigorous' and show the coverage is empirically acceptable under explicit distribution shift. Also, the cost-sensitive decision rule selects eta* on the validation set, so the headline late-detection improvements are partly fitted; no error bars or significance tests anywhere, and the data is proprietary, so external verification is limited.\n\nBottom line: the core direction is sound, the point-prediction result is credible, and the two-stage construction is worth a serious referee looking at the theory. The paper needs a revision that (1) fixes the guarantee claim for the two-stage method, (2) acknowledges the exchangeability caveat from the test filter, and (3) adds error bars or at least a sensitivity analysis. If they do that, it's a solid publication. I'd send it to review.","headline":"Solid applied conformal prediction paper with a genuinely new two-stage construction; the abstract overstates coverage guarantees and the evaluation has selection issues, but it deserves peer review.","tokens_in":16272,"tokens_out":2159,"would_cite":true,"duration_ms":23507,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Conformal distribution forecasts beat rule-based delivery-time estimates by 14%","keywords":["conformal prediction","distributional forecasting","delivery time estimation","order fulfillment","Venn-Abers predictors","cost-sensitive decision rule","e-commerce logistics","quantile regression"],"falsifier":"Re-run the evaluation on the unfiltered test set or on a later two-month window and compare the reported interval coverages; if the coverages drop substantially below the nominal rates once the exchangeability violation is removed, the guarantee collapses.","tokens_in":15206,"feed_emoji":"📦","tokens_out":4952,"duration_ms":50943,"temperature":0.7,"pith_summary":"This paper tries to show that full distributional forecasts of order fulfillment time, built by wrapping machine-learning point predictors in conformal calibration, can replace static rule-based transit tables. On an industrial dataset of more than five million orders, it reports that calibrated multi-class classifiers and conformal regressors produce competitive predictive CDFs, and that point predictions derived from these distributions beat the existing rule-based system by up to 14% in overall accuracy and up to 75% in identifying late deliveries. The authors also propose a two-stage classify-then-regress method whose prediction intervals are the tightest reported, and a cost-sensitive quantile selection rule that trades accuracy for late-delivery detection in a controlled way.","feed_headline":"Conformal forecasts beat rule-based delivery-time system by 14%","feed_subtitle":"Calibrated distributional forecasting also flags up to 75% more late deliveries.","key_machinery":"The central objects are Conformal Predictive Systems (CPS) and Cross Venn-Abers Predictors (CVAP), plus a two-stage truncation combination. CPS converts any regression model's point prediction into a full predictive CDF by shifting the test prediction with calibration residuals and sorting them; Mondrian CPS bins calibration scores by predicted value for local adaptation. CVAP calibrates multi-class classifier scores using cross-validated dual isotonic regression and aggregates fold-wise intervals with a minimax geometric-mean rule. The two-stage method classifies orders as early, on-time, or late, then builds class-conditional distributions from truncated calibration scores and mixes them using class probabilities.","core_discovery":"The central claim is that conformal calibration turns ordinary ML forecasts into reliable full predictive distributions for the fulfillment time deviation, measured in days relative to the promised date, and that doing so improves operational point decisions. The evidence is a test-set comparison in which Conformal Predictive Systems and Cross Venn-Abers calibrated classifiers outperform the existing rule-based system and the QRF and isotonic-regression baselines: up to 14% higher overall accuracy, up to 75% higher late-delivery detection, and 11-13% lower pinball loss than QRF. The two-stage truncation method classifies each order as early, on-time, or late, then mixes class-conditional conformal distributions, yielding the narrowest intervals (0.2-0.4 days narrower than QRF) while keeping interval mass on the correct side of zero.","pith_inferences":["Editorial extension: if temporal drift breaks exchangeability, a sliding-window or weighted conformal calibration would likely restore coverage while preserving the cost-sensitive point rule.","Editorial extension: the decision rule's single global quantile could be replaced by per-order quantile selection conditioned on predicted status, a change the paper does not test but that could improve the accuracy-versus-late-detection frontier.","Editorial extension: since calibration uses only residuals and class scores, the same CPS/CVAP layers should port to deep learning or other point predictors without retraining the conformal machinery.","Editorial extension: the feature-importance evidence points to hub-to-hub transit times and fulfillment-location processing windows as primary signals, suggesting similar gains are plausible for other carriers once similar spatio-temporal features are constructed."],"forward_implications":["If the results hold, e-commerce platforms can replace static promise-date tables with distributional forecasts that support quantile-based service targets and risk-aware planning.","The cost-sensitive rule gives operators a tunable lever: raising the late-delivery penalty weight increases late-detection rates by 1.5-4.2 percentage points while reducing overall accuracy by 4-8 percentage points.","Two-stage predict-then-mix forecasts produce tight intervals around zero for on-time orders and correctly signed intervals for early or late orders, which can reduce unnecessary buffer days in promises and capacity planning.","Because the conformal layers are model-agnostic, any future improvement in the underlying regression or classification model should translate directly into sharper and better-calibrated distributions.","Users of the two-stage method should expect coverage slightly below the nominal level, so those intervals are best treated as approximate guarantees."],"supporting_citations":[{"why":"Supplies the Conformal Predictive System framework, the core distributional forecasting method.","marker":"[30]"},{"why":"Provides Mondrian Conformal Predictive Systems, the binning variant that gives locally adaptive intervals.","marker":"[5]"},{"why":"Provides the Cross Venn-Abers Predictor used to calibrate multi-class classifier probabilities.","marker":"[17]"},{"why":"Defines the fulfillment-time distributional forecasting problem and the QRF/isotonic baselines the paper extends.","marker":"[33]"},{"why":"Establishes the cost-centric decision-rule approach for delivery-time point predictions.","marker":"[25]"},{"why":"Supplies quantile regression forests, the main regression baseline for distributional comparison.","marker":"[19]"}],"fun_headline_variants":["Conformal ML beats rule-based delivery-time forecasts by 14%","Conformal calibration: 14% accuracy gain, 75% better late detection","Conformal forecasts beat rules: 14% higher accuracy, 75% late flag","Calibrated conformal models improve delivery-time accuracy 14%","14% more accurate delivery-time forecasts via conformal ML"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The calibration and test orders must be exchangeable, so that the prediction errors observed during calibration remain representative of future test errors; the test set's temporal separation and carrier filtering make this the load-bearing premise whose failure would void the coverage guarantees.","fun_headline_variants_meta":{"raw":{"variants":["Conformal ML beats rule-based delivery-time forecasts by 14%","Conformal calibration: 14% accuracy gain, 75% better late detection","Conformal forecasts beat rules: 14% higher accuracy, 75% late flag","Calibrated conformal models improve delivery-time accuracy 14%","14% more accurate delivery-time forecasts via conformal ML"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000575,"raw_usage":{"total_tokens":2665,"prompt_tokens":843,"completion_tokens":1822,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":459,"completion_tokens_details":{"reasoning_tokens":1726}},"tokens_in":459,"tokens_out":1822,"duration_ms":7752,"temperature":1.0,"reasoning_tokens":1726,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:48:18.480075+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation on the unfiltered test set or on a later two-month window and compare the reported interval coverages; if the coverages drop substantially below the nominal rates once the exchangeability violation is removed, the guarantee collapses.","supporting_citations":[{"cited_title":"Neurocomputing 397, 292–308 (2020)","cited_arxiv_id":null,"evidence_quote":"Supplies the Conformal Predictive System framework, the core distributional forecasting method."},{"cited_title":"In: Conformal and Prob- abilistic Prediction and Applications","cited_arxiv_id":null,"evidence_quote":"Provides Mondrian Conformal Predictive Systems, the binning variant that gives locally adaptive intervals."},{"cited_title":"In: Con- formal and Probabilistic Prediction and Applications","cited_arxiv_id":null,"evidence_quote":"Provides the Cross Venn-Abers Predictor used to calibrate multi-class classifier probabilities."},{"cited_title":"arXiv preprint arXiv:2409.06918 (2024)","cited_arxiv_id":null,"evidence_quote":"Defines the fulfillment-time distributional forecasting problem and the QRF/isotonic baselines the paper extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the cost-centric decision-rule approach for delivery-time point predictions."}],"review_version":1}