REVIEW 4 major objections 8 minor 1 cited by
Conformal Predictive Distributions for Order Fulfillment Time Forecasting
T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Conformal distribution forecasts beat rule-based delivery-time estimates by 14%
desk verdict Solid applied conformal prediction paper with a genuinely new two-stage construction; the abstract overstates coverage guarantees and the evaluation has selection issues, but it deserves peer review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central objects are Conformal Predictive Systems (CPS) and Cross Venn-Abers Predictors (CVAP), plus a two-stage truncation combination. CPS converts any regression model's point prediction into a full predictive CDF by shifting the test prediction with calibration residuals and sorting them; Mondrian CPS bins calibration scores by predicted value for local adaptation. CVAP calibrates multi-class classifier scores using cross-validated dual isotonic regression and aggregates fold-wise intervals with a minimax geometric-mean rule. The two-stage method classifies orders as early, on-time, or late, then builds class-conditional distributions from truncated calibration scores and mixes them using class probabilities.
What would settle it
Re-run the evaluation on the unfiltered test set or on a later two-month window and compare the reported interval coverages; if the coverages drop substantially below the nominal rates once the exchangeability violation is removed, the guarantee collapses.
Extended reading notes
Core claim
The central claim is that conformal calibration turns ordinary ML forecasts into reliable full predictive distributions for the fulfillment time deviation, measured in days relative to the promised date, and that doing so improves operational point decisions. The evidence is a test-set comparison in which Conformal Predictive Systems and Cross Venn-Abers calibrated classifiers outperform the existing rule-based system and the QRF and isotonic-regression baselines: up to 14% higher overall accuracy, up to 75% higher late-delivery detection, and 11-13% lower pinball loss than QRF. The two-stage truncation method classifies each order as early, on-time, or late, then mixes class-conditional conformal distributions, yielding the narrowest intervals (0.2-0.4 days narrower than QRF) while keeping interval mass on the correct side of zero.
Load-bearing premise
The calibration and test orders must be exchangeable, so that the prediction errors observed during calibration remain representative of future test errors; the test set's temporal separation and carrier filtering make this the load-bearing premise whose failure would void the coverage guarantees.
Editorial extensions
If this is right
- If the results hold, e-commerce platforms can replace static promise-date tables with distributional forecasts that support quantile-based service targets and risk-aware planning.
- The cost-sensitive rule gives operators a tunable lever: raising the late-delivery penalty weight increases late-detection rates by 1.5-4.2 percentage points while reducing overall accuracy by 4-8 percentage points.
- Two-stage predict-then-mix forecasts produce tight intervals around zero for on-time orders and correctly signed intervals for early or late orders, which can reduce unnecessary buffer days in promises and capacity planning.
- Because the conformal layers are model-agnostic, any future improvement in the underlying regression or classification model should translate directly into sharper and better-calibrated distributions.
- Users of the two-stage method should expect coverage slightly below the nominal level, so those intervals are best treated as approximate guarantees.
Reading between the lines
- Editorial extension: if temporal drift breaks exchangeability, a sliding-window or weighted conformal calibration would likely restore coverage while preserving the cost-sensitive point rule.
- Editorial extension: the decision rule's single global quantile could be replaced by per-order quantile selection conditioned on predicted status, a change the paper does not test but that could improve the accuracy-versus-late-detection frontier.
- Editorial extension: since calibration uses only residuals and class scores, the same CPS/CVAP layers should port to deep learning or other point predictors without retraining the conformal machinery.
- Editorial extension: the feature-importance evidence points to hub-to-hub transit times and fulfillment-location processing windows as primary signals, suggesting similar gains are plausible for other carriers once similar spatio-temporal features are constructed.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a distributional forecasting framework for e-commerce order fulfillment time deviations, combining Conformal Predictive Systems (SCPS and MCPS) for regression with Cross Venn-Abers Predictors (CV AP) for multi-class status classification. It adds a two-stage classify-then-regress method that truncates conformal scores by delivery status, a cost-sensitive quantile-selection rule for point predictions, and granular spatio-temporal features. The framework is evaluated on an industrial dataset of more than five million orders, with distributional metrics (CRPS, pinball loss, coverage) and point metrics (accuracy, RMSE, late-delivery detection) reported against rule-based and ML baselines. The paper claims that the proposed methods provide rigorous coverage or validity guarantees and that ML-based point predictions significantly outperform the existing rule-based system.
Significance. If the coverage and validity claims were fully supported, the paper would be a solid applied contribution: it demonstrates conformal predictive distributions on a large real logistics dataset, incorporates practically motivated features, introduces a plausible two-stage structure, and compares against several baselines using standard open-source conformal tools, which aids reproducibility. The cost-sensitive decision rule addresses a real operational need. However, the central advertised guarantee is not established for the main proposed methods: the test-set filtering breaks exchangeability, the two-stage truncation lacks a formal guarantee, and the headline point-prediction gains come from a validation-tuned rule without error bars or significance tests. These issues are load-bearing for the abstract's claims and must be resolved before publication.
major comments (4)
- [Section 5.1, Table 2] The test set is restricted to the last two months and then to orders shipped via the carrier service that was estimated at location selection, but the same filter is not stated for the calibration folds. This post-hoc selection uses realized carrier information that is unavailable at prediction time and can change the score distribution relative to calibration, violating the exchangeability assumption on which the Section 4.2 guarantees rest. Table 2 is consistent with this concern: 2stg-SCPS achieves 79.5%, 87.9%, and 93.5% coverage for nominal 80%, 90%, and 95% intervals, and CV AP MC-CLF achieves only 65.8% at the nominal 80% level. The claim in Section 5.2 that CPS-based methods 'all reach nominal coverages' is true for the SCPS and MCPS rows, but it does not hold for the two-stage or CV AP methods, and the abstract's general guarantee statement is therefore unsupported for the main proposed framework. Please either apply the same selection rule to calibration and test data with a justification of exchangeability, or restrict the coverage-guarantee claims to the settings in which they are proven.
- [Section 4.3] The two-stage truncation method constructs class-conditional predictive distributions by taking signed subsets of calibration scores and mixing them with CV AP class probabilities. This CDF is not a standard Split Conformal Predictive System or Mondrian CPS, and no theorem in the paper shows that its quantiles inherit finite-sample coverage from the cited CPS guarantees. The text in Section 4.3 states that the approach 'ensures that calibration guarantees are preserved across the full range of predictions,' but this assertion is not supported by a proof or by the reported coverage, which falls below nominal for 2stg-SCPS in Table 2. Please provide a formal guarantee for the two-stage mixture distribution or explicitly describe the coverage results as empirical rather than guaranteed.
- [Section 4.4, Table 3] The cost-sensitive decision rule selects a single eta* by minimizing RMSE + beta*Late_RMSE + gamma*Early_RMSE on a validation set, with beta=0.5 and gamma=0 in the experiments. The headline late-detection improvements in Table 3 (for example, SCPS-Rule detecting 79.37% of late deliveries versus 3.82% for the current system) are produced by applying this validation-tuned rule to the same carrier-filtered test set, so the reported gains are partly fitted outcomes. No standard errors, confidence intervals, or significance tests are reported for any metric in Table 3, so the abstract's claim that ML-based predictions 'significantly outperform' the rule-based system is not statistically supported. Please report variability across folds or bootstrap resamples and clarify that eta* is a fitted parameter when presenting the decision rule.
- [Section 4.2, CV AP] The text says that CV AP provides 'distribution-free, finite-sample calibration guarantees' and the abstract extends this to the whole framework. However, Table 2 shows that CV AP MC-CLF coverage is 65.8%, 83.5%, and 93.5% for nominal 80%, 90%, and 95% intervals, which is substantially below nominal at the 80% level. The one-versus-rest extension to multi-class, followed by normalization, is not the standard binary Venn-Abers setting, and the paper does not prove that the normalized multi-class probabilities preserve the binary validity property. Please either provide a validity argument for the multi-class normalized estimator or soften the guarantee language to match the empirical evidence.
minor comments (8)
- [Abstract and Table 3] The phrase 'up to 14% higher prediction accuracy' is ambiguous: Table 3 shows an increase from 49.53% to 64.41% accuracy, which is about 15 percentage points or about 30% relative. Please state the comparison base explicitly.
- [Abstract and Table 3] The phrase 'up to 75% improvement in identifying late deliveries' is an unusually large relative gain from a 3.82% baseline. Please report absolute percentage-point differences alongside relative changes.
- [Section 5.1] The test-set restriction to orders shipped via the matching carrier service is described as enabling 'a direct and unbiased comparison between the proposed models and the existing system,' but the same restriction is not stated for the training and calibration data. Clarify how the rule-based system's historical decisions interact with this filter.
- [Table 2] The note says that coverage values achieving nominal rates are underlined, but the underlining does not appear in the text. Please ensure the formatting is visible or replace it with a marker.
- [Section 4.4] The objective function includes both beta and gamma as user-defined penalties, but the experiments only vary beta (with gamma=0). A sensitivity analysis for gamma would help justify the claim that the rule flexibly balances early and late penalties.
- [Section 2 and Section 4.3] The target Y is described as a discrete set {c1,...,cK}, but the two-stage method defines three status classes from the sign of Y. Please state explicitly how the K discrete deviation values map to the early, on-time, and late groups.
- [Section 4.2, Equation (1)] The geometric-mean aggregation for CV AP is called 'minimax-style'; a derivation or citation showing why this aggregation preserves validity would be helpful for readers.
- [Section 5.2] The text says that MCPS produces '0.1 to 0.3 days narrower intervals' than SCPS, but Table 2 shows this holds at the 90% level while at the 95% level SCPS has a smaller mean interval size. Please reconcile the claim with the table.
Circularity Check
No significant circularity; the conformal calibration procedures are standard external methods and the cost-sensitive decision rule is tuned on validation, not used to construct the reported test distributions.
full rationale
The paper's derivation chain is not circular. The conformal predictive systems (SCPS, MCPS) and Cross Venn-Abers calibration are standard methods with external theoretical guarantees from Vovk et al. and Boström et al., and the implementations rely on published libraries (Crepes, Venn-ABERS), so the coverage claims are imported from independent sources rather than defined in terms of the paper's own outputs. The two-stage classify-then-regress method constructs a mixture of truncated conformal distributions whose coverage is asserted rather than derived, and Table 2 indeed shows below-nominal coverage for the two-stage methods; this is an over-claiming or validity concern, not a by-construction circularity, because the mixture is not defined as whatever would make the guarantee true. The cost-sensitive decision rule in Section 4.4 selects the quantile η* on a validation set by minimizing a weighted loss, and the reported late-detection improvements are measured on a held-out test set; this is standard hyperparameter selection, not a fitted parameter renamed as a prediction. Self-citations, particularly to the authors' prior work [33] for the problem definition and features, are present but not load-bearing: no core result of this paper reduces to a self-cited theorem. The main risks are statistical (exchangeability violations from temporal split and carrier filtering) and evidentiary (no error bars), but these are not circularity.
Assumptions & free parameters
free parameters (4)
- eta* (optimal quantile level) =
Selected by grid search on validation set
- B (number of MCPS bins) =
10
- beta (late penalty weight) =
0.5 (default)
- gamma (early penalty weight) =
0 (default)
assumptions (3)
- domain assumption Exchangeability of calibration and test data
- domain assumption Target distribution is discrete daily deviations
- domain assumption Test set restriction to matching carrier service does not bias comparison
Cite this review
Pith. "Pith review of Conformal Predictive Distributions for Order Fulfillment Time Forecasting." pith.science (2026). https://pith.science/paper/SKUPWPQ5
@misc{pith2026250517340,
author = {Pith},
title = {Pith review of: Conformal Predictive Distributions for Order Fulfillment Time Forecasting},
year = {2026},
howpublished = {\url{https://pith.science/paper/SKUPWPQ5}},
note = {Machine review of arXiv:2505.17340}
}
read the original abstract
Accurate estimation of order fulfillment time is critical for e-commerce logistics, yet traditional rule-based approaches often fail to capture the inherent uncertainties in delivery operations. This paper introduces a novel framework for distributional forecasting of order fulfillment time, leveraging Conformal Predictive Systems and Cross Venn-Abers Predictors -- model-agnostic techniques that provide rigorous coverage or validity guarantees. The proposed machine learning methods integrate granular spatiotemporal features, capturing fulfillment location and carrier performance dynamics to enhance predictive accuracy. Additionally, a cost-sensitive decision rule is developed to convert probabilistic forecasts into reliable point predictions. Experimental evaluation on a large-scale industrial dataset demonstrates that the proposed methods generate competitive distributional forecasts, while machine learning-based point predictions significantly outperform the existing rule-based system -- achieving up to 14% higher prediction accuracy and up to 75% improvement in identifying late deliveries.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
When Does Machine Learning Beat Value Sorting? A Three-Dataset Diagnostic of Exposure-Weighted Shipment Prioritization
Across SCMS, DataCo, and Olist, ML exposure-weighted prioritization beat severity-only ranking but beat a no-model value-sorting baseline on only DataCo.
Reference graph
Works this paper leans on
-
[1]
In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining
Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M.: Optuna: A next-generation hyperparameter optimization framework. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. pp. 2623–2631 (2019)
2019
-
[2]
IEEE Internet of Things Journal 8(23), 17043–17056 (2021)
de Araujo, A.C., Etemad, A.: End-to-end prediction of parcel delivery time with deep learning for smart-city applications. IEEE Internet of Things Journal 8(23), 17043–17056 (2021)
work page 2021
-
[3]
Manufacturing & Service Operations Management 25(4), 1489–1508 (2023)
Arora, S., Taylor, J.W., Mak, H.Y .: Probabilistic forecasting of patient waiting times in an emergency department. Manufacturing & Service Operations Management 25(4), 1489–1508 (2023)
work page 2023
-
[4]
In: Conformal and Probabilistic Prediction with Applications
Boström, H.: crepes: a python package for generating conformal regressors and predictive systems. In: Conformal and Probabilistic Prediction with Applications. pp. 24–41. PMLR (2022)
work page 2022
-
[5]
In: Conformal and Prob- abilistic Prediction and Applications
Boström, H., Johansson, U., Löfström, T.: Mondrian conformal predictive distributions. In: Conformal and Prob- abilistic Prediction and Applications. pp. 24–38. PMLR (2021)
work page 2021
-
[6]
Confidence-Aware Deep Learning for Load Plan Adjustments in the Parcel Service Industry
Bruys, T., Zandehshahvar, R., Hijazi, A., Van Hentenryck, P.: Confidence-aware deep learning for load plan adjustments in the parcel service industry. arXiv preprint arXiv:2411.17502 (2024)
work page Pith review arXiv 2024
-
[7]
Cao, J., Leng, Y ., Wang, H.: Revenue-centered delivery time presentation on platforms: A spatial neural ode approach (2024)
work page 2024
-
[8]
Chen, T., Guestrin, C.: Xgboost: A scalable tree boosting system. In: Proceedings of the 22nd acm sigkdd inter- national conference on knowledge discovery and data mining. pp. 785–794 (2016)
work page 2016
Show all 38 references
-
[9]
arXiv preprint arXiv:2504.09680 (2025)
Cheng, S., Hijazi, A., Konak, J., Erera, A., Van Hentenryck, P.: Spot: Spatio-temporal pattern mining and opti- mization for load consolidation in freight transportation networks. arXiv preprint arXiv:2504.09680 (2025)
2025 arXiv
-
[10]
Manufacturing & Service Operations Management 26(1), 233–251 (2024)
Cui, R., Lu, Z., Sun, T., Golden, J.M.: Sooner or later? promising delivery speed in online retail. Manufacturing & Service Operations Management 26(1), 233–251 (2024)
2024
-
[11]
Faulkner, S., Zandshahvar, R., Akhlaghi, V .E., Hentenryck, P.V .: A data-driven deep learning framework for package delivery time estimation and delay duration prediction (2025), Working Paper
2025
-
[12]
Manufacturing & Service Operations Management 24(6), 3193–3214 (2022)
Guo, X., Grushka-Cockayne, Y ., De Reyck, B.: Forecasting airport transfer passenger flow using real-time data and machine learning. Manufacturing & Service Operations Management 24(6), 3193–3214 (2022)
2022
-
[13]
Applied Energy 361, 122900 (2024)
Jonkers, J., Avendano, D.N., Van Wallendael, G., Van Hoecke, S.: A novel day-ahead regional and probabilistic wind power forecasting framework using deep cnns and conformalized regression forests. Applied Energy 361, 122900 (2024)
2024
-
[14]
arXiv preprint arXiv:2503.04071 (2025)
Li, M., Klamkin, M., Tanneau, M., Zandehshahvar, R., Van Hentenryck, P.: Conformal prediction with upper and lower bound models. arXiv preprint arXiv:2503.04071 (2025)
2025
-
[15]
In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management
Li, Y ., Wu, X., Wang, J., Liu, Y ., Wang, X., Deng, Y ., Miao, C.: Unsupervised categorical representation learning for package arrival time prediction. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management. pp. 3935–3944 (2021)
2021
-
[16]
In: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Min- ing
Liu, H., Jiang, W., Liu, S., Chen, X.: Uncertainty-aware probabilistic travel time prediction for on-demand ride- hailing at didi. In: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Min- ing. pp. 4516–4526 (2023)
2023
-
[17]
In: Con- formal and Probabilistic Prediction and Applications
Manokhin, V .: Multi-class probabilistic classification using inductive and cross venn–abers predictors. In: Con- formal and Probabilistic Prediction and Applications. pp. 228–240. PMLR (2017) 16 Ye et al
2017
-
[18]
Manokhin, V .: Awesome conformal prediction (Apr 2022), https://doi.org/10.5281/zenodo. 6467205
2022 doi
-
[19]
Journal of machine learning research 7(6) (2006)
Meinshausen, N., Ridgeway, G.: Quantile regression forests. Journal of machine learning research 7(6) (2006)
2006
-
[20]
Transportation Science 58(1), 8–11 (2024)
Merchán, D., Arora, J., Pachon, J., Konduri, K., Winkenbach, M., Parks, S., Noszek, J.: 2021 amazon last mile routing research challenge: Data set. Transportation Science 58(1), 8–11 (2024)
2024
-
[21]
arXiv preprint arXiv:2502.07010 (2025)
Moradi, A., Tanneau, M., Zandehshahvar, R., Van Hentenryck, P.: Enhanced renewable energy forecasting and operations through probabilistic forecast aggregation. arXiv preprint arXiv:2502.07010 (2025)
2025 arXiv
-
[22]
Advances in neural information processing systems 31 (2018)
Prokhorenkova, L., Gusev, G., V orobev, A., Dorogush, A.V ., Gulin, A.: Catboost: unbiased boosting with cate- gorical features. Advances in neural information processing systems 31 (2018)
2018
-
[23]
European Journal of Operational Research 316(3), 1114–1128 (2024)
Raj, G., Roy, D., De Koster, R., Bansal, V .: Stochastic modeling of integrated order fulfillment processes with delivery time promise: Order picking, batching, and last-mile delivery. European Journal of Operational Research 316(3), 1114–1128 (2024)
2024
-
[24]
In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
Ruan, S., Long, C., Ma, Z., Bao, J., He, T., Li, R., Chen, Y ., Wu, S., Zheng, Y .: Service time prediction for delivery tasks via spatial meta-learning. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. pp. 3829–3837 (2022)
2022
-
[25]
Salari, N., Liu, S., Shen, Z.J.M.: Real-time delivery time forecasting and promising in online retailing: When will your package arrive? Manufacturing & Service Operations Management 24(3), 1421–1436 (2022)
2022
-
[26]
Operations Research 65(6), 1574–1588 (2017)
Shang, Y ., Dunson, D., Song, J.S.: Exploiting big data in logistics risk assessment via bayesian nonparametrics. Operations Research 65(6), 1574–1588 (2017)
2017
-
[27]
Master’s thesis, University of Gothenburg (2022)
Tang, A., Halmkrona Lahtinen, J.: Applying software engineering principles to develop parcel delay forecasting models using tracking data: A study of the models ARIMA, BSTS, and GAM. Master’s thesis, University of Gothenburg (2022)
2022
-
[28]
V ovk, V ., Gammerman, A., Shafer, G.: Algorithmic learning in a random world, vol. 29. Springer (2005)
2005
-
[29]
Advances in Neural Information Processing Systems 28 (2015)
V ovk, V ., Petej, I., Fedorova, V .: Large-scale probabilistic predictors with and without guarantees of validity. Advances in Neural Information Processing Systems 28 (2015)
2015
-
[30]
Neurocomputing 397, 292–308 (2020)
V ovk, V ., Petej, I., Nouretdinov, I., Manokhin, V ., Gammerman, A.: Computationally efficient versions of con- formal predictive distributions. Neurocomputing 397, 292–308 (2020)
2020
-
[31]
IEEE Transactions on Knowledge and Data Engineering (2024)
Wen, H., Lin, Y ., Wu, L., Mao, X., Cai, T., Hou, Y ., Guo, S., Liang, Y ., Jin, G., Zhao, Y ., et al.: A survey on service route and time prediction in instant delivery: Taxonomy, progress, and prospects. IEEE Transactions on Knowledge and Data Engineering (2024)
2024
-
[32]
In: Proceedings of the AAAI conference on artificial intelligence
Wu, F., Wu, L.: Deepeta: a spatial-temporal sequential neural network model for estimating time of arrival in package delivery system. In: Proceedings of the AAAI conference on artificial intelligence. vol. 33, pp. 774–781 (2019)
2019
-
[33]
arXiv preprint arXiv:2409.06918 (2024)
Ye, T., Cheng, S., Hijazi, A., Van Hentenryck, P.: Contextual stochastic optimization for omnichannel multi- courier order fulfillment under delivery time uncertainty. arXiv preprint arXiv:2409.06918 (2024)
2024
-
[34]
In: Proceedings of the 32nd ACM International Conference on Advances in Geographic Information Systems
Yi, J., Yan, H., Wang, H., Yuan, J., Li, Y .: Learning to estimate package delivery time in mixed imbalanced delivery and pickup logistics services. In: Proceedings of the 32nd ACM International Conference on Advances in Geographic Information Systems. pp. 432–443 (2024)
2024
-
[35]
In: Pro- ceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining
Zadrozny, B., Elkan, C.: Transforming classifier scores into accurate multiclass probability estimates. In: Pro- ceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 694–699 (2002)
2002
-
[36]
Applied Energy 384, 125369 (2025)
Zhang, H., Zandehshahvar, R., Tanneau, M., Van Hentenryck, P.: Weather-informed probabilis- tic forecasting and scenario generation in power systems. Applied Energy 384, 125369 (2025). https://doi.org/https://doi.org/10.1016/j.apenergy.2025.125369, https: //www.sciencedirect.c...
2025
-
[37]
Expert Systems with Applications 238, 121740 (2024)
Zhang, L., Wu, X., Liu, Y ., Zhou, X., Cao, Y ., Xu, Y ., Cui, L., Miao, C.: Estimating package arrival time via heterogeneous hypergraph neural network. Expert Systems with Applications 238, 121740 (2024)
2024
-
[38]
In: 2023 IEEE 39th International Conference on Data Engineering (ICDE)
Zhang, L., Zhou, X., Zeng, Z., Cao, Y ., Xu, Y ., Wang, M., Wu, X., Liu, Y ., Cui, L., Shen, Z.: Delivery time pre- diction using large-scale graph structure learning based on quantile regression. In: 2023 IEEE 39th International Conference on Data Engineering (ICDE). pp. 3403...
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.