REVIEW 4 major objections 6 minor 30 references
SVRPBench: A Realistic Benchmark for Stochastic Vehicle Routing Problem
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper introduces SVRPBench, an open benchmark for the stochastic vehicle routing problem with more than 500 instances (up to 1,000 customers each) where travel times vary with time of day, log-normal delays, random accidents, and…
desk verdict The benchmark artifact is real and potentially useful, but the paper's headline claim about RL solvers degrading over 20% is contradicted by its own tables and appendix. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the stochastic travel-time model embedded in the instance generator: $T(a,b,t) = D(a,b)/V + B(a,b,t)\cdot R(t) + I_{\text{accidents}}(t)\cdot D_{\text{accident}}$, where $B$ couples a Gaussian mixture of morning and evening traffic peaks with a distance-decay term, $R(t)$ is a log-normal random delay whose parameters grow at peak hours, and $I_{\text{accidents}}$ follows a time-inhomogeneous Poisson process. This replaces static travel times with a per-instance time-distance matrix on which all solvers are evaluated, paired with a feasibility-validation step that checks capacity and worst-case time-window demand before an instance is released.
What would settle it
A concrete way to test the realism claim is to take actual last-mile delivery trips with recorded travel times and accident events from one or more cities, generate SVRPBench instances with the paper's parameter values, and compare the simulated travel-time distributions and time-window arrival statistics against the real recordings; if the model's distributions differ significantly (e.g., the log-normal underestimates long-tail delays or accident peaks occur at wrong hours), the high-fidelity realism claim collapses and the solver comparison loses external grounding.
Extended reading notes
Core claim
SVRPBench's core claim is that realistic stochastic dynamics change the ranking of routing solvers. The benchmark models the travel time between two nodes as the sum of a base time, a time-dependent congestion term with morning and evening Gaussian peaks, a log-normal multiplicative delay whose parameters swell near peak hours, and delays from accidents drawn from a time-inhomogeneous Poisson process; customer time windows are sampled from residential and commercial Gaussian mixtures. Across 500+ instances, learning-based solvers POMO and the Attention Model lose more than 20% in cost under this distributional shift, while classical nearest-neighbor plus 2-opt, tabu search, ant colony optimization, and the industrial constraint solver remain robust, with the latter two keeping feasibility near 98%. A secondary finding is that time windows are the hardest constraint, inflating cost by 536–648% for every solver, and that multi-depot configurations improve cost and feasibility for all solver types.
Load-bearing premise
The benchmark's realism rests on hand-picked parametric models—Gaussian traffic peaks, log-normal delays, Poisson accidents, and time-window distributions—that are asserted to match urban delivery conditions but are not validated against real route data.
Editorial extensions
If this is right
- RL solvers trained on static synthetic instances should not be expected to keep their reported performance when deployed on urban routes with congestion, delays, and accidents; SVRPBench quantifies the drop at over 20% cost degradation for POMO and the Attention Model.
- Time windows dominate difficulty: introducing them raises total cost by 536–648% across all tested solvers, making schedule tightness the first thing a logistics planner should check.
- Multi-depot routing is not a burden; across the benchmark it improved cost and feasibility for every solver class, including RL models, suggesting flexible depot placement is a design lever.
- Classical low-latency pipelines such as nearest-neighbor with 2-opt remain competitive or superior to learned policies on stochastic instances, meaning simpler methods are a strong baseline for real-time deployments.
Reading between the lines
- If the reported degradation is driven by the specific noise distributions and time windows in SVRPBench, a natural next test is to retrain POMO and the Attention Model on instances sampled from the benchmark's own generator; the paper does not do this, so the 20% figure likely mixes overfitting to a synthetic training distribution with genuine sensitivity to stochasticity.
- The benchmark's realism claim could be sharpened into a quantitative test: compare its travel-time and time-window distributions to logged delivery data from a real fleet, or run a paired field experiment; absent that, SVRPBench is best read as a controlled stress test rather than a validated emulator of any specific city.
- Because the generator is modular and parameterized, one could create targeted sub-benchmarks that isolate one stochastic component (e.g., accidents only, or congestion only) to attribute solver degradation to a specific source of uncertainty; the current aggregate comparisons do not separate these effects.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SVRPBench, a benchmark suite for stochastic vehicle routing that combines time-dependent congestion, log-normal travel-time delays, Poisson-distributed accidents, and customer-specific time windows into 500+ generated instances with up to 1000 customers. It evaluates classical heuristics (NN+2-opt), metaheuristics (ACO, Tabu Search), OR-Tools, and learning-based solvers (AM, POMO), and releases the dataset, generator, and evaluation code. The abstract and conclusion claim that POMO and AM degrade by over 20% under distributional shift while classical methods remain robust, and the paper positions SVRPBench as the first open benchmark with high-fidelity stochastic dynamics at urban scale. The benchmark infrastructure is potentially useful, but the central empirical claim is not supported by the presented data and is internally contradicted by the paper's own appendix and limitations section.
Significance. If the distribution-shift result were established, SVRPBench would provide a valuable community resource and an important cautionary finding for learning-based VRP solvers. The paper does ship substantial open infrastructure: a Hugging Face dataset, a public GitHub repository, an evaluation suite, and a set of baseline implementations, which is a genuine strength. However, the headline finding is neither defined nor measured, it is contradicted by the aggregate results in Table 2 and by Appendix D.4, and the realism of the stochastic models is asserted rather than validated. As presented, the paper's main scientific message cannot be relied upon, although the released resource could form the basis of a substantially revised contribution.
major comments (4)
- [Abstract; §5.3 (Table 2); §7] The central claim that POMO and AM degrade by over 20% under distributional shift is not defined, measured, or supported anywhere in the manuscript. The aggregate results in Table 2 show POMO at 40,650.4 versus OR-Tools at 40,259.3 (+1.0%) and AM at 41,358.3 (+2.7%); no per-instance or per-configuration comparison against the solvers' training distribution is reported, and no formal definition of 'distributional shift' is given. Since this claim appears in the abstract and conclusion and motivates the paper, it is load-bearing and unsupported.
- [Appendix D.4; Tables 10-11 vs. Table 2] The detailed RL results contradict the aggregate table. Appendix D.4 states that 'Despite domain shift, attention-based RL policies maintained high feasibility and near-optimal costs across all problem sizes,' and Tables 10 and 11 report feasibility 1.000, CVR 0.00, and zero time-window violations for every configuration and size. Table 2, by contrast, reports POMO feasibility 0.933, AM feasibility 0.910, and CVR 1.7-1.9%. These two sets of numbers cannot both describe the same evaluation, and the discrepancy is never explained.
- [Section 2 (Eqs. 1-14)] The realism of the stochastic models is asserted rather than validated. Equations (1)-(11) set parameters such as sigma_peak=1.5, lambda_dist=50, mu_base=0, sigma_base=0.3, delta=0.1, epsilon=0.2, and the accident intensity from literature citations, while the time-window distributions in Eqs. (12)-(14) are said to follow industry white papers; however, no empirical fit, sensitivity analysis, or comparison to real traffic or delivery data is provided. Because the benchmark's 'high-fidelity' realism is a central contribution, this missing validation is a load-bearing gap.
- [Section 6 (Limitations)] Section 6 explicitly concedes that 'our current evaluation protocol lacks standardized procedures to assess robustness across instance scales and distribution shifts.' This directly contradicts the abstract and conclusion, which state that benchmarking reveals over 20% degradation under distributional shift. The narrative of the paper is therefore internally inconsistent on its main empirical finding.
minor comments (6)
- [Tables 6-9] The configuration column contains the typo 'vehicule'; the correct spelling is 'vehicle'.
- [Table 3; Figure 4] The abbreviation 'TWCVRP' is used in Table 3 and Figure 4, while 'TWVRP' is used elsewhere; please standardize the nomenclature.
- [Tables 6-11] Several tables do not explicitly state units for runtime; Table 2 reports seconds, Tables 10-11 report milliseconds, and Tables 6-9 give runtimes that appear to be in different units without a header note.
- [Tables 6-9] The configuration label 'single depot single vehicule sumDemands' is unexplained; the reader cannot determine how vehicle count and capacity are set in those runs.
- [Figure 2] The claim that synthetic layouts 'closely mimic' real cities is based on visual inspection only; a quantitative spatial-comparison metric would be needed to support this statement.
- [References] References [11] and [13] are the same article, as are [22] and [23], resulting in duplicate citations.
Circularity Check
No circular derivation: benchmark parameters come from external literature, and the unsupported >20% RL-degradation claim is an internal-consistency problem, not a circular one.
full rationale
SVRPBench is a benchmark-construction paper rather than a derivation, so there is no chain in which a predicted quantity is defined in terms of the quantity it purports to explain. The stochastic models in Eqs. (1)-(14) use externally cited traffic, delay, and time-window literature (e.g., [3,6,10,19,20,23,25,27,28,29]), and the parameters (morning/evening peaks, log-normal baselines, Poisson accident rates, time-window mixtures) are asserted from those references rather than fitted to the benchmark's own outputs; the reported evaluation numbers are therefore not forced by construction. The central claim that RL solvers 'degrade by over 20% under distributional shift' is not supported by Table 2 (POMO is +1.0% and AM +2.7% versus OR-Tools), Appendix D.4 states the opposite ('Despite domain shift, attention-based RL policies maintained high feasibility and near-optimal costs across all problem sizes'), and Section 6 concedes 'our current evaluation protocol lacks standardized procedures to assess robustness across instance scales and distribution shifts.' That is an evidentiary and consistency failure, not a circular argument. The only author self-citation, [21] (co-author Takáč), appears in a future-work sentence about diagnostic tasks and is not load-bearing for the benchmark's construction or conclusions. Thus no circular step can be exhibited; the score of 2 reflects the minor non-load-bearing self-citation and no deeper circularity.
Assumptions & free parameters
free parameters (10)
- V (average road speed) =
not specified
- alpha, beta, gamma (congestion coefficients) =
not specified
- sigma_peak =
1.5
- lambda_dist =
50
- mu_base, sigma_base, delta, epsilon =
0, 0.3, 0.1, 0.2
- accident intensity lambda_scale =
not specified
- mu_night, sigma_acc =
21, 2
- d_min, d_max accident delay =
0.5, 2.0 hours
- residential and commercial time window parameters =
mu_res_morning=480, mu_res_evening=1140, sigma_res=90/120, mu_com=780, sigma_com=60, w_com_max=120; w_min and w_max…
- customer type split =
60% residential, 40% commercial
assumptions (5)
- domain assumption Euclidean distance divided by average speed approximates real travel time.
- domain assumption Gaussian mixtures and log-normal distributions adequately capture urban traffic dynamics.
- domain assumption Industry white papers provide representative residential and commercial time windows.
- domain assumption The instance validation procedure guarantees a feasible schedule exists.
- domain assumption Baseline solver implementations are correct and unbiased.
Cite this review
Pith. "Pith review of SVRPBench: A Realistic Benchmark for Stochastic Vehicle Routing Problem." pith.science (2026). https://pith.science/paper/JCXHJAZ5
@misc{pith2026250521887,
author = {Pith},
title = {Pith review of: SVRPBench: A Realistic Benchmark for Stochastic Vehicle Routing Problem},
year = {2026},
howpublished = {\url{https://pith.science/paper/JCXHJAZ5}},
note = {Machine review of arXiv:2505.21887}
}
read the original abstract
Robust routing under uncertainty is central to real-world logistics, yet most benchmarks assume static, idealized settings. We present SVRPBench, the first open benchmark to capture high-fidelity stochastic dynamics in vehicle routing at urban scale. Spanning more than 500 instances with up to 1000 customers, it simulates realistic delivery conditions: time-dependent congestion, log-normal delays, probabilistic accidents, and empirically grounded time windows for residential and commercial clients. Our pipeline generates diverse, constraint-rich scenarios, including multi-depot and multi-vehicle setups. Benchmarking reveals that state-of-the-art RL solvers like POMO and AM degrade by over 20% under distributional shift, while classical and metaheuristic methods remain robust. To enable reproducible research, we release the dataset and evaluation suite. SVRPBench challenges the community to design solvers that generalize beyond synthetic assumptions and adapt to real-world uncertainty.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Models and algorithms for stochastic and robust vehicle routing with deadlines
Yossiri Adulyasak and Patrick Jaillet. Models and algorithms for stochastic and robust vehicle routing with deadlines. Transportation Science, 50(2):608–626, 2016
work page 2016
-
[2]
Cock Bastian and Alexander H. G. Rinnooy Kan. The stochastic vehicle routing problem revisited. European Journal of Operational Research, 56(3):407–412, 1992
work page 1992
-
[3]
Scenario-based planning for partially dynamic vehicle routing with stochastic customers
Russell Bent and Pascal Van Hentenryck. Scenario-based planning for partially dynamic vehicle routing with stochastic customers. Operations Research, 52(6):977–987, 2004
work page 2004
- [4]
-
[5]
Bertsimas, Patrick Jaillet, and Amedeo R
Dimitris J. Bertsimas, Patrick Jaillet, and Amedeo R. Odoni. A priori optimization. Operations Research, 38(6):1019–1033, 1990
work page 1990
-
[6]
Reliability of travel times: A stochastic modeling approach
Werner Brilon, Jürgen Geistefeldt, and Markus Regler. Reliability of travel times: A stochastic modeling approach. Transportation Research Record, 2061(1):1–8, 2008
work page 2008
-
[7]
K. Chepuri and T. Homem-de Mello. Solving the vehicle routing problem with stochastic demands using the cross-entropy method. Annals of Operations Research, 134(1):153–181, 2005
work page 2005
-
[8]
George B Dantzig and John H Ramser. The truck dispatching problem. Management science, 6(1):80–91, 1959
work page 1959
Show all 30 references
-
[9]
Vehicle routing with stochastic demands: Properties and solution frameworks
Moshe Dror, Gilbert Laporte, and Pierre Trudeau. Vehicle routing with stochastic demands: Properties and solution frameworks. Transportation Science, 23(3):166–176, 1989
1989
-
[10]
Fedex residential delivery options whitepaper
FedEx Corporation. Fedex residential delivery options whitepaper. Whitepaper, 2020. Flexible delivery time window practices
2020
-
[11]
Stochastic vehicle routing
Michel Gendreau, Gilbert Laporte, and Renaud Séguin. Stochastic vehicle routing. European Journal of Operational Research, 88(1):3–12, 1996
1996
-
[12]
A tabu search heuristic for the vehicle routing problem with stochastic demands and customers
Michel Gendreau, Gilbert Laporte, and Renaud Séguin. A tabu search heuristic for the vehicle routing problem with stochastic demands and customers. Operations Research, 44(3):469–477, 1996
1996
-
[13]
Stochastic vehicle routing
Michel Gendreau, Gilbert Laporte, and Rene Seguin. Stochastic vehicle routing. European Journal of Operational Research, 88(1):3–12, 1996
1996
-
[14]
Solving a dynamic and stochastic vehicle routing problem with a sample scenario hedging heuristic
Lars Magnus Hvattum, Arne Lø kketangen, and Gilbert Laporte. Solving a dynamic and stochastic vehicle routing problem with a sample scenario hedging heuristic. Transportation Science, 40(4):421–438, 2006
2006
-
[15]
Attention, learn to solve routing problems! arXiv preprint arXiv:1803.08475, 2018
Wouter Kool, Herke Van Hoof, and Max Welling. Attention, learn to solve routing problems! arXiv preprint arXiv:1803.08475, 2018
2018 arXiv
-
[16]
Attention, learn to solve routing problems! International Conference on Learning Representations (ICLR), 2019
Wouter Kool, Herke van Hoof, and Max Welling. Attention, learn to solve routing problems! International Conference on Learning Representations (ICLR), 2019
2019
-
[17]
Pomo: Policy optimization with multiple optima for reinforcement learning
Yeong-Dae Kwon, Jinho Choo, Byoungjip Kim, Iljoo Yoon, Youngjune Gwon, and Seungjai Min. Pomo: Policy optimization with multiple optima for reinforcement learning. Advances in Neural Information Processing Systems, 33:21188–21198, 2020
2020
-
[18]
Louveaux, and Hélène Mercure
Gilbert Laporte, François V . Louveaux, and Hélène Mercure. The vehicle routing problem with stochastic travel times. Transportation Science, 26(3):161–170, 1992
1992
-
[19]
Modeling travel time variability with lognormal distribu- tion
Qing Li, Ming Xu, and Yinhai Wang. Modeling travel time variability with lognormal distribu- tion. Transportation Research Record, 2490(1):47–54, 2015
2015
-
[20]
Customer preferences in last-mile deliveries: Flexible windows and urban density effects
Bring Logistics. Customer preferences in last-mile deliveries: Flexible windows and urban density effects. Industry Report, 2021. Available via company white papers. 11
2021
-
[21]
Reinforcement learning for solving the vehicle routing problem
Mohammadreza Nazari, Afshin Oroojlooy, Lawrence Snyder, and Martin Takáˇc. Reinforcement learning for solving the vehicle routing problem. In Proceedings of Advances in Neural Information Processing Systems, pages 9861–9871, 2018
2018
-
[22]
Woodruff
Jorge Oyola, Halvard Arntzen, and David L. Woodruff. The stochastic vehicle routing problem, a literature review, part i: Models. EURO Journal on Transportation and Logistics, 7(3):193–221, 2018
2018
-
[23]
The stochastic vehicle routing problem: A literature review, part i: Models
Jorge Luis Oyola, Halvard Arntzen, and David L Woodruff. The stochastic vehicle routing problem: A literature review, part i: Models. EURO Journal on Transportation and Logistics, 7(3):193–221, 2018
2018
-
[25]
Adaptive memory procedure for solving real-world vehicle routing problem
Nikica Peri ´c, Slaven Begovi ´c, and Vinko Lesi ´c. Adaptive memory procedure for solving real-world vehicle routing problem. arXiv preprint arXiv:2403.04420, 2024
2024 arXiv
-
[26]
Laurent Perron and Frédéric Didier. Cp-sat
-
[27]
2021 urban mobility report
David Schrank, Bill Eisele, Tim Lomax, et al. 2021 urban mobility report. Texas A&M Transportation Institute, 2021
2021
-
[28]
Department of Transportation
Federal Highway Administration U.S. Department of Transportation. Manual on uniform traffic control devices (mutcd), 2009 edition, 2009. Accident and incident classification and duration guidelines
2009
-
[29]
Attended home deliveries: Preferences and behavioral patterns
Ron van Duin, Tolga Bekta¸ s, Murat Bekta¸ s, and Tavares Tan. Attended home deliveries: Preferences and behavioral patterns. Transportation Research Procedia, 16:30–39, 2016
2016
-
[30]
Simple statistical gradient-following algorithms for connectionist reinforce- ment learning
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3):229–256, 1992
1992
-
[31]
Reinforced lin– kernighan–helsgaun algorithms for the traveling salesman problems
Jiongzhi Zheng, Kun He, Jianrong Zhou, Yan Jin, and Chu-Min Li. Reinforced lin– kernighan–helsgaun algorithms for the traveling salesman problems. Knowledge-Based Systems, 260:110144, 2023. A Open Infrastructure To ensure reproducibility, extensibility, and accessibility, we r...
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.