REVIEW 3 major objections 4 minor 21 references
Learning to Search for Vehicle Routing with Multiple Time Windows
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A learned operator-selection policy beats adaptive VNS on multi-window routing.
desk verdict Sensible RL+metaheuristic extension to a realistic VRP variant, but the runtime savings claim is not credible until baselines are run under matched budgets. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is RL-AVNS: a PPO-trained transformer policy network that observes an encoded solution state, including node positions, time-window constraints, arrival times, gaps to the best and last solution, and an improvement indicator, and outputs a probability distribution over a discrete set of intra-route and inter-route operators such as 2-Opt, 2-Opt*, Swap, and Relocate. At each of 2,000 iterations the policy samples one operator and applies local search; a reward of $[f(x_t)-f(x_{t+1})]-100 t_{\text{run}}$, clipped to $[-10,10]$, makes the agent prefer both shorter routes and faster operators. The fitness metric in Equation (12) quantifies a customer's temporal flexibility as the distance from the arrival time to the nearest time-window boundary, and the shaking phase removes and reinserts the 20% most flexible customers, keeping search near high-quality structure.
What would settle it
Run baseline VNS and AVNS on the same 100-instance test sets with a hard 2,000-iteration cap and with a matched wall-clock budget equal to RL-AVNS's reported solve time, then compare route lengths; if AVNS reaches or beats RL-AVNS under either budget, the paper's central efficiency and quality claims would fail.
Extended reading notes
Core claim
The paper claims that replacing the history-based operator weights of adaptive VNS with a transformer-based policy trained by PPO yields a search that finds shorter routes faster. On VRPMTW instances with two, three, or mixed time windows at 50 and 100 customers, RL-AVNS reduces total route length by 3.3-14.5% and solving time by 65-77% relative to AVNS, while also reducing travel duration and the number of vehicles used. On single-window VRPTW it shortens routes by 6-9% and runs three to five times faster than AVNS. The authors further claim the policy generalizes: it is trained only on 50-customer instances with three time windows and then applied, without retraining, to all other instance sizes and window configurations.
Load-bearing premise
The claimed gains rest on the assumption that VNS and AVNS were stopped under rules comparable to RL-AVNS's fixed 2,000-iteration budget; if those baselines were simply given much more search time, the reported quality and speed advantages could shrink.
Editorial extensions
If this is right
- On the paper's vending-machine replenishment instances, a single policy trained on 50-customer, three-window problems transfers to 100-customer, two-window, three-window, and mixed-window instances without retraining.
- Because the reward penalizes runtime, the learned policy tends to avoid slow-but-strong operators, which explains the reported 65-77% time savings and ties solution quality to operator speed.
- The fitness-based shaking operator reduces idle waiting at customers, yielding shorter durations and fewer vehicles in the reported experiments.
- RL-AVNS outperforms the neural constructive solvers AM and JAMPR on single-window VRPTW, suggesting the search-based hybrid is stronger than end-to-end construction for time-constrained routing.
Reading between the lines
- Not reported in the paper: a symmetric generalization test. The authors train on 50-customer, three-window instances and test on larger and mixed settings; training on the harder 100-customer mixed-window distribution and testing on smaller, simpler instances could reveal whether the policy's advantage is tied to training difficulty.
- The runtime term in the reward makes 'fast operator' part of the objective, so a user who cares only about route length could retrain with a smaller runtime penalty and likely trade some speed for lower length; the paper does not map this tradeoff.
- The same RL-operator-selection wrapper could plausibly be applied to other VRP variants with hard timing constraints, such as long-haul routing with driver rest periods, since the policy only needs the state features and operator set to change.
- A direct robustness check not reported in the paper: run AVNS under the same 2,000-iteration cap as RL-AVNS and compare both quality and time; the tables report only wall-clock times for the baselines, so iteration-matched comparisons would clarify how much of the gain comes from the learned policy versus the stopping rule.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes RL-AVNS, a hybrid method that trains a transformer-based PPO policy to select neighborhood operators inside an adaptive variable neighborhood search for the Vehicle Routing Problem with Multiple Time Windows (VRPMTW). The method also introduces a fitness metric that quantifies customers' temporal flexibility for use in the shaking phase. Experiments on VRPTW and VRPMTW instances with N=50 and N=100 compare RL-AVNS against VNS, RVNS, AVNS, OR-Tools, LKH3, AM, and JAMPR, reporting route-length reductions of 3.3–14.5% over AVNS and runtime reductions of 65–77%, as well as generalization from training on N=50 3-TW instances to other settings.
Significance. If the reported gains are reproducible under a controlled comparison protocol, the paper would make a useful contribution: it is the first deep-RL solver for VRPMTW, uses a realistic vending-machine replenishment setting, and demonstrates an interesting integration of a learned policy with VNS. The generalization test (training on N=50 3-TW and testing on N=100 and 2-TW/Mix-TW) is a valuable strength. However, the evidence as presented is not yet convincing: the efficiency comparisons lack a specified baseline termination protocol, the quality comparisons lack statistical support, and the fitness-metric ablation rests on a single illustrative instance. No code or data is provided, which further limits reproducibility.
major comments (3)
- [§5.1, Tables 5–6] The experimental protocol for the efficiency comparison is underspecified and the headline claim depends on it. Section 5.1 states that RL-AVNS runs for a fixed 2,000 iterations, but no termination condition is given for VNS, RVNS, or AVNS; consequently the reported runtimes (and the 'Gain' rows in Tables 5 and 6) compare against baselines whose compute budgets are unknowable. Moreover, the reward in Section 4.4 includes the term -100·t_run, so the learned policy is explicitly rewarded for choosing fast operators; an efficiency advantage over a quality-only selector is therefore an expected consequence of the reward design, not a controlled comparison. Please specify the baseline stopping rules and rerun the comparisons under (i) identical iteration counts and (ii) identical wall-clock time budgets, reporting both settings; this is required before the '65–77% runtime reduction' claim can be evaluated.
- [§5.3, Tables 5–6] All numerical results are averages over 100 instances, but the paper reports no standard deviations, confidence intervals, or significance tests. The abstract and Section 5 repeatedly state that RL-AVNS 'significantly outperforms' baselines; without variance information, the reported gains (e.g., 3.3–14.5% length reductions in Table 6) cannot be distinguished from noise. Please report per-method standard deviations and perform pairwise significance tests (e.g., Wilcoxon signed-rank) across the 100 instances for length, duration, K, and solve time.
- [§5.3, Figure 5] The fitness-metric ablation is based on a single illustrative instance. The two panels of Figure 5 show one routing solution with and without the fitness-based shaking, and the text concludes that the metric 'significantly enhances solution quality.' A single example cannot support a general claim about the metric's contribution. Please add an aggregate comparison (with and without the fitness metric) over the same 100-instance benchmark, reporting means and variances for length, duration, and K.
minor comments (4)
- [§5.3] There are several typos: 'RL-VANS' should be 'RL-AVNS'; 'applly' and 'preceeding' appear in Section 5.1; 'funtion' appears in the hyperparameter paragraph; the Figure 3 caption uses 'architechture'.
- [§4.4] The reward formula mixes route-length units with seconds via the hand-selected coefficient 100; please state the units or normalize the runtime term, and report a sensitivity analysis over the coefficient.
- [Figure 4] The convergence plot shows a single Mix-TW N=100 instance; adding error bands over multiple instances would make the convergence-speed claim more convincing.
- [§5.1] The paper does not specify the implementation language or version for the baseline VNS/RVNS/AVNS methods, which is important for interpreting the reported runtime comparisons.
Circularity Check
The reported computational-efficiency gain is partly circular because runtime is an explicit term in the training reward, so the speed-up over AVNS largely verifies the optimization objective rather than an independent discovery.
-
self definitional
[Section 4.4 (Rewards), interpreted in Section 5.3 / Tables 5-6]
"The reward mechanism is mathematically formulated as: r = [f(x_t) - f(x_{t+1})] - 100 * t_run, where t_run denotes the computational runtime. ... the runtime penalty term introduces the computational efficiency as an explicit optimization criterion, encouraging the algorithm to discover not only high-quality but also computationally tractable solution pathways."
RL-AVNS is trained to maximize this reward, so runtime is an input objective, not an emergent result. The paper then reports 65-77% reductions in solving time over AVNS (Table 6) as evidence of superiority, while AVNS's operator weights do not include runtime. Since both methods share the same operator set and greedy initial solution, the observed speed advantage is the expected outcome of optimizing a hand-tuned runtime-penalized reward; it is not an independent prediction. The solution-quality gains are more independent, so the circularity is partial rather than total.
full rationale
The central derivation chain of RL-AVNS is not circular with respect to solution quality: the policy is tested on unseen instances and the reported length improvements could have failed to materialize. The circularity is confined to the efficiency claim. Equation (13) in Section 4.4 defines runtime as part of the training objective via -100*t_run, and Tables 5-6 present the resulting speed advantage as a discovered improvement over a baseline that does not optimize runtime. That is essentially verifying the training objective. No load-bearing self-citation was found; the authors' prior works (Wu et al. 2022, Xin et al. 2021) appear only as related iterative-solver literature, not as proofs or uniqueness theorems. The unspecified termination criteria for VNS/RVNS/AVNS are an experimental-fairness concern, not circularity. Overall score 6 reflects one central claim that reduces, in part, to the training signal, while the quality gains retain independent empirical content.
Assumptions & free parameters
free parameters (4)
- reward_runtime_penalty_coefficient =
100
- reward_clip_bound =
10
- shaking_disruption_ratio =
0.20
- discount_factor_gamma_and_gae_lambda =
not reported
assumptions (4)
- standard math Big-M constraint (7) correctly models time propagation and subtour elimination.
- domain assumption The synthetic instance generator reflects real-world vending machine time-window patterns.
- domain assumption A policy trained only on N=50, 3-TW instances generalizes to N=100 and to 2-TW and Mix-TW settings.
- ad hoc to paper The fitness metric in Equation (12) identifies customers whose movement improves search.
Cite this review
Pith. "Pith review of Learning to Search for Vehicle Routing with Multiple Time Windows." pith.science (2026). https://pith.science/paper/IFEVGKAF
@misc{pith2026250523098,
author = {Pith},
title = {Pith review of: Learning to Search for Vehicle Routing with Multiple Time Windows},
year = {2026},
howpublished = {\url{https://pith.science/paper/IFEVGKAF}},
note = {Machine review of arXiv:2505.23098}
}
read the original abstract
In this study, we propose a reinforcement learning-based adaptive variable neighborhood search (RL-AVNS) method designed for effectively solving the Vehicle Routing Problem with Multiple Time Windows (VRPMTW). Unlike traditional adaptive approaches that rely solely on historical operator performance, our method integrates a reinforcement learning framework to dynamically select neighborhood operators based on real-time solution states and learned experience. We introduce a fitness metric that quantifies customers' temporal flexibility to improve the shaking phase, and employ a transformer-based neural policy network to intelligently guide operator selection during the local search. Extensive computational experiments are conducted on realistic scenarios derived from the replenishment of unmanned vending machines, characterized by multiple clustered replenishment windows. Results demonstrate that RL-AVNS significantly outperforms traditional variable neighborhood search (VNS), adaptive VNS (AVNS), and state-of-the-art learning-based heuristics, achieving substantial improvements in solution quality and computational efficiency across various instance scales and time window complexities. Particularly notable is the algorithm's capability to generalize effectively to problem instances not encountered during training, underscoring its practical utility for complex logistics scenarios.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Thevehicleroutingproblemwithmultipleprioritizedtimewindows:Acasestudy
Beheshti,A.K.,Hejazi,S.R.,Alinaghian,M.,2015. Thevehicleroutingproblemwithmultipleprioritizedtimewindows:Acasestudy. Computers & Industrial Engineering 90, 402–413. Belhaiza, S., Hansen, P., Laporte, G.,
work page 2015
-
[6]
arXiv preprint arXiv:2006.09100
Learning to solve vehicle routing problems with time windows through joint attention. arXiv preprint arXiv:2006.09100 . Favaretto, D., Moretti, E., Pellegrini, P.,
arXiv 2006
-
[8]
arXiv preprint arXiv:2002.08539
Learn to design the heuristics for vehicle routing problem. arXiv preprint arXiv:2002.08539 . Gendreau, M., Potvin, J.Y. (Eds.),
arXiv 2002
-
[11]
Transportation Science 54, 400–416
Efficient neighborhood evaluations for the vehicle routing problem with multiple time windows. Transportation Science 54, 400–416. Jumper,J.,Evans,R.,Pritzel,A.,Green,T.,Figurnov,M.,Ronneberger,O.,Tunyasuvunakool,K.,Bates,R.,Žídek,A.,Potapenko,A.,etal.,2021. Highly accurate protein structure prediction with alphafold. nature 596, 583–589. Karakostas, P., ...
work page 2021
-
[12]
Expert Systems with Applications 153, 113444
Adaptive variable neighborhood search solution methods for the fleet size and mix pollution location-inventory-routing problem. Expert Systems with Applications 153, 113444. doi:10.1016/j.eswa.2020.113444. First Author et al.:Preprint submitted to Elsevier Page 22 of 25 Learning to search for VRPMTW Karimi-Mamaghan,M.,Mohammadi,M.,Meyer,P.,Karimi-Mamaghan...
-
[15]
On the fluid slip along a solid surface
Edge-direct: A deep reinforcement learning-based method for solving heterogeneous electric vehicle routing problem with time window constraints. ArXiv preprint arXiv:2302.06789 . Nazari, M., Oroojlooy, A., Snyder, L., Takác, M.,
-
[19]
Calibrating 15 years of GOLF data
An adaptive variable neighborhood search algorithm for a vehicle routing problem arising in small package shipping. Transportation Science 47, 64–80. doi:10.1287/trsc.1110.0396. Tang, Q., Kong, Y., Pan, L., Lee, C.,
-
[21]
Society for Industrial and Applied Mathematics
Vehicle routing: Problems, methods, and applications, second edition. Society for Industrial and Applied Mathematics . Wang,Q.,Tang,C.,2021.Deepreinforcementlearningfortransportationnetworkcombinatorialoptimization:Asurvey. Knowledge-BasedSystems 233, 107526. Wei, L., Zhang, Z., Zhang, D., Lim, A.,
work page 2021
Show all 21 references
-
[2001]
European journal of operational research 130, 449–467
Variable neighborhood search: Principles and applications. European journal of operational research 130, 449–467. Helsgaun,K.,2017.Anextensionofthelin-kernighan-helsgauntspsolverforconstrainedtravelingsalesmanandvehicleroutingproblems.Roskilde: Roskilde University 12, 966–980....
2017
-
[2012]
EURO Journal on Transportation and Logistics 1, 285–306
The minimum duration truck driver scheduling problem. EURO Journal on Transportation and Logistics 1, 285–306. Groër,C.,Golden,B.,Wasil,E.,2010. Alibraryoflocalsearchheuristicsforthevehicleroutingproblem. MathematicalProgrammingComputation 2, 79–101. Hansen, P., Mladenović, N.,
2010
-
[2013]
Transportation Science 47, 81–107
Long-haul vehicle routing and scheduling with working hour rules. Transportation Science 47, 81–107. Schaap,H.,Schiffer,M.,Schneider,M.,Walther,G.,2022. Alargeneighborhoodsearchforthevehicleroutingproblemwithmultipletimewindows. Transportation Science 56, 1369–1392. Schulman, ...
2022
-
[2014]
Computers & Operations Research 52, 269–281
A hybrid variable neighborhood tabu search heuristic for the vehicle routing problem with multiple time windows. Computers & Operations Research 52, 269–281. Belhaiza,S.,M’Hallah,R.,BenBrahim,G.,Laporte,G.,2019. Threemulti-startdata-drivenevolutionaryheuristicsforthevehiclerou...
2019 doi
-
[2015]
arXiv preprint arXiv:1506.02438
High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 . Schulman,J.,Wolski,F.,Dhariwal,P.,Radford,A.,Klimov,O.,2017. Proximalpolicyoptimizationalgorithms. arXivpreprintarXiv:1707.06347. First Author et al.:Preprint submitte...
2017 arXiv
-
[2017]
nature 550, 354–359
Mastering the game of go without human knowledge. nature 550, 354–359. Solomon,M.M.,1987.Algorithmsforthevehicleroutingandschedulingproblemswithtimewindowconstraints.OperationsResearch35,254–265. Stenger, A., Vigo, D., Enz, S., Schwind, M.,
1987
-
[2018]
Electronic Notes in Discrete Mathematics 66, 207–214
Variable neighborhood search for vehicle routing problem with multiple time windows. Electronic Notes in Discrete Mathematics 66, 207–214. doi:10.1016/j.endm.2018.03.027. Fisher, M.L.,
2018 doi
-
[2019]
Fast delta evaluation for the vehicle routing problem with multiple time windows. Lin,B.,Ghaddar,B.,Nathwani,J.,2022.Deepreinforcementlearningfortheelectricvehicleroutingproblemwithtimewindows.IEEETransactions on Intelligent Transportation Systems 23, 11528–11538. Lu,H.,Zhang,...
2022
-
[2020]
arXiv preprint arXiv:2005.09330
Dynamic partial removal: A neural network heuristic for large neighborhood search. arXiv preprint arXiv:2005.09330 . Clarke, G., Wright, J.W.,
2005 arXiv
-
[2021]
Computers & Operations Research 134, 105400
Reinforcement learning for combinatorial optimization: A survey. Computers & Operations Research 134, 105400. Morabit,M.,Desaulniers,G.,Lodi,A.,2021.Machine-learning–basedcolumnselectionforcolumngeneration.TransportationScience55,815–831. Mozhdehi, A., Mohammadizadeh, M.,
2021
-
[2022]
arXiv preprint arXiv:2207.09860
Learning to solve soft-constrained vehicle routing problems with lagrangian relaxation. arXiv preprint arXiv:2207.09860 . Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.,
-
[2023]
Computers & Operations Research 149, 106016
A variable neighborhood search-based algorithm with adaptive local search for the vehicle routing problem with time windows and multi-depots aiming for vehicle fleet reduction. Computers & Operations Research 149, 106016. doi:10.1016/j.cor.2022.106016. Bogue, E.T., Ferreira, H...
-
[2024]
arXiv preprint arXiv:2403.05318
Looking ahead to avoid being late: Solving hard-constrained traveling salesman problem. arXiv preprint arXiv:2403.05318 . Chen, M., Gao, L., Chen, Q., Liu, Z.,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.