Pith. sign in

REVIEW 3 major objections 4 minor 21 references

Learning to Search for Vehicle Routing with Multiple Time Windows

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read A learned operator-selection policy beats adaptive VNS on multi-window routing.

desk verdict Sensible RL+metaheuristic extension to a realistic VRP variant, but the runtime savings claim is not credible until baselines are run under matched budgets. read the letter →

arxiv 2505.23098 v1 pith:IFEVGKAF submitted 2025-05-29 cs.LG

classification cs.LG
keywords VehicleroutingwithmultipletimewindowsVariableneighborhoodsearchReinforcementlearningNeuraloperatorselectionPPOTransformerpolicynetworkUnmannedvendingmachinereplenishmentFitnessmetric
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a reinforcement-learning agent can choose which neighborhood operators a variable neighborhood search applies, and that this learned choice beats both fixed-order VNS and adaptive VNS that weighs operators only by past performance. The target problem is vehicle routing with multiple time windows, a hard real-world case because each customer can be served in any of several windows. If the claim holds, logistics planners can get better routes in less compute time for the vending-machine replenishment scenario and possibly for similar multi-window delivery settings, without retraining for each instance size or window configuration.

What carries the argument

The central machinery is RL-AVNS: a PPO-trained transformer policy network that observes an encoded solution state, including node positions, time-window constraints, arrival times, gaps to the best and last solution, and an improvement indicator, and outputs a probability distribution over a discrete set of intra-route and inter-route operators such as 2-Opt, 2-Opt*, Swap, and Relocate. At each of 2,000 iterations the policy samples one operator and applies local search; a reward of $[f(x_t)-f(x_{t+1})]-100 t_{\text{run}}$, clipped to $[-10,10]$, makes the agent prefer both shorter routes and faster operators. The fitness metric in Equation (12) quantifies a customer's temporal flexibility as the distance from the arrival time to the nearest time-window boundary, and the shaking phase removes and reinserts the 20% most flexible customers, keeping search near high-quality structure.

What would settle it

Run baseline VNS and AVNS on the same 100-instance test sets with a hard 2,000-iteration cap and with a matched wall-clock budget equal to RL-AVNS's reported solve time, then compare route lengths; if AVNS reaches or beats RL-AVNS under either budget, the paper's central efficiency and quality claims would fail.

Watch

Extended reading notes

Core claim

The paper claims that replacing the history-based operator weights of adaptive VNS with a transformer-based policy trained by PPO yields a search that finds shorter routes faster. On VRPMTW instances with two, three, or mixed time windows at 50 and 100 customers, RL-AVNS reduces total route length by 3.3-14.5% and solving time by 65-77% relative to AVNS, while also reducing travel duration and the number of vehicles used. On single-window VRPTW it shortens routes by 6-9% and runs three to five times faster than AVNS. The authors further claim the policy generalizes: it is trained only on 50-customer instances with three time windows and then applied, without retraining, to all other instance sizes and window configurations.

Load-bearing premise

The claimed gains rest on the assumption that VNS and AVNS were stopped under rules comparable to RL-AVNS's fixed 2,000-iteration budget; if those baselines were simply given much more search time, the reported quality and speed advantages could shrink.

Editorial extensions

If this is right

  • On the paper's vending-machine replenishment instances, a single policy trained on 50-customer, three-window problems transfers to 100-customer, two-window, three-window, and mixed-window instances without retraining.
  • Because the reward penalizes runtime, the learned policy tends to avoid slow-but-strong operators, which explains the reported 65-77% time savings and ties solution quality to operator speed.
  • The fitness-based shaking operator reduces idle waiting at customers, yielding shorter durations and fewer vehicles in the reported experiments.
  • RL-AVNS outperforms the neural constructive solvers AM and JAMPR on single-window VRPTW, suggesting the search-based hybrid is stronger than end-to-end construction for time-constrained routing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not reported in the paper: a symmetric generalization test. The authors train on 50-customer, three-window instances and test on larger and mixed settings; training on the harder 100-customer mixed-window distribution and testing on smaller, simpler instances could reveal whether the policy's advantage is tied to training difficulty.
  • The runtime term in the reward makes 'fast operator' part of the objective, so a user who cares only about route length could retrain with a smaller runtime penalty and likely trade some speed for lower length; the paper does not map this tradeoff.
  • The same RL-operator-selection wrapper could plausibly be applied to other VRP variants with hard timing constraints, such as long-haul routing with driver rest periods, since the policy only needs the state features and operator set to change.
  • A direct robustness check not reported in the paper: run AVNS under the same 2,000-iteration cap as RL-AVNS and compare both quality and time; the tables report only wall-clock times for the baselines, so iteration-matched comparisons would clarify how much of the gain comes from the learned policy versus the stopping rule.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes RL-AVNS, a hybrid method that trains a transformer-based PPO policy to select neighborhood operators inside an adaptive variable neighborhood search for the Vehicle Routing Problem with Multiple Time Windows (VRPMTW). The method also introduces a fitness metric that quantifies customers' temporal flexibility for use in the shaking phase. Experiments on VRPTW and VRPMTW instances with N=50 and N=100 compare RL-AVNS against VNS, RVNS, AVNS, OR-Tools, LKH3, AM, and JAMPR, reporting route-length reductions of 3.3–14.5% over AVNS and runtime reductions of 65–77%, as well as generalization from training on N=50 3-TW instances to other settings.

Significance. If the reported gains are reproducible under a controlled comparison protocol, the paper would make a useful contribution: it is the first deep-RL solver for VRPMTW, uses a realistic vending-machine replenishment setting, and demonstrates an interesting integration of a learned policy with VNS. The generalization test (training on N=50 3-TW and testing on N=100 and 2-TW/Mix-TW) is a valuable strength. However, the evidence as presented is not yet convincing: the efficiency comparisons lack a specified baseline termination protocol, the quality comparisons lack statistical support, and the fitness-metric ablation rests on a single illustrative instance. No code or data is provided, which further limits reproducibility.

major comments (3)
  1. [§5.1, Tables 5–6] The experimental protocol for the efficiency comparison is underspecified and the headline claim depends on it. Section 5.1 states that RL-AVNS runs for a fixed 2,000 iterations, but no termination condition is given for VNS, RVNS, or AVNS; consequently the reported runtimes (and the 'Gain' rows in Tables 5 and 6) compare against baselines whose compute budgets are unknowable. Moreover, the reward in Section 4.4 includes the term -100·t_run, so the learned policy is explicitly rewarded for choosing fast operators; an efficiency advantage over a quality-only selector is therefore an expected consequence of the reward design, not a controlled comparison. Please specify the baseline stopping rules and rerun the comparisons under (i) identical iteration counts and (ii) identical wall-clock time budgets, reporting both settings; this is required before the '65–77% runtime reduction' claim can be evaluated.
  2. [§5.3, Tables 5–6] All numerical results are averages over 100 instances, but the paper reports no standard deviations, confidence intervals, or significance tests. The abstract and Section 5 repeatedly state that RL-AVNS 'significantly outperforms' baselines; without variance information, the reported gains (e.g., 3.3–14.5% length reductions in Table 6) cannot be distinguished from noise. Please report per-method standard deviations and perform pairwise significance tests (e.g., Wilcoxon signed-rank) across the 100 instances for length, duration, K, and solve time.
  3. [§5.3, Figure 5] The fitness-metric ablation is based on a single illustrative instance. The two panels of Figure 5 show one routing solution with and without the fitness-based shaking, and the text concludes that the metric 'significantly enhances solution quality.' A single example cannot support a general claim about the metric's contribution. Please add an aggregate comparison (with and without the fitness metric) over the same 100-instance benchmark, reporting means and variances for length, duration, and K.
minor comments (4)
  1. [§5.3] There are several typos: 'RL-VANS' should be 'RL-AVNS'; 'applly' and 'preceeding' appear in Section 5.1; 'funtion' appears in the hyperparameter paragraph; the Figure 3 caption uses 'architechture'.
  2. [§4.4] The reward formula mixes route-length units with seconds via the hand-selected coefficient 100; please state the units or normalize the runtime term, and report a sensitivity analysis over the coefficient.
  3. [Figure 4] The convergence plot shows a single Mix-TW N=100 instance; adding error bands over multiple instances would make the convergence-speed claim more convincing.
  4. [§5.1] The paper does not specify the implementation language or version for the baseline VNS/RVNS/AVNS methods, which is important for interpreting the reported runtime comparisons.

Circularity Check

1 steps flagged · score 6.0 of 10

The reported computational-efficiency gain is partly circular because runtime is an explicit term in the training reward, so the speed-up over AVNS largely verifies the optimization objective rather than an independent discovery.

  1. self definitional [Section 4.4 (Rewards), interpreted in Section 5.3 / Tables 5-6]
    "The reward mechanism is mathematically formulated as: r = [f(x_t) - f(x_{t+1})] - 100 * t_run, where t_run denotes the computational runtime. ... the runtime penalty term introduces the computational efficiency as an explicit optimization criterion, encouraging the algorithm to discover not only high-quality but also computationally tractable solution pathways."

    RL-AVNS is trained to maximize this reward, so runtime is an input objective, not an emergent result. The paper then reports 65-77% reductions in solving time over AVNS (Table 6) as evidence of superiority, while AVNS's operator weights do not include runtime. Since both methods share the same operator set and greedy initial solution, the observed speed advantage is the expected outcome of optimizing a hand-tuned runtime-penalized reward; it is not an independent prediction. The solution-quality gains are more independent, so the circularity is partial rather than total.

full rationale

The central derivation chain of RL-AVNS is not circular with respect to solution quality: the policy is tested on unseen instances and the reported length improvements could have failed to materialize. The circularity is confined to the efficiency claim. Equation (13) in Section 4.4 defines runtime as part of the training objective via -100*t_run, and Tables 5-6 present the resulting speed advantage as a discovered improvement over a baseline that does not optimize runtime. That is essentially verifying the training objective. No load-bearing self-citation was found; the authors' prior works (Wu et al. 2022, Xin et al. 2021) appear only as related iterative-solver literature, not as proofs or uniqueness theorems. The unspecified termination criteria for VNS/RVNS/AVNS are an experimental-fairness concern, not circularity. Overall score 6 reflects one central claim that reduces, in part, to the training signal, while the quality gains retain independent empirical content.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on hand-picked reward coefficients and disruption ratios, plus assumptions about the realism of the synthetic instances and the transferability of the trained policy. No new physical or mathematical entities are introduced.

free parameters (4)
  • reward_runtime_penalty_coefficient = 100
    Hand-picked coefficient in the reward function (Section 4.4) that forces the policy to favor fast operators; directly produces the reported runtime reductions.
  • reward_clip_bound = 10
    Reward is clipped to [-10,10] to stabilize learning; chosen without sensitivity analysis (Section 4.4).
  • shaking_disruption_ratio = 0.20
    Shaking disrupts 20% of highest-fitness customers; chosen without ablation (Section 4.2).
  • discount_factor_gamma_and_gae_lambda = not reported
    PPO and GAE hyperparameters appear in equations but their values are never given; they affect the trained policy's behavior.
assumptions (4)
  • standard math Big-M constraint (7) correctly models time propagation and subtour elimination.
    The formulation is standard for VRPTW variants and used in Section 3.
  • domain assumption The synthetic instance generator reflects real-world vending machine time-window patterns.
    Section 5.1 describes three clustered replenishment periods based on a manufacturer's data, but the mapping from real data to the generation parameters is not provided.
  • domain assumption A policy trained only on N=50, 3-TW instances generalizes to N=100 and to 2-TW and Mix-TW settings.
    The paper tests this empirically, but the generalization mechanism is not analyzed; the transferability is assumed by the experimental design.
  • ad hoc to paper The fitness metric in Equation (12) identifies customers whose movement improves search.
    The metric is a new heuristic used to select customers during shaking; it is supported only by a single illustrative example (Figure 5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to Search for Vehicle Routing with Multiple Time Windows." pith.science (2026). https://pith.science/paper/IFEVGKAF

@misc{pith2026250523098,
  author       = {Pith},
  title        = {Pith review of: Learning to Search for Vehicle Routing with Multiple Time Windows},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IFEVGKAF}},
  note         = {Machine review of arXiv:2505.23098}
}
read the original abstract

In this study, we propose a reinforcement learning-based adaptive variable neighborhood search (RL-AVNS) method designed for effectively solving the Vehicle Routing Problem with Multiple Time Windows (VRPMTW). Unlike traditional adaptive approaches that rely solely on historical operator performance, our method integrates a reinforcement learning framework to dynamically select neighborhood operators based on real-time solution states and learned experience. We introduce a fitness metric that quantifies customers' temporal flexibility to improve the shaking phase, and employ a transformer-based neural policy network to intelligently guide operator selection during the local search. Extensive computational experiments are conducted on realistic scenarios derived from the replenishment of unmanned vending machines, characterized by multiple clustered replenishment windows. Results demonstrate that RL-AVNS significantly outperforms traditional variable neighborhood search (VNS), adaptive VNS (AVNS), and state-of-the-art learning-based heuristics, achieving substantial improvements in solution quality and computational efficiency across various instance scales and time window complexities. Particularly notable is the algorithm's capability to generalize effectively to problem instances not encountered during training, underscoring its practical utility for complex logistics scenarios.

Figures

Figures reproduced from arXiv: 2505.23098 by the authors.

Figure 1
Figure 1. An illustration of VRPMTW, where each customer features two optional time windows. ∑ 𝑗∈𝑉 𝑥ℎ𝑗𝑘 = ∑ 𝑖∈𝑉 𝑥𝑖ℎ𝑘, ∀ℎ ∈ 𝐶, 𝑘 ∈ 𝐾, (5) ∑ 𝑖∈𝐶 𝑑𝑖 ∑ 𝑗∈𝑉 𝑥𝑖𝑗𝑘 ≤ 𝑄, ∀𝑘 ∈ 𝐾, (6) 𝑡 𝑖 + 𝑠𝑖 + 𝑐𝑖𝑗 − 𝑀(1 − ∑ 𝑘∈𝐾 𝑥𝑖𝑗𝑘) ≤ 𝑡 𝑗 , ∀𝑖, 𝑗 ∈ 𝑉 , (7) ∑ 𝑚∈𝑊𝑖 𝑦𝑖𝑚 = 1, ∀𝑖 ∈ 𝑉 , (8) 𝑡 𝑖 ≥ ∑ 𝑚∈𝑊𝑖 𝑦𝑖𝑚 ⋅ 𝑒𝑖𝑚, ∀𝑖 ∈ 𝐶, (9) 𝑡 𝑖 ≤ ∑ 𝑚∈𝑊𝑖 𝑦𝑖𝑚 ⋅ 𝑙 𝑖𝑚, ∀𝑖 ∈ 𝐶, (10) 𝑥𝑖𝑗𝑘, 𝑦𝑖𝑚 ∈ {0, 1} ∀𝑖, 𝑗 ∈ 𝑉 , 𝑘 ∈ 𝐾, 𝑚 ∈ 𝑊𝑖 , (11) In particular, the objective function (1)… view at source ↗
Figure 2
Figure 2. Illustration of fitness calculation for node i. The location of the arrow indicates the time when the vehicle arrived at the node 𝑖, and the length covered by the brace is the value of fitness. among different neighborhood structures throughout the search process. The rationale behind VNS is grounded in three foundational observations: (1) A local optimum with respect to one neighborhood structure might not represen… view at source ↗
Figure 3
Figure 3. The architechture of our policy network in RL-AVNS. States. The state representation in our model consists of two main components: information of the current solution and its relation to historical performance. The first component encompasses both static data (customer coordinates, time window constraints) and dynamic data (arrival times, sequence of predecessor and successor nodes). The second component evaluates t… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Convergence curves for different algorithms on a Mix-TW instance with 𝑁 = 100 nodes. times after the vehicle’s arrival, a result of late time windows for customers (18, 6, 4). Conversely, the path in the right subgraph aligns more effectively with the customers’ time w…
Figure 5
Figure 5. Figure 5: The effect of fitness metric on the final solution. of RL-AVNS across various VRP variants beyond those with multiple time windows. Through this expansion of scope and further refinement of its computational efficiency, RL-AVNS would become an even more robust and vers…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 17 canonical work pages

  1. [1]

    Thevehicleroutingproblemwithmultipleprioritizedtimewindows:Acasestudy

    Beheshti,A.K.,Hejazi,S.R.,Alinaghian,M.,2015. Thevehicleroutingproblemwithmultipleprioritizedtimewindows:Acasestudy. Computers & Industrial Engineering 90, 402–413. Belhaiza, S., Hansen, P., Laporte, G.,

  2. [6]

    arXiv preprint arXiv:2006.09100

    Learning to solve vehicle routing problems with time windows through joint attention. arXiv preprint arXiv:2006.09100 . Favaretto, D., Moretti, E., Pellegrini, P.,

  3. [8]

    arXiv preprint arXiv:2002.08539

    Learn to design the heuristics for vehicle routing problem. arXiv preprint arXiv:2002.08539 . Gendreau, M., Potvin, J.Y. (Eds.),

  4. [11]

    Transportation Science 54, 400–416

    Efficient neighborhood evaluations for the vehicle routing problem with multiple time windows. Transportation Science 54, 400–416. Jumper,J.,Evans,R.,Pritzel,A.,Green,T.,Figurnov,M.,Ronneberger,O.,Tunyasuvunakool,K.,Bates,R.,Žídek,A.,Potapenko,A.,etal.,2021. Highly accurate protein structure prediction with alphafold. nature 596, 583–589. Karakostas, P., ...

  5. [12]

    Expert Systems with Applications 153, 113444

    Adaptive variable neighborhood search solution methods for the fleet size and mix pollution location-inventory-routing problem. Expert Systems with Applications 153, 113444. doi:10.1016/j.eswa.2020.113444. First Author et al.:Preprint submitted to Elsevier Page 22 of 25 Learning to search for VRPMTW Karimi-Mamaghan,M.,Mohammadi,M.,Meyer,P.,Karimi-Mamaghan...

  6. [15]

    On the fluid slip along a solid surface

    Edge-direct: A deep reinforcement learning-based method for solving heterogeneous electric vehicle routing problem with time window constraints. ArXiv preprint arXiv:2302.06789 . Nazari, M., Oroojlooy, A., Snyder, L., Takác, M.,

  7. [19]

    Calibrating 15 years of GOLF data

    An adaptive variable neighborhood search algorithm for a vehicle routing problem arising in small package shipping. Transportation Science 47, 64–80. doi:10.1287/trsc.1110.0396. Tang, Q., Kong, Y., Pan, L., Lee, C.,

  8. [21]

    Society for Industrial and Applied Mathematics

    Vehicle routing: Problems, methods, and applications, second edition. Society for Industrial and Applied Mathematics . Wang,Q.,Tang,C.,2021.Deepreinforcementlearningfortransportationnetworkcombinatorialoptimization:Asurvey. Knowledge-BasedSystems 233, 107526. Wei, L., Zhang, Z., Zhang, D., Lim, A.,

Show all 21 references
  1. [2001]

    European journal of operational research 130, 449–467

    Variable neighborhood search: Principles and applications. European journal of operational research 130, 449–467. Helsgaun,K.,2017.Anextensionofthelin-kernighan-helsgauntspsolverforconstrainedtravelingsalesmanandvehicleroutingproblems.Roskilde: Roskilde University 12, 966–980....

  2. [2012]

    EURO Journal on Transportation and Logistics 1, 285–306

    The minimum duration truck driver scheduling problem. EURO Journal on Transportation and Logistics 1, 285–306. Groër,C.,Golden,B.,Wasil,E.,2010. Alibraryoflocalsearchheuristicsforthevehicleroutingproblem. MathematicalProgrammingComputation 2, 79–101. Hansen, P., Mladenović, N.,

  3. [2013]

    Transportation Science 47, 81–107

    Long-haul vehicle routing and scheduling with working hour rules. Transportation Science 47, 81–107. Schaap,H.,Schiffer,M.,Schneider,M.,Walther,G.,2022. Alargeneighborhoodsearchforthevehicleroutingproblemwithmultipletimewindows. Transportation Science 56, 1369–1392. Schulman, ...

  4. [2014]

    Computers & Operations Research 52, 269–281

    A hybrid variable neighborhood tabu search heuristic for the vehicle routing problem with multiple time windows. Computers & Operations Research 52, 269–281. Belhaiza,S.,M’Hallah,R.,BenBrahim,G.,Laporte,G.,2019. Threemulti-startdata-drivenevolutionaryheuristicsforthevehiclerou...

  5. [2015]

    arXiv preprint arXiv:1506.02438

    High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 . Schulman,J.,Wolski,F.,Dhariwal,P.,Radford,A.,Klimov,O.,2017. Proximalpolicyoptimizationalgorithms. arXivpreprintarXiv:1707.06347. First Author et al.:Preprint submitte...

  6. [2017]

    nature 550, 354–359

    Mastering the game of go without human knowledge. nature 550, 354–359. Solomon,M.M.,1987.Algorithmsforthevehicleroutingandschedulingproblemswithtimewindowconstraints.OperationsResearch35,254–265. Stenger, A., Vigo, D., Enz, S., Schwind, M.,

  7. [2018]

    Electronic Notes in Discrete Mathematics 66, 207–214

    Variable neighborhood search for vehicle routing problem with multiple time windows. Electronic Notes in Discrete Mathematics 66, 207–214. doi:10.1016/j.endm.2018.03.027. Fisher, M.L.,

  8. [2019]

    Fast delta evaluation for the vehicle routing problem with multiple time windows. Lin,B.,Ghaddar,B.,Nathwani,J.,2022.Deepreinforcementlearningfortheelectricvehicleroutingproblemwithtimewindows.IEEETransactions on Intelligent Transportation Systems 23, 11528–11538. Lu,H.,Zhang,...

  9. [2020]

    arXiv preprint arXiv:2005.09330

    Dynamic partial removal: A neural network heuristic for large neighborhood search. arXiv preprint arXiv:2005.09330 . Clarke, G., Wright, J.W.,

  10. [2021]

    Computers & Operations Research 134, 105400

    Reinforcement learning for combinatorial optimization: A survey. Computers & Operations Research 134, 105400. Morabit,M.,Desaulniers,G.,Lodi,A.,2021.Machine-learning–basedcolumnselectionforcolumngeneration.TransportationScience55,815–831. Mozhdehi, A., Mohammadizadeh, M.,

  11. [2022]

    arXiv preprint arXiv:2207.09860

    Learning to solve soft-constrained vehicle routing problems with lagrangian relaxation. arXiv preprint arXiv:2207.09860 . Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.,

  12. [2023]

    Computers & Operations Research 149, 106016

    A variable neighborhood search-based algorithm with adaptive local search for the vehicle routing problem with time windows and multi-depots aiming for vehicle fleet reduction. Computers & Operations Research 149, 106016. doi:10.1016/j.cor.2022.106016. Bogue, E.T., Ferreira, H...

  13. [2024]

    arXiv preprint arXiv:2403.05318

    Looking ahead to avoid being late: Solving hard-constrained traveling salesman problem. arXiv preprint arXiv:2403.05318 . Chen, M., Gao, L., Chen, Q., Liu, Z.,

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.