Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

This paper introduces the XL set, a collection of 100 capacitated vehicle routing instances with 1,000 to 10,000 customers, provides initial best-known solutions from eight heuristics, and reports that a 30-day community challenge improved

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 09:58 UTC pith:ZNX4WNQN

load-bearing objection The XL benchmark is a genuinely useful new resource and the experiments are careful, but the abstract's post-competition claims (1,932 BKS improvements, LLM-assisted discovery) appear nowhere in the body and are impossible given the dates; fix that and this deserves serious refereeing. the 3 major comments →

arxiv 2601.11467 v2 pith:ZNX4WNQN submitted 2026-01-16 math.OC

The XL Instances and the CVRPLib Best Known Solution Challenge

Eduardo Queiroga , Rafael Martinelli , Anand Subramanian , Eduardo Uchoa , Thibaut Vidal This is my paper
classification math.OC MSC 90C2790C59
keywords capacitated vehicle routingbenchmark instanceslarge-scale optimizationbest known solutionsmetaheuristicsiterated local searchcommunity challengeinstance generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's goal is to give the vehicle-routing community a large, diverse benchmark for the 1,000-to-10,000-customer range, which previous public sets left almost empty. It generates 100 instances using the same attribute scheme as the established X and XML sets, then runs eight heuristics 60 times each to produce initial best-known solutions. To keep those baselines improving, it launches a 30-day community challenge with lead-time scoring. The 1,932 submitted improvements show the set is tractable yet demanding, and the experiments indicate a shift in effective methodology at scale: single-trajectory refinement (iterated local search) outperforms population-based search on the large instances.

Core claim

The paper's central claim is that the XL instance set, 100 capacitated vehicle routing problems with 1,000 to 10,000 customers, generated with the attribute-based scheme of the X and XML sets, is a suitable and much-needed testbed for large-scale routing. The initial best-known solutions, obtained from 60 runs of eight heuristics with a two-hour limit, are strong enough that 1,932 challenge improvements over 30 days were needed to refine them. The experiments also show a scale-dependent methodological transition: at this size, a single-trajectory iterated local search variant produces the best results, ahead of two other large-scale methods and well ahead of standard-scale population-based m

What carries the argument

The load-bearing machinery is the attribute-based instance generator (depot/customer positioning, demand distributions, average route size) inherited from the X/XML sets and scaled to 1,000–10,000 customers, combined with a community challenge whose lead-time scoring rewards long-held improvements. The generator produces instances spanning a wide range of structural characteristics; the challenge turns the resulting best-known solutions into a continuously moving baseline.

Load-bearing premise

The initial best-known solutions, computed with default parameters and a two-hour limit per run, are strong enough to serve as a demanding baseline for the challenge.

What would settle it

Check the public leaderboard's per-instance percentage improvements: if the 1,932 updates average more than roughly 1% gain per instance, the initial best-known solutions were not a demanding baseline; if they average well below 0.1%, they were. Alternatively, run the standard-scale heuristics on a sample of XL instances with per-instance parameter tuning and longer time budgets; systematic improvements beyond 1% would undercut the baseline claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The XL set fills the 1,001-to-2,999-customer gap and gives researchers a common, diverse testbed for comparing large-scale CVRP algorithms.
  • Because the public generator can produce statistically similar instances, the set supports both evaluation and training of machine-learning methods at scale.
  • The reported method ranking suggests that for instances of this size, single-trajectory improvement methods (like iterated local search) are currently more effective than population-based approaches.
  • The 1,932 challenge improvements imply that the initial solutions, while strong, leave clear headroom, so new methods have a concrete target and a live leaderboard to measure progress.
  • The challenge's lead-time scoring model could be applied to other benchmark families to incentivize sustained improvements rather than one-shot results.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the scale-dependent transition the paper observes is real, we can predict that future large-scale routing heuristics will converge on ILS-style refinement of a single solution, and that challenge winners will be variants of such methods rather than genetic or population-based hybrids.
  • The reported 1,932 improvements were made within a 30-day window; the final best-known solutions are likely to be substantially different from the initial ones, so researchers should treat the initial BKSs as provisional and re-check before using them as references.
  • The generator's public availability could establish a de facto training/evaluation standard for ML-based routing at scale, analogous to what the 100-customer XML set did; a testable consequence is that new ML papers will adopt this generator for their large-scale tests.
  • Re-running the standard-scale methods after calibration would clarify whether the observed ranking is a property of the algorithms themselves or an artifact of default parameters; if calibrated versions close the gap to the leading method, the claim of a methodological transition weakens.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces the XL set, a new collection of 100 CVRP benchmark instances with 1,000 to 10,000 customers, generated according to the attribute scheme of the earlier X/XML sets (depot and customer placement, demand distribution, route size, capacity). Section 4 reports an extensive initial computational study: 60 runs of eight state-of-the-art heuristics per instance, with a two-hour per-run limit on a single CPU thread. The best solutions found become the initial BKSs used for the CVRPLib BKS Challenge described in Section 3. Section 5 reports additional experiments on the X and XML sets. The abstract further claims that over the 30-day challenge participating teams submitted 1,932 BKS improvements and that the results highlight LLM-assisted algorithm discovery, but no such results appear in the body.

Significance. If the benchmark generation is sound, the XL set fills a real gap in the CVRP benchmark literature (the 1,001–2,999 customer range) and usefully extends the influential X/XML family. The public availability of the generator, the transparent experimental protocol (fixed time limit, multiple seeds, linked source codes), and the detailed per-instance results in Table 2 are concrete strengths. The additional X/XML comparisons in Section 5 are also valuable as a cross-method snapshot across instance scales. However, the abstract's unsupported and internally inconsistent claim about 1,932 post-competition BKS improvements is a serious credibility problem: it is the headline quantitative result and it cannot be verified from the manuscript. The core benchmark contribution is sound and publishable after this claim is corrected.

major comments (3)
  1. [Abstract; §3–§4] The abstract claims: 'over 30 days, participating teams submitted 1,932 BKS improvements, substantially refining the initial solution set and highlighting promising research directions for large-scale CVRPs, notably through LLM-assisted algorithm discovery.' No such result is reported anywhere in the body: Section 3 describes only the design of the BKS Challenge, Section 4 reports the authors' own initial BKS experiments, and Section 5 contains retrospective analyses on X/XML instances. Moreover, the arXiv submission is dated 16 Jan 2026, while Section 3 states the challenge starts on 12 Jan 2026 and lasts 30 days; at submission only four days had elapsed. The claim is therefore both unsubstantiated and impossible on the stated timeline. The manuscript must be corrected by either adding the full post-competition data or removing the claim from the abstract.
  2. [§3; §4, Tables 1–2] Section 3 states that the initial BKSs 'already represent a strong baseline' and that the competition is 'demanding and scientifically meaningful.' This claim depends on the quality of the initial BKSs, but Section 4 observes that SISRs, HGS-CVRP, OR-Tools, and LKH-3 'were not originally designed and calibrated for instances with up to 10,000 customers' and were run with default parameters under a two-hour limit identical to the limit used for 1,000-customer instances in the DIMACS Challenge. Since 93 of the 100 BKSs come from a single method (AILS-II), the baseline strength is only as credible as that method's default configuration at this scale. The paper is transparent about this limitation, but it does not provide evidence that the baseline is actually demanding, e.g., a sensitivity test with longer runs or calibrated parameters on a subset of instances. Please either supply such evi
  3. [§5, Tables 3–4] The paper uses the additional X/XML experiments to draw comparative conclusions, e.g., in Section 6 'HGS-CVRP achieves the best average solution quality consistently' on small/medium instances and 'AILS-II clearly outperforms the other approaches' on larger instances. These statements are based on a single time budget per instance family (10 minutes for X, 1 minute for XML) and default parameters, with no variance or significance measures. The aggregate gaps in Table 4 are not accompanied by instance counts per subgroup or standard deviations, so subgroup comparisons are hard to interpret. This does not invalidate the main benchmark contribution, but the comparative conclusions in Section 6 should be presented as descriptive observations under a specific protocol rather than as general findings.
minor comments (5)
  1. [§2] In the route-size bullet, 'The inclusion of instances with ultra long routes was demand from a number of users' is ungrammatical; it should be 'was in demand from' or 'was requested by.'
  2. [§4] The hardware sentence 'All our experiments were run on a single thread of 2 x AMD EPYC 9654 ...' is ambiguous. Clarify whether each run used one logical processor of one of the two CPUs, and how the 'up to 50 parallel executions' interacted with the two-hour wall-clock limit.
  3. [Table 1] The 'Dem' column uses abbreviations (U, Q, SL, 1–10, etc.). Most are defined in Section 2, but a compact legend directly under the table would help readers parse the instance attributes without rereading the generation description.
  4. [Table 4] The table reports average optimality gaps per attribute subgroup but no subgroup sizes or standard deviations. Given the 10,000 instances, per-subgroup counts would be easy to include and would make the comparisons more interpretable.
  5. [Abstract; §1] If the post-competition claim is retained after correction, the phrase 'LLM-assisted algorithm discovery' needs a reference or a pointer to the relevant section; currently reference [14] is cited only in the introduction and no connection is made to the claimed challenge results.

Circularity Check

0 steps flagged

No circularity found in the benchmark construction or experiments; the abstract's post-competition claims are unsupported by the body, but that is an internal-consistency issue, not circularity.

full rationale

The XL set is an instance-generation and benchmarking contribution, not a derivation. The generator follows the attribute scheme of Uchoa et al. [22] and Queiroga et al. [19], both established external benchmarks, so the self-citations are provenance rather than load-bearing proof. The initial BKSs are openly defined as the best values from the authors' own 60-run experiments (Table 1: 'Column BKS is the Best Known Solution and the Method that found it during the experiments described in Section 4'), and Table 2 gaps are computed relative to 'the best solution obtained by any method.' This makes AILS-II's 0.00 Best gap and its '93 out of 100' count a summary of the table by construction, but the paper presents these as experimental observations, not as predictions or first-principles results. Section 5 anchors the same codes against external CVRPLib BKSs and XML optima, so the experimental pipeline is not circularly evaluated. The stated limitations—default parameters, a fixed two-hour limit, no calibration—weaken the claim that the initial BKSs are a strong baseline, but that is a risk to benchmark usefulness, not circularity. Separately, the abstract's assertion of '1,932 BKS improvements' over 30 days and 'LLM-assisted algorithm discovery' appears in no body section and is inconsistent with the 16 Jan 2026 submission date versus the 12 Jan 2026 challenge start; I flag this as an unsupported/inconsistent claim, but it is not a circular step, so it does not change the circularity score.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The paper's central contribution is a dataset plus an empirical protocol; it introduces no free parameters in the mathematical fitting sense. Its load-bearing assumptions are the transferability of X/XML generation principles to 1k–10k scale, the use of default solver parameters as a fair comparison, and the external ground truth of prior BKS/optima. No new theoretical entities are postulated.

axioms (5)
  • domain assumption The Euclidean 2D grid instance structure with depot/customer distributions is a meaningful proxy for real-world CVRPs.
    Section 2 defines generation attributes following the X set; no validation against real routing data is provided.
  • standard math K_min computed by exact bin packing is a sufficient route-count reference; solutions using more routes are permitted.
    Section 2 relies on exact bin-packing solutions to set instance names and capacity; correctness of the bin-packing solver is not discussed.
  • domain assumption The "established generation principles" from X/XML transfer to 1,000–10,000 customer sizes.
    Section 2 follows Uchoa et al. [22] and Queiroga et al. [19]; extrapolation to larger n is assumed rather than demonstrated.
  • domain assumption Existing BKS and optimal values for X and XML sets can be used as ground truth for gap computations.
    Section 5 relies on CVRPLib BKSs and XML optima from prior work; no verification is performed in this paper.
  • domain assumption Default parameter settings and a two-hour time limit yield a fair and meaningful method comparison at 1,000–10,000 customers.
    Section 4 applies the DIMACS two-hour limit to instances ten times larger, and the authors acknowledge that several methods were not calibrated for this scale.

pith-pipeline@v1.3.0-alltime-deepseek · 29127 in / 14363 out tokens · 147950 ms · 2026-08-03T09:58:26.293046+00:00 · methodology

0 comments
read the original abstract

This paper introduces the XL set, a new collection of large-scale benchmark instances for the capacitated vehicle routing problem (CVRP). The set extends previous benchmarks by covering instances with 1,000 to 10,000 customers and a wide range of structural characteristics, following established generation principles from prior CVRP studies. To provide strong reference solutions, we conducted an extensive computational study with several state-of-the-art algorithms and retained the best solutions obtained as the starting point for a community-driven BKS challenge hosted on the CVRPLib website. The XL instances are publicly available to support the experimental evaluation and comparison of future solution methods. The post-competition results demonstrate the impact of the challenge: over 30 days, participating teams submitted 1,932 BKS improvements, substantially refining the initial solution set and highlighting promising research directions for solving large-scale CVRPs, notably through LLM-assisted algorithm discovery.

Figures

Figures reproduced from arXiv: 2601.11467 by Anand Subramanian, Eduardo Queiroga, Eduardo Uchoa, Rafael Martinelli, Thibaut Vidal.

Figure 1
Figure 1. Figure 1: Feasible solutions for four XL instances. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Timeline and lead-time scoring mechanism of the CVRPLib Best Known Solution Challenge for [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SpecAHD: Localize to Specialize for Automated Heuristic Design in Large-Scale Routing Problems

    cs.AI 2026-07 conditional novelty 6.5

    A coupled bilevel LLM search that specializes repair heuristics to local regions within one routing solution cuts held-out cost by up to 57.7% versus competing AHD methods.

Reference graph

Works this paper leans on

24 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    Accessed: December 28, 2025

    12th DIMACS implementation challenge: Capacitated Vehicle Routing Problem (CVRP).http: //dimacs.rutgers.edu/programs/challenge/vrp/cvrp/, 2022. Accessed: December 28, 2025

    2022

  2. [2]

    Accessed: December 28, 2025

    EURO Meets NeurIPS 2022 Vehicle Routing Competition.https://euro-neurips-vrp-2022.chal lenges.ortec.com/, 2023. Accessed: December 28, 2025. 16

    2022

  3. [3]

    A fast and scalable heuristic for the solution of large-scale capacitated vehicle routing problems.Transportation Science, 55(4):832–856, 2021

    Luca Accorsi and Daniele Vigo. A fast and scalable heuristic for the solution of large-scale capacitated vehicle routing problems.Transportation Science, 55(4):832–856, 2021

    2021

  4. [4]

    Routing one million customers in a handful of minutes.Computers & Operations Research, 164:106562, 2024

    Luca Accorsi and Daniele Vigo. Routing one million customers in a handful of minutes.Computers & Operations Research, 164:106562, 2024

    2024

  5. [5]

    Efficiently solving very large-scale routing problems.Computers & Operations Research, 107:32–42, 2019

    Florian Arnold, Michel Gendreau, and Kenneth S¨ orensen. Efficiently solving very large-scale routing problems.Computers & Operations Research, 107:32–42, 2019. ISSN 0305-0548. doi: https://doi.org/ 10.1016/j.cor.2019.03.006

    work page doi:10.1016/j.cor.2019.03.006 2019

  6. [6]

    Augerat, J

    P. Augerat, J. Belenguer, E. Benavent, A. Corber´ an, D. Naddef, and G. Rinaldi. Computational results with a branch and cut code for the capacitated vehicle routing problem. Technical Report 949-M, Universit´ e Joseph Fourier, Grenoble, France, 1995

    1995

  7. [7]

    Slack induction by string removals for vehicle routing problems.Transportation Science, 54(2):417–433, 2020

    Jan Christiaens and Greet Vanden Berghe. Slack induction by string removals for vehicle routing problems.Transportation Science, 54(2):417–433, 2020

    2020

  8. [8]

    An algorithm for the vehicle-dispatching problem.Journal of the Operational Research Society, 20(3):309–318, 1969

    Nicos Christofides and Samuel Eilon. An algorithm for the vehicle-dispatching problem.Journal of the Operational Research Society, 20(3):309–318, 1969

    1969

  9. [9]

    Exact branch-price-and-cut algorithms for vehicle routing.Transportation Science, 53(4):946–985, 2019

    Luciano Costa, Claudio Contardo, and Guy Desaulniers. Exact branch-price-and-cut algorithms for vehicle routing.Transportation Science, 53(4):946–985, 2019

    2019

  10. [10]

    The truck dispatching problem.Management science, 6(1): 80–91, 1959

    George B Dantzig and John H Ramser. The truck dispatching problem.Management science, 6(1): 80–91, 1959

    1959

  11. [11]

    Ferraz and Jo˜ ao Marcelo G

    Arthur M. Ferraz and Jo˜ ao Marcelo G. Soares. Implementation in C++ of slack induction by string removals for vehicle routing problems. Personal communication, 2025. Private source code repository shared directly by the authors

    2025

  12. [12]

    M. Fisher. Optimal solution of vehicle routing problem using minimum k-trees.Operations Research, 42:626–642, 1994

    1994

  13. [13]

    Golden, E

    B. Golden, E. Wasil, J. Kelly, and I. Chao. The impact of metaheuristics on solving the vehicle routing problem: algorithms, problem sets, and computational results. InFleet management and logistics, pages 33–56. Springer, 1998

    1998

  14. [14]

    VR- PAgent: LLM-driven discovery of heuristic operators for vehicle routing problems.arXiv preprint arXiv:2510.07073, 2025

    Andr´ e Hottung, Federico Berto, Chuanbo Hua, Nayeli Gast Zepeda, Daniel Wetzel, Michael R¨ omer, Haoran Ye, Davide Zago, Michael Poli, Stefano Massaroli, Jinkyoo Park, and Kevin Tierney. VR- PAgent: LLM-driven discovery of heuristic operators for vehicle routing problems.arXiv preprint arXiv:2510.07073, 2025

    arXiv 2025

  15. [15]

    M´ aximo, Jean-Fran¸ cois Cordeau, and Mari´ a C

    Vin ´ ıcius R. M´ aximo, Jean-Fran¸ cois Cordeau, and Mari´ a C. V. Nascimento. AILS-II: An Adaptive Iterated Local Search Heuristic for the Large-Scale Capacitated Vehicle Routing Problem.INFORMS Journal on Computing, 36(4):974–986, 2024. doi: 10.1287/ijoc.2023.0106

    arXiv 2024

  16. [16]

    Improved branch-cut-and-price for capacitated vehicle routing.Mathematical Programming Computation, 9(1):61–100, 2017

    Diego Pecin, Artur Pessoa, Marcus Poggi, and Eduardo Uchoa. Improved branch-cut-and-price for capacitated vehicle routing.Mathematical Programming Computation, 9(1):61–100, 2017

    2017

  17. [17]

    A generic exact solver for vehicle routing and related problems.Mathematical Programming, 183(1):483–523, 2020

    Artur Pessoa, Ruslan Sadykov, Eduardo Uchoa, and Fran¸ cois Vanderbeck. A generic exact solver for vehicle routing and related problems.Mathematical Programming, 183(1):483–523, 2020

    2020

  18. [18]

    New exact algorithms for the capacitated vehicle routing problem

    Marcus Poggi and Eduardo Uchoa. New exact algorithms for the capacitated vehicle routing problem. In Paolo Toth and Daniele Vigo, editors,Vehicle Routing: Problems, Methods, and Applications, Second Edition, pages 59–86. SIAM, 2014

    2014

  19. [19]

    10,000 optimal CVRP so- lutions for testing machine learning based heuristics

    Eduardo Queiroga, Ruslan Sadykov, Eduardo Uchoa, and Thibaut Vidal. 10,000 optimal CVRP so- lutions for testing machine learning based heuristics. InAAAI-22 workshop on machine learning for operations research (ML4OR), 2021. 17

    2021

  20. [20]

    Probabilistic diversification and intensification in local search for vehicle routing.Journal of heuristics, 1(1):147–167, 1995

    Yves Rochat and ´Eric D Taillard. Probabilistic diversification and intensification in local search for vehicle routing.Journal of heuristics, 1(1):147–167, 1995

    1995

  21. [21]

    Cluster branching for vehicle routing problems.INFORMS Journal on Computing, 2025

    Jo˜ ao Marcos Pereira Silva, Eduardo Uchoa, and Anand Subramanian. Cluster branching for vehicle routing problems.INFORMS Journal on Computing, 2025

    2025

  22. [22]

    New benchmark instances for the capacitated vehicle routing problem.European Journal of Operational Research, 257(3):845–858, 2017

    Eduardo Uchoa, Diego Pecin, Artur Pessoa, Marcus Poggi, Thibaut Vidal, and Anand Subramanian. New benchmark instances for the capacitated vehicle routing problem.European Journal of Operational Research, 257(3):845–858, 2017

    2017

  23. [23]

    Hybrid genetic search for the CVRP: Open-source implementation and SW AP* neigh- borhood.Computers & Operations Research, 140:105643, 2022

    Thibaut Vidal. Hybrid genetic search for the CVRP: Open-source implementation and SW AP* neigh- borhood.Computers & Operations Research, 140:105643, 2022

    2022

  24. [24]

    RouteOpt: An open-source modular exact solver for vehicle routing problems.Available at SSRN 5314242, 2025

    Zhengzhong You and Yu Yang. RouteOpt: An open-source modular exact solver for vehicle routing problems.Available at SSRN 5314242, 2025. 18

    2025