Pith. sign in

REVIEW 4 major objections 5 minor 62 references

Preference Optimization for Combinatorial Optimization Problems

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that training combinatorial-optimization solvers on qualitative pairwise preferences between sampled solutions, instead of raw reward magnitudes, yields faster convergence and better solutions than REINFORCE-style…

desk verdict Useful empirical recipe for DPO-style training in neural COPs, but the claimed entropy-regularized grounding is wrong. read the letter →

arxiv 2505.08735 v1 pith:II3TKI3I submitted 2025-05-13 cs.LG

Mingjun Pan , Guanquan Lin , You-Wei Luo , Bin Zhu , Zhien Dai , Lijun Sun , Chun Yuan This is my paper · ORCID
classification cs.LG
keywords preferenceoptimizationcombinatorialreinforcementlearningneuralsolversentropy-regularizedRLdirecttravelingsalesmanproblemvehiclerouting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a preference-based training objective, which it calls Preference Optimization (PO), can replace the usual REINFORCE gradient in neural solvers for combinatorial optimization. Instead of scaling policy updates by a numerical advantage, the method labels each sampled solution as better or worse than another sampled solution and trains the policy to predict the winner. Because the reward is reparameterized in terms of the policy, the preference probability between two solutions becomes a function of their log-probability difference, and the intractable partition function cancels. The authors argue that this stabilizes training when rewards stop differentiating good from bad solutions, promotes exploration through an entropy-regularized objective, and lets local search be folded into fine-tuning rather than paid for at inference. A sympathetic reader would take the paper's claim to be that PO is a drop-in replacement for policy-gradient training that converges faster and finds shorter tours and schedules on TSP, CVRP, and FFSP benchmarks.

What carries the argument

The load-bearing object is the reward-to-policy reparameterization identity of Eq. 5, $\hat{r}(x,\tau) = \alpha \log \pi(\tau|x) + \alpha \log Z(x)$, obtained by inverting the analytical form of the entropy-regularized optimal policy. This identity turns a preference model into a policy-only objective: substituting it into the paired-comparison probability gives Eq. 6, $p(\tau_1 \succ \tau_2 | x) = f(\alpha [\log \pi(\tau_1|x) - \log \pi(\tau_2|x)])$, where $Z(x)$ cancels as an instance-dependent constant. The machinery then works by generating conflict-free preference labels from the grounding reward, maximizing the classification likelihood in Eq. 7, and, in the fine-tuning phase, treating local-search-improved solutions as preferred trajectories as in Eq. 9. The identity does the main work: it converts an intractable entropy-regularized RL problem into a tractable pairwise classification problem over sampled trajectories.

What would settle it

Take a small combinatorial instance such as TSP with 8-10 cities, enumerate all tours to compute the exact entropy-regularized optimal policy $\pi^*(\tau|x) \propto \exp(r(x,\tau)/\alpha)$, train a policy with Eq. 7 from reward-generated labels, and compare the learned log-probability differences with the exact Gibbs differences; a systematic mismatch at high label accuracy would falsify the claimed equivalence.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that the analytical relation $\hat{r}(x,\tau) = \alpha \log \pi(\tau|x) + \alpha \log Z(x)$ from the entropy-regularized optimal policy lets the probability that one trajectory is preferred over another be written as $p(\tau_1 \succ \tau_2 | x) = f(\alpha [\log \pi_{\theta}(\tau_1|x) - \log \pi_{\theta}(\tau_2|x)])$. Using the true objective to generate conflict-free preference labels, PO trains the policy by maximizing the likelihood of these preferences rather than by regressing on reward magnitudes. The resulting update in Eq. 8 replaces the advantage $r(x,\tau) - b(x)$ in REINFORCE with a signed comparison term that is invariant to affine reward transformations, so it keeps separating good from bad solutions even when advantage values shrink. With the Bradley-Terry or exponential preference models, the paper reports that PO converges 1.5x-2.5x faster than the corresponding REINFORCE variants on POMO, Sym-NCO, and Pointerformer, improves solution quality on TSP, CVRP, and FFSP, and that fine-tuning with local search cuts the TSP-100 gap to 0.03% and CVRP-100 to 1.19% without extra inference time.

Load-bearing premise

The whole method rests on the unproved assumption that maximizing the likelihood of reward-generated pairwise preference labels drives the policy to the same optimum as the entropy-regularized RL objective; if that equivalence fails, the method is a heuristic whose guarantees evaporate.

Editorial extensions

If this is right

  • Training time shrinks: PO reaches the performance REINFORCE achieves at epoch 200 in roughly 40%-60% of the epochs, with comparable gains on FFSP and large-scale TSP.
  • Final solution quality improves at no inference cost, because local search generates preference pairs during fine-tuning rather than running at test time.
  • PO is architecture-agnostic: swapping the update rule into POMO, Sym-NCO, Pointerformer, MatNet, ELG, DIMES, COMPASS, and Poppy improves or matches each baseline, placing the benefit in the training objective rather than the network design.
  • Preference labels are invariant to positive affine reward transformations, so the learned policy should be insensitive to reward shaping or to rescaling of route lengths and makespans.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the claimed equivalence between preference classification and the entropy-regularized objective holds only approximately, the method still works as an uphill heuristic: what matters is that the gradient increases probability mass on better solutions, not that the final policy is exactly the regularized optimum.
  • The ordinal-signal idea transfers to problems where a numeric reward is unreliable or multi-objective, since 'which solution is better' labels are easier to specify than a scalar reward; the paper gestures at multi-objective optimization but does not claim this extension.
  • Folding local search into fine-tuning turns the solver into an imitation learner of improved trajectories, so a testable extension is to tighten the off-policy story by comparing PO with an explicit DAgger-style correction when local search shifts the solution distribution.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Preference Optimization (PO), a training objective for neural combinatorial optimization solvers that replaces scalar rewards with pairwise preference labels derived from the ground-truth objective (route length or makespan). The method reparameterizes the reward as alpha times the policy log-probability plus a per-instance constant, converts trajectory comparisons into a logistic (or other) preference loss (Eq. 7), and optionally fine-tunes with local search (Eq. 9). It is evaluated on TSP, CVRP, and FFSP across several RL4CO baselines (AM, POMO, Sym-NCO, Pointerformer, ELG, MatNet, DIMES), reporting faster convergence and lower optimality gaps than REINFORCE-based variants, plus a zero-shot generalization study.

Significance. PO is a conceptually appealing direction: it replaces the diminishing advantages of REINFORCE with a scale-invariant comparison signal, and the local-search fine-tuning procedure is a practical way to improve solution quality without extra inference cost. The empirical study is broad, the preference labels are anchored to the true objective, so the method is not circular, and the paper provides an implementation snippet and detailed hyperparameter tables. However, because the central equivalence claim (Eq. 7 versus Eq. 3) is not established and the empirical claims lack error bars or multiple seeds, the current form is not conclusive.

major comments (4)
  1. [Section 3.3, Eq. (7) vs. Eq. (3)] The load-bearing claim that maximizing Eq. (7) is equivalent to solving the entropy-regularized objective Eq. (3) is not established and is incorrect as stated. With deterministic labels y = 1(r(tau1) > r(tau2)), the Bradley-Terry loss in Eq. (7) is a binary classification loss with logit alpha[log pi_theta(tau1|x) - log pi_theta(tau2|x)]. Optimizing this loss over the policy simplex drives the logit to +infinity for every preferred pair, which is achievable only as pi_theta degenerates onto the highest-reward trajectory; the Boltzmann optimum pi* proportional to exp(r/alpha) of Eq. (4) is not even a stationary point, since at pi* the gradient for the preferred trajectory is proportional to sigma(-(r(tau1)-r(tau2))) > 0. Proposition 3.1 only shows that per-instance reward shifts do not change the optimal policy of Eq. (3); it does not supply the missing reference policy or bridge deterministic indicator labels to the BT/Thurstone likelihood. Please either prove the equivalence under explicit assumptions (stochastic preference labels, a reference policy, or an additional regularizer) or remove the claim that PO "formulates an entropy-regularized objective" from the abstract and Section 3.3.
  2. [Section 4, Tables 1-3, Figures 2-5] The empirical comparison lacks any measure of uncertainty. All gaps in Tables 1-3 and all convergence curves in Figures 2, 4, and 5 are single point estimates, and Figure 3 reports only point values; without multiple seeds and error bars (or at least bootstrap confidence intervals over test instances), the claim that PO "significantly outperforms" REINFORCE-based methods is not supported. Please report means and standard deviations over at least 3-5 independent training runs and state the number of seeds in every table and figure.
  3. [Appendix E.2, Tables 4-9] The hyperparameters alpha and the preference model are tuned separately for each problem and solver (e.g., alpha = 0.05 for TSP and 0.03 for CVRP with POMO; BT for TSP and Exponential for CVRP). This tuning makes it difficult to attribute the observed gains to the PO objective rather than to per-task calibration. Please include a sensitivity analysis over alpha and over the preference model, or justify why the chosen values are comparable across methods.
  4. [Section 3.4, Eq. (9)] The claim that PO "naturally aligns with an imitation learning perspective" and therefore handles the off-policy LS-refined solutions without importance sampling is asserted rather than demonstrated. Eq. (9) is a behavior-cloning term on LS outputs, and the paragraph does not analyze how this term interacts with the preference objective or why distribution shift is not a concern. Please provide a formal argument or an ablation that isolates the fine-tuning contribution.
minor comments (5)
  1. [Section 3.3, Eq. (7)] Eq. (7) has a notational inconsistency: the expectation is over tau ~ pi_theta(·|x), but the integrand refers to tau1 and tau2; the objective should be written over pairs, and the normalization in Eq. (8) should match Algorithm 1.
  2. [Section 3.3, Eq. (5)] In Eq. (5), the symbol r is used where the surrounding text distinguishes the latent reward \hat r; please use consistent notation.
  3. [Appendix E.1] There are typos in the code snippet: "preference_optimazation" should be "preference_optimization" and "Brandley-Terry" should be "Bradley-Terry".
  4. [Section 2] In Section 2, "such ask-Opt" should be "such as k-Opt".
  5. [Table 2] Table 2 reports CPLEX times in hours but uses "x" for FFSP50 and FFSP100; please clarify whether these are per-instance or total times and why entries are missing.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the preference labels are grounded in the external route-length/makespan objective, and the DPO-style reward reparameterization follows from the stated entropy-regularized objective rather than assuming the paper's conclusions.

full rationale

The derivation chain is sequential: Eq. 3 states the entropy-regularized objective, Eq. 4 gives the Boltzmann optimal policy, Eq. 5 reparameterizes the reward in terms of that policy, Eq. 6 maps reward differences to preference probabilities, and Eq. 7 turns the preference likelihood into a policy objective. Each transition is presented as a derivation, with the reparameterization explicitly attributed to external prior work (Rafailov et al., 2024; Hejna & Sadigh, 2024), so no load-bearing self-citation or imported uniqueness theorem is involved. The preference labels are generated from the grounding reward function (route length or makespan) and the same external objective is used for evaluation; this is standard RL benchmarking, not a fitted parameter renamed as a prediction. The appearance of the policy in both the reparameterized reward (Eq. 5) and the trained policy (Eq. 7) is the standard DPO mechanism, not a logical circularity, because Eq. 5 is derived from Eq. 4 rather than assumed. The skeptical concern that Eq. 7 lacks an explicit entropy or reference-policy KL term, and therefore may not recover the entropy-regularized optimum, is a mathematical correctness issue, not circularity, and per the review rules it does not raise the circularity score. The paper is self-contained against external benchmarks (TSP, CVRP, FFSP) with external baselines, so the honest finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The ledger is light: no new entities or forces. The main free parameters are alpha and the preference model choice, both tuned per problem. The load-bearing axioms are the standard max-entropy policy form and the unproven DPO-style equivalence between the preference loss and the entropy-regularized objective.

free parameters (3)
  • alpha (entropy regularization / preference temperature) = 0.05 (POMO TSP), 0.03 (POMO CVRP), 0.05 (AM TSP), 0.03 (AM CVRP), 0.05 (Sym-NCO TSP), 0.03 (Sym-NCO CVRP), 1.5/1.5/1…
    Grid-searched over {0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 2.0} per problem and architecture; controls preference sharpness and the implicit exploration/exploitation trade-off in Eq 6-8.
  • preference model (BT vs Exponential vs Thurstone vs Plackett-Luce) = BT for POMO TSP, AM TSP/CVRP, Sym-NCO TSP; Exponential for POMO CVRP, Sym-NCO CVRP, MatNet FFSP, DIMES
    Selected per problem based on empirical performance (Appendix E.2, F.4). The choice changes the gradient weighting in Eq 8 and the final results.
  • local search iterations and fine-tuning epochs = Not explicitly reported; fine-tuning uses 100 epochs for TSP-100, 200 for CVRP-100
    Local search (2-Opt, swap*) is applied during a fine-tuning phase; the number of LS iterations per trajectory is not specified in the main text, affecting the preference pair quality.
assumptions (4)
  • standard math The optimal policy for the entropy-regularized RL objective has the Boltzmann form pi*(tau|x) proportional to exp(r(x,tau)/alpha), with a finite partition function Z(x) (Eq 4).
    Invoked at the start of Section 3.3, citing Ziebart et al. 2008 and Haarnoja et al. 2017. It is a standard result when the action space is finite, which holds for COPs.
  • domain assumption The Bradley-Terry / Thurstone / Plackett-Luce models correctly map reward differences to preference probabilities (Eq 2).
    Standard statistical comparison models, assumed valid for route-length and makespan rewards.
  • ad hoc to paper Substituting the current policy pi_theta for the optimal policy pi* in Eq 6 gives a valid preference model whose optimization (Eq 7) is equivalent to optimizing the entropy-regularized objective (Eq 3).
    This is the DPO assumption (uniform reference policy). The paper does not prove this equivalence; Proposition 3.1 only covers reward shifts. It is load-bearing because the entire method rests on Eq 7 being the right objective.
  • domain assumption Local search outputs can be added to the preference set as expert demonstrations without importance sampling, despite being off-policy.
    Asserted in Section 3.4 with an imitation-learning argument. The policy is updated toward LS-refined solutions even though they were not sampled from pi_theta.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Preference Optimization for Combinatorial Optimization Problems." pith.science (2026). https://pith.science/paper/II3TKI3I

@misc{pith2026250508735,
  author       = {Pith},
  title        = {Pith review of: Preference Optimization for Combinatorial Optimization Problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/II3TKI3I}},
  note         = {Machine review of arXiv:2505.08735}
}
read the original abstract

Reinforcement Learning (RL) has emerged as a powerful tool for neural combinatorial optimization, enabling models to learn heuristics that solve complex problems without requiring expert knowledge. Despite significant progress, existing RL approaches face challenges such as diminishing reward signals and inefficient exploration in vast combinatorial action spaces, leading to inefficiency. In this paper, we propose Preference Optimization, a novel method that transforms quantitative reward signals into qualitative preference signals via statistical comparison modeling, emphasizing the superiority among sampled solutions. Methodologically, by reparameterizing the reward function in terms of policy and utilizing preference models, we formulate an entropy-regularized RL objective that aligns the policy directly with preferences while avoiding intractable computations. Furthermore, we integrate local search techniques into the fine-tuning rather than post-processing to generate high-quality preference pairs, helping the policy escape local optima. Empirical results on various benchmarks, such as the Traveling Salesman Problem (TSP), the Capacitated Vehicle Routing Problem (CVRP) and the Flexible Flow Shop Problem (FFSP), demonstrate that our method significantly outperforms existing RL algorithms, achieving superior convergence efficiency and solution quality.

Figures

Figures reproduced from arXiv: 2505.08735 by the authors.

Figure 1
Figure 1. Algorithmic framework of PO for COPs. In the Preference Comparison module, pairwise comparisons are [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. (a)-(c): Comparison of PO and RFs on TSP-100 on different neural solvers; PO achieves RFs-level performance in only 40% - 60% training epochs, and surpasses RFs’ solution quality consistently. (d): Comparison of different preference models: Bradley-Terry (BT), Plackett-Luce (PL), Thurstone (Th), and unbounded Exponential (Exp) (Azar et al., 2024). 0 20 40 60 80 100 Index of Sorted Solutions 2 1 0 1 2 Scale RF PO (a)… view at source ↗
Figure 3
Figure 3. (a): Advantage values for solutions sorted by their length, sampled from the trained model, PO significantly assigns separable advantage values than RF. (b): Distribution of advantage scales among different algorithms, comparing REINFORCE-based method, PO with the Thurstone model (PO-Th), and PO with the Bradley-Terry model (PO-BT). (c): Consistency measured as p(π(τ1) > π(τ2) | r(τ1) > r(τ2)). PO shows higher consi… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: (a) Training curve for TSP (N=100) over 2000 epochs. (b) Training curve for CVRP (N=100) over 4000 epochs. [PITH_FULL_IMAGE:figures/full_fig_p023_4.png]
Figure 5
Figure 5. Figure 5: Training curves of PO and REINFORCE on Poppy and COMPASS. [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]
Figure 6
Figure 6. Figure 6: Comparison of three preference models—Thurstone, Bradley-Terry, and Exponential— under different reward [PITH_FULL_IMAGE:figures/full_fig_p025_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 41 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Concorde TSP solver

    Applegate, D., Bixby, R., Chvatal, V., and Cook, W. Concorde TSP solver. http://www.math.uwaterloo.ca/tsp/concorde/, 2006

    work page 2006

  3. [3]

    G., Guo, Z

    Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp.\ 4447--4455. PMLR, 2024

    2024

  4. [4]

    G., Naddaf, Y., Veness, J., and Bowling, M

    Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 0 253--279, 2013

    2013

  5. [5]

    V., Norouzi, M., and Bengio, S

    Bello, I., Pham, H., Le, Q. V., Norouzi, M., and Bengio, S. Neural combinatorial optimization with reinforcement learning. arXiv preprint arXiv:1611.09940, 2016

    arXiv 2016

  6. [6]

    Machine learning for combinatorial optimization: a methodological tour d’horizon

    Bengio, Y., Lodi, A., and Prouvost, A. Machine learning for combinatorial optimization: a methodological tour d’horizon. European Journal of Operational Research, 290 0 (2): 0 405--421, 2021

    2021

  7. [7]

    Bi, J., Ma, Y., Wang, J., Cao, Z., Chen, J., Sun, Y., and Chee, Y. M. Learning generalizable models for vehicle routing problems via knowledge distillation. In Advances in Neural Information Processing Systems, 2022

    work page 2022

  8. [8]

    Chalumeau, F., Surana, S., Bonnet, C., Grinsztajn, N., Pretorius, A., Laterre, A., and Barrett, T. D. Combinatorial optimization with policy adaptation using latent space search. In Advances in Neural Information Processing Systems, 2023

    work page 2023

Show all 62 references
  1. [9]

    F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D

    Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017

    2017

  2. [10]

    J., Cunningham, W

    Cook, W. J., Cunningham, W. H., Pulleyblank, W. R., and Schrijver, A. Combinatorial optimization. Unpublished manuscript, 10: 0 75--93, 1994

    1994

  3. [11]

    Cplex, I. I. V12. 1: User’s manual for cplex. International Business Machines Corporation, 46 0 (53): 0 157, 2009

    2009

  4. [12]

    Croes, G. A. A method for solving traveling-salesman problems. Operations research, 6 0 (6): 0 791--812, 1958

    1958

  5. [13]

    R., Rhuggenaath, J., Zhang, Y., and Akcay, A

    d O Costa, P. R., Rhuggenaath, J., Zhang, Y., and Akcay, A. Learning 2-opt heuristics for the traveling salesman problem via deep reinforcement learning. In Asian Conference on Machine Learning, pp.\ 465--480. PMLR, 2020

    2020

  6. [14]

    David, H. A. The method of paired comparisons, volume 12. London, 1963

    1963

  7. [15]

    Bq-nco: Bisimulation quotienting for efficient neural combinatorial optimization

    Drakulic, D., Michel, S., Mai, F., Sors, A., and Andreoli, J.-M. Bq-nco: Bisimulation quotienting for efficient neural combinatorial optimization. In Advances in Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2301.03313

    2023 arXiv

  8. [16]

    Generalize a small pre-trained model to arbitrarily large tsp instances

    Fu, Z.-H., Qiu, K.-B., and Zha, H. Generalize a small pre-trained model to arbitrarily large tsp instances. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 7474--7482, 2021

    2021

  9. [17]

    Towards generalizable neural solvers for vehicle routing problems via ensemble with transferrable local policy

    Gao, C., Shang, H., Xue, K., Li, D., and Qian, C. Towards generalizable neural solvers for vehicle routing problems via ensemble with transferrable local policy. In Larson, K. (ed.), Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJC...

    2024 doi

  10. [18]

    Garey, M. R. and Johnson, D. S. Computers and intractability, volume 174. freeman San Francisco, 1979

    1979

  11. [19]

    Grinsztajn, N., Furelos-Blanco, D., Surana, S., Bonnet, C., and Barrett, T. D. Winner takes it all: Training performant rl populations for combinatorial optimization. In Advances in Neural Information Processing Systems, 2023

    2023

  12. [20]

    Reinforcement learning with deep energy-based policies

    Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, pp.\ 1352--1361. PMLR, 2017

    2017

  13. [21]

    and Sadigh, D

    Hejna, J. and Sadigh, D. Inverse preference learning: Preference-based rl without a reward function. Advances in Neural Information Processing Systems, 36, 2024

    2024

  14. [22]

    An extension of the lin-kernighan-helsgaun tsp solver for constrained traveling salesman and vehicle routing problems

    Helsgaun, K. An extension of the lin-kernighan-helsgaun tsp solver for constrained traveling salesman and vehicle routing problems. Roskilde: Roskilde University, 12, 2017

    2017

  15. [23]

    Orpo: Monolithic preference optimization without reference model

    Hong, J., Lee, N., and Thorne, J. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2 0 (4): 0 5, 2024

    2024 arXiv

  16. [24]

    Ensemble-based deep reinforcement learning for vehicle routing problems under distribution shift

    Jiang, Y., Cao, Z., Wu, Y., Song, W., and Zhang, J. Ensemble-based deep reinforcement learning for vehicle routing problems under distribution shift. Advances in Neural Information Processing Systems, 36, 2024

    2024

  17. [25]

    Pointerformer: Deep reinforced multi-pointer transformer for the traveling salesman problem

    Jin, Y., Ding, Y., Pan, X., He, K., Zhao, L., Qin, T., Song, L., and Bian, J. Pointerformer: Deep reinforced multi-pointer transformer for the traveling salesman problem. arXiv preprint arXiv:2304.09407, 2023

    2023 arXiv

  18. [26]

    Symmetric exploration in combinatorial optimization is free! arXiv preprint arXiv:2306.01276, 2023

    Kim, H., Kim, M., Ahn, S., and Park, J. Symmetric exploration in combinatorial optimization is free! arXiv preprint arXiv:2306.01276, 2023

    2023 arXiv

  19. [27]

    Sym-nco: Leveraging symmetricity for neural combinatorial optimization

    Kim, M., Park, J., and Park, J. Sym-nco: Leveraging symmetricity for neural combinatorial optimization. Advances in Neural Information Processing Systems, 35: 0 1936--1949, 2022

    1936

  20. [28]

    Attention, learn to solve routing problems! In International Conference on Learning Representations, 2019

    Kool, W., Van Hoof, H., and Welling, M. Attention, learn to solve routing problems! In International Conference on Learning Representations, 2019

    2019

  21. [29]

    H., Vygen, J., Korte, B., and Vygen, J

    Korte, B. H., Vygen, J., Korte, B., and Vygen, J. Combinatorial optimization, volume 1. Springer, 2011

    2011

  22. [30]

    Pomo: Policy optimization with multiple optima for reinforcement learning

    Kwon, Y.-D., Choo, J., Kim, B., Yoon, I., Gwon, Y., and Min, S. Pomo: Policy optimization with multiple optima for reinforcement learning. Advances in Neural Information Processing Systems, 33: 0 21188--21198, 2020

    2020

  23. [31]

    Matrix encoding networks for neural combinatorial optimization

    Kwon, Y.-D., Choo, J., Yoon, I., Park, M., Park, D., and Gwon, Y. Matrix encoding networks for neural combinatorial optimization. Advances in Neural Information Processing Systems, 34: 0 5138--5149, 2021

    2021

  24. [32]

    T2t: From distribution learning in training to gradient search in testing for combinatorial optimization

    Li, Y., Guo, J., Wang, R., and Yan, J. T2t: From distribution learning in training to gradient search in testing for combinatorial optimization. In Advances in Neural Information Processing Systems, 2023

    2023

  25. [33]

    Neural combinatorial optimization with heavy decoder: Toward large scale generalization

    Luo, F., Lin, X., Liu, F., Zhang, Q., and Wang, Z. Neural combinatorial optimization with heavy decoder: Toward large scale generalization. In The 37th Anniversary Conference on Neural Information Processing Systems, NeurIPS 2023, 2023

    2023

  26. [34]

    Learning to iteratively solve routing problems with dual-aspect collaborative transformer

    Ma, Y., Li, J., Cao, Z., Song, W., Zhang, L., Chen, Z., and Tang, J. Learning to iteratively solve routing problems with dual-aspect collaborative transformer. In Advances in Neural Information Processing Systems, volume 34, pp.\ 11096--11107, 2021

    2021

  27. [35]

    Simpo: Simple preference optimization with a reference-free reward

    Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024

    2024 arXiv

  28. [36]

    W., Songhori, E., Wang, S., Lee, Y.-J., Johnson, E., Pathak, O., Nazi, A., et al

    Mirhoseini, A., Goldie, A., Yazgan, M., Jiang, J. W., Songhori, E., Wang, S., Lee, Y.-J., Johnson, E., Pathak, O., Nazi, A., et al. A graph placement methodology for fast chip design. Nature, 594 0 (7862): 0 207--212, 2021

    2021

  29. [37]

    Reinforcement learning for solving the vehicle routing problem

    Nazari, M., Oroojlooy, A., Snyder, L., and Tak \'a c, M. Reinforcement learning for solving the vehicle routing problem. Advances in Neural Information Processing Systems, 31, 2018

    2018

  30. [38]

    Generalization in deep rl for tsp problems via equivariance and local search

    Ouyang, W., Wang, Y., Weng, P., and Han, S. Generalization in deep rl for tsp problems via equivariance and local search. arXiv preprint arXiv:2110.03595, 2021

    2021 arXiv

  31. [39]

    Papadimitriou, C. H. and Steiglitz, K. Combinatorial Optimization: Algorithms and Complexity. Dover Publications, 1998

    1998

  32. [40]

    Disentangling length from quality in direct preference optimization

    Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19159, 2024

    2024 arXiv

  33. [41]

    Plackett, R. L. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24 0 (2): 0 193--202, 1975

    1975

  34. [42]

    Pomerleau, D. A. Alvinn: An autonomous land vehicle in a neural network. Advances in Neural Information Processing Systems, 1, 1988

    1988

  35. [43]

    Dimes: A differentiable meta solver for combinatorial optimization problems

    Qiu, R., Sun, Z., and Yang, Y. Dimes: A differentiable meta solver for combinatorial optimization problems. Advances in Neural Information Processing Systems, 35: 0 25531--25546, 2022

    2022

  36. [44]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  37. [45]

    Tsplib—a traveling salesman problem library

    Reinelt, G. Tsplib—a traveling salesman problem library. ORSA journal on computing, 3 0 (4): 0 376--384, 1991

    1991

  38. [46]

    A reduction of imitation learning and structured prediction to no-regret online learning

    Ross, S., Gordon, G., and Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pp.\ 627--635, 2011

    2011

  39. [47]

    A general reinforcement learning algorithm that masters chess, shogi, and go through self-play

    Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362 0 (6419): 0 1140--1144, 2018

    2018

  40. [48]

    K., and Sarwar, T

    Sultana, N., Chan, J., Qin, A. K., and Sarwar, T. Learning vehicle routing problems using policy optimisation. arXiv preprint arXiv:2012.13269, 2020

    2012 arXiv

  41. [49]

    and Yang, Y

    Sun, Z. and Yang, Y. DIFUSCO : Graph-based diffusion solvers for combinatorial optimization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=JV8Ff0lgVV

    2023

  42. [50]

    Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018

    2018

  43. [51]

    Mujoco: A physics engine for model-based control

    Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.\ 5026--5033, 2012

    2012

  44. [52]

    New benchmark instances for the capacitated vehicle routing problem

    Uchoa, E., Pecin, D., Pessoa, A., Poggi, M., Vidal, T., and Subramanian, A. New benchmark instances for the capacitated vehicle routing problem. European Journal of Operational Research, 257 0 (3): 0 845--858, 2017

    2017

  45. [53]

    Hybrid genetic search for the cvrp: Open-source implementation and swap* neighborhood

    Vidal, T. Hybrid genetic search for the cvrp: Open-source implementation and swap* neighborhood. Computers & Operations Research, 140: 0 105643, 2022

    2022

  46. [54]

    Pointer networks

    Vinyals, O., Fortunato, M., and Jaitly, N. Pointer networks. In Advances in Neural Information Processing Systems, pp.\ 2692--2700, 2015

    2015

  47. [55]

    Asp: Learn a universal neural solver! IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    Wang, C., Yu, Z., McAleer, S., Yu, T., and Yang, Y. Asp: Learn a universal neural solver! IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  48. [56]

    A survey of preference-based reinforcement learning methods

    Wirth, C., Akrour, R., Neumann, G., and F \"u rnkranz, J. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18 0 (136): 0 1--46, 2017

    2017

  49. [57]

    Learning improvement heuristics for solving routing problems

    Wu, Y., Song, W., Cao, Z., Zhang, J., and Lim, A. Learning improvement heuristics for solving routing problems. IEEE Transactions on Neural Networks and Learning Systems, 33 0 (9): 0 5057--5069, 2021

    2021

  50. [58]

    Multi-decoder attention model with embedding glimpse for solving vehicle routing problems

    Xin, L., Song, W., Cao, Z., and Zhang, J. Multi-decoder attention model with embedding glimpse for solving vehicle routing problems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.\ 12042--12049, 2021 a

    2021

  51. [59]

    Neurolkh: Combining deep learning model with lin-kernighan-helsgaun heuristic for solving the traveling salesman problem

    Xin, L., Song, W., Cao, Z., and Zhang, J. Neurolkh: Combining deep learning model with lin-kernighan-helsgaun heuristic for solving the traveling salesman problem. Advances in Neural Information Processing Systems, 34: 0 7472--7483, 2021 b

    2021

  52. [60]

    Deepaco: Neural-enhanced ant systems for combinatorial optimization

    Ye, H., Wang, J., Cao, Z., Liang, H., and Li, Y. Deepaco: Neural-enhanced ant systems for combinatorial optimization. In Advances in Neural Information Processing Systems, 2023

    2023

  53. [61]

    Towards omni-generalizable neural methods for vehicle routing problems

    Zhou, J., Wu, Y., Song, W., Cao, Z., and Zhang, J. Towards omni-generalizable neural methods for vehicle routing problems. In International Conference on Machine Learning, 2023

    2023

  54. [62]

    D., Maas, A

    Ziebart, B. D., Maas, A. L., Bagnell, J. A., Dey, A. K., et al. Maximum entropy inverse reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 8, pp.\ 1433--1438, 2008

    2008

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.