Pith. sign in

REVIEW 4 major objections 5 minor 58 references

Sequential Stochastic Combinatorial Optimization Using Hierarchal Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read WS-option jointly learns budget allocation and node selection for sequential stochastic combinatorial optimization, beating uniform-allocation baselines and transferring from 50-node training graphs to 1000-node graphs.

desk verdict WS-option is a genuine and sensible combination of HRL ideas for non-uniform budget allocation in sequential stochastic CO, but its 'significantly improved' claim rests on thin margins with unreported variance, and the theory section is mostly assertion. read the letter →

arxiv 2502.05537 v1 pith:NHZ25LOQ submitted 2025-02-08 cs.AI cs.LG

Xinsong Feng , Zihan Yu , Yanhai Xiong , Haipeng Chen This is my paper · ORCID
classification cs.AIcs.LG
keywords hierarchicalreinforcementlearningoptionframeworksequentialstochasticcombinatorialoptimizationbudgetallocationnodeselectionadaptiveinfluencemaximizationrouteplanningwake-sleeptraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper takes on a class of optimization problems it calls sequential stochastic combinatorial optimization: a fixed budget must be spread over a time horizon, and at each step a set of nodes or cities is chosen, with rewards that are random and partly revealed only as decisions unfold. It claims that the few existing reinforcement-learning treatments of these problems settle for a uniform budget split, which is suboptimal, and that the right structure is a two-layer hierarchy. The proposed WS-option framework learns a budget-allocation policy on the higher layer and a node-selection policy on the lower layer, with the two layers coupled through a shared time-step reward and stabilized by a wake-sleep training schedule. The payoff, if the claims hold, is that one agent trained cheaply on small graphs supplies both levels of the decision for large problem instances: the paper reports better cumulative rewards than hand-designed baselines on adaptive influence maximization and route planning, and successful use of a model trained on 50–100 node graphs at sizes up to 1000 nodes.

What carries the argument

The load-bearing object is the two-layer Markov decision process pair connected by a reward-alignment identity: at each time step the higher layer chooses a budget option $o^I_t = K_t$ and the lower layer selects nodes one at a time, and the lower-layer rewards are scaled marginal rewards $r^{II}_{t,v} = \frac{m_{t,v}}{\sum_{u \in a^{II}_t} m_{t,u}}(r^I_t - r^{II}_{t,\emptyset})$ so that $\sum_u r^{II}_{t,u} + r^{II}_{t,\emptyset} = r^I_t$; the layers therefore optimize exactly the same objective. Around this identity the paper builds three stabilising devices: a wake-sleep training schedule (sleep: freeze the higher layer at an average budget allocation and train the lower layer, then pre-train the higher layer offline on those trajectories; wake: train both layers jointly), a layer-wise learning-method split (Monte Carlo returns for the unbiased higher-layer Q-function, TD/Q-learning for the faster-converging lower layer), and a binary continuation option $o^{II}_t \in \{0,1\}$ with a null action, so the lower layer only needs to know whether to keep selecting nodes rather than the exact budget value. The network also uses an action-in structure for the higher layer so that budget size is an input, not an output layer, which is what lets a model trained on small graphs be evaluated on arbitrarily larger ones.

What would settle it

Train the lower-layer node-selection policy to convergence under several deliberately different budget-allocation policies—all budget on the first day, uniform across days, all budget on the last day—and measure how often the resulting policies choose different nodes in the same state. If they disagree substantially, the similarity assumption behind wake-sleep training is violated; conversely, if joint training from scratch reaches the same final reward as wake-sleep training on a hard instance, the schedule's claimed stabilising role is not essential to the result.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the SSCO problem—maximize $\sum_{t=1}^T r_t(S_t)$ subject to $\sum_t K_t \leq K$ and $|S_t| \leq K_t$—is a bi-level optimization, and that a hierarchical value-based agent can solve it stably instead of flattening it into one giant Markov decision process. WS-option treats the per-time-step budget $K_t$ as a higher-layer option and the per-step node set $S_t$ as a sequence of lower-layer actions, and the two layers are trained so that the lower layer's rewards at each time step sum exactly to the higher layer's reward, keeping both levels optimizing the same objective. The paper further claims that its wake-sleep schedule—first training the lower layer under a frozen average budget policy, then fine-tuning both layers together—avoids the divergent Q-value estimates it observes when the layers are trained jointly from scratch, and its two convergence theorems establish Q-learning convergence for each layer in the tabular case assuming the other layer is fixed or optimally responsive. The reported empirical result is that this agent outperforms the average, static, and normal budget policies with degree- or score-based node selection on adaptive influence maximization, and greedy and genetic-algorithm planners on route planning, with all reported differences at $p \leq 0.05$; it also reports that the learned policies generalize to graphs up to 1000 nodes and to a 2500-node real-world power-grid network.

Load-bearing premise

The whole wake-sleep design rests on the claim, stated explicitly in Appendix A.2, that the best node-selection policy is roughly the same no matter how the budget is spread across time steps: if that similarity fails, the lower-layer policy learned in the sleep stage is a poor starting point and nothing guarantees the two layers converge jointly.

Editorial extensions

If this is right

  • Uniform budget allocation over the time horizon is suboptimal for sequential stochastic combinatorial optimization; an adaptive budget policy learned jointly with node selection yields higher cumulative reward on adaptive influence maximization and route planning in the reported experiments ($p \leq 0.05$).
  • A model trained on graphs of 50–100 nodes transfers to unseen graphs of 200–1000 nodes and to a 2500-node power-grid network without retraining, because the learned networks contain no graph-size-dependent parameters.
  • The lower layer's node-selection policy and the higher layer's budget-allocation policy each contribute independently to the gain: holding one layer fixed at its learned policy while swapping the other for a baseline still favors the learned policy.
  • The wake-sleep schedule matters: an ablation shows that shortening the sleep stage so the lower layer has not converged (one third of the epochs) degrades final performance, while half and two thirds of the epochs give nearly identical results.
  • Using Monte Carlo returns for the higher layer instead of TD bootstrapping prevents the monotone Q-value inflation the paper observes for TD in this coupled setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference — the reward-alignment identity is a generic credit-assignment recipe: any hierarchical agent that must distribute one aggregate time-step reward across many primitive actions could reuse it, with the marginal contribution of each action estimated by simulation, not just in the two problem instances studied here.
  • Editorial inference — the paper's convergence proof is explicitly limited to tabular Q-learning, so an untested question is whether the wake-sleep schedule keeps the function-approximation case stable in regimes beyond the reported $T \le 20$, $K \le 30$ settings, such as very long horizons or budgets large relative to the graph.
  • Editorial inference — the comparison set does not include a flat RL baseline that learns budget allocation and node selection as a single joint MDP; testing whether the hierarchy is genuinely necessary rather than merely convenient is an open comparison the paper does not run.
  • Editorial inference — the paper's own stated limitation is that each new problem class needs its own graph embedding technique; a testable consequence of its design is that the wake-sleep schedule and the reward-alignment identity, rather than the specific embedding, are what carry the performance gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper defines sequential stochastic combinatorial optimization (SSCO) as a two-level problem in which a budget is allocated over time steps and a set of nodes is selected at each step. It proposes WS-option, a two-layer option-based hierarchical RL framework, with a wake-sleep training procedure, MC learning for the higher layer, TD learning for the lower layer, and brief convergence theorems. The method is evaluated on adaptive influence maximization and route planning, with ablation studies and tests on larger graphs and a real-world network. The central claim is that WS-option exhibits significantly improved effectiveness and generalizability over traditional methods.

Significance. The problem formulation is a useful and reasonably general way to expose the budget-allocation and node-selection coupling in SSCO, and the option-based hierarchical decomposition is a sensible design. The wake-sleep idea, the use of different learning updates per layer, and the ablations for sleep length and simplified options are worthwhile contributions. If the empirical claim were fully supported, the generalization to larger graphs would be a practical benefit. However, the current evidence for 'significantly improved' is incomplete: the reported margins over the strongest baselines are small, and the paper never gives the run counts, variances, test statistics, or actual p-values behind the blanket 'p-values ≤ 0.05' statements. The theoretical guarantees are also narrower and more assumption-dependent than the text suggests.

major comments (4)
  1. [§4.1 and Tables 1–4, 7–8] The central empirical claim is not verifiable from the reported numbers. The paper states 'All cases have p-values ≤ 0.05' but never reports the number of independent runs/seeds, standard errors, confidence intervals, test statistics, or the p-values themselves. This matters because the effect sizes are small: in Table 1 the largest advantage over the best score-based baseline is 1.19 out of 74.81 (about 1.6%), and in Table 4 at n=1000 the difference is 221.16 vs 220.81 (about 0.16%). Without per-run statistics, the reader cannot tell whether these differences are within stochastic noise, so the abstract's 'significantly improved effectiveness and generalizability' claim is unsupported as written. The authors should report the number of independent trials, the variance, and the p-values or confidence intervals for at least all headline tables.
  2. [§3.2.2, Eq. (4)] The lower-layer reward defined in Eq. (4) is not a well-posed per-action reward for the sequential decision process. The reward r^II_{t,v} = m_{t,v} / (Σ_{u∈a^II_t} m_{t,u}) · (r^I_t − r^II_{t,∅}) depends on the full lower-layer action set a^II_t and on the realized total reward r^I_t, both of which are known only after the complete node sequence for time step t has been executed. When Algorithm 2 selects nodes one by one (lines 8–13), the reward retrospectively assigned to an earlier node therefore depends on future choices and on the stochastic outcome of the whole step. The lower-layer transition is then not a Markov transition of the sort assumed in Theorem 1, and the contraction proof in Appendix A.5.1 does not apply to the algorithm as implemented. The sentence in §3.3 that scaling 'does not affect the relative Q value of the actions' is also not justified, because the normalizer varies with the action set and can change the ordering of Q-targets. The authors should either define a causal reward, or explicitly treat Eq. (4) as heuristic reward shaping and remove the convergence guarantee that relies on it.
  3. [§3.3.3, Theorem 2, and Appendix A.2] The joint convergence claim is not established. Theorem 2 assumes that for every higher-layer policy the lower-layer policy provides the conditionally optimal response, which is precisely the property that the wake-sleep procedure is supposed to deliver. Appendix A.2 adds a further assertion, 'the conditionally optimal lower-layer policy remains similar across different higher-layer policies πI', with no proof and no direct test. The wake-sleep design is load-bearing on this assumption: during the sleep stage the lower layer is trained under the average budget allocation, and during the wake stage it is co-trained with a learned higher-layer policy. If the conditionally optimal lower-layer policies are not similar across high-layer policies, the sleep-stage solution may be a poor initialization and the joint training may not converge. Table 9 only varies sleep duration; it does not test the similarity assumption. The framing of Figure 3 as a Stackelberg game is an analogy, not a proof. I recommend either proving the assumption under explicit conditions, testing it by training the lower layer to convergence under several fixed high-layer policies and comparing the resulting policies, or substantially weakening the convergence claim to match what is actually shown.
  4. [§4.3, Table 3 and Tables 9–10] The policy-assessment and ablation tables are reported as single point estimates. Table 9 explicitly says 'we use the same random seed as in 4.2', which suggests that at least some results come from a single seed. If that is true, the t-test language in Section 4.1 is internally inconsistent: a two-sample t-test requires a sample of runs, not a single trajectory. The authors should clarify how many seeds underlie each table and, if the ablations are single-seed, label them as illustrative rather than as significance-tested results.
minor comments (5)
  1. [Title and abstract] The title contains a spelling error: 'Hierarchal' should be 'Hierarchical'; the abstract also has 'opitmization' instead of 'optimization' in one place.
  2. [Table 1] In the row 'normal-degree', the entry for (T,K=10,20) reads '101.50.34', which appears to be a typo for either 101.50 or 101.34; please correct it.
  3. [Throughout] There are several typographical errors, including 'transistions' in §3.3, 'repreat' in Appendix A.4.1, 'effiency' in Appendix A.1, and 'Vonference' in reference [22]. A careful proofreading pass is needed.
  4. [Appendix C.5] The t-test rationale states that standard deviation mainly reflects aleatoric uncertainty, but a t-test still requires multiple independent runs to estimate the sampling distribution. The appendix should state the number of runs, the type of t-test (paired or unpaired), and how the stochastic environment is sampled across runs.
  5. [Figure 2] The two panels in Figure 2(b) are hard to interpret because the MC and TD plots are shown as separate small curves with no shared axis labels or legend; please provide full axis labels and a single linked figure or a table of the Q-value trajectories.

Circularity Check

2 steps flagged · score 5.0 of 10

Empirical comparisons are benchmarked externally and are not circular, but the convergence analysis that motivates the wake-sleep design reduces to its own assumptions: Theorem 2 assumes the conditionally optimal lower-layer policy, and Theorem 1's proof equates qI with max qII.

  1. self definitional [Theorem 2 / Appendix A.5.2 (Section 3.3.3)]
    "Theorem 2. ... converges to the optimal Q-value function qI∗(st, ot) with probability 1, assuming that for any given higher-layer policy, the lower-layer policy always provides the corresponding conditionally optimal response. ... Under the assumption ... the lower-layer policy remains fixed and conditionally optimal at each time step t. ... Consequently, the expected value of the observed return gt matches the Q-value estimate qI_t(st, ot)."

    In the paper's own bi-level reformulation (Appendix A.1, Eqs. 5-6), the inner problem is πII*(πI) ∈ arg max J(πI, πII); a conditionally optimal lower-layer policy is exactly the object the hierarchical training is supposed to produce. Theorem 2 assumes that object exists for every higher-layer policy and then concludes qI converges to qI*. Appendix A.5.3 then 'combines' the two theorems to assert global convergence, so the joint convergence of the framework is assumed, not derived. This is load-bearing because Section 3.3.1 says the wake-sleep procedure is 'well-aligned with our theoretical analysis.'

  2. self definitional [Appendix A.5.1, proof of Theorem 1, Eqs. (10)-(11)]
    "Note that qI(st+1, ot+1) = max_a qII(st+1, ot+1, a). Therefore, we have ..."

    This equality identifies the higher-layer option value with the optimal lower-layer action value, which is precisely the cross-layer optimality/consistency that Theorem 1 is meant to establish. If the equality is taken as a definition, then qI is defined in terms of qII* and the contraction argument presupposes the optimality it claims to prove; if it is not a definition, it is an unproved assertion of the very interdependency the paper identifies as the source of training instability. Either way, the proof reduces to its own target.

full rationale

The paper's central empirical claim — that WS-option outperforms the listed baselines on adaptive influence maximization and route planning — is benchmarked against external, non-self-citation baselines (Tables 1-4, 7-8, e.g., average-score, static-score, greedy, GA), so that part of the evaluation is not circular. The weak statistical reporting (no per-run standard deviations, seed counts, or reported p-values despite 'All cases have p-values ≤ 0.05') is a verifiability and correctness concern, not a circularity concern. The circularity burden lies in the theoretical convergence analysis that motivates the wake-sleep training design. Theorem 2's assumption is the conditionally optimal lower-layer response, which is exactly the inner optimization of the paper's bi-level reformulation, so the option-policy convergence is assumed rather than derived. Theorem 1's proof further asserts qI = max_a qII, an identity that either defines qI through qII* or assumes the cross-layer consistency the theorem is meant to prove. Appendix A.5.3 then 'combines' the two theorems to conclude global convergence; the combination does not prove that the iterative wake-sleep process reaches the assumed conditionally optimal response. The empirical contribution remains substantial and independent, so the paper is not wholly circular, but the formal support for the core training mechanism is partially circular.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The framework's effectiveness rests primarily on two unproven modeling assumptions: the transferability of the lower-layer policy across budget policies, and the validity of the normalized marginal reward as a per-action reward. The problem instances introduce several ad hoc constants. No genuinely new entities are postulated.

free parameters (3)
  • Sleep-stage duration = N/2 (10 of 20 epochs)
    The wake-sleep split is a hand-chosen training hyperparameter; the ablation in Appendix D.5 shows that N/3 degrades the AIM result from 118.56 to 104.84, so the choice matters.
  • Number of simulations for marginal reward estimation = 10
    Algorithm 3 uses 10 Monte Carlo simulations to estimate marginal rewards; no sensitivity analysis is provided.
  • Route planning constants = eta=0.1, penalty=5, dmax=1.5*max distance
    Ad hoc constants defining the RP instance in Appendix C.3; they shape the problem and the reported numbers.
assumptions (3)
  • ad hoc to paper Conditionally optimal lower-layer policies are similar across different higher-layer budget allocation policies.
    Stated in Appendix A.2 to justify the wake-sleep training; not proven. In the sleep stage the lower layer is trained against an average budget policy, and the success of the wake stage depends on this transfer.
  • ad hoc to paper The normalized marginal reward in Eq. (4) defines a valid per-action reward for the lower-layer MDP.
    The reward for selecting node v is scaled by the total marginal reward of the final set aII_t (Algorithm 3, lines 11-14), so it depends on future selections and is not a stationary function of the current state-action pair. The paper assumes this preserves the optimal node-selection policy (Appendix A.4.2).
  • domain assumption Independent Cascade model with activation probability p(u,v)=1/d^-(v).
    Used to define the adaptive influence maximization instance (Appendix C.2); this is a standard model choice from the IM literature, adopted rather than derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sequential Stochastic Combinatorial Optimization Using Hierarchal Reinforcement Learning." pith.science (2026). https://pith.science/paper/NHZ25LOQ

@misc{pith2026250205537,
  author       = {Pith},
  title        = {Pith review of: Sequential Stochastic Combinatorial Optimization Using Hierarchal Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NHZ25LOQ}},
  note         = {Machine review of arXiv:2502.05537}
}
read the original abstract

Reinforcement learning (RL) has emerged as a promising tool for combinatorial optimization (CO) problems due to its ability to learn fast, effective, and generalizable solutions. Nonetheless, existing works mostly focus on one-shot deterministic CO, while sequential stochastic CO (SSCO) has rarely been studied despite its broad applications such as adaptive influence maximization (IM) and infectious disease intervention. In this paper, we study the SSCO problem where we first decide the budget (e.g., number of seed nodes in adaptive IM) allocation for all time steps, and then select a set of nodes for each time step. The few existing studies on SSCO simplify the problems by assuming a uniformly distributed budget allocation over the time horizon, yielding suboptimal solutions. We propose a generic hierarchical RL (HRL) framework called wake-sleep option (WS-option), a two-layer option-based framework that simultaneously decides adaptive budget allocation on the higher layer and node selection on the lower layer. WS-option starts with a coherent formulation of the two-layer Markov decision processes (MDPs), capturing the interdependencies between the two layers of decisions. Building on this, WS-option employs several innovative designs to balance the model's training stability and computational efficiency, preventing the vicious cyclic interference issue between the two layers. Empirical results show that WS-option exhibits significantly improved effectiveness and generalizability compared to traditional methods. Moreover, the learned model can be generalized to larger graphs, which significantly reduces the overhead of computational resources.

Figures

Figures reproduced from arXiv: 2502.05537 by the authors.

Figure 1
Figure 1. Hierarchical MDPs for SSCO 4 [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Q-values learned for the AIM problem (T = 10, K = 20). option policy and give Theorem 1. Then, we show the convergence of the option policy and give the Theorem 2. The proofs are provided in Appendix A.5.1 and Appendix A.5.2, respectively. Theorem 1. (Intra-option policy convergence). In our WS-option framework, given any Markov transition (sτ , oτ , aτ , rτ , sτ+1, oτ+1), the Q-value function q II (sτ , oτ , aτ ) c… view at source ↗
Figure 3
Figure 3. Wake-sleep training procedure 4 EXPERIMENTAL RESULTS 4.1 SSCO PROBLEM INSTANCES In this article, we examine two broad classes of problems: the propagation problem (high stochasticity) and the route planning problem (low stochasticity). For the propagation problem, we use an adaptive version of the classic IM problem. For the route planning (RP) problem, we consider a model tailored for travel route planning. Notably… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Network architecture The network architecture consists of two primary components: the higher layer network and the lower layer network, as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]
Figure 5
Figure 5. Figure 5: Cumulative reward during training for AIM [PITH_FULL_IMAGE:figures/full_fig_p025_5.png]
Figure 6
Figure 6. Figure 6: Budget allocation for T = 10, K = 10 1 2 3 4 5 6 7 Time Steps 0 1 2 3 4 5 6 7 8 9 Budget Allocation Policy 1 (Reward: 83.9) Policy 2 (Reward: 99.29) Policy 3 (Reward: 97.29) Policy 4 (Reward: 51.51) Average Reward: 127.51 [PITH_FULL_IMAGE:figures/full_fig_p025_6.png]
Figure 7
Figure 7. Figure 7: Budget allocation for T = 10, K = 20 [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]
Figure 8
Figure 8. Figure 8: Budget allocation for T = 10, K = 30 D.3 GENERALIZATION TO LARGER GRAPHS D.4 EVALUATION ON THE REAL-WORLD DATA Real-world datasets often exhibit certain patterns, such as clustering characteristics. These patterns are easier for our RL-based algorithm to recognize, whi…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 47 canonical work pages

  1. [1]

    Hindsight experience replay

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in Neural Information Processing Systems, 30, 2017

    work page 2017

  2. [2]

    Modern graph neural networks do worse than classical greedy algorithms in solving combinatorial optimization problems like maximum independent set

    Maria Chiara Angelini and Federico Ricci-Tersenghi. Modern graph neural networks do worse than classical greedy algorithms in solving combinatorial optimization problems like maximum independent set. Nature Machine Intelligence, 5(1):29–31, 2023

    2023

  3. [3]

    The option-critic architecture

    Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference on Artificial Intelligence, volume 31, 2017

    work page 2017

  4. [4]

    Neural combina- torial optimization with reinforcement learning

    Irwan Bello, Hieu Pham, Quoc V Le, Mohammad Norouzi, and Samy Bengio. Neural combina- torial optimization with reinforcement learning. arXiv preprint arXiv:1611.09940, 2016

    arXiv 2016

  5. [5]

    Machine learning for combinatorial optimization: A methodological tour d’horizon

    Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial optimization: A methodological tour d’horizon. European Journal of Operational Research, 2020

    work page 2020

  6. [6]

    Rl4co: an extensive reinforcement learning for combina- torial optimization benchmark

    Federico Berto, Chuanbo Hua, Junyoung Park, Minsu Kim, Hyeonah Kim, Jiwoo Son, Haeyeon Kim, Joungho Kim, and Jinkyoo Park. Rl4co: an extensive reinforcement learning for combina- torial optimization benchmark. arXiv preprint arXiv:2306.17100, 2023

    arXiv 2023

  7. [7]

    Models and algorithms for combinatorial optimization problems arising in railway applications

    Valentina Cacchiani. Models and algorithms for combinatorial optimization problems arising in railway applications. 4OR, 7:109–112, 2009

    work page 2009

  8. [8]

    Applying gis and combinatorial optimization to fiber deployment plans

    Buyang Cao, Minghe Sun, and Charles Macleod. Applying gis and combinatorial optimization to fiber deployment plans. Journal of Heuristics, 5:385–402, 1999

    work page 1999

Show all 58 references
  1. [9]

    Improving optimization bounds using machine learning: Decision diagrams meet deep reinforcement learning

    Quentin Cappart, Emmanuel Goutierre, David Bergman, and Louis-Martin Rousseau. Improving optimization bounds using machine learning: Decision diagrams meet deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pp. 1443–1451, 2019

    2019

  2. [10]

    Contingency-aware influence maximization: A reinforcement learning approach

    Haipeng Chen, Wei Qiu, Han-Ching Ou, Bo An, and Milind Tambe. Contingency-aware influence maximization: A reinforcement learning approach. In Uncertainty in Artificial Intelligence, pp. 1535–1545. PMLR, 2021

    2021

  3. [11]

    Learning to perform local rewriting for combinatorial optimization

    Xinyun Chen and Yuandong Tian. Learning to perform local rewriting for combinatorial optimization. Advances in Neural Information Processing Systems, 32, 2019

    2019

  4. [12]

    Discriminative embeddings of latent variable models for structured data

    Hanjun Dai, Bo Dai, and Le Song. Discriminative embeddings of latent variable models for structured data. In International Conference on Machine Learning, pp. 2702–2711. PMLR, 2016

    2016

  5. [13]

    Feudal reinforcement learning

    Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. Advances in Neural Information Processing Systems, 5, 1992

    1992

  6. [14]

    Learning heuristics for the tsp by policy gradient

    Michel Deudon, Pierre Cournut, Alexandre Lacoste, Yossiri Adulyasak, and Louis-Martin Rousseau. Learning heuristics for the tsp by policy gradient. In Integration of Constraint Programming, Artificial Intelligence, and Operations Research: 15th International Conference, CPAIOR...

    2018

  7. [15]

    Patrick Emami and S. Ranka. Learning Permutations with Sinkhorn Policy Gradient. ArXiv, abs/1805.07010, 2018. 11 Published as a conference paper at ICLR 2025

    2018 arXiv

  8. [16]

    Generalize a small pre-trained model to arbitrarily large tsp instances

    Zhang-Hua Fu, Kai-Bin Qiu, and Hongyuan Zha. Generalize a small pre-trained model to arbitrarily large tsp instances. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7474–7482, 2021

    2021

  9. [17]

    Deep sensitivity analysis for objective-oriented combinatorial optimiza- tion

    Ganga Gireesan, Nisha Pillai, Michael J Rothrock, Bindu Nanduri, Zhiqian Chen, and Ma- halingam Ramkumar. Deep sensitivity analysis for objective-oriented combinatorial optimiza- tion. arXiv preprint arXiv:2403.00016, 2024

    2024 arXiv

  10. [18]

    node2vec: Scalable feature learning for networks

    Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 855–864, 2016

    2016

  11. [19]

    Double q-learning

    Hado Hasselt. Double q-learning. Advances in Neural Information Processing Systems, 23, 2010

    2010

  12. [20]

    Efficient active search for combinatorial optimization problems

    Andr´e Hottung, Yeong-Dae Kwon, and Kevin Tierney. Efficient active search for combinatorial optimization problems. In International Conference on Learning Representations, 2021

    2021

  13. [21]

    Convergence of stochastic iterative dynamic programming algorithms

    Tommi Jaakkola, Michael Jordan, and Satinder Singh. Convergence of stochastic iterative dynamic programming algorithms. Advances in neural information processing systems, 6, 1993

    1993

  14. [22]

    Deep reinforcement learning approach to solve dynamic vehi- cle routing problem with stochastic customers

    Waldy Joe and Hoong Chuin Lau. Deep reinforcement learning approach to solve dynamic vehi- cle routing problem with stochastic customers. In Proceedings of the International Vonference on Automated Planning and Scheduling, volume 30, pp. 394–402, 2020

    2020

  15. [23]

    Maximizing the spread of influence through a social network

    David Kempe, Jon Kleinberg, and ´Eva Tardos. Maximizing the spread of influence through a social network. In Proceedings of the ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 137–146, 2003

    2003

  16. [24]

    Learning combinatorial optimization algorithms over graphs

    Elias Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning combinatorial optimization algorithms over graphs. Advances in Neural Information Processing Systems, 30, 2017

    2017

  17. [25]

    Semi-supervised classification with graph convolutional networks

    Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2016

    2016

  18. [26]

    Attention, learn to solve routing problems! In International Conference on Learning Representations, 2018

    Wouter Kool, Herke van Hoof, and Max Welling. Attention, learn to solve routing problems! In International Conference on Learning Representations, 2018

    2018

  19. [27]

    Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation

    Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in Neural Information Processing Systems, 29, 2016

    2016

  20. [28]

    Pomo: Policy optimization with multiple optima for reinforcement learning

    Yeong-Dae Kwon, Jinho Choo, Byoungjip Kim, Iljoo Yoon, Youngjune Gwon, and Seungjai Min. Pomo: Policy optimization with multiple optima for reinforcement learning. Advances in Neural Information Processing Systems, 33:21188–21198, 2020

    2020

  21. [29]

    Mind dataset for diet planning and dietary healthcare with machine learning: dataset creation using combinatorial optimization and controllable generation with domain experts

    Changhun Lee, Soohyeok Kim, Sehwa Jeong, Chiehyeon Lim, Jayun Kim, Yeji Kim, and Minyoung Jung. Mind dataset for diet planning and dietary healthcare with machine learning: dataset creation using combinatorial optimization and controllable generation with domain experts. In Th...

    2021

  22. [30]

    Learning multi-level hierar- chies with hindsight

    Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierar- chies with hindsight. In International Conference on Learning Representations, 2018

    2018

  23. [31]

    Deeper Insights Into Graph Convolutional Networks for Semi-Supervised Learning

    Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper Insights Into Graph Convolutional Networks for Semi-Supervised Learning. In AAAI Conference on Artificial Intelligence (AAAI), pp. 3538–3545, 2018

    2018

  24. [32]

    Deep-learning-based wireless resource allocation with application to vehicular networks

    Le Liang, Hao Ye, Guanding Yu, and Geoffrey Ye Li. Deep-learning-based wireless resource allocation with application to vehicular networks. Proceedings of the IEEE, 108(2):341–356, 2019. 12 Published as a conference paper at ICLR 2025

    2019

  25. [33]

    Towards graph foundation models: A survey and beyond

    Jiawei Liu, Cheng Yang, Zhiyuan Lu, Junze Chen, Yibo Li, Mengmei Zhang, Ting Bai, Yuan Fang, Lichao Sun, Philip S Yu, et al. Towards graph foundation models: A survey and beyond. arXiv preprint arXiv:2310.11829, 2023

    2023 arXiv

  26. [34]

    A learning-based iterative method for solving vehicle routing problems

    Hao Lu, Xingwen Zhang, and Shuang Yang. A learning-based iterative method for solving vehicle routing problems. In International Conference on Learning Representations, 2019

    2019

  27. [35]

    Reinforcement learning for combinatorial optimization: A survey

    Nina Mazyavkina, Sergey Sviridov, Sergei Ivanov, and Evgeny Burnaev. Reinforcement learning for combinatorial optimization: A survey. Computers & Operations Research, 134:105400, 2021

    2021

  28. [36]

    Playing atari with deep reinforcement learning

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013

    2013 arXiv

  29. [37]

    Human-level control through deep reinforcement learning

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015

    2015

  30. [38]

    Data-efficient hierarchical reinforcement learning

    Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018

    2018

  31. [39]

    Reinforcement learning for solving the vehicle routing problem

    Mohammadreza Nazari, Afshin Oroojlooy, Lawrence Snyder, and Martin Tak´ac. Reinforcement learning for solving the vehicle routing problem. Advances in Neural Information Processing Systems, 31, 2018

    2018

  32. [40]

    Solo: search online, learn offline for combinatorial optimization problems

    Joel Oren, Chana Ross, Maksym Lefarov, Felix Richter, Ayal Taitler, Zohar Feldman, Dotan Di Castro, and Christian Daniel. Solo: search online, learn offline for combinatorial optimization problems. In Proceedings of the International Symposium on Combinatorial Search, volume 1...

    2021

  33. [41]

    Active screening for recurrent diseases: A reinforcement learning approach

    Han-Ching Ou, Haipeng Chen, Shahin Jabbari, and Milind Tambe. Active screening for recurrent diseases: A reinforcement learning approach. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 992–1000, 2021

    2021

  34. [42]

    Adaptive influence maximization with myopic feedback.Advances in Neural Information Processing Systems, 32, 2019

    Binghui Peng and Wei Chen. Adaptive influence maximization with myopic feedback.Advances in Neural Information Processing Systems, 32, 2019

    2019

  35. [43]

    Deepwalk: Online learning of social repre- sentations

    Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 701–710, 2014

    2014

  36. [44]

    Scarselli, M

    F. Scarselli, M. Gori, Ah Chung Tsoi, M. Hagenbuchner, and G. Monfardini. The Graph Neural Network Model. IEEE Transactions on Neural Networks, 20(1):61–80, 2009

    2009

  37. [45]

    Combinatorial optimization with physics-inspired graph neural networks

    Martin JA Schuetz, J Kyle Brubaker, and Helmut G Katzgraber. Combinatorial optimization with physics-inspired graph neural networks. Nature Machine Intelligence, 4(4):367–377, 2022

    2022

  38. [46]

    Learning to predict by the methods of temporal differences

    Richard S Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3:9–44, 1988

    1988

  39. [47]

    Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning

    Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2): 181–211, 1999

    1999

  40. [48]

    Time-constrained adaptive influence maximization

    Guangmo Tong, Ruiqi Wang, Zheng Dong, and Xiang Li. Time-constrained adaptive influence maximization. IEEE Transactions on Computational Social Systems, 8(1):33–44, 2020

    2020

  41. [49]

    Graph attention networks

    Petar Veliˇckovi´c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Li`o, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018

    2018

  42. [50]

    Feudal networks for hierarchical reinforcement learning

    Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, pp. 3540–3549. PMLR, 2017. 13 Published as a con...

    2017

  43. [51]

    Q-learning

    Christopher JCH Watkins and Peter Dayan. Q-learning. Machine Learning, 8:279–292, 1992

    1992

  44. [52]

    On efficiency in hierarchical reinforcement learning

    Zheng Wen, Doina Precup, Morteza Ibrahimi, Andre Barreto, Benjamin Van Roy, and Satinder Singh. On efficiency in hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 33:6708–6718, 2020

    2020

  45. [53]

    Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A Comprehensive Survey on Graph Neural Networks. IEEE Transactions on Neural Networks and Learning Systems, 32(1):4–24, 2021

    2021

  46. [54]

    How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018

    Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018

    2018 arXiv

  47. [55]

    Accelerating Exact Combinatorial Optimization via RL-based Ini- tialization - A Case Study in Scheduling

    Jiaqi Yin and Cunxi Yu. Accelerating Exact Combinatorial Optimization via RL-based Ini- tialization - A Case Study in Scheduling. In 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD). IEEE, 2023

    2023

  48. [56]

    Gnnexplainer: Generating explanations for graph neural networks

    Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Generating explanations for graph neural networks. Advances in Neural Information Processing Systems, 32, 2019

    2019

  49. [57]

    Graph transformer networks

    Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph transformer networks. Advances in Neural Information Processing Systems, 32, 2019

    2019

  50. [58]

    Graph neural networks: A review of methods and applications

    Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI Open, 1:57–81, 2020. 14 Published as a conference paper at ICLR 2025 CONTENTS 1 Introduction...

    2020

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.