Pith. sign in

REVIEW 5 major objections 5 minor 42 references

HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising

T0 review · 5 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HOBA claims that online advertising bidding can be made safer and more adaptive by restricting online learning to discrete selection among pre-trained expert models, and reports a +3.6% target-cost improvement in a large-scale production A/

desk verdict The three-tier architecture is genuinely new, but the evaluation protocol is cherry-picked, the constraint guarantee is overstated, and the causal adjustment mechanism is not grounded as written. read the letter →

arxiv 2607.24779 v1 pith:AEM6PWRS submitted 2026-06-17 cs.AI cs.LG

Ji Wu , Yunshan Peng , Wentao Bai , Yunke Bai , Wenzheng Shu , Jinan Pang , Yanxiang Zeng , Xialong Liu This is my paper
classification cs.AIcs.LG
keywords hierarchicalreinforcementlearningonlineadvertisingautomaticbiddingexpertselectionSARSAcausaladjustmentlargelanguagemodelagentA/Btesting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the safety-adaptability tradeoff in real-time bidding is resolvable by hierarchy. Instead of learning continuous bid values online, HOBA learns only which pre-trained bidding expert to activate, while a large language model sets strategic hyperparameters on an hourly cycle and a pool of fixed offline experts executes per-auction bids. The authors argue this confines online exploration to a small discrete action space, making adaptation safe enough for production. They report consistent offline gains of roughly +3% to +12% over baselines and a live A/B test with +3.6% higher target-cost achievement and +8.1% more conversion value. If correct, the paradigm offers a deployable answer to the standing conflict between adapting to non-stationary auctions and avoiding budget-exhausting exploration.

What carries the argument

The central object is the three-tier hierarchy with distinct decision cycles: hourly strategic reasoning, two-minute tactical selection, and per-auction execution. The load-bearing mechanism is the mid-level agent, a linear SARSA (State-Action-Reward-State-Action) Q-function with polynomial features and UCB exploration that selects among a pool of five to ten fixed offline experts; its training signal is augmented by a doubly robust estimator (Eq. 12) that combines an outcome model with inverse propensity weights, intended to purge selection bias from logged historical data. The hierarchy's safety guarantee is enforced by a double-clipping rule (Eq. 15) that caps every bid inside the LLM-set

What would settle it

Inspect the mid-level logs after warmup: for each logged (state, action) pair, compute the actual selection probability under Eq. (13) given the recorded state and visit counts. If the fitted logistic propensity model disagrees materially with those actual probabilities—for example, assigning 0.3 to an action that was chosen with probability 1—then the doubly robust estimator is correcting against the wrong logging policy. A simpler decisive test is to rerun the causal-adjustment ablation with propensities fixed to 1 for the chosen action and 0 otherwise: if performance still changes by roughl

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that a three-tier agent hierarchy—strategic hyperparameter inference by a large language model with a Think-Act-Observe-Reflect loop and memory retrieval, tactical expert selection by an on-policy SARSA agent with doubly robust causal adjustment, and low-level bid execution by a fixed pool of offline experts—outperforms both monolithic RL policies and manually tuned controllers. The online learning is deliberately confined to the mid-level choice among a handful of pre-validated experts, so no bid-generating weights change during deployment. The paper attributes the online gains to this decoupling: the mid-level selection adapts to market shifts, th

Load-bearing premise

The causal debiasing in the mid-level agent assumes the logged expert selections came from a stochastic logging policy whose propensities can be estimated, but after the 1000-step warmup the system selects experts deterministically, so the estimated propensities do not reflect the true logging process.

Editorial extensions

If this is right

  • If the central claim holds, bidding systems can adapt to non-stationary auction markets by switching among a handful of pre-validated experts, without ever updating bid-generating weights during deployment.
  • The reported production numbers (+3.6% target-cost achievement, +8.1% conversion value, +3.8% budget completion) imply that constraint compliance and revenue can improve together rather than trade off.
  • The ablation results make expert selection the hinge: removing SARSA selection costs 10.7%, removing the LLM costs 7.1%, and removing causal adjustment costs 4.9%.
  • Since the low-level experts stay fixed unless dynamically fine-tuned, the framework's online adaptability is bounded by the diversity of the expert pool; the paper's dynamic expansion and pruning mechanisms are what extend that bound.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The hierarchy's separation of time scales suggests a general design rule: place online learning at the coarsest decision granularity that can still react to regime change. HOBA's choice of the two-minute expert switch as the only online decision is one point on that spectrum; a similar decomposition could apply to dynamic pricing or cloud resource arbitration.
  • A testable extension would vary the diversity of the initial expert pool and measure whether HOBA's offline gains shrink when the pool is homogeneous. The paper's ablation (only -2.5% without dynamic experts) suggests most of the value comes from having good static experts rather than from the dynamic expansion machinery.
  • Because the LLM only sets hard-clipped bounds, the marginal value of the language model is capped by the width of those bounds; one could ablate by widening the bounds to see how much headroom the mid-level selection can exploit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes HOBA, a three-tier hierarchical bidding framework for online advertising. A high-level LLM agent (hourly cycle) infers hyperparameters via a Think-Act-Observe-Reflect loop with memory retrieval; a mid-level SARSA agent (2-minute cycle) selects among a pool of pre-trained experts using a doubly robust causal adjustment; a low-level pool (PID, MPC, CQL, IQL, DT) executes per-auction bids. The authors claim that confining online learning to discrete expert selection rather than continuous bid optimization reduces exploration risk while maintaining adaptability. They report consistent offline improvements over baselines on AuctionNet (Table 3: +3.0% to +12.1%) and an online A/B test (Table 5: +3.6% target cost achievement, +8.1% conversion value), and interpret these as validating the hierarchical multi-agent paradigm.

Significance. If the empirical claims are correct, HOBA is an interesting engineering contribution: the three-tier decomposition is a plausible way to bring online adaptation to production bidding while preserving safety, and the phased online A/B test against a strong incumbent system is a genuine strength. The paper also makes a clear falsifiable prediction that discrete expert selection suffices to capture most of the available adaptation gains. However, the current evidence is undermined by a nonstandard evaluation statistic (mean of top-5 of 30 trials), an ungrounded causal-adjustment mechanism that is a stated contribution, and an over-strong safety guarantee for the clipping procedure. These issues are load-bearing for the paper's central claims and must be resolved before the results can be accepted.

major comments (5)
  1. [Section 5.1.2 and Table 3] The evaluation protocol reports 'the mean of the top-5 outcomes' from 30 independent trials. This is a cherry-picked statistic: the expected value of the top-5 mean exceeds the true mean, and the reported standard deviations and p-values are not valid for a selected subset. The claim of consistent improvements (+3.0% to +12.1%) therefore is not supported by the numbers as reported. Table 3's caption also says 'Mean score over 5 runs', contradicting Section 5.1.2. The authors should report the mean and standard deviation over all 30 trials, or all seeds, and re-compute significance. If the top-5 protocol is retained, a distributional justification and the full distribution must be provided.
  2. [Section 4.2.1, Eqs. (12)-(14)] The causal adjustment is not grounded as presented. After the 1000 warmup iterations, Eq. (13) selects actions by argmax over Q + mu + UCB, which is deterministic given the state and counts; the true logging propensity is therefore 1 for the selected action and 0 for all others. The logistic-regression propensity model trained on such data is misspecified, and clipping π̂(a|x) ≥ 0.05 can create a 20x upweight for chosen-action residuals. Even if the propensities were correct, Eq. (12) produces a scalar μ(a) that is added to Q in Eq. (13), while the SARSA update in Eq. (14) fits Q to the raw reward r_τ. A state-independent additive offset cannot remove state-dependent selection bias from the Q-function. Thus the claim that the doubly robust estimator 'eliminates selection bias' is unsupported, and the 4.9% ablation gain in Table 4 ('w/o Causal Adjustment') cannot be attributed to debiasin
  3. [Section 4 and Eq. (15)] The paper claims that double-clipping 'guarantees every bid satisfies Eq. (1)'. This is false. Eq. (1) imposes cumulative constraints on total budget B_i and cost-per-value C_i over all auctions; clipping a single bid to [q_lower*CPA_target, q_upper*CPA_target] and then to b_{τ-1}±Δ does not ensure those sums are satisfied. In addition, the outer clip can move the bid outside the inner bounds when the previous-bid interval does not overlap the bound interval. The guarantee should be rephrased as a per-bid bound, and a separate mechanism for satisfying the campaign-level constraints should be described.
  4. [Section 5.5, Table 5 and Abstract] The abstract states that the online deployment 'proving the effectiveness' of the hierarchical paradigm, but the quantitative evidence in Table 5 comes from Phase 2 (20% of traffic, 4,982 campaigns), while Phases 1 and 3 are described only qualitatively. The 'proving' language is too strong, and the paper should report exact p-values, confidence intervals, and Phase 3 results. This matters because the online A/B test is the most direct evidence for the central claim, and the current presentation does not allow the reader to assess the precision of the +3.6% and +8.1% estimates.
  5. [Table 4 and Section 5.3] The ablation results inherit the questionable top-5 protocol and are not interpreted cautiously. In particular, the row 'w/o Causal Adjustment' removes μ(a) from Eq. (13), which changes the action-selection score even if μ is not a valid causal correction. The observed 4.9% drop could be due to losing a per-action offset rather than to removing bias. Similarly, the 'SARSA→DQN' comparison is described as validating on-policy stability, but the reported switching rates and Q-value overestimation are not tied to a formal metric. The ablation section should be rewritten to state precisely what each ablation changes and what conclusion can be drawn.
minor comments (5)
  1. [Section 4.1.2 and Appendix A] The sentence 'stochasticity in the LLM reasoning trace does not propagate to bid execution' is misleading: the LLM outputs bid bounds, pacing rate, and exploration bonus, which directly affect bid execution. Hard clipping only keeps these values in range; it does not eliminate their influence.
  2. [Section 2.2, reference [38]] Reference [38] is cited for 'budget allocation across campaigns or coordination among multiple advertisers', but it appears to be a paper on hierarchical recommendation. Please verify the citation.
  3. [Table 5, Advertiser Satisfaction] The 'Advertiser Satisfaction' metric is reported as 3.68/5 vs. 3.96/5. The manuscript should describe how this subjective metric was collected and whether the measurement itself could be affected by the treatment.
  4. [Section 5.5.1, latency] The sentence 'LLM hourly overhead: 15ms amortized' is not well-defined. Amortized over what time window, and does it include API call time or only inference on the serving side? Please clarify.
  5. [Table 2 and Section 5.4] Many hyperparameters are hand-set (e.g., μ=0.1, η=0.05, β=2.0, the clipping threshold, and the expert hyperparameters). The sensitivity analysis covers only four mid-level parameters. A brief statement of how the remaining values were chosen, or a sensitivity table in the appendix, would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HOBA's central claims are empirical benchmark results and an online A/B test, not derivations from fitted inputs or self-citations.

full rationale

I found no step in which a claimed prediction or first-principles result reduces by construction to its own inputs. HOBA is presented as an empirical architecture: the headline results are the AuctionNet comparison (Table 3), ablations on that external benchmark (Table 4), and a phased production A/B test against an incumbent system (Table 5). These are measured outcomes, not consequences of how the components are defined. The bibliography contains no papers by the present authors, so self-citation patterns (uniqueness imported from authors, ansatz smuggled via self-citation, self-citation load-bearing) do not arise. The causal adjustment Eq. (12) is a standard doubly robust estimator with citations to external statistics literature [1,7]; the skeptic's concern that after 1000 warmup steps the UCB policy makes the logistic propensity model misspecified is a statistical-validity critique, not a circular identification of output with input. Likewise, Eq. (12)'s \hat\mu(a) being added to Q in Eq. (13) but not used in the SARSA update Eq. (14) is a mechanism/implementation concern, not a tautology. The LLM Think-Act-Observe-Reflect loop is an online learning mechanism; the optional KTO fine-tuning uses the LLM's self-reported quality score, which is a possible self-referential reward signal, but it is optional, not load-bearing for the external benchmark claims, and the paper itself discloses a limitation on sparse campaigns in Section 5.5.3 and a proprietary-template caveat in the appendix. Accordingly the honest finding is no significant circularity (score 0).

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

No invented entities; the framework is composed of an LLM agent, SARSA, and existing offline RL/control experts.

free parameters (7)
  • mu (switching penalty) = 0.1
    Hand-tuned weight in Eq. (11) penalizing expert switches; directly shapes SARSA reward and selection stability.
  • eta (pacing penalty) = 0.05
    Hand-tuned weight in Eq. (11) penalizing pacing deviation; shapes reward.
  • beta (UCB exploration coefficient) = 2.0
    Hand-tuned in Eq. (13); controls exploration intensity and affects the exploration-exploitation balance.
  • q_lower, q_upper (bid bounds) = LLM-generated, clipped to [0.1, 3.0]
    Output by the high-level LLM each hour; directly constrain all bids via Eq. (15).
  • theta_explore (exploration bonus) = LLM-generated, clipped to [0, 0.5]
    Modulates UCB exploration in Eq. (13); a free parameter per hour.
  • Propensity clipping threshold = 0.05
    Chosen in Section 4.2.1 to avoid extreme importance weights; affects the causal adjustment estimate.
  • Expert hyperparameters (PID gains, MPC horizon, CQL alpha, IQL tau, DT context) = PID Kp=0.5, Ki=0.1, Kd=0.05; MPC 1h; CQL alpha=5.0; IQL tau=0.7; DT context 20
    Hand-tuned or from prior literature; determine the expert pool's behavior and coverage.
assumptions (6)
  • domain assumption At least one of the outcome model and propensity model in Eq. (12) is correctly specified (doubly robust property)
    Doubly robust estimation is unbiased only if either model is correct; no model check is provided.
  • domain assumption The AuctionNet agent-replacement protocol with 30 trials is a valid estimator of auto-bidding performance
    Used for all offline comparisons; but the protocol reports mean of top-5, which is a biased statistic.
  • domain assumption The five static experts (PID, MPC, CQL, IQL, DT) collectively cover the market regimes encountered
    If no expert is adequate in a regime, selection cannot adapt; the paper offers no coverage analysis.
  • ad hoc to paper The LLM's text template preserves all decision-relevant numeric state information
    The state vector is rendered via template P_state; no evidence that discretization to language does not lose information.
  • domain assumption UCB exploration in Eq. (13) is sufficient to identify better experts in a non-stationary environment
    Standard UCB assumes stationarity; the paper's KS-triggered exploration doubling is a heuristic.
  • ad hoc to paper The KS test on reward CDFs is a reliable distribution-shift detector for triggering exploration/fine-tuning
    Threshold p<0.05 is chosen without calibration.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising." pith.science (2026). https://pith.science/paper/AEM6PWRS

@misc{pith2026260724779,
  author       = {Pith},
  title        = {Pith review of: HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AEM6PWRS}},
  note         = {Machine review of arXiv:2607.24779}
}
read the original abstract

Online advertising bidding systems typically deploy multiple offline-trained expert models (e.g., PID controllers, model predictive control, offline RL policies) but face two critical limitations: lack of online adaptability to non-stationary auction markets, and reliance on costly manual tuning of hyperparameters such as bid bounds and budget pacing constraints. We propose HOBA (Hierarchical On-policy Bidding Agents), a hierarchical reinforcement learning framework that decouples strategic reasoning, model selection, and bid execution across three time scales. At the high level, a large language model infers hyperparameters from contextual signals through a Think-Act-Observe-Reflect loop with historical experience retrieval. At the mid level, a SARSA agent dynamically selects among expert models, incorporating causal adjustment to eliminate selection bias. At the low level, a dynamic expert pool (PID, MPC, IQL, Decision Transformer) executes bids under high-level constraints. This design confines online learning to discrete expert selection rather than continuous bid optimization, significantly reducing exploration risk while maintaining adaptability. Experiments on the AuctionNet benchmark and a large-scale A/B test demonstrate consistent improvements over state-of-the-art baselines. In a large-scale online deployment, HOBA delivered substantial business value, achieving a +3.6\% increase in target cost, proving the effectiveness of our hierarchical multi-agent bidding paradigm.

Figures

Figures reproduced from arXiv: 2607.24779 by the authors.

Figure 1
Figure 1. Overview of HOBA’s three-tier architecture. High [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Hyperparameter sensitivity analysis showing per [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 6 linked inside Pith

  1. [1]

    Heejung Bang and James M Robins. 2005. Doubly robust estimation in missing data and causal inference models.Biometrics61, 4 (2005), 962–973

    2005

  2. [2]

    Andrew G Barto and Sridhar Mahadevan. 2003. Recent advances in hierarchical reinforcement learning.Discrete event dynamic systems13, 1-2 (2003), 41–77

    2003

  3. [3]

    Haifeng Cai, Ren Kan, Xiaoyan Zhu, et al. 2017. Real-time bidding by reinforce- ment learning in display advertising. InProceedings of the tenth ACM international conference on web search and data mining. 661–670

    2017

  4. [4]

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. 2021. Decision transformer: Reinforcement learning via sequence modeling. InAdvances in Neural Information Processing Systems, Vol. 34. 15084–15097

    2021

  5. [5]

    Yuchen Chen, Zhengliang Huang, Hao Wu, Yupeng Lv, Yao Zhang, et al. 2023. Towards autonomous system: Flexible modular production system enhanced with large language model agents.arXiv preprint arXiv:2304.14721(2023)

    arXiv 2023

  6. [6]

    Peter Dayan and Geoffrey E Hinton. 1992. Feudal reinforcement learning.Ad- vances in neural information processing systems5 (1992)

    1992

  7. [7]

    Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. 2014. Doubly robust policy evaluation and optimization. InStatistical Science, Vol. 29. 485–511

    2014

  8. [8]

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. KTO: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306(2024)

    arXiv 2024

Show all 42 references
  1. [9]

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. 2021. D4RL: Datasets for Deep Data-Driven Reinforcement Learning. InNeurIPS Datasets Track

    2021

  2. [10]

    Scott Fujimoto, David Meger, and Doina Precup. 2019. Off-policy deep rein- forcement learning without exploration. InInternational Conference on Machine Learning. PMLR, 2052–2062

    2019

  3. [11]

    Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dollé. 2018. Offline a/b testing for recommender systems. InProceed- ings of the Eleventh ACM International Conference on Web Search and Data Mining. 198–206

    2018

  4. [12]

    Xu Han, Weinan Wang, and Weinan Zhang. 2020. A Reinforcement Learning Framework for Optimizing Real-Time Bidding.IEEE TKDE32 (2020), 1–14

    2020

  5. [13]

    Daniel G Horvitz and Donovan J Thompson. 1952. A generalization of sampling without replacement from a finite universe.Journal of the American statistical Association47, 260 (1952), 663–685

    1952

  6. [14]

    Yudong Hu, Congying Han, Tiande Guo, and Hao Xiao. 2022. Applying Opponent Modeling for Automatic Bidding in Online Repeated Auctions.arXiv preprint arXiv:2212.02723(2022)

    2022 arXiv

  7. [15]

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Lan- guage models as zero-shot planners: Extracting actionable knowledge for em- bodied agents.International Conference on Machine Learning(2022), 9118–9147

    2022

  8. [16]

    Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs.IEEE Transactions on Big Data7, 3 (2019), 535–547

    2019

  9. [17]

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. 2022. Offline Reinforcement Learning with Implicit Q-Learning. InICLR

    2022

  10. [18]

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conserva- tive Q-Learning for Offline Reinforcement Learning. InNeurIPS

    2020

  11. [19]

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. 2020. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643(2020)

    2020 arXiv

  12. [20]

    Lihong Li, Wei Chu, John Langford, and Robert E Schapire. 2010. A contextual- bandit approach to personalized news article recommendation. InProceedings of the 19th international conference on World wide web. 661–670

    2010

  13. [21]

    Lihong Li, Jimin Kim, and Imed Zitouni. 2015. Toward predicting the outcome of an A/B experiment for search relevance.Proceedings of the Eighth ACM International Conference on Web Search and Data Mining(2015), 37–46

    2015

  14. [22]

    Frank J Massey Jr. 1951. The Kolmogorov-Smirnov test for goodness of fit.Journal of the American statistical Association46, 253 (1951), 68–78

    1951

  15. [23]

    Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timo- thy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchro- nous methods for deep reinforcement learning. InInternational Conference on Machine Learning. PMLR, 1928–1937

    2016

  16. [24]

    Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. 2018. Data- efficient hierarchical reinforcement learning. InAdvances in Neural Information Processing Systems, Vol. 31

    2018

  17. [25]

    Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, and Jan Peters. 2020. An algorithmic perspective on imitation learning.Founda- tions and Trends in Robotics7, 1-2 (2020), 1–179

    2020

  18. [26]

    Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. 2021. Hierarchical reinforcement learning: A comprehensive survey.ACM Computing Surveys (CSUR)54, 5 (2021), 1–35

    2021

  19. [27]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCN...

    2019

  20. [28]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

  21. [29]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems, Vol. 36. 8634–8652

    2023

  22. [30]

    Jasper Snoek, Hugo Larochelle, and Ryan P Adams. 2012. Practical bayesian optimization of machine learning algorithms. InAdvances in Neural Information Processing Systems, Vol. 25

    2012

  23. [31]

    Bartolomeo Stellato, Goran Banjac, Paul Goulart, Alberto Bemporad, and Stephen Boyd. 2020. OSQP: An operator splitting solver for quadratic programs.Mathe- matical Programming Computation12, 4 (2020), 637–669

    2020

  24. [32]

    2018.Reinforcement learning: An intro- duction

    Richard S Sutton and Andrew G Barto. 2018.Reinforcement learning: An intro- duction. MIT press

    2018

  25. [33]

    Richard S Sutton, Doina Precup, and Satinder Singh. 1999. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence112, 1-2 (1999), 181–211

    1999

  26. [34]

    Adith Swaminathan and Thorsten Joachims. 2015. Batch learning from logged bandit feedback through counterfactual risk minimization.Journal of Machine Learning Research16, 1 (2015), 1731–1755

    2015

  27. [35]

    Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. 2017. Feudal networks for hier- archical reinforcement learning. InInternational Conference on Machine Learning. PMLR, 3540–3549

    2017

  28. [36]

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024. Large language models as optimizers.arXiv preprint arXiv:2309.03409(2024)

    2024 arXiv

  29. [37]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations

    2023

  30. [38]

    Weiwen Yuan, Han Wang, Xiao Yu, Nan Liu, and Zilong Li. 2021. Hierarchical reinforcement learning for integrated recommendation. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 4521–4528

    2021

  31. [39]

    Weiwen Yuan, Han Wang, Xiao Yu, Nan Liu, and Zilong Li. 2022. Multi-agent reinforcement learning for online advertising. InProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 4447–4457

    2022

  32. [40]

    Weinan Zhang, Shuai Yuan, and Jun Wang. 2014. Optimal Real-Time Bidding for Display Advertising. InKDD. 1077–1086

    2014

  33. [41]

    mode": "conservative

    Jun Zhao, Weinan Zhang, Jun Wang, et al. 2021. AuctionNet: Benchmark dataset for learning to bid in online advertising auctions. InProceedings of the 27th ACM KDD ’26, August 09–13, 2026, Jeju Island, Republic of Korea Ji Wu et al. SIGKDD Conference on Knowledge Discovery & Da...

    2021

  34. [2017]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347 (2017)

    2017 arXiv

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.