Pith. sign in

REVIEW 4 major objections 5 minor 73 references

AIGB-R1 claims that auto-bidding improves when an LLM plans strategy and a lightweight transformer executes it, trained end-to-end with a decoupled advantage signal; its scores lead two large public auction benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

AIGB-R1 couples an LLM strategy planner with a prompt-conditioned Decision Transformer executor and trains both end-to-end with a decoupled-GRPO self-evolving loop, reporting the highest Score on the AuctionNet benchmark.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection A sensible hierarchical LLM+DT auto-bidding method, but the headline SOTA rests on per-task memory the baselines don't have, so the generalization claim is unsupported. the 4 major comments →

arxiv 2607.17281 v1 pith:K4WZPF5O submitted 2026-07-19 cs.LG cs.AI

AIGB-R1: Self-Evolving Generative Auto-Bidding via Hierarchical Planner-Executor Optimization

Yuejia Dou , Hesong Wang , Xinyu Zhang , Tianyu Wang , Zhilin Zhang , Chuan Yu , Jian Xu , Bo Zheng
show 1 more author
Qi Qi
This is my paper
classification cs.LG cs.AI
keywords auto-biddingAI-generated biddinglarge language modelsdecision transformerhierarchical reinforcement learningcredit assignmentGRPOonline advertising
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that LLM reasoning can be injected into AI-generated bidding without suffering the LLM's usual weaknesses — numerical imprecision, hallucination, latency — by splitting the task between a high-level Planner and a low-level Executor. It further claims that this two-layer policy can be trained end-to-end through an experience-driven self-evolving loop in a simulated auction environment, using a new advantage-decomposition rule (D-GRPO) that allocates credit between the layers. If true, generative auto-bidding systems would surpass offline-trained decision transformers and value-guided search methods across a wide range of budgets. The paper reports the best scores on two public auction datasets, and its ablations show every component contributes.

Core claim

The central claim is that decoupling macro strategy reasoning from fine-grained bid execution, then optimizing both layers jointly, yields a better auto-bidding agent than any single-model generative or value-based baseline. An LLM Planner converts advertiser profiles and historical experience into structured strategy prompts; a prompt-conditioned decision transformer Executor translates those prompts and numerical state histories into log-normal bid distributions. A self-evolving loop stores per-task best-strategy anchors and recent trajectories, and D-GRPO decomposes the rollout reward matrix into orthogonal inter-prompt and intra-prompt components so the Planner and Executor receive separ

What carries the argument

The load-bearing object is the nested reward matrix from hierarchical rollout: M strategy prompts times K trajectories per prompt. D-GRPO writes each centered reward as (mean reward of that prompt minus global mean) plus (trajectory reward minus its prompt mean), an orthogonal decomposition under the Frobenius inner product, so the first term measures the Planner's strategy quality and the second measures the Executor's execution quality under a fixed prompt. The two terms are normalized and feed separate clipped GRPO objectives, enabling stable end-to-end credit assignment.

Load-bearing premise

That AuctionNetEnv reproduces enough of the real second-price auction world — with its fixed pre-trained opponents — so that strategies improved inside it are genuinely better and not just better in simulation.

What would settle it

Train and evaluate AIGB-R1 on a held-out set of bidding tasks from the same dataset that it never saw during post-training, or replace the fixed opponents with adaptive ones during evaluation; if the Score advantage over the best baselines collapses, the reported gains depend on simulator-specific regularities rather than a general planning mechanism.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Auto-bidding can tap LLM world knowledge and reasoning for strategy while keeping request-time latency low, because the Planner runs once per bidding period, not per impression.
  • End-to-end hierarchical RL with decoupled advantage estimation becomes a practical alternative to freezing one layer or relying on offline value functions prone to extrapolation error.
  • Experience buffers that keep best-strategy anchors and recent rollouts per task turn exploration from blind random sampling into targeted neighborhood search.
  • The decomposition principle, if valid, generalizes to any two-tier policy where a language-level planner proposes strategies and a lower-level controller executes them.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the reward-decomposition trick is not bidding-specific; other two-level systems (plan plus controller) that can collect multiple executions per plan could use the same orthogonal advantage split, provided enough rollouts per plan are affordable.
  • Editorial extension: the paper's evaluation fixes the opponent agents; a stronger, untested scenario would use adaptive opponents, where the Planner's strategy prompts would need to anticipate competitor drift.
  • Editorial extension: a cheap test of the self-evolving loop's robustness is to evaluate on held-out bidding tasks never seen during post-training, which would reveal whether the gains come from general planning skill or from memorizing the simulator's task pool.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. AIGB-R1 proposes a hierarchical auto-bidding framework with an LLM Planner that outputs macro-level strategy prompts and a Prompt Decision Transformer (PDT) Executor that maps those prompts plus numerical state sequences to bids. Training is two-stage: offline pre-training of the Executor, followed by RL post-training in AuctionNetEnv, a simulator reconstructed from the AuctionNet benchmark. The post-training loop maintains a per-task experience buffer and uses a proposed D-GRPO objective, which decomposes the reward deviation into inter-group (Planner) and intra-group (Executor) components. Experiments on AuctionNet and AuctionNet-sparse report the best Score against a broad set of RL, generative, and LLM baselines, including across budget ratios.

Significance. If the empirical claims hold, the paper makes a useful engineering contribution: combining an LLM planner with a lightweight DT-style executor is a sensible way to exploit LLM reasoning while keeping numerical decision-making and latency in a small model, and the experience-driven rollout reuses successful strategies rather than sampling blindly. The D-GRPO decomposition is a natural extension of GRPO to a nested rollout hierarchy. The authors also compare against a wide range of recent baselines and test robustness across budget ratios. However, the central empirical claim is currently threatened by an evaluation-protocol confound: training and evaluation appear to use the same 48-task pool, and the Planner conditions on a per-task experience buffer populated during training. Without a held-out task split or a buffer-cleared evaluation, the reported margins over baselines may reflect task-specific memorization rather than generalizable planning. The simulator fidelity and the interpretational claim behind D-GRPO are secondary but still load-bearing concerns.

major comments (4)
  1. [§5.1.1, §4.3.2, Appendix A.4.3] The reported SOTA may be confounded by per-task experience-buffer memorization. The post-training loop in §4.3.2 maintains an experience buffer for each task in the AuctionNetEnv task pool, and Appendix A.4.3 shows that the Planner's prompt includes the BEST-STRATEGY ANCHOR and SLIDING MEMORY WINDOW for that advertiser. The datasets in §5.1.1 are described without any train/eval task split, and no held-out or buffer-cleared evaluation is reported. Consequently, at evaluation the Planner can retrieve and re-issue the best strategy discovered for that exact task during training, an advantage no baseline has. The margins in Table 1 (385±3.61 vs 376; 39.0 vs 38.1) could therefore be dominated by memory rather than by generalized hierarchical planning or D-GRPO. Please add evaluation on tasks unseen during post-training and/or an ablation with an empty/reset buffer; without that, the central
  2. [§A.1.1, Table 4] AuctionNetEnv is both the training and the evaluation arena, and its fidelity to the real environment is only approximately validated. Table 4 shows Sim/Data ratios deviating from 1.0, e.g., LWC std ratio 0.830 and sparse Marketing std ratio 0.780, and no held-out environment or opponent-policy shift is tested. This does not invalidate the internal comparison, since all baselines run in the same simulator, but it does limit the paper's stronger real-world conclusions. The abstract and conclusion imply effectiveness beyond the simulator; please add a clear scope statement and, ideally, a sensitivity analysis over opponent policies or environment parameters.
  3. [§4.3.3, Eq. (12), Appendix A.2] The claim that the inter-group reward component (r̄_m − r̄) measures only Planner quality and the intra-group component (r_m,k − r̄_m) measures only Executor quality is asserted, not derived. The orthogonality proof in A.2 is an algebraic identity for the decomposition; it does not establish that the between-group mean is free of Executor noise, initial-state effects, or other confounds. Since D-GRPO's Planner advantage is exactly (r̄_m − r̄)/σ^P, a systematic prompt-independent variation across strategy groups would inject spurious gradients into the Planner. A conditional-expectation argument, or a control experiment with random prompt permutations, would substantially strengthen the paper.
  4. [§5.3, Table 3] The ablation does not isolate D-GRPO from the effect of updating the Planner. AIGB-R1(SP) freezes the Planner and only post-trains the Executor; AIGB-R1 updates both Planner and Executor with D-GRPO. The performance gap between them conflates (a) making the Planner trainable and (b) using D-GRPO instead of standard GRPO. A variant with the full Planner-Executor architecture trained with standard GRPO (no advantage decoupling) is needed to support the paper's claim that D-GRPO is a necessary component. This is a missing experimental cell, not a presentation issue.
minor comments (5)
  1. [Throughout] Typos and formatting: 'modelns capabilities' in §2.3; 'PRoposition1' and 'PRoof' in Appendix A.2; inconsistent capitalization of 'AuctionNet-Sparse' vs 'AuctionNet-sparse'. Figure 1 contains very small text and is hard to read at normal zoom.
  2. [Table 1] Only AIGB-R1 is reported with standard deviation; baselines are point estimates. Reporting baselines with the same variance across seeds would help assess whether the 9-point AuctionNet margin is meaningful.
  3. [§5.1.4] The Planner is Qwen2.5-32B-Instruct, but Section 2.2 / Introduction emphasize latency; the paper never reports inference latency, response time, or cost for the Planner or AIGB-R1 as a whole. Since latency is one of the motivations, adding this number would strengthen the practical claims.
  4. [Appendix A.4.2] Table 6 lists 'Target return 2' for the PDT Executor, but the text does not explain how this value was chosen or whether it is tied to the RTG scale in Eq. (4). Please clarify.
  5. [Appendix A.1.2] The distributional consistency section is one paragraph plus a table; it would benefit from a brief explanation of which metrics are traffic-side vs cost-side and why deviations such as std ratio 0.830 are acceptable for downstream bidding decisions.

Circularity Check

0 steps flagged

No load-bearing circularity: the Planner-Executor policy is optimized by environment reward and D-GRPO is a standard policy-gradient update with an algebraic decomposition; the main caveats (reward/evaluation-metric alignment, per-task memory during same-task evaluation) are evaluation-validity concerns rather than definitional circularity.

full rationale

The paper's derivation chain is self-contained and does not reduce to its inputs. The Planner and Executor are defined by Eqs. (5)-(6) and trained by NLL pre-training (Eq. (8)) followed by environment-reward RL (Eqs. (9)-(15)); D-GRPO's advantages are explicitly defined as normalized inter-group and intra-group reward components, and the orthogonality proof in Appendix A.2 is an algebraic identity following from the definitions of r̄_m and r̄. The evaluation Score (§5.1.2) has the same penalty shape as the training reward in Eq. (9) (when η=2), so the system optimizes the metric on which it is judged; this is standard RL objective alignment, not a circular prediction. The per-task experience buffer (§4.3.2) is a stated algorithmic component, and using the same 48-task pool for post-training and evaluation is a potential overfitting/external-validity concern, not a definitional equivalence. Self-citations (e.g., GAM [10] for D-GRPO implementation, AuctionNet [45] for the environment) support implementation details and benchmarks rather than carrying the central claim; no uniqueness theorem or unverified prior result is invoked to force the architecture. Hence no circular step can be exhibited.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The framework relies on four explicit assumptions: the linear parametric bid structure (Eq 3), the fidelity of AuctionNetEnv, the stationary opponent setup, and the semantic validity of the D-GRPO advantage decomposition. None of these is independently verified beyond the paper's own claims.

free parameters (2)
  • Reward shaping penalty exponent η in AuctionNetEnv (Eq. 9) = not reported (prompt template uses exponent 2)
    Controls the penalty for CPA violations in the simulator reward. Since the evaluation Score uses the same penalty shape, this choice aligns training objective with the test metric, affecting all reported numbers.
  • PDT target return = 2
    Normalization constant for Return-To-Go used in the Decision Transformer; chosen by the authors and affects pre-training behavior.
axioms (4)
  • domain assumption The optimal bid takes a linear form b_i* = λ0 v_i + Σ_j λ_j p_ij C_j (Eq. 3, from USCB [22]).
    The executor's action is the adjustment of λ_j parameters; if the true optimal policy is not in this parametric family, the planner/executor search space is restricted.
  • domain assumption AuctionNetEnv is distributionally consistent enough with the real auction environment for training and evaluation purposes.
    Appendix A.1.2 reports Sim/Data ratios that are not all close to 1 (e.g., LWC std ratio 0.830; Marketing std 0.780), yet the paper assumes fidelity.
  • ad hoc to paper The inter-group reward component (r̄_m − r̄) measures only the Planner's strategy quality and the intra-group component (r_m,k − r̄_m) measures only the Executor's execution quality.
    Used to construct separate advantages for Planner and Executor in D-GRPO (Eq. 12–13). The orthogonality proof in Appendix A.2 establishes only an algebraic identity, not this semantic credit assignment.
  • domain assumption Opponent agents are fixed during evaluation; the environment is a stationary game.
    All rollouts in AuctionNetEnv use pre-trained opponent policies (PID, Online LP, IQL, etc.). If opponents adapt or change, the learned strategy may not generalize, and the paper does not test this.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of AIGB-R1: Self-Evolving Generative Auto-Bidding via Hierarchical Planner-Executor Optimization." pith.science (2026). https://pith.science/paper/K4WZPF5O

@misc{pith2026260717281,
  author       = {Pith},
  title        = {Pith review of: AIGB-R1: Self-Evolving Generative Auto-Bidding via Hierarchical Planner-Executor Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K4WZPF5O}},
  note         = {Machine review of arXiv:2607.17281}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Auto-bidding plays an essential role in online advertising, automatically adjusting bids for advertisers to optimize their commercial goals. The emerging AI-Generated Bidding (AIGB) paradigm widely adopts generative modeling to optimize bidding strategies, yet suffers from the limited mode coverage of offline datasets and inadequate task-state understanding, hindering effective exploration of optimal strategies. Large Language Models (LLMs), with prior world knowledge and reasoning capabilities, offer a promising approach to overcome these limitations. However, directly applying LLMs to auto-bidding tasks faces inherent challenges in limited numerical precision, hallucinations, and inference latency. To address these limitations, we propose AIGB-R1, a hierarchical self-evolving auto-bidding framework aiming to enhance AI-Generated Bidding via LLMs' Reasoning capabilities, comprising a high-level Planner module for macro-level strategy planning and a low-level Executor module for fine-grained decision-making. Building upon this, we design an experience-driven self-evolving loop, enabling autonomous strategy exploration and optimization from accumulated experience. We adopt a two-stage pipeline of offline pre-training and post-training alignment, and build an interactive bidding simulation environment for strategy rollout. Furthermore, we propose Decoupled Group Relative Policy Optimization (D-GRPO) to achieve end-to-end optimization via advantage decoupling. Experimental results on a large-scale public dataset demonstrate the effectiveness of AIGB-R1.

Figures

Figures reproduced from arXiv: 2607.17281 by Bo Zheng, Chuan Yu, Hesong Wang, Jian Xu, Qi Qi, Tianyu Wang, Xinyu Zhang, Yuejia Dou, Zhilin Zhang.

Figure 1
Figure 1. Figure 1: Overall framework of AIGB-R1. autoregressive sampling of actions at each subsequent timestep through the self-attention mechanism, achieving strategy-conditioned decision-making. The strategy prompt 𝑝 of the PDT Executor contains two forms of information: enumerated strategy labels and macro-level strat￾egy reasoning. The strategy labels cover predefined discrete strat￾egy dimensions such as risk preferenc… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

73 extracted references · 23 linked inside Pith

  1. [1]

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. 2022. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691 (2022)

    Pith/arXiv arXiv 2022

  2. [2]

    Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. 2022. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657 (2022)

    Pith/arXiv arXiv 2022

  3. [3]

    Ageneraltheoret- ical paradigm to understand learning from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, MarkRowland,MichalValko,andDanieleCalandriello.2024. Ageneraltheoret- ical paradigm to understand learning from human preferences. InInternational Conference on Artificial Intelligence and Statistics. PMLR, 4447–4455

    2024

  4. [4]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report.arXiv preprint arXiv:2309.16609 (2023)

    Pith/arXiv arXiv 2023

  5. [5]

    Santiago Balseiro, Yuan Deng, Jieming Mao, Vahab Mirrokni, and Song Zuo

  6. [6]

    LengCai,JunxuanHe,YikaiLi,JunjieLiang,YuanpingLin,ZimingQuan,Yawen Zeng, and Jin Xu. 2025. Rtbagent: A llm-based agent system for real-time bid- ding. InCompanion Proceedings of the ACM on Web Conference 2025 . 104–113

    2025

  7. [7]

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. 2021. Decision transformer:Reinforcementlearningviasequencemodeling. Advances in neural information processing systems 34 (2021), 15084–15097

    2021

  8. [8]

    Ye Chen, Pavel Berkhin, Bo Anderson, and Nikhil R Devanur. 2011. Real-time bidding algorithms for performance-based display ad allocation. InProceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining. 1307–1315

    2011

  9. [9]

    Yuan Deng, Jieming Mao, Vahab Mirrokni, and Song Zuo. 2021. Towards effi- cient auctions in an auto-bidding world. InProceedings of the Web Conference

    2021

  10. [10]

    GAM:AGenerativeAuto-MarketingFrame- work in Online E-commerce Platforms

    YuejiaDou,ShuaiDou,YuchaoMa,BingzheWang,TianyuWang,ZhilinZhang, ChuanYu,JianXu,andQiQi.2026. GAM:AGenerativeAuto-MarketingFrame- work in Online E-commerce Platforms. InProceedings of the ACM Web Confer- ence 2026. 7979–7988

    2026

  11. [11]

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 (2024)

    Pith/arXiv arXiv 2024

  12. [12]

    David S Evans. 2009. The online advertising industry: Economics, evolution, and privacy. Journal of economic perspectives 23, 3 (2009), 37–60

    2009

  13. [13]

    Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. 2026. Group-in-group policy optimization for llm agent training.Advances in Neural Information Pro- cessing Systems 38 (2026), 46375–46408

    2026

  14. [14]

    Asurveyonmathematicalreasoningandoptimizationwith large language models.arXiv preprint arXiv:2503.17726 (2025)

    AliForootani.2025. Asurveyonmathematicalreasoningandoptimizationwith large language models.arXiv preprint arXiv:2503.17726 (2025)

    Pith/arXiv arXiv 2025

  15. [15]

    Scott Fujimoto, David Meger, and Doina Precup. 2019. Off-policy deep rein- forcement learning without exploration. InInternational conference on machine learning. PMLR, 2052–2062

    2019

  16. [16]

    Jingtong Gao, Yewen Li, Shuai Mao, Peng Jiang, Nan Jiang, Yejing Wang, Qing- peng Cai, Fei Pan, Peng Jiang, Kun Gai, et al. 2025. Generative auto-bidding with value-guided explorations. InProceedings of the 48th International ACM SI- GIR Conference on Research and Development in Information Retrieval . 244–254

    2025

  17. [17]

    Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. 2023. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998 (2023)

    Pith/arXiv arXiv 2023

  18. [18]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. 2025. Deepseek-r1: Incen- tivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948 (2025)

    Pith/arXiv arXiv 2025

  19. [19]

    Jiayan Guo, Yusen Huo, Zhilin Zhang, Tianyu Wang, Chuan Yu, Jian Xu, Bo Zheng, and Yan Zhang. 2024. Generative auto-bidding via conditional diffusion modeling. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Dis- covery and Data Mining. 5038–5049

    2024

  20. [20]

    Louisa Ha. 2008. Online advertising research in advertising journals: A review. Journal of Current Issues & Research in Advertising 30, 1 (2008), 31–48

    2008

  21. [21]

    Fromwords to actions: Unveiling the theoretical underpinnings of llm-driven autonomous systems

    JianliangHe,SiyuChen,FengzhuoZhang,andZhuoranYang.2024. Fromwords to actions: Unveiling the theoretical underpinnings of llm-driven autonomous systems. arXiv preprint arXiv:2405.19883 (2024)

    Pith/arXiv arXiv 2024

  22. [22]

    Aunifiedsolutiontoconstrainedbiddinginonlinedisplay advertising

    Yue He, Xiujun Chen, Di Wu, Junwei Pan, Qing Tan, Chuan Yu, Jian Xu, and XiaoqiangZhu.2021. Aunifiedsolutiontoconstrainedbiddinginonlinedisplay advertising. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2993–3001

    2021

  23. [23]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2025. A Yuejia Dou, Hesong Wang, Xinyu Zhang, Tianyu Wang, Zhilin Zhang, Chuan Yu, Jian Xu, Bo Zheng, and Qi Qi survey on hallucination in large language models: Principles, taxonomy, chal- lenges, and open questions...

    2025

  24. [24]

    Yinqiu Huang, Hao Ma, Wenshuai Chen, Zongwei Wang, Shuli Wang, Yongqiang Zhang, Xue Wei, Yinhua Zhu, Haitao Wang, and Xingxing Wang

  25. [25]

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. 2021. Offline reinforcement learning with implicit q-learning.arXiv preprint arXiv:2110.06169 (2021)

    Pith/arXiv arXiv 2021

  26. [26]

    Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. 2019. Stabilizing off-policy q-learning via bootstrapping error reduction.Advances in neural information processing systems 32 (2019)

    2019

  27. [27]

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conser- vative q-learning for offline reinforcement learning.Advances in neural infor- mation processing systems 33 (2020), 1179–1191

    2020

  28. [28]

    Yu Lei, Jiayang Zhao, Yilei Zhao, Zhaoqi Zhang, Linyou Cai, Qianlong Xie, and Xingxing Wang. 2026. Generative Large-Scale Pre-trained Models for Auto- mated Ad Bidding Optimization. InProceedings of the 32nd ACM SIGKDD Con- ference on Knowledge Discovery and Data Mining V. 1 . 2266–2277

    2026

  29. [29]

    Haoming Li, Yusen Huo, Shuai Dou, Zhenzhe Zheng, Zhilin Zhang, Chuan Yu, Jian Xu, and Fan Wu. 2024. Trajectory-wise iterative reinforcement learning framework for auto-bidding. InProceedings of the ACM Web Conference 2024 . 4193–4203

    2024

  30. [30]

    Yewen Li, Jingtong Gao, Nan Jiang, Shuai Mao, Ruyi An, Fei Pan, Xiangyu Zhao, Bo An, Qingpeng Cai, and Peng Jiang. 2025. Generative Auto-Bidding in Large- Scale Competitive Auctions via Diffusion Completer-Aligner. arXiv preprint arXiv:2509.03348 (2025)

    Pith/arXiv arXiv 2025

  31. [31]

    Yewen Li, Zhiyi Lyu, Peng Jiang, Qingpeng Cai, Fei Pan, Bo An, and Peng Jiang

  32. [32]

    Yewen Li, Shuai Mao, Jingtong Gao, Nan Jiang, Yunjian Xu, Qingpeng Cai, Fei Pan, Peng Jiang, and Bo An. 2025. Gas: Generative auto-bidding with post- training search. InCompanion Proceedings of the ACM on Web Conference 2025 . 315–324

    2025

  33. [33]

    Xiaowei Lv, Zhilin Zhang, Yijun Li, Yusen Huo, Siyuan Ju, Xuyan Li, Chunxi- ang Hong, Tianyu Wang, Yongcai Wang, Peng Sun, et al. 2026. DecisionLLM: LargeLanguageModelsforLongSequenceDecisionExploration. arXiv preprint arXiv:2601.10148 (2026)

    arXiv 2026

  34. [34]

    Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward.Advances in Neural Information Pro- cessing Systems 37 (2024), 124198–124235

    2024

  35. [35]

    Zhiyu Mou, Yusen Huo, Rongquan Bai, Mingzhou Xie, Chuan Yu, Jian Xu, and Bo Zheng. 2022. Sustainable online reinforcement learning for auto-bidding. Advances in Neural Information Processing Systems 35 (2022), 2651–2663

    2022

  36. [36]

    Deeplandscapeforecastinginmulti-slotreal- time bidding

    Weitong Ou, Bo Chen, Yingxuan Yang, Xinyi Dai, Weiwen Liu, Weinan Zhang, RuimingTang,andYongYu.2023. Deeplandscapeforecastinginmulti-slotreal- time bidding. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 4685–4695

    2023

  37. [37]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feed- back. Advances in neural information processing systems35(2022),27730–27744

    2022

  38. [38]

    Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Jiadai Sun, Xinyue Yang, Yu Yang, Shuntian Yao, Wei Xu, et al. 2025. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning. InInterna- tional Conference on Learning Representations, Vol. 2025. 79791–79821

    2025

  39. [39]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems 36 (2023), 53728–53741

    2023

  40. [40]

    Thomas Schmied, Jörg Bornschein, Jordi Grau-Moya, Markus Wulfmeier, and Razvan Pascanu. 2025. Llms are greedy agents: Effects of rl fine-tuning on decision-making abilities. arXiv preprint arXiv:2504.16078 (2025)

    Pith/arXiv arXiv 2025

  41. [41]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300 (2024)

    Pith/arXiv arXiv 2024

  42. [42]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learn- ing. Advances in neural information processing systems 36 (2023), 8634–8652

    2023

  43. [43]

    YashShukla,WenchangGao,VasanthSarathy,AlvaroVelasquez,RobertWright, and Jivko Sinapov. 2023. LgTS: Dynamic task sampling using LLM-generated sub-goals for reinforcement learning agents. arXiv preprint arXiv:2310.09454 (2023)

    Pith/arXiv arXiv 2023

  44. [44]

    Mingxuan Song, Yusen Huo, Bohan Zhou, Shenglin Yin, Zhen Xiao, Jieyi Long, Zhilin Zhang, and Chuan Yu. 2026. DARA: Few-shot Budget Allocation in On- line Advertising via In-Context Decision Making with RL-Finetuned LLMs. In Proceedings of the ACM Web Conference 2026. 40–50

    2026

  45. [45]

    Auctionnet:Anovelbenchmarkfordecision-makingin large-scale games

    Kefan Su, Yusen Huo, Zhilin Zhang, Shuai Dou, Chuan Yu, Jian Xu, Zongqing Lu,andBoZheng.2024. Auctionnet:Anovelbenchmarkfordecision-makingin large-scale games. Advances in Neural Information Processing Systems 37 (2024), 94428–94452

    2024

  46. [46]

    Xu Wan, Wenyue Xu, Chao Yang, and Mingyang Sun. 2025. Think twice, act once: A co-evolution framework of llm and rl for large-scale decision making. arXiv preprint arXiv:2506.02522 (2025)

    Pith/arXiv arXiv 2025

  47. [47]

    Voyager:Anopen-endedembod- ied agent with large language models.arXiv preprint arXiv:2305.16291 (2023)

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu,LinxiFan,andAnimaAnandkumar.2023. Voyager:Anopen-endedembod- ied agent with large language models.arXiv preprint arXiv:2305.16291 (2023)

    Pith/arXiv arXiv 2023

  48. [48]

    Hao Wang, Bo Tang, Chi Harold Liu, Shangqin Mao, Jiahong Zhou, Zipeng Dai, Yaqi Sun, Qianlong Xie, Xingxing Wang, and Dong Wang. 2023. Hibid: A cross- channel constrained bidding system with budget allocation by hierarchical of- fline deep reinforcement learning.IEEE Trans. Comput. 73, 3 (2023), 815–828

    2023

  49. [49]

    Jun Wang and Shuai Yuan. 2015. Real-time bidding: A new frontier of com- putational advertising research. InProceedings of the Eighth ACM International Conference on Web Search and Data Mining. 415–416

    2015

  50. [50]

    Jun Wang, Weinan Zhang, and Shuai Yuan. 2017. Display advertising with real- time bidding (RTB) and behavioural targeting.Foundations and Trends® in In- formation Retrieval 11, 4–5 (2017), 297–435

    2017

  51. [51]

    YongdongWang,RunzeXiao,JunYounesLouhiKasahara,RyosukeYajima,Keiji Nagatani, Atsushi Yamashita, and Hajime Asama. 2024. Dart-llm: Dependency- awaremulti-robottaskdecompositionandexecutionusinglargelanguagemod- els. arXiv preprint arXiv:2411.09022 (2024)

    Pith/arXiv arXiv 2024

  52. [52]

    Chao Wen, Miao Xu, Zhilin Zhang, Zhenzhe Zheng, Yuhui Wang, Xiangyu Liu, Yu Rong, Dong Xie, Xiaoyang Tan, Chuan Yu, et al. 2022. A cooperative- competitive multi-agent framework for auto-bidding in online advertising. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. 1129–1139

    2022

  53. [53]

    BinglinWu,YingyiZhang,XiannengLi,RuyueDeng,ChuanYue,WeiruZhang, and Xiaoyi Zeng. 2026. Constraint-Aware Generative Auto-bidding via Pareto- Prioritized Regret Optimization.arXiv preprint arXiv:2602.08261 (2026)

    arXiv 2026

  54. [54]

    RongWu,XiaomanWang,JianbiaoMei,PinlongCai,DaochengFu,ChengYang, Licheng Wen, Xuemeng Yang, Yufan Shen, Yuxin Wang, et al. 2025. Evolver: Self-evolving llm agents through an experience-driven lifecycle.arXiv preprint arXiv:2510.16079 (2025)

    Pith/arXiv arXiv 2025

  55. [55]

    Chaojun Xiao, Jie Cai, Weilin Zhao, Biyuan Lin, Guoyang Zeng, Jie Zhou, Zhi Zheng, Xu Han, Zhiyuan Liu, and Maosong Sun. 2025. Densing law of llms. Nature Machine Intelligence (2025), 1–11

    2025

  56. [56]

    Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024. Contrastive preference optimization: Pushing the boundaries of llm performance in machine transla- tion. arXiv preprint arXiv:2401.08417 (2024)

    Pith/arXiv arXiv 2024

  57. [57]

    Jian Xu, Zhilin Zhang, Zongqing Lu, Xiaotie Deng, Michael P Wellman, Chuan Yu, Shuai Dou, Yusen Huo, Zhiwei Xu, Zhijian Duan, et al. 2024. Auto-bidding in large-scale auctions: Learning decision-making in uncertain and competitive games. InNeurIPS 2024 Competition Track

    2024

  58. [58]

    Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenen- baum, and Chuang Gan. 2022. Prompting decision transformer for few-shot policy generalization. In international conference on machine learning . PMLR, 24631–24645

    2022

  59. [59]

    Qwen3technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, BowenYu,ChangGao,ChengenHuang,ChenxuLv,etal.2025. Qwen3technical report. arXiv preprint arXiv:2505.09388 (2025)

    Pith/arXiv arXiv 2025

  60. [60]

    ShunyuYao,JeffreyZhao,DianYu,NanDu,IzhakShafran,KarthikNarasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language mod- els. arXiv preprint arXiv:2210.03629 (2022)

    Pith/arXiv arXiv 2022

  61. [61]

    Hao Yu, Michael Neely, and Xiaohan Wei. 2017. Online convex optimization with stochastic constraints.Advances in Neural Information Processing Systems 30 (2017)

    2017

  62. [62]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D Goodman. 2024. Star: Self- taught reasoner bootstrapping reasoning with reasoning. InProc. the 36th Inter- national Conference on Neural Information Processing Systems , Vol. 1126. 0–55

    2024

  63. [63]

    Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. 2024. Expel: Llm agents are experiential learners. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 19632–19642

    2024

  64. [64]

    YuzeZhao,JintaoHuang,JinghanHu,XingjunWang,YunlinMao,DaozeZhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. 2024. SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning. arXiv:2408.05517 [cs.CL] https://arxiv.org/abs/2408.05517

    Pith/arXiv arXiv 2024

  65. [65]

    Qinqing Zheng, Amy Zhang, and Aditya Grover. 2022. Online decision trans- former. Ininternational conference on machine learning. PMLR, 27042–27059

    2022

  66. [66]

    Yifei Zhou, Sergey Levine, Jason Weston, Xian Li, and Sainbayar Sukhbaatar

  67. [67]

    GuanyuZhu,JiningLuan,HanwenDu,XinyuFang,SiboXu,ErshengNi,Hongji Li, Jincheng Fang, Ronghao Chen, Huacan Wang, et al. 2026. On the Role of Language Representations in Auto-Bidding: Findings and Implications.arXiv AIGB-R1: Self-Evolving Generative Auto-Bidding via Hierarchical Planner-Executor Optimization preprint arXiv:2605.05833 (2026). A Appendix A.1 Au...

    Pith/arXiv arXiv 2026

  68. [70]

    Self-challenging language model agents.Advances in Neural Information Processing Systems 38 (2026), 113959–113991

    2026

  69. [72]

    Then, based on your chosen adjustment direction, select the most appropriateenumeratedstrategylabelforeachofthefollowingdimen- sions ⟨strategy labels⟩: ⟨option⟩ – e.g., Risk Awareness:AGGRESSIVE / MODERATE / CONSERVATIVE

  70. [73]

    Provide ⟨M⟩distinctstrategiesrankedbypotentialvalueindescending order; they must differ in at least one strategy dimension

    Finally, briefly summarize historical strategy performance and ex- plain the rationale for your chosen adjustment direction, producing ⟨strategy reasoning⟩: ⟨description⟩. Provide ⟨M⟩distinctstrategiesrankedbypotentialvalueindescending order; they must differ in at least one strategy dimension. NOTES: – Each advertiser has unique budget, CPA, and advertis...

  71. [2021]

    Advances in Neural Information Processing Systems 34 (2021), 17777–17788

    Robust auction design in the auto-bidding world. Advances in Neural Information Processing Systems 34 (2021), 17777–17788

    2021

  72. [2025]

    Generative Bid Shading in Real-Time Bidding Advertising.arXiv preprint arXiv:2508.06550 (2025)

    Pith/arXiv arXiv 2025

  73. [2026]

    In Proceedings of the ACM Web Conference 2026

    LBM: Hierarchical Large Auto-Bidding Model via Reasoning and Acting. In Proceedings of the ACM Web Conference 2026. 75–86

    2026

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.