Pith. sign in

REVIEW 4 major objections 5 minor 31 references

Permutation Equivariant Model-based Offline Reinforcement Learning for Auto-bidding

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Advertising auto-bidding policies trained with a permutation equivariant environment model and a pessimistically penalized offline Q target can escape the coverage of fixed offline data and raise GMV by 3.9 to 7.2 percent in A/B tests.

desk verdict The deployment results are worth a look, but the permutation equivariance proof does not hold up, and the paper overclaims theory. read the letter →

arxiv 2506.17919 v1 pith:VUWRWNXP submitted 2025-06-22 cs.LG cs.AI

Zhiyu Mou , Miao Xu , Wei Chen , Rongquan Bai , Chuan Yu , Jian Xu This is my paper · ORCID
classification cs.LGcs.AI
keywords auto-biddingofflinereinforcementlearningmodel-basedpermutationequivariancepessimisticQ-learningonlineadvertisinguncertaintypenalty
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that auto-bidding policies for online advertising can be trained more effectively by learning an environment model from real bidding data and then training the policy on both real and model-generated data, rather than training only on the fixed offline dataset or only on a hand-built auction simulator. It introduces PE-MORL, which makes the learned environment model permutation equivariant, meaning that reordering the list of advertisers reorders the model's predictions in the same way, and adds a pessimistic penalty to the Q-learning target that discounts predicted rewards and state values where the model is uncertain. The paper argues this combination expands the state space the policy can visit, with a distance measure between policy trajectories and real data that is 15.55 percent larger than for a model-free offline baseline, while protecting against model error. In ten-day A/B tests across about three thousand representative advertisers, the authors report GMV gains of +4.4 percent, +7.2 percent, and +3.9 percent over three state-of-the-art baselines, with ROI up and cost close to unchanged. If correct, this establishes a practical middle path between simulator-based and offline RL bidding.

What carries the argument

The load-bearing object is the permutation equivariant environment model combined with a pessimistically penalized offline $Q$ target. The environment model has two heads: a shared-parameter, permutation equivariant network that predicts every advertiser's next local state, and a permutation invariant network that predicts the representative advertiser's reward; each uses multi-head attention followed by mean and max pooling, following the known representation of invariant functions as $\phi(\mathrm{pool}(\psi(\cdot)))$. An ensemble of $K$ such models is trained by maximum log-likelihood with reparameterized sampling. The robust $Q$ target is $\hat B Q(s_t,a_t)=\hat r_t-\lambda\|\hat\sigma_t\|_F+\gamma\min_{\hat s_{t+1}\in\hat M(s_t,a_t)}\max_{a'}Q(\hat s_{t+1},a')$, where $\|\hat\sigma_t\|_F$ is the ensemble's reward variance and the minimum over predicted next states makes the policy pessimistic out of distribution. The theory works by showing that orbit averaging, the operator that averages a function over all advertiser permutations, shrinks the function class's covering number, which tightens the generalization bound.

What would settle it

Take the real transition logs used for training, apply a fixed permutation to the advertiser indices of each tuple, and check whether the observed next-state distribution and reward change only by reordering; if the permuted data produce a different reward or a non-equivariant next-state distribution, Proposition 5.1 is false. A second check: retrain without the equivariance constraint and compare test error on data where advertiser order at inference is reversed; if the equivariant model loses its advantage when advertiser identities matter, the architectural assumption is the bottleneck.

Watch

Extended reading notes

Core claim

The central claim is that the real bidding environment is symmetric under permutations of advertisers, and a model-based offline RL algorithm that builds this symmetry into the environment model, then pessimistically corrects for model error, outperforms state-of-the-art auto-bidding methods. Concretely, the paper proves that the transition rule of the advertising system is permutation equivariant and the reward is permutation invariant (Proposition 5.1), designs an environment model whose next-state head is equivariant and whose reward head is invariant, and proves that imposing this structure reduces the upper bound on the train-test generalization gap (Theorem 5.2). On the policy side, the robust Q target in Eq. (15) penalizes the predicted reward by the ensemble's uncertainty and takes the minimum Q value over the model's predicted next states, with a lower-bound guarantee that the pessimistically trained policy's performance in the real system is at least its performance in the penalized model (Proposition 5.4). The experiments support the claim by showing the equivariant model has lower prediction error than both a non-equivariant learned model and a GSP-based simulator, and that the resulting policies move further from the offline data while increasing GMV in online A/B tests.

Load-bearing premise

The load-bearing premise is that the online auction treats all advertisers symmetrically, so permuting the advertiser list merely permutes the auction outputs; if the real mechanism favors advertisers by identity, position, or other asymmetric features, or if next states depend on market dynamics beyond the cost spent, the built-in permutation symmetry is misspecified and the model's advantage can collapse.

Editorial extensions

If this is right

  • Policies trained with PE-MORL reach states measurably beyond the offline dataset: the distance from real data is 15.55 percent larger than for the model-free offline baseline, so the data-coverage ceiling of offline RL bidding is not fundamental.
  • The learned environment model achieves much lower test error than the GSP-based simulator used in simulation-based bidding, with MAE down 69.6 percent and MSE down 88.2 percent, and a smaller but real improvement over a non-equivariant neural network, with MAE down 5.9 percent and MSE down 3.3 percent.
  • The pessimistic penalty is necessary for the reported gains: removing it, i.e., setting $\lambda=0$, yields the worst offline metrics in the ablation, so the performance claim depends on the uncertainty penalty rather than on the equivariant model alone.
  • In production-style A/B tests, the method raises GMV by 3.9 to 7.2 percent over three baselines while holding cost roughly flat and increasing ROI, implying advertisers obtain more value per unit of spend.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the permutation equivariance premise is not tied to one auction platform; any multi-bidder sequential auction with symmetric allocation rules would admit the same environment model, so the recipe could transfer to sponsored search or display advertising systems elsewhere.
  • Beyond the paper, the uncertainty penalty is a generic cure for model bias: one could test whether the same ensemble-variance reward penalty plus min-Q state penalty improves other model-based offline RL problems where the environment has known symmetries.
  • Beyond the paper, varying the number of background advertisers at test time would be a clean extension; because the pooling layers are set-based, the equivariant model may generalize across different advertiser counts better than a fully connected model, which the paper does not test.
  • Beyond the paper, if the cost-only transition assumption is relaxed, for example if budgets or market prices enter the next state, the architecture would need additional equivariant channels; the paper's performance claims may depend on this simplification.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper frames RL-based auto-bidding as a choice between training on a simplistic GSP simulator (SRLB) and training purely on fixed offline data (ORLB), and proposes a model-based offline RL paradigm (MRLB) that learns an environment model from real data and trains the policy on the union of real and model-generated data. The specific algorithm, PE-MORL, has two main components: a permutation-equivariant environment model that predicts next states and rewards from the joint state-action vector of all advertisers, and a robust offline Q-learning target that penalizes the predicted reward by ensemble variance and pessimistically selects among predicted next states. The paper presents real-world A/B tests on about 3,000 representative advertisers claiming GMV gains of +4.4% over V-CQL, +7.2% over USCB, and +3.9% over MBAB with roughly flat Cost and improved ROI, plus offline comparisons showing the PE environment model has lower prediction error than a fully connected baseline and a GSP simulator. The theoretical claims are that the environment model is justified by permutation invariance/equivariance of the auction process (Proposition 5.1), that the equivariant architecture improves the generalization gap (Theorem 5.2), and that the pessimistic reward penalty improves a lower bound on true system performance (Proposition 5.4).

Significance. If the empirical and theoretical claims hold, the paper makes a useful contribution to industrial auto-bidding: it identifies a concrete middle ground between simulator-based and offline-RL bidding, and it demonstrates a production-scale A/B evaluation, which is a genuine strength and uncommon in the literature. The permutation-equivariant environment model is a sensible inductive bias for auctions, and the offline prediction comparisons in Tables 2 and 3 are encouraging evidence for that architectural choice. However, the contribution is currently weakened by three load-bearing gaps: the proof of Proposition 5.1 is not valid as written for a fixed-index representative advertiser; Theorem 5.2 is proved only for an orbit-averaged model, not for the deployed parameter-shared architecture; and the headline A/B results in Table 1 are reported without confidence intervals or significance tests, so the size and stability of the claimed gains are not established.

major comments (4)
  1. [§5.1, Appendix E.1, Eqs. (37)-(38)] The proof of Proposition 5.1 does not establish the claimed reward permutation invariance. Since R(s,a) is defined as the N-th component of the reward vector r_t(s,a), Eq. (37) gives R(ρs,ρa)=[ρ r_t(s,a)]_N = r_{ρ^{-1}(N)}(s,a), which equals R(s,a) only when ρ fixes N; for a transposition on N=2 with distinct advertisers this fails. This invalidates the PI claim, and consequently the theoretical justification for the reward branch of the PE environment model and for the applicability of Theorem 5.2 to the proposed architecture needs repair.
  2. [§5.1, Appendix D, Eq. (30) and Proposition D.4] Theorem 5.2 is proved for the orbit-averaged model Q\hat M', but the deployed environment model is a parameter-shared network with mean and max pooling, which is not equal to orbit averaging. The proof shows that orbit averaging reduces the r-covering number, yet no argument is given that the specific network architecture also reduces the covering number or preserves the required non-expansiveness. Thus the theorem's conclusion about "making it satisfy (4)" does not directly support the generalization claim for the actual architecture; the claim should either be proved for the architecture or explicitly downgraded to an empirical one.
  3. [Table 1, §6.1] The headline A/B results report only single point estimates for GMV, ROI, and Cost. With roughly 3,000 representative advertisers over 10 days, the claimed differences of +3.9% to +7.2% need confidence intervals, standard errors, or significance tests to be interpretable; without these, the central claim that PE-MORL outperforms state-of-the-art methods in deployment is not statistically supported.
  4. [§5.2, Proposition 5.4 and Appendix E.4, Eqs. (44)-(45)] The lower-bound guarantee is conditional on an imported assumption, and the proof does not match the implemented penalty. The proof uses a function u(s,a) that dominates the transition-model discrepancy d_F, but the actual reward penalty in Eq. (15) is λ||σ̂_t||_F, the ensemble reward variance, with no demonstrated relationship between σ̂_t and u(s,a). The proposition should either be re-proved for the actual penalty or stated as an assumption-based result requiring empirical validation of the uncertainty-dominance condition.
minor comments (5)
  1. [Table 5] The column formatting for different λ values is unclear; for example, the row for λ=5 appears to merge values for two different settings, making it hard to read which R/R* and online rate correspond to which λ.
  2. [Eq. (13)] The notation "E [s^T_{t+1},r_t]^T ~ D_R" is malformed and should be written as an expectation over (s_{t+1},r_t) drawn from the real data distribution, e.g., E_{(s_{t+1},r_t)∼D_R}.
  3. [Appendix A] Several citation placeholders appear as "[?]" and there is a duplicated/incomplete sentence "We highlight that We are the first..."; these should be cleaned before submission.
  4. [Figure 2 and §6.1] The Wasserstein distance plot lacks axis labels and units, and the claim that the distance is 15.55% higher for PE-MORL than for V-CQL needs a precise definition of the reference distance used to compute the percentage.
  5. [Throughout] There are scattered typos such as "representation advertiser" and "natrual language" that should be corrected in a revision.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central GMV claim is tested by external A/B experiments, and no equation reduces a prediction to its fitted inputs.

full rationale

The paper's claimed derivation chain is not circular. The PE/PI environment model (Section 5.1) is justified by Proposition 5.1, whose proof (Appendix E.1) is attempted from assumed properties of the auction mechanism (Eqs. 34-36) and from the cost-dependence of state transitions. Although that proof contains an apparent algebraic gap concerning the fixed-index representative and the distinction between invariant and equivariant transition distributions, this is a correctness or mathematical-rigor issue, not a circular reduction of a prediction to its inputs. Theorem 5.2 is proven by orbit averaging and covering-number arguments (Appendix D) without assuming the conclusion; it states a conditional generalization-gap improvement, not a fitted-value prediction. The pessimistic Q target (Eq. 15) and Proposition 5.4 are conditional on an uncertainty-dominance assumption imported from MOPO [27] (Eq. 45), and the λ ablation (Table 5) is a hyperparameter-selection result on offline metrics; these are caveats about the strength of the guarantee, but they do not make any output equal to an input by construction. No fitted parameter is renamed as a prediction, and the central claim of superiority is validated by external 10-day A/B tests against three baselines (Table 1). Self-citations to prior work [19] for the ORLB baseline and OOD framing are modest and are not load-bearing for the new algorithm's theoretical claims. Overall, no step in the derivation chain equates its output to its input by the paper's own algebra.

Assumptions & free parameters 3 free parameters · 8 assumptions · 0 invented entities

The central claim rests on three kinds of inputs: (a) assumed structure of the real auction system (permutation equivariance of allocation and pricing, cost-only transitions, fixed background policies), asserted about a proprietary platform and not verifiable from the paper; (b) an uncertainty-dominance premise imported from MOPO [27] that the ensemble variance penalty bounds the true model error, never calibrated; and (c) standard mathematical results (Deep Sets representation theorem, covering-number bounds, performance-difference lemma) taken from the cited literature. Free parameters include the penalty scale λ (tuned on the same offline metrics used for the ablation), the ensemble size K, and unreported network hyperparameters. No invented entities are postulated; the PE environment model and the penalized Q target are algorithmic constructions built from existing components.

free parameters (3)
  • λ (reward penalty coefficient) = not clearly reported; ablation values include 3, 4, 5 and smaller values (Table 5 is misaligned)
    Section 5.2, Eq. (15) and Table 5. Scales the ensemble variance penalty; tuned on the offline metrics R/R* and online rate. The guarantee in Proposition 5.4 is conditional on λ being large enough to dominate model error, which the ablation implicitly selects.
  • Ensemble size K = not reported
    Section 5.1.1, 'we train an ensemble of K environment models'. Chosen by hand; the variance estimate σ̂_t used for the penalty depends on it.
  • Pooling choice (mean plus max) and network hyperparameters = not reported
    Section 5.1.1, Eqs. (7)-(8) and Figure 1. Attention heads, embedding dimension B, and FC widths are unreported; the max pool is justified by a domain assumption about highest bids.
assumptions (8)
  • domain assumption The real allocation and pricing mechanisms are permutation equivariant in the advertiser indices: G(ρx, ρa_t, y) = G(x, a_t, y)ρ_c and C(ρx, ρa_t, y) = C(x, a_t, y)ρ_c
    Appendix E.1, Eqs. (34)-(35). Load-bearing for Proposition 5.1; if the production system has advertiser-specific asymmetries, the PE/PI properties fail. Asserted about the proprietary system, not verifiable from the paper.
  • domain assumption State transitions depend only on the cost spent during the period: P(s_{t+1}|s_t, a_t) = P(c_t(s_t, a_t))
    Appendix E.1, before Eq. (39). Ignores state evolution driven by external factors such as traffic or market conditions, which would break the permutation equivariance argument for P.
  • domain assumption There exists u(s,a) with u(s,a) >= sup_f |E_{s'~P̂} f(s') - E_{s'~P} f(s')| >= |G^π_M̂(s,a)|, and the ensemble variance penalty λ||σ̂_t||_F plays this role
    Section 5.2 and Appendix E.4 (Eq. 45), imported from MOPO [27]. The lower bound η_M(π) >= η_M̂(π) holds only if the uncertainty estimate dominates the true model error; the paper asserts consistency ('consistent with the value of σ̂_t [22,27]') without calibration evidence.
  • domain assumption Background advertisers' policies are fixed and known during both training and deployment
    Section 4 and the note after Algorithm 2. If background policies drift, the learned environment model is mis-specified.
  • domain assumption The next local states and the reward follow a Gaussian distribution
    Section 5.1.1, Eq. (12) reparameterization. Justifies the negative log-likelihood loss (13) and the covariance output structure.
  • domain assumption For each advertiser, usually only the highest bid of other advertisers influences its reward and state transition
    Section 5.1.1, Eq. (8). Justifies the max pooling term; if lower bids jointly matter, the max-pooled feature loses information.
  • standard math Every permutation invariant function can be represented as φ(pool(ψ(·))) with sum pooling (Theorem 2 in [28], Deep Sets)
    Section 5.1.1, Lemma 5.3, cited from Zaheer et al. 2017. Used to justify the PI neural network structure.
  • standard math Covering-number generalization bounds (Lemma B.2 and B.6 in [4]) and the performance-difference lemma (Lemma 4.3 in [17])
    Appendices C and D. Imported from cited literature and used directly in the proofs of Theorem 5.2 and Proposition 5.4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Permutation Equivariant Model-based Offline Reinforcement Learning for Auto-bidding." pith.science (2026). https://pith.science/paper/VUWRWNXP

@misc{pith2026250617919,
  author       = {Pith},
  title        = {Pith review of: Permutation Equivariant Model-based Offline Reinforcement Learning for Auto-bidding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VUWRWNXP}},
  note         = {Machine review of arXiv:2506.17919}
}
read the original abstract

Reinforcement learning (RL) for auto-bidding has shifted from using simplistic offline simulators (Simulation-based RL Bidding, SRLB) to offline RL on fixed real datasets (Offline RL Bidding, ORLB). However, ORLB policies are limited by the dataset's state space coverage, offering modest gains. While SRLB expands state coverage, its simulator-reality gap risks misleading policies. This paper introduces Model-based RL Bidding (MRLB), which learns an environment model from real data to bridge this gap. MRLB trains policies using both real and model-generated data, expanding state coverage beyond ORLB. To ensure model reliability, we propose: 1) A permutation equivariant model architecture for better generalization, and 2) A robust offline Q-learning method that pessimistically penalizes model errors. These form the Permutation Equivariant Model-based Offline RL (PE-MORL) algorithm. Real-world experiments show that PE-MORL outperforms state-of-the-art auto-bidding methods.

Figures

Figures reproduced from arXiv: 2506.17919 by the authors.

Figure 1
Figure 1. The proposed PE-MORL algorithm is designed based on the MBRL paradigm. The PE-MORL follows the MBRL [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The Wasserstein distance between the policy trajec [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Policy trajectories of two specific representative [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: An illustration of the bidding process in an industrial advertising system between time step [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 17 canonical work pages

  1. [1]

    Nair Ashvin, Dalal Murtaza, Gupta Abhishek, and L Sergey. 2020. Accelerating online reinforcement learning with offline datasets.CoRR, vol. abs/2006.09359 (2020)

    arXiv 2020

  2. [2]

    Han Cai, Kan Ren, Weinan Zhang, Kleanthis Malialis, Jun Wang, Yong Yu, and Defeng Guo. 2017. Real-time bidding by reinforcement learning in display adver- tising. InProceedings of the tenth ACM international conference on web search and data mining. 661–670. Permutation Equivariant Model-based Offline Reinforcement Learning for Auto-bidding Conference acr...

    work page 2017

  3. [3]

    Shuang Chen, Qisen Xu, Liang Zhang, Yongbo Jin, Wenhao Li, and Linjian Mo

    work page

  4. [4]

    Zhijian Duan, Yunxuan Ma, and Xiaotie Deng. 2023. Are Equivariant Equilibrium Approximators Beneficial?arXiv preprint arXiv:2301.11481(2023)

    work page Pith review arXiv 2023

  5. [5]

    Zhijian Duan, Jingwu Tang, Yutong Yin, Zhe Feng, Xiang Yan, Manzil Zaheer, and Xiaotie Deng. 2022. A context-integrated transformer-based neural network for auction design. InInternational Conference on Machine Learning. PMLR, 5609– 5626

    work page 2022

  6. [6]

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. InInternational conference on machine learning. PMLR, 1861– 1870

    2018

  7. [7]

    Xiaotian Hao, Zhaoqing Peng, Yi Ma, Guan Wang, Junqi Jin, Jianye Hao, Shan Chen, Rongquan Bai, Mingzhou Xie, Miao Xu, et al. 2020. Dynamic knapsack op- timization towards efficient multi-channel sequential advertising. InInternational Conference on Machine Learning. PMLR, 4060–4070

    work page 2020

  8. [8]

    Yue He, Xiujun Chen, Di Wu, Junwei Pan, Qing Tan, Chuan Yu, Jian Xu, and Xiaoqiang Zhu. 2021. A unified solution to constrained bidding in online display advertising. InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2993–3001

    2021

Show all 31 references
  1. [9]

    Junqi Jin, Chengru Song, Han Li, Kun Gai, Jun Wang, and Weinan Zhang. 2018. Real-time bidding with multi-agent reinforcement learning in display advertis- ing. InProceedings of the 27th ACM international conference on information and knowledge management. 2193–2201

    2018

  2. [10]

    Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. 2019. Model-based reinforcement learning for atari.arXiv preprint arXiv:1903.00374(2019)

    2019 arXiv

  3. [11]

    Alex Kendall and Yarin Gal. 2017. What uncertainties do we need in bayesian deep learning for computer vision?Advances in neural information processing systems30 (2017)

    2017

  4. [12]

    Dmytro Korenkevych, Frank Cheng, Artsiom Balakir, Alex Nikulkov, Ling- nan Gao, Zhihao Cen, Zuobing Xu, and Zheqing Zhu. 2023. Offline rein- forcement learning for optimizing production bidding policies.arXiv preprint arXiv:2310.09426(2023)

    2023 arXiv

  5. [13]

    2016.Partially observed Markov decision processes

    Vikram Krishnamurthy. 2016.Partially observed Markov decision processes. Cam- bridge university press

    2016

  6. [14]

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conserva- tive q-learning for offline reinforcement learning.Advances in Neural Information Processing Systems33 (2020), 1179–1191

    2020

  7. [15]

    Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. InInternational conference on machine learning. PMLR, 3744–3753

    2019

  8. [16]

    Yan Li, Lingxiao Wang, Jiachen Yang, Ethan Wang, Zhaoran Wang, Tuo Zhao, and Hongyuan Zha. 2021. Permutation invariant policy optimization for mean- field multi-agent reinforcement learning: A principled approach.arXiv preprint arXiv:2105.08268(2021)

    2021 arXiv

  9. [17]

    Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. 2018. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees.arXiv preprint arXiv:1807.03858(2018)

    2018 arXiv

  10. [18]

    Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. 2020. Monte carlo gradient estimation in machine learning.The Journal of Machine Learning Research21, 1 (2020), 5183–5244

    2020

  11. [19]

    Zhiyu Mou, Yusen Huo, Rongquan Bai, Mingzhou Xie, Chuan Yu, Jian Xu, and Bo Zheng. 2022. Sustainable Online Reinforcement Learning for Auto-bidding. Advances in Neural Information Processing Systems35 (2022), 2651–2663

    2022

  12. [20]

    Athanasios S Polydoros and Lazaros Nalpantidis. 2017. Survey of model-based reinforcement learning: Applications on robotics.Journal of Intelligent & Robotic Systems86, 2 (2017), 153–173

    2017

  13. [21]

    Tian Qin, Fengxiang He, Dingfeng Shi, Wenbing Huang, and Dacheng Tao. 2022. Benefits of permutation-equivariance in auction mechanisms.Advances in Neural Information Processing Systems35 (2022), 18131–18142

    2022

  14. [22]

    Eric Schulz, Maarten Speekenbrink, and Andreas Krause. 2018. A tutorial on Gaussian process regression: Modelling, exploring, and exploiting functions. Journal of Mathematical Psychology85 (2018), 1–16

    2018

  15. [23]

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems12 (1999)

    1999

  16. [24]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.Advances in neural information processing systems30 (2017)

    2017

  17. [25]

    Chao Wen, Miao Xu, Zhilin Zhang, Zhenzhe Zheng, Yuhui Wang, Xiangyu Liu, Yu Rong, Dong Xie, Xiaoyang Tan, Chuan Yu, et al. 2022. A cooperative-competitive multi-agent framework for auto-bidding in online advertising. InProceedings of the Fifteenth ACM International Conference ...

    2022

  18. [26]

    Di Wu, Xiujun Chen, Xun Yang, Hao Wang, Qing Tan, Xiaoxun Zhang, Jian Xu, and Kun Gai. 2018. Budget constrained bidding by model-free reinforcement learning in display advertising. InProceedings of the 27th ACM International Conference on Information and Knowledge Management. ...

    2018

  19. [27]

    Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. 2020. Mopo: Model-based offline policy optimization.Advances in Neural Information Processing Systems33 (2020), 14129– 14142

    2020

  20. [28]

    Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. 2017. Deep sets.Advances in neural information processing systems30 (2017)

    2017

  21. [29]

    Jun Zhao, Guang Qiu, Ziyu Guan, Wei Zhao, and Xiaofei He. 2018. Deep rein- forcement learning for sponsored search real-time bidding. InProceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 1021–1030. A RELATED WORKS A.1 Mean Field G...

    2018

  22. [31]

    HereX andY denote the contextual feature space of the advertiser and the impression opportunity, respectively

    Specifically, there are 𝑁− 1background advertisers and one representative advertiser bidding for𝑀∈N + impression opportu- nities, where𝑥𝑖∈X denotes the contextual feature of advertiser𝑖, and𝑦𝑖∈Y denotes the contextual feature of the𝑗-th impression opportunity,𝑗∈[𝑀] . HereX and...

    2018

  23. [2023]

    InProceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems

    Model-Based Reinforcement Learning for Auto-bidding in Display Adver- tising. InProceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems. 1560–1568

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.