Pith. sign in

REVIEW 5 major objections 7 minor 18 references

Optimal Return-to-Go Guided Decision Transformer for Auto-Bidding in Advertisement

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a self-augmenting Decision Transformer variant, R* DT, learns the optimal return-to-go target and outperforms DT, BC, and IQL on ad auto-bidding benchmarks.

desk verdict A sensible, applied extension of Decision Transformer for auto-bidding with a real gap in its theoretical justification: the claimed monotone convergence to optimal RTG is asserted, not proven. read the letter →

arxiv 2506.21956 v1 pith:Y73HFRDI submitted 2025-06-27 cs.LG

Hao Jiang , Yongxiang Tang , Yanxiang Zeng , Pengjia Yuan , Yanhua Cheng , Teng Sha , Xialong Liu , Peng Jiang This is my paper
classification cs.LG
keywords offlinereinforcementlearningdecisiontransformerauto-biddingreturn-to-goquantileregressiondataaugmentationadvertisingauctionssimulator-basedtrajectoryselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's aim is to remove two weak points of a Decision Transformer when used for auto-bidding in ad auctions: someone has to preset a return-to-go (RTG) value, and the policy is stuck with whatever quality of trajectories the training data contains. The authors build R* DT in three steps: a variant that memorizes RTG per state, a variant that estimates the upper quantile of RTG for a state, and a loop that generates new trajectories with that estimate, keeps the high-reward ones via a simulator, and retrains. The claim is that this loop raises the RTG of the training set monotonically toward the optimal value, so the policy approaches optimality without human-specified targets. If true, generative sequence-modeling agents become more practical for ad bidding, since the model supplies its own target and improves its own data.

What carries the argument

The load-bearing object is the $\hat{R}$ loss, an asymmetric quantile-regression loss $$L(\hat{R}_t(s_t)) = (1-\$\lambda$)\,\mathrm{ReLU}(R_t - \hat{R}_t(s_t)) + \$\lambda$\,\mathrm{ReLU}(\hat{R}_t(s_t) - R_t),$$ whose minimizer is the $(1-\lambda)$-th quantile of RTG for a state. As $\lambda \to 0$, $\hat{R}_t$ approaches the maximum RTG present in the training set, giving the policy a target it can aim at without a human-supplied value. The second mechanism is the augmentation loop: $\hat{R}$ DT rolls out actions in the simulator, keeps trajectories with high total RTG, adds them to the training set, and retrains; with nested training sets $T^{(1)} \subset \cdots \subset T^{(k)}$, the paper argues the estimated RTG ceiling rises monotonically toward the true optimal $R^*_t$.

What would settle it

Give the loop a simulator with a known reward error, such as overestimating conversions by 20 percent, and compare the final policy on the true environment; if the augmented training set degrades rather than improves true performance, the convergence claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the optimal return-to-go $R^*_t(s_t)$ can be approached by a self-augmenting chain of suboptimal estimates. Because each augmentation round adds only high-reward trajectories and the estimator $\hat{R}^{(k)}_t$ is the supremum of RTG over the enlarged training set, the sequence satisfies $\hat{R}^{(1)}_t < \cdots < \hat{R}^{(k)}_t < R^*_t$. The paper reports that $R^*$ DT, trained after several such rounds, outscores DT, BC, and IQL on all three test datasets and produces trajectories whose RTG distribution shifts upward from roughly 2-50 to 20-80 by the fifth iteration.

Load-bearing premise

The loop works only if the simulator's reward estimates faithfully reflect the real auction's rewards; if the simulator is biased, the 'high-quality' trajectories it selects are biased too, and the whole chain of improving RTG estimates inherits that bias.

Editorial extensions

If this is right

  • Advertisers can drop manual RTG tuning because R* DT predicts a state-dependent return-to-go from data.
  • Each augmentation round should push the training set's RTG ceiling upward, so the policy's performance should improve with each iteration.
  • R* DT should outperform DT, BC, and IQL under target-CPA constraints, as it did on all three datasets in the paper.
  • The method's behavior differs by CPA target: high targets favor conversion optimization, low targets favor CPA-penalty minimization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's monotone-improvement argument does not depend on the Transformer itself; the same 'estimate the upper envelope, roll out, filter, retrain' loop could be bolted onto any offline policy that returns a value estimate, so the core idea may transfer beyond Decision Transformers.
  • A diagnostic suggested by the argument: plot predicted $\hat{R}$ on a fixed set of states across iterations; when the curve stops rising, further augmentation rounds are no longer buying information. The paper does not report that plot.
  • In a live advertising system the simulator could be replaced by online exploration with a safety filter, converting the closed-loop augmentation into a continual-improvement mechanism; that online setting is outside the paper's experiments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper introduces the R* Decision Transformer (R* DT) for auto-bidding in online advertising. It first modifies the Decision Transformer input format to memorize return-to-go (RTG) values (R DT), then trains a quantile-regression-based estimator to predict a high RTG value for each state (R-hat DT), and finally iteratively generates trajectories with the resulting policy in a simulator, selects high-RTG trajectories, and augments the training set. Experiments on three variants of a public auto-bidding dataset compare R* DT with DT, BC, and IQL, and an ablation tracks five augmentation rounds. The paper claims that the iterative procedure raises the estimated RTG toward the true optimal R* and thereby gradually approaches the optimal policy.

Significance. The paper targets a genuine limitation of Decision Transformers in auto-bidding: the need to preset RTG and the presence of mixed-quality trajectories in offline data. The proposed iterative self-augmentation is simple and, if validated, could be practically useful. The paper does not provide code or machine-checked proofs, and the central optimality claim rests on an unproven monotonicity assertion; the empirical evidence is suggestive but weakened by single-run results and an unvalidated simulator. The contribution is therefore an interesting heuristic with incomplete theoretical and empirical support.

major comments (5)
  1. [Section 2.3, Eq. (2) and following paragraph] The monotone chain R_hat^(1)_t < ... < R_hat^(k)_t < R*_t is asserted, not proven. Nested training sets imply only that the population supremum over the training set is nondecreasing; the learned R_hat^(k)_t is a neural-network quantile estimator, which can decrease, plateau, or overshoot when the training set expands. The strict upper bound R_hat < R* additionally requires that the estimator never overestimates, which is not established. Because this chain is the stated support for the claim that the method 'paves the way towards approximating R*_t', the central theoretical claim is currently unsupported.
  2. [Figure 2 and Section 3.4] The reported increase in RTG is computed on trajectories generated and selected by the algorithm itself; this increase can be produced purely by selection pressure even if the underlying model is unchanged. The paper does not report R_hat values on a fixed set of validation or test states across iterations, so there is no evidence that the estimator itself moves toward R*. Such fixed-set measurements are needed to support the convergence claim.
  3. [Section 2.3, simulator description] The simulator used in the self-augmentation loop is described only as Bernoulli draws with probabilities p_Exposure and p_CVR, together with opponents' eCPM data. No calibration or validation of the simulator against the real evaluation environment is provided. If the simulator's reward structure is biased, the selection loop will amplify that bias; since the reported gains of R*DT over R-hat DT are entirely mediated by the simulator, this is a load-bearing gap.
  4. [Tables 1 and 2, Section 3.3] The experimental results are single runs without error bars, standard deviations, or significance tests. The claim that performance 'improves progressively' is not monotone even in the reported point estimates: on dataset 1, the score drops from 35.42 for R-hat DT to 34.34 for R*DT(1), and only exceeds the R-hat result after further iterations. Repeated-seed experiments with variance reporting are needed before the improvement can be considered reliable.
  5. [Section 2.2, quantile loss and Eq. (1)] The quantile-loss argument only guarantees that the minimizer over the empirical distribution approaches the maximum observed RTG for each state as lambda tends to 0. In a continuous state space, the empirical maximum at the query state is not available, and the neural-network model is not guaranteed to approximate this supremum with finite samples. The relation between the trained R_hat and the true supremum in Eq. (1) needs a more careful statement or an explicit empirical check.
minor comments (7)
  1. [Section 2.2, Eq. (1)] The notation 'sup_{s_t, RT~T}' is imprecise; the supremum should be over trajectories in T whose state matches s_t.
  2. [Section 3.1.2] The definition of score mixes the penalty function P(Ca;Ct) with the summation; please rewrite the equation and text to make the exact order of operations clear.
  3. [Section 3.1.1] The creation of datasets 2 and 3 by altering the target CPA and the preprocessing of the AIGB dataset are not described; include concrete preprocessing steps.
  4. [Section 2.3] The selection criterion 'keeping those with higher RTG' is never quantified; specify the selection threshold or the proportion of generated trajectories retained in each iteration.
  5. [Abstract] The abstract contains grammatical errors such as 'which is consists of' and 'supreme RTG'; use 'consists of' and 'supremal/upper-bound RTG'.
  6. [Reproducibility] No code, architecture details, training hyperparameters, or number of generated trajectories per iteration are provided, which limits reproducibility.
  7. [References] Reference [14] is a competition URL without a version or access date; provide a persistent citation for the dataset.

Circularity Check

1 steps flagged · score 6.0 of 10

The claimed monotone convergence of the RTG estimator to R* is partly by construction: the augmentation rule keeps only high-RTG trajectories, and ^R is defined/estimated as an upper statistic of the training set, so the reported increase is a selection artifact rather than verified optimality.

  1. self definitional [Section 2.3 (Eq. 2 and following paragraph); definition in Section 2.2 Eq. (1)]
    "Given that the sequence of training sets satisfiesT(1)⊂···⊂T (𝑘), and by the definition of ˆ𝑅(𝑘) 𝑡 , we observe ˆ𝑅(1) 𝑡 <··· < ˆ𝑅(𝑘) 𝑡 < 𝑅∗ 𝑡 . This methodology paves the way towards approximating𝑅∗ 𝑡."

    In Eq. (1), the paper defines ˆR_t(s_t) as the supremum of R_t(s_t) over the training set T. With nested training sets, the supremum is nondecreasing, so the reported monotone chain is a direct consequence of the definition plus the expansion rule 'keeping those with higher RTG,' not an empirical or theoretical verification that the learned estimator approaches the true optimum R*. The estimator actually used is a neural quantile regression fit (Section 2.2), not the exact supremum, so the inference equivocates between the mathematical definition and the fitted model. The strict inequalities also do not follow from nesting alone; the claim of convergence to R* is therefore primarily a restatement of the method's own data-selection rule.

full rationale

The paper's strongest independent content is empirical: R* DT is evaluated on held-out test datasets against DT, BC, and IQL, and those comparisons do not reduce to the construction of the training-set statistic. No load-bearing self-citation chain or imported uniqueness theorem is present. The circularity lies in the theoretical convergence argument in Section 2.3. The paper defines ˆR_t as the supremum of RTG over the training set and then asserts, 'by the definition of ˆR^(k)_t', that the sequence ˆR^(1)_t < ... < ˆR^(k)_t < R*_t follows from nested training sets. This is partly a tautology: if ˆR were exactly the training-set supremum, nesting would give monotonicity, but the implemented ˆR is a neural network trained with a quantile loss, so it is a stochastic function of the data and can decrease, plateau, or overshoot. Moreover, the augmentation step selects only trajectories with high total RTG, so the upward shift in the training-set RTG distribution is by construction, not evidence that the policy is approaching the true optimal return. The paper never reports ˆR^(k) on a fixed validation set; Figure 2 shows RTG of generated/selected trajectories, which can rise from selection pressure even if the policy is stationary. Thus the central theoretical claim that the method 'paves the way towards approximating R*_t' is partly a consequence of the method's own selection rule and definition, while the empirical benchmark results retain independent value. This warrants a partial circularity score of 6 rather than a higher score.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method rests on hand-chosen quantities (lambda, simulator probabilities, exploration noise, selection cutoff) and on two unproven premises: simulator fidelity and monotone improvement of quantile estimates. The self-augmentation loop means the predictor helps create the data it is later trained on, so these assumptions are load-bearing for the optimality claim.

free parameters (4)
  • lambda (quantile loss parameter) = not reported
    Weights the asymmetric ReLU loss for ^R_t in Eq. (2); the paper says 'as lambda approaches 0' but reports no value used in experiments.
  • simulator exposure and conversion probabilities (p_Exposure, p_CVR) = not reported
    Bernoulli probabilities used to simulate ad display and conversion in the trajectory generator (Section 2.3); they are data-derived but never given or validated.
  • exploration noise scale (epsilon_t) = not reported
    Added to the ^R prediction in Eq. (2) to create diverse trajectories; scale is not specified.
  • high-RTG trajectory selection threshold = not reported
    The method 'keeps those with higher RTG' (Section 2.3); the cutoff or top-k fraction is not specified.
assumptions (3)
  • domain assumption The simulator faithfully models the auction environment, with opponents' eCPM and Bernoulli exposure/conversion draws reproducing the real reward distribution.
    Invoked in Section 2.3 to generate and select training trajectories; if the simulator is biased, the self-augmentation loop can reinforce errors.
  • ad hoc to paper Adding selected high-RTG trajectories to the training set monotonically increases the estimated conditional upper quantile of RTG, so ^R(1) < ... < ^R(k).
    Asserted in Section 2.3 after Eq. (2); no proof that neural-quantile estimates on a superset of trajectories move monotonically.
  • domain assumption The asymmetric ReLU loss yields a consistent estimator of the conditional upper quantile with the chosen neural network in finite samples.
    Section 2.2 relies on quantile regression consistency; finite-capacity and finite-data guarantees are not given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimal Return-to-Go Guided Decision Transformer for Auto-Bidding in Advertisement." pith.science (2026). https://pith.science/paper/Y73HFRDI

@misc{pith2026250621956,
  author       = {Pith},
  title        = {Pith review of: Optimal Return-to-Go Guided Decision Transformer for Auto-Bidding in Advertisement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y73HFRDI}},
  note         = {Machine review of arXiv:2506.21956}
}
read the original abstract

In the realm of online advertising, advertisers partake in ad auctions to obtain advertising slots, frequently taking advantage of auto-bidding tools provided by demand-side platforms. To improve the automation of these bidding systems, we adopt generative models, namely the Decision Transformer (DT), to tackle the difficulties inherent in automated bidding. Applying the Decision Transformer to the auto-bidding task enables a unified approach to sequential modeling, which efficiently overcomes short-sightedness by capturing long-term dependencies between past bidding actions and user behavior. Nevertheless, conventional DT has certain drawbacks: (1) DT necessitates a preset return-to-go (RTG) value before generating actions, which is not inherently produced; (2) The policy learned by DT is restricted by its training data, which is consists of mixed-quality trajectories. To address these challenges, we introduce the R* Decision Transformer (R* DT), developed in a three-step process: (1) R DT: Similar to traditional DT, R DT stores actions based on state and RTG value, as well as memorizing the RTG for a given state using the training set; (2) R^ DT: We forecast the highest value (within the training set) of RTG for a given state, deriving a suboptimal policy based on the current state and the forecasted supreme RTG value; (3) R* DT: Based on R^ DT, we generate trajectories and select those with high rewards (using a simulator) to augment our training dataset. This data enhancement has been shown to improve the RTG of trajectories in the training data and gradually leads the suboptimal policy towards optimality. Comprehensive tests on a publicly available bidding dataset validate the R* DT's efficacy and highlight its superiority when dealing with mixed-quality trajectories.

Figures

Figures reproduced from arXiv: 2506.21956 by the authors.

Figure 1
Figure 1. Our 𝑅 ∗ Decision Transformer Framework determined by the actual observed return values, whereas during testing, it is predefined. As we demonstrated, DT in the auto-bidding problem comes with its own set of challenges. Two significant concerns include the issues of presetting the return-to-go[10] and the problem of selecting suboptimal trajectories[7, 17]. First, the concept of Return￾to-Go (RTG) is fundamental for … view at source ↗
Figure 2
Figure 2. RTG (via simulator) performance of trajectories [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 13 canonical work pages

  1. [1]

    Gagan Aggarwal, Ashwinkumar Badanidiyuru, Santiago R Balseiro, Kshipra Bhawalkar, Yuan Deng, Zhe Feng, Gagan Goel, Christopher Liaw, Haihao Lu, Mohammad Mahdian, et al. 2024. Auto-bidding and auctions in online advertising: A survey. ACM SIGecom Exchanges 22, 1 (2024), 159–183

    2024

  2. [2]

    Gagan Aggarwal, Ashwinkumar Badanidiyuru, and Aranyak Mehta. 2019. Au- tobidding with constraints. In Web and Internet Economics: 15th International Conference, WINE 2019, New York, NY, USA, December 10–12, 2019, Proceedings 15 . Springer, 17–30

    work page 2019

  3. [3]

    Santiago Balseiro, Yuan Deng, Jieming Mao, Vahab Mirrokni, and Song Zuo. 2021. Robust auction design in the auto-bidding world. Advances in Neural Information Processing Systems 34 (2021), 17777–17788

    work page 2021

  4. [4]

    Santiago R Balseiro, Kshipra Bhawalkar, Zhe Feng, Haihao Lu, Vahab Mirrokni, Balasubramanian Sivan, and Di Wang. 2023. Joint Feedback Loop for Spend and Return-On-Spend Constraints. arXiv preprint arXiv:2302.08530 3 (2023)

    arXiv 2023

  5. [5]

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. 2021. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems 34 (2021), 15084–15097

    work page 2021

  6. [6]

    Kevin Esslinger, Robert Platt, and Christopher Amato. 2022. Deep transformer q-networks for partially observable reinforcement learning. arXiv preprint arXiv:2206.01078 (2022)

    arXiv 2022

  7. [7]

    Michael Janner, Qiyang Li, and Sergey Levine. 2021. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems 34 (2021), 1273–1286

    work page 2021

  8. [8]

    Roger Koenker. 2005. Quantile regression. Cambridge Univ Pr (2005)

    work page 2005

Show all 18 references
  1. [9]

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. 2021. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169 (2021)

    2021 arXiv

  2. [10]

    Kuang-Huei Lee, Ofir Nachum, Mengjiao Sherry Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, et al. 2022. Multi-game decision transformers. Advances in Neural Information Processing Systems 35 (2022), 27921–27936. Optim...

    2022

  3. [11]

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. 2020. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 (2020)

    2020 arXiv

  4. [12]

    Emilio Parisotto and Ruslan Salakhutdinov. 2021. Efficient transformers in reinforcement learning using actor-learner distillation. arXiv preprint arXiv:2104.01655 (2021)

    2021 arXiv

  5. [13]

    Fransisca Susan, Negin Golrezaei, and Okke Schrijvers. 2023. Multi-platform budget management in ad markets with non-ic auctions. arXiv preprint arXiv:2306.07352 (2023)

    2023 arXiv

  6. [14]

    Tianchi. 2024. AIGB Track: Learning Auto-Bidding Agent with Generative Mod- els. https://tianchi.aliyun.com/competition/entrance/532236/customize448? lang=en-us

    2024

  7. [15]

    Uddeshya Upadhyay, Nikunj Shah, Sucheta Ravikanti, and Mayanka Medhe

  8. [16]

    Kerong Wang, Hanye Zhao, Xufang Luo, Kan Ren, Weinan Zhang, and Dongsheng Li. 2022. Bootstrapped transformer for offline reinforcement learning. Advances in Neural Information Processing Systems 35 (2022), 34748–34761

    2022

  9. [17]

    Yueh-Hua Wu, Xiaolong Wang, and Masashi Hamaya. 2024. Elastic decision transformer. Advances in Neural Information Processing Systems 36 (2024)

    2024

  10. [2019]

    arXiv preprint arXiv:1912.03918 (2019)

    Transformer based reinforcement learning for games. arXiv preprint arXiv:1912.03918 (2019)

    2019 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.