REVIEW 3 major objections 6 minor 36 references
BAT: Benchmark for Auto-bidding Task
T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Avito benchmark opens 11 million real auctions to autobidding research.
desk verdict The genuine contribution here is the dataset — a new public benchmark with both first-price and VCG auction logs — while the baseline comparisons run on an unvalidated simulator and should be treated as sanity checks, not live-auction rankings. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the auction statistics table, where each row corresponds to a campaign-hour-bid-bin and contains 'surplus' fields: AuctionClicksSurplus, AuctionContactsSurplus, AuctionVisibilitySurplus, and AuctionWinBidSurplus. Each surplus is the expected incremental gain from raising the bid by one bin relative to the previous bin, and the WinBidSurplus is the expected additional spend. The simulation uses these fields to compute the reward of each bid: for a given bin, the win price is the sum of WinBidSurplus up to that bin (for VCG) or the bid itself (for FP), and the click yield is the sum of ClickSurplus up to that bin. The algorithm's only decision at each hour is which bin to choose, and the environment is a deterministic function of that choice, with no feedback from the bidder's history or from competing bids.
What would settle it
A direct test would be to run the simulator's recommended bid sequence on a live auction platform for a sample of campaigns and compare the realized hourly spend and clicks to the simulated values. If the observed differences cannot be explained by random noise and systematically grow over time, the assumption that the surplus fields are fixed responses is untenable.
Extended reading notes
Core claim
The central claim is that BAT is a valid and practical benchmark for autobidding research. The dataset contains, for each of roughly 9000 advertising campaigns, hourly aggregated statistics that describe the expected incremental clicks, contacts, visibility, and spend that a campaign would gain by moving its bid from one discrete price bin to the next. The paper demonstrates the benchmark's usability by implementing five algorithms from the literature and by showing that the proposed ALM and TA-PID algorithms, together with the modified M-PID, achieve competitive performance on the three studied tasks. In particular, on the click-sum maximization experiment, TA-PID and ALM outperform the others, while Mystique wins on budget pacing in both auction formats. The authors present their results as evidence that the dataset can support meaningful algorithm development and comparison.
Load-bearing premise
The benchmark's simulation assumes that the aggregated surplus fields (AuctionClicksSurplus, AuctionWinBidSurplus, etc.) give fixed, predictable responses to a campaign's bid, independent of how much the bidder itself affects the auction or how competitors adjust.
Editorial extensions
If this is right
- If the benchmark is correct, researchers can test budget pacing and CPC-constrained bidding algorithms on production-scale data without access to proprietary advertising platforms.
- The inclusion of both FP and VCG auctions allows the same algorithm to be studied under two pricing rules, which matters as platforms continue to migrate from second-price to first-price auctions.
- The five implemented baselines provide ready reference points, so a new autobidding method can be compared against adaptive pacing, PID control, and optimization-based approaches using the same simulator.
- The proposed metrics, RMSE_T and relative CPC, give standard quantitative targets that make hyperparameter tuning and cross-paper comparison more straightforward.
- The hour-level traffic share data can support time-aware pacing algorithms that account for the roughly thirty-fold variation in traffic across the day.
Reading between the lines
- Because the auction feedback is pre-aggregated per bid bin, the dataset could be used for off-policy evaluation of bid policies, not only for the provided closed-loop simulator.
- The two-auction-format design invites a direct study of bid shading: comparing how much a given surplus response changes between VCG and FP pricing could reveal how much of an FP bid should be shaded relative to the equivalent VCG bid.
- The fixed-surplus response format could be replaced with a learned response model from the same fields, which would let future work test how sensitive algorithm rankings are to the stationarity assumption.
- The traffic table and the reported category-level differences suggest a natural extension: benchmarking algorithms that are specialized to particular logical categories rather than tuned globally.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces BAT (Benchmark for Auto-bidding Task), a public dataset from Avito containing roughly 10 million first-price (FP) auctions and 1 million VCG auctions, together with campaign metadata, traffic distributions, and aggregated auction statistics. The authors describe the data collection and filtering process, define a simulated auction environment based on aggregated surplus fields, and propose several baselines: two new algorithms (ALM and TA-PID), a modified M-PID, and reimplementations of Mystique and BROI. They define metrics for budget pacing (RMSE_T), CPC satisfaction (REL_CPC), and total clicks (SCR), and report experiments on a subset of logical category 1, with hyperparameters tuned on an S1 split and evaluated on a disjoint S2 split. The paper claims that the benchmark supports development and evaluation of autobidding algorithms and that two of the proposed baselines have proven effective in production.
Significance. If the benchmark delivers what it promises, it addresses a genuine scarcity of open, large-scale autobidding datasets with two distinct auction formats. The dataset scale (10M FP and 1M VCG auctions, thousands of campaigns) and the public code repository are concrete strengths, and the S1/S2 split with Optuna-based hyperparameter tuning is a sensible attempt at avoiding overfitting in the baseline comparisons. The paper also provides clear metrics and a reproducible simulation harness. However, the central evaluation claim is currently conditional: all baseline rankings are produced inside a simulator whose reward model is an unvalidated map from bids to aggregated surplus values. The data contribution itself is likely valuable, but the benchmark's usefulness for ranking autobidding algorithms depends on evidence that this simulator reflects real auction dynamics, which the paper does not yet provide.
major comments (3)
- [Sections 4.2, 5, 10] The simulation treats the aggregated surplus fields (AuctionClicksSurplus, AuctionWinBidSurplus, AuctionContactsSurplus) as fixed causal responses to a campaign's bid, but these fields are defined in Section 4.2 as marginal differences between adjacent price bins computed from logged auctions, not as structural counterfactuals. Section 10 then applies them deterministically at every timestamp, implicitly assuming that changing one bidder's bid does not change the distribution of competing bids, the winning price, or other auction conditions. The paper provides no off-policy check, stability analysis, or live A/B validation connecting simulated outcomes to realized auction outcomes. Because Tables 6-8 and 10 are all generated inside this fixed-response loop, the benchmark's claim to evaluate autobidding algorithms is currently conditional on an untested invariance assumption.
- [Section 10, FP cost model] For FP auctions, the simulator sets the expected price to the product of AuctionContactsSurplus and the bid, 'as if we buy each auction with the defined bid.' However, AuctionContactsSurplus is defined in Section 4.2 as the expected increase in contacts relative to the previous price bin, which is not a win probability. Multiplying a marginal contact increment by the full bid does not yield the expected cost of winning a first-price auction. This conflation can systematically distort budget depletion and click accounting in the FP simulation, and it may explain the large asymmetry in Table 7 (M-PID REL_CPC 0.49 for FP vs 0.91 for VCG). The authors should either justify the cost model from the data-generating process or replace it with a quantity that is actually interpretable as a win probability or expected payment.
- [Tables 6, 7, 8, 10 and Section 10] The experimental results are reported as single numbers with no error bars, standard deviations, or significance tests. Hyperparameters are selected with a stochastic Bayesian optimizer (Optuna) on S1, and the simulation itself is deterministic given those parameters, so the reported point estimates do not convey uncertainty about the Optuna runs or about the choice of campaign subset. Several published differences are small (e.g., Table 6 VCG: TA-PID 1.42 vs M-PID 1.38; Table 8 FP: ALM 1,085,836 vs BROI 1,098,184), and the text draws comparative conclusions such as 'the leading algorithms, TA-PID and ALM, achieve very similar results.' Without repeated runs, confidence intervals, or a sensitivity analysis, the comparative claims in Section 11 are not statistically supported.
minor comments (6)
- [Appendix B, Algorithms 1 and 2] Algorithm 1 line 6 and Algorithm 2 line 7 write 'Play bid b ← −γ Δ_n'; the minus sign and the missing exponent contradict Section 8, where the bid is defined as b = γ^Δ. This should be corrected to b ← γ^Δ_n.
- [Section 8.1] The update formula 'Δ_n = Δ_{n-1} + B_left · β' mixes a price-bin index with a relative remaining budget; the dimensions and intended semantics of the update should be clarified, and the clip boundaries mentioned in the text should appear explicitly in Algorithm 1.
- [Section 9.1] The text refers to 'the first 4 baselines (Linear, TA-PID, M-PID, Mystique)', but the algorithm introduced in Section 8.1 is called ALM (Adaptive Linear Model); the naming should be consistent.
- [Sections 7 and 10] Section 7 states that the budget decreases by the bid only if the user clicks, while Section 10 says the budget is reduced at every timestamp by the expected price of auction participation; the payment rule for FP and VCG should be stated unambiguously and consistently.
- [Table 2 and Section 5] Table 2 shows AuctionCount only for the VCG example and a dash for the FP example, yet Section 5 reports that FP auctions were logged at 100%; the authors should clarify whether AuctionCount exists for FP and why it is not shown.
- [Table 7] The paper should state the interpretation of REL_CPC explicitly (values below 1 satisfy the CPC constraint, values above 1 violate it) so that the reader can interpret values such as 0.49 and 0.91 without ambiguity.
Circularity Check
No significant circularity: the benchmark dataset and simulation are presented as an environment, and the reported evaluations use held-out splits and external baselines.
full rationale
The paper's central deliverable is a dataset plus a simulation environment, not a derived theoretical prediction. The surplus fields (e.g., AuctionClicksSurplus, AuctionWinBidSurplus) are explicitly defined as observational aggregates in Section 4.2 and then used in Section 10 as the reward and cost model for the simulator; this is an environment definition, not a claim that the environment predicts live outcomes from first principles. Baseline hyperparameters are tuned on S1 and evaluated on S2 (Section 10), so the reported metrics are not fitted-input predictions of the same data used for tuning. The cited algorithms (Mystique, M-PID, BROI) are external prior work; there are no load-bearing self-citations and no uniqueness theorem imported from the authors' own prior papers. The closest element is the standard benchmark-paper practice of the authors proposing baselines and evaluating them on their own benchmark, but that does not make any derivation equivalent to its inputs. Concerns about simulator validity (e.g., treating surplus fields as fixed causal responses) are correctness and generalization risks, not circularity.
Assumptions & free parameters
free parameters (4)
- b0 (cold start bid) =
not reported
- beta (ALM control parameter) =
not reported
- clip boundaries (ALM) =
not reported
- PID coefficients kp, ki, kd (TA-PID and M-PID) =
not reported
assumptions (3)
- domain assumption The auction environment is stationary; aggregated surplus curves are valid counterfactual responses.
- domain assumption CTR and CVR aggregated by item features are sufficient statistics for predicting auction outcomes at any bid.
- standard math Linear programming primal-dual optimal bid formula from Yang et al. [33] is correct.
Cite this review
Pith. "Pith review of BAT: Benchmark for Auto-bidding Task." pith.science (2026). https://pith.science/paper/ZOALXLSV
@misc{pith2026250508485,
author = {Pith},
title = {Pith review of: BAT: Benchmark for Auto-bidding Task},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZOALXLSV}},
note = {Machine review of arXiv:2505.08485}
}
read the original abstract
The optimization of bidding strategies for online advertising slot auctions presents a critical challenge across numerous digital marketplaces. A significant obstacle to the development, evaluation, and refinement of real-time autobidding algorithms is the scarcity of comprehensive datasets and standardized benchmarks. To address this deficiency, we present an auction benchmark encompassing the two most prevalent auction formats. We implement a series of robust baselines on a novel dataset, addressing the most salient Real-Time Bidding (RTB) problem domains: budget pacing uniformity and Cost Per Click (CPC) constraint optimization. This benchmark provides a user-friendly and intuitive framework for researchers and practitioners to develop and refine innovative autobidding algorithms, thereby facilitating advancements in the field of programmatic advertising. The implementation and additional resources can be accessed at the following repository (https://github.com/avito-tech/bat-autobidding-benchmark, https://doi.org/10.5281/zenodo.14794182).
Figures
Reference graph
Works this paper leans on
-
[1]
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Frame- work. In Proceedings of the 25th ACM SIGKDD International Conference on Knowl- edge Discovery and Data Mining . Association for Computing Machinery, AK, Anchorage, USA, 2623 – 2631. https://doi.org/10.1145/3292...
arXiv 2019
-
[2]
Kareem Amin, Michael Kearns, Peter Key, and Anton Schwaighofer. 2012. Budget optimization for sponsored search: Censored learning in mdps. In Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence (Catalina Island, CA) (UAI’12). AUAI Press, Arlington, Virginia, USA, 54–63
work page 2012
-
[3]
Dirk Bergemann, Benjamin Brooks, and Stephen Morris. 2017. First-price auc- tions with general information structures: Implications for bidding and revenue. Econometrica 85, 1 (2017), 107–143. https://doi.org/10.3982/ECTA13958
-
[4]
Han Cai, Kan Ren, Weinan Zhang, Kleanthis Malialis, Jun Wang, Yong Yu, and Defeng Guo. 2017. Real-time bidding by reinforcement learning in display ad- vertising. In Proceedings of the tenth ACM international conference on web search and data mining . Association for Computing Machinery, New York, NY, USA, 661–670. https://doi.org/10.1145/3018661.3018702
arXiv 2017
-
[6]
Ye Chen, Pavel Berkhin, Bo Anderson, and Nikhil R Devanur. 2011. Real-time bidding algorithms for performance-based display ad allocation. InProceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining. Association for Computing Machinery, New York, NY, USA, 1307–1315. https://doi.org/10.1145/2020408.2020604
arXiv 2011
-
[7]
Abebe Hailu and Sophie Thoyer. 2006. Multi-unit auction format design. Journal of Economic Interaction and Coordination 1, 2 (2006), 129–146. https://doi.org/10. 1007/s11403-006-0012-9
work page 2006
-
[8]
Yanjun Han, Zhengyuan Zhou, Aaron Flores, Erik Ordentlich, and Tsachy Weissman. 2020. Learning to bid optimally and efficiently in adversarial first- price auctions. arXiv preprint arXiv:2007.04568 abs/2007.04568 (2020), 1–41. https://doi.org/10.48550/arXiv.2007.04568
-
[9]
Sarma Tangirala Harrison Chu, John M. Trenkle and Andrew Wang. 2020. Using a PID controller engine for controlling the pace of an online campaign in realtime. US 10,878,448 B1
work page 2020
Show all 36 references
-
[10]
Jim Jansen and Tracy Mullen. 2008. Sponsored search: An overview of the concept, history, and technology. IJEB 6 (01 2008), 114–131. https://doi.org/10. 1504/IJEB.2008.018068
2008 arXiv
-
[11]
Grégoire Jauvion, Nicolas Grislain, Pascal Dkengne Sielenou, Aurélien Garivier, and Sébastien Gerchinovitz. 2018. Optimization of a ssp’s header bidding strategy using thompson sampling. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and ...
2018 doi
-
[12]
Olivier Jeunen, Sean Murphy, and Ben Allison. 2022. Learning to bid with AuctionGym. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining AdKDD Workshop (Washington DC, USA) (AdKDD ’22). ACM, New York, NY, USA, 1–6
2022
-
[13]
Olivier Jeunen, Sean Murphy, and Ben Allison. 2023. Off-Policy Learning- to-Bid with AuctionGym. In Proceedings of the 29th ACM SIGKDD Confer- ence on Knowledge Discovery and Data Mining (Long Beach, CA, USA) (KDD ’23). Association for Computing Machinery, New York, NY, USA, 4...
2023
-
[14]
Bernhard Kasberger and Karl H Schlag. 2023. Robust bidding in first-price auctions: How to bid without knowing what others are doing. Management Science 70, 7 (2023), 4219 – 4235. https://doi.org/10.2139/ssrn.3044438
2023 doi
-
[15]
Kuang-Chih Lee, Ali Jalali, and Ali Dasdan. 2013. Real time bid optimization with smooth budget delivery in online advertising. In Proceedings of the seventh international workshop on data mining for online advertising . Association for Computing Machinery, New York, NY, USA, ...
2013 doi
-
[16]
Bin Li, Xiao Yang, Daren Sun, Zhi Ji, Zhen Jiang, Cong Han, and Dong Hao. 2020. Incentive Mechanism Design for ROI-constrained Auto-bidding. In PRICAI 2024: Trends in Artificial Intelligence. Springer, Singapore, 291–296. https://doi.org/10. 1007/978-981-96-0125-7_24
2020
-
[17]
Hairen Liao, Lingxiao Peng, Zhenchuan Liu, and Xuehua Shen. 2014. iPinYou global rtb bidding algorithm competition dataset. In Proceedings of the Eighth International Workshop on Data Mining for Online Advertising . Association for Computing Machinery, New York, NY, USA, 1–6. ...
2014 doi
-
[18]
Junwei Lu, Chaoqi Yang, Xiaofeng Gao, Liubin Wang, Changcheng Li, and Guihai Chen. 2019. Reinforcement learning with sequential information clustering in real-time bidding. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (Beijing...
2019
-
[19]
Brendan Lucier, Sarath Pattathil, Aleksandrs Slivkins, and Mengxiao Zhang
-
[20]
Muthukrishnan
S. Muthukrishnan. 2009. Ad Exchanges: Research Issues. In Internet and Network Economics, Stefano Leonardi (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 1–12
2009
-
[21]
Gali Noti and Vasilis Syrgkanis. 2021. Bid Prediction in Repeated Auctions with Learning. In Proceedings of the Web Conference 2021 (Ljubljana, Slovenia) (WWW ’21). Association for Computing Machinery, New York, NY, USA, 3953–3964. https://doi.org/10.1145/3442381.3449968
2021
-
[22]
Weitong Ou, Bo Chen, Xinyi Dai, Weinan Zhang, Weiwen Liu, Ruiming Tang, and Yong Yu. 2023. A survey on bid optimization in real-time bidding display advertising. ACM Transactions on Knowledge Discovery from Data 18, 3 (2023), 1–31. https://doi.org/10.1145/3628603
2023 doi
-
[23]
Renato Paes Leme, Balasubramanian Sivan, and Yifeng Teng. 2020. Why do competitive markets converge to first-price auctions?. In Proceedings of The Web Conference 2020 (Taipei, Taiwan) (WWW ’20). Association for Computing Ma- chinery, New York, NY, USA, 596–605. https://doi.or...
2020
- [24]
-
[25]
Claudia Perlich, Brian Dalessandro, Rod Hook, Ori Stitelman, Troy Raeder, and Foster Provost. 2012. Bid optimizing and inventory scoring in targeted online advertising. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining (Beiji...
2012
-
[26]
Kan Ren, Yuchen Fang, Weinan Zhang, Shuhao Liu, Jiajun Li, Ya Zhang, and Yong Yu. 2018. Learning multi-touch conversion attribution with dual-attention mechanisms for online advertising. In Proceedings of the 27th ACM International Conference on Information and Knowledge Manag...
2018
-
[27]
Nirmal Roy, David Maxwell, and Claudia Hauff. 2022. Users and Contemporary SERPs: A (Re-) Investigation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (Madrid, Spain) (SIGIR ’22). Association for Computing Ma...
2022
-
[28]
Rotem Stram, Rani Abboud, Alex Shtoff, Oren Somekh, Ariel Raviv, and Yair Koren. 2024. Mystique: A Budget Pacing System for Performance Optimization in Online Advertising. In Companion Proceedings of the ACM Web Conference 2024 (Singapore, Singapore) (WWW ’24). Association for...
2024
-
[29]
Sicco Verwer, Yingqian Zhang, and Qing Chuan Ye. 2017. Auction optimization using regression trees and linear models as integer programs.Artificial Intelligence 244 (2017), 368–395. https://doi.org/10.1016/j.artint.2015.05.004
2017 doi
-
[30]
Yuchen Wang, Kan Ren, Weinan Zhang, Jun Wang, and Yong Yu. 2016. Functional bid landscape forecasting for display advertising. In European Conference on Machine Learning and Knowledge Discovery in Databases - Volume 9851 (Riva del Garda, Italy) (ECML PKDD 2016). Springer-Verla...
2016 doi
-
[31]
Wush Chi-Hsuan Wu, Mi-Yen Yeh, and Ming-Syan Chen. 2015. Predicting win- ning price in real time bidding with censored data. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Sydney, NSW, Australia)(KDD ’15). Association for...
2015
-
[32]
Wellman, Chuan Yu, Shuai Dou, Yusen Huo, Zhiwei Xu, Zhijian Duan, Shaopan Xiong, Chuang Liu, Ningyuan Li, Kefan Su, Wei Gong, and Bo Zheng
Jian Xu, Zhilin Zhang, Zongqing Lu, Xiaotie Deng, Michael P. Wellman, Chuan Yu, Shuai Dou, Yusen Huo, Zhiwei Xu, Zhijian Duan, Shaopan Xiong, Chuang Liu, Ningyuan Li, Kefan Su, Wei Gong, and Bo Zheng. 2024. Auto-Bidding in Large-Scale Auctions: Learning Decision-Making in Unce...
2024
-
[33]
Xun Yang, Yasong Li, Hao Wang, Di Wu, Qing Tan, Jian Xu, and Kun Gai. 2019. Bid Optimization by Multivariable Control in Display Advertising. InProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Anchorage, AK, USA) (KDD ’19). Asso...
2019
-
[34]
Dawei Yin, Yuening Hu, Jiliang Tang, Tim Daly, Mianwei Zhou, Hua Ouyang, Jianhui Chen, Changsung Kang, Hongbo Deng, Chikashi Nobata, et al . 2016. Ranking relevance in yahoo search. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data ...
2016
-
[35]
Shuai Yuan and Jun Wang. 2012. Sequential selection of correlated ads by pomdps. In Proceedings of the 21st ACM International Conference on Information and Knowl- edge Management (Maui, Hawaii, USA) (CIKM ’12) . Association for Comput- ing Machinery, New York, NY, USA, 515–524...
2012 doi
-
[36]
S. Yuan, J. Wang, and X. Zhao. 2013. Real-time bidding for online advertising: measurement and analysis. InProceedings of the Seventh International Workshop on Data Mining for Online Advertising (Chicago, Illinois) (ADKDD ’13). Association for Computing Machinery, New York, NY...
2013
-
[2024]
In Proceedings of Thirty Seventh Conference on Learning Theory (Pro- ceedings of Machine Learning Research, Vol
Autobidders with budget and roi constraints: Efficiency, regret, and pacing dynamics. In Proceedings of Thirty Seventh Conference on Learning Theory (Pro- ceedings of Machine Learning Research, Vol. 247) , Shipra Agrawal and Aaron Roth (Eds.). PMLR, Edmonton, Canada, 3642–3643...
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.