{"id":"17627466-2453-4855-b795-6f82d2919f2e","arxiv_id":"2505.08485","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"BAT provides a new large-scale autobidding benchmark with Avito auction data from first-price and VCG auctions, plus baseline algorithms and metrics for budget pacing and CPC-constrained bidding.","lead":"This paper releases BAT, a benchmark dataset of millions of real online advertising auctions from Avito, covering both first-price and VCG auction formats, along with baseline bidding algorithms and simulation code. It matters because open, realistic datasets for autobidding are scarce, and this resource gives researchers a common testbed for budget pacing and cost-per-click optimization.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation loop is unvalidated: BAT's simulator treats aggregated surplus fields as fixed causal responses, so the reported algorithm rankings may not transfer to live auctions.","rationale":"The reader's weakest assumption is exactly the load-bearing point: the simulation treats the surplus fields as fixed causal responses, independent of the bidder's own influence and competitive dynamics. I agree with that diagnosis. The dataset itself may be a useful public resource, and the paper is transparent about the data collection and filtering, so the central contribution is not fatally undermined. However, the benchmark's evaluation layer, which is the basis for the baseline comparisons and for the claim that the benchmark supports algorithm development, has no external validation. The paper includes no live deployment comparison, no off-policy evaluation against raw logged auctions, and no sensitivity analysis around the response function. Because the reported tables are produced entirely inside the simulator, a misspecified response model changes the numerical results and the ranking of algorithms. This does not require rejecting the paper, but it does require either adding validation or sharply limiting the claims to simulation-based evaluation. The reader's CONDITIONAL verdict already captures this; my stress-test does not move the verdict, but it specifies a concrete test that would settle whether the concern lands.","tokens_in":15863,"tokens_out":7029,"duration_ms":78394,"concrete_test":"Download the public repository and re-run the three experiments on a held-out subset using an off-policy estimator on the logged auction outcomes rather than the surplus aggregates: for the exact bid sequence emitted by each baseline, compute realized spend and clicks by inverse propensity weighting from the logged competition, and compare the resulting SCR and RMSE_T rankings with Tables 6 and 8. If the rank order of the top two algorithms changes, or if the simulated spend deviates by more than 20% from the off-policy estimate for the same bids, the fixed-response simulation is not a reliable evaluation environment and the conditional verdict should be strengthened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's usefulness as an evaluation environment rests on the simulation in Section 10, which maps each campaign's bid to expected cost, clicks, and visibility using the surplus fields from Section 4.2. These fields are observational aggregates over a competitive, non-stationary auction system: AuctionClicksSurplus and AuctionWinBidSurplus are marginal differences between adjacent price bins computed from logged bids, not structural counterfactuals. When an autobidder changes its bid, the set of competing bids, the winning price distribution, and the platform's ranking or reserve logic can all change; the paper provides no evidence that the surplus function is invariant to such changes. All baseline rankings in Tables 6–8 are generated inside this fixed-response loop, so if the response function is misspecified, the headline comparisons no longer support the claim that the benchmark reliably evaluates autobidding algorithms. A related internal concern is that the FP cost model in Section 10 multiplies AuctionContactsSurplus (a marginal contact increment) by the bid as if it were an auction win probability, which is not the same quantity. The paper gives no live A/B test, off-policy check, or stability analysis connecting simulated outcomes to realized auction outcomes. Until such validation exists, the central evaluation claim is conditional on an untested counterfactual assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BAT (Benchmark for Auto-bidding Task), a public dataset from Avito containing roughly 10 million first-price (FP) auctions and 1 million VCG auctions, together with campaign metadata, traffic distributions, and aggregated auction statistics. The authors describe the data collection and filtering process, define a simulated auction environment based on aggregated surplus fields, and propose several baselines: two new algorithms (ALM and TA-PID), a modified M-PID, and reimplementations of Mystique and BROI. They define metrics for budget pacing (RMSE_T), CPC satisfaction (REL_CPC), and total clicks (SCR), and report experiments on a subset of logical category 1, with hyperparameters tuned on an S1 split and evaluated on a disjoint S2 split. The paper claims that the benchmark supports development and evaluation of autobidding algorithms and that two of the proposed baselines have proven effective in production.","tokens_in":16142,"tokens_out":4718,"duration_ms":49614,"significance":"If the benchmark delivers what it promises, it addresses a genuine scarcity of open, large-scale autobidding datasets with two distinct auction formats. The dataset scale (10M FP and 1M VCG auctions, thousands of campaigns) and the public code repository are concrete strengths, and the S1/S2 split with Optuna-based hyperparameter tuning is a sensible attempt at avoiding overfitting in the baseline comparisons. The paper also provides clear metrics and a reproducible simulation harness. However, the central evaluation claim is currently conditional: all baseline rankings are produced inside a simulator whose reward model is an unvalidated map from bids to aggregated surplus values. The data contribution itself is likely valuable, but the benchmark's usefulness for ranking autobidding algorithms depends on evidence that this simulator reflects real auction dynamics, which the paper does not yet provide.","major_comments":[{"comment":"The simulation treats the aggregated surplus fields (AuctionClicksSurplus, AuctionWinBidSurplus, AuctionContactsSurplus) as fixed causal responses to a campaign's bid, but these fields are defined in Section 4.2 as marginal differences between adjacent price bins computed from logged auctions, not as structural counterfactuals. Section 10 then applies them deterministically at every timestamp, implicitly assuming that changing one bidder's bid does not change the distribution of competing bids, the winning price, or other auction conditions. The paper provides no off-policy check, stability analysis, or live A/B validation connecting simulated outcomes to realized auction outcomes. Because Tables 6-8 and 10 are all generated inside this fixed-response loop, the benchmark's claim to evaluate autobidding algorithms is currently conditional on an untested invariance assumption.","section":"Sections 4.2, 5, 10"},{"comment":"For FP auctions, the simulator sets the expected price to the product of AuctionContactsSurplus and the bid, 'as if we buy each auction with the defined bid.' However, AuctionContactsSurplus is defined in Section 4.2 as the expected increase in contacts relative to the previous price bin, which is not a win probability. Multiplying a marginal contact increment by the full bid does not yield the expected cost of winning a first-price auction. This conflation can systematically distort budget depletion and click accounting in the FP simulation, and it may explain the large asymmetry in Table 7 (M-PID REL_CPC 0.49 for FP vs 0.91 for VCG). The authors should either justify the cost model from the data-generating process or replace it with a quantity that is actually interpretable as a win probability or expected payment.","section":"Section 10, FP cost model"},{"comment":"The experimental results are reported as single numbers with no error bars, standard deviations, or significance tests. Hyperparameters are selected with a stochastic Bayesian optimizer (Optuna) on S1, and the simulation itself is deterministic given those parameters, so the reported point estimates do not convey uncertainty about the Optuna runs or about the choice of campaign subset. Several published differences are small (e.g., Table 6 VCG: TA-PID 1.42 vs M-PID 1.38; Table 8 FP: ALM 1,085,836 vs BROI 1,098,184), and the text draws comparative conclusions such as 'the leading algorithms, TA-PID and ALM, achieve very similar results.' Without repeated runs, confidence intervals, or a sensitivity analysis, the comparative claims in Section 11 are not statistically supported.","section":"Tables 6, 7, 8, 10 and Section 10"}],"minor_comments":[{"comment":"Algorithm 1 line 6 and Algorithm 2 line 7 write 'Play bid b ← −γ Δ_n'; the minus sign and the missing exponent contradict Section 8, where the bid is defined as b = γ^Δ. This should be corrected to b ← γ^Δ_n.","section":"Appendix B, Algorithms 1 and 2"},{"comment":"The update formula 'Δ_n = Δ_{n-1} + B_left · β' mixes a price-bin index with a relative remaining budget; the dimensions and intended semantics of the update should be clarified, and the clip boundaries mentioned in the text should appear explicitly in Algorithm 1.","section":"Section 8.1"},{"comment":"The text refers to 'the first 4 baselines (Linear, TA-PID, M-PID, Mystique)', but the algorithm introduced in Section 8.1 is called ALM (Adaptive Linear Model); the naming should be consistent.","section":"Section 9.1"},{"comment":"Section 7 states that the budget decreases by the bid only if the user clicks, while Section 10 says the budget is reduced at every timestamp by the expected price of auction participation; the payment rule for FP and VCG should be stated unambiguously and consistently.","section":"Sections 7 and 10"},{"comment":"Table 2 shows AuctionCount only for the VCG example and a dash for the FP example, yet Section 5 reports that FP auctions were logged at 100%; the authors should clarify whether AuctionCount exists for FP and why it is not shown.","section":"Table 2 and Section 5"},{"comment":"The paper should state the interpretation of REL_CPC explicitly (values below 1 satisfy the CPC constraint, values above 1 violate it) so that the reader can interpret values such as 0.49 and 0.91 without ambiguity.","section":"Table 7"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution is real and timely, and the authors have made code publicly available, which are significant positives. My recommendation for major revision is driven by the load-bearing simulation-validity concern: the benchmark's core evaluative function rests on a fixed-response simulator that is not validated against live or holdout auction dynamics. I would ask the editor to require either a validation study of the simulator or a substantial tempering of the paper's claims, framing the baseline comparisons as illustrative of the benchmark mechanics rather than as established algorithm rankings. I would also suggest checking that the production-effectiveness claim for ALM and TA-PID is either supported by evidence or removed, since it appears in the contribution statement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing about this paper is the dataset, not the algorithms. BAT adds two public collections of real auction logs from Avito — 10 million first-price auctions and 1 million VCG auctions — with campaign metadata, traffic distributions, and aggregated surplus curves per bid bin. That combination does not exist in the open literature as far as I can verify: iPinYou is second-price only and the Alibaba set is GSP. For anyone working on budget pacing or CPC-constrained bidding, this is a genuine new testbed. The authors also did the right structural things: code and data are published, campaigns are split into S1/S2 so hyperparameters are tuned on a held-out basis, and several baselines are included rather than one. That deserves credit.\n\nThe soft spot is the evaluation loop. Section 10 maps each campaign's bid to clicks, costs, and visibility through the aggregated surplus fields; those fields are observational differences between adjacent price bins in logged data, not structural counterfactuals. When a bidder changes its bid, competing bids and reserve logic can shift, and the paper offers no stability analysis, off-policy check, or live A/B test connecting simulated rewards to realized outcomes. The stress-test note lands on this, and on the FP cost model: multiplying AuctionContactsSurplus by the bid as if it were a win probability mixes a marginal contact increment with the bid itself, and the units do not obviously line up. This does not kill the dataset, but the baseline rankings in Tables 6–8 should be read as properties of the simulator, not of live auctions. The missing error bars make that worse: several differences, M-PID versus Mystique on FP RMSE_T, ALM versus TA-PID on SCR, are small enough that single runs do not establish an ordering.\n\nTwo smaller things. The claim that ALM and TA-PID have demonstrated effectiveness in production environments is supported only by the simulation in this paper; if they are actually running in production, show those numbers. And the pseudocode has sign typos — the bid updates should be γ^Δn, not −γ^Δn — which should be fixed in a reference benchmark.\n\nThe citation pattern is fine; the claim that only two open datasets exist checks out. My bottom line: the dataset is the contribution, it is real, and I would engage with it. The comparisons of algorithms need to be reframed as baseline sanity checks, and the simulator's counterfactual assumption needs validation or an explicit caveat before the rankings carry weight.\n\nRecommendation: referee it seriously; require either a validation study for the simulator or a bounded claim about what the rankings mean, plus error bars or multiple seeds.","headline":"The genuine contribution here is the dataset — a new public benchmark with both first-price and VCG auction logs — while the baseline comparisons run on an unvalidated simulator and should be treated as sanity checks, not live-auction rankings.","tokens_in":16689,"tokens_out":3506,"would_cite":true,"duration_ms":32204,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Avito benchmark opens 11 million real auctions to autobidding research.","keywords":["auto-bidding","real-time bidding","first-price auction","VCG auction","benchmark dataset","budget pacing","cost-per-click constraint","online advertising"],"falsifier":"A direct test would be to run the simulator's recommended bid sequence on a live auction platform for a sample of campaigns and compare the realized hourly spend and clicks to the simulated values. If the observed differences cannot be explained by random noise and systematically grow over time, the assumption that the surplus fields are fixed responses is untenable.","tokens_in":1250,"feed_emoji":"📊","tokens_out":3174,"duration_ms":59669,"temperature":0.7,"pith_summary":"BAT (Benchmark for Auto-bidding Task) is a new open dataset of over 10 million first-price auctions and 1 million VCG auctions logged from the Russian classifieds platform Avito. The paper argues that this dataset fills a shortage of realistic, public data for developing automated bidding algorithms, which currently mostly rely on private logs. To make the dataset immediately usable, the authors provide five bidding baselines: two new ones (Adaptive Linear Model and Traffic-aware PID), a modified M-PID, and two published systems (Mystique and BROI). They test these baselines on budget pacing uniformity, cost-per-click constraints, and click maximization, and report results that separate the algorithms clearly. The benchmark's value is that it gives researchers a production-like environment for both first-price and VCG auctions without needing access to a proprietary platform.","feed_headline":"11M real auctions open to autobidding tests","feed_subtitle":"Avito benchmark pits new pacing and CPC baselines on first-price and VCG auctions.","key_machinery":"The central mechanism is the auction statistics table, where each row corresponds to a campaign-hour-bid-bin and contains 'surplus' fields: AuctionClicksSurplus, AuctionContactsSurplus, AuctionVisibilitySurplus, and AuctionWinBidSurplus. Each surplus is the expected incremental gain from raising the bid by one bin relative to the previous bin, and the WinBidSurplus is the expected additional spend. The simulation uses these fields to compute the reward of each bid: for a given bin, the win price is the sum of WinBidSurplus up to that bin (for VCG) or the bid itself (for FP), and the click yield is the sum of ClickSurplus up to that bin. The algorithm's only decision at each hour is which bin to choose, and the environment is a deterministic function of that choice, with no feedback from the bidder's history or from competing bids.","core_discovery":"The central claim is that BAT is a valid and practical benchmark for autobidding research. The dataset contains, for each of roughly 9000 advertising campaigns, hourly aggregated statistics that describe the expected incremental clicks, contacts, visibility, and spend that a campaign would gain by moving its bid from one discrete price bin to the next. The paper demonstrates the benchmark's usability by implementing five algorithms from the literature and by showing that the proposed ALM and TA-PID algorithms, together with the modified M-PID, achieve competitive performance on the three studied tasks. In particular, on the click-sum maximization experiment, TA-PID and ALM outperform the others, while Mystique wins on budget pacing in both auction formats. The authors present their results as evidence that the dataset can support meaningful algorithm development and comparison.","pith_inferences":["Because the auction feedback is pre-aggregated per bid bin, the dataset could be used for off-policy evaluation of bid policies, not only for the provided closed-loop simulator.","The two-auction-format design invites a direct study of bid shading: comparing how much a given surplus response changes between VCG and FP pricing could reveal how much of an FP bid should be shaded relative to the equivalent VCG bid.","The fixed-surplus response format could be replaced with a learned response model from the same fields, which would let future work test how sensitive algorithm rankings are to the stationarity assumption.","The traffic table and the reported category-level differences suggest a natural extension: benchmarking algorithms that are specialized to particular logical categories rather than tuned globally."],"forward_implications":["If the benchmark is correct, researchers can test budget pacing and CPC-constrained bidding algorithms on production-scale data without access to proprietary advertising platforms.","The inclusion of both FP and VCG auctions allows the same algorithm to be studied under two pricing rules, which matters as platforms continue to migrate from second-price to first-price auctions.","The five implemented baselines provide ready reference points, so a new autobidding method can be compared against adaptive pacing, PID control, and optimization-based approaches using the same simulator.","The proposed metrics, RMSE_T and relative CPC, give standard quantitative targets that make hyperparameter tuning and cross-paper comparison more straightforward.","The hour-level traffic share data can support time-aware pacing algorithms that account for the roughly thirty-fold variation in traffic across the day."],"supporting_citations":[{"why":"Supplies the optimization problem formulation with budget and CPC constraints, the primal-dual optimal-bid result, and the M-PID baseline that the paper modifies.","marker":"[33]"},{"why":"Provides the Mystique budget pacing system, one of the main baselines for uniform spending.","marker":"[28]"},{"why":"Gives the BROI autobidding algorithm with budget and ROI constraints, along with its theoretical welfare guarantee.","marker":"[19]"},{"why":"Serves as the primary prior open RTB dataset, whose limitations the paper contrasts with BAT.","marker":"[17]"},{"why":"Describes the Alibaba RTB dataset for reinforcement learning, the other recent open dataset the paper compares against.","marker":"[32]"},{"why":"Documents the scarcity of open datasets in automated bidding, which motivates the benchmark.","marker":"[22]"}],"fun_headline_variants":["Autobidding benchmark: 9k real campaigns, two auction formats","BAT: real-auction benchmark for autobidding algorithms","Test autobidding on real auctions with BAT benchmark","First-price and VCG auctions in new autobidding benchmark","9k campaigns fuel autobidding benchmark"],"cache_read_input_tokens":18816,"weakest_assumption_plain":"The benchmark's simulation assumes that the aggregated surplus fields (AuctionClicksSurplus, AuctionWinBidSurplus, etc.) give fixed, predictable responses to a campaign's bid, independent of how much the bidder itself affects the auction or how competitors adjust.","fun_headline_variants_meta":{"raw":{"variants":["Autobidding benchmark: 9k real campaigns, two auction formats","BAT: real-auction benchmark for autobidding algorithms","Test autobidding on real auctions with BAT benchmark","First-price and VCG auctions in new autobidding benchmark","9k campaigns fuel autobidding benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000312,"raw_usage":{"total_tokens":1728,"prompt_tokens":853,"completion_tokens":875,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":793}},"tokens_in":469,"tokens_out":875,"duration_ms":8448,"temperature":1.0,"reasoning_tokens":793,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:53:26.463804+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to run the simulator's recommended bid sequence on a live auction platform for a sample of campaigns and compare the realized hourly spend and clicks to the simulated values. If the observed differences cannot be explained by random noise and systematically grow over time, the assumption that the surplus fields are fixed responses is untenable.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the BROI autobidding algorithm with budget and ROI constraints, along with its theoretical welfare guarantee."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Serves as the primary prior open RTB dataset, whose limitations the paper contrasts with BAT."},{"cited_title":"Wellman, Chuan Yu, Shuai Dou, Yusen Huo, Zhiwei Xu, Zhijian Duan, Shaopan Xiong, Chuang Liu, Ningyuan Li, Kefan Su, Wei Gong, and Bo Zheng","cited_arxiv_id":null,"evidence_quote":"Describes the Alibaba RTB dataset for reinforcement learning, the other recent open dataset the paper compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the scarcity of open datasets in automated bidding, which motivates the benchmark."}],"review_version":1}