{"id":"07664849-c537-4440-bae2-10aa7ab1d144","arxiv_id":"2506.17919","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PE-MORL combines a permutation equivariant learned auction environment with a pessimistically penalized Q-learning objective and reports 3.9-7.2% GMV gains over three baselines in deployed Taobao A/B tests.","lead":"This paper presents PE-MORL, a model-based offline reinforcement learning method that learns a simulator from real auction data and trains auto-bidding policies on both real and imagined data. The practical hook is a reported 4-7% GMV improvement over three prior auto-bidding methods in A/B tests on Taobao.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 5.1's proof of reward permutation invariance is invalid for a fixed-index representative, so the PE/PI foundation and Theorem 5.2 support are unproven.","rationale":"The paper's central empirical claim is that PE-MORL outperforms SOTA auto-bidding in production, and the reader's CONDITIONAL verdict already reflects missing statistical characterization and under-reported evidence. My stress-test identifies a more specific, internal correctness issue in Proposition 5.1: the proof of the reward's PI property fails when the representative is a fixed physical advertiser indexed by N, because the reward vector equivariance r_t(ρs,ρa)=ρr_t(s,a) does not imply invariance of the N-th component under permutations that move the representative. The transition proof similarly mistakes invariance for equivariance at the step P(ρc)=P(c). This matters because the PE environment model's architecture and the generalization theorem (Theorem 5.2) are motivated by these properties; if the true functions are not PE/PI in the stated sense, the theoretical support for the design is unproven, though the empirical gains could still be real. I do not reject the paper: Tables 1-4 provide direct evidence of improved GMV and model accuracy, and the reader's conditions (error bars, uncertainty calibration, closer alignment between Theorem 5.2 and the implemented architecture) remain necessary. The proof gap adds an explicit condition: the authors must either correct Proposition 5.1 by restricting the symmetry group to permutations fixing the representative (with a marker for the representative's identity) or revise the architecture's reward head accordingly. Because the verdict stays CONDITIONAL with these conditions, I recommend UNCHANGED.","tokens_in":22957,"tokens_out":15823,"duration_ms":156937,"concrete_test":"Construct the N=2 case with distinct advertiser rewards, e.g., r_1=1, r_2=100, and let ρ be the swap. Using Eq. (37), r_t(ρs,ρa)=ρ r_t(s,a)=(100,1), so the representative's reward at index 2 changes from 100 to 1, contradicting R(ρs,ρa)=R(s,a). The same test can be run as a unit check on the implemented reward head: feed a batch with two advertisers, evaluate the reward, permute the two advertisers in the input, and check whether the output is invariant. If the implementation always fixes the representative at index N and only permutes the other N-1 advertisers, the test should restrict to that subgroup and the paper's '∀ρ∈Ω_N' claim should be weakened accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The PE environment model's theoretical basis rests on Proposition 5.1, which claims the reward R is permutation invariant and the transition P is permutation equivariant. The proof in Appendix E.1 does not go through. From the paper's own Eq. (37), the reward vector satisfies r_t(ρs,ρa)=ρ r_t(s,a). Since the representative advertiser is defined as the N-th element of the ordered vector, R(s,a)=[r_t(s,a)]_N, but R(ρs,ρa)=[ρ r_t(s,a)]_N=r_{ρ^{-1}(N)}(s,a), which equals R(s,a) only when ρ fixes N. For arbitrary permutations (e.g., the swap on N=2 with distinct advertiser values), R is not invariant. The transition proof has an analogous gap: after deriving c_t(ρs,ρa)=ρ c_t(s,a), it asserts P(ρ c_t)=P(c_t) to conclude P(ρs_{t+1}|ρs_t,ρa_t)=P(s_{t+1}|s_t,a_t). This is an invariance statement, not the equivariance needed; the correct equivariance would be P(·|ρc)=ρP(·|c). Thus the PE and PI properties, and consequently the architectural bias and Theorem 5.2's applicability, are not established by the provided argument.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper frames RL-based auto-bidding as a choice between training on a simplistic GSP simulator (SRLB) and training purely on fixed offline data (ORLB), and proposes a model-based offline RL paradigm (MRLB) that learns an environment model from real data and trains the policy on the union of real and model-generated data. The specific algorithm, PE-MORL, has two main components: a permutation-equivariant environment model that predicts next states and rewards from the joint state-action vector of all advertisers, and a robust offline Q-learning target that penalizes the predicted reward by ensemble variance and pessimistically selects among predicted next states. The paper presents real-world A/B tests on about 3,000 representative advertisers claiming GMV gains of +4.4% over V-CQL, +7.2% over USCB, and +3.9% over MBAB with roughly flat Cost and improved ROI, plus offline comparisons showing the PE environment model has lower prediction error than a fully connected baseline and a GSP simulator. The theoretical claims are that the environment model is justified by permutation invariance/equivariance of the auction process (Proposition 5.1), that the equivariant architecture improves the generalization gap (Theorem 5.2), and that the pessimistic reward penalty improves a lower bound on true system performance (Proposition 5.4).","tokens_in":23153,"tokens_out":4409,"duration_ms":45295,"significance":"If the empirical and theoretical claims hold, the paper makes a useful contribution to industrial auto-bidding: it identifies a concrete middle ground between simulator-based and offline-RL bidding, and it demonstrates a production-scale A/B evaluation, which is a genuine strength and uncommon in the literature. The permutation-equivariant environment model is a sensible inductive bias for auctions, and the offline prediction comparisons in Tables 2 and 3 are encouraging evidence for that architectural choice. However, the contribution is currently weakened by three load-bearing gaps: the proof of Proposition 5.1 is not valid as written for a fixed-index representative advertiser; Theorem 5.2 is proved only for an orbit-averaged model, not for the deployed parameter-shared architecture; and the headline A/B results in Table 1 are reported without confidence intervals or significance tests, so the size and stability of the claimed gains are not established.","major_comments":[{"comment":"The proof of Proposition 5.1 does not establish the claimed reward permutation invariance. Since R(s,a) is defined as the N-th component of the reward vector r_t(s,a), Eq. (37) gives R(ρs,ρa)=[ρ r_t(s,a)]_N = r_{ρ^{-1}(N)}(s,a), which equals R(s,a) only when ρ fixes N; for a transposition on N=2 with distinct advertisers this fails. This invalidates the PI claim, and consequently the theoretical justification for the reward branch of the PE environment model and for the applicability of Theorem 5.2 to the proposed architecture needs repair.","section":"§5.1, Appendix E.1, Eqs. (37)-(38)"},{"comment":"Theorem 5.2 is proved for the orbit-averaged model Q\\hat M', but the deployed environment model is a parameter-shared network with mean and max pooling, which is not equal to orbit averaging. The proof shows that orbit averaging reduces the r-covering number, yet no argument is given that the specific network architecture also reduces the covering number or preserves the required non-expansiveness. Thus the theorem's conclusion about \"making it satisfy (4)\" does not directly support the generalization claim for the actual architecture; the claim should either be proved for the architecture or explicitly downgraded to an empirical one.","section":"§5.1, Appendix D, Eq. (30) and Proposition D.4"},{"comment":"The headline A/B results report only single point estimates for GMV, ROI, and Cost. With roughly 3,000 representative advertisers over 10 days, the claimed differences of +3.9% to +7.2% need confidence intervals, standard errors, or significance tests to be interpretable; without these, the central claim that PE-MORL outperforms state-of-the-art methods in deployment is not statistically supported.","section":"Table 1, §6.1"},{"comment":"The lower-bound guarantee is conditional on an imported assumption, and the proof does not match the implemented penalty. The proof uses a function u(s,a) that dominates the transition-model discrepancy d_F, but the actual reward penalty in Eq. (15) is λ||σ̂_t||_F, the ensemble reward variance, with no demonstrated relationship between σ̂_t and u(s,a). The proposition should either be re-proved for the actual penalty or stated as an assumption-based result requiring empirical validation of the uncertainty-dominance condition.","section":"§5.2, Proposition 5.4 and Appendix E.4, Eqs. (44)-(45)"}],"minor_comments":[{"comment":"The column formatting for different λ values is unclear; for example, the row for λ=5 appears to merge values for two different settings, making it hard to read which R/R* and online rate correspond to which λ.","section":"Table 5"},{"comment":"The notation \"E [s^T_{t+1},r_t]^T ~ D_R\" is malformed and should be written as an expectation over (s_{t+1},r_t) drawn from the real data distribution, e.g., E_{(s_{t+1},r_t)∼D_R}.","section":"Eq. (13)"},{"comment":"Several citation placeholders appear as \"[?]\" and there is a duplicated/incomplete sentence \"We highlight that We are the first...\"; these should be cleaned before submission.","section":"Appendix A"},{"comment":"The Wasserstein distance plot lacks axis labels and units, and the claim that the distance is 15.55% higher for PE-MORL than for V-CQL needs a precise definition of the reference distance used to compute the percentage.","section":"Figure 2 and §6.1"},{"comment":"There are scattered typos such as \"representation advertiser\" and \"natrual language\" that should be corrected in a revision.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be an industrial submission with access to a production A/B platform, which is a valuable strength. The main concerns for the editor are that the theoretical narrative currently overclaims relative to the proofs (orbit averaging vs. the actual architecture, and the fixed-index representative in Proposition 5.1), and the empirical headline results lack inferential statistics. These are fixable within the scope of a revision if the authors tighten the claims and add the missing statistical support. I would not recommend rejection based on the current content, but I would require the proofs to be corrected or the corresponding claims to be weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. The paper reports real A/B tests on Taobao: PE-MORL beats three deployed baselines by 3.9-7.2% GMV on about 3,000 advertisers over 10 days. That's the strongest part, and it's a useful data point for anyone doing offline RL in bidding. The second thing is that the theory is not in good shape. Proposition 5.1, which is supposed to justify the permutation equivariant architecture, is not proven by the argument given.\n\nThe new idea is reasonable: don't use a hand-built GSP simulator, learn an environment model from logs, and build in permutation equivariance as an inductive bias. The empirical comparison of the learned model against the GSP simulator shows large accuracy gains, and the MRLB framing is sensible. The generalization lemma (orbit averaging shrinks covering number) is correct as a lemma, though it's borrowed from Duan et al. and applied to the orbit-averaged model rather than to the shared-parameter network they actually deploy.\n\nThe soft spots, in order of size. First, the proof of Proposition 5.1 fails. The reward of the representative advertiser is the N-th entry of the joint reward vector. From the paper's own Eq. (37), permuting the input permutes the joint reward. So the N-th entry becomes the reward of whichever advertiser moved into slot N; for a fixed physical advertiser, that's not invariant. If the representative advertiser is meant to be a distinguished index that never gets permuted, then the claim needs to be restated over the background advertisers only, and the proof has to say that. The transition proof has a related problem: it asserts P(ρc)=P(c), which is invariance of the cost vector distribution, not the equivariance you need. The correct statement would be P(·|ρc)=ρP(·|c). So as written, the PE/PI foundation and the promised generalization guarantee are unsupported.\n\nSecond, Table 1 has point estimates only. The word 'significantly' appears, but there are no confidence intervals or significance tests, so the stability of the 3.9-7.2% gains is unknown. Third, Proposition 5.4 leans on MOPO's uncertainty-dominance assumption, and λ is tuned on the same offline metrics used in the ablation, so the pessimism guarantee is weaker than it looks. Fourth, the manuscript is unfinished: template artifacts, a garbled Table 5, missing references.\n\nWho gets value: someone working on auto-bidding or industrial offline RL will want to know about the deployment results. The theory should not be cited until repaired.\n\nRecommendation: this deserves peer review, not desk rejection, because the deployment evidence is real and rare. But tell the authors the equivariance proof has to be rebuilt (or the claims scaled back), the A/B results need error bars, and the paper needs a cleanup pass. It's not acceptable in current form.","headline":"The deployment results are worth a look, but the permutation equivariance proof does not hold up, and the paper overclaims theory.","tokens_in":23789,"tokens_out":3912,"would_cite":false,"duration_ms":38018,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Advertising auto-bidding policies trained with a permutation equivariant environment model and a pessimistically penalized offline Q target can escape the coverage of fixed offline data and raise GMV by 3.9 to 7.2 percent in A/B tests.","keywords":["auto-bidding","offline reinforcement learning","model-based reinforcement learning","permutation equivariance","pessimistic Q-learning","online advertising","uncertainty penalty"],"falsifier":"Take the real transition logs used for training, apply a fixed permutation to the advertiser indices of each tuple, and check whether the observed next-state distribution and reward change only by reordering; if the permuted data produce a different reward or a non-equivariant next-state distribution, Proposition 5.1 is false. A second check: retrain without the equivariance constraint and compare test error on data where advertiser order at inference is reversed; if the equivariant model loses its advantage when advertiser identities matter, the architectural assumption is the bottleneck.","tokens_in":22676,"feed_emoji":"📈","tokens_out":9573,"duration_ms":89103,"temperature":0.7,"pith_summary":"This paper claims that auto-bidding policies for online advertising can be trained more effectively by learning an environment model from real bidding data and then training the policy on both real and model-generated data, rather than training only on the fixed offline dataset or only on a hand-built auction simulator. It introduces PE-MORL, which makes the learned environment model permutation equivariant, meaning that reordering the list of advertisers reorders the model's predictions in the same way, and adds a pessimistic penalty to the Q-learning target that discounts predicted rewards and state values where the model is uncertain. The paper argues this combination expands the state space the policy can visit, with a distance measure between policy trajectories and real data that is 15.55 percent larger than for a model-free offline baseline, while protecting against model error. In ten-day A/B tests across about three thousand representative advertisers, the authors report GMV gains of +4.4 percent, +7.2 percent, and +3.9 percent over three state-of-the-art baselines, with ROI up and cost close to unchanged. If correct, this establishes a practical middle path between simulator-based and offline RL bidding.","feed_headline":"Symmetry-aware RL model lifts ad-bidding GMV by up to 7.2%","feed_subtitle":"A learned auction model plus a pessimistic Q target lets bids escape the offline data without trusting model errors.","key_machinery":"The load-bearing object is the permutation equivariant environment model combined with a pessimistically penalized offline $Q$ target. The environment model has two heads: a shared-parameter, permutation equivariant network that predicts every advertiser's next local state, and a permutation invariant network that predicts the representative advertiser's reward; each uses multi-head attention followed by mean and max pooling, following the known representation of invariant functions as $\\phi(\\mathrm{pool}(\\psi(\\cdot)))$. An ensemble of $K$ such models is trained by maximum log-likelihood with reparameterized sampling. The robust $Q$ target is $\\hat B Q(s_t,a_t)=\\hat r_t-\\lambda\\|\\hat\\sigma_t\\|_F+\\gamma\\min_{\\hat s_{t+1}\\in\\hat M(s_t,a_t)}\\max_{a'}Q(\\hat s_{t+1},a')$, where $\\|\\hat\\sigma_t\\|_F$ is the ensemble's reward variance and the minimum over predicted next states makes the policy pessimistic out of distribution. The theory works by showing that orbit averaging, the operator that averages a function over all advertiser permutations, shrinks the function class's covering number, which tightens the generalization bound.","core_discovery":"The central claim is that the real bidding environment is symmetric under permutations of advertisers, and a model-based offline RL algorithm that builds this symmetry into the environment model, then pessimistically corrects for model error, outperforms state-of-the-art auto-bidding methods. Concretely, the paper proves that the transition rule of the advertising system is permutation equivariant and the reward is permutation invariant (Proposition 5.1), designs an environment model whose next-state head is equivariant and whose reward head is invariant, and proves that imposing this structure reduces the upper bound on the train-test generalization gap (Theorem 5.2). On the policy side, the robust Q target in Eq. (15) penalizes the predicted reward by the ensemble's uncertainty and takes the minimum Q value over the model's predicted next states, with a lower-bound guarantee that the pessimistically trained policy's performance in the real system is at least its performance in the penalized model (Proposition 5.4). The experiments support the claim by showing the equivariant model has lower prediction error than both a non-equivariant learned model and a GSP-based simulator, and that the resulting policies move further from the offline data while increasing GMV in online A/B tests.","pith_inferences":["Beyond the paper, the permutation equivariance premise is not tied to one auction platform; any multi-bidder sequential auction with symmetric allocation rules would admit the same environment model, so the recipe could transfer to sponsored search or display advertising systems elsewhere.","Beyond the paper, the uncertainty penalty is a generic cure for model bias: one could test whether the same ensemble-variance reward penalty plus min-Q state penalty improves other model-based offline RL problems where the environment has known symmetries.","Beyond the paper, varying the number of background advertisers at test time would be a clean extension; because the pooling layers are set-based, the equivariant model may generalize across different advertiser counts better than a fully connected model, which the paper does not test.","Beyond the paper, if the cost-only transition assumption is relaxed, for example if budgets or market prices enter the next state, the architecture would need additional equivariant channels; the paper's performance claims may depend on this simplification."],"forward_implications":["Policies trained with PE-MORL reach states measurably beyond the offline dataset: the distance from real data is 15.55 percent larger than for the model-free offline baseline, so the data-coverage ceiling of offline RL bidding is not fundamental.","The learned environment model achieves much lower test error than the GSP-based simulator used in simulation-based bidding, with MAE down 69.6 percent and MSE down 88.2 percent, and a smaller but real improvement over a non-equivariant neural network, with MAE down 5.9 percent and MSE down 3.3 percent.","The pessimistic penalty is necessary for the reported gains: removing it, i.e., setting $\\lambda=0$, yields the worst offline metrics in the ablation, so the performance claim depends on the uncertainty penalty rather than on the equivariant model alone.","In production-style A/B tests, the method raises GMV by 3.9 to 7.2 percent over three baselines while holding cost roughly flat and increasing ROI, implying advertisers obtain more value per unit of spend."],"supporting_citations":[{"why":"Provides the vanilla model-based auto-bidding baseline that PE-MORL extends and must outperform in the A/B tests.","marker":"[3]"},{"why":"Supplies the simulation-based RL baseline that trains in a GSP-based simulator and represents the simulator-reality gap the paper targets.","marker":"[8]"},{"why":"Defines the offline RL bidding baseline and the limited state-space coverage and out-of-distribution challenge the paper addresses.","marker":"[19]"},{"why":"Supplies the uncertainty-based reward penalty and the lower-bound proof technique used in Proposition 5.4.","marker":"[27]"},{"why":"Provides the covering-number generalization lemmas used to prove that orbit averaging tightens the environment model's generalization bound.","marker":"[4]"},{"why":"Provides the invariant representation $\\phi(\\mathrm{pool}(\\psi(\\cdot)))$ used as the basis for the pooling layer of the environment model.","marker":"[28]"},{"why":"Supplies the multi-head attention encoder-decoder blocks used as the feature extractor inside the invariant network.","marker":"[24]"},{"why":"Supplies the performance-difference lemma used to derive the lower bound in Proposition 5.4.","marker":"[17]"}],"fun_headline_variants":["Symmetry-aware RL lifts ad-bid GMV 7.2%","Equivariant model closes offline RL gap in ad bidding","Pessimistic model-based RL for symmetric auto-bidding","Symmetry in auction model improves offline RL bidding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the online auction treats all advertisers symmetrically, so permuting the advertiser list merely permutes the auction outputs; if the real mechanism favors advertisers by identity, position, or other asymmetric features, or if next states depend on market dynamics beyond the cost spent, the built-in permutation symmetry is misspecified and the model's advantage can collapse.","fun_headline_variants_meta":{"raw":{"variants":["Symmetry-aware RL lifts ad-bid GMV 7.2%","Equivariant model closes offline RL gap in ad bidding","Pessimistic model-based RL for symmetric auto-bidding","Symmetry in auction model improves offline RL bidding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000461,"raw_usage":{"total_tokens":2322,"prompt_tokens":976,"completion_tokens":1346,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":1276}},"tokens_in":592,"tokens_out":1346,"duration_ms":10985,"temperature":1.0,"reasoning_tokens":1276,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:00:01.520177+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the real transition logs used for training, apply a fixed permutation to the advertiser indices of each tuple, and check whether the observed next-state distribution and reward change only by reordering; if the permuted data produce a different reward or a non-equivariant next-state distribution, Proposition 5.1 is false. A second check: retrain without the equivariance constraint and compare test error on data where advertiser order at inference is reversed; if the equivariant model loses its advantage when advertiser identities matter, the architectural assumption is the bottleneck.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the vanilla model-based auto-bidding baseline that PE-MORL extends and must outperform in the A/B tests."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the offline RL bidding baseline and the limited state-space coverage and out-of-distribution challenge the paper addresses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the uncertainty-based reward penalty and the lower-bound proof technique used in Proposition 5.4."},{"cited_title":"Are Equivariant Equilibrium Approximators Beneficial?","cited_arxiv_id":"2301.11481","evidence_quote":"Provides the covering-number generalization lemmas used to prove that orbit averaging tightens the environment model's generalization bound."}],"review_version":2}