{"id":"e2e2d8e5-0196-4ef7-946b-ed5466daeb05","arxiv_id":"2509.05292","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"An RL agent that picks personalized weights for a linear ad ranking utility raised treated-segment CTR by 9.7% and CTR30 by 7.7% in Pinterest's production system.","lead":"This paper describes a reinforcement learning system that automatically chooses the weights in Pinterest's ad ranking score for each user. In an online test, the system improved click-through rate by 9.7% and long clicks by 7.7% on the treated segment.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Off-policy training update in Eq (4.1) omits the importance ratio required for a valid policy gradient, so the learned policy is not shown to maximize the stated reward.","rationale":"The reader's CONDITIONAL verdict is appropriate; the off-policy issue is the same one I would flag. I considered whether the online A/B result by itself is enough to support the central claim. It is the strongest evidence, but the paper's contribution is a deep RL method, and the method's defining update is not a valid policy gradient under the stated data-collection policy. I also considered the reward being based on predicted probabilities; this is a design choice that the ablation study explores, and it does not by itself invalidate the online comparison. The absence of confidence intervals is a reporting weakness, not a correctness flaw. The off-policy bias, however, changes the objective being optimized: maximizing E_{π_B}[r log π] is not maximizing E_π[r], and no argument or experiment in the paper bridges that gap. The paper's own Section 7 flags the off-policy setting, but only as a future-work item, not as a justification for Eq (4.1). A simulation test can settle whether the bias actually changes the deployed argmax policy. If it does not, the empirical result stands; if it does, the theoretical framing should be revised. Thus no verdict change: still CONDITIONAL.","tokens_in":881,"tokens_out":884,"duration_ms":165902,"concrete_test":"Run a synthetic contextual-bandit simulation with a known reward function r(s,a), a state distribution, and a uniform behavior policy; train two agents on the same logged data, one with the paper's Eq (4.1) update and one with the importance-weighted off-policy gradient. Evaluate the greedy (argmax) policies on a held-out state sample by their true expected reward. If the Eq (4.1) agent's greedy reward is materially below the importance-weighted agent's (or below the optimal policy's), the missing importance ratio is consequential and the paper's training objective is not a valid policy gradient.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that Eq (4.1) is a policy-gradient update that maximizes the expected reward. It is not. Actions are logged under a uniform behavior policy π_B = U(A) (Section 5), so the REINFORCE-style update θ ← θ + η/B Σ_i r_i ∇θ log π(a_i|s_i,θ) is an unweighted reward-weighted log-likelihood. The true off-policy gradient for the one-step objective J(θ)=E_s Σ_a π(a|s) r(s,a) is E_{s,a∼π_B}[ (π(a|s)/π_B(a|s)) r(s,a) ∇θ log π(a|s) ]; with uniform π_B, this requires an importance ratio proportional to π(a|s). Omitting it makes the update the gradient of L(θ)=E_{s,a∼π_B}[r log π], not of J. For a single state, maximizing L gives π(a|s) ∝ r(s,a) (for positive rewards), not the optimal degenerate policy; with a shared MLP across many states, the argmax of π need not equal argmax r. The paper provides no simulation, IPS evaluation, or on-policy validation showing the biased surrogate still recovers good decisions. Section 7 acknowledges the off-policy setting but does not correct the estimator. Since the central claim is that DRL-PUT 'directly learn[s] an optimal policy' from logs, this gap is load-bearing. The online A/B may still show empirical improvement, but the causal mechanism attributed to RL is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DRL-PUT, a policy-gradient RL framework that selects discretized weights and a reserve price for a linear ad ranking utility. The policy is an MLP with softmax output; training uses a REINFORCE-style update on data logged from a uniform behavior policy. The method is evaluated in a production A/B test at Pinterest, reporting a +9.7% CTR increase and +7.7% CTR30 increase on the treated segment, with reward-function ablations and a visualization of personalized weight predictions.","tokens_in":14317,"tokens_out":6917,"duration_ms":68947,"significance":"If the empirical results hold, the paper demonstrates a practical framework for automated, personalized utility tuning in industrial ad recommender systems, where static manual weights are still common. The production-scale A/B test, reward ablations, and feature-based personalization analysis are valuable contributions. However, the theoretical grounding of the training objective is incomplete, and the online evidence lacks essential statistical detail; the strength of the central claim is therefore not yet commensurate with the evidence.","major_comments":[{"comment":"The update in Eq. (4.1) is a reward-weighted log-likelihood over actions logged from the uniform behavior policy π_B. This is not the policy gradient for J(θ)=E_s Σ_a π(a|s) r(s,a). The correct off-policy gradient requires the importance ratio π(a|s)/π_B(a|s); with π_B uniform this ratio is proportional to π(a|s). Omitting it means the update maximizes E_{s,a∼π_B}[r log π(a|s)], whose per-state solution is π(a|s) ∝ r(s,a), not a policy that selects the highest-reward action. Section 7 acknowledges the off-policy setting but does not correct the estimator. The central claim that DRL-PUT 'directly learn[s] an optimal policy' is therefore unsupported. Please add an importance-weighted correction, provide a simulation or off-policy evaluation showing the surrogate recovers the optimal action, or substantially soften the claim to a heuristic reward-weighted update.","section":"Section 4.3, Eq. (4.1); Section 5"},{"comment":"The primary evidence for the paper's contribution is the online A/B experiment. However, no confidence intervals, p-values, or details of the test (duration, sample size, randomization unit, multiple-comparison handling) are reported. Only 'statistically insignificant' is marked for some metrics, and the headline CTR/CTR30 gains are unmarked. Given the large number of metrics in Tables 2–5, these numbers could be inflated by multiple comparisons. Please report uncertainty for the headline metrics and for each ablation.","section":"Section 6.2.2, Table 2"},{"comment":"The reward function r in Eqs. (3.2)–(3.4) is a weighted sum of the same estimated engagement probabilities that the action weights w_i multiply in the ranking utility (3.1), with hand-tuned coefficients α, β, γ. Consequently, the reported CTR/CTR30 improvements are direct consequences of the chosen reward, not an independent demonstration that the RL agent discovered a good trade-off. The ablation in Tables 3–5 shows how sensitive the outcomes are to these coefficients. The paper should state explicitly that the agent optimizes an estimated proxy, and discuss the potential for reward hacking (e.g., exploiting correlations among estimated probabilities rather than improving actual user engagement).","section":"Section 3.5, Eq. (3.4); Section 6.2.3"},{"comment":"The offline metrics are not adequate to validate the learned policy. Relative_Gain (6.2) is computed with the same reward r used in training and compares action probabilities to the uniform behavior policy; it therefore measures fit to the surrogate objective, not decision quality. Diversity (6.1) only checks that the policy is not degenerate. No held-out evaluation or counterfactual estimate is provided. Please include an off-policy evaluation (e.g., IPS on a logged validation set) or a small simulation to establish that the learned policy improves on the behavior policy.","section":"Section 6.1, Eqs. (6.1)–(6.2)"}],"minor_comments":[{"comment":"'Reletive_Gain' should be 'Relative_Gain'.","section":"Section 6.1"},{"comment":"The phrase 'their values shall always be the same𝑖𝑡ℎ' is garbled; please rephrase.","section":"Section 3.3"},{"comment":"Algorithm 1 uses α for the step size while Eq. (4.1) uses η; make the notation consistent. Also, the 'policy gradient w.r.t. θ: ∇θ L(θ)' in Algorithm 1 is the negative of the gradient used in Eq. (4.1); please align the signs.","section":"Algorithm 1"},{"comment":"The success/failure indicators (✓/✗) in Table 1 are not legible; consider using text or a clearer table format.","section":"Table 1"},{"comment":"Hiding absolute values in Figure 3 makes it impossible to assess effect sizes; include scale labels or a table with the numerical values.","section":"Section 6.3, Figure 3"},{"comment":"The claim 'first published work in this area' is hard to verify; please soften it unless a thorough literature search is documented.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"The off-policy estimator problem is real and should be fixed or substantially clarified before publication. An importance-weighted update or a careful reframing as a heuristic, together with a validation, would make the claims defensible. The missing confidence intervals for the headline A/B metrics are also a concern for a production-scale study; I would not reject if these issues are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is an industrial RL-for-utility-tuning application with real online results: 9.7% CTR and 7.7% CTR30 lift on the treated segment in Pinterest's ad system. The idea is new—applying a policy-gradient learner to choose personal weights and reserve prices for a linear ranking utility—and the online A/B plus reward ablations are the strongest parts. The authors also show a sensible personalization pattern (higher click weight for click-prone users). Credit where due: deployed, measurable, and they're candid about limitations.\n\nThe main soft spot is the training update in Eq (4.1). The stress-test note is correct: data is logged under a uniform behavior policy, and the update is a reward-weighted log-likelihood without importance sampling or a baseline. It is not the policy gradient for the deployed policy. That said, the practical effect is less bad than it looks: for a single state, the stationary point of this surrogate gives pi(a) proportional to r(a), whose argmax matches the optimal action. With shared function approximation and finite data, that alignment is not guaranteed, and the paper does not validate it. But the online A/B is the empirical test, so the claim that the learned policy improves metrics doesn't collapse. I'd ask the authors to either add an importance ratio or show a counterfactual evaluation (e.g., offline IPS or a small on-policy run) to back the mechanism.\n\nOther weaknesses are minor: no confidence intervals for the headline metrics, an undefined 'treated segment,' and sloppy notation (m^n vs g^n). The reward coefficients are hand-tuned, but that's normal for reward design, not circularity.\n\nBottom line: this is a solid industrial case study, not a theoretical contribution. It deserves a serious referee—send it to review with a request for the off-policy fix or a defense, plus statistical details. I'd cite it for the application and the online results.","headline":"A deployed RL-for-utility-tuning paper with real online gains, but the training update is a heuristic surrogate without importance correction; still worth a serious referee.","tokens_in":14738,"tokens_out":3931,"would_cite":true,"duration_ms":41788,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a policy-gradient RL agent can replace manual tuning of ad-ranking utility weights, raising click-through rate by 9.7% and long click-through rate by 7.7% on the treated segment in a production A/B experiment.","keywords":["Ranking Utility Tuning","Ad Recommender System","Deep Reinforcement Learning","Policy Gradient","REINFORCE","Reward Function","Personalization","Online A/B Testing"],"falsifier":"Take the logged state-action-reward data used for training and estimate the policy's value with an inverse-propensity scorer that accounts for the uniform behavior policy. If the learned policy's corrected expected reward is not higher than the uniform policy's, the reported online CTR gains cannot be attributed to the RL objective. Equivalently, in a live test, scramble the reward coefficients (α, β, γ) and observe whether served weights track the scramble; if they do not, the policy is optimizing something other than the stated reward.","tokens_in":13774,"feed_emoji":"📈","tokens_out":6148,"duration_ms":59621,"temperature":0.7,"pith_summary":"Manual tuning of an ad-ranking utility—a weighted sum of predicted engagement probabilities gated by a reserve price—is interpretable but struggles with combinatorial parameter counts, seasonal shifts, and user heterogeneity. DRL-PUT reframes this tuning as a one-step reinforcement learning problem: given the user and request features, an MLP policy outputs a probability distribution over a discretized, grouped set of hyperparameters, and is trained with a REINFORCE-style update on logged traffic. The paper's main evidence is a production A/B test in which the treated segment saw click-through rate rise 9.7% and long click-through rate rise 7.7% relative to the manually tuned baseline. A reward-function ablation shows that the composition of the reward—revenue versus user-engagement terms—is what steers the trade-off between business metrics, and the predicted weights vary sensibly with users' historical CTR and CVR, indicating personalization.","feed_headline":"RL-tuned ad weights lift click-through rate 9.7% in live test","feed_subtitle":"A policy-gradient agent personalizes ad-ranking weights per request, beating manual tuning on treated users.","key_machinery":"The central object is the ranking utility function U = 1{Estimated_Revenue ≥ b}·(Estimated_Revenue + Σ p(engagement_i)·w_i), together with the policy π(a|s, θ) that selects the hyperparameter vector A = (b, {w_i}) from a discretized, grouped action space. The REINFORCE-style update θ ← θ + η·(1/B)Σ_i r_i ∇_θ log π(a_i|s_i, θ) carries the learning: it raises the log-likelihood of logged actions in proportion to their observed reward, with no value function and no discounting (γ=0). Action-space discretization—ten levels per weight, with semantically related engagement weights forced to share values—reduces the search space to 10^3 combinations and is what makes direct policy learning feasible","core_discovery":"The paper claims that the hyperparameters in the ranking utility U = 1{Estimated_Revenue ≥ b}·(Estimated_Revenue + Σ_i p(engagement_i)·w_i) can be learned rather than hand-set. It models an incoming ad request as a state, the vector A = (b, {w_i}) as an action from a discretized and semantically grouped space of 10^3 candidates, and the immediate reward as a campaign-type-dependent weighted sum of expected revenue and user-engagement probabilities. The agent is a multilayer perceptron trained by the update θ ← θ + η·(1/B)Σ_i r_i ∇_θ log π(a_i|s_i, θ) on batches of about 0.5% of production traffic logged under a uniform behavioral policy. In the online A/B experiment, this approach beat the m","pith_inferences":["Because Eq (4.1) uses reward-weighted log-likelihood without importance sampling or a baseline, the update is not the true policy gradient for the deployed policy; the same objective could be fit by straightforward reward-weighted supervised learning, so the online gains may reflect the reward signal more than RL-specific exploration.","A cheap falsification outside the paper: on the logged uniform data, compare the learned policy's expected reward against an inverse-propensity-weighted estimate; if the corrected estimate shows no gain over uniform sampling, the A/B lift cannot be credited to optimizing the stated reward.","The one-step formulation suggests a natural extension to multi-step or session-level tuning, which the paper lists as future work; reward design would then need to include delayed effects.","The discretization trick (m=10 levels, g=3 groups) is likely transferable to other ranking systems with linear utilities, but the choice of groups and ranges is currently manual and could be a source of bias."],"forward_implications":["Utility weights no longer need to be static: the same framework can re-learn them continuously as user behavior and advertiser demand shift.","An RL agent can be layered on top of existing prediction models without retraining them, since it only re-weights their outputs.","Reward design, not the RL algorithm, is the main lever controlling the revenue-versus-engagement trade-off: dropping the user-value term flips CTR from +9.71% to −0.74%.","Grouping semantically related engagement weights into shared parameters reduces the action space enough (to 10^3) for stable convergence on modest logged data.","Personalization emerges from state features: high-CTR users get higher click weights and lower reserve prices, while high-CVR users get higher reserve prices."],"supporting_citations":[{"why":"Supplies the REINFORCE policy-gradient algorithm that the paper adapts into its batch reward-weighted log-likelihood update.","marker":"[28, 29]"},{"why":"Provides the policy-gradient-versus-action-value framing that justifies learning a policy directly without a value function.","marker":"[23]"},{"why":"Cited as evidence that value-function estimation is hard in recommender systems due to high-variance, unbalanced immediate rewards, motivating the policy-based choice.","marker":"[11]"},{"why":"Multi-agent ranking baseline that the paper contrasts with its single-agent, discrete-action utility-tuning formulation.","marker":"[8]"},{"why":"Multi-agent whole-chain recommendation baseline that the paper contrasts with its simpler one-step utility-weighting approach.","marker":"[39]"},{"why":"Value-based deep RL method for ad display that the paper distinguishes from its policy-based direct learning.","marker":"[36]"},{"why":"Value-based RL pacing baseline that the paper contrasts to highlight the efficiency of learning the policy directly.","marker":"[27]"}],"fun_headline_variants":["RL-tuned ad weights lift CTR 9.7% vs manual","Personalized RL ad ranking boosts click-through 9.7%","Deep RL auto-tunes ad utility, beats manual by 9.7%","Learning ad weights with RL: +9.7% CTR in live test","Reinforcement learning personalizes ad weights, CTR up 9.7%"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the reward-weighted log-likelihood update on actions logged by a uniform random policy, with no importance weights and no baseline, actually maximizes the intended reward for the deployed policy.","fun_headline_variants_meta":{"raw":{"variants":["RL-tuned ad weights lift CTR 9.7% vs manual","Personalized RL ad ranking boosts click-through 9.7%","Deep RL auto-tunes ad utility, beats manual by 9.7%","Learning ad weights with RL: +9.7% CTR in live test","Reinforcement learning personalizes ad weights, CTR up 9.7%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000271,"raw_usage":{"total_tokens":1504,"prompt_tokens":824,"completion_tokens":680,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":580}},"tokens_in":568,"tokens_out":680,"duration_ms":7256,"temperature":1.0,"reasoning_tokens":580,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:23:59.201199+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the logged state-action-reward data used for training and estimate the policy's value with an inverse-propensity scorer that accounts for the uniform behavior policy. If the learned policy's corrected expected reward is not higher than the uniform policy's, the reported online CTR gains cannot be attributed to the RL objective. Equivalently, in a live test, scramble the reward coefficients (α, β, γ) and observe whether served weights track the scramble; if they do not, the policy is optimizing something other than the stated reward.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cited as evidence that value-function estimation is hard in recommender systems due to high-variance, unbalanced immediate rewards, motivating the policy-based choice."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Multi-agent ranking baseline that the paper contrasts with its single-agent, discrete-action utility-tuning formulation."}],"review_version":1}