{"id":"491380dc-a85c-4405-b4f8-8f1e3bc85eb4","arxiv_id":"1908.06698","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Advertisers can increase their products' free organic traffic by learning bid adjustments that exploit how ad clicks feed back into the recommender's training data.","lead":"This paper shows that in e-commerce feeds, advertising a product can change how often the recommendation system later shows it for free, and proposes a reinforcement learning bidding algorithm to exploit that effect. The method was tested on Taobao data and in an online A/B test.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HTLB's transition expansion reuses a single x_{j+1} for M counterfactual actions; if organic traffic responds to ad delivery within the same day, the hybrid samples are invalid and the reported faster convergence is an artifact. The paper never tests the required one-day lag.","rationale":"The paper's central claim has two parts: the existence of the Leverage mechanism and the superiority of the HTLB-DDPG algorithm. The Leverage mechanism has independent support from the observational analysis in Section 4 and the online A/B deployment, even though the deployed algorithm is CEM rather than HTLB-DDPG. The weakest link is the HTLB transition expansion, which relies on the untested assumption that the recommendation-related state x_{t+1} is independent of the current bidding action a_t. This is exactly the reader's weakest_assumption, so I agree. If the assumption fails, the expanded transitions in Eq. (6) reuse one real x_{j+1} for M simulated actions that would, in reality, produce different organic outcomes; HTLB's faster convergence and lower variance would then be an artifact of a misspecified model rather than a genuine property. The paper provides no empirical check of the required delay: the Section 4 analysis is at daily granularity but does not estimate the lag between ad delivery and organic traffic response, and the online deployment's one-hour decision windows make the delay assumption even more questionable. Because the online A/B results validate only the CEM-based system, the offline experiments carry the entire evidential weight for HTLB, making this assumption load-bearing. The reader's CONDITIONAL verdict is appropriate: the concern is concrete and testable, and it does not automatically invalidate the Leverage phenomenon or the deployed CEM approach. My read does not move the verdict, hence UNCHANGED.","tokens_in":18949,"tokens_out":7917,"duration_ms":84452,"concrete_test":"Re-analyze the real logs used in Section 4.1: for the 1,000 stable products, estimate y_it = alpha_i + beta_0 * ad_traffic_it + beta_1 * ad_traffic_{i,t-1} + beta_2 * ad_traffic_{i,t-2} + gamma_t + epsilon_it, where y is organic traffic and ad_traffic is business traffic, with product fixed effects and day dummies. Test H0: beta_0 = 0. If beta_0 is significant (or if a Granger-causality test at lag 0 rejects), same-day dependence exists, x_{t+1} is action-dependent, and Eq. (6)'s reuse of x_{j+1} is invalid. Report the magnitude of beta_0 relative to beta_1 to show whether the one-day lag assumption is a reasonable approximation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central algorithmic claim is the HTLB sample-expansion step (Eq. 6), which is valid only if the recommendation-related state is action-independent: for each real transition <[o_j,x_j], a_j, r_{j+1}, [o_{j+1},x_{j+1}]>, the same x_{j+1} is attached to M simulated actions a_j^(m) that produce different o_{j+1}^(m). This assumes a_t cannot affect x_{t+1} within one time step, i.e., that the organic traffic state at the end of the day is invariant to the day's bidding action. Sec. 5.1 justifies this by 'requires at least two time steps to influence the recommendation-related state,' but no evidence is given for the delay; the Sec. 4 observational analysis is daily and measures levels, not lags. If ad exposure changes user behavior within the same session or day (e.g., an ad click leading to an organic click, or real-time user features affecting ranking), then x_{t+1} depends on a_t, and the expanded transitions are not valid counterfactuals: they attribute the real action's organic outcome to actions that were never taken. The offline simulator also embeds the exposure-effect function U, so it cannot detect this misspecification. Since the online deployment uses CEM rather than HTLB-DDPG, there is no real-world check of HTLB's advantage; Table 2 and Figure 6 may reflect a false model rather than a real property.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies blended advertising and recommendation feeds in e-commerce. The authors argue that consumer behavior on advertised items becomes training data for the recommendation model, so the advertising platform's traffic allocation influences the recommendation platform's organic traffic allocation, a mechanism they call Leverage. They analyze Taobao 'Guess What You Like' data to document organic-traffic changes before, during, and after advertising, then formulate the advertiser's bidding problem as a Partially Known MDP whose state is split into advertisement-related and recommendation-related components. To reduce sample complexity, they propose Hybrid Training Leverage Bidding (HTLB), which expands each real transition by simulating advertisement-related next states with an advertising emulator while reusing the observed recommendation-related next state. Offline experiments compare HTLB-DDPG with DDPG, A2C, CEM, and manual bids inside an exposure-effect simulator; an online A/B test compares a CEM-based deployed policy with the baseline. The paper claims faster convergence, better stability, and superior traffic increments.","tokens_in":19401,"tokens_out":6376,"duration_ms":64364,"significance":"If the central claims hold, the paper opens a new and practically important axis for bidding optimization in blended feeds: advertisers can strategically acquire organic traffic through the shared behavioral-data loop. The real randomized online A/B test is a genuine strength and provides field evidence that Leverage exists and that bidding policy can shift organic traffic; this is valuable regardless of the RL algorithm used. The state-decomposition and hybrid-sample idea is interesting and, if the action-independence assumption can be validated, could be a useful general technique for partially known MDPs. However, the headline claim of HTLB's superiority is not yet strongly supported: the offline simulator is driven by a fitted exposure-effect function, HTLB's advantage over DDPG lies within overlapping error bars, and the online deployment uses CEM rather than HTLB-DDPG. These gaps are load-bearing for the paper's main algorithmic contribution.","major_comments":[{"comment":"Equation (6) expands each real transition by attaching the same observed recommendation-related next state x_{j+1} to M emulated actions. This is valid only if x_{j+1} is independent of the day's bidding action a_j. The paper asserts in §5.1 that it requires at least two time steps for the bidding action to influence the recommendation-related state, but no lag analysis is reported; §4's observational comparison is at the level of before/during/after advertising averages, not a lagged dependence test. The assumption is especially concerning because the offline time step is one day (§6.1) whereas the online system uses a one-hour decision window (§6.6), so same-window feedback is more plausible in deployment. I ask for direct evidence of the lag structure, for example cross-correlation between exposure and next-period organic scores, and a sensitivity analysis in which x_{t+1} is allowed to depend on a_t or on exposure within the window.","section":"§5.1, Eq. (6)"},{"comment":"Table 2 reports converged episode organic-traffic increments of 257,755±11,589 for HTLB-DDPG and 228,610±49,138 for DDPG. The two are well within each other's error bars, and no significance test is reported, so the superior performance of HTLB over DDPG is not established by these numbers. Moreover, the offline environment embeds the non-parametrically fitted exposure-effect function U_i, so every algorithm, including the manual baseline, is evaluated under a model trained on the same product data. This makes the comparison unable to detect misspecification of U_i. Please add out-of-sample predictive checks for U_i, report paired or standard errors over the five runs, and state explicitly what portion of the HTLB advantage is attributable to the emulator expansion rather than to the fitted model.","section":"§6.1, Table 2"},{"comment":"The online A/B test uses the CEM algorithm, not HTLB-DDPG, and additionally changes the decision window from one day to one hour, clips bid adjustments, and adds auxiliary rewards. Table 5 therefore provides evidence that a Leverage-aware bidding policy can increase organic traffic over the baseline, but it does not validate the HTLB training method or its claimed offline advantage. The abstract and Section 7 say the online deployment demonstrates superior performance of the approach; this overstates what the deployment shows. Please either deploy HTLB online or clearly separate the claims: the online experiment validates the Leverage mechanism and the CEM policy, while HTLB remains validated only through the offline simulator.","section":"§6.6, Tables 4-5"},{"comment":"Assumption 2 is the load-bearing Markov assumption z_{t+1}=U_i(p_t) that the exposure-effect function depends only on the current period's exposure. The paper motivates it with the qualitative observation in Figure 3b but presents no estimate of U_i or its lag structure in the main text; the offline experiments then use a fitted U_i as ground truth. Given that this assumption is the mechanism connecting business traffic to organic traffic, the paper should at least provide an empirical check that a one-lag Markov model predicts held-out organic scores no worse than models with longer lags or with additional state, such as recent ad exposure.","section":"§4.2, Eq. (1)"}],"minor_comments":[{"comment":"The text refers to Table 6 but the table caption says Table 1, and the column counts are hard to parse; renumber tables consistently and clarify that the last column counts only the 302 items with a significant af-ad drop.","section":"§4.1, Table 1"},{"comment":"The entry '49..57%' for 20180925, Level 2 should read '49.57%'.","section":"§6.6, Table 5"},{"comment":"The axis labels of Figure 7 are duplicated and misaligned; give each panel its own clearly labeled x- and y-axis.","section":"Figure 7"},{"comment":"Please define the stable-traffic selection threshold used to choose the 1,000 products; the variance threshold is mentioned but its value is not given.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The online A/B results fluctuate strongly across levels and dates, for example the Level 2 gap is -148.78% on 20190106, and only short one-week windows are reported. I would advise the editor that the online evidence is suggestive but not definitive, and that the HTLB-specific claim should not be promoted without additional analysis. The paper is within scope for a conference on applied machine learning and the Leverage phenomenon is worth publishing, but the authors should be required to either strengthen the offline validation or carefully narrow their claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, this paper is the first in the cited bidding/rec-ads literature to treat the recommender's training-data feedback loop as something an advertiser can deliberately exploit, and it backs that up with a real randomized A/B deployment on Taobao's Guess What You Like feed. The core observation—ad exposure changes user behavior, which becomes training data for the recommender, which shifts organic traffic—is worth taking seriously. Second, the headline algorithm HTLB-DDPG is not as well supported as the Leverage phenomenon itself. The offline advantage over plain DDPG is within noise, and the online deployment used CEM, not the proposed method.\n\nWhat is genuinely good: the PKMDP decomposition is sensible. The advertising state is emulatable, the recommendation state is not, and the two-step action-independence assumption, if true, gives a clean Dyna-style hybrid sample expansion. The observational analysis of 1,000 products, showing that organic gains persist after ads stop, is a nice real-data observation. The appendix includes a second online experiment with a different date range, which helps.\n\nThe soft spots are real and mostly center on the transition expansion in Eq. 6. Each real transition contributes one x_{j+1} that gets attached to M counterfactual actions. That is only valid if a day's bidding cannot affect the recommendation state within the same day. The paper asserts this in Sec. 5.1 and gives the two-time-step story, but the Sec. 4 analysis is daily levels, not lags. If same-session clicks or rapid model refreshes matter, the hybrid samples are not counterfactuals and HTLB's faster convergence is an artifact of a false model. The offline simulator embeds the fitted exposure-effect function U_i, so it cannot detect this misspecification. Also, Table 2 shows overlapping error bars (HTLB 257,755 ± 11,589 vs. DDPG 228,610 ± 49,138), and the online A/B gap jumps around before stabilizing, with no significance tests or confidence intervals.\n\nThese are addressable, not fatal. The central Leverage claim is not defined into existence: the randomized A/B split shows a large, sustained positive gap after deployment, which is strong evidence that ad bidding does influence organic traffic. The HTLB advantage, though, needs better evidence.\n\nWho is this for: anyone working on ad-organic interplay, e-commerce feed design, or industrial RL. It deserves a serious referee. My recommendation: accept with major revisions requiring (i) evidence on the lag between ad delivery and recommendation model updates, (ii) statistical tests or confidence intervals for the online A/B, and (iii) an offline evaluation that does not rely on the same fitted U_i used in training. The authors are at Alibaba; that data should be available to them.","headline":"A plausible and novel Leverage mechanism backed by a real deployment, but the proposed HTLB algorithm's advantage is not yet convincingly demonstrated.","tokens_in":19850,"tokens_out":2066,"would_cite":true,"duration_ms":23764,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes that business traffic allocation can affect organic traffic allocation in blended e-commerce feeds, and that a bidding policy trained on this Leverage effect can substantially increase a product's recommended organic…","keywords":["online advertising","organic traffic","e-commerce product feed","reinforcement learning","recommendation systems","bidding strategy","Leverage mechanism","DDPG"],"falsifier":"Run the bidding policy in a live feed while randomly perturbing bid adjustments within a decision window and measure whether recommendation-side exposure or predicted scores move before the next scheduled model update; if same-window changes appear, the action-independence assumption fails, and if no changes appear until the update, it holds.","tokens_in":18753,"feed_emoji":"📈","tokens_out":5541,"duration_ms":55484,"temperature":0.7,"pith_summary":"Most e-commerce feeds blend paid and recommended products selected from the same candidate pool, and both platforms train on the same consumer behavior logs. This paper tries to establish that the paid impressions a product wins are not isolated transactions: they feed into the recommendation model's training data and thereby shift the product's future organic, free traffic. It then argues that advertisers can treat this Leverage effect as a controllable resource and learn bidding policies that maximize cumulative organic traffic, not just clicks or purchases. The proposed hybrid training algorithm is validated offline and in a live deployment, where the reported gap in organic traffic growth between optimized and baseline buckets reaches tens of percentage points within days.","feed_headline":"Bidding strategy lifts organic traffic by up to 64.7%","feed_subtitle":"A reinforcement-learning bidder exploits the way ad exposure feeds recommender training; A/B tests confirm the link.","key_machinery":"The load-bearing object is the Leverage dynamic model built from two functions: the traffic-win function T, mapping a product's average recommended score to its organic exposure, and the exposure-effect function U, mapping exposure back to the next recommended score; composed, they form a Markov chain whose stable point is the product's organic traffic under fixed conditions. On top of this, the paper defines a Partially Known MDP (PKMDP) in which the state is $[o, x]$, where $o$ is the advertisement-related state, simulable via an advertising emulator $f_{ad}$, and $x$ is the recommendation-related state, unknown but independent of the current bidding action. The HTLB transition expansion exploits that independence to synthesize transitions $\\langle[o_j, x_j], a_j^{(m)}, r_{j+1}, [f_{ad}([o_j, x_j], a_j^{(m)}), x_{j+1}]\\rangle$ for $M$ exploratory actions, sharing the real $x_{j+1}$, which is what cuts sample complexity.","core_discovery":"The paper's central claim is that business traffic allocation can affect organic traffic allocation: when a product is advertised, the extra consumer feedback changes the historical data the recommendation platform retrains on, which changes the product's recommended score and hence its organic traffic. The authors first document the phenomenon in real feed data, showing that most advertised products gain organic traffic during an ad campaign and some keep the gain after the campaign ends, including products that escape the recommendation system's cold start. They then formulate the optimization as a Markov decision process whose state decomposes into a model-known advertising state and an action-independent recommendation state, and propose HTLB-DDPG, a hybrid training algorithm that expands experience replay with emulator-simulated bids while reusing the real next recommendation state. Offline learning curves and an online A/B deployment with over a thousand products support the claim that the optimized bidding policy increases leveraged organic traffic, with the reported gap in relative organic traffic increments between treated and control buckets reaching 46.64% to 64.71% at Level 1 by 20180925.","pith_inferences":["One implication the paper leaves implicit: if paid exposure systematically improves organic scores, the auction itself becomes a driver of recommendation quality, so a platform's choice of which exposures enter the shared training data will partly determine which products get recommended.","A testable extension would be to replace the one-day action-independence assumption with lead-lag analysis: compare recommendation-score changes under randomized bid shocks delivered at different times of day to find the true delay between ad delivery and recommender update.","The reported online gaps suggest a feedback flywheel: early wins in organic traffic feed the recommendation model, compounding later gains; measuring how long the gains persist after the policy stops would separate the flywheel from the direct effect."],"forward_implications":["Advertisers can optimize organic traffic as a direct objective rather than treating it as an unresponsive externality.","Because organic traffic is free, gains from the optimized policy translate into added exposure without additional media cost.","Spending more on ads is not enough: the comparison of minimum and maximum bid ratio policies shows that allocation, not volume, determines organic gains.","The HTLB training scheme is portable: applying it to A2C also improves learning speed and stability, suggesting any model-free RL bidder can exploit the action-independence structure.","If the Leverage mechanism operates in other blended feeds, the same bidding-as-optimization perspective should transfer to search result pages and social feeds."],"supporting_citations":[{"why":"Provides the base off-policy actor-critic DDPG algorithm that HTLB-DDPG extends.","marker":"[12]"},{"why":"Defines the Dyna model-based planning framework that HTLB adapts to a partially known transition model.","marker":"[17]"},{"why":"Supplies experience replay and target networks used in the DDPG implementation.","marker":"[14]"},{"why":"Defines deterministic policy gradient, the learning rule underlying the actor.","marker":"[16]"},{"why":"Supplies the win-rate curve analogy used to justify the shape of the traffic-win function T.","marker":"[23]"},{"why":"Introduces the bid-adjustment ratio setting and eCPM ranking used in the platform.","marker":"[25]"},{"why":"Demonstrates multi-agent bidding with bid adjustment ratios, providing context for the action space.","marker":"[11]"}],"fun_headline_variants":["RL bidder raises organic traffic up to 64.7%","Ad bids exploit recommender feedback for organic gains","Bid-to-rank: ad spend boosts organic traffic up to 64.7%","Reinforcement-learned bidding lifts organic traffic via feedback loop","Strategic ad bids turn ad exposure into organic recommendations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The hybrid training method assumes that the recommendation platform's state one day later does not depend on today's bid adjustment, so one observed next state can be reused for many simulated bids; if the recommender reacts to ad delivery within the same day, the expanded training samples are invalid.","fun_headline_variants_meta":{"raw":{"variants":["RL bidder raises organic traffic up to 64.7%","Ad bids exploit recommender feedback for organic gains","Bid-to-rank: ad spend boosts organic traffic up to 64.7%","Reinforcement-learned bidding lifts organic traffic via feedback loop","Strategic ad bids turn ad exposure into organic recommendations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001848,"raw_usage":{"total_tokens":7258,"prompt_tokens":939,"completion_tokens":6319,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":6233}},"tokens_in":555,"tokens_out":6319,"duration_ms":41713,"temperature":1.0,"reasoning_tokens":6233,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:37:32.610518+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the bidding policy in a live feed while randomly perturbing bid adjustments within a decision window and measure whether recommendation-side exposure or predicted scores move before the next scheduled model update; if same-window changes appear, the action-independence assumption fails, and if no changes appear until the update, it holds.","supporting_citations":[{"cited_title":"1998.Reinforcement learning: An intro- duction","cited_arxiv_id":null,"evidence_quote":"Defines the Dyna model-based planning framework that HTLB adapts to a partially known transition model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines deterministic policy gradient, the learning rule underlying the actor."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the win-rate curve analogy used to justify the shape of the traffic-win function T."},{"cited_title":"wh-ad vs. be-ad","cited_arxiv_id":null,"evidence_quote":"Introduces the bid-adjustment ratio setting and eCPM ranking used in the platform."},{"cited_title":"Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising","cited_arxiv_id":"1802.09756","evidence_quote":"Demonstrates multi-agent bidding with bid adjustment ratios, providing context for the action space."}],"review_version":1}