{"id":"8f475fd1-7963-40da-be15-31e12c60ead9","arxiv_id":"2506.23090","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"MTORL jointly learns channel recommendation and budget allocation for online advertising from offline user journeys, and reports better accuracy and reward than prior methods on KuaiRand, Criteo, and a Taobao A/B test.","lead":"A new ad-serving system, MTORL, reads a user's past exposure history and predicts both the next advertising channel and the reward of showing an ad, then uses those predictions to choose which users to advertise to under a budget. It reports better accuracy and revenue than strong baselines on two public datasets and in a short Taobao A/B test, which matters because online advertising is one of the largest deployment surfaces for machine learning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"User ranking uses a reward predicted without conditioning on the chosen action (Eq. 12, Algorithm 1), so the budget allocation optimizes an action-marginal quantity rather than the CMDP objective in Eq. (2); the fixed penalty s=0.5 in Eq. (3) is also never validated.","rationale":"The reader's weakest assumption identifies the same core weakness: reward prediction ignores the current action, and the fixed penalty s=0.5 is arbitrary. This is an internal design issue, not a disagreement with external consensus. The paper explicitly rejects using predicted actions to predict rewards to avoid cumulative errors, but that choice makes the reward action-marginal. The CMDP objective in Eq. (2) requires the reward of a chosen action; a state-value estimate under the behavior policy is not the same quantity. If the reward decoder cannot distinguish between serving channel A and serving channel B, then ranking users by this reward cannot correctly allocate channels under a budget. The concern is load-bearing because budget allocation is one of the two advertised contributions, and the reported average-reward gains in Table 3 could be driven by the user filter selecting users with high baseline conversion propensity rather than by a correct allocation mechanism. A separate possible data-leak issue (whether the touchpoint feature vector q_t contains the channel label c_t) is related but ambiguous from the text: Eq. (3) keeps c_t in the action and q_t in the state, suggesting they are intended to be disjoint, so I do not rest the critique on that ambiguity. The online A/B test also has imprecise effect reporting, but the decisive structural check is the offline budget simulation with a conditional reward decoder. The reader's conditional verdict is appropriate: the paper should run this test and report the sensitivity to s before the budget-allocation claim is accepted.","tokens_in":23569,"tokens_out":8313,"duration_ms":86511,"concrete_test":"Run an offline budget-allocation simulation on KuaiRand-Pure with a fixed budget W and the exact Algorithm 1. Compare (i) the published reward decoder (Eq. 12), (ii) a variant whose reward decoder input is Concat(~s_t, onehot(\\hat{a}_t)), and (iii) an oracle that uses the logged true r_t for the channel actually served. Measure total gained reward and budget violation. Also sweep s in Eq. (3) over {0.1, 0.3, 0.5, 0.7, 0.9} and record which s yields the best total reward under the budget. If the conditional variant (ii) or the oracle (iii) substantially outperforms (i), or if the optimal s differs from 0.5, the current allocation objective is misspecified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.5 defines the reward decoder to predict r_t directly from the causal states ~s_t (Eq. 12), deliberately omitting the current action a_t. Section 3.7.2 then uses these predictions to rank users in Algorithm 1 (line 6) and to construct the implicit channel CTR ratios for the initial policy. But r_t in Eq. (3) is the realized outcome of the exposure, r_t = MinMaxNorm(g_t - s w_t) with s fixed at 0.5, so it is causally action-dependent. An action-marginal estimate can only recover the average reward under the logging policy; it cannot answer the counterfactual question of what this user would yield if a given channel is served. Ranking users by this quantity therefore maximizes a different objective than the CMDP in Eq. (2), which requires conditional reward information for the chosen policy. In addition, the penalty s=0.5 is never tuned or justified against the budget constraint, although in a Lagrangian formulation s should be adjusted to enforce the budget. The offline reward experiments in Table 3 report only the final average reward and provide no ablation that varies s or that compares against a reward decoder conditioned on the predicted action, so it is untested whether the user filter is selecting high-propensity users rather than allocating channels correctly.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MTORL, a multi-task sequence-modeling method for online advertising. It formulates channel recommendation and budget allocation as a constrained MDP, encodes user journeys with a dilated causal convolution encoder and a causal attention module, and decodes channel actions and penalized rewards jointly. Training uses cross-entropy on logged actions, an auxiliary reward prediction loss, and a DPO loss. A deployment procedure ranks users by predicted reward and reweights channels by explicit and implicit CTR ratios to allocate budget. Experiments on KuaiRand-Pure and Criteo compare against DL, offline Q-learning, and Decision Transformer baselines on accuracy and average reward; an online A/B test on Taobao reports a 0.08 CTR and 0.23% RPM gain.","tokens_in":23940,"tokens_out":5781,"duration_ms":57800,"significance":"If the empirical claims are correct, MTORL provides a deployable, integrated solution for channel recommendation and budget allocation that outperforms a broad set of baselines on public benchmarks and in an online test. The paper's strengths are its released code, the breadth of baselines, the component ablation, and the inclusion of an online A/B experiment. However, the central validity of the budget-allocation claims depends on two assumptions that are not established: that rewards can be predicted without conditioning on the served channel, and that a fixed penalty strength s=0.5 enforces the budget constraint. The lack of standard deviations and a precise budget-allocation protocol also makes the reported average-reward superiority difficult to verify. These issues are fixable, but they require substantial additional analysis.","major_comments":[{"comment":"The reward decoder in Eq. (12) predicts r_t from causal states \\tilde{s}_t and deliberately omits the current action a_t, while the training reward in Eq. (3) is the realized outcome r_t = MinMaxNorm(g_t - s w_t) of the exposed channel. Consequently \\hat{r}_t is an action-marginal estimate under the logging distribution, not an estimate of r(s,a). Algorithm 1 (line 6) ranks users by this marginal reward, and Section 3.7.2 uses it to compute per-channel implicit CTR ratios \\hat{CTR}_j = \\sum I(\\hat{r}_t > \\tau | a_j)/N_j; without action-conditioned reward predictions the condition '| a_j' cannot produce a channel-specific quantity. The reported average-reward gains in Table 3 therefore cannot be attributed to correct channel-level or user-level allocation. Please condition the reward decoder on the chosen action (or provide an ablation showing that the action-marginal reward suffices for ranking).","section":"3.5, Eq. (12); 3.7.2; Algorithm 1"},{"comment":"The Lagrangian relaxation in Eq. (2) introduces a penalty strength s that should be adjusted to satisfy the budget constraint in Eq. (1), but Section 3.1.1 fixes s = 0.5 and no ablation or tuning of s is reported. As a result, the budget constraint is never enforced during policy optimization; Algorithm 1 only stops when W reaches zero at serving time. Please report how W and s are set, sweep s, and show constraint satisfaction (e.g., total cost versus W) to support the claim that MTORL performs budget-constrained optimization.","section":"2.2.2, Eq. (2); 3.1.1, Eq. (3); Algorithm 1"},{"comment":"Table 2 claims 'All improvements are statistically significant (i.e., two-sided t-test with p < 0.05)', but no standard deviations, confidence intervals, or test statistics are reported anywhere. Table 3 reports average reward under 'the same budget' without specifying W, the top-N threshold, channel costs, or the exploration-phase budget; without this protocol the budget-fairness of the comparison cannot be verified. Please report means and variances over seeds and a complete budget-allocation evaluation protocol.","section":"4.2, Tables 2 and 3"},{"comment":"The policy objective is cross-entropy against logged actions plus a DPO term; the model has no return-to-go conditioning, no value function, and no constraint-aware policy improvement. Thus MTORL is best described as a supervised sequence model with auxiliary reward prediction, and its comparison with CQL, IQL, and DT should be framed as such. Please either adopt an actual offline RL objective (e.g., return-conditioned or value-based) or moderate the claim that MTORL is an offline RL method.","section":"3.6, Eqs. (13)-(16)"}],"minor_comments":[{"comment":"The binary cross-entropy term should be (1-r_t)\\log(1-\\hat{r}_t); the printed (1-r_t)(1-\\log(\\hat{r}_t)) is incorrect.","section":"3.6, Eq. (14)"},{"comment":"The implicit budget ratio defines \\hat{p}_j but uses p_j on the right-hand side; the notation should be made consistent.","section":"3.7.2"},{"comment":"The DPO loss omits the reference policy \\pi_ref and does not explain how the preference pairs \\{x_w, x_l\\} are sampled; please clarify.","section":"3.6, Eq. (15)"},{"comment":"The online experiment reports +0.08 CTR and +0.23% RPM without confidence intervals or test details; the footnote that a 0.001-level gain is significant is not a statistical test. Also, the online system only uses the pretrained user encoder, not the full MTORL pipeline, so the online gain does not directly validate the budget-allocation modules.","section":"4.5"},{"comment":"The text says 'the video types and campaign categories in three datasets' but only two datasets are used; please correct the count.","section":"4.1.1"}],"recommendation":"major_revision","confidential_remarks":"This manuscript appears to be the KDD '25 conference version submitted as a journal paper; the text does not state what new material is added beyond the conference version, which the editor should clarify. The main technical concern, the action-marginal reward prediction, affects the core budget-allocation contribution and requires a substantive revision, not just additional experiments reporting standard deviations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a credible engineering paper. The authors combine a dilated causal TCN, masked self-attention, and a DPO-style loss into a channel-recommendation policy, and the supervised action-prediction results on KuaiRand and Criteo (Table 2) are internally consistent and impressive in absolute terms. The online Taobao numbers are small and thinly documented, but the offline accuracy gains over DT/ODT are large enough to take seriously.\n\nWhat's genuinely new is the deployment recipe: predicting rewards from causal states (deliberately skipping the predicted action) and then using those reward predictions both to rank users for budget allocation and to construct an implicit channel-CTR policy. That specific combination is not in the existing literature, as far as I can tell.\n\nNow the soft spots, in rough order of importance.\n\nFirst, the reward decoder's omission of the current action is a real problem for the budget-allocation half of the paper. Eq. (12) predicts r_t from causal states that include a_{t-1} but not a_t, while Eq. (3) defines r_t as the realized outcome of the chosen a_t. So the user ranking in Algorithm 1 is optimizing a marginal quantity, not the CMDP objective in Eq. (2). The authors justify the design as avoiding error accumulation from a two-hop action-then-reward prediction, but they never test whether conditioning on the predicted action changes the ranking or the final reward. If the reward truly depends on the channel, their user filter is selecting users for the wrong reason. This is the biggest gap, and it is fixable: add an ablation with a reward decoder that takes \\hat{a}_t as an additional input, and report whether the budget-allocation results change.\n\nSecond, the penalty s=0.5 is fixed and never varied. For a Lagrangian formulation, s should be adjusted to meet the budget; the paper just asserts 0.5 is consistent with prior work. A small sensitivity analysis would settle this.\n\nThird, the reporting is statistically thin. Tables 2 and 3 give no standard deviations, despite claiming 10 runs and p<0.05 t-tests. The 'same budget' evaluation protocol for Table 3 is not described, so I can't tell what exactly is being compared. The online experiment is 1% traffic for 48 hours with no confidence interval; the RPM gain of 0.23% could be noise.\n\nFourth, the 'offline RL/CMDP' framing is generous. What is implemented is supervised sequence prediction plus heuristic filtering. That framing doesn't invalidate the results, but reviewers should ask the authors to be precise about what is learned vs. hand-designed.\n\nBottom line: if this is under review, send it out. The action-prediction core is sound, the pipeline is useful, and the budget-allocation flaw is identifiable and repairable rather than fundamental. I would ask for the reward-decoder ablation, error bars, a full budget-allocation protocol, and a sensitivity analysis on s before accepting.","headline":"MTORL is a credible industrial recipe whose action-prediction results stand up, but the budget-allocation claims rest on a reward decoder that ignores the chosen action.","tokens_in":24427,"tokens_out":3607,"would_cite":false,"duration_ms":39656,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that one offline reinforcement learning model can jointly pick ad channels and allocate budgets, beating strong baselines on two public datasets and in a live A/B test.","keywords":["offline reinforcement learning","online advertising","channel recommendation","budget allocation","causal state encoding","causal attention","direct preference optimization","sequence modeling"],"falsifier":"On a held-out logged dataset, if the reward decoder's predictions become significantly more accurate when the actual channel of the next exposure is added to the causal-state input, the reward model is misspecified and the user-ranking signal is unjustified. In a live test, if users ranked in the top tier by predicted reward do not convert at a higher rate than randomly selected users with the same expected cost, the user-level budget allocation mechanism fails.","tokens_in":23374,"feed_emoji":"📈","tokens_out":8787,"duration_ms":86591,"temperature":0.7,"pith_summary":"This paper claims that one offline reinforcement learning model can jointly handle two advertising decisions that platforms usually make separately: which channel should serve a user's next ad exposure, and which users should get ads under a limited budget. The proposed model, MTORL, recasts both tasks as conditional sequence modeling over users' past exposure journeys, predicting the next channel action and a budget-penalized reward from a causal state representation. On two public benchmark datasets the model reports the best channel-prediction accuracy and the highest average reward under a fixed budget, compared with conservative value-based offline RL (CQL, IQL), constrained-RL methods, and transformer sequence models (Decision Transformer). In a live platform A/B test, integrating the learned user encoder into the deployed click-through-rate model yielded a 0.08 CTR gain and a 0.23% revenue-per-mille gain. If these results hold, offline-trained multi-task models of this kind could be a practical way to deploy budget-aware advertising policies without online exploration.","feed_headline":"Offline RL model beats baselines at ad channel and budget allocation","feed_subtitle":"Pre-trained on logged data, the same model also lifted click rate and per-thousand revenue in a live A/B test.","key_machinery":"The load-bearing component is the causal state encoder: a dilated causal temporal convolutional network that forces the hidden state at time $t$ to depend only on states at times $k \\le t$, preserving temporal causality in the user sequence. Around this, the model stacks a causal attention module (masked self-attention in the style of a generative transformer) to emphasize relevant historical exposures, and two decoders: an action decoder that produces the channel policy, and a reward decoder that maps the causal states $\\tilde{\\mathbf{s}}_t$ to the penalized reward. The reward estimates feed the online advertising procedure, where they rank users (user-level budget allocation) and reweight channel policies in proportion to reliable conversion counts (channel-level allocation). A direct preference optimization loss over high-return versus low-return trajectories is added to the cross-entropy action loss and reward loss, giving the multi-task objective. The fixed penalty $s=0.5$ in the reward definition is what lets the budget constraint be folded into the reward signal.","core_discovery":"The central claim is that the sparse-reward, budget-constrained advertising problem is better approached by supervised causal sequence modeling than by value-based offline reinforcement learning. Instead of estimating Q-values, MTORL treats the observed channel as the action label and the normalized penalized reward $r_t = \\mathrm{MinMaxNorm}(g_t - s\\,w_t)$ with $s=0.5$ as the reward label, and trains two decoders on top of a shared sequence encoder. The encoder first applies dilated causal convolutions to a user's exposure history to produce causal states $\\tilde{\\mathbf{s}}_t$ that depend only on past exposures, then applies causal attention to weigh relevant exposures. One decoder predicts the next channel; the other predicts the reward directly from the causal states, deliberately avoiding the two-hop path through predicted actions to prevent error accumulation. Those predicted rewards are then used to rank and filter target users and to reweight channels, turning budget allocation into an inference-time procedure rather than a separate optimization. The paper reports consistent gains over value-based offline RL and transformer sequence baselines on both benchmark datasets, and a live A/B test in which the encoder's user embeddings improved an existing CTR model. The intended conclusion is that a single multi-task offline RL model can form the core of an automated advertising pipeline.","pith_inferences":["The live experiment validates the learned user encoder inside an existing CTR model, but not the full channel-recommendation and budget-allocation loop; the end-to-end value of the complete MTORL pipeline remains an extrapolation.","The penalty term $s=0.5$ is taken as a constant without calibration; if an advertiser's true cost-gain tradeoff differs, the learned rewards would rank users incorrectly, so tuning $s$ against realized profit is a natural next test.","A direct ablation that feeds the actual next channel into the reward decoder would reveal whether the causal-state-only reward model is well specified; if accuracy improves substantially, the ranking signal should condition on the served channel."],"forward_implications":["If MTORL's benchmark results hold, supervised causal sequence modeling can outperform value-based offline RL methods in reward-sparse advertising data.","Reward predictions can serve as a user-ranking signal, so channel recommendation and budget allocation can be learned jointly and executed at inference time under the budget.","The DPO loss provides a sparse-reward remedy that improves policy quality without explicit Q-value estimation.","The automated advertising procedure offers a deployment pattern in which offline pretraining, initial exploration with a blended policy, and rank-then-exploit serve for online use.","The reported online gains suggest that injecting an offline-pretrained user encoder into an existing CTR model can produce measurable lifts in click rate and revenue per mille."],"supporting_citations":[{"why":"Supplies the dilated causal temporal convolutional network used as the causal state encoder.","marker":"[10]"},{"why":"Decision Transformer, the conditional sequence modeling baseline that MTORL extends and outperforms.","marker":"[15]"},{"why":"IQL, a value-based offline RL baseline compared in the experiments.","marker":"[40]"},{"why":"CQL, the conservative Q-learning baseline that motivates the paper's contrast with supervised sequence modeling.","marker":"[43]"},{"why":"Provides the causal attention mechanism (GPT-style causal mask) used to weigh prior exposures.","marker":"[60]"},{"why":"Provides the direct preference optimization loss used to prefer high-return trajectories.","marker":"[61]"},{"why":"Provides the self-attention and causal-mask formulation underlying the causal attention module.","marker":"[68]"},{"why":"Introduces the Lagrangian-constrained offline RL baseline (R-BCQ) and the penalized-reward formulation that the paper adapts.","marker":"[79]"}],"fun_headline_variants":["Multi-task offline RL unifies ad channel and budget decisions","Single model handles ad channel choice and budget allocation","Budget-aware multi-task offline RL beats baselines in ads","Causal encoder improves offline RL for ad channel and budget","Multi-task offline RL with causal attention for ad channel and budget"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on the assumption that a user's next ad outcome can be predicted from her past exposure history alone, without knowing which channel will actually be served, and that the fixed cost penalty of 0.5 reflects the advertiser's true tradeoff.","fun_headline_variants_meta":{"raw":{"variants":["Multi-task offline RL unifies ad channel and budget decisions","Single model handles ad channel choice and budget allocation","Budget-aware multi-task offline RL beats baselines in ads","Causal encoder improves offline RL for ad channel and budget","Multi-task offline RL with causal attention for ad channel and budget"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000388,"raw_usage":{"total_tokens":2065,"prompt_tokens":983,"completion_tokens":1082,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":1002}},"tokens_in":599,"tokens_out":1082,"duration_ms":10945,"temperature":1.0,"reasoning_tokens":1002,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:50:16.074824+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out logged dataset, if the reward decoder's predictions become significantly more accurate when the actual channel of the next exposure is added to the causal-state input, the reward model is misspecified and the user-ranking signal is unjustified. In a live test, if users ranked in the top tier by predicted reward do not convert at a higher rate than randomly selected users with the same expected cost, the user-level budget allocation mechanism fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CQL, the conservative Q-learning baseline that motivates the paper's contrast with supervised sequence modeling."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Lagrangian-constrained offline RL baseline (R-BCQ) and the penalized-reward formulation that the paper adapts."}],"review_version":1}