{"id":"73bec89e-a1a7-4a5b-a9f7-0f765f955d2a","arxiv_id":"2412.10381","paper_version":6,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"SL-MGAC combines supervised reward prediction, user-group decomposition, and actor-critic RL to allocate live streams in a feed; offline and online tests report gains, but the reward predictor is partly fed the true reward bin.","lead":"This paper describes an algorithm for deciding when to insert a live stream into a short-video feed, using reinforcement learning with extra supervised training signals and user-group splitting. The authors report higher live-stream watch time and stable behavior in a large deployment, but the core reward-learning component appears to leak the true answer into its own input.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 10 feeds the true reward-bin one-hot into the reward prediction network, so the critic's Q-value is conditioned on the realized reward and the claimed variance reduction is partly an oracle effect.","rationale":"The reader's weakest assumption identifies the same weak point I would select: the posterior one-hot inputs in Eq. 10 are label leakage. In standard critic learning, Q(s,a) is an expectation over the reward distribution; conditioning the network on the realized reward bin changes the object being learned and makes the actor loss a reward-weighted regression on privileged information. The paper's own note that o(t+1) can be recovered from r_{t+1} shows the authors are aware of the one-hot inputs, but it does not make the construction a valid Q-function. If the leakage is benign, the experiment I propose would show no material drop; if it is not, the central mechanism claim and the empirical gains are not attributable to the proposed supervised learning enhancement. I therefore agree with the reader's REJECT verdict and do not recommend changing it. Other concerns, such as tuning lambda on the same online objectives used to claim constraint satisfaction and the absence of confidence intervals, are secondary; the label leakage alone is sufficient to block the central claim.","tokens_in":15918,"tokens_out":7772,"duration_ms":77407,"concrete_test":"Retrain SL-MGAC from the provided repository with the RPN in Eq. 10 modified to use a predicted time bin (e.g., the argmax or soft mixture over bins from state-action features) instead of the true one-hot o(t)_l and o(t)_v, keeping all other hyperparameters and data identical. Then recompute the Table 1 offline cumulative reward and the Table 3 online gains; if the leakage-free variant loses the 15.5-point offline margin over RLUR and the online watch-time and retention gains, the label leakage is load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing mechanism is the supervised reward-learning critic described in Section 3.3.2. In Eq. 10, R_theta_i(s_t,a_t) is computed using o(t)_l and o(t)_v, the one-hot vectors of the actual live-stream and short-video watch-time bins of the sample. These bins are derived from the reward r_t that the Reward Prediction Network is supposed to predict. Consequently, Q_phi_i(s_t,a_t) in Eq. 8 is not a function of (s_t,a_t); it is a posterior estimate conditioned on the realized reward bin at time t. This Q value is then used in the actor loss (Eq. 13) and in the TD label (Eq. 7), so the reported variance reduction and the offline/online gains in Tables 1 and 3 can be explained by the critic having access to the label, rather than by the multi-group decomposition or multi-task learning. The paper's remark that o(t+1) can be obtained from r_{t+1} in the dataset confirms the leakage for the target term but does not address the current-term leakage. Unless a leakage-free variant with matched performance is demonstrated, the central claim that supervised reward learning stabilizes critic learning is unsupported as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SL-MGAC, an actor-critic algorithm for deciding whether to insert a live stream into a short-video feed at Kuaishou. The algorithm combines a multi-group user decomposition, layer normalization, a critic decomposed into a reward prediction network and a Q residual network, distribution-discretized reward learning, and a softmax-normalized actor loss. Offline NCIS evaluation and a five-day online A/B test report improvements over SAC and other baselines in live-stream watch time while maintaining app usage duration and retention. The central claimed contribution is that multi-task supervised reward learning plus multi-group decomposition reduces variance and improves stability in this final-stage live-stream allocation problem.","tokens_in":16127,"tokens_out":11013,"duration_ms":103968,"significance":"If the claims held, this would be a practically valuable industrial RL deployment with released code and a clear risk story. The variance-reduction components (multi-group decomposition, normalization, Q-normalization) are plausible and worth studying. However, the main validation is undermined by a label-leakage flaw in the reward prediction network and by tuning of the constraint weight on the evaluation metrics, so the experiments do not currently support the central attribution.","major_comments":[{"comment":"The current-term reward prediction network is conditioned on the realized reward. In Eq. (10), R_theta_i(s_t,a_t) is computed using o_l(t) and o_v(t), the one-hot vectors of the actual live-stream and short-video watch-time bins of the logged sample, so these inputs are derived from r_t, the reward the RPN is supposed to predict. Consequently Q_phi_i(s_t,a_t) in Eq. (8) is a posterior quantity conditioned on the observed outcome of a_t, and this leaked Q is used both in the actor loss (Eq. (13)) and in the critic regression (Eq. (7)). The same inconsistency appears in the target term: Eq. (7) takes a max over a_{t+1}, but o(t+1) encodes the realized reward of the logged next action, so for the non-logged next action the bin is unobserved and the target cannot be evaluated as written unless the max is silently replaced by evaluation at the logged action. The paper's remark after Eq. (10) only justifies the availability of o(t+1); it does not address the current-term leakage or the max-over-action problem. A leakage-free ablation, in which o_l/o_v are removed from the current RPN input, is needed to support the central claim that supervised reward learning stabilizes critic learning; without it, the offline and online gains in Tables 1 and 3 cannot be attributed to the proposed mechanism.","section":"Section 3.3.2, Eq. (10) and Eq. (7)"},{"comment":"The constraint weight lambda is selected on the same online metrics used to claim constraint satisfaction. Table 5 reports live-stream DAU, watch times, app usage duration, and user retention for lambda in {0.2, 0.1, -0.1, -0.2}, and lambda = 0.1 is then presented as the setting in which SL-MGAC improves app usage duration and retention. Because this is selection on the evaluation metric, the constraint-satisfying behavior reported in Table 3 is at least partly a tuning artifact and is not evidence that the algorithm is inherently better at satisfying platform-level constraints. Please pre-specify a validation period or report the selection rule, provide the same lambda sweep for the baselines, and treat the final lambda as a selected hyperparameter with associated uncertainty.","section":"Section 4.3 and Appendix C, Table 5"},{"comment":"All offline and online comparisons are point estimates without confidence intervals, standard errors, number of independent runs, or significance tests. The offline advantage over the best baseline is 458.49 versus 443.01 for RLUR, and the online gains are +0.197% video watch time, +0.121% app usage duration, and +0.086% user retention; these magnitudes are small enough that they could easily be within day-to-day or user-segment noise. The stability comparisons in Fig. 4 and Fig. 8 also lack a statistical test and rest on visual inspection of shaded areas and amplitude densities. In addition, the NCIS offline evaluation does not state the logging policy, the propensity model, or the clipping constant, which are essential for interpreting the cumulative-reward numbers. Without this statistical and methodological support, the abstract's claims of outperforming baselines and exhibiting enhanced stability are not established.","section":"Section 4.4 and Section 4.7, Tables 1-3"}],"minor_comments":[{"comment":"The relationship between the N_l+1 and N_v+1 one-hot vectors in Eq. (10) and the eight-dimensional output of the RPN should be spelled out; the extra bin for the a_t = 0 case is mentioned in the text but is not reflected in the bin-boundary table.","section":"Section 3.3.2 and Table 4"},{"comment":"The notation Q_label(s_{t+1}) is misleading because the target depends on (s_t, a_t) through r(s_t, a_t); please rename it to Q_label(s_t, a_t) or simply 'target'.","section":"Eq. (7)"},{"comment":"The phrase 'over 10 rounds of training' should state whether the curves are averages over 10 random seeds, 10 dataset shuffles, or 10 independent training runs.","section":"Figure 4 caption"},{"comment":"The caption should state explicitly that all values are relative improvements over the SAC baseline; otherwise the rows for L2R, Dummy, and SL-MGAC-0 are hard to interpret.","section":"Table 3 caption"},{"comment":"The section titled 'More Ablation Studies' is a hyperparameter sensitivity analysis for lambda, not an ablation of the model components; the title should be adjusted to reflect its content.","section":"Appendix C"},{"comment":"The reward y_l - (lambda/B)y_v is not exactly the penalized reward of Eq. (4), which contains the factor (1+lambda)y_l; the text should explain that lambda has been rescaled by 1/(1+lambda), otherwise the connection to the Lagrange multiplier in Eq. (3) is confusing.","section":"Eq. (5)"}],"recommendation":"reject","confidential_remarks":"The label-leakage issue in Eq. (10) is severe enough that I cannot recommend acceptance of the present version. The code release is a positive step, but any resubmission would need a leakage-free variant of the reward prediction network, a pre-specified lambda selection protocol, and statistical support for the offline and online claims. The current evidence does not support the central attribution of the reported gains to the proposed variance-reduction mechanism."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this is a real industrial RL deployment, but the central claim that supervised reward learning stabilizes the critic is undercut by a label-leakage problem in Eq. 10, so the reported gains are not attributable as written.\n\nWhat's new: the paper applies multi-group state decomposition to both actor and critic, combines supervised reward learning with TD learning, and uses distribution discretization plus layer normalization to fight Q-value divergence. The problem setup—binary live-stream allocation at the final stage of a mixed feed—is a real and risky deployment point, and they show online A/B results on a 20% slice of Kuaishou. That's worth taking seriously.\n\nThe good: the engineering is detailed, the offline evaluation uses a standard OPE estimator, and the online results are consistent with the story (SL-MGAC lifts live-stream watch time while holding app usage and retention roughly flat). The ablation confirms that the Q-normalization trick matters a lot (w/o Q-norm drops from 458 to 392), which is a useful empirical fact. The code link is provided, though I couldn't verify it without running.\n\nThe problem: Eq. 10 feeds the true watch-time bin one-hot, o(t)_l and o(t)_v, into the reward prediction network for the current time step. Those one-hots are derived from r_t, the very reward the network is supposed to predict. So Q(s_t,a_t) is conditioned on the realized outcome at time t. The paper argues this doesn't contaminate the target Q(s_{t+1},·) because you can compute o(t+1) from r_{t+1} in the dataset—true, but that doesn't address the current-step leakage. The critic, and therefore the actor's training signal, sees the label. That makes the variance reduction partly an oracle effect. Without an ablation that removes the one-hot input (e.g., using a predicted bin or no bin at all) and shows matched performance, the central mechanism is unsupported.\n\nOther soft spots: no confidence intervals or significance tests anywhere, and lambda is selected from online A/B results (Table 5), so the constraint-satisfying behavior is partly a tuned artifact. Minor: the comparison to RLUR is thin—RLUR is cited but not described in the method section.\n\nWho this is for: anyone building final-stage RL for recommendation systems will find the deployment details and the instability discussion useful. The label-leakage issue is also a good teaching example for why posterior conditioning in critic networks is dangerous.\n\nFor peer review: this deserves a serious referee, but the current form should be rejected because the key claim is not supported. I'd ask for a leakage-free variant, confidence intervals, and an honest treatment of lambda selection before accepting.","headline":"A genuine industrial RL deployment whose headline mechanism is undermined by label leakage in Eq. 10, so the empirical gains don't yet support the central claim.","tokens_in":16692,"tokens_out":3859,"would_cite":false,"duration_ms":34700,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SL-MGAC stabilizes reinforcement learning for live-stream allocation in a large mixed feed, outperforming all baselines in offline evaluation and an SAC-based baseline in a production A/B test.","keywords":["Reinforcement Learning","Recommendation System","Variance Reduction","Live Stream Allocation","Actor-Critic","Constrained Markov Decision Process","Offline Policy Evaluation","Online A/B Testing"],"falsifier":"Retrain SL-MGAC on the same offline dataset with the posterior one-hot bin vectors in Eq. 10 replaced by a feature-only estimate (or dropped) while keeping multi-group decomposition, distribution discretization, and normalization fixed, and compare cumulative reward and Q-value variance: if performance stays near 458.49 the mechanism survives, and if it drops toward 449.13 the label conditioning is the load-bearing part.","tokens_in":15686,"feed_emoji":"📺","tokens_out":9888,"duration_ms":84172,"temperature":0.7,"pith_summary":"At the final stage of a mixed short-video and live-stream feed, the system must decide for each user request whether to inject at most one live stream into the video feed. The paper argues that this decision cannot be made greedily: over-injecting live streams raises live-stream metrics in the short run but reduces total app usage and retention. To make reinforcement learning usable for this high-risk decision, the authors propose SL-MGAC, an actor-critic algorithm whose critic is augmented with multi-task supervised reward prediction, whose state is decomposed across user activity groups, and whose Q-values and rewards are normalized. The reported evidence is an offline policy evaluation where SL-MGAC reaches a cumulative reward of 458.49 versus 443.01 for the best baseline, and a five-day online A/B test against a SAC-based allocation baseline showing +7.431% live-stream watch time, +0.197% video watch time, +0.121% app usage duration, and +0.086% user retention, with a visibly more stable live-stream injection ratio.","feed_headline":"7.4% more live-stream watch time, and app time rises too","feed_subtitle":"In a production A/B test, SL-MGAC keeps video watch time and retention positive while raising live-stream engagement.","key_machinery":"The load-bearing identity is the additive critic decomposition $Q_{\\phi_i}(s_t,a_t)=R_{\\theta_i}(s_t,a_t)+\\gamma T_{\\xi_i}(s_t,a_t)$ (Eq. 8), in which the Reward Prediction Network $R$ is trained both by a supervised loss on the reconstructed watch time and by the TD target of the critic. The supervised branch uses posterior one-hot bin vectors $o(t)_l, o(t)_v$ and linear reconstruction of the watch time from a predicted within-bin proportion $\\delta$ (Eq. 10), which the paper argues reduces variance relative to predicting raw watch time. Independent Multi-Group State Decomposition modules in actor and critic partition users into $K$ activity-level groups, each with its own MLP, so that sparse live-stream feedback is shared within groups rather than learned per sample. Layer normalization on inputs and softmax normalization of Q-values in the actor loss complete the stabilization machinery.","core_discovery":"On the paper's own terms, SL-MGAC establishes that the instability that blocks RL deployment in live-stream allocation can be traced to high-variance reward and Q-value learning, and that this variance can be contained by three coordinated changes. First, the critic is split into a Reward Prediction Network and a Q Residual Network; the reward branch is trained by supervised multi-task learning to predict live-stream and video watch-time proportions inside discretized time bins, which restricts bootstrapping error accumulation. Second, a multi-group state decomposition partitions users into K groups by historical live-stream activity and runs separate group-wise MLPs for actor and critic, reducing prediction variance. Third, Layer Normalization, reward normalization, and a softmax-normalized Q-value in the actor loss keep gradients stable. With these changes, the paper reports both higher cumulative reward than all baselines in offline evaluation and positive gains on every online engagement metric while keeping the live-stream injection ratio steadier.","pith_inferences":["A testable next step would ablate only the posterior one-hot inputs in Eq. 10 while keeping multi-group decomposition and distribution discretization fixed; the current ablations remove several components at once, so the individual contribution of the label conditioning is not yet isolated.","If this mechanism is the true source of stability, the same split of a critic into a supervised reward predictor plus a residual Q-value could apply to any recommender-side RL task with noisy, heavy-tailed feedback, not just live-stream injection.","The ablation data suggest the softmax-normalized actor loss (Q-norm) is the single largest contributor, since removing it drops offline reward from 458.49 to 392.12; exploring a temperature-scaled variant or a learned baseline might push performance further."],"forward_implications":["If the reported offline and online results hold, RL can be deployed at the final allocation stage of a large feed, where a policy collapse would directly harm ad and e-commerce exposure.","The policy network alone is small enough for sub-20ms inference on a high-request-rate system, because the complex critic is used only as a training-time teacher.","The platform-level constraint becomes a tunable knob: the penalty coefficient $\\lambda$ in the reward $\\tilde r = y_l - \\frac{\\lambda}{B} y_v$ makes the allocator more conservative as $\\lambda$ grows and more greedy as it turns negative.","The multi-group state decomposition is transferable to other binary injection scenarios such as advertising and e-commerce allocations, as the authors state."],"supporting_citations":[{"why":"supplies the theoretical result that Layer Normalization bounds the neural tangent kernel and thereby limits Q-value divergence, which motivates the normalization used throughout SL-MGAC.","marker":"[45]"},{"why":"provides Clipped Double Q-Learning, the four-critic architecture SL-MGAC adopts and one of the strongest baselines (TD3) it is compared against.","marker":"[13]"},{"why":"is the RLUR baseline that achieves the best offline cumulative reward among baselines and represents prior RL for user retention in short-video recommendation.","marker":"[5]"},{"why":"gives the reward-constrained policy optimization trick that converts the constrained CMDP into the penalized reward used in Eq. 4 and Eq. 5.","marker":"[38]"},{"why":"underpins the actor loss by showing the softmax-normalized advantage-weighted objective is equivalent to a KL-constrained RL problem.","marker":"[33]"},{"why":"supplies the offline policy evaluation protocol that the paper follows to compare policies on the collected real-world dataset.","marker":"[44]"},{"why":"defines the Normalized Capped Importance Sampling estimator used to compute the offline cumulative-reward metric.","marker":"[37]"},{"why":"defines Layer Normalization, which is applied to the inputs of actor and critic networks for training stability.","marker":"[4]"}],"fun_headline_variants":["Stable RL lifts live-stream watch time 7.4%","Variance reduction keys RL for live-stream allocation","Multi-group actor-critic improves feed stability and engagement","RL that doesn't destabilize: live-stream gains 7.4%","Curbing Q-variance enables RL for live-stream feeds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reward-prediction branch of the critic is fed the watch-time bin that each training sample actually fell into, which is the very quantity it is meant to predict; if this label conditioning distorts the critic rather than merely reducing variance, the reported stability and gains are not attributable to the proposed mechanism.","fun_headline_variants_meta":{"raw":{"variants":["Stable RL lifts live-stream watch time 7.4%","Variance reduction keys RL for live-stream allocation","Multi-group actor-critic improves feed stability and engagement","RL that doesn't destabilize: live-stream gains 7.4%","Curbing Q-variance enables RL for live-stream feeds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1547,"prompt_tokens":1012,"completion_tokens":535,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":450}},"tokens_in":628,"tokens_out":535,"duration_ms":4614,"temperature":1.0,"reasoning_tokens":450,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:46:20.850930+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain SL-MGAC on the same offline dataset with the posterior one-hot bin vectors in Eq. 10 replaced by a feature-only estimate (or dropped) while keeping multi-group decomposition, distribution discretization, and normalization fixed, and compare cumulative reward and Q-value variance: if performance stays near 458.49 the mechanism survives, and if it drops toward 449.13 the label conditioning is the load-bearing part.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the theoretical result that Layer Normalization bounds the neural tangent kernel and thereby limits Q-value divergence, which motivates the normalization used throughout SL-MGAC."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"is the RLUR baseline that achieves the best offline cumulative reward among baselines and represents prior RL for user retention in short-video recommendation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the offline policy evaluation protocol that the paper follows to compare policies on the collected real-world dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines the Normalized Capped Importance Sampling estimator used to compute the offline cumulative-reward metric."}],"review_version":1}