{"id":"d6451465-4d4d-472f-80c0-fe3f1b952fb5","arxiv_id":"2505.07257","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"DARLR uses a selector agent to pick similar and diverse reference users, then averages their predicted rewards to dynamically refine the reward and uncertainty used to train a recommender policy in offline RL.","lead":"This paper describes DARLR, a dual-agent framework for offline reinforcement learning in recommender systems that dynamically reshapes reward estimates by selecting similar reference users. On four benchmark datasets it reports modest gains over static reward-shaping baselines, though statistical significance is not established.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 15 averages predictions from the same frozen world model; because the selector's own reward (Eq. 12) includes that same \\hat{r}, dynamic shaping may inherit or amplify systematic reward bias rather than correct it, and Figure 4's error metric is not defined tightly enough to rule this out.","rationale":"The paper proposes a plausible mechanism: a learned selector pools predicted rewards from similar but diverse users, hoping to cancel user-specific prediction noise. In good faith, this could work if errors are independent across users. My concern is not that dynamic reward shaping is impossible, but that the paper does not establish the necessary condition, and the specific selector design may push against it. The selector's reward (Eq. 12) includes the raw predicted reward \\hat{r}, so high-predicted-reward users are rewarded as references even when their predictions are overestimates; the similarity and diversity terms do not remove a common bias. Figure 4 is the key evidence but is under-specified: no definition of which pairs are used, whether target-user ground truth is employed, or how the average is computed across epochs. This matters because the main headline claim is precisely that Eq. 15 reduces reward inaccuracy. The paper's ablations show the full method beats its variants, which is useful, but they do not compare reward-estimation accuracy between Eq. 3 and Eq. 15. Thus the conditional acceptance is appropriate: the algorithm may work, but the central mechanism is not yet verified. The proposed test would settle it by measuring MAE directly on held-out interactions and by checking whether selected users' rewards are systematically inflated. I do not see a reason to reject outright, as there is a real empirical signal and the method is reproducible in principle with the provided code. The reader's weakest assumption captures the same core issue, so I agree with that assessment and recommend keeping the conditional verdict.","tokens_in":20066,"tokens_out":5194,"duration_ms":53651,"concrete_test":"On held-out interactions in KuaiRec and KuaiRand, compute (a) MAE of Eq. 15 against Eq. 3 for the same (user, item) pairs relative to ground-truth reward, stratified by item popularity and user interaction density; (b) the mean predicted reward of the selected reference set minus the target user's own predicted reward, and the corresponding difference against ground truth. If Eq. 15 does not reduce MAE, or if selected users' mean prediction is systematically above the target's while ground truth does not support that gap, the dynamic reward is amplifying rather than mitigating world-model inaccuracy. Also re-run Figure 4 with a fully specified protocol, using the same sampled state-action pairs for all methods and measuring error on the target user's own ground-truth reward, to confirm the published comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that iteratively averaging reference-user rewards, Eq. 15, makes reward estimates more accurate and thereby improves offline RL policies. This requires \\hat{r}(u,i_t) = mean_{u' in S} \\hat{r}(u',i_t) to be closer to the true reward for u than the direct prediction \\hat{r}(u,i_t). Nothing in the construction guarantees this: all predictions come from the same frozen world model, so any systematic bias shared across users, such as popularity- or density-dependent overestimation, is inherited by the average. The selector is not trained to minimize reward error; its intrinsic reward, Eq. 12, includes \\hat{r} itself, so the selection policy is rewarded for choosing users the world model already rates highly. If those users are overestimated, the averaged reward is biased upward rather than corrected. The similarity term (Eq. 10) and diversity term (Eq. 11) only shape which users are pooled; they do not debias the common prediction errors. The only direct evidence offered, Figure 4, defines 'Error' as deviation between ground truth and estimated rewards but does not specify which state-action pairs are included, whether the comparison uses the target user's own ground-truth reward, or how trajectories are sampled. Without that protocol, the lower DARLR curve could reflect selection of reference users whose true rewards are easier to predict, rather than improved estimates for the target user. The ablation in Table 4 shows the full method beats removing parts, but it does not directly compare Eq. 15 against Eq. 3 in reward accuracy. Thus the mechanism claimed for the main result is unverified and plausibly biased in the direction of the world model's own errors.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes DARLR, a dual-agent offline model-based RL framework for recommender systems. A selector agent chooses a set of reference users for each recommendation step using an intrinsic reward that combines the world-model reward prediction, user similarity, and diversity (Eqs. 10-12). A recommender agent then uses the average world-model reward over the selected users as the reward estimate (Eq. 15) and an adaptive uncertainty penalty (Eq. 16) when training the recommendation policy. The authors claim that this dynamic reward estimation mitigates the impact of world-model reward inaccuracies, and they report improvements over baselines on KuaiRand, KuaiRec, Coat, and Yahoo, together with ablations and hyperparameter sensitivity studies.","tokens_in":1941,"tokens_out":2022,"duration_ms":71464,"significance":"If the central mechanism were established, the work would offer a practical way to use peer-user information in model-based offline RL for recommendations, a direction with limited prior work. The manuscript has clear strengths: all model-based baselines share the same pretrained world model; the ablation study isolates the similarity/diversity components and static-versus-dynamic designs; the GT-reward upper bound provides a useful reference; and the authors release code. However, the load-bearing claims, namely that Eq. (15) yields more accurate rewards for the target user and that the empirical gains over ROLeR are statistically meaningful, are not yet supported by the evidence presented, so the significance is conditional on substantial revision.","major_comments":[{"comment":"Eq. (15) replaces the target user's predicted reward with the average of predictions from a reference-user set drawn from the same frozen world model, but the paper gives no argument or empirical demonstration that this average is closer to the true reward for the target user than the direct prediction \\hat{r}(u,i_t). Because the selector's intrinsic reward in Eq. (12) includes \\hat{r} itself, a selector trained to maximize that reward tends to choose users for whom the world model predicts high rewards; if those predictions share a systematic bias (e.g., popularity- or density-dependent overestimation), the averaged reward in Eq. (15) inherits and may amplify that bias. Please provide a direct correctness check: on a held-out set, compare the mean absolute error of \\hat{r}(u,i), of the Eq. (15) average, and of a random-user average against ground truth, stratified by interaction density, and add an ablation in which the selector's reward excludes \\hat{r}.","section":"§3.3.1, Eq. (15)"},{"comment":"The 'Error' plotted in Figure 4 is not defined tightly enough to support the claim that DARLR produces more accurate reward estimates. The text says only that it is the deviation between ground truth and estimated rewards; it does not state which state-action pairs are included, whose ground truth is used (the target user's or the reference users'), how trajectories are sampled, or whether the DARLR curve uses Eq. (15) or the direct prediction. Without this protocol, the lower DARLR curve could reflect selection of reference users whose true rewards are easier to predict, rather than improved estimates for the target user. Please specify the exact metric, the aggregation, and the evaluation protocol, and include error bars or multiple seeds.","section":"§4.4, Figure 4"},{"comment":"The headline empirical claim is not supported by the reported statistics. On KuaiRand, DARLR's cumulative reward (13.8152±1.9351) differs from ROLeR (13.4553±1.5086) by about one third of the pooled standard deviation; on Yahoo the difference (68.5418 vs. 68.3637) is within one standard deviation; on Coat the difference (78.0429 vs. 76.1603) is less than one pooled standard deviation; and on KuaiRec (35.2203 vs. 33.2457) the gap is also smaller than the reported variability. No significance tests are reported, so statements such as 'superior performance' and 'outperforms ROLeR significantly' are not justified. Please add paired bootstrap or equivalent tests across the common seeds, or present effect sizes with confidence intervals and soften the claims accordingly.","section":"Tables 2 and 3; §4.2"},{"comment":"The manuscript repeatedly describes DARLR as dynamically 'updating world models' and as having an 'evolving world model', but Algorithm 1, Eq. (15), and Eq. (17) update only the selector, the recommender, and the shaped reward/uncertainty; the pretrained world-model parameters and the per-user feedback vectors p_u and \\hat{r} in Eq. (3) are not updated during policy learning. This distinction matters because the claimed contribution over ROLeR/DORL is the dynamic correction of reward inaccuracy, and the implemented mechanism is best described as dynamic reward estimation via reference-user averaging. Please either revise the terminology throughout (including the abstract and related-work comparison) or actually update the world model and then evaluate whether that parameter update is responsible for the gains.","section":"Abstract; §3.4; §5"},{"comment":"The adaptive uncertainty penalty in Eq. (16) has two undeclared technical issues. First, the denominator r^{sel}_s + r^{sel}_d can be negative, zero, or very small because r^{sel}_s is a cosine similarity in its raw range [-1,1], so the penalty can flip sign or blow up; no clipping or additive constant is specified. Second, the numerator uses the difference between consecutive reward estimates \\hat{r} - \\hat{r}_{-1}, but after user selection \\hat{r} is the average over a potentially different reference-user set at each step, so the difference may reflect the changing composition of the selected set rather than the evolving accuracy of the world model. Please clarify what exactly is tracked in \\hat{r}_{-1}, add a stability mechanism for the denominator, and test sensitivity to this choice.","section":"§3.3.2, Eq. (16)"}],"minor_comments":[{"comment":"Notation: Eq. (3) defines \\hat{r} as an average over K world models, while Eq. (15) redefines \\hat{r} as an average over reference users; using the same symbol for two different estimators is confusing. Use a distinct symbol such as \\bar{r} for the reference-user average.","section":"Eqs. (3) and (15)"},{"comment":"The selector state in Eq. (7) concatenates the recommender state with p_u, described as a row of the predicted user-item feedback matrix, but the exact dimensionality and how p_u is constructed from the world-model predictions or embeddings are not stated. Clarify this, since it affects the selector state size and the cosine similarity in Eq. (10).","section":"§3.2.1, Eqs. (7) and (10)"},{"comment":"The hyperparameters K_sel, lambda_s, lambda_d, lambda_U, and lambda_E are tuned over dataset-specific ranges, but no validation-split or model-selection procedure is described. If the ranges were chosen on the test set, the hyperparameter sensitivity analysis does not demonstrate robustness; state how the final values were selected.","section":"§4.1.4"},{"comment":"For the variants DARLR w. r_static and DARLR w. P_U,static, the text says these change the reward function and uncertainty penalty to those of ROLeR, but it is not explicit whether the selector is still used. State that explicitly so the ablation isolates the intended components.","section":"§4.3, Table 4"},{"comment":"There are several typos and formatting artifacts: 'Acknowledegments' in Section 7, 'coeffients' in Section 4.5, and the broken word 'Reco- mmender' in the introduction should be corrected.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the SIGIR audience, and the code release and shared-world-model comparisons are good practice. My main concern for the editor is the mismatch between the claims of 'dynamic world-model update' and the actually implemented reference-user averaging, which, together with the lack of significance tests, currently makes the contribution difficult to evaluate as stated. I do not see evidence of a novelty disclosure problem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper proposes DARLR, a dual-agent offline RL recommender where a selector learns to pick reference users similar and diverse to the current user, and the recommender uses the average of the world model's predicted rewards over those users as a dynamically shaped reward (Eq. 15), plus a dynamic uncertainty penalty. That combination is new relative to ROLeR and DORL, and the paper does several things right: code is released, all model-based baselines share the same world model, there are ablations and hyperparameter sensitivity experiments, and on KuaiRec and Coat the gains over ROLeR are substantial (35.22 vs 33.25, 78.04 vs 76.16). So there is real evidence the mechanism can help.\n\nThe soft spots are real, though. First, the 'world model update' framing is misleading: the world model weights are frozen; what changes is only the reward estimate via averaging. Second, the central mechanism, averaging \\hat{r} over selected users, cannot correct systematic bias in the world model because every prediction comes from the same frozen model. The selector's intrinsic reward (Eq. 12) includes \\hat{r} itself, so the selection policy is partly rewarded for choosing users the world model already rates highly, which can amplify rather than fix bias. The stress-test note about Figure 4 is fair: the 'Error' metric is not defined tightly enough to show the method improves the target user's reward estimate rather than selecting easier-to-predict reference users. Third, the empirical claim of 'superior performance on all four datasets' is too strong: on KuaiRand and Yahoo the gap over ROLeR is within one standard deviation (13.82±1.94 vs 13.46±1.51; 68.54±1.90 vs 68.36±1.86), and no significance tests are reported. Finally, there's no simple non-RL baseline (e.g., user-based collaborative filtering) to show what the RL machinery buys over a standard neighborhood average.\n\nNone of these are fatal. The idea is coherent, the ablations in Table 4 support the importance of the components, and the empirical gains on two datasets are nontrivial. The paper would benefit from sharper claims, significance tests, and a better-defined evaluation of reward accuracy. I think it deserves serious refereeing: a reviewer can push for those fixes without starting from scratch. Who's it for? Researchers working on model-based offline RL for recommenders; it's a useful incremental contribution rather than a breakthrough.","headline":"Plausible incremental advance in model-based offline RL for RecSys, but the dynamic reward mechanism is oversold and two of four benchmark gains are within noise.","tokens_in":21003,"tokens_out":2829,"would_cite":true,"duration_ms":27874,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two-agent reward shaping tops offline RL recommenders on four datasets","keywords":["offline reinforcement learning","recommender systems","dynamic reward shaping","dual-agent framework","uncertainty penalty","reference user selection","model-based RL"],"falsifier":"On any of the four datasets, compare DARLR's reward estimates against ground-truth rewards for a held-out set of users and items, then run a variant where the selector picks reference users uniformly at random. If the random-selection variant achieves the same or lower reward error, or if DARLR's dynamic reward curves do not stay below the static baselines' error curves, the central claim that learned selection drives the improvement is falsified.","tokens_in":1475,"feed_emoji":"🤖","tokens_out":2317,"duration_ms":38697,"temperature":0.7,"pith_summary":"This paper argues that model-based offline reinforcement learning for recommender systems fails not because the world model is inaccurate, but because existing methods freeze the world model's reward function and uncertainty penalty during policy learning, propagating those inaccuracies. The authors propose DARLR, a dual-agent framework in which a selector agent picks reference users by balancing similarity and diversity, and a recommender agent averages the world model's predicted rewards over those users to dynamically reshape the reward signal. They also derive an uncertainty penalty from the selector's own statistics, so conservatism adapts as training proceeds. If correct, this means offline recommenders can approach the performance of policies trained on ground-truth rewards without any online interaction, just by treating the world model as an evolving object rather than a fixed look-up table.","feed_headline":"Two-agent reward shaping tops offline RL recommenders on four datasets","feed_subtitle":"A selector finds similar-but-diverse users; their averaged rewards keep the world model honest during policy training.","key_machinery":"The central mechanism is the dual-agent loop: a selector policy $\\pi_{\\text{sel}}$ chooses reference users by maximizing an intrinsic reward $r^{\\text{sel}} = \\hat{r} + \\lambda_s r_s^{\\text{sel}} + \\lambda_d r_d^{\\text{sel}}$, where $r_s^{\\text{sel}}$ is cosine similarity to the target user and $r_d^{\\text{sel}}$ is average dissimilarity from already selected users. The recommender then uses the averaged predicted reward $\\hat{r}(u, i_t) = \\frac{1}{|\\mathcal{U}_{S,t}|}\\sum_{u' \\in \\mathcal{U}_{S,t}} \\hat{r}(u', i_t)$ as its reward signal, and the uncertainty penalty $P'_U = |\\hat{r} - \\hat{r}_{-1}| / (r_s^{\\text{sel}} + r_d^{\\text{sel}})$ adapts to how representative the selected set is. This machinery converts reward refinement and risk estimation from one-shot static computations into actions taken inside the same reinforcement learning loop that trains the recommender.","core_discovery":"DARLR claims that continuously refining reward estimations during policy learning—rather than treating rewards as static look-up tables—mitigates the impact of inaccuracies in the world model's reward predictions for offline recommender systems. A selector agent sequentially gathers a small set of reference users whose predicted feedback vectors are similar to the target user's yet diverse among themselves; the recommender then replaces the target user's predicted reward with the average reward over that set at each interaction step. The same selection statistics feed a dynamically adjusted uncertainty penalty, so risky actions are penalized more when the selected reference set is not representative. Empirically, DARLR achieves the highest single-step reward and cumulative reward among all compared baselines on KuaiRand, KuaiRec, Coat, and Yahoo, with curves consistently closer to the ground-truth-reward upper bound than static-reward methods.","pith_inferences":["The selector's averaging trick may work even with a systematically biased world model if the bias is roughly additive across users, but if the bias is user-specific and correlated with similarity, the averaging could reinforce rather than cancel the error; this is testable with synthetic bias injection.","The same dual-agent reward-refinement idea could transfer to offline RL outside recommender systems, such as robotics or healthcare, wherever a learned reward model is used as a frozen proxy for a sparse environment.","Constraining the selector to a small user subset (the paper notes clustering can speed it up) suggests a practical path to scaling dynamic reward shaping to millions of users."],"forward_implications":["Offline recommender policies can be trained to near-ground-truth performance without online interaction, as long as rewards are refined during learning rather than frozen.","The dual-agent design suggests a general recipe: any model-based offline RL system can allocate a second policy to actively select which data points to trust when estimating rewards.","Adaptive uncertainty penalties tied to the selection process should be preferred over static ensemble-variance penalties when offline logs are sparse or biased.","On dense datasets like KuaiRec, fewer reference users suffice; on sparse datasets like KuaiRand, more reference users help, indicating the method self-tunes to data sparsity."],"supporting_citations":[{"why":"DORL supplies the entropy penalty and the world-model/uncertainty-penalty formulation that DARLR extends from static to dynamic.","marker":"[17]"},{"why":"ROLeR provides the static reward-shaping baseline and the reference-user clustering idea that DARLR turns into a learned selector.","marker":"[70]"},{"why":"MOPO introduces the ensemble-based uncertainty penalty paradigm that DARLR's dynamic penalty builds on.","marker":"[65]"},{"why":"CIRS demonstrates the practice of averaging multiple world-model reward predictions, which DARLR keeps as its base reward estimate.","marker":"[20]"},{"why":"DeepFM is the supervised reward-prediction model used to learn the world model shared by all compared methods.","marker":"[21]"},{"why":"KuaiRand is one of the two sparse short-video datasets used to demonstrate the dynamic-reward advantage.","marker":"[19]"}],"fun_headline_variants":["Dual-agent RL refines rewards dynamically for offline recommenders","Selector-recommender duo boosts offline RL recommendation","Dynamic reward shaping lifts offline RL recommenders on four benchmarks","Reward refinement beats static models in offline recommender RL","Two-agent reward shaping improves offline RL recommenders"],"cache_read_input_tokens":23040,"weakest_assumption_plain":"The averaged reward over users selected for similarity and diversity is more accurate than the target user's own predicted reward, even though all predictions come from the same error-prone frozen world model; if the world model is systematically biased, the selector may pick similarly biased users and the averaged reward will inherit rather than correct that bias.","fun_headline_variants_meta":{"raw":{"variants":["Dual-agent RL refines rewards dynamically for offline recommenders","Selector-recommender duo boosts offline RL recommendation","Dynamic reward shaping lifts offline RL recommenders on four benchmarks","Reward refinement beats static models in offline recommender RL","Two-agent reward shaping improves offline RL recommenders"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000517,"raw_usage":{"total_tokens":2504,"prompt_tokens":941,"completion_tokens":1563,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":1495}},"tokens_in":557,"tokens_out":1563,"duration_ms":11022,"temperature":1.0,"reasoning_tokens":1495,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:21:12.585349+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On any of the four datasets, compare DARLR's reward estimates against ground-truth rewards for a held-out set of users and items, then run a variant where the selector picks reference users uniformly at random. If the random-selection variant achieves the same or lower reward error, or if DARLR's dynamic reward curves do not stay below the static baselines' error curves, the central claim that learned selection drives the improvement is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DORL supplies the entropy penalty and the world-model/uncertainty-penalty formulation that DARLR extends from static to dynamic."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ROLeR provides the static reward-shaping baseline and the reference-user clustering idea that DARLR turns into a learned selector."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MOPO introduces the ensemble-based uncertainty penalty paradigm that DARLR's dynamic penalty builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CIRS demonstrates the practice of averaging multiple world-model reward predictions, which DARLR keeps as its base reward estimate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DeepFM is the supervised reward-prediction model used to learn the world model shared by all compared methods."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"KuaiRand is one of the two sparse short-video datasets used to demonstrate the dynamic-reward advantage."}],"review_version":1}