{"id":"119d8171-85fa-4fbc-85a2-6c772e8904e2","arxiv_id":"2501.13816","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"LLM-distilled item preferences pre-train an RL recommender, and two online adaptation schemes (fine-tuning and adaptive blending) improve cumulative rewards in simulated online recommendation.","lead":"This paper pre-trains a recommender policy using item preferences generated by a large language model, then adapts it online with user feedback. Its two adaptive variants improve early and long-term recommendation returns over standard online RL baselines in three simulated environments.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed gains are not attributable to LLM-distilled preferences; the paper lacks a non-LLM pretraining control.","rationale":"The reader's CONDITIONAL verdict is appropriate. I agree with the reader's weakest assumption but sharpen it: the unvalidated LLM judge is not merely a risk that the LLM might misalign with real users; it is a missing control for the causal claim. Because frozen iALP is the worst final policy in Table 5 on all three datasets, the LLM labels cannot be shown to encode useful preferences; the improvements of the adaptive variants might stem from generic pretraining or representation learning. A one-line ablation replacing LLM choices with popularity or random labels while keeping everything else identical would settle the attribution. I do not see an internal contradiction that forces rejection, but the central claim is not fully supported as stated. Hence the verdict stays CONDITIONAL, with the ablation as the condition. The absence of released code and error bars further supports the conditional status. The reader's concern about missing human validation is real but secondary; the decisive test is whether the LLM-specific signal matters at all.","tokens_in":14600,"tokens_out":9220,"duration_ms":83989,"concrete_test":"Run Algorithms 1 and 2 on LFM, Industry, and Coat with the LLM component in Eq. (2) replaced by a semantically empty label generator that yields 'None' responses with the same marginal frequency as the LLM but otherwise picks actions uniformly from the candidate list (or from item popularity), keeping all hyperparameters, seeds, and the online alpha schedule fixed. If this control matches A-iALP_ap within run-to-run variance on Table 5, the reported improvements are not caused by LLM-distilled preferences; if it degrades clearly, the preference content is load-bearing. Report the control's learning curves and final returns alongside Table 5.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is causal: LLM-distilled preferences (Section 4.1.1) are what make A-iALP_ap outperform DQN/PG/A2C in Table 5. That requires the LLM's item choices to be informative for the reward models used in LFM/Industry/Coat. The paper never checks this. Section 5.4 says the LoRA tuning on 1000 samples only learns the response format, and no comparison of LLM judgments against held-out user interactions or against the reward model is reported. The paper's own numbers make the attribution doubtful: the frozen iALP policy is worse than every online baseline in Table 5 (LFM 11.2 vs A2C 28.1; Industry 25.3 vs 46.3; Coat 31.2 vs 81.7), so the LLM-derived policy itself does not behave like a good user-preference policy in these environments. The eventual A-iALP gains could instead come from the pretrained state encoder/action prior or the alpha-blending schedule, not from the semantic content of LLM preferences. Without ablating the label source, the abstract's 'substantial performance improvements' conflates 'pretraining helps' with 'LLM preference distillation helps.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes iALP, a policy for recommender systems pretrained offline using rewards and actions distilled from an LLM prompted with user interaction histories and item attributes. It then introduces two online adaptation strategies: A-iALP_ft, which fine-tunes the pretrained policy with simulated online feedback, and A-iALP_ap, which blends a frozen pretrained policy with a learnable online policy using a time-varying weight alpha. Experiments on three simulated environments (LFM, Industry, Coat) compare against DQN, PG, A2C, and frozen iALP in terms of Return, Length, and Average Reward, with claims of substantial improvements in initial and long-term performance.","tokens_in":14823,"tokens_out":3773,"duration_ms":34452,"significance":"The core idea of using LLM judgments as an offline pretraining signal to improve cold-start and long-term recommendation is practically relevant and the adaptive alpha-blending mechanism is a sensible way to transition from LLM-derived behavior to environment-driven behavior. The paper gives a clear formulation of the pretraining and online phases, and the three-environment evaluation is broader than many prior works. However, the empirical support for the central causal claim is incomplete: the LLM preference signal is never validated against real user behavior or the reward models, there is no non-LLM pretraining control, and the reported margins lack error bars or significance tests. If these gaps are addressed, the method could be a useful contribution; as it stands, the evidence is suggestive rather than conclusive.","major_comments":[{"comment":"The central claim that LLM-distilled preferences cause the observed improvements is not supported because there is no ablation isolating the label source. The pretraining phase in Eqs. (2)-(3) and (6)-(10) uses LLM choices as both action labels and rewards, and Section 5.4 states that the LoRA fine-tuning on 1000 samples only learns the response format, not user preferences. The paper never checks whether LLM judgments agree with held-out user interactions or with the reward models used for evaluation. The authors' own Table 5 makes the attribution doubtful: frozen iALP is worse than every online baseline in all three environments (e.g., LFM 11.2 vs A2C 28.1; Industry 25.3 vs 46.3; Coat 31.2 vs 81.7). The gains of A-iALP_ap could therefore come from the pretrained state encoder, the pretrained initialization of the online policy, or the alpha-blending schedule, rather than from the semantic content of LLM preferences. A control with a non-LLM pretraining signal (e.g., random labels, popularity-based labels, or reward-model labels) is needed to attribute the improvement to the LLM.","section":"§4.2.1, §5.1, Table 5"},{"comment":"The evaluation is entirely against simulated reward models trained on the same public datasets used to build the recommendation scenarios. No calibration or agreement analysis is reported between these reward models and actual user behavior, and no validation of the LLM's preference judgments against the reward model is provided. Consequently, the reported Returns are internal consistency checks within a simulator, not measures of real-world user satisfaction. In addition, Table 5 reports single numbers without variance across seeds, confidence intervals, or significance tests; margins such as 33.1 vs 28.1 on LFM or 51.8 vs 46.3 on Industry could be within run-to-run noise. At minimum, multiple random seeds with standard deviations and a significance test are needed for the headline claims.","section":"§5.2, Table 3"},{"comment":"The RQ1 comparison at epoch 0 is not a meaningful competition: iALP is a fully pretrained policy, while DQN, PG, and A2C are randomly initialized at epoch 0. The statement that iALP 'significantly outperforms' these baselines is true by construction and is not accompanied by any statistical test. A more informative comparison would report the number of online steps required for each baseline to reach the initial return of iALP, or would compare all methods from the same initialization schedule. The current framing overstates the contribution of the LLM signal.","section":"§4.2.3, Eq. (14), Algorithm 2"},{"comment":"The adaptive scheme A-iALP_ap depends on a weight alpha that is stated to increase to 1 as training proceeds, but the exact schedule is never specified. Section 4.2.3 says only that the initial value depends on the scenario, and Algorithm 2 does not list alpha as a tunable parameter or provide its values for LFM, Industry, or Coat. Since alpha controls the relative contribution of the frozen pretrained policy versus the learnable policy, the reported results are not reproducible without this information, and the sensitivity of the method to the alpha schedule is unknown.","section":"§4.2.3, Eq. (14), Algorithm 2"}],"minor_comments":[{"comment":"The policy parameter is denoted as psi in Eq. (1) and as theta in Section 4.1.2; the notation should be made consistent throughout.","section":"§1, §2.1"},{"comment":"There are several typos and inconsistent abbreviations: 'A-iALT_ap' appears in the introduction, 'iAPT' appears in the contributions, and 'A-iALP_ap' is sometimes written as 'A-iALP_ap'. These should be corrected for clarity.","section":"§4.2.1, §5.1"},{"comment":"The description of the reward models is incomplete: for Coat, the text says 'Matrix Factorization (DeepFM)', which conflates two distinct model families, and for LFM/Industry the reward model is said to follow 'a sequential recommender method' from [24] without specifying the architecture, training objective, or hyperparameters.","section":"§5.4, §6.3"},{"comment":"The LLMOnline baseline used for RQ3 is not fully specified: it is unclear whether it uses the same Mistral 7B model, the same LoRA format tuning, the same prompt template, and the same candidate sampling procedure as iALP. Without this information, the comparison in Figures 8 and 9 is difficult to interpret.","section":"§6.4"},{"comment":"The exploration-strategy experiments are reported only for LFM and Industry (Figures 10 and 11); no results are given for Coat, despite the paper claiming 'three simulated environments'. The absence of Coat in RQ1, RQ2, and RQ4 should be acknowledged or addressed.","section":"§6.4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Main takeaway: the paper has a sensible warm-start recipe, but the headline claim that LLM-distilled preferences drive the gains is not actually pinned down. What's new is the combination: using an LLM as a judge to generate both actions and rewards for offline A2C pretraining, then adapting online either by fine-tuning (A-iALP_ft) or by blending the frozen pretrained policy with a learnable policy (A-iALP_ap). The blending scheme is a practical idea, and the comparison against directly using the LLM online (RQ3) is a useful sanity check. The writing is clear and the algorithms are specified well enough to reimplement.\n\nThe soft spots are concentrated in the evaluation. The biggest one is attribution: there is no ablation that replaces the LLM labels with a non-LLM signal before the same online adaptation. Given that frozen iALP underperforms all the trained baselines in Table 5 (LFM 11.2 vs A2C 28.1), the LLM's preferences are not obviously aligned with the simulated rewards. So the gains of A-iALP_ap could come from the warm-started state encoder, the alpha-blending schedule, or simply from having any initial policy, rather than from the semantic content of the LLM's choices. A control with random or supervised pretraining labels would settle this.\n\nSecond, there are no error bars or significance tests. The margins in Table 5, e.g., 33.1 vs 28.1, could be within run-to-run noise. Third, the alpha schedule is left unspecified beyond 'increases to 1', which leaves the main adaptive mechanism under-determined. Fourth, Coat is missing from the epoch-0 table, which is exactly where iALP's advantage is weakest (in Table 5, iALP gets 31.2 on Coat vs 81.7 for A2C). That looks like selective reporting.\n\nNone of this is fatal. The idea is sound and the results are consistently in favor of the adaptive variants. What's missing is isolating the causal contribution of the LLM. I'd send this to review. It needs major revisions: a non-LLM pretraining baseline, statistical tests, the alpha schedule, and a full reporting of all datasets at epoch 0. Then it could be a solid contribution for the recommender-systems RL crowd.","headline":"A sensible warm-start recipe for RL recommenders, but the paper doesn't isolate whether the LLM label source is what makes it work.","tokens_in":15387,"tokens_out":4281,"would_cite":true,"duration_ms":35324,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using an LLM's item preferences to pre-train a recommender policy improves both cold-start quality and long-term returns once the policy is adapted online.","keywords":["recommender systems","reinforcement learning","large language models","policy pre-training","online adaptation","cold-start","exploration","actor-critic"],"falsifier":"Measure agreement between the LLM's top-1 item choice and the actual next item in a held-out slice of the LFM or Industry interaction logs; if agreement is at or below random-item chance, then the reward signal in Equation (3) carries no user-preference information and the claimed pre-training advantage should disappear.","tokens_in":14389,"feed_emoji":"🤖","tokens_out":6472,"duration_ms":52674,"temperature":0.7,"pith_summary":"This paper seeks to establish that a large language model can stand in for human users during the risky early phase of an online recommender, and that policies pre-trained on that stand-in do better than policies trained from scratch once they go live. The authors prompt an LLM with a user's interaction history and ten candidate items, ask it to choose the next item, and convert each choice into a reward of 1 or 0. Those signals train an actor-critic recommendation policy offline, producing a policy that yields much better initial recommendations than randomly initialized online RL baselines. They then ship the policy online in two ways: fine-tuning it on real feedback, or blending it with a fresh learnable policy whose influence grows over time. Experiments across three simulated environments report that the adaptive blend achieves the highest long-term cumulative returns while converging faster and more stably than A2C, DQN, or policy-gradient baselines.","feed_headline":"LLM tastes pre-train recommender policies for better returns","feed_subtitle":"Pretraining on LLM preferences lets online recommenders skip the cold-start stumble and converge faster.","key_machinery":"The load-bearing mechanism is LLM-based preference distillation used as an offline reward and action generator. Given a user state encoded by SASRec, the actor samples k candidate actions, the LLM is prompted to pick one or 'None', the picked action receives reward 1, and this reward feeds the actor-critic update. The second key mechanism is the adaptive online blend: A-iALP_ap acts according to (1-alpha)pi_theta + alpha pi_beta, with the pretrained policy pi_theta frozen and the learnable policy pi_beta eventually taking over; this preserves pretrained behavior early while allowing the agent to escape any bad preferences the LLM encoded.","core_discovery":"The paper's central discovery is that preference signals distilled from an LLM are a viable offline pre-training signal for RL-based recommendation, and that the resulting policy can be transferred online without the usual cold-start penalty. Concretely, the authors construct prompts that ask the LLM to choose among ten candidate items given the user's history; a chosen item gets reward 1, 'None' gets 0, and these state-action-reward triples train an A2C actor-critic whose state encoder is SASRec. The resulting iALP policy produces markedly better initial sequences than randomly initialized DQN, PG, or A2C. To move online, A-iALP_ft fine-tunes the same network on simulated user rewards, while A-iALP_ap freezes the pretrained policy and learns a new policy alongside it, mixing the two with a weight alpha that shifts from 0 to 1 as training progresses. Across LFM, Industry, and Coat, A-iALP_ap reports the highest cumulative returns, and both adaptive variants converge faster and more stably than training from scratch.","pith_inferences":["If LLM preferences are good enough, this recipe could extend to other interactive domains with textual item attributes but no obvious reward model, such as dialogue or tutoring systems.","The paper fine-tunes the LLM only on response format, not on user preferences; a testable extension is to fine-tune on real feedback when available, which might fix the failure mode visible in the Coat environment where iALP alone underperforms from-scratch A2C.","Because the simulated environments share data with the reward models used for evaluation, a harder test is to evaluate against held-out user behavior or a different platform's reward model; the paper does not show how the gains transfer.","The alpha schedule of A-iALP_ap is scenario-dependent; making alpha adaptive to the online policy's measured performance, rather than a fixed time schedule, is a natural next step."],"forward_implications":["LLM-pretrained policies can be dropped into an online recommender to cut the initial poor-recommendation phase, addressing a main reason online RL recommenders are rarely deployed.","Fine-tuning the LLM-pretrained policy on real feedback already improves over offline training alone, so online adaptation is complementary to LLM distillation.","The adaptive blend scheme avoids catastrophic forgetting of pretrained behavior when the pretrained policy's preferences mismatch the real environment.","A-iALP outperforms directly querying the LLM online, since the learned policy continues improving while the LLM's choices stay static and expensive.","Under all tested exploration strategies, the pretrained start gives faster convergence and higher stable returns than A2C."],"supporting_citations":[{"why":"Provides the SASRec state encoder that maps interaction histories to states.","marker":"[15]"},{"why":"Defines the actor-critic (A2C) framework that iALP is built on.","marker":"[16]"},{"why":"Provides the method for constructing simulated online environments and reward models from public datasets.","marker":"[24]"},{"why":"Supplies the simulated-environment setting and a comparable LLM-based planner baseline.","marker":"[36]"},{"why":"Motivates using LLM judgments as reward signals for RL.","marker":"[19]"},{"why":"Enables low-rank fine-tuning of the LLM to follow the prompt format.","marker":"[13]"},{"why":"Is the base LLM (Mistral 7B) that generates preferences.","marker":"[14]"},{"why":"LFM music dataset used to build one simulated environment.","marker":"[33]"},{"why":"Amazon Industrial and Scientific category dataset used for the Industry environment.","marker":"[27]"},{"why":"Coat dataset used for the third simulated environment.","marker":"[34]"}],"fun_headline_variants":["LLM preferences jump-start recommender policies","Pretrain recommender policies with LLM tastes","LLM-distilled rewards boost online RL recommenders","Adaptive LLM pretraining beats from-scratch RS","LLM pretraining cuts recommender cold start"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an LLM prompted with item attributes and a user's history picks the items the user would actually pick, so its 1/0 choices can serve as a trustworthy pre-training reward.","fun_headline_variants_meta":{"raw":{"variants":["LLM preferences jump-start recommender policies","Pretrain recommender policies with LLM tastes","LLM-distilled rewards boost online RL recommenders","Adaptive LLM pretraining beats from-scratch RS","LLM pretraining cuts recommender cold start"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1311,"prompt_tokens":1032,"completion_tokens":279,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":205}},"tokens_in":648,"tokens_out":279,"duration_ms":3123,"temperature":1.0,"reasoning_tokens":205,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:34:34.754049+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure agreement between the LLM's top-1 item choice and the actual next item in a held-out slice of the LFM or Industry interaction logs; if agreement is at or below random-item chance, then the reward signal in Equation (3) carries no user-preference information and the claimed pre-training advantage should disappear.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the simulated-environment setting and a comparable LLM-based planner baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates using LLM judgments as reward signals for RL."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LFM music dataset used to build one simulated environment."}],"review_version":1}