{"id":"c3d230cf-f108-42c6-b81e-ee863bb002af","arxiv_id":"2501.13344","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ReLLaX combines semantic behavior retrieval, collaborative soft prompts, and a new fully interactive LoRA variant to improve LLM-based CTR prediction on long user histories.","lead":"This paper introduces ReLLaX, a three-part method that helps large language models extract useful information from long user purchase or watch histories to predict whether a user will like a new item. It combines smarter retrieval of past interactions, soft prompt tokens from a classic recommender, and a flexible low-rank adapter, and reports better click-through-rate predictions than earlier methods on three public datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The empirical case for CFLoRA's full interaction matrix is confounded: no ablation replaces full W with diagonal W while holding SUBR and SPA fixed, so the central 'degraded versions' claim lacks direct support.","rationale":"The reader's weakest_assumption focuses on whether the CRM representation h is informative enough to generate a generalizing W, especially on sparse data such as BookCrossing. That is a legitimate robustness concern, but the paper's own BookCrossing results (with held-out users) suggest h carries at least some signal. The more decisive gap is that the paper's central theoretical claim -- that full W interaction is what makes CFLoRA better than existing LoRA variants -- is never directly tested. The comparison against iLoRA is confounded by SUBR and SPA, and the w/o-CFLoRA ablation removes LoRA entirely rather than constraining W to diagonal. If full W offers no gain over diagonal W, then the 'degraded versions' framing, while mathematically true, is not practically meaningful, and the paper's parameter-level contribution is substantially weakened. This is an addressable omission, not a refutation: the algebra in Section 3.4 is correct, and the framework is plausible. The reader's CONDITIONAL verdict already reflects the need for stronger empirical evidence; my concern adds a specific missing experiment. I therefore recommend no change to the verdict, but the condition should include the diagonal-W ablation as a required check.","tokens_in":31852,"tokens_out":13854,"duration_ms":123761,"concrete_test":"Run an ablation of ReLLaX in which the CFLoRA interaction matrix W of Equation 21 is constrained to be diagonal (only w_ii learned, off-diagonal entries fixed to 0), generated by the same projector from the same CRM representation h, while keeping SUBR, SPA, the frozen CRM, LoRA rank, and all training hyperparameters identical. Compare this 'ReLLaX-diagW' against ReLLaX with full W on BookCrossing, MovieLens-1M, and MovieLens-25M under the Table 3 few-shot settings, with at least 5 random seeds and paired significance tests. If ReLLaX-diagW matches ReLLaX within 0.001 AUC or the difference is not statistically significant, then the paper's claim that full atom-component interaction is responsible for CFLoRA's superiority over prior 'degraded' methods is not supported by the current experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4.2 claims existing LLM4Rec LoRA methods (TALLRec, CoLLM, iLoRA, ReLLa) are 'degraded versions' of CFLoRA because they constrain the interaction matrix W to identity or (block) diagonal, while CFLoRA allows an arbitrary full W (Equation 18). This containment is mathematically correct, but the paper also implies that the full W is the reason for CFLoRA's superior performance. The experiments do not isolate this. Table 3 compares ReLLaX (which includes SUBR and SPA) against iLoRA (which has neither), so any observed gain could come from the data-level and prompt-level components. The ablation 'ReLLaX (w/o CFLoRA)' in Table 4 removes CFLoRA entirely, falling back to vanilla LoRA with W = I, not to a per-sample diagonal W (which is exactly iLoRA's constraint). A diagonal W still provides per-sample adaptation, so if it performs as well as the full W, then the 'full interaction' is not the source of the improvement. In that case the theoretical framing is a containment result without practical bite, and the parameter-level novelty reduces to per-sample LoRA plus SUBR/SPA. This is a concrete, testable gap in the central argument.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formulates the 'lifelong sequential behavior incomprehension' problem for LLM-based CTR prediction, in which LLMs fail to exploit long textual user behavior sequences even within their context window. It proposes ReLLaX, a three-level framework: SUBR (semantic retrieval of user behaviors), SPA (soft prompts derived from a conventional recommendation model), and CFLoRA (LoRA with a per-sample full interaction matrix W between down/up projection components). The authors show algebraically that existing LoRA-based LLM4Rec methods correspond to constrained versions of CFLoRA (W=I or block-diagonal W), and report experiments on BookCrossing, MovieLens-1M, and MovieLens-25M where ReLLaX outperforms ID-based and LM-based baselines.","tokens_in":32175,"tokens_out":5951,"duration_ms":50891,"significance":"If the empirical claims hold, the paper makes two contributions: a reusable diagnosis of why long textual sequences hurt LLM recommenders, and a unified view of LoRA variants in LLM4Rec. The theoretical containment result (Eqs. 13-20) is clean and correct: identity, block-diagonal, and full W form a clear hierarchy, and the 'degraded versions' framing is a useful organizing perspective. The framework is accompanied by code and evaluated on three public datasets. However, the practical significance of the full-interaction W is not yet established: the key ablation replaces CFLoRA with vanilla LoRA (W=I) rather than with a per-sample diagonal W, so the empirical advantage of the full matrix over intermediate constraints is untested. The lack of error bars and the imported ReLLa baseline further weaken the state-of-the-art claim as currently presented.","major_comments":[{"comment":"The central claim that the full interaction matrix W is the source of CFLoRA's gains is not directly tested. The ablation 'ReLLaX (w/o CFLoRA)' falls back to vanilla LoRA, i.e. W=I; it does not replace W with a per-sample diagonal (or block-diagonal) matrix, which is exactly the iLoRA/RecLoRA constraint. Thus the comparison cannot separate the effect of full W from the effect of per-sample adaptation. Please add an ablation that keeps SUBR and SPA fixed and compares full W against diagonal W (and, if feasible, block-diagonal W) generated by the same projector. If a diagonal W matches full W, the 'degraded versions' claim is mathematically clean but has no demonstrated performance consequence.","section":"3.4.2 / Table 4"},{"comment":"The ReLLa baseline results are imported verbatim from the prior conference paper, while ReLLaX and the other baselines are executed in the present setup. Because ReLLa is one of the two strongest LM baselines in Table 3 (e.g., 0.8033 vs 0.8091 AUC on MovieLens-1M), a difference in hardware, library versions, prompt formatting, or data preprocessing can bias the comparison in either direction. Please re-run ReLLa under the same code and environment as ReLLaX and report both sets of numbers.","section":"4.2 / Table 3 / Appendix B.2"},{"comment":"No significance-test procedure is described. The table reports '*' for p<0.001, but does not state which test was used, whether the comparison is paired, or how many seeds/runs are aggregated. Given the small margins at stake (e.g., 0.0005 AUC between some Table 4 variants), please report means and standard deviations over at least 3 random seeds for the main table and ablations, and specify the test used for the asterisks.","section":"4.2 / Table 3"},{"comment":"On BookCrossing the CRM is trained on only 17,714 samples but must produce per-sample interaction matrices W for a dataset with 278,858 users. The paper does not analyze how CFLoRA behaves for cold users or when the CRM's final representation h is unreliable. Please report performance broken down by user history length or include a cold-start experiment; at minimum, discuss this limitation explicitly.","section":"3.4.3 / Eq. (21) / Table 2"}],"minor_comments":[{"comment":"The sentence 'Each user u has a has a chronological interaction sequence' contains a duplicated phrase, and 'the objective of is to predict' is missing a word.","section":"2.1"},{"comment":"The notation alpha_{j//N} is not defined; clarify how the block index maps to the r1-dimension blocks after merging N LoRA sets.","section":"3.4.1 / Eq. (16)"},{"comment":"Figure 1 uses two different y-axes (left for SIM, right for the LLMs) without explicit axis labels in the caption; the reader cannot immediately tell the scales.","section":"Figure 1"},{"comment":"The text says ReLLa (few-shot) achieves continuous improvement as K increases; consider stating explicitly the K ranges for which this holds, since the curves in Figure 6 are not trivially monotonic in all panels.","section":"4.4.1"},{"comment":"The statement that ReLLa results are taken directly from the original paper should be moved to or repeated in the main text as a limitation of the comparison.","section":"Appendix B.2"},{"comment":"There are several typos ('comparision', 'sequnce', 'Ragnaro', 'but also but also'); a careful proofread is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a substantial extension of the authors' own WWW'24 ReLLa paper, and the novelty disclosure is transparent. The main concern is not the mathematical containment result but whether the full-W design is empirically load-bearing; the missing diagonal-W ablation should be the focus of the revision. Re-running ReLLa in-house would remove a potentially unfair baseline comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: worth reading and worth refereeing, but the headline claim is not actually tested. What is genuinely new is per-sample dense W in CFLoRA, written as ΔΘ = B W A, with W generated from a CRM's final representation h. The containment algebra in Section 3.4 is correct: vanilla LoRA is W = I, iLoRA/RecLoRA are block-diagonal W, so those methods are special cases. That is bookkeeping rather than a theory of performance, and the abstract overstates it a bit by calling it a theoretical demonstration. Still, it is useful bookkeeping: it gives a unified matrix form for LoRA-based LLM4Rec and a concrete new design.\n\nThe experiments support the overall framework: ReLLaX beats ReLLa, iLoRA, and CoLLM on three datasets, with AUC gains around 0.006 on MovieLens-1M. The ablation shows each module helps. Good. But the central attribution is under-supported. The w/o CFLoRA variant uses vanilla LoRA (W = I); a diagonal per-sample W (the iLoRA constraint) is never tested while holding SUBR and SPA fixed. So the gain could come from per-sample adaptation, not from full off-diagonal interaction. This is the single most important gap, and it is fixable with one run. Also missing: error bars, seeds, and description of the significance test despite the p < 0.001 stars; ReLLa baseline numbers are imported from the prior paper rather than rerun; no cold-user or low-data failure analysis, which matters on BookCrossing with only 17k training samples. These are addressable, not fatal.\n\nWho this is for: anyone working on LLM4Rec or parameter-efficient adaptation of LLMs for recommender systems. I would bring it to a reading group, and the discussion will center on the missing ablation. I would cite it for the unified BWA formulation if I were doing LLM4Rec work. Recommendation: yes, send to peer review, with a referee who will ask for the full-W versus diagonal-W ablation and a rerun of the baselines. Do not desk-reject.","headline":"Solid, niche-important paper with correct containment algebra and a plausible new LoRA design, but the claim that the full interaction matrix drives the gains is not isolated by the ablations.","tokens_in":693,"tokens_out":669,"would_cite":true,"duration_ms":29236,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReLLaX claims LLMs fail to comprehend long user behavior sequences, and fixes it with a three-level framework: semantic retrieval, soft prompt augmentation, and a fully interactive LoRA variant.","keywords":["lifelong sequential behavior incomprehension","LLM4Rec","CTR prediction","semantic user behavior retrieval","soft prompt augmentation","component fully-interactive LoRA","low-rank adaptation","collaborative knowledge injection"],"falsifier":"Run a matched comparison on a sparse dataset such as BookCrossing: train ReLLaX once with $W$ generated by the conventional recommender and once with $W$ fixed to the identity matrix (vanilla LoRA), keeping all other settings identical. If the AUC gap does not persist across random seeds and especially on cold-user subsets, the claim that full atom-component interaction drives the gain collapses. A second check is to shuffle the conventional recommender's embeddings before generating $W$; if ReLLaX's performance does not drop, the $W$ generator is not actually using collaborative signal.","tokens_in":31686,"feed_emoji":"🎯","tokens_out":4405,"duration_ms":54609,"temperature":0.7,"pith_summary":"This paper claims that large language models lose the ability to extract useful signal from user behavior sequences once those sequences grow long, even when the token count stays well below the context window. To fix this, the authors propose ReLLaX, a framework that intervenes at three levels: data, prompt, and parameters. At the data level, semantic user behavior retrieval (SUBR) replaces the most recent behaviors with the most semantically relevant ones toward the target item. At the prompt level, soft prompt augmentation (SPA) injects collaborative knowledge from a conventional recommender into the LLM's token embeddings. At the parameter level, component fully-interactive LoRA (CFLoRA) lets all atom components of the LoRA matrices interact, going beyond the constrained interaction patterns of prior LoRA-based LLM4Rec methods. If the claims hold, ReLLaX offers both a practical recipe for long-history CTR prediction and a unified theoretical view of how LoRA is used across the LLM4Rec literature.","feed_headline":"Three-part fix helps LLMs read long user histories","feed_subtitle":"Semantic retrieval, soft prompts, and a fuller LoRA lift CTR prediction on MovieLens and BookCrossing datasets.","key_machinery":"The central object is the LoRA interaction matrix $W \\in \\mathbb{R}^{r \\times r}$ in the composite form $\\Delta\\Theta = B W A$. In the decomposed view, $\\Delta\\Theta = \\sum_{i,j} w_{ij} B_i A_j$, each pair of rank-one atom components interacts with weight $w_{ij}$; vanilla LoRA fixes $W = I$, personalized LoRA methods fix $W$ as block-diagonal with per-block scalars, and CFLoRA leaves $W$ fully general. $W$ is generated per data sample by a two-layer MLP that projects the conventional recommender's final representation $h$ (from Equation 21), which is how the method injects collaborative, lifelong-sequence information into the otherwise frozen LLM parameters. This object carries both the expressiveness argument and the theoretical unification of prior LLM4Rec LoRA uses.","core_discovery":"The paper identifies a specific failure mode, lifelong sequential behavior incomprehension, in which LLM-based pointwise CTR scorers peak at short histories (around 15 items for MovieLens-1M) and then decline as the sequence lengthens, despite the context staying within the model's window. The proposed fix, ReLLaX, combines three mechanisms: SUBR homogenizes the behavior sequence by retrieving the most semantically relevant items for the target; SPA appends soft prompt tokens derived from a pretrained conventional recommender's item embeddings, aligning the language space with collaborative signal; and CFLoRA extends LoRA by inserting a per-sample interaction matrix $W$ between the down- and up-projections, so the update becomes $\\Delta\\Theta = B W A$. The paper further argues theoretically that existing LoRA-based LLM4Rec methods are degraded versions of CFLoRA: vanilla LoRA uses $W = I$, and personalized variants like iLoRA use a block-diagonal $W$, whereas CFLoRA leaves $W$ fully general and learns it from the conventional recommender's final representation. Experiments on BookCrossing, MovieLens-1M, and MovieLens-25M show ReLLaX achieving the best AUC among all baselines, including ReLLa, iLoRA, and CoLLM, and the ablation study attributes the gain to all three components working together.","pith_inferences":["The paper's unification claim suggests a practical benchmark recipe: report every LoRA variant by its constraint on $W$ (identity, block-diagonal, or full). This would make future LLM4Rec comparisons more structural than the paper itself demonstrates.","The heterogeneity metric used to justify SUBR (number of unique movie genres) is only a proxy; a direct measure of token-level attention or perplexity over retrieved versus recent sequences would make the comprehension benefit more inspectable.","Because the LLM baselines are trained with only a few thousand samples while the conventional recommenders use the full training set, part of ReLLaX's margin may reflect sample efficiency rather than comprehension per se; a full-shot LLM comparison would test that interpretation.","If CFLoRA's gains come from full interaction, a natural extension is to learn $W$ end-to-end without a conventional recommender, which would test whether the collaborative signal is necessary or merely a convenient generator."],"forward_implications":["If ReLLaX is correct, LLMs can be made to keep improving as user history grows, matching conventional models like SIM instead of peaking and degrading.","The CFLoRA formulation gives a common language for LoRA-based LLM4Rec: every existing method becomes a specific constraint on the interaction matrix $W$, which makes comparing methods a matter of comparing matrix structures.","SPA shows that injecting collaborative embeddings as soft tokens can help an LLM reason about item relationships, pointing to a general design where conventional recommenders supply side information to language models.","The framework still operates under few-shot instruction tuning, so the reported gains are achievable with relatively little training data while beating full-shot ID-based baselines."],"supporting_citations":[{"why":"The conference version ReLLa supplies the SUBR retrieval mechanism and the few-shot instruction-tuning setup that ReLLaX extends.","marker":"[46]"},{"why":"iLoRA is the instance-wise personalized LoRA baseline whose block-diagonal interaction structure CFLoRA generalizes.","marker":"[34]"},{"why":"CoLLM injects collaborative embeddings as soft prompts, serving as the direct comparison for SPA.","marker":"[96]"},{"why":"TALLRec establishes the pointwise scoring and LoRA instruction-tuning protocol that the paper follows for LLM-based CTR prediction.","marker":"[2]"},{"why":"Lifelong personalized LoRA is another personalized-LoRA method with constrained atom interactions that CFLoRA subsumes.","marker":"[100]"},{"why":"LoRA defines the low-rank adaptation form $\\Delta\\Theta = BA$ that CFLoRA rewrites as $BWA$.","marker":"[28]"},{"why":"SIM provides the conventional search-based user interest modeling baseline whose steady improvement with sequence length motivates the target behavior.","marker":"[60]"}],"fun_headline_variants":["LLMs get lost in long user histories; ReLLaX fixes it","ReLLaX: A full-stack fix for LLM long-sequence comprehension","Interactive LoRA helps LLMs understand long user histories","One framework to fix LLM short-sightedness in recommendations","CFLoRA: Interactive LoRA for lifelong sequential behavior in LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole CFLoRA argument rests on the assumption that the conventional recommender's final representation $h$, trained with plain binary cross-entropy, is informative enough to generate a per-sample interaction matrix $W$ that generalizes to users and items the recommender has never seen.","fun_headline_variants_meta":{"raw":{"variants":["LLMs get lost in long user histories; ReLLaX fixes it","ReLLaX: A full-stack fix for LLM long-sequence comprehension","Interactive LoRA helps LLMs understand long user histories","One framework to fix LLM short-sightedness in recommendations","CFLoRA: Interactive LoRA for lifelong sequential behavior in LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001064,"raw_usage":{"total_tokens":4529,"prompt_tokens":1081,"completion_tokens":3448,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":697,"completion_tokens_details":{"reasoning_tokens":3355}},"tokens_in":697,"tokens_out":3448,"duration_ms":26487,"temperature":1.0,"reasoning_tokens":3355,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:14:44.644980+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a matched comparison on a sparse dataset such as BookCrossing: train ReLLaX once with $W$ generated by the conventional recommender and once with $W$ fixed to the identity matrix (vanilla LoRA), keeping all other settings identical. If the AUC gap does not persist across random seeds and especially on cold-user subsets, the claim that full atom-component interaction drives the gain collapses. A second check is to shuffle the conventional recommender's embeddings before generating $W$; if ReLLaX's performance does not drop, the $W$ generator is not actually using collaborative signal.","supporting_citations":[],"review_version":1}