{"id":"a245f0ad-b987-4f88-959c-e81abc4d9956","arxiv_id":"2506.21579","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM2Rec combines next-item prediction fine-tuning with masked token reconstruction and contrastive learning to produce item embeddings that outperform existing text-embedding baselines for sequential recommendation.","lead":"This paper presents LLM2Rec, a two-stage method that fine-tunes a large language model to predict next purchases and then converts it into an item-embedding model. It reports consistent gains over existing text-embedding baselines on in-domain and out-of-domain recommendation datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CSFT's claimed contribution of CF signals to item embeddings is untested: the ablation cannot separate collaborative structure from corpus/vocabulary adaptation, and no check shows CF information survives the transition to bidirectional mean-pooled embeddings.","rationale":"The paper's headline distinction over LLMEmb and LLM2Vec is the explicit integration of CF signals. If the CSFT gain is really vocabulary or domain adaptation, the method reduces to a supervised next-item fine-tune followed by LLM2Vec-style encoder adaptation, and the central conceptual claim is lost. The shuffled-sequence control isolates the collaborative information channel: randomizing sequence order removes exactly the multi-hop co-occurrence structure that CF signals are supposed to capture, while leaving the input text distribution nearly unchanged. This is the minimal intervention that tests the mechanism. The same control also addresses the OOD tension: because unseen items have no co-occurrence history, a large OOD gain under shuffled CSFT would suggest the model transfers category-level regularities rather than item-level CF. I agree with the reader that the mechanism is the weakest assumption; the missing ID-based recommenders and significance tests are additional concerns but do not change the conditional verdict. If the shuffled control shows no difference, the paper should be revised to weaken the CF claim or add direct embedding-geometry evidence; if it shows a clear drop, the current conditional acceptance is appropriate. I therefore keep the reader's CONDITIONAL verdict unchanged.","tokens_in":19712,"tokens_out":7512,"duration_ms":80367,"concrete_test":"Train the full pipeline identically, but during CSFT randomly permute the items in each user sequence before constructing the input, destroying co-occurrence while preserving item frequencies and the marginal next-item distribution. Evaluate this shuffled-CSFT variant with the same IEM stage using SASRec on Games and Sports under the paper's protocol, and compare with full CSFT and the no-CSFT bidirectional baseline. If shuffled-CSFT matches full CSFT within seed noise, the gains attributed to CF are not due to collaborative structure; if it is clearly worse, the CF mechanism is supported. Report per-seed standard deviations alongside the averages.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central novelty is that Collaborative Supervised Fine-Tuning (CSFT, Sec. 3.2) injects collaborative filtering (CF) signals into item hidden states and that these survive Item-level Embedding Modeling (IEM, Sec. 3.3). This is not directly shown. Figure 1 is illustrative; the Sec. 4.3 ablation only reports downstream accuracy, a confounded proxy. CSFT is a 10,000-step next-item-title generation fine-tune on 1.03M interactions; its gains could reflect item-title vocabulary adaptation, category-transition memorization, or popularity effects rather than CF structure in single-item embeddings. The 'Bidirectional' vs 'CSFT' comparison in Table 4 lacks a control that destroys co-occurrence while preserving token distributions; without it, the CF interpretation is unsupported. A second gap is out-of-domain transfer: Sports/Baby/Goodreads items have no interaction history in pre-training, so no CF signal can be encoded for them; the small CSFT gain on Sports R@10 (0.1079 to 0.1119) is consistent with transferable semantic/category regularities rather than item-level CF. Also, IEM consists of masked single-title prediction and SimCSE-style contrastive learning; nothing in Eq. 3 or Eq. 4 ties the resulting mean-pooled embeddings back to sequence co-occurrence, so the persistence of CF information is assumed, not demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces LLM2Rec, a two-stage framework for turning a decoder-only LLM into an item embedding model for sequential recommendation. In Stage 1 (Collaborative Supervised Fine-Tuning, CSFT), the LLM is fully fine-tuned for 10,000 steps on 1.03M user interactions from six Amazon categories to predict the next item title given the titles of previous interactions. In Stage 2 (Item-level Embedding Modeling, IEM), the model is converted to bidirectional attention, trained with masked next-token prediction and item-level SimCSE-style contrastive learning, and item embeddings are obtained by mean pooling. These embeddings are passed through a linear adapter into GRU4Rec and SASRec. Experiments cover three in-domain Amazon datasets (Games, Arts, Movies) and three out-of-domain datasets (Sports, Baby, Goodreads), with Recall@10/20 and NDCG@10/20. The authors report consistent gains over BERT, GTE, BGE, LLM2Vec, BLAIR, EasyRec, and LLMEmb, and provide ablations, backbone studies, mixed-dataset studies, and efficiency analysis.","tokens_in":19935,"tokens_out":6273,"duration_ms":60202,"significance":"If the results are taken at face value, the paper makes a useful practical contribution: a lightweight recipe (Qwen2-0.5B backbone) for producing recommendation-aware embeddings that improve downstream sequential recommenders over existing text embeddings, including on categories never seen during pretraining. The evidence for the narrow claim is solid: Table 3 reports a fair text-embedding comparison under a fixed protocol with three-seed averaging, and Table 5 and Figure 4 support the generalization and backbone claims. The code is promised and the experimental setup follows standard datasets and full-ranking evaluation. However, the broader framing rests on two unsupported pillars: the absence of ID-based recommenders as anchors, and the claim that CSFT specifically encodes collaborative-filtering structure that survives mean-pooled item embeddings. Both are testable and should be addressed before the paper can support its title-level claim.","major_comments":[{"comment":"The evaluation compares only text-embedding-initialized recommenders; no ID-based sequential recommender trained from scratch is reported. Because ID-based SASRec and GRU4Rec are the dominant paradigm for sequential recommendation, the absence of these numbers means Table 3 cannot establish the paper's title-level claim that LLM2Rec is a 'powerful embedding model for sequential recommendation' in an absolute sense. It only establishes superiority over the listed text-embedding baselines. Please add ID-based GRU4Rec and SASRec on the same data splits under the same full-ranking protocol, or explicitly reframe the contribution as 'text-embedding models for sequential recommendation' and soften the title and abstract accordingly.","section":"§4.1.2, Table 3"},{"comment":"The central mechanism — that CSFT injects item-level collaborative-filtering (CF) signals and that these signals survive IEM — is not directly evidenced. The only quantitative support is the downstream accuracy ablation in Table 4, which is a confounded proxy: fine-tuning on next-item-title prediction over 1.03M interactions can improve accuracy by adapting to item-title vocabulary, category-transition regularities, or popularity, without inducing pairwise co-occurrence geometry in single-item embeddings. There is no control that destroys interaction co-occurrence while preserving token distributions (e.g., shuffling items within sequences or permuting user-item assignments), and Eq. (3)–(4) contain no sequence-level objective, so the persistence of CF information is assumed. Moreover, Sports, Baby, and Goodreads items have no interaction history in the pretraining mixture, so CSFT gains on these datasets (e.g., Sports R@10 from 0.1079 to 0.1119 in Table 4) cannot stem from item-level CF; they are more plausibly explained by transferable category or semantic regularities. Please add a direct test of embedding geometry (e.g., co-purchased pairs closer than category-matched non-co-purchased pairs), a shuffled-co-occurrence control, and a discussion of how CSFT could benefit out-of-domain items.","section":"§3.2–§3.3, Eq. (2)–(4), Table 4"},{"comment":"The comparison between the 'Causal' and 'Bidirectional' settings changes both the attention mask and the pooling strategy: the causal row uses the last hidden state of the [EOS] token, while the bidirectional row uses mean pooling over all token hidden states. The reported improvement therefore cannot be attributed to bidirectional attention alone, which is the stated design motivation for IEM. A controlled comparison should hold the pooling strategy fixed, e.g., causal attention with mean pooling versus bidirectional attention with mean pooling, to isolate the effect of the attention mask.","section":"Table 4, rows 'Causal' and 'Bidirectional'"}],"minor_comments":[{"comment":"The header 'Casual' should be 'Causal'.","section":"Table 4"},{"comment":"The sentence 'we only retrain the item titles with some necessary separators, like commas' appears to be a typo; 'retrain' should likely be 'retain' or 'keep', since the surrounding text describes restricting the input to item titles rather than training the titles.","section":"§3.2"},{"comment":"The text 'LLM2Rec consistently outperforms the general-purpose embedding baseline, LLM2Rec' should refer to 'LLM2Vec' rather than 'LLM2Rec'.","section":"§4.4.1"},{"comment":"References [23] and [24] are the same paper; please deduplicate the bibliography and adjust the in-text citations accordingly.","section":"References"},{"comment":"The heading 'Evaluation Mertrics' contains a typo and should read 'Evaluation Metrics'.","section":"§4.1.1"},{"comment":"Algorithm 1 uses epoch counts E1, E2, and E3, but Section 4.1.3 reports step counts (10,000, 1,000, 1,000); please align the two descriptions for reproducibility.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"None."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a solid empirical paper with a new two-stage recipe for adapting an LLM into a recommendation embedding model, but the central mechanistic claim about collaborative filtering signals is more asserted than demonstrated, and the evaluation compares only against text-embedding baselines, not the ID-based recommenders that dominate the field.\n\nWhat's new: the specific combination of collaborative supervised fine-tuning (next-item title generation on mixed interaction data) followed by bidirectional attention, masked next-token prediction, and item-level contrastive learning. LLMEmb and LLM2Vec each have pieces, but not this full pipeline. The experiments are broad: six pretraining datasets, three in-domain and three out-of-domain test sets, two downstream recommenders, four LLM backbones, and an efficiency analysis. The code is available, and the numbers consistently favor LLM2Rec over the listed text-embedding baselines on both GRU4Rec and SASRec. That is a genuine, reproducible result.\n\nThe soft spots are real but proportionate. Most importantly, there is no ID-based baseline (e.g., SASRec trained on item IDs from scratch) in the comparisons, so readers cannot tell whether the embeddings actually beat standard sequential recommendation practice in-domain. The absolute R@10 numbers (0.08-0.15) look low, but without the ID benchmark it is hard to interpret. Second, there are no error bars or significance tests, and several reported gains are small (e.g., under 1% relative on Goodreads with SASRec). The three-run average helps, but not enough to rule out noise. Third, the mechanism story is not fully supported. The ablation shows CSFT helps, but on out-of-domain datasets the model has never seen those items during pretraining, so the gain cannot come from item-level CF memory. It likely reflects transferable semantic/category regularities or vocabulary adaptation. The stress-test note is right that there is no control that destroys co-occurrence while preserving token distributions, so the 'CF injection' claim remains an interpretation rather than a proven effect. That does not invalidate the recipe, but it does mean the title overstates what is established.\n\nOverall, this is a useful paper for anyone working on text-based or hybrid recommenders, and the empirical work is careful enough to deserve a serious referee. I would expect the authors to add ID-based comparisons, significance testing, and ideally a direct analysis of embedding geometry to support the CF claim. With those, it could be a solid KDD-level contribution; as is, it is a well-executed but somewhat overclaimed empirical study.\n\nRecommendation: send to peer review, but with major-revision expectations around baselines and mechanism.","headline":"A solid and reproducible empirical pipeline for injecting collaborative signals into LLM embeddings, but the mechanism is asserted rather than proven and the evaluation dodges ID-based baselines.","tokens_in":20509,"tokens_out":3720,"would_cite":true,"duration_ms":35961,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM2Rec claims that fine-tuning a large language model on user interaction histories yields item embeddings that combine semantics with collaborative-filtering signals and outperform text-embedding baselines in sequential recommendation.","keywords":["Sequential recommendation","Large language models","Item embeddings","Collaborative filtering","Contrastive learning","Supervised fine-tuning","Out-of-domain generalization","Bidirectional attention"],"falsifier":"Replace each item title in the fine-tuning sequences with a random but consistent token, or shuffle the item order within each sequence, while keeping everything else fixed; if the resulting embeddings still produce the reported gains on the out-of-domain datasets, then co-occurrence structure is not what carries the improvement. A direct geometry check—measuring whether co-purchased item pairs become closer in LLM2Rec's embedding space than in the base LLM's space—would make the CF claim observable.","tokens_in":19474,"feed_emoji":"🧠","tokens_out":6814,"duration_ms":57010,"temperature":0.7,"pith_summary":"The paper argues that an ideal item embedding for sequential recommendation must carry two kinds of information at once: what an item is (semantics) and how it co-occurs with other items in user histories (collaborative filtering, or CF). It proposes LLM2Rec, a two-stage training recipe that first fine-tunes an LLM to predict the next item in real user sequences, then reshapes it into an embedding model with bidirectional attention, masked token prediction, and item-level contrastive learning. The reported experiments show that embeddings from this recipe improve downstream recommenders over all tested text-embedding baselines, on datasets from the training categories and on categories and platforms the model never saw. That matters because conventional recommenders either capture CF but fail to transfer to new domains, or capture semantics but ignore CF; LLM2Rec is an attempt to get both in a single embedding space.","feed_headline":"LLM2Rec joins recommendation signals with language-model item embeddings","feed_subtitle":"Fine-tuning a small language model on purchase histories beats pure text embeddings on known and unseen domains.","key_machinery":"The load-bearing object is the trained item embedder $E = \\mathrm{avg} \\circ \\pi_\\theta$: an LLM whose causal attention has been replaced by bidirectional attention, whose output is mean-pooled over title tokens, and whose weights are obtained by two sequential training stages. CSFT is the stage that injects CF signals: the model is supervised to autoregressively generate the next item title in a user's sequence, which forces hidden states of item titles to reflect co-occurrence statistics. IEM then reconciles this CF-aware model with embedding-style inference: MNTP adapts the weights to bidirectional attention, and item-level contrastive learning with randomly masked views pulls augmentations of the same item together while separating distinct items, producing distinguishable item-level vectors.","core_discovery":"The central claim is that a decoder-only LLM can be turned into a recommendation embedding model that is simultaneously semantic and collaborative. The mechanism is the order of training: first, Collaborative Supervised Fine-Tuning (CSFT) makes the LLM predict the next item given a user's previous item titles, exposing the hidden states to CF-driven co-occurrence patterns; second, Item-level Embedding Modeling (IEM) replaces causal attention with bidirectional attention, adds masked next-token prediction, and applies item-level contrastive learning so the model produces fixed-sized item embeddings rather than token predictions. The paper reports that LLM2Rec consistently outperforms all compared baselines on both in-domain and out-of-domain datasets, with the largest gains on out-of-domain categories, and that the CSFT stage contributes the largest share of the improvement in the ablation.","pith_inferences":["Editorial inference: if CSFT truly embeds CF structure, then LLM2Rec embeddings should reconstruct known co-purchase or co-click graphs better than text embeddings; a direct nearest-neighbor test on held-out interaction pairs would make the claim's mechanism visible.","Editorial inference: the framework suggests a path to a single 'universal' item-embedding model trained once on many platforms and categories, then frozen for downstream recommenders; the paper's cross-platform result is a first step but larger cross-platform evaluations would test it.","Editorial inference: the same two-stage recipe—supervised collaborative fine-tuning followed by bidirectional embedding adaptation—could generalize to other structured prediction tasks where an embedding must encode both content and relational context, such as citation recommendation or link prediction on text-attributed networks.","Editorial inference: because the reported gains rest on an ablation rather than on controlled perturbation of sequence statistics, a natural stress test is to shuffle item order or replace item titles with random identifiers during CSFT; if gains persist, the credit may be vocabulary or domain adaptation rather than CF structure."],"forward_implications":["Sequential recommenders can be improved without learning item IDs from scratch: a fixed, general embedding from LLM2Rec, plus a small linear adapter, outperforms both pure text embeddings and recommendation-specific embedding models.","CF signals can be transferred across domains: because the embedding is trained on mixed categories, items from categories unseen at training time still receive embeddings that carry useful behavioral information.","The two-stage order matters: causal attention fine-tuning for recommendation, followed by a bidirectional embedding adaptation, is what yields the gains; reversing or skipping the collaborative stage degrades performance.","Small LLM backbones suffice for practical gains: the reported results are based on a 0.5B-parameter backbone, with larger backbones improving further but at higher cost.","Dataset diversity in pre-training improves out-of-domain generalization, while a focused single-category pre-training can be better for that same category."],"supporting_citations":[{"why":"Supplies the bidirectional-attention and masked-next-token-prediction recipe for turning decoder-only LLMs into embedding models, forming the basis of IEM.","marker":"[3]"},{"why":"Supplies the unsupervised contrastive learning objective adapted for item-level contrastive learning.","marker":"[8]"},{"why":"Provides the pre-training datasets and serves as a recommendation-specific embedding baseline that motivates the need for CF-aware embeddings.","marker":"[13]"},{"why":"Transformer-based downstream sequential recommender used to evaluate the generated embeddings.","marker":"[15]"},{"why":"Recurrent downstream sequential recommender used to evaluate the generated embeddings.","marker":"[12]"},{"why":"LLM-based recommendation embedding baseline; comparison against it demonstrates the gain from explicitly injecting CF signals.","marker":"[29]"},{"why":"Recommendation-specific embedding baseline and earlier example of contrastive alignment for CF signals.","marker":"[39]"},{"why":"General-purpose text embedding baseline whose weaker performance motivates the need for recommendation-specialized embeddings.","marker":"[17]"}],"fun_headline_variants":["LLM2Rec fuses collaborative signals into LLM item embeddings","LLM2Rec: LLM embeddings that encode collaborative filtering for recommendation","LLM2Rec trains LLMs to produce collaborative-aware item embeddings","LLM2Rec: from language models to recommendation embeddings with collaborative signals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that making the LLM predict the next item in real user sequences really does bake 'items that tend to be bought together' into the hidden states of individual item titles, and that this information survives the later switch to bidirectional attention and averaging.","fun_headline_variants_meta":{"raw":{"variants":["LLM2Rec fuses collaborative signals into LLM item embeddings","LLM2Rec: LLM embeddings that encode collaborative filtering for recommendation","LLM2Rec trains LLMs to produce collaborative-aware item embeddings","LLM2Rec: from language models to recommendation embeddings with collaborative signals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1576,"prompt_tokens":992,"completion_tokens":584,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":507}},"tokens_in":608,"tokens_out":584,"duration_ms":5367,"temperature":1.0,"reasoning_tokens":507,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:00:25.764605+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace each item title in the fine-tuning sequences with a random but consistent token, or shuffle the item order within each sequence, while keeping everything else fixed; if the resulting embeddings still produce the reported gains on the out-of-domain datasets, then co-occurrence structure is not what carries the improvement. A direct geometry check—measuring whether co-purchased item pairs become closer in LLM2Rec's embedding space than in the base LLM's space—would make the CF claim observable.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Recurrent downstream sequential recommender used to evaluate the generated embeddings."}],"review_version":2}