{"id":"d9584675-91f9-405a-b61d-0abe27c94045","arxiv_id":"2508.20587","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fusing frozen LLM item embeddings into MSGAT and NISER improves Recall@20 on Amazon datasets, and re-ranking with the vanilla model restores MRR, though the gains are not consistent across all configurations.","lead":"SemSR adds frozen text embeddings from a large language model to two session-based recommendation models, and the authors report higher recall and, after re-ranking, better mean reciprocal rank on Amazon datasets. A generalist should care because this is a cheap, end-to-end way to make recommenders use item semantics without fine-tuning an LLM.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fused model alone does not consistently beat baselines on MRR; the abstract's 'both Recall and MRR' claim depends on an external NISER re-ranking step not described as part of the fusion.","rationale":"The reader identified the dependence on the external re-ranker as the weakest assumption; I agree. This is the most load-bearing issue because the paper's headline claim about 'both Recall and MRR' rests on the '+' variants, which are not described in the abstract and are not part of the fused model itself. The Recall gains from semantic fusion are plausible and consistently present, but the MRR gains are not a property of the fusion across all base models. My proposed test directly isolates whether the re-ranker alone produces the MRR boost, which would settle whether the abstract's claim is an overstatement or a description of a legitimate two-stage pipeline. No deeper methodological flaw was found: the architecture in Section 3.3 is coherent, the experimental setup is standard, and the comparison against NARM/SRGNN is reasonable. Therefore the verdict remains CONDITIONAL rather than moving to REJECT or ACCEPT, because the concern is about claim precision and attribution, not about the validity of the underlying method.","tokens_in":11177,"tokens_out":6471,"duration_ms":58823,"concrete_test":"Run a re-ranking control: apply the same NISER-based re-ranking procedure to the top-100 candidate lists produced by vanilla NISER and vanilla MSGAT (e.g., NISER+ and MSGAT+). If these controls achieve MRR@20 gains comparable to SemNISER-F+ and SemMSGAT-F+, then the MRR improvement is due to the re-ranker, not to the semantic fusion. Also report paired bootstrap confidence intervals for SemNISER-F vs NISER on both datasets to check whether the MRR differences are statistically distinguishable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, stated in the abstract and echoed in RQ2, is that the integration of LLM embeddings with data-driven SR models significantly outperforms standalone LLM approaches and baseline SR models in both Recall and MRR. Table 2 does not support this for the fused models without post-processing. For NISER, SemNISER-F has lower MRR@20 than vanilla NISER on both datasets: on Amazon-M2, 26.41 vs 26.45 (Angle) and 25.19 vs 26.45 (Llama 3); on Beauty, 6.10 vs 6.95 (Angle) and 5.42 vs 6.95 (Llama 3). The only reported MRR improvements over NISER come from the '+' variants (SemNISER-I+, SemNISER-F+), which re-rank the candidate list using a plain, unmodified NISER model (RQ3). This re-ranking is an external step: it is not part of the fused architecture described in Section 3.3, and the abstract does not mention it. Consequently, the claim that the integration itself improves fine-grained ranking is not supported for a substantial subset of the reported configurations. The Recall gains are consistent, but the MRR gains are attributable to a separate re-ranker, which also makes the comparison asymmetric because vanilla NISER is embedded inside the SemSR pipeline while being used as a baseline. Additionally, no error bars, significance tests, or ablations separating the fusion from the re-ranking are provided, so the word 'significantly' is unsubstantiated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SemSR, a family of methods that incorporate frozen LLM-generated item embeddings into two session-based recommender models, MSGAT and NISER. Variants include semantic initialization (SemSR-I), feature fusion (SemSR-F), and re-ranking of the top-K list by a vanilla NISER model (SemSR-I+ and SemSR-F+). Experiments on Amazon-M2 (UK) and Amazon-Beauty report Recall@K and MRR@K. The paper claims that LLM-based in-context methods are strong at coarse retrieval but weak at ranking, and that the integration of LLM embeddings with data-driven SR models significantly improves both Recall and MRR over all baselines.","tokens_in":11496,"tokens_out":4884,"duration_ms":42166,"significance":"If the central claim were fully supported, the contribution would be practically useful: it would give a model-agnostic, end-to-end recipe for injecting frozen semantic embeddings into established SR models, evaluated on public benchmarks with two LLM encoders. The recall gains in Table 2 are large and consistent, e.g., MSGAT-F on Amazon-M2 raises R@20 from 38.86 to 52.66, and the framework is clearly described in Section 3.3. However, the MRR improvements are not consistent for the fused models alone and depend on an external re-ranking step that is not part of the stated architecture; the paper also provides no uncertainty quantification, so the rhetorical claim of 'significantly outperforms' is currently unsupported.","major_comments":[{"comment":"The abstract claims that the integration of LLMs with data-driven SR models significantly outperforms baseline SR models in both Recall and MRR, but Table 2 shows that the fused models alone do not consistently improve MRR. For example, SemNISER-F with Angle embeddings on Beauty has MRR@20 6.10 versus NISER's 6.95, and SemNISER-F with Llama 3 has 5.42 versus 6.95; SemNISER-I on Amazon-M2 has MRR@20 14.25 versus NISER's 26.45. The consistent MRR improvements appear only in the '+' variants, which additionally use re-ranking. The claim must either be restricted to Recall for the fused models, or the re-ranking step must be explicitly treated as part of the method and evaluated symmetrically.","section":"Abstract and RQ2; Table 2"},{"comment":"The re-ranking procedure that produces SemSR-I+ and SemSR-F+ is not part of the architecture described in Section 3.3, and no algorithmic details are given for it: Section 4 only states that the lists 'obtained from SemSR-I, and SemSR-F' are re-ranked 'using NISER model.' This is load-bearing because re-ranking is the mechanism by which the paper obtains its claimed MRR improvements. Without a description of the candidate-list size, whether the NISER re-ranker is trained or frozen, and how re-ranking interacts with the fusion, the results cannot be reproduced. Since vanilla NISER is both the baseline and the re-ranker embedded in the pipeline, the comparison is also asymmetric.","section":"RQ3 and Section 3.3"},{"comment":"No error bars, standard deviations, or significance tests are reported anywhere in Section 4, so the repeated use of 'significantly' in RQ2 and RQ3 is not substantiated. In addition, the LLM-as-RS comparison in Table 3 is performed on a 10,000-session subset of Amazon-M2, as described in Section 4, whereas Table 2 uses the full test set; the paper should state explicitly that all models in Table 3 are evaluated on this subset and should temper the RQ1 conclusions accordingly, because the comparison to the full-test-set numbers in Table 2 is not direct.","section":"Section 4.1; Table 3"}],"minor_comments":[{"comment":"There are typos: 'knowlege' should be 'knowledge', and 'utilizes a a two step strategy' contains a duplicated article.","section":"Section 2"},{"comment":"The footnote markers after 'Llama3-8B-Instruct' and 'Chromadb' render as asterisks that are easy to confuse with mathematical symbols; these should be replaced with standard numbered citations or footnotes.","section":"Section 3.2"},{"comment":"The caption refers to Step-1 and Step-2, but the figure does not clearly delineate these steps; adding visual separators or labels would improve readability.","section":"Figure 1"},{"comment":"Reference [2] appears to be a mistaken or secondary citation for SRGNN; the original work is cited as [36], so [2] should be corrected or removed.","section":"References"},{"comment":"The statement 'adam optimizer with mini-batch size 100, momentum 0.9' is ambiguous for Adam; please report the exact optimizer settings such as beta1, beta2, and epsilon.","section":"Hyperparameter Setup, Section 4"}],"recommendation":"major_revision","confidential_remarks":"The use of the authors' own NISER model as both backbone and baseline is not improper, but it makes the asymmetry of the re-ranking comparison a genuine correctness concern that needs to be addressed in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper has a real empirical result — fusing frozen LLM item embeddings with trainable interaction embeddings, plus an attention-based LLM session embedding and end-to-end training, lifts Recall@20 substantially on two Amazon datasets (e.g. MSGAT-F on M2 goes from 38.86 to 52.66). The catch: the abstract’s 'both Recall and MRR' claim doesn’t hold for the fused models alone. Table 2 shows SemNISER-F’s MRR@20 is below vanilla NISER on both datasets (26.41 vs 26.45 on M2, 6.10 vs 6.95 on Beauty). The MRR wins come only after re-ranking the candidate list with a plain NISER model (the '+' variants) — a step that is described in RQ3 but is external to the fusion and makes the baseline comparison asymmetric.\n\nThat’s the main thing to know. Credit where due: the architecture is clean, the variants are sensible, and the experiments cover two scales. Unlike SAID’s two-stage learning or LLM-ESR’s self-distillation, SemSR is genuinely end-to-end, and using soft-attention over LLM item embeddings to get the session representation is a small but real novelty. The paper is also honest enough to report Recall@100 and MRR@100, not just a single K.\n\nSoft spots, in order of severity. First, the RQ2 text says the fused variants 'consistently outperform' vanilla models in recall and MRR, which Table 2 contradicts. That overstatement needs a careful rewrite — the fusion delivers recall, the re-ranker delivers MRR. Second, there are no error bars, significance tests, or ablation that separates the fusion from the re-ranking, so 'significantly' is unsubstantiated. Third, the LLM-as-Recommender comparison on Amazon-M2 uses a 10k-session subset without any analysis of representativeness; it’s a cost-saving choice, but it should be justified or flagged. Fourth, the novelty is incremental relative to KAR, AlphaFuse, LLM-ESR, and SAID — the fusion idea is a variation, not a new paradigm. None of that is fatal for a workshop paper, but it limits where you can place it.\n\nBottom line: I’d take this paper for the Recall finding and the clear description of what re-ranking does, but I would not cite the 'both Recall and MRR' line. With the claims corrected and some significance testing, a conference version would be solid. As it stands, it deserves a serious referee — send it to peer review, but expect heavy revision.","headline":"SemSR's fusion of frozen LLM embeddings with GNN session encoders gives real Recall gains, but the MRR claim only holds after an external NISER re-ranker, so the abstract overstates what the fused model itself does.","tokens_in":12132,"tokens_out":3392,"would_cite":false,"duration_ms":29117,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that fusing frozen LLM item embeddings with trainable interaction embeddings in session-based recommenders such as MSGAT and NISER raises Recall@K substantially, while a separate re-ranking step by vanilla NISER recovers…","keywords":["session-based recommendation","large language models","semantic item embeddings","item embedding fusion","re-ranking","Recall@K","MRR","graph attention networks"],"falsifier":"Reproduce Table 2 on Amazon-M2(UK) without any re-ranking step. If SemNISER-F or SemMSGAT-F fails to beat its vanilla counterpart on MRR@20 while still improving Recall@20, then the recall claim stands but the claim that fusion improves fine-grained ranking is false and depends entirely on the external ranker.","tokens_in":10934,"feed_emoji":"🛒","tokens_out":8217,"duration_ms":75215,"temperature":0.7,"pith_summary":"This paper tries to establish that a session-based recommender can become semantically aware without expensive LLM fine-tuning: freeze the text embeddings an LLM produces for item titles and metadata, and inject them into an existing data-driven recommender. The authors test three ways to use an LLM: as a standalone prompt-based recommender, as a source of initial item embeddings, and as a source of semantic embeddings fused with learned interaction embeddings. Their experiments on two retail datasets show that the standalone LLM trails data-driven models badly on ranking, that semantic initialization lifts Recall at large K, and that the fused variants outperform vanilla MSGAT and NISER on Recall and, after re-ranking, on MRR as well. If the result holds, practitioners can upgrade deployed recommender backbones by swapping in a frozen-embedding fusion layer rather than retraining an LLM.","feed_headline":"Frozen LLM embeddings buy session recommenders up to 19 recall points","feed_subtitle":"Merging frozen text embeddings with learned interaction embeddings lifts top-K recall, and a re-ranking pass restores MRR.","key_machinery":"The load-bearing mechanism is a fusion block placed between the item representation and the scorer. Each item gets a trainable interaction embedding $\\mathbf{i}^m_j \\in \\mathbb{R}^{d_1}$ and a frozen LLM embedding $\\mathbf{i}^l_j \\in \\mathbb{R}^{d_2}$; the two are concatenated and passed through a linear projection to form the semantic item embedding. The session is represented twice, once by the base SR model as $\\mathbf{s}^m$ and once by a soft-attention aggregation of the LLM embeddings with the most recent item highlighted, giving $\\mathbf{s}^l$; the two session vectors are concatenated and projected. Relevance is then the softmax of $\\mathbf{i}_k^\\top \\mathbf{s}$. The same block is dropped into MSGAT and NISER, and a final re-ranking pass by vanilla NISER is what the paper credits for the MRR improvements.","core_discovery":"The paper's central claim is that the fusion variant, SemSR-F, dominates both data-driven baselines and LLM-only recommenders: on Amazon-M2(UK), Recall@20 rises from 38.86 for MSGAT to 52.66 for SemMSGAT-F with Llama-3 embeddings, and from 49.84 for NISER to 54.98 for SemNISER-F with BERT-based embeddings. The authors attribute the recall gain to semantic item representations capturing title, category, brand, and price signals that interaction-only embeddings miss. They also report that re-ranking the fused model's top-K list with an unmodified NISER model raises MRR for every variant, and they interpret that re-ranking step as part of the SemSR pipeline. The paper is explicit that LLM-derived knowledge helps at coarse retrieval while interaction-based signals carry fine-grained ranking, and that the two are complementary.","pith_inferences":["A natural production architecture, which the paper does not develop, would run SemSR-F as candidate generator and a cheap interaction model as final ranker; the paper's own numbers suggest MRR would match or beat the fused model alone.","The drop in MRR when semantic embeddings are used without re-ranking suggests the frozen text vectors distort the learned ranking geometry; a testable extension is to constrain the fusion projection so the interaction embedding lives in a subspace complementary to the semantic embedding, to see whether recall gains can be had without an external ranker.","Because the paper reports only aggregate Recall and MRR, a testable extension is to split results by item frequency: if the recall gains concentrate on long-tail or cold items, the method is a cheap way to address cold-start, which the paper does not explicitly claim."],"forward_implications":["Any graph-attention or normalization-based SR model can receive semantic awareness by concatenating frozen LLM embeddings to its item representations, with no LLM retraining.","A two-stage design, semantic retrieval for candidate generation and interaction-based re-ranking for final order, should beat either stage alone.","LLM prompting alone is not competitive on fine-grained ranking, so systems relying on in-context LLMs for next-item ranking would lose MRR.","Semantic initialization alone yields its largest recall advantage at high K (200 to 500), meaning it is suited to large candidate sets."],"supporting_citations":[{"why":"Supplies the NISER base model that the SemNISER variants wrap.","marker":"[11]"},{"why":"Supplies the MSGAT base model that the SemMSGAT variants wrap.","marker":"[29]"},{"why":"Supplies the BERT-based frozen semantic item embeddings used in the fusion variants.","marker":"[23]"},{"why":"Supplies the LLM-based frozen semantic item embeddings used in the fusion variants.","marker":"[1]"},{"why":"Provides the NARM baseline that the proposed method is compared against.","marker":"[22]"},{"why":"Provides the graph-based SRGNN baseline that the proposed method is compared against.","marker":"[2]"}],"fun_headline_variants":["Fusion of LLM and interaction embeddings boosts session recall","LLM embeddings add 19 recall points to session recommenders","SemSR fusion beats LLM-only and data-driven session models","Frozen LLM embeddings lift session recall up to 19 points","LLM semantics + learned interactions: best of both worlds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline MRR improvement depends on counting an external vanilla NISER re-ranking step as part of SemSR; for the fused model alone, MRR@20 can fall below plain NISER (SemNISER-F on Beauty: 6.10 vs 6.95).","fun_headline_variants_meta":{"raw":{"variants":["Fusion of LLM and interaction embeddings boosts session recall","LLM embeddings add 19 recall points to session recommenders","SemSR fusion beats LLM-only and data-driven session models","Frozen LLM embeddings lift session recall up to 19 points","LLM semantics + learned interactions: best of both worlds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1360,"prompt_tokens":993,"completion_tokens":367,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":282}},"tokens_in":609,"tokens_out":367,"duration_ms":4179,"temperature":1.0,"reasoning_tokens":282,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:42:31.781337+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce Table 2 on Amazon-M2(UK) without any re-ranking step. If SemNISER-F or SemMSGAT-F fails to beat its vanilla counterpart on MRR@20 while still improving Recall@20, then the recall claim stands but the claim that fusion improves fine-grained ranking is false and depends entirely on the external ranker.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the NISER base model that the SemNISER variants wrap."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MSGAT base model that the SemMSGAT variants wrap."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the graph-based SRGNN baseline that the proposed method is compared against."}],"review_version":2}