{"id":"be9916df-f8b9-4e41-8e3b-0ef9a24ab964","arxiv_id":"2608.12184","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"GALLM injects item co-occurrence and item-text relations into LLM attention as learnable biases, improving sequential recommendation accuracy on four benchmarks.","lead":"A new framework, GALLM, adds collaborative graph information to large language models for sequential recommendation by turning item relationships into attention biases. It reports consistent gains over strong baselines on four benchmarks, though the reported margins are small and no code or error bars are provided.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Item–Item bias uses raw co-occurrence counts, so high buckets may encode item popularity rather than collaborative dependency; a PMI or popularity-control experiment is needed to support the central claim.","rationale":"The reader's weakest assumption is that item co-occurrence counts encode collaborative dependency rather than popularity or session artifacts. I agree and sharpen the concern: because Eq. (7) bins raw counts, equal-frequency binning makes the high buckets systematically track the product of item frequencies, so the injected Item–Item bias can act as a popularity prior. This is testable and directly relevant to the paper's central claim that GALLM captures global collaborative signals. The paper's internal numbers are consistent and the method is clearly described; I am not claiming the results are fabricated or that the method is incoherent. However, the existing ablation evidence (Table 4) cannot distinguish the proposed collaborative mechanism from a popularity-proxy mechanism, and the reported improvements over the strongest baseline are modest on several metrics (e.g., 2.56% NDCG@10 on Book, 2.82% NDCG@10 on Clothing). The proposed control experiment would settle whether the 9.76% HR@5 average is evidence for graph-aware collaborative modeling or for item popularity. Since the reader already issued a CONDITIONAL verdict and my concern specifies an additional condition rather than overturning the empirical claim, the verdict remains CONDITIONAL (UNCHANGED).","tokens_in":14560,"tokens_out":8899,"duration_ms":79979,"concrete_test":"Re-run the four benchmark experiments with the Item–Item relation replaced by: (i) PMI-normalized co-occurrence C(i,j)/(C(i,·)C(·,j)) with smoothing; (ii) a popularity-only control whose bucket depends only on the two items' marginal frequencies (e.g., sum or product of frequency bins); (iii) the paper's raw-count buckets. If the popularity-only control reproduces GALLM's HR@5 gains over LLaRA, or if PMI normalization collapses the gains by more than ~10% relative, the collaborative-signal interpretation of the headline result is not supported. Report per-dataset HR@5 and NDCG@5 for each condition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the discretized raw co-occurrence count C(i,j) of Eq. (1), mapped to buckets by Eq. (7) and injected as attention bias in Eq. (12), encodes pairwise collaborative preference. Raw co-occurrence under independent, random co-occurrence scales with the product of the two items' marginal frequencies. Because Eq. (7) uses equal-frequency binning on raw counts, the highest buckets are dominated by popular-popular item pairs even when no true preference relation exists, and the per-bucket bias in Eq. (12) cannot distinguish such pairs from genuinely related ones. With full-item-space HR/NDCG evaluation and no negative sampling (Appendix A.3), a systematic boost to popular items can inflate the reported 9.76% HR@5 gain. The Table 4 ablation w/o I–I does not rule out this alternative: removing a popularity-proxy bias would also hurt. No PMI normalization, frequency-matched control, or popularity-only baseline is reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces GALLM, a graph-aware large language model framework for sequential recommendation. It constructs a hybrid prompt in which each item is represented by both its textual description and a dedicated item token, then builds a token-level graph with three relation types: Text–Text, Item–Text, and Item–Item, where Item–Item relations are derived from global item co-occurrence counts. The relations are converted into learnable attention biases injected into every Transformer layer, avoiding an external graph encoder. The authors report evaluations on Amazon Toys, Clothing, Books, and MovieLens-10M, claiming consistent improvements over existing LLM-based and traditional recommender baselines, and they provide ablations and attention-pattern analyses.","tokens_in":14792,"tokens_out":5507,"duration_ms":47297,"significance":"If the results hold, the paper offers a simple, parameter-efficient way to combine semantic and collaborative signals within a single LLM, and the idea of converting token-level graph relations into attention biases is well motivated. The paper includes clear ablations, full-item-space evaluation, and an attention analysis, and the method is specified in enough detail to be reimplemented. However, the significance is currently qualified by the absence of released code, the lack of variance or significance testing, and the unvalidated use of raw co-occurrence counts as a proxy for collaborative dependency.","major_comments":[{"comment":"The central claim that C(i,j) encodes pairwise collaborative preference is not supported against the alternative that it encodes item popularity. Under a null model in which items co-occur independently, the expected co-occurrence count is proportional to the product of the two items' marginal frequencies, so equal-frequency binning on raw counts places popular–popular pairs in the highest buckets even when there is no behavioral association. Because evaluation uses full-item ranking without negative sampling (Appendix A.3), preferentially attending to popular items can inflate HR/NDCG. The w/o I-I ablation in Table 4 does not rule out this explanation, since removing a popularity-proxy bias would also hurt. I recommend reporting PMI or another margin-controlled association measure, a frequency-matched control, or an explicit popularity-only baseline.","section":"Collaborative Graph Construction, Eqs. (1), (7), (12)"},{"comment":"The claimed 'best performance' is not supported by tests of statistical significance. The improvements over the strongest baseline are small in several cells (e.g., Book N@10: 2.56%; MovieLens N@10: 3.66%). Although Appendix A.3 states that each experiment was repeated three times, no variance, confidence intervals, or significance tests are reported. Without such evidence, the observed differences could be within run-to-run noise.","section":"Performance Comparison, Table 2"},{"comment":"The contribution of the Item–Item relation is modest on some datasets and metrics; for example, on Toy, removing I–I changes H@5 from 0.0252 to 0.0244 and N@10 from 0.0206 to 0.0200, and on MovieLens H@10 changes from 0.0664 to 0.0644. The text states that removing Item–Item 'consistently harms performance,' but no significance test is given for any of these differences. A paired test across seeds (or more seeds) is needed before this ablation can support the central claim.","section":"Ablation Study, Table 4"},{"comment":"The paper says 'Additional implementation details are included in the supplementary code' but no code or repository is provided. It is also unclear whether the co-occurrence C(i,j) in Eq. (1) is computed from full user histories or from the length-11 sliding-window subsequences, and whether the statistics are computed before or after the chronological split. Since the method depends critically on this statistic, the exact definition must be stated and the code released to make the experiments reproducible.","section":"Appendix A.3 and Dataset Details"}],"minor_comments":[{"comment":"The bucket boundaries used for equal-frequency binning are never reported, so the reader cannot tell how strong a co-occurrence is required to reach the highest relation bucket. Please report the bucket edges or the quantiles used.","section":"Figure 5 and Eq. (7)"},{"comment":"The reference list contains duplicate entries for Wang et al. 2026a and 2026b, which appear to be the same paper; please consolidate and verify all citations.","section":"References"},{"comment":"Eq. (12) is slightly underspecified: b_{II,R} has subscripts R=r_{ij} but the bucket index set is also denoted R in Eq. (7); please use consistent notation to avoid confusion between the relation-type set and the bucket index.","section":"Eq. (12)"},{"comment":"The 'Impr.' row is computed relative to the second-best method for each metric, but the caption does not state this explicitly; please clarify.","section":"Table 2 caption"},{"comment":"For Amazon Books and MovieLens-10M, the authors randomly retain 100,000 items before preprocessing. Random item subsampling changes the co-occurrence distribution on which GALLM relies and can distort the evaluation space; the choice deserves justification or a sensitivity analysis.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript makes heavy use of unpublished arXiv papers, including several self-citations and very recent references (e.g., HeLLM, G2Rec, HatLLM, TCA4Rec, GraphLoRA). While this is not unusual in a fast-moving area, the editor may want to confirm that the baseline implementations are publicly available and that the comparisons are not affected by the authors' own tuning. The central method is plausible and the paper is in scope for cs.IR, but the decisive point is whether the authors can address the statistical and confounding concerns raised in the major comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that it has a clean idea and an unresolved confound in the same place: the Item-Item attention bias. GALLM injects three relation types as learnable attention biases into an LLM, avoiding an external graph encoder. That is a sensible and lightweight design, and the ablations in Table 4 show each relation contributes. The paper is clearly written, and the experimental setup follows the usual protocol: chronological 8:1:1 split, full-item-space evaluation, co-occurrence computed only from training data.\n\nThe main problem is the stress-test concern, which I think lands. C(i,j) is raw co-occurrence; equal-frequency binning puts popular-popular pairs in the top buckets. The learned bias can then act as a popularity boost. Under full-item-space ranking, that can inflate HR. The w/o I-I ablation does not refute this: removing a popularity proxy would also hurt. The paper needs a frequency-matched control or PMI-normalized co-occurrence, or at least a comparison against a popularity-only bias baseline.\n\nThere are secondary issues: no variance or significance tests, no code, and some gains are small (under 3% on Book and MovieLens N@10). These are minor relative to the popularity confound, but they matter for confidence.\n\nWho this is for: people working on LLM-based recommendation who want a plug-in collaborative signal. The method is plausible, but the evidence for the I-I relation specifically is not yet convincing.\n\nA serious referee should see it, because the idea is worth testing and the confound is addressable. I would send it to review, ask for the popularity control and significance tests, and check whether the I-I effect survives.","headline":"Clean idea, but the Item-Item attention bias is confounded with item popularity and the paper does not run the control that would clear it.","tokens_in":15274,"tokens_out":1739,"would_cite":false,"duration_ms":16612,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GALLM encodes item co-occurrence as attention biases and reports up to 16.30% HR@5 gains over the strongest baseline.","keywords":["sequential recommendation","large language models","collaborative filtering","graph-aware attention","item co-occurrence","attention biases","generative recommendation","LLM-based recommender systems"],"falsifier":"Run the same GALLM pipeline but replace each item-pair co-occurrence count with independent random values before bucketing, keeping everything else identical. If HR@5 and NDCG@5 stay within noise of the true co-occurrence version, the Item–Item graph is not carrying collaborative signal; if performance drops sharply, the co-occurrence encoding is doing the work.","tokens_in":14390,"feed_emoji":"📈","tokens_out":6679,"duration_ms":52381,"temperature":0.7,"pith_summary":"The paper argues that large language models can capture collaborative filtering signals internally, without relying on external recommendation models, by encoding a token-level collaborative graph directly into attention. GALLM builds a graph whose nodes are text tokens and special item tokens, with three edge types: text–text semantic links, item–text alignment links, and item–item global co-occurrence links. The edges are turned into small learnable biases added to the attention logits of each Transformer layer, so each layer implicitly aggregates over the collaborative graph while keeping the LLM's pretrained language behavior. On four real-world benchmarks the method reports the best results among compared baselines, improving HR@5 by 9.76% on average and by up to 16.30%.","feed_headline":"Graph-aware attention biases lift LLM recommendation by ~10%","feed_subtitle":"Item co-occurrence edges enter LLM attention directly and beat external collaborative-injection baselines.","key_machinery":"The central object is the heterogeneous token-level collaborative graph $G=(V,E)$: nodes are text tokens and item tokens in the hybrid prompt, and edges are the three relation sets $E_{TT}$, $E_{IT}$, and $E_{II}$. Item–item edges are assigned discrete relation types $r_{kk'} = \\mathrm{Bucket}(C(i_k,i_{k'}))$ by equal-frequency binning of global co-occurrence. The relation type $\\phi(x_i,x_j)$ indexes a learnable scalar attention bias $b^l_{\\phi}$ added to the content-based logit in every layer, which turns each Transformer layer into an implicit graph aggregation step over the collaborative graph.","core_discovery":"GALLM claims that raw item co-occurrence counts $C(i,j)$ from user histories encode collaborative dependencies that can be discretized into relation buckets and injected as attention biases, and that this recovers global collaborative signals without a separate graph encoder. The pre-softmax attention logit becomes $\\tilde{e}^l_{ij} = (h^l_i W^l_Q)(h^l_j W^l_K)^\\top/\\sqrt{d} + b^l_{\\phi(x_i,x_j)}$, where the learned scalar $b^l_{\\phi}$ is shared by relation type. The paper reports consistent gains over the strongest baseline across Toy, Clothing, Book, and MovieLens, with the largest single improvement of 16.30% in HR@5, and shows that the gains persist across 1B, 3B, and 8B backbone scales. Ablations removing any one of the three relation types hurt performance, and attention analysis shows item–text attention rises relative to a standard collaborative-injection baseline, while item–item attention grows with co-occurrence strength.","pith_inferences":["Outside sequential recommendation, the same bias-injection recipe could turn any relational side information—citation links, knowledge-graph edges, or social ties—into attention biases for an LLM, provided the relation types can be discretized.","The item–item relation depends on equal-frequency bucketing of co-occurrence counts; a natural stress test is to vary bucket count and observe whether gains concentrate at the correct granularity for each dataset.","Because item tokens can be initialized from pretrained recommender embeddings, GALLM still profits from those embeddings, but the central claim implies the graph biases, not the embeddings, carry most of the collaborative signal in the final model.","The attribution analysis suggests external collaborative embeddings contribute little; if that holds generally, GALLM-style attention biases could replace collaborative embedding injection altogether."],"forward_implications":["Because the collaborative signal enters as additive scalar biases, the approach adds only a small number of trainable parameters and works with existing LLM architectures without structural changes.","The same three-relation graph can be applied to any hybrid prompt that pairs an item token with its textual description, so the gains do not depend on a task-specific prompt template.","Reported gains persist across 1B, 3B, and 8B LLM backbones, suggesting the mechanism scales with model capacity rather than compensating for a weak base model.","Ablations indicate that removing any single relation degrades results, so semantic preservation, semantic–collaborative alignment, and global item dependencies are complementary sources of accuracy."],"supporting_citations":[{"why":"Supplies hybrid prompting with textual tokens plus item tokens and the collaborative-injection baseline LLaRA that GALLM improves on.","marker":"Liao et al. 2024"},{"why":"SASRec provides the sequential baseline and the item representations used to initialize GALLM's item tokens.","marker":"Kang and McAuley 2018"},{"why":"Graph Transformer formulation gives the relation-aware attention bias form that GALLM adapts into the LLM.","marker":"Ying et al. 2021"},{"why":"HeLLM is a graph-enhanced LLM baseline used in the attribution comparison and performance tables.","marker":"Guo et al. 2025"},{"why":"G2Rec constructs an item co-engagement graph and serves as a direct graph-based generative recommendation baseline.","marker":"Qiu et al. 2026"},{"why":"Provides the preprocessing protocol, chronological split, and HR/NDCG evaluation setting that all methods follow.","marker":"Wang et al. 2025a"},{"why":"LightGCN establishes that propagation over interaction graphs captures collaborative filtering signals, motivating the graph bias design.","marker":"He et al. 2020"}],"fun_headline_variants":["Co-occurrence edges in attention lift LLM rec by ~10%","No graph encoder: LLM rec gets collaborative attention biases","Graph-aware LLM rec: inject co-occurrence as attention bias","LLM recommendation gains ~10% with co-occurrence attention biases","GALLM: collaborative attention biases boost sequential rec by ~10%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that items co-occurring in user histories share genuine preference dependence, so bucketed co-occurrence counts encode collaborative structure that an LLM can exploit; if co-occurrence mainly reflects popularity or session artifacts, the injected biases would not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Co-occurrence edges in attention lift LLM rec by ~10%","No graph encoder: LLM rec gets collaborative attention biases","Graph-aware LLM rec: inject co-occurrence as attention bias","LLM recommendation gains ~10% with co-occurrence attention biases","GALLM: collaborative attention biases boost sequential rec by ~10%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000282,"raw_usage":{"total_tokens":1673,"prompt_tokens":954,"completion_tokens":719,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":628}},"tokens_in":570,"tokens_out":719,"duration_ms":6402,"temperature":1.0,"reasoning_tokens":628,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:13:02.369824+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same GALLM pipeline but replace each item-pair co-occurrence count with independent random values before bucketing, keeping everything else identical. If HR@5 and NDCG@5 stay within noise of the true co-occurrence version, the Item–Item graph is not carrying collaborative signal; if performance drops sharply, the co-occurrence encoding is doing the work.","supporting_citations":[],"review_version":1}