{"id":"bcbdbcbe-cf56-4356-b654-0d13e54020d1","arxiv_id":"2412.13771","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A recommender system that converts item IDs into shared code tokens and fine-tunes Llama with alignment tasks plus negative sampling beats LC-Rec on three Amazon datasets by a few percent.","lead":"This paper presents a two-stage method for using large language models in product recommendation: item IDs are converted into short code sequences, then the LLM is fine-tuned on alignment tasks and negative examples. The authors report modest but consistent gains over the previous best LLM-based recommender, LC-Rec, on three Amazon datasets, and use pre-caching to keep online inference fast.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline improvement over LC-Rec is under-supported because the metric protocol is unspecified and key hyperparameters are selected on the same Games test set used in Table 1.","rationale":"The paper attempts an incremental but plausible LLM recommendation method. The central claim is concrete and checkable, and the tables are internally consistent; I do not see a mathematical flaw in the tokenization or a missing derivation that would make the method impossible. The weakest link is the empirical comparison. The reader's weakest_assumption correctly identifies the metric candidate-set ambiguity and the same-dataset hyperparameter selection, and I agree with that assessment. The paper's own ablations show that the disputed choices matter: HR@5 in Table 4 ranges from 0.0791 to 0.0871 across negative sampling ratios, and HR@5 in Table 5 ranges from 0.0700 to 0.0858 across codebook counts, while the reported Ours-vs-LC-Rec HR@5 gap on Games is 0.0103. Thus the tuning protocol can move results by a substantial fraction of the claimed improvement. Without a held-out split or a fixed evaluation protocol, the reported margins are not falsifiable from the text. This does not make the result false; it makes it under-supported. A conditional acceptance that requires code, exact candidate-set definitions, significance or error bars, and validation-based hyperparameter selection is the right level. The paper explicitly acknowledges limitations about inference caching and the small scale of the scaling-law experiments, which further supports tempering the broader claims, but those are secondary to the evaluation-protocol concern. I would keep the reader's CONDITIONAL verdict, so no verdict adjustment is needed.","tokens_in":12813,"tokens_out":6695,"duration_ms":66013,"concrete_test":"Request the exact evaluation script and run one reproduction: define the candidate set identically for all methods (e.g., full-corpus scoring for non-generative baselines and the same beam-width top-K valid code sequences for Ours and LC-Rec, or a fixed sampled candidate set of size M for all methods), choose C and the negative sampling ratio on a validation split of Games that excludes the test users, then evaluate once on the held-out test users and report bootstrap confidence intervals. If, under this protocol, Ours's HR@10 advantage over LC-Rec on Games is no longer the reported +9.17% (0.1167 vs 0.1274), or the confidence intervals overlap, the central claim is not supported by the current tables.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is an empirical comparison: Ours beats LC-Rec on HR@K and NDCG@K on three Amazon datasets. For that comparison to be valid, two conditions must hold: the HR/NDCG numbers must be computed under a protocol that is identical and unbiased across methods, and the hyperparameters behind the reported numbers must not have been selected on the same test set used for the headline. Neither condition is documented. The Metrics paragraph states only the K values; it never says whether the ground-truth item is ranked against the full item corpus, a sampled candidate set, or the beam-search output of each LLM. Because Ours and LC-Rec generate item-code sequences via beam search while SASRec/BERT4Rec/DCCF score the full item space, the candidate sets can differ in size and composition; a hit rate computed against a short generated list is not directly comparable to one computed against full-corpus top-K ranking unless the evaluation code is exactly aligned. Separately, Tables 4 and 5 choose the negative sampling ratio and number of codebooks on the Games dataset by comparing test metrics, and the same Games rows appear as the headline result in Table 1. If those choices were made by looking at Table 1 itself, part of the +15.77% HR@1 gain on Games is selection on the test set. The paper reports no held-out validation split, no error bars, and no significance tests, so it is impossible to tell how much of the gap is protocol or selection artifact rather than method effect. This concern is about the evidence supporting the central claim, not about the method's plausibility; the cascaded-codebook plus negative-sampling idea can be sound while still lacking the support needed to establish the claimed margin.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage framework for LLM-based recommendation. In the first stage, Alignment Tokenization maps item embeddings (from DCCF) into a compact sequence of discrete codes drawn from cascaded codebooks, with an additional loss aligning the code embeddings with the LLM's input embedding space. In the second stage, Alignment Task fine-tunes Llama-7B on several instruction-style tasks (sequential, text, query) plus a negative-sampling task, using the learned code indices as item tokens. For inference, the paper pre-caches the top-K beam-search outputs per user to reduce online latency. The main empirical claim is that the method outperforms LC-Rec and other baselines on Amazon Games, Arts, and Instruments in HR@K and NDCG@K, with relative gains over LC-Rec ranging from +3.25% to +15.77%.","tokens_in":13075,"tokens_out":5878,"duration_ms":51790,"significance":"The work addresses a genuine problem in LLM-based recommendation, namely representing large item vocabularies as discrete tokens that are learnable by an LLM, and it does so with a clear, well-motivated design. The ablation studies (Table 3) and the 'behavior-only' control (Table 8) are useful and help isolate the contributions of the alignment loss and negative sampling. The pre-caching inference strategy is a practical step toward deployment. However, the empirical claims are not yet supported by the evidence as presented: the evaluation protocol is underspecified, the hyperparameters appear to be tuned on the same Games dataset used for the headline result, and no statistical significance is reported. If these issues are resolved, this could be a solid contribution; as it stands, the central comparison is not fully credible.","major_comments":[{"comment":"The paper never specifies how HR@K and NDCG@K are computed over the item corpus. It is essential to state whether the ground-truth item is ranked against the full item set, a sampled candidate set, or only the items produced by each method's own generation/beam-search procedure. Since LC-Rec and Ours decode item-code sequences via beam search while SASRec/BERT4Rec/DCCF score all items, the candidate sets can differ in size and composition. Without a common evaluation protocol, the numbers in Table 1 are not directly comparable across methods, and the headline claim of improvement over LC-Rec is under-supported. Please specify the protocol precisely and, if possible, provide evaluation code or a concrete example.","section":"Experiment Setup, Metrics"},{"comment":"The negative sampling ratio (1:3) and the number of codebooks (4) are selected by comparing test metrics on the Games dataset (Tables 4 and 5), and the same Games dataset is used as the first block of Table 1. No validation split is described. If these hyperparameters were chosen based on test-set performance, the reported Games improvements (e.g., +15.77% HR@1) are at least partly an artifact of test-set selection. Please clarify whether a validation split was used, or re-run the selection on a hold-out set and report the resulting performance for all datasets.","section":"Ablation Study, Tables 4 and 5"},{"comment":"The reported differences between Ours and LC-Rec are small for several metrics (Instruments HR@1 +3.25%; Arts HR@5 +4.20%). The paper reports no error bars, standard deviations, or significance tests. It is therefore unclear whether the observed improvements are statistically meaningful or within run-to-run variance. Please report multiple independent runs with standard errors, or at least a paired significance test (e.g., bootstrap or t-test) for the main comparison in Table 1.","section":"Experiment Results, Table 1"}],"minor_comments":[{"comment":"The notation is inconsistent: in the Method section and Eq. (3), N denotes the number of codebook levels and C denotes the number of codes per level, with N=4 and C=256. In Table 5, however, C denotes the number of codebooks. This makes Table 5 hard to interpret; please use disjoint symbols for the two quantities.","section":"Method, Alignment Tokenization; Table 5"},{"comment":"The text states that 'our method shows a slight improvement compared to \"w/o AM\"', but Table 3 does not include a 'w/o AM' row; it includes 'ED', 'w/o LA', and 'w/o NS'. This appears to be a typo for 'w/o LA'. Please correct the text or the table.","section":"Ablation Study"},{"comment":"There are several typos, including 'effciency' in the abstract and 'negtive sampling' in the Ablation Study section. Please proofread the manuscript.","section":"Abstract and Ablation Study"},{"comment":"The paragraph claiming that new items can be handled without retraining the LLM or codebooks ('mitigating the cold start problem') is not supported by any experiment. Please either add a small experiment demonstrating cold-start capability or soften the claim.","section":"Inference, cold-start discussion"},{"comment":"The paper does not report training hyperparameters for the LLM fine-tuning stage, such as LoRA rank, learning rate, batch size, number of epochs, or hardware used. Including these details would improve reproducibility.","section":"Experiment Setup"},{"comment":"The inference prompt is written as 'The user has interacted with <itemi>,... in chronological order . Can you predict the next possible item that the user may expect?' with an extra space before the period; the surrounding sentence also contains 'which mentioned in LC-Rec'. Please correct the grammar and punctuation.","section":"Inference"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising idea and a clear method, but the evaluation section requires substantial strengthening before the claims can be accepted. The most serious issue is the unspecified evaluation protocol, which could invalidate the entire comparison if the candidate sets differ across methods. The apparent test-set tuning on the Games dataset is also a major concern. I would encourage the editor to ask for a precise description of the evaluation protocol, a validation split for hyperparameter selection, and error bars or significance tests. Additionally, the paper uses DCCF embeddings as input to the tokenizer while also listing DCCF as a baseline; the authors should discuss this dependency explicitly so readers can judge the comparison fairly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the short version: this is a competent, incremental extension of LC-Rec, and the central claim—that the two-stage alignment plus negative sampling beats LC-Rec on three Amazon datasets—is probably correct. But the paper doesn't currently make that case convincingly enough to trust the extra few percentage points. The stress-test note is right: the metrics paragraph never specifies whether HR/NDCG are computed over the full item corpus or a sampled candidate set. Since LLM-based methods decode from a constrained set of valid item-code sequences while traditional baselines score the full item space, the comparison could be apples to oranges unless the evaluation code is exactly aligned.\n\nWhat's genuinely good: the method is a clean variant of LC-Rec—drop the RQ-VAE auto-encoding loss, add an LLM alignment loss on codebook embeddings, and inject negative sampling. The ablations show each piece earns its keep, and the pre-caching inference scheme addresses a real practical bottleneck. The reported gains are consistent across three datasets, which is encouraging.\n\nThe soft spots, in order of severity: (1) the metric protocol is unspecified; (2) key hyperparameters (negative sampling ratio 1:3, four codebooks) are selected on the Games dataset, and the same Games rows appear as the headline result in Table 1—without a held-out split, some of the +15.77% HR@1 gain may be selection on the test set; (3) no error bars or significance tests, so the 3-5% gains on Arts and Instruments could be noise; (4) the cold-start claim in the inference section is not directly tested, and the \"scaling law\" is inferred from two model sizes; (5) no code release. None of these alone sinks the paper, but together they leave the central quantitative claim under-supported.\n\nBottom line: send it to a serious referee, but the reviewers should require code, the exact candidate-set definition, and a held-out split for hyperparameter selection. The idea is worth engaging with; the specific numbers aren't yet. I'd also ask the authors to temper the \"remarkable scalability\" language in the abstract and to be explicit that Tables 4 and 5 are model selection, not final results.","headline":"A clean incremental follow-up to LC-Rec with a plausible but under-supported headline gain; the evaluation protocol and hyperparameter selection need to be pinned down before the numbers can be trusted.","tokens_in":13738,"tokens_out":2694,"would_cite":true,"duration_ms":23755,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a two-stage alignment framework, combining cascaded codebook tokenization with alignment fine-tuning and negative sampling, makes an LLM-based recommender outperform the previous best LLM-based method, LC-Rec, on…","keywords":["large language models","recommender systems","alignment tokenization","cascaded codebooks","negative sampling","sequential recommendation","semantic alignment"],"falsifier":"An independent re-run of the Games experiment with full-corpus ranking (scoring all 16,859 items, not only items whose codes appear in the beam-search output) and with the codebook count and negative-sampling ratio chosen on a validation split would settle whether the reported gains over LC-Rec are genuine.","tokens_in":12535,"feed_emoji":"🎯","tokens_out":6528,"duration_ms":51298,"temperature":0.7,"pith_summary":"This paper proposes a two-stage framework that lets large language models (LLMs) act directly as recommenders without being overwhelmed by large item catalogs. The first stage, Alignment Tokenization, compresses item IDs into short sequences of shared, discrete codes using cascaded codebooks, and aligns those codes with the LLM's input embedding space. The second stage, Alignment Tasks, fine-tunes the LLM on prompts that mix user behavior, item text, and queries, including a negative-sampling task that imitates the generalization behavior of traditional recommenders. The authors claim the resulting system beats the previous best LLM-based recommender, LC-Rec, on all three Amazon datasets (Games, Arts, Instruments), with relative HR@1 gains from 3.25% to 15.77% and similar NDCG improvements. The framework also pre-caches per-user top-K item codes at inference time, making online deployment practical.","feed_headline":"Two-stage alignment boosts LLM recommendations by up to 16%","feed_subtitle":"Four shared codebooks plus negative-sampling make Llama-7B beat the previous best LLM recommender.","key_machinery":"The load-bearing object is the Alignment Tokenization module: a cascaded set of four codebooks, each with 256 codes, that maps an item's behavioral embedding (from a contrastive collaborative filtering model) into a four-token index sequence via residual quantization. The first codebook captures the coarsest semantics and each subsequent codebook approximates the residual, so related items share codes. A second, LLM alignment loss penalizes the distance between each chosen code and the mean-pooled LLM embedding of the item's title and description, pulling the code space toward the LLM's semantic space. The other half of the mechanism is the Alignment Task suite: sequential, text, and query alignment prompts plus a negative sampling task, all used to fine-tune the LLM with cross-entropy loss. Together these turn LLM fine-tuning into a tractable ranking problem over a compact, shared vocabulary.","core_discovery":"The central claim is that the gap between sparse collaborative semantics and dense LLM token representations can be closed by a two-stage alignment. In the paper's telling, representing each item as four codes drawn from four cascaded codebooks (256 codes per book) makes the LLM vocabulary compact enough to fine-tune, while an LLM alignment loss during codebook training pre-synchronizes the code space with the LLM's embeddings. A set of supervised alignment tasks, namely sequential prediction, text-to-item translation, query-to-item matching, and a deliberately added negative-behavior task, then teach the LLM to combine behavioral and textual signals. The paper reports that this approach yields top Hit Ratio and NDCG on three Amazon review datasets, improving on LC-Rec by 3.25% to 15.77% depending on metric and dataset. The same tokenization also supports practical inference by pre-caching beam-searched top-K codes per user.","pith_inferences":["The reported superiority over LC-Rec may partly reflect a comparison set: if the metrics rank only items whose code sequences appear in the beam-search output, traditional baselines that score the full corpus are at a disadvantage.","The token-sharing property of the codebooks suggests a cross-domain transfer test the paper does not run: items from an unseen domain could be tokenized with the same codebooks and fed to the fine-tuned LLM to probe whether the alignment generalizes.","The ablation tunes negative-sampling ratio and codebook count on Games and then reports Games as the main result; validating those choices on a held-out split would give a stricter estimate of the gains.","Because the inference cache stores per-user top-K codes, a user's evolving interests between cache refreshes are not reflected; the paper notes this as a storage and latency constraint but leaves hybrid strategies (cache plus vector retrieval for new items) untested."],"forward_implications":["LLM-based recommenders can handle industrial-scale item pools because shared codes keep new-token counts small: 16,859 items fit in 1,024 new tokens with zero collisions.","Adding an LLM alignment loss during tokenization improves downstream recommendation accuracy, since the code space starts closer to the LLM's representation space.","Negative sampling during LLM fine-tuning adds training signal and mitigates sample selection bias, as shown by the ablation that removes it.","Larger LLMs scale recommendation quality: Llama-13B outperforms Llama-7B, suggesting LLM-based recommenders inherit a scaling law.","Pre-caching top-K beam-search codes per user makes online inference fast, and new items can be added by tokenizing them without retraining the LLM or codebooks."],"supporting_citations":[{"why":"LC-Rec, the strongest baseline and the method whose quantization approach and inference prompt this work extends; the paper's headline claim is beating it.","marker":"Zheng et al. 2023"},{"why":"DCCF, the behavior-only contrastive model whose item embeddings feed the Alignment Tokenization codebooks.","marker":"Ren et al. 2023"},{"why":"RQ-VAE cascaded codebooks, the inspiration for the residual quantized tokenization, which this work adapts by dropping the auto-encoding objective.","marker":"Rajput et al. 2024"},{"why":"Llama-7B, the base LLM used for all fine-tuning experiments.","marker":"Touvron et al. 2023"},{"why":"Amazon review datasets (Games, Arts, Instruments) that provide the interaction sequences, titles, and descriptions used for evaluation.","marker":"Ni, Li, and McAuley 2019"},{"why":"P5, a unified recommendation-as-language-processing baseline that must be outperformed; it is among the comparison methods in Table 1.","marker":"Geng et al. 2022"}],"fun_headline_variants":["Two-stage alignment lifts LLM recommender recall by 16%","Tokenizing items as codes closes LLM-recommender gap","LLM recommender: dual alignment yields up to 16% gain","Behavioral tokenization aligns LLMs with collaborative semantics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains rest on the evaluation being fair and transferable: the paper does not state how HR/NDCG are computed over the item corpus (full ranking vs. ranking within a candidate set), and the negative-sampling ratio and number of codebooks are tuned on the Games dataset that later appears as the headline result.","fun_headline_variants_meta":{"raw":{"variants":["Two-stage alignment lifts LLM recommender recall by 16%","Tokenizing items as codes closes LLM-recommender gap","LLM recommender: dual alignment yields up to 16% gain","Behavioral tokenization aligns LLMs with collaborative semantics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000876,"raw_usage":{"total_tokens":3765,"prompt_tokens":897,"completion_tokens":2868,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":2796}},"tokens_in":513,"tokens_out":2868,"duration_ms":20085,"temperature":1.0,"reasoning_tokens":2796,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:49:23.275607+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent re-run of the Games experiment with full-corpus ranking (scoring all 16,859 items, not only items whose codes appear in the beam-search output) and with the codebook count and negative-sampling ratio chosen on a validation split would settle whether the reported gains over LC-Rec are genuine.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DCCF, the behavior-only contrastive model whose item embeddings feed the Alignment Tokenization codebooks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"P5, a unified recommendation-as-language-processing baseline that must be outperformed; it is among the comparison methods in Table 1."}],"review_version":1}