{"id":"3da645d4-21bc-4085-ab4f-46e706ad59b7","arxiv_id":"2412.18176","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Molar combines multimodal LLM item embeddings with a contrastive post-alignment to an ID-based sequential recommender, reporting improved NDCG and Recall on three datasets.","lead":"Molar is a recommendation framework that encodes item images and text with a multimodal language model, then aligns the resulting content-based user profiles with collaborative-filtering signals from an ID-based model. The authors report consistent gains over LLM-based baselines on Amazon, PixelRec, and MovieLens sequential recommendation benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation protocol is underspecified: 'full amount of data' does not state the candidate item set, so the headline empirical claim is not independently verifiable; the reported p-value lacks run counts.","rationale":"We read the paper in good faith. The architecture is plausible: MIRM produces multimodal item embeddings, DUEG aggregates them into user embeddings, and a contrastive post-alignment loss ties the content-based user embedding to an ID-based SASRec embedding. The ablation in Table 5 shows that removing the alignment loss degrades performance, which is evidence that the mechanism does something. However, the paper's own headline is empirical superiority over eight baselines. That claim rests entirely on Table 2. Our review of the experimental section found that the evaluation protocol is underspecified in a way that directly affects every number in Table 2. 'Evaluated on the full amount of data' is not a standard description of a ranking protocol; standard terminology is 'full ranking' over all items. Given the dataset sizes, full ranking is infeasible for Amazon without heavy engineering, yet no sampling or ANN procedure is documented. If the authors used negative sampling for evaluation, the metrics are not comparable across methods unless the exact same candidate set and random seed are used. The paper gives no evidence of that. The missing error bars compound the issue: the p-value claim cannot be checked without per-run results. The reader's 'weakest_assumption' focused on the alignment mechanism, but the empirical claim is even more fragile at the measurement level. We therefore propose the concrete check above: inspect the code and recompute a small dataset with full ranking. If the protocol turns out to be sound, the concern evaporates. Since the paper is a preprint with anonymous code, the right verdict is CONDITIONAL: the authors should specify the evaluation protocol, release the evaluation script, and report variance. We do not see a reason to change the reader's CONDITIONAL verdict, but we believe this concern should be stated explicitly as a condition.","tokens_in":12582,"tokens_out":6033,"duration_ms":52377,"concrete_test":"Inspect the anonymous repository (Molar-8B06) and locate the evaluation script. Determine how the candidate set is built: all items, a fixed random sample, or an ANN index. Then, on MovieLens (3,706 items), re-run the Molar evaluation under the published split using full ranking and compare NDCG@10 to the reported 0.1768. If the code uses sampling, recompute with 3 random seeds, report mean±std, and state whether the same candidate set was used for CoLLM and HLLM. If the script is absent or the candidate set is not specified, the central comparison is unverifiable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical (Section 4.2, Table 2). For it to hold, the reported NDCG/Recall numbers must be computed under a well-defined, reproducible evaluation protocol. Section 4.1.1 says metrics are 'evaluated on the full amount of data' but never specifies the candidate item set used for ranking. On Amazon (301,312 items, 993,087 users), full ranking with a 2B-parameter MLLM is computationally prohibitive, so the evaluation almost certainly uses a sampled candidate set or an approximate nearest-neighbor index. Neither is described in the paper or appendix. If the candidate set is sampled (e.g., 100 or 1000 negatives per user), the metrics are biased and may not be comparable to CoLLM/HLLM baselines unless they use the identical sampling process and seed. The paper reports no standard deviations, no number of runs, and no seed, while Table 2 marks results with 'p-value << 0.05'. Appendix B describes 1:1 negative sampling only for training loss, not for evaluation. Thus the headline claim 'Molar consistently outperforms all baseline models across three datasets' cannot be verified from the manuscript alone. This is more load-bearing than the missing mechanism analysis of post-alignment, because a flaw in the evaluation protocol would invalidate the empirical basis of the whole contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Molar, a multimodal LLM-based sequential recommendation framework that decouples item and user modeling. A Multimodal Item Representation Model (MIRM), built on Qwen2-VL, encodes each item's text and image into a compact embedding; a Dynamic User Embedding Generator (DUEG) builds a user embedding from the sequence of item embeddings; and a post-alignment contrastive loss aligns this content-based user embedding with an ID-based user embedding produced by a traditional sequential model such as SASRec. Training combines a point-wise BCE next-item loss with the alignment loss. Experiments on Amazon, PixelRec, and MovieLens report consistent improvements over traditional, content-based, and LLM-based baselines, with ablations on DUEG choice, input modalities, post-alignment models, fine-tuning data, and MLLM backbone.","tokens_in":12905,"tokens_out":2701,"duration_ms":27615,"significance":"If the reported results are reproducible, the paper makes a useful empirical contribution: it demonstrates that a post-alignment architecture, in which ID-based collaborative signals are aligned with content-based user embeddings only after multimodal LLM processing, can outperform early-fusion LLM recommenders such as CoLLM and HLLM. The decoupled design also has a practical efficiency motivation, and the ablation study is fairly complete, covering fine-tuning objectives, modality inputs, DUEG variants, and post-alignment models. The claimed gains are meaningful (roughly 4-8% relative over strong LLM baselines). However, the paper's central claim is entirely empirical, and the evaluation protocol is currently underspecified to the point that the headline numbers cannot be independently verified; this is the main barrier to accepting the paper. The authors also provide an anonymous code link, which is a positive step, but without a commit hash or configuration details it does not yet constitute reproducible artifacts.","major_comments":[{"comment":"The evaluation protocol is not fully specified. The text states that NDCG and Recall are 'evaluated on the full amount of data,' but it never states the candidate item set used for ranking. On Amazon, with 301,312 items, full-ranking evaluation with a 2B-parameter MLLM is computationally prohibitive, so the evaluation almost certainly uses sampled negatives, a truncated candidate set, or an approximate nearest-neighbor index; none of these is described. The reported metrics are therefore not independently verifiable, and unless the baselines use the identical candidate-generation process and seed, cross-method comparisons in Table 2 are not valid. This is load-bearing because the paper's central claim is that Molar 'consistently outperforms all baseline models across three datasets.' The authors must specify the exact candidate set construction, including the number of negatives, the sampling distribution, the seed, and whether the same protocol is applied to every baseline.","section":"§4.1.1 and Table 2"},{"comment":"The paper reports 'p-value << 0.05' in Table 2 but provides no number of runs, no standard deviations, no confidence intervals, and no description of the statistical test used. A significance claim of this strength requires at least three to five independent runs with seeds stated, and ideally paired tests across baselines. Without this information, the reader cannot distinguish real improvements from noise, especially for the small absolute differences on PixelRec (e.g., N@10 0.0197 vs. 0.0189 for HLLM). Please report run counts, standard deviations, and the exact test.","section":"Table 2 and §4.2"},{"comment":"The DUEG architecture is described only as removing the word embedding layer from the MLLM and adding a [User] token. This is insufficient to reproduce the method: the paper does not state the number of remaining transformer layers, the initialization of the [User] token, the output projection, or how the predicted logits are computed for the BCE loss. Additionally, Appendix B describes a 1:1 negative-sampling strategy for training, which is reasonable, but the relationship between this train-time sampling and the claimed full-ranking evaluation is not explained; if evaluation uses a different candidate set, the train/eval mismatch should be acknowledged and analyzed.","section":"§3.4 and Appendix B"},{"comment":"The reproducibility of the experiments is currently incomplete. The anonymous code link has no commit hash or version identifier, and the paper does not report the values of several free parameters that affect the results: the alignment loss weight alpha, the temperature tau, the number of contrastive negatives K, the number of BCE negatives beyond the stated 1:1 pairing, the sizes of the three fine-tuning data mixtures beyond '10,000 data points,' and the LoRA configuration used in Appendix A. Please provide a configuration table or a tagged code release so that the reported numbers can be reproduced.","section":"§4.1.2 and code availability"},{"comment":"The paper motivates post-alignment by claiming it 'preserves collaborative signals better than early ID fusion,' but it provides no mechanism-level analysis or diagnostic experiments supporting this specific claim. The ablation in Table 5 shows that removing the alignment loss hurts performance, which is evidence that the loss helps, but it does not test whether the benefit comes specifically from post-alignment rather than from any auxiliary ID-based signal, nor does it compare directly against an early-fusion variant under matched conditions. A small experiment with early ID fusion using the same MIRM and DUEG would substantially strengthen the paper's central conceptual claim.","section":"§3.4 and Figure 1"}],"minor_comments":[{"comment":"The Molar row contains a typo: '01478' under Amazon N@20 should be '0.1478'.","section":"Table 2"},{"comment":"Figure 3 reports comparisons of DUEG variants but provides no numeric values, error bars, or dataset labels in the caption; please add a table or axis-visible values.","section":"Figure 3"},{"comment":"The text says 'As shown in Table 5' when referring to the backbone comparison, but the table is numbered Table 6; please fix the cross-reference.","section":"Appendix A"},{"comment":"The phrase 'over a 7% improvement' should state explicitly whether this is a relative or absolute improvement, and over which baseline and metric, since the values in Table 2 do not make this unambiguous.","section":"§4.2"},{"comment":"The phrase 'evaluated on the full amount of data' is ambiguous and should be replaced with a precise description of the evaluation candidate set, as discussed in the major comments.","section":"§4.1.1"},{"comment":"Some references are cited only by arXiv preprint versions (e.g., Devlin 2018, Dosovitskiy 2020); consider citing the published versions where available.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is potentially solid, and the post-alignment idea is worth publishing if the evaluation protocol is made rigorous. The main risk is not the method but the verifiability of the empirical claims: the candidate-set specification, run counts, and hyperparameter disclosure must be supplied before the numbers in Table 2 can be trusted. I would suggest asking the authors to release the exact evaluation code path, including candidate sampling, as part of the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: Molar's post-alignment idea is a genuine contribution. Instead of fusing ID embeddings into the LLM early (as in CoLLM and HLLM), it fine-tunes the MLLM on multimodal content, then aligns the resulting content-based user embedding with an ID-based user embedding via a contrastive loss. That is a real architectural difference, and the reported gains of roughly 4-8% relative over strong LLM baselines are plausible given the ablation. The ablations are internally consistent: removing any fine-tuning objective or the alignment loss degrades performance.\n\nThe main problem is the evaluation protocol. The paper states metrics are 'evaluated on the full amount of data.' On Amazon, with 301K items and a 2B-parameter MLLM backbone, full ranking is not credible; a sampled candidate set or approximate index must have been used, but neither is described. This matters because the headline 'consistently outperforms' claim depends on the candidate set being identical across methods and seeds. Without that detail the numbers are not independently verifiable. This is load-bearing, not cosmetic.\n\nSecondary issues: Table 2 reports p-value << 0.05 with no standard deviations, run counts, or seeds. Appendix B only describes the training-time negative sampling. DUEG's architecture is under-specified ('removing the word embedding layer from the MLLM' isn't enough to reproduce). The anonymous code link is unpinned. These are all addressable, but they need addressing before the empirical claims are solid.\n\nWhat the paper does well: the motivation for late alignment is clearly argued, the ablation study is informative, and Table 4 shows a useful trend across different ID-based aligners. The backbone comparison in Appendix A is also a practical addition.\n\nWho this is for: researchers working on LLM-based or multimodal sequential recommendation. The post-alignment design is worth trying in other pipelines.\n\nRecommendation: serious referee, but I would require the authors to state the candidate item set, sampling procedure, and number of runs before accepting the headline numbers.","headline":"Molar's post-alignment is a real new variant in LLM sequential recommendation, but the reported numbers can't be checked until the evaluation protocol is specified.","tokens_in":13406,"tokens_out":2904,"would_cite":true,"duration_ms":24188,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Molar claims a sequential recommender should fuse IDs after, not before, the multimodal LLM builds item representations.","keywords":["sequential recommendation","multimodal large language models","collaborative filtering","post-alignment","contrastive learning","item representation","user embedding","LLM-based recommendation"],"falsifier":"Run Molar and an early-fusion baseline (CoLLM or HLLM) on a dataset where item text and images are near-duplicate or uninformative (e.g., anonymized purchases with generic titles) so IDs carry most of the signal; if Molar's NDCG@10 gain over early fusion vanishes or reverses, the post-alignment advantage is not about alignment timing but about content quality. A cheaper check is to ablate the alignment loss while holding everything else fixed and see whether the remaining content-only model still beats early fusion.","tokens_in":12419,"feed_emoji":"🎯","tokens_out":5403,"duration_ms":42508,"temperature":0.7,"pith_summary":"The paper sets out to show that a sequential recommender can get the best of both worlds—multimodal content understanding and collaborative filtering—by keeping them separate until the end. It proposes Molar, which uses a multimodal LLM to turn text-plus-image item descriptions into compact embeddings, builds a user embedding from a history of those embeddings, and only then aligns that content-based user embedding with the user embedding from a classic ID-based sequential model via a contrastive loss. The claim is that this post-alignment fusion beats feeding ID tokens into the LLM early, as prior LLM recommenders do, and the experiments report consistent gains in NDCG and Recall over all baselines on Amazon, PixelRec, and MovieLens. If right, it means the practical recipe for LLM-based recommendation is to use the LLM purely as a multimodal item encoder and add collaborative signals at the user-representation level, not earlier.","feed_headline":"Post-aligned IDs lift multimodal LLM recommenders by up to 8%","feed_subtitle":"Contrastive alignment with a classic SR model beats early-fusion LLM baselines on three datasets.","key_machinery":"The load-bearing mechanism is the post-alignment contrastive loss $L_{align}$ (Eq. 4), a symmetric InfoNCE-style objective that aligns each content-based user embedding $E^{con}_u$ from the multimodal LLM path with the ID-based user embedding $E^{id}_u$ from a traditional sequential recommender, trained jointly with a pointwise BCE next-item loss $L_{bce}$. This is what lets Molar keep the MLLM's multimodal semantics while importing collaborative-filtering structure at the user-embedding level rather than at the token level.","core_discovery":"Molar's central claim is that collaborative-filtering signals should be injected into an LLM-based sequential recommender after, not before, the LLM has built its content representations. The framework first fine-tunes a multimodal LLM (Qwen2-VL) as a Multimodal Item Representation Model, extracting an item embedding from a special [Cur_Item] token over a description that fuses text, image, and structured attributes. A Dynamic User Embedding Generator, also built on the LLM backbone, consumes the sequence of item embeddings and emits a content-based user embedding from a [User] token. A traditional sequential model (SASRec or DuoRec) trained on the same interaction sequences emits an ID-based user embedding, and a symmetric contrastive loss pulls the two user representations together. The experiments report that this design outperforms LLM baselines that concatenate ID embeddings with text early in the LLM input (CoLLM, HLLM), content-only variants, and classical sequential models on all three datasets, with the largest relative gains on NDCG-type metrics.","pith_inferences":["The paper argues that early ID fusion 'can cause LLMs to learn shortcuts,' but it does not measure shortcut behavior directly; a natural test is to compare how much Molar's content and ID user embeddings diverge on users with long vs cold-start histories.","If the post-alignment benefit is really about preserving collaborative structure, then Molar should transfer to non-LLM user encoders and to other sequence tasks (e.g., click-through prediction), providing a cheap way to retrofit multimodal understanding into existing recommender stacks.","The reported gains are on full-corpus evaluation of next-item prediction; on candidate-ranking or re-ranking settings where negative sampling differs, the relative advantage may shrink, which the paper does not test.","Molar's advantage likely depends on item content being genuinely informative; on datasets where IDs dominate (e.g., anonymous sessions with little metadata), the content path may add noise and the alignment loss would then dilute ID signal."],"forward_implications":["Molar reports consistent improvements over the strongest baselines on all three datasets, e.g., over 7% relative gains in NDCG/Recall on MovieLens, meaning late ID alignment is a viable alternative to early fusion in LLM recommenders.","The decoupled design (item encoder MIRM, user encoder DUEG) means user histories are compressed to embeddings rather than long token sequences, reducing inference cost relative to feeding full histories to an LLM.","The post-alignment module can use any traditional sequential model; stronger models such as DuoRec yield better final performance, so the method scales with improvements in ID-based recommenders.","Multimodal input (image + text) outperforms either modality alone on PixelRec, indicating the MLLM fusion is the source of a substantial part of the gain."],"supporting_citations":[{"why":"CoLLM is the early-fusion LLM baseline that Molar must beat to justify late alignment.","marker":"Zhang et al., 2023b"},{"why":"HLLM is the hierarchical LLM baseline that early-fuses IDs and is the strongest LLM competitor in Table 2.","marker":"Chen et al., 2024"},{"why":"SASRec is the ID-based sequential model used in post-alignment and a primary traditional baseline.","marker":"Kang and McAuley, 2018"},{"why":"DuoRec supplies the contrastive-learning baseline and, when swapped in as the ID model, shows stronger post-alignment yields better results.","marker":"Qiu et al., 2022"},{"why":"Qwen2-VL is the multimodal LLM backbone used for both MIRM and DUEG.","marker":"Wang et al., 2024"},{"why":"ViT is the image encoder in the SASRec-Content baseline that Molar's fused multimodal embeddings outperform.","marker":"Dosovitskiy, 2020"},{"why":"BERT is the text encoder in the SASRec-Content baseline used to show the value of unified MLLM fusion.","marker":"Devlin, 2018"}],"fun_headline_variants":["Post-aligned CF signals boost multimodal LLM recommenders","Molar: align after content, not before, for better recs","Late CF alignment beats early-fusion in LLM recommenders","Multimodal LLM + post-aligned IDs: new SOTA in SR","Molar's post-alignment lifts multimodal sequential recs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that aligning content-based and ID-based user embeddings after the multimodal LLM has processed the content preserves collaborative-filtering signals better than feeding ID tokens into the LLM early; if that late-alignment benefit does not generalize beyond these three datasets and this training setup, Molar's edge over CoLLM and HLLM would disappear.","fun_headline_variants_meta":{"raw":{"variants":["Post-aligned CF signals boost multimodal LLM recommenders","Molar: align after content, not before, for better recs","Late CF alignment beats early-fusion in LLM recommenders","Multimodal LLM + post-aligned IDs: new SOTA in SR","Molar's post-alignment lifts multimodal sequential recs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000694,"raw_usage":{"total_tokens":3162,"prompt_tokens":990,"completion_tokens":2172,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":2082}},"tokens_in":606,"tokens_out":2172,"duration_ms":13808,"temperature":1.0,"reasoning_tokens":2082,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:57:15.989150+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Molar and an early-fusion baseline (CoLLM or HLLM) on a dataset where item text and images are near-duplicate or uninformative (e.g., anonymized purchases with generic titles) so IDs carry most of the signal; if Molar's NDCG@10 gain over early fusion vanishes or reverses, the post-alignment advantage is not about alignment timing but about content quality. A cheaper check is to ablate the alignment loss while holding everything else fixed and see whether the remaining content-only model still beats early fusion.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DuoRec supplies the contrastive-learning baseline and, when swapped in as the ID model, shows stronger post-alignment yields better results."}],"review_version":1}