{"id":"755f73ab-e099-4a14-930e-9a2fa463abbb","arxiv_id":"2501.15183","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"NegGen masks and replaces key attributes of items via a multi-modal LLM, generating hard negative descriptions, and uses a contrastive causal module to improve multi-modal recommendation across four Amazon datasets.","lead":"This paper proposes NegGen, a framework that uses a vision-language model to rewrite product descriptions into harder 'negative' training examples for multi-modal recommender systems. A smart generalist might read it to see whether frozen large language models can cheaply improve recommendation quality by generating better supervision signals.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"NegGen's gains may hinge on false negatives: per-item generated negatives are used for all interacting users in Eq. 19 with no per-user filtering or false-negative audit, so the headline improvement could reflect demoting items some users actually prefer.","rationale":"The reader identified the same weakest assumption: the generated negative for item i is treated as a valid negative for every user who interacted with i, with no per-user filtering or false-negative analysis. Their central claim and reasoning are sound. My attack adds a concrete mechanism for why this matters in the loss: the same e*_c enters Eq. 19 for all users, and the causal module's e_t = ~e_m - ~e*_m means a user who likes the generated variant gets a negative causal score for the true positive, actively demoting it. I also independently note that the causal framing (Eqs. 10-12) is not implemented as causal inference; the module is a contrastive subtraction, which is a separate unsupported claim. Both issues are empirical and testable. I do not see a more load-bearing concern: the method could still work as a hard-negative sampler even if the causal interpretation is wrong, but the false-negative risk is the one that could invalidate the headline empirical claim. The paper does not provide released code, error bars, or final hyperparameter values; those are reproducibility gaps, not correctness gaps. The reader's conditional verdict is appropriate: the central empirical claim is plausible but not yet established, and the causal claim is overstated relative to the math.","tokens_in":18460,"tokens_out":2102,"duration_ms":17412,"concrete_test":"Run a per-user validity audit on the generated negatives. Sample 500 (user, positive item, generated negative) triples from the test (or validation) users; be sure that each triple comes from a user who actually interacted with the positive item. For each triple, ask a separate MLLM (or human annotators) to classify whether the generated attribute is (a) clearly worse for that user, (b) roughly equivalent, or (c) possibly preferred. Then measure the fraction of (b)+(c) triples and re-train NegGen after excluding them. If excluding them causes a performance drop on Recall@10/NDCG@10, the gains come from reliance on false negatives; if performance is retained or improves, the claim survives. Separately, re-derive Eq. 19 from Eqs.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that NegGen's MLLM generate-mask-complete pipeline plus causal module outperforms all MMRS and negative samplers. The weakest load-bearing point is the validity of the generated negative for each user. In Section 3.2, negatives are generated per item from the item's description, independent of the user. In Eq. 19, that same e*_c is used in the BPR loss for every user who interacted with positive item i. There is no per-user filtering and no false-negative analysis. If a generated attribute is actually preferable to the user (e.g., a different color of a product the user already owns, or a complementary accessory), the \"negative\" descriptor is false for that user and the model is trained to demote an item the user would want. The paper's Proposition 2 also assumes negatives are truly negative: the NDCG lower bound is only meaningful when e*_i is a valid negative. Furthermore, the causal module of Section 3.3 computes e_t = ~e_m - ~e*_m (Eq. 16) and then, in Eq. 19, y = e_u^T(e_c - e*_c). For a user who prefers the modified attribute, this term is negative on average for the positive item, pushing the score of the true positive down. The paper reports no human evaluation of generated negatives, no false-negative rate, and no robustness check against MLLM generation errors. The ablation in Table 5 cannot separate generation quality from this false-negative effect; it only shows that removing either module hurts. Finally, the causal language is not supported by the math: the do-operator equations (Eqs. 10-12) are never instantiated with causal adjustment; the implemented module is a contrastive subtraction of projected embeddings, which does not estimate a total causal effect under the stated SCM because no intervention, confounding control, or backdoor adjustment is performed. This makes the \"causal learning\" label an unsupported interpretation of a standard contrastive term.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NegGen, a framework for negative sampling in multi-modal recommendation. NegGen uses a multi-modal large language model (MLLM) in three steps to generate counterfactual item descriptions (description generation, attribute masking, attribute completion), encodes them with a text encoder, and then combines a LightGCN base recommender with a 'causal learning' module that computes a difference between positive and negative attribute embeddings. The combined model is trained with a BPR-style loss and an alignment loss. Experiments on four Amazon subsets (Baby, Beauty, Clothing, Sports) report consistent improvements over several negative-sampling and multi-modal baselines, with paired t-tests.","tokens_in":18825,"tokens_out":10103,"duration_ms":85650,"significance":"The empirical results are potentially useful: the idea of using MLLMs to generate hard, modality-balanced negatives is timely, and the consistent gains across datasets (e.g., 9.02% R@10 on Baby) are non-trivial. The paper provides ablations and hyperparameter sensitivity, which support the reproducibility of the main pipeline. However, the claimed causal contribution is not supported by the evidence, and the validity of generated negatives as true negatives is not established. If these issues are addressed, the paper could be a solid contribution to multi-modal recommender systems.","major_comments":[{"comment":"The generated negative e*_c is produced per item from the item's attributes, independent of the target user, and is then used in the BPR loss for every user u who interacted with i. This design assumes the counterfactual description is a valid negative for each such user, but the paper provides no per-user filtering, no human evaluation of generated negatives, and no analysis of the false-negative rate. Since the headline improvements in Table 2 rest on this negative-sampling module, the claim that NegGen generates high-quality negatives (RQ3) is not yet established; the ablation in Table 5 cannot separate generation quality from false-negative contamination.","section":"Section 3.2 and Eq. (19)"},{"comment":"The paper labels e_t = e_m_tilde - e*_m_tilde as an instantiation of the 'total causal effect' (TE). However, Eq. (12) is simply the definition of a difference between two potential outcome values under a deterministic SCM, and Eq. (16) is a difference of two learned embeddings. No identification step, no backdoor adjustment, and no do-calculus operation is performed; the SCM in Figure 5 is assumed without unobserved confounding. Therefore the claim that the module 'disentangle[s] the effect of intervened key features and irrelevant item attributes' is unsupported. The module may still be a useful contrastive feature, but it should be framed as such, not as causal inference.","section":"Section 3.3, Eqs. (12)–(16)"},{"comment":"The text states that e_c is 'the positive embedding of item i' and e*_c is 'the corresponding negative embedding', but Eqs. (17)–(18) define e_c as a projection of e_t = e_m_tilde - e*_m_tilde, i.e., a difference embedding, while e*_c is the projection of the negative attribute embedding. This discrepancy makes the training objective ambiguous: it is unclear whether the causal score is a difference term or an item-level positive score. The alignment loss in Eq. (20) suffers from the same ambiguity. The authors should correct the notation and clarify the exact forward pass.","section":"Eq. (19)"},{"comment":"The lower bound on NDCG in Eq. (1) is stated as a sum of sigmoid terms over positive items and their corresponding negatives, but NDCG is a listwise ranking metric and the bound is not generally valid without additional assumptions about the remaining items and ranking positions. The citation to [32] does not supply the precise conditions. If the lemma is incorrect, the modality-imbalance analysis in Section 2.2 and the motivation for the generation pipeline are weakened. Please provide a proof or a precise citation, or soften the claims accordingly.","section":"Section 2.2, Eq. (1), Proposition 2"}],"minor_comments":[{"comment":"The phrase 'precious study' should be 'previous study'.","section":"Section 2.1"},{"comment":"The caption contains 'Ours Ours' duplicated; only one 'Ours' is needed.","section":"Figure 6 caption"},{"comment":"In the Beauty row, V&T reports R@10 = 0.0832, but Table 2 lists NegGen Beauty R@10 = 0.0823; please reconcile this inconsistency.","section":"Table 4"},{"comment":"The text says 'we remove the the causal learning module' with a duplicated 'the'; also clarify what 'Ours' denotes in Figure 6 relative to the baselines.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely within scope for ACM MM and has promising empirical results. The main risk is the overclaimed causal contribution and the unvalidated false-negative assumption. I would advise the editor to encourage the authors to either substantially revise the causal claims or reframe the module as a contrastive embedding difference, and to add a false-negative analysis. The numeric inconsistency in Table 4 should also be corrected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a useful empirical paper, not a foundational one. The MLLM mask-and-complete pipeline for negative generation is genuine novelty, and the reported gains over strong baselines look consistent. What does not hold up is the causal framing, and the paper's silence on false negatives is a real gap.\n\nWhat's new: the three-stage generation—describe, mask, complete—is a clean way to force the MLLM to produce hard but semantically related negatives. The preliminary result that naive MLLM generation hurts (Fig. 2) is a useful negative result for the field. Experiments on four Amazon datasets show 2–9% relative gains over strong baselines like DRAGON, LGMRec, and AHNS, with p-values, and the ablations make a plausible case that both the generation module and the contrastive module contribute.\n\nSoft spots, in order of severity:\n\n1. The causal language is unsupported. Eq. 12 writes the total effect as a difference of two outcome functions, which is just the definition of TE. Eq. 16 implements it as a difference of two learned embeddings. No intervention, no confounding control, no backdoor adjustment. The module is a contrastive subtraction. That doesn't make it useless, but it is not causal estimation. Ask the authors to drop the causal claims or do a proper adjustment.\n\n2. The false-negative concern is legitimate. The same generated negative description is used in Eq. 19 for every user who interacted with item i. If a generated attribute is something a particular user would actually like—a different color, a complementary accessory—the loss actively demotes a positive item for that user. The paper reports no per-user filtering, no false-negative rate, no human evaluation, and no robustness check against MLLM generation errors. This is a load-bearing assumption for the BPR objective. I am not calling it a demonstrated flaw, because the held-out gains are real regardless of the mechanism, but the authors need to address it with analysis before the claim 'better negative samples' is fully established.\n\n3. Reproducibility: no code, no final hyperparameter values (lambda, alpha, tau are only ranges), no error bars. For an LLM-based method, that handicaps independent verification.\n\nWho this is for: recsys researchers working on negative sampling or LLM-based data augmentation. They will get value from the generation pipeline and the failure analysis. It deserves peer review, and I would send it out with a request for code, exact hyperparameters, a false-negative audit, and a tone-down of the causal claims.\n\nRecommendation: accept for peer review. My own verdict is conditional—the empirical contribution is likely right, but the causal claim is not, and the false-negative issue could change how we interpret the gains.","headline":"A useful negative-sampling pipeline with an overstated causal claim; the false-negative issue needs an audit before the gains are fully trusted.","tokens_in":19421,"tokens_out":4201,"would_cite":true,"duration_ms":35371,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes NegGen, which uses a multi-modal large language model to turn positive item descriptions into masked-and-completed counterfactual negatives, and argues that these generated negatives, combined with a causal contrast…","keywords":["multi-modal recommendation","negative sampling","multi-modal large language models","attribute masking","attribute completion","counterfactual negatives","causal learning","Bayesian personalized ranking"],"falsifier":"Ask a held-out set of users to choose between each positive item and its MLLM-generated negative description; if a large share of users prefer the generated variant for items they have interacted with, those training pairs are false negatives and the claimed supervision signal is corrupted. A cleaner quantitative check is to retrain NegGen with only generated negatives that pass a per-user preference filter and compare Recall and NDCG against the unfiltered run.","tokens_in":18280,"feed_emoji":"🛒","tokens_out":6696,"duration_ms":59909,"temperature":0.7,"pith_summary":"Multi-modal recommender systems often train on weak negatives: random uninteracted items or GAN-generated ones that do not exploit the image and text information attached to products. This paper argues that the right negative for a multi-modal recommender is a counterfactual item that resembles the positive in structure but differs in key attributes, and that such negatives can be produced by a multi-modal large language model with a generate-mask-complete prompt pipeline. The proposed NegGen framework first describes an item's image in text, masks its salient feature words, and asks the model to fill the masks with alternatives; it then encodes both original and generated attributes and trains a LightGCN base recommender with a BPR loss plus a causal contrast module. The paper reports that NegGen outperforms both state-of-the-art multi-modal recommenders and existing negative-sampling methods on four Amazon categories, and that the visual-plus-textual setting beats either modality alone. If correct, the payoff is a practical way to turn cheap MLLM generation into harder, more informative training signals for multimedia recommendation.","feed_headline":"MLLM-written counterfeits improve top-K recommendation","feed_subtitle":"NegGen masks key attributes of positive items and fills them with alternatives, lifting Recall and NDCG on four Amazon categories.","key_machinery":"The load-bearing mechanism is the three-prompt MLLM pipeline followed by a causal contrast module. Description Generation turns an item image into detailed text; Attribute Masking identifies and replaces core features, distinctive characteristics, and key specifications with [MASK] tokens; Attribute Completion fills each mask with plausible alternative words, producing a structurally parallel but semantically shifted description. The original and generated attribute strings are mapped by a frozen text encoder into embeddings $e_m$ and $e^*_m$, and a self-attention layer produces context-aware versions $\\tilde{e}_m$ and $\\tilde{e}^*_m$. The total causal effect of moving from positive to negative attributes is instantiated as their difference $e_t = \\tilde{e}_m - \\tilde{e}^*_m$, which is projected into the collaborative embedding space and used in the BPR loss $-\\log\\sigma(\\lambda e_u^\\top e_c + e_u^\\top e_i - e_u^\\top e^*_c)$; this difference is what the paper claims disentangles intervened key features from irrelevant attributes.","core_discovery":"The central discovery claimed is that a generate-mask-complete procedure over multi-modal item attributes yields negative samples that are simultaneously cohesive (semantically related to the positive) and hard (difficult for the recommender to distinguish), and that this improves top-K recommendation beyond what uniform, hard-negative, GAN-based, or diffusion-based samplers achieve. NegGen operationalizes this by using an MLLM to convert visual content into natural-language descriptions, masking key feature words, and completing the masks with alternative words; the resulting text attributes are encoded with a pretrained text encoder and fed through a self-attention module into a causal-effect embedding $e_t = \\tilde{e}_m - \\tilde{e}^*_m$ that is added to the collaborative score. The training objective combines a BPR ranking loss and a contrastive alignment loss, with the same generated negative embedding used for all users who interacted with the positive item. The paper supports the claim with experiments on Baby, Beauty, Clothing, and Sports, where NegGen reports consistent gains over second-best baselines in Recall and NDCG at K=10 and K=20.","pith_inferences":["Editorial extension: a per-user false-negative audit is the natural stress test: if a generated 'negative' (for example, the same shirt in a different color) is actually preferred by a user, that BPR pair is mislabeled, and adding a preference filter before training could either strengthen or bound NegGen's gains.","Editorial extension: the same generate-mask-complete recipe could be transplanted to other implicit-feedback problems with rich side information, such as sequential recommendation or product search, wherever the bottleneck is the informativeness of negatives rather than the model architecture.","Editorial extension: the paper claims the causal module disentangles key features from irrelevant attributes, but the implementation subtracts two learned embeddings; a direct test would intervene on a single attribute such as color, brand, or size and check whether the score changes only along that dimension."],"forward_implications":["On all four datasets, NegGen's Recall and NDCG at K=10 and K=20 exceed every baseline, with relative gains over the best baseline ranging from roughly 2.3% to 9.0%.","The ablation that removes the negative generation module and falls back to uniform sampling drops performance, showing that the generated attributes carry a substantial part of the improvement.","The visual-plus-textual variant of NegGen beats both unimodal variants, indicating that the mask-complete pipeline reduces modality imbalance rather than simply adding text.","The paper reports that NegGen's negatives reach a given NDCG in fewer epochs than uniform sampling, IRGAN, DNS, DENS, AHNS, and MixGCF, which it reads as evidence that the negatives are both cohesive and hard."],"supporting_citations":[{"why":"Supplies the Bayesian Personalized Ranking objective on which NegGen's recommendation loss is built.","marker":"[52]"},{"why":"Provides the LightGCN base recommender whose collaborative embeddings are combined with the generated multi-modal attributes.","marker":"[18]"},{"why":"FREEDOM is the representative multi-modal recommender used in the preliminary modality-imbalance analysis and a main baseline.","marker":"[81]"},{"why":"Provides the Amazon five-core datasets with images and metadata used in all experiments.","marker":"[43]"},{"why":"Sentence-BERT is the pretrained text encoder that maps original and generated attributes into a shared vector space.","marker":"[50]"},{"why":"Gives the lemma that smaller negative-sample gradients improve the NDCG lower bound, which motivates the hardness and cohesion analysis.","marker":"[32]"},{"why":"MixGCF is a hard-negative sampling baseline whose neighborhood-synthesized negatives NegGen must outperform.","marker":"[20]"},{"why":"Diffusion-based negative sampling on graphs is the generative baseline that motivates the comparison to MLLM generation.","marker":"[45]"}],"fun_headline_variants":["NegGen: MLLM-crafted negatives boost top-K ranking","MLLM-generated balanced negatives improve multi-modal recall","MLLM-masked attributes yield harder negative samples","MLLM counterfeits teach recommenders better","Balanced hard negatives from MLLM masking boost top-K"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that, for every user who interacted with a positive item, the MLLM-generated altered description of that item is a genuine negative that the user does not want, even though the paper reports no per-user filtering or human evaluation of generated negatives.","fun_headline_variants_meta":{"raw":{"variants":["NegGen: MLLM-crafted negatives boost top-K ranking","MLLM-generated balanced negatives improve multi-modal recall","MLLM-masked attributes yield harder negative samples","MLLM counterfeits teach recommenders better","Balanced hard negatives from MLLM masking boost top-K"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000594,"raw_usage":{"total_tokens":2790,"prompt_tokens":958,"completion_tokens":1832,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":1751}},"tokens_in":574,"tokens_out":1832,"duration_ms":12832,"temperature":1.0,"reasoning_tokens":1751,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:32:15.524464+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask a held-out set of users to choose between each positive item and its MLLM-generated negative description; if a large share of users prefer the generated variant for items they have interacted with, those training pairs are false negatives and the claimed supervision signal is corrupted. A cleaner quantitative check is to retrain NegGen with only generated negatives that pass a per-user preference filter and compare Recall and NDCG against the unfiltered run.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Bayesian Personalized Ranking objective on which NegGen's recommendation loss is built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FREEDOM is the representative multi-modal recommender used in the preliminary modality-imbalance analysis and a main baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Sentence-BERT is the pretrained text encoder that maps original and generated attributes into a shared vector space."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the lemma that smaller negative-sample gradients improve the NDCG lower bound, which motivates the hardness and cohesion analysis."}],"review_version":1}