REVIEW 3 major objections 4 minor 13 references
Synchronized text-and-image poisoning drives a target item into top-20 recommendations for 73% of users in multimodal LLM recommender systems.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 03:54 UTC pith:CWWTYX2W
load-bearing objection Strong novel idea and impressive numbers, but the visual perturbation is defined on internal features, not images, so the paper as written does not realize its own threat model. the 3 major comments →
VENOMREC: Cross-Modal Interactive Poisoning for Targeted Promotion in Multimodal LLM Recommender Systems
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
VENOMREC formalizes a new poisoning threat, cross-modal interactive poisoning, against recommender systems built on multimodal large language models. The core claim is that if an attacker simultaneously edits both the textual description and the visual appearance of a target item's user-generated content, the cross-modal fusion mechanism—normally a stabilizer that lets one modality correct noise in the other—instead locks onto the coordinated perturbation and moves the item's fused representation toward a chosen high-exposure region. The paper reports that on three product datasets this yields a mean exposure rate at top-20 of 0.73, an average gain of +0.52 over the strongest baseline, while
What carries the argument
The load-bearing object is the high-exposure centroid z⋆: a normalized average of proxy-model embeddings of publicly popular items in the same category as the target. This centroid is the attack's destination. The second mechanism is the cross-modal interactive perturbation loop, which alternates gradient-free visual updates on attention-masked patches and greedy discrete token edits, recomputing the cross-modal attention map after each round so the two modalities co-adapt; the pair is constrained to stay within a stealthiness set that preserves unimodal naturalness and image-text coherence.
Load-bearing premise
The attack's steering direction is a centroid of publicly popular items' embeddings computed with a public proxy model, and the whole method presumes this centroid points to a high-exposure region in the victim's privately fine-tuned model; if that transfer fails, the poisoned edits push the target toward a direction that does not raise its exposure.
What would settle it
Fine-tune the victim recommender with a fusion backbone that differs from the public proxy used to craft the poisoned samples and measure ER@20: if exposure falls back toward interaction-only baselines, the proxy-to-victim transfer is the load-bearing part. A complementary check is to replace the high-exposure centroid with a random or category-mismatched centroid; if exposure remains high, the coupled perturbation by itself—not exposure alignment—is producing the effect.
If this is right
- Cross-modal consensus should not be assumed to defend MLLM-based recommenders: coordinated edits turn the fusion mechanism into an amplifier, so any robustness claim needs to be tested against synchronized multimodal poisoning.
- Existing interaction-level and single-modality attacks show near-zero exposure in these settings, so threat models for content-grounded recommenders must include the coupled text-image channel to be realistic.
- The attack works at very small poisoning ratios—measurable exposure at 0.05% of users and strong gains at 0.7%—meaning the cost of launching it is low.
- The attack succeeds in zero-shot conditions, indicating the high-exposure destination is derived from public popularity signals rather than from the victim's private fine-tuning data.
- Benign ranking quality stays close to the clean baseline, so the attack is unlikely to be flagged by utility monitoring alone.
Where Pith is reading between the lines
- If the centroid transfer assumption holds across broader model families, the same two-stage recipe might apply to any retrieval system with public popularity signals and fused embeddings—multimodal search, product ranking, content feeds—though the paper does not test those settings.
- A natural defense suggested by the result would be to break the stable centroid, for example by randomizing or ensembling the fusion backbone or by monitoring content edits that move an item's embedding toward a same-category popularity centroid; these defenses are editorial inferences, not evaluated in the paper.
- The anchor-mining step restricts anchors to the target item's category, which implies that items in categories with few popular same-category examples may be harder to promote; a direct test would measure ER@20 against category size or anchor count.
- Because the attack transfers zero-shot, it also implies that a handful of poisoned multimodal samples shape the model's generalization behavior, not just its training-set memorization; if true, detection should look for systematic embedding drift rather than individual outlier samples.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper formalizes cross-modal interactive poisoning against MLLM-based recommender systems and proposes VENOMREC, a two-stage attack. Stage 1 (Exposure Alignment) mines high-exposure anchor items from public popularity signals and constructs a centroid z* in the proxy embedding space as the attack destination. Stage 2 (Cross-modal Interactive Perturbation) uses attention rollout to select salient text tokens and visual patches, then alternately updates textual tokens by greedy search and visual patch features by sign-projection toward z*, with masks recomputed each round. The experiments on three Amazon datasets report mean ER@20 of roughly 0.73, large gains over ten interaction-level, unimodal, and multimodal baselines, and little degradation of HR/NDCG. The paper claims that synchronized multimodal perturbations exploit rather than are mitigated by cross-modal consensus.
Significance. If the result holds under the stated threat model, it would be an interesting and important demonstration that content-grounded, fusion-based MLLM recommender systems are not inherently robust to coordinated multimodal poisoning. The paper contributes a crisp threat model (Definition 3.1), a clean separation between 'where to move' and 'how to move', and an unusually broad baseline comparison. The code is promised, and the empirical magnitudes are large. However, the significance is currently conditional on resolving a load-bearing realizability problem: the visual perturbation in Eq. (7) operates on projected patch features, not on images, so the attack may not correspond to the user-generated-content poisoning scenario defined in the paper. The deferred appendix also contains no content, so transferability, defensive robustness, and multi-target claims are unverifiable.
major comments (3)
- [§4.3, Eq. (7)] The visual update rule optimizes 'projected patch features' and clips them in feature space to [v_orig − ε, v_orig + ε]. The threat model in Definition 3.1 requires a coupled transformation T: (t,v) → (t~,v~) applied to user-generated content, with stealthiness assessed on actual images and text. The manuscript never specifies how modified patch features are mapped back to pixel-space images. Table 2 reports FID scores, implying that images were produced, but the algorithm as written does not contain an inversion step. As a result, the reported ER@20 may be achieved by manipulating internal tensors, which is outside the stated UGC poisoning threat. This needs either an explicit pixel-space optimization/decoding step or a revised threat model and revised claims.
- [Appendix A.2–A.5] The paper repeatedly defers load-bearing evidence to the appendix: implementation details (A.2), T5-base transfer (A.3), robustness against defensive filtering (A.4), and multi-target scalability (A.5). In the submitted manuscript these sections contain only headings and no content. Since the main evaluation uses a single victim model (VIP5 with T5-small), no defense evaluation, and no multi-target results, the claims of transferability, robustness, and scalability are currently unsupported. These sections must be filled in before the paper can be assessed.
- [§5.2, Table 1] All ER/HR/NDCG numbers in Table 1 are point estimates with no standard deviations, confidence intervals, or number of random seeds. The attack pipeline involves stochastic candidate sampling, greedy search, and fine-tuning, so run-to-run variation could be non-negligible. The paper's language ('consistently', 'reliably') requires at least seed-level variance reporting to support the central claim of reliability.
minor comments (4)
- [Abstract vs. §5.1] The abstract states 'four real-world multimodal datasets', while the full-text abstract and Section 5.1 describe experiments on three datasets (Clothing, Sports, Toys). Please reconcile.
- [§5.4, Table 3] The ablation variant 'Tab' is described as 'Interaction injection (EA)', which conflates interaction injection with Exposure Alignment. The textual explanation says Tab alone 'isolates the effect of Exposure Alignment', but the operation appears to be injecting interaction sessions co-locating the target with anchors. Please clarify the naming and exactly what is ablated.
- [Table 3 formatting] The numeric columns in Table 3 are difficult to read because some entries lack clear separators (e.g., HR/NDCG/ER triples). Use consistent column alignment or explicit separators.
- [§5.1, Figure 2 caption] Minor typo: 'textural perturbation' should likely be 'textual perturbation'. Also, Figure 3 heatmaps lack labeled axes for the subsets shown; the text references values such as 0.0309 and 0.8543 that are hard to verify from the figure alone.
Circularity Check
No significant circularity: the attack optimizes a surrogate cosine objective against a public proxy backbone and evaluates a distinct downstream ER@20 on a victim model; no load-bearing result reduces to its inputs by construction.
full rationale
The derivation chain is self-contained with respect to circularity. The formal attack objective (Eq. 3) is a bilevel exposure-maximization problem. VENOMREC replaces it with a surrogate: Eq. (4) defines the destination z* as the l2-normalized mean of proxy embeddings of high-exposure anchor items, and Eq. (5) minimizes 1 - cos(phi(tilde_t, tilde_v), z*). This surrogate is not the same object as the reported metric ER@20, which is measured on benign test users of the victim VIP5 model after fine-tuning; there is no equation in the paper identifying cosine similarity to z* with top-K exposure. The assumption that high-exposure items share a stable semantic centroid (Section 4.2) is a substantive empirical premise, not an identity. The comparison against ten baselines on three datasets and the reported utility metrics (Table 1) provide independent empirical content. The main validity concerns are not circularity: Eq. (7) updates 'projected patch features' in feature space and the paper does not specify an inversion to pixel-space images, so the realizability of the poisoned UGC under Definition 3.1 is not established; and the appendix contains only headings (A.1-A.5) while the main text defers implementation details, T5-base results, and defense evaluations there. These are completeness/mechanism gaps, not instances of an output being equivalent to an input by definition. No load-bearing step relies on a self-citation chain; the cited victim model VIP5 is external prior work by other authors.
Axiom & Free-Parameter Ledger
free parameters (5)
- compromised user ratio ρ =
not fixed; varied in {0.05%, 0.1%, 0.3%, 0.5%, 0.7%} (Fig. 3); value used for main Table 1 not stated
- perturbation budget ε (ℓ∞ clip) =
not reported
- top-k sensitive tokens/patches =
not reported
- iteration budget R and alignment threshold =
not reported
- anchor set size |H'| and category restriction =
not reported
axioms (4)
- domain assumption Naturally high-exposure items share latent semantic characteristics stable across model instantiations
- domain assumption Public CLIP/T5 backbone is a usable proxy for the victim's fusion operator ΦΘ
- domain assumption Popularity lists are available and overlap with the item distribution
- domain assumption Victim fine-tuning includes poisoned UGC in the prompt via Eq. (1)
read the original abstract
Multimodal large language models (MLLMs) are pushing recommender systems (RecSys) toward content-grounded retrieval and ranking via cross-modal fusion. We find that while cross-modal consensus often mitigates conventional poisoning that manipulates interaction logs or perturbs a single modality, it also introduces a new attack surface where synchronised multimodal poisoning can reliably steer fused representations along stable semantic directions during fine-tuning. To characterise this threat, we formalise cross-modal interactive poisoning and propose VENOMREC, which performs Exposure Alignment to identify high-exposure regions in the joint embedding space and Cross-modal Interactive Perturbation to craft attention-guided coupled token--patch edits. Experiments on four real-world multimodal datasets demonstrate that VENOMREC consistently outperforms strong baselines, achieving 0.73 mean ER@20 and improving over the strongest baseline by +0.52 absolute ER points on average, while maintaining comparable recommendation utility. Code is available at https://github.com/GuoweiGuan666/VenomRec.
Figures
Reference graph
Works this paper leans on
-
[1]
L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,
-
[13]
Zhang, C., Hu, B., Chen, W., and Mao, Z. Histllm: A unified framework for llm-based multimodal recommen- dation with user history encoding and compression.arXiv preprint arXiv:2504.10150, 2025a. Zhang, C., Zhang, H., Wu, S., Wu, D., Xu, T., Zhao, X., Gao, Y ., Hu, Y ., and Chen, E. Notellm-2: Multimodal large representation models for recommendation. InPr...
Pith/arXiv arXiv 2020
-
[2003]
Liu, Y ., Yuan, Z., Tie, G., Shi, J., Zhou, P., Sun, L., and Gong, N. Z. Poisoned-mrag: Knowledge poisoning attacks to multimodal retrieval augmented generation. arXiv preprint arXiv:2503.06254,
-
[2012]
D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 1877–1901,
1901
-
[2013]
Ning, L., Fan, W., and Li, Q. Exploring backdoor attack and defense for llm-empowered recommendations.arXiv preprint arXiv:2504.11182,
-
[2015]
Not one less: Exploring interplay between user profiles and items in untargeted attacks against federated recommendation
Hao, Y ., Chen, X., Lyu, X., Liu, J., Zhu, Y ., Wan, Z., Mauw, S., and Wang, W. Not one less: Exploring interplay between user profiles and items in untargeted attacks against federated recommendation. InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pp. 2889–2903, 2024a. Hao, Y ., Chen, X., Wang, W., Liu, J., Li...
2024
-
[2016]
Bert-attack: Adversarial attack against bert using bert.Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing,
Li, L., Ma, R., Guo, Q., Xue, X., and Qiu, X. Bert-attack: Adversarial attack against bert using bert.Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing,
2020
-
[2018]
Vip5: Towards multimodal foundation models for recommenda- tion
Geng, S., Tan, J., Liu, S., Fu, Z., and Zhang, Y . Vip5: Towards multimodal foundation models for recommenda- tion. InThe 2023 Conference on Empirical Methods in Natural Language Processing. Geng, S., Liu, S., Fu, Z., Ge, Y ., and Zhang, Y . Recommen- dation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). InP...
2023
-
[2020]
Carlini, N. and Terzis, A. Poisoning and backdooring con- trastive learning.arXiv preprint arXiv:2106.09667,
-
[2021]
Kang, W.-C., Ni, J., Mehta, N., Sathiamoorthy, M., Hong, L., Chi, E., and Cheng, D. Z. Do llms understand user preferences? evaluating llms on user rating prediction. arXiv preprint arXiv:2305.06474,
-
[2022]
Poisoning attacks and defenses in recommender systems: A survey.arXiv preprint arXiv:2406.01022,
Wang, Z., Yu, J., Gao, M., Yuan, W., Ye, G., Sadiq, S., and Yin, H. Poisoning attacks and defenses in recommender systems: A survey.arXiv preprint arXiv:2406.01022,
-
[2023]
Vl-adapter: Parameter- efficient transfer learning for vision-and-language tasks
Sung, Y .-L., Cho, J., and Bansal, M. Vl-adapter: Parameter- efficient transfer learning for vision-and-language tasks. InProceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5227– 5237,
2022
-
[2025]
and Larson, M
Liu, Z. and Larson, M. Adversarial item promotion: Vul- nerabilities at the core of top-n recommenders that use images to address cold start. InProceedings of the Web Conference 2021, pp. 3590–3602,
2021
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.