Pith. sign in

REVIEW 3 major objections 4 minor 13 references

Synchronized text-and-image poisoning drives a target item into top-20 recommendations for 73% of users in multimodal LLM recommender systems.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-03 03:54 UTC pith:CWWTYX2W

load-bearing objection Strong novel idea and impressive numbers, but the visual perturbation is defined on internal features, not images, so the paper as written does not realize its own threat model. the 3 major comments →

arxiv 2602.06409 v2 pith:CWWTYX2W submitted 2026-02-06 cs.CR

VENOMREC: Cross-Modal Interactive Poisoning for Targeted Promotion in Multimodal LLM Recommender Systems

classification cs.CR
keywords multimodal LLM recommender systemsdata poisoningcross-modal interactive poisoningtargeted promotionexposure alignmentcross-modal fusionadversarial machine learningfine-tuning attacks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the cross-modal fusion at the heart of multimodal LLM recommender systems creates a new poisoning surface: when an attacker edits an item's text and image together, in a coordinated and attention-guided way, the fusion mechanism that normally filters out unimodal noise instead amplifies the malicious signal. It formalizes this as cross-modal interactive poisoning and proposes VENOMREC, which first computes a high-exposure 'hotspot' from publicly popular same-category items and then iteratively co-adapts textual tokens and visual patches so the target item's fused representation moves toward that hotspot. On three product datasets the attack raises target exposure at top-20 to an average of 0.73, about +0.52 absolute over the best compared attack, while keeping overall recommendation quality largely unchanged. The authors' point is that the same consensus mechanism believed to make multimodal systems robust can be repurposed as an amplifier, so security evaluations of MLLM-based recommenders need to consider synchronized multimodal poisoning.

Core claim

VENOMREC formalizes a new poisoning threat, cross-modal interactive poisoning, against recommender systems built on multimodal large language models. The core claim is that if an attacker simultaneously edits both the textual description and the visual appearance of a target item's user-generated content, the cross-modal fusion mechanism—normally a stabilizer that lets one modality correct noise in the other—instead locks onto the coordinated perturbation and moves the item's fused representation toward a chosen high-exposure region. The paper reports that on three product datasets this yields a mean exposure rate at top-20 of 0.73, an average gain of +0.52 over the strongest baseline, while

What carries the argument

The load-bearing object is the high-exposure centroid z⋆: a normalized average of proxy-model embeddings of publicly popular items in the same category as the target. This centroid is the attack's destination. The second mechanism is the cross-modal interactive perturbation loop, which alternates gradient-free visual updates on attention-masked patches and greedy discrete token edits, recomputing the cross-modal attention map after each round so the two modalities co-adapt; the pair is constrained to stay within a stealthiness set that preserves unimodal naturalness and image-text coherence.

Load-bearing premise

The attack's steering direction is a centroid of publicly popular items' embeddings computed with a public proxy model, and the whole method presumes this centroid points to a high-exposure region in the victim's privately fine-tuned model; if that transfer fails, the poisoned edits push the target toward a direction that does not raise its exposure.

What would settle it

Fine-tune the victim recommender with a fusion backbone that differs from the public proxy used to craft the poisoned samples and measure ER@20: if exposure falls back toward interaction-only baselines, the proxy-to-victim transfer is the load-bearing part. A complementary check is to replace the high-exposure centroid with a random or category-mismatched centroid; if exposure remains high, the coupled perturbation by itself—not exposure alignment—is producing the effect.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Cross-modal consensus should not be assumed to defend MLLM-based recommenders: coordinated edits turn the fusion mechanism into an amplifier, so any robustness claim needs to be tested against synchronized multimodal poisoning.
  • Existing interaction-level and single-modality attacks show near-zero exposure in these settings, so threat models for content-grounded recommenders must include the coupled text-image channel to be realistic.
  • The attack works at very small poisoning ratios—measurable exposure at 0.05% of users and strong gains at 0.7%—meaning the cost of launching it is low.
  • The attack succeeds in zero-shot conditions, indicating the high-exposure destination is derived from public popularity signals rather than from the victim's private fine-tuning data.
  • Benign ranking quality stays close to the clean baseline, so the attack is unlikely to be flagged by utility monitoring alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the centroid transfer assumption holds across broader model families, the same two-stage recipe might apply to any retrieval system with public popularity signals and fused embeddings—multimodal search, product ranking, content feeds—though the paper does not test those settings.
  • A natural defense suggested by the result would be to break the stable centroid, for example by randomizing or ensembling the fusion backbone or by monitoring content edits that move an item's embedding toward a same-category popularity centroid; these defenses are editorial inferences, not evaluated in the paper.
  • The anchor-mining step restricts anchors to the target item's category, which implies that items in categories with few popular same-category examples may be harder to promote; a direct test would measure ER@20 against category size or anchor count.
  • Because the attack transfers zero-shot, it also implies that a handful of poisoned multimodal samples shape the model's generalization behavior, not just its training-set memorization; if true, detection should look for systematic embedding drift rather than individual outlier samples.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper formalizes cross-modal interactive poisoning against MLLM-based recommender systems and proposes VENOMREC, a two-stage attack. Stage 1 (Exposure Alignment) mines high-exposure anchor items from public popularity signals and constructs a centroid z* in the proxy embedding space as the attack destination. Stage 2 (Cross-modal Interactive Perturbation) uses attention rollout to select salient text tokens and visual patches, then alternately updates textual tokens by greedy search and visual patch features by sign-projection toward z*, with masks recomputed each round. The experiments on three Amazon datasets report mean ER@20 of roughly 0.73, large gains over ten interaction-level, unimodal, and multimodal baselines, and little degradation of HR/NDCG. The paper claims that synchronized multimodal perturbations exploit rather than are mitigated by cross-modal consensus.

Significance. If the result holds under the stated threat model, it would be an interesting and important demonstration that content-grounded, fusion-based MLLM recommender systems are not inherently robust to coordinated multimodal poisoning. The paper contributes a crisp threat model (Definition 3.1), a clean separation between 'where to move' and 'how to move', and an unusually broad baseline comparison. The code is promised, and the empirical magnitudes are large. However, the significance is currently conditional on resolving a load-bearing realizability problem: the visual perturbation in Eq. (7) operates on projected patch features, not on images, so the attack may not correspond to the user-generated-content poisoning scenario defined in the paper. The deferred appendix also contains no content, so transferability, defensive robustness, and multi-target claims are unverifiable.

major comments (3)
  1. [§4.3, Eq. (7)] The visual update rule optimizes 'projected patch features' and clips them in feature space to [v_orig − ε, v_orig + ε]. The threat model in Definition 3.1 requires a coupled transformation T: (t,v) → (t~,v~) applied to user-generated content, with stealthiness assessed on actual images and text. The manuscript never specifies how modified patch features are mapped back to pixel-space images. Table 2 reports FID scores, implying that images were produced, but the algorithm as written does not contain an inversion step. As a result, the reported ER@20 may be achieved by manipulating internal tensors, which is outside the stated UGC poisoning threat. This needs either an explicit pixel-space optimization/decoding step or a revised threat model and revised claims.
  2. [Appendix A.2–A.5] The paper repeatedly defers load-bearing evidence to the appendix: implementation details (A.2), T5-base transfer (A.3), robustness against defensive filtering (A.4), and multi-target scalability (A.5). In the submitted manuscript these sections contain only headings and no content. Since the main evaluation uses a single victim model (VIP5 with T5-small), no defense evaluation, and no multi-target results, the claims of transferability, robustness, and scalability are currently unsupported. These sections must be filled in before the paper can be assessed.
  3. [§5.2, Table 1] All ER/HR/NDCG numbers in Table 1 are point estimates with no standard deviations, confidence intervals, or number of random seeds. The attack pipeline involves stochastic candidate sampling, greedy search, and fine-tuning, so run-to-run variation could be non-negligible. The paper's language ('consistently', 'reliably') requires at least seed-level variance reporting to support the central claim of reliability.
minor comments (4)
  1. [Abstract vs. §5.1] The abstract states 'four real-world multimodal datasets', while the full-text abstract and Section 5.1 describe experiments on three datasets (Clothing, Sports, Toys). Please reconcile.
  2. [§5.4, Table 3] The ablation variant 'Tab' is described as 'Interaction injection (EA)', which conflates interaction injection with Exposure Alignment. The textual explanation says Tab alone 'isolates the effect of Exposure Alignment', but the operation appears to be injecting interaction sessions co-locating the target with anchors. Please clarify the naming and exactly what is ablated.
  3. [Table 3 formatting] The numeric columns in Table 3 are difficult to read because some entries lack clear separators (e.g., HR/NDCG/ER triples). Use consistent column alignment or explicit separators.
  4. [§5.1, Figure 2 caption] Minor typo: 'textural perturbation' should likely be 'textual perturbation'. Also, Figure 3 heatmaps lack labeled axes for the subsets shown; the text references values such as 0.0309 and 0.8543 that are hard to verify from the figure alone.

Circularity Check

0 steps flagged

No significant circularity: the attack optimizes a surrogate cosine objective against a public proxy backbone and evaluates a distinct downstream ER@20 on a victim model; no load-bearing result reduces to its inputs by construction.

full rationale

The derivation chain is self-contained with respect to circularity. The formal attack objective (Eq. 3) is a bilevel exposure-maximization problem. VENOMREC replaces it with a surrogate: Eq. (4) defines the destination z* as the l2-normalized mean of proxy embeddings of high-exposure anchor items, and Eq. (5) minimizes 1 - cos(phi(tilde_t, tilde_v), z*). This surrogate is not the same object as the reported metric ER@20, which is measured on benign test users of the victim VIP5 model after fine-tuning; there is no equation in the paper identifying cosine similarity to z* with top-K exposure. The assumption that high-exposure items share a stable semantic centroid (Section 4.2) is a substantive empirical premise, not an identity. The comparison against ten baselines on three datasets and the reported utility metrics (Table 1) provide independent empirical content. The main validity concerns are not circularity: Eq. (7) updates 'projected patch features' in feature space and the paper does not specify an inversion to pixel-space images, so the realizability of the poisoned UGC under Definition 3.1 is not established; and the appendix contains only headings (A.1-A.5) while the main text defers implementation details, T5-base results, and defense evaluations there. These are completeness/mechanism gaps, not instances of an output being equivalent to an input by definition. No load-bearing step relies on a self-citation chain; the cited victim model VIP5 is external prior work by other authors.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The attack rests on five unreported hyperparameters and four domain assumptions. The most consequential assumptions are the stability of the high-exposure centroid and the equivalence between the public proxy backbone and the victim's fusion mechanism. No new physical or mathematical entities are introduced.

free parameters (5)
  • compromised user ratio ρ = not fixed; varied in {0.05%, 0.1%, 0.3%, 0.5%, 0.7%} (Fig. 3); value used for main Table 1 not stated
    Controls attack budget; main results presumably use one value but the paper never states which.
  • perturbation budget ε (ℓ∞ clip) = not reported
    Eq. 7 clips visual edits to [v_orig−ε, v_orig+ε]; ε is not specified.
  • top-k sensitive tokens/patches = not reported
    Binary masks m_txt, m_vis select top-k elements; k is unspecified.
  • iteration budget R and alignment threshold = not reported
    CIP loop stops after R rounds or when cos exceeds threshold; neither value is given.
  • anchor set size |H'| and category restriction = not reported
    EA uses same-category high-exposure anchors; the size of H' is unspecified.
axioms (4)
  • domain assumption Naturally high-exposure items share latent semantic characteristics stable across model instantiations
    §4.2 'High-Exposure Anchor Mining'; the attack destination z* is only meaningful if this holds.
  • domain assumption Public CLIP/T5 backbone is a usable proxy for the victim's fusion operator ΦΘ
    §4.1; the entire surrogate optimization assumes proxy-victim alignment, and the victim uses the same CLIP ViT-B/32 and T5 backbones.
  • domain assumption Popularity lists are available and overlap with the item distribution
    §4.2; the paper does not state how H' is obtained per dataset or whether anchors are in the training split.
  • domain assumption Victim fine-tuning includes poisoned UGC in the prompt via Eq. (1)
    §3.2; the attack's success depends on how the poisoned content enters the training prompt.

pith-pipeline@v1.3.0-alltime-deepseek · 17136 in / 11843 out tokens · 112392 ms · 2026-08-03T03:54:38.485485+00:00 · methodology

0 comments
read the original abstract

Multimodal large language models (MLLMs) are pushing recommender systems (RecSys) toward content-grounded retrieval and ranking via cross-modal fusion. We find that while cross-modal consensus often mitigates conventional poisoning that manipulates interaction logs or perturbs a single modality, it also introduces a new attack surface where synchronised multimodal poisoning can reliably steer fused representations along stable semantic directions during fine-tuning. To characterise this threat, we formalise cross-modal interactive poisoning and propose VENOMREC, which performs Exposure Alignment to identify high-exposure regions in the joint embedding space and Cross-modal Interactive Perturbation to craft attention-guided coupled token--patch edits. Experiments on four real-world multimodal datasets demonstrate that VENOMREC consistently outperforms strong baselines, achieving 0.73 mean ER@20 and improving over the strongest baseline by +0.52 absolute ER points on average, while maintaining comparable recommendation utility. Code is available at https://github.com/GuoweiGuan666/VenomRec.

Figures

Figures reproduced from arXiv: 2602.06409 by Cyril Leung, Fuyao Zhang, Guowei Guan, Jiaming Zhang, Longtao Huang, Tiantong Wu, Tianxiang Chen, Wei Yang Bryan Lim, Yurong Hao.

Figure 1
Figure 1. Figure 1: Overview of poisoning paradigms on MLLM-RecSys. Top: Interaction-level attacks manipulate discrete user-item records (e.g., clicks/ratings) but have limited targeted influence against MLLM-RecSys because they do not directly steer the model’s semantic reasoning process. Middle: Single-modality attacks perturb either text or image in isolation and are often miti￾gated by cross-modal consensus in the fusion … view at source ↗
Figure 2
Figure 2. Figure 2: The schematic overview of VENOMREC. The framework operates in two strategic phases: (1) Exposure Alignment (EA) constructs a latent target centroid from high-exposure anchors to guide the attack direction. (2) Cross-modal Interactive Perturbation (CIP) leverages attention-guided saliency to optimise visual features and textual tokens iteratively. 4.1. Method Overview The adversary’s objective is to craft p… view at source ↗
Figure 3
Figure 3. Figure 3: Impact of the compromised user ratio ρ on attack effec￾tiveness (ER@5) and recommendation utility (HR@5, NDCG@5) under Few-Shot and Zero-Shot settings on the Clothing dataset. injecting adversarial interaction sessions that co-locate the target with high-exposure anchors while keeping the item content unchanged, we observe a non-trivial lift in broad exposure compared to the NoAttack baseline. In the Few￾S… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 7 linked inside Pith

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

  2. [13]

    Histllm: A unified framework for llm-based multimodal recommen- dation with user history encoding and compression.arXiv preprint arXiv:2504.10150, 2025a

    Zhang, C., Hu, B., Chen, W., and Mao, Z. Histllm: A unified framework for llm-based multimodal recommen- dation with user history encoding and compression.arXiv preprint arXiv:2504.10150, 2025a. Zhang, C., Zhang, H., Wu, S., Wu, D., Xu, T., Zhao, X., Gao, Y ., Hu, Y ., and Chen, E. Notellm-2: Multimodal large representation models for recommendation. InPr...

  3. [2003]

    Liu, Y ., Yuan, Z., Tie, G., Shi, J., Zhou, P., Sun, L., and Gong, N. Z. Poisoned-mrag: Knowledge poisoning attacks to multimodal retrieval augmented generation. arXiv preprint arXiv:2503.06254,

  4. [2012]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 1877–1901,

  5. [2013]

    Exploring backdoor attack and defense for llm-empowered recommendations.arXiv preprint arXiv:2504.11182,

    Ning, L., Fan, W., and Li, Q. Exploring backdoor attack and defense for llm-empowered recommendations.arXiv preprint arXiv:2504.11182,

  6. [2015]

    Not one less: Exploring interplay between user profiles and items in untargeted attacks against federated recommendation

    Hao, Y ., Chen, X., Lyu, X., Liu, J., Zhu, Y ., Wan, Z., Mauw, S., and Wang, W. Not one less: Exploring interplay between user profiles and items in untargeted attacks against federated recommendation. InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pp. 2889–2903, 2024a. Hao, Y ., Chen, X., Wang, W., Liu, J., Li...

  7. [2016]

    Bert-attack: Adversarial attack against bert using bert.Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing,

    Li, L., Ma, R., Guo, Q., Xue, X., and Qiu, X. Bert-attack: Adversarial attack against bert using bert.Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing,

  8. [2018]

    Vip5: Towards multimodal foundation models for recommenda- tion

    Geng, S., Tan, J., Liu, S., Fu, Z., and Zhang, Y . Vip5: Towards multimodal foundation models for recommenda- tion. InThe 2023 Conference on Empirical Methods in Natural Language Processing. Geng, S., Liu, S., Fu, Z., Ge, Y ., and Zhang, Y . Recommen- dation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). InP...

  9. [2020]

    and Terzis, A

    Carlini, N. and Terzis, A. Poisoning and backdooring con- trastive learning.arXiv preprint arXiv:2106.09667,

  10. [2021]

    Kang, W.-C., Ni, J., Mehta, N., Sathiamoorthy, M., Hong, L., Chi, E., and Cheng, D. Z. Do llms understand user preferences? evaluating llms on user rating prediction. arXiv preprint arXiv:2305.06474,

  11. [2022]

    Poisoning attacks and defenses in recommender systems: A survey.arXiv preprint arXiv:2406.01022,

    Wang, Z., Yu, J., Gao, M., Yuan, W., Ye, G., Sadiq, S., and Yin, H. Poisoning attacks and defenses in recommender systems: A survey.arXiv preprint arXiv:2406.01022,

  12. [2023]

    Vl-adapter: Parameter- efficient transfer learning for vision-and-language tasks

    Sung, Y .-L., Cho, J., and Bansal, M. Vl-adapter: Parameter- efficient transfer learning for vision-and-language tasks. InProceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5227– 5237,

  13. [2025]

    and Larson, M

    Liu, Z. and Larson, M. Adversarial item promotion: Vul- nerabilities at the core of top-n recommenders that use images to address cold start. InProceedings of the Web Conference 2021, pp. 3590–3602,