Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

MOON2.0 claims state-of-the-art zero-shot e-commerce product understanding by rebalancing modalities, aligning image–text within products, and co-augmenting data with an MLLM.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 22:01 UTC pith:UKAGWDAX

load-bearing objection Real new benchmark and a plausible recipe, but the SOTA claim is not yet attributable: uncontrolled proprietary backbone and missing evaluation protocol undermine the headline numbers. the 4 major comments →

arxiv 2511.12449 v3 pith:UKAGWDAX submitted 2025-11-16 cs.CV cs.AIcs.IRcs.LG

MOON2.0: Dynamic Modality-balanced Multimodal Representation Learning for E-commerce Product Understanding

Zhanheng Nie , Chenghan Fu , Daoze Zhang , Junxian Wu , Wanxian Guan , Pengjie Wang , Jian Xu , Bo Zheng This is my paper
classification cs.CV cs.AIcs.IRcs.LG
keywords multimodal representation learninge-commerce product understandingmodality imbalancemixture-of-expertsimage-text alignmentcontrastive learningzero-shot retrievaldata augmentation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that three obstacles—modality imbalance from fixed-ratio mixed training, underused image–text alignment inside each product, and noisy e-commerce data—can be overcome in one framework. It proposes MOON2.0, which routes inputs through a modality-aware mixture-of-experts, adds intra-product alignment on top of inter-product contrastive learning, and uses a multimodal LLM to enrich titles and synthesize image variants while dynamically down-weighting unreliable training triplets. If the reported numbers hold, a single zero-shot model covers retrieval, classification, and attribute prediction across e-commerce benchmarks. The authors also release MBE2.0, a 6.7-million-sample co-augmented benchmark, making the claims testable.

Core claim

The central claim is that dynamic modality balancing and explicit intra-product image–text alignment substantially improve e-commerce multimodal representations. On zero-shot evaluation over MBE2.0, M5Product, and Fashion200K, the paper reports top results across multimodal retrieval, product classification, and attribute prediction, with the largest jump on M5Product classification: 95.50% accuracy versus 73.12% for the prior MOON model and roughly 78% for the strongest previous baseline. The ablation study shows that removing the dual-level alignment causes the steepest drop, while removing the MoE, co-augmentation, or dynamic filtering also degrades performance measurably.

What carries the argument

The central mechanism is the Modality-driven Mixture-of-Experts (MoE), placed in the feed-forward layers of a multimodal large language model (MLLM) backbone. Unlike standard token-level routing, a learnable Dual-alignment Matrix assigns each expert a preferred subset of alignment objectives (image-only, text-only, or multimodal queries paired with multimodal targets), and a sparsity loss pushes expert preferences to be peaked. A Dual-level Alignment objective then combines inter-product contrastive losses across modality pairs with an intra-product loss that pulls image and text of the same item together. On the data side, an MLLM-based co-augmentation pipeline enriches titles and generates

Load-bearing premise

The load-bearing premise is that a purchase following a query marks a semantically matching product and a skipped exposure with a low relevance score marks a mismatch; if purchases are driven by price, popularity, or display position instead of product-match, the contrastive labels do not encode product understanding and the reported gains measure click correlation.

What would settle it

Take a random sample of MBE2.0 test queries, have human annotators judge whether the ground-truth positive item is a genuine semantic match to the query (same product type, brand, and key attributes), and check whether model rankings agree with human judgments; if agreement is low, or if retrieval accuracy collapses when popularity-and-price-matched distractors are added to the candidate set, the purchase-based labels are not carrying product-match signal.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A single zero-shot model can serve multimodal retrieval in both directions, cross-modal retrieval (image-to-text and text-to-image), product classification, and attribute prediction without task-specific heads.
  • Modality-aware expert routing mitigates the degradation caused by fixed-ratio mixed training, benefiting downstream tasks regardless of their query-modality distribution.
  • Explicit intra-product alignment is the highest-leverage component: removing it produces the largest performance drop, especially on cross-modal retrieval tasks.
  • The released MBE2.0 benchmark, with co-augmented training data and 6.7M real-world samples, offers a reproducible evaluation protocol for future e-commerce representation learning.
  • Each of the four components contributes: ablations show measurable declines when any one is removed, confirming they are complementary rather than redundant.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the M5Product classification jump transfers to other domains, the same co-augmentation recipe—entity-aware title enrichment plus multi-granularity image synthesis—could lift representation quality in other image–text domains with sparse or noisy data, such as furniture catalogs or second-hand marketplaces.
  • The paper leaves implicit that purchase-based positives may encode popularity or price rather than semantic match; a natural extension is to validate the training labels with human relevance judgments on a sample of triplets before trusting the benchmark's numbers.
  • The Modality-driven MoE with a Dual-alignment Matrix is a general architectural idea: applying it to non-e-commerce multimodal models could address similar modality imbalance in general image–text retrieval or visual question answering.
  • The dynamic filtering threshold is fixed at 0.6 on the logit scale; a testable extension is to make the threshold adaptive per alignment objective, since some modality pairs may be noisier than others.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MOON2.0, a multimodal representation-learning framework for e-commerce built on an internal generative MLLM. The method combines a modality-driven Mixture-of-Experts, dual-level inter/intra-product alignment, MLLM-based image-text co-augmentation, and dynamic sample filtering. The authors also release MBE2.0, a 6.7M-sample co-augmented benchmark with retrieval, classification, and attribute-prediction tasks. They report state-of-the-art zero-shot results on MBE2.0, M5Product, and Fashion200K, plus an ablation study on MBE2.0.

Significance. If the claims hold, the paper makes a useful contribution to e-commerce multimodal representation learning: it releases a large-scale benchmark, introduces four well-motivated components, and provides an internal ablation showing that each component contributes. The reported gains on cross-modal retrieval and on public benchmarks are substantial. The public-benchmark results are the strongest evidence for generalization, and the MBE2.0 release is a community resource. However, the central SOTA claim is currently supported by an uncontrolled backbone comparison and an underspecified evaluation protocol, so the significance is conditional on addressing those points.

major comments (4)
  1. [§5.1 (Training), Tables 2–3] The SOTA claim is not yet attributable because the backbone is uncontrolled. The paper only states that training uses an 'inner-developed generative-model-based MLLM for e-commerce' with no parameter count, architecture, or pretraining corpus. Baselines include 2B/3B open models (Qwen2.5-VL-3B, InternVL3-2B) and other retrievers. The ablation in Table 4 shows each module helps on the same backbone, but it does not establish superiority to baselines under controlled capacity. The M5Product classification jump from 78.70% (MM-Embed) and 73.12% (MOON) to 95.50% (MOON2.0) is unexplained. Please provide a same-backbone baseline without the proposed modules, report backbone details, and audit pretraining overlap with M5Product/Fashion200K.
  2. [§5.1 (Evaluation Tasks), Tables 1–3] The evaluation protocol is underspecified. For retrieval, the size and construction of the candidate set, the handling of duplicate query-target pairs, and the negative sampling scheme are not described. For classification and attribute prediction, it is unclear whether the reported numbers come from a linear probe, kNN, label-text embedding matching, or a task-specific head. Without this information, the metrics are not reproducible and cross-model comparisons may be unfair. Please specify the exact evaluation pipeline and, ideally, publish the evaluation code.
  3. [§4.1 (Data Collection and Processing)] The weak-supervision assumption is load-bearing for all MBE2.0 numbers: purchases after a query are treated as positives and skipped exposures with low relevance scores as negatives. Purchases can be driven by price, popularity, recommendation exposure, or browsing position rather than semantic product-match. This affects every training triplet and test query. Dynamic Sample Filtering (Eq. 11) reweights based on the model's own current embedding, so it cannot correct a systematic bias in the definition of positives/negatives. Please validate the supervision with a human agreement study or a popularity-controlled subset, and report how 'low relevance scores' are thresholded and sampled.
  4. [§3.2–3.3, Eq. (8), Eq. (11)] Several design choices that are needed to reproduce the method are missing. The objective weights ω_t, ω_i, ω_mm, ω_pos_intra, ω_neg_intra, temperatures τ and τ̃, MoE regularization weights α and β, the sharpness α in Eq. (11), the number of augmented images n_c, and the CLIP filtering threshold are all unspecified. Only δ=0.6 is given. There is no sensitivity analysis for these hyperparameters. This is a reproducibility concern, especially because the paper claims state-of-the-art results.
minor comments (4)
  1. [§5.2] The text refers to 'CASILE' after Table 1; the baseline is named CASLIE-S in the tables and references. Please correct the typo.
  2. [Figure 2] The figure lacks axis labels and a legend, and the 'dashed line corresponds to model performance following multi-objective joint training' is not visible in the caption. Please clarify.
  3. [Figure 7] The heatmap figure contains the label 'Our DART' in the lower row, though the caption refers to MOON2.0. This seems to be a leftover artifact and should be corrected.
  4. [Table 3] Column headers use 'R10' instead of 'R@10'; make the notation consistent with Tables 1 and 2.

Circularity Check

0 steps flagged

No circularity: MOON2.0's reported gains are empirical, measured on held-out and external benchmarks, and are not derived by construction from fitted parameters or from load-bearing self-citation.

full rationale

The paper's central claim is that MOON2.0 achieves state-of-the-art zero-shot performance on MBE2.0, M5Product, and Fashion200K. This is supported by direct evaluation on held-out test data (Tables 1–3) and by ablation of each proposed component on the same backbone (Table 4). No equation reduces a predicted quantity to a fitted input: Eq. (6)–(7) are standard contrastive objectives, and Eq. (11) is a training-time reliability weight, not a test-time prediction fitted to benchmark labels. The M5Product and Fashion200K results are external benchmarks, so the SOTA claim is falsifiable outside the authors' own MBE2.0 benchmark. The only same-group citation is MOON [50], used as a baseline and as an example of fixed-ratio mixed training, not as load-bearing justification for MOON2.0's architecture or results. Concerns about the weak-supervision assumption in §4.1, the undisclosed proprietary backbone, and the uncontrolled comparison to MOON are correctness and comparability risks, not circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central claim rests on weak user-log supervision, a proprietary base MLLM, and several unreported hyperparameters. The model itself is a standard contrastive pipeline, not a derivation; the main burden is that the benchmark and model come from the same lab and the backbone is opaque. No new physical entities are involved.

free parameters (6)
  • Dynamic Sample Filtering threshold δ = 0.6
    Eq. 11: 'We set a fixed reliability threshold δ=0.6'; chosen by hand, no sensitivity analysis or sweep reported.
  • Dynamic filter sharpness α = not reported
    Eq. 11: alpha controls how sharply reliability weights saturate; no value or sensitivity analysis is given.
  • Contrastive temperatures τ, τ̃ = not reported
    Eqs. 6–7: both inter- and intra-product temperatures affect alignment sharpness; numerical values are omitted.
  • Objective weights ω_t, ω_i, ω_mm, ω_pos_intra, ω_neg_intra, α, β = not reported
    Eq. 8: the holistic loss contains at least seven weighting hyperparameters; none are specified or ablated.
  • Number of augmented images n_c = not reported
    §3.2: the formulation includes n_c augmented images per product; the actual count and generation prompts are not stated.
  • CLIP filtering threshold for generated images = not reported
    §3.3: CLIP similarity filters low-quality augmented images, but the cutoff and failure rate are not reported.
axioms (5)
  • domain assumption A purchase following a search query marks the item as a semantically relevant positive; skipped low-relevance exposures mark negatives.
    §4.1: user-log data is converted into triplets under this label assumption; if purchases reflect popularity or recommendation bias, the contrastive signal is noisy.
  • domain assumption The internal, unnamed MLLM backbone provides a suitable initialization and its pretraining is not a confound.
    §5.1: 'inner-developed generative-model-based MLLM' is used but never specified; gains on public classification could come from prior exposure to the same product categories.
  • domain assumption CLIP similarity between a generated image and the product title is a valid proxy for augmentation quality.
    §3.3: generated images are filtered by CLIP-title similarity; this assumes the title covers the visual content and the CLIP score reflects semantic consistency.
  • domain assumption M5Product and Fashion200K labels and the zero-shot protocol are used consistently across all methods.
    §5.2: comparisons assume identical evaluation; no protocol details are given to confirm.
  • ad hoc to paper Mean pooling of LLM last-layer hidden states yields a representative product embedding.
    §3.2: representation is obtained by mean pooling; no evidence this pooling is optimal or comparable to baselines' pooling choices.

pith-pipeline@v1.3.0-alltime-deepseek · 16582 in / 16887 out tokens · 137299 ms · 2026-08-03T22:01:04.332920+00:00 · methodology

0 comments
read the original abstract

Recent Multimodal Large Language Models (MLLMs) have significantly advanced e-commerce product understanding. However, they still face three challenges: (i) the modality imbalance induced by modality mixed training; (ii) underutilization of the intrinsic alignment relationships among visual and textual information within a product; and (iii) limited handling of noise in e-commerce multimodal data. To address these, we propose MOON2.0, a dynamic modality-balanced MultimOdal representation learning framework for e-commerce prOduct uNderstanding. It comprises: (1) a Modality-driven Mixture-of-Experts (MoE) that adaptively processes input samples by their modality composition, enabling Multimodal Joint Learning to mitigate the modality imbalance; (2) a Dual-level Alignment method to better leverage semantic alignment properties inside individual products; and (3) an MLLM-based Image-text Co-augmentation strategy that integrates textual enrichment with visual expansion, coupled with Dynamic Sample Filtering to improve training data quality. We further release MBE2.0, a co-augmented Multimodal representation Benchmark for E-commerce representation learning and evaluation at https://huggingface.co/datasets/ZHNie/MBE2.0. Experiments show that MOON2.0 delivers state-of-the-art zero-shot performance on MBE2.0 and multiple public datasets. Furthermore, attention-based heatmap visualization provides qualitative evidence of improved multimodal alignment of MOON2.0.

Figures

Figures reproduced from arXiv: 2511.12449 by Bo Zheng, Chenghan Fu, Daoze Zhang, Jian Xu, Junxian Wu, Pengjie Wang, Wanxian Guan, Zhanheng Nie.

Figure 1
Figure 1. Figure 1: Overall results on all the downstream tasks. The arrows [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Modality imbalance under the training set of mixed train [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Pipeline of our MOON2.0. Given a training triplet consisting of a query, a positive item, and a negative item, the model processes each element into three input modalities: multimodal (x mm, combining both image and text), image-only (x i ), and text-only (x t ). In addition, positive item and negative item will further include enriched title and augmented images. on MBE2.0 and public datasets show that MO… view at source ↗
Figure 4
Figure 4. Figure 4: (a) Modality-driven MoE. We adopt the MoE mod￾ule for the feed-forward layers of the LLM backbone. Each of {hˆq, hˆp, hˆn} includes hidden states for t, i, mm input modalities. (b) Dual-level Alignment. Besides inter-product alignment, we in￾troduce intra-product alignment to further leverage the semantic consistency within the e-commerce products. The arrow symbol (→) denotes the alignment relationship. S… view at source ↗
Figure 5
Figure 5. Figure 5: The MLLM-based Image-text Co-augmentation pipeline. (q, p, n), denoting a query, a positive, and a negative. For each triplet, we apply the following contrastive objective: L φ inter = − log exp r φ q · r mm p /τ exp r φ q · r mm p /τ + P Nq exp (r φ q · r mm n /τ ) , (6) where r φ q (φ ∈ {t, i, mm}) is the representation of the text￾only, image-only, or multimodal query, and r mm p , rmm n are the multi… view at source ↗
Figure 6
Figure 6. Figure 6: Illustration of one training sample in our [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The heatmap visualization of our MOON2.0. Fashion200K benchmark. MOON2.0 demonstrates compet￾itive performance on both datasets, further validating its effectiveness on our proposed benchmark and on public datasets. These findings confirm the robustness and general￾ity of the proposed improvements across diverse evaluation tasks and data distributions. 5.3. Ablation Study To quantify the contribution of th… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. JD Oxygen AI Item Center (Oxygen AIIC) V1: An Industrial-Scale LLM/VLM-Centric Solution for Item Understanding, Management, and Applications

    cs.AI 2026-06 unverdicted novelty 3.0

    Oxygen AIIC is an industrial platform using LLMs and VLMs for scalable item knowledge production and service at JD.com, reporting 94.2% precision and 82.8% recall along with business metric improvements.

  2. JD Oxygen AI Item Center (Oxygen AIIC) V1: An Industrial-Scale LLM/VLM-Centric Solution for Item Understanding, Management, and Applications

    cs.AI 2026-06 unverdicted novelty 2.0

    JD.com's Oxygen AIIC applies LLMs/VLMs with a Semantic Search then Discrimination architecture and self-evolving models to produce item knowledge at industrial scale, reporting 94.2% precision and 82.8% recall.

Reference graph

Works this paper leans on

56 extracted references · 19 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [2]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736,

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736,

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 1(2):3,

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities.arXiv preprint arXiv:2308.12966, 1(2):3,

    Pith/arXiv arXiv

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 7

    Pith/arXiv arXiv 2025

  5. [5]

    Product2vec: Leveraging representation learning to model consumer product choice in large assortments.NYU Stern School of Business, 2022

    Fanglin Chen, Xiao Liu, Davide Proserpio, and Isamar Tron- coso. Product2vec: Leveraging representation learning to model consumer product choice in large assortments.NYU Stern School of Business, 2022. 3

    2022

  6. [6]

    On scaling up a multilingual vision and language model

    Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Se- bastian Goodman, Xiao Wang, Yi Tay, et al. On scaling up a multilingual vision and language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14432–14444, 2024. 3

    2024

  7. [7]

    Contrastive language and vi- sion learning of general fashion concepts.Scientific Reports, 12(1):18958, 2022

    Patrick John Chia, Giuseppe Attanasio, Federico Bianchi, Silvia Terragni, Ana Rita Magalhaes, Diogo Goncalves, Ciro Greco, and Jacopo Tagliabue. Contrastive language and vi- sion learning of general fashion concepts.Scientific Reports, 12(1):18958, 2022. 2, 3, 7

    2022

  8. [8]

    Unified generative and discriminative training for multi-modal large language models.Advances in Neural Information Processing Systems, 37:23155–23190, 2024

    Wei Chow, Juncheng Li, Qifan Yu, Kaihang Pan, Hao Fei, Zhiqi Ge, Shuai Yang, Siliang Tang, Hanwang Zhang, and Qianru Sun. Unified generative and discriminative training for multi-modal large language models.Advances in Neural Information Processing Systems, 37:23155–23190, 2024. 2

    2024

  9. [9]

    Uniembedding: Learning universal multi- modal multi-domain item embeddings via user-view con- trastive learning

    Boqi Dai, Zhaocheng Du, Jieming Zhu, Jintao Xu, De- qing Zou, Quanyu Dai, Zhenhua Dong, Rui Zhang, and Hai-Tao Zheng. Uniembedding: Learning universal multi- modal multi-domain item embeddings via user-view con- trastive learning. InProceedings of the 33rd ACM Interna- tional Conference on Information and Knowledge Manage- ment, pages 4446–4453, 2024. 1, 3

    2024

  10. [10]

    M5product: Self- harmonized contrastive learning for e-commercial multi- modal pretraining

    Xiao Dong, Xunlin Zhan, Yangxin Wu, Yunchao Wei, Michael C Kampffmeyer, Xiaoyong Wei, Minlong Lu, Yaowei Wang, and Xiaodan Liang. M5product: Self- harmonized contrastive learning for e-commercial multi- modal pretraining. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 21252–21262, 2022. 2, 7

    2022

  11. [11]

    Pmr: Prototypical modal rebalance for multi- modal learning

    Yunfeng Fan, Wenchao Xu, Haozhao Wang, Junxiao Wang, and Song Guo. Pmr: Prototypical modal rebalance for multi- modal learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20029– 20038, 2023. 2

    2023

  12. [12]

    Switch transformers: Scaling to trillion parameter models with sim- ple and efficient sparsity.Journal of Machine Learning Re- search, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with sim- ple and efficient sparsity.Journal of Machine Learning Re- search, 23(120):1–39, 2022. 4, 5

    2022

  13. [13]

    Moon embedding: Multi- modal representation learning for e-commerce search adver- tising.arXiv preprint, 2025

    Chenghan Fu, Daoze Zhang, Yukang Lin, Zhanheng Nie, Xi- ang Zhang, Jianyu Liu, Yueran Liu, Wanxian Guan, Pengjie Wang, Jian Xu, and Bo Zheng. Moon embedding: Multi- modal representation learning for e-commerce search adver- tising.arXiv preprint, 2025. 1

    2025

  14. [14]

    Fashionbert: Text and im- age matching with adaptive loss for cross-modal retrieval

    Dehong Gao, Linbo Jin, Ben Chen, Minghui Qiu, Peng Li, Yi Wei, Yi Hu, and Hao Wang. Fashionbert: Text and im- age matching with adaptive loss for cross-modal retrieval. In Proceedings of the 43rd International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, pages 2251–2260, 2020. 3

    2020

  15. [15]

    Compodiff: Versa- tile composed image retrieval with latent diffusion.arXiv preprint arXiv:2303.11916, 2023

    Geonmo Gu, Sanghyuk Chun, Wonjae Kim, HeeJae Jun, Yoohoon Kang, and Sangdoo Yun. Compodiff: Versa- tile composed image retrieval with latent diffusion.arXiv preprint arXiv:2303.11916, 2023. 6

    Pith/arXiv arXiv 2023

  16. [16]

    Multi-modal preference modeling for product search

    Yangyang Guo, Zhiyong Cheng, Liqiang Nie, Xin-Shun Xu, and Mohan Kankanhalli. Multi-modal preference modeling for product search. InProceedings of the 26th ACM interna- tional conference on Multimedia, pages 1865–1873, 2018. 1

    2018

  17. [17]

    Au- tomatic spatially-aware fashion concept discovery

    Xintong Han, Zuxuan Wu, Phoenix X Huang, Xiao Zhang, Menglong Zhu, Yuan Li, Yang Zhao, and Larry S Davis. Au- tomatic spatially-aware fashion concept discovery. InPro- ceedings of the IEEE international conference on computer vision, pages 1463–1471, 2017. 2, 7

    2017

  18. [18]

    Multimodal retrieval in e-commerce: From categories to images, text, and back

    Mariya Hendriksen. Multimodal retrieval in e-commerce: From categories to images, text, and back. InEuropean Con- ference on Information Retrieval, pages 505–512. Springer,

  19. [19]

    A multi- modal recommender system for large-scale assortment gen- eration in e-commerce.arXiv preprint arXiv:1806.11226,

    Murium Iqbal, Adair Kovac, and Kamelia Aryafar. A multi- modal recommender system for large-scale assortment gen- eration in e-commerce.arXiv preprint arXiv:1806.11226,

    Pith/arXiv arXiv

  20. [20]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. InInternational conference on machine learning, pages 4904–4916. PMLR,

  21. [21]

    Mrse: An efficient multi-modality retrieval system for large scale e-commerce.arXiv preprint arXiv:2408.14968, 2024

    Hao Jiang, Haoxiang Zhang, Qingshan Hou, Chaofeng Chen, Weisi Lin, Jingchang Zhang, and Annan Wang. Mrse: An efficient multi-modality retrieval system for large scale e-commerce.arXiv preprint arXiv:2408.14968, 2024. 1

    Pith/arXiv arXiv 2024

  22. [22]

    Vlm2vec: Training vision-language models for massive multimodal embedding tasks.arXiv preprint arXiv:2410.05160, 2024

    Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. Vlm2vec: Training vision-language models for massive multimodal embedding tasks.arXiv preprint arXiv:2410.05160, 2024. 2

    Pith/arXiv arXiv 2024

  23. [23]

    Learn- ing instance-level representation for large-scale multi-modal pretraining in e-commerce

    Yang Jin, Yongzhi Li, Zehuan Yuan, and Yadong Mu. Learn- ing instance-level representation for large-scale multi-modal pretraining in e-commerce. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11060–11069, 2023. 1

    2023

  24. [24]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.arXiv preprint arXiv:2407.07895, 2024. 2

    Pith/arXiv arXiv 2024

  25. [25]

    Align before fuse: Vision and language representation learn- ing with momentum distillation.Advances in neural infor- mation processing systems, 34:9694–9705, 2021

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learn- ing with momentum distillation.Advances in neural infor- mation processing systems, 34:9694–9705, 2021. 3

    2021

  26. [26]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InInterna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 3

    2022

  27. [27]

    Embedding-based product retrieval in taobao search

    Sen Li, Fuyu Lv, Taiwei Jin, Guli Lin, Keping Yang, Xiaoyi Zeng, Xiao-Ming Wu, and Qianli Ma. Embedding-based product retrieval in taobao search. InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3181–3189, 2021. 1

    2021

  28. [28]

    Adversarial multimodal representation learning for click-through rate prediction

    Xiang Li, Chao Wang, Jiwei Tan, Xiaoyi Zeng, Dan Ou, Dan Ou, and Bo Zheng. Adversarial multimodal representation learning for click-through rate prediction. InProceedings of The Web Conference 2020, pages 827–836, 2020. 1

    2020

  29. [29]

    Uniecs: Unified multimodal e-commerce search framework with gated cross-modal fusion.arXiv preprint arXiv:2508.13843, 2025

    Zihan Liang, Yufei Ma, ZhiPeng Qian, Huangyu Dai, Zi- han Wang, Ben Chen, Chenyi Lei, Yuqing Ding, and Han Li. Uniecs: Unified multimodal e-commerce search framework with gated cross-modal fusion.arXiv preprint arXiv:2508.13843, 2025. 3

    Pith/arXiv arXiv 2025

  30. [30]

    Mm-embed: Universal multimodal retrieval with multimodal llms.arXiv preprint arXiv:2411.02571, 2024

    Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. Mm-embed: Universal multimodal retrieval with multimodal llms.arXiv preprint arXiv:2411.02571, 2024. 3, 7

    Pith/arXiv arXiv 2024

  31. [31]

    Captions speak louder than images (caslie): Generalizing foundation models for e-commerce from high-quality multimodal instruction data.arXiv preprint arXiv:2410.17337, 2024

    Xinyi Ling, Bo Peng, Hanwen Du, Zhihui Zhu, and Xia Ning. Captions speak louder than images (caslie): Generalizing foundation models for e-commerce from high-quality multimodal instruction data.arXiv preprint arXiv:2410.17337, 2024. 2, 3, 7

    arXiv 2024

  32. [32]

    Ecom- mmmu: Strategic utilization of visuals for robust multimodal e-commerce models.arXiv preprint arXiv:2508.15721,

    Xinyi Ling, Hanwen Du, Zhihui Zhu, and Xia Ning. Ecom- mmmu: Strategic utilization of visuals for robust multimodal e-commerce models.arXiv preprint arXiv:2508.15721,

    arXiv

  33. [33]

    Multimodal pretraining, adaptation, and generation for recommendation: A survey

    Qijiong Liu, Jieming Zhu, Yanting Yang, Quanyu Dai, Zhaocheng Du, Xiao-Ming Wu, Zhou Zhao, Rui Zhang, and Zhenhua Dong. Multimodal pretraining, adaptation, and generation for recommendation: A survey. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Dis- covery and Data Mining, pages 6566–6576, 2024. 1

    2024

  34. [34]

    Multimodal pre-training with self-distillation for prod- uct understanding in e-commerce

    Shilei Liu, Lin Li, Jun Song, Yonghua Yang, and Xiaoyi Zeng. Multimodal pre-training with self-distillation for prod- uct understanding in e-commerce. InProceedings of the Six- teenth ACM International Conference on Web Search and Data Mining, pages 1039–1047, 2023. 2

    2023

  35. [35]

    Pretraining representations of multi-modal multi-query e- commerce search

    Xinyi Liu, Wanxian Guan, Lianyun Li, Hui Li, Chen Lin, Xubin Li, Si Chen, Jian Xu, Hongbo Deng, and Bo Zheng. Pretraining representations of multi-modal multi-query e- commerce search. InProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3429–3437, 2022. 1

    2022

  36. [36]

    ecellm: generalizing large language models for e-commerce from large-scale, high-quality instruction data

    Bo Peng, Xinyi Ling, Ziru Chen, Huan Sun, and Xia Ning. ecellm: generalizing large language models for e-commerce from large-scale, high-quality instruction data. InProceed- ings of the 41st International Conference on Machine Learn- ing, pages 40215–40257, 2024. 2

    2024

  37. [37]

    Kosmos-2: Grounding multimodal large language models to the world, 2023

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world, 2023. 3

    2023

  38. [38]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021. 3

    2021

  39. [39]

    Outra- geously large neural networks: The sparsely-gated mixture- of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outra- geously large neural networks: The sparsely-gated mixture- of-experts layer.arXiv preprint arXiv:1701.06538, 2017. 4

    Pith/arXiv arXiv 2017

  40. [40]

    Llama-e: Empowering e-commerce authoring with multi-aspect instruction following.arXiv preprint arXiv:2308.04913, 2023

    Kaize Shi, Xueyao Sun, Dingxian Wang, Yinlin Fu, Guan- dong Xu, and Qing Li. Llama-e: Empowering e-commerce authoring with multi-aspect instruction following.arXiv preprint arXiv:2308.04913, 2023. 2

    Pith/arXiv arXiv 2023

  41. [41]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. 3

    Pith/arXiv arXiv 2023

  42. [42]

    Merlin: Multimodal & multilingual embed- ding for recommendations at large-scale via item associa- tions

    Sambeet Tiady, Arihant Jain, Dween Rabius Sanny, Khushi Gupta, Srinivas Virinchi, Swapnil Gupta, Anoop Saladi, and Deepak Gupta. Merlin: Multimodal & multilingual embed- ding for recommendations at large-scale via item associa- tions. InProceedings of the 33rd ACM International Con- ference on Information and Knowledge Management, pages 4914–4921, 2024. 2

    2024

  43. [43]

    Stablerep: Synthetic images from text-to- image models make strong visual representation learners

    Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to- image models make strong visual representation learners. Advances in Neural Information Processing Systems, 36: 48382–48402, 2023. 6

    2023

  44. [44]

    Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muham- mad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language en- coders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025. 7

    Pith/arXiv arXiv 2025

  45. [45]

    Missrec: Pre-training and transfer- ring multi-modal interest-aware sequence representation for recommendation

    Jinpeng Wang, Ziyun Zeng, Yunxiao Wang, Yuting Wang, Xingyu Lu, Tianxiang Li, Jun Yuan, Rui Zhang, Hai-Tao Zheng, and Shu-Tao Xia. Missrec: Pre-training and transfer- ring multi-modal interest-aware sequence representation for recommendation. InProceedings of the 31st ACM Interna- tional Conference on Multimedia, pages 6548–6557, 2023. 1

    2023

  46. [46]

    Mim: Multi-modal content interest mod- eling paradigm for user behavior modeling.arXiv preprint arXiv:2502.00321, 2025

    Bencheng Yan, Si Chen, Shichang Jia, Jianyu Liu, Yueran Liu, Chenghan Fu, Wanxian Guan, Hui Zhao, Xiang Zhang, Kai Zhang, et al. Mim: Multi-modal content interest mod- eling paradigm for user behavior modeling.arXiv preprint arXiv:2502.00321, 2025. 1

    Pith/arXiv arXiv 2025

  47. [47]

    Commercemm: Large-scale commerce multimodal representation learning with omni retrieval

    Licheng Yu, Jun Chen, Animesh Sinha, Mengjiao Wang, Yu Chen, Tamara L Berg, and Ning Zhang. Commercemm: Large-scale commerce multimodal representation learning with omni retrieval. InProceedings of the 28th ACM SIGKDD conference on knowledge discovery and data min- ing, pages 4433–4442, 2022. 1, 2, 3

    2022

  48. [48]

    Florence: A new foundation model for computer vision.arXiv preprint arXiv:2111.11432, 2021

    Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision.arXiv preprint arXiv:2111.11432, 2021. 3

    Pith/arXiv arXiv 2021

  49. [49]

    Prod- uct1m: Towards weakly supervised instance-level product retrieval via cross-modal pretraining

    Xunlin Zhan, Yangxin Wu, Xiao Dong, Yunchao Wei, Min- long Lu, Yichi Zhang, Hang Xu, and Xiaodan Liang. Prod- uct1m: Towards weakly supervised instance-level product retrieval via cross-modal pretraining. InProceedings of the IEEE/CVF international conference on computer vision, pages 11782–11791, 2021. 2

    2021

  50. [50]

    Moon: Generative mllm-based multimodal representation learning for e-commerce product understand- ing.arXiv preprint arXiv:2508.11999, 2025

    Daoze Zhang, Zhanheng Nie, Jianyu Liu, Chenghan Fu, Wanxian Guan, Yuan Gao, Jun Song, Pengjie Wang, Jian Xu, and Bo Zheng. Moon: Generative mllm-based multimodal representation learning for e-commerce product understand- ing.arXiv preprint arXiv:2508.11999, 2025. 2, 3, 7

    arXiv 2025

  51. [51]

    Sharper and faster mean better: Towards more efficient vision-language model for hour-scale long video understand- ing

    Daoze Zhang, Yuze Zhao, Jintao Huang, and Yingda Chen. Sharper and faster mean better: Towards more efficient vision-language model for hour-scale long video understand- ing. InProceedings of the 63rd Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Pa- pers), pages 4423–4439, 2025. 3

    2025

  52. [52]

    Modality-balanced learning for multimedia recom- mendation

    Jinghao Zhang, Guofan Liu, Qiang Liu, Shu Wu, and Liang Wang. Modality-balanced learning for multimedia recom- mendation. InProceedings of the 32nd ACM International Conference on Multimedia, pages 7551–7560, 2024. 2

    2024

  53. [53]

    Gme: Improving universal multimodal retrieval by multimodal llms.arXiv preprint arXiv:2412.16855, 2024

    Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. Gme: Improving universal multimodal retrieval by multimodal llms.arXiv preprint arXiv:2412.16855, 2024. 2, 3, 6, 7

    Pith/arXiv arXiv 2024

  54. [54]

    Delving into e-commerce product retrieval with vision-language pre-training

    Xiaoyang Zheng, Fuyu Lv, Zilong Wang, Qingwen Liu, and Xiaoyi Zeng. Delving into e-commerce product retrieval with vision-language pre-training. InProceedings of the 46th International ACM SIGIR Conference on Research and De- velopment in Information Retrieval, pages 3385–3389, 2023. 1

    2023

  55. [55]

    Megapairs: Massive data synthesis for universal mul- timodal retrieval.arXiv preprint arXiv:2412.14475, 2024

    Junjie Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, and Yongping Xiong. Megapairs: Massive data synthesis for universal mul- timodal retrieval.arXiv preprint arXiv:2412.14475, 2024. 7

    Pith/arXiv arXiv 2024

  56. [56]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shen- glong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 7

    Pith/arXiv arXiv 2025