Pith. sign in

REVIEW 4 major objections 3 minor 55 references

A two-stage training recipe lets a dense multimodal retriever match reasoning-heavy models at near-zero latency overhead.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 13:42 UTC pith:BBK6ZLJI

load-bearing objection A plausible industrial recipe with a clean ablation, but the load-bearing teacher-supervision assumption is unvalidated and the reported gains need error bars before they are taken as SOTA. the 4 major comments →

arxiv 2608.02148 v1 pith:BBK6ZLJI submitted 2026-08-03 cs.IR cs.CLcs.CV

Douyin Multimodal Embedding Model Technical Report

Haonan Chen , Chu Li , Zhicheng Wang , Yuanwei Liu , Yuanjiang Wang , Shaohua Jiang , Zhicheng Dou This is my paper
classification cs.IR cs.CLcs.CV
keywords multimodal embeddingretrievallatent reasoningcontrastive learningsemantic sufficiencyreconstruction bottleneckNTP/MTPbi-encoder
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that fine-grained, evidence-aware retrieval does not require explicit chain-of-thought generation or reranking during serving. Instead, it trains a standard bi-encoder in two stages: first, large-scale contrastive pre-training builds a broad multimodal embedding space; second, two training-only mechanisms — Evidence-Grounded Typed Latent Reasoning and Cross-Conditional Reconstruction — inject evidence grounding and counterpart-side semantics into the embedding itself. If correct, this means the retrieval encoder can stay cheap to serve while retaining the fine-grained discrimination usually associated with much heavier reasoning-based models. The paper reports state-of-the-art scores on MMEB-v2 at both 2B and 9B scales, plus measurable gains in Douyin's production search, suggesting the recipe transfers beyond academic benchmarks.

Core claim

The central claim is that semantic sufficiency — an embedding being grounded in retrieval-relevant evidence and preserving fine-grained counterpart information — can be learned through latent, training-time objectives rather than explicit generation. Stage 2-A introduces anchor tokens that localize evidence (text spans, image regions, video frames) and typed latent states (localize, align_pos, reject_neg, summarize) that organize retrieval roles in the hidden space. Stage 2-B uses the same embedding as a prefix condition to reconstruct counterpart-side text via next-token and multi-token prediction, forcing the vector to retain token-level semantics. At inference, latent tokens stay inside o

What carries the argument

The load-bearing mechanism is the Stage-2 readout with two intertwined components: (1) evidence-grounded typed latent reasoning, where query-side anchor tokens attend to localized evidence and a few typed latent states organize retrieval roles, with the terminal state plus pooled evidence forming the readout; and (2) cross-conditional reconstruction, where the pre-normalization embedding is fed back as a prefix token and used to decode the counterpart's text under NTP and MTP losses. These objectives act only during training, turning the embedding itself into a semantic bottleneck that must be both evidence-aware and information-complete, while inference stays a single encoder pass.

Load-bearing premise

Stage-2A's measured improvement depends on the teacher model's structured evidence and typed trajectories being correct and correctly aligned to the input; if those synthetic annotations are noisy or misaligned, the evidence-grounding losses train the model to attend to the wrong locations with the wrong roles.

What would settle it

A human-annotation audit on a sample of training triplets, comparing teacher-marked text spans, image regions, and frame references against independent human judgments of what justifies relevance; low overlap (e.g., below about 60%) would predict that Stage-2A's contribution does not come from genuine evidence grounding. Additionally, a matched control with shuffled or randomized teacher targets — if the model still improves, the specific supervision content is not the causal factor.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Retrieval systems can adopt reasoning-style discrimination without sacrificing billion-scale index serving or online query latency.
  • The representation-completeness metric (teacher-forced token recover accuracy) provides a bounded, interpretable proxy for semantic sufficiency that can guide data selection and embedding optimization in industrial settings.
  • If the recipe generalizes, other dense-retrieval domains (recommendation, agent tool retrieval, cross-modal search) could use latent-reasoning objectives to inject evidence grounding without changing the serving interface.
  • The reported gains concentrate in video and visual-document tasks, where localized evidence such as frames and OCR text matters most, suggesting those are the highest-value application targets.
  • Even a small online metric gain (0.1% Lifetime) is treated as consequential, indicating that retrieval quality improvements translate directly to business-relevant outcomes in large-scale platforms.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The latent reasoning here is effectively a differentiable, soft version of chain-of-thought; a natural extension is to adapt the number and roles of latent states per query or task, which could further improve the quality-efficiency frontier without changing the architecture.
  • A control experiment that replaces teacher-generated trajectories with random or heuristic supervision could isolate whether Stage-2A's 1.3-point gain comes from the semantic content of the teacher annotations or from the architectural change itself.
  • The 'information-complete' claim could be stress-tested by conditioning decoding on a random projection of the same hidden state; if recovery accuracy stays high, the reconstruction loss may be measuring language-prior regularities rather than embedding content.
  • The production A/B result is reported as a single aggregate number; segment-level stability (e.g., across query types, modalities, or traffic slices) is left open and would clarify where the gains are robust.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes DME, a two-stage multimodal embedding model for large-scale retrieval. Stage 1 performs large-scale contrastive pre-training over 25M heterogeneous query-document pairs. Stage 2 adds two training-only mechanisms: Evidence-Grounded Typed Latent Reasoning, which uses anchor tokens and typed latent states supervised by teacher-generated evidence records, and Cross-Conditional Reconstruction, which uses NTP and MTP to reconstruct counterpart text from the embedding. On MMEB-v2, DME-2B and DME-9B achieve 74.8 and 78.4, respectively, which the authors describe as state-of-the-art at comparable scales. The paper also reports a cumulative ablation (Table 3), a representation completeness metric (acc@K), query-side latency measurements, and production results (2.92% offline relative gain, 0.1% online LT gain).

Significance. If the results are correct, the core design is valuable: it combines bi-encoder serving efficiency with fine-grained retrieval discrimination by moving reasoning into hidden-space latent tokens and enforcing counterpart-side semantic preservation through reconstruction. The external anchor to MMEB-v2 and the cumulative ablation in Table 3 are appropriate and give the main retrieval claim a solid empirical base. The latency analysis and production deployment report are also useful practical evidence. The main weakness is that the Stage 2-A mechanism—the source of the largest per-domain gains—relies entirely on unvalidated teacher-generated supervision, so the causal attribution of those gains to evidence grounding rather than added model capacity is not yet established. The paper's significance would be materially higher if this gap were closed with a control experiment.

major comments (4)
  1. [§4.1, §4.3, Table 3] The Stage 2-A gains (Table 3: +1.3 overall, +4.4 on video) are attributed to Evidence-Grounded Typed Latent Reasoning, but every supervision signal for this mechanism—anchor hit loss (Eq. 36), summary loss (Eq. 43), typed latent losses (Eqs. 49–51)—is derived from Seed-2.0-Pro-generated structured records. The paper reports no human agreement check, no alignment error rate, and no analysis of failure cases in the token/patch/frame-level alignment described in §4.1. Since +Stage2-A also adds anchor tokens, trajectory tokens, and a fused readout, the observed improvement could come from extra latent capacity rather than correct evidence grounding. A required control is to shuffle or randomize the teacher evidence targets while keeping the same token counts and loss structure; if the gain persists, the evidence-grounding interpretation is unsupported. Reporting basic alignment statistics (e
  2. [§5.2, Table 1] The headline claim 'state-of-the-art at comparable scales' is based on single numbers from the MMEB-v2 leaderboard, without error bars, multiple seeds, or significance tests. The margins are not always large: DME-2B leads Qwen3-VL-Embedding-2B by 1.6 overall, and DME-9B leads Qwen3-VL-Embedding-8B by 0.6 overall. Additionally, the TTE-v2 row is explicitly on a 76-task setting while other rows use 78 tasks, making the comparison not fully apples-to-apples. The authors should either provide repeated-run variance or clearly state that the differences are point estimates, and re-run or re-annotate the TTE-v2 comparative setting so all rows use the identical task set.
  3. [§4.4, Eqs. (21)–(23)] The statement that 'any information required to reconstruct the counterpart text is forced to pass through the embedding' is too strong. The decoder is the same generative backbone with strong language priors, and the input template/instruction is available as context; therefore the reconstruction objective does not isolate what the embedding itself contributes. Table 4's acc@K would likely be nontrivial even for a poorly informed embedding because of these priors. To support the semantic-sufficiency claim, the authors should compare against a no-reconstruction baseline or a random-prefix condition, and demonstrate a clear gap attributable to the embedding content.
  4. [§5.5, Eq. (31)] The representation completeness metric uses teacher-forced top-K accuracy with a shared backbone and the same target distribution used in training. This makes high acc@K partly a measure of language-model memorization/prior predictability rather than a direct measure of embedding informativeness. It would be informative to report the same metric for a model trained with contrastive loss only (or with the reconstruction loss ablated), and to compare against a prefix that is a random vector or a mean-pooled baseline. As written, the claim that DME embeddings are 'information-complete' is not uniquely supported by the presented diagnostic.
minor comments (3)
  1. [§4.1 and §4.5] The paper omits hyperparameters for the Stage-2 losses (λ_hit, λ_sum, λ_bal, λ_sem, λ_align, λ_reject, λ_NTP, λ_MTP), temperatures τ_CL, τ_anc, τ_traj, rejection margin μ, number of anchor/trajectory tokens, MTP depth D, and data mixture ratios. For a technical report intended to guide replication, these values should be listed or referenced.
  2. [Abstract and §5.6] Minor text issues: 'Y ouT ube' appears in the abstract and introduction, and the phrase 'only marginal query-side overhead' is repeated several times. The latency table reports p50 with batch size 4 only; reporting p95 or another tail percentile would be more informative for production serving.
  3. [§5.4, Table 3] The cumulative ablation is helpful, but the caption should clarify that all configurations use the Stage-2 contrastive objective and that 'Stage 2-A' includes both the new supervision and the additional tokens/readout. This makes the marginal attribution to the supervision itself ambiguous, which is related to Major Comment 1.

Circularity Check

1 steps flagged

Core MMEB-v2 retrieval claim is externally anchored and not circular; the only self-referential element is the §5.5 representation-completeness metric, which recasts the NTP training objective as evidence of semantic sufficiency.

specific steps
  1. other [Section 5.5, Eq. (31) vs Eq. (24)]
    "To quantify how much content a single embedding actually preserves, we recast the NTP objective of Eq. (24) as a bounded, interpretable metric. ... This directly operationalizes the notion of semantic sufficiency that motivates DME."

    The completeness metric acc@K in Eq. (31) is computed from the same teacher-forced next-token distribution P_theta of Eq. (23) that Stage 2-B optimizes via the NTP loss in Eq. (24). High acc@K on the training branch is therefore a restatement of successful optimization of the training objective, not an independent verification that the embedding preserves counterpart semantics. The property being confirmed (semantic sufficiency) is quantified by the very loss used to train it, so this supporting evidence is self-referential by construction. This does not affect the externally benchmarked retrieval result, but it does mean the 'information-complete' claim is not independently demonstrated.

full rationale

The central retrieval claim is anchored to an external benchmark: MMEB-v2 scores are compared against official leaderboard baselines, and Table 3 is a cumulative ablation that isolates the marginal contributions of Stage 1, Stage 2-A, and Stage 2-B on the same data. No load-bearing step reduces to a fitted parameter or to a self-citation chain. The self-citations present in the related-work discussion (e.g., Pailitao-VL [8], SAIL-Embedding [33], and the authors' own parallel embedding work [44]) provide context and do not carry the derivation of the reported gains. The identified self-referential element is the §5.5 representation-completeness analysis: Eq. (31) is a bounded reformulation of the NTP objective of Eq. (24), so reporting high recovery accuracy from the same teacher-forced distribution confirms optimization of that objective rather than independently substantiating semantic sufficiency. This is a secondary interpretability claim, not the core MMEB-v2 result, hence the low score. The unvalidated teacher-generated supervision in Stage 2-A is a data-quality/validation concern, not a circularity, because the teacher (Seed-2.0-Pro) is external to the model being trained.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 3 invented entities

The retrieval claims are anchored to an external benchmark (MMEB-v2) and a clean cumulative ablation, so the central result is not self-referential. What the paper pulls from outside without independent evidence: the Qwen3.5 backbone's quality, the Seed-2.0-Pro teacher's annotation accuracy, the validity of MMEB-v2 as the quality proxy, and the premise that reconstructing counterpart text from an embedding enforces retrieval-relevant semantics. All Stage-2 loss weights, temperatures, margins, the MTP depth, and anchor/latent-token counts are undisclosed hand choices. No new physical entities are introduced; the new ML constructs (anchor tokens, typed latent states, the completeness measure) have no independent confirmatory evidence outside the paper's own ablations and self-defined metric.

free parameters (6)
  • Stage-2 loss weights λ_hit, λ_sum, λ_bal, λ_sem, λ_align, λ_reject, λ_NTP, λ_MTP
    All Stage-2 loss terms in Eq. 20/29/30 depend on these hand-chosen weights; no values appear in the paper or Appendix A.
  • Temperatures τ_CL, τ_anc, τ_traj and rejection margin μ
    Scale the contrastive (Eq. 14), anchor-assignment (Eq. 35), trajectory-contrastive (Eq. 50), and rejection (Eq. 51) losses; undisclosed.
  • Readout coefficient α and evidence projection W_e
    Gates how much pooled evidence enters the readout (Eq. 18); α chosen by hand, value undisclosed.
  • Anchor-token and typed-latent-token counts
    'A small number of typed query-side latent states' (§3.3) and R_m anchor tokens (Eq. 33) — the counts are never specified.
  • MTP depth D and completeness truncation L_max = L_max = 10; D undisclosed
    D is the number of MTP prediction depths (Eq. 26-28), undisclosed; L_max=10 is stated in §5.5.
  • Stage-1/Stage-2 data mixture ratios
    25M/5M pair totals are given, but per-dataset sampling ratios are hand-set and undisclosed (§4.1).
axioms (6)
  • domain assumption Qwen3.5 provides a strong generative multimodal backbone whose understanding transfers to embeddings.
    The entire architecture builds on this backbone (§3.2); the paper cites the Qwen3.5 blog [38] rather than demonstrating the transfer.
  • domain assumption Seed-2.0-Pro's structured evidence/trajectory annotations are accurate enough to supervise anchors and latent states.
    Stage 2-A supervision (§4.1, §4.3) depends entirely on teacher outputs; no human agreement check is reported.
  • domain assumption MMEB-v2 aggregate score is a valid proxy for universal multimodal retrieval quality.
    The SOTA claim (§5.2, Table 1) rests on this benchmark and its official protocol.
  • domain assumption Reconstructing counterpart-side text from an embedding enforces retrieval-relevant semantics (semantic sufficiency).
    The paper's own premise for Stage 2-B (§4.4); not established independently of the retrieval gains.
  • standard math Causal attention lets retrieval tokens at sequence end attend to all preceding multimodal content.
    Standard transformer property invoked in §3.2 (Eq. 8-9).
  • domain assumption In-batch negatives with task-aware grouping provide sufficient contrastive signal at both stages.
    Training uses only in-batch negatives in Stage 1 and in-batch plus hard negatives in Stage 2 (§4.1).
invented entities (3)
  • Typed latent states (localize / align_pos / reject_neg / summarize) no independent evidence
    purpose: Hidden-space reasoning roles that organize retrieval evidence before embedding readout (§3.3, §4.3).
    Validated only through the paper's own ablation (Table 3) and teacher-supervised objectives (Eq. 46-51); no outside confirmatory study.
  • Modality-aware anchor tokens no independent evidence
    purpose: Soft-attention evidence grounding over text spans, image regions, and video frames (§4.3, Eq. 15-16).
    Gains attributable to anchors are internal to the recipe ablation; anchor quality depends on unvalidated teacher targets.
  • Representation completeness measure (acc@K) no independent evidence
    purpose: Quantifies 'semantic sufficiency' / information-completeness (§5.5, Eq. 31).
    Defined from the same NTP objective used in training (Eq. 24-28), so it is not independent of the training signal.

pith-pipeline@v1.3.0-daily-deepseek · 26035 in / 22024 out tokens · 162255 ms · 2026-08-04T13:42:14.126615+00:00 · methodology

0 comments
read the original abstract

Multimodal representation learning is a cornerstone of modern AI. By encoding multimodal queries and targets into vectors, it powers industrial search and recommendation and underpins modern agents. Real-world platforms with complex modalities and massive-scale content, such as Douyin, Xiaohongshu, and YouTube, demand both efficiency under billion-scale indexing and fine-grained discrimination for hard matching. Existing MLLM embedding models rarely satisfy both. Contrastive models are efficient but rely on pair-level supervision too coarse for fine-grained distinctions, while CoT-based models improve discrimination through explicit generation impractical to serve online. We present Douyin Multimodal Embedding (DME), a model trained in two stages to combine both strengths. Stage 1 performs large-scale contrastive pre-training that establishes a unified multimodal embedding space with broad modality and task coverage. Stage 2 supplements semantic sufficiency, the property that an embedding is grounded in retrieval-relevant evidence and preserves fine-grained counterpart-side semantics, via two mechanisms. Evidence-Grounded Typed Latent Reasoning organizes retrieval evidence through hidden-space latent reasoning, and Cross-Conditional Reconstruction enforces counterpart-side semantics through cross-directional autoregressive reconstruction. Both act only during training and add only marginal query-side overhead, so DME serves as efficiently as a standard contrastive encoder. On MMEB-v2, DME reaches state-of-the-art results at comparable scales for its 2B and 9B variants (74.8 and 78.4), with especially strong video and visual-document tasks. In production, DME delivers a 2.92% relative gain on Douyin's in-house offline evaluation set, is deployed across Douyin scenarios such as generative, image, and AI search, and yields a 0.1% Lifetime (LT) gain in online A/B testing on Douyin search.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 2 canonical work pages · 1 internal anchor

  1. [1]

    MM-BRIGHT: A multi-task multi- modal benchmark for reasoning-intensive retrieval

    Abdelrahman Abdallah, Mohamed Darwish Mounis, Mahmoud Abdalla, Mahmoud SalahEldin Kasem, Mostafa Farouk Senussi, Mohamed Mahmoud, Mohammed Ali, Adam Jatowt, and Hyun-Soo Kang. MM-BRIGHT: A multi-task multi- modal benchmark for reasoning-intensive retrieval. CoRR, abs/2601.09562, 2026. doi: 10.48550/ARXIV.2601.09562. URL https://doi.org/10.48550/arXiv.2601.09562

    doi:10.48550/arxiv.2601.09562 2026

  2. [2]

    Qwen3-vl technical report

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Y ang Liu, Dayiheng Liu, Shixuan...

    Pith/arXiv arXiv doi:10.48550/arxiv.2511.21631 2025

  3. [3]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Y ang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Y e, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Y ang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical rep...

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.13923 2025

  4. [4]

    Ms marco: A human generated machine reading comprehension dataset

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268, 2016

    Pith/arXiv arXiv 2016

  5. [5]

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558–3568, 2021

    2021

  6. [6]

    mme5: Improving multimodal multilingual embeddings via high-quality synthetic data

    Haonan Chen, Liang Wang, Nan Y ang, Yutao Zhu, Ziliang Zhao, Furu Wei, and Zhicheng Dou. mme5: Improving multimodal multilingual embeddings via high-quality synthetic data. CoRR, abs/2502.08468, 2025. doi: 10.48550/ARXIV.2502.08468. URL https://doi.org/10.48550/arXiv.2502.08468

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.08468 2025

  7. [7]

    Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning

    Jiahan Chen, Da Li, Hengran Zhang, Yinqiong Cai, Lixin Su, Jiafeng Guo, Daiting Shi, Dawei Yin, and Keping Bi. Recon- structing content via collaborative attention to improve multimodal embedding quality. CoRR, abs/2603.01471, 2026. doi: 10.48550/ARXIV.2603.01471. URL https://doi.org/10.48550/arXiv.2603.01471

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2603.01471 2026

  8. [8]

    Pailitao-vl: Unified embedding and reranker for real-time multi-modal industrial search

    Lei Chen, Chen Ju, Xu Chen, Zhicheng Wang, Yuheng Jiao, Hongfeng Zhan, Zhaoyang Li, Shihao Xu, Zhixiang Zhao, Tong Jia, et al. Pailitao-vl: Unified embedding and reranker for real-time multi-modal industrial search. arXiv preprint arXiv:2602.13704, 2026

    arXiv 2026

  9. [9]

    Think then embed: Generative context improves multimodal embedding

    Xuanming Cui, Jianpeng Cheng, Hong-you Chen, Satya Narayan Shukla, Abhijeet Awasthi, Xichen Pan, Chaitanya Ahuja, Shlok Kumar Mishra, Y onghuan Y ang, Jun Xiao, Qi Guo, Ser-Nam Lim, Aashu Singh, and Xiangjun Fan. Think then embed: Generative context improves multimodal embedding. CoRR, abs/2510.05014, 2025. doi: 10.48550/ARXIV.2510.05014. URL https://doi....

    doi:10.48550/arxiv.2510.05014 2025

  10. [10]

    Reason to contrast: A cascaded multimodal retrieval frame- work

    Xuanming Cui, Hong- Y ou Chen, Hao Yu, Hao Yuan, Zihao Wang, Shlok Kumar Mishra, Hanchao Yu, Y onghuan Y ang, Jun Xiao, Ser-Nam Lim, Jianpeng Cheng, Qi Guo, and Xiangjun Fan. Reason to contrast: A cascaded multimodal retrieval frame- work. CoRR, abs/2602.23369, 2026. doi: 10.48550/ARXIV.2602.23369. URL https://doi.org/10.48550/arXiv.2602. 23369

    doi:10.48550/arxiv.2602.23369 2026

  11. [11]

    Deepseek-v3 technical report

    DeepSeek-AI. Deepseek-v3 technical report. CoRR, abs/2412.19437, 2024. doi: 10.48550/ARXIV.2412.19437. URL https: //doi.org/10.48550/arXiv.2412.19437

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.19437 2024

  12. [12]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAA...

    2019

  13. [13]

    Colpali: Ef- ficient document retrieval with vision language models

    Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. Colpali: Ef- ficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Represen- 19 tations, ICLR 2025, Singapore, April 24-28, 2025 . OpenReview.net, 2025. URL https://openreview.net/forum?id= ogjBpZ8uSi

    2025

  14. [14]

    Moon embedding: Multimodal representation learning for e-commerce search advertising

    Chenghan Fu, Daoze Zhang, Yukang Lin, Zhanheng Nie, Xiang Zhang, Jianyu Liu, Yueran Liu, Wanxian Guan, Pengjie Wang, Jian Xu, et al. Moon embedding: Multimodal representation learning for e-commerce search advertising. arXiv preprint arXiv:2511.11305, 2025

    arXiv 2025

  15. [15]

    Better & faster large lan- guage models via multi-token prediction

    Fabian Gloeckle, Badr Y oubi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large lan- guage models via multi-token prediction. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine...

    2024

  16. [16]

    Breaking the modality barrier: Universal embedding learning with multimodal llms

    Tiancheng Gu, Kaicheng Y ang, Ziyong Feng, Xingjun Wang, Y anzhao Zhang, Dingkun Long, Yingda Chen, Weidong Cai, and Jiankang Deng. Breaking the modality barrier: Universal embedding learning with multimodal llms. In Proceedings of the 33rd ACM International Conference on Multimedia , pages 2860–2869, 2025

    2025

  17. [17]

    TRACE: task-adaptive reasoning and representation learning for universal multimodal retrieval

    Xiangzhao Hao, Shijie Wang, Tianyu Y ang, Tianyue Wang, Haiyun Guo, and Jinqiao Wang. TRACE: task-adaptive reasoning and representation learning for universal multimodal retrieval. CoRR, abs/2603.02929, 2026. doi: 10.48550/ARXIV.2603. 02929. URL https://doi.org/10.48550/arXiv.2603.02929

    work page doi:10.48550/arxiv.2603.02929 2026

  18. [18]

    Girshick

    Kaiming He, Xinlei Chen, Saining Xie, Y anghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 15979–15988. IEEE, 2022. doi: 10.1109/CVPR52688.2022.01553. URL https://doi.org/10. 1109/CVPR52...

    arXiv 2022

  19. [19]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Y e, Fei Zhao, Zhe Xu, Xu Tang, Y ao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig

    Chao Jia, Yinfei Y ang, Y e Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V . Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, ...

    2021

  21. [21]

    Rzenembed: Towards compre- hensive multimodal retrieval

    Weijian Jian, Y ajun Zhang, Dawei Liang, Chunyu Xie, Yixiao He, Dawei Leng, and Yuhui Yin. Rzenembed: Towards compre- hensive multimodal retrieval. arXiv preprint arXiv:2510.27350, 2025

    arXiv 2025

  22. [22]

    Embed-rl: Rein- forcement learning for reasoning-driven multimodal embeddings

    Haonan Jiang, Yuji Wang, Y ongjie Zhu, Xin Lu, Wenyu Qin, Meng Wang, Pengfei Wan, and Y ansong Tang. Embed-rl: Rein- forcement learning for reasoning-driven multimodal embeddings. CoRR, abs/2602.13823, 2026. doi: 10.48550/ARXIV.2602. 13823. URL https://doi.org/10.48550/arXiv.2602.13823

    doi:10.48550/arxiv.2602.13823 2026

  23. [23]

    E5-V: universal embeddings with multimodal large language models

    Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-V: universal embeddings with multimodal large language models. CoRR, abs/2407.12580, 2024. doi: 10.48550/ ARXIV.2407.12580. URL https://doi.org/10.48550/arXiv.2407.12580

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.12580 2024

  24. [24]

    Vlm2vec: Training vision-language models for massive multimodal embedding tasks

    Ziyan Jiang, Rui Meng, Xinyi Y ang, Semih Y avuz, Yingbo Zhou, and Wenhu Chen. Vlm2vec: Training vision-language models for massive multimodal embedding tasks. arXiv preprint arXiv:2410.05160, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, 2017

    2017

  26. [26]

    Natural questions: a benchmark for question answering research

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics , 7:453–466, 2019

    2019

  27. [27]

    UME-R1: exploring reasoning-driven generative multi- modal embeddings

    Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, and Jinsong Su. UME-R1: exploring reasoning-driven generative multi- modal embeddings. CoRR, abs/2511.00405, 2025. doi: 10.48550/ARXIV.2511.00405. URL https://doi.org/10.48550/ arXiv.2511.00405

    doi:10.48550/arxiv.2511.00405 2025

  28. [28]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, 20...

    2022

  29. [29]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Ho...

    2023

  30. [30]

    Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking

    Mingxin Li, Y anzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Y ang, Pengjun Xie, An Y ang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking. CoRR, abs/2601.04720, 2026. doi: 10.48550/ARXIV.2601.04720. URL https://doi. org/10...

    Pith/arXiv arXiv doi:10.48550/arxiv.2601.04720 2026

  31. [31]

    U-marvel: Unveiling key factors for universal multimodal retrieval via embedding learning with mllms

    Xiaojie Li, Chu Li, Shi-Zhe Chen, and Xi Chen. U-marvel: Unveiling key factors for universal multimodal retrieval via embedding learning with mllms. arXiv preprint arXiv:2507.14902, 2025

    arXiv 2025

  32. [32]

    Towards general text embeddings with multi-stage contrastive learning

    Zehan Li, Xin Zhang, Y anzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. CoRR, abs/2308.03281, 2023. doi: 10.48550/ARXIV.2308.03281. URL https://doi. org/10.48550/arXiv.2308.03281

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.03281 2023

  33. [33]

    Sail-embedding technical report: Omni-modal embedding foundation model

    Lin Lin, Jiefeng Long, Zhihe Wan, Yuchi Wang, Dingkang Y ang, Shuang Y ang, Yueyang Y ao, Xu Chen, Zirui Guo, Shengqiang Li, et al. Sail-embedding technical report: Omni-modal embedding foundation model. arXiv preprint arXiv:2510.12709, 2025

    arXiv 2025

  34. [34]

    CREM: compression-driven representation enhancement for multimodal retrieval and comprehension

    Lihao Liu, Y an Wang, Biao Y ang, Da Li, Jiangxia Cao, Yuxiao Luo, Xiang Chen, Xiangyu Wu, Wei Yuan, Fan Y ang, Guiguang Ding, Tingting Gao, and Guorui Zhou. CREM: compression-driven representation enhancement for multimodal retrieval and comprehension. CoRR, abs/2602.19091, 2026. doi: 10.48550/ARXIV.2602.19091. URL https://doi.org/10.48550/ arXiv.2602.19091

    doi:10.48550/arxiv.2602.19091 2026

  35. [35]

    Vlm2vec-v2: Advancing multimodal embedding for videos, images, and visual documents

    Rui Meng, Ziyan Jiang, Y e Liu, Mingyi Su, Xinyi Y ang, Yuepeng Fu, Can Qin, Raghuveer Thirukovalluru, Xuan Zhang, Zeyuan Chen, Ran Xu, Caiming Xiong, Yingbo Zhou, Wenhu Chen, and Semih Y avuz. Vlm2vec-v2: Advancing multimodal embedding for videos, images, and visual documents. Trans. Mach. Learn. Res. , 2026, 2026. URL https://openreview. net/forum?id=TpU38jbKIJ

    2026

  36. [36]

    Zhanheng Nie, Chenghan Fu, Daoze Zhang, Junxian Wu, Wanxian Guan, Pengjie Wang, Jian Xu, and Bo Zheng. Moon2. 0: Dynamic modality-balanced multimodal representation learning for e-commerce product understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22975–22985, 2026

    2026

  37. [37]

    Through the lens of contrast: Self-improving visual reasoning in vlms

    Zhiyu Pan, Yizheng Wu, Jiashen Hua, Junyi Feng, Shaotian Y an, Bing Deng, Zhiguo Cao, and Jieping Y e. Through the lens of contrast: Self-improving visual reasoning in vlms. arXiv preprint arXiv:2603.02556, 2026

    arXiv 2026

  38. [38]

    Qwen3.5: Towards native multimodal agents.https://qwen.ai/blog?id=qwen3.5, February 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents.https://qwen.ai/blog?id=qwen3.5, February 2026. Accessed: 2026-05-30

    2026

  39. [39]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine...

    2021

  40. [40]

    Bytedance Seed. Seed2. 0 model card: Towards intelligence frontier for real-world complexity. arXiv preprint arXiv:2607.00248, 2026

    Pith/arXiv arXiv 2026

  41. [41]

    Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Y e Xia, Basil Mustafa, Olivier J

    Michael Tschannen, Alexey A. Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Y e Xia, Basil Mustafa, Olivier J. Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense featu...

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.14786 2025

  42. [42]

    Representation learning with contrastive predictive coding

    Aäron van den Oord, Y azhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1807.03748, 2018. URL http://arxiv.org/abs/1807.03748

    Pith/arXiv arXiv 2018

  43. [43]

    Text embeddings by weakly-supervised contrastive pre-training

    Liang Wang, Nan Y ang, Xiaolong Huang, Binxing Jiao, Linjun Y ang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. CoRR, abs/2212.03533, 2022. doi: 10.48550/ARXIV.2212.03533. URL https://doi.org/10.48550/arXiv.2212.03533. 21

    Pith/arXiv arXiv doi:10.48550/arxiv.2212.03533 2022

  44. [44]

    Explore more, learn better: Parallel mllm embeddings under mutual information minimization

    Zhicheng Wang, Chen Ju, Xu Chen, Shuai Xiao, Jinsong Lan, Xiaoyong Zhu, Ying Chen, and Zhiguo Cao. Explore more, learn better: Parallel mllm embeddings under mutual information minimization. arXiv preprint arXiv:2511.01588, 2025

    arXiv 2025

  45. [45]

    Junxian Wu, Chenghan Fu, Zhanheng Nie, Daoze Zhang, Bowen Wan, Wanxian Guan, Chuan Yu, Jian Xu, and Bo Zheng. Moon3. 0: Reasoning-aware multimodal representation learning for e-commerce product understanding. arXiv preprint arXiv:2604.00513, 2026

    arXiv 2026

  46. [46]

    Deepseek-v4: Towards highly efficient million-token context intelligence

    Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. Deepseek-v4: Towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348, 2026

    arXiv 2026

  47. [47]

    Llava-cot: Let vision language models reason step-by-step

    Guowei Xu, Peng Jin, Ziang Wu, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2087–2098, 2025

    2087

  48. [48]

    R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization

    Yi Y ang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Y an Deng, Xingtao Y ang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2376–2385, 2025

    2025

  49. [49]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Y ang, Peng Qi, Saizheng Zhang, Y oshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2369–2380, 2018

    2018

  50. [50]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Y eung, Mojtaba Seyedhosseini, and Y onghui Wu. Coca: Contrastive captioners are image-text foundation models. Trans. Mach. Learn. Res. , 2022, 2022. URL https://openreview.net/forum?id= Ee277P3AYC

    2022

  51. [51]

    Visrag: Vision-based retrieval-augmented generation on multi-modality documents

    Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Y an, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, and Maosong Sun. Visrag: Vision-based retrieval-augmented generation on multi-modality documents. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 . OpenReview.net, 2025. URL https:...

    2025

  52. [52]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023 , pages 11941–11952. IEEE, 2023. doi: 10.1109/ICCV51070.2023.01100. URL https://doi.org/10.1109/ICCV51070.2023.01100

    arXiv 2023

  53. [53]

    Gme: Improving universal multimodal retrieval by multimodal llms, 2024

    Xin Zhang, Y anzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. Gme: Improving universal multimodal retrieval by multimodal llms, 2024. URL http://arxiv.org/abs/2412. 16855

    2024

  54. [54]

    Multimodal chain-of-thought reasoning in language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023

    Pith/arXiv arXiv 2023

  55. [55]

    Query Emb Decode

    Junjie Zhou, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, Defu Lian, and Y ongping Xiong. Megapairs: Massive data synthesis for universal multimodal retrieval. arXiv preprint arXiv:2412.14475, 2024. 22 Appendix A Detailed Formulation of Stage 2-A This appendix provides the implementation-level formulation of the evidence assignme...

    Pith/arXiv arXiv 2024