Pith. sign in

REVIEW 3 major objections 6 minor 4 references

When task-vector fusion shifts the backbone, a grafted audio projector goes out of calibration; freezing the backbone and fine-tuning only the projector restores the lost audio retrieval.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 11:57 UTC pith:GXU3XXDE

load-bearing objection Projector Drift is a real, useful finding, but the paper's own ablation says balanced rehearsal—not fusion—composes the visual capabilities. the 3 major comments →

arxiv 2606.09331 v2 pith:GXU3XXDE submitted 2026-06-08 cs.MM cs.AIcs.LG

Conan-embedding-v3: Fusing Modality-Specific Models for Omni-Modal Embedding

Shiyu Li , Zhiyuan Hu , Yifan Wang , Peiming Li , Zheng Wei , Yang Tang This is my paper
classification cs.MM cs.AIcs.LG
keywords omni-modal retrievalembedding modeltask-vector fusionprojector driftaudio retrievalmodel mergingmulti-task interferencedecouple-fuse-recover
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that a single omni-modal embedding model can be assembled from independently trained modality specialists rather than one jointly optimized network. It trains separate image, video, document, and audio specialists, then fuses their shared-backbone updates with task vectors. The strong technical claim is that this fusion silently breaks the audio pathway, even though the audio encoder and projector are copied unchanged: the backbone the projector was trained against is no longer the backbone it faces after fusion, a mismatch the paper calls Projector Drift. The paper's fix is to freeze the fused backbone and fine-tune only the lightweight audio projector, then run a short balanced rehearsal over all modalities. This raises audio retrieval on the 30-task MAEB suite from 32.68 to 55.61 while keeping the visual MMEB score at 74.96.

Core claim

The central phenomenon is Projector Drift. When a modality enters the model through an external encoder plus a learned projector—audio here—the projector is trained in tandem with a specialist backbone. Task-vector fusion replaces that backbone with a weighted average of all specialists' updates, so the copied projector now faces a representation space it was never calibrated to. The paper shows this is not a missing-module problem: audio retrieval drops from specialist-level performance (R@1 92% on diagnostic AudioCaps) to 56% after direct fusion, and recovers to 91% after Projector Recovery—freezing the backbone and full-parameter fine-tuning only the 19M-parameter projector. The paper pro

What carries the argument

The load-bearing object is the audio projector, a two-layer MLP that maps 2048-dimensional audio features into the backbone's 4096-dimensional hidden space. Its work is to keep the audio encoder's output compatible with whatever backbone follows it. The paper's central claim is that a projector trained against backbone A is not automatically valid against backbone A + Σ α_m Δ_m; the resulting mismatch is Projector Drift. Decoupled Specialist Fusion is the composition mechanism—weighted task-vector addition over shared parameters with audio-only modules copied—and Projector Recovery is the repair mechanism: freezing the fused backbone and fine-tuning only the projector.

Load-bearing premise

The load-bearing premise is that the balanced rehearsal stage is only gently coordinating skills the model already has—but the paper's own table shows visual scores moving from 68.5/56.7/68.9 to 77.2/65.1/79.0 only after that rehearsal, which suggests the rehearsal, not the fusion, is doing the visual work.

What would settle it

Run the pipeline without the decoupled specialists and task-vector fusion—just copy the audio encoder and projector onto the base visual-language model and run Projector Recovery plus Balanced Rehearsal with the same data and steps. If the final MMEB and MAEB scores match the paper's reported 74.96 and 55.61, then specialisation and fusion contribute nothing beyond the recovery and rehearsal stages, falsifying the central claim about fusion composing capabilities.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the recipe holds, an existing strong visual-language embedding model can gain a new modality by training one specialist and fusing it, rather than retraining the entire network from scratch.
  • Projector Drift is a general failure of parameter-space merging for any projector-based modality, since TIES and DARE merging also produced pre-recovery MAEB scores at or below 32.68.
  • Post-fusion recovery—not the fusion itself—is the stage that determines final audio quality; the paper's ablations show audio goes from 32.68 to 55.82 after projector-only tuning.
  • Balanced rehearsal, not task-vector fusion, is what lifts the visual scores from 68.5/56.7/68.9 to 77.2/65.1/79.0 on Image/Video/VisDoc in the paper's own ablation table.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: The ablation tables suggest that the 'fusion composes visual capabilities' part of the story is not actually demonstrated—after Direct Fusion and Projector-Only Recovery the visual scores remain 68.5/56.7/68.9, and they jump only after Balanced Rehearsal. If that is right, the method's visual gain may be an ordinary fine-tuning effect, and the unique contribution reduces to th
  • Editorial extension: A testable follow-up is to graft a second projector-based modality, such as depth maps or point clouds, onto the same backbone. Projector Drift predicts the same collapse under direct fusion and the same recovery pattern, with the caveat that multiple projectors may interact in ways the paper does not explore.
  • Editorial extension: The paper reports that audio task vectors are nearly orthogonal to visual updates and have the largest global norm (about 42.3); a cheap pre-fusion predictor of drift severity could be the per-layer cosine similarity between the fused backbone and the audio-specialist backbone at the projector's output layers, which the paper reports dropping to 0.591 at the deepest layer it e

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Conan-embedding-v3, an omni-modal embedding model built on Qwen3-VL-8B. Its pipeline has three stages: (1) train image, video, visual-document, and audio specialists independently from a common initialization; (2) fuse their shared-backbone updates via task-vector arithmetic while copying audio-only modules; and (3) repair the resulting audio-pathway mismatch through projector-only fine-tuning followed by balanced multi-modal rehearsal. The paper identifies Projector Drift as the cause of audio degradation after fusion and claims that Decoupled Specialist Fusion composes visual, video, and document capabilities. Final reported scores are 74.96 on MMEB and 55.61 on MAEB.

Significance. If fully supported, the decouple-fuse-recover recipe would be a practical alternative to joint multi-task training for omni-modal retrieval: it avoids a single mixed-modality optimizer, is parallelizable, and isolates modality-specific failures. The paper also contributes a concrete and plausible failure mechanism, Projector Drift, with diagnostic evidence in Figures 1, 4, and 5, and a simple repair. Strengths include the internally consistent MMEB weighted average, the use of held-out validation to set fusion coefficients, and the breadth of ablations over merging methods and recovery strategies. However, the ablation data in Tables 3 and 4 directly undercut the central claim that fusion composes visual capabilities: direct fusion is far below the pre-fusion specialists, and most of the visual performance appears only after balanced rehearsal, which is an ordinary fine-tuning stage. This weakens the paper's main narrative and needs to be addressed before publication.

major comments (3)
  1. [Table 3, §4.3.1] The central claim that task-vector fusion 'composes visual, video, and document retrieval capabilities' is contradicted by the reported ablation. Direct Fusion scores 68.5/56.7/68.9 on Image/Video/VisDoc versus the pre-fusion specialists' 80.3/67.2/82.0, a drop of roughly 11-13 points. Projector-Only Recovery leaves the visual scores unchanged, and Balanced Rehearsal produces the +8.7/+8.4/+10.1 recovery to 77.2/65.1/79.0. Thus the data indicate that the final visual performance is supplied by the rehearsal stage, a standard multi-modal fine-tuning step, not by parameter-space fusion. The authors should either provide a controlled isolation of fusion versus rehearsal (e.g., applying the same rehearsal budget to the base model or to the pre-fusion specialists) or revise the contribution claim to describe fusion as a warm-start initialization and rehearsal as the main compositional stage.
  2. [§4.3.1, Table 3; Appendix A, Table 5] The 'Joint Training (I+V+D+A)' baseline is described in the main text as 'to be filled after full evaluation,' yet Table 3 and Table 5 report numerical scores for exactly this row and use them to conclude that PCGrad 'performs slightly better than naive joint multi-task training' and that decoupled fusion outperforms joint training. This is an internal inconsistency that affects the validity of the comparison against the joint/PCGrad baselines. Please clarify whether this baseline was actually run, report its training setup, or remove the row and all conclusions based on it.
  3. [§4.3.2, Table 4] The interpretation of Balanced Rehearsal as 'a gentle global coordination of already learned representations' is not supported by the magnitude of the observed changes. A 2000-step LoRA run at learning rate 1e-5 moves Image/Video/VisDoc by +8.7/+8.4/+10.1 points, which is the main capability-acquisition step in the pipeline rather than a gentle coordination. This hypothesis should be tested by varying the rehearsal budget and learning rate, or the description should be changed to acknowledge that balanced rehearsal is the stage that produces most of the final visual performance.
minor comments (6)
  1. [Eq. (9)] The notation θPA = Train(θPA, D_A) uses the same symbol for initial and final projector parameters. Specify that the initial projector comes from the direct-fusion checkpoint and only the projector parameters are updated.
  2. [Figure 1] The semicircle visualization and the listed cosine values are hard to interpret without a definition of the reference direction and the angle being plotted. Add a concise explanation.
  3. [Table 3 caption] The caption says the I+V+D+A row is 'to be filled after full evaluation,' but the row is populated. This is confusing and should be corrected with the actual training details.
  4. [Appendix A] The PCGrad baseline is 'evaluated at checkpoint-700'; please define the checkpoint criterion and report the data mixture, number of steps, and learning rate used for this baseline.
  5. [Abstract/§3.1] The paper says 'one backbone' while the audio pathway uses an external encoder and projector; consider 'one dense backbone plus external encoders' to avoid overstating architectural unity.
  6. [References] The reference list contains a spacing typo ('LLaV A-Hound') and several 2026-dated preprints; verify the final citation details before resubmission.

Circularity Check

0 steps flagged

No construction-level circularity; the main risk is an attribution gap, not circular derivation.

full rationale

The paper's chain is empirical rather than equationally circular. Equations (1)-(9) define an InfoNCE loss, task vectors, fusion, and projector-only tuning; no reported number is the identity of a fitted parameter renamed as a prediction. The fusion coefficients are selected on held-out diagnostic tasks stated to be disjoint from the MMEB/MAEB evaluation splits, and final scores are measured on separate benchmarks. 'Projector Drift' is an empirical label for the observed audio regression after fusion, not a result derived from its own definition. Citations to Qwen3-VL-Embedding and Qwen3-Omni are external base-model/data-recipe references, not load-bearing self-citations. The Limitations section even concedes that recovery is 'a repair rather than a complete solution' and that 'the recovery configuration is selected through a directed empirical search,' which is consistent with a tuned but non-circular pipeline. The real concern is an internal-validity/credit-assignment problem rather than circularity: Table 3 shows Direct Fusion drops Image/Video/VisDoc to 68.5/56.7/68.9, Projector-Only Recovery leaves those scores unchanged, and Balanced Rehearsal supplies the +8-10 point recovery to 77.2/65.1/79.0. Thus the abstract's claim that 'fusion composes visual, video, and document retrieval capabilities' is not supported by the paper's own ablation, because the final visual performance is achieved by the balanced rehearsal fine-tuning stage, not by parameter-space fusion alone. That is a substantiveness/attribution weakness, but it does not make any equation or prediction equivalent to its input by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claim rests on standard contrastive retrieval, task-vector merging from prior work, and the paper-specific assumption that a projector-only repair closes the interface gap after fusion. Two free coefficient choices (fusion weights, rehearsal budget) and the projector architecture are chosen by hand, and the training data includes unspecified in-house synthetic pairs. No new physical or architectural entities are introduced; 'Projector Drift' is a label for an observed interface mismatch.

free parameters (3)
  • Fusion coefficients α_visual and α_audio = αI = αV = αD = 0.3, αA = 0.5
    Selected by grid sweep on held-out validation tasks disjoint from MMEB/MAEB; the main-model numbers depend on this choice (Table 7, §3.3).
  • Rehearsal step budget and learning rate = 2000 steps, lr 1e-5
    Chosen empirically as a 'conservative update budget'; no sensitivity analysis is provided beyond the recovery-strategy ablation in Table 4.
  • Audio projector architecture = two-layer MLP, 19M parameters, 2048→4096
    Hand-selected component; because the projector is the only trainable part during Projector Recovery, its capacity constrains the repair and thus the reported audio scores.
axioms (5)
  • standard math InfoNCE contrastive loss with temperature τ=0.02 (Eq. 1) is a valid retrieval objective
    Used for all specialist training and rehearsal; standard contrastive retrieval assumption.
  • domain assumption Task-vector arithmetic θ0 + Σ αm Δm produces a meaningful merged backbone (Eq. 4)
    Borrowed from model-merging literature; the paper tests it empirically but provides no formal guarantee, and Table 3 shows fusion alone underperforms the specialists.
  • domain assumption LoRA adapters can be merged into dense checkpoints without meaningful loss before task-vector subtraction
    Used throughout Stage 1/2; no analysis of merge-induced approximation error is given.
  • domain assumption MMEB and MAEB evaluation splits are disjoint from all training/validation data
    Stated in §4.1 but not auditable without released data; the unspecified in-house synthetic data is the main unverifiable part.
  • ad hoc to paper Projector Drift is repairable by fine-tuning only the projector on the frozen fused backbone (Eq. 9)
    This is the paper's central repair hypothesis; supported only by the specific ablations reported, with no general argument about when projector-only recovery succeeds.

pith-pipeline@v1.3.0-alltime-deepseek · 14196 in / 14254 out tokens · 145864 ms · 2026-08-02T11:57:04.609645+00:00 · methodology

0 comments
read the original abstract

Omni-modal retrieval promises a single embedding space for text, image, video, document, and audio inputs, but building such a unified retriever is difficult since these modalities differ in data distribution, architecture, and optimization dynamics. In this work, we present Conan-embedding-v3, a decouple--fuse--recover framework for omni-modal retrieval. Conan-embedding-v3 first trains modality specialists independently and fuses their task vectors into a single dense backbone, a strategy we call Decoupled Specialist Fusion. We show that this fusion composes visual, video, and document retrieval capabilities, but also exposes a failure mode for projector-based modalities: when audio is attached through an external encoder and projector, fusing the backbone leaves the projector calibrated to the audio-specialist backbone, causing a large audio retrieval regression despite copying all audio-specific modules unchanged. We call this failure Projector Drift. To repair it, Conan-embedding-v3 applies Projector Recovery (i.e., full-parameter fine-tuning of the projector while keeping the backbone frozen) followed by balanced multi-modal rehearsal. The resulting model supports these retrieval pathways in one backbone, achieving 74.9 scores on MMEB while obtaining 55.61 on the 30-task MAEB audio suite.

Figures

Figures reproduced from arXiv: 2606.09331 by Peiming Li, Shiyu Li, Yang Tang, Yifan Wang, Zheng Wei, Zhiyuan Hu.

Figure 1
Figure 1. Figure 1: Projector Drift across Transformer depth. The semicircles show audio-token directions at layers 0, 17, and 35, measured relative to the audio specialist (blue). Direct fusion (red) drifts away in deeper layers, while Projector Recovery (green) stays closer to the specialist and restores AudioCaps performance. space for text, image, video, visual-document, and audio retrieval (Zhan et al., 2024; Lin et al.,… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of Conan-embedding-v3. Stage 1 trains modality specialists from a shared initialization θ0 using modality-specific LoRA, where the audio specialist additionally grafts an audio encoder and projector. These specialists are then combined in Stage 2 by fusing their shared-backbone task vectors while copying audio modules directly, which induces Projector Drift. Finally, Stage 3 applies Projector Reco… view at source ↗
Figure 3
Figure 3. Figure 3: Ablation on fusion weights before recov￾ery. Increasing the audio task-vector coefficient αA improves audio capability but degrades visual retrieval performance, illustrating the inherent conflict. 4.4 Qualitative Experiments 4.4.1 Task-Vector Geometry To understand why direct fusion disproportionately affects the grafted audio pathway, we analyze the model-space task vectors of the specialists in Fig￾ure … view at source ↗
Figure 4
Figure 4. Figure 4: Geometry of the four modality task vectors. (a) Pairwise cosine similarity indicates audio is nearly orthogonal to visual updates (cos ≤ 0.001). (b) Audio has the largest global update norm (≈ 42.3). (c) Audio updates remain large across the backbone and peak in deeper layers. Specialist (R@1 = 92%) Direct merge (R@1 = 56%) Recovered (R@1 = 91%) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Output-space view of Projector Drift on diagnostic audio–text retrieval pairs. The audio specialist (left) aligns paired embeddings (R@1 = 92%). Direct fusion (middle) separates the audio and text manifolds (R@1 = 56%). Projector Recovery (right) reduces this separation and restores retrieval performance (R@1 = 91%). sual updates provide little direct directional support for audio during fusion. Second, Fi… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

4 extracted references · 3 linked inside Pith

  1. [2]

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Man- nat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra

    Colpali: Efficient document retrieval with vi- sion language models.Preprint, arXiv:2407.01449. Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Man- nat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. Imagebind: One embed- ding space to bind them all. InProceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition ...

    Pith/arXiv arXiv 2023

  2. [3]

    InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 9637– 9662, Bangkok, Thailand

    AnyGPT: Unified multimodal LLM with dis- crete sequence modeling. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 9637– 9662, Bangkok, Thailand. Association for Computa- tional Linguistics. Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyua...

    Pith/arXiv arXiv 2024

  3. [700]

    against our naive Joint Training, independent Decoupled Specialists (pre-fusion upper bounds), and our final merged Conan-embedding-v3. Comparison with Joint Training:PCGrad per- forms slightly better than naive joint multi-task training overall, demonstrating that projecting con- flicting gradients mitigates severe cross-modal in- terference to some exte...

    2023

  4. [2024]

    Qwen2-audio technical report.Preprint, arXiv:2407.10759. Xuanming Cui, Jianpeng Cheng, Hong you Chen, Satya Narayan Shukla, Abhijeet Awasthi, Xi- chen Pan, Chaitanya Ahuja, Shlok Kumar Mishra, Yonghuan Yang, Jun Xiao, Qi Guo, Ser-Nam Lim, Aashu Singh, and Xiangjun Fan. 2026. Think then embed: Generative context improves multimodal em- bedding.Preprint, ar...

    Pith/arXiv arXiv 2026