{"id":"23322980-72b2-4039-bf84-18be82fc2dd3","arxiv_id":"2606.09331","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Separately trained modality specialists are merged by task vectors, and a projector-only repair followed by balanced rehearsal yields 74.96 on MMEB and 55.61 on MAEB.","lead":"Conan-embedding-v3 builds one embedding model for text, image, video, document, and audio by training each modality separately, merging the models in weight space, and then repairing the audio connector, which they call Projector Drift. The recipe is relevant because it is a modular route to omni-modal retrieval and names a concrete failure mode for model merging.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Visual gains attributed to fusion come from balanced rehearsal; Table 3 shows direct fusion drops Image/Video/VisDoc to 68.5/56.7/68.9, and rehearsal supplies the +8–10 point recovery.","rationale":"The reader's weakest_assumption—that balanced rehearsal, not fusion, drives the reported visual performance—is directly supported by Table 3. Direct fusion alone is well below the specialist upper bounds and even below several baselines; all visual recovery occurs during rehearsal. The paper's assertion that rehearsal is 'lightweight' and 'gentle' is questionable given the magnitude of the improvements (up to +10.1 on VisDoc). This concern is load-bearing because it undercuts the central claim that task-vector fusion 'composes' visual, video, and document capabilities. The audio-side story (Projector Drift) is well-supported separately, and the final model is competitive, so a reject is not warranted. However, the current wording overstates what fusion alone achieves; the paper should either temper its claims or provide evidence that the rehearsal stage is modest. The proposed no-fusion ablation would settle this cleanly. I agree with the reader's verdict and recommend keeping CONDITIONAL, as the concern is addressable with a single experiment.","tokens_in":14517,"tokens_out":5313,"duration_ms":60416,"concrete_test":"Run the full pipeline with Stage 2 (task-vector fusion) disabled for the visual specialists: start from the base model θ0, copy the trained audio modules (as in the current pipeline), then apply the identical Projector Recovery and Balanced Rehearsal stages (same data mix, learning rate 1e-5, 2000 steps). If Image/Video/VisDoc scores land within ~1 point of 77.2/65.1/79.0, then the visual task-vector fusion contributes nothing to the reported visual performance; the gains are entirely due to the balanced rehearsal fine-tuning. If the no-fusion baseline falls short, the fusion provides a necessary initialization, but the paper must still demonstrate that the benefit is compositional rather than a warm-start effect.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that task-vector fusion 'composes visual, video, and document retrieval capabilities' (§1, abstract). But its own ablation (Table 3) contradicts this. Direct Fusion scores are 68.5/56.7/68.9 on Image/Video/VisDoc, far below the specialists' upper bounds (80.3/67.2/82.0). Projector-Only Recovery leaves these visual scores unchanged (68.5/56.7/68.9); Balanced Rehearsal then jumps them to 77.2/65.1/79.0—a gain of 8.7/8.4/10.1 points. The paper describes rehearsal as a 'gentle global coordination' of already-learned representations (hypothesis, §4.3.2), but a 10-point VisDoc improvement is not gentle; it is the main source of the final visual performance. The Projector Drift narrative (§3.4) explains the audio regression and its repair, but it does not explain why fusion degrades visual scores by ~12 points relative to specialists. Thus the claim that fusion composes visual capabilities is unsupported by the paper's own data: the composition appears to happen in the rehearsal stage, which is a standard multi-modal fine-tuning step. If so, the novel 'Decoupled Specialist Fusion' contribution reduces to a warm-start initialization, and the final MMEB score is achieved by ordinary fine-tuning, not by parameter-space fusion. This is the load-bearing weakness of the paper's central narrative.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Conan-embedding-v3, an omni-modal embedding model built on Qwen3-VL-8B. Its pipeline has three stages: (1) train image, video, visual-document, and audio specialists independently from a common initialization; (2) fuse their shared-backbone updates via task-vector arithmetic while copying audio-only modules; and (3) repair the resulting audio-pathway mismatch through projector-only fine-tuning followed by balanced multi-modal rehearsal. The paper identifies Projector Drift as the cause of audio degradation after fusion and claims that Decoupled Specialist Fusion composes visual, video, and document capabilities. Final reported scores are 74.96 on MMEB and 55.61 on MAEB.","tokens_in":14953,"tokens_out":5711,"duration_ms":61045,"significance":"If fully supported, the decouple-fuse-recover recipe would be a practical alternative to joint multi-task training for omni-modal retrieval: it avoids a single mixed-modality optimizer, is parallelizable, and isolates modality-specific failures. The paper also contributes a concrete and plausible failure mechanism, Projector Drift, with diagnostic evidence in Figures 1, 4, and 5, and a simple repair. Strengths include the internally consistent MMEB weighted average, the use of held-out validation to set fusion coefficients, and the breadth of ablations over merging methods and recovery strategies. However, the ablation data in Tables 3 and 4 directly undercut the central claim that fusion composes visual capabilities: direct fusion is far below the pre-fusion specialists, and most of the visual performance appears only after balanced rehearsal, which is an ordinary fine-tuning stage. This weakens the paper's main narrative and needs to be addressed before publication.","major_comments":[{"comment":"The central claim that task-vector fusion 'composes visual, video, and document retrieval capabilities' is contradicted by the reported ablation. Direct Fusion scores 68.5/56.7/68.9 on Image/Video/VisDoc versus the pre-fusion specialists' 80.3/67.2/82.0, a drop of roughly 11-13 points. Projector-Only Recovery leaves the visual scores unchanged, and Balanced Rehearsal produces the +8.7/+8.4/+10.1 recovery to 77.2/65.1/79.0. Thus the data indicate that the final visual performance is supplied by the rehearsal stage, a standard multi-modal fine-tuning step, not by parameter-space fusion. The authors should either provide a controlled isolation of fusion versus rehearsal (e.g., applying the same rehearsal budget to the base model or to the pre-fusion specialists) or revise the contribution claim to describe fusion as a warm-start initialization and rehearsal as the main compositional stage.","section":"Table 3, §4.3.1"},{"comment":"The 'Joint Training (I+V+D+A)' baseline is described in the main text as 'to be filled after full evaluation,' yet Table 3 and Table 5 report numerical scores for exactly this row and use them to conclude that PCGrad 'performs slightly better than naive joint multi-task training' and that decoupled fusion outperforms joint training. This is an internal inconsistency that affects the validity of the comparison against the joint/PCGrad baselines. Please clarify whether this baseline was actually run, report its training setup, or remove the row and all conclusions based on it.","section":"§4.3.1, Table 3; Appendix A, Table 5"},{"comment":"The interpretation of Balanced Rehearsal as 'a gentle global coordination of already learned representations' is not supported by the magnitude of the observed changes. A 2000-step LoRA run at learning rate 1e-5 moves Image/Video/VisDoc by +8.7/+8.4/+10.1 points, which is the main capability-acquisition step in the pipeline rather than a gentle coordination. This hypothesis should be tested by varying the rehearsal budget and learning rate, or the description should be changed to acknowledge that balanced rehearsal is the stage that produces most of the final visual performance.","section":"§4.3.2, Table 4"}],"minor_comments":[{"comment":"The notation θPA = Train(θPA, D_A) uses the same symbol for initial and final projector parameters. Specify that the initial projector comes from the direct-fusion checkpoint and only the projector parameters are updated.","section":"Eq. (9)"},{"comment":"The semicircle visualization and the listed cosine values are hard to interpret without a definition of the reference direction and the angle being plotted. Add a concise explanation.","section":"Figure 1"},{"comment":"The caption says the I+V+D+A row is 'to be filled after full evaluation,' but the row is populated. This is confusing and should be corrected with the actual training details.","section":"Table 3 caption"},{"comment":"The PCGrad baseline is 'evaluated at checkpoint-700'; please define the checkpoint criterion and report the data mixture, number of steps, and learning rate used for this baseline.","section":"Appendix A"},{"comment":"The paper says 'one backbone' while the audio pathway uses an external encoder and projector; consider 'one dense backbone plus external encoders' to avoid overstating architectural unity.","section":"Abstract/§3.1"},{"comment":"The reference list contains a spacing typo ('LLaV A-Hound') and several 2026-dated preprints; verify the final citation details before resubmission.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The MMEB weighted average is internally consistent and the Projector Drift story is well supported by the audio-side numbers. The main problem is the attribution of visual gains: Tables 3 and 4 show that direct fusion degrades visual performance and balanced rehearsal supplies nearly all the final visual scores, which undercuts the abstract's core claim. This is fixable with a re-analysis or a substantial reframing, but it is not merely a presentation issue. The unresolved 'to be filled' baseline row in Tables 3 and 5 is also a data-integrity concern that should be corrected before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Projector Drift is the real contribution here. The recovery recipe is clean: freeze the fused backbone, tune only the 19M-parameter audio projector, then run a short multi-modal rehearsal. That restores audio MAEB from 32.68 to 55.82 in the recovery-only ablation and 55.61 in the full model, while keeping MMEB at 74.96. The diagnostic figures—task-vector geometry, t-SNE neighborhoods—support the mechanism. That part deserves credit.\n\nWhat does not hold up is the abstract's claim that fusion composes visual, video, and document retrieval capabilities. Table 3 shows Direct Fusion with no recovery at 68.5/56.7/68.9 on Image/Video/VisDoc, well below the specialists (80.3/67.2/82.0). Projector-only recovery leaves those scores unchanged. Balanced rehearsal moves them to 77.2/65.1/79.0—a gain of 8.7/8.4/10.1 points. That is a standard LoRA fine-tuning stage over all modalities, and a 10-point VisDoc jump is not a gentle coordination. The paper's own data says the visual composition happens in rehearsal, not in parameter-space fusion. The fusion appears to be a warm start, and the narrative should be reframed.\n\nThe Projector Drift story is about audio, and it is solid. The paper isolates the mechanism well: task-vector fusion shifts the backbone, the projector stays calibrated to the specialist backbone, and the ablation comparing projector-only tuning to backbone LoRA is informative. That is a useful caution for anyone merging multimodal specialists.\n\nMinor issues: no code, data, or error bars; the Table 3 caption says the I+V+D+A joint baseline is 'to be filled' while the row already has values; fusion coefficients are tuned on held-out diagnostics, leaving a few free knobs.\n\nOverall, this is a solid engineering paper with an overstated central narrative. It deserves peer review because the Projector Drift identification and recovery recipe are real, reproducible contributions. The authors should be pushed to correct the fusion-versus-rehearsal attribution and release artifacts.","headline":"Projector Drift is a real, useful finding, but the paper's own ablation says balanced rehearsal—not fusion—composes the visual capabilities.","tokens_in":15382,"tokens_out":3657,"would_cite":true,"duration_ms":36716,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"When task-vector fusion shifts the backbone, a grafted audio projector goes out of calibration; freezing the backbone and fine-tuning only the projector restores the lost audio retrieval.","keywords":["omni-modal retrieval","embedding model","task-vector fusion","projector drift","audio retrieval","model merging","multi-task interference","decouple-fuse-recover"],"falsifier":"Run the pipeline without the decoupled specialists and task-vector fusion—just copy the audio encoder and projector onto the base visual-language model and run Projector Recovery plus Balanced Rehearsal with the same data and steps. If the final MMEB and MAEB scores match the paper's reported 74.96 and 55.61, then specialisation and fusion contribute nothing beyond the recovery and rehearsal stages, falsifying the central claim about fusion composing capabilities.","tokens_in":14441,"feed_emoji":"🎧","tokens_out":7989,"duration_ms":82730,"temperature":0.7,"pith_summary":"This paper tries to establish that a single omni-modal embedding model can be assembled from independently trained modality specialists rather than one jointly optimized network. It trains separate image, video, document, and audio specialists, then fuses their shared-backbone updates with task vectors. The strong technical claim is that this fusion silently breaks the audio pathway, even though the audio encoder and projector are copied unchanged: the backbone the projector was trained against is no longer the backbone it faces after fusion, a mismatch the paper calls Projector Drift. The paper's fix is to freeze the fused backbone and fine-tune only the lightweight audio projector, then run a short balanced rehearsal over all modalities. This raises audio retrieval on the 30-task MAEB suite from 32.68 to 55.61 while keeping the visual MMEB score at 74.96.","feed_headline":"Fusing models broke audio; one projector retune restores it","feed_subtitle":"The fix preserves visual retrieval (74.96 MMEB) and revives audio (55.61 MAEB); rehearsal, not fusion, does the visual lifting.","key_machinery":"The load-bearing object is the audio projector, a two-layer MLP that maps 2048-dimensional audio features into the backbone's 4096-dimensional hidden space. Its work is to keep the audio encoder's output compatible with whatever backbone follows it. The paper's central claim is that a projector trained against backbone A is not automatically valid against backbone A + Σ α_m Δ_m; the resulting mismatch is Projector Drift. Decoupled Specialist Fusion is the composition mechanism—weighted task-vector addition over shared parameters with audio-only modules copied—and Projector Recovery is the repair mechanism: freezing the fused backbone and fine-tuning only the projector.","core_discovery":"The central phenomenon is Projector Drift. When a modality enters the model through an external encoder plus a learned projector—audio here—the projector is trained in tandem with a specialist backbone. Task-vector fusion replaces that backbone with a weighted average of all specialists' updates, so the copied projector now faces a representation space it was never calibrated to. The paper shows this is not a missing-module problem: audio retrieval drops from specialist-level performance (R@1 92% on diagnostic AudioCaps) to 56% after direct fusion, and recovers to 91% after Projector Recovery—freezing the backbone and full-parameter fine-tuning only the 19M-parameter projector. The paper pro","pith_inferences":["Editorial extension: The ablation tables suggest that the 'fusion composes visual capabilities' part of the story is not actually demonstrated—after Direct Fusion and Projector-Only Recovery the visual scores remain 68.5/56.7/68.9, and they jump only after Balanced Rehearsal. If that is right, the method's visual gain may be an ordinary fine-tuning effect, and the unique contribution reduces to th","Editorial extension: A testable follow-up is to graft a second projector-based modality, such as depth maps or point clouds, onto the same backbone. Projector Drift predicts the same collapse under direct fusion and the same recovery pattern, with the caveat that multiple projectors may interact in ways the paper does not explore.","Editorial extension: The paper reports that audio task vectors are nearly orthogonal to visual updates and have the largest global norm (about 42.3); a cheap pre-fusion predictor of drift severity could be the per-layer cosine similarity between the fused backbone and the audio-specialist backbone at the projector's output layers, which the paper reports dropping to 0.591 at the deepest layer it e"],"forward_implications":["If the recipe holds, an existing strong visual-language embedding model can gain a new modality by training one specialist and fusing it, rather than retraining the entire network from scratch.","Projector Drift is a general failure of parameter-space merging for any projector-based modality, since TIES and DARE merging also produced pre-recovery MAEB scores at or below 32.68.","Post-fusion recovery—not the fusion itself—is the stage that determines final audio quality; the paper's ablations show audio goes from 32.68 to 55.82 after projector-only tuning.","Balanced rehearsal, not task-vector fusion, is what lifts the visual scores from 68.5/56.7/68.9 to 77.2/65.1/79.0 on Image/Video/VisDoc in the paper's own ablation table."],"fun_headline_variants":["Projector drift: why fusing specialists broke audio retrieval","Fusion broke audio; a 19M-parameter projector retune restored it","Audio collapsed after fusion; projector-only retune recovered 91% R@1","Projector Drift exposed: a tiny retune restores audio after model fusion"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the balanced rehearsal stage is only gently coordinating skills the model already has—but the paper's own table shows visual scores moving from 68.5/56.7/68.9 to 77.2/65.1/79.0 only after that rehearsal, which suggests the rehearsal, not the fusion, is doing the visual work.","fun_headline_variants_meta":{"raw":{"variants":["Projector drift: why fusing specialists broke audio retrieval","Fusion broke audio; a 19M-parameter projector retune restored it","Audio collapsed after fusion; projector-only retune recovered 91% R@1","Projector Drift exposed: a tiny retune restores audio after model fusion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001134,"raw_usage":{"total_tokens":4569,"prompt_tokens":787,"completion_tokens":3782,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":3700}},"tokens_in":531,"tokens_out":3782,"duration_ms":27478,"temperature":1.0,"reasoning_tokens":3700,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T11:57:04.609645+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline without the decoupled specialists and task-vector fusion—just copy the audio encoder and projector onto the base visual-language model and run Projector Recovery plus Balanced Rehearsal with the same data and steps. If the final MMEB and MAEB scores match the paper's reported 74.96 and 55.61, then specialisation and fusion contribute nothing beyond the recovery and rehearsal stages, falsifying the central claim about fusion composing capabilities.","supporting_citations":[],"review_version":2}