{"id":"dfd578f9-3cfa-483f-b213-65271a7612bb","arxiv_id":"2607.18666","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Trained connectors and audio-only gated adapters integrate audio into a frozen vision-language embedding space, preserving base outputs bit-exactly and yielding emergent audio-image retrieval.","lead":"This paper adds audio to a frozen text-image-video embedding model by training a small connector (and later gated adapters), so the original model's outputs stay bit-identical. It reports audio-text retrieval on AudioCaps and audio-to-image retrieval that emerges with no audio-image training pairs. Why read it: it shows how to extend an existing embedding index to a new modality without re-embedding stored vectors.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Emergent audio-image retrieval may be inflated by base-pretraining overlap with VGGSound; the paper only blacklists VGGSound from connector training, not from the frozen base's own text-image data.","rationale":"The reader's verdict is CONDITIONAL with moderate confidence, citing single-seed results and single-backbone generalization as concerns. I find a more specific and load-bearing concern: the emergent audio-image result may be confounded by the frozen base's own pretraining data. The reader's weakest_assumption focuses on whether the base's text-image alignment is semantically trustworthy for the caption register; my concern is narrower and sharper: the base may have been trained on VGGSound itself, so the text-image link is not merely 'trustworthy' but directly supervised on the evaluation set. This would inflate the headline 'emergent' number without any flaw in the connector training. The paper's own controls do not rule this out: VGGSound is blacklisted only from the audio-text corpus, and the text-image control rows in Table 9 (0.331/0.319) are identical across checkpoints by construction—they do not tell us whether that baseline itself is contaminated. The paper is otherwise strong: bitwise invariance is carefully verified, negative results are controlled, and audio-text retrieval is well-supported. Thus the overall verdict should remain CONDITIONAL, but the condition must include demonstrating that the base's text-image alignment on the evaluation benchmark is not a product of base-pretraining overlap. My agreement_with_reader is partial because our concerns overlap on the base's text-image geometry, but mine identifies a concrete contamination mechanism the reader did not articulate.","tokens_in":27097,"tokens_out":5909,"duration_ms":56658,"concrete_test":"Inspect the Qwen3-VL-Embedding-2B model card / training-data documentation for inclusion of VGGSound (or its video-text pairs). If it is included, re-run the emergent audio→image evaluation on an audio-visual benchmark released after the base's training cutoff (e.g., a 2026+ dataset or a fresh VGGSound split verified absent from the base's pretraining). If audio→image R@10 drops materially below 0.418, the emergent claim is inflated by base-pretraining overlap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central novelty is 'emergent' audio-image retrieval on VGGSound-696 with 'zero paired audio-visual training data.' That claim is load-bearing, and it has a soft spot: the paper's blacklist applies only to the connector/adapters training corpus (§6.1), not to the frozen base Qwen3-VL-Embedding-2B. The mechanism explicitly depends on the base's pre-existing text-image geometry (§7): 'captions of a barking dog live near images of barking dogs by the base’s own training.' If the base's own pretraining included VGGSound video frames and their text captions—a common ingredient in vision-language pretraining, and not disclosed here—then the text→image leg of the chain is directly supervised on the exact evaluation clips. The audio connector only learns audio→text; the text→image association for those clips would be memorized in the base, not emergent. The measured audio→image R@10 0.418 (29× chance) would then partly reflect base-pretraining leakage, not a clean zero-shot cross-modal transfer. This is distinct from the reader's concern about 'semantic trustworthiness': even a perfectly trustworthy base alignment can be contaminated on this specific benchmark. The paper provides no evidence about the base's pretraining data, and the 'zero paired audio-visual training data' phrasing, while technically true for the trained components, is potentially misleading for the full system. The bitwise-invariance and audio-text results stand independently, but the emergent cross-modal result—a headline contribution—needs a contamination check.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents two generations of an audio extension to a frozen vision-language embedding model. Generation 1 trains only a 16.4M-parameter perceiver-resampler connector between a frozen Qwen2.5-Omni audio tower and a byte-frozen Qwen3-VL-Embedding-2B base; generation 2 adds 44.2M modality-gated bottleneck adapters at every decoder layer, hard-gated so non-audio inputs are bitwise identical to the released base. Training uses a symmetric InfoNCE objective at all Matryoshka rungs with a full-corpus frozen-text negative bank, native chat-template targets, diagonal whitening, and an in-domain fine-tuning stage. The paper reports AudioCaps audio-to-text R@10 of 0.741/0.743, strictly zero-shot Clotho transfer, emergent audio-to-image retrieval on VGGSound-696 at R@10 0.418/0.392 (29× chance), three controlled negative results, and a training-protocol effect of +14.5 R@10 points. Weights, code, and evaluation harnesses are released.","tokens_in":27458,"tokens_out":11815,"duration_ms":115297,"significance":"If the results hold, this is a practically valuable recipe: it adds audio to a deployed frozen embedding stack at small training cost while preserving, by construction and by test, the base's text/image/video outputs bit-for-bit. The paper is unusually thorough: machine-checked invariance tests, a loss-floor audit, controlled A/B experiments, explicit single-seed noise caveats, and open weights/code are all real strengths. The central novelty, however, is the 'emergent' audio-to-image retrieval claim, and that claim currently lacks a critical control on the frozen base's own pretraining data. The audio-text retrieval results and the invariance guarantee stand independently; the emergence interpretation needs tightening before the paper's strongest conclusion is fully supported.","major_comments":[{"comment":"The headline 'emergent audio→image' result has a load-bearing gap: the VGGSound blacklist applies to the connector/adapter training corpus, not to the pretraining data of the frozen base. If Qwen3-VL-Embedding-2B saw VGGSound frames/captions, the text→image leg is directly supervised for the exact evaluation clips, and 'zero paired audio-visual training data' is true only for the trained components. Please add an audit/control (e.g., evaluate on clips/captions that cannot be in the base's pretraining, or compare with a base ablated at the text-image leg) or explicitly reframe the claim as compositional transfer through a pretrained text→image space, not zero-shot cross-modal emergence.","section":"§7, §6.1"},{"comment":"The stated mechanism — audio inherits text→image geometry via caption embeddings — is at odds with the reported numbers: audio→image R@10 is 0.418 for v0.2, while text↔image R@10 on the same VGGSound-696 protocol is 0.331/0.319. If audio embeddings are trained to sit near whitened text embeddings, the caption→image retrieval should upper-bound the audio→image result; the observed inversion needs a quantitative explanation (differences in text-query formatting, whitening statistics, or per-modality mean-centering between the text and audio query paths). Without this, the mechanism claim is not supported by the paper's own tables.","section":"§7"}],"minor_comments":[{"comment":"Typo: 'widening the connector eachreduceretrieval' should read 'each reduce retrieval'.","section":"Abstract"},{"comment":"The header 'Gen. 1 (v0.3) Gen. 2' is visually ambiguous; add column separators or vertical rules for clarity.","section":"Table 6 caption"},{"comment":"The binomial standard errors assume independent queries; VGGSound items are grouped by class and the caption pools overlap, so the effective noise may be larger than stated. This does not change the paper's qualitative conclusions but should be noted.","section":"§6.1, Uncertainty"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is transparent and reproducible, and the audio-text and invariance contributions are solid. The main issue is the scope of the emergence claim: the paper needs either a base-pretraining leakage control or a careful qualification. If the authors add that control and explain the audio→image vs text→image inversion, I would be happy to accept."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short take: the core engineering is real, and the paper is unusually honest about its own negative results. The load-bearing claim to scrutinize is 'emergent audio-image retrieval,' because the frozen base's pretraining data isn't disclosed and VGGSound is a common ingredient.\n\nWhat's new: the gated deep adapters that keep text/image/video outputs bit-identical after training is a clean mechanism, and the bitwise verification is the right kind of evidence. The controlled negative results (caption rewriting hurts, a better audio tower on CLAP-style benchmarks loses inside this splice, wider connectors overfit) are genuinely informative. The +14.5 point protocol effect is a useful reminder for anyone fine-tuning against a frozen decoder-LM embedding. Training cost is low, and the release is complete enough to reproduce.\n\nWhere it's soft: the audio-image retrieval on VGGSound-696. The paper blacklists VGGSound from the connector's training data, but the base model Qwen3-VL-Embedding-2B was trained on an undisclosed mixture—likely including VGGSound's video frames and captions, since it's a standard vision-language pretraining set. If that's the case, the chain 'audio→text→image' has a memorized text→image link for the exact evaluation clips, and calling the result 'emergent with zero paired audio-visual data' is misleading. The audio-text and bitwise-invariance results don't depend on this, so the paper's main practical contribution stands. But the cross-modal headline needs a leakage check—e.g., measuring how often the text-image link for these clips appears in the base's own retrieval, or evaluating on a held-out set of audio-visual pairs with no web presence.\n\nOther caveats you already know: single-seed numbers, one backbone, a few baselines single-sourced. They're appropriately flagged in the paper, so I'd call them minor.\n\nBottom line: this deserves a serious referee. The authors are doing careful, honest work, and the frozen-base extension recipe is publishable. I'd want the contamination question answered before the 'emergence' language is used, but I'd rather have this paper in the record than not. Yes, send it to review.","headline":"Core recipe is sound and honestly reported; the 'emergent' audio–image claim needs a VGGSound-contamination check before it's taken at face value.","tokens_in":28001,"tokens_out":2288,"would_cite":true,"duration_ms":23928,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a frozen vision-language embedding base can be extended to audio without touching a single base weight, using a small connector and modality-gated adapters, so audio joins text, image, and video in one space while base outp","keywords":["multimodal embedding space","audio-text retrieval","frozen backbone adaptation","emergent cross-modal retrieval","modality-gated adapters","contrastive learning","zero-shot transfer","retrieval invariance"],"falsifier":"Train the same connector and adapters against the same audio-caption data but attach them to a vision-language base whose text-image alignment has been deliberately weakened or scrambled, for example by randomizing or freezing the text projector on sound-event captions. If emergent audio-to-image retrieval stays well above chance, the 'emergent' claim is not carrying the result; if it collapses to chance while audio-to-text retrieval stays intact, the paper's mechanism is confirmed.","tokens_in":26964,"feed_emoji":"🔊","tokens_out":5035,"duration_ms":66129,"temperature":0.7,"pith_summary":"This paper tries to establish that a single embedding space for text, image, video, and audio can be built without retraining the underlying vision-language model. It claims that training only a small connector between a frozen audio encoder and a frozen vision-language embedding base, aligning audio to text alone, makes audio a first-class modality, and that audio-to-image retrieval emerges as a side effect because the base already binds text to images. A second generation shows where the remaining gap lives: inside the frozen decoder layers, so it adds per-layer adapters gated to fire only on audio inputs, keeping text, image, and video outputs bit-for-bit identical to the released base. A sympathetic reader should care because the recipe preserves already-deployed embeddings and benchmark results by construction, and it maps the design space with controlled negative results rather than only positive numbers.","feed_headline":"Frozen base gains audio without changing a single base weight","feed_subtitle":"A 16M-parameter connector adds audio; gated adapters lift retrieval while text, image, and video stay bit-identical.","key_machinery":"The load-bearing mechanism is the text pivot: audio is aligned only to text, and audio-to-image and audio-to-video retrieval inherit the base's existing text-image geometry. Two trained components carry this. Generation 1 uses a perceiver-style resampler connector (about 16.4M parameters), a small attention module that compresses audio frames into a fixed number of tokens injected at placeholder positions in the frozen decoder. Generation 2 keeps the connector and adds a bottleneck adapter to the residual stream of every decoder layer, hard-gated so the adapter runs only while audio is being encoded. The gate converts 'base outputs are preserved' from an empirical hope into a construction-ti","core_discovery":"The central claim is that a frozen vision-language embedding model can be extended with audio without updating any of its weights. The paper demonstrates this with two generations: a 16.4M-parameter connector trained contrastively against cached frozen-text embeddings already reaches strong audio-to-text retrieval and produces emergent audio-to-image retrieval at 29 times chance despite zero audio-visual training pairs; a second generation adds 44.2M parameters of per-layer adapters, gated so they execute only for audio, improving text-to-audio retrieval while text, image, and video outputs remain bit-for-bit identical to the released base. The paper locates the remaining bottleneck in the f","pith_inferences":["Editorial inference: the large protocol effect implies that published comparisons among LLM-backbone embedding models are only meaningful when the input template is matched; many reported gaps may be partly formatting artifacts.","Editorial inference: the negative results suggest scaling this recipe is not primarily a data problem; the next testable lever is matching the caption register of the target benchmark rather than making captions cleaner.","Editorial inference: the modality-gated adapter pattern could transfer to any future modality, such as depth, sensor streams, or new audio formats, with the same bitwise-preservation guarantee, and that extension is directly testable.","Editorial inference: the full-corpus negative bank ceiling near half a million pairs implies that the scaling path requires improved negative sampling, not just more data; soft-label and false-negative masking terms are a first step."],"forward_implications":["Text, image, and video embeddings produced by the base remain valid bit-for-bit, so an existing index never needs re-embedding when audio is added.","Audio-to-text retrieval reaches roughly 0.74 recall at rank 10 on a standard sound-caption benchmark with both towers frozen, and transfers zero-shot to a held-out caption benchmark.","Audio-to-image retrieval emerges at about 29 times chance with no audio-visual supervision, and improves in lockstep with audio-to-text quality.","The negative results redraw the design space: cleaner rewritten captions, a leaderboard-stronger audio tower, and a wider connector all reduce retrieval, while gated in-layer capacity helps.","Both generations train in hours on a single accelerator, making the recipe reproducible without a large compute budget."],"fun_headline_variants":["Add audio to frozen embedding base, zero weight updates","16M connector adds audio to frozen vision-language base","Audio joins frozen embedding; emergent image retrieval","Gated adapters add audio, keep base outputs bit-identical"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The base model's existing text-image alignment is a trustworthy bridge for the caption register used in training and evaluation, and the paper does not independently measure or control that geometry.","fun_headline_variants_meta":{"raw":{"variants":["Add audio to frozen embedding base, zero weight updates","16M connector adds audio to frozen vision-language base","Audio joins frozen embedding; emergent image retrieval","Gated adapters add audio, keep base outputs bit-identical"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000962,"raw_usage":{"total_tokens":3960,"prompt_tokens":795,"completion_tokens":3165,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":3113}},"tokens_in":539,"tokens_out":3165,"duration_ms":19955,"temperature":1.0,"reasoning_tokens":3113,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T14:41:09.984370+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same connector and adapters against the same audio-caption data but attach them to a vision-language base whose text-image alignment has been deliberately weakened or scrambled, for example by randomizing or freezing the text projector on sound-event captions. If emergent audio-to-image retrieval stays well above chance, the 'emergent' claim is not carrying the result; if it collapses to chance while audio-to-text retrieval stays intact, the paper's mechanism is confirmed.","supporting_citations":[],"review_version":1}