{"id":"deeb08ae-ff19-4e42-abe3-3231a7348350","arxiv_id":"2502.02885","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A video-text retrieval method that automatically generates and selects multi-perspective video captions with a prompt-optimized multimodal LLM and a mixture-of-experts module, reporting state-of-the-art performance on MSR-VTT, MSVD, and DiDeMo.","lead":"This paper presents a video-text retrieval method that automatically writes several descriptions of each video using a large AI model, then lets the model learn which descriptions best match text queries. It reports top results on three standard video search benchmarks without using extra training video-text data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The caption scoring proxy in Eq. (1) is never validated against retrieval accuracy, so the CSI 'self-improvement' may be optimizing the wrong objective.","rationale":"The reader identified the same load-bearing assumption: that the caption scoring function in Eq. (1) is a valid proxy for downstream retrieval accuracy. This is indeed the most load-bearing concern because the CSI module's entire 'self-improvement' mechanism relies on it. If the proxy is miscalibrated, the method's core novelty—automatic prompt optimization without human empiricism—collapses, even though the final empirical results might still be plausible through other means (e.g., the ECS module or the raw captioning quality of GPT-4o). The concern is concrete and testable without external dependencies: one only needs to generate captions with several prompts and compare Score_t against an actual R@1 measure. The paper does not provide such a validation, and the ablation study is insufficient because it only contrasts one optimized prompt against one initial prompt. I agree with the reader's verdict of CONDITIONAL: the paper should be accepted only after this proxy is validated or the CSI mechanism is re-derived. The contamination risk from GPT-4o is a related but secondary external-validity concern; if the proxy test passes, the method's internal logic would still need scrutiny against non-contaminated captioners. No change to the reader's verdict is needed because the reader already conditioned acceptance on exactly this kind of evidence.","tokens_in":13649,"tokens_out":10570,"duration_ms":99601,"concrete_test":"On a held-out split of MSR-VTT (e.g., validation), generate a diverse set of candidate prompts by running the Prompt Engineer with multiple random seeds or by perturbing the initial prompt. For each prompt, generate K=10 captions per video, compute Score_t via Eq. (1), and compute text-to-video R@1 using a fixed retrieval pipeline (e.g., CLIP4Clip with mean-pooled video frames plus caption features, without the learned ECS experts, to isolate caption quality). Compute Spearman's rank correlation coefficient between Score_t and R@1 across prompts. If the correlation is not significantly positive (p < 0.05), the proxy is not a valid indicator of retrieval performance and the CSI module's optimization target is unjustified. Ideally, repeat for the full ExCae pipeline on a few prompts to confirm the correlation holds after ECS training.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section III-A, ExCae's CSI module uses Score_t (Eq. 1) to guide prompt selection. The score combines semantic consistency between generated captions and the original text (via M3-embedding cosine similarity) with caption diversity. The authors state they 'hope' this proxy is positively correlated with retrieval performance, but the paper provides no empirical or theoretical evidence of such a correlation. The proxy lives in M3-embedding space, whereas retrieval is performed with CLIP-based encoders; there is no inherent reason that a higher score in one space implies better CLIP-space retrieval. Because the CSI module greedily keeps prompts only when Score_t improves, a miscalibrated proxy would make the claimed 'self-improvement' nothing more than hill-climbing on an unrelated objective. The ablation in Table IV shows that the CSI module improves R@1 by 4.5 points, but that only verifies that the final prompt beats the initial prompt—it does not show that Score_t is predictive across a range of prompts. Without a correlation analysis between Score_t and downstream R@1, the central claim that the method 'self-improves' and 'mitigates empiricism' is unsupported, and the reported gains could stem from the Prompt Engineer's arbitrary better rewrites or from overfitting to the proxy. This is the weakest link in the method's causal chain from prompt optimization to retrieval accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes ExCae, a caption auto-enhancement method for video-text retrieval. The Caption Self-Improvement (CSI) module iteratively refines a prompt for an mLLM Captioner using a Prompt Engineer, guided by a caption score that combines M3-embedding-based semantic consistency and diversity (Eq. (1)). The Expertized Caption Selection (ECS) module then fuses sampled video frames with K generated captions through a mixture-of-experts layer before the standard contrastive alignment. Experiments on MSR-VTT, MSVD, and DiDeMo report strong R@1 gains over ViT-based baselines, ablations of the two modules, and plug-in improvements for DRL, TS2-Net, and Clip4Clip. The code is publicly available.","tokens_in":13924,"tokens_out":8821,"duration_ms":83155,"significance":"If the reported results are reproducible, the paper makes a useful practical contribution: it improves video-text retrieval without collecting extra training data, offers a plug-in formulation, and releases code. The core idea of generating captions from the video side and selecting them per sample is clearly motivated and reasonably presented. I agree with the reader's assessment that the method is not circular: prompt selection uses training data and a heuristic score, not test labels. The main weaknesses are the unvalidated caption-scoring proxy that drives CSI, the absence of variance estimates, and overstatements in the benchmark comparisons; these are fixable with additional experiments and tighter claims.","major_comments":[{"comment":"The caption scoring function Score_t is the only feedback signal for the CSI prompt search, but the paper never shows that a higher Score_t corresponds to higher retrieval R@1. The text itself says \"we hope\" that the adjustment is positively correlated with VTR results. Since Score_t is computed in M3-embedding space while retrieval is performed in CLIP space, a correlation analysis is needed, for example by reporting Score_t and R@1 across CSI iterations or across a set of candidate prompts. Table IV only shows that the final learned prompt beats the initial prompt; it does not establish that hill-climbing on Score_t is optimizing the retrieval objective. Also, M3-embedding is a text-only model, so the score cannot detect whether a caption is faithful to the video content, only whether it resembles the original text.","section":"Section III-A, Eq. (1)"},{"comment":"All reported numbers are single-run point estimates with no error bars, standard deviations, or significance tests. This is especially problematic because several head-to-head comparisons are close: on MSVD text-to-video with ViT-B/16, ExCae (51.3) is below Cap4Video++ (51.8), and on DiDeMo several ExCae entries are within a few points of existing methods. The claim of state-of-the-art performance cannot be evaluated from unreplicated point estimates; the authors should provide variance over multiple seeds or at least state that the reported runs are representative.","section":"Tables I and II"},{"comment":"The abstract and Section IV-C claim state-of-the-art results on all three benchmarks, but Table II shows that on DiDeMo the best ExCae text-to-video R@1 (62.0 with ViT-G/14) is substantially below the pre-trained foundation models UMT-L (70.4) and VAST (72.0). The paper's claim is valid only within the \"ViT-based\" category, so the benchmark claim in the abstract should be revised or explicitly scoped to that category.","section":"Table II (DiDeMo)"},{"comment":"The CSI prompt-selection protocol is underspecified. The Prompt Engineer is given \"the corresponding video-text pairs (x,t) involved in this iteration,\" but the paper does not state how many pairs are used, how they are sampled, or whether they come strictly from the training split. Because the selected prompt affects all downstream retrieval scores, the authors must document the data split and the stopping or iteration budget (e.g., the claim of convergence in 400 iterations) so that test-set leakage can be ruled out.","section":"Algorithm 1 and Section IV-B"}],"minor_comments":[{"comment":"The abstract reports 68.5% on MSR-VTT and 68.1% on MSVD, which appear to be the means of the text-to-video and video-to-text R@1 entries in the tables rather than a single reported direction; the abstract should state exactly which metric is being reported.","section":"Abstract vs. Tables I and II"},{"comment":"The statement that existing methods \"could hardly transcend 50%\" without extra data is contradicted by Cap4Video++, which reports 52.3 R@1 on MSR-VTT with no extra data in Table I.","section":"Section IV-C"},{"comment":"The claimed improvements of \"7% on MSVD\" and \"13% on DiDeMo\" are actually percentage-point differences and are not computed correctly: against the best ViT-based method in Table II, the differences are 7.4 points and 9.5 points, respectively.","section":"Section IV-C"},{"comment":"The row labels and checkmark columns of Table IV are ambiguous: several rows have no checkmark yet still report numbers, and it is not clear which checkmark indicates the CSI module versus the ECS module for each row; the table should be reformatted.","section":"Table IV"},{"comment":"The implementation details should state the exact GPT-4o version, sampling temperature, and number of stochastic runs, because caption generation from a closed LLM is non-deterministic and affects the reproducibility of all reported numbers.","section":"Section IV-B"},{"comment":"The method name \"TSR-Net\" in the plug-in experiment appears to be a typo for \"TS2-Net\"; please correct it.","section":"Section IV-D"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the strongest fixable weakness is the unvalidated caption-scoring proxy in Eq. (1), which is the engine of the claimed self-improvement; the authors should add a direct correlation study between Score_t and R@1. The absence of error bars and the overstatement on DiDeMo also need attention. The reliance on a closed commercial model (GPT-4o) is a reproducibility risk, but it is not by itself disqualifying if the prompt, version, and decoding parameters are fully documented. I do not see a circularity problem in the prompt selection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ExCae is a genuinely new combination for video-text retrieval: it auto-tunes the caption-generation prompt via an LLM-based prompt engineer and then uses a learned mixture-of-experts to select among video-derived captions. That is a real contribution, and the paper shows it works. The ablations in Table IV are coherent: both CSI and ECS contribute, and the original video plus generated captions is the combination that matters. The plug-in gains over DRL, TS2-Net, and Clip4Clip are meaningful. The writing is clear and the code is released, so the method is reproducible in principle.\n\nWhat is actually new is the direction: instead of rewriting the text side, they augment the video side with generated captions and then use a learned router to pick the right expressions. Combining automatic prompt optimization with a caption-selection expert module for VTR appears to be genuinely first, and it is a sensible idea.\n\nThe main soft spot is the caption-scoring proxy in Eq. (1). The paper explicitly says it 'hopes' that Score_t correlates with retrieval, but no evidence is provided. The ablation only shows the final optimized prompt beats the initial prompt; it does not show that Score_t is predictive across a range of prompts. That matters, because CSI is hill-climbing on that score. If the proxy is miscalibrated, the 'self-improvement' could be chasing an unrelated objective. The stress-test note lands: this is the weakest link in the causal chain.\n\nSecond, all results are single-run point estimates with no error bars. The reported SOTA numbers on MSR-VTT and DiDeMo are large, and they come from a proprietary GPT-4o captioner. There is real contamination risk, and no control for the extra test-time caption generation compute. These issues are fixable with validation of the proxy, multiple seeds, and either an open captioner or a contamination check.\n\nA smaller complaint: the 'mitigates empiricism' framing overstates things. The prompt engineer is itself an empirical component, and the whole pipeline depends on the choice of mLLM and the initial prompt. That is not a flaw in the method, but the rhetoric should be toned down.\n\nOverall, the core idea is worth refereeing. The unvalidated proxy and the missing error bars are exactly what a serious reviewer should ask for. If those come back solid, this is a useful contribution to the subfield.\n\nFor peer review: yes, send it out. It deserves a serious referee even though I am skeptical of the absolute numbers until the proxy is validated.","headline":"A genuinely new video-side caption augmentation pipeline for video-text retrieval, with a load-bearing but unvalidated caption-scoring proxy and single-run numbers that need scrutiny before the SOTA claims are trusted.","tokens_in":14450,"tokens_out":2629,"would_cite":true,"duration_ms":24423,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Generating diverse captions on the video side and learning to select the best per query narrows the modality gap, letting ExCae achieve state-of-the-art retrieval accuracy without extra data.","keywords":["video-text retrieval","caption augmentation","prompt optimization","multimodal large language model","mixture of experts","modality gap","self-improvement"],"falsifier":"Take a set of candidate prompts, run the CSI scoring function on each, then fine-tune the retrieval model with each prompt's captions and measure R@1 on a held-out validation split; if the ranking of prompts by Score_t does not match the ranking by R@1, the central mechanism fails.","tokens_in":13422,"feed_emoji":"🎬","tokens_out":7598,"duration_ms":64764,"temperature":0.7,"pith_summary":"Video-text retrieval usually fails because the text describing a video is short and one-sided, while the video itself is rich, so the two modalities sit far apart. This paper argues that the better fix is not to rewrite the text, but to add automatically generated, multi-perspective captions on the video side, bringing video representations closer to text. To do that it introduces ExCae, a pipeline with two parts: Caption Self-Improvement, which iteratively refines the prompt of a multimodal language model to make captions both faithful to the original text and diverse, and Expertized Caption Selection, a mixture-of-experts router that chooses which captions best match each query. The authors report state-of-the-art Top-1 recall (68.5% on MSR-VTT, 68.1% on MSVD, 62.0% on DiDeMo) using no extra training data, and show the same modules can be plugged into existing retrieval models to raise their accuracy.","feed_headline":"Self-refined captions top video-text retrieval without extra data","feed_subtitle":"ExCae generates diverse video captions, picks the best per clip, and hits 68.5% top-1 recall on MSR-VTT.","key_machinery":"The two load-bearing pieces are the Caption Self-Improvement module (CSI) and the Expertized Caption Selection module (ECS). CSI runs a loop: a multimodal LLM (the Captioner) generates $K$ captions per video under a query prompt; a separate Prompt Engineer rewrites the prompt; and the rewrite is kept only if the caption score rises. The score is $Score_t = \\mathbb{E}_{i,k}[\\mathrm{sim}(\\psi(c^t_{i,k}), \\psi(t_i))] + \\mathrm{div}(\\psi(c^t_i))$, where $\\psi$ is a semantic embedding extractor, the first term rewards semantic consistency with the original text, and the second term (a mean pairwise dissimilarity over the $K$ captions) rewards diversity. ECS takes the $N$ sampled frames plus the $K$ generated captions as $N{+}K$ video-side expressions and routes them through $M$ learnable experts with top-$R$ gating: $\\tilde{e}_v = \\frac{1}{M}\\sum_m f_m(r_m(e_v)\\otimes e_v) + e_v$. The router weights are learned under the same contrastive loss as the encoders, so the model personalizes which caption viewpoints are used for each video-text pair.","core_discovery":"On the paper's own terms, the central discovery is that directional augmentation — moving from video to text rather than text to video — is what actually expands the representation space enough to shrink the modality gap. The authors show that generated captions alone help little, and original videos alone help little; the large gain appears only when the two are combined and filtered through learnable experts. The prompt optimizer is not chasing a hand-written style: it maximizes a score that rewards semantic agreement with the original caption and diversity among the ten generated captions, and the learned prompt transfers across datasets. With a ViT-G/14 backbone the method reaches R@1 of 67.7% text-to-video and 69.3% video-to-text on MSR-VTT, and similar records on MSVD and DiDeMo, surpassing methods that use extra pretraining data. The paper also reports that the same caption-auto-enhancement and expert-selection modules, attached to existing retrieval encoders, improve their Top-1 recall by roughly five to nine points.","pith_inferences":["Editorial inference: the same CSI loop should transfer to image-text retrieval, where captions are equally short; the scoring function already uses a text embedding extractor, so only the Captioner would need to accept still images.","Editorial inference: the ECS mixture-of-experts router could be applied as a general cross-modal fusion layer — for example, to choose among multiple human-written captions or multiple text rewrites — not just among video-derived captions.","Editorial inference: the paper's convergence claim (CSI stabilizes in about 400 iterations) suggests the learned prompt has converged to a style rather than to dataset-specific details, which is consistent with the reported cross-dataset transfer; a direct test would be to initialize a different dataset with the prompt learned on MSR-VTT and compare against starting from scratch."],"forward_implications":["Retrieval accuracy no longer depends on expensive extra corpora: ExCae on a ViT-B/16 backbone already beats ViT-based methods that use additional data, and the ViT-G/14 variant surpasses pretrained foundation models on MSR-VTT.","Existing video-text retrieval models can be upgraded without redesign: the plug-in experiments raise Top-1 recall by 4.8, 9.4, and 7.7 points for three base methods, indicating a general add-on.","The learned caption prompt is reusable: because prompt optimization is a one-off process and the resulting prompt transfers across datasets, the cost is amortized over many retrieval tasks.","The method keeps improving with model scale: R@1 grows roughly seven points from the ViT-L/14 to ViT-G/14 backbone, implying the best results are not yet saturated."],"supporting_citations":[{"why":"Supplies the pretrained image-text encoder whose weights initialize ExCae and whose contrastive objective aligns the two modalities.","marker":"[23]"},{"why":"One of the baseline video-text retrieval methods that ExCae compares against and plugs into for the plug-in experiment.","marker":"[3]"},{"why":"Prior work that augments video understanding with auxiliary captions; ExCae positions its video-side generation as the extension and reports higher recall.","marker":"[15]"},{"why":"Automatic prompt optimization framework that the CSI module borrows the iterative refinement idea from.","marker":"[20]"},{"why":"Language-rewrites approach that motivates the Prompt Engineer's rewrite-based prompt improvement.","marker":"[31]"},{"why":"Provides the semantic embedding extractor used to compute both semantic consistency and diversity in the caption score.","marker":"[38]"},{"why":"Supplies the modality-gap measurement that the discussion uses to quantify how much ExCae shrinks the gap.","marker":"[39]"},{"why":"Baseline token-shift selection method that ExCae outperforms and also uses as a plug-in testbed.","marker":"[24]"}],"fun_headline_variants":["Video-to-text captioning narrows modality gap, sets retrieval records","Experts auto-pick captions to beat benchmarks without extra data","Self-taught prompt learns to caption, boosting video-text recall","Directional captioning flips the script for video retrieval","Auto-enhanced captions claim top spots in video-text matching"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The caption-scoring function in Equation (1) is assumed to be a valid proxy for retrieval accuracy, but the paper does not demonstrate that a higher score actually predicts higher recall; if the proxy is miscalibrated, the self-improvement loop could be optimizing the wrong target.","fun_headline_variants_meta":{"raw":{"variants":["Video-to-text captioning narrows modality gap, sets retrieval records","Experts auto-pick captions to beat benchmarks without extra data","Self-taught prompt learns to caption, boosting video-text recall","Directional captioning flips the script for video retrieval","Auto-enhanced captions claim top spots in video-text matching"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000315,"raw_usage":{"total_tokens":1838,"prompt_tokens":1047,"completion_tokens":791,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":706}},"tokens_in":663,"tokens_out":791,"duration_ms":8471,"temperature":1.0,"reasoning_tokens":706,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T10:46:14.738129+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of candidate prompts, run the CSI scoring function on each, then fine-tune the retrieval model with each prompt's captions and measure R@1 on a held-out validation split; if the ranking of prompts by Score_t does not match the ranking by R@1, the central mechanism fails.","supporting_citations":[{"cited_title":"Learning transferable visual models from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained image-text encoder whose weights initialize ExCae and whose contrastive objective aligns the two modalities."},{"cited_title":"Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning,","cited_arxiv_id":null,"evidence_quote":"One of the baseline video-text retrieval methods that ExCae compares against and plugs into for the plug-in experiment."},{"cited_title":"Cap4video++: Enhancing video understanding with auxiliary captions,","cited_arxiv_id":null,"evidence_quote":"Prior work that augments video understanding with auxiliary captions; ExCae positions its video-side generation as the extension and reports higher recall."},{"cited_title":"Automatic prompt optimization with","cited_arxiv_id":null,"evidence_quote":"Automatic prompt optimization framework that the CSI module borrows the iterative refinement idea from."},{"cited_title":"Improving clip training with language rewrites,","cited_arxiv_id":null,"evidence_quote":"Language-rewrites approach that motivates the Prompt Engineer's rewrite-based prompt improvement."},{"cited_title":"Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the modality-gap measurement that the discussion uses to quantify how much ExCae shrinks the gap."},{"cited_title":"Ts2-net: Token shift and selection trans- former for text-video retrieval,","cited_arxiv_id":null,"evidence_quote":"Baseline token-shift selection method that ExCae outperforms and also uses as a plug-in testbed."}],"review_version":1}