{"id":"8c5203e6-4f4d-4b5c-9934-25e1c84abecb","arxiv_id":"2412.07619","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning a CLIP retriever with a list-wise loss derived from LVLM feedback improves few-shot performance, although the loss formula as written would reward worse demonstrations, creating an internal contradiction.","lead":"This paper proposes a method to choose better example images and questions when showing a vision-language AI how to do a new task. It reports gains on seven benchmarks, but the training rule as described appears to push the retriever in the wrong direction, which casts doubt on the results.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 6's list-wise ranking loss is sign-reversed relative to its stated goal: for any pair where demonstration zj is better than zi, the loss is minimized when sim(xq, zi) > sim(xq, zj), training the retriever to favor the worse demonstration.","rationale":"After reading the paper, I concur with the reader's REJECT verdict. The central claim is that the list-wise ranking loss in Eq. 6 trains the retriever to rank demonstrations according to LVLM feedback, thereby improving ICL. However, Eq. 6 is internally inconsistent: for any pair where zj is ranked better than zi (r(zj) < r(zi)), m(i,j) > 0 and the logistic loss is minimized when sim(xq, zi) > sim(xq, zj). This trains the retriever to favor the worse demonstration. The paper's own interpretation in Section 3.2.2 confirms this direction. Thus the described training objective cannot explain the observed improvements; the reported gains would be inexplicable or attributable to a different (unreported) loss. This is not an 'outside consensus' issue but a formal inconsistency between equations and intended semantics. The reader's weakest_assumption about the SIT-IPDR train/test gap is also plausible and untested, but it is secondary: even if that gap were closed, the sign error would still invalidate the argument. I therefore recommend REJECT, consistent with the reader, though we disagree on which concern is most load-bearing. A concrete check (implementing Eq. 6 exactly and flipping the sign) would decisively resolve the matter.","tokens_in":12195,"tokens_out":4656,"duration_ms":34690,"concrete_test":"Implement Eq. 6 exactly as written on a subset of one dataset (e.g., VQAv2, 2,000 training queries, n=8 candidates) and monitor S(E) on Dclip_dev (Eq. 10) across training steps. If S(E) decreases while the loss decreases, the sign is wrong. Then run the same training with the sign flipped: replace sim(xq, zj) − sim(xq, zi) with sim(xq, zi) − sim(xq, zj) in Eq. 6. If the flipped version yields ICL accuracy on the test set comparable to Table 1's DRUM numbers (e.g., > 52.4 on VQAv2), while the as-written version does not, this confirms the reported gains rely on the opposite sign, making the paper's equation an error.","verdict_should_be":"REJECT","load_bearing_attack":"Section 3.2.2 defines the training loss as Lr = Σ_{i≠j} m(i,j) * log(1 + exp(sim(xq, zj) − sim(xq, zi))) with m(i,j) = max(0, 1/√r(zj) − 1/√r(zi)). Because r() is the rank in ascending order (Eq. 4), smaller r means a better demonstration. For a pair where zj ranks higher than zi (r(zj) < r(zi)), m(i,j) > 0 and the loss is increasing in sim(xq, zj) − sim(xq, zi); minimizing it drives sim(xq, zi) > sim(xq, zj), i.e., the retriever is trained to assign higher similarity to the worse demonstration. The text confirms this reading: 'Lr will draw sim(xq, zi) up and optimize the retriever towards sim(xq, zi) > sim(xq, zj)' (Section 3.2.2), which is the opposite of the stated goal of retrieving 'more proper demonstrations.' The paper's own ablation variants (DRUM-1 through DRUM-4) all retain the same sign convention, so they do not resolve the inconsistency. As written, the loss is anti-correlated with LVLM feedback; the reported improvements cannot be causally attributed to the described optimization. If the authors intended the opposite sign, the manuscript describing Eq. 6 is wrong and must be corrected; if they implemented it as written, the method would degrade retrieval, contradicting the experimental claims.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DRUM, a framework for fine-tuning a CLIP-based visual-language embedding model to retrieve in-context demonstrations for large vision-language models (LVLMs). The method defines retrieval strategies over images, prompts, and draft responses; uses the LVLM's conditional log-likelihood to rank candidate demonstrations; trains the retriever with a list-wise ranking loss; and iteratively mines demonstration candidates. Experiments are reported on seven benchmark datasets spanning VQA, image classification, and image captioning, using Open-Flamingo 9B as the inference LVLM, with claims of consistent improvement over baselines and transferability to commercial LVLMs.","tokens_in":12479,"tokens_out":4302,"duration_ms":37351,"significance":"If the described training objective were correct, DRUM would offer a general, task-agnostic way to adapt an embedding model to an LVLM's in-context learning preferences, which is a useful and timely contribution. The paper also addresses an important gap: most prior demonstration-retrieval work targets text-only LLMs, whereas this work targets multimodal LVLMs. However, the central training loss in Eq. (6) is, as written, anti-correlated with the LVLM feedback that it is supposed to inject, and the paper explicitly describes that anti-correlated behavior. The reported experimental gains therefore cannot be attributed to the described method. The paper also lacks a defined baseline, has a mismatched caption/table entry, and leaves a significant train/test representation gap unexamined. The contribution is not acceptable in its current form.","major_comments":[{"comment":"The list-wise ranking loss is sign-reversed relative to its stated purpose. For any pair with r(zj) < r(zi), meaning demonstration zj is ranked better by the LVLM, the coefficient m(i,j) = max(0, 1/sqrt(r(zj)) - 1/sqrt(r(zi))) is positive, and the term log(1 + exp(sim(xq,zj) - sim(xq,zi))) is minimized when sim(xq,zi) > sim(xq,zj). Thus the loss trains the retriever to assign higher similarity to the worse demonstration. The accompanying text confirms this: \"Lr will draw sim(xq, zi) up and optimize the retriever towards sim(xq, zi) > sim(xq, zj)\". This directly contradicts the stated goal of retrieving more helpful demonstrations and the validation metric S(E) in Eq. (10), which rewards agreement with the LVLM ranking. Since all ablation variants retain the same sign convention, the reported improvements cannot be causally explained by the described optimization. The authors must either correct the sign in Eq. (6) (and the surrounding text) or, if the implementation used a different objective, clearly restate the actual training objective and rerun the analysis.","section":"Section 3.2.2, Eq. (6) and Eq. (7)"},{"comment":"There is an unexamined train/test gap in the query representation. During training, the querying example xq includes the ground-truth response responseq, and the retriever's similarity sim(xq, zj) is evidently based on the full triplet (imageq, promptq, responseq) under the SIT-IPDR strategy. At inference, the SIT-IPDR strategy uses a draft response of at most two tokens generated by the LVLM, which can be incomplete or incorrect. The paper provides no analysis, bound, or ablation showing that the learned retriever remains effective when the ground-truth response is replaced by this short draft. This gap is load-bearing because the retriever's ranking signal during training may rely heavily on the full response text, which is unavailable at deployment time.","section":"Sections 3.1 and 3.2.1"},{"comment":"The experimental presentation is internally inconsistent. Section 4.4 lists baselines (a) through (g) and does not define any method called \"Dr-VL\", yet Table 1 reports a \"Dr-VL\" row. The table caption states that results for \"Random, Fixed, EPR, UDR and DRUM\" are averaged over five runs, but the table contains no \"UDR\" row. The conclusion (Section 5) also says \"UDR significantly outperforms the baseline demonstration retrieval methods\" instead of DRUM. These inconsistencies make it impossible to determine which methods were actually compared and which of the reported numbers correspond to the authors' own prior UDR work. The authors must clarify the baseline set, correct the table and caption, and ensure the conclusion refers to the method proposed in this paper.","section":"Section 4.4, Table 1, and Section 5"}],"minor_comments":[{"comment":"The text refers to \"w will be a high weight\" when discussing m(i,j), but no variable w is defined; this should be m(i,j) throughout.","section":"Section 3.2.2"},{"comment":"Equation (9) and Equation (10) present the identical formula for S(E); the duplicate should be removed, and the validation metric should be defined once.","section":"Section 3.2.2 and Section 3.2 (Eq. 9 vs Eq. 10)"},{"comment":"The reference for Open-Flamingo is given as \"Lin, 2004\" (the ROUGE paper); the correct citation is Awadalla et al. (2023).","section":"Section 4.3"},{"comment":"There are several typos and inconsistent spellings: \"Flicker30K\" for Flickr30K, \"lantency\" for latency, \"visiion-language\" for vision-language, \"concate\" for concatenate, and \"unifined\" for unified. These should be corrected.","section":"Throughout"},{"comment":"The conclusion claims transferability across LVLMs from \"4B to 175B\", but the experiments use Open-Flamingo 9B, GPT-4o, and Claude 3 Opus; the paper does not report experiments with a 4B or 175B model, so this claim is unsupported.","section":"Section 5"}],"recommendation":"reject","confidential_remarks":"The sign error in Eq. (6) is not a minor typo: the manuscript explicitly describes the anti-correlated behavior, and the validation metric would necessarily decrease under the stated objective. This suggests the reported experiments may have used a different objective than the one described. Given that the central claim depends on this objective, I cannot recommend publication without a corrected derivation and re-validation. The authors should also be asked to clarify the relationship to UDR (Li et al., 2023c), since the iterative mining strategy is directly adapted from that work and the conclusion confuses the two methods."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the central training signal is backwards. Eq. 6's loss increases when the better demonstration gets higher similarity, so minimizing it pushes the retriever to prefer worse demonstrations. The text even says so ('Lr will draw sim(xq, zi) up...') while claiming the opposite. This is not a nitpick; it breaks the causal story for every reported gain.\n\nWhat's genuinely useful: the SIT-IPDR retrieval strategy (concatenating image and text embeddings, using a short draft response) is a reasonable empirical idea, and the paper evaluates it across seven datasets with several baselines. The ambition to replace task-specific training signals with a unified list-wise loss is also sensible, even if the implementation is broken here.\n\nThe rest of the soft spots are secondary but real. Inference uses a two-token draft response while training uses ground-truth responses in the query embedding; there is no analysis of that gap. No code is released, and the numbers appear without error bars despite the caption claiming five-seed averages. The conclusion overclaims transfer 'across different LVLMs (4B to 175B)' when only Open-Flamingo 9B to GPT-4o/Claude is tested. The baseline table is sloppy: the caption mentions UDR but the table lists 'Dr-VL' with no definition.\n\nCould the sign be a typo? Possibly, and if so the empirical results might be salvageable. But as written, the loss is anti-correlated with the LVLM feedback, so the paper is internally contradictory. That warrants a desk reject rather than reviewer time. I'd encourage the authors to fix the sign, release code, and add variance estimates; then the SIT-IPDR idea plus a corrected loss could be a useful incremental contribution.","headline":"Equation 6 is sign-reversed: the loss trains the retriever to favor worse demonstrations, so the claimed improvements cannot be caused by the described optimization; the SIT-IPDR idea is worth salvaging, but the paper needs a major fix.","tokens_in":13110,"tokens_out":2908,"would_cite":false,"duration_ms":24295,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DRUM fine-tunes a multimodal embedding retriever with a list-wise ranking loss based on the LVLM's own conditional log-likelihood, and reports improved in-context learning on seven benchmark tasks.","keywords":["in-context learning","large vision-language models","demonstration retrieval","list-wise ranking loss","multimodal embeddings","CLIP fine-tuning","visual question answering","image captioning"],"falsifier":"Compare the ranking produced by DRUM for a fixed set of queries when the query embedding is built from the ground-truth response versus from the two-token draft response. If the rank correlation or the downstream in-context learning accuracy drops markedly when the draft response replaces the ground truth, the train/test gap is real and load-bearing; measuring the cosine distance between the two query embeddings on a held-out set would also indicate whether the assumption holds.","tokens_in":11917,"feed_emoji":"🎯","tokens_out":8278,"duration_ms":69474,"temperature":0.7,"pith_summary":"This paper claims that for in-context learning in large vision-language models (LVLMs), the choice of demonstrations should be learned from the LVLM's own preferences rather than fixed or similarity-based heuristics. DRUM fine-tunes a standard image-text embedding model so that its retrieval rankings match the LVLM's conditional log-likelihood of the correct answer under each candidate demonstration, using a list-wise ranking loss with rank-distance weights. At inference it retrieves demonstrations by concatenating image, prompt, and a two-token draft-response embedding for each test sample. On seven datasets spanning visual question answering, image classification, and image captioning, the fine-tuned retriever is reported to outperform random, fixed, CLIP-only, and EPR baselines, and to transfer to API-based LVLMs that cannot provide the training feedback themselves.","feed_headline":"Fine-tuned retriever beats fixed and CLIP-only demos on 7 tasks","feed_subtitle":"Choosing demonstrations the LVLM itself prefers lifts in-context answers, and transfers to GPT-4o and Claude 3 Opus.","key_machinery":"The central object is the list-wise ranking loss $L_r = \\sum_{i \\neq j} m(i,j) \\log(1 + e^{\\text{sim}(x_q, z_j) - \\text{sim}(x_q, z_i)})$, with $m(i,j) = \\max(0, 1/\\sqrt{r(z_j)} - 1/\\sqrt{r(z_i)})$, where $r(z)$ is the rank assigned by the LVLM's conditional log-likelihood and $\\text{sim}$ is the cosine similarity from the fine-tuned CLIP embedding model. This loss converts the LVLM's preference ordering into dense gradient signal for the retriever. The companion mechanism is the SIT-IPDR retrieval strategy, which concatenates image, prompt, and draft-response embeddings so that both modalities and the predicted answer contribute to similarity; a two-token draft response keeps the added inference cost small.","core_discovery":"On its own terms, the paper's discovery is that a retriever can be trained to predict which demonstrations an LVLM will find useful, and that this prediction transfers across tasks and across LVLMs. Given a query sample, candidate demonstrations are ranked by the LVLM's conditional log-likelihood of the ground-truth response; the embedding model is trained with a list-wise ranking loss that pulls up the similarity of demonstrations the LVLM ranked higher and pushes down the rest, with per-pair weights proportional to the difference of inverse square roots of their ranks. The same loss works without task-specific modifications for visual question answering, image classification, and image captioning. The paper further claims that retrieval with concatenated image, prompt, and draft-response embeddings outperforms image-only or prompt-only variants, that iterative re-mining of candidates improves training, and that a retriever fine-tuned on feedback from an open LVLM improves the in-context performance of GPT-4o and Claude 3 Opus.","pith_inferences":["The unvalidated train/test gap around the draft response suggests a concrete extension: train with a small amount of response noise or a consistency regularizer so the retriever is robust to draft-response quality.","The same list-wise feedback loop could be applied to text-only LLM in-context learning or multimodal retrieval-augmented generation, wherever a frozen teacher can score candidate context blocks by likelihood.","The cross-LVLM transfer result implies demonstration preferences are partly shared across models; measuring rank correlation between different LVLMs on identical candidate sets would test how far that sharing extends.","Because the retriever is tuned to one LVLM's likelihood, updating the target LVLM may require re-mining candidates; the paper does not study how quickly DRUM's advantage decays across model generations."],"forward_implications":["Demonstration quality can outweigh demonstration count: the paper reports DRUM with one or two demonstrations outperforming EPR with four on generation-heavy tasks.","The same list-wise ranking loss applies across task types without task-specific objectives, so new vision-language tasks need only a supporting set and a likelihood-capable LVLM to train a retriever.","Retrievers fine-tuned on one LVLM's feedback can improve other LVLMs, including API-only models that cannot provide likelihood feedback during training.","Combining image, prompt, and draft-response embeddings is the strongest retrieval configuration in the paper's ablations, suggesting text-only or image-only retrieval leaves useful signal unused.","Iterative demonstration candidate mining contributes to the gains, since removing it in the ablation lowers performance."],"supporting_citations":[{"why":"Supplies the CLIP image-text embedding model that DRUM initializes and fine-tunes.","marker":"(Radford et al., 2021)"},{"why":"Introduces EPR, the learning-based demonstration-retrieval baseline, and the idea of using LM feedback as a training signal.","marker":"(Rubin et al., 2021)"},{"why":"Contributes the unified demonstration retriever with iterative candidate mining that DRUM adapts.","marker":"(Li et al., 2023c)"},{"why":"Defines the in-context demonstration retrieval strategies for LVLMs that DRUM extends with SIT-IPDR.","marker":"(Li et al., 2024)"},{"why":"Flamingo is the LVLM whose in-context learning capability motivates the task and supplies the VQA evaluation setting.","marker":"(Alayrac et al., 2022)"},{"why":"Open-Flamingo 9B is the LVLM used to compute feedback rankings and to evaluate ICL performance.","marker":"(Awadalla et al., 2023)"},{"why":"Faiss provides the vector index used for efficient demonstration retrieval.","marker":"(Douze et al., 2024)"}],"fun_headline_variants":["LVLM feedback trains retriever that picks better demos for 7 tasks","Retriever learns from LVLM feedback, transfers to GPT-4o and Claude 3","Fine-tuned retriever on LVLM picks better demos for 7 tasks","Demonstration retriever trained with LVLM feedback boosts ICL on 7 benchmarks","Retriever fine-tuned on LVLM preferences outperforms fixed and CLIP demos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"At test time the query embedding uses a two-token draft response generated by the LVLM, but during training the query embedding includes the ground-truth response; the paper assumes these two representations place the query close enough in the embedding space that the fine-tuned rankings remain correct, and it provides no ablation or analysis of this mismatch.","fun_headline_variants_meta":{"raw":{"variants":["LVLM feedback trains retriever that picks better demos for 7 tasks","Retriever learns from LVLM feedback, transfers to GPT-4o and Claude 3","Fine-tuned retriever on LVLM picks better demos for 7 tasks","Demonstration retriever trained with LVLM feedback boosts ICL on 7 benchmarks","Retriever fine-tuned on LVLM preferences outperforms fixed and CLIP demos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000674,"raw_usage":{"total_tokens":3102,"prompt_tokens":1013,"completion_tokens":2089,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":1976}},"tokens_in":629,"tokens_out":2089,"duration_ms":14437,"temperature":1.0,"reasoning_tokens":1976,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:39:20.101362+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the ranking produced by DRUM for a fixed set of queries when the query embedding is built from the ground-truth response versus from the two-token draft response. If the rank correlation or the downstream in-context learning accuracy drops markedly when the draft response replaces the ground truth, the train/test gap is real and load-bearing; measuring the cosine distance between the two query embeddings on a held-out set would also indicate whether the assumption holds.","supporting_citations":[],"review_version":1}