{"id":"bf38e29d-6ccf-4152-aba0-691f4fe871e0","arxiv_id":"2411.18947","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ICLERB evaluates retrievers by their effect on LLM in-context learning accuracy, and the RLRAIF-trained model cm-rerank-mxbai-rlaif-v0.1 tops the benchmark with only 10k DPO queries.","lead":"This paper introduces ICLERB, a benchmark that ranks document retrievers by how much they improve an LLM's in-context learning accuracy, and RLRAIF, a method that fine-tunes a small retriever on a minimal budget of LLM queries. A reader interested in RAG systems would care because the benchmark reorders today's retrieval leaderboard and suggests utility to the LLM, not semantic similarity, should guide retriever selection and training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ICLERB's central claim rests on the untested assumption that the DPO metric (Eq. 7) tracks true ICL accuracy; no evidence links DPO to end-task performance, and RLRAIF is trained and evaluated on the same proxy.","rationale":"The reader's verdict identified the DPO metric as the load-bearing assumption, and my analysis agrees: the entire benchmark and the RLRAIF training signal depend on Eq. 7 being a valid measure of utility. The paper's own text states that DPO has 'desirable properties' without offering any validation against actual ICL accuracy, and the only reference for these properties is a workshop paper, not a benchmark validation study. This is not a discrepancy with external consensus; it is an internal gap between the claimed objective ('enhance LLM accuracy') and the operationalized metric (log-probability ratios in a fixed single-demo prompt). The concrete test I propose would settle the question directly: compute DPO and actual accuracy on the same pairs and compare their rankings. If the correlation is low, the leaderboard can be reordered by accuracy, and RLRAIF should be retrained with an accuracy-based reward. The paper's central claims about the limitations of MTEB and the superiority of RLRAIF would then need to be re-examined. However, the paper's methodology is transparent and internally consistent, so a conditional verdict with a request for this validation is appropriate rather than outright rejection. I chose UNCHANGED because the reader's CONDITIONAL verdict already reflects this concern; my proposed test strengthens the conditionality rather than changing it.","tokens_in":23204,"tokens_out":4723,"duration_ms":42521,"concrete_test":"On a held-out sample of queries from each dataset (e.g., 200 per dataset), compute DPO(q,d) for all documents, then measure the LLM's actual accuracy when prompted with each document as a 1-shot demonstration, using the same prompt template. Compare the ranking induced by DPO with the ranking induced by accuracy (or by the probability assigned to the correct answer). Report Spearman rank correlation per dataset. If the correlation is not high (e.g., below 0.6), the DPO proxy does not faithfully measure ICL utility, and both the ICLERB leaderboard and RLRAIF training need to be re-grounded in measured accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark and the RLRAIF training loop both depend on the DPO metric (Eq. 7, Section 2.4) as a proxy for how much a retrieved document improves ICL. The paper asserts that DPO has 'desirable properties' (Section 3.1) but provides no empirical evidence that documents ranked higher by DPO actually increase the LLM's end-task accuracy. This matters because ICLERB's ground truth is DPO, so the leaderboard (Table 2) ranks models by their agreement with this proxy, not by measured task accuracy. Similarly, RLRAIF (Section 5.2) uses DPO as its reward, so the top-ranked model, cm-rerank-mxbai-rlaif-v0.1, is specifically optimized for the evaluation metric. If DPO diverges from real ICL utility, the comparison against large off-the-shelf models is misleading. The risk is concrete: DPO is computed with a single fixed demonstration template, open-weight LLM log probabilities, one (optionally) incorrect answer r̄, and 1-shot prompting; any of these choices can create a proxy that rewards documents which look helpful in log-probability space but do not improve accuracy. The paper reports no correlation analysis between DPO and accuracy, no human or downstream validation, and no ablation of the template or the choice of r̄. Until such evidence is provided, the central claim is conditional on an untested surrogate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes reframing retrieval for in-context learning as a recommendation problem rather than a search problem, and introduces ICLERB, a benchmark that ranks embedding models and rerankers by how well the documents they retrieve increase the probability an LLM assigns to the correct answer, as quantified by a DPO-based metric (Eq. 7). The benchmark is instantiated on three MCQ-style datasets (TruthfulQA, Emotion, ProductER) and three open-weight LLMs, and the resulting leaderboard (Table 2) is compared against MTEB. The paper then introduces RLRAIF, an active-learning/contextual-bandit procedure that fine-tunes a small cross-encoder adapter using DPO rewards, and reports that the resulting model, cm-rerank-mxbai-rlaif-v0.1, outperforms much larger existing retrieval models on ICLERB with a budget of only 10k DPO evaluations.","tokens_in":23549,"tokens_out":3654,"duration_ms":35077,"significance":"If the central claims hold, ICLERB provides a utility-oriented complement to semantic-similarity benchmarks such as MTEB, and the RLRAIF result is a practically valuable demonstration that a small, cheaply fine-tuned retriever can be competitive with much larger models for retrieval-augmented ICL. The paper is also careful to split queries into train/test sets, to average results over repeated splits, and to report per-dataset and per-LLM breakdowns, which is a strength. However, the significance of both contributions is conditional on the validity of the DPO metric as a proxy for actual ICL accuracy; the manuscript does not yet supply that validation, and the RLRAIF method is described at a level that prevents independent reproduction.","major_comments":[{"comment":"The benchmark's ground truth is the DPO metric, and Section 3.1 asserts that DPO has 'desirable properties' citing [11], but the paper provides no empirical evidence that documents ranked higher by DPO actually improve the LLM's end-task accuracy. This is load-bearing because Table 2 ranks all models by DPO-based nDCG and because RLRAIF (Section 5.2) uses DPO as its reward. Please add a direct validation study: for held-out queries, compute the correlation (or rank agreement) between DPO scores and the LLM's task accuracy when the top document is used as a 1-shot demonstration, and probe sensitivity to the fixed prompt template, the choice of the incorrect answer r̄, and the use of a single demonstration. Without such evidence, ICLERB measures agreement with a specific surrogate, not utility for ICL.","section":"§3.1, Eq. (7)"},{"comment":"The paper states that dataset splits and experiments are repeated and averaged, but no standard deviations, confidence intervals, or significance tests are reported anywhere. The difference between the first and second rows of Table 2 is 0.0047 nDCG@10 (0.7238 vs. 0.7191), and a large cluster of models lies within 0.01 of each other, so the ranking order may be within noise. Please report paired bootstrap confidence intervals over test queries (and per dataset/LLM) or equivalent significance tests to support the claim that RLRAIF's model significantly outperforms the baselines.","section":"§3.2, Table 2"},{"comment":"The RLRAIF acquisition function is described only as a qualitative bullet list ('exploitation in the document space,' 'information for ranking loss,' 'exploration,' 'diversity in the batch'), with no equations or hyperparameter values. Since RLRAIF is a major contribution and the empirical result depends on its specific exploration/exploitation trade-off, the description is not reproducible; the reader cannot tell what was actually optimized. Please provide the exact acquisition function, its coefficients, batch sizes, number of acquisition rounds, the training hyperparameters, and ideally release the code at submission.","section":"§5.2, §5.3"},{"comment":"ProductER is a new dataset introduced in this paper, described as 'manually curated' but in fact generated with OpenAI o1-preview, with no human validation or inter-annotator statistics reported. It is also not released ('we aim to release ... in the near future'), so the benchmark cannot currently be reproduced on one of its three datasets. Please describe the generation and validation protocol in detail, report label-quality checks, and release the dataset or clearly mark ProductER results as preliminary and separate from the main leaderboard.","section":"§3.3.1, Table 1"},{"comment":"Because RLRAIF is trained to maximize the DPO reward and ICLERB evaluates retrievers using the same DPO-based nDCG, the reported superiority of cm-rerank-mxbai-rlaif-v0.1 is partly a measure of alignment between the training objective and the evaluation metric. The test queries are held out, so this is not formally circular, but it does mean that the headline claim ('small models fine-tuned with our RLRAIF algorithm outperform large state-of-the-art retrieval models') is only as strong as the validity of DPO as a proxy for ICL accuracy. I therefore see this as connected to the validation requested in the first major comment, and recommend that the paper either provide that validation or temper the abstract and conclusion claims until it is available.","section":"§5.2, §3.2"}],"minor_comments":[{"comment":"The sentence 'the DPO metric ... is defined as the negative of the DPO loss [52]' is confusing because the original DPO loss is a training loss to be minimized; clarify the sign conventions and define σ explicitly.","section":"§2.4, Eq. (7)"},{"comment":"There is a typo in 'inital version' (should be 'initial').","section":"§3.3.2"},{"comment":"The table is labeled as 'fully extracted from the MTEB benchmark on November 27th, 2024' in the caption, but the body text says 'Table 9 summarizes the performance ... as reported by MTEB' without a date; please ensure the access date is stated consistently.","section":"Table 9"},{"comment":"Equation (10) is the standard logistic pairwise ranking loss; please state this explicitly and cite [6] in the text near the equation rather than only in the reference list.","section":"§5.2, Eq. (10)"},{"comment":"The claim that DPO is 'additive over independent queries' is used to justify aggregate evaluation, but the additive quantity is not written out; a brief statement of how DCG aggregations are pooled across queries would help.","section":"§3.1"},{"comment":"The text says test queries and documents are randomly subsampled, while §3.3.1 says the corpus of documents is fixed to be the training-set ground-truth responses; clarify how documents from the test split, if any, are treated to avoid ambiguity.","section":"§3.2"},{"comment":"Figure 2 is not referenced in the main text; add a reference in Section 5.1 or 5.3.","section":"Appendix C, Figure 2"},{"comment":"Models with undisclosed sizes are marked with '–'; for clarity, consider labeling them as 'not disclosed' or 'API' in the table caption.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central claims are conditional on the validity of a surrogate metric (DPO) that is not validated against actual ICL accuracy. The authors co-authored the paper [11] that introduced this metric, and the RLRAIF model is trained and evaluated on the same benchmark; while the train/test split avoids formal circularity, the lack of any external anchor makes the leaderboard and the RLRAIF comparison hard to interpret independently. I would encourage the editor to treat the requested DPO validation as a required revision rather than a nice-to-have, and to ask for code/data release before any final acceptance, since the benchmark's reproducibility currently depends on unreleased artifacts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, ICLERB is a genuinely new benchmark: it ranks 26 retrievers and rerankers by how much a retrieved document changes an LLM's log-probability of the right answer (the DPO metric from the authors' RAGSys paper), across three datasets and three open-weight LLMs. That reordering relative to MTEB is real and worth discussing. Second, the paper's headline result—a 335M model fine-tuned with RLRAIF on 10k DPO queries beats 7B-parameter retrievers—is conditional on that same DPO metric being a faithful proxy for ICL utility, and the paper never shows that it is.\n\nThe benchmark construction is careful in several ways: train/test splits with held-out queries, repeats, and nDCG aggregation; a clear statement that the corpus is fixed and only cold-start queries are evaluated; and an honest accounting of the compute involved (about 1.5B tokens). The per-dataset and per-LLM appendices show the rankings vary, which is useful for anyone wanting to know how robust the aggregate leaderboard is. The RLRAIF result, if reproduced, would be practically significant: a 150k-parameter adapter on a single GPU is a low barrier to entry.\n\nThe soft spots are exactly where the reader put them. The DPO metric (Eq. 7) is the benchmark's ground truth and RLRAIF's reward, but there is no correlation analysis between DPO and actual multiple-choice accuracy, no ablation of the prompt template or the choice of the incorrect answer r̄, and no evidence that documents ranked high by DPO actually improve end-task performance. The paper calls DPO 'desirable' and cites its own prior work, but that is an assertion, not evidence. This is load-bearing because the leaderboard and the training target are both DPO. If DPO diverges from accuracy, the comparison against off-the-shelf models is misleading. The RLRAIF section is also under-specified: the acquisition function is described qualitatively (exploitation, uncertainty, diversity) without equations or hyperparameters, so the algorithm is not reproducible from the text. No error bars or significance tests on the nDCG differences, and the code and ProductER dataset are not released yet.\n\nI don't think the paper is fatally flawed; the idea is sound and the authors are transparent about scope. But the central claim is premature. The benchmark needs a validation study linking DPO to measured ICL accuracy, and RLRAIF needs a precise specification. Both are doable.\n\nWho should read this: anyone building RAG systems or retrieval benchmarks. It deserves a serious referee, but with major revision, not acceptance as is. I'd like to see the DPO-accuracy correlation before citing it.","headline":"Useful benchmark idea, but the leaderboard and the RLRAIF claim both rest on an unvalidated DPO proxy; worth reviewing, needs empirical anchoring.","tokens_in":24072,"tokens_out":2753,"would_cite":false,"duration_ms":25967,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Retrieval for in-context learning should be scored by how much a document improves the LLM's answer, and a 335M-parameter reranker trained with 10,000 LLM queries outperforms models over twenty times larger on ICLERB.","keywords":["in-context learning","retrieval-augmented generation","retrieval as recommendation","learning to rank","direct preference optimization","active learning","LLM feedback","benchmark"],"falsifier":"Take a held-out set of queries with known answers, compute DPO scores for all documents, and also measure actual answer accuracy when each document is used as the one-shot demonstration; if documents with higher DPO do not systematically yield higher accuracy, or if a retriever trained to maximize DPO does not improve end-task accuracy, the benchmark and the training algorithm lose their justification.","tokens_in":23028,"feed_emoji":"🎯","tokens_out":10080,"duration_ms":78938,"temperature":0.7,"pith_summary":"This paper argues that when retrieval supplies examples to an LLM's prompt, the retriever should be judged by how much it improves the LLM's answers, not by semantic similarity to the query. It introduces ICLERB, a benchmark that scores embedding models and rerankers with the DPO metric, which measures the change in the LLM's log-probability of the correct answer when a document is added to a one-shot prompt. It also introduces RLRAIF, a reinforcement learning-to-rank algorithm that actively chooses which query-document pairs to score and fine-tunes a reranker using about 10,000 such scores. On ICLERB, the resulting 335M-parameter reranker ranks first, ahead of state-of-the-art retrieval models that are more than twenty times larger. The paper concludes that retrieval for ICL is a recommendation problem and that training retrievers against LLM utility rather than search relevance can compensate for much smaller model size.","feed_headline":"Small retriever tuned on LLM feedback tops models 20x larger","feed_subtitle":"A new utility-based benchmark ranks retrievers by how much they improve LLM answers, and the small model wins.","key_machinery":"The central object is the DPO metric, defined for a query $q$, a document $d$, a correct response $r$, and an optional incorrect response $\\bar r$ as $\\mathrm{DPO}(q,d)=\\log\\sigma\\left(\\log\\frac{p_M(r\\mid q,d)}{p_M(r\\mid q)}-\\log\\frac{p_M(\\bar r\\mid q,d)}{p_M(\\bar r\\mid q)}\\right)$, which measures how much adding $d$ to a one-shot prompt increases the LLM's relative log-probability of the correct answer. ICLERB uses this value as the ground-truth relevance label for ranking documents, and RLRAIF uses it as the reward signal in an active-learning loop that selects $(q,d)$ pairs by balancing high expected reward, high uncertainty, and batch diversity. The fine-tuning step is a pairwise ranking loss applied to the resulting reward comparisons, updating a small adapter on top of pre-trained embeddings. This single metric therefore carries both the benchmark's definition of document utility and the training signal that lets a small retriever outperform much larger models.","core_discovery":"The core claim is that retrieval for in-context learning should be reframed from a search problem to a recommendation problem: given a query, the retriever should rank documents by their utility in improving the LLM's response, and that utility can be measured by the DPO metric. ICLERB operationalizes this by building ground-truth relevance labels from DPO scores across multiple multiple-choice datasets and open-weight LLMs, then ranks embedding models and rerankers with nDCG@10 and nDCG@50. The same DPO signal is used as the reward in RLRAIF, which treats data acquisition as a contextual bandit problem that balances exploration and exploitation in both the query and document spaces, and trains a small non-linear adapter on top of a frozen embedding model. With roughly 10,000 DPO evaluations, the fine-tuned 335M-parameter reranker reaches the top of the ICLERB leaderboard, beating models an order of magnitude larger, which the paper takes as evidence that alignment with ICL utility matters more than raw model capacity.","pith_inferences":["Editorial inference: the single-document, one-shot DPO measure ignores interactions among multiple demonstrations, so a multi-document extension that scores sets of documents might rank retrievers differently and could change what RLRAIF optimizes.","Editorial inference: because ICLERB uses one fixed prompt template and one demonstration, the stability of its rankings under prompt phrasing is untested; a prompt-variation study would clarify how much of the leaderboard reflects retrieval quality rather than template effects.","Editorial inference: the benchmark and training loop depend on open-weight LLM log probabilities, so extending the approach to proprietary or closed LLMs would require an alternative feedback signal, such as sampled-completion accuracy.","Editorial inference: the RLRAIF acquisition strategy could be tested on other ranking models and other base embeddings to see whether the dual exploration-exploitation trade-off gives the reported gains beyond the single model and datasets used here."],"forward_implications":["Retriever rankings produced by ICLERB disagree with rankings from semantic-similarity benchmarks on several models, so RAG system builders should expect different component choices when the goal is in-context learning rather than search.","A 335M-parameter model fine-tuned with roughly 10,000 DPO evaluations can beat retrieval models more than twenty times larger, indicating that aligning the training signal with ICL utility can matter more than model capacity.","Search-optimized rerankers can rank below their own embedding counterparts on ICLERB, suggesting that fine-tuning for semantic relevance can actively hurt retrieval for ICL.","Because RLRAIF requires only log-probability access to an open-weight LLM, the same recipe can be applied to any domain with a query set and response labels without building a dedicated retrieval training dataset.","Future ICLERB releases with more datasets and LLMs may change the leaderboard, and RLRAIF can be applied to other base models to test whether the reported gains persist across architectures."],"supporting_citations":[{"why":"Defines the DPO metric that serves as ICLERB's relevance label and RLRAIF's reward signal.","marker":"[11]"},{"why":"Provides the reference retrieval benchmark whose model rankings are compared with ICLERB to demonstrate the discrepancy.","marker":"[47]"},{"why":"Pioneers supervised few-shot demonstration retrieval with LLM utility scores, the baseline approach RLRAIF is designed to improve.","marker":"[56]"},{"why":"Introduces the DPO loss whose negative form is the metric in Equation 7.","marker":"[52]"},{"why":"Iterative demonstration-retriever training that motivates RLRAIF's exploration beyond pure exploitation.","marker":"[35]"},{"why":"Supplies the pairwise ranking loss used to train the retriever from reward pairs.","marker":"[6]"},{"why":"Formulates contextual-bandit learning to rank, which underpins the acquisition strategy.","marker":"[33]"},{"why":"Provides the pre-trained embedding model that the RLRAIF fine-tuned reranker is initialized from.","marker":"[30]"},{"why":"Supplies one of the three few-shot multiple-choice datasets used to construct the ICLERB benchmark.","marker":"[37]"},{"why":"Supplies another of the three few-shot multiple-choice datasets used to construct the ICLERB benchmark.","marker":"[57]"}],"fun_headline_variants":["Retrieval as recommendation: small model tops giants in ICL","ICL retrieval reimagined: 335M model beats 20x bigger","Utility-based benchmark: small reranker wins with LLM feedback","RL-tuned small retriever beats 20x larger on ICL tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that the DPO score of a single document in a one-shot prompt is a faithful measure of how much that document improves the LLM's true end-task accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Retrieval as recommendation: small model tops giants in ICL","ICL retrieval reimagined: 335M model beats 20x bigger","Utility-based benchmark: small reranker wins with LLM feedback","RL-tuned small retriever beats 20x larger on ICL tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000454,"raw_usage":{"total_tokens":2299,"prompt_tokens":978,"completion_tokens":1321,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":1241}},"tokens_in":594,"tokens_out":1321,"duration_ms":9808,"temperature":1.0,"reasoning_tokens":1241,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:42:11.686944+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of queries with known answers, compute DPO scores for all documents, and also measure actual answer accuracy when each document is used as the one-shot demonstration; if documents with higher DPO do not systematically yield higher accuracy, or if a retriever trained to maximize DPO does not improve end-task accuracy, the benchmark and the training algorithm lose their justification.","supporting_citations":[{"cited_title":"Learning to Rank with Nonsmooth Cost Functions","cited_arxiv_id":null,"evidence_quote":"Supplies the pairwise ranking loss used to train the retriever from reward pairs."},{"cited_title":"Schapire","cited_arxiv_id":null,"evidence_quote":"Formulates contextual-bandit learning to rank, which underpins the acquisition strategy."},{"cited_title":"RAGSys: Item-Cold-Start Recommender as RAG System","cited_arxiv_id":null,"evidence_quote":"Defines the DPO metric that serves as ICLERB's relevance label and RLRAIF's reward signal."},{"cited_title":"MTEB: Massive Text Embedding Benchmark","cited_arxiv_id":null,"evidence_quote":"Provides the reference retrieval benchmark whose model rankings are compared with ICLERB to demonstrate the discrepancy."},{"cited_title":"Learning To Retrieve Prompts for In- Context Learning","cited_arxiv_id":null,"evidence_quote":"Pioneers supervised few-shot demonstration retrieval with LLM utility scores, the baseline approach RLRAIF is designed to improve."},{"cited_title":"Direct Preference Optimization: Your Language Model is Secretly a Reward Model","cited_arxiv_id":null,"evidence_quote":"Introduces the DPO loss whose negative form is the metric in Equation 7."},{"cited_title":"Unified Demonstration Retriever for In-Context Learning","cited_arxiv_id":null,"evidence_quote":"Iterative demonstration-retriever training that motivates RLRAIF's exploration beyond pure exploitation."},{"cited_title":"Open Source Strikes Bread - New Fluffy Embeddings Model, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained embedding model that the RLRAIF fine-tuned reranker is initialized from."},{"cited_title":"Lin, Jacob Hilton, and Owain Evans","cited_arxiv_id":null,"evidence_quote":"Supplies one of the three few-shot multiple-choice datasets used to construct the ICLERB benchmark."},{"cited_title":"CARER: Contextualized affect representations for emotion recognition","cited_arxiv_id":null,"evidence_quote":"Supplies another of the three few-shot multiple-choice datasets used to construct the ICLERB benchmark."}],"review_version":1}