{"id":"a9aa22a9-3358-46f2-8a7f-9f947a646837","arxiv_id":"2501.07391","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"An empirical RAG ablation study finds contrastive in-context learning and sentence-level focus retrieval helpful, but the main ICL result is undermined by using the evaluation dataset as the knowledge base.","lead":"This paper runs a large battery of RAG system configurations, testing how prompt, chunk size, knowledge base size, query expansion, in-context examples, and sentence-level retrieval affect answer quality. It reports that contrastive in-context learning and a sentence-level Focus Mode improve answers, but the top method's evaluation leaks test-set answers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Contrastive ICL's reported superiority is not supported because its knowledge base consists of gold-annotated evaluation examples; the active-query mask does not remove test-label leakage, so the comparison against Wikipedia-based baselines is confounded.","rationale":"The reader's weakest_assumption pinpoints the load-bearing flaw. The paper's central claim is explicitly that Contrastive ICL RAG outperforms all other RAG variants, and this claim is supported only by experiments in which the Contrastive ICL variants retrieve from the evaluation dataset itself, including correct and incorrect gold answers. Masking the active query removes only the trivial case of retrieving the same question; it does not remove the more general leakage whereby a model can retrieve a nearby test question together with its gold answer and use that to answer the current question. This is especially severe on MMLU, where questions within a subject are topically clustered and the reported gains are dramatic. The comparison is therefore not between RAG variants over equivalent knowledge sources; it is between few-shot prompting with test labels and retrieval-augmented generation over Wikipedia. The concern is internal to the evaluation design and is acknowledged in the paper's own description of Q7, so it is not a matter of disagreeing with community consensus. I give credit for the released code and the breadth of the ablations, and I would be glad to see the headline claim restored by a held-out knowledge-base experiment. Until that check is done, the strongest claim in the paper is not supported, and the submitted version does not meet the bar for acceptance.","tokens_in":15278,"tokens_out":2955,"duration_ms":29677,"concrete_test":"Re-run the Contrastive ICL experiment using a knowledge base that contains no test instances and no gold answers from the evaluation split. For MMLU, build demonstrations from the dev split (or a disjoint sampled set of questions with answer keys held out entirely from the 1,824 evaluated items); for TruthfulQA, use its training/validation split if available and ensure the active query's gold answer is excluded. Evaluate ICL1Doc+, ICL2Doc+, and the Wikipedia-based baseline on the same 1,824 MMLU and 817 TruthfulQA items with identical metrics. If the Contrastive ICL advantage largely disappears, the superiority claim is an evaluation artifact. Additionally, log the retrieved demonstration for 100 random queries and check whether any retrieved question is a paraphrase of the active query or has a gold answer matching the active query's correct answer.","verdict_should_be":"REJECT","load_bearing_attack":"The headline finding—Contrastive In-Context Learning RAG outperforms all variants (Section 6, finding 1)—rests on the Contrastive ICL evaluation in Section 5.1 (Q7) and Appendix A.3. There, the 'knowledge base' is the evaluation dataset itself: correct and incorrect answers from TruthfulQA/MMLU are indexed, retrieved, and inserted into the prompt as demonstrations. Masking the active query (Section 3.1 Q7) prevents retrieving the exact current question, but it does not prevent test-label leakage: for MMLU, all 32 per-subject evaluation examples sit in the retrieval index, and the prompt supplies the gold correct answer (and, for '+' variants, the gold incorrect answer) of the nearest neighbor. Because the baseline and all other variants retrieve from Wikipedia Vital Articles, they never receive gold answers in the prompt. Thus the comparison measures in-context learning from the test distribution against RAG over external knowledge, not two comparable RAG designs. The observed advantage, especially the large MMLU jumps (e.g., ICL2Doc+ ROUGE-2 from ~1.9 to ~17.5; FActScore 74.44 vs 63.73), is exactly what this confound would predict. This is a stated design choice, not a hidden bug, but it invalidates the central comparative claim. The reader's rejection is well-founded; the paper's other ablations may be useful, but they do not rescue the headline result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of Retrieval-Augmented Generation (RAG) configurations, introducing four new variants: Query Expansion, Contrastive In-Context Learning (ICL), a Multilingual knowledge base, and Focus Mode. The authors evaluate these and other ablations (LLM size, prompt design, chunk size, knowledge base size, retrieval stride) on TruthfulQA and MMLU using ROUGE, embedding cosine similarity, MAUVE, and FActScore. The central claim is that Contrastive ICL RAG outperforms all other variants, especially on MMLU, followed by Focus Mode RAG. The paper also reports best practices such as the relative unimportance of knowledge base size and the importance of prompt design.","tokens_in":15623,"tokens_out":3923,"duration_ms":36279,"significance":"If the results were valid, the paper would be a useful reference for practitioners, and the public code and broad ablation coverage are commendable. However, the headline finding about Contrastive ICL is compromised by a fundamental evaluation design flaw: the ICL knowledge base is the evaluation dataset itself, so the model is prompted with gold-annotated answers from the test distribution. This makes the comparison against the Wikipedia-based baselines invalid for the paper's central claim. The other ablations (e.g., Focus Mode, prompt design) may still be informative, but they do not rescue the main conclusion as presented.","major_comments":[{"comment":"The Contrastive ICL experiment uses the evaluation dataset as the knowledge base, and the prompt templates in Appendix A.3 insert the correct answer (and, for '+' variants, the incorrect answer) of a retrieved example. Masking the active query prevents retrieving the exact current question, but it does not prevent retrieving other test examples with gold labels. For MMLU, all 1,824 evaluation samples are indexed, so the model receives the gold answer of a similar question in its prompt. All other variants retrieve from Wikipedia Vital Articles and never receive such gold answers. This is test-label leakage, not a comparable RAG design. The large MMLU gains (Table 2: ROUGE-2 from 1.90 to 17.46; Table 3: FActScore from 63.73 to 74.44) are exactly what this confound predicts. This invalidates finding (1) in Section 6. The authors need to use a knowledge base disjoint from the evaluation data (e.g., training splits or an external corpus) and re-run the ICL experiments to support the claimed superiority.","section":"Section 3.1 Q7 and Appendix A.3"},{"comment":"Table 2's footnote states that bold scores denote 'statistical significance over the baseline,' but no significance test, p-value, confidence interval, or multiple-comparison correction is reported anywhere in the paper. Phrases such as 'significantly outperforming' (Section 5.1, item 2) and 'statistically significant' are therefore unsupported. The authors should either report appropriate statistical tests (with correction for the many comparisons across 74 runs) or remove the significance claims.","section":"Table 2 and Section 5.1"},{"comment":"The qualitative example for ICL1D+ on MMLU ('The correct answer is I. Bending and III. Asymmetric stretching.') is the gold answer itself. This demonstrates concretely that the prompt transfers test labels into the generated response, confirming that the measured ICL advantage reflects answer leakage rather than improved retrieval or generation capability.","section":"Table 5 and Section 5.3"},{"comment":"The Limitations section does not acknowledge the evaluation-data leakage in the Contrastive ICL setup, despite this being the most serious threat to the paper's central claim. Any revised version must address this issue explicitly and explain how the headline result can be validated under a leakage-free protocol.","section":"Section 8 Limitations"}],"minor_comments":[{"comment":"The phrase 'are the still the top models' should read 'are still the top models.'","section":"Section 6, Finding 5"},{"comment":"The header 'Artivles' appears to be a typo for 'Articles.'","section":"Table 4"},{"comment":"The naming 'ExpendS', 'ExpendM', 'ExpendL' is inconsistent with the term 'Query Expansion'; consider renaming to 'ExpandS', 'ExpandM', 'ExpandL' for clarity.","section":"Section 5.1, Item 6"},{"comment":"In the ICL prompt templates, the placeholder 'Question: q' is used for both the retrieved example's question and the active query. Clarify the notation to distinguish the example question from the current query, as this is central to understanding the leakage issue.","section":"Appendix A.3"},{"comment":"The MAUVE description states that it is evaluated on the entire dataset to ensure stability, but no standard errors or confidence intervals are reported. Given the small per-subject sample size in MMLU, the authors should report uncertainty or at least discuss the sensitivity of MAUVE to sample size.","section":"Section 4.3"}],"recommendation":"reject","confidential_remarks":"The paper has a clear strength in its breadth of ablations and public code, but the central comparative claim is invalidated by the evaluation design. The ICL experiments retrieve from the evaluation dataset itself, leaking gold answers into the prompt, while baselines retrieve from Wikipedia. This is not a minor fix; it requires redesigning the ICL evaluation with a disjoint knowledge base and re-running the experiments. The unsupported 'statistical significance' claims in Table 2 further weaken the paper. I recommend rejection. If the authors re-run the experiments with proper separation and significance testing, a resubmission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the headline result is not supported. The Contrastive ICL variant uses the gold-annotated evaluation set as its retrieval knowledge base, so the prompt gets correct (and sometimes incorrect) answers from the same distribution the model is scored on. Masking the active query prevents exact self-retrieval, but it does not remove test-label leakage. Baselines retrieve from Wikipedia Vital Articles and never see gold answers. That makes the central comparison \"test-time supervision versus external-knowledge RAG,\" not two comparable RAG designs. The large MMLU gains (ROUGE-2 from ~1.9 to ~17.5; FActScore from 63.73 to 74.44) are exactly the footprint of this confound. The stress-test note is correct, and the reader's rejection is well-founded.\n\nCredit where due: this is a systematic ablation study—nine research questions, two datasets, 74 runs, public code. Some negative results are useful for practitioners: knowledge base size matters less than expected, query expansion gives marginal gains, small retrieval strides hurt, adversarial prompts hurt. The four \"novel\" configurations are mostly known techniques under new names (sentence-level retrieval, query expansion, few-shot contrastive prompting, multilingual retrieval), so the novelty is incremental. That alone is not fatal, but it lowers the bar for what a best-practices paper should add.\n\nOther soft spots: Table 2 says bold scores are statistically significant over baseline, but no test, p-value, or confidence interval appears anywhere. For a paper making best-practice claims, that is a real reporting gap. Section 8 lists limitations but never acknowledges that the ICL knowledge base is the gold-labeled evaluation set. That omission is as worrying as the design itself.\n\nWho is this for? A practitioner skimming for RAG configuration sanity checks might get something from the non-ICL ablations. The ICL numbers should be treated as artifacts. The paper would make a good teaching example of evaluation contamination. I would not accept it as is. If the authors rerun the ICL experiments against an external knowledge base, add actual significance tests, and reposition the claims, this could become a modest but useful benchmark paper. Deserves a serious referee rather than a desk reject because there is enough systematic empirical work to salvage, but the current form needs major revision.","headline":"A fatal test-set leakage sinks the Contrastive ICL headline; the rest is a useful but incremental ablation study.","tokens_in":16111,"tokens_out":4571,"would_cite":false,"duration_ms":44582,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that showing a RAG model retrieved correct and incorrect answer pairs for the same question improves accuracy and factuality more than any other configuration tested, especially on specialized-knowledge questions.","keywords":["retrieval-augmented generation","contrastive in-context learning","query expansion","focus mode","RAG best practices","factuality evaluation","ablation study","LLM prompting"],"falsifier":"Hold out the target question's own example from the retrieval knowledge base, or swap in demonstrations drawn from a disjoint development set, and re-run the Contrastive ICL experiments. If the advantage over the baseline disappears or shrinks sharply, the reported gains came from revealing the answer inside the prompt, not from contrastive learning.","tokens_in":15111,"feed_emoji":"🎯","tokens_out":7829,"duration_ms":65179,"temperature":0.7,"pith_summary":"This paper tries to establish which design choices most improve retrieval-augmented generation (RAG), the technique of feeding an LLM retrieved context at inference time. It runs 74 ablation experiments varying model size, prompt wording, chunk size, knowledge-base size, retrieval frequency, query expansion, multilingual context, sentence-level focusing, and in-context demonstrations. The central finding is that Contrastive In-Context Learning RAG, which retrieves correct and incorrect answer examples for the same question and puts them in the prompt, outperforms every other variant on both general-knowledge and specialized-knowledge questions. A second proposed configuration, Focus Mode, which retrieves individual sentences rather than whole documents, ranks second. The paper concludes that the relevance and quality of retrieved documents matter more than their quantity, and that prompt phrasing remains decisive even inside a RAG system.","feed_headline":"RAG does best when prompts show right and wrong answers","feed_subtitle":"A 74-run ablation finds contrastive example pairs beat larger models, bigger knowledge bases, and query expansion.","key_machinery":"The load-bearing mechanism is Contrastive In-Context Learning: the retrieved context is not general encyclopedic text but a small set of demonstration examples consisting of the same question with its correct answer and a deliberately incorrect answer, inserted into the prompt before the active query. The retrieval pipeline that carries the method is built from a vector-similarity search index (FAISS) over sentence-transformer embeddings of the knowledge base, a T5-based query-expansion module, and a Mistral instruction-tuned generator. Focus Mode, the second-ranked configuration, works by splitting retrieved documents into sentences and re-ranking them against the query so the generator sees only the most relevant sentences.","core_discovery":"The paper's central claim is that adding contrastive demonstrations to a RAG prompt is the single most effective configuration among those tested. In this design, the knowledge base is the evaluation dataset itself: each query retrieves example question–answer pairs of the same type, and the prompt presents both a correct answer and an incorrect (contrastive) answer before asking the model to answer the active question, with the active query masked during retrieval. Across ROUGE, embedding similarity, MAUVE, and FActScore, these Contrastive ICL variants beat the baseline RAG and all other variants, with the margin widening on MMLU, the dataset requiring specialized knowledge. The authors interpret this as the model learning to discriminate correct from incorrect response patterns.","pith_inferences":["A stricter evaluation that draws demonstrations from a disjoint development set would separate the method's genuine effect from answer leakage; the paper's current setup does not fully rule this out.","Combining Contrastive ICL with Focus Mode or Query Expansion, which the paper leaves untested, could yield additional gains beyond any single module.","The contrastive demonstration idea transfers naturally to other conditional generation tasks such as summarization or simplification, where correct and incorrect outputs for the same input can be obtained.","A practical takeaway for practitioners: treat the module ranking with caution until leakage-free reproduction is run, but expect the qualitative message that demonstrations help and sentence-level focus helps to be robust."],"forward_implications":["RAG practitioners should consider adding contrastive demonstrations (correct and incorrect answer pairs) to prompts for question-answering, especially on knowledge-heavy tasks.","Knowledge-base size can be kept small if the retrieved documents are on-topic; the paper finds 1K and 10K article bases perform comparably.","Retrieving individual sentences (Focus Mode) is a low-cost way to sharpen responses when full documents add noise.","Frequent retrieval updates during generation (small strides) hurt coherence; larger strides preserve context stability.","Prompt wording has a measurable effect even within RAG, so prompt design deserves the same attention as retrieval choices."],"supporting_citations":[{"why":"Supplies the TruthfulQA dataset used for demonstrations and general-knowledge evaluation.","marker":"(Lin et al., 2022)"},{"why":"Supplies the MMLU dataset used for specialized-knowledge evaluation and demonstrations.","marker":"(Hendrycks et al., 2021)"},{"why":"Provides the sentence-transformer encoder for query and document embeddings and for similarity metrics.","marker":"(Reimers and Gurevych, 2019)"},{"why":"Provides the FAISS vector index used for retrieval in all RAG variants.","marker":"(Douze et al., 2024)"},{"why":"Provides the Mistral instruction-tuned generator models (7B and 45B) used across experiments.","marker":"(Jiang et al., 2023)"},{"why":"Provides FActScore, the factuality metric used in the factuality assessment.","marker":"(Min et al., 2023)"}],"fun_headline_variants":["Contrastive examples beat bigger models in RAG","RAG prompts with right and wrong answers win","Show contrastive pairs in RAG for best results","Contrastive ICL boosts RAG more than model size"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results for Contrastive ICL assume that retrieving from the evaluation dataset with the active query masked still hides the answer; if the same question appears in the retrieved demonstrations with its correct answer visible, the measured advantage is an artifact of answer leakage rather than a genuine property of the method.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive examples beat bigger models in RAG","RAG prompts with right and wrong answers win","Show contrastive pairs in RAG for best results","Contrastive ICL boosts RAG more than model size"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000364,"raw_usage":{"total_tokens":1937,"prompt_tokens":898,"completion_tokens":1039,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":975}},"tokens_in":514,"tokens_out":1039,"duration_ms":7249,"temperature":1.0,"reasoning_tokens":975,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:42:48.219610+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out the target question's own example from the retrieval knowledge base, or swap in demonstrations drawn from a disjoint development set, and re-run the Contrastive ICL experiments. If the advantage over the baseline disappears or shrinks sharply, the reported gains came from revealing the answer inside the prompt, not from contrastive learning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the TruthfulQA dataset used for demonstrations and general-knowledge evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the sentence-transformer encoder for query and document embeddings and for similarity metrics."}],"review_version":1}