{"id":"542029ab-161d-4e28-8025-57fe6db14f6d","arxiv_id":"2411.13691","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A RAG pipeline with BM25, FAISS, a reranker, and a Mistral 7B generator raises domain-specific QA F1 from 5.45% to 42.21% on a self-built Pittsburgh/CMU test set.","lead":"This paper builds a retrieval-augmented question answering system for questions about Pittsburgh and Carnegie Mellon University, combining BM25, FAISS, and a reranker with a Mistral model. On a 165-question test set it reports F1 improving from 5.45% for the non-RAG baseline to 42.21% for the best RAG configuration.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 165 test questions are authored from the same scraped corpus that RAG retrieves from (Section 3), so the reported F1 gap partly reflects a guaranteed-answerable test set rather than general domain-QA superiority; the central comparison needs an independent test set.","rationale":"The reader's weakest assumption was that the 165 test questions are representative and that the corpus contains their answers. My concern is sharper: Section 3 explicitly says the test questions were written from segments of the scraped corpus, so answer presence is guaranteed by construction. This makes the RAG-versus-non-RAG comparison asymmetric in a way that directly inflates the headline gap, especially for time-sensitive items. I still agree with the reader's CONDITIONAL verdict: the paper is a plausible case study, but the central numerical claim lacks external validation. I do not move the verdict because the test-set concern is already embedded in the reader's call for a more rigorous evaluation; it reinforces rather than reverses it. The recall discrepancy (56.18% in the abstract vs 45.39% in Table 1 for the best config) is an additional reporting error that further lowers confidence, but it is secondary to the evaluation-design issue. No ad hominem is intended; the critique targets the experimental design, not the authors.","tokens_in":6819,"tokens_out":7218,"duration_ms":61147,"concrete_test":"Recruit independent annotators who have not seen the scraped corpus to write 100-200 naturally occurring Pittsburgh/CMU questions (e.g., from real user queries or general-interest prompts). Separately verify which answers are present in the corpus. Then run the best RAG configuration and the non-RAG baseline using the same SQuAD F1 script on this independent set, and report the F1 gap on the subset whose answers are in the corpus. If the gap on these independent questions is materially smaller than the 36.76-point gap in Table 1, the paper's headline improvement is an artifact of test-set authorship. As a secondary check, also run the non-RAG baseline with the same 2-shot prompt to isolate retrieval from prompting effects.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is the F1 gap between RAG (42.21%) and non-RAG (5.45%). Section 3 states that the 165 test questions were written by manually selecting segments from the scraped resources and composing Q&A pairs from each segment. The retrieval index (Section 4.2) is built from the same scraped corpus. Consequently, every test question is guaranteed to have an answer in the documents given to RAG, while the non-RAG baseline receives no documents. For time-sensitive queries, such as the Alumni Awards Ceremony example, the baseline cannot possibly know the answer if it postdates the model's training data, so RAG's advantage is built into the test design. The reported 'statistically significant' claim is unsupported by any significance test, but the deeper issue is that the comparison measures retrieval accuracy on a corpus-derived test set, not whether RAG improves domain QA on a representative query distribution. The paper's own Section 6.1 shows RAG's recall on time-sensitive questions (0.2335) is lower than on non-time-sensitive ones (0.3994), suggesting retrieval is challenging, but this does not remove the authorship confound. Without an independent test set, the headline gap cannot be attributed to RAG's general utility.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes a domain-specific retrieval-augmented generation system for questions about Pittsburgh and Carnegie Mellon University. The authors scrape about 1,820 subpages and related documents, construct QA pairs through manual annotation and Mistral-generated examples, and build a RAG pipeline combining BM25 and FAISS retrievers, a MiniLM reranker, and a Mistral 7B generator. On a manually constructed test set of 165 questions, they report a non-RAG baseline F1 of 5.45% and a best RAG F1 of 42.21%, with additional analyses by time sensitivity and question complexity. The central claim is that RAG substantially outperforms the bare LLM, especially for time-sensitive and complex queries.","tokens_in":7178,"tokens_out":5225,"duration_ms":68783,"significance":"The paper is a straightforward empirical case study. If the reported numbers were fully supported, it would provide a useful demonstration that a standard RAG stack with off-the-shelf retrievers and an open 7B model can give large gains over a bare LLM in a narrow geographic and institutional domain. The disaggregated results by time sensitivity and complexity, together with the qualitative error analysis, are potentially informative. However, the significance is tempered by the construction of the test set from the retrieval corpus, the absence of any statistical testing, and a direct inconsistency between the abstract and Table 1 for the headline recall. The paper is transparent about many design choices, and the ablation table is a useful element, but the current evidence does not support the strength of the claims as written.","major_comments":[{"comment":"The abstract and Section 5.3 state that the best configuration achieves a recall of 56.18%, but Table 1 shows that the best configuration (RAG + reranker + few-shot learning + ensembled retriever) has a recall of 45.39%. The 56.18% value belongs to a different row, the configuration without few-shot learning, which has an F1 of only 20.50%. This inconsistency must be corrected in the abstract, in Section 5.3, and in any summary of the results.","section":"Abstract and Section 5.3"},{"comment":"The claim that the best results are \"statistically significant\" is unsupported by any significance test, confidence interval, or error bar. The evaluation uses a single fixed set of 165 questions, so there is no reported measure of variability. Either remove the significance language or support it with an appropriate test, such as a bootstrap over the per-question scores.","section":"Section 5.3"},{"comment":"The 165-question test set is constructed from the same scraped corpus that populates the retrieval index. Section 3 states that the manual QA pairs were created by selecting segments from the scraped resources, and Section 4.2 builds the BM25/FAISS index from the same corpus. This design guarantees that every test question has an answer in the documents available to RAG, while the non-RAG baseline receives no documents. The reported F1 gap therefore reflects, at least in part, the construction of the test set rather than a general advantage of RAG on a representative domain query distribution. The paper should either add an independently authored test set or explicitly narrow the claim to performance on questions derived from the scraped corpus.","section":"Sections 3, 4.2, and 6"}],"minor_comments":[{"comment":"The text says \"As shown in Figure 2\" when referring to the comparison of RAG and non-RAG models across metrics; the relevant display is Table 2, while Figure 2 is the data annotation pipeline.","section":"Section 6.1"},{"comment":"The sentence-transformers embedding models all-MiniLM-L6-v2 and multi-qa-mpnet-base-dot-v1 are cited as Jiang et al. (2023), which is the Mistral 7B citation; these models need to be cited to their actual sources.","section":"Sections 4.1 and 4.2"},{"comment":"The statement that setting top_p = 1 \"means every token is chosen as the token with highest probability\" is imprecise. Top-p sampling with p = 1 is equivalent to no nucleus sampling cutoff, but it does not by itself force greedy decoding; a temperature and seed would be needed for exact reproducibility.","section":"Section 5.1"},{"comment":"The inter-annotator agreement score is computed as the F1 overlap between model and human answers. This is an answer-similarity score rather than a standard inter-annotator agreement measure, and the label should be qualified accordingly.","section":"Section 3.1"},{"comment":"Several typographical and grammatical errors should be corrected, including \"cominbined\", \"an Embedding model\", \"ariations\", and inconsistent hyphenation of \"reranker\" and \"re-ranker\".","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads more like a course project report than a fully developed research paper. The most load-bearing concerns are the overlap between the test set and the retrieval corpus, the unsupported significance claim, and the incorrect recall value in the abstract. These are fixable by reframing the evaluation and correcting the claims, but as written the headline comparison overstates the evidence. The citation errors for the embedding models also suggest the paper needs a careful proofreading pass before it could be considered for publication."},"author_rebuttal":null,"desk_editor":null,"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding retrieval to a bare 7-billion-parameter language model lifts F1 from 5.45% to 42.21% on a local question-answering domain.","keywords":["Retrieval-Augmented Generation","domain-specific question answering","BM25","FAISS","reranking","few-shot learning","time-sensitive queries","inter-annotator agreement"],"falsifier":"Have independent annotators write a fresh test set from real user queries without seeing the scraped corpus, then run the exact same best configuration; if the F1 gap over the bare model shrinks substantially, the reported improvement was an artifact of test-set or corpus construction.","tokens_in":6547,"feed_emoji":"🎯","tokens_out":11349,"duration_ms":102914,"temperature":0.7,"pith_summary":"This paper reports an end-to-end experiment in which a plain 7-billion-parameter language model is turned into a more reliable question answerer for a specific local domain, Pittsburgh and CMU, by adding a retrieval step. The authors claim that the retrieval-augmented system dramatically outperforms the bare model: F1 rises from 5.45% to 42.21%, exact match from 0.00% to 20.25%, and recall reaches 56.18%. The largest gains appear on time-sensitive questions, where the bare model has no up-to-date knowledge, and on long, multi-step questions. The study would matter because it suggests that a standard, off-the-shelf retrieval stack can deliver large accuracy gains for domain-specific question answering without fine-tuning the generator.","feed_headline":"Retrieval lifts local Q&A F1 from 5.45% to 42.21%","feed_subtitle":"With sparse-plus-dense retrieval, reranking, and few-shot prompts, a bare LLM becomes a usable domain answerer.","key_machinery":"The load-bearing mechanism is a retrieval pipeline that takes the user's question, pulls candidate passages from a locally scraped document collection using both BM25 (a lexical keyword-matching retriever) and FAISS (a dense vector-search index), reranks the combined results with a small reranker model, and inserts the top chunks into the prompt of a 7-billion-parameter generator with two worked examples. The documents are chunked recursively at size 1000 with overlap 200, embedded, and indexed for dense search, while the sparse retriever compensates for the dense index's accuracy loss. The argument's force rests on the comparison between this pipeline and the same generator without retrieval, and on ablation runs showing that removing the reranker, the few-shot examples, or the ensembled retrieval lowers F1.","core_discovery":"The central claim is that retrieval augmentation, not model size or fine-tuning, is what makes a general-purpose language model usable for answering questions about a specific city-university domain. On the paper's own numbers, the best configuration—combining a sparse lexical retriever with a dense vector index, reranking the retrieved chunks, and prompting a 7-billion-parameter generator with two worked examples—raises F1 from 5.45% to 42.21%, exact match from 0.00% to 20.25%, precision to 47.29%, and recall to 56.18%. The paper further claims the advantage is concentrated where a static model is weakest: time-sensitive queries (e.g., a specific event date) and long, complex questions requiring multi-step reasoning. It also reports that the hybrid manual-plus-model annotation process reached an inter-annotator agreement of 0.7625 on a 100-question sample.","pith_inferences":["Editorial inference: the reported F1 gap likely overstates real-world benefit, because the manual test questions were written from segments of the scraped corpus, so the correct answer is almost always present in the retrievable documents by construction.","Editorial inference: the same pipeline should transfer to other geographically or institutionally scoped domains, and the error analysis suggests the highest-value adaptation is tuning the reranker and retrieval depth rather than swapping the generator.","Editorial inference: the paper's time-sensitive split hints at a cheap benchmark—build a QA set with known correct-date versions and measure whether retrieved documents actually contain the current version, which would isolate retrieval freshness from generation quality."],"forward_implications":["If the central claim holds, organizations with local, fast-changing knowledge bases can get large QA accuracy gains from retrieval alone, without fine-tuning the generator.","Time-sensitive questions become addressable because retrieval supplies current dates and event details that a static model cannot know.","Long, multi-step questions improve most when the system retrieves and reranks relevant documents, since the generator no longer reasons from memory alone.","Retrieval quality, not generator capacity, is the main performance lever: the paper's error analysis attributes the worst failures to irrelevant or misleading retrieved documents.","The paper's ablations show the three enhancements are complementary: the best configuration needs the ensembled retriever, the reranker, and few-shot prompting together."],"supporting_citations":[{"why":"Supplies the 7-billion-parameter generator and the model used to auto-generate QA pairs, so the whole system depends on its behavior.","marker":"Jiang et al., 2023"},{"why":"Defines BM25, the sparse retriever whose precision compensates for the dense index's recall.","marker":"Robertson and Zaragoza, 2009"},{"why":"Provides FAISS, the dense vector index used to search embedded chunks and the other half of the ensembled retriever.","marker":"Johnson et al., 2017"},{"why":"Supplies the evaluation script and token-overlap F1 metric used for all reported scores, so the headline numbers inherit its normalization choices.","marker":"Rajpurkar et al., 2016"},{"why":"Provides the pipeline framework that wires loading, chunking, retrieval, and generation into one workflow.","marker":"list or maintainers of LangChain, 2023"}],"fun_headline_variants":["RAG lifts F1 from 5.45% to 42.21% for CMU/Pittsburgh Q&A","Sparse+dense retrieval boosts LLM F1 by 8x for local Q&A","Retrieval, not larger models, makes LLMs answer local questions: F1 up 8x","From 5% to 42% F1: Retrieval-augmented LLM answers Pittsburgh/CMU queries","RAG turns a 7B LLM into a useful local Q&A system: F1 42.21%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the 165 manually created test questions fairly represent real queries about the target city and university, and that token-overlap F1 captures answer quality; the paper does not describe a random sampling protocol or verify that the scraped corpus contains the answer to every test question.","fun_headline_variants_meta":{"raw":{"variants":["RAG lifts F1 from 5.45% to 42.21% for CMU/Pittsburgh Q&A","Sparse+dense retrieval boosts LLM F1 by 8x for local Q&A","Retrieval, not larger models, makes LLMs answer local questions: F1 up 8x","From 5% to 42% F1: Retrieval-augmented LLM answers Pittsburgh/CMU queries","RAG turns a 7B LLM into a useful local Q&A system: F1 42.21%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000949,"raw_usage":{"total_tokens":4037,"prompt_tokens":918,"completion_tokens":3119,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":2981}},"tokens_in":534,"tokens_out":3119,"duration_ms":19356,"temperature":1.0,"reasoning_tokens":2981,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:59:30.033016+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent annotators write a fresh test set from real user queries without seeing the scraped corpus, then run the exact same best configuration; if the F1 gap over the bare model shrinks substantially, the reported improvement was an artifact of test-set or corpus construction.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides FAISS, the dense vector index used to search embedded chunks and the other half of the ensembled retriever."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the pipeline framework that wires loading, chunking, retrieval, and generation into one workflow."}],"review_version":1}