{"id":"75a1cd37-399c-4c91-98a3-52b01aa8d864","arxiv_id":"2501.13880","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A RAG assistant for University of São Paulo documents lifts LLM answer accuracy from 13.68% (no context) to 54.02% (oracle context), but the best retriever reaches only 30% Top-5 accuracy, making retrieval the limiting factor.","lead":"The paper builds and tests a question-answering assistant for University of São Paulo rules, using retrieval-augmented generation (RAG). It finds retrieval is the bottleneck: providing the right document passage lifts GPT-3.5's judged accuracy from 22% to 54%, while the search step itself finds that passage only 30% of the time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 54.02% vs 22.04% comparison compares a 194-question retrieval-success subset with all 592 questions at different k, so the claimed retrieval bottleneck is not yet demonstrated.","rationale":"The paper builds a useful RAG system, curates a Portuguese QA dataset, and provides a modular evaluation of retrievers and generators, which are real contributions. However, the headline quantitative claim that retrieval is the main bottleneck rests on a comparison that mixes conditions: 54.02% is measured only on the subset of questions where the retriever already succeeded, while 22.04% is measured on all questions with a different k. The correct way to estimate the upper bound of the RAG pipeline is to supply the correct chunk to every question, not to condition on retrieval success. The reader's concern about GPT-4 judge validity is also legitimate, and the reported manual audit discrepancy (88.20% manual vs 42.97% GPT-4 for Llama-3 on the same subset) reinforces the need for caution. But the subset-confound issue is more basic: it affects the interpretation even under the assumption that the judge is perfect. The finding is fixable by re-running the generation experiment with oracle chunks on the full dataset, or at least by reporting the all-question accuracy on the same k and comparing within the same subset. Because the paper's code and data are available and the reanalysis is straightforward, the appropriate verdict remains CONDITIONAL, matching the reader's assessment. I disagree, however, that judge validity is the single weakest assumption; the subset comparison is the more load-bearing flaw.","tokens_in":8694,"tokens_out":4403,"duration_ms":38573,"concrete_test":"Recompute the oracle condition by supplying the correct chunk to GPT-3.5 for all 592 questions (or a random sample) at k=3 and k=8, using the same GPT-4 judge, and compare the resulting full-oracle scores with the 22.04% and 13.68% all-question baselines. If the full-oracle score is close to 54.02%, the subset-selection concern is minor; if it is substantially lower (e.g., below 40%), the paper's central 30-point gap is an artifact of comparing a retrieval-success subset to the full set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central comparison is confounded by subset selection and differing k. In Table 3, the 54.02% figure is GPT-3.5's GPT-4-judged score for k=3 restricted to the 194 questions where the Mpnet retriever had already placed the correct chunk in context. The 22.04% figure is GPT-3.5's score for k=8 over all 592 questions in the top block, and the 13.68% no-context figure is also over all 592 questions. Filtering on retrieval success selects questions whose chunks are easier to retrieve (often those with higher lexical overlap with the query), and the k values differ between the two numbers. Therefore, the statement that 'when the correct document chunks are supplied ... accuracy significantly improves to 54.02%' is only supported for a nonrandom subset, not as an estimate of oracle retrieval on the full dataset. Even a perfectly valid GPT-4 judge would not fix this mismatch because the compared quantities are not drawn from the same question set or the same k.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a retrieval-augmented generation (RAG) assistant for answering questions about University of São Paulo regulations. The authors curate 866 Portuguese normative documents, chunk them at three sizes, and build a 592-question QA dataset (original and paraphrased versions) using GPT-4. They evaluate five retrievers (four multilingual Sentence-BERT models plus BM25 and a random baseline) by top-1/top-5 accuracy, and four generators (GPT-3.5, Llama-3, Mixtral, Sabiá-2) under varying numbers of retrieved chunks (k = 3, 5, 8), using F1, cosine similarity, and a GPT-4-based qualitative score. The central claim is that the main bottleneck is retrieval: with correct chunks supplied, GPT-3.5's GPT-4-judged accuracy reaches 54.02%, versus 22.04% with the system's best retriever and 13.68% with no context.","tokens_in":8933,"tokens_out":4569,"duration_ms":38212,"significance":"If the quantitative claims were reliable, this would be a useful applied study for Portuguese-language institutional QA, with public code and data. The paper addresses a real deployment scenario and includes both lexical and neural retrievers and several generators. It also attempts to mitigate the artificial lexical overlap of generated questions with a paraphrased dataset. However, the main numeric comparisons are currently confounded by subset selection and by reliance on a single GPT-4 judge whose agreement with human assessment is not established; the manual Llama-3 audit reported in Section 5.2 shows an 88.20% human score versus a 42.97% GPT-4 score on the same questions, a discrepancy that is not resolved. Consequently, the headline result—that retrieval, not generation, is the limiting factor—is plausible but not yet quantitatively supported.","major_comments":[{"comment":"The headline comparison is confounded by subset selection and differing k. The 54.02% figure is GPT-3.5's score on the 194 questions for which the Mpnet retriever already placed the correct chunk among the three provided chunks, while the 22.04% figure is the score over all 592 questions at k = 8, and the 13.68% no-context figure is also over all 592 questions. These are not comparable quantities: the 194-question subset is selected by retrieval success (which correlates with lexical overlap and question ease), and k differs between the two numbers. The statement in the abstract that \"when the correct document chunks are supplied to the LLMs, accuracy significantly improves to 54.02%\" is therefore only supported for a nonrandom subset, not as an estimate of oracle retrieval on the full dataset. The authors should report oracle-retrieval results on a fixed question set (e.g., all 592 questions with the correct chunk included, or a matched random subset) and compare against the same question set at the same k.","section":"Abstract; Section 5.2, Table 3"},{"comment":"The reliability of the GPT-4 judge is load-bearing and is contradicted by the paper's own manual audit. For Llama-3 on the k = 5 correct-chunk subset, the GPT-4 judge assigns 42.97% while the manual evaluation using the same criteria gives 88.20%. The paper does not explain this large gap or calibrate the GPT-4 judge against human judgments for the other models, particularly GPT-3.5, whose scores drive the central comparison. If the judge is systematically biased (e.g., toward longer or differently worded answers), then the quantitative claims (54.02%, 22.04%, 13.68%, and the reported gaps) lose support. The authors should provide human evaluation on a sample for each model and k, report inter-annotator or judge-human agreement, and, if the gap persists, reconsider using the GPT-4 score as the primary metric.","section":"Section 5.2, Table 3 and manual evaluation paragraph"},{"comment":"The choice of Mpnet as the retriever for the generation experiments is not justified, and the inconsistency between retrieval and generation question sets obscures the bottleneck claim. In Table 2, BM25 outperforms Mpnet on the original questions at the 2K chunk size (top-5 0.51 vs. 0.36), yet BM25 is never used in the full RAG pipeline; the paper says Mpnet \"achieved the highest score in the 2K database\" without clarifying that this is only among neural embedders. Furthermore, the abstract's \"optimal retriever model ... Top-5 accuracy of 30%\" corresponds to Mpnet on the paraphrased questions, while the generation numbers in Table 3 are presumably from the original questions, but Section 5.2 never states which question set was used. This makes it impossible to connect retrieval performance to downstream generation accuracy. The authors should test generation with BM25 as the retriever, state explicitly whether original or paraphrased questions are used in generation, and report the relevant retrieval recall for the actual question set at the chosen k.","section":"Section 5.1 and Section 5.2"},{"comment":"No confidence intervals or significance tests are reported for any of the F1, cosine, or LLM-score comparisons. Given the small sizes of the correct-chunk subsets (194, 178, and 204 for k = 3, 5, 8) and the apparent differences between models and k values, the reader cannot tell whether the ordering of models (e.g., GPT-3.5 vs. Mixtral at k = 8) is reliable. At minimum, the authors should report bootstrap confidence intervals or a statistical test for the main comparisons.","section":"Section 5.2, Table 3"}],"minor_comments":[{"comment":"There is a typo in the first paragraph: \"due to to these models' impressive in-context learning abilities\" should read \"due to these models' impressive in-context learning abilities.\"","section":"Section 1, Introduction"},{"comment":"Several captions and labels use Portuguese (\"Tabela\", \"Figura\", \"A vg.\") while the text is in English; these should be translated to \"Table\", \"Figure\", and \"Avg.\" for consistency.","section":"Tables 1, 2, 3 and Figures 1, 3, 4"},{"comment":"The sentence \"The results of these evaluations are shown in the bottom rows of Tabl 3\" contains a truncated \"Tabl\" and should be \"Table 3.\" Also \"completly\" in the definition of \"Totally correct\" should be \"completely.\"","section":"Section 5.2, final paragraph"},{"comment":"The paper mentions that all questions were manually reviewed to ensure no two chunks could answer a single question, but it does not describe the review procedure or report inter-annotator agreement. A sentence on how many reviewers were involved and how conflicts were resolved would increase confidence in the dataset quality.","section":"Section 3, Dataset"},{"comment":"Figure 5 is referenced as a correlation matrix between metrics, but the figure does not appear in the text provided and the correlations are not discussed numerically. Please add the figure and describe the key correlations, especially between the GPT-4 score and the human assessment.","section":"Section 5.2, Figure 5"},{"comment":"The prompt used for the GPT-4 judge is not given, nor is the exact scoring rubric (beyond the three labels). Providing the full prompt would improve reproducibility and would also help the reader assess possible judge bias.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful empirical contribution and releases code/data, but the main claim rests on a comparison that mixes different question subsets and k values, and on a single GPT-4 judge that the paper's own manual audit contradicts. I would like to see the authors reanalyze the oracle-retrieval results on a matched question set and provide human evaluation for at least GPT-3.5 across k values. The circularity of using GPT-4 both to generate questions and to judge answers is a concern, though it is partially mitigated by the no-context and with-context comparisons; a human-calibrated judge would address this more directly. The paper is not ready for acceptance in its current form, but the issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper gives you a real thing: a new QA dataset built from USP normative documents, with code and data on GitHub, and a clean comparison of BM25 against multilingual embedding retrievers on original versus paraphrased questions. The finding that BM25 wins on original questions but loses its edge on paraphrased ones is worth knowing. I also credit them for running a manual evaluation and reporting the discrepancy with the GPT-4 judge, even though they don't resolve it.\n\nThe soft spot is the load-bearing claim. The abstract says supplying correct chunks lifts accuracy to 54.02%, versus 22.04% with the best retriever and 13.68% without context. But those numbers are not comparable. The 54.02% is computed only on the 194 questions where the retriever had already placed the correct chunk in the top 3, while the 22.04% and 13.68% are over all 592 questions at k=8 and k=0. Filtering on retrieval success selects easier questions, so the 30-point gap is not an estimate of what oracle retrieval would do on the full set. Even a perfect judge wouldn't fix this mismatch.\n\nOn the judge: the manual audit gave Llama-3 an 88.20% on the k=5 correct-chunk subset, while GPT-4 judged the same setting at 42.97%. That is a huge gap and it undermines confidence in the absolute LLM scores. The authors mention it but don't calibrate the judge or offer an explanation.\n\nAll that said, the qualitative conclusion that retrieval limits end-to-end performance is plausible and probably true. The paper just doesn't demonstrate it with the reported numbers. The dataset and the retrieval comparisons are the real contribution, and they stand on their own. Also, the lack of confidence intervals on small subsets (194 answers) makes the ranking across models fragile.\n\nThis is a paper for practitioners building RAG assistants in Portuguese or for researchers who want a benchmark in a non-English institutional domain. It deserves a serious referee, but the revision needs to be heavy: redo the analysis on matched question sets, report error bars, and validate the judge properly. As is, I wouldn't cite the headline numbers, but I might cite the dataset after it's cleaned up.\n\nIf I were the editor, I'd send it out. The flaws are fixable, and the resource is genuinely useful.","headline":"Useful Portuguese RAG dataset and honest engineering, but the headline accuracy comparison is confounded by subset selection and the judge reliability is shaky.","tokens_in":9437,"tokens_out":1693,"would_cite":false,"duration_ms":17316,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a RAG assistant for university regulations is limited by retrieval, not generation, with accuracy jumping from 22.04% to 54.02% when the correct document chunk is provided.","keywords":["retrieval-augmented generation","question answering","institutional assistant","Portuguese language models","semantic search","BM25 baseline","GPT-4 evaluation","university regulations"],"falsifier":"Manually score GPT-3.5's answers on the 178 questions where the k=5 prompt contained the correct chunk, using the paper's three-level scale, and compare with GPT-4's 51.96% for that setting; if human scores diverge by more than about 15 points, the reported retrieval-bottleneck numbers are an artifact of the judge rather than a measure of answer quality.","tokens_in":8527,"feed_emoji":"🔎","tokens_out":7877,"duration_ms":62075,"temperature":0.7,"pith_summary":"The paper builds a retrieval-augmented question-answering assistant for University of São Paulo regulations and runs controlled experiments to find where errors originate. It claims that retrieval, not answer generation, is the bottleneck: the best full system answers 22.04% of questions correctly, but when the correct document chunk is forced into the prompt the same generator scores 54.02%, and with no context it drops to 13.68%. The authors also find that a lexical BM25 retriever outperforms multilingual neural embedding models, especially on original questions, though the gap narrows on paraphrased questions. This matters because it separates a fixable engineering problem, finding the right document, from the harder question of whether current language models can use that document once they have it.","feed_headline":"Retrieval, not generation, is the RAG bottleneck","feed_subtitle":"Supplying the right document chunks lifts correct answers from 22% to 54%; no context drops it to 13.7%.","key_machinery":"The central instrument is a three-condition comparison inside one modular RAG pipeline: the system's best retriever feeding the generator, the correct chunk force-fed into the prompt, and no context at all. The accuracy gap between the first and second conditions attributes error to retrieval, while the gap between the second and third attributes error to the value of external knowledge. The retriever is a dot-product similarity search over multilingual Sentence-BERT-style embeddings, with BM25 as a lexical baseline, and the dataset is built so each question has exactly one chunk that contains the answer. Answer quality is scored by a GPT-4 judge on a three-level scale, converted to 0-100.","core_discovery":"The paper's central claim is that, in a modular RAG system for Portuguese-language institutional QA, end-to-end accuracy is set by retrieval quality rather than by the generator's ability. With the Mpnet retriever and GPT-3.5, the system scores 22.04% on the GPT-4 judgment scale; when the correct chunk is guaranteed to be among the k=3 provided chunks, GPT-3.5 rises to 54.02%; with k=0, no context, it falls to 13.68%. Table 2 shows BM25 reaching 57% Top-5 accuracy on original questions and 37% on paraphrased questions, while the best neural model reaches 40% and 30% respectively, so the retriever frequently fails to surface the relevant chunk. The paper interprets this as evidence that database access is the dominant factor in RAG performance.","pith_inferences":["The manual-vs-GPT-4 gap hints that the absolute accuracy numbers are conservative, yet the relative ordering across retrieval conditions could still hold; a fair test would human-score all models, not just Llama-3.","Because each question is tied to a single chunk, the retrieval task is simpler than real multi-document questions; real USP queries may need several norms at once, so practical end-to-end accuracy could be lower than 22%.","A hybrid retriever combining BM25 and embedding scores could outperform either alone, especially on paraphrased questions where lexical and semantic signals diverge.","The paper's reported cosine-similarity scores barely move across conditions, suggesting that embedding similarity is not a reliable answer-quality signal here."],"forward_implications":["Improving the retriever is the highest-leverage next step: better retrieval should raise end-to-end accuracy more than switching generators, since correct-chunk accuracy is roughly 2.4 times full-system accuracy.","Current LLMs can handle institutional documents when given the right chunk, so the 54% ceiling is a retrieval ceiling, not proof that generation is the limiting factor.","Extra chunks hurt when the right one is already present: GPT-3.5 falls from 54.02% at k=3 to 50.24% at k=8, so systems should avoid indiscriminately increasing context size.","BM25's strong showing is partly a dataset artifact, because original questions share vocabulary with source chunks; on paraphrased questions the neural models become competitive, so deployment should expect lexical overlap to be lower.","Without external context, all tested models perform much worse, confirming that RAG's value for institutional QA is real and not a modeling detail."],"supporting_citations":[{"why":"Defines the RAG architecture that the paper builds and evaluates.","marker":"Lewis et al. 2020"},{"why":"Supplies the dense-passage dot-product retrieval method the system follows.","marker":"Karpukhin et al. 2020"},{"why":"Provides Sentence-BERT, the basis of all tested embedding models.","marker":"Reimers and Gurevych 2019"},{"why":"Defines BM25, the lexical baseline that outperformed neural retrievers.","marker":"Robertson et al. 2009"},{"why":"GPT-4 is used to generate the QA dataset and to judge answer quality.","marker":"OpenAI 2024"},{"why":"Motivates restricting chunk count and size because LLMs are not robust to long contexts.","marker":"Liu et al. 2024"}],"fun_headline_variants":["RAG's weak link: finding the right document, not generating text","Correct context triples RAG accuracy, but retrieval often misses it","Without the right docs, RAG drops to 14%; with them, 54%","Retrieval quality, not LLM size, decides RAG success"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline percentages depend on GPT-4's ratings being a valid measure of answer correctness, and the paper's own manual evaluation of Llama-3 scored 88.20% where GPT-4 scored 42.97% on the same subset, so a biased judge would change the conclusion.","fun_headline_variants_meta":{"raw":{"variants":["RAG's weak link: finding the right document, not generating text","Correct context triples RAG accuracy, but retrieval often misses it","Without the right docs, RAG drops to 14%; with them, 54%","Retrieval quality, not LLM size, decides RAG success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000623,"raw_usage":{"total_tokens":2887,"prompt_tokens":949,"completion_tokens":1938,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":1856}},"tokens_in":565,"tokens_out":1938,"duration_ms":11964,"temperature":1.0,"reasoning_tokens":1856,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:30:28.767434+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually score GPT-3.5's answers on the 178 questions where the k=5 prompt contained the correct chunk, using the paper's three-level scale, and compare with GPT-4's 51.96% for that setting; if human scores diverge by more than about 15 points, the reported retrieval-bottleneck numbers are an artifact of the judge rather than a measure of answer quality.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the dense-passage dot-product retrieval method the system follows."},{"cited_title":"and Gurevych, I","cited_arxiv_id":null,"evidence_quote":"Provides Sentence-BERT, the basis of all tested embedding models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines BM25, the lexical baseline that outperformed neural retrievers."},{"cited_title":"F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P","cited_arxiv_id":null,"evidence_quote":"Motivates restricting chunk count and size because LLMs are not robust to long contexts."}],"review_version":1}