Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

A RAG-Based Institutional Assistant

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a RAG assistant for university regulations is limited by retrieval, not generation, with accuracy jumping from 22.04% to 54.02% when the correct document chunk is provided.

desk verdict Useful Portuguese RAG dataset and honest engineering, but the headline accuracy comparison is confounded by subset selection and the judge reliability is shaky. read the letter →

arxiv 2501.13880 v1 pith:GQNJSKUQ submitted 2025-01-23 cs.CL

Gustavo Kuratomi , Paulo Pirozelli , Fabio G. Cozman , Sarajane M. Peres This is my paper · ORCID
classification cs.CL
keywords retrieval-augmentedgenerationquestionansweringinstitutionalassistantPortugueselanguagemodelssemanticsearchBM25baselineGPT-4evaluationuniversityregulations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper builds a retrieval-augmented question-answering assistant for University of São Paulo regulations and runs controlled experiments to find where errors originate. It claims that retrieval, not answer generation, is the bottleneck: the best full system answers 22.04% of questions correctly, but when the correct document chunk is forced into the prompt the same generator scores 54.02%, and with no context it drops to 13.68%. The authors also find that a lexical BM25 retriever outperforms multilingual neural embedding models, especially on original questions, though the gap narrows on paraphrased questions. This matters because it separates a fixable engineering problem, finding the right document, from the harder question of whether current language models can use that document once they have it.

What carries the argument

The central instrument is a three-condition comparison inside one modular RAG pipeline: the system's best retriever feeding the generator, the correct chunk force-fed into the prompt, and no context at all. The accuracy gap between the first and second conditions attributes error to retrieval, while the gap between the second and third attributes error to the value of external knowledge. The retriever is a dot-product similarity search over multilingual Sentence-BERT-style embeddings, with BM25 as a lexical baseline, and the dataset is built so each question has exactly one chunk that contains the answer. Answer quality is scored by a GPT-4 judge on a three-level scale, converted to 0-100.

What would settle it

Manually score GPT-3.5's answers on the 178 questions where the k=5 prompt contained the correct chunk, using the paper's three-level scale, and compare with GPT-4's 51.96% for that setting; if human scores diverge by more than about 15 points, the reported retrieval-bottleneck numbers are an artifact of the judge rather than a measure of answer quality.

Watch

Extended reading notes

Core claim

The paper's central claim is that, in a modular RAG system for Portuguese-language institutional QA, end-to-end accuracy is set by retrieval quality rather than by the generator's ability. With the Mpnet retriever and GPT-3.5, the system scores 22.04% on the GPT-4 judgment scale; when the correct chunk is guaranteed to be among the k=3 provided chunks, GPT-3.5 rises to 54.02%; with k=0, no context, it falls to 13.68%. Table 2 shows BM25 reaching 57% Top-5 accuracy on original questions and 37% on paraphrased questions, while the best neural model reaches 40% and 30% respectively, so the retriever frequently fails to surface the relevant chunk. The paper interprets this as evidence that database access is the dominant factor in RAG performance.

Load-bearing premise

The headline percentages depend on GPT-4's ratings being a valid measure of answer correctness, and the paper's own manual evaluation of Llama-3 scored 88.20% where GPT-4 scored 42.97% on the same subset, so a biased judge would change the conclusion.

Editorial extensions

If this is right

  • Improving the retriever is the highest-leverage next step: better retrieval should raise end-to-end accuracy more than switching generators, since correct-chunk accuracy is roughly 2.4 times full-system accuracy.
  • Current LLMs can handle institutional documents when given the right chunk, so the 54% ceiling is a retrieval ceiling, not proof that generation is the limiting factor.
  • Extra chunks hurt when the right one is already present: GPT-3.5 falls from 54.02% at k=3 to 50.24% at k=8, so systems should avoid indiscriminately increasing context size.
  • BM25's strong showing is partly a dataset artifact, because original questions share vocabulary with source chunks; on paraphrased questions the neural models become competitive, so deployment should expect lexical overlap to be lower.
  • Without external context, all tested models perform much worse, confirming that RAG's value for institutional QA is real and not a modeling detail.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The manual-vs-GPT-4 gap hints that the absolute accuracy numbers are conservative, yet the relative ordering across retrieval conditions could still hold; a fair test would human-score all models, not just Llama-3.
  • Because each question is tied to a single chunk, the retrieval task is simpler than real multi-document questions; real USP queries may need several norms at once, so practical end-to-end accuracy could be lower than 22%.
  • A hybrid retriever combining BM25 and embedding scores could outperform either alone, especially on paraphrased questions where lexical and semantic signals diverge.
  • The paper's reported cosine-similarity scores barely move across conditions, suggesting that embedding similarity is not a reliable answer-quality signal here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper describes a retrieval-augmented generation (RAG) assistant for answering questions about University of São Paulo regulations. The authors curate 866 Portuguese normative documents, chunk them at three sizes, and build a 592-question QA dataset (original and paraphrased versions) using GPT-4. They evaluate five retrievers (four multilingual Sentence-BERT models plus BM25 and a random baseline) by top-1/top-5 accuracy, and four generators (GPT-3.5, Llama-3, Mixtral, Sabiá-2) under varying numbers of retrieved chunks (k = 3, 5, 8), using F1, cosine similarity, and a GPT-4-based qualitative score. The central claim is that the main bottleneck is retrieval: with correct chunks supplied, GPT-3.5's GPT-4-judged accuracy reaches 54.02%, versus 22.04% with the system's best retriever and 13.68% with no context.

Significance. If the quantitative claims were reliable, this would be a useful applied study for Portuguese-language institutional QA, with public code and data. The paper addresses a real deployment scenario and includes both lexical and neural retrievers and several generators. It also attempts to mitigate the artificial lexical overlap of generated questions with a paraphrased dataset. However, the main numeric comparisons are currently confounded by subset selection and by reliance on a single GPT-4 judge whose agreement with human assessment is not established; the manual Llama-3 audit reported in Section 5.2 shows an 88.20% human score versus a 42.97% GPT-4 score on the same questions, a discrepancy that is not resolved. Consequently, the headline result—that retrieval, not generation, is the limiting factor—is plausible but not yet quantitatively supported.

major comments (4)
  1. [Abstract; Section 5.2, Table 3] The headline comparison is confounded by subset selection and differing k. The 54.02% figure is GPT-3.5's score on the 194 questions for which the Mpnet retriever already placed the correct chunk among the three provided chunks, while the 22.04% figure is the score over all 592 questions at k = 8, and the 13.68% no-context figure is also over all 592 questions. These are not comparable quantities: the 194-question subset is selected by retrieval success (which correlates with lexical overlap and question ease), and k differs between the two numbers. The statement in the abstract that "when the correct document chunks are supplied to the LLMs, accuracy significantly improves to 54.02%" is therefore only supported for a nonrandom subset, not as an estimate of oracle retrieval on the full dataset. The authors should report oracle-retrieval results on a fixed question set (e.g., all 592 questions with the correct chunk included, or a matched random subset) and compare against the same question set at the same k.
  2. [Section 5.2, Table 3 and manual evaluation paragraph] The reliability of the GPT-4 judge is load-bearing and is contradicted by the paper's own manual audit. For Llama-3 on the k = 5 correct-chunk subset, the GPT-4 judge assigns 42.97% while the manual evaluation using the same criteria gives 88.20%. The paper does not explain this large gap or calibrate the GPT-4 judge against human judgments for the other models, particularly GPT-3.5, whose scores drive the central comparison. If the judge is systematically biased (e.g., toward longer or differently worded answers), then the quantitative claims (54.02%, 22.04%, 13.68%, and the reported gaps) lose support. The authors should provide human evaluation on a sample for each model and k, report inter-annotator or judge-human agreement, and, if the gap persists, reconsider using the GPT-4 score as the primary metric.
  3. [Section 5.1 and Section 5.2] The choice of Mpnet as the retriever for the generation experiments is not justified, and the inconsistency between retrieval and generation question sets obscures the bottleneck claim. In Table 2, BM25 outperforms Mpnet on the original questions at the 2K chunk size (top-5 0.51 vs. 0.36), yet BM25 is never used in the full RAG pipeline; the paper says Mpnet "achieved the highest score in the 2K database" without clarifying that this is only among neural embedders. Furthermore, the abstract's "optimal retriever model ... Top-5 accuracy of 30%" corresponds to Mpnet on the paraphrased questions, while the generation numbers in Table 3 are presumably from the original questions, but Section 5.2 never states which question set was used. This makes it impossible to connect retrieval performance to downstream generation accuracy. The authors should test generation with BM25 as the retriever, state explicitly whether original or paraphrased questions are used in generation, and report the relevant retrieval recall for the actual question set at the chosen k.
  4. [Section 5.2, Table 3] No confidence intervals or significance tests are reported for any of the F1, cosine, or LLM-score comparisons. Given the small sizes of the correct-chunk subsets (194, 178, and 204 for k = 3, 5, 8) and the apparent differences between models and k values, the reader cannot tell whether the ordering of models (e.g., GPT-3.5 vs. Mixtral at k = 8) is reliable. At minimum, the authors should report bootstrap confidence intervals or a statistical test for the main comparisons.
minor comments (6)
  1. [Section 1, Introduction] There is a typo in the first paragraph: "due to to these models' impressive in-context learning abilities" should read "due to these models' impressive in-context learning abilities."
  2. [Tables 1, 2, 3 and Figures 1, 3, 4] Several captions and labels use Portuguese ("Tabela", "Figura", "A vg.") while the text is in English; these should be translated to "Table", "Figure", and "Avg." for consistency.
  3. [Section 5.2, final paragraph] The sentence "The results of these evaluations are shown in the bottom rows of Tabl 3" contains a truncated "Tabl" and should be "Table 3." Also "completly" in the definition of "Totally correct" should be "completely."
  4. [Section 3, Dataset] The paper mentions that all questions were manually reviewed to ensure no two chunks could answer a single question, but it does not describe the review procedure or report inter-annotator agreement. A sentence on how many reviewers were involved and how conflicts were resolved would increase confidence in the dataset quality.
  5. [Section 5.2, Figure 5] Figure 5 is referenced as a correlation matrix between metrics, but the figure does not appear in the text provided and the correlations are not discussed numerically. Please add the figure and describe the key correlations, especially between the GPT-4 score and the human assessment.
  6. [Section 5.2] The prompt used for the GPT-4 judge is not given, nor is the exact scoring rubric (beyond the three labels). Providing the full prompt would improve reproducibility and would also help the reader assess possible judge bias.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical evaluation whose headline numbers are measurements, not derivations or predictions forced by construction.

full rationale

The paper does not derive any quantity from an equation or from a fitted parameter; all reported numbers are measured outcomes of a retrieval and generation pipeline. The GPT-4 dataset-generation and GPT-4 judging loop (Sections 3 and 5.2) is a shared-bias validity concern, not a circular reduction: the 54.02% figure is not equal to the judge's prompt or to the dataset by construction, and the same judge is applied to the with-context and no-context arms. The 'correct document chunks supplied' comparison (Table 3, middle block) is conditioned on retrieval success and uses different k and different question subsets, so the abstract's 'increase of over 30 percentage points' is statistically confounded; however, this is a validity and selection-bias issue, not an equivalence-by-definition. No self-citations are load-bearing; no uniqueness theorem or ansatz is imported from the authors' prior work. The paper even acknowledges the lexical-overlap artifact in its original questions and constructs a paraphrased set to address it (Section 3). The limitations section (Section 8) calls for expert evaluation and real-user questions, which further shows the authors treat the GPT-4 judge as a proxy rather than a definitional ground truth. Consequently, there is no circular step to quote and exhibit; the correct score is 0 on the circularity scale.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the QA dataset and the evaluation protocol; both depend on GPT-4 generation and judgment, plus design choices for chunking and retrieval.

free parameters (4)
  • chunk_size = 2000, 4000, and 8000 characters
    Chunk size is a design choice varied across experiments; retrieval accuracy varies strongly with it, and the generation stage uses the 2000-character database.
  • chunk_overlap_ratio = 1:10 (200 characters for 2K chunks)
    Chosen to reduce information loss at chunk boundaries; no sensitivity analysis is reported.
  • number_of_retrieved_chunks_k = 3, 5, and 8
    Used as a hyperparameter in the generation experiment; performance changes slightly as k increases.
  • generation_retriever_choice = Mpnet-base-v2 with 2K chunks
    The authors fixed this retriever for all generation experiments because it scored highest in the 2K database; this choice affects the measured end-to-end performance.
assumptions (4)
  • domain assumption The GPT-4-generated questions, answers, and paraphrases are correct, self-contained, and representative of real user queries.
    Every retrieval and generation metric is computed against this dataset (Section 3); the paraphrased set is not manually reviewed, and the questions share vocabulary with the source documents.
  • domain assumption Each question has exactly one answerable chunk, so top-k accuracy is unambiguous.
    The dataset is constructed to satisfy this by manual review of original questions, but real institutional queries are often multi-hop and span multiple chunks, as the Limitations section acknowledges.
  • domain assumption GPT-4's automated ratings are a valid measure of answer correctness.
    The headline generation scores depend on GPT-4 as judge (Section 5.2); the reported manual audit of Llama-3 k=5 gives 88.20% versus GPT-4's 42.97%, so this assumption is suspect.
  • domain assumption Dot-product similarity in multilingual Sentence-BERT embedding space is a valid relevance measure for Portuguese normative documents.
    Embedding retrieval is the core of the retriever module (Section 4), and BM25 outperforms it on original questions, indicating the semantic similarity assumption is weak for this corpus.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A RAG-Based Institutional Assistant." pith.science (2026). https://pith.science/paper/GQNJSKUQ

@misc{pith2026250113880,
  author       = {Pith},
  title        = {Pith review of: A RAG-Based Institutional Assistant},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GQNJSKUQ}},
  note         = {Machine review of arXiv:2501.13880}
}
read the original abstract

Although large language models (LLMs) demonstrate strong text generation capabilities, they struggle in scenarios requiring access to structured knowledge bases or specific documents, limiting their effectiveness in knowledge-intensive tasks. To address this limitation, retrieval-augmented generation (RAG) models have been developed, enabling generative models to incorporate relevant document fragments into their inputs. In this paper, we design and evaluate a RAG-based virtual assistant specifically tailored for the University of S\~ao Paulo. Our system architecture comprises two key modules: a retriever and a generative model. We experiment with different types of models for both components, adjusting hyperparameters such as chunk size and the number of retrieved documents. Our optimal retriever model achieves a Top-5 accuracy of 30%, while our most effective generative model scores 22.04\% against ground truth answers. Notably, when the correct document chunks are supplied to the LLMs, accuracy significantly improves to 54.02%, an increase of over 30 percentage points. Conversely, without contextual input, performance declines to 13.68%. These findings highlight the critical role of database access in enhancing LLM performance. They also reveal the limitations of current semantic search methods in accurately identifying relevant documents and underscore the ongoing challenges LLMs face in generating precise responses.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. An Agile Method for Implementing Retrieval Augmented Generation Tools in Industrial SMEs

    cs.CL 2025-08 conditional novelty 6.0 of 10

    EASI-RAG is a structured agile method for deploying RAG tools in industrial SMEs, validated by one case study where a no-experience team built a working assistant in three weeks.

  2. AfriEconQA: A Benchmark for Quantitative and Temporal Reasoning over World Bank Economic Reports

    cs.CL 2026-01 reject novelty 4.0 of 10

    An LLM-generated QA benchmark over World Bank African economic reports is evaluated on a small sample and found hard, but the abstract and body report conflicting dataset sizes, model names, and scores.

Reference graph

Works this paper leans on

24 extracted references · 14 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

  2. [2]

    T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M

    Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M. T., and Zhang, Y. (2023). Sparks of artificial general intelligence: Early experiments with gpt-4

    work page 2023

  3. [3]

    Chen, W., Zha, H., Chen, Z., Xiong, W., Wang, H., and Wang, W. Y. (2020). H ybrid QA : A dataset of multi-hop question answering over tabular and textual data. In Cohn, T., He, Y., and Liu, Y., editors, Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 1026--1036, Online. Association for Computational Linguistics

    work page 2020

  4. [4]

    Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., and Wang, H. (2024). Retrieval-augmented generation for large language models: A survey

    work page 2024

  5. [5]

    ILIN, I. (2023). Advanced rag techniques: an illustrated overview

    work page 2023

  6. [6]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. (2023). Mistral 7b

    2023

  7. [7]

    Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and tau Yih, W. (2020). Dense passage retrieval for open-domain question answering

    work page 2020

  8. [8]

    u ttler, H., Lewis, M., Yih, W., Rockt \

    Lewis, P. S. H., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K \" u ttler, H., Lewis, M., Yih, W., Rockt \" a schel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. CoRR , abs/2005.11401

    arXiv 2020

Show all 24 references
  1. [9]

    D., Yue, X., and Chen, W

    Li, T., Zhang, G., Do, Q. D., Yue, X., and Chen, W. (2024). Long-context llms struggle with long in-context learning. arXiv preprint arXiv:2404.02060

    2024 arXiv

  2. [10]

    F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P

    Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics , 12:157--173

    2024

  3. [11]

    Ma, X., Gong, Y., He, P., Zhao, H., and Duan, N. (2023). Query rewriting for retrieval-augmented large language models

    2023

  4. [12]

    Nguyen, T., Rosenberg, M., Song, X., Gao, J., Tiwary, S., Majumder, R., and Deng, L. (2016). MS MARCO: A human generated machine reading comprehension dataset. CoRR , abs/1611.09268

    2016 arXiv

  5. [13]

    Gpt-4 technical report

    OpenAI (2024). Gpt-4 technical report

    2024

  6. [14]

    Rajpurkar, P., Jia, R., and Liang, P. (2018). Know what you don't know: Unanswerable questions for squad. CoRR , abs/1806.03822

    2018 arXiv

  7. [15]

    Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100, 000+ questions for machine comprehension of text. CoRR , abs/1606.05250

    2016 arXiv

  8. [16]

    Rawte, V., Sheth, A., and Das, A. (2023). A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922

    2023 arXiv

  9. [17]

    and Gurevych, I

    Reimers, N. and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks

    2019

  10. [18]

    Robertson, S., Zaragoza, H., et al. (2009). The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3(4):333--389

    2009

  11. [19]

    Sales Almeida, T., Abonizio, H., Nogueira, R., and Pires, R. (2024). Sabi \'a -2: A new generation of portuguese large language models

    2024

  12. [20]

    Shao, Z., Gong, Y., Shen, Y., Huang, M., Duan, N., and Chen, W. (2023). Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy

    2023

  13. [21]

    Team, G. (2024). Gemini: A family of highly capable multimodal models

    2024

  14. [22]

    and et al

    Touvron, H. and et al. (2023). Llama 2: Open foundation and fine-tuned chat models

    2023

  15. [23]

    Wang, X., Yang, Q., Qiu, Y., Liang, J., He, Q., Gu, Z., Xiao, Y., and Wang, W. (2023). Knowledgpt: Enhancing large language models with retrieval and storage access on knowledge bases

    2023

  16. [24]

    Zhuang, S., Liu, B., Koopman, B., and Zuccon, G. (2023). Open-source large language models are strong zero-shot query likelihood models for document ranking

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.