Towards Optimizing a Retrieval Augmented Generation using Large Language Model on Academic Data

Andrei Staradubets; Anum Afzal; Florian Matthes; Gentrit Fazlija; Juraj Vladika

REVIEW 5 major objections 5 minor 31 references

Towards Optimizing a Retrieval Augmented Generation using Large Language Model on Academic Data

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Retrieval-augmented generation expands each question into several LLM-generated variants before searching, and this multi-query step consistently lifts the fraction of queries that retrieve the correct study program and topic across five…

desk verdict A modest, readable RAG benchmark with a credible directional multi-query result; the tiny samples and GPT-4-everywhere setup keep it from being more than that. read the letter →

arxiv 2411.08438 v1 pith:6S5MQX62 submitted 2024-11-13 cs.AI

Anum Afzal , Juraj Vladika , Gentrit Fazlija , Andrei Staradubets , Florian Matthes This is my paper · ORCID

classification cs.AI

keywords retrievalaugmentedgenerationmulti-queryexpansionquestionansweringacademicdomainLLMevaluationRAGconfusionmatrixhitratedomain-specificcorpus

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether cheap, modular optimizations can make a retrieval-augmented generation (RAG) system reliably answer questions about university study programs, a domain where off-the-shelf LLM knowledge is thin. It claims that adding a Multi-Query step to the retrieval phase—expanding each student question into several LLM-generated paraphrases before searching—substantially improves the hit rate, defined as the fraction of queries for which the correct study program and topic appear in the top-five retrieved documents. The improvement appears across all five tested LLMs (Llama 2 7B and 13B, Mistral 7B, GPT-3.5, GPT-4) and in both English and German, with GPT-4's English hit rate rising from 65.4% to 75.3%. The paper also introduces a RAG Confusion Matrix to separate retrieval errors from generation errors, and contributes the CurriculumQA dataset of 200 curated question-answer pairs.

What carries the argument

The Multi-Query module: a pre-retrieval step in which GPT-4 rewrites the user's question into several semantically similar variants; the retriever then searches a vector store with each variant and merges the results, which is what the paper credits for the hit-rate gains. The evaluation setup is carried by two additional inventions: a two-stage LLM-based filter that first identifies the study program and then the topic within that program, and the RAG Confusion Matrix, which classifies each answer by whether the right document was retrieved and whether the generated answer was judged acceptable, separating retrieval failures from generation failures.

What would settle it

Re-run the retrieval comparison on a freshly written evaluation set where students—without GPT-4 involvement—write the questions and mark the correct study program and topic, then check whether multi-query still beats single-query retrieval by roughly the same margin; if the margin shrinks substantially, the original gain is partly an artifact of the GPT-4-generated labels.

Watch

Extended reading notes

Core claim

The central claim is that multi-query retrieval is the single optimization with the clearest payoff: whenever the Multi-Query module is included, hit rate rises relative to the same configuration without it, and configurations without it perform 'significantly worse.' The module works by keeping the identified study program fixed and generating several rephrasings of the user's question via GPT-4, then retrieving for each variant. The paper reports the largest gains for GPT-4 (English hit rate 65.43% to 75.31%) and consistent gains for open-source models, though their absolute performance stays lower, especially in German. A second, supporting claim is that the Ensemble Retriever (BM25 plus cosine similarity) produces the most faithful generations when paired with multi-query. Finally, the paper argues that the RAG Confusion Matrix—classifying outcomes by whether the correct document was retrieved and whether the generated answer passed a quality threshold—can decouple retriever quality from generator quality, something a single retrieval or generation score cannot do.

Load-bearing premise

The load-bearing premise is that the 200-question evaluation set with its (study program, topic) relevance labels is an unbiased measure of retrieval quality; the questions and labels were generated by GPT-4 and then manually corrected, so if the labels favor phrasing that GPT-4's multi-query rewrites naturally match, the observed hit-rate gain could be partly a label artifact rather than a true retrieval improvement.

Editorial extensions

If this is right

RAG deployments on specialized corpora should try multi-query expansion before more invasive retrieval overhauls, since it is a drop-in addition to the retrieval phase with consistent hit-rate gains.
Combining Multi-Query with the Ensemble Retriever yields the highest faithfulness scores, indicating that query diversity and hybrid lexical/semantic retrieval address different retrieval weaknesses.
The RAG Confusion Matrix can be used to attribute failures: a high false-positive rate means the generator produces acceptable answers from wrong context, pointing to a retriever problem, not a generation problem.
Open-source Llama 2 13B with multi-query approaches GPT-3.5's retrieval hit rate, suggesting that model size matters less than the query expansion step for this task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the Multi-Query paraphrases are always generated by GPT-4, the measured gain may partly depend on GPT-4's paraphrase quality; a direct comparison using a small open-source model or fixed templates as the query generator would test whether the benefit is inherent to multi-query or specific to GPT-4.
The hit-rate label requires both the correct study program and the correct topic to appear in the top-five documents; multi-query may be improving topic coverage more than program identification, which an ablation that scores the two components separately would reveal.
The evaluation set's 200 QA pairs were produced semi-automatically by GPT-4 and then human-corrected; if the paraphrase style of GPT-4's questions aligns with GPT-4's own multi-query rewrites, part of the advantage could be a label-phrasing artifact, which would be checked by re-evaluating on independently human-written questions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Rough take: this is a useful, narrow empirical paper. If you work on RAG pipelines, the comparison across five LLMs, two languages, and four standard optimizations is easy to read, and the multi-query result is consistent in direction everywhere. That consistency is the paper's real asset.

What's new: the CurriculumQA dataset (72 scraped study programs, 200 curated QA pairs, English/German) and the specific benchmark numbers. The RAG confusion matrix is a relabeling of the standard one with a threshold, so I wouldn't call it novel, but it's a reasonable way to separate retrieval and generation failures. The pre-retrieval LLM filtering step (predict study program and topic) is also not entirely new, but its positive effect is worth noting.

Soft spots: the samples are small—82 for the hit-rate table, 20 for the LLM evaluation, 30 for the human correlation—and there are no error bars or significance tests. Several cells in Table 2 are identical across configurations, which suggests the optimizations often don't change retrieval behavior or the table collapses over settings; you'd want to know which. The GPT-4 style-matching concern is real: the evaluation questions were generated by GPT-4, the multi-query expansions come from GPT-4, and GPT-4 also scores the outputs. That doesn't kill the internal comparison, but it does mean the headline improvement may not transfer to real student queries, which are not all GPT-4-like. The authors should run a control where the evaluation questions come from a different source (e.g., real student emails or a non-GPT-4 generation) or at least acknowledge this as a boundary on generality. Also, no dataset or code is released, which makes the empirical claims hard to verify.

The paper is honest about its own limits—it says 'definitive conclusions are challenging to draw'—and the direction of the effect is consistent, so I don't think the central claim is a fabrication. It's just under-powered and under-specified.

Bottom line: appropriate for a workshop or a short conference paper if the authors add error bars, release the data/code, and address the query-source confound. A serious referee could help them get there. I'd send it to review, but I wouldn't put much weight on the exact numbers yet.

Referee Report

5 major / 5 minor

Summary. The paper evaluates a retrieval-augmented generation (RAG) system for answering student questions about study programs at a technical university. It constructs a dataset of 200 GPT-4-generated, manually corrected QA pairs in English and German, and tests four optimization modules—Multi-Query, Child-Parent-Retriever, Ensemble Retriever, and In-Context-Learning—across five LLMs (Llama 2 7B/13B, GPT-3.5, GPT-4, Mistral 7B). The central empirical claim is that adding Multi-Query to the retrieval phase yields a consistent increase in top-5 hit rate, while other modules have smaller or inconsistent effects. The paper also proposes a RAG Confusion Matrix for separating retrieval and generation errors.

Significance. The paper provides a useful empirical benchmark of RAG optimizations on a domain-specific use case, and the multi-query effect is consistent across all five models and both languages in Table 2, which is a genuine strength. The RAG Confusion Matrix is a simple, interpretable diagnostic that practitioners may find valuable. However, the evaluation is limited by small sample sizes, lack of significance tests, and reliance on GPT-4 for both query generation and evaluation, so the quantitative claims should be treated as preliminary until the identified controls are added.

major comments (5)

[§5.1, Table 2] The claim of a 'significant performance increase' is not backed by any statistical test. Hit rates are computed on 82 samples, and the reported differences (e.g., Llama 2 7B English: 43.21 vs 53.09) are presented without confidence intervals, p-values, or effect sizes. Since the same queries are evaluated under different retrieval configurations, a paired test such as McNemar's test would be appropriate. Without this, the headline claim is only descriptive and could be due to chance, especially for the smaller German-language differences.
[§3 and §4.2.2] The evaluation questions and the multi-query expansions are both generated by GPT-4. Because hit-rate scoring relies on BM25 and embedding similarity, which are sensitive to lexical and surface-form overlap, the consistent multi-query advantage may partly reflect GPT-4's ability to rephrase its own question style, rather than a general improvement for arbitrary user queries. The manual correction of answers in Section 3 does not change the question text, so this confound remains. A control experiment that evaluates retrieval on human-written queries (or queries generated by a different model) is required to support the abstract's general claim.
[§5.2, Table 4] The LLM-based evaluation and the resulting RAG Confusion Matrix are computed on only 20 answers (10 with correct context, 10 with incorrect context). The confusion matrix entries are therefore counts out of 10, and the threshold of 5 (a perfect score) makes the acceptable-response criterion extremely strict. The reported differences between configurations (e.g., Faithfulness false positives of 5 vs 4 vs 5) are within the noise of this sample size and should not be used to rank configurations without uncertainty estimates.
[§4.2.2 and Table 2] In many configurations the Child-Parent-Retriever and Ensemble Retriever modules have no effect on hit rate at all (e.g., Llama 2 7B German: 8.64 for er, cpr, and icl; the same value for all three). The paper does not explain this degeneracy or analyze whether the pre-retrieval LLM filtering in Section 4.2.1 already restricts the candidate set so much that later retrieval modules are irrelevant. This is important because it affects which of the four proposed optimizations are actually shown to be useful.
[§5.3, Table 5] The optimized 'Retriever+' configuration is claimed to increase hit rate, but no hit-rate numbers for this configuration are reported; the table only lists generation metrics (Faithfulness, Relevance, Coherence, Fluency, ROUGE, BERTScore). Without the hit-rate comparison, the reader cannot verify the stated improvement in Section 5.3.

minor comments (5)

[Section 3] The word 'scrapping' should be 'scraping'.
[Section 5.3] The phrase 'GPT-4’s its strong ability' contains a typo; it should read 'GPT-4’s strong ability'.
[Authorship header] The author name 'Andrei Staradube ts' appears to have an inserted space and should be 'Andrei Staradubets'.
[Table 5] The notation 'x/x' for human vs GPT-4 evaluation is explained only in the caption; the text should reference it explicitly when discussing alignment.
[Section 6] The sentence 'the human annotators as the humans almost always rated the answer lower than the LLM rating' is grammatically incomplete; suggest rewording to 'the human annotators almost always rated the answers lower than the LLM did'.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the multi-query hit-rate claim is an empirical comparison on a fixed, manually corrected evaluation set, not a fitted prediction.

full rationale

The abstract's central claim is about retrieval Hit Rate (Sec. 4.4.1), which is computed by checking whether the correct study program and topic appear in the top-5 retrieved documents. The evaluation pairs were created semi-automatically with GPT-4, but the authors state that 'the two authors manually went over all the questions and filtered the dataset down to 200 QA pairs' and 'All 200 answers were checked and any observed errors that stem from GPT generation were manually corrected' (Sec. 3). The Multi-Query module is a fixed prompt-based GPT-4 query-expansion step (Sec. 4.2.2), not a parameter fitted to the evaluation labels, and the same expansion procedure is applied across all compared configurations. No equation or construction in the paper defines a configuration's hit rate in terms of the expansion model's outputs, so the claimed improvement is not equivalent to its inputs by construction. The overlap between GPT-4 as question generator and as query expander is a possible surface-form bias in the experimental design, which is a validity concern rather than a logical circularity. The only self-citation ([2], used to justify LLM-based evaluation) is not load-bearing for the central hit-rate claim, which is independent of LLM-as-judge scores, and the paper itself reports discrepancies between human and LLM evaluation in Sec. 5.3.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim depends on hand-set evaluation parameters (top-k, threshold, chunk sizes, ensemble weights, ICL examples) and on the validity of the GPT-4-generated and GPT-4-scored evaluation set. No new physical or mathematical entities are introduced; the RAG Confusion Matrix is an evaluation rubric built from the standard confusion matrix, not a postulated entity.

free parameters (5)

Hit Rate top-k (top-5 retrieval window) = 5 (top-5)
Used to define whether a retrieval is a hit; changing k would change every hit-rate number that supports the main multi-query claim. Chosen by the authors without sensitivity analysis.
Acceptable-response threshold for RAG Confusion Matrix = 5 on a 1-5 scale
The confusion matrix counts an answer as acceptable only if the GPT-4 judge gives the maximum score. This hand-set threshold directly determines all TP/FP/TN/FN counts in Table 4.
Chunk sizes for Child-Parent Retriever = 300 and 1500 characters
Child chunks are 300 characters and parent chunks are 1500. These sizes were chosen a priori and affect what the retriever can match.
Ensemble Retriever weighting = 0.5 BM25 + 0.5 cosine similarity
The paper states equal weight was assigned during experiments; no tuning or sensitivity analysis is reported.
In-context learning examples = 3 manually selected QA pairs
Three-shot examples were chosen for semantic diversity by hand; the choice can shift generation quality and evaluation scores.

assumptions (3)

domain assumption The 200 QA pairs and their relevant-context labels, generated semi-automatically by GPT-4 and corrected manually, are accurate ground truth.
All hit-rate, ROUGE, BERTScore, and confusion-matrix numbers depend on these labels; Section 3 describes the curation process.
domain assumption GPT-4-as-judge scores reflect true answer quality on a 1-5 scale for relevance, coherence, faithfulness, and fluency.
The optimal-configuration search and confusion matrix rely on LLM evaluation; only a 30-sample human comparison is reported in Section 5.3.1 and the paper notes human scores were often lower.
domain assumption The 72 scraped study program pages fully and accurately represent the university's programs at the time of evaluation.
The corpus is the sole knowledge source for retrieval; if pages are stale or incomplete, the evaluation labels and hit rates rest on a partial corpus. Introduced in Section 3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Optimizing a Retrieval Augmented Generation using Large Language Model on Academic Data." pith.science (2026). https://pith.science/paper/6S5MQX62

@misc{pith2026241108438,
  author       = {Pith},
  title        = {Pith review of: Towards Optimizing a Retrieval Augmented Generation using Large Language Model on Academic Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6S5MQX62}},
  note         = {Machine review of arXiv:2411.08438}
}

read the original abstract

Given the growing trend of many organizations integrating Retrieval Augmented Generation (RAG) into their operations, we assess RAG on domain-specific data and test state-of-the-art models across various optimization techniques. We incorporate four optimizations; Multi-Query, Child-Parent-Retriever, Ensemble Retriever, and In-Context-Learning, to enhance the functionality and performance in the academic domain. We focus on data retrieval, specifically targeting various study programs at a large technical university. We additionally introduce a novel evaluation approach, the RAG Confusion Matrix designed to assess the effectiveness of various configurations within the RAG framework. By exploring the integration of both open-source (e.g., Llama2, Mistral) and closed-source (GPT-3.5 and GPT-4) Large Language Models, we offer valuable insights into the application and optimization of RAG frameworks in domain-specific contexts. Our experiments show a significant performance increase when including multi-query in the retrieval phase.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 19 canonical work pages

[1]

Sallam Abualhaija, Chetan Arora, Amin Sleimi, and Lionel C Briand. 2022. Automated question answering for im- proved understanding of compliance requirements: A multi-document st udy. In 2022 IEEE 30th International Require- ments Engineering Conference (RE) . IEEE, 39–50

work page 2022
[2]

Anum Afzal, Alexander Kowsik, Rajna Fani, and Florian Matthes. 20 24. Towards Optimizing and Evaluating a Re- trieval Augmented QA Chatbot using LLMs with Human-in-the-Loop. In Proceedings of the Fifth Workshop on Data Science with Human-in-the-Loop (DaSH 2024) . Association for Computational Linguistics, Mexico City, Mexico

work page 2024
[3]

Chenxin An, Ming Zhong, Zhichao Geng, Jianqiang Yang, and Xipeng Qiu. 2021. R etrievalSum: A Retrieval Enhanced Framework for Abstractive Summarization. arXiv: 2109.07943 [cs.CL]

arXiv 2021
[4]

Deng Cai, Yan Wang, Huayang Li, Wai Lam, and Lemao Liu. 2021. Neura l Machine Translation with Monolingual Translation Memory. In Proceedings of the 59th Annual Meeting of the Association fo r Computational Linguistics and the 11th International Joint Conference on Natural Languag e Processing (Volume 1: Long Papers) . Association for Com- putational Ling...

work page 2021
[5]

Deng Cai, Yan Wang, Lemao Liu, and Shuming Shi. 2022. Recent advances in retrieval-augmented text generation. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval

work page 2022
[6]

Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Re ading Wikipedia to Answer Open-Domain Questions. In Proceedings of the 55th Annual Meeting of the Association fo r Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada. https://doi.org/10.18653/v1/P17-1171

doi:10.18653/v1/p17-1171 2017
[7]

Yu Chen, Scott Jensen, Leslie J Albert, Sambhav Gupta, and Terr i Lee. 2023. Artiﬁcial intelligence (AI) student as- sistants in the classroom: Designing chatbots to support student suc cess. Information Systems Frontiers 25, 1 (2023), 161–182

work page 2023
[8]

Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Precise Z ero-Shot Dense Retrieval without Relevance La- bels. In Proceedings of the 61st Annual Meeting of the Association fo r Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Toronto, Canada, 1762–1777. https://doi.org/10.18653/v1/2023.acl-long.99 , V...

doi:10.18653/v1/2023.acl-long.99 2023

Show all 31 references

[9]

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Da i, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-Augmented Generation for Large Language Mo dels: A Survey. arXiv: 2312.10997 [cs.CL]

2024 arXiv
[10]

Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhu ry, Ankita Naik, Pengshan Cai, and Alﬁo Gliozzo
[11]

Roman Grundkiewicz, Marcin Junczys-Dowmuntz, and Kenneth Heaﬁe ld. 2019. Neural grammatical error correction systems with unsupervised pre-training on synthetic data. In 14th Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational...

2019
[12]

Prakhar Gupta, Jeﬀrey Bigham, Yulia Tsvetkov, and Amy Pavel. 2021. Controlling Dialogue Generation with Semantic Exemplars. In Proceedings of the 2021 Conference of the North American Cha pter of the Association for Computational Linguistics: Human Language Technologies . Asso...

2021
[13]

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Ch ris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas...

2023
[14]

Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Ba nnert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023. ChatGPT for good? On opportunities and challenges of large language models for education. Learning a...

2023
[15]

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-a ugmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing ...

2020
[16]

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out. Association for Computational Linguistics, 74–81

2004
[17]

Jingjing Liu, Chang Liu, and Nicholas J. Belkin. 2020. Personalization in text information retrieval: A survey. Journal of the Association for Information Science and Technology 71, 3 (2020), 349–369

2020
[18]

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Ch enguang Zhu. 2023. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. arXiv: 2303.16634 [cs.CL]

2023 arXiv
[19]

Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023 . Query Rewriting in Retrieval-Augmented Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in N atural Language Processing . Association for Computational Linguistics, Singapore, ...

2023 doi
[20]

Dadi Ramesh and Suresh Kumar Sanampudi. 2022. An automated essay scoring systems: a systematic literature review. Artiﬁcial Intelligence Review 55, 3 (2022), 2495–2527

2022
[21]

Stephen Robertson and Hugo Zaragoza. 2009. The Probabilis tic Relevance Framework: BM25 and Beyond.Foundations and Trends in Information Retrieval 3, 4 (2009), 333–389. https://doi.org/10.1561/1500000019

2009 doi
[22]

Burr Settles, Chris Brust, Erin Gustafson, Masato Hagiwara , and Nitin Madnani. 2018. Second language acquisition modeling. In Proceedings of the thirteenth workshop on innovative use of NLP for building educational applications . 56–65

2018
[23]

Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. 2023. Improving the domain adaptation of retrieval a ugmented generation (RAG) models for open domain question answering. Transactions of the Association for Com...

2023
[24]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad A lmahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, ...

2023
[25]

Fangkai Yang, Pu Zhao, Zezhong Wang, Lu Wang, Jue Zhang, Mohit Ga rg, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang. 2023. Empower Large Language Model to Perform Be tter on Industrial Domain-Speciﬁc Question Answering. arXiv: 2305.11541

2023 arXiv
[26]

Ke Zhang and Ayse Begum Aslan. 2021. AI technologies for educa tion: Recent research & future directions. Computers and Education: Artiﬁcial Intelligence 2 (2021), 100025. , Vol. 1, No. 1, Article . Publication date: November 2024. Towards Optimizing a Retrieval Augmented Gene...

2021
[27]

Weinberger, and Y oav Artzi

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Y oav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. arXiv: 1904.09675 [cs.CL]

2020 arXiv
[28]

Hashimoto

Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathlee n McKeown, and Tatsunori B. Hashimoto. 2023. Benchmarking Large Language Models for News Summarization. arXiv: 2301.13848 [cs.CL]

2023 arXiv
[29]

Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, a nd Tat-Seng Chua. 2021. Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering. ar Xiv:2101.00774 [cs.AI]

2021 arXiv
[30]

Brian Zylich, Adam Viola, Brokk Toggerson, Lara Al-Hariri, and Andrew Lan. 2020. Exploring automated question answering methods for teaching assistance. In Artiﬁcial Intelligence in Education: 21st International C onference, AIED 2020, Ifrane, Morocco, July 6–10, 2020, Proceed...

2020
[2022]

In Proceedings of the 2022 Conference of the North American Cha pter of the Association for Computational Linguistics: Human Languag e Technologies

Re2G: Retrieve, Rerank, Generate. In Proceedings of the 2022 Conference of the North American Cha pter of the Association for Computational Linguistics: Human Languag e Technologies. Association for Computational Linguistics, Seattle, United States, 2701–2715. https://doi.org/...

2022 doi

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Sallam Abualhaija, Chetan Arora, Amin Sleimi, and Lionel C Briand. 2022. Automated question answering for im- proved understanding of compliance requirements: A multi-document st udy. In 2022 IEEE 30th International Require- ments Engineering Conference (RE) . IEEE, 39–50

work page 2022

[2] [2]

Anum Afzal, Alexander Kowsik, Rajna Fani, and Florian Matthes. 20 24. Towards Optimizing and Evaluating a Re- trieval Augmented QA Chatbot using LLMs with Human-in-the-Loop. In Proceedings of the Fifth Workshop on Data Science with Human-in-the-Loop (DaSH 2024) . Association for Computational Linguistics, Mexico City, Mexico

work page 2024

[3] [3]

Chenxin An, Ming Zhong, Zhichao Geng, Jianqiang Yang, and Xipeng Qiu. 2021. R etrievalSum: A Retrieval Enhanced Framework for Abstractive Summarization. arXiv: 2109.07943 [cs.CL]

arXiv 2021

[4] [4]

Deng Cai, Yan Wang, Huayang Li, Wai Lam, and Lemao Liu. 2021. Neura l Machine Translation with Monolingual Translation Memory. In Proceedings of the 59th Annual Meeting of the Association fo r Computational Linguistics and the 11th International Joint Conference on Natural Languag e Processing (Volume 1: Long Papers) . Association for Com- putational Ling...

work page 2021

[5] [5]

Deng Cai, Yan Wang, Lemao Liu, and Shuming Shi. 2022. Recent advances in retrieval-augmented text generation. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval

work page 2022

[6] [6]

Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Re ading Wikipedia to Answer Open-Domain Questions. In Proceedings of the 55th Annual Meeting of the Association fo r Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada. https://doi.org/10.18653/v1/P17-1171

doi:10.18653/v1/p17-1171 2017

[7] [7]

Yu Chen, Scott Jensen, Leslie J Albert, Sambhav Gupta, and Terr i Lee. 2023. Artiﬁcial intelligence (AI) student as- sistants in the classroom: Designing chatbots to support student suc cess. Information Systems Frontiers 25, 1 (2023), 161–182

work page 2023

[8] [8]

Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Precise Z ero-Shot Dense Retrieval without Relevance La- bels. In Proceedings of the 61st Annual Meeting of the Association fo r Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Toronto, Canada, 1762–1777. https://doi.org/10.18653/v1/2023.acl-long.99 , V...

doi:10.18653/v1/2023.acl-long.99 2023