REVIEW 3 major objections 4 minor 3 cited by
Automated Literature Review Using NLP Techniques and LLM-Based Retrieval-Augmented Generation
T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A retrieval-augmented GPT-3.5-turbo system beats a fine-tuned T5 and a frequency-based spaCy extractor at generating literature-review segments from PDFs, according to ROUGE scores on the SciTLDR test set.
desk verdict A concrete but flawed comparison of NLP methods for automatic literature review generation; the main ROUGE claim is compromised by likely test-set contamination. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the retrieval-augmented generation (RAG) pipeline built around GPT-3.5-turbo. A custom assistant has retrieval enabled, and the SciTLDR dataset is loaded into its knowledge base; a prompt instructs the model to summarize a given PDF's extracted text in the style of a literature review, mentioning the first author and paper title, in at most 80 words. The system extracts full text from each PDF, submits it as a thread message, collects the assistant's per-paper summary, and merges the summaries into one coherent review segment. The other two pipelines, spaCy's frequency-based sentence scoring and a fine-tuned T5 summarizer, serve as baselines for comparison, and ROUGE-N, ROUGE-L, and ROUGE-Lsum are used as the shared evaluation metric.
What would settle it
Check whether any SciTLDR test reference summaries were present in the assistant's uploaded knowledge base; if so, remove them and rerun the ROUGE evaluation to see whether the 0.364 ROUGE-1 persists.
Extended reading notes
Core claim
On the SciTLDR test split, the retrieval-augmented GPT-3.5-turbo system achieved a ROUGE-1 of 0.364, ROUGE-2 of 0.123, ROUGE-L of 0.181, and ROUGE-Lsum of 0.182. The T5 model scored 0.268, 0.115, 0.204, and 0.204; spaCy scored 0.257, 0.055, 0.144, and 0.146. The paper takes these numbers as evidence that the LLM-based approach outperforms both older NLP approaches, places T5 second, and uses the LLM pipeline as the backend of a user interface that accepts uploaded PDFs and produces a merged literature review.
Load-bearing premise
The reported ROUGE advantage is trustworthy only if the SciTLDR test-set summaries were not part of the assistant's retrieval knowledge base when it was evaluated; the paper adds the dataset to the assistant's knowledge base but does not state that test targets were excluded.
Editorial extensions
If this is right
- A PDF-only input is enough for the RAG-LLM pipeline to produce literature-review-style summaries of individual papers.
- On ROUGE-1 and ROUGE-2, the LLM system scores highest, so the paper concludes LLMs are the strongest of the three compared approaches for this task.
- The T5 transformer, though second, still beats the frequency-based extractor on ROUGE-2 and ROUGE-L.
- The LLM-backed UI lets a user upload several PDFs and receive a merged review that names each paper's first author and title.
- The results imply that older frequency-based extractive summarization is the weakest of the three for this literature-review use case.
Reading between the lines
- If the SciTLDR test targets were present in the assistant's knowledge base, the reported ROUGE gap may come from retrieving reference summaries rather than from generation, and a stricter evaluation without those targets could shrink the gap.
- ROUGE-1 rewards word overlap and says little about whether a generated review is coherent, complete, or useful to a researcher, so a blinded human judgment on the same outputs would be a more informative comparison.
- A natural next experiment is to run the same three pipelines on papers published after the model's training cutoff, where no reference summary can leak through retrieval.
- The current system merges per-paper summaries by concatenation; a pipeline that plans a shared narrative structure across the selected papers might improve readability and coverage.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes three end-to-end pipelines for generating literature review segments from PDFs: a frequency-based extractive approach using spaCy, a fine-tuned Simple T5 transformer, and a retrieval-augmented GPT-3.5-TURBO-0125 assistant that uses the SciTLDR dataset as a knowledge base. The three systems are compared on the SciTLDR test split using ROUGE-1/2/L/Lsum, with GPT-3.5-turbo reported as best on ROUGE-1 (0.364) and ROUGE-2 (0.123). A GUI for the LLM-based system is described, along with an informal user evaluation. The central claim is that the LLM-based approach outperforms both T5 and spaCy for automated literature review generation.
Significance. If the empirical comparison were valid, the paper would provide a useful practical data point on RAG-based LLM pipelines for scientific text summarization and a working end-to-end tool. The manuscript documents the three pipelines, gives the exact prompt used for the OpenAI assistant, and reports standard ROUGE metrics. However, the central claim is not established: the evaluation is potentially contaminated because the SciTLDR test targets may have been accessible to the LLM via its knowledge base, and the evaluation measures single-paper summarization rather than the multi-paper literature review task the paper claims to address. The paper also provides no code, no test-set sizes, no error bars, and no significance tests, so the reported ranking is not reproducible or statistically supported.
major comments (3)
- [III-D and IV-C] The evaluation is potentially circular. Section III-D states that the SciTLDR dataset is added as the knowledge base for the OpenAI assistant with retrieval enabled, and the prompt explicitly instructs the model to use the data.json file in its knowledge base. Section IV-C then evaluates the assistant on the SciTLDR test data. The paper never states that test-set source-target pairs, or at least the target summaries, were removed from data.json. With retrieval enabled, the assistant could retrieve the gold reference summary for a test input, so the reported ROUGE-1 score of 0.364 may reflect lookup or retrieval of the reference rather than generation of a literature review. This is load-bearing for the central claim that the LLM outperforms T5 and spaCy. The authors must specify exactly which SciTLDR split was uploaded and confirm that no test targets were accessible, or rebuild the knowledge base from the training split only and re-run the evaluation.
- [III and IV] The evaluation task does not match the claimed task. The SciTLDR dataset is a single-document extreme summarization benchmark: each source is one paper (or its abstract/introduction/conclusion/full text) and each target is a TLDR summary of that same paper. The ROUGE scores in Tables I-IV compare per-paper model outputs against these single-paper targets. However, the paper's stated objective is to generate a literature review segment by merging summaries from multiple PDFs. No evaluation is reported on the final merged literature review output; the user evaluation in Section IV-E is anecdotal and unscored. As presented, the scores support a claim about single-document summarization, not about automated literature review generation. The authors should either reframe the contribution as scientific summarization or evaluate on a multi-document literature review test set.
- [Tables I-IV and Section IV-D] The comparison is not statistically grounded. Only point estimates are reported, with no test-set size, standard deviation, confidence interval, or significance test. The ROUGE-1 difference between T5 (0.268) and spaCy (0.257) is only 0.011, which is typically within the noise of ROUGE evaluation; the ranking of the three systems is therefore not established. Furthermore, no details are given for the SciTLDR train/test split sizes or for the Simple T5 fine-tuning hyperparameters, and no code is released. These omissions prevent replication and make it impossible to assess whether the observed differences are meaningful. The authors should provide test statistics or bootstrap intervals and full experimental configuration.
minor comments (4)
- [III-D and IV-A-C] The systems are not matched on output length: the LLM prompt caps the output at 80 words, while no equivalent length constraint is reported for the spaCy or T5 systems. Since ROUGE F1 scores are sensitive to summary length, this confound should be controlled or discussed.
- [References] Several references are incomplete, e.g., [8], [10], and [11] lack full venue and year information, and [13] appears both as a footnote-style dataset citation and in the main reference list with inconsistent formatting.
- [Figures] Figures 1-7 are referenced in the text but not included in the submitted manuscript, which makes it difficult to verify the pipeline descriptions; please include them.
- [Abstract and Section V] The phrase 'satisfactory results' and 'above the acceptable mark' are undefined; the authors should specify a threshold or justify why these ROUGE values are considered satisfactory.
Circularity Check
LLM evaluation is contaminated: SciTLDR test targets are loaded into the GPT-3.5-turbo knowledge base, so the reported ROUGE-1 of 0.364 may be retrieval, not generation.
-
fitted input called prediction
[Section III-A, Section III-D (OpenAI Assistant prompt), Section IV-C]
""For the LLM-based approach, this dataset is used as the knowledge base for the model." / "The retrieval is turned on and the dataset is added for the knowledge of the LLM." / "The user will give you a pdf file as input, similar to the 'input' field of the given 'data.json' file in your knowledge base. You have to produce a summarized 'output' for the given pdf based on the file given to your knowledge." / "The LLM-based model was evaluated on the test data utilizing the ROUGE scores.""
The claimed prediction target — ROUGE-1 on SciTLDR test data — is also an input to the system: the entire SciTLDR dataset is uploaded into the OpenAI assistant's retrieval knowledge base, with retrieval enabled, and the prompt instructs the model to produce an 'output' for a PDF 'based on the file given to your knowledge.' The data.json file contains input/output pairs, so if the test split is included in that knowledge base, the assistant can retrieve the reference summary for a test paper and return it. The paper never states that test targets were removed from the knowledge base or that only the training split was uploaded.
full rationale
The central circular step is the LLM evaluation contamination. The T5 model is fine-tuned on SciTLDR and then evaluated on its test split, which is standard supervised evaluation and not circular. spaCy is unsupervised and evaluated on the same test data, also not circular. The paper's self-citation [5] for ROUGE analysis is not load-bearing because ROUGE is an external, standard metric and the citation does not define the evaluation. However, the LLM-based system is built by adding the SciTLDR dataset as the assistant's knowledge base with retrieval enabled, and the same dataset's test split is then used as the evaluation benchmark. The prompt tells the model to generate the 'output' field for an 'input' similar to those in the data.json knowledge file. Unless the test split was explicitly excluded from that knowledge base — which the paper never states — the model can retrieve the reference summary for a test paper, so the reported ROUGE-1 of 0.364 is not an independent measure of generation ability. This compromises the paper's central claim that the LLM outperforms T5 and spaCy. Score 7 reflects that one central 'prediction' reduces by construction to retrieval, while the other two baselines and the ROUGE toolkit remain normal external evaluation steps.
Assumptions & free parameters
free parameters (4)
- spaCy sentence selection threshold =
top 10% of sentences
- T5 fine-tuning hyperparameters =
not reported
- LLM summary length cap =
max 80 words
- SciTLDR train/test split =
not reported
assumptions (4)
- domain assumption SciTLDR TLDR summaries are appropriate gold references for literature review generation.
- domain assumption ROUGE scores adequately measure literature review quality.
- domain assumption The LLM knowledge base contains no test-set reference summaries.
- domain assumption Concatenating per-paper summaries yields a coherent literature review.
Cite this review
Pith. "Pith review of Automated Literature Review Using NLP Techniques and LLM-Based Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/FOYTI3OL
@misc{pith2026241118583,
author = {Pith},
title = {Pith review of: Automated Literature Review Using NLP Techniques and LLM-Based Retrieval-Augmented Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/FOYTI3OL}},
note = {Machine review of arXiv:2411.18583}
}
read the original abstract
This research presents and compares multiple approaches to automate the generation of literature reviews using several Natural Language Processing (NLP) techniques and retrieval-augmented generation (RAG) with a Large Language Model (LLM). The ever-increasing number of research articles provides a huge challenge for manual literature review. It has resulted in an increased demand for automation. Developing a system capable of automatically generating the literature reviews from only the PDF files as input is the primary objective of this research work. The effectiveness of several Natural Language Processing (NLP) strategies, such as the frequency-based method (spaCy), the transformer model (Simple T5), and retrieval-augmented generation (RAG) with Large Language Model (GPT-3.5-turbo), is evaluated to meet the primary objective. The SciTLDR dataset is chosen for this research experiment and three distinct techniques are utilized to implement three different systems for auto-generating the literature reviews. The ROUGE scores are used for the evaluation of all three systems. Based on the evaluation, the Large Language Model GPT-3.5-turbo achieved the highest ROUGE-1 score, 0.364. The transformer model comes in second place and spaCy is at the last position. Finally, a graphical user interface is created for the best system based on the large language model.
Forward citations
Cited by 3 Pith papers
-
Whose Name Comes Up? II: Benchmarking and Intervention-Based Auditing of LLM-Based Scholar Recommendation
A new benchmark auditing LLM-based scholar recommendation across 22 models shows that inference-time interventions (temperature, constrained prompting, RAG) trade off technical quality against social representation ra...
-
Who Gets Recommended? Investigating Gender, Race, and Country Disparities in Paper Recommendations from Large Language Models
LLM recommendations of important AI research favor recent, well-cited, team-authored papers, but do not measurably over-represent male, white, or developed-country scholars relative to a human-curated benchmark.
-
How Far Are AI Scientists from Changing the World?
This survey proposes a four-level capability framework for AI Scientist systems and, using an AI reviewer, finds that current systems produce papers rated well below normal scientific standards.
Reference graph
Works this paper leans on
-
[1]
Automating systematic literature review
Felizardo KR, Carver JC. Automating systematic literature review. Contemporary empirical methods in software engineering. 2020:327-55
work page 2020
-
[2]
Nlp based machine learning approaches for text summariza- tion
Adhikari S. Nlp based machine learning approaches for text summariza- tion. In2020 Fourth International Conference on Computing Methodolo- gies and Communication (ICCMC) 2020 Mar 11 (pp. 535-538). IEEE
work page 2020
-
[3]
TLDR: Extreme summarization of scientific documents
Cachola I, Lo K, Cohan A, Weld DS. TLDR: Extreme summarization of scientific documents. arXiv preprint arXiv:2004.15011. 2020 Apr 30
arXiv 2004
-
[4]
Extractive automatic text summarization using SpaCy in Python & NLP
Jugran S, Kumar A, Tyagi BS, Anand V. Extractive automatic text summarization using SpaCy in Python & NLP. In2021 International conference on advance computing and innovative technologies in en - gineering (ICACITE) 2021 Mar 4 (pp. 582-585). IEEE
work page 2021
-
[5]
Ali NF, Tanvin JU, Islam MR, Ahmed J, Akhtaruzzaman M. ROUGE Score Analysis and Performance Evaluation Between Google T5 and SpaCy for YouTube News Video Summarization. In2023 26th Interna - tional Conference on Computer and Information Technology (ICCIT) 2023 Dec 13 (pp. 1-6). IEEE
work page 2023
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.