{"id":"ef436cf1-5be2-40a2-95eb-e9662816eee8","arxiv_id":"2411.18583","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"On SciTLDR, a GPT-3.5-turbo RAG pipeline scores ROUGE-1 0.364 versus 0.268 for T5 and 0.257 for spaCy, and is used to build a PDF-to-literature-review tool.","lead":"The authors compare three ways to auto-generate literature review text from research PDFs: a spaCy frequency baseline, a fine-tuned T5, and GPT-3.5-turbo with retrieval-augmented generation. The LLM pipeline wins on ROUGE-1 (0.364) and is packaged into a UI, but the evaluation may be contaminated by test data in the model's knowledge base.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central comparison may be invalid because SciTLDR test targets were uploaded into the GPT-3.5-turbo knowledge base; the paper never states that they were excluded.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the SciTLDR test set is used both as the LLM knowledge base and as the evaluation target, creating a potential circularity that could inflate the reported ROUGE scores. My independent reading confirms that Section III-D describes adding the SciTLDR dataset to the OpenAI assistant's knowledge base, while Section IV-C evaluates on the test data, with no explicit statement that test pairs were removed. Since this concern is unresolved, the central claim that GPT-3.5-turbo outperforms T5 and spaCy is not established as reported. The paper does provide a useful end-to-end pipeline design and a UI prototype, but the empirical comparison rests on an unverified assumption about data separation. I do not see a different, more load-bearing flaw that would change the reader's rejection; the recommendation therefore remains unchanged.","tokens_in":6859,"tokens_out":2402,"duration_ms":21803,"concrete_test":"Inspect the assistant's uploaded knowledge-base file (data.json) and retrieval logs to verify that no SciTLDR test split examples are present. If test sources or targets appear in the knowledge base, re-run the GPT-3.5-turbo RAG evaluation with a knowledge base built from the train split only and recompute ROUGE-1 and ROUGE-L; if the scores drop substantially (e.g., ROUGE-1 falls below T5's 0.268), the reported ranking is an artifact of retrieval and the headline claim fails. Also record the number of test instances and the exact ROUGE computation settings to confirm reproducibility.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-D says the SciTLDR dataset is added as the knowledge base for the OpenAI assistant, with retrieval enabled, and the prompt instructs the model to use the data.json file in its knowledge base. Section IV-C then evaluates the same assistant on the SciTLDR test split. If data.json contains test split source-target pairs, or even just the target summaries, the RAG tool can retrieve the reference summary, so the reported ROUGE-1 of 0.364 would measure retrieval fidelity rather than review-generation ability. The paper never states that test examples were removed from the knowledge base, nor does it specify which split of SciTLDR was uploaded. This is the single load-bearing assumption: the conclusion that the LLM-based model outperforms T5 and spaCy requires that the LLM did not have access to the test targets during evaluation. Without that guarantee, the central empirical claim is not established. Secondary reproducibility gaps (no code, no split sizes, no error bars) compound the problem, but the contamination issue is decisive.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes three end-to-end pipelines for generating literature review segments from PDFs: a frequency-based extractive approach using spaCy, a fine-tuned Simple T5 transformer, and a retrieval-augmented GPT-3.5-TURBO-0125 assistant that uses the SciTLDR dataset as a knowledge base. The three systems are compared on the SciTLDR test split using ROUGE-1/2/L/Lsum, with GPT-3.5-turbo reported as best on ROUGE-1 (0.364) and ROUGE-2 (0.123). A GUI for the LLM-based system is described, along with an informal user evaluation. The central claim is that the LLM-based approach outperforms both T5 and spaCy for automated literature review generation.","tokens_in":7052,"tokens_out":6953,"duration_ms":64932,"significance":"If the empirical comparison were valid, the paper would provide a useful practical data point on RAG-based LLM pipelines for scientific text summarization and a working end-to-end tool. The manuscript documents the three pipelines, gives the exact prompt used for the OpenAI assistant, and reports standard ROUGE metrics. However, the central claim is not established: the evaluation is potentially contaminated because the SciTLDR test targets may have been accessible to the LLM via its knowledge base, and the evaluation measures single-paper summarization rather than the multi-paper literature review task the paper claims to address. The paper also provides no code, no test-set sizes, no error bars, and no significance tests, so the reported ranking is not reproducible or statistically supported.","major_comments":[{"comment":"The evaluation is potentially circular. Section III-D states that the SciTLDR dataset is added as the knowledge base for the OpenAI assistant with retrieval enabled, and the prompt explicitly instructs the model to use the data.json file in its knowledge base. Section IV-C then evaluates the assistant on the SciTLDR test data. The paper never states that test-set source-target pairs, or at least the target summaries, were removed from data.json. With retrieval enabled, the assistant could retrieve the gold reference summary for a test input, so the reported ROUGE-1 score of 0.364 may reflect lookup or retrieval of the reference rather than generation of a literature review. This is load-bearing for the central claim that the LLM outperforms T5 and spaCy. The authors must specify exactly which SciTLDR split was uploaded and confirm that no test targets were accessible, or rebuild the knowledge base from the training split only and re-run the evaluation.","section":"III-D and IV-C"},{"comment":"The evaluation task does not match the claimed task. The SciTLDR dataset is a single-document extreme summarization benchmark: each source is one paper (or its abstract/introduction/conclusion/full text) and each target is a TLDR summary of that same paper. The ROUGE scores in Tables I-IV compare per-paper model outputs against these single-paper targets. However, the paper's stated objective is to generate a literature review segment by merging summaries from multiple PDFs. No evaluation is reported on the final merged literature review output; the user evaluation in Section IV-E is anecdotal and unscored. As presented, the scores support a claim about single-document summarization, not about automated literature review generation. The authors should either reframe the contribution as scientific summarization or evaluate on a multi-document literature review test set.","section":"III and IV"},{"comment":"The comparison is not statistically grounded. Only point estimates are reported, with no test-set size, standard deviation, confidence interval, or significance test. The ROUGE-1 difference between T5 (0.268) and spaCy (0.257) is only 0.011, which is typically within the noise of ROUGE evaluation; the ranking of the three systems is therefore not established. Furthermore, no details are given for the SciTLDR train/test split sizes or for the Simple T5 fine-tuning hyperparameters, and no code is released. These omissions prevent replication and make it impossible to assess whether the observed differences are meaningful. The authors should provide test statistics or bootstrap intervals and full experimental configuration.","section":"Tables I-IV and Section IV-D"}],"minor_comments":[{"comment":"The systems are not matched on output length: the LLM prompt caps the output at 80 words, while no equivalent length constraint is reported for the spaCy or T5 systems. Since ROUGE F1 scores are sensitive to summary length, this confound should be controlled or discussed.","section":"III-D and IV-A-C"},{"comment":"Several references are incomplete, e.g., [8], [10], and [11] lack full venue and year information, and [13] appears both as a footnote-style dataset citation and in the main reference list with inconsistent formatting.","section":"References"},{"comment":"Figures 1-7 are referenced in the text but not included in the submitted manuscript, which makes it difficult to verify the pipeline descriptions; please include them.","section":"Figures"},{"comment":"The phrase 'satisfactory results' and 'above the acceptable mark' are undefined; the authors should specify a threshold or justify why these ROUGE values are considered satisfactory.","section":"Abstract and Section V"}],"recommendation":"reject","confidential_remarks":"The contamination issue in Section III-D/IV-C is decisive: unless the test targets were provably excluded from the knowledge base, the main empirical result is invalid. The task mismatch between single-paper SciTLDR summarization and multi-paper literature review generation is a second fundamental gap. These are not local presentation issues; they require a new evaluation design and likely new experiments. I would not recommend acceptance in the current form, though a substantially revised submission with a clean held-out evaluation and a proper literature-review benchmark could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take: this is a clearly written engineering paper that builds three end-to-end pipelines for turning PDFs into a literature review segment, and compares them with ROUGE. The work is new in the narrow sense—none of the cited prior work appears to implement a full PDF/DOI-to-review pipeline with a UI. The pipeline descriptions are concrete, and the user-evaluation output is a coherent sample of what the system produces. The authors also engage with the relevant prior work on automating literature reviews.\n\nThe soft spot is the one the stress-test flagged, and it's decisive. Section III-D states that the SciTLDR dataset is added as the knowledge base for the GPT-3.5-turbo assistant, with retrieval enabled, and the prompt tells the model to use the data.json file. Section IV-C then evaluates that same assistant on the SciTLDR test split. The paper never says that test split source-target pairs were removed from the knowledge base. If they were not, the assistant can retrieve the gold summaries, so the reported ROUGE-1 of 0.364 would measure retrieval fidelity, not generation ability. That kills the central claim that the LLM-based system outperforms T5 and spaCy. There are also secondary issues: the paper reports point estimates with no error bars or split sizes, no code or data, and the conclusion overstates the result—on ROUGE-L, T5 (0.204) actually beats the LLM (0.181), so \"outperformed both on ROUGE scores\" is only true for ROUGE-1 and ROUGE-2.\n\nThe citation pattern is fine, and the authors give credit to the earlier T5/spaCy comparison they built on. I don't see any of the typical tricks; the contamination looks like an oversight, not a deliberate attempt to inflate.\n\nWho is this for? People building similar tools, and it could be a useful case study in evaluation pitfalls. It deserves a serious referee, because the system is real and the fix is clear: re-run the LLM evaluation after explicitly excluding test targets from the knowledge base, report the split details and error bars, and correct the ROUGE-L overclaim. If the authors do that, it could become a solid application paper. As it stands, I would not cite the empirical comparison in my own work.","headline":"A concrete but flawed comparison of NLP methods for automatic literature review generation; the main ROUGE claim is compromised by likely test-set contamination.","tokens_in":7599,"tokens_out":2735,"would_cite":false,"duration_ms":23485,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A retrieval-augmented GPT-3.5-turbo system beats a fine-tuned T5 and a frequency-based spaCy extractor at generating literature-review segments from PDFs, according to ROUGE scores on the SciTLDR test set.","keywords":["T5","SpaCy","Large Language Model","GPT","ROUGE","Literature Review","Natural Language Processing","Retrieval-augmented generation"],"falsifier":"Check whether any SciTLDR test reference summaries were present in the assistant's uploaded knowledge base; if so, remove them and rerun the ROUGE evaluation to see whether the 0.364 ROUGE-1 persists.","tokens_in":6662,"feed_emoji":"📚","tokens_out":5942,"duration_ms":45101,"temperature":0.7,"pith_summary":"This paper asks whether an automated system can generate a literature-review segment from nothing but the PDFs of related papers, and it builds three such systems to find out. It reports that a retrieval-augmented GPT-3.5-turbo pipeline beats a fine-tuned T5 model and a frequency-based spaCy extractor on the SciTLDR benchmark, with the highest ROUGE-1 score (0.364) of the three. If correct, this would mean a large language model with retrieval can serve as the backend of a practical, PDF-in/literature-review-out tool, which the paper then wraps in a graphical interface.","feed_headline":"Retrieval LLM beats T5 and spaCy at writing literature reviews","feed_subtitle":"On the SciTLDR benchmark, the LLM scores ROUGE-1 0.364 vs 0.268 for T5 and 0.257 for spaCy.","key_machinery":"The central mechanism is the retrieval-augmented generation (RAG) pipeline built around GPT-3.5-turbo. A custom assistant has retrieval enabled, and the SciTLDR dataset is loaded into its knowledge base; a prompt instructs the model to summarize a given PDF's extracted text in the style of a literature review, mentioning the first author and paper title, in at most 80 words. The system extracts full text from each PDF, submits it as a thread message, collects the assistant's per-paper summary, and merges the summaries into one coherent review segment. The other two pipelines, spaCy's frequency-based sentence scoring and a fine-tuned T5 summarizer, serve as baselines for comparison, and ROUGE-N, ROUGE-L, and ROUGE-Lsum are used as the shared evaluation metric.","core_discovery":"On the SciTLDR test split, the retrieval-augmented GPT-3.5-turbo system achieved a ROUGE-1 of 0.364, ROUGE-2 of 0.123, ROUGE-L of 0.181, and ROUGE-Lsum of 0.182. The T5 model scored 0.268, 0.115, 0.204, and 0.204; spaCy scored 0.257, 0.055, 0.144, and 0.146. The paper takes these numbers as evidence that the LLM-based approach outperforms both older NLP approaches, places T5 second, and uses the LLM pipeline as the backend of a user interface that accepts uploaded PDFs and produces a merged literature review.","pith_inferences":["If the SciTLDR test targets were present in the assistant's knowledge base, the reported ROUGE gap may come from retrieving reference summaries rather than from generation, and a stricter evaluation without those targets could shrink the gap.","ROUGE-1 rewards word overlap and says little about whether a generated review is coherent, complete, or useful to a researcher, so a blinded human judgment on the same outputs would be a more informative comparison.","A natural next experiment is to run the same three pipelines on papers published after the model's training cutoff, where no reference summary can leak through retrieval.","The current system merges per-paper summaries by concatenation; a pipeline that plans a shared narrative structure across the selected papers might improve readability and coverage."],"forward_implications":["A PDF-only input is enough for the RAG-LLM pipeline to produce literature-review-style summaries of individual papers.","On ROUGE-1 and ROUGE-2, the LLM system scores highest, so the paper concludes LLMs are the strongest of the three compared approaches for this task.","The T5 transformer, though second, still beats the frequency-based extractor on ROUGE-2 and ROUGE-L.","The LLM-backed UI lets a user upload several PDFs and receive a merged review that names each paper's first author and title.","The results imply that older frequency-based extractive summarization is the weakest of the three for this literature-review use case."],"supporting_citations":[{"why":"Introduces the SciTLDR dataset of scientific TLDRs that the paper uses for training and evaluation.","marker":"[3]"},{"why":"Provides the spaCy library used in the frequency-based extractive baseline.","marker":"[4]"},{"why":"Supplies the ROUGE metric definitions used to compare all three systems.","marker":"[5]"},{"why":"Distributes the SciTLDR dataset in the exact form loaded into the LLM knowledge base.","marker":"[13]"}],"fun_headline_variants":["LLM-based RAG tops T5 and spaCy in literature review generation","Retrieval-augmented GPT-3.5 outwrites T5 and spaCy for reviews","Auto lit reviews: LLM with retrieval beats T5 and spaCy","SciTLDR lit review: RAG LLM hits 0.364 ROUGE-1, best of three"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported ROUGE advantage is trustworthy only if the SciTLDR test-set summaries were not part of the assistant's retrieval knowledge base when it was evaluated; the paper adds the dataset to the assistant's knowledge base but does not state that test targets were excluded.","fun_headline_variants_meta":{"raw":{"variants":["LLM-based RAG tops T5 and spaCy in literature review generation","Retrieval-augmented GPT-3.5 outwrites T5 and spaCy for reviews","Auto lit reviews: LLM with retrieval beats T5 and spaCy","SciTLDR lit review: RAG LLM hits 0.364 ROUGE-1, best of three"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000315,"raw_usage":{"total_tokens":1798,"prompt_tokens":971,"completion_tokens":827,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":730}},"tokens_in":587,"tokens_out":827,"duration_ms":6727,"temperature":1.0,"reasoning_tokens":730,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:03:54.707827+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check whether any SciTLDR test reference summaries were present in the assistant's uploaded knowledge base; if so, remove them and rerun the ROUGE evaluation to see whether the 0.364 ROUGE-1 persists.","supporting_citations":[{"cited_title":"Extractive automatic text summarization using SpaCy in Python & NLP","cited_arxiv_id":null,"evidence_quote":"Provides the spaCy library used in the frequency-based extractive baseline."},{"cited_title":"ROUGE Score Analysis and Performance Evaluation Between Google T5 and SpaCy for YouTube News Video Summarization","cited_arxiv_id":null,"evidence_quote":"Supplies the ROUGE metric definitions used to compare all three systems."}],"review_version":1}