{"id":"768ca1a7-55a2-46a7-a882-cfb400fe63f3","arxiv_id":"2506.03704","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"ScoreRAG combines RAG with LLM-based consistency scoring, reranking, and score-graded summarization to generate news articles that beat a zero-shot baseline in reported quality scores.","lead":"This paper describes ScoreRAG, a multi-stage news generation system that retrieves articles, scores relevance with a language model, reranks, and produces score-graded summaries before generating a final story. The authors report that ScoreRAG beats a zero-shot baseline on accuracy, informativeness, and stability in both LLM and expert evaluations.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM judge is unnamed and potentially format-biased, and expert evaluation covers only 10 of 50 articles without selection details; the claim of significant gains in accuracy and informativeness is not yet established.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the expert subset is undocumented and the LLM evaluator is unspecified. My stress-test agrees and sharpens it: the central claim depends on evaluation validity, and the risk is concrete rather than stylistic. The manuscript gives the architecture of ScoreRAG in reasonable detail, and the pipeline components are coherent; the code link and structured summaries are positive signs. The problem is that the empirical evidence for the headline claim is under-specified at exactly the points where bias could enter. The LLM evaluation lacks a named model, and if the model is the same one used for summarization, it may prefer its own output style. The expert evaluation covers at most 20 ratings, with no sampling plan or agreement metric. The paper openly acknowledges misalignment between LLM and expert scores, which further weakens the chain from numbers to conclusion. This does not require rejecting the framework; it requires re-running the evaluation in a blinded, full-sample, pre-registered form. Since the reader already issued CONDITIONAL, my read does not change the verdict, so I keep it UNCHANGED.","tokens_in":7744,"tokens_out":3096,"duration_ms":29687,"concrete_test":"Have both expert journalists blindly rate all 50 article pairs (or a pre-registered random sample of 30) with citation markers and section headers stripped from ScoreRAG outputs so raters cannot distinguish the system, and report per-item scores plus inter-annotator agreement (Cohen's kappa). Simultaneously, run the same blind protocol with a specified LLM judge (e.g., GPT-4 or LLaMA-3.1-70B with a published prompt and temperature). If the ScoreRAG advantage on accuracy and informativeness persists at full sample with formatting removed and agrees across raters, the claim survives; if the advantage shrinks or disappears on the full sample or when formatting is removed, the reported superiority is an artifact of subset selection and format bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is that ScoreRAG significantly outperforms the Zero-shot baseline on accuracy and informativeness. This rests on two evaluation pillars: an LLM-based evaluation of all 50 outputs (Table 2, Figures 5-6) and an expert evaluation of a subset of 10 articles (Section 3.1.2, Table 3, Figures 7-8). The load-bearing weakness is that neither pillar is validated as measuring what it claims. First, the LLM judge is never named; Section 3.2.1 reports p-values without stating the model, prompt, temperature, or test. If the judge is the same LLaMA 3.1 8B model used in the pipeline (Section 2.2.4), or any model sensitive to formatting, the more structured, citation-rich ScoreRAG output may be scored higher for reasons unrelated to factual accuracy. Informativeness in particular correlates with length, and ScoreRAG deliberately produces longer graded summaries; a length confound is not controlled. Second, the expert evaluation covers only 10 articles (one reporter per 10, or 20 ratings total, as the text is ambiguous), with no description of how the subset was chosen, no inter-annotator agreement, and no per-item breakdown. With such a small, possibly selected sample, the 'notably higher' 3.83 vs. 3.08 average cannot support a general claim of significant superiority. The paper itself admits LLM and expert scores were 'sometimes misaligned' (Section 5), which undermines using unvalidated LLM scores as evidence. Therefore the central claim is not yet supported; it is a hypothesis requiring a blinded, full-sample evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ScoreRAG, a multi-stage retrieval-augmented generation (RAG) framework for generating Traditional Chinese news articles. It retrieves news chunks from a vector database (multilingual-e5 embeddings stored in ChromaDB), maps retrieved chunks back to full articles using news_id metadata, and applies a self-consistency LLM scoring procedure (LLaMA 3.1 8B, averaged over three evaluations) to rerank and filter documents. The framework then produces score-graded summaries and prompts a generation LLM to produce a news article with explicit per-reference citations. The experiments compare ScoreRAG against a zero-shot baseline on 50 target articles, using an LLM-based evaluation with four weighted criteria (coherence, accuracy, professionalism, informativeness) and a small expert evaluation by two journalists each reviewing a subset of 10 articles. The paper reports that ScoreRAG outperforms zero-shot across all dimensions, especially in informativeness and accuracy, and shows greater output stability.","tokens_in":8086,"tokens_out":5302,"duration_ms":45545,"significance":"If the reported results were fully supported, ScoreRAG would be a useful, practical contribution to grounded news generation: the multi-stage design is clearly motivated, and the open-source code and demo (GitHub repository) are valuable for reproducibility and follow-up work. However, the current evidence is not sufficient to establish the central claim of significant superiority over ungrounded generation. The evaluation lacks a RAG-only baseline, the LLM evaluator is unidentified, and the expert sample consists of only 10 articles without selection or agreement details. These gaps weaken the conclusion that consistency scoring and graded summarization, rather than the simple presence of retrieved context, drive the gains.","major_comments":[{"comment":"The only comparison is against a zero-shot baseline, so the reported gains cannot be attributed to ScoreRAG's specific components (consistency scoring, filtering, and graded summarization) rather than to simply providing retrieved articles in the prompt. Add at least one RAG-only baseline that retrieves the same top-k documents and inserts them either as full articles or as plain summaries without reranking or score-based grading, and report the same evaluation metrics on that baseline.","section":"Section 3.1.1"},{"comment":"The LLM evaluator is not described. The paper reports p-value ranges for informativeness and accuracy but does not state which model was used as judge, the prompt, temperature, decoding parameters, or number of repetitions. Since ScoreRAG outputs are longer and contain structured references such as '(Reference X)', a format- or length-sensitive judge could systematically favor them for reasons unrelated to factual accuracy. Specify the evaluator, include a length-controlled comparison or length as a covariate, and validate the LLM scores against the expert ratings, especially because Section 5 acknowledges that LLM and expert scores were sometimes misaligned.","section":"Section 3.2.1"},{"comment":"The expert evaluation is too small and underspecified to support the claim of significant superiority. The text says two journalists each independently reviewed a subset of 10 articles, but it does not state how the subset was chosen, whether the subsets overlapped, how disagreements were resolved, or what the inter-annotator agreement was. With only 10 articles (or 20 ratings), the reported averages of 3.83 versus 3.08 cannot establish statistical significance or generalizability. Provide the sampling procedure, agreement statistics, and per-article scores; if feasible, enlarge the expert sample.","section":"Section 3.1.2, Table 3, Figures 7-8"},{"comment":"The statistical reporting is incomplete. The paper does not state which significance test was used, whether the two systems were evaluated on the same 50 queries in a paired design, whether multiple comparisons were adjusted for, or what the effective sample size was in each criterion. Report the exact test, test statistics, effect sizes, and confidence intervals for both the LLM and expert evaluations.","section":"Section 3.2.1, Section 3.2.2"}],"minor_comments":[{"comment":"The introduction contains a duplicated, incomplete sentence: 'LLMs frequently suffer from hallucinations, factual In zero-shot or instruction-based generation settings, ...' Please rewrite this passage.","section":"Section 1"},{"comment":"The filtering threshold of 20 is described without justification or sensitivity analysis; please report how varying the threshold affects the downstream scores.","section":"Section 2.2.3"},{"comment":"The expert rubric defines anchors at 1, 3, and 5 but does not explain how scores of 2 and 4 should be assigned; clarify the intermediate levels.","section":"Section 3.1.2"},{"comment":"Report standard deviations, ranges, and the number of samples alongside the averages, and define precisely what 'no overlap' refers to in Figure 8 (e.g., interquartile ranges vs. full distributions).","section":"Tables 2 and 3, Figures 6 and 8"},{"comment":"There are frequent shifts between first-person singular ('I') and first-person plural ('we'); use a consistent style throughout the paper.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The central claim of significant improvement over zero-shot rests on two evaluation pillars that are not yet validated: an unnamed LLM judge and a 10-article expert sample with no selection or agreement details. The absence of a RAG-only baseline is a notable weakness for a paper whose title emphasizes 'retrieval-augmented generation.' The open-source code and demo are strengths, and the framework is plausible, but the evaluation must be substantially strengthened before the paper can be accepted. There is also a mismatch between the modest evidence and the broad generalization claims in Section 5 (fake news detection, educational content, financial news)."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. ScoreRAG is a sensible RAG pipeline for Traditional Chinese news generation: retrieve chunks, map to full articles, have an LLM score them three times for consistency-relevance, rerank and filter, then generate graded summaries and a guided article. The components are established, so the novelty is only in the specific combination. That is fine for an engineering paper, but the evaluation does not yet support the conclusion.\n\nWhat the paper does well: it targets a real problem (hallucination in automated news), the system is implemented, and code and demo are linked. The dual evaluation with LLM scores and two practicing journalists is a good-faith attempt at grounding, and the paper is honest in Section 5 that LLM and expert scores were sometimes misaligned. The consistency-scoring and graded-summarization ideas are reasonable and worth testing.\n\nThe soft spots are in the evidence. There is no RAG-only baseline, so you cannot tell whether gains come from grounding itself or from the scoring/summarization scheme. The LLM judge is never named; if it is the same model family as the generator, format bias and a length confound could drive the informativeness scores. Expert evaluation covers only 10 of 50 articles, with no description of how the subset was chosen, no inter-rater agreement, and p-values are reported without test details. These are fixable, but as it stands the central claim of significant superiority is not established.\n\nThis is an incremental, dataset-specific contribution—not a new capability or theoretical advance. The right reader is someone working on automated journalism or RAG pipelines who wants a concrete baseline or design pointer. I would not cite it in its current form, but after revision it could be a useful reference.\n\nIf this lands on my desk, I would send it to peer review rather than desk-reject, with a major-revision request: add a RAG-only baseline and ablations, specify the LLM judge and prompt, expand or clearly label the expert evaluation as a pilot, and tone down the conclusion. The core idea is coherent enough to deserve referee time.","headline":"Solid engineering pipeline for RAG-based news generation, but the evaluation is too weakly specified to back the headline claim of significant gains.","tokens_in":8580,"tokens_out":1831,"would_cite":false,"duration_ms":18012,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ScoreRAG grounds news generation in retrieved articles scored for consistency with the query, and its experiments show this produces more accurate, informative, professional, and stable articles than generating from a topic alone.","keywords":["Retrieval-Augmented Generation","News Generation","Large Language Models","Semantic Reranking","Graded Summarization","Hallucination Reduction","Traditional Chinese News","Self-Consistency Scoring"],"falsifier":"Take the 50 generated outputs, or a larger random sample from the same retrieval pool, and have at least three journalists who are blind to which outputs came from which method score every article on the four stated criteria with reported inter-annotator agreement. If the accuracy and informativeness gaps between ScoreRAG and the zero-shot baseline are not statistically significant in that blind scoring, or if an LLM evaluator awards the same high scores to any text simply because it contains 'Reference X' markers, the paper's central claim is falsified.","tokens_in":7556,"feed_emoji":"📰","tokens_out":7942,"duration_ms":72948,"temperature":0.7,"pith_summary":"ScoreRAG is a pipeline for automatic news writing that retrieves past news articles matching a topic, scores each retrieved article for how directly it addresses the query using repeated LLM judgments, removes weak matches, and writes summaries whose detail grows with the relevance score. The paper's central claim is that this grounded, relevance-controlled pipeline produces articles that are more accurate, informative, professional, and stable than asking the same model to write from the topic alone. In the paper's experiments on 50 generated articles, an LLM evaluator gave ScoreRAG an average total score of 4.64 against 4.34 for zero-shot generation, and two practicing journalists, each reviewing 10 articles, gave it 3.83 against 3.08. If true, this matters because the main obstacle to using language models in newsrooms is not fluency but factual reliability.","feed_headline":"News generator with scored retrieval beats zero-shot on accuracy","feed_subtitle":"Retrieving, scoring, and summarizing sources before writing lifts accuracy, informativeness, and stability in news articles.","key_machinery":"The consistency relevance score is the central object: a 0-100 number produced by averaging three LLM assessments of how directly each retrieved article matches the query. It does three jobs in the pipeline: it reranks the retrieved chunks, it filters out articles scoring below 20, and it sets the level of detail for each article's summary. The graded summary is the second mechanism, with four depth tiers tied to the score bands above 70, 50-70, 30-50, and 20-30, so that context tokens are concentrated on the most relevant sources. Mapping each retrieved chunk back to its complete article through stored identifiers prevents the fragmentary context that would otherwise dilute generation.","core_discovery":"ScoreRAG claims that a retrieval-augmented generator becomes substantially more reliable when a consistency-relevance judgement is inserted between retrieval and generation. Each retrieved article gets a score from 0 to 100, computed as the average of three LLM evaluations of how directly the article's date, title, and summary address the query. Articles scoring below 20 are filtered out, and the survivors are summarized at a depth set by their score: high-scoring articles keep key data, quotes, background, and impact assessments, while low-scoring articles are reduced to core facts. The generator is then instructed to write a Traditional Chinese news article with explicit references, basing all key facts and data on the scored sources. The reported experiments show this design outperforming a zero-shot baseline on every evaluated dimension, with statistically significant gains in accuracy and informativeness, and expert journalists judging its professionalism higher.","pith_inferences":["The paper does not test whether three evaluation passes are necessary; a single pass with score variance used as an uncertainty signal might reach the same reranking quality at one-third of the scoring cost.","The graded-summary idea generalizes to any generation task with limited context and uneven source relevance, including the financial and educational applications the paper lists as future work.","Because the LLM evaluator saw the structured ScoreRAG outputs in full, part of the reported gap may reflect the presence of citations and headings rather than improved facts; a human evaluation blind to method would separate those factors.","The small expert sample is an invitation to re-run the comparison on a larger, stratified set of topics; the framework's value depends on whether the accuracy and informativeness gains replicate outside the 50 evaluated outputs."],"forward_implications":["Newsroom systems could rely on grounded generation to reduce post-editing effort, because each paragraph is tied to a numbered reference drawn from scored source articles.","Generation context budgets become controllable: high-relevance sources receive detailed summaries, low-relevance sources are stripped to facts, so token limits do not force a uniform loss of evidence.","The lower score variance reported for ScoreRAG means publication quality is more predictable, so editorial review can concentrate on rare anomalies rather than pervasive errors.","The expert evaluation result indicates that the method does not just add citations; it moves the writing style closer to what professional journalists expect."],"supporting_citations":[{"why":"Supplies the retrieval-augmented generation grounding approach that ScoreRAG builds on.","marker":"[5]"},{"why":"Supplies the retrieve-rerank-generate pattern used before summarization.","marker":"[6]"},{"why":"Supplies the self-consistency mechanism: repeated LLM evaluations are averaged to stabilize relevance scores.","marker":"[8]"},{"why":"Supplies the multilingual embedding model used to index and search Chinese news articles.","marker":"[9]"},{"why":"Supplies the document store keyed by news_id that maps retrieved chunks back to full articles.","marker":"[15]"}],"fun_headline_variants":["Scored retrieval before writing beats zero-shot on news accuracy","Score sources, rerank, then summarize: better news from RAG","Consistency scoring in RAG slashes news hallucinations and boosts accuracy","News generator that scores sources before writing outperforms zero-shot","Grade sources, filter weak ones, then write: ScoreRAG lifts news quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported superiority rests on the assumption that the 10 articles scored by two journalists are a representative sample of the 50 generated outputs, and that the unspecified LLM evaluator does not systematically favor the more structured ScoreRAG format.","fun_headline_variants_meta":{"raw":{"variants":["Scored retrieval before writing beats zero-shot on news accuracy","Score sources, rerank, then summarize: better news from RAG","Consistency scoring in RAG slashes news hallucinations and boosts accuracy","News generator that scores sources before writing outperforms zero-shot","Grade sources, filter weak ones, then write: ScoreRAG lifts news quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1402,"prompt_tokens":915,"completion_tokens":487,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":396}},"tokens_in":531,"tokens_out":487,"duration_ms":5950,"temperature":1.0,"reasoning_tokens":396,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:56:10.527767+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 50 generated outputs, or a larger random sample from the same retrieval pool, and have at least three journalists who are blind to which outputs came from which method score every article on the four stated criteria with reported inter-annotator agreement. If the accuracy and informativeness gaps between ScoreRAG and the zero-shot baseline are not statistically significant in that blind scoring, or if an LLM evaluator awards the same high scores to any text simply because it contains 'Reference X' markers, the paper's central claim is falsified.","supporting_citations":[{"cited_title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","cited_arxiv_id":null,"evidence_quote":"Supplies the retrieval-augmented generation grounding approach that ScoreRAG builds on."},{"cited_title":"Re2g: Retrieve, rerank, and generate for factual open-domain question answering","cited_arxiv_id":null,"evidence_quote":"Supplies the retrieve-rerank-generate pattern used before summarization."},{"cited_title":"Self-consistency improves chain of thought reasoning in language models","cited_arxiv_id":null,"evidence_quote":"Supplies the self-consistency mechanism: repeated LLM evaluations are averaged to stabilize relevance scores."},{"cited_title":"MongoDB: The developer data platform.https://www.mongodb.com/","cited_arxiv_id":null,"evidence_quote":"Supplies the document store keyed by news_id that maps retrieved chunks back to full articles."}],"review_version":1}