{"id":"91b4e8a6-19a1-4c9b-806d-a6a32c15ddb2","arxiv_id":"1908.06216","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"NDORGS, a pipeline of filtering, single-document summarization, LDA clustering, multi-document summarization, and title generation, produces two-level overview reports from thousands of documents, and a TOPSIS evaluation finds 20 percent length summaries best on two corpora.","lead":"This paper presents NDORGS, a system that turns thousands of news articles into a sectioned overview report by summarizing each article, grouping the summaries by topic, and summarizing each group. The authors' evaluation says reports built from summaries at 20 percent of the original length are the best across human ratings, speed, and topic coverage.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unreported AHP weight matrix makes the TOPSIS-based '0.2 best' claim uncheckable; Table 4 raw scores alone do not support it.","rationale":"The paper's central claim is a TOPSIS ranking, so the missing AHP weights are not a peripheral detail. Table 4's raw scores do not obviously prefer lambda=0.2: on BBC News, human evaluation favors 0.3 and running time favors 0.1; on Factiva-Marx, 0.2 wins human evaluation and diversity but loses time and coverage. The 'best overall' conclusion is therefore produced by the choice of weights. The sensitivity analysis in Fig. 9 starts from an unreported base weight vector and only explores 0.02 increments; it cannot establish the unconditional claim, nor does it show the ranking is stable over the whole preference-consistent weight region. In fact, as the human-evaluation weight tends to 1, the BBC ranking must converge to the human-only ranking with lambda=0.3 first, so Fig. 9's claimed stability is inherently range-limited. This is the same reason the reader assigned CONDITIONAL; with the AHP matrix disclosed and robustness checks over the weight simplex, the claim could be supported. No issue with the system architecture or internal consistency of the pipeline was found; the datasets and sample outputs are provided, which is helpful. Thus I keep CONDITIONAL and see no reason to change the reader's verdict.","tokens_in":16495,"tokens_out":7821,"duration_ms":75254,"concrete_test":"Recompute the TOPSIS ranking from Table 4 using vector normalization and sample the weight simplex consistent with the stated preference order (w_human > w_time > w_coverage > w_diversity > 0), recording the fraction and region of the simplex for which lambda=0.2 is rank 1 on each dataset. If the AHP matrix is recovered from the authors, check that its weight vector falls inside that region; if the matrix is unavailable, this grid search alone determines whether the unreported weights are decisive. A large winning region would answer the concern; a small or empty region would show the headline is an artifact of one weight choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is a statement about a TOPSIS ranking, but the input to that ranking—the 4x4 Saaty pairwise-comparison matrix and the resulting AHP weight vector—is never reported in Section 4.5 or Table 4. The raw scores in Table 4 do not entail the headline: on BBC News, the 0.3-report has the best human score (4.03 vs 3.71) and the 0.1-report the best time (3310s vs 5060s), while 0.2 wins only on the two self-defined text-mining metrics; on Factiva-Marx, 0.2 wins on human and diversity but loses on time and coverage. Which lambda is 'best overall' is therefore a function of four weights, and the abstract's unqualified claim is not checkable without them. The sensitivity analysis (Fig. 9) starts from an unreported base weight vector and varies each weight in 0.02 increments, so at most it shows local stability within an unspecified neighborhood; it cannot show the ranking is robust across the preference-consistent simplex. Analytically, as w_human approaches 1 the TOPSIS ranking must approach the human-only ranking, which on BBC News puts 0.3 first, so any claimed stability is range-limited. The missing AHP matrix is thus load-bearing: it is the difference between a reproducible multi-criteria decision and a claim that depends on an unspecified parameter choice.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents NDORGS, a five-module pipeline (preprocessing, hierarchical topic clustering, cluster summarizing, cluster titling, and report generating) for producing a two-level overview report from thousands of documents. The pipeline uses Semantic WordRank for single-document summaries at length ratios 0.1, 0.2, and 0.3, LDA-based hierarchical clustering, GFLOW for cluster summaries, and DTATG for title generation. The authors evaluate six generated reports (three length ratios on BBC News and three on Factiva-Marx) using DUC-style human ratings, running time, and self-defined information coverage and topic diversity scores, then combine these four criteria with AHP-derived weights and TOPSIS to claim that 0.2-summaries produce the best overall report on both datasets. The paper also includes a sensitivity analysis and trending graphs.","tokens_in":16776,"tokens_out":5730,"duration_ms":53459,"significance":"If the evaluation were fully specified and reproducible, the paper would contribute a useful modular architecture for large-corpus overview generation and a concrete practical finding about the SDS compression ratio in such pipelines. The provision of the six generated reports at ndorg.net is a strength, as is the explicit use of a multi-criteria decision framework rather than a single metric. However, the headline result currently rests on an unreported AHP weight vector and on self-defined metrics that have not been validated, so the practical recommendation is not yet reliably established.","major_comments":[{"comment":"The headline claim that 0.2-summaries are best overall is not reproducible because the Saaty pairwise comparison matrix and the resulting AHP weight vector are never reported. The raw scores in Table 4 do not imply the stated ranking: on BBC News, the 0.3 report has the best human evaluation (4.03 vs. 3.71), the 0.1 report has the best running time (3310 s vs. 5060 s), and 0.2 wins only on the two self-defined text-mining metrics; on Factiva-Marx, 0.2 wins on human and diversity but loses on time and coverage. Consequently, which alternative TOPSIS selects is entirely determined by the four undisclosed weights. The sensitivity analysis in Fig. 9 starts from an unstated base weight vector and perturbs weights in 0.02 increments, so it demonstrates only local stability; as the weight on human evaluation approaches 1, TOPSIS must converge to the human-only ranking, which on BBC News places 0.3 first. Please report the full pairwise comparison matrix, the derived weight vector, and the region of the weight simplex in which 0.2 is ranked first.","section":"§4.5, Table 4"},{"comment":"The human evaluation component has four annotators per report and no inter-annotator agreement measure or significance test. The raw scores in Appendix Table 5 show large disagreement, for example BBC-0.1 receives scores of 1, 2, 3, and 4 on the UselessText criterion, so the reported means may not be stable. Since human evaluation is the highest-weighted criterion in the AHP/TOPSIS procedure, the absence of agreement statistics and inferential tests undermines the comparative claim between λ=0.2 and λ=0.3. Please report per-criterion annotator agreement, such as Krippendorff's alpha, and at minimum a paired or bootstrap test for the human-score differences.","section":"§4.5, Human evaluations"},{"comment":"The information coverage score S_k(A,B) and the topic diversity score based on CSD F1 are author-defined measures, and the value of k is never stated in Section 4.5 even though the appendix lists top 50 words. These two metrics are the only criteria on which the 0.2 report beats the 0.3 report on BBC News, so the central ranking depends on metrics that have not been validated against reading quality or shown to be stable under the choice of k. Please report the exact k used, justify the choice, and provide evidence that these text-mining proxies correlate with human judgments or at least are robust to k in a reasonable range.","section":"§4.5, Information coverage and Topic diversity"},{"comment":"The cluster score formula is garbled: it reads SC = 1/n^2 ∑_{j=i} 2p_i, with an undefined summation index and an unexplained factor of 2. Because cluster scores determine the ordering of sections and subsections in the Report Generating module, this is part of the algorithm's specification and must be corrected, presumably to a sum over i of 2p_i or similar, with p_i's role clarified. Without a correct formula, the structure of the generated reports cannot be reproduced.","section":"§3, Step 2"},{"comment":"The evaluation compares only three configurations of NDORGS (λ=0.1, 0.2, and 0.3) and does not include any baseline system, such as T-DMCA [25], a flat MDS pipeline, or a random/lead-based summary. Consequently, the experiments cannot support the contribution that NDORGS is an effective approach for large-corpus overview generation; they only rank the system's own compression choices. Please add at least one competitive baseline or a meaningful ablation, such as a non-hierarchical MDS report or a pipeline without title generation, and evaluate it with the same criteria.","section":"§4, Evaluations"}],"minor_comments":[{"comment":"The system name is written as T-CMDA in the Introduction but as T-DMCA in Section 2.2.2; please use a single consistent acronym for reference [25].","section":"§1 and §2.2.2"},{"comment":"The text says 'Parameters α, β, and λ are learned using the DUC-03 dataset,' but λ is elsewhere the SDS length ratio that the authors set to 0.1, 0.2, and 0.3, and the displayed ILP formulation does not contain λ. Please clarify what λ means in the GFLOW parameters and how this relates to the three fixed values used in the experiments.","section":"§4.2"},{"comment":"The caption states that the x-axis shows increments/decrements of 0.02 each time, but the plotted x-axis values appear to be absolute weight values starting near 0. Please mark the base weight vector and the direction of perturbation on the figure.","section":"Fig. 9 caption"},{"comment":"Table 5 lists individual annotator scores, but the main text reports a single human-evaluation mean in Table 4; please state explicitly how the seven criterion scores are aggregated into that single number and whether all seven criteria are weighted equally.","section":"Appendix Table 5"}],"recommendation":"major_revision","confidential_remarks":"The central claim is currently under-specified because the AHP weight matrix is missing, and the sensitivity analysis is local in nature; the analytic counterexample with the human-evaluation weight approaching 1 should be addressed. The paper would also benefit from stronger statistical rigor in the human evaluation and from at least one baseline comparison. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is an honest engineering paper: it builds a two-level overview report over thousands of documents using existing components (SWR, LDA, GFLOW, DTATG) and it works. Second, the central empirical claim—that lambda=0.2 gives the best report—is not actually supported by the evidence as presented. The TOPSIS ranking depends on a 4x4 Saaty weight matrix that is never shown. On BBC News the raw human score for lambda=0.3 is 4.03 vs 3.71 for lambda=0.2; only the two self-defined text-mining metrics favor 0.2. So the \"best overall\" label is a function of unreported weights.\n\nWhat the paper does well: the task is real and under-served; scaling summary generation to thousands of documents with a two-level structure is worth trying. The pipeline description is clear, the two datasets are public, and the six generated reports are hosted, so someone could re-annotate or re-run. The clustering F1 comparison for choosing K is a reasonable empirical step. They also report running times, which is more than many summarization papers do.\n\nThe soft spots are real but not fatal to the whole paper. There is no baseline at all: no T-CMDA, no SUMMA, no flat MDS, no human-written report. The lambda comparison only pits three configurations of the same pipeline against each other, so it tells you the best lambda within NDORGS, not whether NDORGS beats anything. Four annotators per report is thin, and there are no significance tests. The coverage and diversity metrics are constructed by the authors, and the coverage scores are close (0.52 vs 0.54 on BBC; 0.70 vs 0.64 vs 0.70 on Marx), so the ranking partly hinges on small, self-defined differences. The sensitivity analysis starts from an unspecified weight vector and varies each weight by 0.02 increments, so it only shows stability in a small neighborhood—not that 0.2 is robust across preference-consistent weights. The abstract's unqualified \"best overall\" is an overstatement.\n\nWho is this for? Anyone building report-generation systems for analysts or journalists; the architecture and parameter choices are useful even if the evaluation is underpowered. The paper deserves a serious referee, but a revised version needs baselines, more annotators, a significance test, and full disclosure of the AHP matrix. I would not cite the lambda=0.2 claim, but I would cite the pipeline if I worked on hierarchical summarization.","headline":"A coherent pipeline for a real task, but the headline claim about lambda=0.2 is not reproducible until the AHP weight matrix is reported; raw human scores on BBC favor lambda=0.3.","tokens_in":17346,"tokens_out":2288,"would_cite":false,"duration_ms":23162,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A five-module pipeline called NDORGS generates structured overview reports over thousands of documents, and its 20% summary-length setting ranks best overall.","keywords":["multi-document summarization","overview report generation","hierarchical topic clustering","single-document summarization","TOPSIS","multi-criteria evaluation","Latent Dirichlet Allocation","large-scale text summarization"],"falsifier":"Recompute the TOPSIS ranking for the classified corpus with the pairwise-comparison weight vector made explicit: because the 0.3 report has the highest human-evaluation score (4.03 versus 3.71), there is a computable threshold on the human-evaluation weight at which the 0.3 report overtakes the 0.2 report, and the claim that the decision is stable fails if that threshold lies inside the weight range covered by the sensitivity analysis.","tokens_in":16276,"feed_emoji":"📑","tokens_out":17194,"duration_ms":139978,"temperature":0.7,"pith_summary":"The paper describes NDORGS, a five-module pipeline for turning thousands of related documents into a well-structured, multi-level overview report, a task existing multi-document summarization systems do not address. The pipeline filters noise, compresses each document to a fixed length fraction $\\lambda$, clusters the compressed documents into a two-level topic hierarchy, summarizes each cluster, generates section titles, and assembles the report in order of topic salience. To decide how much compression works best, the authors combine human judgments, running time, information coverage, and topic diversity into a single ranking using a 9-point pairwise-comparison weighting and the TOPSIS decision method. Their central finding is that on both a 2,225-article classified corpus and a 5,300-article unclassified corpus, the reports built from 20% single-document summaries rank best overall, and a sensitivity analysis indicates the ranking survives weight perturbations. If that finding holds, NDORGS gives a practical recipe for producing bounded, multi-level overview reports at a scale no prior summarization scheme was designed for.","feed_headline":"20% summaries win the overview-report comparison","feed_subtitle":"A five-stage pipeline condenses thousands of articles into a report; the 20% setting ranks first on both datasets.","key_machinery":"The load-bearing object is the $\\lambda$-summary: each original document is first compressed by a single-document summarizer to a fixed length fraction $\\lambda$ of itself, and that compressed set is what the rest of the pipeline clusters and re-summarizes. $\\lambda$ acts as the information budget for the whole report, because all later stages see only these summaries. The two-level topic hierarchy is built by LDA, with second-level sub-clusters created whenever a top-level cluster exceeds a preset size; each node's cluster score orders sections by salience. GFLOW then produces a coherent multi-document summary for each cluster, DTATG generates the section and subsection titles, and the evaluation uses a 9-point pairwise-comparison weighting of the four criteria feeding into TOPSIS, which ranks the three $\\lambda$ alternatives by closeness to an ideal solution.","core_discovery":"On the paper's own terms, the discovery is that the NDORGS configuration — filtering plus Semantic WordRank single-document summarization at $\\lambda = 0.2$, two-level hierarchical clustering with LDA, GFLOW cluster summarization, DTATG title generation, and salience-ordered report assembly — is the best overall choice on both test corpora. On BBC News the 0.2 report has the best information-coverage score (0.54), the best topic-diversity score (0.1444), and the middle human-evaluation score; on Factiva-Marx it has the best human-evaluation score (4.03) and the best diversity score (0.1167). The paper therefore claims that $\\lambda = 0.2$ is the best overall setting under its four-criteria ordering, and that the TOPSIS ranking is stable when criterion weights are adjusted one at a time.","pith_inferences":["The 'best overall' label is tied to the authors' criterion ordering: because human annotators scored the 0.3 report above the 0.2 report on the classified corpus (4.03 vs 3.71), a stakeholder who weights readability heavily would likely prefer $\\lambda = 0.3$, so the result should be read as best under the stated weights, not best in an absolute sense.","Treating $\\lambda$ as an information budget suggests a principled extension: set $\\lambda$ as a function of corpus size, average document length, and desired report length rather than grid-searching three fixed values.","Because coverage and diversity are measured as top-word overlap and cluster-overlap scores, both metrics reward lexical and topical resemblance to the corpus; substituting a semantic or human-rated novelty metric could reorder the three candidates and is a cheap robustness test.","The pipeline is modular, so the $\\lambda$ result is not a claim about any single component: swapping the single-document summarizer, clustering method, multi-document summarizer, or title generator while keeping $\\lambda = 0.2$ could change the ranking, and jointly searching module choice and $\\lambda$ is a natural next experiment."],"forward_implications":["A corpus of several thousand documents can be reduced to a bounded two-level overview report in hours of CPU time on a commonplace desktop, with sections ordered by salience and each section carrying an automatically generated title.","For teams that prioritize speed, $\\lambda = 0.1$ is the fastest configuration, while for readability alone the BBC News human scores favor $\\lambda = 0.3$; the $\\lambda = 0.2$ recommendation is specifically the best balance of the four weighted criteria.","The observation that the much larger unclassified corpus preferred a smaller $\\lambda$ suggests that optimal compression should scale down as corpus size and document length grow, so deployments on even larger corpora should start near $\\lambda = 0.2$ or below.","The evaluation protocol itself — standard multi-document summarization quality scoring plus text-mining proxies for coverage and diversity, aggregated by TOPSIS — can be reused to compare other summarization pipelines when reading the whole corpus is infeasible."],"supporting_citations":[{"why":"Supplies the single-document summarizer used to produce the length-ratio summaries that define the three compared configurations.","marker":"[47]"},{"why":"Supplies the multi-document summarization algorithm that produces coherent summaries for each cluster.","marker":"[6]"},{"why":"Supplies the topic model used for the two-level hierarchical clustering that forms the report's section structure.","marker":"[2]"},{"why":"Supplies the title generator that creates the section and subsection headings.","marker":"[40]"},{"why":"Supplies the TOPSIS multi-attribute decision method that ranks the three alternatives in the overall evaluation.","marker":"[17]"},{"why":"Supplies the benchmark whose evaluation criteria shape the human scoring of report quality.","marker":"[9]"},{"why":"Supplies the quality questions that human annotators followed when scoring each report.","marker":"[8]"},{"why":"Provides the classified corpus of 2,225 articles, one of the two datasets on which the $\\lambda = 0.2$ result is demonstrated.","marker":"[14]"},{"why":"Provides the unclassified corpus of 5,300 articles, the second dataset on which the $\\lambda = 0.2$ result is demonstrated.","marker":"[11]"}],"fun_headline_variants":["Overview reports: 20% summaries rank best across corpora","Five-stage pipeline: 20% summary length wins both tests","NDORGS: 20% summaries top evaluation on both corpora","Best overview reports use 20% summaries, study finds","Two corpora, one winner: 20% summaries in NDORGS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that the 20% setting is best overall rests on the authors' four evaluation criteria and the weights assigned through pairwise comparisons; if those weights shift toward human readability, the 20% report no longer wins on at least one dataset.","fun_headline_variants_meta":{"raw":{"variants":["Overview reports: 20% summaries rank best across corpora","Five-stage pipeline: 20% summary length wins both tests","NDORGS: 20% summaries top evaluation on both corpora","Best overview reports use 20% summaries, study finds","Two corpora, one winner: 20% summaries in NDORGS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000578,"raw_usage":{"total_tokens":2713,"prompt_tokens":924,"completion_tokens":1789,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":1699}},"tokens_in":540,"tokens_out":1789,"duration_ms":11435,"temperature":1.0,"reasoning_tokens":1699,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:52:27.653550+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the TOPSIS ranking for the classified corpus with the pairwise-comparison weight vector made explicit: because the 0.3 report has the highest human-evaluation score (4.03 versus 3.71), there is a computable threshold on the human-evaluation weight at which the 0.3 report overtakes the 0.2 report, and the claim that the decision is stable fails if that threshold lies inside the weight range covered by the sensitivity analysis.","supporting_citations":[{"cited_title":"Zhang and J","cited_arxiv_id":null,"evidence_quote":"Supplies the single-document summarizer used to produce the length-ratio summaries that define the three compared configurations."},{"cited_title":"Christensen, Mausam, S","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-document summarization algorithm that produces coherent summaries for each cluster."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the topic model used for the two-level hierarchical clustering that forms the report's section structure."},{"cited_title":"Shao and J","cited_arxiv_id":null,"evidence_quote":"Supplies the title generator that creates the section and subsection headings."},{"cited_title":"Hwang and K","cited_arxiv_id":null,"evidence_quote":"Supplies the TOPSIS multi-attribute decision method that ranks the three alternatives in the overall evaluation."},{"cited_title":"Document understanding confer- ence","cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark whose evaluation criteria shape the human scoring of report quality."},{"cited_title":"Duc2004 quality questions","cited_arxiv_id":null,"evidence_quote":"Supplies the quality questions that human annotators followed when scoring each report."},{"cited_title":"Greene and P","cited_arxiv_id":null,"evidence_quote":"Provides the classified corpus of 2,225 articles, one of the two datasets on which the $\\lambda = 0.2$ result is demonstrated."},{"cited_title":"Marx dataset","cited_arxiv_id":null,"evidence_quote":"Provides the unclassified corpus of 5,300 articles, the second dataset on which the $\\lambda = 0.2$ result is demonstrated."}],"review_version":1}