{"id":"ff5a31ca-0c7b-4ba1-96e2-a346e2a04904","arxiv_id":"1908.11723","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Across nine corpora, summarization bias toward position, importance, and diversity differs by domain and by system type, with news showing strong position bias and academic papers showing balance.","lead":"This paper breaks summarization into three sub-functions, position, importance, and diversity, and measures how much each is used by nine datasets and many summarization systems. It finds news and personal posts lean on sentence position, while academic papers and meetings are more balanced, and simple ensembles can match single systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validating aspect algorithms is the crux: ConvexFall's outlier-prone diversity measure and N-Nearest's centrality-based importance are unvalidated, so the cross-corpus bias comparisons may be algorithm artifacts.","rationale":"Good-faith reading: the paper is an empirical survey aimed at measuring the extent to which position, importance, and diversity biases characterize nine corpora and a set of summarization systems. The position finding is well supported: First-k dramatically beats random on news (CNNDM R 30.7 vs 19.1; Newsroom 32.2 vs 10.1) and not on PubMed (27.6 vs 30.1) or AMI (11.4 vs 12.0). The diversity and importance measurements, however, depend entirely on the ConvexFall and N-Nearest algorithms. ConvexFall is admitted to select outliers; N-Nearest is an unvalidated centrality heuristic. The paper provides no external validation that these algorithms isolate the intended aspects. The internal contradiction between the abstract ('not the case for academic papers') and Section 6.4 ('PeerRead ... highly related to the position') suggests the qualitative claim is fragile at the margins, but this is secondary. The load-bearing issue is operationalization validity. A focused validation study with human aspect-specific gold summaries would settle whether the cross-corpus diversity and importance comparisons are meaningful. Since the reader's conditional verdict already identifies this assumption as the weakest point, our read does not change the verdict.","tokens_in":18401,"tokens_out":7692,"duration_ms":67326,"concrete_test":"Annotate a sample (e.g., 100 documents across CNNDM, PubMed, and AMI) with human gold summaries produced under each aspect instruction: select the k most important sentences, the k most diverse sentences, and the k lead sentences. Then recompute the corpus-bias values (ROUGE/SO) for ConvexFall, N-Nearest, and First against these aspect-specific gold summaries. If ConvexFall's agreement with the diversity gold summaries is no higher than with the importance gold summaries (or is near random), the diversity bias results in Figure 1 are artifacts and the central claim must be restricted to position. Additionally, compare against MMR with a sweep of diversity weights and against TextRank to test robustness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that sub-aspect biases vary across corpora rests on the assumption that the three algorithm families in Section 3 each isolate their intended aspect. The position algorithms are transparent, but the diversity and importance operationalizations are not validated. In Section 3.2, ConvexFall selects convex-hull vertices in BERT embedding space to maximize semantic volume; the authors themselves note in Section 6.1 that it 'chooses the outlier sentences over the convex hull.' The pruning step (keeping the sentence with the lowest volume reduction ratio) is left underspecified and is motivated by Marcu's waterfall phenomenon, not by a demonstrated link to extractive diversity. Similarly, Section 3.3 defines importance as maximum average Pearson correlation to all other sentences (N-Nearest), a centrality measure that may select generic or repetitive sentences rather than summary-relevant ones. No human annotation, no comparison to established aspect-specific baselines (e.g., MMR with varying diversity weights, TextRank centrality), and no ablation against alternative sentence encodings is provided. Since the corpus bias analysis in Section 6 and Figure 1 measures each aspect by the ROUGE score of these heuristic extractive summaries against target summaries, a low ROUGE for ConvexFall could simply mean that outlier selection is a poor extractive strategy for that corpus, not that the corpus lacks a diversity bias. The same reasoning applies to the system-bias measurements in Section 7. Thus the headline qualitative difference (news versus non-news, position versus diversity/importance) is secure only for the position aspect; the diversity and importance comparisons are not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper characterizes summarization behavior through three sub-aspects—position, importance, and diversity—and measures each by the ROUGE score of a heuristic extractive algorithm (First, ConvexFall, N-Nearest) against target summaries on nine corpora and against system outputs on a further set of systems. It reports that news and personal posts are strongly position-biased, academic papers are more balanced across aspects, and meeting, book, and movie-script summarization is very difficult. It also reports system-level aspect biases and finds that simple ensembles of aspect systems perform comparably to single-aspect baselines. The central empirical claim is that sub-aspect biases vary systematically across domains and across system families.","tokens_in":18695,"tokens_out":5730,"duration_ms":51933,"significance":"The study's breadth is its main strength: it compares position, importance, and diversity biases across nine domains and multiple system families in a single framework, and the position aspect is transparently defined via First, Last, and Middle baselines. The oracle-extraction and sentence-overlap controls are sensible for extractive comparisons. If the diversity and importance operationalizations are valid, the results would give useful guidance for dataset collection and system design. However, the diversity and importance algorithms are geometric proxies whose construct validity is not established, and at least one headline conclusion ('neural systems rarely have bias') is contradicted by the paper's own table. The paper would be substantially strengthened by validation studies for the aspect algorithms and by uncertainty estimates for the cross-corpus comparisons.","major_comments":[{"comment":"The diversity sub-aspect is operationalized solely as convex-hull volume in BERT embedding space, with no validation that hull vertices or the pruned subset correspond to diverse summary content. The paper itself states in §6.1 that ConvexFall 'chooses the outlier sentences over the convex hull,' yet the corpus-level diversity bias in Figure 1(a) is derived from the ROUGE score of this same algorithm. Without a validation against human annotations or against a controlled diversity baseline such as MMR with varying redundancy penalties, a low diversity score for a corpus can simply mean that hull-vertex extraction is a poor extractive strategy for that corpus, not that the corpus lacks diversity bias.","section":"§3.2, Table 2, §6.1"},{"comment":"The importance aspect is measured by N-Nearest and K-Nearest average Pearson correlations, a centrality heuristic that may select generic or repetitive sentences rather than summary-relevant ones. No comparison is made with established importance baselines such as LexRank, TextRank, or the MMR relevance term, and no ablation on the sentence encoder is provided. Because the system-bias columns R(P/D/I) in Table 4 are ROUGE scores between system outputs and these aspect-algorithm summaries, the system-bias analysis inherits the validity of the heuristics; an unvalidated importance proxy makes the reported system-level importance biases (e.g., MMR's high importance on XSum and Reddit) difficult to interpret.","section":"§3.3, Table 4"},{"comment":"The volume-overlap metric is computed by reducing sentence embeddings to 2D PCA before computing convex-hull overlap, but the paper gives no evidence that 2D hull overlap preserves the intended semantic-volume relation. The very high VO values for ConvexFall (e.g., 98.6 on AMI, 99.7 on BookSum, 93.5 on PubMed) likely reflect hull vertices spanning the 2D projection rather than semantic overlap, and VO is undefined for single-sentence target summaries such as XSum and Reddit. These values should be accompanied by a sensitivity analysis over PCA dimensions or replaced by a high-dimensional overlap measure.","section":"§4, Eq. (3), footnote 6"},{"comment":"The conclusion that 'neural systems rarely do [have bias]' is contradicted by Table 4: on CNNDM, CL, SumRun, and S2SExt have position components of 86, 68, and 74, respectively, with much smaller diversity and importance components. This is a load-bearing claim in the system-bias analysis and should be reworded or reconciled with the reported numbers.","section":"§7, Table 4, Conclusion"},{"comment":"All corpus-bias comparisons are reported as point estimates without confidence intervals or significance tests. Several corpora are small (Reddit has 404 training documents and AMI has 98), and some are preprocessed by truncation (BookSum uses the first 1,000 source sentences and first 50 target sentences), so the cross-corpus differences highlighted in Figure 1(a) may not be stable. The main claims, especially the news-versus-academic and news-versus-meeting position findings, should include bootstrap intervals or per-document variance.","section":"§6, Table 1"}],"minor_comments":[{"comment":"The Reddit Last-k ROUGE-1 score is listed as 116.4, which is impossible for an F-measure and appears to be a typo for 16.4.","section":"Table 5"},{"comment":"The captions use 'KN' where the text and Figure 4 use 'NN' for the N-Nearest algorithm; the notation should be made consistent.","section":"Figures 7 and 8"},{"comment":"The text says ConvexFall 'guarantees the maximum volume size,' but the subsequent pruning step removes this guarantee; the claim should be stated as applying only before pruning.","section":"§3.2"},{"comment":"The paper does not state whether code and data splits will be released; a reproducibility statement would strengthen confidence in the empirical claims.","section":"General"},{"comment":"Minor typos include 'anarphorcity' in the Related Work section, 'V enn' in the Appendix B heading, and 'the the sub-aspects' in the Figure 6 caption.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope and addresses an interesting empirical question. The core problem is construct validity of the diversity and importance measures, plus a few claims that are contradicted by the paper's own tables. I believe these issues can be fixed within the scope of a revision through validation experiments and by softening or correcting the unsupported claims. I did not see any issue with citation practices or novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a genuinely useful empirical survey, and the breadth is the contribution: nine corpora, fourteen systems, one consistent set of diagnostics. The position result—news is lead-biased, academic papers and meeting minutes are not—is well supported, and I’d trust it. The authors also deserve credit for a sensible idea: measure a system’s bias by ROUGE between its output and summaries produced by aspect-specific extractive algorithms.\n\nThe soft spot is exactly where the stress-test lands. Position algorithms (First/Last/Middle) are transparent. Diversity and importance are not validated. ConvexFall selects convex-hull vertices in BERT space—outliers by construction—and the pruning step is motivated by Marcu’s waterfall, not by a demonstrated link to extractive diversity. N-Nearest is a centrality measure; it could easily pick generic or repetitive sentences. No human annotation, no comparison to established aspect baselines such as MMR with varying trade-off or TextRank, no ablation of the sentence encoder. So when ConvexFall does poorly on meetings, that might mean outlier selection is a bad extractive strategy for that domain, not that meetings lack a diversity bias. Same for the system-bias numbers. This doesn’t sink the paper, but it means the headline cross-corpus comparisons are only secure for position.\n\nSmaller issues: the volume-overlap metric is explicitly reduced to 2D PCA, which limits what it can tell you, and the claimed ensemble advantage is not consistent across Table 4. The writing is honest about some of this—they note ConvexFall chooses outliers—but the implications are not fully absorbed into the conclusions. The citations are fair and engage with Kedzie et al. and Peyrard rather than ignoring them.\n\nWho is this for? Anyone choosing a summarization dataset or wondering why their model behaves differently across domains. It deserves a serious referee. I would send it out and ask for a validation section: either compare the aspect algorithms against human judgments or established methods, or soften the diversity/importance claims to what the current evidence supports.\n\nMy own verdict: conditional. Use it for the position findings, cite it cautiously for the diagnostic idea, but don’t build on the diversity/importance numbers until they’re validated.","headline":"Broad and practical empirical survey of summarization biases; the position findings are solid, but the diversity and importance measurements need validation before the cross-domain claims can be trusted.","tokens_in":19235,"tokens_out":2727,"would_cite":true,"duration_ms":26378,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Summarization systems and datasets each carry a measurable bias toward position, diversity, or importance—and news is the only domain where position dominates.","keywords":["text summarization","position bias","extractive summarization","corpus bias","system bias","semantic diversity","sentence importance","ROUGE"],"falsifier":"Take a new domain absent from the nine (for instance, legal opinions) and run the same three probe algorithms against human summaries. If First-k, ConvexFall, and N-Nearest all score no better than random selection, the claimed aspect structure does not exist there; if their relative ordering matches the news pattern in a non-news domain, the claim that position bias is domain-specific would be contradicted.","tokens_in":18202,"feed_emoji":"📊","tokens_out":6885,"duration_ms":60819,"temperature":0.7,"pith_summary":"The paper argues that \"earlier isn't always better\": the well-known tendency of summarizers to pick opening sentences holds strongly for news, but not for academic papers, meeting minutes, or several other domains. It decomposes summarization into three sub-aspects—position, importance, and diversity—and measures each with a purpose-built extractive algorithm across nine corpora. The measurements show that every corpus and every summarization system has a distinct bias profile; neural extractive systems are comparatively balanced, and simple ensembles of aspect-specific selectors match or beat individual systems on most domains. If true, this means benchmark results on news overstate the role of sentence position and understate the need for importance and diversity in other genres.","feed_headline":"News summaries favor lead sentences; papers and minutes don't","feed_subtitle":"A sub-aspect analysis of nine corpora and many systems ties summary success to position, diversity, and importance.","key_machinery":"The measurement instrument is a set of three extractive probe algorithms operating on BERT sentence embeddings. Position is probed by selecting the first, last, or middle k sentences; diversity by ConvexFall, which picks sentences whose convex hull in embedding space has maximum volume and then prunes least-volume-reducing sentences, plus a greedy Heuristic baseline; importance by N-Nearest and K-Nearest, which select sentences most similar to their neighbors. Each probe's output is compared with reference summaries or a system's output using averaged ROUGE, sentence overlap with the extractive oracle, and volume-overlap ratio, yielding a position/diversity/importance profile for every corpus and system.","core_discovery":"The central claim is that both corpora and systems can be profiled by how much they rely on sentence position, content importance, and semantic diversity, and that these profiles differ systematically. Concretely, First-k position selection achieves high ROUGE and sentence overlap on CNNDM, Newsroom, Reddit, and BookSum, but roughly random-level performance on PubMed, AMI, and XSum; academic papers (PeerRead, PubMed) need all three aspects at once. On the system side, LexRank is heavily position-biased, MMR leans importance, and neural extractive systems (CL, SumRun, S2SExt) are more balanced, while an ensemble built from the three aspect algorithms outperforms single-aspect systems on five of eight datasets. The paper also finds that reference summaries in most corpora contain fewer than 15% novel words, except academic abstracts, and that semantic volume overlap between model and reference summaries is not correlated with ROUGE.","pith_inferences":["The same probe procedure could be run on evaluation metrics themselves, scoring each metric's own position/diversity/importance bias, which would turn the paper's closing suggestion into a concrete benchmark.","The near-random performance on meetings, books, and scripts suggests these domains may need sub-aspects beyond the three studied (for example, discourse structure or speaker role); testing additional probes would show whether the three-aspect space is closed.","A direct test of the decomposition: construct synthetic source documents with known aspect composition and check whether First, ConvexFall, and N-Nearest recover the planted profile; that would validate or invalidate the measurement itself.","Dataset designers could publish an aspect profile alongside each corpus, letting users match a summarizer's bias to the target domain before training."],"forward_implications":["Benchmarking on news alone will keep rewarding lead-biased models; new datasets should report an aspect profile so position-biased gains are not mistaken for general progress.","For academic-paper summarization, a system must balance all three aspects; position-only baselines near random on PubMed show why abstract-based evaluation needs importance-aware selection.","Long documents (books, movie scripts) and conversations (meetings) remain far from solved; current aspect probes score near random, indicating the task needs additional mechanisms beyond these three aspects or noise filtering.","Simple ensembles of aspect-specific selectors are a cheap, effective alternative: on five of eight datasets they beat single systems in ROUGE or sentence overlap while keeping the three aspects in balance.","The weak correlation between volume overlap and ROUGE implies semantic diversity and lexical overlap are measuring different qualities; systems should be evaluated on both."],"supporting_citations":[{"why":"Supplies the sub-aspect theory that summarization systems are mixtures of sub-functions such as coverage, diversity, and information, motivating the paper's three-aspect decomposition.","marker":"Lin and Bilmes (2011)"},{"why":"Supplies the semantic-volume maximization and greedy heuristic for diversity; the diversity probes and volume-overlap metric derive from this work.","marker":"Yogatama et al. (2015)"},{"why":"Provides the position hypothesis and a method to identify preferred sentence positions, which the paper's position probe extends across domains.","marker":"Lin and Hovy (1997)"},{"why":"Documents the intentional lead bias in news writing, the baseline 'position bias in news' that the paper tests against other domains.","marker":"Hong and Nenkova (2014)"},{"why":"Shows position bias differs across domains, supplies the PubMed and Reddit splits, and provides the S2SExt neural extractive framework used in the system-bias analysis.","marker":"Kedzie et al. (2018)"},{"why":"Introduces XSum, the extreme-summarization corpus designed to avoid position bias; the paper uses it as the key non-position-biased news corpus.","marker":"Narayan et al. (2018a)"},{"why":"BERT is the sentence encoder shared by all three aspect algorithms and by the volume-overlap metric.","marker":"Devlin et al. (2018)"},{"why":"ROUGE is the metric that operationalizes each aspect's contribution; all corpus and system bias scores are computed with it.","marker":"Lin (2004)"}],"fun_headline_variants":["Summary bias varies by domain: position dominates news, not papers","Lead-sentence bias is a news quirk, not a universal summary rule","Position, importance, diversity: summary systems have distinct profiles","Ensemble of sub-aspect algorithms beats single-focus summary systems","Corpus and system biases in summarization: not all about position"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on the assumption that the three probe algorithms—First for position, ConvexFall for diversity, N-Nearest for importance—each cleanly isolate one sub-aspect, so their ROUGE scores against reference summaries reveal how much that aspect contributes to a corpus or system.","fun_headline_variants_meta":{"raw":{"variants":["Summary bias varies by domain: position dominates news, not papers","Lead-sentence bias is a news quirk, not a universal summary rule","Position, importance, diversity: summary systems have distinct profiles","Ensemble of sub-aspect algorithms beats single-focus summary systems","Corpus and system biases in summarization: not all about position"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000161,"raw_usage":{"total_tokens":1227,"prompt_tokens":931,"completion_tokens":296,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":206}},"tokens_in":547,"tokens_out":296,"duration_ms":3493,"temperature":1.0,"reasoning_tokens":206,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:08:04.424581+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a new domain absent from the nine (for instance, legal opinions) and run the same three probe algorithms against human summaries. If First-k, ConvexFall, and N-Nearest all score no better than random selection, the claimed aspect structure does not exist there; if their relative ordering matches the news pattern in a non-news domain, the claim that position bias is domain-specific would be contradicted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the sub-aspect theory that summarization systems are mixtures of sub-functions such as coverage, diversity, and information, motivating the paper's three-aspect decomposition."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the semantic-volume maximization and greedy heuristic for diversity; the diversity probes and volume-overlap metric derive from this work."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the position hypothesis and a method to identify preferred sentence positions, which the paper's position probe extends across domains."},{"cited_title":"Content Selection in Deep Learning Models of Summarization","cited_arxiv_id":"1810.12343","evidence_quote":"Shows position bias differs across domains, supplies the PubMed and Reddit splits, and provides the S2SExt neural extractive framework used in the system-bias analysis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ROUGE is the metric that operationalizes each aspect's contribution; all corpus and system bias scores are computed with it."}],"review_version":1}