{"id":"356bb246-4a5a-42d9-997d-80d6e141503a","arxiv_id":"2411.16613","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey of linear text segmentation covering unsupervised, supervised, and LLM-based approaches, datasets, evaluation metrics, and open challenges.","lead":"This paper reviews recent work on linear text segmentation, the task of identifying where topics change in documents, with emphasis on transformer-based language models and large language models. It organizes the field by method family, summarizes datasets and metrics, and compares reported system performance.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Tables 2 and 3, the sole support for the survey's empirical claims, misattribute several Pk values to works that cannot contain them (e.g., C99/Choi on en_city/en_disease; TextTiling/Solbiati on SuperDialseg); without a provenance protocol the headline ranking is not verifiable.","rationale":"The reader's verdict is CONDITIONAL, and my check supports that verdict rather than moving it. The survey is a useful synthesis, and the central narrative—transformer-based features and supervised learning becoming the framework of choice, with LLMs just starting to gain traction—is consistent with the wider literature, so I do not think the paper should be rejected. However, the empirical backbone is weaker than the text implies. The table rows are not all traceable to the cited references; the C99 en_city/en_disease example is unambiguous because the dataset postdates the citation, and the TextTiling SuperDialseg example is equally impossible. This is not a mere citation typo: if the table's numbers come from unknown third-party runs with unknown pre-processing, then the comparison lacks the reproducibility needed to support 'supervised models largely outperform unsupervised systems' and 'ChatGPT largely outperforms other unsupervised models.' The reader flagged the lack of error bars and selection protocol; I add that the source attribution itself fails. A provenance audit is cheap and settles the question. If the audit finds the values do appear in later cited papers or in the cited papers under a different name, the concern is resolved and the paper only needs a protocol statement.","tokens_in":13480,"tokens_out":6806,"duration_ms":64533,"concrete_test":"Perform a provenance audit of Tables 2 and 3: for every non-empty cell, open the cited reference, search for the exact benchmark and Pk value, and record whether (a) the reference's publication date is after the dataset's release and (b) the number actually appears in that paper. If any cell fails (e.g., C99/Choi on en_city, TextTiling/Solbiati on SuperDialseg), trace it to the actual source or rerun the system under a common evaluation split; if more than two cells are untraceable, the empirical ranking in the survey should be downgraded to 'not established.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest empirical conclusions—that supervised systems largely outperform unsupervised ones and that LLMs outperform all unsupervised methods—are supported only by Tables 2 and 3. The table caption says 'Results are reported from the works cited in the table,' but that is not true for multiple cells. The row 'C99 (Choi, 2000)' lists Pk=36.8 on en_city and 37.4 on en_disease, yet Choi (2000) predates the Wikisection datasets (Arnold et al., 2019) by 19 years. The row 'TextTiling (Solbiati et al., 2021)' lists Pk=44.1 on SuperDialseg, but SuperDialseg was introduced by Jiang et al. (2023), two years later. These numbers must have been produced or re-run by other papers, not reported by the cited works. The system-selection paragraph mentions that metrics and datasets vary, but it gives no protocol for choosing which reported value to include, no pre-processing or split compatibility check, and no error bars. Because the table entries cannot be traced to a stable source, the conclusion that supervised frameworks dominate—and that ChatGPT outperforms all unsupervised methods—cannot be independently checked. This is more load-bearing than the Pk metric objection alone: even with a perfect metric, the numbers in these tables would still lack provenance.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This survey reviews linear text segmentation, tracing approaches from count-based unsupervised methods through topic models, embeddings, transformer-based supervised classifiers, and initial LLM-based zero-shot segmentation. It surveys written-text and dialogue datasets, discusses evaluation metrics with particular attention to the shortcomings of Pk, and presents two comparison tables of reported Pk scores across systems and benchmarks. The paper concludes that transformer-based supervised learning has become the dominant framework, that the main bottlenecks are data availability/adoption and metric flaws, and that future work should pursue LLMs, multi-task learning, and better evaluation infrastructure.","tokens_in":13702,"tokens_out":4778,"duration_ms":42677,"significance":"If the survey's central claims are accepted, it fills a genuine gap: prior surveys are outdated and do not cover transformer-based and LLM-based methods. The organization of methods into count-based, topic-modelling, embedding-based, LLM-based, and single/multi-task supervised categories is clear and generally faithful to the cited literature, and the explicit Limitations section is a useful scoping statement. The survey's main empirical contribution is the claim, based on Tables 2 and 3, that supervised methods outperform unsupervised ones and that LLMs currently lead unsupervised methods. However, because those tables contain attribution errors and no comparison protocol, the quantitative evidence for these conclusions is presently not verifiable. The survey is therefore more reliable as a narrative review than as a benchmark comparison, and the empirical claims need to be re-anchored before the contribution is fully sound.","major_comments":[{"comment":"The provenance of several table entries is impossible. Table 2 row 'C99 (Choi, 2000)' reports Pk=36.8 on en_city and 37.4 on en_disease, but Choi (2000) predates the Wikisection datasets (Arnold et al., 2019) by nearly two decades, so that citation cannot be the source of those numbers. Similarly, Table 3 row 'TextTiling (Solbiati et al., 2021)' reports Pk=44.1 on SuperDialseg, a dataset introduced by Jiang et al. (2023); the cited Solbiati et al. work cannot contain that result. The caption 'Results are reported from the works cited in the table' is therefore false for at least these cells, and possibly others. Because §5 uses these tables to conclude that supervised models largely outperform unsupervised systems and that ChatGPT outperforms all unsupervised methods, the empirical core of the survey currently lacks traceable evidence. Please provide the exact source (paper, table, and experimental setup) for every reported value, or re-run the systems under a documented protocol, and correct the table citations accordingly.","section":"§5, Table 2 and Table 3"},{"comment":"The comparison lists a single Pk value per system-dataset pair with no error bars, no statistical significance tests, no tokenization or preprocessing details, and no description of how values were selected when multiple configurations exist. The paper itself notes in §4 that Pk has well-documented flaws and in §5 that metrics and datasets vary across works, yet the inclusion criteria are not given. Consequently, the quantitative ranking—for example, 'models based on Longformer... show best performance on most benchmarks' and 'ChatGPT largely outperforms other unsupervised models'—may reflect differences in test splits, Pk window sizes, or preprocessing rather than system quality. Please make the comparison protocol explicit (datasets, splits, basic units, Pk parameters, and whether numbers come from original papers or from re-runs) and either add confidence measures or limit the conclusions to qualitative statements supported by direct citation.","section":"§5, Tables 2 and 3; §4"},{"comment":"The claim that LLMs 'outperform all other unsupervised methods after careful prompt optimization' is stronger than the evidence presented. In Table 3, the only LLM result is a single ChatGPT value on SuperDialseg, with no comparison on other dialogue benchmarks and no measure of variance; the other unsupervised rows on the same dataset come from methods whose reported values are themselves of uncertain provenance, as noted above. Please either provide a reproducible evaluation protocol or soften the claim to state that, under the cited setups, ChatGPT reports the best unsupervised score on SuperDialseg in the authors' collected comparison.","section":"§2.2.4 and §5"}],"minor_comments":[{"comment":"The phrase 'mole of content' appears to be a typographical error for 'mass of content' or 'bulk of content'; please correct it.","section":"Abstract"},{"comment":"The row labeled 'TextTiling (Choi, 2000)' misattributes TextTiling to Choi; the algorithm originates from Hearst (1994) and the Choi reference is for the dataset. Please correct the system attribution.","section":"Table 2"},{"comment":"The dataset name 'Non-NewsSBBC' is not expanded or explained; if this is intended to refer to a BBC news-related corpus, please clarify the name and the distinction from 'Media'.","section":"Table 1"},{"comment":"In the sentence about en_disease, 'at it includes a variety of rare medical terms' should be 'as it includes a variety of rare medical terms'.","section":"§3.1"},{"comment":"The sentence 'In our systems comparison, We also used Pk' has an erroneous capital 'W' in 'We'; please restructure or lowercase.","section":"§4"},{"comment":"The word 'hierachical' should be 'hierarchical' in the future-directions paragraph.","section":"§6"},{"comment":"The figures report metric frequencies in 'a sample of recent works we cited,' but the sample selection process and the exact set of works counted are not described; please define the sample so the reader can interpret the counts.","section":"§4, Figures 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The two impossible citations in Tables 2 and 3 are concrete and easily verifiable; before any revision is considered further, I would ask the authors to supply a per-cell source table or the scripts used to produce each number. If values were taken from other papers that re-ran older systems, those papers must be cited. This is a matter of scholarly provenance and reproducibility, not a stylistic preference."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. This survey fills a real gap: there hasn't been a broad overview of linear text segmentation since Purver (2011), and the authors do a decent job of charting the shift from count-based and topic-model methods to transformer- and LLM-based supervised systems. The taxonomy, dataset summaries, and metric-co-occurrence counts are genuinely useful. The limitations section is honest.\n\nThe problem is the empirical core. Tables 2 and 3 are the only support for the headline claims—supervised beats unsupervised, ChatGPT beats all unsupervised methods—and several cells are misattributed. C99 (Choi, 2000) cannot have reported Pk on en_city and en_disease; those datasets came from Arnold et al. (2019). TextTiling (Solbiati et al., 2021) cannot have reported Pk on SuperDialseg, which was introduced by Jiang et al. (2023). The caption says the numbers come from the cited works, but that is not true. BayesSeg is cited to Barzilay and Lapata (2008), which is not the BayesSeg paper, and QMSUM is cited to a Zhang et al. (2022) paper that isn't QMSUM. These are easy to verify.\n\nMore structurally: there is no protocol for which reported Pk value to include, no split or preprocessing checks, no error bars, and Pk alone carries the comparison even though the paper itself says Pk is flawed. The stress-test note is right: even with a perfect metric, the provenance problem makes the rankings unverifiable. The claim in Section 2.2.4 that ChatGPT outperforms all other unsupervised methods is also broader than the single dataset shown.\n\nI want to be fair. The survey's central narrative is plausible and the organization is solid. The problems are fixable, not fatal to the whole enterprise. But as it stands, the tables cannot be trusted, and the conclusions lean on them.\n\nThis paper is for newcomers to topic segmentation and researchers who want a map of methods, datasets, and metrics. It deserves a serious referee, but the verdict should be conditional: major revision to re-source every number in Tables 2 and 3, correct the citations, and qualify the claims to match the evidence. I would not cite it in its current form.","headline":"Useful survey of linear text segmentation, but its comparison tables—the only evidence for the headline empirical claims—have load-bearing citation and provenance problems.","tokens_in":14232,"tokens_out":2840,"would_cite":false,"duration_ms":25317,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey argues that linear text segmentation has shifted to supervised transformer-based systems, with LLM methods just beginning to compete, and that missing data and flawed metrics—especially Pk—are now the field's real bottlenecks.","keywords":["linear text segmentation","topic segmentation","transformer language models","large language models","Pk metric","boundary similarity","dialogue topic segmentation","benchmark datasets"],"falsifier":"Run the systems in Tables 2 and 3 on the same benchmarks with identical tokenization, train/dev/test splits, and metric code, and add confidence intervals; if the supervised-versus-unsupervised ordering changes materially, or if a tuned ChatGPT prompt no longer leads the unsupervised column, the survey's empirical conclusions would not hold.","tokens_in":13236,"feed_emoji":"📚","tokens_out":8453,"duration_ms":71953,"temperature":0.7,"pith_summary":"The survey is trying to establish where linear text segmentation stands after the transformer wave. It claims that the field has consolidated around a standard recipe: use sentence- or word-level features from pretrained transformer language models, train a supervised sequence tagger, and add auxiliary objectives to fight overfitting. It also claims that large language models are only beginning to enter the task, but that prompted zero-shot segmentation already outperforms older unsupervised methods. The deeper point is that model design is no longer the main obstacle; the bottlenecks are scarce and fragmented datasets, especially for dialogue and media, and evaluation metrics that the field itself knows are unreliable. A sympathetic reader should finish believing that progress depends less on inventing a better model than on building shared benchmarks and better metrics.","feed_headline":"Survey: data and metrics, not models, bottleneck topic segmentation","feed_subtitle":"Supervised transformer features now lead the field; the unsolved problems are shared benchmarks and the flawed Pk metric.","key_machinery":"The analytical machinery is a two-axis taxonomy—method family (count-based, topic-modelling, embedding-based, LLM-based, supervised single-task, or multi-task) crossed with basic unit (word, sentence or pseudo-sentence, paragraph, or speaker turn)—supported by two comparison tables that line systems up on benchmarks using the Pk metric. The taxonomy is what lets the survey read fifty years of work as a single arc toward transformer features and supervised learning. The tables carry the empirical claims: supervised systems hold the lowest (best) reported Pk values on written text, multi-task systems dominate their single-task rivals, and ChatGPT's 31.8 on SuperDialseg is the best reported unsupervised dialogue result. Pk itself is part of the argument: because its flaws are documented and it still dominates usage, the survey uses it to say the field lacks a trustworthy, widely shared evaluation standard.","core_discovery":"The paper's central claim is that the history of linear text segmentation is a shift in where semantic features come from: from count-based window comparisons, through topic-model word distributions and word embeddings, to transformer-based sentence embeddings feeding supervised classifiers, and now to LLM prompting. It finds that supervised systems largely outperform unsupervised ones on written-text benchmarks, that multi-task variants are the strongest performers, and that ChatGPT, after careful prompt optimization, beats all earlier unsupervised systems on the SuperDialseg dialogue benchmark. On evaluation, it argues that the most-used metric, Pk, has documented flaws, that newer alternatives such as Boundary Similarity are rarely adopted, and that no single dataset has become a standard benchmark. The survey concludes that dataset availability and adoption, not model architecture, are the central challenges for the field's near-term progress.","pith_inferences":["A controlled re-evaluation on shared splits with confidence intervals could reorder the systems in the paper's own tables, since the Pk numbers are copied from heterogeneous sources; this is an editorial caution, not the survey's claim.","If evaluation infrastructure is the real bottleneck, then test sets that record annotator disagreement about boundary placement, rather than one ground-truth boundary, would make Pk's tolerance flaws visible and could push the field toward metrics like Boundary Similarity.","The ChatGPT result suggests a direct test: evaluate prompted LLM segmentation on podcast and TV transcripts, the domains the survey identifies as lacking supervised datasets; good transfer would confirm that LLMs can fill the data gap.","A concrete design target implied by the survey is a multi-thousand-transcript dialogue benchmark with per-boundary agreement scores and both Pk and Boundary Similarity reported; such a resource would do more for the field than another model."],"forward_implications":["Future systems should build on transformer-based features; older count-based and topic-modelling pipelines are unlikely to be competitive except where training data are very scarce.","Releasing larger and more diverse dialogue and media datasets should matter more for progress than novel architectures, since supervised systems improve sharply when enough data exist.","The reported superiority of supervised over unsupervised systems should be read as provisional until a shared benchmark and metric protocol exist, because the comparison tables use Pk values from papers with different setups.","LLM zero-shot segmentation, if it keeps improving with prompt optimization, is a credible way to address domains that lack annotated training data.","Multi-task objectives such as topic classification and coherence discrimination are the main reported remedy for overfitting to domain-specific cue phrases, so they are likely to remain central to system design."],"supporting_citations":[{"why":"Defines linear text segmentation and supplies the previous survey that this work updates.","marker":"Purver, 2011"},{"why":"Introduces TextTiling, the count-based unsupervised algorithm that anchors the early part of the taxonomy.","marker":"Hearst, 1994"},{"why":"Introduces the wiki-727k dataset and the TextSeg supervised baseline that showed large gains over earlier methods.","marker":"Koshorek et al., 2018"},{"why":"Introduces Cross-segment BERT, the sentence-level transformer supervised system used in the comparison tables.","marker":"Lukasik et al., 2020"},{"why":"Provides Longformer + TSSP + CSSL, the reported state of the art on written-text benchmarks in the comparison.","marker":"Yu et al., 2023"},{"why":"Contributes the SuperDialseg dataset and reports both the ChatGPT zero-shot result and a supervised TextSeg result on it.","marker":"Jiang et al., 2023"},{"why":"Supplies evidence that prompt-optimized ChatGPT outperforms other unsupervised methods, grounding the LLM claim.","marker":"Fan and Jiang, 2023"},{"why":"Documents specific flaws in Pk and WindowDiff, supporting the survey's evaluation critique.","marker":"Georgescul et al., 2006"},{"why":"Proposes Boundary Similarity, the alternative metric the survey recommends as a replacement for Pk.","marker":"Fournier, 2013"}],"fun_headline_variants":["Supervised transformers win text segmentation; benchmarks and Pk metric lag","LLM prompting beats unsupervised segmentation on dialogue","Text segmentation stuck without standard benchmarks, says survey","The real bottleneck in topic segmentation: data, not models","Pk metric flaws hold back text segmentation progress"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central ordering—supervised systems beat unsupervised ones, and ChatGPT beats other unsupervised methods—rests on Pk scores copied from papers that used different datasets, splits, and metric implementations, with no error bars or significance tests, so the numbers may not be directly comparable.","fun_headline_variants_meta":{"raw":{"variants":["Supervised transformers win text segmentation; benchmarks and Pk metric lag","LLM prompting beats unsupervised segmentation on dialogue","Text segmentation stuck without standard benchmarks, says survey","The real bottleneck in topic segmentation: data, not models","Pk metric flaws hold back text segmentation progress"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000658,"raw_usage":{"total_tokens":2959,"prompt_tokens":842,"completion_tokens":2117,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":2041}},"tokens_in":458,"tokens_out":2117,"duration_ms":14383,"temperature":1.0,"reasoning_tokens":2041,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:55:31.782821+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the systems in Tables 2 and 3 on the same benchmarks with identical tokenization, train/dev/test splits, and metric code, and add confidence intervals; if the supervised-versus-unsupervised ordering changes materially, or if a tuned ChatGPT prompt no longer leads the unsupervised column, the survey's empirical conclusions would not hold.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines linear text segmentation and supplies the previous survey that this work updates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Cross-segment BERT, the sentence-level transformer supervised system used in the comparison tables."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents specific flaws in Pk and WindowDiff, supporting the survey's evaluation critique."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Proposes Boundary Similarity, the alternative metric the survey recommends as a replacement for Pk."}],"review_version":1}