{"id":"12a0a8a2-75b9-4eb6-885c-f3abde1dc7e6","arxiv_id":"2501.07723","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ESURF, a simple random forest with lexical and character n-gram features, reaches state-of-the-art EDU segmentation on RST-DT and modestly improves RST parsing.","lead":"A new paper proposes ESURF, a random forest classifier that uses lexical and character n-gram features to locate boundaries between elementary discourse units. The authors report it beats neural and handcrafted segmenters on RST-DT and slightly improves an existing RST parser, which suggests simple local cues may be enough for this task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed SOTA margins rest on uncontrolled literature baselines and a 0.3-point F1 gap with no significance test, while the CNN/Daily Mail EDU label source is undisclosed.","rationale":"The reader's CONDITIONAL verdict is appropriate. My stress-test sharpens the same weakest assumption: the empirical superiority claim depends on comparable, significance-tested evaluation, which the paper does not provide. The core method is plausible and the RST-DT results are not internally contradictory, so this is not a rejection-level flaw. The missing CNN/Daily Mail label provenance is a concrete, fixable reporting gap, and the absence of error bars or a unified baseline harness means the 0.3-point F1 lead over the Joint Model cannot yet be distinguished from noise. The proposed test would settle the matter directly, and the reader's conditional acceptance already requires such evidence before the claim is taken as established.","tokens_in":6335,"tokens_out":6342,"duration_ms":63631,"concrete_test":"Have the authors run a single controlled RST-DT evaluation: obtain or re-implement the Joint Model (Lin, 2023) and at least one neural baseline, apply ESURF's exact preprocessing and sentence splitting, and compute paired bootstrap 95% confidence intervals over the 38 test documents for the ESURF-versus-Joint F1 difference. If the confidence interval includes zero, Table 2's superiority claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing link in the abstract's claim is the comparison in Table 2: ESURF at 95.8 F1 versus the Joint Model (Lin, 2023) at 95.5 F1 on RST-DT, plus Table 3's parser deltas (+0.4 Span, +1.3 Relation, +0.3 Full). These numbers are compared with baselines taken from prior publications rather than re-run under one control. Section 4.2 states only that evaluation used a similarly sized subset of data points (50% positive / 50% negative) with preprocessing consistent with our previous approach; it does not define the sentence-splitting, boundary-matching, or class-balancing protocol for the literature baselines. A 0.3-point F1 difference is within typical test-set noise, and no confidence intervals, bootstrap, or paired test are reported. In addition, Section 4.1 introduces CNN/Daily Mail but never states where EDU labels come from; since that corpus has no gold RST annotations, Table 1's CNN column is uninterpretable unless the label-generation procedure is disclosed. If those labels come from an automatic segmenter (e.g., Muller et al. 2019), the result measures agreement with that segmenter rather than EDU quality. These are fixable gaps, but without them the headline superiority claim is not established as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ESURF, a random-forest EDU boundary classifier that uses nine-token windows with lexical and character n-gram features. The system is evaluated in three ways: against generic classifiers on RST-DT and CNN/Daily Mail (Table 1), against established EDU segmenters on RST-DT (Table 2), and as the segmentation component inside the Yu et al. (2022) RST parser (Table 3). The authors report a state-of-the-art F1 of 95.8 on RST-DT segmentation, 0.3 points above the Joint Model (Lin, 2023), and small improvements in parser Span, Relation, and Full scores when ESURF replaces the original segmenter.","tokens_in":6602,"tokens_out":4351,"duration_ms":41282,"significance":"If substantiated, the result is valuable: it would show that simple local lexical and morphological features can rival or exceed large neural models on EDU segmentation, with potential training-efficiency and interpretability benefits. The simplicity of the model and the use of an external parser for downstream evaluation are positive features. However, the empirical support as reported is incomplete: the CNN/Daily Mail EDU label provenance is undisclosed, the literature baselines in Tables 2 and 3 are not shown to be evaluated under the same protocol, and no significance or stability testing accompanies the small reported margins. The paper also does not include a code or data availability statement, which limits reproducibility of the comparisons.","major_comments":[{"comment":"The paper never states where the EDU labels for CNN/Daily Mail come from; this corpus has no gold RST annotations normally associated with it. Without disclosing whether the labels were produced by an automatic segmenter, projected from another corpus, or manually annotated, the CNN/Daily Mail columns in Table 1 are uninterpretable and the claim of a two-dataset evaluation is not established.","section":"Section 4.1, Table 1"},{"comment":"The comparison with prior segmenters (HILDA, SPADE, CRF, JCN, WYL, Fisher and Roark, and the Joint Model) relies on scores taken from earlier publications rather than re-running those systems under the same evaluation protocol. Section 4.2 specifies a balanced 50/50 subset and 'preprocessing consistent with our previous approach' only for the Table 1 classifiers, so it is unclear whether the literature baselines in Table 2 were evaluated under the same boundary-matching, sentence-splitting, and class-balancing setup. In addition, a 0.3-point F1 gap over the Joint Model is within typical test-set noise, and no confidence intervals, bootstrap resampling, or significance tests are reported.","section":"Section 4.2, Table 2"},{"comment":"The parser experiment does not state whether the Yu et al. (2022) baseline numbers were re-run under identical conditions or taken from the original paper, and it does not report variance across runs or paired tests. The reported improvements (+0.004 Span, +0.013 Relation, +0.003 Full, with Full tying Zhang et al. 2021b at 0.538) are small enough that they may not be distinguishable from noise. The paper should clarify the setup and provide significance or stability evidence before claiming that ESURF improves a state-of-the-art parser.","section":"Section 4.3, Table 3"},{"comment":"The reported XLNet F1 of 0.330 with precision 0.253 is near chance and inconsistent with published results for XLNet on text classification. This suggests a degenerate training setup, an implementation issue, or a label misalignment. The authors should either report the exact training configuration, diagnose the failure, or remove the XLNet comparison, since as presented the row does not constitute a meaningful baseline.","section":"Table 1, XLNet row"}],"minor_comments":[{"comment":"The manuscript contains several typos and stylistic errors, including 'well-acccepted', 'assymetrical', 'the the', and 'highly effectiveness'; a careful proofread is needed.","section":"Throughout"},{"comment":"Figure 1 is referenced but no diagram appears in the manuscript text; either include the figure or remove the reference.","section":"Section 3 and Figure 1"},{"comment":"There are duplicate references: Yu et al. 2018a and 2018b are identical, and Zhang et al. 2021a and 2021b are identical; these should be consolidated.","section":"References"},{"comment":"The 'Joint Model (Lin, 2023)' citation points to a book titled 'Natural language processing as autoregressive generation', which does not appear to describe an EDU segmentation system; please verify the citation and provide the correct source.","section":"References, Joint Model"},{"comment":"The feature description lacks details needed for reproducibility: the character n-gram lengths, the frequency filter thresholds, and the random forest hyperparameters are not specified.","section":"Section 3"},{"comment":"The limitation section states that the method assumes accurate sentence segmentation, but Section 3 says that windows do not cross sentence boundaries; the paper should clarify whether the evaluation used gold sentence boundaries or an automatic sentence segmenter.","section":"Section 6, Limitation"}],"recommendation":"major_revision","confidential_remarks":"The paper does not include a code or data availability statement. Given the reliance on literature-reported baselines and the undisclosed CNN/Daily Mail label source, the editor may wish to request that the authors release their preprocessing scripts, evaluation code, and label-generation pipeline during revision. The Lin (2023) reference also appears mismatched and should be verified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the core method is genuinely simple and the reported numbers are good enough that the result would matter if it holds up: a random forest over lexical and character n-gram features reaching 95.8 F1 on RST-DT, slightly above a 95.5 F1 neural joint model, and improving a state-of-the-art parser by a small but consistent amount. Second, as reported, the empirical case is not yet solid. The CNN/Daily Mail experiments have no stated EDU label source — that corpus has no gold RST annotations — so Table 1 is uninterpretable until the authors say where the labels came from. If they came from an automatic segmenter, the comparison measures agreement with that segmenter, not segmentation quality. The RST-DT comparison in Table 2 uses baselines taken from prior papers rather than re-run under one preprocessing pipeline, and the 0.3-point F1 gap has no error bars or significance test. Also, the XLNet baseline in Table 1 sits at 0.330 F1, which is implausibly low; that suggests the experimental protocol for at least one condition was broken, and it makes you wonder about the others. These are fixable issues, not fatal contradictions, but they mean the headline \"outperforms other methods\" is not established as written. What the paper does well: it is clearly written, the method is described in enough detail to reimplement, and the limitation section is honest about scope. The idea that local lexical and morphological cues do most of the work in EDU segmentation is credible, and the parser experiment with Yu et al. is a sensible way to test downstream value, even though the gains are modest. The citation pattern looks reasonable, though the “Joint Model” reference is vague. My take: this is a desk-reviewable paper with a potentially useful baseline, but the SOTA claim needs verification. I would send it to peer review with the expectation that the authors release code, disclose the CNN/Daily Mail label generation, and either re-run the literature baselines under a unified setup or present significance-tested comparisons. If they cannot do that, the claim should be softened to \"competitive with neural segmenters\" rather than \"outperforms.\" A serious referee could get this into shape.","headline":"A plausible simple EDU segmenter whose SOTA claim rests on uncontrolled literature baselines and an undisclosed CNN/Daily Mail label source.","tokens_in":7127,"tokens_out":1533,"would_cite":false,"duration_ms":18319,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A random forest trained only on words and character fragments around each candidate boundary segments text into discourse units more accurately than large neural models, and swapping it into a top RST parser improves parse scores.","keywords":["EDU segmentation","rhetorical structure theory","random forest","discourse parsing","lexical features","character n-grams","RST-DT","CNN/Daily Mail"],"falsifier":"Re-run all baselines on the same RST-DT split and the same CNN/Daily Mail subset under identical preprocessing and label definitions; if the CNN/Daily Mail EDU labels come from an automatic segmenter, check whether ESURF's advantage persists when those labels are replaced with human-annotated boundaries.","tokens_in":6111,"feed_emoji":"✂️","tokens_out":6059,"duration_ms":52007,"temperature":0.7,"pith_summary":"This paper claims that a deliberately simple classifier—a random forest trained on the words and character fragments around each candidate boundary—segments text into Elementary Discourse Units (EDUs), the leaf nodes of Rhetorical Structure Theory, more accurately than far more complex neural models. The authors report that ESURF reaches 95.8% F1 on the RST-DT test set, slightly beating the previous best of 95.5%, and that swapping ESURF into the Yu et al. (2022) RST parser improves its span, relation, and full-structure scores. If true, the result matters because it suggests that local lexical and morphological cues carry most of the information needed for EDU identification, pointing toward cheaper and more training-efficient discourse analysis.","feed_headline":"Random forest beats big neural models at discourse segmentation","feed_subtitle":"ESURF uses only word and character cues to split text into discourse units — and improves RST parsing too.","key_machinery":"The central object is the nine-token context window around a candidate EDU boundary, with tokens tagged as Before, Leading, or Continuing, combined with filtered character n-grams from those tokens. A random forest classifier assigns a boundary/non-boundary label to each window, and consecutive tokens between positive boundaries become EDUs. This design isolates local surface cues and deliberately avoids parse trees, hand-crafted syntax features, and pretrained language models, which is what makes the comparison against neural baselines informative.","core_discovery":"ESURF treats EDU segmentation as binary classification of each token gap: for every nine-token window with three tokens before the candidate boundary and six after, it extracts the tokens themselves and informative character substrings, marks each by position (before, leading, or continuing), and trains a random forest. The paper's central discovery is that this low-tech feature set matches or beats state-of-the-art EDU segmenters: on RST-DT it achieves 0.958 F1 against 0.955 for the Joint Model, and when used with the shift-reduce parser of Yu et al. (2022), parse span F1 rises from 0.764 to 0.768 and relation F1 from 0.545 to 0.558. The authors interpret this as evidence that lexical and morphological context, rather than deep syntax or pretrained representations, are the dominant cues for basic discourse unit boundaries.","pith_inferences":["An extension the paper leaves implicit: applying the same position-tagged word and character features inside a linear classifier would reveal whether the random forest's nonlinearities are necessary or whether the features themselves carry the signal.","Because ESURF needs only boundary-annotated text and no parse trees, the authors' future plan implies a semi-supervised loop—train on a small gold corpus, segment a large unlabeled corpus, retrain on machine-labeled output—though the paper does not demonstrate this.","The paper does not ablate its feature groups, but its framing implies a diagnostic: removing character n-grams while keeping word tokens (and vice versa) would quantify how much morphology contributes, and if character cues dominate, the method should transfer more readily to morphologically rich languages."],"forward_implications":["If ESURF's advantage holds, simpler feature-based EDU segmentation can replace heavy neural segmenters without losing accuracy, saving compute and training data.","Plugging ESURF into existing RST parsers yields immediate parse-quality gains (span 0.768, relation 0.558, full 0.538 on the reported metrics), showing segmentation quality directly limits parser performance.","The success of position-tagged word and character features supports semi-supervised use on unlabeled corpora, as the model needs only boundary labels and no syntactic preprocessing.","The approach promises easier adaptation to lower-resource languages, since random forests with n-gram features need far less data than fine-tuned transformers."],"supporting_citations":[{"why":"Supplies the RST-DT corpus, the standard benchmark on which ESURF's segmentation and parsing scores are measured.","marker":"(Carlson et al., 2002)"},{"why":"The state-of-the-art RST parser into which ESURF is plugged; Table 3 shows ESURF's segmentation improving its span, relation, and full scores.","marker":"(Yu et al., 2022)"},{"why":"The Joint Model whose 0.955 F1 is the strongest segmentation baseline that ESURF claims to beat with 0.958.","marker":"(Lin, 2023)"},{"why":"The EDU segmenter that Yu et al. (2022) reimplemented for large-scale unlabeled text; ESURF replaces it in the parser.","marker":"(Muller et al., 2019)"},{"why":"Provides the four-metric (span, nuclearity, relation, full) evaluation framework used for the parse comparisons in Table 3.","marker":"(Morey et al., 2017)"},{"why":"Source of the CNN/Daily Mail corpus on which ESURF's classifier comparison is run.","marker":"(Nallapati et al., 2016)"}],"fun_headline_variants":["Simple random forest outperforms neural EDU segmenters","ESURF: low-tech features win at discourse segmentation","Random forest on n-grams beats deep models for EDUs","EDU segmentation: random forest > neural networks","Cheap features, top score: ESURF beats neural parsers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on all methods being compared under identical data, preprocessing, and EDU label definitions; if the baselines were not run in the same setup, or if the CNN/Daily Mail EDU labels come from an automatic system, the reported advantage may not be real.","fun_headline_variants_meta":{"raw":{"variants":["Simple random forest outperforms neural EDU segmenters","ESURF: low-tech features win at discourse segmentation","Random forest on n-grams beats deep models for EDUs","EDU segmentation: random forest > neural networks","Cheap features, top score: ESURF beats neural parsers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001104,"raw_usage":{"total_tokens":4528,"prompt_tokens":793,"completion_tokens":3735,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":409,"completion_tokens_details":{"reasoning_tokens":3654}},"tokens_in":409,"tokens_out":3735,"duration_ms":25253,"temperature":1.0,"reasoning_tokens":3654,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:36:06.712442+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run all baselines on the same RST-DT split and the same CNN/Daily Mail subset under identical preprocessing and label definitions; if the CNN/Daily Mail EDU labels come from an automatic segmenter, check whether ESURF's advantage persists when those labels are replaced with human-annotated boundaries.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the RST-DT corpus, the standard benchmark on which ESURF's segmentation and parsing scores are measured."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Joint Model whose 0.955 F1 is the strongest segmentation baseline that ESURF claims to beat with 0.958."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The EDU segmenter that Yu et al. (2022) reimplemented for large-scale unlabeled text; ESURF replaces it in the parser."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the four-metric (span, nuclearity, relation, full) evaluation framework used for the parse comparisons in Table 3."}],"review_version":1}