{"id":"e7e46aff-3fd4-4f8d-879d-4b9c7db6522d","arxiv_id":"1908.10993","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new 13-class scientific statement classification task over 10.5 million arXiv paragraphs, with baselines up to 0.91 F1, but the improved score reflects a confusion-based regrouping of labels.","lead":"This paper builds a large dataset of 10.5 million scientific paragraphs from arXiv, labeled by statement type (theorem, proof, definition, etc.), and tests standard NLP models on a 13-class version. It reports a best F1 of 0.91 with a BiLSTM, but the class grouping was derived from the same model's confusion matrix, which inflates the headline score.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.91 F1 headline is compromised because the 13-class task was derived from the same BiLSTM's confusion matrix on the same test split, making the subsequent evaluation an optimistic, partly circular estimate.","rationale":"The reader's weakest assumption correctly identifies the circularity: the 13 classes were selected from the confusion matrix of the same BiLSTM family, and the reported 0.91 F1 improvement is partly by construction. I agree with that core concern. My attack sharpens it with a concrete technical detail: the confusion matrix used for grouping appears to be evaluated on the same 20% test split later used for the 13-class report, which constitutes test-set leakage in task design and makes the 0.91 number an optimistic estimate. The reader did not explicitly call out the test-split issue, hence 'partial' agreement. I still support the reader's CONDITIONAL verdict: the dataset and preprocessing pipeline are real contributions, the paper is transparent about its methodology, and the underlying resource is not invalidated. However, the headline F1 requires reframing or re-evaluation under a cleaner protocol. Since the reader's verdict already captures this conditionality, my stress-test does not change the verdict; it strengthens the rationale for why the condition is necessary.","tokens_in":8379,"tokens_out":3604,"duration_ms":35748,"concrete_test":"Re-run the pipeline with strict separation: (1) compute the 50-class BiLSTM confusion matrix only on the 80% training split (e.g., via internal cross-validation), (2) derive the 13-class grouping from that training-only confusion matrix, and (3) evaluate the 13-class BiLSTM on the untouched 20% test split. If the F1 drops substantially below 0.91 (e.g., below 0.85), the original number is inflated by test-set leakage. If the F1 remains near 0.91, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central performance claim—that a BiLSTM encoder-decoder reaches 0.91 F1 on the reduced 13-class task—rests on a task-design step that is not independent of the evaluation. In Section 3, the authors pre-partition the original 50-class data into an 80/20 train/test split, train a 50-class BiLSTM, and inspect its confusion matrix (Figure 2) to guide the grouping into 13 nests. The wording in Section 4 ('Its confusion matrix, also evaluated on the unseen test set') indicates that the Figure 2 confusion matrix was computed on the same test split that is later used to report the 13-class F1 in Table 2. Thus the test labels directly influenced which classes were merged and which were preserved. The 13-class evaluation is then performed on the same test set, so the reported F1 is not a clean measure of how well the model generalizes to a predefined task; it is partly an artifact of having selected the class structure to match the model's own confusions on that exact data. This is a form of test-set leakage, even though no gradient updates touched the test set, because the test set was used for task design. The paper honestly describes the procedure as 'empirical curation,' but the abstract's claim that 'the task setup aligns with known success rates from the state of the art, peaking at a 0.91 F1-score' overstates the significance of the number. The dataset itself remains valuable, and the 13-class grouping may be sensible, but the headline F1 should not be taken as an unbiased benchmark of task difficulty.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new large-scale scientific statement classification task derived from the arXMLiv HTML5 conversion of arXiv, extracting 10.5 million paragraph-level samples with 50 author-annotated labels. To make the task tractable, the authors inspect the confusion matrix of a BiLSTM encoder-decoder trained on an 80/20 split of the 50-class data and group the 50 labels into 13 'nests,' retaining 99% of the original paragraphs. They then evaluate several baselines on the reduced 13-class task, reporting that a BiLSTM encoder-decoder achieves a 0.91 F1 score, and they perform a control experiment omitting mathematical formulas to assess the contribution of the symbolic modality. The paper also releases the preprocessed dataset, preprocessing code, and baseline implementations.","tokens_in":115,"tokens_out":3394,"duration_ms":87641,"significance":"If the proposed benchmark is valid, this is a substantial community resource: the dataset is two orders of magnitude larger than most prior statement-classification corpora, the preprocessing pipeline is described in detail and made publicly available, and the math-lexeme serialization is a novel contribution that may benefit the MathIR community. The paper also provides several reproducible baselines. However, the central evaluation is weakened by a circularity in the task design: the 13-class grouping is derived from the confusion matrix of the same model family on the same test split that is later used to report the headline F1, so the 0.91 figure is not an unbiased estimate of generalization to a predefined task. The data collection and preprocessing strengths are real, but the performance claims need to be re-framed or re-validated.","major_comments":[{"comment":"The grouping into 13 nests is motivated solely by the confusion patterns of a single architecture (BiLSTM). No external evidence—such as human annotation agreement, functional taxonomy of scientific discourse, or an independent corpus study—is provided to establish that these nests are natural or meaningful classes. The paper acknowledges this is 'empirical curation' but does not discuss the associated validity threat. For a benchmark intended for the wider community, the task definition should be justified independently of any particular model's errors. At minimum, the authors should provide a post-hoc analysis showing that the groupings are stable across different model families or that a human-labeled subset confirms the nest structure.","section":"§3, §5"}],"minor_comments":[{"comment":"The phrase 'aligns with known success rates from the state of the art' is vague. Please specify which known success rates are being referred to and how the comparison is quantified.","section":"Abstract, §1"}],"recommendation":"major_revision","confidential_remarks":"The core issue is the circularity of the 13-class task construction and evaluation, which the authors openly describe but do not identify as a validity threat. The dataset and tooling are solid contributions, and the math lexeme work is interesting, but the headline performance claim needs to be re-derived under a non-leaky evaluation protocol. I would be willing to see a revised version that addresses this point; a rejection seems too harsh given the value of the released resources."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe real contribution here is the resource: 10.5 million paragraph-level scientific statement labels extracted from arXiv using author \\newtheorem markup, with an openly released preprocessing pipeline, math lexeme serialization, and clean train/test splits. That is new at this scale, and the authors are careful about filtering, language detection, and reproducibility. The paper also does a decent job of positioning the work against earlier small-scale efforts. Credit is due for shipping code, data, and a live demo.\n\nThe soft spot is the one the reader's report flags: the 13-class task is defined after inspecting the confusion matrix of the same BiLSTM family that is later used to produce the headline 0.91 F1. The paper calls this 'empirical curation' and is transparent about it, but that does not remove the circularity. Even if the confusion matrix was computed on the held-out test split, using that split to design the label grouping and then reporting performance on it is selection on the evaluation target. The zero-rule baseline jumping from 0.201 to 0.388 is a simple confirmation that the task became structurally easier. So I would not treat 0.91 as an unbiased benchmark of task difficulty. The stress-test note about test-set leakage is in the right family of concern, though I'd call it task-design leakage rather than classical overfitting.\n\nNone of this kills the dataset. The 13-class grouping is not crazy—proposition/lemma/theorem etc. do form a natural nest—but it needs external justification. A human baseline would help, and so would presenting the 50-class result with full-document context rather than the reduced task as the main event. The limitations section is honest; the paper itself acknowledges the lack of human evaluation.\n\nThis is a paper for NLP people working on scientific text and math-aware models. The resource deserves a serious referee, and I would accept it in peer review with requests to reframe the F1 claim and add an independent grouping justification. I'd cite the dataset if I worked in the area, but I would not cite 0.91 as evidence without the caveat.","headline":"A valuable open dataset and pipeline for scientific statement classification, but the headline 0.91 F1 is partly by construction and needs reframing.","tokens_in":9232,"tokens_out":2621,"would_cite":true,"duration_ms":24838,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a 13-class scientific statement classification task built from 10.4 million author-marked arXiv paragraphs and reports a 0.91 F1 bidirectional LSTM baseline.","keywords":["scientific statement classification","paragraph classification","arXiv","author markup annotations","mathematical formula lexemes","confusion matrix grouping","BiLSTM encoder-decoder","scholarly NLP benchmark"],"falsifier":"A human-annotation study on a random sample of the held-out test paragraphs, asking trained readers to assign the thirteen labels, would settle whether the 0.91 F1 reflects genuine separability: if human agreement is materially below or far above the model's score, or if humans cannot reliably distinguish the merged proposition nest, the benchmark's validity would be in question. Alternatively, training the same BiLSTM on thirteen classes chosen independently of the confusion analysis, for example by a panel of domain experts, and comparing F1 would reveal how much of the reported improvement is due to the grouping procedure.","tokens_in":8144,"feed_emoji":"📚","tokens_out":5887,"duration_ms":49455,"temperature":0.7,"pith_summary":"The paper argues that author-supplied LaTeX statement environments, such as theorem, proof, and remark, can be harvested at scale from a machine-readable conversion of the arXiv preprint corpus, yielding more than 10 million labeled paragraphs. It proposes grouping the fifty original author-annotated categories into thirteen 'nest' classes based on where a strong baseline model confuses them, and reports that this reduced task reaches a 0.91 F1 score with a bidirectional LSTM encoder-decoder, up from 0.67 on the full fifty-class version. The central claim is that this constitutes a valid large-scale task for scientific statement classification, with reproducible baselines and a publicly released resource. A sympathetic reader would care because distinguishing definitions, proofs, results, examples, and related work is a necessary step toward semantic enrichment of the scientific literature, and the paper offers a tractable, data-rich starting point.","feed_headline":"BiLSTM hits 0.91 F1 on 13-class scientific statement benchmark","feed_subtitle":"A 10.5-million-paragraph dataset of marked-up scientific statements gives NLP models a reproducible target with baselines.","key_machinery":"The load-bearing machinery is a machine-readable HTML5 conversion of the preprint corpus that preserves the exact environment scoping of author markup, together with a preprocessing pipeline that extracts the first logical paragraph of each marked-up environment, filters by language and conversion quality, and serializes mathematical expressions as font- and case-preserving lexeme tokens. The task design itself is carried by confusion-matrix nesting: classes that a BiLSTM systematically confuses are merged into abstract parent classes, most notably a proposition nest that absorbs lemma, theorem, corollary, conjecture, and related labels, while clearly separable classes such as abstract, acknowledgement, and proof are kept intact. This nesting procedure is what converts a noisy fifty-way labeling problem into the thirteen-class benchmark.","core_discovery":"On the paper's own terms, the central discovery is that the fifty author-annotated paragraph labels in arXiv submissions collapse, under a confusion-matrix analysis of a BiLSTM classifier, into thirteen separable classes while retaining 99% of the 10.5 million annotated paragraphs. The same model family that scored 0.67 F1 on the fifty-class version reaches 0.91 F1 on the thirteen-class version, and a hierarchical attention network reaches 0.89, establishing the reduced task as well-posed for current methods. The paper also introduces a lexeme serialization for mathematical formulas that preserves font, weight, and case distinctions, and observes a small but consistent 0.01 F1 gain for context-aware models when the math modality is included.","pith_inferences":["The headline F1 improvement from 0.67 to 0.91 is substantially an artifact of label regrouping; because the nests are constructed from the same model family's errors, the benchmark is partly circular and would be strengthened by an independent human-agreement study. ","If the thirteen classes reflect genuine linguistic nests, the same grouping should transfer to other machine-readable scholarly corpora without retraining the confusion analysis, which is a testable prediction. ","The math-modality gain being confined to context-aware models suggests that sequence models with longer context, such as transformers, may show a larger effect from formula lexemes than the 0.01 F1 observed with LSTMs."],"forward_implications":["If the task is accepted, follow-up work has a 10.4-million-paragraph benchmark with thirteen labels, a held-out test set of 2.1 million paragraphs, and baselines ranging from 0.77 to 0.91 F1. ","The math lexeme serialization lets standard NLP pipelines include symbolic modality; the paper finds a 0.01 F1 gain for context-sensitive models when math tokens are retained. ","The confusion-nesting method offers a general recipe for task design on weakly labeled scholarly corpora: use a strong baseline's confusion matrix to define separable label nests. ","The paper's stated next step is a sequence-of-paragraphs classification task with full-document context, which should particularly help positionally anchored labels like abstract and conclusion."],"supporting_citations":[{"why":"Supplies the machine-readable HTML5 corpus of 1.2 million documents from which the labeled paragraphs are extracted.","marker":"[2]"},{"why":"The LaTeX-to-XML converter that produces the structured representation preserving environment markup and math structure.","marker":"[3]"},{"why":"Implements paragraph extraction, language filtering, tokenization, and math lexeme serialization used for the dataset.","marker":"[5]"},{"why":"Provides the 300-dimensional word embeddings that all baseline models use as input representation.","marker":"[6]"},{"why":"The Hierarchical Attention Network baseline whose 0.89 F1 score confirms the reduced task is learnable.","marker":"[15]"},{"why":"The encoder-decoder architecture family used for the best-performing baseline model.","marker":"[17]"},{"why":"The earlier segment-classification attempt over arXiv that this task extends and contrasts with.","marker":"[21]"}],"fun_headline_variants":["13 classes, 0.91 F1: arXiv statements get a cleaner benchmark","Collapsing 50 labels to 13 yields 0.91 F1 for scientific statements","10.5M paragraph dataset condenses to 13 high-scoring classes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that merging the original fifty labels into thirteen nests according to the confusion matrix of the same BiLSTM model family produces a task that is meaningfully aligned with how scientific statements are actually distinguished, rather than a redefinition tailored to what that model can already separate.","fun_headline_variants_meta":{"raw":{"variants":["13 classes, 0.91 F1: arXiv statements get a cleaner benchmark","Collapsing 50 labels to 13 yields 0.91 F1 for scientific statements","10.5M paragraph dataset condenses to 13 high-scoring classes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000428,"raw_usage":{"total_tokens":2132,"prompt_tokens":828,"completion_tokens":1304,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":1235}},"tokens_in":444,"tokens_out":1304,"duration_ms":11027,"temperature":1.0,"reasoning_tokens":1235,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:26:32.069356+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A human-annotation study on a random sample of the held-out test paragraphs, asking trained readers to assign the thirteen labels, would settle whether the 0.91 F1 reflects genuine separability: if human agreement is materially below or far above the model's score, or if humans cannot reliably distinguish the merged proposition nest, the benchmark's validity would be in question. Alternatively, training the same BiLSTM on thirteen classes chosen independently of the confusion analysis, for example by a panel of domain experts, and comparing F1 would reveal how much of the reported improvement is due to the grouping procedure.","supporting_citations":[{"cited_title":"arXMLiv:08.2018 dataset, an HTML5 conversion of arXiv.org","cited_arxiv_id":null,"evidence_quote":"Supplies the machine-readable HTML5 corpus of 1.2 million documents from which the labeled paragraphs are extracted."},{"cited_title":"LaTeXML: A LATEX to XML converter","cited_arxiv_id":null,"evidence_quote":"The LaTeX-to-XML converter that produces the structured representation preserving environment markup and math structure."},{"cited_title":"LLaMaPUn: common language and mathematics processing algorithms","cited_arxiv_id":null,"evidence_quote":"Implements paragraph extraction, language filtering, tokenization, and math lexeme serialization used for the dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the 300-dimensional word embeddings that all baseline models use as input representation."},{"cited_title":"Smola, and Eduard H","cited_arxiv_id":null,"evidence_quote":"The Hierarchical Attention Network baseline whose 0.89 F1 score confirms the reduced task is learnable."},{"cited_title":"Logical structure analysis of scientific publications in mathematics","cited_arxiv_id":null,"evidence_quote":"The earlier segment-classification attempt over arXiv that this task extends and contrasts with."}],"review_version":1}