{"id":"df642d2f-88e3-46b8-87b0-ee443a7539dc","arxiv_id":"2411.16618","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using section headers as global attention tokens during masked-language-model pretraining improves downstream document-structure tasks such as SciREX salient clustering, with no clear drop on GLUE.","lead":"StructFormer pre-trains a Longformer model with document headings used as global attention tokens, and compares it with an identical model pre-trained on the same scientific corpus without structural attention. On the SciREX document extraction benchmark it improves salient-cluster detection, while GLUE results stay roughly flat, suggesting structure-aware pretraining can help without hurting general language ability.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central attribution rests on single-run comparisons with no variance; the main SciREX gap (Salient Clusters F1 0.3419 vs 0.3182) is small enough that seed noise could erase it.","rationale":"The reader's weakest assumption—that the two pretraining runs are identical except for the global attention mask and that observed differences exceed run-to-run noise—is exactly the load-bearing weak point. I agree with that assessment and find no independent fatal flaw. The corpus construction from LaTeX, the parallel text-only counterpart, the controlled Longformer ablation, and the use of SciREX as a document-level downstream task are all appropriate and give the paper a sensible experimental skeleton. The problem is evidentiary: every claimed benefit is presented as a point estimate from an unreported number of runs. For the primary result, Table 4, the structure-aware gain over vanilla Longformer is 0.0237 F1 on salient clustering; without a variance estimate or significance test it is impossible to distinguish this from seed-level noise in the multi-stage SciREX pipeline. The GLUE results are even less decisive, since several differences are fractions of a point in magnitude and go in opposite directions across tasks. The attention analysis is the only direct evidence for the proposed mechanism, but it also lacks the basic reporting needed to rule out cherry-picking or annotation subjectivity. None of this invalidates the idea; it means the paper should be read as a preliminary demonstration whose quantitative claims are conditional on a variance analysis. The Limitations section itself says the findings are preliminary, which is honest but does not supply the missing evidence. A CONDITIONAL verdict is therefore appropriate, matching the reader's judgment.","tokens_in":9533,"tokens_out":3183,"duration_ms":30411,"concrete_test":"Re-run the SciREX end-to-end fine-tuning and the GLUE fine-tuning for both StructFormer and vanilla Longformer with at least five random seeds (ideally including independent pretraining seeds for at least two pretrained checkpoints), then report per-seed means, standard deviations, and a paired bootstrap or permutation test on the salient-cluster F1 difference. If the 95% confidence interval for the difference overlaps zero, the causal attribution to structure-aware global attention is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that adding header tokens as global attention during pretraining (StructFormer) causes downstream gains over an identically pretrained Longformer. For that attribution to hold, observed differences must exceed run-to-run noise. No random seeds, repeated runs, or variance estimates are reported anywhere. In Table 4, the headline comparison is Salient Clusters F1 0.3419 vs 0.3182, a 2.37-point gap with precision and recall gaps of about 2.1 and 1.8 points. The SciREX pipeline is multi-stage and stochastic, and single-run end-to-end differences of this size are commonly within seed noise. The GLUE evidence is weaker still: QNLI 0.910 vs 0.908, SST2 0.933 vs 0.928, and CoLA 0.469 vs 0.502 are below or near typical fine-tuning variance, and the one larger GLUE gap (MRPC accuracy 0.900 vs 0.870) is also from a single run. The proposed mechanism, 'more than 20%' higher header-keyword attention, rests on a manually annotated set of 2023 arXiv documents with no stated size, annotation protocol, inter-annotator agreement, or error bars. The controlled corpus and vanilla ablation are the right experimental design, but without repeated runs the causal attribution to structure-aware global attention is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes StructFormer, a Longformer-based masked language model pre-trained with structure-aware global attention restricted to header tokens, on a corpus of arXiv LaTeX documents. It compares this model against a vanilla Longformer pre-trained on the same corpus without global tokens, and against a SciREX baseline, on SciREX document-level information extraction and on several GLUE tasks, and it analyzes attention patterns between header tokens and manually annotated keywords. The central claim is that structure-aware pre-training improves performance on document-understanding tasks such as SciREX salient clustering without harming generalizability on GLUE.","tokens_in":9864,"tokens_out":3173,"duration_ms":28451,"significance":"If the central claim holds, the work offers a low-cost way to inject document structure into BERT-style pre-training using only markup already present in LaTeX, with a controlled ablation isolating the effect of global attention. Strengths include a genuinely controlled comparison (same architecture, same 100k-document corpus, differing only in the global attention mask), evaluation on a held-out external benchmark (SciREX) plus GLUE, and a mechanistic attention analysis that directly tests the proposed mechanism. However, the empirical evidence is currently preliminary: single-run comparisons without variance estimates or significance tests leave the attribution to structure-aware pre-training insufficiently supported, and the attention analysis lacks dataset size, annotation details, and uncertainty quantification.","major_comments":[{"comment":"The central comparison on SciREX rests on a single run of each pre-training and fine-tuning configuration; no random seeds, repeated runs, or variance estimates are reported anywhere, and the 2.37-point F1 gap in Salient Clusters (0.3419 vs 0.3182) is small relative to typical run-to-run variability in multi-stage IE pipelines. Please report results over multiple seeds (e.g., 3–5 runs) with means and standard deviations, or provide statistical significance tests, before attributing this gap to structure-aware pre-training.","section":"§5.1, Table 4"},{"comment":"Several GLUE gaps used to support the generalizability and positive-effect claims are within typical fine-tuning noise (e.g., QNLI 0.910 vs 0.908, SST2 0.933 vs 0.928, CoLA 0.469 vs 0.502), and no repeated runs or confidence intervals are reported; the claim that structure-aware pre-training 'leads to positive effects' in MRPC, QNLI, and MNLI therefore needs multi-seed fine-tuning results to be substantiated.","section":"§5.4, Table 5"},{"comment":"The attention analysis dataset is described only as 'a novel dataset' of 2023 arXiv documents with manually annotated keywords, without reporting the number of documents, headings, keyword instances, annotation protocol, inter-annotator agreement, or how attention scores were aggregated across layers and heads; the 'more than 20%' increase in header-keyword attention thus lacks the information needed to evaluate its reliability and generalizability. Please provide these details and error bars over documents and annotators.","section":"§5.2"},{"comment":"The BPC evaluation does not define the held-out test split (size, document selection, masking procedure) used for Table 2, and the phrase '9,000 runs' is ambiguous (training steps, optimizer updates, or something else); clarify the experimental protocol so the reported BPC difference (2.2136 vs 2.3051) is interpretable.","section":"§5.1, Table 2"}],"minor_comments":[{"comment":"The model name is written inconsistently as 'StructFormer' and 'STRUCT FORMER'; unify the notation.","section":"Throughout"},{"comment":"Citation style is inconsistent (e.g., 'Kevin Clark et al.' in prose vs author-year elsewhere), and several references lack venue or page information.","section":"§2"},{"comment":"There are several typos and spacing errors, including 'V ocabularyV' in the Introduction, 'pertraining', 'decarease', and 'Mathews' for Matthews correlation coefficient.","section":"Abstract/Introduction"},{"comment":"Figure 3 is referenced in Section 5.2, but Appendix A.1 appears empty; provide the actual attention figure or remove the placeholder.","section":"Figure 3 / Appendix A.1"},{"comment":"The phrase '9, 000 runs' is ambiguous and the local attention window size is stated twice; rephrase to specify the number of training steps and state the window size once.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The controlled pretraining setup is a good experimental design for the research question, but the single-run evidence falls short of supporting the causal claims; I would ask for multi-seed results and a properly documented attention dataset before further consideration. The paper may also need a scope check with the target venue regarding the balance between empirical findings and methodological novelty."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"One thing to know: this is a reasonable incremental contribution to the sparse-attention pretraining line. Making LaTeX headers into global tokens during BERT-style MLM pretraining, with a matched Longformer trained on the same corpus without global tokens, is the right controlled design for the question. If the effect is real, it is a cheap way to inject document structure into pretraining. But the paper does not yet show the effect is real: all key comparisons are single runs, with no seeds, variance, or significance tests, and some gaps are within normal fine-tuning noise.\n\nWhat is actually new: the paper contributes the method (global tokens for headers during pretraining), an arXiv-derived structure-aware pretraining corpus with a matched text-only corpus, and a manually annotated attention dataset. The SciREX ablation is the right experiment: Vanilla Longformer on the same corpus is a fair control, and the StructFormer gain on salient clusters (34.2 vs 31.8 F1) plus the mostly flat GLUE results support the no-harm claim. The BPC difference (2.21 vs 2.31) is consistent with the mechanism.\n\nSoft spots: the biggest is missing variance. The SciREX pipeline is multi-stage and stochastic; a 2.37-point gap from one run is not enough to attribute to the mask. GLUE differences are mostly tiny except MRPC, also single-run. There are no released artifacts, no seeds, and no test-split definition for BPC. The attention analysis is suggestive but rests on manually annotated keywords with no stated dataset size, annotation protocol, or inter-annotator agreement. The paper's own Limitations section says the findings are preliminary, which is honest and should keep the authors' claims in check.\n\nOverall: for a reader working on efficient or document-aware pretraining, this is a useful pointer and a sensible design to build on, but not yet a demonstrated result. It deserves a serious referee; I would ask for repeated runs (at least three seeds), error bars, and release of model and corpus artifacts before acceptance. In the meantime, I would not cite it as evidence of a stable improvement.","headline":"A sensible controlled study of header-global attention during MLM pretraining, but the central effect is not yet established because every comparison is a single run with no variance estimates.","tokens_in":10350,"tokens_out":1554,"would_cite":false,"duration_ms":15866,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pretraining with document headings as global attention tokens improves downstream document understanding while preserving general language ability.","keywords":["StructFormer","global attention","document structure","language model pre-training","Longformer","SciREX","GLUE"],"falsifier":"Run both pretraining recipes with, say, five random seeds each and compare the distributions of SciREX salient-cluster F1; if the 0.3419 versus 0.3182 gap falls within seed noise, the claimed structure-aware benefit collapses. Recomputing the more-than-20-percent header-keyword attention advantage with automatically extracted keywords rather than hand-annotated ones would test whether the mechanism is robust.","tokens_in":9394,"feed_emoji":"📄","tokens_out":5658,"duration_ms":44533,"temperature":0.7,"pith_summary":"This paper claims that injecting document structure into language-model pretraining, by marking title and section-heading tokens as global attention points, improves the model's document understanding without sacrificing general-purpose language ability. The authors build two parallel corpora from arXiv LaTeX sources, pretrain an identical Longformer on each, and compare the structure-aware version (StructFormer) with the text-only version. On the SciREX scientific information-extraction benchmark, StructFormer raises salient mention clustering F1 to 0.3419 from 0.3182 for the same Longformer trained without global tokens, and attention analysis shows more than 20 percent higher attention from section headers to annotated keywords. If the attribution is right, document markup is a free and reusable source of supervision for pretraining, and structural awareness can be learned rather than hard-coded at fine-tune time.","feed_headline":"Document headings as global attention tokens lift extraction","feed_subtitle":"A structure-aware pretraining run improves document-level clustering on SciREX, while GLUE scores stay level.","key_machinery":"The central object is a masked-attention configuration over a sparse transformer: tokens extracted from LaTeX headings (title, section, subsection, subsubsection) are designated global tokens, so every token can attend to them and they attend to all tokens, while all other tokens only attend within a 256-token local window. This converts document structure into a pretraining signal without any new labels, and the attention from header tokens to content keywords becomes the mechanism the paper offers for why downstream document understanding improves.","core_discovery":"StructFormer pretrains the Longformer architecture with LaTeX-derived headings as global tokens (mask set to 1) while all other tokens keep windowed local attention with a window size of 256; the control model is identical with all global masks set to zero. The paper reports that this single change lowers held-out bits-per-character from 2.3051 to 2.2136, produces qualitatively different attention patterns with header tokens attending to keywords with more than 20 percent higher scores, and improves SciREX salient mention clustering to F1 0.3419 from 0.3182 for the vanilla Longformer and 0.3070 for the SciREX baseline. GLUE results stay comparable to both BERT-base and the vanilla Longformer on most tasks. The paper interprets this as evidence that structure-aware pretraining teaches the model to use document organization, not just surface text, and that this transfers to document-level extraction tasks.","pith_inferences":["The same recipe should transfer to other structured corpora with explicit markup, such as books with chapter headings, HTML pages, patents, or legal documents, because the pretraining signal is positional and markup-based rather than domain-specific.","Header-keyword attention strength could be turned into a zero-shot signal for extractive tasks like section-aware summarization or keyword identification, although the paper does not test this.","If the SciREX gain is real, fine-tuning effort on document information extraction could be reduced: a structure-aware model may need fewer epochs or examples to reach baseline performance, which is not reported and would be a direct test.","Re-running the controlled comparison with several seeds would likely show the true size of the effect; small GLUE differences such as QNLI 90.8 versus 91.0 are within typical run-to-run noise."],"forward_implications":["Document structure can be added to BERT-style pretraining at no annotation cost beyond the markup already present in the source.","The gap on SciREX salient mention clustering, 0.3419 versus 0.3182 for the same model without global tokens, suggests structure-aware pretraining helps document-level information extraction.","Attention patterns shift from uniform recency-biased distributions to structured header-keyword attention, offering a visible route for why the model improves.","GLUE results indicate the structure-aware model does not lose general language ability compared with vanilla Longformer or BERT-base on most tasks.","The approach is architecture-agnostic: any sparse-attention BERT variant can be trained with header tokens as global attention."],"supporting_citations":[{"why":"Supplies the Longformer sparse-attention architecture with local windowed and global attention, plus the 4096_base checkpoint used as the starting point for both pretraining runs.","marker":"Beltagy et al., 2020a"},{"why":"Supplies the SciREX benchmark, its end-to-end pipeline, and the baseline scores StructFormer is compared against.","marker":"Jain et al., 2020"},{"why":"Supplies the GLUE benchmark used to test whether structure-aware pretraining preserves general language understanding.","marker":"Wang et al., 2018"},{"why":"Supplies the DocBank-style LaTeX structure-extraction idea used to build the structure-aware pretraining corpus.","marker":"Li et al., 2020"},{"why":"Supplies arXiv as the source of the 1,129,787 LaTeX documents used to create both the structure-aware and text-only corpora.","marker":"Ginsparg, 2011"},{"why":"Supplies the original Transformer attention mechanism that sparse attention modifies.","marker":"Vaswani et al., 2017"},{"why":"Supplies the masked language modeling masking strategy used for the pretraining objective.","marker":"Wettig et al., 2022"}],"fun_headline_variants":["Document structure as global attention improves extraction","Headings as global tokens lift SciREX clustering","Structure-aware pretraining aids document understanding","Global heading attention boosts document-level tasks","Pretraining with LaTeX headings boosts extraction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The controlled comparison assumes that the only difference between StructFormer and Vanilla Longformer is the global-attention mask, and that the reported score gaps (for example, SciREX salient clusters 0.3419 versus 0.3182) are larger than run-to-run noise, but no seeds, repeats, or variance estimates are reported.","fun_headline_variants_meta":{"raw":{"variants":["Document structure as global attention improves extraction","Headings as global tokens lift SciREX clustering","Structure-aware pretraining aids document understanding","Global heading attention boosts document-level tasks","Pretraining with LaTeX headings boosts extraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1508,"prompt_tokens":890,"completion_tokens":618,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":553}},"tokens_in":506,"tokens_out":618,"duration_ms":5917,"temperature":1.0,"reasoning_tokens":553,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:54:48.568796+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run both pretraining recipes with, say, five random seeds each and compare the distributions of SciREX salient-cluster F1; if the 0.3419 versus 0.3182 gap falls within seed noise, the claimed structure-aware benefit collapses. Recomputing the more-than-20-percent header-keyword attention advantage with automatically extracted keywords rather than hand-annotated ones would test whether the mechanism is robust.","supporting_citations":[{"cited_title":"SciREX: A Challenge Dataset for Document-Level Information Extraction","cited_arxiv_id":"2005.00512","evidence_quote":"Supplies the SciREX benchmark, its end-to-end pipeline, and the baseline scores StructFormer is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies arXiv as the source of the 1,129,787 LaTeX documents used to create both the structure-aware and text-only corpora."}],"review_version":1}