Pith. sign in

REVIEW 4 major objections 5 minor 22 references

StructFormer: Document Structure-based Masked Attention and its Impact on Language Model Pre-Training

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pretraining with document headings as global attention tokens improves downstream document understanding while preserving general language ability.

desk verdict A sensible controlled study of header-global attention during MLM pretraining, but the central effect is not yet established because every comparison is a single run with no variance estimates. read the letter →

arxiv 2411.16618 v1 pith:ANRJBGQ7 submitted 2024-11-25 cs.CL

Kaustubh Ponkshe , Venkatapathy Subramanian , Natwar Modani , Ganesh Ramakrishnan This is my paper · ORCID
classification cs.CL
keywords StructFormerglobalattentiondocumentstructurelanguagemodelpre-trainingLongformerSciREXGLUE
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that injecting document structure into language-model pretraining, by marking title and section-heading tokens as global attention points, improves the model's document understanding without sacrificing general-purpose language ability. The authors build two parallel corpora from arXiv LaTeX sources, pretrain an identical Longformer on each, and compare the structure-aware version (StructFormer) with the text-only version. On the SciREX scientific information-extraction benchmark, StructFormer raises salient mention clustering F1 to 0.3419 from 0.3182 for the same Longformer trained without global tokens, and attention analysis shows more than 20 percent higher attention from section headers to annotated keywords. If the attribution is right, document markup is a free and reusable source of supervision for pretraining, and structural awareness can be learned rather than hard-coded at fine-tune time.

What carries the argument

The central object is a masked-attention configuration over a sparse transformer: tokens extracted from LaTeX headings (title, section, subsection, subsubsection) are designated global tokens, so every token can attend to them and they attend to all tokens, while all other tokens only attend within a 256-token local window. This converts document structure into a pretraining signal without any new labels, and the attention from header tokens to content keywords becomes the mechanism the paper offers for why downstream document understanding improves.

What would settle it

Run both pretraining recipes with, say, five random seeds each and compare the distributions of SciREX salient-cluster F1; if the 0.3419 versus 0.3182 gap falls within seed noise, the claimed structure-aware benefit collapses. Recomputing the more-than-20-percent header-keyword attention advantage with automatically extracted keywords rather than hand-annotated ones would test whether the mechanism is robust.

Watch

Extended reading notes

Core claim

StructFormer pretrains the Longformer architecture with LaTeX-derived headings as global tokens (mask set to 1) while all other tokens keep windowed local attention with a window size of 256; the control model is identical with all global masks set to zero. The paper reports that this single change lowers held-out bits-per-character from 2.3051 to 2.2136, produces qualitatively different attention patterns with header tokens attending to keywords with more than 20 percent higher scores, and improves SciREX salient mention clustering to F1 0.3419 from 0.3182 for the vanilla Longformer and 0.3070 for the SciREX baseline. GLUE results stay comparable to both BERT-base and the vanilla Longformer on most tasks. The paper interprets this as evidence that structure-aware pretraining teaches the model to use document organization, not just surface text, and that this transfers to document-level extraction tasks.

Load-bearing premise

The controlled comparison assumes that the only difference between StructFormer and Vanilla Longformer is the global-attention mask, and that the reported score gaps (for example, SciREX salient clusters 0.3419 versus 0.3182) are larger than run-to-run noise, but no seeds, repeats, or variance estimates are reported.

Editorial extensions

If this is right

  • Document structure can be added to BERT-style pretraining at no annotation cost beyond the markup already present in the source.
  • The gap on SciREX salient mention clustering, 0.3419 versus 0.3182 for the same model without global tokens, suggests structure-aware pretraining helps document-level information extraction.
  • Attention patterns shift from uniform recency-biased distributions to structured header-keyword attention, offering a visible route for why the model improves.
  • GLUE results indicate the structure-aware model does not lose general language ability compared with vanilla Longformer or BERT-base on most tasks.
  • The approach is architecture-agnostic: any sparse-attention BERT variant can be trained with header tokens as global attention.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same recipe should transfer to other structured corpora with explicit markup, such as books with chapter headings, HTML pages, patents, or legal documents, because the pretraining signal is positional and markup-based rather than domain-specific.
  • Header-keyword attention strength could be turned into a zero-shot signal for extractive tasks like section-aware summarization or keyword identification, although the paper does not test this.
  • If the SciREX gain is real, fine-tuning effort on document information extraction could be reduced: a structure-aware model may need fewer epochs or examples to reach baseline performance, which is not reported and would be a direct test.
  • Re-running the controlled comparison with several seeds would likely show the true size of the effect; small GLUE differences such as QNLI 90.8 versus 91.0 are within typical run-to-run noise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes StructFormer, a Longformer-based masked language model pre-trained with structure-aware global attention restricted to header tokens, on a corpus of arXiv LaTeX documents. It compares this model against a vanilla Longformer pre-trained on the same corpus without global tokens, and against a SciREX baseline, on SciREX document-level information extraction and on several GLUE tasks, and it analyzes attention patterns between header tokens and manually annotated keywords. The central claim is that structure-aware pre-training improves performance on document-understanding tasks such as SciREX salient clustering without harming generalizability on GLUE.

Significance. If the central claim holds, the work offers a low-cost way to inject document structure into BERT-style pre-training using only markup already present in LaTeX, with a controlled ablation isolating the effect of global attention. Strengths include a genuinely controlled comparison (same architecture, same 100k-document corpus, differing only in the global attention mask), evaluation on a held-out external benchmark (SciREX) plus GLUE, and a mechanistic attention analysis that directly tests the proposed mechanism. However, the empirical evidence is currently preliminary: single-run comparisons without variance estimates or significance tests leave the attribution to structure-aware pre-training insufficiently supported, and the attention analysis lacks dataset size, annotation details, and uncertainty quantification.

major comments (4)
  1. [§5.1, Table 4] The central comparison on SciREX rests on a single run of each pre-training and fine-tuning configuration; no random seeds, repeated runs, or variance estimates are reported anywhere, and the 2.37-point F1 gap in Salient Clusters (0.3419 vs 0.3182) is small relative to typical run-to-run variability in multi-stage IE pipelines. Please report results over multiple seeds (e.g., 3–5 runs) with means and standard deviations, or provide statistical significance tests, before attributing this gap to structure-aware pre-training.
  2. [§5.4, Table 5] Several GLUE gaps used to support the generalizability and positive-effect claims are within typical fine-tuning noise (e.g., QNLI 0.910 vs 0.908, SST2 0.933 vs 0.928, CoLA 0.469 vs 0.502), and no repeated runs or confidence intervals are reported; the claim that structure-aware pre-training 'leads to positive effects' in MRPC, QNLI, and MNLI therefore needs multi-seed fine-tuning results to be substantiated.
  3. [§5.2] The attention analysis dataset is described only as 'a novel dataset' of 2023 arXiv documents with manually annotated keywords, without reporting the number of documents, headings, keyword instances, annotation protocol, inter-annotator agreement, or how attention scores were aggregated across layers and heads; the 'more than 20%' increase in header-keyword attention thus lacks the information needed to evaluate its reliability and generalizability. Please provide these details and error bars over documents and annotators.
  4. [§5.1, Table 2] The BPC evaluation does not define the held-out test split (size, document selection, masking procedure) used for Table 2, and the phrase '9,000 runs' is ambiguous (training steps, optimizer updates, or something else); clarify the experimental protocol so the reported BPC difference (2.2136 vs 2.3051) is interpretable.
minor comments (5)
  1. [Throughout] The model name is written inconsistently as 'StructFormer' and 'STRUCT FORMER'; unify the notation.
  2. [§2] Citation style is inconsistent (e.g., 'Kevin Clark et al.' in prose vs author-year elsewhere), and several references lack venue or page information.
  3. [Abstract/Introduction] There are several typos and spacing errors, including 'V ocabularyV' in the Introduction, 'pertraining', 'decarease', and 'Mathews' for Matthews correlation coefficient.
  4. [Figure 3 / Appendix A.1] Figure 3 is referenced in Section 5.2, but Appendix A.1 appears empty; provide the actual attention figure or remove the placeholder.
  5. [§5.1] The phrase '9, 000 runs' is ambiguous and the local attention window size is stated twice; rephrase to specify the number of training steps and state the window size once.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claim rests on a controlled pretraining comparison evaluated on external benchmarks, not on self-citation or definitional reduction.

full rationale

The paper's central claim is that adding header tokens as global attention during pretraining (StructFormer) improves downstream performance relative to an otherwise identical Longformer pretrained on the same corpus without global tokens. This comparison is a genuine controlled experiment: both models use the same architecture, the same 100,000-document arXiv corpus, and the only difference is the global attention mask, as stated in Section 5.1. The downstream evaluations on SciREX and GLUE are held-out, externally defined benchmarks, so the results are not fitted inputs renamed as predictions. The attention-pattern analysis in Section 5.2 uses a manually annotated keyword dataset, but this is an evaluation of the model's internal attention, not a parameter fitted to the downstream outcome; it may be a weak or biased probe, but it does not make the derivation circular. The paper contains no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled through prior work; its references to Longformer, SciREX, and GLUE are external sources. The Limitations section explicitly acknowledges that the findings are preliminary, which further indicates the authors do not claim a forced or definitional result. Accordingly, no step in the claimed derivation chain reduces to its own inputs, and the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim is empirical rather than derivational, so there are no fitted constants in the mathematical sense. The listed free parameters are hand-selected experimental conditions that both models share; the listed axioms are the domain assumptions needed for the comparison to mean what the paper says it means.

free parameters (3)
  • Pre-training corpus size
    Subset of 1,129,787 arXiv LaTeX files, filtered to 2,000 to 12,000 tokens each and limited to 100,000 documents; chosen for compute feasibility, not fitted to downstream results, but the central comparison depends on this single corpus size.
  • Local attention window size
    Fixed at 256 tokens for both models following Longformer's windowed attention; it shapes the attention patterns used as evidence in Section 5.2.
  • Document length filter
    Hand-chosen bounds of 2,000 to 12,000 tokens to 'avoid outliers' in Section 5.1; this affects which documents enter pretraining and is not varied or justified.
assumptions (3)
  • domain assumption LaTeX headers are the correct and sufficient global tokens to encode document structure for any downstream use.
    Introduced in Sections 3 and 4; the entire pretraining signal is derived from marking section and subsection headings as global tokens, with no alternative structure encodings tested.
  • domain assumption The manually annotated keyword dataset in Section 5.2 is representative and unbiased.
    The attention comparison relies on human-annotated keywords from 2023 arXiv documents; no annotation guidelines, inter-annotator agreement, or random baseline are reported.
  • domain assumption The two pretraining runs differ only in the global attention mask.
    Sections 5.1 and 5.4 attribute downstream differences to structure-aware pretraining; no seeds, repetitions, or variance are provided, so exact equivalence of training conditions is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of StructFormer: Document Structure-based Masked Attention and its Impact on Language Model Pre-Training." pith.science (2026). https://pith.science/paper/ANRJBGQ7

@misc{pith2026241116618,
  author       = {Pith},
  title        = {Pith review of: StructFormer: Document Structure-based Masked Attention and its Impact on Language Model Pre-Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ANRJBGQ7}},
  note         = {Machine review of arXiv:2411.16618}
}
read the original abstract

Most state-of-the-art techniques for Language Models (LMs) today rely on transformer-based architectures and their ubiquitous attention mechanism. However, the exponential growth in computational requirements with longer input sequences confines Transformers to handling short passages. Recent efforts have aimed to address this limitation by introducing selective attention mechanisms, notably local and global attention. While sparse attention mechanisms, akin to full attention in being Turing-complete, have been theoretically established, their practical impact on pre-training remains unexplored. This study focuses on empirically assessing the influence of global attention on BERT pre-training. The primary steps involve creating an extensive corpus of structure-aware text through arXiv data, alongside a text-only counterpart. We carry out pre-training on these two datasets, investigate shifts in attention patterns, and assess their implications for downstream tasks. Our analysis underscores the significance of incorporating document structure into LM models, demonstrating their capacity to excel in more abstract tasks, such as document understanding.

Figures

Figures reproduced from arXiv: 2411.16618 by the authors.

Figure 1
Figure 1. Illustration of our approach to empirically ana [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. SciREX pipeline with structure-aware corpus [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Attention patterns of structure-aware pre-training and vanilla pre-training between header and keywords [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Example of the extraction and storage of document structure in a text file [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Example (cont.) of the extraction and storage of document structure in a text file [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 6 canonical work pages

  1. [1]

    Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2016. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. arXiv preprint arXiv:1608.04207

    arXiv 2016

  2. [2]

    Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.19 ETC : Encoding long and structured inputs in transformers . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNL...

    doi:10.18653/v1/2020.emnlp-main.19 2020

  3. [4]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020 b . https://arxiv.org/abs/2004.05150 Longformer: The long-document transformer . Preprint, arXiv:2004.05150

    arXiv 2020

  4. [5]

    Shuyang Cao and Lu Wang. 2022. https://doi.org/10.18653/v1/2022.acl-long.58 HIBRIDS : Attention with hierarchical biases for structure-aware long document summarization . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 786--807, Dublin, Ireland. Association for Computational Linguistics

    doi:10.18653/v1/2022.acl-long.58 2022

  5. [6]

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509

    arXiv 2019

  6. [7]

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.04341

    arXiv 2019

  7. [8]

    Paul Ginsparg. 2011. Arxiv at 20. Nature, 476(7359):145--147

    work page 2011

  8. [9]

    Mario Giulianelli, Jacqueline Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. arXiv preprint arXiv:1808.08079

    arXiv 2018

Show all 22 references
  1. [10]

    Sarthak Jain, Madeleine Van Zuylen, Hannaneh Hajishirzi, and Iz Beltagy. 2020. Scirex: A challenge dataset for document-level information extraction. arXiv preprint arXiv:2005.00512

    2020 arXiv

  2. [11]

    Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, and Ming Zhou. 2020. https://doi.org/10.18653/v1/2020.coling-main.82 Docbank: A benchmark dataset for document layout analysis . pages 949--960

    2020 doi

  3. [12]

    Gomez, ukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, page 6000–6010, ...

    2017

  4. [13]

    Jesse Vig. 2019. A multiscale visualization of attention in the transformer model. arXiv preprint arXiv:1906.05714

    2019 arXiv

  5. [14]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/W18-5446 GLUE : A multi-task benchmark and analysis platform for natural language understanding . In Proceedings of the 2018 EMNLP Workshop B lackbox NLP : A...

    2018 doi

  6. [15]

    Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. 2022. Should you mask 15\ arXiv preprint arXiv:2202.08005

    2022 arXiv

  7. [16]

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020 a . Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283--17297

    2020

  8. [17]

    Manzil Zaheer, Guru Prashanth Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Minh Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Mahmoud El Houssieny Ahmed. 2020 b . https://proceedings.neurips.cc/paper/2020/file/c8512d142a2d849725f31a9a...

    2020

  9. [18]

    Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.692 HEGEL : Hypergraph transformer for long document summarization . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10167--10176, Abu ...

    2022 doi

  10. [19]

    Kelly W Zhang and Samuel R Bowman. 2018. Language modeling teaches you more syntax than translation does: Lessons learned through auxiliary task analysis. arXiv preprint arXiv:1809.10040

    2018 arXiv

  11. [20]

    Xu Zhong, Jianbin Tang, and Antonio Jimeno-Yepes. 2019. https://doi.org/10.1109/ICDAR.2019.00166 Publaynet: Largest dataset ever for document layout analysis

    2019

  12. [21]

    Mahdi Zolnouri, Xinlin Li, and Vahid Partovi Nia. 2020. Importance of data loading pipeline in training deep neural networks. arXiv preprint arXiv:2005.02130

    2020 arXiv

  13. [22]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  14. [23]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.