Using section headers as global attention tokens during masked-language-model pretraining improves downstream document-structure tasks such as SciREX salient clustering, with no clear drop on GLUE.
Title resolution pending
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.CL 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
StructFormer: Document Structure-based Masked Attention and its Impact on Language Model Pre-Training
Using section headers as global attention tokens during masked-language-model pretraining improves downstream document-structure tasks such as SciREX salient clustering, with no clear drop on GLUE.