From Global to Local: Learning Context-Aware Graph Representations for Document Classification and Summarization

Gerard de Melo; Margarita Bugue\~no; Ruangrin Ldallitsakool

REVIEW 4 major objections 6 minor 6 references

From Global to Local: Learning Context-Aware Graph Representations for Document Classification and Summarization

T0 review · 4 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Document graphs built from sliding-window attention match or beat full-attention graphs at a fraction of the computational cost, with unfiltered local edges working best on medium-length documents.

desk verdict SWA graph construction is a modest but real win on medium-length documents; the computational-efficiency claim is not supported by the paper's own numbers. read the letter →

arxiv 2603.00021 v2 pith:LV4EMN7C submitted 2026-02-03 cs.CL

Ruangrin Ldallitsakool , Margarita Bugueño , Gerard de Melo This is my paper

classification cs.CL

keywords documentclassificationgraphneuralnetworksattentionsliding-windowlearnedconstructiongraphsextractivesummarizationlongdocuments

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most document classifiers read text as a token sequence, which makes long-range structure hard to model. This paper argues that a better representation is a graph whose nodes are sentences and whose edges are learned by sliding-window attention: each sentence attends only to nearby sentences, and the resulting attention weights become weighted edges. Trained on such graphs, graph attention networks match or beat both hand-built heuristic graphs and graphs built with full self-attention on three document-classification benchmarks, while needing substantially less computation. A notable empirical finding is that, on medium-length documents, unfiltered local graphs outperform statistically filtered ones, suggesting that weak local edges carry genuinely useful signal. The paper also finds that the same graph construction gives semantically central sentence selections for extractive summarization but does not beat simple ordering baselines under oracle-label evaluation, locating the bottleneck in the selection objective rather than the graph structure.

What carries the argument

The load-bearing object is the sliding-window attention (SWA) module, a multi-head self-attention layer where each sentence embedding attends only to sentences within a window defined as a percentage of document length. The attention weights, optionally transformed by a nonlinear activation (softmax, ReLU, sigmoid, or annealed softmax) and optionally pruned by row-wise mean-bound or max-bound thresholds, become edge weights of an undirected sentence graph; sentence nodes with identical embeddings are merged, and self-loops are redistributed to neighbors. A Graph Attention Network trained from scratch on that graph performs classification or sentence labeling. The window converts global quadr

What would settle it

Run the same sliding-window attention graph construction and GAT training on BBC, HND, and AX under the exact splits and preprocessing used for the baselines, and compare with independently reproduced full-attention graphs. If unfiltered SWA graphs do not beat or match the best full-attention graphs on BBC/HND, or if the gap on AX grows well beyond the reported 0.3 F1, the paper's central claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that a data-driven document graph can be induced from local, sliding-window attention rather than global self-attention without sacrificing, and in some settings improving, task performance. Concretely, on BBC News and Hyperpartisan News Detection, unfiltered sliding-window attention graphs give the best GAT classification results, outperforming both heuristic graphs and full-attention graphs; on arXiv classification they remain competitive, trailing the strongest full-attention graph by 0.3 F1 while beating all heuristics. On summarization, the induced graphs identify sentences near the semantic center of document clusters but over-predict summary-worthy sentences, so t

Load-bearing premise

The central comparison assumes that the previously reported full-attention and heuristic baseline results were obtained under exactly the same dataset splits, preprocessing, and evaluation protocol; if that assumption fails, the improvements claimed on the three classification datasets could be smaller or disappear.

Editorial extensions

If this is right

On BBC and HND, unfiltered sliding-window attention graphs give the best classification accuracy, outperforming both heuristic graphs and full-attention-induced graphs; the improvement on HND is roughly 1–3 points.
Replacing full self-attention with sliding-window attention changes graph construction from quadratic to linear cost in sentence count, and the resulting GAT models are shallow, reducing computational requirements.
The same induced graphs, used for extractive summarization, achieve high BERTScore to reference summaries but lower oracle-label F1 than order-based baselines, indicating the graphs capture semantic centrality, not the precise oracle selection pattern.
GAT classifiers trained on the induced graphs consistently outperform the standalone sliding-window attention classifiers used to produce the graphs, so converting attention to graph structure adds value beyond the transformer representation alone.
Window size matters only mildly: on HND, 10% and 50% windows are statistically indistinguishable, so the method does not hinge on a precisely tuned context width.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because unfiltered local edges help on medium-length documents, an adaptive sparsifier that keeps weak local edges while pruning weak global ones may outperform both all-or-nothing filtering regimes used here; this is directly testable on the same datasets.
The linear complexity of sliding-window graph construction suggests the approach could scale to documents of thousands of sentences where full attention is infeasible; a natural test is the same pipeline on a long-document benchmark with 500+ sentences per document.
The summarization result implies that the same graphs could support extractive summarization better with a redundancy-aware, ranking-based, or calibrated-threshold objective; replacing the binary sentence classifier is a concrete next experiment.
The paper's comparisons to prior full-attention graphs reuse previously reported numbers; independent reproduction under identical splits and preprocessing would be the cleanest way to confirm the magnitude of the claimed gains.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things you should know. First, the paper's main empirical finding does hold up on reading: swapping full attention for sliding-window attention when building learned document graphs gives small accuracy gains on BBC and HND, and notably the best configurations are unfiltered graphs, which contradicts the earlier filtering wisdom from the authors' own 2025 paper. Second, the paper's efficiency claim does not hold up: the configurations that win are denser and have more edges than the full-attention baselines, and no runtime or memory measurements are reported.

What's actually new: the window-size ablation on HND, the evidence that unfiltered local graphs can be competitive, and the negative result on extractive summarization, which is honestly discussed. The statistical analysis in the appendix is solid; they report multiple runs with standard deviations and do proper ANOVA/Tukey tests for their internal comparisons. The structural analysis of degree distributions is a nice touch.

Soft spots. The abstract and Section 5.1 claim 'substantially more computationally efficient than unrestricted self-attention.' Table 2 contradicts that for the best configurations: the SWA graphs that achieve the best accuracy have average degrees of 27.8 and 41.6 on BBC/HND versus 9.5 and 8.9 for the full-attention baselines, and on AX the best SWA mean-bound graph has 14,662 edges and 40 GB disk vs 1,092 edges and 25 GB for the full-attention max-bound baseline. The paper's own Section 5.2 acknowledges this, but the abstract still makes the efficiency claim. No wall-clock or peak-memory measurements are provided anywhere, so the efficiency half of the contribution is either false for the best configurations or unverified. The reuse of Bugueño and de Melo (2025) numbers for the full-attention baselines is pragmatic but not fully independent; if any preprocessing step differed, the gains could shift. The headline Table 2 also lacks error bars, though the appendix covers that for the internal comparisons. Minor but real problems: the HND sample count is 1,270 in Table 1 but 645 in Appendix A.1, and the text references 'Table 10' for the summarization results when it appears in Table 4, plus a literal '??' placeholder in Section 5.

Who should read this: anyone working on attention-induced graph construction for document classification. It's an incremental extension of the authors' own framework, but the empirical finding that unfiltered local attention graphs make the best use of GATs is worth knowing. It deserves a serious referee, but not a clean accept: the efficiency claim needs to be removed or backed with real measurements, and the comparability assumptions should be stated more carefully.

Referee Report

4 major / 6 minor

Summary. This paper extends the graph-structure-learning framework of Bugueño and de Melo (2025) by replacing full self-attention with dynamic sliding-window attention. Sentence embeddings are fed to a multi-head SWA model, the resulting attention weights are thresholded (or not) to build document graphs, and GAT models are trained on these graphs for document classification (BBC, HND, AX) and, exploratorily, extractive summarization (GovReport). The authors report that SWA-induced graphs outperform heuristic and full-attention baselines on two medium-length datasets, are competitive on a long-document dataset, and claim lower computational resource requirements. They also study activation functions, statistical filtering, window size, and provide an analysis of the summarization setup.

Significance. If the efficiency and accuracy claims were fully supported, the paper would offer a practical simplification of learned graph construction for document classification, with a useful analysis of local versus global attention. The strengths include a public implementation, multiple independent runs (five per configuration), statistical tests for internal variant comparisons (ANOVA/Tukey HSD), a window-size ablation, and an honest exploratory summarization evaluation that does not overstate its findings. However, the central efficiency claim is contradicted by the paper's own structural statistics, and the comparisons against prior work rely on previously published point estimates without significance testing or explicit verification of identical protocols. The empirical contribution is therefore real but currently overstated.

major comments (4)

[Abstract; §5.1; §5.2; Table 2] The claim that the method 'require[s] lower computational resources than previous approaches' is not supported—and in the best-performing configurations is contradicted—by the reported data. On BBC and HND, the best SWA configurations are the unfiltered graphs, with average degrees of 27.8 and 41.6 and 540.8/842.7 edges, versus 9.5/8.9 and 245.8/329.6 for the best full-attention baselines; the BBC disk size is also larger (109 M vs 90 M). On AX, the best SWA result (mean-bound) has 14,662.3 edges and 40 GB storage versus 1,092.2 edges and 25 GB for the full-attention max-bound baseline, and is 0.3 F1 lower. Only the mean-bound SWA graphs on BBC/HND are cheaper, but they are not the best-performing variants. The paper reports no wall-clock time or peak-memory measurements, so graph edge/disk counts are an incomplete proxy. The efficiency claim should be either removed, substantially weake
[§5.1; Table 2; Appendix C] The reported gains over the full-attention and heuristic baselines are point estimates without confidence intervals or significance tests. The text states that the unfiltered SWA graphs yield 'statistically significant gains (p < 0.05)', but the ANOVA/Tukey analyses in Appendix C compare only the SWA variants among themselves; they do not include the Bugueño and de Melo (2025) baseline runs. Without per-run results for the baselines, or a paired test under identical conditions, the significance claim cannot be verified. Please either provide the baseline runs with the same protocol and perform a proper statistical comparison, or report effect sizes/error bars for all entries in Table 2.
[§4.1; Table 2] The comparison to Bugueño and de Melo (2025) reuses published numbers from that paper. Section 4.1 states that the datasets are reused to ensure 'direct comparability', but it does not specify that the train/validation/test splits, preprocessing (e.g., sentence tokenization, embedding model, truncation), and evaluation protocol are identical to those used in the cited work. If any of these differ, the claimed improvements over the baselines could disappear. Please state the exact split and preprocessing protocol, or rerun the baselines in the same codebase.
[§5.3; Table 3; Table 5] The main experiments use a window size of 30%, but the ablation on HND shows that window size 30 is consistently and statistically significantly worse than window sizes 10, 20, 40, and 50. The paper interprets this as making the main results 'conservative', yet it also means the headline improvements were obtained with a suboptimal hyperparameter while better settings exist. This does not invalidate the approach, but it should be made more prominent in the main text and reflected in the interpretation of the absolute gains.

minor comments (6)

[§3.1, Eq. (3)] The definition 'd = dim(A − 1)' is confusing. Presumably d is the last dimension of A (i.e., A.shape[-1]). Please clarify the notation.
[§5.4] The text refers to 'the results in Table 10' but the main-text GR results appear in Table 4. Table 10 is the appendix table. Please correct the cross-reference.
[References] The citation 'Waswani et al. (2017)' should be 'Vaswani et al. (2017)'.
[§3.1] The method operates on sentences, but the text repeatedly says 'tokens within the window'. Please use consistent terminology.
[§5.1] There is a typo: 'These results exreveal' should be 'These results reveal'.
[§5.1] The sentence 'The main limitation of these locally-aware graphs is that it cannot directly model dependencies' has a subject-verb agreement issue.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the method is an empirical evaluation with self-cited baselines that constitute independent evidence.

full rationale

This paper does not contain a derivation chain in which an output is defined in terms of its own input. The core pipeline is: sentence embeddings are encoded, sliding-window attention (SWA) weights are learned, graphs are built from thresholded attention weights, and GATs are trained on those graphs and evaluated on held-out test data. The attention weights are used to construct graph structure, not to directly produce the final classification, so the GAT evaluation is not a renaming of the SWA outputs. No parameter is fitted to a target quantity and then reported as a prediction of that same quantity. The reuse of Bugueño and de Melo (2025) full-attention and heuristic baseline numbers is a self-citation, but it is not load-bearing in the circularity sense: these are external empirical results independently produced in prior work, and the current paper's own Table 2 and statistical tests provide new evidence for its claims. The efficiency claim may be weakened by the density figures in Table 2 and the absence of runtime measurements, but that is an evidence-strength and correctness concern, not a circularity concern. The paper also honestly reports limitations, including the summary evaluation being against oracle labels only and the absence of broad domain coverage. No circular step meeting the required evidentiary standard was found.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method rests on a small number of hand-set hyperparameters (window size, tolerance, annealing rate) and on the assumption that a frozen sentence encoder and shallow task-trained attention weights produce semantically meaningful graphs. It introduces no new theoretical entities.

free parameters (3)

attention window size = 30% of document length (main experiments); ablated 10-50%
Chosen based on prior work and ablation; main experiments use 30%, though ablation shows 10 and 50 better on HND.
tolerance degree δ = 0.5 (all settings), 1.0 for AX mean-bound
Hand-set threshold adjustment for edge filtering; altered for AX to improve resource efficiency (Appendix B.3).
annealing rate λ = 0.0004 per batch
Controls temperature schedule in Eq. 4; chosen without reported tuning.

assumptions (4)

domain assumption Sentence embeddings from frozen paraphrase-MiniLM-L6-v2 capture sufficient semantic information for sentence-level attention.
Used as input to SWA models (Section B.2); if embeddings are inadequate, attention weights and graphs are compromised.
domain assumption Attention weights from a shallow task-trained SWA model express meaningful semantic dependencies between sentences.
Core to graph construction (Section 3); the paper provides indirect evidence via downstream GAT performance, but this is assumed.
domain assumption The datasets, splits, and evaluation protocol are identical to Bugueño and de Melo (2025) for the reused baselines.
Section 4.1 states reuse of datasets, but baselines are taken from prior paper; any mismatch would invalidate comparisons.
domain assumption Row-wise mean-bound and max-bound filtering with a tolerance degree generalizes from prior work.
Thresholds adopted from Bugueño and de Melo (2025) without re-derivation (Section 3.2).

how reviews work

0 comments

Cite this review

Pith. "Pith review of From Global to Local: Learning Context-Aware Graph Representations for Document Classification and Summarization." pith.science (2026). https://pith.science/paper/LV4EMN7C

@misc{pith2026260300021,
  author       = {Pith},
  title        = {Pith review of: From Global to Local: Learning Context-Aware Graph Representations for Document Classification and Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LV4EMN7C}},
  note         = {Machine review of arXiv:2603.00021}
}

read the original abstract

Recent NLP systems commonly represent documents as linear token sequences. Although this captures sequential order, it can hinder modeling long-range dependencies and global document structure, especially for long texts. This paper proposes a data-driven method to automatically construct graph-based document representations. Building upon the recent work of Bugue\~no and de Melo (2025), we leverage the dynamic sliding-window attention module to effectively capture local and mid-range semantic dependencies between sentences, as well as structural relations within documents. Graph Attention Networks (GATs) trained on our learned graphs achieve competitive results on document classification while requiring lower computational resources than previous approaches. We further present an exploratory evaluation of the proposed graph construction method for extractive document summarization, highlighting both its potential and current limitations. The implementation of this project can be found on GitHub.

Figures

Figures reproduced from arXiv: 2603.00021 by the authors.

**Figure 2.** Overview of our experiment pipeline. The [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** A sample of attention matrices from a randomly selected document in the GovReport dataset. From Top [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: The figure shows the distribution of the summary sentences in relative positions (%) of the documents in [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]

**Figure 5.** Figure 5: The Figure shows the occurrences of summary sentences from random documents in the test split. Note [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]

**Figure 6.** Figure 6: Left: The numbers of Oracle summary sentences (y-axis), compared to the numbers of predicted summary [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]

**Figure 7.** Figure 7: Text similarity scores on summaries generated from the single-layer GAT model trained on ReLU-based [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]

**Figure 8.** Figure 8: t-SNE visualizations of random GR documents. Left-side images show the input feature representations, [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

6 extracted references · 3 linked inside Pith

[6]

Dongsheng Wang, Zhiqiang Ma, Armineh Nourbakhsh, Kang Gu, and Sameena Shah

Graph attention networks.arXiv preprint arXiv:1710.10903. Dongsheng Wang, Zhiqiang Ma, Armineh Nourbakhsh, Kang Gu, and Sameena Shah. 2023. Docgraphlm: Documental graph language model for information extraction. InProceedings of the 46th International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, pages 1944–1948. Kunze Wang,...

arXiv 2023
[2007]

Jun He, Liqun Wang, Liu Liu, Jiao Feng, and Hao Wu

Random walk term weighting for improved text classification.International Journal of Semantic Computing, 1(04):421–439. Jun He, Liqun Wang, Liu Liu, Jiao Feng, and Hao Wu
[2017]

John W Tukey

Activation functions in neural networks.To- wards Data Sci, 6(12):310–316. John W Tukey. 1949. Comparing individual means in the analysis of variance.Biometrics, pages 99–114. Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, Koray Kavukcuoglu, and 1 others. 2016. Wavenet: A genera...

arXiv 1949
[2019]

Yilun Hua, Zhaoyuan Deng, and Kathleen McKeown

Long document classification from local word glimpses via recurrent attention learning.IEEE Ac- cess, 7:40707–40718. Yilun Hua, Zhaoyuan Deng, and Kathleen McKeown
[2022]

InProceedings of the 29th international conference on computational linguis- tics, pages 1163–1168

Contexting: granting document-wise contex- tual embeddings to graph neural networks for induc- tive text classification. InProceedings of the 29th international conference on computational linguis- tics, pages 1163–1168. G Thomas Hudson and Noura Al Moubayed. 2022. Muld: The multitask long document benchmark. arXiv preprint arXiv:2202.07362. Johannes Kies...

arXiv 2022
[2023]

InFindings of the As- sociation for Computational Linguistics: ACL 2023, pages 13851–13883

Improving long dialogue summarization with semantic graph representation. InFindings of the As- sociation for Computational Linguistics: ACL 2023, pages 13851–13883. Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. Efficient attentions for long document summarization.arXiv preprint arXiv:2104.02112. Yen-Hao Huang, Yi-Hsin Chen, an...

arXiv 2023

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [6]

Dongsheng Wang, Zhiqiang Ma, Armineh Nourbakhsh, Kang Gu, and Sameena Shah

Graph attention networks.arXiv preprint arXiv:1710.10903. Dongsheng Wang, Zhiqiang Ma, Armineh Nourbakhsh, Kang Gu, and Sameena Shah. 2023. Docgraphlm: Documental graph language model for information extraction. InProceedings of the 46th International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, pages 1944–1948. Kunze Wang,...

arXiv 2023

[2] [2007]

Jun He, Liqun Wang, Liu Liu, Jiao Feng, and Hao Wu

Random walk term weighting for improved text classification.International Journal of Semantic Computing, 1(04):421–439. Jun He, Liqun Wang, Liu Liu, Jiao Feng, and Hao Wu

[3] [2017]

John W Tukey

Activation functions in neural networks.To- wards Data Sci, 6(12):310–316. John W Tukey. 1949. Comparing individual means in the analysis of variance.Biometrics, pages 99–114. Aaron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, Koray Kavukcuoglu, and 1 others. 2016. Wavenet: A genera...

arXiv 1949

[4] [2019]

Yilun Hua, Zhaoyuan Deng, and Kathleen McKeown

Long document classification from local word glimpses via recurrent attention learning.IEEE Ac- cess, 7:40707–40718. Yilun Hua, Zhaoyuan Deng, and Kathleen McKeown

[5] [2022]

InProceedings of the 29th international conference on computational linguis- tics, pages 1163–1168

Contexting: granting document-wise contex- tual embeddings to graph neural networks for induc- tive text classification. InProceedings of the 29th international conference on computational linguis- tics, pages 1163–1168. G Thomas Hudson and Noura Al Moubayed. 2022. Muld: The multitask long document benchmark. arXiv preprint arXiv:2202.07362. Johannes Kies...

arXiv 2022

[6] [2023]

InFindings of the As- sociation for Computational Linguistics: ACL 2023, pages 13851–13883

Improving long dialogue summarization with semantic graph representation. InFindings of the As- sociation for Computational Linguistics: ACL 2023, pages 13851–13883. Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. Efficient attentions for long document summarization.arXiv preprint arXiv:2104.02112. Yen-Hao Huang, Yi-Hsin Chen, an...

arXiv 2023