REVIEW 3 major objections 6 minor 35 references
Lightweight Person-Place Relation Extraction from Historical Newspapers with Dependency Graphs and Proximity Features
T0 review · 3 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read A lightweight system with no pretrained language model at classification can compete on person–place relation extraction from historical newspapers, with character distance carrying most of the signal.
desk verdict Honest shared-task notes with a solid external result, but the headline leakage claim overstates what document-grouped CV achieves and the internal comparisons have a tuning leak. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is a document-level graph assembled from dependency parses of each sentence, with three bridges—merging repeated mentions of the same entity, linking content words across sentences by embedding similarity, and adding geographic-containment edges—that connect otherwise isolated sentence trees. From this graph the system computes fifteen proximity and part-of-speech features per entity pair, then classifies with either small tree ensembles or compact graph attention networks (all under 847K parameters). The argument runs through two mechanisms: the minimum character-distance feature (Group A) is shown by ablation to dominate, and document-grouped cross-validation is sho
What would settle it
Compute grouped-CV estimates under entity-disjoint splits (ensuring no person or location mention appears in both training and validation folds) and compare them to the paper's document-grouped CV numbers; if macro recall drops by several points, the document-grouped protocol still leaks. Alternatively, train a classifier that predicts document identity from entity-pair features; if it succeeds at above-chance accuracy under document-grouped splits, leakage remains.
Extended reading notes
Core claim
The discovery is that on this task, distance is the signal: a single scalar—the minimum character distance between any mention of the person and any mention of the location—accounts for most of what a classifier can learn, and adding fifteen engineered features often hurts (e.g., German at drops from 0.408 to 0.385). A deterministic rule that thresholds graph path length outperformed all trained models in internal evaluation (0.549 vs 0.520 for the best tabular pass). The same proximity dominance explains cross-validated leakage: because entity mentions recur across documents, naive pair-level folds let models memorize document-specific patterns, inflating scores by 15–37 percentage points;
Load-bearing premise
The analysis assumes that grouping cross-validation folds by document removes the data leakage caused by entity mentions recurring across documents; but the same person or location can still appear in both a training document and a validation document, so mention-level leakage may persist and internal estimates may remain inflated.
Editorial extensions
If this is right
- Historical archives with millions of entity pairs can be classified on CPU-only hardware with sub-second training, making large-scale annotation practical.
- Any future relation-extraction benchmark on corpora where entities recur across documents must use document-grouped (or entity-disjoint) splits, or risk inflated scores of up to 37 points.
- Feature design: engineered feature sets beyond distance add noise, so lightweight baselines should start with proximity-only features.
- Graph message passing helps mainly when parses are noisy: tabular classifiers match or beat graph attention networks on clean French parses, while graph-based models help on noisy English.
- A parameter-free path-length rule can outperform trained models on small noisy data, suggesting simple topological heuristics are strong baselines.
Reading between the lines
- The 15–37 percentage-point leakage inflation measured here implies that prior relation-extraction results on historical corpora using pair-level splits may overstate accuracy; re-evaluating with document-grouped folds could shift reported rankings.
- The dominance of minimum character distance suggests the task as annotated rewards surface proximity over semantic understanding; a testable extension is to measure how much a coreference-aware distance feature improves recall on TRUE pairs at long distance.
- The deterministic path-length rule beating all trained models hints that the effective sample size is too small to learn better boundaries; one could test whether the ranking reverses with 10x or 100x more training data.
- The reuse of newspaper models on the literary surprise set underperformed; a testable extension is to add a small amount of in-domain literary data with domain adaptation, which the paper suggests but does not test.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes the DS@GT HIPE submission to the HIPE-2026 shared task on person–place relation extraction from historical newspapers. The system builds document-level graphs from Stanza dependency parse trees with three bridging mechanisms, extracts 15 proximity/POS features per entity pair, and classifies with scikit-learn ensembles or small GATs. On the official Test A, Run 1 achieves macro recall 0.5142, ranking 26th/46 on accuracy and 3rd on efficiency. The paper claims that minimum character distance dominates the classification signal and that document-grouped cross-validation is essential because pair-level splits inflate scores by 25–37 percentage points.
Significance. If the findings hold, the paper makes a useful contribution: it demonstrates that a lightweight, interpretable system without a pretrained LM at the classification stage can reach mid-table accuracy on a new shared task, with parameter counts two to three orders of magnitude below transformer baselines. The external Test A result is a real, reproducible data point. The paper is also transparent about the cost of parse errors and provides a detailed error analysis. However, the internal methodological claims (distance dominance, leakage removal) rest on a CV protocol that may still be leaky and on ablations with no significance testing; these need to be strengthened before the conclusions can be taken at face value.
major comments (3)
- [Abstract; §5.2; Table 7] The paper claims that document-grouped CV 'removes' the leakage caused by entity mentions recurring across documents. Grouping folds by document prevents pairs from the same document being split, but the same person or location entity can still appear in both training and validation documents, as the paper itself notes in §3. A classifier can memorize entity-pair associations (e.g., 'P at L = TRUE') from training documents and exploit them on validation documents even when no document is shared. Table 7 only compares naive vs. grouped CV; it does not measure residual entity overlap under grouped CV. This matters because all internal ablations (Tables 8–9), architecture comparisons (Tables 4, 13), and the path-length rule evaluation rely on this protocol. Please re-run with entity-grouped folds (or otherwise demonstrate that entity overlap is controlled), and temper the 'removes' claim to
- [§5.3, Tables 8–9, Abstract] The headline claim that 'minimum character distance alone captures most of the classification signal' is not well supported by the cumulative ablation. In Table 8, adding groups beyond A improves macro recall in 3 of 6 cells (EN at .325→.348; EN isAt .471→.542; FR at .402→.408) and degrades in the other 3. The EN isAt improvement of 7.1 pp is substantial relative to the .47 baseline. Leave-one-group-out deltas (Table 9) are mostly within the stated ~2 pp fold variance, and no significance tests are reported. Feature importance (Table 19) is not a substitute for classification-signal analysis. Please provide significance testing or a more nuanced statement, e.g., 'proximity features explain a large share of the variance but additional features help in some language/relation combinations.'
- [§5.5, Table 13] The path-length tiered rule is reported to achieve 0.549 on the newspaper held-out set, outperforming all trained models. However, the text states that its thresholds were 'empirically tuned' (presumably on the same held-out set). Evaluating a rule on the data used to tune its thresholds is a form of selection bias, and the comparison to trained models that were not optimized on that set is unfair. If the thresholds were tuned on the held-out set, the 0.549 figure is optimistic and the conclusion that a deterministic rule beats all learned models is unreliable. Please clarify the tuning procedure and, if necessary, use a nested or separate validation set.
minor comments (6)
- [§5.2, Figure 2] Figure 2 includes classifiers (SVC-rbf, LGBM, HistGBM) not listed in Table 7, and its caption says 'four classifier/relation combinations' while appearing to show six; please align the figure with the table or clarify.
- [Table 12] The efficiency profile table appears garbled in the manuscript (e.g., 'Run 1323 5 4'); please ensure the columns are readable.
- [§4.1, Table 2] The bridging ablation quantifies the effect of each mechanism on connectivity, but it would be helpful to report the corresponding downstream classification performance (e.g., macro recall with and without content-word linking) rather than only connectivity metrics.
- [§5.4] The paper notes that the official efficiency rank used a global parameter count of 2,087,375 that is inconsistent with per-run counts. Please state explicitly in the final version which numbers are official and which are corrected, to avoid confusion.
- [Table 19] The feature 'min_char_dist_via_pron' appears in Table 19 but is not defined in Table 3; clarify that it belongs to Group D.
- [§1] The phrase 'macro recall, also called balanced accuracy' may confuse readers; these are not always equivalent.
Circularity Check
No significant circularity: the central claims rest on an external held-out benchmark and on ablations that are not definitionally tied to their inputs.
full rationale
The main load-bearing result, Run 1's macro recall of 0.5142 on the official Test A set, is an external benchmark result and is not produced by fitting to that target. The feature-ablation finding that minimum character distance captures most of the signal is an empirical result from grouped cross-validation on training data, not a quantity that is equal to an input by construction. The leakage analysis in Section 5.2 is a methodological argument about CV protocols; grouping by document is not defined in terms of the claimed outcome, and even if the claim overstates the removal of cross-document entity leakage, that is a validity concern rather than circularity. The only quasi-circular element is the path-length rule in Section 5.5, whose thresholds are described as 'empirically tuned' before being reported as outperforming trained models on the held-out set; however, the paper does not state that the tuning was performed on that same held-out set, so the specific reduction required to score a fitted-input-called-prediction step is not established. No load-bearing self-citations or imported uniqueness arguments appear. Overall, the derivation chain is self-contained and benchmark-anchored.
Assumptions & free parameters
free parameters (5)
- content-word linking cosine threshold =
0.7
- GAT hyperparameters (hidden_dim, n_heads, n_layers, dropouts) =
hidden 32; heads 4; layers 2; dropouts 0.3/0.3/0.5
- sklearn/HistGBM hyperparameters =
RF 500/max_depth 10; ET 300; LR C=1.0; HistGBM 500 iters, lr 0.05
- path-length tiered rule thresholds =
not stated (empirically tuned)
- Group E movement/residence verb lexicon =
hand-curated, not listed
assumptions (5)
- domain assumption Stanza dependency parses of 19th-20th century OCR-degraded newspapers are accurate enough that graph paths and POS features carry usable signal.
- domain assumption FastText embeddings trained on modern text provide meaningful cross-sentence content-word similarity for historical language at threshold 0.7.
- domain assumption The HIPE-2026 annotations and official test set are a fair, consistent operationalization of person-place at/isAt relations.
- ad hoc to paper Grouping cross-validation folds by document removes the data leakage from entity mentions recurring across documents.
- ad hoc to paper The document-level graph with entity merging, content-word bridges, and Wikidata P131 edges is a faithful discourse representation for relation classification.
Cite this review
Pith. "Pith review of Lightweight Person-Place Relation Extraction from Historical Newspapers with Dependency Graphs and Proximity Features." pith.science (2026). https://pith.science/paper/4OY6OODO
@misc{pith2026260719718,
author = {Pith},
title = {Pith review of: Lightweight Person-Place Relation Extraction from Historical Newspapers with Dependency Graphs and Proximity Features},
year = {2026},
howpublished = {\url{https://pith.science/paper/4OY6OODO}},
note = {Machine review of arXiv:2607.19718}
}
read the original abstract
The HIPE-2026 shared task introduces person-place relation extraction from multilingual historical newspapers as a new evaluation track, classifying the at and isAt relations between pre-annotated person and location mentions in English, French, and German. Motivated by the cost of processing historical archives at scale, our team (DS@GT HIPE, team 2 in the official results) investigates how far a lightweight, interpretable system can go without any pretrained language model at the relation classification stage. Our approach builds a document-level graph from dependency parses, extracts proximity-based and part-of-speech features for each entity pair, and classifies them with small scikit-learn ensembles or compact Graph Attention Networks, keeping every submitted run under 847K parameters. On the official evaluation (Test A, the newspaper test set), our best run reached a macro recall of 0.5142, ranking 3rd on the Efficiency profile while placing mid-table on Accuracy among the 17 participating teams. Two findings stand out. First, minimum character distance alone captures most of the classification signal; adding further engineered features yields inconsistent gains and sometimes degrades performance, echoing prior evidence that argument distance dominates relation extraction. Second, document-grouped cross-validation is essential on this corpus: pair-level splits inflate scores by 25-37 percentage points because entity mentions recur across documents, a data-leakage effect that grouped cross-validation removes.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Ehrmann, M
M. Ehrmann, M. Romanello, A. Flückiger, S. Clematide, Extended Overview of CLEF HIPE 2020: Named Entity Processing on Historical Newspapers, in: L. Cappellato, C. Eickhoff, N. Ferro, A. Névéol (Eds.), Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum, volume 2696, CEUR-WS, Thessaloniki, Greece, 2020, p. 38. URL: https://infoscience....
2020
-
[2]
M. Ehrmann, M. Romanello, S. Najem-Meyer, A. Doucet, S. Clematide, Extended overview of HIPE-2022: Named Entity Recognition and Linking in Multilingual Historical Documents, in: G. Faggioli, N. Ferro, A. Hanbury, M. Potthast (Eds.), Proceedings of the Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, volume 3180, CEUR-WS, 2022. URL...
-
[3]
J. Opitz, M. Ehrmann, C. Raclé, A. Michail, M. Romanello, E. Boros, S. Gabay, S. Clematide, Extended Overview of HIPE-2026: Evaluating Accurate and Efficient Person–Place Relation Extraction from Multilingual Historical Texts, in: E. Sánchez Salido, A. Barrón-Cedeño, A. García Seco de Herrera, S. MacAvaney, J. M. Struß (Eds.), CLEF 2026 Working Notes, CEU...
-
[4]
Opitz, M
J. Opitz, M. Ehrmann, C. Raclé, A. Michail, M. Romanello, S. Clematide, Overview of HIPE-2026: Person–Place Relation Extraction from Multilingual Historical Texts, in: M. Hagen, M. Potthast, B. Stein, P. Schaer, E. Zangerle, S. MacAvaney, J. M. Struß, E. Sánchez Salido, A. Barrón-Cedeño, A. García Seco de Herrera (Eds.), Experimental IR Meets Multilingual...
2026
-
[5]
Opitz, S
J. Opitz, S. Burst, A closer look at classification evaluation metrics and a critical reflection of common evaluation practice, Transactions of the Association for Computational Linguistics (TACL) 12 (2024)
2024
-
[6]
Schwartz, J
R. Schwartz, J. Dodge, N. A. Smith, O. Etzioni, Green AI, Communications of the ACM 63 (2020) 54–63
2020
-
[7]
Strubell, A
E. Strubell, A. Ganesh, A. McCallum, Energy and policy considerations for deep learning in NLP, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019, pp. 3645–3650
2019
-
[8]
Bojanowski, E
P. Bojanowski, E. Grave, A. Joulin, T. Mikolov, Enriching word vectors with subword information, Transactions of the Association for Computational Linguistics 5 (2017) 135–146
2017
Show all 35 references
-
[9]
Pedregosa, G
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al., Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830
2011
-
[10]
M. Fey, J. E. Lenssen, Fast graph representation learning with PyTorch Geometric, in: ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019
2019
-
[11]
Ehrmann, A
M. Ehrmann, A. Hamdi, E. Linhares Pontes, M. Romanello, A. Doucet, Named entity recognition and classification in historical documents: A survey, ACM Computing Surveys 56 (2023) 27
2023
-
[12]
Hamdi, E
A. Hamdi, E. Linhares Pontes, N. Sidère, M. Coustaty, A. Doucet, In-depth analysis of the impact of OCR errors on named entity recognition and linking, Natural Language Engineering 29 (2023) 1–24
2023
-
[13]
Bunescu, R
R. Bunescu, R. Mooney, A shortest path dependency kernel for relation extraction, in: Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), 2005, pp. 724–731
2005
-
[14]
Zhang, P
Y. Zhang, P. Qi, C. D. Manning, Graph convolution over pruned dependency trees improves relation extraction, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018, pp. 2205–2215
2018
-
[15]
Z. Guo, Y. Zhang, W. Lu, Attention guided graph convolutional networks for relation extraction, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019, pp. 241–251
2019
-
[16]
G. Nan, Z. Guo, I. Sekulić, W. Lu, Reasoning with latent structure refinement for document- level relation extraction, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020, pp. 1546–1557
2020
-
[17]
N. Peng, H. Poon, C. Quirk, K. Toutanova, W.-t. Yih, Cross-sentence n-ary relation extraction with graph LSTMs, Transactions of the Association for Computational Linguistics 5 (2017) 101–115
2017
-
[18]
Y. Yao, D. Ye, P. Li, X. Han, Y. Lin, Z. Liu, Z. Liu, L. Huang, J. Zhou, M. Sun, DocRED: A large-scale document-level relation extraction dataset, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019, pp. 764–777
2019
-
[19]
Jiang, C
J. Jiang, C. Zhai, A systematic exploration of the feature space for relation extraction, in: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), 2007, pp. 113–120
2007
-
[20]
G. Zhou, J. Su, J. Zhang, M. Zhang, Exploring various knowledge in relation extraction, in: Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL), 2005, pp. 427–434
2005
-
[21]
D. Zeng, K. Liu, S. Lai, G. Zhou, J. Zhao, Relation classification via convolutional deep neural network, in: Proceedings of the 25th International Conference on Computational Linguistics (COLING), 2014, pp. 2335–2344
2014
-
[22]
Zhang, V
Y. Zhang, V. Zhong, D. Chen, G. Angeli, C. D. Manning, Position-aware attention and supervised data improve slot filling, in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017, pp. 35–45
2017
-
[23]
C. Alt, A. Gabryszak, L. Hennig, Probing linguistic features of sentence-level representations in neural relation extraction, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020, pp. 1547–1559
2020
-
[24]
de-Dios-Flores, J
I. de-Dios-Flores, J. R. Pichel, F. Silva, Exploring the effectiveness of linguistic knowledge for biographical relation extraction, Natural Language Engineering (2015)
2015
-
[25]
Giuliano, A
C. Giuliano, A. Lavelli, L. Romano, Exploiting shallow linguistic information for relation extraction from biomedical literature, in: Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2006, pp. 401–408
2006
-
[26]
Minard, A.-L
A.-L. Minard, A.-L. Ligozat, B. Grau, Multi-class SVM for relation extraction from clinical reports, in: Proceedings of Recent Advances in Natural Language Processing (RANLP), 2011, pp. 604–609
2011
-
[27]
Renslow, G
T. Renslow, G. Neumann, LightRel SemEval-2018 task 7: Lightweight and fast relation classification, in: Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), 2018, pp. 778–782
2018
-
[28]
Plum, Biographical Information Extraction: A Language-Agnostic Methodology for Datasets and Models, Ph.D
A. Plum, Biographical Information Extraction: A Language-Agnostic Methodology for Datasets and Models, Ph.D. thesis, University of Wolverhampton, 2022
2022
-
[29]
P. Qi, Y. Zhang, Y. Zhang, J. Bolton, C. D. Manning, Stanza: A Python natural language processing toolkit for many human languages, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020, pp. 101–108
2020
-
[30]
Veličković, G
P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, Y. Bengio, Graph attention networks, in: International Conference on Learning Representations (ICLR), 2018
2018
-
[31]
Brody, U
S. Brody, U. Alon, E. Yahav, How attentive are graph attention networks?, in: International Conference on Learning Representations (ICLR), 2022
2022
-
[32]
Mandya, D
A. Mandya, D. Bollegala, F. Coenen, Graph convolution over multiple dependency sub-graphs for relation extraction, in: Proceedings of the 28th International Conference on Computational Linguistics (COLING), 2020, pp. 6424–6435
2020
-
[33]
Elangovan, J
A. Elangovan, J. He, K. Verspoor, Memorization vs. generalization: Quantifying data leakage in NLP performance evaluation, arXiv preprint arXiv:2102.01818 (2021)
2021 arXiv
-
[34]
D. R. Roberts, V. Bahn, S. Ciuti, M. S. Boyce, J. Elith, G. Guillera-Arroita, S. Hauenstein, J. J. Lahoz-Monfort, B. Schröder, W. Thuiller, D. I. Warton, B. A. Wintle, F. Hartig, C. F. Dormann, Cross-validation strategies for data with temporal, spatial, hierarchical, or phylo...
2017
-
[35]
Søgaard, S
A. Søgaard, S. Ebert, J. Bastings, K. Filippova, We need to talk about random splits, in: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2021, pp. 1823–1832
2021
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.