{"id":"442d80db-b069-43a9-b9e0-93cca029eda0","arxiv_id":"2607.19718","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A feature-based, no-pretrained-LM classifier for person–place relations in historical newspapers reaches 0.5142 macro recall on HIPE-2026, with minimum character distance dominating the signal and document-grouped CV removing much of the leakage inflation.","lead":"A HIPE-2026 shared-task entry shows that a lightweight system with no pretrained model at the classification stage reaches mid-table accuracy (macro recall 0.5142) on person–place relation extraction from historical newspapers, ranking 3rd on efficiency. It also reports that a single character-distance feature carries most of the signal and that pair-level cross-validation inflates scores by 15–37 points.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Document-grouped CV does not remove cross-document entity leakage; the paper's claim that it 'removes' leakage and in turn the feature-ablation findings may be contaminated.","rationale":"The reader's weakest assumption is exactly the load-bearing concern I identify: document-grouped CV does not prevent cross-document entity recurrence. This is a real logical gap, not a matter of consensus. The paper's own text (§3, §5.2) demonstrates that the authors know entity mentions recur across documents, yet they choose grouping by document and assert that this removes leakage. The assertion does not follow. The concern is load-bearing because the paper's second central finding — that minimum character distance dominates the signal — is derived from an ablation run under this CV protocol. If entity-level leakage persists, the model may rely on entity identity shortcuts, making proximity features appear more important than they are. The path-length rule and official test result provide some independent support, but the internal feature analysis is a key contribution and should be corrected. The official Test A score and efficiency rank are external validations that remain credible, so the verdict should stay CONDITIONAL: the paper needs to either fix the CV protocol or qualify its methodological claims. My proposed test directly quantifies residual leakage and, more importantly, assesses whether the findings change under an entity-disjoint split. This is feasible with the released task data and would settle the matter objectively.","tokens_in":14784,"tokens_out":5788,"duration_ms":65508,"concrete_test":"Reproduce the document-grouped 5-fold splits from the released training data and compute, for each validation fold, the fraction of entity pairs where the person entity, the location entity, or the exact person–location pair also appears in the training fold. If the fraction is non-trivial (>5%), re-run the feature ablation (Table 8) and classifier comparisons using group-by-person (or group-by-connected-entity-component) CV. If macro recall drops substantially or the 'Group A dominates' ranking changes, the paper's leakage-free claim is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central methodological claim is that pair-level splits inflate scores by 25–37 pp because entity mentions recur across documents, and that document-grouped cross-validation 'removes' this leakage (Abstract; §5.2). This is logically incomplete. Grouping by document only ensures that all pairs from the same document stay in the same fold. Since the paper itself notes in §3 that 'entity mentions recur across documents within the same newspaper', the same person or location entity can still appear in a training document and in a validation document. A model can then memorize entity-level associations (e.g., a particular person–location pair is TRUE) from training and exploit them on validation, even though no single document is split across folds. Table 7 quantifies the naive-vs-grouped gap, but it does not measure the residual entity overlap under grouped CV. All internal analyses — feature ablation (Table 8), GAT-vs-sklearn comparisons (Table 13), and architecture selection (Table 4) — are performed under this grouped protocol. If entity-level leakage persists, these results are inflated and the reported model rankings and feature-importance findings may reflect memorization of entity identities rather than the claimed proximity signal. The official Test A result (0.5142) is independent and supports the viability claim, but the paper's broader conclusions about distance dominance and the necessity of a particular CV scheme rest on the internal, potentially still-leaky evaluation. The paper should either demonstrate that entity overlap is negligible under document grouping or adopt a stricter grouping criterion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes the DS@GT HIPE submission to the HIPE-2026 shared task on person–place relation extraction from historical newspapers. The system builds document-level graphs from Stanza dependency parse trees with three bridging mechanisms, extracts 15 proximity/POS features per entity pair, and classifies with scikit-learn ensembles or small GATs. On the official Test A, Run 1 achieves macro recall 0.5142, ranking 26th/46 on accuracy and 3rd on efficiency. The paper claims that minimum character distance dominates the classification signal and that document-grouped cross-validation is essential because pair-level splits inflate scores by 25–37 percentage points.","tokens_in":15050,"tokens_out":7200,"duration_ms":68176,"significance":"If the findings hold, the paper makes a useful contribution: it demonstrates that a lightweight, interpretable system without a pretrained LM at the classification stage can reach mid-table accuracy on a new shared task, with parameter counts two to three orders of magnitude below transformer baselines. The external Test A result is a real, reproducible data point. The paper is also transparent about the cost of parse errors and provides a detailed error analysis. However, the internal methodological claims (distance dominance, leakage removal) rest on a CV protocol that may still be leaky and on ablations with no significance testing; these need to be strengthened before the conclusions can be taken at face value.","major_comments":[{"comment":"The paper claims that document-grouped CV 'removes' the leakage caused by entity mentions recurring across documents. Grouping folds by document prevents pairs from the same document being split, but the same person or location entity can still appear in both training and validation documents, as the paper itself notes in §3. A classifier can memorize entity-pair associations (e.g., 'P at L = TRUE') from training documents and exploit them on validation documents even when no document is shared. Table 7 only compares naive vs. grouped CV; it does not measure residual entity overlap under grouped CV. This matters because all internal ablations (Tables 8–9), architecture comparisons (Tables 4, 13), and the path-length rule evaluation rely on this protocol. Please re-run with entity-grouped folds (or otherwise demonstrate that entity overlap is controlled), and temper the 'removes' claim to","section":"Abstract; §5.2; Table 7"},{"comment":"The headline claim that 'minimum character distance alone captures most of the classification signal' is not well supported by the cumulative ablation. In Table 8, adding groups beyond A improves macro recall in 3 of 6 cells (EN at .325→.348; EN isAt .471→.542; FR at .402→.408) and degrades in the other 3. The EN isAt improvement of 7.1 pp is substantial relative to the .47 baseline. Leave-one-group-out deltas (Table 9) are mostly within the stated ~2 pp fold variance, and no significance tests are reported. Feature importance (Table 19) is not a substitute for classification-signal analysis. Please provide significance testing or a more nuanced statement, e.g., 'proximity features explain a large share of the variance but additional features help in some language/relation combinations.'","section":"§5.3, Tables 8–9, Abstract"},{"comment":"The path-length tiered rule is reported to achieve 0.549 on the newspaper held-out set, outperforming all trained models. However, the text states that its thresholds were 'empirically tuned' (presumably on the same held-out set). Evaluating a rule on the data used to tune its thresholds is a form of selection bias, and the comparison to trained models that were not optimized on that set is unfair. If the thresholds were tuned on the held-out set, the 0.549 figure is optimistic and the conclusion that a deterministic rule beats all learned models is unreliable. Please clarify the tuning procedure and, if necessary, use a nested or separate validation set.","section":"§5.5, Table 13"}],"minor_comments":[{"comment":"Figure 2 includes classifiers (SVC-rbf, LGBM, HistGBM) not listed in Table 7, and its caption says 'four classifier/relation combinations' while appearing to show six; please align the figure with the table or clarify.","section":"§5.2, Figure 2"},{"comment":"The efficiency profile table appears garbled in the manuscript (e.g., 'Run 1323 5 4'); please ensure the columns are readable.","section":"Table 12"},{"comment":"The bridging ablation quantifies the effect of each mechanism on connectivity, but it would be helpful to report the corresponding downstream classification performance (e.g., macro recall with and without content-word linking) rather than only connectivity metrics.","section":"§4.1, Table 2"},{"comment":"The paper notes that the official efficiency rank used a global parameter count of 2,087,375 that is inconsistent with per-run counts. Please state explicitly in the final version which numbers are official and which are corrected, to avoid confusion.","section":"§5.4"},{"comment":"The feature 'min_char_dist_via_pron' appears in Table 19 but is not defined in Table 3; clarify that it belongs to Group D.","section":"Table 19"},{"comment":"The phrase 'macro recall, also called balanced accuracy' may confuse readers; these are not always equivalent.","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"The paper's external Test A result (0.5142) is credible and supports the lightweight-viability claim. The main weakness is the internal CV protocol: document-grouping does not eliminate entity-level leakage, so the internal ablations and the 'distance dominates' finding may be contaminated. This is fixable but requires re-analysis. The path-length rule tuning issue also needs clarification. I would not reject, but the authors should address these points before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent working-notes paper from a shared task, and the official Test A result is the real contribution. A sub-847K-parameter system with no pretrained LM at classification reaches 0.5142 macro recall and ranks 3rd on efficiency. That is externally evaluated and it supports the viability claim. The paper also gives a useful, honest quantification of how much pair-level CV inflates scores on this corpus (15–37 pp). Credit where due: it frames distance-dominance as echoing prior work rather than a new discovery, and the error analysis is detailed and credible.\n\nThe stress-test concern is right and should be fixed. Grouping CV folds by document does not remove cross-document entity leakage. The paper itself says entity mentions recur across documents, but it never measures how much entity overlap remains between train and validation documents after grouping. The 25–37 pp gap is between pair-level and document-grouped splits, not between leaky and leakage-free. So the claim that grouped CV 'removes' the leakage is unsupported, and the internal ablations and architecture choices made under that protocol may still be optimistic. The official Test A result is independent, so the core viability finding survives, but the stronger conclusions about distance dominance and an 'essential' CV protocol rest on internal numbers that could be contaminated.\n\nTwo smaller issues. The path-length rule (Tier 2, 0.549) beats trained models on the newspaper held-out set, but its thresholds were empirically tuned on exactly that set, so that comparison is not a fair head-to-head. And the efficiency ranks were computed from the official global parameter total (2.09M for all runs), not the per-run counts in Table 6; the author discloses this, but it means the reader cannot reproduce the rank from the paper's tables. The ablation also has no significance tests, and many deltas are within the stated fold variance — worth saying plainly rather than marketing small differences.\n\nWho this is for: anyone working on lightweight RE, historical document NLP, or CV-leakage pitfalls in shared-task settings. It deserves a serious referee — the external result and the honest writeup earn that — but the revision needs to address the entity-overlap issue, either with a stricter grouping criterion (by entity or entity-pair) or an explicit measurement showing residual overlap is negligible, and it needs to qualify the path-length comparison.\n\nRecommendation: send to peer review with a request for major revision on those points. The fixes are tractable and the paper would be stronger for them.","headline":"Honest shared-task notes with a solid external result, but the headline leakage claim overstates what document-grouped CV achieves and the internal comparisons have a tuning leak.","tokens_in":15639,"tokens_out":2794,"would_cite":true,"duration_ms":30235,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A lightweight system with no pretrained language model at classification can compete on person–place relation extraction from historical newspapers, with character distance carrying most of the signal.","keywords":["relation extraction","historical newspapers","dependency parsing","proximity features","cross-validation leakage","graph attention networks","efficiency","person-place relations"],"falsifier":"Compute grouped-CV estimates under entity-disjoint splits (ensuring no person or location mention appears in both training and validation folds) and compare them to the paper's document-grouped CV numbers; if macro recall drops by several points, the document-grouped protocol still leaks. Alternatively, train a classifier that predicts document identity from entity-pair features; if it succeeds at above-chance accuracy under document-grouped splits, leakage remains.","tokens_in":14526,"feed_emoji":"📰","tokens_out":4936,"duration_ms":50719,"temperature":0.7,"pith_summary":"This paper claims that a lightweight relation-extraction system—no pretrained language model at classification, just dependency-parse graphs, fifteen proximity and part-of-speech features, and small tabular or graph classifiers—is viable for person–place relations in multilingual historical newspapers. On the official newspaper test set the best run reached 0.5142 macro recall, placing mid-table on accuracy yet 3rd on the efficiency profile among 17 teams. The paper's central analytic findings are that minimum character distance between entities carries most of the classification signal, and that document-grouped cross-validation is essential because pair-level splits inflate scores by 15–37 percentage points due to recurring entity mentions. A sympathetic reader would care because historical-archive processing at scale demands cheap, transparent models, and the paper shows the proximity signal is strong enough to make such models competitive.","feed_headline":"Tiny no-LM system ranks 3rd in efficiency on historical newspaper RE","feed_subtitle":"Dependency graphs and character distance hit 0.5142 macro recall; all runs stay under 847K parameters.","key_machinery":"The load-bearing object is a document-level graph assembled from dependency parses of each sentence, with three bridges—merging repeated mentions of the same entity, linking content words across sentences by embedding similarity, and adding geographic-containment edges—that connect otherwise isolated sentence trees. From this graph the system computes fifteen proximity and part-of-speech features per entity pair, then classifies with either small tree ensembles or compact graph attention networks (all under 847K parameters). The argument runs through two mechanisms: the minimum character-distance feature (Group A) is shown by ablation to dominate, and document-grouped cross-validation is sho","core_discovery":"The discovery is that on this task, distance is the signal: a single scalar—the minimum character distance between any mention of the person and any mention of the location—accounts for most of what a classifier can learn, and adding fifteen engineered features often hurts (e.g., German at drops from 0.408 to 0.385). A deterministic rule that thresholds graph path length outperformed all trained models in internal evaluation (0.549 vs 0.520 for the best tabular pass). The same proximity dominance explains cross-validated leakage: because entity mentions recur across documents, naive pair-level folds let models memorize document-specific patterns, inflating scores by 15–37 percentage points;","pith_inferences":["The 15–37 percentage-point leakage inflation measured here implies that prior relation-extraction results on historical corpora using pair-level splits may overstate accuracy; re-evaluating with document-grouped folds could shift reported rankings.","The dominance of minimum character distance suggests the task as annotated rewards surface proximity over semantic understanding; a testable extension is to measure how much a coreference-aware distance feature improves recall on TRUE pairs at long distance.","The deterministic path-length rule beating all trained models hints that the effective sample size is too small to learn better boundaries; one could test whether the ranking reverses with 10x or 100x more training data.","The reuse of newspaper models on the literary surprise set underperformed; a testable extension is to add a small amount of in-domain literary data with domain adaptation, which the paper suggests but does not test."],"forward_implications":["Historical archives with millions of entity pairs can be classified on CPU-only hardware with sub-second training, making large-scale annotation practical.","Any future relation-extraction benchmark on corpora where entities recur across documents must use document-grouped (or entity-disjoint) splits, or risk inflated scores of up to 37 points.","Feature design: engineered feature sets beyond distance add noise, so lightweight baselines should start with proximity-only features.","Graph message passing helps mainly when parses are noisy: tabular classifiers match or beat graph attention networks on clean French parses, while graph-based models help on noisy English.","A parameter-free path-length rule can outperform trained models on small noisy data, suggesting simple topological heuristics are strong baselines."],"fun_headline_variants":["Distance beats features in lightweight historical newspaper RE","No-LM RE: one scalar dominates, group CV essential","Tiny model, big insight: distance is key in old news RE","847K params, 3rd in efficiency: graph distance wins","Grouped CV fixes 25-37 pt leakage in person-place RE"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The analysis assumes that grouping cross-validation folds by document removes the data leakage caused by entity mentions recurring across documents; but the same person or location can still appear in both a training document and a validation document, so mention-level leakage may persist and internal estimates may remain inflated.","fun_headline_variants_meta":{"raw":{"variants":["Distance beats features in lightweight historical newspaper RE","No-LM RE: one scalar dominates, group CV essential","Tiny model, big insight: distance is key in old news RE","847K params, 3rd in efficiency: graph distance wins","Grouped CV fixes 25-37 pt leakage in person-place RE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000618,"raw_usage":{"total_tokens":2730,"prompt_tokens":794,"completion_tokens":1936,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":1849}},"tokens_in":538,"tokens_out":1936,"duration_ms":12442,"temperature":1.0,"reasoning_tokens":1849,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T11:53:27.489959+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute grouped-CV estimates under entity-disjoint splits (ensuring no person or location mention appears in both training and validation folds) and compare them to the paper's document-grouped CV numbers; if macro recall drops by several points, the document-grouped protocol still leaks. Alternatively, train a classifier that predicts document identity from entity-pair features; if it succeeds at above-chance accuracy under document-grouped splits, leakage remains.","supporting_citations":[],"review_version":1}