{"id":"7182f30f-30b6-4df1-9d02-6e483900f96a","arxiv_id":"2505.14699","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GraphSAGE with a dual-branch multimodal design and a k-closest-neighbor page graph performs best among tested GNNs for layout classification in Spanish official documents, but remains below a Random Forest baseline overall.","lead":"This paper benchmarks four graph neural network models for classifying text blocks in official Spanish gazettes, comparing graph structures and text-plus-image features. It finds that GraphSAGE with a local-neighbor graph and separate text and image branches is the best GNN setup, though a simple random forest baseline still scores higher overall.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'outperforms baseline in some sources' claim is not checkable: Table 2 omits baseline values, and the RF baseline was trained on a different split, so the headline result may not survive a matched comparison.","rationale":"The reader's weakest assumption identifies the right area: the comparison against the Random Forest baseline is the load-bearing part of the paper's claim. I would sharpen it: the decisive problem is not only that the GNNs and baseline use different training data, but that the per-source baseline numbers needed to support 'some sources' are absent. Table 1 shows the baseline winning on average, and Table 2, the only source-level table, omits the baseline entirely. The conclusion phrase 'one third of them' has no corresponding table or supplementary material in the manuscript. This matters because the abstract's claimed result is specifically about beating the baseline in some sources, not about ordering among GNNs. The internal GNN comparisons are informative and the k-closest versus complete graph finding is plausible, but the headline comparison is currently unverifiable from the paper. With the baseline re-run under the same split and per-source results reported, the claim could be substantiated, so the conditional verdict stands.","tokens_in":10018,"tokens_out":6449,"duration_ms":54808,"concrete_test":"Re-run the Random Forest baseline of [26] under the exact protocol used for the GNNs: for each of the 24 sources, train on the same 80% of the validation set and test on the same 20%, with the same 5-fold cross-validation, and report per-source overall and per-class accuracy alongside the GraphSAGE k-closest dual-branch model. Then count the sources in which the GNN has higher overall accuracy, and test whether differences exceed the reported standard deviations. If the count is not about one third of sources, or if the gaps are within noise, the abstract and conclusions should be revised to remove the claim of outperforming the baseline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that GraphSAGE on the k-closest graph in a dual-branch configuration outperforms the Random Forest baseline in some sources. This claim is not supported by the evidence as presented. Table 1 reports only aggregate accuracy over all sources and shows the baseline (97.48% overall) above every GNN configuration (best dual-branch 95.98%), so the 'some sources' improvement must come from a per-source comparison. No such comparison is shown: Table 2 gives GNN accuracy for five selected sources but contains no baseline column, and the conclusion that overall accuracy improves in 'one third' of sources is not backed by a per-source table. Additionally, the comparison is protocol-mismatched: GNNs are trained and tested exclusively on the PAL validation set (80/20 document split, Section 4.2), while the RF baseline numbers come from models trained on the original training set (Section 4.1). If the validation set is cleaner or differently distributed, the GNN advantage could be due to training data choice rather than to graph-based modeling. The paper itself acknowledges possible optimism from selecting best configurations, but the baseline comparison is the more serious issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks four GNN architectures (GCN, GAT, GraphSAGE, TAGCN) for fine-grained classification of text-block layout categories (Identifier, Title, Summary, Body) in digital-born official documents from the PAL dataset. It introduces two page-level graph constructions (k-closest-neighbor and complete), initializes node features with pretrained RoBERTa (text) and ResNet (vision) embeddings, and evaluates three frameworks: text-only, vision-only, concatenated single-branch, and dual-branch multimodal. Experiments are per-source, with 80/20 document-level splits and 5-fold cross-validation on the PAL validation set. The authors report that a dual-branch GraphSAGE on the k-closest graph achieves the best GNN accuracy and claim it outperforms the Random Forest baseline in some sources and in about one third of sources overall.","tokens_in":10378,"tokens_out":2228,"duration_ms":22138,"significance":"If the central comparison were well controlled, the paper would provide a useful benchmark for graph-based multimodal layout analysis in a domain (public affairs documents) with practical value. The use of pretrained text and vision embeddings avoids manual feature engineering, the k-closest graph is a sensible inductive bias for local layout relations, and the comparison across four GNN backbones and three fusion schemes is informative. The paper also ships a public dataset link and states reproducible experimental choices. However, the significance is materially reduced by the uncontrolled baseline comparison: the GNNs are trained on the validation set while the Random Forest baseline was trained on the original training set, and the per-source comparisons that underlie the headline claims are not shown in any table. As presented, the strongest defensible claim is limited to relative ranking among the tested GNN configurations, not superiority over the baseline.","major_comments":[{"comment":"The comparison against the Random Forest baseline is not protocol-matched. The GNN models are trained and tested exclusively on the PAL validation set (80/20 document split per source, Section 4.2), while the baseline numbers in Table 1 come from models trained on the original training set of PAL, as described in Section 4.1. If the validation set differs in label quality or distribution, any improvement over the baseline could be due to the training-data choice rather than to graph-based modeling. Please retrain the Random Forest under the same validation-set protocol, or clearly restrict all claims to a comparison among GNN variants and add an explicit statement that no controlled comparison with the baseline is claimed.","section":"Section 4.2 and Table 1"},{"comment":"The headline claim that the proposed model 'outperforms the baseline in some sources' is not verifiable from the reported results. Table 2 reports accuracy for the five selected sources but contains no baseline column, and Table 1 reports only source-averaged accuracy, where the Random Forest baseline (97.48% overall) exceeds every GNN configuration (best dual-branch 95.98%). Please provide a per-source table that includes the baseline accuracy for each source and each class, so that the claimed per-source improvements can be checked.","section":"Table 2 and Section 4.3"},{"comment":"The conclusion states that the dual-branch GraphSAGE configuration improves overall accuracy 'in one third of them' (Section 5), but the results section states that, among the five selected sources, the model achieves higher overall accuracy 'in one of the cases' (Section 4.3), which is 20%, not one third. These statements are inconsistent, and neither is backed by a per-source comparison table. Please harmonize the quantification and support it with the actual per-source baseline comparison.","section":"Section 5 vs. Section 4.3"},{"comment":"The claim that the dual-branch framework outperforms the other frameworks in four of the five top configurations is based on selecting the best configuration per framework after seeing the results. The paper acknowledges this selection bias in general terms, but the central comparison to the baseline is still affected by it, since the baseline is a single fixed model. Please report results for all configurations or apply a multiple-comparison correction, and provide error bars or confidence intervals for the aggregated accuracies, given the large variance implied by the per-source variation.","section":"Table 1 and Section 4.3"}],"minor_comments":[{"comment":"The abstract and introduction mention 'more than 20 sources' and '20 sources', while Section 4.1 states the dataset contains 24 official gazettes; please make the source count consistent.","section":"Section 4.1"},{"comment":"The table uses 'Páis Vasco' but the correct spelling is 'País Vasco'; also, the formatting of the mean and standard deviation values (e.g., '99, 520,76') is inconsistent and should be cleaned to avoid confusion.","section":"Table 2"},{"comment":"The caption of Figure 2 says 'four different models are trained independently' while the figure illustrates three frameworks; please clarify the caption to avoid ambiguity.","section":"Section 3.2"},{"comment":"The choice of k=4 in the k-closest graph is stated without any sensitivity analysis; a brief comment on how the results vary with k would strengthen the benchmark.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The central novelty relative to the authors' previous work on the PAL dataset appears to be the GNN-based modeling and the multimodal fusion, but the claimed improvement over the Random Forest baseline is not substantiated by a matched comparison. Given that the baseline and dataset originate from the same group, the protocol mismatch should be resolved before publication; otherwise the paper should be reframed as a comparison among GNN variants only."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper benchmarks four GNN architectures, two graph-construction schemes, and three fusion frameworks for fine-grained layout classification of text blocks on the PAL dataset of Spanish official gazettes. The internal comparison is the useful part: GraphSAGE with a k-closest-neighbor graph and a dual-branch text/vision setup consistently comes out on top among the GNN variants, and the authors are careful enough to acknowledge that cherry-picking the best configuration per framework overstates performance.\n\nThe soft spot is the headline comparison to the Random Forest baseline. The baseline numbers in Table 1 come from Peña et al.'s models trained on the original PAL training set, while all GNNs are trained and tested exclusively on the validation set (80/20 split). That is not a controlled comparison, and it is the kind of difference that can explain a few accuracy points. On top of that, Table 1 shows the baseline (97.48% overall) above every GNN configuration (best 95.98%), so the claimed per-source advantages in \"some sources\" have to come from a per-source comparison that is never shown: Table 2 gives GNN accuracy for five sources but omits the baseline column entirely. The conclusion that overall accuracy improves in one third of sources is therefore not checkable from the paper. This is the central claim in the abstract, and the evidence does not currently support it.\n\nSmaller issues: Table 1 has no error bars, hyperparameters like k (fixed at 4) and per-source batch sizes are described but not fully specified, and code is not provided, which limits reproducibility of the benchmark.\n\nNone of this sinks the internal GNN comparison. The paper is a reasonable benchmark for a domain (digital-born official Spanish documents) that hasn't been explored with GNNs before, and the graph construction choices are clearly motivated. It just needs major revision: rerun the baseline under the same validation-set protocol (or at least report its performance on the same test split), add per-source tables with baseline columns and error bars, and recalibrate the abstract and conclusions to what the data actually show.\n\nI would send this to peer review rather than desk-reject: the empirical setup is solid enough in its internal comparisons, and the dataset work is useful. But I would push for the baseline comparison to be fixed or the claims walked back.","headline":"A useful internal benchmark of GNN variants for Spanish official-document layout analysis, but the headline comparison to the Random Forest baseline is not supported by the evidence as presented.","tokens_in":10784,"tokens_out":2198,"would_cite":false,"duration_ms":19340,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GraphSAGE with local links leads GNN benchmark for official document layout","keywords":["Graph Neural Networks","Document Layout Analysis","digital-born PDF","multimodal fusion","GraphSAGE","k-closest neighbor graph","public affairs documents","text block classification"],"falsifier":"Retrain the Random Forest baseline on the same 80/20 validation split used for the GNNs and compare per-source overall accuracy; if the baseline then matches or exceeds GraphSAGE on these sources, the claimed gains over the baseline would not hold under a matched protocol.","tokens_in":9833,"feed_emoji":"📄","tokens_out":3050,"duration_ms":26996,"temperature":0.7,"pith_summary":"The paper asks whether graph neural networks can classify the semantic role of text blocks in born-digital official documents, such as Spanish gazettes, where layout objects are exactly placed and need no OCR. It claims that among the tested GNN configurations, GraphSAGE operating on a k-closest-neighbor graph with text and visual features processed in separate branches achieves the highest per-class and overall accuracy, and that it beats a Random Forest baseline on some sources and on overall accuracy in about one third of them. If true, this would show that local layout relationships and multimodal fusion are exploitable by GNNs for fine-grained layout analysis, without manual feature engineering.","feed_headline":"GraphSAGE leads GNN benchmark for official document layout","feed_subtitle":"Local neighbor graphs plus separate text and visual branches give top GNN accuracy, beating the baseline on some gazette sources.","key_machinery":"The central object is the page-level document graph, where each layout object is a node and edges are built either by connecting each node to its k closest neighbors by centroid distance (k=4 in all experiments) or by connecting all nodes. Node features come from two pretrained models: a Spanish RoBERTa-base for text and a ResNet-18 for cropped page regions. The GNN architectures—GCN, GAT, GraphSAGE, and TAGCN—are then run in three frameworks (text-only, visual-only, concatenated, and dual-branch), with the dual-branch framework using separate text and visual GNNs whose outputs are fused before classification.","core_discovery":"The paper's central claim is that a dual-branch GraphSAGE model on a k-closest-neighbor document graph performs best among the GNN settings tested, and that this configuration improves per-class accuracy over the Random Forest baseline for some classes in most sources and overall accuracy in one third of the sources. The discovery is framed as the first GNN-based layout analysis applied to Spanish official digital-born documents, with the finding that local neighborhood structure and separate text and visual processing branches are the key ingredients for the gains.","pith_inferences":["Because Table 1 shows the Random Forest baseline averages 97.48% overall while the best GNN averages 95.98%, the claim of beating the baseline rests entirely on source-level and per-class comparisons; a matched Random Forest trained on the same validation split could well erase or reverse the reported gains.","The choice k=4 is not tested for sensitivity; probing k values from 2 to 8 on a few sources would indicate whether the result is robust or tied to that specific neighborhood size.","The paper's decision to train on the validation set while citing the original training set as unreliable implicitly questions the baseline's training source; a fairer comparison would retrain the baseline under the same protocol.","The method could plausibly extend to classify non-text objects (images, tables, links) rather than only text blocks, since those nodes are retained in the graph and their embeddings are updated during training."],"forward_implications":["If local layout relationships are as informative as the paper suggests, then graph-based methods can be used instead of manual layout heuristics for born-digital documents, with node features from pretrained transformers.","The dual-branch fusion pattern—separate text and visual GNNs followed by concatenation—could transfer to other document types and languages where the same four semantic classes apply.","The k-closest graph with small k=4 keeps the graph diameter small while focusing on local context, implying that only nearby layout objects matter most for classifying a text block.","The benchmark provides a fixed evaluation protocol on a multi-source public affairs dataset, which could serve as a comparison point for future layout analysis methods on digital-born official documents.","The finding that GraphSAGE outperforms GCN, GAT, and TAGCN in these settings suggests that inductive neighborhood aggregation is well suited to layout graphs with variable structure across pages and sources."],"supporting_citations":[{"why":"Supplies the PAL dataset, the Random Forest baseline, and the source-wise train/test protocol that the paper adopts.","marker":"[26]"},{"why":"Introduces GraphSAGE, the architecture that appears in all top-performing configurations.","marker":"[12]"},{"why":"Defines GCN, one of the four GNN architectures benchmarked as a baseline.","marker":"[17]"},{"why":"Defines GAT, one of the four GNN architectures benchmarked as a baseline.","marker":"[34]"},{"why":"Defines TAGCN, one of the four GNN architectures benchmarked as a baseline.","marker":"[6]"},{"why":"Provides the Spanish RoBERTa-base model used to generate text node embeddings.","marker":"[11]"},{"why":"Provides ResNet-18, the backbone used to generate visual node embeddings from cropped page regions.","marker":"[13]"},{"why":"Establishes the prior approach to text embedding for public affairs documents that this work builds on.","marker":"[27]"}],"fun_headline_variants":["GraphSAGE on local graphs wins document layout benchmark","Dual-branch GraphSAGE tops GNN layout test for official docs","Local GNN outperforms baseline on public affairs layouts","GraphSAGE with k-nearest neighbors bests GNN baselines","Best layout GNN: GraphSAGE with multimodal branches"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The protocol assumes the PAL validation set is reliable and sufficient for training the GNNs, while the original training set used for the compared baseline is too unreliable to use.","fun_headline_variants_meta":{"raw":{"variants":["GraphSAGE on local graphs wins document layout benchmark","Dual-branch GraphSAGE tops GNN layout test for official docs","Local GNN outperforms baseline on public affairs layouts","GraphSAGE with k-nearest neighbors bests GNN baselines","Best layout GNN: GraphSAGE with multimodal branches"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000264,"raw_usage":{"total_tokens":1567,"prompt_tokens":874,"completion_tokens":693,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":607}},"tokens_in":490,"tokens_out":693,"duration_ms":5944,"temperature":1.0,"reasoning_tokens":607,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:16:20.766040+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the Random Forest baseline on the same 80/20 validation split used for the GNNs and compare per-source overall accuracy; if the baseline then matches or exceeds GraphSAGE on these sources, the claimed gains over the baseline would not hold under a matched protocol.","supporting_citations":[{"cited_title":"Procesamiento del Lenguaje Natural68 (2022)","cited_arxiv_id":null,"evidence_quote":"Provides the Spanish RoBERTa-base model used to generate text node embeddings."},{"cited_title":"In: ICLR (2018)","cited_arxiv_id":null,"evidence_quote":"Defines GAT, one of the four GNN architectures benchmarked as a baseline."},{"cited_title":"Advances in Neural Information Processing Systems30 (2017)","cited_arxiv_id":null,"evidence_quote":"Introduces GraphSAGE, the architecture that appears in all top-performing configurations."},{"cited_title":"In: International Conference on Document Analysis and Recognition","cited_arxiv_id":null,"evidence_quote":"Establishes the prior approach to text embedding for public affairs documents that this work builds on."}],"review_version":1}