pith. sign in

arxiv: 2606.18721 · v1 · pith:HVJBB6CXnew · submitted 2026-06-17 · 💻 cs.CV

Rethinking the Pointer Loss in Table Structure Recognition: Geometry-Aware Pointer Loss for Spatial Locality

Hong-Jun Choi , Jongho Lee , Jaeyoung Kim This is my paper

Pith reviewed 2026-06-26 21:24 UTC · model grok-4.3

classification 💻 cs.CV
keywords table structure recognitionpointer networkgeometry-aware lossinverse distance weightingspatial localityadjacent cell errorstsr
0
0 comments X

The pith

Reweighting cross-entropy loss by inverse cell distance reduces adjacent errors in table structure recognition.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Pointer networks for table structure recognition fail most often by mixing up cells that sit next to each other. Standard cross-entropy loss treats every wrong prediction the same regardless of location. The Geometry-Aware Pointer loss changes the objective so that mistakes between nearby cells receive larger gradient updates through inverse distance weighting. This adjustment requires only a change in the loss calculation and leaves the network and inference unchanged. Experiments on two standard table datasets show fewer local errors and higher overall accuracy.

Core claim

The paper establishes that reweighting the pointer network's cross-entropy loss according to the Manhattan distance between predicted and ground-truth cells directs stronger gradients toward spatially adjacent mistakes, which account for 79.6 percent of errors, and that this yields improved recognition accuracy on PubTabNet and SynthTabNet.

What carries the argument

Geometry-Aware Pointer Loss applies inverse distance weighting to negative candidates in the cross-entropy objective so that closer cells exert stronger influence on the gradient.

If this is right

  • Adjacent-cell errors decrease because gradients emphasize local mistakes over distant ones.
  • State-of-the-art results are reached on the evaluated table datasets.
  • The same model architecture works with only a change in the loss calculation.
  • No additional cost occurs during inference.
  • Geometric information incorporated at the loss level aids robustness in table structure tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This reweighting could be adapted to other pointer-based prediction tasks that involve spatial layouts.
  • Future work might examine whether the same weighting helps when tables have irregular layouts not seen in the training data.
  • Combining this loss with other geometric features in the model architecture might produce further gains.
  • Testing on tables from different domains like financial reports or scientific papers would check if the error pattern persists.

Load-bearing premise

The pattern that most errors occur between cells within a Manhattan distance of two is typical for table structure recognition in general.

What would settle it

Training a pointer network on a table dataset where most errors occur between cells farther apart and observing that the geometry-aware loss performs worse than standard cross-entropy.

Figures

Figures reproduced from arXiv: 2606.18721 by Hong-Jun Choi, Jaeyoung Kim, Jongho Lee.

Figure 1
Figure 1. Figure 1: Pointer error locality. Histogram of errors by Manhattan distance d between the predicted and ground-truth tags (bars), with cumulative percentage (green). 59.2% of errors occur at d = 1, 79.6% within d ≤ 2, and >90% within d ≤ 4. Recent TSR systems have converged on pointer-based architectures [7,6,12] that jointly predict structural tags (HTML/OTSL [10]) and align them to de￾tected text regions. These mo… view at source ↗
Figure 2
Figure 2. Figure 2: OTSL representation of table structure. From left to right: (a) Orig￾inal table with detected text regions, (b) Reconstructed grid structure showing cell boundaries and spans, (c) OTSL token sequence in grid format where green C tokens represent cells requiring pointer mapping to text regions, colored tokens (L, U, X) indicate span continuations, and NL marks row endings. The pointer mechanism maps each C … view at source ↗
Figure 3
Figure 3. Figure 3: Analysis of pointer network confusion patterns. (a) Attention heatmap showing that errors concentrate around the correct region: ground truth receives 6.39% attention while nearby cells (d = 1, 2) collectively capture 47.6%. (b) Mismatch between error distribution (79.6% at d ≤ 2) and gradient distri￾bution (26.7% at d ≤ 2) reveals the spatial blindness of standard cross-entropy loss. Learning Inefficiency… view at source ↗
Figure 4
Figure 4. Figure 4: Weight distribution comparison in a 7×7 table grid. (a) Standard pointer loss assigns uniform weights (w=1.0) to all negative candidates, ignoring spatial structure. (b) Our GAP applies distance-based weighting with geometric decay. The center cell (GT) represents the ground truth position. 4 Methodology 4.1 Spatial Weight Design Our analysis motivates the GAP loss, a training objective designed to integra… view at source ↗
Figure 5
Figure 5. Figure 5: TEDS vs Position Accuracy comparison. Ground truth table (left) and model prediction (right) where all data columns are systematically shifted. TEDS-structure achieves 100% and TEDS evaluates this as 90.0% due to pre￾served structural relationships and partial content scores, while Position Accu￾racy reveals the true severity with only 50.0% of cells correctly positioned. This discrepancy highlights TEDS’s… view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of TFLOP and TFLOP+GAP performance across different [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
read the original abstract

Table Structure Recognition (TSR) using a pointer network achieves impressive results by predicting HTML sequences while aligning tags to detected text (or cell) regions. However, our analysis reveals that when pointer networks fail, 79.6% of errors occur between spatially adjacent cells (Manhattan distance <= 2). Despite this, standard cross-entropy loss weights all negative candidates equally. In this work, we propose Geometry-Aware Pointer (GAP) Loss, which reweights the cross-entropy objective based on spatial proximity to ground truth. By applying inverse distance weighting, GAP focuses gradient flow where the model struggles most: immediate neighbors receive stronger gradients than distant cells. Our approach requires only a straightforward modification to the loss computation, maintaining the same model architecture with zero additional inference cost. Extensive experiments on PubTabNet and SynthTabNet demonstrate that GAP consistently reduces adjacent-cell errors, achieving new state-of-the-art performance. Our findings suggest that incorporating geometric inductive biases at the loss level provides a simple yet effective approach to robust TSR. Our code is available at https://github.com/teamreboott/GAP

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper analyzes failures in pointer-network-based table structure recognition (TSR), reporting that 79.6% of errors occur between cells with Manhattan distance ≤2. It proposes Geometry-Aware Pointer (GAP) Loss, a modification to cross-entropy that applies inverse-distance reweighting to negative candidates so that spatially closer cells receive higher loss weight. The method requires no architecture changes or inference overhead. Experiments on PubTabNet and SynthTabNet show reduced adjacent-cell errors and new state-of-the-art results; code is released.

Significance. If the empirical gains hold under the proposed reweighting, the work shows that a lightweight geometric bias injected only at the loss level can measurably improve pointer-based TSR without altering model capacity or runtime. The public code release is a concrete strength that aids reproducibility. The significance is limited by the extent to which the motivating error distribution generalizes beyond the two evaluated datasets.

major comments (2)
  1. [Introduction / §3] Error analysis (Introduction and §3): the 79.6% adjacent-cell failure rate is presented as the dominant, general failure mode that justifies inverse-distance reweighting. No additional datasets, table-density variants, or cross-domain splits are used to test whether this statistic is representative; if the distribution is PubTabNet/SynthTabNet-specific, the reweighting may over-penalize local confusions while under-penalizing long-range pointer errors on other topologies.
  2. [§4] §4 (Experiments) and loss definition: the manuscript reports consistent gains and SOTA numbers but supplies neither per-run standard deviations, statistical significance tests, nor an ablation isolating the exact inverse-distance formula (including distance metric, zero-distance handling, and normalization). These omissions make it impossible to verify that the reported improvements are robustly attributable to the geometric reweighting rather than other training factors.
minor comments (2)
  1. [§3.2] The precise mathematical definition of the GAP loss (reweighting term, handling of the ground-truth cell itself) should be stated as an explicit equation rather than described only in prose.
  2. [Tables 1-2] Figure captions and table footnotes should explicitly state whether the reported metrics are on the official test splits and whether any post-processing differs from the baselines.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We respond to each major comment below, indicating planned revisions where appropriate.

read point-by-point responses

  1. Referee: [Introduction / §3] Error analysis (Introduction and §3): the 79.6% adjacent-cell failure rate is presented as the dominant, general failure mode that justifies inverse-distance reweighting. No additional datasets, table-density variants, or cross-domain splits are used to test whether this statistic is representative; if the distribution is PubTabNet/SynthTabNet-specific, the reweighting may over-penalize local confusions while under-penalizing long-range pointer errors on other topologies.

    Authors: The reported 79.6% statistic is derived directly from error cases on PubTabNet and SynthTabNet, the standard benchmarks used throughout the TSR literature and on which GAP Loss is evaluated. The geometric reweighting is motivated by and validated against the failure modes observed in these datasets. While we agree that analysis on further datasets or topologies would help establish broader generality, the current results show consistent error reduction and SOTA performance on the evaluated benchmarks. We will add a short discussion of this scope limitation in the revised manuscript. revision: partial

  2. Referee: [§4] §4 (Experiments) and loss definition: the manuscript reports consistent gains and SOTA numbers but supplies neither per-run standard deviations, statistical significance tests, nor an ablation isolating the exact inverse-distance formula (including distance metric, zero-distance handling, and normalization). These omissions make it impossible to verify that the reported improvements are robustly attributable to the geometric reweighting rather than other training factors.

    Authors: We agree that additional statistical reporting and targeted ablations would improve verifiability. In the revised manuscript we will report mean performance and standard deviations over multiple independent runs with different random seeds, include statistical significance tests, and add an ablation isolating the inverse-distance formulation (distance metric, zero-distance handling, and normalization). revision: yes

Circularity Check

0 steps flagged

No circularity: loss modification is an independent design choice motivated by empirical failure analysis

full rationale

The paper's central contribution is an empirical observation of error distribution (79.6% adjacent-cell failures) followed by a direct, non-circular reweighting of standard cross-entropy loss via inverse distance. This does not reduce to any self-definition, fitted-input prediction, or self-citation chain. The reweighting is presented as a straightforward modification with no parameters fitted to the target metric and no uniqueness theorem invoked. The derivation chain is self-contained against external benchmarks (standard pointer-network TSR losses) and does not rely on load-bearing self-citations.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The approach rests on the standard cross-entropy objective for pointer networks plus the domain assumption that spatial adjacency is the dominant error mode; no new free parameters or invented entities are introduced.

axioms (1)
  • domain assumption Cross-entropy is the appropriate base loss for training pointer networks on HTML tag sequences
    Standard practice in sequence-to-sequence pointer models for TSR.

pith-pipeline@v0.9.1-grok · 5729 in / 1125 out tokens · 20128 ms · 2026-06-26T21:24:36.157161+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 3 canonical work pages

  1. [1]

    Complicated table structure recognition, 2019

    Chi, Z., Huang, H., Xu, H.D., Yu, H., Yin, W., Mao, X.L.: Complicated table structure recognition. arXiv preprint arXiv:1908.04729 (2019) 3

    work page arXiv 1908

  2. [2]

    In: Swiss Text Analytics Conference (SwissText)

    Dimitri Romaric, N.M., Petrucci, A., Marinai, S., Hennebert, J.: Slanet-1m: a lightweight and efficient model for table recognition with minimal computational cost. In: Swiss Text Analytics Conference (SwissText). vol. 2025, pp. To–be (2025) 3

    2025

  3. [3]

    In: 2019 International Conference on Document Analysis and Recognition (ICDAR)

    Gao, L., Huang, Y., Déjean, H., Meunier, J.L., Yan, Q., Fang, Y., Kleber, F., Lang, E.M.: ICDAR 2019 competition on table detection and recognition (cTDaR). In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 1510–1515. IEEE (2019) 4

    2019

  4. [4]

    In: Proceedings of the 12th ACM symposium on Document engineering

    Göbel, M., Hassan, T., Oro, E., Orsi, G.: A methodology for evaluating algorithms for table understanding in PDF documents. In: Proceedings of the 12th ACM symposium on Document engineering. pp. 45–48. ACM (2012) 4

    2012

  5. [5]

    In: Proceedings of the IEEE International Conference on Computer Vision (ICCV)

    He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). pp. 2961–2969 (2017) 4

    2017

  6. [6]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Huang, Y., Lv, N., Zhang, L., Cai, S., Liang, R., Chen, Z., Li, X., Pan, B., Chen, X.: Improving table structure recognition with visual-alignment sequential coordinate modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11134–11143 (2023) 2, 3 Geometry-Aware Pointer Loss for Spatial Locality 15

    2023

  7. [7]

    In: Proceedings of the Thirty-Third International Joint Con- ference on Artificial Intelligence

    Khang, M., Hong, T.: Tflop: Table structure recognition framework with layout pointer mechanism. In: Proceedings of the Thirty-Third International Joint Con- ference on Artificial Intelligence. pp. 947–955 (2024) 2, 4, 6, 10

    2024

  8. [8]

    In: International conference on document analysis and recognition

    Kudale, D., Kasuba, B.V., Subramanian, V., Chaudhuri, P., Ramakrishnan, G.: Sprint: script-agnostic structure recognition in tables. In: International conference on document analysis and recognition. pp. 350–367. Springer (2024) 3

    2024

  9. [9]

    Folio: natural language reasoning with first-order logic, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp

    Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoy- anov, V., Zettlemoyer, L.: BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J. (eds.) Proceedings of the 58th Annual Meet- ing of the Association for Computati...

    work page doi:10.18653/v1/ 2020

  10. [10]

    Lysak, M., Nassar, A., Livathinos, N., Fornés, A., Staar, P.: Optimized table tok- enizationfortablestructurerecognition.In:InternationalConferenceonDocument Analysis and Recognition. pp. 37–52. Springer (2023) 2

    2023

  11. [11]

    Pattern Recognition133, 109006 (2023) 3, 4

    Ma, C., Lin, W., Sun, L., Huo, Q.: Robust table detection and structure recognition from heterogeneous document images. Pattern Recognition133, 109006 (2023) 3, 4

    2023

  12. [12]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Nassar, A., Livathinos, N., Lysak, M., Staar, P.: Tableformer: Table structure un- derstanding with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4614–4623 (2022) 2, 3, 10

    2022

  13. [13]

    In: 2019 International Conference on Document Analysis and Recognition (ICDAR)

    Qasim, S.R., Mahmood, H., Shafait, F.: Rethinking table recognition using graph neural networks. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 142–147. IEEE (2019) 3

    2019

  14. [14]

    In: 2019 International Conference on Document Analysis and Recognition (ICDAR)

    Riba, P., Dutta, A., Goldmann, L., Fornés, A., Ramos, O., Lladós, J.: Table de- tection in invoice documents by graph neural networks. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 122–127. IEEE (2019) 3

    2019

  15. [15]

    In: IJCAI

    Shen, H., Gao, X., Wei, J., Qiao, L., Zhou, Y., Li, Q., Cheng, Z.: Divide rows and conquer cells: Towards structure recognition for large tables. In: IJCAI. pp. 1369–1377 (2023) 3

    2023

  16. [16]

    Grits: Grid table similarity metric for table structure recognition, 2022

    Smock, B., Pesala, R., Abraham, R.: Grits: Grid table similarity metric for table structure recognition. arXiv preprint arXiv:2203.12555 (2022) 4

    work page arXiv 2022

  17. [17]

    In: Advances in Neural Information Processing Systems

    Vinyals, O., Fortunato, M., Jaitly, N.: Pointer networks. In: Advances in Neural Information Processing Systems. vol. 28 (2015) 4

    2015

  18. [18]

    Pattern Recognition135, 109150 (2023) 3, 4

    Wang, J., Liu, W., Du, J.: Robust table structure recognition with dynamic queries enhanced detection transformer. Pattern Recognition135, 109150 (2023) 3, 4

    2023

  19. [19]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Xue, W., Yu, B., Wang, W., Tao, D., Li, Q.: Tgrnet: A table graph reconstruc- tion network for table structure recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1295–1304 (2021) 3

    2021

  20. [20]

    In: ICDAR 2021 Competition on Scientific Literature Parsing

    Ye, J., Qi, X., He, Y., Chen, Y., Gu, D., Gao, P., Xiao, R.: Pingan-vcgroup’s solution for icdar 2021 competition on scientific literature parsing task b: table recognition to html. In: ICDAR 2021 Competition on Scientific Literature Parsing. pp. 1–8 (2021) 3

    2021

  21. [21]

    Pattern Recognition126, 108565 (2022) 3

    Zhang, Z., Zhang, J., Du, J., Wang, F.: Split, embed and merge: An accurate table structure recognizer. Pattern Recognition126, 108565 (2022) 3

    2022

  22. [22]

    In: European conference on computer vision

    Zhong, X., ShafieiBavani, E., Jimeno Yepes, A.: Image-based table recognition: data, model, and evaluation. In: European conference on computer vision. pp. 564–580. Springer (2020) 4, 10

    2020