Pith. sign in

REVIEW 4 major objections 5 minor 34 references

L2GNet: Optimal Local-to-Global Representation of Anatomical Structures for Generalized Medical Image Segmentation

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Pith's one-line read L2GNet replaces self-attention with optimal-transport pooling and reports higher segmentation accuracy on two medical benchmarks.

desk verdict A plausible application of optimal-transport pooling to medical segmentation that is currently undermined by unverified baseline comparability and internal inconsistencies in the reported numbers. read the letter →

arxiv 2502.05229 v1 pith:PSKGBXE2 submitted 2025-02-06 cs.CV

classification cs.CV MSC 68T0749Q2292C55
keywords medicalimagesegmentationoptimaltransportvectorquantizationdiscretelatentspaceSinkhorndistancesself-attentionreplacementmulti-organcardiac
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces L2GNet, a segmentation model whose bottleneck aligns vector-quantized image codes to a small set of learnable references using optimal transport and Sinkhorn distances, instead of computing dot-product self-attention. The authors argue that this captures long-range dependencies among pertinent anatomical regions while avoiding attention between redundant regions, which they identify as a source of false negatives and poor generalization in CLS, DLS, and CDLS models. On the Synapse multi-organ and ACDC cardiac benchmarks, L2GNet with four references reports mean Dice scores of 82.23 and 91.44, respectively, exceeding the CDLS model SynergyNet and other Transformer-based baselines. The significance would be a computationally cheaper bottleneck that generalizes better across organs and cardiac structures.

What carries the argument

The load-bearing component is the L2GMapper, a bottleneck that replaces the quadratic self-attention matrix with a linear-cost optimal-transport pooling step. It works by taking vector-quantized codes $z_{\rm dis}$, mapping them through a kernel embedding $\psi$ into an RKHS, computing the Sinkhorn transport plan $T(\psi(z_{\rm dis}), z_{\rm ref})$ between codes and a trainable reference $z_{\rm ref}$, and pooling the codes into $t$ bins with weights from the transport plan multiplied by a position kernel $S_{ij}=e^{-\frac{1}{\sigma^2_{\rm pos}}(i/n-j/t)^2}$. Multiple references $z^1_{\rm ref},\dots,z^q_{\rm ref}$ play the role of attention heads, and all Sinkhorn operations are differentiable, so the references are learned by backpropagation; the Nyström method is used as an approximation when code sets are large.

What would settle it

Re-run SynergyNet, TransUNet, SSNet, TranSSNet, and VQUNet in the exact L2GNet training configuration across multiple random seeds with identical data splits. If the mean Dice gaps on Synapse and ACDC shrink below run-to-run variance, or L2GNet no longer leads, the paper's central superiority claim is refuted. A complementary check: compute the transport plans and test whether removing the position kernel $S$ or replacing Sinkhorn with random assignment eliminates the reported gains.

Watch

Extended reading notes

Core claim

The central claim is that global context for segmentation can be obtained by optimal-transport alignment of discrete codes to a trainable reference rather than by self-attention. After the encoder produces a continuous latent and vector quantization converts it to discrete codes, the L2GMapper embeds those codes in a reproducing kernel Hilbert space, computes a Sinkhorn transport plan between the codes and a learnable reference, and pools codes into buckets weighted by both transport mass and a position-dependent similarity. The authors report that this procedure yields state-of-the-art results on Synapse and ACDC, with L2GNet(4-ref) reaching 82.23 mean DSC and 14.17 HD on Synapse and 91.44 DSC and 1.24 HD on ACDC, outperforming CLS, DLS, and CDLS baselines including SynergyNet, TransUNet, and VQUNet. They interpret the gains as evidence that the model learns inter-class and intra-class anatomical dependencies and reduces false negatives.

Load-bearing premise

The reported superiority assumes every baseline was measured under the same preprocessing, optimizer, epochs, data splits, and augmentation as L2GNet; the paper only states that it follows SynergyNet's configuration and does not show that the baseline numbers come from that same protocol.

Editorial extensions

If this is right

  • Self-attention bottlenecks in segmentation networks can be replaced by a linear-complexity optimal-transport pooling module without sacrificing accuracy on multi-organ and cardiac tasks.
  • The reported margins over SynergyNet, TransUNet, and VQUNet imply that modeling dependencies between pertinent regions, rather than all pairwise patch interactions, is sufficient for strong segmentation performance.
  • Codebook size and the number of references are tunable and affect results: L2GNet peaks at $K=512$ and $q=4$ in the reported experiments, and performance degrades at smaller codebooks and at $q=2$.
  • Because the module is a drop-in bottleneck, the same design could be attached to other encoders and decoders, potentially improving generalization on additional anatomy segmentation tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the comparison would be strongest if all baselines were re-trained under L2GNet's exact protocol with multiple seeds, since the paper's tables do not report variance and only state that SynergyNet's configuration was followed.
  • Editorial inference: the position-aware transport plan could be visualized as attention maps to test the claim that it focuses on pertinent regions and avoids redundant pairs; the paper does not provide such a quantitative analysis.
  • Editorial inference: the same optimal-transport bottleneck may extend to 3D volumes and semi-supervised settings, where the authors' claimed linear complexity and annotation efficiency would matter most.
  • Editorial inference: the method's benefit may depend on the entropy regularization $\varepsilon$ in Sinkhorn and the position-kernel width $\sigma_{\rm pos}$; ablating those would clarify whether the gain comes from optimal transport or from the added positional prior.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes L2GNet, a medical image segmentation architecture that combines vector-quantized discrete representations with a trainable optimal-transport (OT) embedding to replace dot-product self-attention. The authors argue that CLS, DLS, and CDLS methods either miss long-range dependencies or capture redundant ones, and that L2GNet's OT-based pooling onto learned references yields discriminative global representations at linear cost. The method is evaluated on the Synapse multi-organ and ACDC cardiac datasets, reporting improvements over several baselines including the authors' own SynergyNet. The central empirical claim is that L2GNet outperforms state-of-the-art CLS, DLS, and CDLS methods, but the experimental validation is incomplete and internally inconsistent.

Significance. If the central claim were rigorously established, L2GNet would be a useful contribution: it offers a concrete alternative to quadratic self-attention bottlenecks, is applicable to 2D medical image segmentation, and its integration of VQ with an OT embedding is a plausible way to encourage structured, anatomy-aware aggregation. The paper explicitly builds on the trainable OT embedding of Mialon et al. [22] and Sinkhorn's algorithm [21], and the novelty lies in adapting these tools for discrete-code aggregation in segmentation rather than inventing a new OT formalism. The complexity argument (reducing the attention matrix from O(n^2) to O(n)) is reasonable, and the use of multiple references resembles multi-head attention. However, the significance is currently limited by the lack of controlled experiments, missing error bars, and several presentation defects. No code or trained models are released, which further hampers verification of the empirical claims.

major comments (4)
  1. [Section 4, Dataset and Experiment Settings] The experimental section never specifies the preprocessing and training configuration claimed to follow SynergyNet [16] (e.g., input resolution, optimizer, learning rate, batch size, number of epochs, loss terms, augmentation, and official train/validation splits), nor does it state whether the CLS/DLS/CDLS baselines in Tables 1 and 2 were retrained under L2GNet's protocol or their numbers were taken from the original publications. Because SynergyNet, SSNet, and TranSSNet are prior work by the same authors, a controlled re-run of all baselines in the same setting is feasible and is required to support the claimed superiority; otherwise the margins in Tables 1 and 2 may reflect protocol differences rather than architectural benefit.
  2. [Tables 1 and 2] No error bars, confidence intervals, or significance tests are reported for any metric. The ACDC DSC margin between L2GNet(4-ref) and TranSSNet is only 0.12 points (91.44 vs 91.32), and between L2GNet(4-ref) and SynergyNet is 1.66 points; with the run-to-run variance typical of medical image segmentation, these differences cannot be judged without multiple seeds. The HD metric is particularly sensitive to outliers and should be reported with variance.
  3. [Tables 1, 3, 4, and 5] The baseline numbers are internally inconsistent: SynergyNet's Synapse HD is 23.59 in Table 1 but 23.29 in Tables 4 and 5, and its Synapse DSC is 79.65 in Table 1 but 79.61 in Table 3. This suggests that baseline scores were not produced by a single controlled evaluation protocol, and it directly undermines the credibility of the comparisons in the main tables. The authors should rerun all baselines in exactly the L2GNet setting and report a single consistent set of numbers.
  4. [Section 3, Eq. (3)] Equation (3), which defines the proposed embedding phi_{z_ref}(z_dis), is not readable as typeset: it mixes the indices 1i and tit, contains an unexplained '×S,...,' and the expression is not a well-formed vector. In addition, the preceding sentence defines the cost as M = -psi(z_dis), but M should be a pairwise cost matrix between codes and reference elements. Because this equation is the core of the proposed L2GMapper, it must be rewritten precisely and consistently.
minor comments (5)
  1. [Abstract and Keywords] The listed keywords ('Cirrhotic liver segmentation, Abdominal MRI dataset, liver segmentation, ...') do not match the paper's content, which concerns multi-organ and cardiac segmentation via optimal transport; these keywords should be corrected.
  2. [Tables 3 and 6] Tables 3 and 6 are identical (codebook embedding size analysis) and should be merged or one should be removed.
  3. [Abstract] The phrase 'an novel approach' contains a grammatical error; it should be 'a novel approach.'
  4. [Section 2, Eq. (1)] The notation is inconsistent: the text introduces pairwise costs C, but Eq. (1) uses M_{ij}; the variable should be matched to avoid ambiguity.
  5. [References] Reference [1] (Kawaguchi et al.) appears not to be cited in the body of the paper; either cite it or remove it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core method is an external OT embedding and the claims are benchmark evaluations.

full rationale

The derivation chain of L2GNet is not circular. The L2GMapper module relies on a trainable optimal transport embedding, and the paper explicitly attributes that embedding and the Nyström approximation to external references ([21], [22], [23], [25]) rather than to the authors' own prior work. The main claims are empirical results on the Synapse and ACDC benchmarks. No fitted constant is relabeled as a prediction, and no output is defined in terms of the quantity it is supposed to predict. The paper does use the authors' own SynergyNet/SSNet/TranSSNet as baselines and cites SynergyNet [16] for the training configuration, but this is not a load-bearing mathematical self-citation: the comparison numbers are benchmark scores, and the configuration reference affects experimental protocol, not the logical derivation of the method. Concerns that baseline numbers may have been produced under different training protocols, and the internal inconsistencies in the reported SynergyNet values across Tables 1, 3, 4, and 5, are experimental-comparability and reporting-quality issues, not circularity, because they do not make L2GNet's result equivalent to its inputs by construction. No circular step can be exhibited by quoting a specific equation whose output is identical to its input by definition. The paper is self-contained against external benchmarks, so the honest finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard building blocks: VQ from [14], Sinkhorn OT from [21], and the OT embedding from [22]. Free parameters include codebook size, latent dimension, number of references, Sinkhorn iterations, and positional scaling. The paper tunes K and q without a disclosed validation protocol. Axioms include standard OT convergence, the fidelity of the linear-kernel Nystrom approximation, and the assumption that VQ codes retain enough information for pooling. No invented entities are introduced.

free parameters (5)
  • Codebook size K = 512
    Table 3 sweeps K from 64 to 1024 and reports K=512 as best; no validation protocol is stated for this choice.
  • Latent dimension dim = 1024
    Used in the quantizer; not swept in the paper and chosen by hand.
  • Number of references q = 4
    Tables 4 and 5 compare q=2 and q=4 and select q=4; no validation split is disclosed.
  • Sinkhorn iterations = 10
    Set to 10 for inner optimization; no sensitivity analysis is provided.
  • Positional scaling sigma_pos = not reported
    Used in the definition of the positional weighting matrix S in Eq. 3; the value is not stated.
assumptions (4)
  • standard math Sinkhorn's algorithm converges to a unique solution of the entropic OT problem (Eq. 1).
    Relied on in Section 3 to compute transport plans T between discrete codes and references.
  • domain assumption The linear-kernel Nystrom approximation of the RKHS embedding is faithful.
    Used in Section 3 to make the embedding explicit for high-dimensional MRI features; its accuracy is assumed and not verified.
  • domain assumption Vector quantization preserves enough anatomical information for downstream optimal-transport pooling.
    Assumed throughout the method; no analysis of information loss is provided.
  • domain assumption The positional weighting matrix S encodes meaningful spatial relationships and improves alignment.
    Introduced in Eq. 3 with a specific Gaussian form taken from [22]; its effectiveness on these medical datasets is not validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of L2GNet: Optimal Local-to-Global Representation of Anatomical Structures for Generalized Medical Image Segmentation." pith.science (2026). https://pith.science/paper/PSKGBXE2

@misc{pith2026250205229,
  author       = {Pith},
  title        = {Pith review of: L2GNet: Optimal Local-to-Global Representation of Anatomical Structures for Generalized Medical Image Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PSKGBXE2}},
  note         = {Machine review of arXiv:2502.05229}
}
read the original abstract

Continuous Latent Space (CLS) and Discrete Latent Space (DLS) models, like AttnUNet and VQUNet, have excelled in medical image segmentation. In contrast, Synergistic Continuous and Discrete Latent Space (CDLS) models show promise in handling fine and coarse-grained information. However, they struggle with modeling long-range dependencies. CLS or CDLS-based models, such as TransUNet or SynergyNet are adept at capturing long-range dependencies. Since they rely heavily on feature pooling or aggregation using self-attention, they may capture dependencies among redundant regions. This hinders comprehension of anatomical structure content, poses challenges in modeling intra-class and inter-class dependencies, increases false negatives and compromises generalization. Addressing these issues, we propose L2GNet, which learns global dependencies by relating discrete codes obtained from DLS using optimal transport and aligning codes on a trainable reference. L2GNet achieves discriminative on-the-fly representation learning without an additional weight matrix in self-attention models, making it computationally efficient for medical applications. Extensive experiments on multi-organ segmentation and cardiac datasets demonstrate L2GNet's superiority over state-of-the-art methods, including the CDLS method SynergyNet, offering an novel approach to enhance deep learning models' performance in medical image analysis.

Figures

Figures reproduced from arXiv: 2502.05229 by the authors.

Figure 1
Figure 1. Illustrates the workflow of Proposed L2GNet. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Segmentation maps on Synapse and ACDC datasets are shown with color-code (First three rows, yellow: liver, blue: right kidney, green: [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 30 canonical work pages

  1. [22]

    A Trainable Optimal Transport Embedding for Feature Aggregation and its Relationship to Attention

    G. Mialon et al., “A trainable optimal transport em- bedding for feature aggregation and its relationship to attention,” arXiv preprint arXiv:2006.12065, 2020

  2. [21]

    Sinkhorn distances: Lightspeed com- putation of optimal transport,

    M. Cuturi, “Sinkhorn distances: Lightspeed com- putation of optimal transport,” Advances in Neural Information Processing Systems, vol. 26, 2013

  3. [16]

    Synergy- Net: Bridging the gap between discrete and continuous representations for precise medical image segmenta- tion,

    V . Gorade, S. Mittal, D. Jha, and U. Bagci, “Synergy- Net: Bridging the gap between discrete and continuous representations for precise medical image segmenta- tion,” in Proc. WACV, pp. 7768–7777, 2024. 7

  4. [1]

    Generalization in deep learning,

    K. Kawaguchi, L. P. Kaelbling, and Y . Bengio, “Generalization in deep learning,” arXiv preprint 6 Table 6: Codebook embedding size analysis. Kdim Synapse ACDC SynergyNet L2GNet SynergyNet L2GNet DSC HD DSC HD DSC HD DSC HD 1024 77.61 29.53 80.27 24.00 88.64 2.12 91.41 1.36 512 79.61 23.89 82.23 14.17 88.89 1.86 91.44 1.24 256 79.21 30.07 81.35 22.92 89.1...

  5. [2]

    Survey on deep learning for radiotherapy,

    P. Meyer, V . Noblet, C. Mazzara, and A. Lallement, “Survey on deep learning for radiotherapy,”Computers in Biology and Medicine , vol. 98, pp. 126–146, 2018

  6. [3]

    Learning mutually local-global U-nets for high- resolution retinal lesion segmentation in fundus im- ages,

    Z. Yan, X. Han, C. Wang, Y . Qiu, Z. Xiong, and S. Cui, “Learning mutually local-global U-nets for high- resolution retinal lesion segmentation in fundus im- ages,” in Proc. IEEE ISBI, pp. 597–600, 2019

  7. [4]

    Col- laborative global-local networks for memory-efficient segmentation of ultra-high resolution images,

    W. Chen, Z. Jiang, Z. Wang, K. Cui, and X. Qian, “Col- laborative global-local networks for memory-efficient segmentation of ultra-high resolution images,” inProc. IEEE/CVF CVPR, pp. 8924–8933, 2019

  8. [5]

    Contrastive learning of global and local features for medical image segmentation with limited annotations,

    K. Chaitanya, E. Erdil, N. Karani, and E. Konukoglu, “Contrastive learning of global and local features for medical image segmentation with limited annotations,” Advances in Neural Information Processing Systems , vol. 33, pp. 12546–12558, 2020

Show all 34 references
  1. [6]

    Attention is all you need,

    A. Vaswani et al., “Attention is all you need,”Advances in Neural Information Processing Systems , vol. 30, 2017

  2. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020

  3. [8]

    Swin transformer: Hierarchical vi- sion transformer using shifted windows,

    Z. Liu et al., “Swin transformer: Hierarchical vi- sion transformer using shifted windows,” in Proc. IEEE/CVF ICCV, pp. 10012–10022, 2021

  4. [9]

    UNet ++: A nested U-Net architecture for medical image segmentation,

    Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “UNet ++: A nested U-Net architecture for medical image segmentation,” in Proc. Int. Workshop Deep Learning Med. Image Analysis , pp. 3–11, 2018

  5. [10]

    Attention U-Net: Learning where to look for the pancreas,

    O. Oktay et al., “Attention U-Net: Learning where to look for the pancreas,” arXiv preprint arXiv:1804.03999, 2018

  6. [11]

    Swin-UNet: UNet-like pure trans- former for medical image segmentation,

    H. Cao et al., “Swin-UNet: UNet-like pure trans- former for medical image segmentation,” in Proc. ECCV, pp. 205–218, 2022

  7. [12]

    TransUNet: Transformers make strong encoders for medical image segmentation,

    J. Chen et al., “TransUNet: Transformers make strong encoders for medical image segmentation,” arXiv preprint arXiv:2102.04306, 2021

  8. [13]

    Vector quantisation for robust segmentation,

    A. Santhirasekaram et al., “Vector quantisation for robust segmentation,” in Proc. MICCAI, pp. 663–672, 2022

  9. [14]

    Neural discrete representation learning,

    A. Van Den Oord and O. Vinyals, “Neural discrete representation learning,” Advances in Neural Informa- tion Processing Systems, vol. 30, 2017

  10. [15]

    HiFormer: Hierarchical multi- scale representations using transformers for medical image segmentation,

    M. Heidari et al., “HiFormer: Hierarchical multi- scale representations using transformers for medical image segmentation,” in Proc. WACV, pp. 6202–6212, 2023

  11. [17]

    Harmonized spatial and spectral learning for robust and generalized medical image segmentation,

    V . Gorade, S. Mittal, D. Jha, R. Singhal, and U. Bagci, “Harmonized spatial and spectral learning for robust and generalized medical image segmentation,” arXiv preprint arXiv:2401.10373, 2024

  12. [18]

    Supervised contrastive learning,

    P. Khosla et al., “Supervised contrastive learning,” Advances in Neural Information Processing Systems , vol. 33, pp. 18661–18673, 2020

  13. [19]

    Domain adaptive relational reasoning for 3D multi-organ segmentation,

    S. Fu et al., “Domain adaptive relational reasoning for 3D multi-organ segmentation,” in Proc. MICCAI, pp. 656–666, 2020

  14. [20]

    U-Net: Convolutional networks for biomedical image segmen- tation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmen- tation,” in Proc. MICCAI, pp. 234–241, 2015

  15. [23]

    Using the Nystr ¨om method to speed up kernel machines,

    C. Williams and M. Seeger, “Using the Nystr ¨om method to speed up kernel machines,” Advances in Neural Information Processing Systems, vol. 13, 2000

  16. [24]

    Computational opti- mal transport,

    G. Peyr ´e, M. Cuturi, et al., “Computational opti- mal transport,” Center for Research in Economics and Statistics Working Papers, no. 2017-86, 2017

  17. [25]

    End-to-end kernel learning with super- vised convolutional kernel networks,

    J. Mairal, “End-to-end kernel learning with super- vised convolutional kernel networks,” Advances in Neural Information Processing Systems, vol. 29, 2016

  18. [26]

    Biological se- quence modeling with convolutional kernel networks,

    D. Chen, L. Jacob, and J. Mairal, “Biological se- quence modeling with convolutional kernel networks,” Bioinformatics, vol. 35, no. 18, pp. 3294–3302, 2019

  19. [27]

    [Online]

    Multi-Atlas Abdomen Labeling Challenge: Synapse Multi-Organ Segmentation Dataset , Synapse Consortium, 2015. [Online]. Avail- able: https://www.synapse.org/#!Synapse: syn3193805/wiki/217789

  20. [28]

    ACDC (Automated Cardiac Diagnosis Challenge) ,

  21. [29]

    Mean teachers are bet- ter role models: Weight-averaged consistency targets improve semi-supervised deep learning results,

    A. Tarvainen and H. Valpola, “Mean teachers are bet- ter role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” Ad- vances in Neural Information Processing Systems , vol. 30, 2017

  22. [30]

    Advent: Adversarial entropy mini- mization for domain adaptation in semantic segmen- tation,

    T.-H. Vu et al., “Advent: Adversarial entropy mini- mization for domain adaptation in semantic segmen- tation,” in Proc. IEEE /CVF CVPR , pp. 2517–2526, 2019

  23. [31]

    Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmenta- tion,

    L. Yu et al., “Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmenta- tion,” in Proc. MICCAI, pp. 605–613, 2019

  24. [32]

    Interpolation consistency training for semi-supervised learning,

    V . Verma et al., “Interpolation consistency training for semi-supervised learning,” Neural Networks, vol. 145, pp. 90–106, 2022

  25. [33]

    Semi-supervised semantic seg- mentation with cross pseudo supervision,

    X. Chen et al., “Semi-supervised semantic seg- mentation with cross pseudo supervision,” in Proc. IEEE/CVF CVPR, pp. 2613–2622, 2021. 8

  26. [2017]

    Available: https://www.creatis

    [Online]. Available: https://www.creatis. insa-lyon.fr/Challenge/acdc

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.