Pith. sign in

REVIEW 5 major objections 5 minor 24 references

Query Nearby: Offset-Adjusted Mask2Former enhances small-organ segmentation

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Offset-adjusted Mask2Former reaches state-of-the-art scores on small-organ segmentation.

desk verdict The offset-adjustment idea is worth a look, but the SOTA claim does not survive a matched-protocol check: two datasets, two evaluation mismatches. read the letter →

arxiv 2506.05897 v1 pith:ASICWMES submitted 2025-06-06 cs.CV

classification cs.CV
keywords Mask2Formerdeformableattentionsmall-organsegmentationheadandneckCTorgan-at-riskfeaturefusionauxiliarydecodermedicalimage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that the standard Mask2Former segmentation model can be made reliable on small and mid-sized organs in head-and-neck CT by changing how deformable attention samples query points. The proposed modifications are three offset-adjustment strategies that keep sampling near compact foreground regions, a fusion of the fourth feature map that Mask2Former normally discards, and an FCN-based auxiliary decoder that supplies coarse organ location and background contrast. The authors report a mean Dice of 81.60 on HaN-Seg against a previous state of the art of 81.26, and 87.77 on SegRap2023 against 86.70, with the largest gains on small organs such as cochlea and optic nerve. If these results hold, a lightweight 2D transformer can compete with heavier 3D and multi-modal systems in a clinically demanding setting.

What carries the argument

The load-bearing mechanism is offset control inside deformable attention, the sampling scheme introduced by Deformable DETR and reused by Mask2Former. In that scheme each query predicts an offset and attention is computed over points at the query position plus the offset; the paper modifies this computation so the offsets are pulled toward the query. Strategy 1 divides any offset above a fixed threshold by a constant, strategy 2 applies softmax to contract all offsets, and strategy 3 applies softmax and then multiplies by a constant greater than 1, which the experiments identify as most effective. Two supporting mechanisms are the fusion of the fourth feature map, used as a coarse memory that provides organ location and margin information, and an FCN-based background-location sensitive decoder that gives a quick coarse segmentation against the background. Together they let the encoder spend attention on foreground and accelerate training.

What would settle it

Re-run the two comparisons under matched protocols: on SegRap2023 use the official overlap-aware metric on the full test set, and on HaN-Seg evaluate on the same 56-case CT/MR_T1 protocol used by SegReg. If the resulting mDice values are no longer above 86.70 and 81.26 respectively, the claimed state-of-the-art margins are an artifact of protocol differences.

Watch

Extended reading notes

Core claim

The authors' central claim is that transformer-based segmentation fails on small organs mainly because deformable attention lets sampling points drift too far from the query, wasting computation on background. Their remedy is to control the offset so sampled points stay within the same compact organ, tested with three strategies: a manual threshold that shrinks large offsets, a softmax contraction of offsets, and a softmax contraction followed by scaling by a constant greater than one, the last proving most effective. They further claim that reusing the fourth feature map as coarse location guidance and adding a background-location-sensitive FCN decoder both improve accuracy and training speed. With all components, their model reaches a mean Dice of 81.60 on HaN-Seg and 87.77 on SegRap2023, reported as state of the art, especially for mid-sized and small organs.

Load-bearing premise

The reported state-of-the-art margins depend on comparing scores that were not computed under exactly the same protocol: on SegRap2023 the authors count only non-overlapping organ regions, and on HaN-Seg they evaluate on 42 cases while the previous SOTA used 56; if those differences are removed, the margins could shrink or disappear.

Editorial extensions

If this is right

  • Since the offset-adjustment operates only on the deformable-attention computation, any transformer that uses that sampling mechanism can adopt it without changing the backbone.
  • The fourth feature map already exists in Mask2Former, so fusing it adds no new feature-extraction cost while giving the model coarse organ-location information.
  • The background-location sensitive decoder gives a coarse foreground/background segmentation that helps training converge faster, so the approach is promising for low-resource training settings.
  • If the reported margins hold, average Dice above 80 percent on both benchmarks means the method reaches clinically acceptable territory for head-and-neck organs while using only 2D CT slices.
  • The largest per-organ gains are on small structures such as cochlea and optic nerve, while symmetric or irregular organs like the lacrimal gland remain unstable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same offset-control principle should transfer to deformable-attention models outside segmentation, such as small-object detection in aerial or pathology imagery, where foreground is sparse and compact.
  • If a matched-protocol evaluation confirms the reported margins, the practical implication is that registration-based multi-modal preprocessing can be replaced by a cheaper attention-level change plus a localization head, lowering the compute barrier for clinical deployment.
  • A natural extension would be to apply the offset adjustment within 3D deformable attention, since the current experiments use 2D slices and the paper's own framing suggests the gains should persist in volumetric sampling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes three modifications to Mask2Former for small-organ segmentation: offset-adjustment strategies for deformable attention, fusion of the fourth feature map, and an FCN-based background-location sensitive auxiliary decoder. The authors report mDice 81.60 on HaN-Seg and 87.77 on SegRap2023, claiming state-of-the-art performance, and release code. The internal ablation on SegRap shows gains over a naive Mask2Former baseline, but the external SOTA comparisons use incompatible evaluation protocols.

Significance. If the results are confirmed under matched evaluation, the proposed modifications are simple and could be useful for small-organ segmentation, particularly the offset-adjustment idea and the auxiliary decoder. The paper ships code, which supports reproducibility, and the ablation demonstrates internal improvements on the SegRap dataset. However, the headline SOTA claim is currently unsupported because the comparison to prior work is not apples-to-apples, and the ablation contains an internal inconsistency about which configuration constitutes the proposed method.

major comments (5)
  1. [Table 1 (SegRap2023 footnote)] The SegRap2023 comparison is not valid under the official challenge metric. The footnote states "we just count those not overlapped regions of different organs," whereas the quoted SOTA (Y. Zhong et al., 86.70) is the official overlap-aware result. Excluding overlapping voxels changes the denominator of the Dice score and can inflate the reported value; the 1.17% margin is within the range such a protocol change could produce. Please re-evaluate on the official metric or clearly state that the comparison is against a non-standard baseline.
  2. [Table 1 (HaN-Seg rows)] The HaN-Seg SOTA comparison is undermined by a test-set mismatch: "Ours" is evaluated on 42 cases, while the SegReg(SOTA) row uses 56 cases. Since 42 is the open subset of the HaN-Seg data, the excluded cases are not random, and the 0.34% mDice margin could disappear under a matched evaluation. The row "Ours 35 cases CT 72.26 (val)" also mixes a validation score with a test score for the nnU-Net baseline and should be clarified.
  3. [Table 2 and Section 3.2] The configuration reported as the SegRap SOTA row, "Ours*" with 87.77 mDice, corresponds to "trick+Sigmoid*2+BLS(2)" in Table 2, which does not include the proposed feature fusion. The best ablation row is "trick+FF (inside)" at 88.01, and the full combination including both fusion and BLS(2) gives 87.47. The paper's contributions claim feature fusion improves results, but the headline result omits it. Please specify the exact architecture behind each reported number and justify why the full method is not the best configuration.
  4. [Table 2 (offset adjustment ablation)] The ablation does not isolate the offset-adjustment strategy that the paper emphasizes. There is no row with "trick+Sigmoid*2" alone; the row "trick+Sigmoid*2+BLS" confounds the offset strategy with the BLS decoder, and Strategy 2 (softmax alone) is never evaluated. Consequently, the claim that the softmax-based strategy with a scaling constant is "particularly outstanding" is not supported by a clean comparison.
  5. [Section 3.2 and Table 1] No error bars, confidence intervals, or significance tests are reported for any of the mDice or mIoU values. The claimed SOTA margins are small (0.34% on HaN-Seg and 1.17% on SegRap), and without an estimate of variability it is impossible to tell whether these differences are meaningful, especially given the protocol mismatches.
minor comments (5)
  1. [Section 2.2] The paper repeatedly calls Mask2Former a "deformable-attention-based transformer," but the cited Mask2Former [5] uses masked attention rather than deformable attention. If the implementation is a modified variant, this should be stated explicitly.
  2. [Sections 2.1 and 2.3] The fourth feature map is described in Section 2.1 as "the biggest" but in Section 2.3 as containing "low-resolution information." These descriptions conflict; please clarify the resolution and role of the fused feature map.
  3. [Section 3.1] The SegRap dataset description states there are 120 training cases and 20 test cases, but then says the training, validation, and test sets contain 100, 7, and 10 images. Please reconcile these numbers and state the official challenge split used for the quoted prior results.
  4. [Figure 3] The claim of improvement on mid-sized and small organs is illustrated only for selected classes. A per-organ Dice table would be needed to support the claim quantitatively.
  5. [Table 2] The text says the softmax strategy with a constant greater than 1 is "particularly outstanding," but Table 2 does not include a direct comparison among Strategy 1, Strategy 2, and Strategy 3 at matched settings.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: all reported scores are measured evaluations, not predictions derived from the paper's own fitted parameters.

full rationale

The paper is an empirical engineering study. The mDice and mIoU values in Tables 1 and 2 are obtained by training and evaluating Mask2Former variants on HaN-Seg and SegRap2023; they are direct measurements, not quantities derived from fitted constants or from the paper's own assumptions. The offset-adjustment strategies, feature-fusion positions, and BLS decoder are compared through ablations, and no equation in Section 2 defines the reported performance in terms of the method's own parameters. There are no self-citations, uniqueness theorems, or imported ansatze that carry the load of the central claim. The SegRap footnote ('we just count those not overlapped regions of different organs') and the HaN-Seg comparison of 42 vs 56 cases are potential threats to the fairness and validity of the SOTA comparison, but they are not circularity: the numbers are still measured under a stated, if mismatched, protocol, and the paper does not predict them from its inputs. Hyperparameter choices such as the scaling constant 2 were selected on the same validation data, which is a mild tuning concern, but no reported result is forced to equal a fitted parameter by construction. Therefore no circular step is identified.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper relies on several hand-chosen hyperparameters and domain assumptions. The most consequential is the comparability of evaluation protocols, which the paper does not verify. No new physical or conceptual entities are introduced.

free parameters (5)
  • Offset threshold (Strategy 1) = not reported
    Offsets larger than the threshold are divided by a constant; the threshold value is chosen by hand and not disclosed.
  • Scaling constant for Strategy 3 = 2
    The final configuration uses Sigmoid*2, i.e., softmax-scaled offsets multiplied by 2; this constant is selected from ablation results.
  • Fusion position = late fusion
    Three fusion positions are tested; late fusion is used in the final model and is chosen based on validation performance.
  • Number of BLS auxiliary heads = 2
    Ablation shows BLS(2) improves mDice to 87.77; the choice of two heads is empirical.
  • Preprocessing scaling factors = 2x and 0.5x
    The preprocessing trick stacks the original image with a 2x-enlarged and a 0.5x-reduced copy; the scaling factors are chosen ad hoc.
assumptions (4)
  • domain assumption Sampling points in deformable attention can be constrained to nearby positions without degrading learning
    The offset adjustment assumes organs are compact and sampling nearby is beneficial; no theoretical or empirical proof is provided that this holds across all organs.
  • domain assumption The fourth feature map of Mask2Former provides useful coarse location information
    The paper assumes this map contains location cues useful for segmentation; no analysis is provided.
  • domain assumption Adding an FCN head with Dice loss accelerates training and does not harm final performance
    The auxiliary head is standard practice, but the paper does not isolate its effect from other changes.
  • domain assumption Reported mDice/mIoU numbers from prior papers are directly comparable to the paper's numbers
    The SOTA claim rests on comparing numbers from different evaluation setups (case counts, overlap handling, modality use).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Query Nearby: Offset-Adjusted Mask2Former enhances small-organ segmentation." pith.science (2026). https://pith.science/paper/ASICWMES

@misc{pith2026250605897,
  author       = {Pith},
  title        = {Pith review of: Query Nearby: Offset-Adjusted Mask2Former enhances small-organ segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ASICWMES}},
  note         = {Machine review of arXiv:2506.05897}
}
read the original abstract

Medical segmentation plays an important role in clinical applications like radiation therapy and surgical guidance, but acquiring clinically acceptable results is difficult. In recent years, progress has been witnessed with the success of utilizing transformer-like models, such as combining the attention mechanism with CNN. In particular, transformer-based segmentation models can extract global information more effectively, compensating for the drawbacks of CNN modules that focus on local features. However, utilizing transformer architecture is not easy, because training transformer-based models can be resource-demanding. Moreover, due to the distinct characteristics in the medical field, especially when encountering mid-sized and small organs with compact regions, their results often seem unsatisfactory. For example, using ViT to segment medical images directly only gives a DSC of less than 50\%, which is far lower than the clinically acceptable score of 80\%. In this paper, we used Mask2Former with deformable attention to reduce computation and proposed offset adjustment strategies to encourage sampling points within the same organs during attention weights computation, thereby integrating compact foreground information better. Additionally, we utilized the 4th feature map in Mask2Former to provide a coarse location of organs, and employed an FCN-based auxiliary head to help train Mask2Former more quickly using Dice loss. We show that our model achieves SOTA (State-of-the-Art) performance on the HaNSeg and SegRap2023 datasets, especially on mid-sized and small organs.Our code is available at link https://github.com/earis/Offsetadjustment\_Background-location\_Decoder\_Mask2former.

Figures

Figures reproduced from arXiv: 2506.05897 by the authors.

Figure 1
Figure 1. Results on Different Models for the HaN-Seg Dataset and Visualization of Selected Results. 2 Methods 2.1 Overview An overview of our framework is presented in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of Our Offset-Adjusted Mask2Former with Feature Fusion, Background-Location Sensitive Auxiliary Branch, and Three Offset Adjustment Strate￾gies. 2.2 Offset Adjustment Methods for Middle and Small Organ Segmentation Improvement As mentioned above, Mask2Former [5] utilizes a simple linear layer to compute the offset for each input point. The offset is used to acquire candidate points to be queried by the orig… view at source ↗
Figure 3
Figure 3. Comparison in some mid-sized and small organ classes with nn-UNet baseline and previous SOTA [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative results of some re-implemented models on SegRap2023. to the naïve version, our offset adjustment strategy, feature fusion, and BLS Decoder all help improve the results. The data perprocessing trick and offset adjustment strategy with BLS helps to increase t…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 11 canonical work pages

  1. [1]

    ArXivabs/2005.12872 (2020)

    Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End- to-end object detection with transformers. ArXivabs/2005.12872 (2020)

  2. [2]

    https://doi.org/10.48550/arXiv.2102.04306

    Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A., Zhou, Y.: TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation (2021). https://doi.org/10.48550/arXiv.2102.04306

  3. [3]

    IEEE Transactions on Pattern Analysis and Machine Intelli- gence 40(4), 834–848 (2018).https://doi.org/10.1109/TPAMI.2017.2699184

    Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Se- mantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelli- gence 40(4), 834–848 (2018).https://doi.org/10.1109/TPAMI.2017.2699184

  4. [4]

    Radiotherapy and Oncology160, 175–184 (2021)

    Chen, X., Sun, S., Bai, N., Han, K., Liu, Q., Yao, S., Tang, H., Zhang, C., Lu, Z., Huang, Q., Zhao, G., Xu, Y., Chen, T., Xie, X., Liu, Y.: A deep learning-based auto-segmentation system for organs-at-risk on whole-body computed tomography images for radiation therapy. Radiotherapy and Oncology160, 175–184 (2021). https://doi.org/https://doi.org/10.1016/...

  5. [5]

    In: 2022 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR)

    Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: 2022 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR). pp. 1280–1289. https://doi.org/10.1109/CVPR52688.2022.00135

  6. [6]

    In: Neural Information Processing Systems

    Cheng, B., Schwing, A.G., Kirillov, A.: Per-pixel classification is not all you need for semantic segmentation. In: Neural Information Processing Systems

  7. [7]

    https://github.com/open-mmlab/mmsegmentation (2020)

    Contributors, M.: MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https://github.com/open-mmlab/mmsegmentation (2020)

  8. [8]

    2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) pp

    Hatamizadeh, A., Yang, D., Roth, H.R., Xu, D.: Unetr: Transformers for 3d med- ical image segmentation. 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) pp. 1748–1758 (2021)

Show all 24 references
  1. [9]

    ArXivabs/2412.08345 (2024), https://api.semanticscholar.org/CorpusID:274638646

    Lei, M., Wu, H., Lv, X., Wang, X.: Condseg: A general medical image segmenta- tion framework via contrast-driven feature enhancement. ArXivabs/2412.08345 (2024), https://api.semanticscholar.org/CorpusID:274638646

  2. [10]

    International Journal of Radiation Oncology*Biology*Physics 113(4), 893–902 (2022)

    Liao, W., He, J., Luo, X., Wu, M., Shen, Y., Li, C., Xiao, J., Wang, G., Chen, N.: Automatic delineation of gross tumor volume based on magnetic resonance imaging by performing a novel semisupervised learning framework in nasopharyngeal carcinoma. International Journal of Radi...

  3. [11]

    In: 2020 IEEE17thInternationalSymposiumonBiomedicalImaging(ISBI).pp.1522–1525

    Liu, Z., Wang, H., Lei, W., Wang, G.: Csaf-cnn: Cross-layer spatial attention map fusion network for organ-at-risk segmentation in head and neck ct images. In: 2020 IEEE17thInternationalSymposiumonBiomedicalImaging(ISBI).pp.1522–1525. https://doi.org/10.1109/ISBI45749.2020.909...

  4. [12]

    Medical Image Analysis 101, 103447 (2025)

    Luo, X., Fu, J., Zhong, Y., Liu, S., Han, B., Astaraki, M., Bendazzoli, S., Toma-Dasu, I., Ye, Y., Chen, Z., Xia, Y., Su, Y., Ye, J., He, J., Xing, Z., Wang, H., Zhu, L., Yang, K., Fang, X., Wang, Z., Lee, C.W., Park, S.J., Chun, J., Ulrich, C., Maier-Hein, K.H., Ndipenoch, N....

  5. [13]

    Medical Physics 50(3), 1917–1927 (2023)

    Podobnik, G., Strojan, P., Peterlin, P., Ibragimov, B., Vrtovec, T.: Han- seg: The head and neck organ-at-risk ct and mr segmentation dataset. Medical Physics 50(3), 1917–1927 (2023). https://doi.org/https://doi. org/10.1002/mp.16197, https://aapm.onlinelibrary.wiley.com/doi/a...

  6. [14]

    ArXivabs/1505.04597 (2015)

    Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. ArXivabs/1505.04597 (2015)

  7. [15]

    In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV)

    Takikawa, T., Acuna, D., Jampani, V., Fidler, S.: Gated-scnn: Gated shape cnns for semantic segmentation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 5228–5237.https://doi.org/10.1109/ICCV.2019. 00533

  8. [16]

    Nature Machine Intelligence 1(10), 480–491 (2019)

    Tang, H., Chen, X., Liu, Y., Lu, Z., You, J., Yang, M., Yao, S., Zhao, G., Xu, Y., Chen, T., Liu, Y., Xie, X.: Clinically applicable deep learning frame- work for organs at risk delineation in ct images. Nature Machine Intelligence 1(10), 480–491 (2019). https://doi.org/10.103...

  9. [17]

    In: Neural Information Pro- cessing Systems

    Vaswani, A., Shazeer, N.M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Neural Information Pro- cessing Systems

  10. [18]

    ArXivabs/1903.11816 (2019), https://api.semanticscholar.org/CorpusID:85542864

    Wu, H., Zhang, J., Huang, K., Liang, K., Yu, Y.: Fastfcn: Rethinking dilated convolution in the backbone for semantic segmentation. ArXivabs/1903.11816 (2019), https://api.semanticscholar.org/CorpusID:85542864

  11. [19]

    In: Neural Information Processing Systems

    Xie, E., Wang, W., Yu, Z., Anandkumar, A., Álvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. In: Neural Information Processing Systems

  12. [20]

    Yan, K., Yin, X., Xia, Y., Wang, F., Wang, S., Gao, Y., Yao, J., Li, C., Bai, X., Zhou, J., Zhang, L., Lu, L., Shi, Y.: Liver Tumor Screening and Diagno- sis in CT with Pixel-Lesion-Patient Network (2023).https://doi.org/10.48550/ arXiv.2307.08268

  13. [21]

    Asian journal of surgery (2024)

    Yuan, Y., Hou, S., Wu, X., Wang, Y., Sun, Y., Yang, Z., Yin, S., Zhang, F.: Application of deep-learning to the automatic segmentation and classification of lateral lymph nodes on ultrasound images of papillary thyroid carcinoma. Asian journal of surgery (2024)

  14. [22]

    2022 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR) pp

    Zhang, G., Luo, Z., Yu, Y., Cui, K., Lu, S.: Accelerating detr convergence via semantic-aligned matching. 2022 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR) pp. 939–948 (2022), https://api. semanticscholar.org/CorpusID:247446935

  15. [23]

    In: 2024 IEEE International Symposium on Biomedical Imaging (ISBI)

    Zhang, Z., Qi, X., Zhang, B., Wu, B., Le, H., Jeong, B., Liao, Z., Liu, Y., Verjans, J., To, M.S., Hartley, R.: Segreg: Segmenting oars by registering mr images and ct annotations. In: 2024 IEEE International Symposium on Biomedical Imaging (ISBI). pp. 1–5.https://doi.org/10.1...

  16. [24]

    ArXivabs/2010.04159 (2020)

    Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable detr: Deformable transformers for end-to-end object detection. ArXivabs/2010.04159 (2020)

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.