REVIEW 5 major objections 5 minor 24 references
Query Nearby: Offset-Adjusted Mask2Former enhances small-organ segmentation
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Offset-adjusted Mask2Former reaches state-of-the-art scores on small-organ segmentation.
desk verdict The offset-adjustment idea is worth a look, but the SOTA claim does not survive a matched-protocol check: two datasets, two evaluation mismatches. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is offset control inside deformable attention, the sampling scheme introduced by Deformable DETR and reused by Mask2Former. In that scheme each query predicts an offset and attention is computed over points at the query position plus the offset; the paper modifies this computation so the offsets are pulled toward the query. Strategy 1 divides any offset above a fixed threshold by a constant, strategy 2 applies softmax to contract all offsets, and strategy 3 applies softmax and then multiplies by a constant greater than 1, which the experiments identify as most effective. Two supporting mechanisms are the fusion of the fourth feature map, used as a coarse memory that provides organ location and margin information, and an FCN-based background-location sensitive decoder that gives a quick coarse segmentation against the background. Together they let the encoder spend attention on foreground and accelerate training.
What would settle it
Re-run the two comparisons under matched protocols: on SegRap2023 use the official overlap-aware metric on the full test set, and on HaN-Seg evaluate on the same 56-case CT/MR_T1 protocol used by SegReg. If the resulting mDice values are no longer above 86.70 and 81.26 respectively, the claimed state-of-the-art margins are an artifact of protocol differences.
Extended reading notes
Core claim
The authors' central claim is that transformer-based segmentation fails on small organs mainly because deformable attention lets sampling points drift too far from the query, wasting computation on background. Their remedy is to control the offset so sampled points stay within the same compact organ, tested with three strategies: a manual threshold that shrinks large offsets, a softmax contraction of offsets, and a softmax contraction followed by scaling by a constant greater than one, the last proving most effective. They further claim that reusing the fourth feature map as coarse location guidance and adding a background-location-sensitive FCN decoder both improve accuracy and training speed. With all components, their model reaches a mean Dice of 81.60 on HaN-Seg and 87.77 on SegRap2023, reported as state of the art, especially for mid-sized and small organs.
Load-bearing premise
The reported state-of-the-art margins depend on comparing scores that were not computed under exactly the same protocol: on SegRap2023 the authors count only non-overlapping organ regions, and on HaN-Seg they evaluate on 42 cases while the previous SOTA used 56; if those differences are removed, the margins could shrink or disappear.
Editorial extensions
If this is right
- Since the offset-adjustment operates only on the deformable-attention computation, any transformer that uses that sampling mechanism can adopt it without changing the backbone.
- The fourth feature map already exists in Mask2Former, so fusing it adds no new feature-extraction cost while giving the model coarse organ-location information.
- The background-location sensitive decoder gives a coarse foreground/background segmentation that helps training converge faster, so the approach is promising for low-resource training settings.
- If the reported margins hold, average Dice above 80 percent on both benchmarks means the method reaches clinically acceptable territory for head-and-neck organs while using only 2D CT slices.
- The largest per-organ gains are on small structures such as cochlea and optic nerve, while symmetric or irregular organs like the lacrimal gland remain unstable.
Reading between the lines
- The same offset-control principle should transfer to deformable-attention models outside segmentation, such as small-object detection in aerial or pathology imagery, where foreground is sparse and compact.
- If a matched-protocol evaluation confirms the reported margins, the practical implication is that registration-based multi-modal preprocessing can be replaced by a cheaper attention-level change plus a localization head, lowering the compute barrier for clinical deployment.
- A natural extension would be to apply the offset adjustment within 3D deformable attention, since the current experiments use 2D slices and the paper's own framing suggests the gains should persist in volumetric sampling.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes three modifications to Mask2Former for small-organ segmentation: offset-adjustment strategies for deformable attention, fusion of the fourth feature map, and an FCN-based background-location sensitive auxiliary decoder. The authors report mDice 81.60 on HaN-Seg and 87.77 on SegRap2023, claiming state-of-the-art performance, and release code. The internal ablation on SegRap shows gains over a naive Mask2Former baseline, but the external SOTA comparisons use incompatible evaluation protocols.
Significance. If the results are confirmed under matched evaluation, the proposed modifications are simple and could be useful for small-organ segmentation, particularly the offset-adjustment idea and the auxiliary decoder. The paper ships code, which supports reproducibility, and the ablation demonstrates internal improvements on the SegRap dataset. However, the headline SOTA claim is currently unsupported because the comparison to prior work is not apples-to-apples, and the ablation contains an internal inconsistency about which configuration constitutes the proposed method.
major comments (5)
- [Table 1 (SegRap2023 footnote)] The SegRap2023 comparison is not valid under the official challenge metric. The footnote states "we just count those not overlapped regions of different organs," whereas the quoted SOTA (Y. Zhong et al., 86.70) is the official overlap-aware result. Excluding overlapping voxels changes the denominator of the Dice score and can inflate the reported value; the 1.17% margin is within the range such a protocol change could produce. Please re-evaluate on the official metric or clearly state that the comparison is against a non-standard baseline.
- [Table 1 (HaN-Seg rows)] The HaN-Seg SOTA comparison is undermined by a test-set mismatch: "Ours" is evaluated on 42 cases, while the SegReg(SOTA) row uses 56 cases. Since 42 is the open subset of the HaN-Seg data, the excluded cases are not random, and the 0.34% mDice margin could disappear under a matched evaluation. The row "Ours 35 cases CT 72.26 (val)" also mixes a validation score with a test score for the nnU-Net baseline and should be clarified.
- [Table 2 and Section 3.2] The configuration reported as the SegRap SOTA row, "Ours*" with 87.77 mDice, corresponds to "trick+Sigmoid*2+BLS(2)" in Table 2, which does not include the proposed feature fusion. The best ablation row is "trick+FF (inside)" at 88.01, and the full combination including both fusion and BLS(2) gives 87.47. The paper's contributions claim feature fusion improves results, but the headline result omits it. Please specify the exact architecture behind each reported number and justify why the full method is not the best configuration.
- [Table 2 (offset adjustment ablation)] The ablation does not isolate the offset-adjustment strategy that the paper emphasizes. There is no row with "trick+Sigmoid*2" alone; the row "trick+Sigmoid*2+BLS" confounds the offset strategy with the BLS decoder, and Strategy 2 (softmax alone) is never evaluated. Consequently, the claim that the softmax-based strategy with a scaling constant is "particularly outstanding" is not supported by a clean comparison.
- [Section 3.2 and Table 1] No error bars, confidence intervals, or significance tests are reported for any of the mDice or mIoU values. The claimed SOTA margins are small (0.34% on HaN-Seg and 1.17% on SegRap), and without an estimate of variability it is impossible to tell whether these differences are meaningful, especially given the protocol mismatches.
minor comments (5)
- [Section 2.2] The paper repeatedly calls Mask2Former a "deformable-attention-based transformer," but the cited Mask2Former [5] uses masked attention rather than deformable attention. If the implementation is a modified variant, this should be stated explicitly.
- [Sections 2.1 and 2.3] The fourth feature map is described in Section 2.1 as "the biggest" but in Section 2.3 as containing "low-resolution information." These descriptions conflict; please clarify the resolution and role of the fused feature map.
- [Section 3.1] The SegRap dataset description states there are 120 training cases and 20 test cases, but then says the training, validation, and test sets contain 100, 7, and 10 images. Please reconcile these numbers and state the official challenge split used for the quoted prior results.
- [Figure 3] The claim of improvement on mid-sized and small organs is illustrated only for selected classes. A per-organ Dice table would be needed to support the claim quantitatively.
- [Table 2] The text says the softmax strategy with a constant greater than 1 is "particularly outstanding," but Table 2 does not include a direct comparison among Strategy 1, Strategy 2, and Strategy 3 at matched settings.
Circularity Check
No circularity found: all reported scores are measured evaluations, not predictions derived from the paper's own fitted parameters.
full rationale
The paper is an empirical engineering study. The mDice and mIoU values in Tables 1 and 2 are obtained by training and evaluating Mask2Former variants on HaN-Seg and SegRap2023; they are direct measurements, not quantities derived from fitted constants or from the paper's own assumptions. The offset-adjustment strategies, feature-fusion positions, and BLS decoder are compared through ablations, and no equation in Section 2 defines the reported performance in terms of the method's own parameters. There are no self-citations, uniqueness theorems, or imported ansatze that carry the load of the central claim. The SegRap footnote ('we just count those not overlapped regions of different organs') and the HaN-Seg comparison of 42 vs 56 cases are potential threats to the fairness and validity of the SOTA comparison, but they are not circularity: the numbers are still measured under a stated, if mismatched, protocol, and the paper does not predict them from its inputs. Hyperparameter choices such as the scaling constant 2 were selected on the same validation data, which is a mild tuning concern, but no reported result is forced to equal a fitted parameter by construction. Therefore no circular step is identified.
Assumptions & free parameters
free parameters (5)
- Offset threshold (Strategy 1) =
not reported
- Scaling constant for Strategy 3 =
2
- Fusion position =
late fusion
- Number of BLS auxiliary heads =
2
- Preprocessing scaling factors =
2x and 0.5x
assumptions (4)
- domain assumption Sampling points in deformable attention can be constrained to nearby positions without degrading learning
- domain assumption The fourth feature map of Mask2Former provides useful coarse location information
- domain assumption Adding an FCN head with Dice loss accelerates training and does not harm final performance
- domain assumption Reported mDice/mIoU numbers from prior papers are directly comparable to the paper's numbers
Cite this review
Pith. "Pith review of Query Nearby: Offset-Adjusted Mask2Former enhances small-organ segmentation." pith.science (2026). https://pith.science/paper/ASICWMES
@misc{pith2026250605897,
author = {Pith},
title = {Pith review of: Query Nearby: Offset-Adjusted Mask2Former enhances small-organ segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/ASICWMES}},
note = {Machine review of arXiv:2506.05897}
}
read the original abstract
Medical segmentation plays an important role in clinical applications like radiation therapy and surgical guidance, but acquiring clinically acceptable results is difficult. In recent years, progress has been witnessed with the success of utilizing transformer-like models, such as combining the attention mechanism with CNN. In particular, transformer-based segmentation models can extract global information more effectively, compensating for the drawbacks of CNN modules that focus on local features. However, utilizing transformer architecture is not easy, because training transformer-based models can be resource-demanding. Moreover, due to the distinct characteristics in the medical field, especially when encountering mid-sized and small organs with compact regions, their results often seem unsatisfactory. For example, using ViT to segment medical images directly only gives a DSC of less than 50\%, which is far lower than the clinically acceptable score of 80\%. In this paper, we used Mask2Former with deformable attention to reduce computation and proposed offset adjustment strategies to encourage sampling points within the same organs during attention weights computation, thereby integrating compact foreground information better. Additionally, we utilized the 4th feature map in Mask2Former to provide a coarse location of organs, and employed an FCN-based auxiliary head to help train Mask2Former more quickly using Dice loss. We show that our model achieves SOTA (State-of-the-Art) performance on the HaNSeg and SegRap2023 datasets, especially on mid-sized and small organs.Our code is available at link https://github.com/earis/Offsetadjustment\_Background-location\_Decoder\_Mask2former.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End- to-end object detection with transformers. ArXivabs/2005.12872 (2020)
arXiv 2020
-
[2]
https://doi.org/10.48550/arXiv.2102.04306
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A., Zhou, Y.: TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation (2021). https://doi.org/10.48550/arXiv.2102.04306
-
[3]
Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Se- mantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelli- gence 40(4), 834–848 (2018).https://doi.org/10.1109/TPAMI.2017.2699184
arXiv 2018
-
[4]
Radiotherapy and Oncology160, 175–184 (2021)
Chen, X., Sun, S., Bai, N., Han, K., Liu, Q., Yao, S., Tang, H., Zhang, C., Lu, Z., Huang, Q., Zhao, G., Xu, Y., Chen, T., Xie, X., Liu, Y.: A deep learning-based auto-segmentation system for organs-at-risk on whole-body computed tomography images for radiation therapy. Radiotherapy and Oncology160, 175–184 (2021). https://doi.org/https://doi.org/10.1016/...
-
[5]
In: 2022 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR)
Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: 2022 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR). pp. 1280–1289. https://doi.org/10.1109/CVPR52688.2022.00135
arXiv 2022
-
[6]
In: Neural Information Processing Systems
Cheng, B., Schwing, A.G., Kirillov, A.: Per-pixel classification is not all you need for semantic segmentation. In: Neural Information Processing Systems
-
[7]
https://github.com/open-mmlab/mmsegmentation (2020)
Contributors, M.: MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https://github.com/open-mmlab/mmsegmentation (2020)
2020
-
[8]
2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) pp
Hatamizadeh, A., Yang, D., Roth, H.R., Xu, D.: Unetr: Transformers for 3d med- ical image segmentation. 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) pp. 1748–1758 (2021)
work page 2021
Show all 24 references
-
[9]
ArXivabs/2412.08345 (2024), https://api.semanticscholar.org/CorpusID:274638646
Lei, M., Wu, H., Lv, X., Wang, X.: Condseg: A general medical image segmenta- tion framework via contrast-driven feature enhancement. ArXivabs/2412.08345 (2024), https://api.semanticscholar.org/CorpusID:274638646
2024 arXiv
-
[10]
International Journal of Radiation Oncology*Biology*Physics 113(4), 893–902 (2022)
Liao, W., He, J., Luo, X., Wu, M., Shen, Y., Li, C., Xiao, J., Wang, G., Chen, N.: Automatic delineation of gross tumor volume based on magnetic resonance imaging by performing a novel semisupervised learning framework in nasopharyngeal carcinoma. International Journal of Radi...
2022 doi
-
[11]
In: 2020 IEEE17thInternationalSymposiumonBiomedicalImaging(ISBI).pp.1522–1525
Liu, Z., Wang, H., Lei, W., Wang, G.: Csaf-cnn: Cross-layer spatial attention map fusion network for organ-at-risk segmentation in head and neck ct images. In: 2020 IEEE17thInternationalSymposiumonBiomedicalImaging(ISBI).pp.1522–1525. https://doi.org/10.1109/ISBI45749.2020.909...
2020
-
[12]
Medical Image Analysis 101, 103447 (2025)
Luo, X., Fu, J., Zhong, Y., Liu, S., Han, B., Astaraki, M., Bendazzoli, S., Toma-Dasu, I., Ye, Y., Chen, Z., Xia, Y., Su, Y., Ye, J., He, J., Xing, Z., Wang, H., Zhu, L., Yang, K., Fang, X., Wang, Z., Lee, C.W., Park, S.J., Chun, J., Ulrich, C., Maier-Hein, K.H., Ndipenoch, N....
2025
-
[13]
Medical Physics 50(3), 1917–1927 (2023)
Podobnik, G., Strojan, P., Peterlin, P., Ibragimov, B., Vrtovec, T.: Han- seg: The head and neck organ-at-risk ct and mr segmentation dataset. Medical Physics 50(3), 1917–1927 (2023). https://doi.org/https://doi. org/10.1002/mp.16197, https://aapm.onlinelibrary.wiley.com/doi/a...
2023 doi
-
[14]
ArXivabs/1505.04597 (2015)
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. ArXivabs/1505.04597 (2015)
2015 arXiv
-
[15]
In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
Takikawa, T., Acuna, D., Jampani, V., Fidler, S.: Gated-scnn: Gated shape cnns for semantic segmentation. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 5228–5237.https://doi.org/10.1109/ICCV.2019. 00533
2019 doi
-
[16]
Nature Machine Intelligence 1(10), 480–491 (2019)
Tang, H., Chen, X., Liu, Y., Lu, Z., You, J., Yang, M., Yao, S., Zhao, G., Xu, Y., Chen, T., Liu, Y., Xie, X.: Clinically applicable deep learning frame- work for organs at risk delineation in ct images. Nature Machine Intelligence 1(10), 480–491 (2019). https://doi.org/10.103...
2019 doi
-
[17]
In: Neural Information Pro- cessing Systems
Vaswani, A., Shazeer, N.M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Neural Information Pro- cessing Systems
-
[18]
ArXivabs/1903.11816 (2019), https://api.semanticscholar.org/CorpusID:85542864
Wu, H., Zhang, J., Huang, K., Liang, K., Yu, Y.: Fastfcn: Rethinking dilated convolution in the backbone for semantic segmentation. ArXivabs/1903.11816 (2019), https://api.semanticscholar.org/CorpusID:85542864
2019 arXiv
-
[19]
In: Neural Information Processing Systems
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Álvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. In: Neural Information Processing Systems
- [20]
-
[21]
Asian journal of surgery (2024)
Yuan, Y., Hou, S., Wu, X., Wang, Y., Sun, Y., Yang, Z., Yin, S., Zhang, F.: Application of deep-learning to the automatic segmentation and classification of lateral lymph nodes on ultrasound images of papillary thyroid carcinoma. Asian journal of surgery (2024)
2024
-
[22]
2022 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR) pp
Zhang, G., Luo, Z., Yu, Y., Cui, K., Lu, S.: Accelerating detr convergence via semantic-aligned matching. 2022 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR) pp. 939–948 (2022), https://api. semanticscholar.org/CorpusID:247446935
2022
-
[23]
In: 2024 IEEE International Symposium on Biomedical Imaging (ISBI)
Zhang, Z., Qi, X., Zhang, B., Wu, B., Le, H., Jeong, B., Liao, Z., Liu, Y., Verjans, J., To, M.S., Hartley, R.: Segreg: Segmenting oars by registering mr images and ct annotations. In: 2024 IEEE International Symposium on Biomedical Imaging (ISBI). pp. 1–5.https://doi.org/10.1...
2024
-
[24]
ArXivabs/2010.04159 (2020)
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable detr: Deformable transformers for end-to-end object detection. ArXivabs/2010.04159 (2020)
2020 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.