REVIEW 3 major objections 1 cited by
A compact student can match or beat a large 3D medical segmenter by copying latent displacement trajectories instead of voxel activations.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-11 16:38 UTC pith:IWEU4HFU
load-bearing objection Solid engineering KD for 3D medical seg: vector displacements of ROI-pooled embeddings beat standard baselines and can edge a MedNeXt teacher on AMOS, but the GT-mask training dependency and internal-only splits keep the headline claim from being pure relational transfer. the 3 major comments →
Displacement Preserving Relational Distillation for Robust Medical Segmentation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Displacement-Preserving Relational Distillation (DPRD) shows that matching batch-wise, scale-normalized displacement vectors of ROI-masked case embeddings across encoder stages transfers 3D anatomical structure more effectively than logit, feature, or scalar-relational distillation, enabling a compact student to equal or surpass a much larger frozen teacher on standard medical segmentation benchmarks.
What carries the argument
Displacement-Preserving Relational Alignment (DPRA): after ROI-aware density-normalized pooling of multi-stage encoder features into case embeddings, all-pairs displacement vectors are computed, divided by the batch-average L2 norm, and matched with Smooth-L1 trajectory and relative-distance losses.
Load-bearing premise
The method assumes dense ground-truth organ or lesion masks are available at training time so that ROI pooling can exclude background voxels; without those labels the relational signal may again be diluted.
What would settle it
Retrain the identical MobileUNetV3 student on AMOS 2022 using only teacher-predicted or randomly thresholded masks instead of ground-truth ROIs; if Dice falls back near the non-ROI baseline (~81.7 %) and HD95 rises above 15 mm, the load-bearing role of true anatomical masks is confirmed.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Displacement-Preserving Relational Distillation (DPRD) for compressing 3D medical segmenters. DPRD combines ROI-Aware Feature Masking (RAFM), which density-normalizes encoder features with ground-truth foreground masks (Eqs. 6–7), and Displacement-Preserving Relational Alignment (DPRA), which matches batch-wise pairwise displacement vectors after scale normalization (Eqs. 1–5) across multi-stage encoder embeddings. Integrated into nnU-Net with a frozen MedNeXt teacher, the method is evaluated on ISLES 2022 and AMOS 2022 against Logits KD, FitNet, RKD, and CIRKD. On AMOS, a MobileUNetV3 student (0.63 M params, 39 G FLOPs) reaches 85.46% Dice / 11.72 mm HD95, slightly above the teacher (85.37% / 14.52 mm) while using ~5% of the parameters and ~3% of the FLOPs; ablations (Table 3) and qualitative figures support the contributions of the components.
Significance. If the gains hold under fairer conditions, DPRD would be a practical contribution to resource-efficient 3D medical segmentation: it targets the well-known problems of architectural heterogeneity, background dilution, and scalar collapse in relational KD, and reports consistent improvements on two public benchmarks with standard metrics, multi-stage design, and released code. The student-beats-teacher margin on AMOS boundary metrics and the clear ablation isolation of RAFM / trajectory / distance are useful empirical signals for the community. The work is incremental rather than foundational, but the combination of vector displacement matching with ROI-masked case-level embeddings is a concrete, implementable design that could be adopted in nnU-Net-style pipelines for clinical deployment under compute constraints.
major comments (3)
- Abstract and §3.1 / Table 2 claim that DPRD “edges out” the MedNeXt teacher on AMOS (85.46% vs 85.37% Dice; 11.72 vs 14.52 mm HD95). Table 3 shows that removing RAFM drops Dice to 81.68% and raises HD95 to 21.65 mm—well below the teacher—so the headline margin is largely driven by privileged GT-mask supervision (Eq. 7, Sec. 2.2) rather than by displacement-vector alignment alone. The paper itself notes this limitation in the Conclusion but does not quantify performance with teacher-predicted or unsupervised masks. Either reframe the claim as “GT-guided relational distillation” or add an experiment that replaces GT masks with teacher predictions / soft masks so the student-beats-teacher statement is supported under realistic training conditions.
- All reported numbers (Tables 1–3) use the official internal validation splits of ISLES 2022 and AMOS 2022 with no external, multi-center, or held-out test set. Given the paper’s emphasis on robustness to anatomical variability and cross-scanner drift (Introduction), the absence of an external evaluation leaves open whether the gains generalize beyond the training distribution. At minimum, a multi-fold or leave-center-out protocol on AMOS, or an external cohort, is needed to substantiate the robustness claim that underpins the clinical-deployment narrative.
- Batch size is restricted to 2–4 (Implementation Details) because of 3D memory, so the O(N²) pairwise displacements (Eq. 1) are estimated from very small samples. The paper asserts that stochastic accumulation across patches, stages, and epochs is sufficient, but provides no sensitivity analysis on batch size or number of pairs. A short experiment (e.g., gradient accumulation to effective N=8, or random pair sampling) would strengthen the claim that the relational geometry is reliably estimated.
Circularity Check
No load-bearing circularity: empirical student performance on public Dice/HD95/NSD is independent of the proposed displacement losses; only minor non-essential author self-citations appear in related-work lists.
specific steps
-
self citation load bearing
[Section 1, paragraph on relational KD priors]
"Prior work has shown that distilling inter-sample relations (e.g., pairwise similarities or distances) can provide a softer, architecture-agnostic signal than pointwise feature matching [21,28,27,16,1,12]."
References [1] and [12] are prior papers by overlapping authors (Ding/Chu/Tian and Lan/Tian). They supply background context only; they are not used to prove uniqueness of the displacement formulation or to derive the AMOS/ISLES numbers, so the circularity is minor and non-load-bearing.
full rationale
The paper defines a concrete training objective (ROI-masked pairwise displacement vectors with batch-scale normalization and Smooth-L1 trajectory/distance terms, Eqs. 1–8) and then measures the resulting student segmenters on the official ISLES 2022 and AMOS 2022 validation partitions with standard, externally defined metrics (Dice, HD95, NSD). Those metrics are not algebraic rearrangements of L_traj or L_dist; the ablation table shows each term contributes non-trivially, and the student-beats-teacher margin is an observed numerical outcome rather than a quantity forced by construction. Hyper-parameters (λ_traj=1, λ_dist=0.5, stage weights) are fixed a priori, not fitted to the reported scores. Self-citations ([1], [11], [12]) merely list related KD papers by overlapping authors and are never invoked as uniqueness theorems or as the sole justification for the performance claims. Consequently the derivation chain from method definition through training to tabulated results is self-contained and externally falsifiable; the only residual is ordinary, non-load-bearing self-citation of the kind that scores 1–2.
Axiom & Free-Parameter Ledger
free parameters (4)
- stage-wise weights ω_l =
{0.05,0.1,0.1,0.15,0.25,0.35}
- λ_traj, λ_dist =
1.0 and 0.5
- projection dimension =
128
- batch size =
2 or 4
axioms (4)
- ad hoc to paper Teacher-induced pairwise displacement vectors in ROI-pooled embedding space encode transferable anatomical structure that a heterogeneous student can usefully imitate.
- domain assumption Ground-truth foreground union is a valid and sufficient ROI for distillation masking during training.
- ad hoc to paper Batch-wise L2-norm normalization of displacements removes activation-scale mismatch without destroying relational geometry.
- domain assumption nnU-Net Dice+CE loss plus the relational terms yields a well-behaved multi-task optimum under SGD.
invented entities (2)
-
Displacement-Preserving Relational Alignment (DPRA) objective
no independent evidence
-
ROI-Aware Feature Masking (RAFM) with density-normalized pooling
no independent evidence
read the original abstract
Accurate 3D medical segmentation is limited by anatomical variability and high computational costs. While knowledge distillation (KD) offers a route for model compression, conventional methods often fail to preserve complex structures and are overwhelmed by background noise. We propose Displacement-Preserving Relational Distillation (DPRD), which distills latent anatomical trajectories via vector based alignment to preserve the orientation and relative scale of the teacher's manifold, and prevents signal dilution by anchoring distillation in task-relevant structures. Integrated into nnU-Net, DPRD outperforms established baselines on ISLES 2022 and AMOS 2022 benchmarks. Notably, on the AMOS dataset, DPRD achieves a Dice score of 85.46%, edging out the high-capacity MedNeXt teacher while significantly reducing boundary errors. Despite utilizing only ~5% of the teacher's parameters and ~3% of its FLOPs, our approach maintains high structural consistency. This provides a robust, efficient solution for deploying high performance segmenters in resource-constrained clinical environments. Code: https://github.com/ClinicaAlpha/DPRD-3D-MedSeg
Figures
Forward citations
Cited by 1 Pith paper
-
MIS-HCC: Hierarchical Channel Clustering for Efficient Medical Image Segmentation
MIS-HCC prunes medical segmentation networks by Wasserstein-based hierarchical clustering of channels followed by parameter averaging, reporting near-baseline accuracy at 87.5% pruning.
Reference graph
Works this paper leans on
-
[1]
In: Pro- ceedings of the 2026 IEEE Intelligent Vehicles Symposium (IV) (2026)
Ding, Z., Chu, X., Tian, Q.: Difference feature map distillation: Transferring inter- sample relational knowledge towards efficient transformer-based tracking. In: Pro- ceedings of the 2026 IEEE Intelligent Vehicles Symposium (IV) (2026)
2026
-
[2]
IEEE transactions on medical imaging39(7), 2415–2425 (2020)
Dou, Q., Liu, Q., Heng, P.A., Glocker, B.: Unpaired multi-modal segmentation via knowledge distillation. IEEE transactions on medical imaging39(7), 2415–2425 (2020)
2020
-
[3]
IEEE Transactions on Biomedical Engineering69(3), 1173–1185 (2021)
Guan, H., Liu, M.: Domain adaptation for medical image analysis: a survey. IEEE Transactions on Biomedical Engineering69(3), 1173–1185 (2021)
2021
-
[4]
In: International MICCAI brainlesion workshop
Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H.R., Xu, D.: Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In: International MICCAI brainlesion workshop. pp. 272–284. Springer (2021)
2021
-
[5]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
He, T., Shen, C., Tian, Z., Gong, D., Sun, C., Yan, Y.: Knowledge adaptation for efficient semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 578–587 (2019)
2019
-
[6]
Scientific data9(1), 762 (2022) 10 Z
Hernandez Petzsche, M.R., De La Rosa, E., Hanning, U., Wiest, R., Valenzuela, W., Reyes, M., Meyer, M., Liew, S.L., Kofler, F., Ezhov, I., et al.: Isles 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset. Scientific data9(1), 762 (2022) 10 Z. Ding et al
2022
-
[7]
arXiv preprint arXiv:1503.02531 (2015)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
Pith/arXiv arXiv 2015
-
[8]
Nature methods18(2), 203–211 (2021)
Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods18(2), 203–211 (2021)
2021
-
[9]
Advances in neural information processing systems 35, 36722–36732 (2022)
Ji, Y., Bai, H., Ge, C., Yang, J., Zhu, Y., Zhang, R., Li, Z., Zhanng, L., Ma, W., Wan, X., et al.: Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. Advances in neural information processing systems 35, 36722–36732 (2022)
2022
-
[10]
arXiv preprint arXiv:2602.23752 (2026)
Jia, J., Wu, Y., Chen, H., Jing, H., Wang, H., Bu, J., Wu, L.: Unsupervised causal prototypical networks for de-biased interpretable dermoscopy diagnosis. arXiv preprint arXiv:2602.23752 (2026)
arXiv 2026
-
[11]
arXiv preprint arXiv:2601.09191 (2026)
Lan, Q., Choi, A., Ma, J., Wang, B., Zhao, Z., Jiang, X., Hsu, Y.C.: From perfor- mance to practice: Knowledge-distilled segmentator for on-premises clinical work- flows. arXiv preprint arXiv:2601.09191 (2026)
arXiv 2026
-
[12]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision
Lan,Q.,Tian,Q.:Acam-kd:adaptiveandcooperativeattentionmaskingforknowl- edge distillation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3957–3966 (2025)
2025
-
[13]
arXiv preprint arXiv:2209.15076 (2022)
Lee, H.H., Bao, S., Huo, Y., Landman, B.A.: 3d ux-net: A large kernel volumet- ric convnet modernizing hierarchical transformer for medical image segmentation. arXiv preprint arXiv:2209.15076 (2022)
Pith/arXiv arXiv 2022
-
[14]
Authorea Preprints (2026)
Li, Y., Meng, S., Yang, C., Feng, W., Liu, J., An, Z., Wang, Y., Tian, Y.: A comprehensive survey of interaction techniques in 3d scene generation. Authorea Preprints (2026)
2026
-
[15]
Medical image analysis42, 60–88 (2017)
Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., Van Der Laak, J.A., Van Ginneken, B., Sánchez, C.I.: A survey on deep learning in medical image analysis. Medical image analysis42, 60–88 (2017)
2017
-
[16]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Liu, Y., Chen, K., Liu, C., Qin, Z., Luo, Z., Wang, J.: Structured knowledge dis- tillation for semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2604–2613 (2019)
2019
-
[17]
Langenbeck’s archives of surgery398(4), 501–514 (2013)
Mezger, U., Jendrewski, C., Bartels, M.: Navigation in surgery. Langenbeck’s archives of surgery398(4), 501–514 (2013)
2013
-
[18]
In: 2016 fourth international confer- ence on 3D vision (3DV)
Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 2016 fourth international confer- ence on 3D vision (3DV). pp. 565–571. Ieee (2016)
2016
-
[19]
NeuroImage: Clinical22, 101766 (2019)
Nitsch, J., Klein, J., Dammann, P., Wrede, K., Gembruch, O., Moltz, J., Meine, H., Sure, U., Kikinis, R., Miller, D.: Automatic and efficient mri-us segmentations for improving intraoperative image fusion in image-guided neurosurgery. NeuroImage: Clinical22, 101766 (2019)
2019
-
[20]
Journal of Medical Imaging9(5), 052407–052407 (2022)
Noothout, J.M., Lessmann, N., Van Eede, M.C., Van Harten, L.D., Sogancioglu, E., Heslinga, F.G., Veta, M., Van Ginneken, B., Išgum, I.: Knowledge distillation with ensembles of convolutional neural networks for medical image segmentation. Journal of Medical Imaging9(5), 052407–052407 (2022)
2022
-
[21]
In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Park, W., Kim, D., Lu, Y., Cho, M.: Relational knowledge distillation. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3967–3976 (2019)
2019
-
[22]
IEEE Transactions on Medical Imaging40(12), 3820–3831 (2021)
Qin, D., Bu, J.J., Liu, Z., Shen, X., Zhou, S., Gu, J.J., Wang, Z.H., Wu, L., Dai, H.F.: Efficient medical image segmentation based on knowledge distillation. IEEE Transactions on Medical Imaging40(12), 3820–3831 (2021)
2021
-
[23]
Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: Fitnets: Hints for thin deep nets (2015),https://arxiv.org/abs/1412.6550 DPRD for Robust Medical Segmentation 11
Pith/arXiv arXiv 2015
-
[24]
In: International conference on medical image computing and computer-assisted intervention
Roy, S., Koehler, G., Ulrich, C., Baumgartner, M., Petersen, J., Isensee, F., Jaeger, P.F., Maier-Hein, K.H.: Mednext: transformer-driven scaling of convnets for med- ical image segmentation. In: International conference on medical image computing and computer-assisted intervention. pp. 405–415. Springer (2023)
2023
-
[25]
In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)
Shu, C., Liu, Y., Gao, J., Yan, Z., Shen, C.: Channel-wise knowledge distillation for dense prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 5311–5320 (2021)
2021
-
[26]
In: International Workshop on Deep Learning in Medical Image Analysis
Sudre, C.H., Li, W., Vercauteren, T., Ourselin, S., Jorge Cardoso, M.: Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In: International Workshop on Deep Learning in Medical Image Analysis. pp. 240–
-
[27]
arXiv preprint arXiv:1910.10699 (2019)
Tian, Y., Krishnan, D., Isola, P.: Contrastive representation distillation. arXiv preprint arXiv:1910.10699 (2019)
Pith/arXiv arXiv 1910
-
[28]
In: Proceedings of the IEEE/CVF international conference on computer vision
Tung, F., Mori, G.: Similarity-preserving knowledge distillation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1365–1374 (2019)
2019
-
[29]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Yang, C., Zhou, H., An, Z., Jiang, X., Xu, Y., Zhang, Q.: Cross-image rela- tional knowledge distillation for semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 12319–12328 (2022)
2022
-
[30]
In: Proceedings of the 33rd ACM International Conference on Multimedia
Zhu, C., Lin, Y., Shao, J., Lin, J., Wang, Y.: Pathology-aware prototype evolution via llm-driven semantic disambiguation for multicenter diabetic retinopathy diag- nosis. In: Proceedings of the 33rd ACM International Conference on Multimedia. pp. 9196–9205 (2025)
2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.