Pith. sign in

REVIEW 4 major objections 4 minor 24 references

Enjoying Information Dividend: Gaze Track-based Medical Weakly Supervised Segmentation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read This paper claims that the order in which a physician's gaze moves across an image is a learnable supervisory signal that brings weakly supervised segmentation within 0.33 Dice of fully supervised U-Net.

desk verdict A novel reverse-truncation mechanism for gaze-based segmentation, but the evaluation is confounded by a U-Net/nnU-Net backbone inconsistency and test-set-tuned threshold. read the letter →

arxiv 2505.22230 v1 pith:3I6WKR7N submitted 2025-05-28 cs.CV

classification cs.CV
keywords gazesupervisionweaklysupervisedsegmentationmedicalimageeyetrackingtemporalordertrackattentionpolypprostateMRI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that gaze annotations for medical image segmentation carry more information than previous methods use: the order in which a physician looks at regions, not just where and how long, encodes the diagnostic reasoning. To exploit that order, GradTrack builds a gaze track from fixation points and their start timestamps, then truncates the track in reverse to produce several soft attention maps. These maps supervise the decoder at multiple depths, giving the model a progressive, staged view of the diagnostic process. On polyp and prostate MRI datasets, the method reports Dice gains of 3.21 and 2.61 points over the prior gaze-based approach and lands within about a point of fully supervised U-Net.

What carries the argument

The load-bearing object is the reversed, truncated gaze track. Given fixation points with start timestamps, GradTrack constructs the gaze track $T$, then produces $T_{50\%}$, $T_{75\%}$, $T_{100\%}$ by removing the earliest fixations in reverse order. For each truncated track it computes a distance map $D^r(\mathbf{p}) = \min_{t\in T^r}\|\mathbf{p}-t\|_2$ and converts it to a soft attention map $G^r = e^{-D^r/\beta}\cdot \mathbb{I}(e^{-D^r/\beta}>\tau)$. These maps supervise predictions at decoder blocks 2, 4, and 6 through a weighted cross-entropy loss, providing staged prior information so that later decoder stages are guided by fuller gaze context. The Track Attention module fuses the predicted foreground map into the decoder features with a stop-gradient, letting gaze track knowledge shape segmentation without backpropagating through the fusion.

What would settle it

Run GradTrack's training protocol on a public gaze dataset with fixation timestamps shuffled but all other signals unchanged; if Dice does not drop appreciably below the reported 81.01, the temporal-order claim is not doing the work attributed to it. Equivalently, retrain the 'w/o Sequential' variant on NCI-ISBI and check whether the gap generalises beyond Kvasir-SEG.

Watch

Extended reading notes

Core claim

GradTrack's central claim is that the temporal order of gaze fixations is a usable supervisory signal for weakly supervised segmentation, and that reversing that order during training stabilises learning. The Gaze Track Map Generation module converts raw gaze data into a continuous track, then forms three track attention maps by keeping the last 50%, 75%, and 100% of the fixation sequence. A distance-based exponential decay turns each truncated track into a soft attention map that supervises the decoder; the Track Attention modules inject the learned maps back into the feature stream. The paper reports that this scheme outperforms GazeMedSeg by 3.21% Dice on Kvasir-SEG and 2.61% on NCI-ISBI, and comes within 1.11% and 0.33% of a fully supervised U-Net on the two datasets.

Load-bearing premise

The method presumes that every training image has gaze annotations with reliable start timestamps, and that the comparison numbers inherited from GazeMedSeg were produced on the same annotations and data split.

Editorial extensions

If this is right

  • Gaze-based weakly supervised segmentation can close most of the gap to full supervision: on NCI-ISBI the gap to U-Net is 0.33 Dice, so gaze annotation may serve as a practical substitute for pixel-level labels.
  • Multi-level supervision from truncated tracks gives increasing benefit at deeper decoder stages; placing guidance in the encoder hurts, so decoder-only integration is the design to keep.
  • Reverse truncation is intentionally conservative: it biases predictions toward under-activation, trading recall of uncertain regions for precision against noise.
  • The component ablation on Kvasir-SEG shows a cumulative climb from 63.63 Dice with only the under-activated VAM to 81.01 for the full design, so each module adds measurable value.
  • The field-of-view threshold is a sensitive knob: 0.25 works best, small values inject redundant noise and large values starve the model, so the method needs calibration per dataset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same ordering signal might transfer to other sequential human annotations, such as mouse-trace or touch-screen scribbles with timestamps, turning any time-stamped annotation trace into this style of staged supervision.
  • The 0.73 Dice gain of the full design over the non-sequential variant (81.01 vs 80.28) suggests that temporal order contributes only modestly on this benchmark; a larger gain would need datasets where diagnostic scanning order varies more between images.
  • A testable extension would replace the hand-set truncation ratios of 50/75/100% with learned or per-image ratios, letting the model decide how much of a physician's early scanning to trust.
  • If the gaze annotations were released, re-running the exact protocol on a second institution's data would clarify whether the 3.21% and 2.61% gains are tied to the specific annotators or generalise across gaze patterns.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes GradTrack, a gaze-based weakly supervised segmentation framework that augments a U-Net-style decoder with gaze track attention. The GTMG module converts fixation points and start timestamps into three reverse-truncated track attention maps (Eqs. 1-2), and TA modules predict and fuse these maps into decoder features (Eqs. 3-4), with total loss Eq. 5. On Kvasir-SEG and NCI-ISBI, GradTrack reports Dice 81.01/80.25, versus 77.80/77.64 for the inherited GazeMedSeg baseline, and 0.33 points below fully supervised U-Net on NCI-ISBI. The paper claims that temporal order of gaze fixations is the key information dividend.

Significance. If the empirical claims are valid, the paper makes a useful contribution by incorporating temporal order into gaze-based WSSS, and the module design is concrete and well-specified. The ablation shows that each component improves over the under-activation baseline, and the multi-dataset evaluation is a strength. However, the headline comparison is not currently architecture-matched, and the gaze annotations are not described or released in sufficient detail; the contribution is therefore conditional on a controlled re-evaluation.

major comments (4)
  1. [Sec. 2.1 / Sec. 3.2 / Table 1] Section 2.1 states that GradTrack adopts a U-Net model as the backbone, while Section 3.2 states that nnU-Net is the backbone architecture. These are not interchangeable: Table 1 reports fully supervised Dice of 82.12/80.58 for U-Net versus 88.41/82.43 for nnU-Net. Because the GazeMedSeg baseline is inherited from [24] and its backbone is not specified, the reported 3.21/2.61 Dice gains and the '0.33 gap to fully supervised U-Net' could stem from the backbone or training recipe rather than from GTMG and TA. Please reconcile the backbone statements and rerun GazeMedSeg under GradTrack's exact training schedule and backbone.
  2. [Sec. 3.3 / Sec. 3.1] Section 3.3 states that the PointSup, AGMM, Boxinst, BoxTeacher, and GazeMedSeg results are inherited from [24]. Section 3.1 says only that training annotations were 'meticulously generated by annotators,' without specifying the gaze-tracking device, fixation/timestamp format, or whether these annotations match the gaze data used by [24]. Without a shared gaze-annotation protocol and data split, the comparison is not controlled; please provide the gaze data or a precise specification and rerun the inherited baseline under the same conditions.
  3. [Fig. 4(b) / Sec. 3.2] The field-of-view threshold tau=0.25 is chosen on Kvasir-SEG by scanning test-set Dice across tau values (0.05-0.5) and selecting the maximum (81.01 at tau=0.25). Since tau appears directly in Eq. (2) and thresholds the attention maps used as supervision, selecting it on the test set can optimistically bias the reported results. Please use a validation split or report the selection procedure with held-out test evaluation.
  4. [Table 2] The case for reverse truncation is weak as reported. The full model achieves 81.01 +/- 0.66 while the 'w/o Sequential' variant achieves 80.28 +/- 0.02, a 0.73-point difference with overlapping standard deviations; the central claim that temporal order of fixation points is the 'information dividend' would benefit from statistical testing or a more direct comparison isolating temporal order from the GTMG/TA construction.
minor comments (4)
  1. [Eq. (2)] Equation (2) contains the typo 'can be is defined as'; replace with 'is defined as'.
  2. [Abstract / Table 1 / Sec. 3.2] The abstract writes 'nnUNet' while Table 1 and Section 3.2 write 'nnU-Net'; use one consistent notation throughout.
  3. [Sec. 2.4] The phrase 'basic overly under-activation hard label supervision' is confusing; consider 'basic under-activation hard-label supervision' or define what 'overly' modifies.
  4. [Eq. (5)] In Eq. (5), both L50 and L75 are weighted by the same lambda_1; if the authors intend different weights at different decoder depths, this should be stated explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: gaze track maps are constructed directly from gaze annotations, the losses use those maps as supervision, and the reported Dice gains rest on external empirical comparisons rather than on any fitted or self-referential term.

full rationale

The derivation chain is self-contained. The GTMG module builds a gaze track T from fixation points and start timestamps, computes distance maps via Eq. (1), and converts them into attention maps via Eq. (2). The TA module predicts these maps from decoder features, and Eq. (3) supervises those predictions with the precomputed G_r maps. The total loss Eq. (5) is a weighted sum of these gaze-track supervision terms plus an under-activated VAM loss. Nothing in this chain is defined in terms of the test Dice, the evaluation ground truth, or the reported final numbers. The hyperparameters tau, beta, lambda_1, and lambda_2 are hand-set empirical choices, with a sensitivity analysis for tau (Fig. 4b); they are not fitted to the evaluation labels, so the reported improvements are not forced by construction. The comparisons to GazeMedSeg and other baselines are inherited from external prior work [24], not from the current authors, and the backbone inconsistency (U-Net in Sec. 2.1 vs nnU-Net in Sec. 3.2) is a legitimate experimental-comparability concern, but it does not make the central claim reduce to its own inputs. The only self-citation, ref. [21], appears in a list of scribble-based methods and is not load-bearing for any claim. Thus there is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper's central method depends on the availability of timestamped gaze tracks and on the assumption that temporal order carries diagnostic signal. The main free parameters are chosen by hand or by test-set scanning. No new physical or abstract entity is introduced; the 'track attention map' is a derived supervision signal, not an independent invention.

free parameters (5)
  • Field-of-view threshold tau = 0.25
    Selected by scanning values 0.05-0.5 on the test set, Fig. 4b; peak Dice 81.01 used in final model.
  • Decay rate beta = 10 (Kvasir-SEG), 15 (NCI-ISBI)
    Set per dataset in Sec. 3.2, no sensitivity analysis given.
  • Under-activation threshold = 0.7
    Chosen as 'more conservative' than GazeMedSeg's 0.3/0.2; no ablation on this value reported.
  • Balancing coefficients lambda1, lambda2 = 0.5 each
    Set 'empirically' in Sec. 3.2.
  • Reverse truncation ratios = 50%, 75%, 100%
    Hand-chosen design; Table 2 compares to w/o Sequential but not to other ratio sets.
assumptions (4)
  • domain assumption Temporal order of gaze fixations encodes progressive diagnostic reasoning
    Motivation in Sec. 1; if false, the reverse-truncation supervision is arbitrary.
  • domain assumption Gaze annotations for the training images exist, are accurate, and include start timestamps
    Assumed in Sec. 3.1; no gaze dataset release or protocol described.
  • standard math The D-CRF-refined visual attention map from [12] is a sound basis for hard pseudo labels
    Used in Sec. 2.4; external prior accepted from the literature.
  • domain assumption nnU-Net preprocessed at 224x224 retains enough detail for these segmentation tasks
    Implementation detail in Sec. 3.2; resizing may discard resolution information.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enjoying Information Dividend: Gaze Track-based Medical Weakly Supervised Segmentation." pith.science (2026). https://pith.science/paper/3I6WKR7N

@misc{pith2026250522230,
  author       = {Pith},
  title        = {Pith review of: Enjoying Information Dividend: Gaze Track-based Medical Weakly Supervised Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3I6WKR7N}},
  note         = {Machine review of arXiv:2505.22230}
}
read the original abstract

Weakly supervised semantic segmentation (WSSS) in medical imaging struggles with effectively using sparse annotations. One promising direction for WSSS leverages gaze annotations, captured via eye trackers that record regions of interest during diagnostic procedures. However, existing gaze-based methods, such as GazeMedSeg, do not fully exploit the rich information embedded in gaze data. In this paper, we propose GradTrack, a framework that utilizes physicians' gaze track, including fixation points, durations, and temporal order, to enhance WSSS performance. GradTrack comprises two key components: Gaze Track Map Generation and Track Attention, which collaboratively enable progressive feature refinement through multi-level gaze supervision during the decoding process. Experiments on the Kvasir-SEG and NCI-ISBI datasets demonstrate that GradTrack consistently outperforms existing gaze-based methods, achieving Dice score improvements of 3.21\% and 2.61\%, respectively. Moreover, GradTrack significantly narrows the performance gap with fully supervised models such as nnUNet.

Figures

Figures reproduced from arXiv: 2505.22230 by the authors.

Figure 1
Figure 1. Three frameworks of gaze-based WSSS. (a) VAM-based: Directly using VAM as supervision. (b) GazeMedSeg: Applying multi thresholds to VAM to generate over￾and under-activation maps for joint supervision. (c) GradTrack: Using multiple gaze track attention maps to provide stronger information for overly under-activation maps. “Activ.”: Abbreviation of activation. Existing gaze-based methods focus on utilizing gaze annot… view at source ↗
Figure 2
Figure 2. Overview of the proposed GradTrack. (a) The training pipeline of GradTrack, where the red dashed lines denote the supervision information. (b) Track Attention (TA) Module: Composed of three convolutional blocks, it enhances the model’s prior information by truncating the learned GTMG information and fusing it with the main features. (c) Gaze Track Map Generation (GTMG) Module: Generates gaze attention maps by applyi… view at source ↗
Figure 3
Figure 3. Visualization of segmentation results obtained using gaze-based methods, in￾cluding VAM, VAMD-CRF, GazeMedSeg, and GradTrack, on the NCI-ISBI and Kvasir￾SEG datasets [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Ablation study and hyper-parameter τ discussion on the Kvasir-SEG dataset. (a) Analyzing the contribution of each component within our GradTrack. (b) Perfor￾mance of our GradTrack with various values of τ . In [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 17 canonical work pages

  1. [24]

    In: International Conference on Medical Image Computing and Computer-Assisted Intervention

    Zhong, Y., Tang, C., Yang, Y., Qi, R., Zhou, K., Gong, Y., Heng, P.A., Hsiao, J.H., Dou, Q.: Weakly-supervised medical image segmentation with gaze annotations. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 530–540. Springer (2024)

  2. [1]

    Image and Vision Computing144, 104961 (2024)

    Bisogni, C., Nappi, M., Tortora, G., Del Bimbo, A.: Gaze analysis: A survey on its applications. Image and Vision Computing144, 104961 (2024)

  3. [2]

    The Cancer Imaging Archive (2015)

    Bloch, N., Madabhushi, A., Huisman, H., Freymann, J., Kirby, J., Grauer, M., Enquobahrie, A., Jaffe, C., Clarke, L., Farahani, K.: Nci-isbi 2013 challenge: Au- tomated segmentation of prostate structures. The Cancer Imaging Archive (2015). https://doi.org/http://doi.org/10.7937/K9/TCIA.2015.zF0vlOPv

  4. [3]

    IEEE Transactions on Image Processing (2025)

    Chen, J., Huang, W., Zhang, J., Debattista, K., Han, J.: Addressing inconsistent labeling with cross image matching for scribble-based medical image segmentation. IEEE Transactions on Image Processing (2025)

  5. [4]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition

    Cheng, B., Parkhi, O., Kirillov, A.: Pointly-supervised instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition. pp. 2617–2626 (2022)

  6. [5]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Cheng, T., Wang, X., Chen, S., Zhang, Q., Liu, W.: Boxteacher: Exploring high- quality pseudo labels for weakly supervised instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3145–3154 (2023)

  7. [6]

    IEEE Transactions on Pattern Analysis and Machine Intelligence46(1), 61–84 (2023)

    Ghosh, S., Dhall, A., Hayat, M., Knibbe, J., Ji, Q.: Automatic gaze analysis: A survey of deep learning based approaches. IEEE Transactions on Pattern Analysis and Machine Intelligence46(1), 61–84 (2023)

  8. [7]

    IEEE Journal of Biomedical and Health Infor- matics (2024)

    Ibragimov, B., Mello-Thoms, C.: The use of machine learning in eye tracking stud- ies in medical imaging: A review. IEEE Journal of Biomedical and Health Infor- matics (2024)

Show all 24 references
  1. [8]

    Nature methods18(2), 203–211 (2021)

    Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods18(2), 203–211 (2021)

  2. [9]

    In: MultiMedia modeling: 26th international conference, MMM 2020, Daejeon, South Korea, January 5–8, 2020, proceedings, part II 26

    Jha, D., Smedsrud, P.H., Riegler, M.A., Halvorsen, P., De Lange, T., Johansen, D., Johansen, H.D.: Kvasir-seg: A segmented polyp dataset. In: MultiMedia modeling: 26th international conference, MMM 2020, Daejeon, South Korea, January 5–8, 2020, proceedings, part II 26. pp. 451...

  3. [10]

    IEEE Journal of Biomedical and Health Informatics (2024)

    Jiang, H., Gao, M., Liu, Z., Tang, C., Zhang, X., Jiang, S., Yuan, W., Liu, J.: Glanceseg: Real-time microaneurysm lesion segmentation with gaze-map-guided foundation model for early detection of diabetic retinopathy. IEEE Journal of Biomedical and Health Informatics (2024)

  4. [11]

    In: International Conference on Medical Image Computing and Computer-Assisted Intervention

    Kong,Y.,Wang,S.,Cai,J.,Zhao,Z.,Shen,Z.,Li,Y.,Fei,M.,Wang,Q.:Gaze-detr: Using expert gaze to reduce false positives in vulvovaginal candidiasis screening. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 133–143. Springer (2024)

  5. [12]

    Advances in neural information processing systems24(2011)

    Krähenbühl, P., Koltun, V.: Efficient inference in fully connected crfs with gaussian edge potentials. Advances in neural information processing systems24(2011)

  6. [13]

    IEEE Transactions on Medical Imaging (2024)

    Li, Z., Zheng, Y., Shan, D., Yang, S., Li, Q., Wang, B., Zhang, Y., Hong, Q., Shen, D.: Scribformer: Transformer makes cnn work better for scribble-based medical image segmentation. IEEE Transactions on Medical Imaging (2024)

  7. [14]

    Pattern recognition122, 108341 (2022)

    Liu, X., Yuan, Q., Gao, Y., He, K., Wang, S., Tang, X., Tang, J., Shen, D.: Weakly supervised segmentation of covid19 infection with scribble annotation on ct images. Pattern recognition122, 108341 (2022)

  8. [15]

    In: International Conference on Medical Image Computing and Computer-Assisted Intervention

    Luo, X., Hu, M., Liao, W., Zhai, S., Song, T., Wang, G., Zhang, S.: Scribble- supervised medical image segmentation via dual-branch network and dynamically mixed pseudo labels supervision. In: International Conference on Medical Image Computing and Computer-Assisted Interventi...

  9. [16]

    In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, Oc- tober 5-9, 2015, proceedings, part III 18

    Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomed- ical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, Oc- tober 5-9, 2015, proceedings, part III 18....

  10. [17]

    In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion

    Tang, F., Xu, Z., Qu, Z., Feng, W., Jiang, X., Ge, Z.: Hunting attributes: Context prototype-aware learning for weakly supervised semantic segmentation. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion. pp. 3324–3334 (2024)

  11. [18]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Tian, Z., Shen, C., Wang, X., Chen, H.: Boxinst: High-performance instance seg- mentation with box annotations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5443–5452 (2021)

  12. [19]

    IEEE Transactions on Medical Imaging 40(8), 1990–2001 (2021)

    Valvano, G., Leo, A., Tsaftaris, S.A.: Learning to segment from scribbles using multi-scale adversarial attention gates. IEEE Transactions on Medical Imaging 40(8), 1990–2001 (2021)

  13. [20]

    In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision

    Wang, B., Pan, H., Aboah, A., Zhang, Z., Keles, E., Torigian, D., Turkbey, B., Krupinski, E., Udupa, J., Bagci, U.: Gazegnn: A gaze-guided graph neural network for chest x-ray classification. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision....

  14. [21]

    arXiv preprint arXiv:2408.12814 (2024)

    Wang, Z., Ye, Y., Chen, Z., Shu, M., Xia, Y.: From few to more: Scribble-based medical image segmentation via masked context modeling and continuous pseudo labels. arXiv preprint arXiv:2408.12814 (2024)

  15. [22]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wu, L., Zhong, Z., Fang, L., He, X., Liu, Q., Ma, J., Chen, H.: Sparsely annotated semantic segmentation with adaptive gaussian mixtures. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15454– 15464 (2023)

  16. [23]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Zhao, X., Tang, F., Wang, X., Xiao, J.: Sfc: Shared feature calibration in weakly supervised semantic segmentation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 7525–7533 (2024)

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.