Pith. sign in

REVIEW 4 major objections 6 minor 21 references

Self-Supervised Ultrasound-Video Segmentation with Feature Prediction and 3D Localised Loss

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding a 3D localisation task to V-JEPA pre-training improves cardiac ultrasound segmentation by up to 8.35% when only 10% of labels are available.

desk verdict A solid, incremental SSL-for-ultrasound paper with consistent gains, but the 'locality' mechanism is under-supported and the eval protocol needs tightening. read the letter →

arxiv 2507.18424 v1 pith:FFQKJNCH submitted 2025-07-24 cs.CV

Edward Ellis , Robert Mendel , Andrew Bulpitt , Nasim Parsa , Michael F Byrne , Sharib Ali This is my paper · ORCID
classification cs.CV
keywords Self-supervisedlearningUltrasoundvideosegmentationV-JEPAFeaturepredictionVisionTransformer3DlocalisationauxiliarytaskCardiacLimitedannotation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that V-JEPA, a self-supervised video model that learns by predicting latent features of masked patches rather than by reconstructing pixels, is well suited to noisy ultrasound video, and that its main weakness on small medical datasets—limited spatial locality in vision transformers—can be fixed with a lightweight auxiliary task. That task, a 3D localisation loss, samples pairs of predicted patch embeddings from masked regions and trains a small MLP to predict their normalised relative spatio-temporal offsets. The paper reports that adding this loss to V-JEPA pre-training improves downstream frozen-encoder cardiac ultrasound segmentation on the CAMUS dataset, with Dice gains up to 3.40% when all training labels are used and up to 8.35% when only 10% are used. A sympathetic reader would care because ultrasound labels are scarce and expensive, so pre-training that extracts more from unlabelled video could reduce the annotation burden in clinical practice.

What carries the argument

The load-bearing mechanism is the 3D localisation auxiliary task. During pre-training, a video clip is tokenised into $t \times i \times j$ spatio-temporal patch embeddings; predicted masked embeddings $M_{t,i,j}$ are used to sample 100 random pairs per batch item, and the normalised ground-truth offset is $\Delta^{(b)}_{m_1,m_2} = ((t_1-t_2)/t,\; (i_1-i_2)/i,\; (j_1-j_2)/j)$. A three-layer MLP $F(\cdot)$ takes the concatenated pair and predicts the offset, and the MSE between prediction and target is $L_{\text{ll}}$. This is combined with V-JEPA's feature-prediction loss through the weight $\lambda$. The task's role is to force the ViT encoder and predictor to encode relative position of patches, countering the transformer's weak spatial locality on small medical datasets.

What would settle it

Re-run pre-training with shuffled patch-index offsets as ground truth: if the Dice gains persist, the auxiliary loss helps through generic regularisation rather than spatial learning. Separately, inspect whether any test patient's videos appeared in pre-training; if they did, the improvement claims would collapse.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a 3D localisation auxiliary task, added during V-JEPA pre-training, makes the learned representations more spatially and temporally aware and thereby improves downstream segmentation of cardiac ultrasound video. The combined loss is $L_{\text{combined}} = \lambda \cdot L_{\text{jepa}} + (1-\lambda) \cdot L_{\text{ll}}$, with the auxiliary loss $L_{\text{ll}}$ computed as the mean squared error between an MLP's prediction and the normalised relative translation offset $\Delta^{(b)}_{m_1,m_2}$ of randomly sampled pairs of predicted masked patches. With $\lambda=0.25$, the authors report statistically significant Dice improvements ($p<0.05$) for all V-JEPA configurations tested: 1.07%, 3.40% and 0.7% with 100% of training data and 7.45%, 8.35% and 2.31% with 10% of training data, for V-JEPA, V-JEPA (12b) and V-JEPA (16b) respectively. The same pattern holds for the Jaccard index, with recall improving more than precision as labels become scarce.

Load-bearing premise

The central claim rests on the assumption that the videos used for pre-training and for tuning the loss weight are strictly separate from the test videos, so the reported gains measure real transfer rather than data overlap.

Editorial extensions

If this is right

  • If the central claim holds, V-JEPA pre-training is a viable alternative to mask-reconstruction SSL for ultrasound video, outperforming VideoMAE and supervised ViT baselines on CAMUS.
  • The 3D localisation loss improves segmentation most when labels are scarce: up to 8.35 points of Dice at 10% training data for the 12-block configuration.
  • Freezing more transformer blocks (12b, 16b) during pre-training consistently helps, suggesting that limiting trainable parameters reduces overfitting on small medical datasets.
  • Because the auxiliary task only touches the predictor output and the loss, it can be added to other JEPA-style video models without modifying their encoder architectures.
  • Larger Dice gains at 10% than at 100% imply the method's value grows as the annotation budget shrinks, which is the clinically relevant regime.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the same offset-prediction head could be attached to other masked-latent video SSL methods, not just V-JEPA; a direct test would be adding the 3D localisation loss to VideoMAE-style training and comparing Dice gains.
  • Inference: because the targets in Eq. 2 are computed from patch indices that the predictor already receives as positional conditioning, some of the reported gain may come from a regularisation or auxiliary-gradient effect rather than genuinely new spatial knowledge; ablating with shuffled indices would separate these.
  • Inference: if the gains transfer across ultrasound sites and scanners, this recipe lowers annotation cost for cardiac segmentation and could extend to other video-based point-of-care imaging settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript studies self-supervised video segmentation of cardiac ultrasound using V-JEPA. It proposes a 3D relative localisation auxiliary task: from pairs of predictor-produced masked patch embeddings, an MLP regresses normalized spatio-temporal offsets, and this loss is combined with V-JEPA's feature-prediction loss. On CAMUS, with frozen ViT-L encoders, the authors report that adding the localisation loss improves DSC over V-JEPA baselines across three block-freezing configurations and at 10-100% label fractions, and that V-JEPA variants outperform VideoMAE and supervised training. The paper claims that the auxiliary task improves the spatial-temporal locality of ViT representations.

Significance. If the empirical gains are clean, this is a potentially useful contribution: V-JEPA is applied to ultrasound video for the first time, and the reported improvements are consistent across configurations and label budgets, with the largest DSC gains in the 10% training regime (up to 8.35 percentage points). The use of a public benchmark, multiple encoder configurations, and per-configuration p-values are strengths. However, the manuscript provides no code, no patient-level split details, and its mechanistic claim that the auxiliary task enforces locality is not backed by an analysis. The empirical finding is promising, but the evaluation and interpretation need substantial revision.

major comments (4)
  1. [Section 3.1] The paper never specifies the CAMUS patient split or the subset used for pre-training. Because pre-training is itself performed on CAMUS, the reported comparisons are only valid if pre-training and downstream evaluation are separated at the patient level. Please report the number of patients in train/validation/test, state explicitly that no test patient videos were used for pre-training or for lambda/checkpoint selection, and make the exact pre-training subset (patients, clips) explicit.
  2. [Section 2.2, Eq. (2)-(3)] The localisation target is derived from patch indices that the predictor already receives as positional conditioning (Section 2.1). Since the MLP F sees concatenated predictor outputs, it can in principle regress the relative offsets by decoding positional information alone, without requiring the encoder or predictor to encode spatial content. The paper offers no ablation to rule this out (for example, training F on position-only or content-shuffled inputs, or testing whether F transfers to unseen relative offsets), and no diagnostic that representations become more local. Therefore the stated mechanism, that the task 'improves locality', is unsupported; the DSC gains may stem from the changed loss landscape or optimisation dynamics rather than from spatial supervision.
  3. [Table 1, Section 4.1] Lambda=0.25 is selected on the validation set using 100% training samples, and the same value is then used for all reported test results. The assertion that this lambda remains optimal for 10%, 20% and 50% subsets is not shown in any table or figure. Because lambda is tuned on the same dataset used for evaluation, this can inflate the reported gains. Please provide per-subset lambda ablations or use a nested validation procedure, and report how the validation set was used.
  4. [Section 4.2] The p-values are reported for DSC gains but the test procedure is not described: was it a paired test across patients, and were multiple comparisons across the three configurations and label fractions adjusted? Without this, the word 'significant' cannot be assessed. Please add the exact statistical test, the pairing unit, and the multiple-comparison correction method, or justify why no correction is needed.
minor comments (6)
  1. [Section 3.1] The freeze configurations V-JEPA (12b) and V-JEPA (16b) are not defined precisely; please state which transformer blocks are frozen, at what stage, and how many parameters are trainable in each variant.
  2. [Equation (2)] The notation i, j, t is used both for token counts and for the claimed range [-1,1]; please clarify the normalization denominators and use distinct symbols for the total token counts.
  3. [Table 2] In the 10% row for V-JEPA (12b) + LL, the JI value '0.671±0.967' appears to be a typo; the standard deviation is implausible and inconsistent with neighboring rows.
  4. [Figure 2] The qualitative discussion refers to orange annotations that are difficult to see in print; please add zoomed crops or clearer arrows and labels.
  5. [References] Reference [6] formats author names in the reference text instead of using the standard citation style; please make the bibliography consistent.
  6. [Reproducibility] Please include a data/code availability statement; the paper does not mention whether the code and the pre-training splits will be released.

Circularity Check

1 steps flagged · score 4.0 of 10

Downstream CAMUS benchmark is external and non-circular, but the 3D localisation auxiliary task defines its target from the same patch indices that V-JEPA already feeds to the predictor, so the claimed locality mechanism is self-referential and unsupported.

  1. self definitional [Section 2.1 and Section 2.2, Eq. (2)]
    "In practice this corruption is implemented via masking, which is contextualised for the network through positional embeddings, εp. We define εp ← ∆y, where ∆y denotes the spatio-temporal positions of the masked regions of y. ... ∆(b)m1,m2 = ( (t1 − t2)/t , (i1 − i2)/i , (j1 − j2)/j )"

    The localisation target in Eq. (2) is computed from the same (t,i,j) patch indices that V-JEPA already uses as positional conditioning for the predictor: εp ← Δy. The inputs to the MLP F are predictor outputs e_{m1} and e_{m2}, which therefore already carry the positional codes of the two sampled masked tokens. The relative offset Δ is a deterministic function of those position codes, so F can minimise L_ll by decoding the positions already present in its input, without requiring the encoder to learn any spatial-content locality. The paper gives no ablation or diagnostic showing that F cannot solve the task from positions alone, and no evidence that the learned representations become more local.

full rationale

The central segmentation claims are evaluated against external baselines (VideoMAE and supervised ViT-L) on the CAMUS test set, so the reported DSC improvements are genuinely empirical and not circular. The λ=0.25 weighting was chosen on the validation set (Table 1) before reporting test results; this is standard hyperparameter tuning and does not make the test-set comparison circular. There is no load-bearing self-citation chain: the cited V-JEPA and VideoMAE results are external prior work, and the authors' own previous paper (Ref. [7]) is not used to justify the novel auxiliary task. The one substantive issue is the 3D localisation auxiliary task itself: its ground-truth offset is defined from the same spatio-temporal patch indices that V-JEPA's predictor already receives as positional embeddings, so the auxiliary prediction can in principle be solved by reading position codes rather than by learning spatial locality. This makes the claimed locality mechanism self-referential and unsupported, but it does not undermine the independent external benchmark comparison that forms the paper's central empirical result.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on one fitted hyperparameter (lambda) and several unstated data-protocol assumptions. The auxiliary-task justification is ad hoc because the task may be solvable from positional conditioning.

free parameters (1)
  • lambda (combined loss weight) = 0.25
    Weight balancing V-JEPA loss and 3D localisation loss; selected by validation set performance in Table 1, claimed optimal across data fractions but only 100% subset shown.
assumptions (5)
  • standard math L1 masked feature prediction loss and MSE localisation loss are appropriate optimization targets
    Eqs. 1 and 3; standard practice, no proof of optimality needed.
  • domain assumption CAMUS annotations of LV endocardium, LV epicardium, and LA wall are accurate ground truth
    Section 3; downstream evaluation relies on these annotations.
  • domain assumption The validation set used to pick lambda is representative and the test set was not used for hyperparameter selection
    Section 4.1 and Table 1; if hyperparameters were selected with test-set feedback, gains are inflated.
  • domain assumption Pre-training on CAMUS video is separated from downstream test evaluation
    Section 3.1 does not state which patients' videos are used for pre-training.
  • ad hoc to paper The 3D localisation task forces the ViT to learn spatial-temporal locality rather than being solved by reading positional embeddings
    Section 2.2; the predictor receives positional conditioning, so offsets may be recoverable without semantic features.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Self-Supervised Ultrasound-Video Segmentation with Feature Prediction and 3D Localised Loss." pith.science (2026). https://pith.science/paper/FFQKJNCH

@misc{pith2026250718424,
  author       = {Pith},
  title        = {Pith review of: Self-Supervised Ultrasound-Video Segmentation with Feature Prediction and 3D Localised Loss},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FFQKJNCH}},
  note         = {Machine review of arXiv:2507.18424}
}
read the original abstract

Acquiring and annotating large datasets in ultrasound imaging is challenging due to low contrast, high noise, and susceptibility to artefacts. This process requires significant time and clinical expertise. Self-supervised learning (SSL) offers a promising solution by leveraging unlabelled data to learn useful representations, enabling improved segmentation performance when annotated data is limited. Recent state-of-the-art developments in SSL for video data include V-JEPA, a framework solely based on feature prediction, avoiding pixel level reconstruction or negative samples. We hypothesise that V-JEPA is well-suited to ultrasound imaging, as it is less sensitive to noisy pixel-level detail while effectively leveraging temporal information. To the best of our knowledge, this is the first study to adopt V-JEPA for ultrasound video data. Similar to other patch-based masking SSL techniques such as VideoMAE, V-JEPA is well-suited to ViT-based models. However, ViTs can underperform on small medical datasets due to lack of inductive biases, limited spatial locality and absence of hierarchical feature learning. To improve locality understanding, we propose a novel 3D localisation auxiliary task to improve locality in ViT representations during V-JEPA pre-training. Our results show V-JEPA with our auxiliary task improves segmentation performance significantly across various frozen encoder configurations, with gains up to 3.4\% using 100\% and up to 8.35\% using only 10\% of the training data.

Figures

Figures reproduced from arXiv: 2507.18424 by the authors.

Figure 1
Figure 1. Block Diagram of our 3D localisation auxiliary task incorporated in the V￾JEPA SSL framework. Our auxiliary task takes a random pair of patch embeddings, (em1 , em2 ), from the predictor and predicts the relative temporal, vertical and horizon￾tal distances between the samples. (a) presents the general SSL framework of V-JEPA with our localisation task. (b) demonstrates the relative localisation between a sample pai… view at source ↗
Figure 2
Figure 2. Qualitative evaluation on CAMUS dataset. 3 example videos were chosen at frame 9. Segmentation predictions across all methods are presented, using 100% and 10% training samples. LV endocardium, LV epicardium, LA wall are indicated in green, red and blue respectively. Orange annotations highlight key points discussed in the qualitative results section (see 4.3). 5 Conclusions In this work we explored the performance … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 18 canonical work pages

  1. [1]

    Pattern Recognition 153, 110510 (Sep 2024)

    Akkaya, I.B., Kathiresan, S.S., Arani, E., Zonooz, B.: Enhancing performance of vision transformers on small datasets through local inductive bias incorporation. Pattern Recognition 153, 110510 (Sep 2024)

    work page 2024

  2. [2]

    arXiv (2024), https://arxiv.org/abs/2404.08471

    Bardes, A., Garrido, Q., Ponce, J., Chen, X., Rabbat, M., LeCun, Y., Assran, M., Ballas, N.: Revisiting Feature Prediction for Learning Visual Representations from Video. arXiv (2024), https://arxiv.org/abs/2404.08471

    arXiv 2024

  3. [3]

    Abdominal Radiology 43(4), 786–799 (Apr 2018)

    Brattain, L.J., Telfer, B.A., Dhyani, M., Grajo, J.R., Samir, A.E.: Machine learn- ing for medical ultrasound: status, methods, and future opportunities. Abdominal Radiology 43(4), 786–799 (Apr 2018)

    work page 2018

  4. [4]

    Chen, L., Rubin, J., Ouyang, J., Balaraju, N., Patil, S., Mehanian, C., Kulhare, S., Millin, R., Gregory, K.W., Gregory, C.R.: Contrastive self-supervised learning for spatio-temporal analysis of lung ultrasound videos. pp. 1–5. IEEE (2023)

    work page 2023

  5. [5]

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S.: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020)

    work page 2020

  6. [6]

    Lamoureux, S

    E. Lamoureux, S. Ayromlou, S. N. Ahmadi Amiri, H. Rhodin: Segmenting Cardiac Ultrasound Videos Using Self-Supervised Learning. In: 2023 45th Annual Interna- tionalConferenceoftheIEEEEngineeringinMedicine&BiologySociety(EMBC). pp. 1–7 (Jul 2023)

    work page 2023

  7. [7]

    A Self-Supervised Framework for Improved Generalisability in Ultrasound B-mode Image Segmentation

    Ellis, E., Bulpitt, A., Parsa, N., Byrne, M.F., Ali, S.: A Self-Supervised Framework for Improved Generalisability in Ultrasound B-mode Image Segmentation. arXiv preprint arXiv:2502.02489 (2025)

    work page Pith review arXiv 2025

  8. [8]

    Com- puter vision - ECCV

    Fu, Z., Jiao, J., Yasrab, R., Drukker, L., Papageorghiou, A.T., Noble, J.A.: Anatomy-Aware Contrastive Representation Learning for Fetal Ultrasound. Com- puter vision - ECCV. European Conference on Computer Vision: proceedings. European Conference on Computer Vision2022, 422–436 (Oct 2022)

    work page 2022

Show all 21 references
  1. [9]

    Proceedings

    Jiao, J., Droste, R., Drukker, L., Papageorghiou, A.T., Noble, J.A.: Self-Supervised Representation Learning for Ultrasound Video. Proceedings. IEEE International Symposium on Biomedical Imaging2020, 1847–1850 (Apr 2020)

    2020

  2. [10]

    Medical Image Analysis 96, 103202 (2024)

    Jiao, J., Zhou, J., Li, X., Xia, M., Huang, Y., Huang, L., Wang, N., Zhang, X., Zhou, S., Wang, Y.: Usfm: A universal ultrasound foundation model generalized to tasks and organs towards label efficient image analysis. Medical Image Analysis 96, 103202 (2024)

    2024

  3. [11]

    IEEE Transactions on Medical Imag- ing 38(9), 2198–2210 (2019)

    Leclerc, S., Smistad, E., Pedrosa, J., Østvik, A., Cervenansky, F., Espinosa, F., Espeland, T., Berg, E.A.R., Jodoin, P.M., Grenier, T., Lartizien, C., D’hooge, J., Lovstakken, L., Bernard, O.: Deep Learning for Segmentation Using an Open Large-Scale Dataset in 2D Echocardiogr...

    2019

  4. [12]

    2, 2022- 06-27

    LeCun, Y.: A path towards autonomous machine intelligence version 0.9. 2, 2022- 06-27. Open Review62(1), 1–62 (2022)

    2022

  5. [13]

    IEEE Transactions on Knowledge and Data Engineering 35(1), 857–876 (2023)

    Liu,X.,Zhang,F.,Hou,Z.,Mian,L.,Wang,Z.,Zhang,J.,Tang,J.:Self-Supervised Learning: Generative or Contrastive. IEEE Transactions on Knowledge and Data Engineering 35(1), 857–876 (2023)

    2023

  6. [14]

    Advances in Neural Information Process- ing Systems 34, 23818–23830 (2021)

    Liu, Y., Sangineto, E., Bi, W., Sebe, N., Lepri, B., Nadai, M.: Efficient training of visual transformers with small datasets. Advances in Neural Information Process- ing Systems 34, 23818–23830 (2021)

    2021

  7. [15]

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. pp. 10012– 10022 (2021) 10 E. Ellis et al

    2021

  8. [16]

    Echocardiography35(9), 1388–1401 (2018)

    Quien, M.M., Saric, M.: Ultrasound imaging artifacts: How to recognize them and how to avoid them. Echocardiography35(9), 1388–1401 (2018)

    2018

  9. [17]

    Szijártó, A., Magyar, B., Szeier, T.A., Tolvaj, M., Fábián, A., Lakatos, B.K., Ladányi, Z., Bagyura, Z., Merkely, B., Kovács, A.: Masked Autoencoders for Med- ical Ultrasound Videos Using ROI-Aware Masking. pp. 167–176. Springer (2024)

    2024

  10. [18]

    In: Proceedings of the 36th International Conference on Neural Information Processing Systems

    Tong, Z., Song, Y., Wang, J., Wang, L.: VideoMAE: masked autoencoders are data-efficient learners for self-supervised video pre-training. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. NIPS ’22, Red Hook, NY, USA (2022)

    2022

  11. [19]

    Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., Shao, L.: Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. pp. 568–578 (2021)

    2021

  12. [20]

    In: Simplifying Medical Ultrasound, vol

    Zhang, K., Jiao, J., Noble, J.A.: Fetal Ultrasound Video Representation Learning Using Contrastive Rubik’s Cube Recovery. In: Simplifying Medical Ultrasound, vol. 15186, pp. 187–197. Springer Nature Switzerland, Cham (2025)

    2025

  13. [21]

    arXiv preprint arXiv:2302.03751 (2023)

    Zhu, H., Chen, B., Yang, C.: Understanding why vit trains badly on small datasets: An intuitive perspective. arXiv preprint arXiv:2302.03751 (2023)

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.