REVIEW 3 major objections 5 minor 30 references
VidFuncta: Towards Generalizable Neural Representations for Ultrasound Videos
T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read VidFuncta encodes ultrasound videos as one video-level code plus per-frame codes, and reconstructs them better than 2D and 3D baselines.
desk verdict Solid extension of Functa to ultrasound video with real efficiency gains, but the autoregressive inference scheme needs per-frame validation before the reconstruction claim is fully trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is two-level modulation of a shared sinusoidal MLP $M_\theta$: the meta-model parameters $\theta$ capture anatomy and patterns shared across the whole dataset, the video-specific vector $v\in\mathbb{R}^{2048}$ is passed through linear layers whose outputs are added as shift modulations at every network layer to capture time-invariant properties such as anatomy, gain, and depth, and per-frame vectors $\phi_t\in\mathbb{R}^{512}$ are projected similarly to capture temporal dynamics. Training follows the Functa/MedFuncta meta-learning recipe: an outer loop updates $\theta$, while an inner loop optimizes $v$ and $\phi_t$ on a random batch of $b$ frames. Because whole videos cannot fit in memory, inference is autoregressive: $v$ is fitted on the first batch of frames, frozen, and only $\phi_t$ is optimized for all subsequent batches.
What would settle it
Record an ultrasound clip in which the operator changes the gain or probe angle after the first few frames, reconstruct the later frames with VidFuncta, and compare against a version that re-fits the video-level code on all frames; if the frozen-code reconstruction is visibly worse on the changed frames, the central assumption fails.
Extended reading notes
Core claim
The paper proposes VidFuncta, an implicit neural representation for ultrasound videos that writes each video as a static video-specific vector $v$ plus a time-resolved sequence of per-frame modulations $\{\phi_t\}_{t=1}^T$, both conditioning a shared sinusoidal MLP meta-model. On cardiac, lung, and breast ultrasound videos, this representation reconstructs held-out videos with higher SSIM3D and PSNR than MedFuncta applied per-frame, MedFuncta applied to 3D chunks, and Spatial Functa, at a compression rate of roughly 24. The same learned modulations can be fed directly to a transformer or MLP for ejection fraction regression, B-line detection, and breast lesion classification, with about a 25x reduction in downstream training cost compared with convolutional video models. A model trained on a mixed dataset of all three modalities retains most of the reconstruction quality, and reconstructions of an out-of-distribution dataset remain visually plausible.
Load-bearing premise
The method assumes that the video-level code learned from the first few frames captures every permanent feature of the scan; if later frames introduce new stable patterns, such as a changed gain setting or probe angle, the frozen code cannot represent them.
Editorial extensions
If this is right
- A single VidFuncta model can encode cardiac, lung, and breast ultrasound videos without dataset-specific retraining, with only a small reconstruction-quality loss relative to per-dataset models.
- Downstream clinical models can be trained directly on the 1D modulation sequences rather than on raw pixels, cutting GPU memory and training time by roughly 25x.
- Compressed reconstructions preserve enough task-relevant structure that a convolutional model trained on reconstructed videos performs nearly as well as on the original videos.
- The disentanglement into a static video vector and a time-resolved sequence separates stable anatomy and acquisition settings from temporal dynamics, which is useful for domain generalization and style transfer.
Reading between the lines
- A natural extension not tested in the paper is to re-estimate the video-level vector $v$ on later frames, or to predict it from a learned prior on the first batch; this would remove the freeze assumption while keeping memory bounded.
- Because the video-level codes cluster by modality in the paper's t-SNE visualization, the same representation could serve as an exam-level embedding for retrieval, quality control, or longitudinal comparison across visits.
- The gap between strong reconstruction and weaker downstream results suggests the bottleneck is how the 1D modulations are structured and read out, not the information they contain; imposing spatial or hierarchical structure on $\phi_t$ is a concrete next test.
- If the result that convolutional models work on reconstructed videos transfers beyond the tested baseline, clinical models could be trained and stored entirely in the compressed modulation space, making multi-site ultrasound studies with differing scanners more practical.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes VidFuncta, an extension of the Functa framework to ultrasound videos, representing each video by a static video-specific modulation vector v and a sequence of frame-specific modulation vectors phi_t that condition a shared sinusoidal MLP meta-model. Training uses a meta-learning inner/outer loop (Algorithm 1) on randomly sampled b-frame batches; at inference, reconstruction is performed in a batch-wise manner, fitting v on the first batch and freezing it thereafter. Reconstruction quality is reported as SSIM3D and PSNR on three public ultrasound datasets (cardiac, lung, breast), comparing against MedFuncta 2D/3D and Spatial Functa, plus mixed-dataset and out-of-distribution (OOD) variants. Downstream tasks (ejection fraction regression, B-line detection, breast lesion classification) are trained on the learned modulations and compared with convolutional baselines, with substantial reductions in downstream training time and memory.
Significance. If the reconstruction results hold, VidFuncta offers a compact, time-resolved neural representation for ultrasound video, with a single model usable across multiple datasets and a roughly 25x reduction in downstream training cost. Strengths of the paper include public code, consistent reconstruction improvements over three baselines across three datasets with mean and standard deviation in Table 1, and the demonstration that downstream regressors/classifiers can operate directly on the 1D modulation vectors. The OOD-generalization and downstream-performance claims are currently less supported than the reconstruction claim; additional evidence is needed before those broader conclusions can be accepted.
major comments (3)
- [§2, 'Reconstruction During Inference' (Algorithm 1, Figure 2); Table 1] The autoregressive inference scheme creates a train/inference mismatch: during training, v is optimized jointly with phi_t on randomly sampled b-frame batches, while at inference v is optimized only on the first b frames and then frozen (Figure 2). The paper reports only whole-video mean SSIM3D/PSNR in Table 1, so a systematic degradation in later frames due to appearance changes (gain, probe motion, shadowing) would be hidden in the average; the batch size b is never stated. Please report per-frame or per-batch reconstruction metrics, specify b, and analyze the effect of freezing v, since the reconstruction superiority claim is the central evidence for the representation.
- [§3.1, 'Reconstruction Task'; Table 1; §5 Conclusion] The claim that the method 'generalizes well to out-of-distribution data' is not supported by the experiments as presented. The OOD rows in Table 1 report only VidFuncta, with no MedFuncta or Spatial Functa baselines evaluated under the same train-on-two/inference-on-third protocol, so the large score drops (e.g., cardiac SSIM3D from 92.8 to 68.0) cannot be attributed to the method itself rather than to general domain shift. Please add OOD baselines or temper the conclusion to a descriptive statement.
- [§4.2, 'Results on the Downstream Tasks'; Table 2] Table 2 reports downstream performance as means across 5 folds without standard deviations in the main text, unlike the reconstruction table, so the reader cannot assess whether the observed differences between VidFuncta and the convolutional baselines, or between the v-only and phi-only settings, are reliable. The statement that PocovidNet on reconstructed videos performs 'similarly' to on original videos is also made without uncertainty quantification. Please include error bars or full statistics in the main table rather than deferring them to the code repository.
minor comments (5)
- [§3, Experiments] The sentence 'All remaining hyperparameters follow the configuration suggested in [10]' is insufficient for reproducibility; the batch size b, the number of sampled coordinates N, and the exact inner/outer optimization settings should be stated explicitly.
- [§4.1, Figure 3] The reconstruction of a natural image is presented as qualitatively 'ultrasound-like'; without a quantitative metric or task, this anecdote does not support the style-transfer claim in the conclusion and could be moved to future work.
- [§4.2, Figure 5] The claim that reconstruction quality does not correlate with downstream performance is based on a scatter plot; report a correlation coefficient or similar quantitative summary.
- [§4.1, Table 1] The statement that training on the mixed dataset 'does not significantly degrade performance' is not backed by a statistical test; provide confidence intervals or a significance test for the mixed versus per-dataset comparisons.
- [§2, 'Reconstruction During Inference'] The inference procedure is batch-wise rather than autoregressive in the temporal-prediction sense; consider renaming it to avoid confusion with standard autoregressive video models.
Circularity Check
No significant circularity: the reconstruction and downstream claims are evaluated on external public ultrasound datasets, and the only self-citations supply training configuration rather than the claimed outcome.
full rationale
VidFuncta's central claims are empirical: reconstruction quality is measured on held-out test videos from BEDLUS, EchoNet-Dynamic, and the Breast Ultrasound Video dataset against MedFuncta 2D/3D, Spatial Functa, and convolutional baselines (Tables 1-2), so the reported superiority is not derived from its own definitions. The paper does cite the authors' prior MedFuncta work for the meta-learning scheme and hyperparameters ('Following Friedrich et al. [10], we adopt a meta-learning strategy...' and 'All remaining hyperparameters follow the configuration suggested in [10]'), but this self-citation is not load-bearing in a circular sense: the cited work supplies an optimization recipe, not the target result, and VidFuncta is compared against MedFuncta itself on external data. The autoregressive inference step ('We assume that this initialization is enough to capture video-specific features... We therefore freeze v...') is an explicit assumption rather than a derivation; moreover, each later batch still optimizes its own phi_t, so later-frame reconstructions are fits rather than predictions forced by the frozen v, meaning the train/inference mismatch is a correctness risk, not a circular reduction. The paper's stated limitations (high-frequency detail loss, downstream models struggling to use modulations) are honest and do not conceal a self-referential argument. No equation is equivalent to an input by construction, and no fitted parameter is relabeled as a prediction.
Assumptions & free parameters
free parameters (7)
- K (number of layers) =
10
- l (hidden dimension) =
256
- s (video modulation dimension) =
2048
- r (frame modulation dimension) =
512
- G (inner loop steps) =
10
- gamma1 (inner learning rate) =
0.1
- gamma2 (meta learning rate) =
0.5e-6
assumptions (4)
- domain assumption Fitting v on the first batch captures all time-invariant video features
- domain assumption Meta-learning with random b-frame batches converges to a meta-model that generalizes across datasets
- domain assumption Downsampling to 112x112 and normalization preserve clinically relevant information
- ad hoc to paper Hyperparameters chosen for MedFuncta transfer to the video extension
invented entities (2)
-
Video-specific modulation vector v
-
Frame-specific modulation sequence phi_t
Cite this review
Pith. "Pith review of VidFuncta: Towards Generalizable Neural Representations for Ultrasound Videos." pith.science (2026). https://pith.science/paper/EEUMOCSK
@misc{pith2026250721863,
author = {Pith},
title = {Pith review of: VidFuncta: Towards Generalizable Neural Representations for Ultrasound Videos},
year = {2026},
howpublished = {\url{https://pith.science/paper/EEUMOCSK}},
note = {Machine review of arXiv:2507.21863}
}
read the original abstract
Ultrasound is widely used in clinical care, yet standard deep learning methods often struggle with full video analysis due to non-standardized acquisition and operator bias. We offer a new perspective on ultrasound video analysis through implicit neural representations (INRs). We build on Functa, an INR framework in which each image is represented by a modulation vector that conditions a shared neural network. However, its extension to the temporal domain of medical videos remains unexplored. To address this gap, we propose VidFuncta, a novel framework that leverages Functa to encode variable-length ultrasound videos into compact, time-resolved representations. VidFuncta disentangles each video into a static video-specific vector and a sequence of time-dependent modulation vectors, capturing both temporal dynamics and dataset-level redundancies. Our method outperforms 2D and 3D baselines on video reconstruction and enables downstream tasks to directly operate on the learned 1D modulation vectors. We validate VidFuncta on three public ultrasound video datasets -- cardiac, lung, and breast -- and evaluate its downstream performance on ejection fraction prediction, B-line detection, and breast lesion classification. These results highlight the potential of VidFuncta as a generalizable and efficient representation framework for ultrasound videos. Our code is publicly available under https://github.com/JuliaWolleb/VidFuncta_public.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Asgari-Targhi, A., Ungi, T., Jin, M., Harrison, N., Duggan, N., Duhaime, E., Gold- smith, A., Kapur, T.: Can crowdsourced annotations improve ai-based congestion scoring for bedside lung ultrasound? In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 580–590. Springer (2024)
work page 2024
-
[2]
IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control67(11), 2312–2320 (2020)
Baloescu, C., Toporek, G., Kim, S., McNamara, K., Liu, R., Shaw, M.M., McNa- mara, R.L., Raju, B.I., Moore, C.L.: Automated lung ultrasound b-line assessment using a deep learning algorithm. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control67(11), 2312–2320 (2020)
work page 2020
-
[3]
arXiv preprint arXiv:2302.03130 (2023)
Bauer, M., Dupont, E., Brock, A., Rosenbaum, D., Schwarz, J.R., Kim, H.: Spatial functa: Scaling functa to imagenet classification and generation. arXiv preprint arXiv:2302.03130 (2023)
arXiv 2023
-
[4]
POCOVID-Net: Automatic Detection of COVID-19 From a New Lung Ultrasound Imaging Dataset (POCUS)
Born, J., Brändle, G., Cossio, M., Disdier, M., Goulet, J., Roulin, J., Wiedemann, N.: Pocovid-net: automatic detection of covid-19 from a new lung ultrasound imag- ing dataset (pocus). arXiv preprint arXiv:2004.12084 (2020) 10 J. Wolleb et al
work page Pith review arXiv 2020
-
[5]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Chen,Z., Chen,Y., Liu,J., Xu, X.,Goel,V., Wang, Z.,Shi, H.,Wang,X.: Videoinr: Learning video implicit neural representation for continuous space-time super- resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2047–2057 (2022)
work page 2022
-
[6]
arXiv preprint arXiv:2010.11929 (2020)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
arXiv 2020
-
[7]
arXiv preprint arXiv:2201.12204 (2022)
Dupont, E., Kim, H., Eslami, S., Rezende, D., Rosenbaum, D.: From data to functa: Your data point is a function and you can treat it like one. arXiv preprint arXiv:2201.12204 (2022)
arXiv 2022
-
[8]
arXiv preprint arXiv:2201.12904 (2022)
Dupont, E., Loya, H., Alizadeh, M., Goliński, A., Teh, Y.W., Doucet, A.: Coin++: Neural compression across modalities. arXiv preprint arXiv:2201.12904 (2022)
arXiv 2022
Show all 30 references
-
[9]
In: 2025 IEEE 22nd International Symposium on Biomedical Imaging (ISBI)
Eid, M.C., Yeung, P.H., Wyburd, M.K., Henriques, J.F., Namburete, A.I.: Rapid- vol: Rapid reconstruction of 3d ultrasound volumes from sensorless 2d scans. In: 2025 IEEE 22nd International Symposium on Biomedical Imaging (ISBI). pp. 1–5. IEEE (2025)
2025
-
[10]
arXiv preprint arXiv:2502.14401 (2025)
Friedrich, P., Bieder, F., Cattin, P.C.: Medfuncta: Modality-agnostic representa- tions based on efficient neural fields. arXiv preprint arXiv:2502.14401 (2025)
2025
-
[11]
In: Medical Imaging with Deep Learning (2022)
Gu, A.N., Abolmaesumi, P., Luong, C., Yi, K.M.: Representing 3d ultrasound with neural fields. In: Medical Imaging with Deep Learning (2022)
2022
-
[12]
arXiv preprint arXiv:2501.12524 (2025)
Guo, J., Wu, Y., Kaimakamis, E., Petmezas, G., Papageorgiou, V.E., Maglaveras, N., Katsaggelos, A.K.: Efficient lung ultrasound severity scoring using dedicated feature extractor. arXiv preprint arXiv:2501.12524 (2025)
2025 arXiv
-
[13]
Journal of medical artificial intelligence3, 4 (2020)
Howard, J.P., Tan, J., Shun-Shin, M.J., Mahdi, D., Nowbar, A.N., Arnold, A.D., Ahmad, Y., McCartney, P., Zolgharni, M., Linton, N.W., et al.: Improving ultra- sound video classification: an evaluation of novel deep learning methods in echocar- diography. Journal of medical art...
2020
-
[14]
1st Workshop on Machine Learning and Global Health (ICLR 2023) (2023)
Hu, Yurong, e.a.: Self-supervised learning to predict ejection fraction using motion- mode images. 1st Workshop on Machine Learning and Global Health (ICLR 2023) (2023)
2023
-
[15]
In: 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI)
Huang, Y., Hu, H., Zhu, Y., Xu, Y.: Breast lesion diagnosis using static images and dynamic video. In: 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI). pp. 1–5. IEEE (2023)
2023
-
[16]
Ultrasonography40(3), 313 (2021)
Kim, Y.H.: Artificial intelligence in medical ultrasonography: driving on an un- paved road. Ultrasonography40(3), 313 (2021)
2021
-
[17]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Lin, Z., Lin, J., Zhu, L., Fu, H., Qin, J., Wang, L.: A new dataset and a baseline model for breast lesion detection in ultrasound videos. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 614–623. Springer (2022)
2022
-
[18]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Liu, Z., Zhu, H., Zhang, Q., Fu, J., Deng, W., Ma, Z., Guo, Y., Cao, X.: Finer: Flexible spectral-bias tuning in implicit neural representation by variable-periodic activation functions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp....
2024
-
[19]
IEEE journal of biomedical and health informatics27(9), 4352–4361 (2023)
Lucassen, R.T., Jafari, M.H., Duggan, N.M., Jowkar, N., Mehrtash, A., Fischetti, C., Bernier, D., Prentice, K., Duhaime, E.P., Jin, M., et al.: Deep learning for detection and localization of b-lines in lung ultrasound. IEEE journal of biomedical and health informatics27(9), 4...
2023
-
[20]
Journal of machine learning research 9(11) (2008) VidFuncta for Ultrasound Videos 11
Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research 9(11) (2008) VidFuncta for Ultrasound Videos 11
2008
-
[21]
In: NeurIPS ML4H Workshop: Vancouver, BC, Canada
Ouyang, D., He, B., Ghorbani, A., Lungren, M.P., Ashley, E.A., Liang, D.H., Zou, J.Y.: Echonet-dynamic: a large new cardiac motion video data resource for medical machine learning. In: NeurIPS ML4H Workshop: Vancouver, BC, Canada. vol. 5 (2019)
2019
-
[22]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Papa, S., Valperga, R., Knigge, D., Kofinas, M., Lippe, P., Sonke, J.J., Gavves, E.: How to train neural field representations: A comprehensive study and benchmark. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22616–22625 (2024)
2024
-
[23]
In: Proceedings of the AAAI conference on artificial intelligence
Perez, E., Strub, F., De Vries, H., Dumoulin, V., Courville, A.: Film: Visual rea- soning with a general conditioning layer. In: Proceedings of the AAAI conference on artificial intelligence. vol. 32 (2018)
2018
-
[24]
Elsevier Health Sciences (2023)
Rumack, C.M., Levine, D.: Diagnostic ultrasound E-book. Elsevier Health Sciences (2023)
2023
-
[25]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Saragadam, V., LeJeune, D., Tan, J., Balakrishnan, G., Veeraraghavan, A., Bara- niuk, R.G.: Wire: Wavelet implicit neural representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18507– 18516 (2023)
2023
-
[26]
Advances in neural information processing systems 33, 7462–7473 (2020)
Sitzmann, V., Martel, J., Bergman, A., Lindell, D., Wetzstein, G.: Implicit neural representations with periodic activation functions. Advances in neural information processing systems 33, 7462–7473 (2020)
2020
-
[27]
International Journal of Maternal and Child Health and AIDS 9(1), 103 (2020)
Stewart, K.A., Navarro, S.M., Kambala, S., Tan, G., Poondla, R., Lederman, S., Barbour, K., Lavy, C.: Trends in ultrasound use in low and middle income coun- tries: a systematic review. International Journal of Maternal and Child Health and AIDS 9(1), 103 (2020)
2020
-
[28]
In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition
Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., Paluri, M.: A closer look at spatiotemporal convolutions for action recognition. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 6450–6459 (2018)
2018
-
[29]
Ieee Access9, 54310–54324 (2021)
Wang, Y., Ge, X., Ma, H., Qi, S., Zhang, G., Yao, Y.: Deep learning in medical ultrasound image analysis: a review. Ieee Access9, 54310–54324 (2021)
2021
-
[30]
IEEE Journal of Biomedical and Health Informatics (2025)
Wiedemann, N., de Korte-De Boer, D., Richter, M., van de Weijer, S., Buhre, C., Eggert, F.A., Aarnoudse, S., Grevendonk, L., Röber, S., Remie, C.M., et al.: Covid-blues-a prospective study on the value of ai in lung ultrasound analysis. IEEE Journal of Biomedical and Health In...
2025
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.