REVIEW 4 major objections 6 minor 29 references
Enhancing Free-hand 3D Photoacoustic and Ultrasound Reconstruction using Deep Learning
T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read MoGLo-Net estimates 6-degree-of-freedom scan motion from B-mode ultrasound frames more accurately than prior sensorless methods, and uses those estimates to build 3D volumes from B-mode, power Doppler, and photoacoustic data.
desk verdict A genuinely new architecture for sensorless freehand ultrasound with an evaluation that is promising but needs more rigor; worth a careful peer review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing pieces are the patch-wise correlation volume and the global-local self-attention module. The correlation volume is built by taking two feature maps from the same frames of two shifted B-mode sequences, fixing a patch in one map, moving a patch across a region of interest in the other, and storing all pairwise correlations; it encodes in-plane motion as a shift of the peak correlation and out-of-plane motion as an overall decay. The attention module divides a mid-level feature map into 256 local blocks, recalibrates them with channel attention, and reweights each block by its cosine similarity to a spatially and channel-attended global feature, so regions with reliable speckle or persistent echogenic anatomy dominate the motion estimate. The fused global and local representations are fed to two LSTM motion estimators, and the motion-based mean-absolute-error loss up-weights fast-motion samples to counter the class imbalance while the correlation loss stabilizes the direction of motion.
What would settle it
Re-run the evaluation with ground-truth labels produced by an independent optical tracker synchronized to the B-mode frame times and compare accumulated trajectory error and final drift on the same long forearm sweeps; if MoGLo-Net no longer beats the LSTM baseline on accumulated error, the central claim of superior sensorless tracking fails.
Extended reading notes
Core claim
MoGLo-Net's central claim is that relative scan motion between consecutive B-mode frames can be recovered by explicitly modeling inter-frame correlation at the feature level and by attending to the image regions that carry motion information. The network processes two overlapping frame sequences through a shared residual encoder, builds a correlation volume from patch-wise comparisons of adjacent feature maps, and then uses a global-local self-attention module to weight local patches, such as fully developed speckle or high-echogenic tissue, by their similarity to a globally pooled feature. Two LSTM-based estimators predict global and local motion vectors, which are fused and supervised by a weighted mean-absolute error, a correlation loss, and a margin triplet loss. The paper reports that MoGLo-Net outperforms the compared sensorless methods on both its in-house forearm dataset and an independent public dataset, and that the estimated trajectories are accurate enough to reconstruct 3D B-mode volumes and, for the first time, 3D vascular structures from power Doppler and photoacoustic acquisitions.
Load-bearing premise
The supervised training and all reported error metrics assume the electromagnetic tracker's ground-truth motion labels are accurately synchronized with the 20 Hz B-mode frames and are reliable under the scanning conditions, but the paper does not describe a synchronization procedure or quantify tracker accuracy.
Editorial extensions
If this is right
- If the reported accuracy holds, sensorless freehand 3D ultrasound can be performed with a standard 1D array transducer, removing the cost and bulk of electromagnetic or optical trackers.
- Because the motion estimates come from B-mode frames, the same estimated trajectory can be reused to place power Doppler and photoacoustic images in 3D, giving volume renderings of vasculature without additional sensing.
- The ablation results imply that speckle content is a necessary cue: denoising B-mode images degrades motion accuracy, while adding IQ data or raw log-compressed frames does not help.
- The consistent performance ordering on the in-house and public datasets suggests the method's advantage is tied to the network design rather than to one acquisition setup.
- For long S-shaped sweeps exceeding 200 mm, the lower final drift rate implies accumulated errors grow slowly enough that volumes of that length remain coherent.
Reading between the lines
- The correlation-volume mechanism could extend to other speckle-based imaging modalities, such as intravascular ultrasound or optical coherence tomography pullbacks, where a similar patch-wise decorrelation signal exists.
- A direct next experiment is to test MoGLo-Net on anatomies with weak or heterogeneous speckle, such as the thyroid or abdomen; the attention module would need to fall back on high-echogenic tissue boundaries, and performance there is not yet established.
- If the electromagnetic-tracker ground-truth assumption is relaxed, a self-supervised variant could be trained by minimizing 3D volume consistency between overlapping sweeps, eliminating the need for synchronized external labels.
- The attention maps the authors visualize suggest a practical byproduct: real-time visualization of which speckle regions are being tracked could warn operators when a scan is moving too fast or when the probe is losing the needed texture.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents MoGLo-Net, a supervised deep network for estimating relative 6-DoF transducer motion between consecutive B-mode ultrasound frames. The architecture combines a ResNet-based encoder, a patch-wise correlation volume between adjacent feature maps, a global-local self-attention module, and two LSTM-based motion estimators; training uses a composite loss comprising motion-weighted MAE, correlation loss, and margin triplet loss. Evaluation is carried out on an in-house forearm PAUS dataset (Dataset 1) and a public freehand ultrasound dataset (Dataset 3), with comparisons to CNN, DCL, DC2, EfficientNet/Li et al., and LSTM baselines, plus ablations and qualitative 3D reconstruction. The authors also demonstrate 3D vascular reconstruction from power Doppler and photoacoustic data (Dataset 2).
Significance. If the results are taken at face value, the paper provides a useful addition to sensorless freehand 3D ultrasound reconstruction: the proposed components are clearly described, the code is public, and the cross-dataset consistency of performance rankings is a strength. The claimed first demonstration of freehand 3D vascular imaging from PD/PA data is of interest, although it is not quantitatively validated. The main limitations, however, concern validation of the tracking ground truth, statistical reliability of the reported margins, and lack of quantitative assessment for the vascular imaging contribution; these need to be addressed before the central claim of superiority over state-of-the-art methods can be considered fully established.
major comments (4)
- [§4.1.1] The electromagnetic tracker (Polhemus, 60 Hz) and B-mode frames (20 Hz) are not shown to be synchronized. The tracker poses are used both as supervised training targets in Eqs. (7)-(9) and as the evaluation reference for Tables 1, 3, 4 and Figs. 5, 8, so an unknown temporal offset or tracker error would corrupt every reported comparison. Please provide a synchronization procedure (e.g., timestamp alignment, interpolation, calibration) and quantify tracker accuracy under scanning conditions; without this, the improvements over LSTM/RecON (aFE 16.19 vs 21.95 on Dataset 1 and 17.81 vs 19.50 on Dataset 3) cannot be attributed to the proposed model.
- [Tables 1-4] All results are reported as single point estimates on small test sets (18 in-house scans, 5 open-set subjects), with no confidence intervals, error bars, or significance tests. Since several performance margins are modest (e.g., rAE 0.1047 vs 0.1159 for LSTM in Table 1; aFE 17.81 vs 19.50 in Table 4), the claimed ranking could be within run-to-run or subject-level variability. Please add per-scan or bootstrap confidence intervals and, if possible, paired tests across models.
- [§4.5, Figs. 6-7] The power Doppler and photoacoustic 3D vascular reconstructions are presented only qualitatively, with no quantitative metrics (e.g., vessel diameter/geometry error, comparison against tracker-based reconstruction) and no comparison to a baseline. Since the paper states this is the first reported application of this approach, the claim would be strengthened by an objective evaluation; as written, this contribution is an anecdotal demonstration.
- [Eqs. (7) and (9)] The loss function definitions contain ambiguities that affect reproducibility. In Eq. (7), w_i is described as a weighting vector, but it is used as a scalar in the sum over k; specify w_i^k = |Δθ_i^k| + ε or an equivalent per-component weighting. In Eq. (9), the 'margin triplet loss' is written without a margin term, and the text does not specify how anchor, positive, and negative features are selected from the sequence. Since the ablation study in Table 2 attributes part of the gain to these losses, these details should be made precise.
minor comments (6)
- [Throughout] There are numerous typographical and grammatical errors, including 'Meterials' (§4.1), 'Iimmediately' (§4.1.2), 'recongnized' (§4.2), 'contrastve' and 'sequencial' (§4.4.1), 'futher' (§4.4.2), and 'dignostic' (Discussion); a careful proofreading pass is needed.
- [§4.2] The list of compared models omits DCL, although DCL appears in Tables 1 and 4; please add it to the enumeration or clarify that it is an additional baseline.
- [Fig. 9] The text says Fig. 9(a) shows the correlation map from B-mode images and Fig. 9(b) from encoded feature maps, but the caption says the opposite; please correct the mismatch.
- [§4.3] The metrics rAE, aAE, rFE, aFE, Corr, FD, and FDR are described only verbally; providing formulas or explicit references would make the evaluation reproducible.
- [References and Table labels] The label 'Efficient[15]' is confusing because reference [15] is cited as the long-term dependency paper by Li et al.; please clarify which model is actually used and how it relates to EfficientNet.
- [§4.2] Please clarify whether '20,000 epochs' means 20,000 full passes over the training set or 20,000 sampled sequences, given the stated protocol of sampling one sequence per person per epoch.
Circularity Check
No significant circularity found: MoGLo-Net is a supervised model trained and evaluated on held-out external tracker labels, with no fitted input renamed as a prediction.
full rationale
The paper's derivation chain is a conventional supervised learning pipeline: MoGLo-Net takes pairs of B-mode frames and outputs relative 6-DoF motion, trained to minimize a weighted MAE plus correlation and triplet losses against electromagnetic or optical tracker labels (Eqs. 7-10), and evaluated on held-out scans with the same type of tracker labels (Tables 1-4). The correlation volume and global-local attention are learned representations and architectural modules; they are not defined in terms of the output motion, and no fitted parameter is later renamed as a prediction. The custom MMAE reweights the regression loss by ground-truth motion magnitude, but this is a training weighting choice, not a quantity that is subsequently reported as a result. There are no self-citations that carry a load-bearing premise: the cited prior works (Prevost, Guo, Luo, Li) are external, and the open dataset [15] is independently acquired. The only notable risk is that the Polhemus tracker labels at 60 Hz are not explicitly synchronized to the 20 Hz B-mode frames (Section 4.1.1), which threatens label quality and hence the validity of the comparison, but this is an experimental-validity concern, not a circular-derivation concern.
Assumptions & free parameters
free parameters (5)
- Loss weights alpha1, alpha2, alpha3 in Eq (10) =
Not reported
- Smoothing factor epsilon in MMAE (Eq 7) =
Not reported
- Triplet loss margin and distance function (Eq 9) =
Not specified
- Sequence length s =
Not specified
- Correlation operation patch size and RoI size =
Not specified
assumptions (5)
- domain assumption Speckle patterns in B-mode ultrasound are preserved between successive frames and provide information about out-of-plane probe motion.
- domain assumption The Polhemus electromagnetic tracker yields ground-truth motion labels that are accurate and synchronized with the 20 Hz B-mode frames.
- domain assumption The forearm scans in Dataset 1 and Dataset 3 are representative of the clinical freehand PAUS use cases the paper targets.
- ad hoc to paper The global-local attention module improves motion estimation by focusing on fully-developed speckle or high-echogenic tissue regions.
- ad hoc to paper The chosen hyperparameters, including the loss weights and training schedule, lead to a well-converged run for all models under comparison.
Cite this review
Pith. "Pith review of Enhancing Free-hand 3D Photoacoustic and Ultrasound Reconstruction using Deep Learning." pith.science (2026). https://pith.science/paper/J6NB47CD
@misc{pith2026250203505,
author = {Pith},
title = {Pith review of: Enhancing Free-hand 3D Photoacoustic and Ultrasound Reconstruction using Deep Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/J6NB47CD}},
note = {Machine review of arXiv:2502.03505}
}
read the original abstract
This study introduces a motion-based learning network with a global-local self-attention module (MoGLo-Net) to enhance 3D reconstruction in handheld photoacoustic and ultrasound (PAUS) imaging. Standard PAUS imaging is often limited by a narrow field of view and the inability to effectively visualize complex 3D structures. The 3D freehand technique, which aligns sequential 2D images for 3D reconstruction, faces significant challenges in accurate motion estimation without relying on external positional sensors. MoGLo-Net addresses these limitations through an innovative adaptation of the self-attention mechanism, which effectively exploits the critical regions, such as fully-developed speckle area or high-echogenic tissue area within successive ultrasound images to accurately estimate motion parameters. This facilitates the extraction of intricate features from individual frames. Additionally, we designed a patch-wise correlation operation to generate a correlation volume that is highly correlated with the scanning motion. A custom loss function was also developed to ensure robust learning with minimized bias, leveraging the characteristics of the motion parameters. Experimental evaluations demonstrated that MoGLo-Net surpasses current state-of-the-art methods in both quantitative and qualitative performance metrics. Furthermore, we expanded the application of 3D reconstruction technology beyond simple B-mode ultrasound volumes to incorporate Doppler ultrasound and photoacoustic imaging, enabling 3D visualization of vasculature. The source code for this study is publicly available at: https://github.com/guhong3648/US3D
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Ultrasonic technologies in imaging and drug de- livery,
Y .-J. Ho, C.-C. Huang, C.-H. Fan, H.-L. Liu, and C.-K. Yeh, “Ultrasonic technologies in imaging and drug de- livery,” Cellular and Molecular Life Sciences , vol. 78, pp. 6119–6141, 2021
work page 2021
-
[2]
Ultrasonic colour doppler imaging,
D. H. Evans, J. A. Jensen, and M. B. Nielsen, “Ultrasonic colour doppler imaging,” Interface focus , vol. 1, no. 4, pp. 490–502, 2011
work page 2011
-
[3]
Panoramic volumet- ric clinical handheld photoacoustic and ultrasound imag- ing,
C. Lee, S. Cho, D. Lee, J. Lee, J.-I. Park, H.-J. Kim, S. H. Park, W. Choi, U. Kim, and C. Kim, “Panoramic volumet- ric clinical handheld photoacoustic and ultrasound imag- ing,” Photoacoustics, vol. 31, p. 100512, 2023
work page 2023
-
[4]
Freehand 3-d ultrasound imaging: a systematic review,
M. H. Moza ffari and W.-S. Lee, “Freehand 3-d ultrasound imaging: a systematic review,” Ultrasound in medicine & biology, vol. 43, no. 10, pp. 2099–2124, 2017
work page 2017
-
[5]
A. Sorriento, M. B. Porfido, S. Mazzoleni, G. Calvosa, M. Tenucci, G. Ciuti, and P. Dario, “Optical and electro- magnetic tracking systems for biomedical applications: A critical review on potentialities and limitations,”IEEE re- views in biomedical engineering , vol. 13, pp. 212–232, 2019
work page 2019
-
[6]
J.-F. Chen, J. B. Fowlkes, P. L. Carson, and J. M. Ru- bin, “Determination of scan-plane motion using speckle decorrelation: Theoretical considerations and initial test,” International Journal of Imaging Systems and Technol- ogy, vol. 8, no. 1, pp. 38–44, 1997
work page 1997
-
[7]
Automated three-dimensional us frame positioning com- puted from elevational speckle decorrelation.,
T. A. Tuthill, J. Krücker, J. B. Fowlkes, and P. L. Carson, “Automated three-dimensional us frame positioning com- puted from elevational speckle decorrelation.,”Radiology, vol. 209, no. 2, pp. 575–582, 1998
work page 1998
-
[8]
A. H. Gee, R. J. Housden, P. Hassenpflug, G. M. Treece, and R. W. Prager, “Sensorless freehand 3d ultrasound in real tissue: speckle decorrelation without fully developed speckle,” Medical image analysis, vol. 10, no. 2, pp. 137– 149, 2006
work page 2006
Show all 29 references
-
[9]
3d freehand ultrasound without external tracking using deep learning,
R. Prevost, M. Salehi, S. Jagoda, N. Kumar, J. Sprung, A. Ladikos, R. Bauer, O. Zettinig, and W. Wein, “3d freehand ultrasound without external tracking using deep learning,” Medical image analysis, vol. 48, pp. 187–202, 2018
2018
-
[10]
Sensorless free- hand 3d ultrasound reconstruction via deep contextual learning,
H. Guo, S. Xu, B. Wood, and P. Yan, “Sensorless free- hand 3d ultrasound reconstruction via deep contextual learning,” in Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part III...
2020
-
[11]
Self context and shape prior for sensorless freehand 3d ultrasound recon- struction,
M. Luo, X. Yang, X. Huang, Y . Huang, Y . Zou, X. Hu, N. Ravikumar, A. F. Frangi, and D. Ni, “Self context and shape prior for sensorless freehand 3d ultrasound recon- struction,” in Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Con...
2021
-
[12]
Spa- tial position estimation method for 3d ultrasound recon- struction based on hybrid transfomers,
G. Ning, H. Liang, L. Zhou, X. Zhang, and H. Liao, “Spa- tial position estimation method for 3d ultrasound recon- struction based on hybrid transfomers,” in2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI) , pp. 1–5, IEEE, 2022
2022
-
[13]
Pose estimation of 2d ultrasound probe from ultrasound image sequences using cnn and rnn,
K. Miura, K. Ito, T. Aoki, J. Ohmiya, and S. Kondo, “Pose estimation of 2d ultrasound probe from ultrasound image sequences using cnn and rnn,” in Simplifying Medical Ul- trasound: Second International Workshop, ASMUS 2021, Held in Conjunction with MICCAI 2021, Strasbourg, Fra...
2021
-
[14]
Re- con: Online learning for sensorless freehand 3d ultra- 11 sound reconstruction,
M. Luo, X. Yang, H. Wang, H. Dou, X. Hu, Y . Huang, N. Ravikumar, S. Xu, Y . Zhang, Y . Xiong, et al. , “Re- con: Online learning for sensorless freehand 3d ultra- 11 sound reconstruction,” Medical Image Analysis , vol. 87, p. 102810, 2023
2023
-
[15]
Long-term depen- dency for 3d reconstruction of freehand ultrasound with- out external tracker,
Q. Li, Z. Shen, Q. Li, D. C. Barratt, T. Dowrick, M. J. Clarkson, T. Vercauteren, and Y . Hu, “Long-term depen- dency for 3d reconstruction of freehand ultrasound with- out external tracker,” IEEE Transactions on Biomedical Engineering, 2023
2023
-
[16]
Speckle pattern correlation with lateral aperture translation: Ex- perimental results and implications for spatial compound- ing,
G. E. Trahey, S. W. Smith, and O. V on Ramm, “Speckle pattern correlation with lateral aperture translation: Ex- perimental results and implications for spatial compound- ing,” IEEE transactions on ultrasonics, ferroelectrics, and frequency control, vol. 33, no. 3, pp. 257–264, 1986
1986
-
[17]
Ultrasound volume reconstruction from freehand scans without tracking,
H. Guo, H. Chao, S. Xu, B. J. Wood, J. Wang, and P. Yan, “Ultrasound volume reconstruction from freehand scans without tracking,” IEEE Transactions on Biomedical En- gineering, vol. 70, no. 3, pp. 970–979, 2022
2022
-
[18]
Deep mo- tion network for freehand 3d ultrasound reconstruction,
M. Luo, X. Yang, H. Wang, L. Du, and D. Ni, “Deep mo- tion network for freehand 3d ultrasound reconstruction,” in International Conference on Medical Image Comput- ing and Computer-Assisted Intervention , pp. 290–299, Springer, 2022
2022
-
[19]
Multi-imu with online self-consistency for freehand 3d ultrasound recon- struction,
M. Luo, X. Yang, Z. Yan, J. Li, Y . Zhang, J. Chen, X. Hu, J. Qian, J. Cheng, and D. Ni, “Multi-imu with online self-consistency for freehand 3d ultrasound recon- struction,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 342– 35...
2023
-
[20]
Displacement estimation in ultrasound elastography using pyramidal convolutional neural network,
A. K. Tehrani and H. Rivaz, “Displacement estimation in ultrasound elastography using pyramidal convolutional neural network,” IEEE transactions on ultrasonics, ferro- electrics, and frequency control, vol. 67, no. 12, pp. 2629– 2639, 2020
2020
-
[21]
Image- based 3d ultrasound reconstruction with optical flow via pyramid warping network,
Y . Xie, H. Liao, D. Zhang, L. Zhou, and F. Chen, “Image- based 3d ultrasound reconstruction with optical flow via pyramid warping network,” in 2021 43rd Annual Interna- tional Conference of the IEEE Engineering in Medicine& Biology Society (EMBC), pp. 3539–3542, IEEE, 2021
2021
-
[22]
Deep residual learn- ing for image recognition,
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learn- ing for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 770–778, 2016
2016
-
[23]
Glo- ria: A multimodal global-local representation learning framework for label-efficient medical image recognition,
S.-C. Huang, L. Shen, M. P. Lungren, and S. Yeung, “Glo- ria: A multimodal global-local representation learning framework for label-efficient medical image recognition,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3942–3951, 2021
2021
-
[24]
Gla: Global–local attention for image description,
L. Li, S. Tang, Y . Zhang, L. Deng, and Q. Tian, “Gla: Global–local attention for image description,”IEEE Transactions on Multimedia, vol. 20, no. 3, pp. 726–737, 2017
2017
-
[25]
Global-local attention for emotion recognition,
N. Le, K. Nguyen, A. Nguyen, and B. Le, “Global-local attention for emotion recognition,”Neural Computing and Applications, vol. 34, no. 24, pp. 21625–21639, 2022
2022
-
[26]
Facenet: A unified embedding for face recognition and clustering,
F. Schro ff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815–823, 2015
2015
-
[27]
Digital image enhancement and noise filtering by use of local statistics,
J.-S. Lee, “Digital image enhancement and noise filtering by use of local statistics,” IEEE transactions on pattern analysis and machine intelligence , no. 2, pp. 165–168, 1980
1980
-
[28]
Sparse 2-d array design for real time rectilinear volumetric imaging,
J. T. Yen, J. P. Steinberg, and S. W. Smith, “Sparse 2-d array design for real time rectilinear volumetric imaging,” IEEE transactions on ultrasonics, ferroelectrics, and fre- quency control, vol. 47, no. 1, pp. 93–110, 2000
2000
-
[29]
Vascular imaging with a three-dimensional power doppler system,
D. DB, “Vascular imaging with a three-dimensional power doppler system,” AJR Am J Roentgenol , vol. 165, pp. 665–668, 1995
1995
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.