REVIEW 4 major objections 7 minor 18 references
Representation Disentanglement for Multi-task Learning with application to Fetal Ultrasound
T0 review · 4 major / 7 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A multi-task adversarial training scheme that disentangles latent features improves generalization to unseen combinations of image properties, such as acoustic shadows over fetal anatomy.
desk verdict Useful incremental result with a striking synthetic OOD demo, but the clinical headline lacks error bars and repeatability; worth reviewing with fair scrutiny. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
A two-encoder adversarial architecture with two classification heads. Encoder A maps the image to $Z_A$, which a classifier uses to predict task A; an adversarial classifier tries to predict task B from $Z_A$. Encoder B and its adversary play the mirror game for task B. The joint objective is $\min_{\{\theta_A,\theta_B,\varphi_A,\varphi_B\}}(L_A + L_B - \lambda (L_{\text{adv}}^A + L_{\text{adv}}^B))$ with adversaries minimizing $L_{\text{adv}}^A + L_{\text{adv}}^B$, so the encoders must shed information that would reveal the other task while keeping information needed for their own task. Six residual blocks form each encoder, and the adversarial pressure is applied through dense layers that output softmax predictions of the irrelevant labels. The paper observes that disentanglement emerges at the final dense layer of the classifiers rather than in earlier layers.
What would settle it
Train the same architecture but replace the adversarial losses with simple augmentation that adds synthetic shadows to the training set; if the augmented baseline matches the proposed model's 73.68% accuracy on the held-out shadowed LVOT view, the reported generalization gain is not attributable to disentanglement. More directly, evaluate on a held-out combination of standard plane with a different artifact type, such as motion blur or gain dropout: if the disentangled model does not beat the no-adversary baseline, the claim that it generalizes to previously unseen artifact properties is falsified.
Extended reading notes
Core claim
The central claim is that imperfect disentanglement of latent representations—separating image content into a feature set for the anatomy and a feature set for the artifact—is enough to make a multi-task network generalize to new combinations of previously seen properties. The paper argues that ordinary CNNs on fetal ultrasound fail on shadowed images because they encode anatomy and shadow correlations together; when a standard plane and a shadow pattern appear in a combination not seen in training, the anatomical classifier has no reliable cue. The proposed training objective, a min-max game between encoders/classifiers and adversarial classifiers, penalizes the mutual information between each feature set and the irrelevant task's labels, so the anatomical feature set stops depending on the accidental shadow pattern. The experiments show the anatomical feature set $Z_A$ classifies standard planes well (94.44% overall accuracy on the held-out test set) but is poor at predicting shadows, while the shadow feature set $Z_B$ is poor at predicting standard planes, exactly the intended division.
Load-bearing premise
The method assumes the two tasks correspond to two independent generative factors that can be cleanly separated into the two feature sets, and that the adversarial competition actually drives the mutual information between each feature set and the other task's labels toward zero; the paper concedes perfect disentanglement is likely infeasible.
Editorial extensions
If this is right
- Standard-plane classification in fetal ultrasound becomes robust to acoustic shadows, so clinical screening models need far fewer shadowed example images for every anatomical plane.
- The same adversarial recipe can be transferred to any pair of co-occurring image properties, such as lesion texture versus surrounding tissue appearance, provided the two factors can be separated.
- Because the shadow feature set $Z_B$ is learned rather than discarded, the framework produces a shadow-artifact detector as a by-product alongside the anatomical classifier.
- Imperfect disentanglement suffices for practical gains, so the method remains usable in settings where the two generative factors are not truly independent.
- In principle the architecture scales to more than two tasks by adding one encoder and one adversary per additional task.
Reading between the lines
- If the disentangled shadow features truly discard anatomy, they could be reused for shadow detection, image-quality scoring, or shadow-corrected reconstruction—uses the paper does not explore.
- The same mechanism may serve as a form of adversarial domain generalization: treating scanner vendor or patient population as task B could make a diagnostic model invariant to acquisition differences without extra domain labels.
- A natural next experiment is to replace the softmax adversaries with a mutual-information estimator or gradient-reversal layer; comparing the residual accuracy on irrelevant tasks would show how much of the effect comes from the specific adversarial game versus the general principle.
- The synthetic circle/rectangle benchmark with fully known ground-truth factors could become a standard sanity check for disentanglement research, since it cleanly separates the two factors and exposes a 99% vs 10% generalization gap.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an adversarial multi-task representation disentanglement method for two classification tasks. Two encoders produce feature sets ZA and ZB; each feature set is trained to maximize accuracy on its own task via a cross-entropy loss and, through an adversarial classifier, to minimize informativeness about the other task. The training objective combines the task losses with an adversarial term weighted by λ (Eq. 1), with a separate minimization for the adversarial networks (Eq. 2). The method is evaluated on synthetic circle/rectangle images with background color as the second task, and on fetal ultrasound data where the tasks are standard-plane classification and acoustic-shadow artifact classification. The main claimed results are 99% accuracy on an out-of-distribution synthetic combination (circles on black background) versus 10% for a no-adversarial baseline, and 73.68% accuracy on fetal LVOT images with shadows, a shadow/plane combination unseen in training, compared with about 35 to 38% for comparison methods. The authors acknowledge that perfect disentanglement is likely infeasible, and they interpret their results as evidence that imperfect disentanglement can still improve out-of-distribution generalization.
Significance. If the results hold, the paper offers a practical and cleanly motivated way to improve robustness to unseen combinations of image factors in medical imaging, where artifacts such as acoustic shadows frequently confound classifiers. The synthetic experiment is a useful, falsifiable proof of concept: the jump from 10% to 99% on an unseen property combination is striking and directly supports the mechanism the authors claim. The fetal ultrasound application is clinically relevant, and the idea of separating anatomy from artifact information is well aligned with an important practical need. The main weaknesses are empirical: the headline fetal results are single-run, single-split point estimates with no uncertainty quantification, λ is tuned per data set without a sensitivity analysis, and baseline architectures are not explicitly matched to the proposed architecture. The paper does not provide code, random seed details, or repeated experiments, so the reproducibility of the headline numbers cannot be assessed from the manuscript alone.
major comments (4)
- [Section 3, Table 2] The central claim that the proposed method achieves 73.68% accuracy on LVOT(W S), roughly 36 percentage points above the comparison methods, rests on a single training run and a single fixed data split. No confidence intervals, standard deviations, or repeated-seed results are reported in the paper. Because the LVOT(W S) test set has only 418 images, the large gap could be due to an unusually favorable or unfavorable initialization or to the specific split. The authors should report mean and standard deviation over at least five random seeds for every row of Table 2, or provide bootstrap confidence intervals or a paired statistical test such as McNemar's test.
- [Section 2, Eq. (1), and Section 3] The adversarial weight λ is chosen heuristically and independently for each data set using validation data, but no sensitivity analysis is presented. If the 36-point gain on LVOT(W S) occurs only in a narrow λ range, the paper's attribution of the improvement to the adversarial disentanglement term is unsupported. Please report validation and LVOT(W S) accuracies, together with the irrelevant-task accuracies ZA→TB and ZB→TA, as functions of λ over a range that includes 0.01 and 0.1.
- [Section 3, 'Evaluation approach' and Table 2] The comparison methods are not explicitly matched in architecture to the proposed method. The proposed method uses a ClsA with three dense layers of 256/256/3 units, while the descriptions of 'Std plane only', 'Artifacts only', and 'Proposed w/o adv' do not state whether they use the same classifier sizes. If the baselines use smaller classifiers, part of the accuracy differences in Table 2 could be due to capacity rather than to disentanglement. The authors should state explicitly that all comparison methods use identical encoder and classifier architectures, differing only in the presence of the adversarial branch or the auxiliary task head.
- [Section 3, Table 2 and Discussion] The evidence for the causal role of disentanglement is indirect and the measured disentanglement is only partial: on Test seen, the proposed method still obtains 72.57% for ZA→TB and 64.35% for ZB→TA, and the paper itself states that perfect disentanglement is 'likely infeasible'. Since the headline generalization benefit is attributed to disentanglement, a more direct control is needed. In particular, replacing the adversarial branch with a non-adversarial auxiliary classifier or another regularization penalty with the same capacity would test whether the LVOT(W S) advantage comes specifically from minimizing the mutual information with the irrelevant task, rather than from a generic regularizing effect of the additional network.
minor comments (7)
- [Section 2] There is a typo in 'We define the the cost functions', which should read 'We define the cost functions'.
- [Section 2] The notation 'Yi B = 0 and Yi B = 1' in the fetal ultrasound setting is inconsistent with the earlier subscript style; it should be written as y_i^B = 0 and y_i^B = 1 or equivalently Y_B ∈ {0,1}.
- [Table 1] The header 'Train Validation Test seen LVOT(W S) Artifacts(OTHS)' and the parenthetical counts such as 'W/O S (W S)' are difficult to parse; please rewrite the table so that column names and cell counts are unambiguous, for example by listing W/O S and W S as separate rows.
- [Figure 2] The training framework figure is very small and the labels for the adversarial branches and the loss terms are hard to read at normal print size; a larger version or a vector figure would improve readability.
- [Section 3, synthetic data] The sentence 'To keep the balance between image properties in the training split, we use circle:rectangle=1:1 and black:white=7:5' appears contradictory, since 7:5 is not a balanced ratio; please clarify whether the 7:5 ratio refers to the training distribution rather than to a balanced design.
- [Section 3] The Discussion makes a qualitative observation that disentanglement occurs only in the last dense layer, but no quantitative measure is provided; a simple metric of cluster separation or an information-based estimate would make the claim testable.
- [General] The manuscript does not provide code, trained models, or a precise description of random seeds and initialization, which are needed to reproduce the reported accuracies; including these details or a public repository would strengthen the paper.
Circularity Check
No significant circularity: the held-out generalization tests are independent of the adversarial training objective and the fitted trade-off parameter.
full rationale
The paper's derivation chain has no load-bearing step that reduces to its own input. The central claim is that the adversarial objective in Eqs. (1)-(2) produces features which generalize to previously unseen combinations of entangled properties. This is tested on held-out data sets that are not used to fit any parameter: synthetic circles-on-black images and fetal ultrasound LVOT with shadows (LVOT(W S)) as well as other standard planes with artifacts (Artifacts(OTHS)). The hyperparameter lambda is chosen on validation data with the same distribution as the training data, not on the held-out test sets, so the reported improvements are not fitted-input-called-prediction. The diagnostic evidence for disentanglement (Table 2 'Proposed irr task' and PCA in Fig. 3) measures how well the learned features perform on irrelevant tasks using the trained adversarial classifiers; this is an optimization check rather than an external prediction, and the paper does not rest its generalization claim on this diagnostic alone. The cited prior work by the same authors ([2] on shadow confidence maps, [17] DLTK for residual blocks) supplies implementation and related-work context, but it is not the basis of the main result. The paper itself acknowledges that perfect disentanglement is 'likely infeasible,' which is a limitation statement rather than an admission of circularity. The residual robustness concern is that results are single-run and lack confidence intervals, but that is an empirical-evidence weakness, not a circularity in the derivation. No equation is used both as assumption and conclusion, and no prediction is equivalent by construction to a fitted parameter.
Assumptions & free parameters
free parameters (1)
- lambda (adversarial regularization weight) =
0.01 (synthetic), 0.1 (fetal US)
assumptions (3)
- domain assumption The data generating process separates into two independent factor sets corresponding exactly to the two tasks.
- domain assumption The adversarial training competition converges to a state where encoded features are minimally informative about irrelevant tasks.
- standard math Standard deep learning training assumptions: the network architectures, optimization, and data splits are sufficient for learning.
Cite this review
Pith. "Pith review of Representation Disentanglement for Multi-task Learning with application to Fetal Ultrasound." pith.science (2026). https://pith.science/paper/Y5B4DKSB
@misc{pith2026190807885,
author = {Pith},
title = {Pith review of: Representation Disentanglement for Multi-task Learning with application to Fetal Ultrasound},
year = {2026},
howpublished = {\url{https://pith.science/paper/Y5B4DKSB}},
note = {Machine review of arXiv:1908.07885}
}
read the original abstract
One of the biggest challenges for deep learning algorithms in medical image analysis is the indiscriminate mixing of image properties, e.g. artifacts and anatomy. These entangled image properties lead to a semantically redundant feature encoding for the relevant task and thus lead to poor generalization of deep learning algorithms. In this paper we propose a novel representation disentanglement method to extract semantically meaningful and generalizable features for different tasks within a multi-task learning framework. Deep neural networks are utilized to ensure that the encoded features are maximally informative with respect to relevant tasks, while an adversarial regularization encourages these features to be disentangled and minimally informative about irrelevant tasks. We aim to use the disentangled representations to generalize the applicability of deep neural networks. We demonstrate the advantages of the proposed method on synthetic data as well as fetal ultrasound images. Our experiments illustrate that our method is capable of learning disentangled internal representations. It outperforms baseline methods in multiple tasks, especially on images with new properties, e.g. previously unseen artifacts in fetal ultrasound.
Figures
Reference graph
Works this paper leans on
-
[1]
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Fe- lix A. Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv:1811.12231, 2018
arXiv 2018
-
[2]
Weakly supervised estimation of shadow confidence maps in fetal ultrasound imaging
Qingjie Meng, Matthew Sinclair, Veronika Zimmer, Benjamin Hou, Mar- tin Rajchl, Nicolas Toussaint, Ozan Oktay, Jo Schlemper, Alberto Gomez, James Housden, Jacqueline Matthew, Daniel Rueckert, Julia A Schnabel, and Bernhard Kainz. Weakly supervised estimation of shadow confidence maps in fetal ultrasound imaging. IEEE transactions on medical imaging ,
-
[3]
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. CoRR, arXiv/1802.05983, 2018
arXiv 2018
-
[4]
A. Hyv¨ arinen and E. Oja. Independent component analysis: Algorithms and applications. Neural Netw., 13(4-5):411–430, May 2000. ISSN 0893-6080
work page 2000
-
[5]
Joshua B. Tenenbaum and William T. Freeman. Separating style and con- tent with bilinear models. Neural Comput. , 12(6):1247–1283, June 2000. ISSN 0899-7667
work page 2000
-
[6]
Infogan: Interpretable representation learning by information maximizing generative adversarial nets
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In NeurIPS’16, pages 2180–2188, USA, 2016. Curran Associates Inc. ISBN 978-1-5108-3881-9
work page 2016
-
[7]
beta-vae: Learning basic visual concepts with a constrained variational framework
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR’17, 2017
work page 2017
-
[8]
Christopher P. Burgess, Irina Higgins, Arka Pal, Lo¨ ıc Matthey, Nick Wat- ters, Guillaume Desjardins, and Alexander Lerchner. Understanding disen- tangling in β-vae. arXiv:1804.03599, 2018
arXiv 2018
Show all 18 references
-
[9]
Representation learn- ing: A review and new perspectives
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learn- ing: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35(8):1798–1828, August 2013. ISSN 0162-8828
2013
-
[10]
Variational lossy autoencoder
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhari- wal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. In ICLR’17, 2017
2017
-
[11]
Image-to- image translation for cross-domain disentanglement
Abel Gonzalez-Garcia, Joost van de Weijer, and Yoshua Bengio. Image-to- image translation for cross-domain disentanglement. In NeurIPS’18, pages 1287–1298. Curran Associates, Inc., 2018
2018
-
[12]
Liu, Yen-Cheng Liu, Yu-Ying Yeh, and Yu-Chiang Frank Wang
Alexander H. Liu, Yen-Cheng Liu, Yu-Ying Yeh, and Yu-Chiang Frank Wang. A unified feature disentangler for multi-domain image translation and manipulation. In NeurIPS, pages 2590–2599. Curran Associates, Inc., 2018. 10 Qingjie Meng et al
2018
-
[13]
A two-step disentanglement method
Naama Hadad, Lior Wolf, and Moni Shahar. A two-step disentanglement method. In CVPR’18, 2018
2018
-
[14]
Disentangling factors of variation in deep representations using adversarial training
Michael F Mathieu, Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann, and Yann LeCun. Disentangling factors of variation in deep representations using adversarial training. In NeurIPS’16, pages 5040–5048, 2016
2016
-
[15]
Unsupervised domain adaptation in brain lesion segmentation with adversarial networks
Konstantinos Kamnitsas, Christian Baumgartner, Christian Ledig, Virginia Newcombe, Joanna Simpson, Andrew Kane, David Menon, Aditya Nori, An- tonio Criminisi, Daniel Rueckert, and Ben Glocker. Unsupervised domain adaptation in brain lesion segmentation with adversarial network...
2017
-
[16]
Im- proving CNN training using disentanglement for liver lesion classification in CT
Avi Ben-Cohen, Roey Mechrez, Noa Yedidia, and Hayit Greenspan. Im- proving CNN training using disentanglement for liver lesion classification in CT. arXiv:1811.00501, 2018
2018 arXiv
-
[17]
Ira Ktena, Matthew Lee, Bernhard Kainz, Daniel Rueck- ert, Ben Glocker, and Martin Rajchl
Nick Pawlowski, S. Ira Ktena, Matthew Lee, Bernhard Kainz, Daniel Rueck- ert, Ben Glocker, and Martin Rajchl. Dltk: State of the art reference imple- mentations for deep learning on medical images. arXiv:1711.06853, 2017
2017 arXiv
-
[18]
Fetal anomaly screening programme: programme handbook June 2015
NHS. Fetal anomaly screening programme: programme handbook June 2015. Public Health England, 2015
2015
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.