REVIEW 2 major objections 4 minor 1 cited by
Supervised Contrastive Block Disentanglement
T0 review · 2 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper claims that a single contrastive invariance loss, scaled by one hyperparameter α, can block-disentangle content from environment and beat domain-generalization baselines on CMNIST and Camelyon17-WILDS while removing batch…
desk verdict Genuinely new contrastive invariance loss, but the domain-generalization superiority claim needs a protocol-fair alpha selection; the batch-correction result is the stronger evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the invariance loss $L^{\mathrm{inv}}_{z_c,e}$, built from the same softmax matching probabilities as supervised contrastive learning. For each anchor, it compares the summed log-probabilities of matching zc to same-environment pairs versus different-environment pairs and takes the absolute difference; minimizing it drives the two sums together, which prevents any linear readout of e from zc. It shares the spirit of adversarial domain discrimination but removes the minimax optimization, and it is combined with two supervised-contrastive terms, one clustering zc by y and one clustering zs by e (or by the pair (y,e)), plus an optional additive decoder that lets the user swap zc or zs to inspect counterfactual images.
What would settle it
A direct check would be to select α by validation accuracy on a held-out environment (or on the training hospital split) and then compare that SCBD checkpoint with ERM under the same selection rule; if SCBD at the selected α no longer beats ERM on Camelyon17-WILDS, the reported superiority depends on the test-informed choice α=192 rather than on the invariance loss itself.
Extended reading notes
Core claim
The central claim is that the invariance of the content embedding zc to the environment e can be enforced by a purely supervised-contrastive term rather than by adversarial training or variational regularization. The loss treats same-environment pairs of zc as positives and different-environment pairs as negatives, then minimizes the absolute gap between their summed matching log-probabilities; this makes the environment unpredictable from zc while the two supervised contrastive terms keep zc informative about y and zs informative about e. The paper reports that α=192 achieves 82.9±12.1% test accuracy on CMNIST and 72.7±3.0% on Camelyon17-WILDS, above all listed baselines, and that α=1 on the batch-correction task outperforms the baselines on both measures of biological content and of residual sensitivity to wells.
Load-bearing premise
The reported gains assume α can be picked without seeing the target distribution, yet the chosen α=192 is exactly what a validation-set protocol would not select, since the paper states that optimizing α during model selection would pick α=0.
Editorial extensions
If this is right
- Domain generalization gains a single-knob method: raising α monotonically trades in-distribution accuracy for out-of-distribution accuracy on datasets where the two are negatively correlated.
- Batch correction can reuse the same objective: zc preserves perturbation-related biology while the ability to predict the well from zc drops, which is what the paper demonstrates on 26 million cell images.
- The absence of a discriminator and of variational bounds removes two common failure modes, minimax instability and posterior collapse, making invariance training more stable with standard image encoders.
- On datasets where in-distribution and out-of-distribution performance are positively correlated, the paper's own experiments on PACS and VLCS show SCBD has no advantage over ERM, so the method's applicability is conditional.
- The optional reconstruction view, by swapping zc and zs, gives a direct check that the two blocks carry distinct information.
Reading between the lines
- Beyond the paper: the same α-controlled invariance loss could be applied to a single contrastive embedding (with y as the target) as a drop-in debiasing regularizer for pretrained features, though the paper only studies the two-block setting.
- A testable extension the paper leaves implicit is to choose α by leave-one-environment-out validation, for example training on four hospitals and picking α on a fifth, to check whether SCBD still beats ERM under a fully target-free selection rule.
- Because the invariance loss is symmetric in the pair labels, it may be reusable to enforce invariance to any categorical nuisance variable such as donor, scanner, or stain whenever such labels exist, not only the well or hospital labels used here.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Supervised Contrastive Block Disentanglement (SCBD), a method that learns two embeddings, zc and zs, from observations x. A supervised contrastive loss clusters zc by target y, a second supervised contrastive loss clusters zs by environment e, and a new invariance loss is added to make zc invariant to e; an optional additive decoder provides reconstruction for qualitative inspection. The method is evaluated on Colored MNIST and Camelyon17-WILDS for domain generalization and on a 26-million-cell Optical Pooled Screening dataset for batch correction. The headline empirical claims are that SCBD with alpha=192 outperforms all baselines on both domain-generalization datasets and that SCBD removes inter-well batch effects while preserving biological signal.
Significance. Block disentanglement with a purely contrastive objective is a potentially useful contribution, and the paper is transparent about the central difficulty of tuning alpha: it explicitly states that validation-based model selection would choose alpha=0. The algorithm is simple, the code is provided in Appendix B, and the Optical Pooled Screening experiments are on a large real-world dataset. If the empirical claims held under a protocol-fair comparison, the method would be practically valuable. However, the main quantitative support for domain generalization is not protocol-fair, and the batch-effect metric largely measures the property that the loss already optimizes, so the strength of the stated claims is currently not commensurate with the evidence.
major comments (2)
- [Section 4.1.6 / Table 1 / Appendix Table 5] The central claim that 'SCBD with alpha=192 significantly outperforms all baselines on both CMNIST and Camelyon17-WILDS' is not established by a protocol-fair comparison. The paper states that alpha is not optimized during model selection because that would choose alpha=0, and then reports the alpha=192 row as the headline result; thus the reported SCBD test accuracy is selected with access to the test distribution, while all baseline hyperparameters are selected on the in-distribution validation set. Under the paper's own validation-selection rule, alpha=0 is selected and SCBD achieves 25.5±3.0 on CMNIST and 61.9±3.8 on Camelyon17, below ERM (37.8±2.6 and 65.8±4.9, respectively). In addition, Appendix Table 5 restricts all non-SCBD baselines to batch size 32, while SCBD uses batch size 2048, which is a further confound and is particularly relevant for contrastive methods. The comparison should be rerun under a common model-selection protocol, or the claims should be explicitly reframed as an oracle-alpha analysis.
- [Section 4.2.3 / Figure 4] The well-F1 metric trains a linear classifier on zc to predict the well label e, which is essentially the same property that the invariance loss Linv is explicitly designed to eliminate. A low well-F1 for SCBD is therefore partly by construction and does not provide independent evidence of batch correction; the CORUM prediction result is the independent part of the evaluation. The authors should state this caveat and, ideally, add a metric that is not directly aligned with the training objective, such as the degree of well mixing in a nearest-neighbor graph or the preservation of within-well perturbation structure.
minor comments (4)
- [Section 3] The sentence 'Our approach differs from theirs because we do not use contrastive learning' contradicts the paper's central claim that SCBD is based purely on supervised contrastive learning; presumably the intended contrast is that SCBD does not use a variational autoencoder framework.
- [Section 2 / Appendix B] Equation (2) defines the invariance loss with sums over positive and negative pairs, but the code in Appendix B computes means over these sets and then averages over the batch; the notation should be aligned with the implementation.
- [Table 1] The word 'significantly' is used without a statistical significance test; given the large standard deviation on CMNIST (82.9 ± 12.1), the authors should report confidence intervals or a paired test across the ten seeds.
- [Figure 3] The text describes the relationship as 'monotonic', but only four values of alpha are tested; it would be clearer to say that the trade-off is observed over the tested values, rather than claiming monotonicity in general.
Circularity Check
Table 1's claimed superiority over baselines rests on test-selected α, which the paper's own selection rule would reject (α=0), and the batch-effect F1 metric is the direct objective being optimized; the domain-generalization claim is therefore partially circular.
-
fitted input called prediction
[Section 4.1.6, Table 1]
"For SCBD, we apply the same model selection procedure to optimize the learning rate and weight decay. We do not optimize α during model selection, since this would result in choosing α = 0. We report the test accuracy for α = 0 and α = 192 as evidence that the invariance loss in SCBD is effective at removing spurious correlations and improving out-of-distribution performance. With α = 192, SCBD significantly outperforms all baseline algorithms across both datasets. Tuning α corresponds to model selection with respect to an unknown test distribution, which is a difficult open problem."
The headline claim 'SCBD significantly outperforms all baselines' is reported for α=192, a value the paper itself says would not be selected by its own model-selection rule, which picks α=0. Baseline hyperparameters are selected on the in-distribution validation set, while α=192 is chosen with knowledge of the test distribution; the paper even labels this 'model selection with respect to an unknown test distribution.' Under the paper's own protocol, SCBD at α=0 scores 25.5±3.0 on CMNIST and 61.9±3.8 on Camelyon17-WILDS, below ERM's 37.8±2.6 and 65.8±4.9.
-
other
[Section 2 (Eq. 1, invariance loss) and Section 4.2.1/4.2.3 (batch-correction evaluation)]
"Since our goal is to make zc invariant to e, we optimize zc to make this classifier fail. We do this by minimizing Linv zc,e = |Σp∈Pe(i) log P(M p i,c = 1) − Σn∈Ne(i) log P(M n i,c = 1)|, which makes it equally probable that zic is matched with its positive and negative pairs with respect to e. ... The second task is to use the perturbation embeddings to predict e, which measures the sensitivity to inter-well batch effects. We fit a linear classifier on top of each of the embeddings, and compute the F1 score. In contrast to the first task, we want the performance on this task to be weak."
The invariance loss directly optimizes zc so that a softmax classifier over pairs cannot distinguish same-e from different-e pairs, i.e., it makes e uninformative from zc. The well-F1 metric then fits a linear classifier on zc to predict e. The low F1 of SCBD is therefore the optimization objective itself, not an independent measurement of batch-effect removal. Comparing SCBD to baselines on this metric largely restates that only SCBD includes the invariance term. The CORUM AUPRC result is independent and supports the method, but the 'less sensitive to inter-well batch effects' claim is, for this metric, true by construction.
full rationale
The core derivation of SCBD—the SCL clustering terms, the invariance loss, and the reconstruction term—is not circular: each objective is defined from the data labels and the loss equations, and no term is secretly defined in terms of the headline results. The qualitative CMNIST swap results and the CORUM biological-content evaluation provide independent evidence. However, the paper's central empirical claim of domain-generalization superiority (abstract, Section 4.1.6, Table 1) is circular in the specific sense of a fitted input called a prediction. The paper states that α is not optimized during model selection because doing so would yield α=0, and then reports α=192 as the headline value. Since all baseline hyperparameters were selected on the validation set, the comparison 'SCBD with α=192 significantly outperforms all baselines' is a comparison between a test-oracle-selected model and validation-selected baselines. Following the paper's own selection rule (α=0) makes SCBD worse than ERM on both datasets, so the claimed superiority is not a prediction of the method but an artifact of reporting the best test grid point. The batch-correction well-F1 result is similarly aligned by construction with the invariance loss—the loss is designed to make a classifier over e fail on zc, and the metric measures exactly that classifier's F1—so the low F1 is a directly optimized outcome rather than an emergent discovery. The CORUM AUPRC result is independent and supports the method, which is why the score is 6 rather than 8. There is no load-bearing self-citation chain; citations to prior work by the authors (e.g., Makino et al. 2022a, Wang et al. 2023) are for baselines or acknowledged limitations and do not force the central result.
Assumptions & free parameters
free parameters (5)
- alpha (invariance strength) =
192 (domain generalization headline), 1 (batch correction)
- softmax temperature tau =
0.1
- batch size =
2048 (SCBD)
- embedding dimensions D_zc, D_zs =
128 (domain generalization), 64 (batch correction)
- learning rate and weight decay =
1e-4 and 0.01
assumptions (6)
- domain assumption The data are generated by a process where x = f(zc, zs), with zc capturing y-correlated, e-invariant variation and zs capturing e-correlated variation.
- domain assumption The environment label e is observed and exhaustively captures all spurious variation.
- domain assumption The chosen datasets exhibit a negative correlation between in- and out-of-distribution performance.
- ad hoc to paper Equalizing positive and negative SCL log-probabilities with respect to e is sufficient to make zc invariant to e.
- ad hoc to paper The additive decoder log p(x|zc,zs) = Decc(zc) + Decs(zs) is a valid model for reconstruction.
- standard math Standard probability and optimization machinery (softmax, log-softmax, stochastic gradient training) is valid.
Cite this review
Pith. "Pith review of Supervised Contrastive Block Disentanglement." pith.science (2026). https://pith.science/paper/FXQZLK6W
@misc{pith2026250207281,
author = {Pith},
title = {Pith review of: Supervised Contrastive Block Disentanglement},
year = {2026},
howpublished = {\url{https://pith.science/paper/FXQZLK6W}},
note = {Machine review of arXiv:2502.07281}
}
abstract
Real-world datasets often combine data collected under different experimental conditions. This yields larger datasets, but also introduces spurious correlations that make it difficult to model the phenomena of interest. We address this by learning two embeddings to independently represent the phenomena of interest and the spurious correlations. The embedding representing the phenomena of interest is correlated with the target variable $y$, and is invariant to the environment variable $e$. In contrast, the embedding representing the spurious correlations is correlated with $e$. The invariance to $e$ is difficult to achieve on real-world datasets. Our primary contribution is an algorithm called Supervised Contrastive Block Disentanglement (SCBD) that effectively enforces this invariance. It is based purely on Supervised Contrastive Learning, and applies to real-world data better than existing approaches. We empirically validate SCBD on two challenging problems. The first problem is domain generalization, where we achieve strong performance on a synthetic dataset, as well as on Camelyon17-WILDS. We introduce a single hyperparameter $\alpha$ to control the degree of invariance to $e$. When we increase $\alpha$ to strengthen the degree of invariance, out-of-distribution performance improves at the expense of in-distribution performance. The second problem is batch correction, in which we apply SCBD to preserve biological signal and remove inter-well batch effects when modeling single-cell perturbations from 26 million Optical Pooled Screening images.
Figures
Figures from the paper (21 more)
Forward citations
Cited by 1 Pith paper
-
XFACTORS: Disentangled Information Bottleneck via Contrastive Supervision
XFACTORS separates latent factors into per-factor subspaces with InfoNCE supervision, achieving near-perfect FactorVAE scores on synthetic benchmarks and qualitative factor swapping on CelebA.
Reference graph
Works this paper leans on
-
[1]
A.1.2. V ARIATIONAL AUTOENCODERS For our experiments with V AE-based approaches, we use the same experimental setup used in Wang et al. (2023), including the architecture and hyperparameters. We resize the images to 64 × 64 pixels and use a batch size of
work page 2023
-
[3]
11 Supervised Contrastive Block Disentanglement Sun, B. and Saenko, K. Deep coral: Correlation alignment for deep domain adaptation. ECCV 2016 Workshops ,
work page 2016
-
[4]
12 Supervised Contrastive Block Disentanglement Appendix A. Experiments A.1. Experimental setup All of our experiments were done using a single NVIDIA A100 GPU on our institutions’ high-performance computing clusters. A.1.1. S UPERVISED CONTRASTIVE BLOCK DISENTANGLEMENT Our experimental setup for SCBD is remarkably similar across all of our experiments, w...
work page 2020
-
[9]
optimizer with 1 × 10−4 learning rate and 0.01 weight decay. We additionally skip gradients with a norm above 1 × 1012, and clip gradients with a norm above 1 × 106, as done in Child (2021). We train for a maximum of 50,000 steps for domain generalization, and three epochs for batch correction, and select the weights with minimum validation loss. We repor...
work page 2021
-
[11]
The results are generally better for larger batch sizes, which was also observed by the authors of SCL (Khosla et al., 2020). 16 Supervised Contrastive Block Disentanglement 0 0.001 0.01 Weight decay 0.92 0.94 0.96Val accuracy 0 0.001 0.01 Weight decay 0.8 0.9T est accuracy Val T est Figure 10: CMNIST weight decay ablation study for SCBD with ResNet-18 en...
work page 2020
-
[12]
The results are relatively consistent across different degrees of weight decay. A.2.2. C AMELYON 17-WILDS 0.96 0.98 Val accuracy 0.60 0.65 0.70 0.75T est accuracy Figure 11: In- and out-of-distribution performance are negatively correlated on Camelyon17-WILDS. This is consistent with Teney et al. (2024), and therefore this dataset satisfies the assumption...
work page 2024
-
[13]
The results are relatively consistent across different embedding sizes. 512 1024 2048 4096 Batch size 0.91 0.92Val accuracy 512 1024 2048 4096 Batch size 0.65 0.70 0.75T est accuracy Val T est Figure 15: Camelyon17-WILDS batch size ablation study for SCBD with DenseNet-121 encoders and α =
-
[14]
The results are generally better for larger batch sizes, which was also observed by the authors of SCL (Khosla et al., 2020). 18 Supervised Contrastive Block Disentanglement 0 0.001 0.01 Weight decay 0.915 0.920Val accuracy 0 0.001 0.01 Weight decay 0.65 0.70 0.75T est accuracy Val T est Figure 16: Camelyon17-WILDS weight decay ablation study for SCBD wit...
work page 2020
Show all 15 references
-
[15]
Batch correction CORUM prediction Our CORUM prediction task mirrors that of (Wang et al., 2023), with some modifications to ensure a fair comparison with CellProfiler
94.7 ± 1.8 67 .1 ± 1.3 74 .7 ± 3.0 71 .4 ± 1.3 77 .0 ERM 97.6 ± 1.0 63 .3 ± 0.9 72 .2 ± 0.5 76 .4 ± 1.5 77 .4 CORAL 98.8 ± 0.1 64 .6 ± 0.8 71 .7 ± 1.4 75 .8 ± 0.4 77 .7 DANN 98.5 ± 0.2 64 .9 ± 1.1 73 .1 ± 0.7 78 .3 ± 0.3 78 .7 IRM 97.6 ± 0.3 65 .0 ± 0.9 72 .2 ± 0.5 76 .4 ± 1.5...
2023
-
[192]
The results are relatively consistent across different embedding sizes. 512 1024 2048 4096 Batch size 0.6 0.8 1.0Val accuracy 512 1024 2048 4096 Batch size 0.4 0.6 0.8 1.0T est accuracy Val T est Figure 9: CMNIST batch size ablation study for SCBD with ResNet-18 encoders and α =
-
[1986]
Our decoder Dec(zc, zs) architecture is shown in Appendix Table 2, with GELU activations (Hendrycks & Gimpel,
with hidden sizes of Drc and Drs, and GELU activations (Hendrycks & Gimpel, 2016). Our decoder Dec(zc, zs) architecture is shown in Appendix Table 2, with GELU activations (Hendrycks & Gimpel,
2016
-
[2016]
We use an additive decoder (Lachapelle et al., 2024), and found this to be necessary to achieve sensible visualization results on CMNIST
between layers. We use an additive decoder (Lachapelle et al., 2024), and found this to be necessary to achieve sensible visualization results on CMNIST. That is, we define log p(x | zc, zc) = Decc(zc) + Decs(zs), where both Decc and Decs have the same architecture. Table 2: S...
2024
-
[2019]
We set the batch size to 2,048
with 1 × 10−4 learning rate and 0.01 weight decay. We set the batch size to 2,048. We chose these values because they resulted in stable training and validation curves across our experiments, and did not tune them extensively. For domain generalization, we resize the images to...
2020
-
[2022]
and Lacoste-Julien, S
10 Supervised Contrastive Block Disentanglement Lachapelle, S. and Lacoste-Julien, S. Partial disentangle- ment via mechanism sparsity. UAI 2022 Workshop on Causal Representation Learning,
2022
-
[2024]
CORUM: the comprehensive resource of mammalian protein complexes—2009
Ruepp, A., Waegele, B., Lechner, M., Brauner, B., Dunger- Kaltenbach, I., Fobo, G., Frishman, G., Montrone, C., and Mewes, H.-W. CORUM: the comprehensive resource of mammalian protein complexes—2009. Nucleic Acids Research,
2009
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.