REVIEW 2 major objections 6 minor 30 references
Learning What's Real: Disentangling Signal and Measurement Artifacts in Multi-Sensor Data, with Applications to Astrophysics
T0 review · 2 major / 6 minor · reviewed 2026-07-12 · grok-4.5
Pith's one-line read Overlapping multi-telescope observations can train dual encoders that separate a galaxy’s true physical signal from instrument-specific distortions.
desk verdict Solid generative dual-encoder extension of multi-instrument disentanglement on real Legacy–HSC pairs; useful and mostly well-checked, with residual leakage and neighbor conditioning as the main soft spots. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Counterfactual generation with dual encoders: the physics encoder is given same-source, different-instrument views; the instrument encoder is given different-source, same-instrument views; a conditional flow-matching decoder reconstructs an anchor image neither encoder has seen, creating an architectural information bottleneck that forces the separation.
What would settle it
Replace the five nearest spatial neighbors with random galaxies from the same survey; if instrument-property prediction from the instrument latent collapses and physics-space outliers again become dominated by saturated pixels and noise bands, the shared-local-systematics premise has failed.
Extended reading notes
Core claim
A dual-encoder flow-matching model trained solely with a counterfactual reconstruction objective on overlapping multi-instrument observations learns representations that explicitly separate intrinsic physical signals from sensor-specific distortions and noise. Demonstrated on paired DESI Legacy and Hyper Suprime-Cam galaxy images, the physics latents align matched sources across surveys and yield morphologically meaningful outliers, while the instrument latents cluster by survey and predict local observing conditions, enabling unconfounded inference, instrument-independent search, and cross-instrument generation.
Load-bearing premise
Images of other galaxies lying within a few arcminutes of the target are assumed to share essentially the same instrument conditions, so they can teach the model the sensor’s behavior without leaking the target’s own physics.
Editorial extensions
If this is right
- Physics embeddings support outlier detection of rare morphologies without instrument artifacts dominating the ranking.
- Counterfactual generation can predict how a low-resolution survey object would appear under a deeper, higher-resolution instrument to prioritize follow-up.
- Nearest-neighbor search in physics space retrieves physically similar objects independent of which survey supplied the query.
- Analysis pipelines trained on one survey’s images can be applied, zero-shot, to counterfactual images generated from another survey.
- The same training recipe—overlap pairs, dual encoders, counterfactual reconstruction—extends to other multi-sensor scientific domains.
Reading between the lines
- Tightening or widening the instrument-encoder bottleneck is a practical dial for residual leakage of physical properties into the instrument space.
- Outside dense overlapping footprints the method needs an unpaired extension; otherwise coverage remains limited to shared sky regions.
- The same structural split could reduce instrument-driven false positives in multi-mission time-series archives where detector systematics currently dominate anomaly lists.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a dual-encoder architecture with a counterfactual flow-matching objective that uses overlapping multi-instrument observations to disentangle physics-invariant signals from instrument-specific artifacts. Training triplets consist of an anchor image (never seen by the encoders), a same-source different-instrument view for the physics encoder, and same-instrument different-source neighbors for the instrument encoder; the decoder reconstructs the anchor conditioned on both latents (Eq. 1, Fig. 1). On ~100k cross-matched Legacy–HSC galaxy images the method produces UMAP-separated latent spaces with pair alignment (Fig. 2), physics-space outliers free of instrumental pathologies (Fig. 3), asymmetric R^{2} probes against AION-1 and random-ResNet baselines (Fig. 4 / Table 5), instrument-independent nearest-neighbor retrieval (Fig. 5), and usable cross-instrument posterior samples (Figs. 6–7, MSE and pipeline-transfer checks). The authors present the approach as a general recipe for scientific multi-modal pretraining that treats sensor effects as natural augmentations.
Significance. If the disentanglement holds at the claimed level, the work supplies a practical architectural recipe for multi-instrument foundation models that avoids hand-engineered contrastive losses and explicit instrument forward models. The generative decoder additionally yields a data-driven noise model and counterfactual survey translation, both of direct utility for follow-up prioritization and pipeline transfer across surveys. Strengths that raise the contribution above a pure methods paper include the external-catalog probes (PROVABGS spectroscopy, shape catalogs, survey metadata), the cross-predict ResNet control that bounds shared spatial structure, the zero-shot ellipticity pipeline transfer (R^{2} 0.82 vs 0.81), and the public simplified codebase. Residual leakage is acknowledged and partially quantified; the result is therefore a solid, usable advance for astro-ph.IM and related multi-sensor domains rather than a complete solution to causal disentanglement.
major comments (2)
- §3 Data Triplets / Application: the instrument encoder is conditioned on up to five nearest spatial neighbors within 3 arcmin, under the premise that PSF/depth vary smoothly enough to share essentially the same instrument conditions. Residual R^{2} of the instrument latent for redshift, stellar mass and morphology (Fig. 4 / Table 5) and of the physics latent for some instrument quantities leave open how much private physics or large-scale structure co-varies with the neighbors. A controlled ablation that degrades neighbor quality (random same-survey images, larger angular radii, or shuffled sky positions) and re-measures the R^{2} asymmetry and pair alignment would quantify the load-bearing assumption; without it the claim of unconfounded physics representations remains only partially stress-tested.
- §4.1 Parameter inference and Appendix A.5: the cross-predict ResNet baseline shows that physics latents do not capture private instrument information above the shared spatial floor, which is a useful control. However, the instrument latent still retains non-trivial predictive power for physical properties that are only partially recoverable from imaging. The paper should either tighten the instrument bottleneck (capacity study already flagged as future work) or report a quantitative leakage metric (e.g., mutual information or residual R^{2} after regressing out the cross-predict baseline) so that the degree of residual confounding is stated rather than left as a qualitative caveat.
minor comments (6)
- Fig. 4 and Table 5: the main-text figure is hard to read at print scale; the appendix table is clearer. Consider moving the full numerical table into the main text or enlarging the figure panels.
- §4 Data Preprocessing: the decision to discard the HSC y-band and to upsample Legacy 31×31 to 48×48 via linear interpolation is stated but not ablated; a short note on sensitivity would help reproducibility.
- Eq. (1) and surrounding text: the notation for the variable-length sets of physics and instrument embeddings is dense; a short algorithmic box listing the sampling of j, k, neighbors and the attention conditioning would improve clarity.
- Fig. 3 caption and text: the full 160×160 cutouts are useful, but the main-text figure is small; consider a supplementary gallery of the top outliers with survey labels.
- Related work: the comparison to Audenaert et al. (contrastive dual-encoder) is clear; a one-sentence quantitative note on why the generative objective preserves high-resolution features that contrastive alignment discards would strengthen the positioning.
- Code availability: the simplified public codebase is welcome; stating the exact commit or DOI used for the reported numbers would aid exact reproduction.
Circularity Check
No load-bearing circularity; mild non-essential self-citation to related dual-encoder work by overlapping authors, while central claims rest on held-out reconstruction and external probes.
full rationale
The paper's core claim is architectural and empirical: a dual-encoder flow-matching model trained to reconstruct a held-out anchor (never seen by either encoder) from physics latents of the same source under a different instrument and instrument latents of different sources under the same instrument produces disentangled representations. Success is measured by UMAP separation (Fig. 2), outlier morphology free of artifacts (Fig. 3), R^{2} asymmetry on external spectroscopic/shape/instrument catalogs versus AION-1 and random-ResNet baselines (Fig. 4 / Table 5), nearest-neighbor retrieval, and generative MSE/posterior calibration on held-out pairs. None of these reduce by construction to a fitted free parameter of the claim, nor to a self-definitional identity. The only self-citation of note is to Audenaert et al. (2025) [11] (overlapping authors), which introduced a dual-encoder contrastive approach on simulated light curves; the present work explicitly distinguishes its generative objective, applies it to real multi-survey imaging, and does not invoke any uniqueness theorem or load-bearing result from that paper. The spatial-neighbor conditioning assumption is an empirical modeling choice, not a circular derivation. The paper is therefore self-contained against external benchmarks; score 1 only for the non-load-bearing related-work citation.
Assumptions & free parameters
free parameters (4)
- Instrument-neighbor count and angular radius =
5 neighbors, 3 arcmin
- Physics/instrument latent token geometry =
(B, 4, 16)
- UNet and training hyperparameters =
see Appendix Tables 1–2
- Image preprocessing (crop, upsample, arcsinh, zero-point) =
48×48, 4 channels, ZP 22.5
assumptions (5)
- domain assumption Observation = f(Signal, Instrument) + Noise, with physics factors independent of measurement and instrument factors acting as confounders.
- ad hoc to paper Cross-matched multi-instrument observations of the same source provide a sufficient training signal to separate physics and instrument without an explicit forward model of either instrument.
- domain assumption Instrument systematics (PSF, depth, noise) vary smoothly on ~arcminute scales so nearby sources share instrument conditions with the anchor.
- standard math Conditional flow matching can learn p(x_anchor | z_physics from other instruments, z_instrument from other sources) from finite paired data.
- ad hoc to paper An architectural information bottleneck (dual encoders never seeing the anchor) is sufficient to drive disentanglement without hand-engineered contrastive or β-VAE-style losses.
invented entities (1)
-
Physics latent space z_phy and instrument latent space z_ins as structural factors of variation
independent evidence
Cite this review
Pith. "Pith review of Learning What's Real: Disentangling Signal and Measurement Artifacts in Multi-Sensor Data, with Applications to Astrophysics." pith.science (2026). https://pith.science/paper/INZWB3BG
@misc{pith2026260409787,
author = {Pith},
title = {Pith review of: Learning What's Real: Disentangling Signal and Measurement Artifacts in Multi-Sensor Data, with Applications to Astrophysics},
year = {2026},
howpublished = {\url{https://pith.science/paper/INZWB3BG}},
note = {Machine review of arXiv:2604.09787}
}
read the original abstract
Data collected from the physical world is always a combination of multiple sources: an underlying signal from the physical process of interest and a signal from measurement-dependent artifacts from the sensor or instrument. This secondary signal acts as a confounding factor, limiting our ability to extract information about the physics underlying the phenomena we observe. Furthermore, it complicates the combination of observations in heterogeneous or multi-instrument settings. We propose a deep learning framework that leverages overlapping observations, a dual-encoder architecture, and a counterfactual generation objective to disentangle these factors of variation. The resulting representations explicitly separate intrinsic signals from sensor-specific distortions and noise, and can be used for counterfactual view generation, parameter inference unconfounded by measurement distortions, and instrument-independent similarity search. We demonstrate the effectiveness of our approach on astrophysical galaxy images from the DESI Legacy Imaging Survey (Legacy) and the Hyper Suprime-Cam (HSC) Survey as a representative multi-instrument setting. This framework provides a general recipe for scientific and multi-modal self-supervised pretraining: construct training pairs from overlapping observations of the same physical system, treat sensor- or modality-specific effects as augmentations, and learn invariant representations through counterfactual generation.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
J. B. Tenenbaum and W. T. Freeman. Separating style and content with bilinear models.Neural Computation, 12:1247 – 1283, 2000
2000
-
[2]
beta-V AE: Learning basic visual concepts with a constrained variational framework
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-V AE: Learning basic visual concepts with a constrained variational framework. InInternational Conference on Learning Representations,
-
[3]
URLhttps://openreview.net/forum?id=Sy2fzU9gl
-
[4]
Toward causal representation learning.Proceedings of the IEEE, 109(5):612–634, 2021
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning.Proceedings of the IEEE, 109(5):612–634, 2021. doi: 10.1109/JPROC.2021.3058954
-
[5]
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomic- level protein structure with a language model.Science, 379(6637):1123–1130, 2023. doi:...
-
[6]
Large-scale chemical language representations capture molecular structure and properties
Jerret Ross, Brian Belgodere, Vijil Chenthamarakshan, Inkit Padhi, Youssef Mroueh, and Payel Das. Large-scale chemical language representations capture molecular structure and properties. Nature Machine Intelligence, 4(12):1256–1264, 2022. URL https://arxiv.org/abs/2106. 09553
2022
-
[7]
Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David Lobell, and Stefano Ermon. Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery.Advances in Neural Information Processing Systems, 35: 197–211, 2022. URLhttps://arxiv.org/abs/2207.08051
arXiv 2022
-
[8]
Mike Walmsley, Inigo Val Slijepcevic, Micah Bowles, and Anna M. M. Scaife. Towards galaxy foundation models with hybrid contrastive learning, 2022. URL https://arxiv.org/abs/ 2206.11927
arXiv 2022
Show all 30 references
-
[9]
Aion-1: Omnimodal foundation model for astronomical sciences
Liam Parker, Francois Lanusse, Jeff Shen, Ollie Liu, Tom Hehir, Leopoldo Sarra, Lucas Meyer, Micah Bowles, Sebastian Wagner-Carena, Helen Qu, Siavash Golkar, Alberto Bietti, Hatim Bourfoune, Nathan Casserau, Pierre Cornette, Keiya Hirashima, Geraud Krawezik, Ruben Ohana, Nicho...
2025
-
[10]
The multimodal universe: Enabling large-scale machine learning with 100tb of astronomical scientific data, 2024
The Multimodal Universe Collaboration. The multimodal universe: Enabling large-scale machine learning with 100tb of astronomical scientific data, 2024. URL https://arxiv. org/abs/2412.02527
2024 arXiv
-
[11]
Siudek, M
Euclid Collaboration, M. Siudek, M. Huertas-Company, M. Smith, G. Martinez-Solaeche, F. Lanusse, S. Ho, E. Angeloudi, P. A. C. Cunha, H. Domínguez Sánchez, M. Dunn, Y . Fu, P. Iglesias-Navarro, J. Junais, J. H. Knapen, B. Laloux, M. Mezcua, W. Roster, G. Stevens, J. Vega-Ferre...
-
[12]
Gregory, David W
Jeroen Audenaert, Daniel Muthukrishna, Paul F. Gregory, David W. Hogg, and V . Ashley Villar. Causal foundation models: Disentangling physics from instrument properties, 2025. URL https://arxiv.org/abs/2507.05333. 11
2025 arXiv
-
[13]
Hogg, Dun Wang, Daniel Foreman-Mackey, Dominik Janzing, Carl-Johann Simon-Gabriel, and Jonas Peters
Bernhard Schölkopf, David W. Hogg, Dun Wang, Daniel Foreman-Mackey, Dominik Janzing, Carl-Johann Simon-Gabriel, and Jonas Peters. Modeling confounding by half-sibling regression. Proceedings of the National Academy of Sciences, 113(27):7391–7398, 2016. doi: 10.1073/ pnas.15116...
2016 doi
-
[14]
Hogg, Benjamin T
Soichiro Hattori, Daniel Foreman-Mackey, David W. Hogg, Benjamin T. Montet, Ruth Angus, T. A. Pritchard, Jason L. Curtis, and Bernhard Schölkopf. The unpopular Package: A Data- driven Approach to Detrending TESS Full-frame Image Light Curves.The Astronomical Journal, 163(6):28...
2022 doi
-
[15]
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, 2023. URLhttps://arxiv.org/abs/2210.02747
2023 arXiv
-
[16]
Schlegel, Dustin Lang, Robert Blum, Kaylan Burleigh, Xiaohui Fan, Joseph R
Arjun Dey, David J. Schlegel, Dustin Lang, Robert Blum, Kaylan Burleigh, Xiaohui Fan, Joseph R. Findlay, Doug Finkbeiner, David Herrera, Stéphanie Juneau, Martin Landriau, Michael Levi, Ian McGreer, Aaron Meisner, Adam D. Myers, John Moustakas, Peter Nugent, Anna Patej, Edward...
2019
-
[19]
Deep residual learning for image recognition, 2015
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URLhttps://arxiv.org/abs/1512.03385
2015 arXiv
-
[20]
Diffusers: State-of-the-art diffusion models
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/ diffusers, 2022
2022
-
[21]
Umap: Uniform manifold approximation and projection for dimension reduction, 2020
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction, 2020. URL https://arxiv.org/abs/1802.03426
2020 arXiv
-
[22]
ChangHoon Hahn, K. J. Kwon, Rita Tojeiro, Malgorzata Siudek, Rebecca E. A. Canning, Mar Mezcua, Jeremy L. Tinker, David Brooks, Peter Doel, Kevin Fanning, Enrique Gaztañaga, Robert Kehoe, Martin Landriau, Aaron Meisner, John Moustakas, Claire Poppett, Gregory Tarle, Benjamin W...
2023 doi
-
[23]
Saxe, Pang Wei Koh, Zhenghao Chen, Maneesh Bhand, Bipin Suresh, and Andrew Y
Andrew M. Saxe, Pang Wei Koh, Zhenghao Chen, Maneesh Bhand, Bipin Suresh, and Andrew Y . Ng. On random weights and unsupervised feature learning. InProceedings of the 28th International Conference on International Conference on Machine Learning, ICML’11, page 1089–1096, Madiso...
2011
-
[24]
Ricker, Joshua N
George R. Ricker, Joshua N. Winn, Roland Vanderspek, David W. Latham, Gáspár Á. Bakos, Jacob L. Bean, Zachory K. Berta-Thompson, Timothy M. Brown, Lars Buchhave, Nathaniel R. Butler, R. Paul Butler, William J. Chaplin, David Charbonneau, Jørgen Christensen-Dalsgaard, Mark Clam...
2015 doi
-
[25]
Borucki, David Koch, Gibor Basri, Natalie Batalha, Timothy Brown, Douglas Caldwell, John Caldwell, Jørgen Christensen-Dalsgaard, William D
William J. Borucki, David Koch, Gibor Basri, Natalie Batalha, Timothy Brown, Douglas Caldwell, John Caldwell, Jørgen Christensen-Dalsgaard, William D. Cochran, Edna DeV ore, Edward W. Dunham, Andrea K. Dupree, Thomas N. Gautier, John C. Geary, Ronald Gilliland, Alan Gould, Ste...
2010 doi
-
[26]
Koch, William J
David G. Koch, William J. Borucki, Gibor Basri, Natalie M. Batalha, Timothy M. Brown, Douglas Caldwell, Jørgen Christensen-Dalsgaard, William D. Cochran, Edna DeV ore, Ed- ward W. Dunham, III Gautier, Thomas N., John C. Geary, Ronald L. Gilliland, Alan Gould, Jon Jenkins, Yoji...
2010 doi
-
[27]
Bahcall, Steven Bickerton, James Bosch, Kevin Bundy, Peter L
Hiroaki Aihara, Nobuo Arimoto, Robert Armstrong, Stéphane Arnouts, Neta A. Bahcall, Steven Bickerton, James Bosch, Kevin Bundy, Peter L. Capak, James H. H. Chan, Masashi Chiba, Jean Coupon, Eiichi Egami, Motohiro Enoki, Francois Finet, Hiroki Fujimori, Seiji Fujimoto, Hisanori...
-
[28]
doi: 10.1093/pasj/psx066
-
[29]
Schlegel, Dustin Lang, Robert Blum, Kaylan Burleigh, Xiaohui Fan, Joseph R
Arjun Dey, David J. Schlegel, Dustin Lang, Robert Blum, Kaylan Burleigh, Xiaohui Fan, Joseph R. Findlay, Doug Finkbeiner, David Herrera, Stéphanie Juneau, Martin Landriau, Michael Levi, Ian McGreer, Aaron Meisner, Adam D. Myers, John Moustakas, Peter Nugent, Anna Patej, Edward...
2019 doi
-
[30]
AstroCLIP: Cross-modal pre-training for astronomical foundation models.arXiv preprint arXiv:2310.03024, 2023
François Lanusse, Liam Parker, Siavash Golkar, Miles Cranmer, Alberto Bietti, Michael Eick- enberg, Geraud Krawezik, Michael McCabe, Ruben Ohana, Mariel Pettee, Bruno Régaldo- Saint Blancard, Tiberiu Tesileanu, Kyunghyun Cho, and Shirley Ho. AstroCLIP: Cross-modal pre-training...
2023 arXiv
-
[31]
Neural spline flows
Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems, volume 32, 2019
2019
-
[32]
Zuko: Normalizing flows in PyTorch, 2023
François Rozet, Felix Andry, François Lanusse, and Gilles Louppe. Zuko: Normalizing flows in PyTorch, 2023. URLhttps://github.com/probabilists/zuko. 15 A Appendix A.1 Model Architecture We use a ResNet-18 [17] architecture for the encoders, plus a UNet2DConditionModel [18] for...
2023
Reviewed July 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.