REVIEW 9 cited by
AstroCLIP: A Cross-Modal Foundation Model for Galaxies
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
We present AstroCLIP, a single, versatile model that can embed both galaxy images and spectra into a shared, physically meaningful latent space. These embeddings can then be used - without any model fine-tuning - for a variety of downstream tasks including (1) accurate in-modality and cross-modality semantic similarity search, (2) photometric redshift estimation, (3) galaxy property estimation from both images and spectra, and (4) morphology classification. Our approach to implementing AstroCLIP consists of two parts. First, we embed galaxy images and spectra separately by pretraining separate transformer-based image and spectrum encoders in self-supervised settings. We then align the encoders using a contrastive loss. We apply our method to spectra from the Dark Energy Spectroscopic Instrument and images from its corresponding Legacy Imaging Survey. Overall, we find remarkable performance on all downstream tasks, even relative to supervised baselines. For example, for a task like photometric redshift prediction, we find similar performance to a specifically-trained ResNet18, and for additional tasks like physical property estimation (stellar mass, age, metallicity, and sSFR), we beat this supervised baseline by 19\% in terms of $R^2$. We also compare our results to a state-of-the-art self-supervised single-modal model for galaxy images, and find that our approach outperforms this benchmark by roughly a factor of two on photometric redshift estimation and physical property prediction in terms of $R^2$, while remaining roughly in-line in terms of morphology classification. Ultimately, our approach represents the first cross-modal self-supervised model for galaxies, and the first self-supervised transformer-based architectures for galaxy images and spectra.
Forward citations
Cited by 9 Pith papers
-
Learning What's Real: Disentangling Signal and Measurement Artifacts in Multi-Sensor Data, with Applications to Astrophysics
Overlapping multi-instrument galaxy images plus dual encoders and flow-matching counterfactual generation yield physics latents unconfounded by sensor artifacts.
-
Star-forming clump detection in nearby galaxies using Faster R-CNN and $ugrizy$ imaging data from CLAUDS and HSC-SSP
A six-band Faster R-CNN with the Zoobot backbone detects star-forming clump candidates in ~700,000 local galaxies, claiming ~90% completeness and ~80% purity for clumps brighter than the surveys' detection limits.
-
AGNFormer I: Reconstruction of AGN spectra using a probabilistic transformer model
An uncertainty-aware transformer reconstructs masked AGN broad lines and spectral halves with 4-16% flux errors and beats eleven purpose-built Lyα-reconstruction algorithms on a blind benchmark.
-
Mantis Shrimp: Exploring Photometric Band Utilization in Computer Vision Networks for Photometric Redshift Estimation
A multi-survey CNN estimates photometric redshifts from GALEX, PanSTARRS, and UnWISE cutouts, with early and late image fusion performing comparably.
-
DeepDISC-photoz: Deep Learning-Based Photometric Redshift Estimation for Rubin LSST
DeepDISC photo-z, an instance-segmentation network with a mixture-density redshift head, produces better photometric redshifts than catalog-based BPZ and FlexZBoost on simulated Rubin LSST images.
-
Beyond Scaling Curves: Internal Dynamics of Neural Networks Through the NTK Lens
Using NTK trace and effective rank, this paper shows that model and data scaling improve test loss at similar rates but drive internal dynamics in opposite directions, and estimates a feature-learning width limit well...
-
SpecPT (Spectroscopy Pre-trained Transformer) Model for Extragalactic Spectroscopy: I. Architecture and Automated Redshift Measurement
SpecPT, a transformer autoencoder, reconstructs DESI spectra and predicts redshifts with NMAD 0.0006 (BGS) and 0.0008 (ELG), with outlier fractions under 1%.
-
Machine Learning and the SKA for Cosmic Dawn and the Epoch of Reionization
A multi-author overview of machine-learning algorithms proposed for instrument modelling, data analysis, simulation and inference in SKA Cosmic Dawn and Epoch of Reionization science.
-
From stellar light to astrophysical insight: automating variable star research with machine learning
An invited review of machine learning for automated variable star research, covering data cleaning, variability classification, stellar parameter inference, and foundation models.
Discussion (0). Continue with ORCID to comment.