Recognition: unknown
Calibration-Induced Systematics in SALT3 Training and Their Impact on Dark Energy Constraints from Stage IV Supernova Surveys
Pith reviewed 2026-05-10 01:14 UTC · model grok-4.3
The pith
Small calibration errors during light-curve fitting reduce the dark energy figure of merit by 50 percent for next-generation surveys
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Zero-point shifts of 5 mmag and filter mean wavelength shifts of 5 angstrom lead to a ∼50% decrease in the FoM relative to a statistical-only case when calibration uncertainties are propagated only through light-curve fitting. The same calibration shifts applied only during model training produce a smaller ∼13% degradation. Contrary to previous analyses, calibration uncertainties in light-curve fitting dominate over those from model training. Their effect during light-curve fitting varies smoothly with redshift and is nearly degenerate with cosmology, preventing mitigation through self-calibration. Finally, the FoM dependence on the size of the calibration uncertainties is roughly linear.
What carries the argument
SALT3 spectro-photometric model with calibration perturbations (zero-point and filter wavelength shifts) propagated separately through training and light-curve fitting stages
If this is right
- Calibration uncertainties in light-curve fitting dominate the systematic error budget for dark energy measurements.
- The smooth redshift variation of fitting errors makes them nearly degenerate with dark energy equation-of-state parameters.
- The degradation of the figure of merit scales roughly linearly with the amplitude of calibration shifts.
- Self-calibration methods are ineffective against these fitting-stage systematics.
Where Pith is reading between the lines
- Efforts to improve on-sky calibration monitoring could disproportionately benefit cosmological constraints by targeting the fitting stage.
- The linear dependence implies that incremental calibration improvements will yield proportional gains in precision.
- This separation of training and fitting effects could be tested with other models like SALT2 to see if the dominance of fitting holds generally.
- Combining supernova data with other probes might help break the degeneracy between calibration errors and cosmology.
Load-bearing premise
The sizes of the applied zero-point and wavelength shifts and the characteristics of the simulated survey data accurately represent the real-world calibration uncertainties without introducing unaccounted biases.
What would settle it
Reanalyzing the problem with real calibration data from current or future surveys or with varied shift sizes to check whether the reported 50% and 13% FoM degradations persist.
Figures
read the original abstract
In the coming years, the Vera Rubin Observatory's Legacy Survey of Space and Time (Rubin-LSST) and the Nancy Grace Roman Space Telescope's (Roman) High Latitude Time Domain Survey (HLTDS) are expected to discover more than a million Type Ia supernovae (SNe Ia), several orders of magnitude more than current samples and with a tighter control on systematic uncertainties. One of the largest systematic uncertainties in cosmological analyses with SNe Ia is the accuracy of the spectro-photometric model for SNe Ia time series data, which depends on the photometric calibration of the surveys. To quantify the impact of this uncertainty, we analyze simulated Rubin-LSST and HLTDS data, perturb the photometric zero-points and filter mean wavelengths, and propagate these systematics to spectral model recovery, estimated distances, and dark energy figure of merit (FoM) based on the $w_0 w_a$CDM model. Zero-point shifts of 5 mmag and filter mean wavelength shifts of 5 angstrom lead to a $\sim 50\%$ decrease in the FoM relative to a statistical-only case when calibration uncertainties are propagated only through light-curve fitting. The same calibration shifts applied only during model training produce a smaller $\sim 13\%$ degradation. Contrary to previous analyses, calibration uncertainties in light-curve fitting dominate over those from model training. Their effect during light-curve fitting varies smoothly with redshift and is nearly degenerate with cosmology, preventing mitigation through self-calibration. Finally, we show that the FoM dependence on the size of the calibration uncertainties (in the range of expected sizes) is roughly linear.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper uses forward simulations of Rubin-LSST and Roman HLTDS Type Ia supernova light curves to quantify calibration systematics in the SALT3 model. Zero-point shifts of 5 mmag and filter mean-wavelength shifts of 5 Å are applied either only during model training or only during light-curve fitting; the resulting distance biases are propagated to the w0waCDM figure of merit (FoM). The analysis finds a ~50% FoM degradation when shifts affect only fitting, a ~13% degradation when they affect only training, and a near-degeneracy between the fitting-stage bias and cosmological parameters that prevents self-calibration. The FoM dependence on calibration uncertainty size is reported as roughly linear.
Significance. If the simulated propagation faithfully reproduces real-data covariances, the result is significant for Stage-IV survey planning: it indicates that calibration resources should be prioritized for the fitting stage rather than training, quantifies the FoM penalty, and shows why self-calibration is ineffective. The forward-simulation framework itself is a strength, as it allows controlled isolation of training versus fitting contributions.
major comments (2)
- [§3] §3 (Simulation and propagation pipeline): the central 50%-versus-13% dominance claim rests on the fidelity of the simulated SN SEDs, training-sample composition, and noise model. No quantitative validation against real calibration residuals or assessment of omitted covariances between zero-point/wavelength errors and color/luminosity parameters is provided; if such covariances exist in the data, the reported relative importance of fitting over training could be an artifact.
- [§5] §5 (FoM results and degeneracy): the statement that the fitting-stage bias 'varies smoothly with redshift and is nearly degenerate with cosmology' is load-bearing for the self-calibration conclusion, yet the manuscript does not show the explicit redshift-dependent bias curves or the Fisher-matrix eigenvectors that would demonstrate the degeneracy strength.
minor comments (2)
- [Abstract / §4] The abstract and §4 state that the FoM dependence is 'roughly linear' over the explored range, but no figure or table quantifies the slope or reports the goodness-of-fit to linearity.
- [Tables] Table captions and axis labels should explicitly state whether the reported FoM values include or exclude the calibration-induced bias term.
Simulated Author's Rebuttal
We thank the referee for their thoughtful review and for recognizing the significance of our forward-simulation framework for isolating calibration systematics in SALT3. We address each major comment below. Where the manuscript was incomplete, we have revised it by adding discussion and a new figure; we also clarify the controlled nature of the simulation and note limitations honestly.
read point-by-point responses
-
Referee: [§3] §3 (Simulation and propagation pipeline): the central 50%-versus-13% dominance claim rests on the fidelity of the simulated SN SEDs, training-sample composition, and noise model. No quantitative validation against real calibration residuals or assessment of omitted covariances between zero-point/wavelength errors and color/luminosity parameters is provided; if such covariances exist in the data, the reported relative importance of fitting over training could be an artifact.
Authors: We agree that direct quantitative validation against observed calibration residuals from existing surveys would be valuable. Our simulations adopt the SALT3 SED model and noise properties calibrated to the published Rubin-LSST and Roman HLTDS specifications, with training-sample composition drawn from realistic redshift and magnitude distributions used in prior SALT3 analyses. We have added a new paragraph in §3 explicitly discussing the assumptions underlying the SED fidelity and noise model, together with a qualitative assessment of how covariances between zero-point/wavelength shifts and color/luminosity parameters could propagate. Because the forward-modeling approach isolates the training versus fitting stages by construction, any unmodeled covariance would affect both stages; we therefore retain the reported 50 % versus 13 % contrast as a lower bound on the fitting-stage dominance. We acknowledge that a full end-to-end validation against proprietary calibration data lies outside the present scope and have noted this limitation. revision: partial
-
Referee: [§5] §5 (FoM results and degeneracy): the statement that the fitting-stage bias 'varies smoothly with redshift and is nearly degenerate with cosmology' is load-bearing for the self-calibration conclusion, yet the manuscript does not show the explicit redshift-dependent bias curves or the Fisher-matrix eigenvectors that would demonstrate the degeneracy strength.
Authors: We accept the referee’s point that explicit visualization strengthens the claim. We have inserted a new figure (Figure 8) in §5 that displays the redshift-dependent distance-modulus bias curves arising from the fitting-stage zero-point and wavelength shifts, together with the leading eigenvectors of the Fisher matrix for the w0–wa plane. These eigenvectors confirm the near-degeneracy between the smooth redshift-dependent bias and the cosmological parameters, directly supporting the conclusion that self-calibration cannot remove the systematic. The revised text now references this figure when stating the degeneracy. revision: yes
Circularity Check
No circularity: forward simulation of external perturbations
full rationale
The paper's central results are obtained by generating simulated Rubin-LSST and Roman HLTDS light curves, imposing independent external zero-point (5 mmag) and wavelength (5 Å) shifts, then propagating those shifts separately through SALT3 model training versus light-curve fitting, and finally computing the w0waCDM FoM from the resulting distance estimates. These FoM values are direct numerical outputs of the pipeline applied to the perturbed mocks; they are not fitted parameters, not defined in terms of themselves, and not obtained by renaming or re-using the input perturbations. No load-bearing self-citations, ansatzes, or uniqueness theorems are invoked in the abstract or described chain. The derivation is therefore self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption The SALT3 model provides an adequate description of Type Ia supernova spectral energy distributions for cosmological distance estimation.
- domain assumption The simulated Rubin-LSST and Roman HLTDS datasets faithfully reproduce the statistical and systematic properties of future observations.
Reference graph
Works this paper leans on
-
[1]
Abdul Karim, M., Aguilar, J., Ahlen, S., et al. 2025, Physical Review D, 112, doi: 10.1103/tr6y-kpc6
-
[2]
G., Aguilar, J., Ahlen, S., et al
Adame, A., Aguilar, J., Ahlen, S., et al. 2025, Journal of Cosmology and Astroparticle Physics, 2025, 021, doi: 10.1088/1475-7516/2025/02/021
-
[3]
2021, prd, 103, 083533, doi: 10.1103/PhysRevD.103.083533
Alam, S., Aubert, M., Avila, S., et al. 2021, Phys. Rev. D, 103, 083533, doi: 10.1103/PhysRevD.103.083533
-
[4]
Report of the Dark Energy Task Force
Albrecht, A., Bernstein, G., Cahn, R., et al. 2006, arXiv e-prints, astro, doi: 10.48550/arXiv.astro-ph/0609591
work page Pith review doi:10.48550/arxiv.astro-ph/0609591 2006
-
[5]
Astier, P., Guy, J., Regnault, N., et al. 2006, A&A, 447, 31, doi: 10.1051/0004-6361:20054185
-
[6]
C., Vincenzi, M., Scolnic, D., et al
Bailey, A. C., Vincenzi, M., Scolnic, D., et al. 2023, MNRAS, 524, 5432, doi: 10.1093/mnras/stad2179
-
[7]
2014, , 568, A22, 10.1051/0004-6361/201423413
Betoule, M., Kessler, R., Guy, J., et al. 2014, A&A, 568, A22, doi: 10.1051/0004-6361/201423413
-
[8]
2019 a , , 874, 150, 10.3847/1538-4357/ab08a0
Brout, D., Scolnic, D., Kessler, R., et al. 2019, ApJ, 874, 150, doi: 10.3847/1538-4357/ab08a0
-
[9]
and Carr, Anthony and Zuntz, Joe and Kessler, Rick and Davis, Tamara M
Brout, D., Scolnic, D., Popovic, B., et al. 2022a, The Astrophysical Journal, 938, 110, doi: 10.3847/1538-4357/ac8e04
-
[11]
Brout, D., Taylor, G., Scolnic, D., et al. 2022, ApJ, 938, 111, doi: 10.3847/1538-4357/ac8bcc
-
[12]
C., Guo, Z., Scolnic, D., et al
Chen, R. C., Guo, Z., Scolnic, D., et al. 2026, ApJ, 999, 81, doi: 10.3847/1538-4357/ae42be
-
[13]
LSST Science Book, Version 2.0
Collaboration, L. S., Abell, P. A., Allison, J., et al. 2009, LSST Science Book, Version 2.0. https://arxiv.org/abs/0912.0201
work page Pith review arXiv 2009
-
[14]
2010, The Astrophysical Journal Supplement Series, 192, 1, 10.1088/0067-0049/192/1/1
Conley, A., Guy, J., Sullivan, M., et al. 2011, ApJS, 192, 1, doi: 10.1088/0067-0049/192/1/1
-
[15]
Dai, M., Jones, D. O., Kenworthy, W. D., et al. 2023, ApJS, 267, 1, doi: 10.3847/1538-4365/acd051
-
[16]
2006, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol
Pierfederici, F. 2006, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 6270, Observatory Operations: Strategies, Processes, and Systems, ed. D. R. Silva & R. E. Doxsey, 62701D, doi: 10.1117/12.671992
-
[17]
Dilday, B., Kessler, R., Frieman, J. A., et al. 2008, ApJ, 682, 262, doi: 10.1086/587733
-
[18]
Guy, J., Astier, P., Nobili, S., Regnault, N., & Pain, R. 2005, A&A, 443, 781, doi: 10.1051/0004-6361:20053025
-
[19]
2007, , 466, 11, 10.1051/0004-6361:20066930
Guy, J., Astier, P., Baumont, S., et al. 2007, A&A, 466, 11, doi: 10.1051/0004-6361:20066930
-
[20]
Hamuy, M., Phillips, M. M., Suntzeff, N. B., et al. 1996, The Astronomical Journal, 112, 2391, doi: 10.1086/118190
-
[21]
2020, The Journal of Open Source Software, 5, 2122, 10.21105/joss.02122
Hinton, S., & Brout, D. 2020, The Journal of Open Source Software, 5, 2122, doi: 10.21105/joss.02122
-
[22]
and Campbell, Heather and Dilday, Ben and et al
Hlozek, R., Kunz, M., Bassett, B., et al. 2012, The Astrophysical Journal, 752, 79, doi: 10.1088/0004-637X/752/2/79
-
[23]
HLTDS, H. 2025, arXiv e-prints, arXiv:2505.10574, doi: 10.48550/arXiv.2505.10574
-
[24]
Hounsell, R., Scolnic, D., Foley, R. J., et al. 2018, ApJ, 867, 23, doi: 10.3847/1538-4357/aac08b
-
[25]
Ivezic, Z., Kahn, S. M., Tyson, J. A., et al. 2019, The Astrophysical Journal, 873, 111, doi: 10.3847/1538-4357/ab042c
-
[26]
Jones, D. O., Kenworthy, W. D., Dai, M., et al. 2023, ApJ, 951, 22, doi: 10.3847/1538-4357/acd195
-
[27]
Jones, D. O., Scolnic, D. M., Riess, A. G., et al. 2018, ApJ, 857, 51, doi: 10.3847/1538-4357/aab6b1
-
[28]
Jones, D. O., Scolnic, D. M., Foley, R. J., et al. 2019, ApJ, 881, 19, doi: 10.3847/1538-4357/ab2bec
-
[29]
Kirshner, R. P. 2010, ApJ, 715, 743, doi: 10.1088/0004-637X/715/2/743
-
[30]
Kenworthy, W. D., Jones, D. O., Dai, M., et al. 2021, ApJ, 923, 265, doi: 10.3847/1538-4357/ac30d8
-
[31]
Kenworthy, W. D., Goobar, A., Jones, D. O., et al. 2025, A&A, 697, A125, doi: 10.1051/0004-6361/202452578
-
[32]
2025, arXiv e-prints, arXiv:2506.04402
Kessler, R., Hounsell, R., Joshi, B., et al. 2025, arXiv e-prints, arXiv:2506.04402. https://arxiv.org/abs/2506.04402
-
[33]
Kessler, R., & Scolnic, D. 2017, ApJ, 836, 56, doi: 10.3847/1538-4357/836/1/56
-
[34]
2023, ApJL, 952, L8, doi: 10.3847/2041-8213/ace34d
Kessler, R., Vincenzi, M., & Armstrong, P. 2023, ApJL, 952, L8, doi: 10.3847/2041-8213/ace34d
-
[35]
Kessler, R., Becker, A. C., Cinabro, D., et al. 2009a, The Astrophysical Journal Supplement Series, 185, 32, doi: 10.1088/0067-0049/185/1/32
-
[36]
Kessler, R., Bernstein, J. P., Cinabro, D., et al. 2009b, Publications of the Astronomical Society of the Pacific, 121, 1028, doi: 10.1086/605984
-
[37]
Kessler, R., Narayan, G., Avelino, A., et al. 2019a, PASP, 131, 094501, doi: 10.1088/1538-3873/ab26f1 —. 2019b, PASP, 131, 094501, doi: 10.1088/1538-3873/ab26f1
-
[38]
Kunz, M., Bassett, B. A., & Hlozek, R. A. 2007, PhRvD, 75, 103508, doi: 10.1103/PhysRevD.75.103508
-
[39]
Lampeitl, H., Smith, M., Nichol, R. C., et al. 2010, ApJ, 722, 566, doi: 10.1088/0004-637X/722/1/566
-
[40]
2022, ApJS, 259, 58, doi: 10.3847/1538-4365/ac5033
Lochner, M., Scolnic, D., Almoubayyed, H., et al. 2022, ApJS, 259, 58, doi: 10.3847/1538-4365/ac5033 17
-
[41]
Marriner, J., Bernstein, J. P., Kessler, R., et al. 2011, ApJ, 740, 72, doi: 10.1088/0004-637X/740/2/72
-
[42]
doi:10.48550/arXiv.2512.06319 , archivePrefix =
Mitra, A., Kessler, R., Chen, R. C., et al. 2025, arXiv e-prints, arXiv:2512.06319. https://arxiv.org/abs/2512.06319
-
[43]
2024, arXiv e-prints, arXiv:2501.05632, doi: 10.48550/arXiv.2501.05632
Collaboration, The Roman HLIS Project Infrastructure Team, et al. 2024, arXiv e-prints, arXiv:2501.05632, doi: 10.48550/arXiv.2501.05632
-
[44]
Measurements of Omega and Lambda from 42 High-Redshift Supernovae
Perlmutter, S., Aldering, G., Goldhaber, G., et al. 1999, ApJ, 517, 565, doi: 10.1086/307221
work page internal anchor Pith review doi:10.1086/307221 1999
-
[45]
Phillips, M. M. 1993, ApJL, 413, L105, doi: 10.1086/186970
-
[46]
Pierel, J. D. R., Jones, D. O., Dai, M., et al. 2021, ApJ, 911, 96, doi: 10.3847/1538-4357/abe867
-
[47]
Pierel, J. D. R., Jones, D. O., Kenworthy, W. D., et al. 2022, ApJ, 939, 11, doi: 10.3847/1538-4357/ac93f9 Planck Collaboration, Aghanim, N., Akrami, Y., et al. 2020, A&A, 641, A1, doi: 10.1051/0004-6361/201833880
-
[48]
Popovic, B., Brout, D., Kessler, R., & Scolnic, D. 2023, ApJ, 945, 84, doi: 10.3847/1538-4357/aca273
-
[49]
Popovic, B., Kenworthy, W. D., Ginolin, M., et al. 2025, arXiv e-prints, arXiv:2506.05471, doi: 10.48550/arXiv.2506.05471
-
[51]
2021, The Astronomical Journal, 162, 67, doi: 10.3847/1538-3881/ac0824
Qu, H., Sako, M., M¨ oller, A., & Doux, C. 2021, The Astronomical Journal, 162, 67, doi: 10.3847/1538-3881/ac0824
-
[52]
Ridgway, S. T. 2016, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9911, Modeling, Systems Engineering, and Project Management for Astronomy VI, ed. G. Z. Angeli & P. Dierickx, 991125, doi: 10.1117/12.2232680
-
[53]
Observational Evidence from Supernovae for an Accelerating Universe and a Cosmological Constant
Riess, A. G., Filippenko, A. V., Challis, P., et al. 1998, AJ, 116, 1009, doi: 10.1086/300499 Roman Core Community Survey Definition Committees. 2025, Interim Report: The Community Definition of Roman’s Core Community Surveys, Tech. rep., NASA Goddard Space Flight Center. https: //asd.gsfc.nasa.gov/roman/comm forum/forum 17/ Core Community Survey Reports-...
-
[54]
M., Baltay, C., Hounsell, R., et al
Rose, B. M., Baltay, C., Hounsell, R., et al. 2021, arXiv e-prints, arXiv:2111.03081, doi: 10.48550/arXiv.2111.03081
-
[55]
M., Vincenzi, M., Hounsell, R., et al
Rose, B. M., Vincenzi, M., Hounsell, R., et al. 2025, arXiv e-prints, arXiv:2506.05161, doi: 10.48550/arXiv.2506.05161
-
[56]
Rubin, D., Aldering, G., Betoule, M., et al. 2025, The Astrophysical Journal, 986, 231, doi: 10.3847/1538-4357/adc0a5
-
[57]
Rubin, D., Aldering, G., Fruchter, A., et al. 2025, arXiv e-prints, arXiv:2506.04327, doi: 10.48550/arXiv.2506.04327 S´ anchez, B. O., Kessler, R., Scolnic, D., et al. 2022, ApJ, 934, 96, doi: 10.3847/1538-4357/ac7a37
-
[58]
Schmidt, B. P., Suntzeff, N. B., Phillips, M. M., et al. 1998, ApJ, 507, 46, doi: 10.1086/306308
-
[59]
, archivePrefix = "arXiv", eprint =
Scolnic, D., Rest, A., Riess, A., et al. 2014, ApJ, 795, 45, doi: 10.1088/0004-637X/795/1/45
-
[60]
Scolnic, D. M., Jones, D. O., Rest, A., et al. 2018, ApJ, 859, 101, doi: 10.3847/1538-4357/aab9bb
-
[61]
Wide-Field InfrarRed Survey Telescope-Astrophysics Focused Telescope Assets WFIRST-AFTA 2015 Report
Spergel, D., Gehrels, N., Baltay, C., et al. 2015, arXiv e-prints, arXiv:1503.03757, doi: 10.48550/arXiv.1503.03757
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1503.03757 2015
-
[62]
Sullivan, M., Conley, A., Howell, D. A., et al. 2010, MNRAS, 406, 782, doi: 10.1111/j.1365-2966.2010.16731.x
-
[63]
2012, , 746, 85, 10.1088/0004-637X/746/1/85
Suzuki, N., Rubin, D., Lidman, C., et al. 2012, The Astrophysical Journal, 746, 85, doi: 10.1088/0004-637X/746/1/85
-
[64]
Taylor, G., Lidman, C., Popovic, B., & Abbot, H. J. 2024, MNRAS, 528, 4643, doi: 10.1093/mnras/stae293
-
[65]
Taylor, G., Jones, D. O., Popovic, B., et al. 2023, MNRAS, 520, 5209, doi: 10.1093/mnras/stad320 The LSST Dark Energy Science Collaboration,
-
[66]
The lsst dark energy science collaboration (desc) science requirements document,
Mandelbaum, R., Eifler, T., et al. 2018, arXiv e-prints, arXiv:1809.01669, doi: 10.48550/arXiv.1809.01669
-
[67]
Vincenzi, M., Brout, D., Armstrong, P., et al. 2024a, ApJ, 975, 86, doi: 10.3847/1538-4357/ad5e6c —. 2024b, ApJ, 975, 86, doi: 10.3847/1538-4357/ad5e6c
-
[68]
Wang, Q., Jones, D. O., Pierel, J. D. R., et al. 2025, arXiv e-prints, arXiv:2512.25064, doi: 10.48550/arXiv.2512.25064
-
[69]
Wang, Y. 2008, PhRvD, 77, 123525, doi: 10.1103/PhysRevD.77.123525 18 APPENDIX A.VALIDATION OF TRAINED MODELS We compare both models’ performance at recovering SN parameters for some simulatedRomanlight curves. Both models show broadly similar distributions for most of the key SN parameters and their uncertainties as well as redshift and redshift errors. T...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.