Optimization of the multiple sampling and signal extraction in non-destructive exposures

A. Castera; B. Kubik; E. Chabanat; G. Smadja; R. Barbier; S. Ferriol

REVIEW 3 major objections 8 minor 6 references

Optimization of the multiple sampling and signal extraction in non-destructive exposures

T0 review · 3 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fitting nondestructive ramps with the full covariance matrix raises SNR by 5–7% and halves intercept noise in long spectroscopic exposures.

desk verdict Correct covariance derivations and useful readout guidance, but the headline SNR gain is mostly weighting, not correlation—worth a serious referee with an added baseline. read the letter →

arxiv 2506.04904 v1 pith:JBW4MPVI submitted 2025-06-05 astro-ph.IM

B. Kubik , R. Barbier , A. Castera , E. Chabanat , S. Ferriol , G. Smadja This is my paper

classification astro-ph.IM

keywords nondestructivereadoutHgCdTedetectorscovariancematrixsignal-to-noiserationoiseshotMACClinearfitting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to show that the noise in non-destructive infrared exposures is correlated in a way that ordinary least-squares fitting ignores, and that using the full covariance matrix in the fit recovers a higher signal-to-noise ratio. The authors derive closed-form covariance matrices for averaged frame groups and for differences between groups, and test them on simulated ramps matching photometric and long spectroscopic exposures. They find that above about 1 electron per second per pixel the covariance-based fits beat the usual slope fit by 5–7% in SNR, and cut the noise on the t=0 intercept by 50–60% in long spectroscopic exposures. They also argue that the best readout strategy is to use every frame in the ramp rather than coadding frames once the flux exceeds that threshold.

What carries the argument

The central objects are two covariance matrices, C for group values and D for group differences, whose entries are derived analytically from the independence assumptions of Equations (1): inter-frame fluxes are Poisson with mean f, inter-group fluxes Poisson with mean d, and readout noise is Gaussian with width σ_R. The diagonal of C includes both the readout variance after coadding and the shot-noise variance weighted by the triangular correlations inside a group, while the off-diagonal entries carry the accumulated Poisson correlations between groups; for D the off-diagonal terms mix a positive shot-noise correlation with a negative readout-noise correlation. The paper then uses these matrices in a generalized least-squares (COV/COVd) fit, which is what turns the analytic covariance into better SNR and a much better y-intercept.

What would settle it

Take long dark ramps on a real nondestructive-readout array, compute the sample covariance between group differences, and compare its off-diagonal entries with the readout-plus-shot-noise prediction of this paper; if the off-diagonal entries are significantly larger (as they would be with 1/f noise), the covariance model is incomplete and the SNR improvements would not be reached in practice.

Watch

Extended reading notes

Core claim

The paper derives closed-form expressions for the full covariance matrix of coadded frame groups and of differences between groups in a non-destructive ramp, including both the Poisson shot-noise correlations accumulated along the ramp and the frame-to-frame readout noise. Feeding these matrices into a straight-line fit (instead of giving every sample equal weight) yields a higher signal-to-noise ratio than an unweighted least-squares fit: about 5–7% higher in long spectroscopic exposures when the flux is above 1 electron per second per pixel, and about 1% in shorter photometric exposures. The biggest gain is on the fitted intercept at t=0, whose noise falls by 50–60% in spectroscopic exposures, giving a more reliable cross-check of signal quality. The same analysis shows that the readout mode using every sampled frame in the ramp maximizes SNR at all fluxes studied, and that for fluxes above 1 electron per second per pixel coadding frames within groups should be avoided.

Load-bearing premise

The analysis assumes readout noise is uncorrelated from frame to frame and the source flux is steady, leaving out the low-frequency correlated noise that real detectors show; if that omitted noise is large, the derived covariance matrix understates the true correlations and the reported gains shrink.

Editorial extensions

If this is right

In long spectroscopic exposures above 1 electron per second per pixel, pipelines using plain unweighted least squares are leaving roughly 5–7% of the achievable SNR unused.
The much tighter y-intercept estimate gives cosmic-ray rejection and persistence or nonlinearity corrections a more accurate baseline from the same data.
For fluxes above 1 electron per second per pixel, coadding frames into groups should be avoided; using all sampled frames directly is the SNR-optimal readout mode.
Below 1 electron per second per pixel, the SNR is insensitive to how frames are grouped, so grouping can be chosen for cosmic-ray rejection rather than noise.
Because the covariance matrix can accept extra noise terms, the same fitting machinery can be extended to real-detector effects such as 1/f noise.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If real detectors' 1/f noise is significant, the off-diagonal entries of the empirical covariance will exceed the model's predictions, so the quoted SNR gains should be read as an upper bound until a 1/f term is added.
The y-intercept improvement suggests a practical test: in spectroscopic dark exposures, comparing the scatter of fitted intercepts between COV and LSF fits over many ramps would show the gain directly without needing simulated data.
The optimal-readout conclusion rests on ignoring cosmic rays; with realistic hit rates, the best mode may trade a few percent of SNR for more groups to identify and remove hits.
The same covariance construction should apply to any detector with nondestructive reads, not only the HgCdTe arrays modeled here.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the covariance matrices are right, the Monte Carlo backs them, and the readout-mode guidance is useful. Your stress-test note lands. The abstract over-credits the off-diagonal correlations.

I re-derived Eqs. 3, 4, 6, and 7 and they match the text. The simulated MACC(4,16,0) matrix agrees at the few-percent level, which is what you want from a closed-form noise model. The genuinely new content is the explicit covariance matrices for group averages and group differences; Refs. 3 and 5 give total variances or cosmic-ray recipes but not these matrices. That is a real, citable contribution for anyone building a ramp-fitting pipeline.

The soft spot is attribution. The only baseline is unweighted LSF. There is no fit that uses diagonal variances only, so you cannot tell how much of the 5–7% SNR gain comes from the off-diagonal terms. Table 1 already shows LSFd matching COV to within 0.1%, which tells you the gain over LSF is essentially weighting, not correlation. The paper is honest that LSFd and COV perform the same, but the abstract and conclusions still say 'improvement arising from the covariance matrix.' That overstates the role of the correlations, and the same applies to the y-intercept improvement: real, but not shown to be a correlation effect.

Minor issues: Tables 1 and 2 have no error bars, though with 10,000 sims the quoted numbers are stable. The criticism of Ref. 3 ('covariance matrix terms are not properly organized') is vague and unsupported; either point to the specific problem or drop the claim. The 1/f limitation is acknowledged, and it means real HgCdTe data may show smaller gains than simulated.

Who is this for? Detector and pipeline people on Euclid NISP or any HgCdTe ramp readout. It deserves a serious referee. The math is correct, the simulations verify it, and the readout-mode scan is useful. I would accept for review and ask for two additions: a diagonal-only weighted baseline and a rewrite of the abstract/conclusions to say 'proper weighting' rather than 'covariance matrix.'

Referee Report

3 major / 8 minor

Summary. The paper derives analytic covariance matrices for the noise in MACC (multiple accumulated) nondestructive readouts, separately for coadded group values (matrix C) and for group differences (matrix D), under a noise model consisting of uncorrelated readout noise and Poisson shot noise. The matrices are introduced into linear fits of the up-the-ramp signal, and the resulting SNR and y-intercept noise are compared with unweighted least-squares fits on simulated spectroscopic (MACC(15,16,9), 549 s) and photometric (MACC(4,16,0), 96 s) exposures. The authors report 5-7% SNR improvements and 50-60% reductions in intercept noise for the spectroscopic case, and conclude that the optimal readout mode uses all frames without coadding for fluxes above 1 e-/s/pix.

Significance. If the results hold, the paper provides a useful, self-contained derivation of covariance matrices for a standard readout mode of HgCdTe detectors, with a transparent noise model and Monte Carlo verification. The explicit acknowledgement that 1/f noise is excluded and the analytic form of the matrices are strengths. However, the central quantitative claim - that the improvement arises from the covariance matrix - is not yet separated from the benefit of optimal variance weighting, and the numerical example in Section 2 contains an internal inconsistency. The practical recommendation about the optimal readout mode is of direct interest to Euclid NISP and similar instruments.

major comments (3)

[Section 3, Tables 1-2, Figures 2-3] The claimed benefit of the full covariance matrix is not isolated from the benefit of optimal variance weighting. The only baselines reported are unweighted least squares on groups (LSF) and on differences (LSFd); no weighted least-squares fit using only the diagonal variances diag(C) or diag(D) is reported. Since the diagonal variances in the MACC(4,16,0) example grow from about 28 to 208 e-^2 along the ramp, the unweighted LSF gives excessive weight to the noisier late groups; a large part of the reported SNR and especially the y-intercept improvement may therefore be a pure weighting effect rather than a correlation effect. Indeed, Table 1 shows COVd and LSFd agreeing to <0.1%, indicating that the off-diagonal terms of D contribute essentially nothing; the analogous control for C is missing. Please add diagonal-only weighted baselines (or at least report the corresponding SNR and intercept noise) so that the improvement can be attributed to the off-diagonal correlations, as claimed in the abstract and conclusions.
[Section 2.2-2.3, Eqs. (3)-(7)] The numerical example is internally inconsistent. For MACC(4,16,0) with a 96-s exposure (64 frames) and a quoted flux of 2.5 e-/s, the per-frame signal is f = 3.75 e- and the inter-group signal is d = 3.75 e-; these values reproduce the quoted C matrix (e.g., C11 = 28.16, C44 = 208.16). However, the D matrix quoted immediately below for the same readout and flux is reproduced only with f = 3.0 e- and d = 3.0 e- (giving Dkk = 44.56 and off-diagonal 1.72). Thus the two covariance matrices are not evaluated for the same input parameters. Please correct the example or specify the actual f and d values used.
[Section 3, Tables 1-2] The SNR and y-intercept noise values are Monte Carlo estimates over 10,000 simulated ramps, but no statistical uncertainties are reported. The relative error on a sample standard deviation with 10,000 trials is about 0.7%, so the reported ~1% SNR gain for the photometric exposure (e.g., 8.16 to 8.24 at 1 e-/s) is within the Monte Carlo noise. Please quote error bars or confidence intervals, or state the number of trials and the standard error, so that the significance of the claimed improvements can be assessed.

minor comments (8)

[Abstract] "We derive the full covariance matrix formulae are derived" should be corrected to "We derive the full covariance matrix formulae", and "shell be avoided" should be "should be avoided".
[Section 2.1 and throughout] Clarify that the quantity f in the equations is the expected signal per frame, not the flux per second, and give the conversion f = flux × frame_time for each readout mode; the example says "about 100 sec" but Table 1 uses 96 s for MACC(4,16,0).
[Section 2.2, Example] When quoting a flux of 2.5 e-/s, specify the frame time used (1.5 s for a 96-s exposure) so the reader can reproduce f = 3.75 e-/frame.
[Section 3] Define the SNR as a/σ_a (slope divided by its standard deviation) and clarify that the y-intercept noise is σ_b.
[Figures 4-5] The axes are labeled with f in e-/frame, while the text and conclusions refer to fluxes in e-/s/pix; make the conversion explicit in the captions.
[Figure 1 caption] "lats frame" should be "last frame".
[Section 4] "trig on a sudden brake" should be "trigger on a sudden break", and "shell be avoided" should be "should be avoided".
[Section 2.3, after Eq. (7)] State explicitly that the off-diagonal entries of D are non-zero only for adjacent differences (|k−l|=1), as the quoted matrix shows.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the covariance matrices are derived analytically from a stated noise model, verified by Monte Carlo simulation, and the reported gains are measured consequences of inverse-covariance weighting, with no fitted parameters renamed as predictions and no load-bearing self-citations.

full rationale

The derivation chain is self-contained. Section 2 defines the noise model (Eq. 1: independent Poisson inter-frame fluxes with mean f, independent inter-group fluxes with mean d, uncorrelated Gaussian readout noise of width sigma_R) and computes every covariance element C_kk (Eq. 3), C_kl (Eq. 4), and the difference-fit elements D (Eqs. 5-7) by direct error propagation from the explicit coadded-group expressions (Eq. 2). These analytic matrices are then cross-checked against Monte Carlo simulations, with C_simu and D_simu agreeing with the analytic C and D; that is independent verification, not a circular input. In Section 3, the SNR and y-intercept noise of the weighted fits (COV, COVd) versus unweighted fits (LSF, LSFd) are computed from 10,000 simulated ramps generated under the same stated model, and the reported 5-7% SNR gain and 50-60% intercept-noise reduction follow from Gauss-Markov inverse-covariance weighting rather than from any fitted quantity; no parameter is fitted to the data whose outcome is then reported as a prediction. The paper invokes no load-bearing self-citation: all six references are external (Fowler & Gatley 1990, Garnett & Forrest 1993, Rauscher et al. 2007, Vacca et al. 2004, Anderson & Gordon 2011, Hogg et al. 2010), and the paper explicitly re-derives rather than merely imports the total variance of Rauscher et al., extending it to the off-diagonal terms. The acknowledged 1/f exclusion (Section 2.2: 'this would be the case of the 1/f noise which correlates all the groups in the ramp and should be incorporated into the covariance matrix to fit real data. It is however out of the scope of this paper') is an explicit scope limitation on transfer to real HgCdTe detectors, weighed here as a conditionality caveat rather than a circularity. One non-circular robustness caveat remains: Tables 1-2 compare full-covariance fits only against unweighted LSF, with no diagonal-variances-only weighted baseline, so the abstract's 'improvement arising from the covariance matrix' does not separately identify how much of the gain comes from off-diagonal correlations versus optimal weighting alone; that concerns attribution of the cause of a measured gain, not a reduction of any claim to its own inputs.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new physical entities and fits no free parameters. All inputs (flux f, readout noise sigma_R, group structure) are treated as given properties of the detector and observation. The correlation structure is derived, not tuned. The load-bearing axioms are the standard noise model of near-infrared integrating detectors and the stationarity assumption.

assumptions (6)

domain assumption Incident photons follow Poisson statistics with a single mean flux per frame
Invoked throughout Section 2.1 via eq. 1 to model the inter-frame integrated fluxes as independent Poisson variables.
domain assumption Readout noise is Gaussian, uncorrelated between frames, with a common width sigma_R
Stated in Section 2.1 in eq. 1 and used to construct the covariance matrices.
domain assumption The average signal between reset and first readout equals the average signal between successive frames
Footnote in Section 2.2; this implies f0 has the same variance f as other frame increments, which enters the covariance formulas.
domain assumption The incident flux is stationary during the exposure
Implicitly assumed because every inter-frame increment has the same mean f (eq. 1). Violated by cosmic rays, persistence, or source variability.
domain assumption No 1/f or other correlated noise beyond readout and shot noise
Explicitly stated in Section 2 as out of scope; if present, the covariance matrix understates correlations.
domain assumption The signal is well modeled by a straight line s = at + b
Used throughout Section 3 for the fits; the intercept b is the object of the claimed improvement.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Optimization of the multiple sampling and signal extraction in non-destructive exposures." pith.science (2026). https://pith.science/paper/JBW4MPVI

@misc{pith2026250604904,
  author       = {Pith},
  title        = {Pith review of: Optimization of the multiple sampling and signal extraction in non-destructive exposures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JBW4MPVI}},
  note         = {Machine review of arXiv:2506.04904}
}

read the original abstract

We derive the full covariance matrix formulae are derived for proper treatment of correlations in signal fitting procedures, extending the results from previous publications. The straight line fits performed with these matrices demonstrate that a significantly higher signal to noise is obtained when the fluence exceeds 1 e/sec/pix in particular in long (several hundreds of seconds) spectroscopic exposures. The improvement arising from the covariance matrix is particularly strong for the initial intercept of the fit at t=0, a quantity which provides a useful redundancy to cross check the signal quality. We demonstrate that the mode that maximizes the signal to noise ratio in all ranges of fluxes studied in this paper is the one that uses all the frames sampled during the exposure. While at low flux there is no restriction on the organization of frames within groups for fluxes lower than 1 e/sec/pix, for fluxes exceeding this value the coadding of frames shell be avoided.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

6 extracted references · 5 canonical work pages

[1]

A. M. Fowler, I. Gatley, ”Demonstration of an algorithm for read-noise reduction in infrared arrays,” The Astrophysical Journal, 353:L33-L34 (1990)

work page 1990
[2]

J. D. Garnett, W. Forrest, ”Multiply sampled read limited and background limited noise performance,” Proc. SPIE, 1946, 395 (1993)

work page 1993
[3]

B. J. Rauscher et al. ”Detectors for the James Webb Space Telescope Near-Infrared Spectrograph. I. Readout Mode, Noise Model, and Calibration Considerations,” PASP, 119, 768-786 (2007)

work page 2007
[4]

W. D. Vacca, M. C. Cushing, J. T. Rayner, ”Nonlinearity Corrections and Statistical Uncertainties As- sociated with Near-Infrared Arrays,” PASP, 116, 352-361 (2004)

work page 2004
[5]

R. E. Anderson, K. D. Gordon, ”Optimal Cosmic-Ray Detection for Nondestructive Read Ramps.” PASP, 123, 1237-1248 (2011)

work page 2011
[6]

D. W. Hogg, J. Bovy, D. Lang, ”Data analysis recipes: Fitting a model to data,” arXiv:1008.4686 [astro- ph.IM] (2010) 18 gn0 10 20 30 40 50 60 SNR 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 0.22 - = 10.0 eRσ/sec, - = 96.00 sec, f = 0.01 eexpt LSF COV nf = 1 nf = 2 nf = 4 nf = 8 nf = 16 gn 0 10 20 30 40 50 60 SNR 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 0.22 - = 1...

arXiv 2010

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

A. M. Fowler, I. Gatley, ”Demonstration of an algorithm for read-noise reduction in infrared arrays,” The Astrophysical Journal, 353:L33-L34 (1990)

work page 1990

[2] [2]

J. D. Garnett, W. Forrest, ”Multiply sampled read limited and background limited noise performance,” Proc. SPIE, 1946, 395 (1993)

work page 1993

[3] [3]

B. J. Rauscher et al. ”Detectors for the James Webb Space Telescope Near-Infrared Spectrograph. I. Readout Mode, Noise Model, and Calibration Considerations,” PASP, 119, 768-786 (2007)

work page 2007

[4] [4]

W. D. Vacca, M. C. Cushing, J. T. Rayner, ”Nonlinearity Corrections and Statistical Uncertainties As- sociated with Near-Infrared Arrays,” PASP, 116, 352-361 (2004)

work page 2004

[5] [5]

R. E. Anderson, K. D. Gordon, ”Optimal Cosmic-Ray Detection for Nondestructive Read Ramps.” PASP, 123, 1237-1248 (2011)

work page 2011

[6] [6]

D. W. Hogg, J. Bovy, D. Lang, ”Data analysis recipes: Fitting a model to data,” arXiv:1008.4686 [astro- ph.IM] (2010) 18 gn0 10 20 30 40 50 60 SNR 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 0.22 - = 10.0 eRσ/sec, - = 96.00 sec, f = 0.01 eexpt LSF COV nf = 1 nf = 2 nf = 4 nf = 8 nf = 16 gn 0 10 20 30 40 50 60 SNR 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 0.22 - = 1...

arXiv 2010