{"id":"706f57be-1033-4023-afae-51e8b10147f0","arxiv_id":"2608.10539","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On MagAO-X, incoherent speckle probes can estimate optical gain in real time and roughly track a benchmark measurement, but they underestimate at high gain and the focal-plane Strehl method is unreliable.","lead":"This paper compares three ways to measure optical gain, a calibration error in pyramid wavefront sensors, on the MagAO-X adaptive optics system. The pupil-plane speckle method is the most practical, but it underestimates gain at high values and needs more work.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The speckle OG estimator's lab-to-sky transfer is unvalidated: §4.3 reports sky KL patterns 'aren't seen as clearly,' and the averaging rule was chosen post hoc against selfRM, so the claimed realtime OG estimate lacks independent support.","rationale":"The reader's conditional verdict is appropriate. I agree with the reader's weakest assumption: the lab-to-sky transfer of the PCA basis is the load-bearing link, and the paper's own note in Fig. 8 that sky patterns are not seen as clearly is an explicit admission that this link is not established. I would emphasize further that the averaging rule was chosen post hoc on the same 8 datasets, so the agreement with selfRM in Fig. 10 is not independent confirmation. The high-OG underestimation is acknowledged but unresolved, and it is exactly where the method would need to be reliable for realtime control. The proposed concrete test is a pre-registered holdout validation; this is standard practice and would settle whether the estimator is transportable. I do not raise concerns about the selfRM benchmark itself, and the paper is honest about its limitations. The method is plausible and the implementation is described, but the central claim is not yet independently validated, so the verdict remains conditional rather than full acceptance or rejection.","tokens_in":11294,"tokens_out":5905,"duration_ms":77960,"concrete_test":"Pre-register the estimator exactly as specified in §4.3 (lab PCA basis per probe configuration, simple mean of the three KL-mode RMS ratios), then apply it to the next on-sky datasets with simultaneous selfRM measurements before comparing to the benchmark. If the frozen estimator does not track selfRM across the 0.4–1.0 OG range, or if the high-OG bias persists, the central claim fails because the estimator was overfit and/or the lab-to-sky transfer is not valid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3's conclusion—'We will use the average of the three projected modes as our OG scalar estimate'—requires that the PCA basis built from a 2-second lab dataset (Sec. 4.1) represent the on-sky WFS response, and that the lab-normalized RMS of the projection be proportional to OG. The paper's own Fig. 8 caption states that on sky 'these patterns aren't seen as clearly on the sky dataset, which is still being investigated.' If the sky projections are degraded or contaminated by turbulence, the ratio to lab RMS is not an OG measurement. The calibration also assumes the temporal misalignment between DM command and WFS frame is identical on sky; the third KL mode is specifically a transition mode, and any change in loop delay or integration phase would change the basis. In addition, the averaging rule was adopted after observing that no single mode matched the selfRM; with only 8 datasets and a systematic high-OG underestimate (Fig. 10), the agreement of the average could be a post hoc artifact rather than a validated estimator. The claim that the method is 'usable' for realtime control therefore rests on an unresolved transfer assumption and an unvalidated averaging rule.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares three methods for measuring optical gain (OG) on the MagAO-X pyramid wavefront sensor: a self response matrix (selfRM) benchmark, a focal-plane Strehl-ratio measurement with the CamTip camera, and a pupil-plane method using incoherent speckle probes already applied for coronagraphic observations. The speckle method is the paper's main new contribution: a PCA decomposition of a 2-second lab dataset yields three KL modes that are projected onto individual WFS frames on sky, and the lab-normalized rolling RMS of these projections is averaged to produce a realtime OG scalar. The authors report 8 on-sky speckle datasets with corresponding selfRM measurements, finding close agreement in 5 cases and systematic underestimation at high OG, and conclude that the averaged three-mode projection is the preferred realtime OG estimate moving forward. The CamTip SR method is found to overestimate relative to an error budget, which the authors attribute to an inability to capture fitting error.","tokens_in":11576,"tokens_out":4133,"duration_ms":39043,"significance":"If validated, the incoherent-speckle OG estimator would be a valuable passive diagnostic, because MagAO-X already applies these probes for coronagraphic centering and photometric references, so OG could be monitored without interrupting science. The paper is honest and specific about its limitations: it explicitly notes that CamTip cannot capture fitting error, that the speckle method saturates at high OG, that only 8 speckle datasets were collected, and that the sky-side behavior of the KL patterns is still under investigation. The authors also provide a clear benchmark in the selfRM and give reproducible definitions of their reduction steps. However, the central claim that the averaged KL-mode RMS is a usable realtime OG estimate is not yet independently established: the lab-to-sky transfer of the PCA basis is unvalidated, the averaging rule was chosen post hoc on the same datasets used for validation, and the high-OG regime shows a systematic bias.","major_comments":[{"comment":"The lab-to-sky transfer of the PCA-derived KL basis is not validated. Figure 8's caption states that the expected four-frame and two-frame patterns \"aren't seen as clearly on the sky dataset, which is still being investigated,\" yet the estimator is exactly the ratio of sky-frame projections to lab-frame RMS in a fixed lab-built basis. Any sky-specific change in the spatial structure of the WFS response, not just an overall gain change, will alter this ratio. The paper should quantify how well the sky projections reproduce the expected KL templates, for example by reporting per-mode explained variance, template correlation, or a comparison against a sky-derived basis. Without such evidence, the claim that the ratio measures OG is not independently supported.","section":"Section 4.3, Fig. 8"},{"comment":"The decision to average the three KL-mode ratios was made after observing that the average matched selfRM better than any individual mode: the text reads \"When the KL mode returns are averaged, we see that the signal has a better match to the selfRM OG. We will use the average of the three projected modes as our OG scalar estimate moving forward.\" With only 8 datasets, five of which are close to selfRM, this is a post hoc selection of the estimator definition on the validation set. The authors should either pre-specify the averaging rule, validate it with a leave-one-out or nested procedure, or report per-mode agreement and show that the average improves performance in a way that is not simply a regression artifact.","section":"Section 4.3 and Section 4.4"},{"comment":"The temporal alignment between the DM command and the WFS integration is a load-bearing assumption. The authors explain that a simple Fourier projection failed because the WFS frames integrate over the DM probe mid-transition, and that the third PCA mode specifically captures this transition. The estimator therefore assumes that the lab and on-sky loop timing, including loop delay and integration phase, are identical. Any change in these would alter the projection coefficients even at fixed OG. The paper should demonstrate timing stability across the eight datasets, or estimate and correct the per-dataset delay, before the estimator can be considered reliable for realtime control.","section":"Section 4.1 and Section 4.3"},{"comment":"The high-OG behavior is not adequately characterized. The paper reports that for higher optical gains the method underestimates OG, and Figure 10 shows a systematic deviation from the unity line in this regime. Because the stated goal is realtime OG control, and because the high-OG regime is precisely where undercorrection would be most problematic, the conclusion that the method is \"usable\" is only supported for the mid-range OG cases. The authors should either restrict the claim to the regime where the estimator is calibrated, model the saturation effect, or show that the underestimation is correctable with a simple transformation.","section":"Section 4.4, Fig. 10"}],"minor_comments":[{"comment":"The abstract contains a typo: \"asses\" should be \"assess.\"","section":"Abstract and Section 1"},{"comment":"The paragraph beginning \"Each set of likely separations\" would benefit from a table listing the specific probe configurations (separation, strength, angle) that were calibrated, since the paper later relies on these configurations being reproducible on sky.","section":"Section 4.2"},{"comment":"The caption states that error bars indicate variance around the average measurement, but does not specify whether this is variance over time samples, over independent speckle cycles, or over spatial modes. Please define the averaging domain and the number of independent samples used.","section":"Figure 10 caption"},{"comment":"The phrase \"2-seconds set\" should be \"2-second set,\" and the earlier hyphenation in \"Karhunen–Lo`eve\" should be consistent with standard notation.","section":"Section 4.1"},{"comment":"The claim that bright-star outliers are \"most likely due to saturation\" on the focal plane is plausible but unsupported; a check of peak counts against the detector full well, or a note in Table 1 flagging saturated frames, would make the statement testable.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"This is an honest and useful engineering report, but the central claim about the speckle-based realtime OG estimator needs stronger validation before acceptance. The two key issues are the unvalidated lab-to-sky transfer of the PCA basis and the post hoc selection of the averaging rule. Both can be addressed with additional analysis of existing data, so I do not recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one. The real new content is the on-sky comparison of three optical-gain estimators on MagAO-X, with selfRM as benchmark, plus the PCA/KL averaging scheme for incoherent speckles and a documented high-OG saturation. The individual methods are not new—selfRM and incoherent speckle probes come from the same group's earlier work, and focal-plane gain sensing has been done elsewhere—but the side-by-side test on sky and the failure modes are real data.\n\nWhat the paper does well: it separates a direct modal OG measurement (selfRM) from practical proxies, and it is honest that the focal-plane camera cannot capture fitting error. The CamTip versus error-budget discussion is candid, and the speckle method's advantage—using probes already applied for coronagraphy, with no extra light loss—is genuine. The lab PCA/KL projection is a reasonable response to the DM/WFS timing mismatch, and the authors explicitly flag the high-OG underestimation as unresolved.\n\nThe soft spots are real and should be kept in proportion. The averaging rule for the three KL modes was adopted after seeing that the average matched selfRM better, on eight datasets. That is post hoc selection, even if not intentional, and with five close points and three underestimates the estimator is not independently validated. The lab-to-sky transfer is also shaky: the paper's own figure caption notes that sky KL patterns 'aren't seen as clearly' and that this is 'still being investigated.' If the sky projections are contaminated by turbulence, the lab-normalized ratio is not a clean OG measurement. No raw data or scripts are released, which matters for an engineering claim with this little statistics. None of this kills the paper, but it should keep the central claim provisional.\n\nThe citation pattern looks fine: prior LBT, SCExAO, and gain-scheduling work is credited, and the self-citations point to the earlier selfRM and speckle papers where they belong.\n\nThis paper is for people working on PyWFS real-time control, non-common-path correction, or PSF reconstruction. It deserves a serious referee and, in my view, publication as an engineering report after minor revision. I would ask for more datasets, a clearer statement that the averaged speckle OG is a provisional estimator rather than 'the one we use moving forward,' and ideally a release of the reduction scripts.","headline":"Honest on-sky comparison of three optical-gain estimators; the speckle method is promising but not yet validated, and the authors are mostly open about that.","tokens_in":12161,"tokens_out":2578,"would_cite":true,"duration_ms":26386,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The average of three projected Karhunen–Loève modes from incoherent speckle probes provides a usable realtime optical-gain estimate on MagAO-X.","keywords":["adaptive optics","pyramid wavefront sensor","optical gain","real-time control","incoherent speckles","Karhunen–Loève modes","Strehl ratio","MagAO-X"],"falsifier":"A concrete test: on a night when the selfRM OG is high and varying, if the averaged KL-mode RMS stays pinned low while the selfRM value changes by more than the reported variance bars, the scalar is not tracking OG in that regime. Rebuilding the PCA basis from a different two-second lab segment and seeing whether the five close matches turn into mismatches would also falsify the calibration transfer; and recomputing the average without the third transition mode tests whether the high-gain underestimation is caused by the misalignment term.","tokens_in":11150,"feed_emoji":"🔭","tokens_out":6726,"duration_ms":55937,"temperature":0.7,"pith_summary":"Pyramid wavefront sensors lose sensitivity between the conditions used to calibrate them and the turbulent residuals they correct, an effect called optical gain (OG), and an unknown OG makes real-time control and post-processing less accurate. This paper shows that MagAO-X can estimate OG continuously from the incoherent speckle probes it already applies for coronagraphic calibration, without taking time away from science. The method projects wavefront-sensor frames onto the three dominant patterns (Karhunen–Loève modes) of a two-second laboratory PCA decomposition, computes a rolling RMS in the lab and on sky, and averages the three mode ratios; the paper reports this average tracks the benchmark selfRM optical gain in five of eight on-sky datasets. A focal-plane Strehl-ratio method was also tested and found to overestimate, mainly because the camera's field of view cannot capture fitting error.","feed_headline":"Averaged speckle probes track optical gain in real time","feed_subtitle":"MagAO-X can monitor pyramid wavefront sensor gain passively from probes it already uses; five of eight on-sky checks match.","key_machinery":"The central object is the rolling, lab-normalized RMS of the projections of the three leading KL modes. Because the deformable-mirror command and the wavefront-sensor integration are not frame-aligned, the probe signal is not a clean Fourier pattern; the PCA decomposition of a two-second lab dataset separates the two primary Fourier modes from a transition mode that carries the misalignment. Projecting each sky frame onto these three modes, dividing the rolling RMS by the lab RMS, and averaging across modes produces the scalar optical-gain estimate. The same KL basis also gets projected onto the control matrix to show which spatial modes each probe configuration excites.","core_discovery":"The central claim is that the average of the projected KL-mode returns is a usable realtime optical-gain scalar on MagAO-X. In Section 4.3 the authors state: \"When the KL mode returns are averaged, we see that the signal has a better match to the selfRM OG. We will use the average of the three projected modes as our OG scalar estimate moving forward.\" The selfRM is their benchmark: a mode-by-mode measurement of the full application and degradation of a deformable-mirror probe, too invasive to run continuously. The averaged projection is intended to reproduce that benchmark passively from probes already in the loop, and the paper reports that five of eight datasets land close to the selfRM value, with higher optical gains underestimated.","pith_inferences":["If the lab-to-sky transfer of the KL basis holds for other instruments that already use incoherent speckles, this passive estimator could be adopted without hardware changes; the paper only demonstrates it on MagAO-X.","The high-gain saturation suggests a concrete experiment: exclude the third, transition KL mode from the average and see whether the saturation disappears; if it does, the misalignment term, not the physics of optical gain, is limiting the estimate.","The choice of a single two-second lab dataset for the PCA basis is a free parameter; rebuilding the basis from longer or diversified lab data and measuring scatter against selfRM would show how much of the residual error is calibration-limited.","The natural next test is to close the loop on the estimate: divide the reconstructed wavefront by the reported OG and check whether the on-sky Strehl or the selfRM improves, a step the paper leaves future."],"forward_implications":["MagAO-X can monitor optical gain continuously with no change to observing operations, as long as the applied speckle configuration has a lab calibration.","The focal-plane CamTip Strehl method is not a trustworthy OG proxy in its current configuration, because it systematically overestimates Strehl by missing fitting error and saturating on bright targets.","The observed saturation at high optical gain means the averaged KL method needs a correction or a different weighting before it can be used for closed-loop gain compensation in good seeing.","Because the speckle responses are expressed in the control basis, the same data could support per-mode gain calibration rather than a single scalar, which the paper lists as immediate follow-up.","With a reliable live OG estimate, the reconstructor can be rescaled by the inverse gain, which is the step needed to correct non-common-path aberrations and to reconstruct point-spread functions from telemetry."],"supporting_citations":[{"why":"Provides the selfRM benchmark: the mode-by-mode measurement of optical gain that the averaged KL projection is meant to reproduce passively.","marker":"[21]"},{"why":"Introduces incoherent speckles as the probe pattern from which the wavefront-sensor signal is extracted.","marker":"[27]"},{"why":"Supplies the implementation of incoherent speckles that the MagAO-X pupil-plane method closely follows.","marker":"[28]"},{"why":"Supplies the precedent of slowly varying probes used to estimate optical gain and rescale the pyramid reconstructor.","marker":"[15]"},{"why":"Provides the end-to-end simulation used to attribute the flat selfRM OG curve to the pyramid's 1.9 arcsecond spatial filter.","marker":"[22]"},{"why":"Gives the convolutional-model framework for compensating pyramid optical gain, the broader control context this work feeds into.","marker":"[17]"}],"fun_headline_variants":["Averaged probes match optical gain benchmark on MagAO-X","Real-time gain from averaged speckle probes on MagAO-X","Passive optical gain tracking via probe averages","Five of eight on-sky checks confirm probe-based gain","Probe averages track optical gain in real time"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a pattern learned from two seconds of lab data describes how the wavefront sensor responds on sky, so that comparing lab and sky projections of that pattern yields the true optical gain.","fun_headline_variants_meta":{"raw":{"variants":["Averaged probes match optical gain benchmark on MagAO-X","Real-time gain from averaged speckle probes on MagAO-X","Passive optical gain tracking via probe averages","Five of eight on-sky checks confirm probe-based gain","Probe averages track optical gain in real time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1372,"prompt_tokens":904,"completion_tokens":468,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":390}},"tokens_in":520,"tokens_out":468,"duration_ms":4341,"temperature":1.0,"reasoning_tokens":390,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:16:06.986065+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: on a night when the selfRM OG is high and varying, if the averaged KL-mode RMS stays pinned low while the selfRM value changes by more than the reported variance bars, the scalar is not tracking OG in that regime. Rebuilding the PCA basis from a different two-second lab segment and seeing whether the five close matches turn into mismatches would also falsify the calibration transfer; and recomputing the average without the third transition mode tests whether the high-gain underestimation is caused by the misalignment term.","supporting_citations":[{"cited_title":"On-sky, real-time optical gain calibration on MagAO-X using incoherent speckles","cited_arxiv_id":"2407.13022","evidence_quote":"Provides the selfRM benchmark: the mode-by-mode measurement of optical gain that the averaged KL projection is meant to reproduce passively."},{"cited_title":"Artificial incoherent speckles enable precision astrometry and photometry in high-contrast imaging","cited_arxiv_id":"1510.04368","evidence_quote":"Introduces incoherent speckles as the probe pattern from which the wavefront-sensor signal is extracted."},{"cited_title":"Precision Photometric and Astrometric Calibration for Exoplanet Imaging,","cited_arxiv_id":null,"evidence_quote":"Supplies the implementation of incoherent speckles that the MagAO-X pupil-plane method closely follows."},{"cited_title":"Non common path aberration correction with non linear WFSs,","cited_arxiv_id":null,"evidence_quote":"Supplies the precedent of slowly varying probes used to estimate optical gain and rescale the pyramid reconstructor."},{"cited_title":"High Contrast Imaging for Python (HCIPy): an open-source adaptive optics and coronagraph simulator,","cited_arxiv_id":null,"evidence_quote":"Provides the end-to-end simulation used to attribute the flat selfRM OG curve to the pyramid's 1.9 arcsecond spatial filter."},{"cited_title":"Pyramid wavefront sensor optical gains compensation using a convolutional model","cited_arxiv_id":"2006.08294","evidence_quote":"Gives the convolutional-model framework for compensating pyramid optical gain, the broader control context this work feeds into."}],"review_version":1}