{"id":"f7f8be9a-be60-4500-9385-ddfb423d6e55","arxiv_id":"2501.07004","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Realistic MWA simulations show that an incomplete sky model is the dominant source of power leakage into the EoR window, and that subtracting sources down to roughly 1 mJy is required to recover the 21-cm signal.","lead":"This paper uses realistic computer simulations of the Murchison Widefield Array telescope to test how different instrumental defects corrupt attempts to measure the 21-cm signal from hydrogen during the epoch of reionization. It finds that an incomplete model of the radio sky is the dominant source of contamination, and that very faint point sources must be removed before the cosmological signal can be recovered.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unmodelled direction-dependent systematics likely change the leakage ranking; the paper's own data comparison shows simulations leak less than real data, so the 'incomplete sky model dominates' claim is only established within the simulated effect set.","rationale":"The reader's verdict is CONDITIONAL, with the weakest assumption being that simulations faithfully represent the MWA signal chain so that the ranking of systematics transfers to real data. The paper itself provides direct evidence that this assumption fails at some level: Section 5.5 states 'there is clearly less leakage into the window from the simulations, indicating further unmodelled systematics.' The full text also explicitly excludes direction-dependent effects in Section 2.2 and lists the ionosphere and missing dipoles as future work in Section 7. Since the headline claim ('single greatest cause of leakage is an incomplete sky model') is comparative, the unmodelled terms are not just a caveat but a known missing component of the comparison. My stress-test pass therefore agrees with the reader: the concern is load-bearing and the verdict should remain CONDITIONAL until a direction-dependent test is performed or the claim is scoped explicitly to direction-independent effects. I did not find an internal inconsistency in the simulation logic beyond the abstract/body flux-cut discrepancy already noted by the reader. The diffuse double-counting issue is acknowledged in Section 3.3 and justified for power-level purposes; it is a secondary risk. I would not move to REJECT because the paper's within-scope experiments are coherent: the flux-cut result is demonstrated with controlled visibility subtraction, the calibration-leakage experiment in Section 4.2 cleanly separates calibration from subtraction, and the instrumental-effect injections follow established formalisms such as Beardsley et al. (2016) cable reflections and Thompson et al. noise. The weakest link is external validity rather than internal consistency. Machine-checked proofs or full reproducibility artefacts are absent, but the WODEN pipeline is a published open-source code and the qualitative match to real images in Figure 2 supports realism at image level. The single strongest countervailing evidence is the paper's own admission of unmodelled systematics; without a quantitative bound on that residual, the 'single greatest cause' claim must be read as 'largest among the modelled direction-independent effects.' The concrete test I propose, adding a measured-level ionospheric phase screen or per-tile beam errors and comparing the resulting window power to the incomplete-sky-model term, would settle whether the ranking survives the transfer to real data. Agreement with the reader is 'agree' because they identified the same weakest assumption and recommended the same verdict.","tokens_in":15239,"tokens_out":2594,"duration_ms":22457,"concrete_test":"Re-run the Section 5.5 comparison including a direction-dependent effect at measured MWA levels, for example by adding a refractive ionospheric phase screen with amplitudes matching Jordan et al. (2017) for the 2015 EoR0 night, or by injecting per-tile primary-beam errors from realistic dead-dipole patterns, then recalibrating with 10,000 sources and recomputing the 1D EoR-window power. If the added direction-dependent leakage exceeds the incomplete-sky-model contribution seen in Figures 7 and 8, the ranking and the 1-10 mJy flux-cut threshold change.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is the ranking of systematics: incomplete sky model dominates over cable reflections, flagged channels, and gain errors. That ranking is established only among direction-independent effects, as stated in Section 2.2: 'only focus on instrumental effects that are independent of direction upon the sky.' The paper's own matched comparison (Figure 6, Section 5.5) shows 'clearly less leakage into the window from the simulations, indicating further unmodelled systematics.' Direction-dependent effects explicitly deferred to future work include the ionosphere (Section 2.2 and Section 7) and missing dipoles with distinct primary beams (Section 7, noted as 'currently being undertaken'). The ionosphere in particular is known to add spectral structure and direction-dependent phase errors that leak foreground power into the EoR window; if its leakage is comparable to or larger than the incomplete-sky-model term, the headline conclusion, and the mJy-level flux-cut threshold derived from it, need not transfer to real data. The reader flagged this same weakness, and the paper self-identifies it. The internal numerical inconsistency, where the abstract states 10 mJy and more than 90% while Section 4.1 and Table 1 show 1 mJy and 98.5%, reinforces that the quantitative threshold is not yet stable enough to guide MWA EoR strategy. The diffuse model double-counting (Section 3.3) is a secondary concern but is explicitly excused for power-level purposes; the dominant risk is the unmodelled-systematics gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper uses the WODEN simulator to generate realistic MWA EoR0 zenith-pointing observations containing a 21-cm model, a discrete foreground model, and a diffuse foreground model, then runs them through the AusEoRPipe analysis pipeline. It tests four direction-independent instrumental effects (edge/centre channel flagging, tile gain errors, cable reflections, and thermal noise) both in isolation and in combination, compares the resulting power spectra to matched real MWA data, and ranks the contributions of these effects to leakage into the EoR window. The headline claims are that an apparent-flux cut near 1 mJy (claimed in the abstract as 10 mJy) is needed to recover the 21-cm signal, that diffuse emission prevents access to some k-modes, that an incomplete sky model is the single greatest cause of leakage, and that the other tested effects are comparable to each other and subdominant.","tokens_in":1537,"tokens_out":1848,"duration_ms":45501,"significance":"If the results hold, the paper provides a concrete, simulation-based prioritisation for MWA EoR analysis: invest in deeper/higher-resolution source catalogues, implement gain spectral smoothness, and develop diffuse-emission treatment. The strengths are the controlled simulation experiment, the use of external sky catalogues for the discrete model and EDA2-derived diffuse map, the matched comparison to real observations, and the open-source WODEN simulator. The paper is also honest about its own limitations, explicitly stating in Section 5.5 that the simulations show less leakage than real data and in Section 2.2 that only direction-independent effects are considered. The central ranking among the simulated effects is a solid contribution, but the quantitative threshold in the abstract and conclusion is inconsistent with the body, and the 'single greatest cause' claim is only established within the simulated direction-independent effect set, which the paper itself acknowledges.","major_comments":[{"comment":"The abstract and conclusion state that more than 90% of unresolved point-source flux down to 10 mJy apparent must be subtracted, but Section 4.1 and Table 1 show that the necessary cut is at 10^-3 Jy (1 mJy), which removes 98.5% of apparent flux. This is an order-of-magnitude discrepancy in the headline quantitative result. The abstract and conclusion (including the bullet 'sources down to less than 10 mJy need to be removed') must be corrected to match the body's 1 mJy / 98.5% threshold, or the body must be re-examined if 10 mJy was intended.","section":"Abstract, Section 4.1, Table 1, Section 8"},{"comment":"The claim that 'the single greatest cause of leakage is an incomplete sky model' is established only among the set of direction-independent effects simulated. Section 2.2 explicitly restricts the study to 'instrumental effects that are independent of direction upon the sky', and Section 5.5 states 'there is clearly less leakage into the window from the simulations, indicating further unmodelled systematics'. Direction-dependent effects such as the ionosphere and per-tile primary-beam errors are deferred (Section 7). Because the ionosphere is known to introduce spectral structure and direction-dependent phase errors that leak foreground power, the ranking may not transfer to real data. Please qualify the headline claim as 'among the tested direction-independent effects' and attach the same qualifier to the flux-cut threshold derived from this ranking.","section":"Section 2.2, Section 5.5, Section 7"},{"comment":"The diffuse sky model contains all discrete sources, so simulations combining the diffuse and discrete models double-count flux. The paper acknowledges this ('will result in some double counting of flux') but does not quantify the resulting spurious power at high k-modes, even though Figure 4 notes residual power there 'may be false power'. Since the full sky model is used for the ranking experiments in Section 5, the double-counting could in principle change the relative sizes of the leakage terms that underpin the 'incomplete sky model dominates' conclusion. Please quantify the double-counted component, for example by comparing a simulation with the diffuse map alone against one with the discrete model alone in the same k-range, and show that the ranking is robust to this effect.","section":"Section 3.3, Section 4.1, Figure 4"}],"minor_comments":[{"comment":"The first-person phrasing in Section 5.6 ('In these tests on real data, I have assumed all dipoles are alive') should be converted to formal third-person wording, and the caveat that this may introduce a different calibration systematic should be integrated into the main analysis rather than left as a parenthetical aside.","section":"Section 5.6"},{"comment":"The caption says 'All real and simulated data have been calibrated using 10,000 sources, with the real data calibrated using 8000 sources' but Section 5.5 states that simulations use 'the apparent 10,000 brightest sources'; please clarify whether the real data use 8000 or 10,000 sources and reconcile the caption with the text.","section":"Figure 6 caption"},{"comment":"The notation '10-1, 10-2, 10-3 Jy' in the text should be typeset with superscripts, e.g., 10^-1 Jy, and Table 1 caption should use the same formatting.","section":"Section 4.1, Table 1"},{"comment":"The reference 'Jordan et al, 2024, submitted' is incomplete; either provide a full citation or remove the reference marker.","section":"Section 6"},{"comment":"The manuscript inconsistently uses 'sky model' and 'skymodel' (e.g., Section 2.1 vs. Section 5.6); please standardise the spelling.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a well-executed controlled simulation study that delivers a useful ranking among the tested effects, and the authors are appropriately cautious about unmodelled systematics. The two issues that prevent acceptance are (1) the internal inconsistency in the headline flux threshold between the abstract/conclusion and the body, which cannot be left unresolved, and (2) the overbroad phrasing of the 'single greatest cause' claim relative to the direction-independent scope. Both are fixable in revision. The double-counting concern in Section 3.3 is secondary but should be addressed quantitatively. I recommend major revision; the paper's scope fits the journal and the central methodology is sound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this paper is worth a serious referee. It is a controlled simulation study that does something concrete: it ranks the leakage contributions from an incomplete sky model, flagged channels, gain errors, and cable reflections through the AusEoRPipe, and identifies a 1 mJy apparent flux cut as necessary to recover the 21-cm signal in their sky model. That threshold and the ranking are new, and they give the EoR community a clear priority list: improve the sky model, add gain smoothing, fit cable reflections, and start treating diffuse emission.\n\nWhat it does well: the simulations are built with WODEN, the sky models are external catalogues (GLEAM, LoBES, EDA2), the instrumental effects are drawn from measured or published ranges, and the results are compared against matched real MWA observations. The comparison to real data is honest—the paper explicitly shows less leakage in the simulations than in the real data, which is the right way to flag unmodelled systematics. The conclusion that per-channel calibration handles cable reflections well is a useful practical result.\n\nThe soft spots are real but not fatal. The abstract quotes '10 mJy apparent' and 'more than 90%', while Table 1 and Section 4.1 show a 10^-3 Jy cut (1 mJy) removing 98.5% of apparent flux. That is an internal inconsistency that must be fixed before publication. Second, the 'single greatest cause' claim is only established among direction-independent effects. The paper itself defers the ionosphere, direction-dependent RFI, and missing dipoles with distinct primary beams to future work, and its own Figure 6 shows real data leaking more than the simulations. So the ranking is a ranking of the modelled effects, not a guarantee about real data. The diffuse model double-counting is a minor issue because the paper only claims comparable power, but it is worth a sentence of clarification.\n\nWho is this for? Anyone working on MWA EoR analysis or planning SKA-Low calibration/foreground strategies. The paper gives them a sanity check and a set of quantitative targets. I would cite it for the flux-cut threshold and the systematics ranking, and I would bring it to a reading group to discuss the calibration experiments.\n\nRecommendation: send it to peer review. The referee should focus on the abstract/body inconsistency, on whether the 1 mJy threshold is robust to the sky model completeness assumptions, and on whether the ranking would survive including direction-dependent effects. None of these are fatal; they are the normal work of revision.","headline":"Quantitative systematics ranking for MWA EoR, useful and mostly sound, but internal numbers conflict and the claim only covers direction-independent effects.","tokens_in":16122,"tokens_out":1672,"would_cite":true,"duration_ms":15596,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that recovering the 21-cm signal with sky-based calibration requires subtracting unresolved point-source flux down to about 1 mJy apparent ($10^{-3}$ Jy, 98.5% of the apparent flux in the adopted sky model), and that the…","keywords":["epoch of reionisation","21-cm power spectrum","MWA EoR pipeline","sky model completeness","radio interferometric calibration","instrumental systematics","WODEN simulations"],"falsifier":"Run the same WODEN/AusEoRPipe test with the sky model extended below the $10^{-3}$ Jy apparent-flux cut (or with direction-dependent peeling of the brightest sources) and measure the residual power in the EoR window; the paper's claim predicts the signal is recovered once 98.5% of apparent flux is subtracted, so any simulation in which the 21-cm signal still fails to appear above that residual would falsify the threshold result.","tokens_in":14986,"feed_emoji":"📡","tokens_out":9000,"duration_ms":72113,"temperature":0.7,"pith_summary":"This paper tries to establish where the Australian MWA EoR pipeline (AusEoRPipe) loses the ability to detect the cosmological 21-cm signal, by simulating realistic MWA data with WODEN and running known instrumental effects through the pipeline in isolation. It finds that sky-model incompleteness dominates: without subtracting unresolved point sources down to an apparent flux of $10^{-3}$ Jy (98.5% of the apparent flux in the adopted sky model), calibration and subtraction errors inject enough power to mask the 21-cm signal even when no instrumental effects are present. When diffuse emission is added, some $k$-modes are never accessible, so diffuse foreground removal is also required. The payoff is a concrete priority list for the experiment: fix the sky model first, then add gain smoothness, cable-reflection fitting, and diffuse treatment. That is what a detection attempt should spend its effort on.","feed_headline":"Subtract 98.5% of point-source flux to see the 21-cm signal","feed_subtitle":"MWA simulations show an incomplete sky model, not cables or flags, is the main blocker to the Epoch of Reionisation detection.","key_machinery":"The central object is the apparent-flux threshold on the discrete sky model: sources are weighted by the MWA primary beam at 182 MHz, and three sub-models cut at $10^{-1}$, $10^{-2}$, and $10^{-3}$ Jy are subtracted from the test bed. The power-spectrum machinery is CHIPS with a wedge cut that excludes modes with $k_\\parallel < 0.08\\,h\\,\\mathrm{Mpc}^{-1}$ and $k_\\perp > 0.06\\,h\\,\\mathrm{Mpc}^{-1}$ plus the horizon line. The mechanism that carries the calibration result is per-channel, direction-independent calibration, whose small spectral ripples transfer smooth foreground power into the EoR window; the same catalogue used for calibration is also used for subtraction. The diffuse model is an EDA2 m-mode map upgraded and smoothed to MWA resolution, which contains discrete sources and therefore double-counts flux when combined with the discrete model, a point the paper accepts as long as total power matches real data.","core_discovery":"In a single-observation test bed with the discrete and 21-cm sky models but no instrumental effects, the paper shows that an apparent-flux cut at $10^{-3}$ Jy (35% of the sources, 98.5% of the apparent flux) is necessary and sufficient for the 21-cm signal to be recovered after direct subtraction in visibility space. Adding the diffuse sky model makes low-$k$ modes unrecoverable without diffuse emission removal. When calibration is included, percent-level frequency-dependent amplitude fluctuations in the per-channel gain solutions couple low-$k$ foreground power into higher $k$-modes, masking the signal; applying the same calibration solutions to the 21-cm model alone does not bias that signal. Across the 15-observation test bed with noise, the incomplete sky model is the single greatest cause of leakage, while cable reflections, flagged coarse-band channels, and tile gain errors each add comparable power and less than the sky model. Averaging calibration solutions over the 30-minute pointing reduces window leakage in both simulations and real data.","pith_inferences":["Inference: if the ranking transfers to real data, extending the southern-sky catalogue below 1 mJy should buy more EoR sensitivity per unit effort than any other single pipeline change; early SKA-Low arrays may find their detection limited by the same sky-model gap until their catalogues catch up.","Inference: because the diffuse map already contains the discrete sources, the residual diffuse power at high $k$ is an upper limit in this test; a cleaner diffuse/discrete separation would turn 'some k-modes cannot be accessed' into a quantitative statement.","Inference: the paper tests only direction-independent effects; ionospheric refraction and per-tile primary-beam variation (missing dipoles) are explicitly left out and could plausibly inject leakage at or above the sky-model level, changing the priority order.","Inference: the calibration-averaging result suggests a direct upgrade: enforce spectral smoothness of gain solutions (LOFAR-style regularization) to attack the leakage mechanism at its source rather than after the fact."],"forward_implications":["Sky-based calibration and power-spectrum recovery of the 21-cm signal requires removing unresolved point sources down to about $10^{-3}$ Jy apparent flux (more than 90% of apparent flux; 98.5% in the adopted model), not just the bright calibrators.","With diffuse emission in the simulation, some $k$-modes are never accessible from 30 minutes of zenith data, so diffuse foreground removal is a necessary part of the pipeline, not an optional refinement.","The ordering of systematics is concrete: an incomplete sky model causes more EoR-window leakage than flagged coarse-band channels, cable reflections, or tile gain errors at the tested levels, so sky-model development should take priority.","Averaging calibration solutions over the 15 snapshots of a pointing reduces window leakage in simulations and real data, pointing to a cheap sensitivity gain."],"supporting_citations":[{"why":"Supplies the 21-cm sky model, the AusEoRPipe methodology, and the Paper I baseline that the pipeline does not lose signal.","marker":"(Line et al. 2024)"},{"why":"WODEN simulator computes all model visibilities for the discrete, diffuse, and 21-cm components.","marker":"(Line 2022)"},{"why":"GLEAM catalogue is the base for the discrete sky model.","marker":"(Hurley-Walker et al. 2017)"},{"why":"LoBES provides the EoR0 catalogue that sets the 90% completeness at 32 mJy, the limit behind the flux-cut result.","marker":"(Lynch et al. 2021)"},{"why":"EDA2 m-mode map is the source of the diffuse sky model.","marker":"(Kriele et al. 2022)"},{"why":"Gives the cable-reflection formalism and amplitude range used to inject that systematic.","marker":"(Beardsley et al. 2016)"},{"why":"CHIPS power-spectrum estimator forms the 2D and 1D power spectra used for all comparisons.","marker":"(Trott et al. 2016)"},{"why":"Identifies the ionospherically quiet 2015 EoR0 observations used as matched real data.","marker":"(Jordan et al. 2017)"}],"fun_headline_variants":["Incomplete sky model tops MWA EoR leakage culprits","EoR detection: subtract 98.5% of point-source flux","Sky model errors dominate MWA EoR leakage","Incomplete sky model outranks cable reflections in EoR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The simulated data must reproduce the real MWA signal chain closely enough that the ranking of systematics transfers to reality; the paper itself reports that simulations leak less into the window than matched real data, so unmodelled effects such as ionospheric refraction, direction-dependent errors, or missing dipoles could change the priority order and the required flux threshold.","fun_headline_variants_meta":{"raw":{"variants":["Incomplete sky model tops MWA EoR leakage culprits","EoR detection: subtract 98.5% of point-source flux","Sky model errors dominate MWA EoR leakage","Incomplete sky model outranks cable reflections in EoR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000937,"raw_usage":{"total_tokens":4028,"prompt_tokens":989,"completion_tokens":3039,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":2965}},"tokens_in":605,"tokens_out":3039,"duration_ms":19320,"temperature":1.0,"reasoning_tokens":2965,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:50:17.699720+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same WODEN/AusEoRPipe test with the sky model extended below the $10^{-3}$ Jy apparent-flux cut (or with direction-dependent peeling of the brightest sources) and measure the residual power in the EoR window; the paper's claim predicts the signal is recovered once 98.5% of apparent flux is subtracted, so any simulation in which the 21-cm signal still fails to appear above that residual would falsify the threshold result.","supporting_citations":[],"review_version":1}