{"id":"c9197498-187d-4713-8016-fd686f9b0464","arxiv_id":"2412.15893","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Convolutional networks trained on 21 cm reionization images often memorize simulation boxes rather than physics, yielding high same-box test scores but poor performance on unseen simulations.","lead":"This paper trains convolutional neural networks on 21 cm reionization simulations and shows that they often memorize the specific simulation boxes they were trained on rather than learning real physics, so their high test scores collapse when tested on fresh simulations. It matters because many published ML-based forecasts for 21 cm cosmology may be overoptimistic and need stricter validation.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing identifiability baseline: without evidence that a single 2D slice can constrain all six parameters, the same-box/new-box R2 gap alone does not diagnose memorization; the P/120 experiment is the decisive test but is not connected to this gap.","rationale":"I agree with the reader that the absence of an identifiability analysis or baseline estimator is the weakest point of the paper's argument. The same-box/new-box R2 contrast is striking, but its interpretation as memorization rather than ill-posedness is not self-evident. This concern is load-bearing because it affects the central claim's diagnostic strength, not just a side detail. However, the paper contains an independent piece of evidence, the P/120 scaling experiment in Sec. 6, which directly manipulates the degree of box sharing and shows that performance on held-out slices from training boxes increases with the number of training slices per box. This is strong evidence that the network exploits box-specific information, regardless of whether the task is identifiable. Therefore the core conclusion that the network is not useful for real data (where P=0) survives, but the paper's stronger statement that it has 'learnt to associate slices with individual boxes' is not fully secured without a baseline. The reader's CONDITIONAL verdict is appropriate: the paper should either provide a simple baseline showing that single-slice inference is feasible, or soften the interpretation of Fig. 7 and rely on the P/120 experiment as the primary memorization evidence. Secondary issues (no repeated seeds, the Sec. 7.1/Sec. 9 datacube contradiction, and unavailable data) also support CONDITIONAL, but the identifiability gap is the most consequential for the central claim.","tokens_in":19997,"tokens_out":12832,"duration_ms":117330,"concrete_test":"Add a baseline estimator for the six-parameter single-slice task, using the same train/test split and evaluated on new boxes. For example, train a linear regression on summary statistics (21 cm power spectrum, PDF, etc.) or a k-nearest-neighbor regressor on the training set. If the baseline also yields R2 ≤ 0 on new boxes, single-slice inference is effectively ill-posed and Fig. 7 alone cannot diagnose memorization; the P/120 trend is then the necessary evidence. If the baseline achieves positive R2, the CNN's new-box failure is more directly attributable to memorization rather than task difficulty.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the CNN memorizes simulation boxes rather than learning physics rests on the R2 collapse between same-box tests (Fig. 6, average R2 ≈ 0.994) and new-box tests (Fig. 7, R2 near zero or negative). The paper interprets this collapse in Sec. 5.2 as direct evidence that the network 'has instead learnt to associate slices with individual boxes'. This interpretation assumes that a well-trained network could in principle recover all six parameters from a single 2D slice of a 150 Mpc Simfast21 box, i.e., that the task is identifiable. No identifiability analysis or baseline estimator is provided. If single-slice inference is fundamentally ill-posed, then a perfect network would also fail on new boxes, and the high same-box R2 would reflect exploitation of box-specific density-field correlations (a form of memorization) rather than a failure to learn a valid physical mapping. In that case Fig. 7 alone does not establish the memorization diagnosis; it merely shows lack of generalization, which could be an information-theoretic limitation. The P/120 experiment (Fig. 8) does provide controlled evidence for memorization, because same-box test performance increases with the fraction of slices per box in training even at fixed total training size. However, the paper does not explicitly tie this experiment to the identifiability question, leaving the strongest interpretation of Fig. 7 underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an empirical benchmark of a convolutional neural network used for 21 cm reionization parameter inference. The authors reproduce the architecture of Hassan et al. (2020), train it on 30,000 Simfast21 simulations, and compare performance when test slices come from boxes already represented in training versus from freshly simulated boxes. They find that a six-parameter regression (Ωm, h, σ8, fesc, Cion, Dion) achieves an average R2 ≈ 0.994 on shared-box test slices but near-zero or negative R2 for four of the six parameters on new boxes. A controlled experiment (Sec. 6) varies the fraction P/120 of slices per box shared between training and test at fixed total training size, and shows a monotonic rise in average R2 with P/120. The paper also studies recovery of the neutral fraction, reionization midpoint, duration, and CMB optical depth, with and without redshift-stacked inputs. It concludes that the CNN often memorizes box-specific features rather than learning physical mappings, so that current architectures are not suitable for real 21 cm data.","tokens_in":20129,"tokens_out":11904,"duration_ms":99238,"significance":"The paper is a timely and valuable cautionary contribution to a field where ML-based inference from 21 cm maps is being adopted rapidly. Its central empirical findings are stark and well controlled: the near-perfect shared-box R2 and the collapse to near-zero/negative R2 on new simulation boxes, complemented by the P/120 trend that isolates the effect of shared-box information at fixed training-set size. The authors ship the code, which supports reproducibility. If the conclusions hold, the paper should motivate the community to adopt stricter train/test separation protocols (new simulation boxes, not just new slices) and to test the robustness of published CNN-based parameter inference. The main qualifications are that the evidence comes from a single architecture and from single training runs, and that the memorization diagnosis would be strengthened by an explicit identifiability argument or baseline estimator.","major_comments":[{"comment":"The R2 collapse in Fig. 7 is interpreted in Sec. 5.2 as direct evidence that the network 'has instead learnt to associate slices with individual boxes.' This interpretation presumes that a well-trained network could in principle recover all six parameters from a single 2D slice of a 150 Mpc Simfast21 box. If the single-slice problem is not identifiable, then a perfect network would also fail on new boxes, and the high same-box R2 would reflect box-specific correlations rather than a failure to learn physics. The P/120 experiment in Fig. 8 provides controlled evidence for memorization, because same-box test performance rises with the fraction of training slices per box at fixed total training size. However, the paper does not explicitly explain why this trend rules out the identifiability alternative; a reader must supply that argument. Please add a brief identifiability discussion and, ideally, a simple baseline estimator (e.g., linear regression on power-spectrum or topology summary statistics, or a nearest-neighbour predictor) to quantify the new-box performance achievable without memorization.","section":"Sec. 5.2 / Sec. 6, Figs. 7 and 8"},{"comment":"The 3D case studies (Secs. 7–9) feed datacubes of shape (200,200,20) into the network, but Table 1 specifies an input shape of (200,200,1). The paper does not state whether the first convolutional layer was changed to accept 20 input channels, whether 3D convolutions were used, or how the redshift/line-of-sight dimension was handled. Because the paper's stated purpose is reproducibility, this architecture detail is essential and should be specified explicitly (or the table amended for the 3D runs).","section":"Secs. 7–9, Table 1"},{"comment":"Every configuration in the paper is trained once, and Fig. 8 is reported without error bars. The main qualitative conclusions are probably robust given the size of the R2 differences, but the paper should at least state that single runs are used and provide some estimate of the expected variability (e.g., from a few random seeds for a subset of configurations, or from multiple data splits). Without this, a reader cannot assess whether the monotonic trend in Fig. 8 and the exact positions of the orange/red points are stable.","section":"General / Fig. 8"}],"minor_comments":[{"comment":"The abstract states the paper applies networks to '21cmFast and Simfast21 simulations,' but the full text only uses Simfast21; please correct the abstract or clarify that 21cmFast refers to the original papers being reproduced.","section":"Abstract"},{"comment":"Sec. 2.3 says each simulation had 21 boxes, while Sec. 7.1 refers to 20 redshift boxes; the counting should be made consistent.","section":"Sec. 2.3 / Sec. 7.1"},{"comment":"The text mentions 'minor adjustments to the code' to vary Cion and Dion, but the exact modifications are not described. Since the code is available, a brief description or pointer would improve reproducibility.","section":"Sec. 2.3"},{"comment":"The claim that the network is 'able to infer four out of six parameters' is somewhat strong given the R2 values in Fig. 12 (e.g., σ8 R2 = 0.283, Dion R2 = 0.494); please temper the wording or define a threshold for 'able to infer.'","section":"Sec. 9 / Fig. 12"},{"comment":"The paper generalizes from a single CNN architecture (Network I of Hassan et al. 2020) to 'CNNs often learn...' in the abstract; consider qualifying the claim as 'the tested CNN' or 'at least one widely used architecture.'","section":"Abstract / Conclusions"},{"comment":"Minor typos include 'criterea' (Sec. 7.1), 'refered' (Sec. 3.1), and 'telecopes' (Sec. 2).","section":"Throughout"}],"recommendation":"minor_revision","confidential_remarks":"This is a solid cautionary study that fits the journal well. The main weakness is not the empirical core but the framing: the abstract overstates the breadth of the conclusion and the mechanism (memorization) is inferred rather than demonstrated against an identifiability baseline. The paper would be strengthened by a short baseline experiment and by uncertainty quantification. I do not see a need for rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline result is real and worth knowing: a CNN that looks great on held-out slices (R2 ≈0.994) collapses to near-zero or negative R2 for four of six parameters when tested on slices from freshly simulated boxes. The controlled P/120 experiment makes the diagnosis convincing—average R2 rises monotonically with the fraction of training slices per box, even with total training set size held roughly constant. That is direct evidence of box memorization, not just poor generalization.\n\nThe paper also does things well. It reproduces the architectures and training setups from Mangena, Hassan, La Plante, and Billings rather than straw-manning them, and it is candid about its own preprocessing choices. The Sec 9 synthesis, showing that providing redshift structure helps some parameters, is a useful counterpoint and suggests a path forward.\n\nNow the soft spots, roughly in order. First, the identifiability question: the paper interprets the same-box/new-box R2 gap as “the network is not learning physics.” But without a baseline estimator or identifiability analysis, you don’t know whether a single 2D slice of a 150 Mpc box can in principle constrain all six parameters. For parameters like h or sigma8, the slice may contain too little information. If so, a perfect network would also fail on new boxes, and the high same-box R2 would reflect memorization of density-field correlations—still bad, but less clearly “not learning physics.” The P/120 experiment does support memorization regardless, but the paper never connects it to the identifiability issue.\n\nSecond, there’s an internal contradiction: Sec 7.1 says each datacube is built by drawing one slice from each of the 20 redshift boxes of a single simulation (same realization), while Sec 9 says the slices in a datacube are drawn from different realizations. Those are different inputs, and it matters for interpreting the Sec 9 improvement. That needs to be resolved.\n\nThird, the R2 values come from single training runs. No seeds, no repeats, no uncertainties. For a paper whose entire point is that apparent performance can be misleading, that’s ironic. At minimum they should run a few seeds for the key figures. Fourth, the dataset is not archived; “reasonable request” is not reproducibility.\n\nNone of these sink the central practical conclusion: as it stands, this network is not useful for real data. But the paper overstates the diagnosis. Anyone doing ML-based inference from 21 cm maps should read it; really anyone training CNNs on simulated cosmology. It deserves peer review—a good referee will ask for the identifiability baseline, resolution of the Sec 7.1/9 mismatch, and multi-seed runs. I'd engage with it.","headline":"Valuable cautionary study showing CNN box-memorization in 21 cm inference, but the diagnosis is stronger than the evidence without an identifiability baseline.","tokens_in":20802,"tokens_out":4640,"would_cite":true,"duration_ms":38001,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A CNN trained on 21 cm reionization maps can score near-perfectly on slices from boxes it already saw while failing on fresh simulation boxes, meaning it has memorized the boxes rather than learned the physics.","keywords":["21 cm cosmology","reionization","convolutional neural networks","parameter inference","generalization","out-of-distribution","neural network memorization","brightness temperature maps"],"falsifier":"Take a fixed set of new simulation boxes and compare the CNN against a non-neural baseline that uses explicit summaries of the same slices, such as a power-spectrum and bispectrum measurement fed into a Gaussian process emulator. If the baseline also fails to recover the six parameters from single slices, the poor new-box R2 is a property of the inference problem rather than evidence of memorization; if the baseline succeeds, the CNN's failure is confirmed as a learning failure. A second decisive test is to train the CNN with exactly one slice per box and check whether new-box R2 stays high.","tokens_in":19669,"feed_emoji":"📡","tokens_out":5909,"duration_ms":47775,"temperature":0.7,"pith_summary":"This paper reports a reproduction study of convolutional neural networks built to infer cosmological and astrophysical parameters from simulated 21 cm reionization maps. Its central claim is that these networks often achieve high scores not by learning the physics of reionization but by learning to recognise individual simulation boxes: slices drawn from the same box share an underlying density field, and the network can exploit that fingerprint. When tested on slices from boxes seen in training, a six-parameter network reaches an average R2 of about 0.994; when tested on slices from freshly simulated boxes, the R2 values collapse to near zero or negative, for example -1.276 for the reionization index Dion. The paper concludes that such a network is not useful as it stands for application to real observations, where the true density field cannot be matched by the training set. A reader should care because this exposes a failure mode that can make published ML-based inference look successful while being fragile.","feed_headline":"CNNs memorize 21 cm boxes instead of learning physics","feed_subtitle":"A network that scores R2≈0.99 on familiar boxes collapses on fresh simulations, a warning for real data.","key_machinery":"The central object is the CNN architecture taken from Hassan et al. (2020): four convolutional blocks (two convolutional layers, batch normalisation, ReLU) joined by max pooling, followed by three fully connected blocks, trained with mean-squared error and the Adam optimiser. The argument is carried by the R2 score measured under two testing protocols: slices from boxes represented in the training set versus slices from newly simulated boxes, and by a control scan in which the fraction P/120 of slices per box shared between training and testing is varied. That shared-slice fraction is the mechanism of memorization: when the network sees many slices from one box, it can identify the box through its shared density field and simply return the box's parameter labels, which produces inflated same-box R2 but fails at P=0, the regime relevant to real observations.","core_discovery":"The central discovery, stated on the paper's own terms, is that CNNs trained on 21 cm brightness-temperature maps frequently learn box-level memorization rather than a transferable mapping from image features to parameters. The evidence chain is: reproducing the network architecture of Hassan et al. (2020) on 30,000 Simfast21 boxes; showing same-box test R2 of about 0.994 for six parameters; showing new-box R2 values of -1.276, -1.157, -0.236, -0.094, 0.433 and 0.511 across the six parameters; and demonstrating that R2 rises as the fraction of slices per box shared with the training set increases. The paper also finds that some targets generalise: the neutral fraction and the CMB optical depth remain recoverable on new boxes, while the reionization duration fails because the training set samples it poorly. The interpretation is that a network which appears to perform excellently can be silently exploiting artificial properties of the training set, placing responsibility on users to test generalization on completely new simulations before trusting CNN-based parameter inference.","pith_inferences":["If the single-slice inference problem is fundamentally ill-posed, the same-box R2 could be inflated by correlations between the density field and the parameters even without memorization; a summary-statistic baseline on the same new boxes would separate those cases.","The redshift-cube improvement in Section 9 may be driven as much by adding extra constraints as by the network learning physics; ablating single redshift slices would show which slices carry the recoverable signal.","The same shared-box diagnostic could be applied to other image-based cosmological inference pipelines, since any training set that draws multiple views from a single realisation carries the same memorization risk.","A practical discipline suggested by the paper is to train with at most one slice per simulation realisation, or to generate enough independent boxes that shared density fields cannot be a shortcut."],"forward_implications":["A reported R2 above 0.99 on a held-out test set is not evidence of generalisation when training and test slices share simulation boxes; new-box performance must be reported separately.","Targets like the neutral fraction and the CMB optical depth appear learnable across new boxes, while parameters such as h, sigma8 and fesc do not, so claims about those parameters should be treated cautiously.","Providing the network with a cube of slices across redshift improves new-box recovery of some parameters, suggesting that temporal evolution carries much of the usable information.","A network trained and tested under the common shared-box protocol is not ready for real 21 cm observations, because real data correspond to P=0.","Users of CNN-based inference in 21 cm cosmology need to check the training-set simulation statistics and add out-of-distribution tests before drawing physical conclusions."],"supporting_citations":[{"why":"Supplies the Network I architecture, the six-parameter inference task, and the same-box test protocol that the paper reproduces and then breaks.","marker":"Hassan et al. (2020)"},{"why":"Supplies the neutral-fraction recovery setup that the paper reproduces and shows to generalise only partially to new boxes.","marker":"Mangena et al. (2020)"},{"why":"Supplies the zmid and Delta-z inference setup whose poor duration recovery is traced to poor training-set sampling.","marker":"La Plante & Ntampaka (2019)"},{"why":"Supplies the CMB optical-depth inference setup that generalises well in the paper's reproduction.","marker":"Billings et al. (2021)"},{"why":"Provides the earlier finding that a CNN trained on one semi-numeric reionization code fails on another, supporting the paper's generalisation warning.","marker":"Zhou & Plante (2022)"},{"why":"Justifies the choice of 150 Mpc boxes by showing that reionization histories converge at this volume.","marker":"Iliev et al. (2014)"},{"why":"Introduces the Simfast21 semi-numerical method that generates all of the paper's training and test simulations.","marker":"Santos et al. (2008)"}],"fun_headline_variants":["CNNs in 21 cm cosmology memorize boxes, miss physics","R2 0.99 on training, negative on new sims: ML warning","21 cm ML nets learn simulation quirks, not real physics","Test ML on fresh simulations before trusting 21 cm results","Memorized boxes, not maps: why 21 cm CNNs fail to generalize"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a single 2D slice of a 150 Mpc simulation box contains enough information to determine all six astrophysical and cosmological parameters; if that inference problem is ill-posed, a network could fail on new boxes even while learning real physics.","fun_headline_variants_meta":{"raw":{"variants":["CNNs in 21 cm cosmology memorize boxes, miss physics","R2 0.99 on training, negative on new sims: ML warning","21 cm ML nets learn simulation quirks, not real physics","Test ML on fresh simulations before trusting 21 cm results","Memorized boxes, not maps: why 21 cm CNNs fail to generalize"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000738,"raw_usage":{"total_tokens":3275,"prompt_tokens":899,"completion_tokens":2376,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":2281}},"tokens_in":515,"tokens_out":2376,"duration_ms":14130,"temperature":1.0,"reasoning_tokens":2281,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:59:52.138078+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed set of new simulation boxes and compare the CNN against a non-neural baseline that uses explicit summaries of the same slices, such as a power-spectrum and bispectrum measurement fed into a Gaussian process emulator. If the baseline also fails to recover the six parameters from single slices, the poor new-box R2 is a property of the inference problem rather than evidence of memorization; if the baseline succeeds, the CNN's failure is confirmed as a learning failure. A second decisive test is to train the CNN with exactly one slice per box and check whether new-box R2 stays high.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the zmid and Delta-z inference setup whose poor duration recovery is traced to poor training-set sampling."}],"review_version":1}