{"id":"46b374ba-06c1-4cdf-97e8-af56a18b6df3","arxiv_id":"2411.12629","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A graph neural network predicts simulated cluster-galaxy halo masses from stellar mass and projected galaxy spatial and velocity structure with RMSE 0.242 dex on TNG300, beating random forests.","lead":"The authors train a graph neural network that estimates dark matter halo masses of galaxy cluster members from stellar mass plus projected positions and line-of-sight velocities, using TNG-Cluster for training and TNG300 for testing. It beats random forest baselines in the simulation, but the paper does not establish that the test clusters are truly independent of the training clusters.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The TNG300 test set is not demonstrated to be independent: TNG-Cluster zoom-ins are selected from the TNG300 volume, so the claimed generalization may only measure resolution transfer. A cluster-ID cross-match and disjoint-subset re-evaluation is needed.","rationale":"After reading in good faith, the central claim is not that GNNs can fit TNG-Cluster, but that a GNN trained on TNG-Cluster transfers to an independent TNG300 test set and beats RF baselines there. The only evidence for transfer is the Table 1 test row; the only argument for independence is the one-sentence assertion in Section 2 that the two simulations are 'otherwise independent.' That assertion is questionable: zoom-in suites such as TNG-Cluster are constructed by selecting target halos from a parent volume — TNG300 — and re-simulating them. Unless the authors explicitly selected TNG300 clusters that are not among those 352 targets (which the text does not state), the test set is contaminated with lower-resolution copies of training clusters. This is not an internal inconsistency in the GNN mechanics, but it is a correctness risk in the central generalization claim. The reader's conditional verdict is already appropriate: the paper needs a disjoint-cluster test. My concrete check would settle it. If the disjoint-subset metrics hold, accept; if not, the claim of cross-simulation generalization is unsupported. Because the reader already required exactly this check, I do not change the verdict.","tokens_in":11119,"tokens_out":7407,"duration_ms":75673,"concrete_test":"Use the public IllustrisTNG and TNG-Cluster catalogs at z=0 to cross-match the 352 TNG-Cluster target halos (by FoF halo ID or central position, e.g., within 0.5 R200) against TNG300 FoF group IDs. Then recompute the Section 4/Table 1 TNG300 metrics on the subset of TNG300 galaxies belonging to clusters with no counterpart among the 352 training clusters. If RMSE and R2 on this disjoint subset remain within about 10-20% of the reported 0.242/0.785, the generalization claim is substantiated; if they degrade toward the RF baseline (0.344/0.567), the headline claim is not supported and the test set must be rebuilt from a truly independent simulation. Also report the number of TNG300 test clusters and galaxies surviving the exclusion, and the Rmax used for Delta_G.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is in Section 2: 'the two simulations are otherwise independent.' This is what licenses the central claim in Sections 4 and 6 that the GNN 'maintains its predictive power when tested on the independent TNG300 dataset.' But TNG-Cluster is a zoom-in suite whose 352 target halos were selected from the parent TNG300 volume and re-simulated at higher resolution. Consequently, the TNG300 test set — defined as galaxies within 10 Mpc of z=0 clusters with Mhalo > 1e14 M_sun — includes the same clusters used for training, at lower resolution. The reported TNG300 RMSE=0.242, R2=0.785 therefore conflates generalization to unseen halos with transfer across numerical resolution of already-seen halos. If the model partially memorizes cluster-specific structure from the high-res training realizations, the low-res versions of those same clusters could be easier to predict, inflating the headline numbers. The paper provides no cluster-ID cross-match, no demonstration that the two cluster populations are disjoint, and no test restricted to TNG300 clusters absent from the TNG-Cluster sample. This is the weakest link in the central claim, and it is directly testable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains a graph neural network (GNN) on TNG-Cluster zoom-in simulations to predict dark matter halo masses from stellar mass, projected positions, and line-of-sight velocities, and tests it on galaxies within 10 Mpc of massive clusters in TNG300. The authors report that the GNN outperforms random forest baselines, including an RF augmented with a stellar-mass overdensity feature, on both the TNG-Cluster cross-validation set and the TNG300 test set (RMSE 0.242 dex and R2 0.785 for the GNN, versus 0.344 dex and R2 0.567 for the best RF baseline). The central claim is that the GNN generalizes across the IllustrisTNG simulation suite.","tokens_in":11381,"tokens_out":5030,"duration_ms":51227,"significance":"If the generalization result is correct, the paper demonstrates a useful practical route to using cluster substructure, encoded as spatial and kinematic edges in a graph, to improve halo mass estimates relative to non-graph baselines. The paper has clear strengths: cross-validation splits based on cluster IDs, reported k-fold scatter, a public code repository, and an explicit discussion of domain shift and future observational tests. The main limitation is that the independence of the TNG300 test set from the TNG-Cluster training set is asserted but not demonstrated; because TNG-Cluster is a zoom-in suite selected from the parent TNG300 volume, the headline test may measure resolution transfer rather than generalization to unseen clusters. This makes the central generalization claim currently unverified.","major_comments":[{"comment":"The claim that TNG-Cluster and TNG300 are 'otherwise independent' is not established and is likely false: TNG-Cluster is a zoom-in suite whose target halos were selected from the TNG300 volume and re-simulated at higher resolution. The TNG300 test set, defined as galaxies within 10 Mpc of clusters with Mhalo > 1e14 Msun, therefore probably contains lower-resolution realizations of the same clusters used in training. The headline numbers in Table 1 (RMSE 0.242 dex, R2 0.785 on TNG300) would then conflate generalization to unseen halos with transfer across numerical resolution of already-seen halos. Please provide a cluster-ID cross-match between the TNG-Cluster sample and the TNG300 cluster population, and report the test-set metrics restricted to TNG300 clusters that are absent from the TNG-Cluster sample.","section":"Section 2 and Section 6"},{"comment":"The overdensity radius Rmax is introduced as a free parameter for the RF baseline, but its value is never stated anywhere in the text, tables, or appendix. Since the RF+M*+DeltaG model is the principal baseline in Table 1, the experiment is not reproducible and the sensitivity of the baseline comparison to Rmax is unknown. Please specify Rmax (and any tuning procedure), or report results for a range of Rmax values.","section":"Section 3, Random Forest Baseline Models"},{"comment":"The selection criteria for the training and test samples are not matched in a way that is clear from the text. Table 2 gives the TNG-Cluster selection as log(Mhalo/Msun) > 11 and within 10 Mpc of the cluster halo, while the TNG300 test sample is described as galaxies within 10 Mpc of clusters with Mhalo > 1e14 Msun, with no stated lower mass cut. Please state the exact cuts applied to both samples and verify that the two selections define comparable galaxy populations; otherwise the comparison between the TNG-Cluster cross-validation metrics and the TNG300 test metrics in Table 1 is difficult to interpret.","section":"Section 2 and Table 2"}],"minor_comments":[{"comment":"The phrase 'galaxy neighbour' should be 'galaxy neighbors' or 'neighboring galaxies'.","section":"Abstract and Section 1"},{"comment":"In the TNG300 test-set block, the first random forest row is labeled 'RF' while the corresponding TNG-Cluster row is labeled 'RF:M*'; these labels should be made consistent.","section":"Table 1"},{"comment":"The scatter plots are colored by distance from cluster center, but no color bar is shown; please add a color bar and clear panel labels identifying each model.","section":"Figure 2"},{"comment":"Please define R2, NMAD, Bias, and outlier fraction explicitly, and state that all metrics are computed on log10 Mhalo rather than linear Mhalo.","section":"Section 3 and Table 1"},{"comment":"Please specify whether the projected positions (x, y) are cluster-centric and whether the line-of-sight velocity vz is the peculiar velocity or includes the Hubble flow; the units of x, y, and vz should also be stated.","section":"Section 3 and Appendix A"},{"comment":"The statement that the GNN uses 'unshared layers' is ambiguous; please clarify what is unshared across layers, and in Figure 3 specify whether the distance is projected three-dimensional distance and whether the RMSE is computed per galaxy or per distance bin.","section":"Section 3 and Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The independence issue is decisive for the headline claim and is directly testable; the authors should be required to perform the cluster-ID cross-match and report metrics on a disjoint subset. The unspecified Rmax for the RF baseline is a smaller but still necessary fix for reproducibility. The paper is otherwise a reasonable workshop-length contribution, but the generalization claim should be reworded or re-supported before I would recommend acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline result is that a GNN trained on TNG-Cluster cluster members beats random forests on a so-called independent TNG300 test set. The in-sample part is solid and worth a look. The independence part is not established, and the stress-test has it right: TNG-Cluster is a zoom-in re-simulation of 352 halos selected from the TNG300 volume. So the TNG300 test set, defined as galaxies within 10 Mpc of Mhalo>1e14 clusters, likely contains the same clusters seen in training, only at lower resolution. The RMSE=0.242 and R2=0.785 may be measuring resolution transfer, not generalization to unseen halos.\n\nWhat the paper does well: it applies the architecture of Wu et al. to a new regime, cluster overdensities; it uses cluster-ID k-fold cross-validation to avoid galaxy-level leakage; it reports scatter across folds; it compares against two RF baselines; and it ships code via a GitHub link. The table is readable, and the limitations section at least acknowledges domain shift. For a workshop paper, that is a decent package.\n\nThe soft spots are real but narrow. The independence assertion in Section 2 is the load-bearing claim, and it is not supported by any cross-match. A simple fix: list cluster IDs in TNG300 that are not the targets of TNG-Cluster zoom-ins, and rerun the test on that disjoint subset. They should also report the radius Rmax used for the overdensity parameter, since that is a free parameter of the RF baseline. Minor: the abstract has a stray word ('neighbour'), but that does not affect the content.\n\nFor a workshop, the paper is borderline acceptable if the generalization claim is softened. For a journal, the disjoint-cluster test or a reframing as resolution transfer is required before acceptance. The authors are not sloppy in the obvious places; the k-fold design shows they know about cluster-level leakage, which makes the omission more likely an oversight than a deliberate overclaim.\n\nI would give this paper a serious review, with a required revision on the independence test. It is not yet ready to cite as evidence that GNNs generalize across the IllustrisTNG suite, but the method itself has promise and the fix is cheap.","headline":"In-sample GNN gains are real, but the 'independent TNG300' test likely shares clusters with training, so the generalization claim needs a disjoint-cluster rerun.","tokens_in":11908,"tokens_out":2388,"would_cite":false,"duration_ms":24613,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A graph neural network predicts galaxy cluster dark matter masses from stellar masses, projected positions, and line-of-sight velocities, cutting the prediction error of the best random forest baseline by roughly a third.","keywords":["graph neural networks","dark matter halos","galaxy clusters","IllustrisTNG","stellar-halo mass relation","halo mass estimation","machine learning","cosmological simulations"],"falsifier":"Cross-match the 352 TNG-Cluster zoom-in halos against the TNG300 halo catalog at $z=0$; if a substantial fraction of the TNG300 test galaxies belong to halos that are also the targets of the TNG-Cluster zooms, the independence premise is false, and a clean generalisation test would need to train on clusters absent from the test volume.","tokens_in":10938,"feed_emoji":"🌌","tokens_out":8424,"duration_ms":75494,"temperature":0.7,"pith_summary":"The paper sets out to show that a graph neural network can estimate the dark matter halo mass of galaxies in clusters from their stellar masses, projected positions, and line-of-sight velocities, and that it does so markedly better than random forest baselines. On the TNG300 test set the GNN achieves an RMSE of 0.242 dex and $R^2=0.785$, compared with 0.344 dex and $R^2=0.567$ for the best random forest, which also receives an overdensity parameter. The GNN represents each cluster as a graph, letting the model use pairwise spatial and kinematic information rather than a scalar density. A sympathetic reader would care because cluster halo masses are hard to measure and are usually estimated from crude proxies; the paper suggests that the substructure already visible in galaxy positions and velocities carries a large part of the answer.","feed_headline":"GNN slices halo-mass prediction error by a third","feed_subtitle":"On an independent simulation, it needs only stellar masses plus projected galaxy positions and velocities.","key_machinery":"The central object is the graph built from the cluster galaxy sample. Each node is a galaxy with stellar mass $M_*$ as its feature; edges connect galaxies separated by less than 3 Mpc, and each edge carries two features: the squared Euclidean distance between the projected positions and the squared relative line-of-sight velocity. The GNN processes this graph with eight unshared layers of two-layer MLPs, max-pools the edge information back to each node, concatenates the node's original stellar mass, and passes the result through a three-layer output MLP that predicts both $\\log M_{\\rm halo}$ and a log-variance. This construction is what allows the model to use the spatial and kinematic arrangement of neighbours, instead of a hand-crafted scalar overdensity.","core_discovery":"The central claim is that a graph neural network, trained on the TNG-Cluster zoom-in simulations and tested on the independent TNG300 simulation, predicts galaxy cluster halo masses with RMSE $0.242$ dex and $R^2 = 0.785$, compared with $0.344$ dex and $R^2 = 0.567$ for the best random forest baseline. The GNN uses each galaxy's stellar mass as the sole node feature and builds edges to galaxies within 3 Mpc, carrying the squared projected separation and the squared line-of-sight velocity difference. It outperforms the random forest on every metric in Table 1, and its error stays low near cluster centres, where the random forest degrades. The paper interprets this as evidence that the graph representation captures substructure that matters for the stellar–halo mass relation.","pith_inferences":["The paper's 'independent test' claim would be overturned if the TNG-Cluster zoom-in halos were originally selected from the TNG300 volume; testing whether the two halo populations are disjoint is a necessary check before interpreting the TNG300 numbers as generalisation.","The paper does not isolate the contributions of the spatial edge feature from the velocity edge feature; an ablation on the same data would show which piece of cluster substructure carries the predictive gain.","The same graph representation could be applied to observed cluster catalogs with spectroscopic redshifts, where the GNN's per-galaxy variance could flag galaxies whose halo masses are poorly constrained by the environment."],"forward_implications":["On the TNG-Cluster cross-validation, the GNN reaches RMSE 0.273 dex, reducing the best random forest's 0.385 dex by about 29 percent.","The GNN's advantage over the random forest persists at all cluster-centric distances, with the largest gap in the dense central regions where tidal stripping weakens the stellar–halo mass relation.","Because the model takes projected positions and line-of-sight velocities as inputs, the same graph construction transfers directly to spectroscopic galaxy surveys, which provide exactly these observables.","Training with a Gaussian negative log-likelihood loss gives the model a per-galaxy predicted variance, so each halo mass estimate comes with an uncertainty.","The comparable performance between the TNG-Cluster validation set and the TNG300 test set suggests the model is robust to domain shift within the IllustrisTNG suite."],"supporting_citations":[{"why":"Supplies the TNG-Cluster zoom-in simulation used for training and validation data.","marker":"[24]"},{"why":"Supplies the TNG300 simulation used as the independent test set and defines the IllustrisTNG suite.","marker":"[25]"},{"why":"Establishes the stellar–halo mass relation that motivates predicting M_halo from M_star.","marker":"[39]"},{"why":"Provides the GNN architecture, unshared layers, and edge-feature construction that the paper adopts.","marker":"[41]"},{"why":"Prior GNN work predicting halo masses that the paper builds on and extends to the cluster regime.","marker":"[18]"},{"why":"Provides the Gaussian negative log-likelihood loss used to predict both halo mass and its log-variance.","marker":"[15]"},{"why":"Defines the random decision forest algorithm used as the baseline regressor.","marker":"[14]"},{"why":"Provides the graph partitioning strategy used to train the GNN on the large cluster graph.","marker":"[6]"}],"fun_headline_variants":["GNN trims halo-mass prediction error by a third","Graph neural nets outshine random forests on halo masses","GNN improves galaxy cluster halo mass estimates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The generalisation claim rests on the premise that TNG300 is an independent simulation from TNG-Cluster; if the zoom-in clusters were selected from the TNG300 volume, then the TNG300 test would only show transfer across resolution, not generalisation to unseen clusters.","fun_headline_variants_meta":{"raw":{"variants":["GNN trims halo-mass prediction error by a third","Graph neural nets outshine random forests on halo masses","GNN improves galaxy cluster halo mass estimates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00028,"raw_usage":{"total_tokens":1618,"prompt_tokens":860,"completion_tokens":758,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":709}},"tokens_in":476,"tokens_out":758,"duration_ms":7600,"temperature":1.0,"reasoning_tokens":709,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:20:16.755386+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Cross-match the 352 TNG-Cluster zoom-in halos against the TNG300 halo catalog at $z=0$; if a substantial fraction of the TNG300 test galaxies belong to halos that are also the targets of the TNG-Cluster zooms, the independence premise is false, and a clean generalisation test would need to train on clusters absent from the test volume.","supporting_citations":[{"cited_title":"Introducing the TNG-Cluster simulation: Overview and the physical properties of the gaseous intracluster medium","cited_arxiv_id":null,"evidence_quote":"Supplies the TNG-Cluster zoom-in simulation used for training and validation data."},{"cited_title":"The IllustrisTNG simulations: Public data release","cited_arxiv_id":null,"evidence_quote":"Supplies the TNG300 simulation used as the independent test set and defines the IllustrisTNG suite."},{"cited_title":"Wechsler and Jeremy L","cited_arxiv_id":null,"evidence_quote":"Establishes the stellar–halo mass relation that motivates predicting M_halo from M_star."},{"cited_title":"Wu, Christian Kragh Jespersen, and Risa H","cited_arxiv_id":null,"evidence_quote":"Provides the GNN architecture, unshared layers, and edge-feature construction that the paper adopts."},{"cited_title":"Larson, John F","cited_arxiv_id":null,"evidence_quote":"Prior GNN work predicting halo masses that the paper builds on and extends to the cluster regime."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Gaussian negative log-likelihood loss used to predict both halo mass and its log-variance."},{"cited_title":"Random decision forests","cited_arxiv_id":null,"evidence_quote":"Defines the random decision forest algorithm used as the baseline regressor."},{"cited_title":"Cluster- GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks","cited_arxiv_id":null,"evidence_quote":"Provides the graph partitioning strategy used to train the GNN on the large cluster graph."}],"review_version":1}