{"id":"be6af8b3-1baf-4b8d-9198-673b732ab785","arxiv_id":"2607.11996","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":5.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Coordination-resolved tree ensembles on 320 cubic spinels reach ~0.12 eV/atom formation-energy MAE and 0.85 metallicity accuracy under grouped holdouts, while band-gap regression fails to beat a trivial baseline.","lead":"Tree-ensemble models trained on 320 Materials Project spinels predict formation energy, hull distance, magnetization, and metallicity using coordination-based cation grouping. Careful group-aware validation and an honest negative result on band gaps make the work useful for materials screening practice.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"Central MAE claims rest on an asserted but unverifiable leakage-free grouped protocol; abstract-only evidence cannot confirm that transforms, CrystalNN assignments, and champion selection stayed strictly inside training folds.","rationale":"The reader correctly flags both label quality (acknowledged by the authors for band gaps) and the necessity of full-text/code verification of the grouped CV pipeline. The performance numbers themselves are the load-bearing claim; their validity hinges on the protocol having been executed without leakage. Because only the abstract is available, that execution cannot be audited, so the CONDITIONAL verdict with low confidence remains appropriate. No stronger internal inconsistency is visible in the abstract, and the authors’ own negative band-gap result and mild-optimism caveat already limit over-claim. Release of artifacts would settle the issue; until then the numbers stay provisional.","tokens_in":2250,"tokens_out":494,"duration_ms":17717,"concrete_test":"Obtain the full manuscript, code, and exact train/holdout indices; re-execute the twenty repeated grouped holdouts from the published scripts (transforms and CrystalNN fit only on each train fold) and verify that the mean MAEs and standard deviations match the abstract within the reported uncertainties. Any global pre-computation or post-selection of champions on the holdout would falsify the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim (formation-energy MAE 0.121±0.030 eV/atom, hull 0.048±0.013 eV/atom, magnetization 1.27±0.19 μB/fu, metallicity 0.85±0.06 under twenty repeated grouped holdouts) is credible only if every step—reduced-formula grouping, CrystalNN site assignment, feature transforms, hyper-parameter tuning, and champion selection—was performed strictly inside each training fold before any holdout was examined. The abstract asserts this protocol and notes mild optimism from single-seed refits, yet supplies no fold-size statistics, leakage diagnostics, or confirmation that coordination numbers and electronegativity descriptors were recomputed solely on train structures. With N=320 and chemistry-grouped splits, even modest global pre-processing would produce optimistically biased holdout numbers that no longer support the claimed out-of-chemistry performance.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The manuscript curates 320 cubic (Fd-3m) spinel entries from the Materials Project (nitrides, oxides, sulfides, selenides, including mixed-valence A3X4) and trains tree-ensemble surrogates for formation energy, energy above hull, band gap, total magnetization, and metallicity. Cations are assigned to tetrahedral-like and octahedral-like groups via CrystalNN coordination rather than element identity. Evaluation is group-aware: splits by reduced formula, transforms fit on training folds only, and champions selected on CV before holdout. Over twenty repeated grouped holdouts the reported MAEs are 0.121±0.030 eV/atom (formation energy), 0.048±0.013 eV/atom (hull), 1.27±0.19 μB/fu (magnetization), with metallicity accuracy 0.85±0.06; band-gap regression fails to beat a trivial baseline on the 19-member non-metal holdout under paired bootstrap. On the same split the tabular champion beats an untuned MEGNet trained from scratch on formation energy. SHAP, grouped conformal intervals, permutation nulls, and leave-one-chemistry-out tests are used to interpret models and bound applicability.","tokens_in":2481,"tokens_out":1403,"duration_ms":18568,"significance":"If the leakage-free grouped protocol and reported errors hold under full scrutiny, the work supplies practical, chemistry-aware surrogates for thermodynamic stability and magnetization of cubic spinels at a data scale where graph networks are not yet decisive, and it documents an honest negative result for band-gap regression. Strengths that should be credited include: reduced-formula grouping, train-only transforms, champion selection before holdout, paired bootstrap for the band-gap null, explicit comparison to untuned MEGNet on an identical split, and post-hoc domain bounds (SHAP, conformal, leave-one-chemistry-out). These practices raise the bar for small-N materials ML reporting even if the absolute errors remain modest and label-limited by semi-local DFT.","major_comments":[{"comment":"The central MAE and accuracy claims (formation energy 0.121±0.030 eV/atom, hull 0.048±0.013 eV/atom, magnetization 1.27±0.19 μB/fu, metallicity 0.85±0.06 over twenty repeated grouped holdouts) are load-bearing and rest on the assertion that reduced-formula grouping, CrystalNN site assignment, feature transforms, hyperparameter tuning, and champion selection all stayed strictly inside training folds. The abstract states this protocol and notes mild optimism from single-seed refits, but does not supply fold-size statistics, per-fold chemistry composition, or leakage diagnostics confirming that coordination numbers and electronegativity descriptors were recomputed solely on train structures. With N=320 and chemistry-grouped splits, even modest global pre-processing would bias the holdout numbers. The full methods must document these steps with enough detail (and preferably code or fold indi","section":null},{"comment":"The band-gap negative result is appropriately framed (does not beat a trivial baseline on the 19-member non-metal holdout under paired bootstrap) and attributed to sample scarcity plus semi-local DFT labels. That honesty is a strength, but the same scarcity implies that any screening use case for non-metals is currently unsupported. The manuscript should state explicitly which screening decisions the surrogates are and are not intended to support, and whether metallicity classification (accuracy 0.85±0.06) is the recommended substitute for gap regression.","section":null},{"comment":"The MEGNet comparison (tabular champion 0.087 vs untuned MEGNet 0.209 eV/atom formation-energy MAE on the primary holdout) is correctly described as bounding rather than settling the descriptor-versus-graph question. Because the graph model is untuned and trained from scratch on this small set, the comparison cannot be read as evidence that tabular features are intrinsically superior. Either a modest hyperparameter search for MEGNet on the same grouped protocol should be added, or the claim should remain strictly limited to “untuned MEGNet from scratch” throughout the discussion and abstract.","section":null},{"comment":"Applicability is described as uneven across anions and cations (grouped conformal intervals, permutation nulls, leave-one-chemistry-out). Those diagnostics are load-bearing for any claim of transfer beyond the curated 320-entry set. The full text must report which anion/cation groups drive the leave-one-chemistry-out failures and whether the conformal coverage is calibrated under the same grouped splits; without that, the domain-of-applicability paragraph remains qualitative.","section":null}],"minor_comments":[{"comment":"Abstract: clarify whether the twenty repeated holdouts re-draw the grouped splits each time or only re-fit with fixed splits and fixed hyperparameters; the phrase “single-seed refits that reuse the tuned hyperparameters” is slightly ambiguous.","section":null},{"comment":"Abstract: “nitrides, oxides, sulfides, and selenides” appears in the opening list while the title emphasizes oxides, sulfides, and selenides; confirm nitride count and whether nitride chemistry is retained in all reported metrics.","section":null},{"comment":"Notation: define μB/fu and eV/atom consistently on first use in the main text; the abstract mixes “μBfu” and “eV/atom” styles.","section":null},{"comment":"When the full manuscript is prepared, include a small table of dataset composition (counts by anion, by A/B cation family, metal vs non-metal) so readers can judge the 19-member non-metal holdout in context.","section":null},{"comment":"Cite the specific Materials Project database version and the CrystalNN settings used for tetrahedral/octahedral assignment so the 320-entry curation is reproducible.","section":null}],"recommendation":"major_revision","confidential_remarks":"This assessment is based on the abstract alone (full text not provided). The described protocol is defensible and the negative band-gap result is a credit, but the MAE claims cannot be verified without methods, fold diagnostics, and preferably code. I would not recommend accept or minor_revision until the leakage-free pipeline is auditable. Scope fit for a materials-informatics or computational materials journal is reasonable if the full methods match the abstract’s hygiene claims."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing to know is that this is a careful materials-informatics paper on tree-ensemble surrogates for 320 cubic spinels from Materials Project. Under repeated grouped holdouts they report usable MAEs (formation energy 0.121±0.030 eV/atom, hull 0.048±0.013 eV/atom, magnetization 1.27±0.19 μB/fu, metallicity accuracy 0.85±0.06) and openly show that band-gap regression fails to beat a trivial baseline on the 19 non-metals.\n\nWhat is new is the coordination-number grouping via CrystalNN rather than element identity, plus the insistence that every transform and champion selection stay inside training folds before the holdout is touched. That protocol description is stronger than most abstracts in this space. They also run an untuned MEGNet on the identical split (tabular wins at this scale), use SHAP to link magnetization to octahedral d-occupancy, and bound domain of applicability with leave-one-chemistry-out and conformal intervals. The negative band-gap result is handled cleanly with paired bootstrap and attributed to sample scarcity plus semi-local DFT labels.\n\nSoft spots are real but proportionate. The central numbers are credible only if CrystalNN assignments, feature transforms, and hyper-parameter search never saw the holdout; with only the abstract we cannot check fold sizes or leakage diagnostics. N=320 with chemistry grouping is small, uncertainties are wide, and the curated set membership plus tree hyper-parameters are free parameters. Label quality for gaps is a known Materials Project limitation they acknowledge. None of that invents a fatal flaw; it just means the out-of-chemistry claims stay provisional until code and data appear.\n\nThis is for people screening spinels or related oxides/sulfides/selenides who want cheap filters before DFT. Methodologists will get value from the evaluation design and the honest negative. It deserves a serious referee—send it out. I would not cite it myself unless I am working on spinel screening, but the work is coherent and engaged with the literature.","headline":"Careful spinel surrogate work with group-aware protocol and an honest negative band-gap result; useful screening numbers, but abstract-only so leakage claims stay unverified.","tokens_in":3093,"tokens_out":528,"would_cite":false,"duration_ms":19547,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Coordination-aware tree ensembles predict spinel thermodynamics and magnetism from Materials Project data, but not band gaps.","keywords":["spinel oxides","surrogate models","formation energy","convex hull","magnetization","band gap","coordination chemistry","Materials Project"],"falsifier":"A new set of higher-rung (hybrid or GW) calculations, or experimental measurements, on a chemically diverse hold-out of spinels that shows the reported formation-energy or magnetization MAEs rising well above the stated error bars, or that shows the band-gap model suddenly beating a constant predictor once better labels are used.","tokens_in":3160,"feed_emoji":"⚡","tokens_out":701,"duration_ms":5760,"temperature":0.7,"pith_summary":"This paper builds surrogate models that predict key properties of cubic spinels—oxides, sulfides, selenides, and related compounds—without running new density-functional calculations for every candidate. The authors curated 320 Materials Project entries, assigned cations to tetrahedral-like and octahedral-like sites by local coordination rather than by chemical identity, and trained tree ensembles under strictly group-aware evaluation so that no reduced formula leaks between train and test. On repeated held-out groups the models recover formation energies to roughly 0.12 eV/atom, distances to the convex hull to 0.05 eV/atom, and magnetizations to about 1.3 µB per formula unit, while correctly classifying metals versus non-metals most of the time. Band-gap regression fails to beat a trivial baseline, a negative result the authors attribute to scarce non-metal examples and the known underestimation of gaps by semi-local DFT. The same protocol also shows that a carefully tuned tabular model can outperform an untuned graph network at this data scale, and SHAP analysis plus conformal intervals map which chemistries the surrogates can be trusted for.","feed_headline":"Spinel surrogates hit 0.12 eV/atom on formation energy","feed_subtitle":"Coordination-aware trees work for thermodynamics and magnetism; band gaps stay out of reach.","key_machinery":"Coordination-resolved tabular descriptors: cations are partitioned into tetrahedral-like and octahedral-like groups by CrystalNN coordination numbers rather than by element identity, then fed to carefully regularized tree ensembles whose entire preprocessing and model-selection pipeline is fit only on training folds of group-aware splits.","core_discovery":"Under repeated grouped holdouts that keep every reduced formula entirely on one side of the split, tree-ensemble surrogates trained on CrystalNN coordination features reach mean absolute errors of 0.121 ± 0.030 eV/atom for formation energy, 0.048 ± 0.013 eV/atom for hull distance, and 1.27 ± 0.19 µB/fu for magnetization, with metallicity accuracy 0.85 ± 0.06; band-gap regression does not beat a trivial baseline on the 19-member non-metal holdout.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Coordination trees hit 0.12 eV/atom spinel formation energy","Spinel surrogates: 0.048 eV/atom hull MAE via CrystalNN groups","Grouped holdouts: 1.27 µB magnetization MAE for spinels","Trees reach 0.85 metallicity accuracy on cubic spinels","Band-gap models fail baseline; thermodynamics succeed"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The semi-local DFT labels and CrystalNN site assignments from Materials Project are accurate and consistent enough that models trained on them remain useful for real screening of spinels.","fun_headline_variants_meta":{"raw":{"variants":["Coordination trees hit 0.12 eV/atom spinel formation energy","Spinel surrogates: 0.048 eV/atom hull MAE via CrystalNN groups","Grouped holdouts: 1.27 µB magnetization MAE for spinels","Trees reach 0.85 metallicity accuracy on cubic spinels","Band-gap models fail baseline; thermodynamics succeed"]},"model":"grok-4.5","effort":"low","cost_usd":0.00422,"raw_usage":{"total_tokens":1408,"prompt_tokens":1009,"num_sources_used":0,"completion_tokens":81,"cost_in_usd_ticks":42200000,"prompt_tokens_details":{"text_tokens":1009,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":318,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":1009,"tokens_out":81,"duration_ms":3156,"temperature":1.0,"reasoning_tokens":318,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-15T08:38:30.349375+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A new set of higher-rung (hybrid or GW) calculations, or experimental measurements, on a chemically diverse hold-out of spinels that shows the reported formation-energy or magnetization MAEs rising well above the stated error bars, or that shows the band-gap model suddenly beating a constant predictor once better labels are used.","supporting_citations":[],"review_version":1}