{"id":"b6bfcaf3-37c8-40a7-b7c3-577e150eee5d","arxiv_id":"2509.03547","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"MatterVial combines pretrained graph-network latent features, compressed descriptors, and SISSO formulas to make MODNet competitive with state-of-the-art GNNs on MatBench with improved interpretability.","lead":"This paper introduces MatterVial, a toolkit that adds hidden insights from several pretrained material models, fast approximations of slow descriptors, and symbolic formulas to a classic feature-based predictor. On the MatBench benchmark, the boosted predictor often matches or beats end-to-end graph networks while keeping features readable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark leakage through pretrained GNN features, not SISSO, is the load-bearing risk: MVL/ROOST/ORB were pretrained on Materials Project data overlapping MatBench's mp_gap, mp_is_metal, and mp_e_form test sets, so the headline SOTA-competitive error reductions on those tasks may be inflated.","rationale":"The reader's conditional verdict is appropriate, but the most load-bearing concern is not SISSO. The abstract's headline claim is about beating state-of-the-art GNNs on MatBench, and that claim depends on a fair benchmark. The task-oriented GNN features—especially the MVL MEGNet features—are extracted from models trained on the same Materials Project data that forms the MatBench mp_gap, mp_is_metal, and mp_e_form test sets, including the same target properties. This is a direct form of label leakage through feature extraction, and it affects the largest and most impressive numbers in Table 1. The SISSO concern identified by the reader is real because SISSO formulas are selected using target correlations on the full dataset, but it is structurally separate and less damaging: removing SISSO would not remove the GNN-feature leakage, whereas removing the leaked GNN features would directly change the core comparison. The paper's own caveat admits the issue in one sentence but does not quantify overlap or exclude the affected tasks from the headline. A concrete overlap audit and a rerun with the pretrained task-oriented feature groups ablated would settle whether the claimed SOTA-competitive performance survives on those tasks. I therefore keep the reader's CONDITIONAL verdict rather than moving it, since the framework may still be useful and interpretable, but the benchmark numbers need this check before they can be taken at face value.","tokens_in":30152,"tokens_out":8686,"duration_ms":84173,"concrete_test":"Quantify overlap: match every structure in matbench_mp_gap, matbench_mp_is_metal, and matbench_mp_e_form against the MP snapshots used to train the MVL models (2018.6.1 and 2019.4.1) by canonical structure/formula matching, and report the fraction of test entries with exact matches. Then rerun MODNet@MM+MV on those three tasks with the MVL, ROOST, and ORB feature groups disabled, keeping only ℓ-MM, ℓ-OFM, and SISSO features. If the overlap fraction is high and the error reductions drop below the claimed 37–77% (or MODNet no longer beats the MatBench record on those tasks), the headline SOTA comparison is inflated by pretraining leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central claim—MODNet augmented with MatterVial features is competitive with, and in several cases superior to, state-of-the-art end-to-end GNNs on MatBench—the pivotal condition is that the evaluation is leakage-free. That condition is least secure for the three large Materials-Project-based tasks: matbench_mp_gap, matbench_mp_is_metal, and matbench_mp_e_form. The task-oriented GNN features come from MVL MEGNet models pretrained on MP snapshots (band gap: 2018.6.1; formation energy: 2019.4.1; elastic and metal classification models are also MP-derived). For these MatBench test splits, the test crystals and their target labels are very likely inside the pretraining set. The feature vector MODNet sees for a test object is therefore produced by a model whose weights were fitted to that object's target value—direct target leakage through the frozen features. ROOST and ORB add composition- and structure-level overlap of the same kind from other corpora. The paper itself flags this in Results: 'some of the larger MatBench tasks can no longer be considered truly independent test sets for models exposed to vast amounts of similar ab initio data during pretraining.' However, the caveat is not quantified, and the affected tasks still drive the headline '>40% error reduction' numbers (77% for mp_is_metal, 67% for mp_e_form, 37.6% for mp_gap). If those entries are corrected or removed, the claim of superiority over SOTA GNNs on large tasks is substantially weakened. The SISSO formula-selection issue is genuine but secondary: SISSO features are chosen with full-data target correlations, yet even if they were removed, the pretrained-GNN leakage would remain.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MatterVial, an open-source Python framework that augments the feature-based model MODNet with latent representations from pretrained GNNs (MVL/MEGNet, ROOST, ORB), GNN-approximated descriptors (ℓ-MM and ℓ-OFM), and SISSO-derived symbolic-regression features. The authors evaluate the approach on all 13 MatBench v0.1 tasks, reporting error reductions relative to a MatMiner-featurized MODNet baseline and, in several cases, claiming performance competitive with or superior to state-of-the-art end-to-end GNNs. They also present an interpretability module that uses SHAP and SISSO to decode GNN latent features into chemical formulas, and an adjacent-GNN featurizer that trains a task-specific GNN per fold for feature extraction. The central claim is that transparent feature-based models can be made competitive with black-box GNNs while preserving interpretability.","tokens_in":30456,"tokens_out":9544,"duration_ms":84676,"significance":"If the results are leakage-free, this is a practically valuable contribution: the framework is modular and open-source, and the controlled perovskite ablation (Table 2) shows that combining pretrained GNN features with traditional descriptors can substantially improve MODNet. The interpretability module, which translates latent GNN features into explicit formulas, is a useful step toward explainable materials ML. However, the headline claims depend heavily on the three largest Materials-Project-based tasks (matbench_mp_gap, matbench_mp_is_metal, matbench_mp_e_form), where the frozen MVL features are generated by models pretrained on the same MP snapshots that constitute the MatBench test sets. The paper acknowledges this concern in one sentence but does not quantify or exclude the affected results, and the SISSO and feature-preselection steps are not described as per-fold operations. The significance is therefore conditional on a leakage-free evaluation protocol.","major_comments":[{"comment":"The central claim of superiority over state-of-the-art GNNs is supported in Table 1 primarily by the matbench_mp_gap (37.6% error reduction), matbench_mp_is_metal (77.1%), and matbench_mp_e_form (67.2%) rows. The MVL MEGNet feature extractors were pretrained on the Materials Project snapshots 2018.6.1 (band gap) and 2019.4.1 (formation energy and other targets), which are exactly the corpora from which the MatBench test sets for these three tasks are drawn. For any test object, the frozen MVL features are therefore produced by a model whose weights were fitted with that object's target value; ROOST and ORB pretraining corpora (MP, OQMD, Alexandria, OMat) overlap similarly. The paper's caveat that 'some of the larger MatBench tasks can no longer be considered truly independent test sets' is not quantified, and the abstract and conclusion still report these results as unqualified successes. This is load-bearing: without these rows, the 'superior in several cases' claim loses its main support. The authors should either separate pretraining-overlap tasks from clean tasks, provide a quantitative overlap analysis, or reframe the contribution around the non-overlapping benchmark results.","section":"Results and discussion, Table 1; Methods (MVL MatterVial featurizers); Results caveat"},{"comment":"The SISSO formulas are derived once per task from the full MatBench datasets (Supplementary Table S12 lists the same tasks, e.g., matbench_mp_gap, matbench_perovskites, matbench_log_gvrh), and the initial feature preselection for all experiments is described as an XGBoost-based recursive feature elimination to 800 features without stating that it is performed inside each cross-validation fold. If either the SISSO formula selection or the RFE preselection uses test-fold target labels, every result involving SISSO features (Table 1 Best-groups columns; Table 2 step MODNet@ℓ-MM+ℓ-OFM+MVL+SISSO) and potentially all reported feature-selection outcomes are optimistically biased. The sentence 'By opting for the expressions produced at the SIS step instead of the final SISSO formula, versatility and generalization are assured' does not rule out label leakage. Please specify the exact cross-validation nesting (e.g., performing SISSO and RFE on training folds only) or re-evaluate the affected models with a leakage-free protocol.","section":"Methods (MODNet model training; SISSO-based formula featurizer); Supplementary Table S12"},{"comment":"The strongest controlled evidence is Table 2's perovskite ablation, which shows a clear improvement from the ORB features (0.0639 to 0.0386 eV/unit cell). However, the incremental gains attributed to SISSO (0.0673 to 0.0653) and to ROOST after SISSO (0.0653 to 0.0639) are within the reported standard deviations of the models, and the final full-feature result (0.0386 ± 0.0009) is only marginally better than the published HackNIP MODNet@ORB baseline (0.0397). The paper's claim that all MatterVial feature groups act synergistically is therefore not convincingly established. Please provide confidence intervals or significance tests for the incremental additions, and state clearly which improvements are meaningful beyond the ORB features.","section":"Table 2; Results (Synergy of MatterVial features)"}],"minor_comments":[{"comment":"The footnote formulas for % error reduction are corrupted by Unicode rendering and are not legible; please rewrite with standard mathematical notation.","section":"Table 1 footnote"},{"comment":"The benchmark model is listed as 'coNGN', but the intended model is coGN; fix the typo.","section":"Table 1, row Logarithmic Kvrh"},{"comment":"The statement that MatterVial reduces feature generation time by 'a minimum of two orders of magnitude' is presented without a benchmark table or measurement description; please add supporting data or soften the claim.","section":"Methods, MatterVial implementation"},{"comment":"The abstract's 'accuracy increases exceeding 40%' should be qualified by the pretraining-overlap caveat; as written, it conflicts with the Results sentence acknowledging that the larger MatBench tasks are not truly independent test sets.","section":"Abstract; Results caveat"},{"comment":"The footnote states that the reported MODNet baseline of 0.960 AUROC could not be replicated; please clarify the exact MODNet configuration and hyperparameters used, and whether the published MODNet results are reproduced for the other tasks.","section":"Table 1, glass formability footnote"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a timely topic, and the MatterVial package is likely to be a useful community resource. The main risks are the pretraining-overlap leakage on the largest MP-derived tasks and the absence of a described per-fold protocol for SISSO and the XGBoost-RFE preselection; both are load-bearing for the headline claims. I do not recommend rejection because the underlying framework is sound and the controlled perovskite ablation provides a credible core, but the evaluation must be made leakage-free before the claims can be accepted. The authors should also position their contribution relative to HackNIP (ref. 26) more explicitly, since the incremental gains over that work appear modest in Table 2."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is a well-executed example of the 'GNN as featurizer' trend, but the central benchmark claim has a load-bearing leak. The new thing here is the packaging: MatterVial combines pretrained task-oriented GNN features (MVL MEGNet, ROOST, ORB), autoencoder-compressed MatMiner/OFM descriptors generated by proxy GNNs, and first-rung SISSO formulas, all in one open-source package. The interpretability loop—SHAP + surrogate XGBoost + SISSO to decode latent GNN features—is genuinely useful and goes beyond similar prior work. The controlled ablation on the perovskite task (Table 2) is careful and credible: each feature group adds a small but consistent gain, and the ORB features do most of the heavy lifting. Code, data, and hyperparameters are provided; citations to related work (HackNIP) are fair.\n\nThe soft spot is the evaluation protocol on the big MP-derived MatBench tasks. The MVL MEGNet featurizers were pretrained on Materials Project snapshots that overlap with the mp_gap, mp_is_metal, and mp_e_form test sets. For those tasks, the feature vector MODNet sees for a test crystal was produced by a model whose weights were fitted to that crystal's target value. That is direct target leakage through the frozen feature extractor. The paper acknowledges this in a sentence, but the abstract still leads with the >40% error reductions on those very tasks (77% for mp_is_metal, 67% for mp_e_form, 37.6% for mp_gap). Remove or correct those entries, and the claim of beating SOTA GNNs on large tasks is substantially weakened. This is not a minor caveat; it is load-bearing.\n\nA secondary issue: the SISSO formulas were selected using target correlations on the full datasets, with no described per-fold split. That can inflate the SISSO and hiSISSO contributions, though the effect is smaller than the pretraining leak. The perovskite ablation is cleaner because that task is not MP-derived, so the incremental results there are probably trustworthy.\n\nIn short: the framework is a useful engineering contribution for small, non-MP datasets and for interpretability; the benchmark claims overreach. The paper deserves peer review, but the authors should be asked to rerun the MP-overlap tasks with a leakage-controlled protocol or explicitly disentangle pretrained vs. non-overlapping features. As is, I would not cite the headline numbers.","headline":"Useful hybrid featurizer with a real evaluation leak: the headline SOTA-competitive numbers rest on pretrained GNN features that have already seen the MatBench test sets.","tokens_in":31112,"tokens_out":3210,"would_cite":false,"duration_ms":29662,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hidden GNN features lift transparent MODNet past black-box rivals","keywords":["MatterVial","MODNet","graph neural networks","feature-based machine learning","symbolic regression","SISSO","materials informatics","interpretable machine learning"],"falsifier":"Rerun all 13 MatBench tasks with a strictly fold-local protocol that regenerates SISSO formulas inside each training fold and retrains or constrains the pretrained feature extractors on a corpus explicitly excluding every benchmark structure; if the error reductions above 40% shrink toward the MatMiner baseline, the central comparison is optimistic.","tokens_in":29892,"feed_emoji":"🧪","tokens_out":7051,"duration_ms":68687,"temperature":0.7,"pith_summary":"This paper claims that a feature-based materials model, MODNet, can be made competitive with black-box graph neural networks if its ordinary chemical descriptors are supplemented with three extra feature families: latent representations extracted from pretrained GNNs, fast GNN-generated copies of expensive descriptors, and formulas found by symbolic regression. Across the 13-task MatBench benchmark, the additions reduce prediction errors on nearly every task, with reported gains exceeding 40% in several cases, and the augmented model beats the previous record on tasks such as elastic moduli, band gap, metallicity, and formation energy. The point of the hybrid is to keep the transparency of feature-based models while borrowing representational power from deep learning. An integrated interpreter decodes the most important GNN-derived features into short physical formulas, so the accuracy gain is meant to come without losing explainability.","feed_headline":"Hidden GNN features lift transparent MODNet past black-box rivals","feed_subtitle":"Latent GNN features and symbolic formulas cut materials benchmark errors by over 40 percent, keeping descriptors interpretable.","key_machinery":"The load-bearing object is MatterVial, a modular featurizer that turns pretrained graph models into feature generators instead of end-to-end predictors. It supplies three feature families: descriptor-oriented latent features, which are autoencoded MatMiner and orbital-field-matrix descriptors reproduced by a MEGNet proxy; task-oriented latent features, which are activations from MEGNet, ROOST, and ORB layers; and SISSO rung-one formulas built from MatMiner features. The argument then runs through MODNet's feature-selection and deep-ensemble pipeline, which chooses the useful subset of this enlarged pool and thereby reveals which feature groups carry each task. A separate interpreter uses XGBoost surrogates plus SHAP to identify the interpretable descriptors most correlated with each latent feature, and SISSO converts those descriptors into explicit formulas.","core_discovery":"On its own terms, the paper's central discovery is that the gap between interpretable feature-based models and end-to-end graph neural networks is not inherent. Feeding MODNet a feature vector that concatenates hidden-layer activations of pretrained structure-based, composition-based, and equivariant models, together with latent encodings of MatMiner and orbital-field-matrix descriptors that a neural network predicts directly from the crystal structure, and paired-feature symbolic-regression formulas, yields consistently lower errors than MatMiner features alone. In the paper's reported numbers, the hybrid reaches or surpasses the best GNN-only results on several MatBench tasks, including the large band-gap and formation-energy datasets, and it sets new feature-based records on tasks where MODNet was already strong. The authors further claim that the most important latent features can be approximated by compact SISSO formulas based on interpretable chemical descriptors, and that feeding those formulas back as new features improves accuracy even further.","pith_inferences":["One natural extension is to regenerate the SISSO formulas inside each cross-validation fold for every task; the paper does not report doing this, and a fold-local version would make the benchmark numbers robust against target leakage.","Because the ORB features dominate the gains, substituting other large-scale equivariant machine-learning potentials, or chaining several such models, would directly test whether larger pretraining corpora translate into further error reductions.","The same GNN-as-featurizer plus symbolic-decoder template could be carried over to molecular or polymeric datasets, where the interpretable formulas would take the form of fragment or group-contribution rules rather than crystal packing descriptors.","If the decoded formulas remain stable across datasets, they could be used proactively as screening rules rather than only as post-hoc explanations."],"forward_implications":["Feature-based models can be treated as the backbone of a materials prediction pipeline; expensive end-to-end architecture search may be unnecessary when pretrained GNN features are available.","Computationally heavy descriptors like orbital-field matrices can be replaced by GNN-predicted latent approximations with little or no loss, making large-scale featurization orders of magnitude faster.","The interpreter's decoded formulas give researchers a route from model predictions to testable chemical hypotheses, such as which orbital interactions or packing motifs stabilize a perovskite.","Feeding interpretable formulas back into the model as hiSISSO features improves accuracy, suggesting that explainability and performance can reinforce each other rather than trade off.","On small experimental datasets, where GNNs usually struggle, the hybrid preserves and extends the advantages of feature-based models."],"supporting_citations":[{"why":"Defines MODNet, the feature-based model whose MatMiner baseline and training pipeline all MatterVial comparisons extend.","marker":"[4]"},{"why":"Supplies the MatMiner descriptor set that MatterVial augments and that its SISSO formulas start from.","marker":"[7]"},{"why":"Provides the MatBench leaderboard records that the augmented MODNet model is compared against.","marker":"[12]"},{"why":"Supplies the pretrained composition-based ROOST model whose latent layers become MatterVial features.","marker":"[20]"},{"why":"Supplies the equivalently invariant ORB model whose layers form the most influential feature group in the paper.","marker":"[21]"},{"why":"Provides the concurrent MODNet@ORB baseline that MatterVial is compared with on the perovskite task.","marker":"[26]"},{"why":"Defines the 13 MatBench tasks and the five-fold protocol on which all reported metrics are evaluated.","marker":"[30]"},{"why":"Provides the hierarchical symbolic-regression idea the paper adapts when feeding decoded formulas back as hiSISSO features.","marker":"[48]"},{"why":"Provides the SISSO++ implementation used to derive rung-one paired-feature formulas and to decode GNN features into interpretable expressions.","marker":"[51]"}],"fun_headline_variants":["Transparent hybrid model rivals black-box GNNs on materials benchmarks","Interpretable AI: GNN latent features boost MODNet by 40%","Feature-based model + GNN latents: transparent, competitive, accurate","Hidden GNN features make interpretable MODNet beat black-box AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported benchmark gains assume that every added feature group was generated fairly: per-task SISSO formulas were not selected using held-out labels, and pretrained models' exposure to similar structures did not inflate test scores.","fun_headline_variants_meta":{"raw":{"variants":["Transparent hybrid model rivals black-box GNNs on materials benchmarks","Interpretable AI: GNN latent features boost MODNet by 40%","Feature-based model + GNN latents: transparent, competitive, accurate","Hidden GNN features make interpretable MODNet beat black-box AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00079,"raw_usage":{"total_tokens":3481,"prompt_tokens":946,"completion_tokens":2535,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":2457}},"tokens_in":562,"tokens_out":2535,"duration_ms":19020,"temperature":1.0,"reasoning_tokens":2457,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:37:42.650142+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun all 13 MatBench tasks with a strictly fold-local protocol that regenerates SISSO formulas inside each training fold and retrains or constrains the pretrained feature extractors on a corpus explicitly excluding every benchmark structure; if the error reductions above 40% shrink toward the MatMiner baseline, the central comparison is optimistic.","supporting_citations":[{"cited_title":"& Rignanese, G","cited_arxiv_id":null,"evidence_quote":"Defines MODNet, the feature-based model whose MatMiner baseline and training pipeline all MatterVial comparisons extend."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MatMiner descriptor set that MatterVial augments and that its SISSO formulas start from."},{"cited_title":"MatBench Leaderboard","cited_arxiv_id":null,"evidence_quote":"Provides the MatBench leaderboard records that the augmented MODNet model is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained composition-based ROOST model whose latent layers become MatterVial features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the hierarchical symbolic-regression idea the paper adapts when feeding decoded formulas back as hiSISSO features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SISSO++ implementation used to derive rung-one paired-feature formulas and to decode GNN features into interpretable expressions."}],"review_version":2}