Pith. sign in

REVIEW 3 major objections 43 references

Bayesian search over material-benchmark configs recovers standard OOD conclusions in twelve steps while cutting more than half the cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 04:58 UTC pith:YW7MXTBJ

load-bearing objection Useful cost-aware OOD+ID config search for materials ML, but the half-cost claim mostly shows BO can match a hand-picked Omee reference inside a discrete grid, not that those configs validate discovery. the 3 major comments →

arxiv 2607.11526 v1 pith:YW7MXTBJ submitted 2026-07-13 cs.LG cs.AI

AutoMatBench: An Automatic Optimization Toolkit for the Acceleration of Material Properties Prediction Benchmarking

Hongxiao Li , Wanling Gao This is my paper
classification cs.LG cs.AI
keywords material property predictionout-of-distribution evaluationBayesian optimizationMatBenchAutoMatBenchbenchmark configurationgraph neural networkscost-efficient evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Material property prediction models look strong on standard MatBench splits, but those splits stay inside the training distribution and therefore overstate how well the same models will handle new, out-of-distribution materials. Performance also swings sharply when three simple knobs—the mix of source databases, the number of material clusters, and how many neighbors sit around each cluster center—are changed. Exhaustive enumeration of every combination is too expensive. AutoMatBench therefore treats the configuration space as a black-box optimization problem and uses Bayesian optimization to locate useful settings. Within a fixed budget of twelve evaluations the search recovers rankings and error levels that match both MatBench and earlier OOD studies, while discarding more than half of the usual computational work and surfacing practical rules of thumb for future benchmarks.

Core claim

Across six property-prediction tasks and five models, Bayesian optimization over the discrete triple (r, n, s) finds configurations that reproduce the conclusions of MatBench and prior OOD work inside twelve steps, cuts evaluation cost by more than half, and consistently recommends n ≤ 24 and s = 7 or 8 (s = 1 only for band-gap), with the database-mix ratio r exerting negligible influence.

What carries the argument

A Bayesian optimizer that treats each (model, configuration) pair as a black-box evaluation: after four random seeds it updates a surrogate and acquisition function to choose the next informative (r, n, s) triple, thereby approximating average performance, model discrimination, or distance to a reference ground-truth without enumerating the full 858-point space.

Load-bearing premise

The paper assumes that matching a single hand-chosen reference configuration, or an aggregate over its own discrete space, is enough to certify that a cheap configuration has recovered the true out-of-distribution ability of a model.

What would settle it

Re-run the same twelve-step optimizer on a larger held-out set of experimentally verified novel materials whose property distributions lie outside both Materials Project and JARVIS; if the recovered (n, s) settings no longer rank models or approximate the reference MAE, the claim that the cheap configurations are reliable fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The manuscript proposes AutoMatBench, a Bayesian-optimization toolkit that searches a discrete configuration space C = R × N × S (dataset ratio r, K-means fold count n, nearest-neighbor count s; 858 points) for material-property-prediction (MPP) benchmarking. Building on MatBench tasks and an Omee-style OOD pipeline (t-SNE + KDE sparse screening + K-means, Section 6.1), it formalizes three problems: best average model performance (first-class, Eq. 1), configuration of maximum model discrimination (dual, Eq. 2), and configuration closest to a defined ground-truth (second-class, Eqs. 3–4). Five models (ALIGNN, SchNet, CrabNet, RF-SCM, MEGNet) are evaluated on six tasks. Empirically, MAE varies by up to ~10 imes across configurations; within a budget of T = 12 BO steps (T0 = 4), the optimizer recovers configurations whose results are claimed to be similar to MatBench/prior OOD work while cutting more than half the cost, with recommended regimes mostly n ≤ 24 and s = 7 or 8 (s = 1 for band gap) and r largely insignificant.

Significance. If the cost-saving and reliability claims hold, AutoMatBench would give the materials-informatics community a practical, sample-efficient way to explore OOD-aware benchmarking configurations instead of exhaustive enumeration or a single fixed split. The explicit framing of configuration causality, the three formal problem classes, and the reproducible BO setup (seed 42, fixed initial D) are useful contributions. The empirical demonstration of large MAE swings across (r, n, s) and the concrete recommended regimes (n ≤ 24, s = 7/8) are actionable for leaderboard design and for reducing experimental cost in MPP model development. Strengths include a clear pipeline (Section 6), multiple experiment sets, and a case study (Table 1) checking approximate vs real labels.

major comments (3)
  1. Experiment set 3 (Section 7.4) defines ground-truth as the single Omee-style point (r, n, s) = (0, 48, 10) (or an aggregate over the same hand-chosen C). Optimizing for configs whose MAE is close to that reference then claiming “similar results … while more than half of the cost are saved” is partly circular: success shows BO can approximate a declared reference inside C, not that the recovered configs validate models under the distribution shifts that matter for new-material discovery. The paper itself reports MAE swings up to ~10 imes (Section 7.2, Figure 4) and a failed shear-modulus case; the reference choice is therefore load-bearing and needs either independent justification or a sensitivity analysis over alternative gt definitions.
  2. Algorithm 2’s Ensure clause maximizes a min pairwise |m(oi1,c)−m(oi2,c)| while Formula (2) states range over models; Algorithm 3’s Ensure returns an option ô ∈ O that minimizes average distance to gt, whereas the second-class problem (Eq. 4 and surrounding text) asks for a configuration. These mismatches mean the reported 12-step “optima” may not be the quantities claimed in the problem statements. The algorithms and ensure clauses should be aligned with Eqs. (2)–(4) and the experimental figures re-checked.
  3. Section 6.1 constructs OOD via t-SNE + KDE sparse screening of 480 candidates then K-means. The manuscript does not show that this proxy captures the compositional/structural/Y-extrapolation shifts that actually impede discovery (cf. the motivation in Section 3 and related work on Y-extrapolation and UQ). Without that link, or at least a comparison to alternative OOD constructions, the claim that AutoMatBench positively contributes to “the cost and efficiency of new material discovery” remains under-supported.

Circularity Check

1 steps flagged

Experiment set 3 defines ground-truth MAE as the single Omee reference config (0,48,10) then optimizes configs for closeness to it, so the “similar results … half the cost” claim holds largely by construction of that objective; Exp. 1–2 and the recommended (n,s) regimes remain independent.

specific steps
  1. other [Section 4.4 (formula 3–4) and Section 7.4 Experiment set 3]
    "The ground-truth MAEs of models are defined as the suggested configuration (r, n, s) = (0,48,10) by the existing study [2]. This assumption ensure that main conclusions with AutoMatBench is capable of reproducing existing findings. A configuration that yields results seriously inconsistent with the ground-truth is considered as unreliable. … Experiments with AutoMatBench reports that, within twelve steps of optimization, the similar results with MatBench and former OOD research can be accessed while more than half of the cost are saved."

    Reliability (and therefore the central cost-saving claim) is defined as MAE closeness to a single externally chosen reference point inside C. Bayesian optimization is then run with exactly that distance as the acquisition objective. Any configuration returned by the optimizer therefore matches the reference by construction of the search; the paper’s assertion that “similar results … can be accessed” is the optimization target itself, not an independent validation that the recovered cheaper configs are reliable under real discovery shifts.

full rationale

The paper is a benchmarking-optimization toolkit, not a first-principles derivation of material properties. Its three formal problems (best model by mean MAE, max-range discrimination config, min-distance-to-gt config) and the Bayesian-optimization algorithms that solve them are self-contained once the discrete space C = R imes N imes S and the OOD pipeline (t-SNE + KDE + K-means) are fixed. The only load-bearing circularity is confined to the second-class problem / Experiment set 3: ground-truth is declared to be the MAE obtained at the single hand-chosen point (r,n,s)=(0,48,10) taken from Omee et al., after which BO simply searches for other points whose MAE is close to that reference. Consequently the reported success (“similar results with MatBench and former OOD research \ldots more than half of the cost are saved”) is the direct objective of the search rather than an independent empirical discovery. The cost-saving observation itself (n≤24, s=7 or 8 suffice) and the discrimination results of Experiment set 2 are not forced by this definition and constitute independent content. No self-definitional equations, no fitted parameters renamed as predictions, no load-bearing self-citations or uniqueness theorems appear. Formal mismatches between Algorithm 2/3 Ensure clauses and the surrounding formulas are correctness bugs, not circularity. Overall circularity is therefore moderate and localized (score 4).

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The central cost-and-agreement claims rest on a hand-built discrete config space, a specific OOD construction pipeline, Bayesian optimization with fixed budgets, and a ground-truth definition borrowed from or aggregated over that space—not on new physical axioms. Free parameters are mostly experimental design choices (budgets, sample sizes, reference config). Domain assumptions import MatBench tasks, MP/JARVIS data, and Omee-style sparse OOD splits. No new physical entities are postulated; AutoMatBench is a software method.

free parameters (5)
  • evaluation budget T and initial samples T0
    Set to T=12 and T0=4 for all BO runs; the headline “within twelve steps” claim is defined by this budget choice, not derived.
  • n_total=1600 and n_sample=480 OOD candidates
    Fixed sampling and sparse-screening sizes in Section 6.1 that shape every dataset before clustering; not justified as optimal.
  • discrete grids R, N, S (858 configs) and constraint s≤8 when n=60
    Hand-chosen ranges for r, n, s define the searchable space C and thus what “optimum” can mean.
  • ground-truth reference (r,n,s)=(0,48,10) in Experiment set 3
    Declared as gt from prior OOD study; Exp. 3 optimizes distance to this choice, so reported “closest to ground-truth” depends on this free reference.
  • random seed 42 and validation proportion p=0.1
    Control reproducibility and fold construction; single seed limits statistical robustness of BO paths.
axioms (5)
  • domain assumption Bayesian optimization with a surrogate and acquisition function is an appropriate sample-efficient solver for the three black-box evaluation problems over discrete (o,c) or c.
    Section 5 adopts BO without comparing optimizers or proving regret bounds for this discrete materials-benchmark landscape.
  • domain assumption OOD materials for discovery can be approximated by low-density points in a 2D t-SNE of OFM features, then K-means clusters and s nearest neighbors in original space (SparseX-style).
    Section 6.1 pipeline inherited from Omee et al.; load-bearing for all OOD claims.
  • ad hoc to paper MAE is a valid scalar performance index for all six tasks, including metallicity treated as regression.
    Section 4.1 states all tasks use MAE and metallicity is “treated similar to a regression problem,” which is nonstandard for binary classification.
  • ad hoc to paper Aggregate or reference performance over the finite config space C is a meaningful stand-in for ground-truth model ability when a true gt is undefined.
    Formulas (3)–(4) and Exp. 3; without this, “closest to gt” and average-performance ranking are not well-defined.
  • standard math Standard ML and materials-informatics background (K-means, t-SNE, KDE, DFT-derived MP/JARVIS labels, GNN/composition models) is reliable enough for comparative benchmarking.
    Used throughout Sections 2 and 6 as unproved infrastructure.
invented entities (1)
  • AutoMatBench configuration-optimization problems (first-class, dual, second-class) over C=R×N×S no independent evidence
    purpose: Formalize model ranking, max discrimination, and gt-matching as optimizable objectives for MPP benchmarking.
    Section 4 introduces these problem classes; they are definitional scaffolding rather than independently evidenced scientific objects.

pith-pipeline@v1.1.0-grok45 · 22482 in / 4206 out tokens · 43675 ms · 2026-07-14T04:58:33.660080+00:00 · methodology

0 comments
read the original abstract

Material property prediction (MPP) infers key properties from chemical composition and structure, accelerating the discovery and optimization of novel materials. In the realm of MPP, MatBench is a widely accepted benchmarking tool that defines over ten significant problems and provides the paradigm of performance evaluation for AI prediction models. Even though MatBench works well in benchmarking the performances of prediction models on in-distribution (ID) tasks and datasets, it lacks the ability to reflect their performances on out-of-distribution (OOD) material data, resulting failure in new material discovery. By combining the pipelines of MatBench and the existing researches on OOD performance evaluation, this study enables a huge space of benchmarking configurations, comprehensively reflecting the performances, abilities, and disadvantages of various AI prediction models. This work reports that the discrepancy of performances at different configuration values is huge and can be illustrated with prior knowledge and novel insights, therefore consideration of causal effect of configurations on performance results is necessary. In case of the impossibility of enumerative benchmarking at every configuration, this work further proposes AutoMatBench, an automatic toolkit with Bayesian optimization. Experiments with AutoMatBench reports that, within twelve steps of optimization, the similar results with MatBench and former OOD research can be accessed while more than half of the cost are saved. Besides, this tool also yields more essential findings on MPP benchmarking, positively contributing to the cost and efficiency of new material discovery.

Figures

Figures reproduced from arXiv: 2607.11526 by Hongxiao Li, Wanling Gao.

Figure 1
Figure 1. Figure 1: The motivation: this study provide a novel, cost￾effective, and near-reliable benchmarking method for material that solves both problems of failure in OOD evaluation and the lack of generalizability. First Author et al.: Preprint submitted to Elsevier Page 4 of 16 [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The methodology of this research. The other existing benchmarking methods do not include configuration adjustment and optimization parts. The optimization part is the essence of our method. First Author et al.: Preprint submitted to Elsevier Page 7 of 16 [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: This figure presents the framework of AutoMatBench that consists if several modules. The pipeline is represented with procedures with arrows. First Author et al.: Preprint submitted to Elsevier Page 11 of 16 [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: This figure reports the wide discrepancies among evaluated MAEs while taking different configurations across different models and tasks. Sphere color/scale: MAE. Position: config [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: This figure shows two examples [(a) and (b)] of optimization in Experiment 1. The optimal configurations are found within 12 steps. The full results are presented in the appendix. significant influence is found. For the number of folds 𝑛, every experiment outcome reports that 𝑠 ≤ 24 is enough. This eliminates at least 50% benchmarking cost compared to Omee et al.’s study [2] without reliability loss. As fo… view at source ↗
Figure 6
Figure 6. Figure 6: This figure shows two examples [(a) and (b)] of optimization in Experiment 2. The optimal configurations are found within 12 steps. The full results are presented in the appendix [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: This figure shows two examples [(a) and (b)] of optimization in Experiment 3. The optimal configurations are found within 12 steps. The full results are presented in the appendix. materials, the relative permeability is approximately 1, so the refractive index can be approximated as the square root of the relative permittivity, which describes the material’s response to an electric field. This approach may… view at source ↗
Figure 8
Figure 8. Figure 8: This figure shows the entire processesand results of optimization in Experiment 1. (Part(a)∼(f)) First Author et al.: Preprint submitted to Elsevier Page 17 of 16 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: This figure shows the entire processesand results of optimization in Experiment 1. (Part(g)∼(j)) First Author et al.: Preprint submitted to Elsevier Page 18 of 16 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: This figure shows the entire processesand results of optimization in Experiment 2. Theoptimization aim is the maximize the performancegap between models. [Part (a)∼(f)] First Author et al.: Preprint submitted to Elsevier Page 19 of 16 [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: This figure shows the entire processesand results of optimization in Experiment 3. Theoptimization aim is the minimize the performancedistance with the ground-truth results. [Part (a)∼(f)] First Author et al.: Preprint submitted to Elsevier Page 20 of 16 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: This figure shows the entire processesand results of optimization in Experiment 3. Theoptimization aim is the minimize the performancedistance with the ground-truth results. [Part (g)∼(l)] First Author et al.: Preprint submitted to Elsevier Page 21 of 16 [PITH_FULL_IMAGE:figures/full_fig_p021_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 3 linked inside Pith

  1. [1]

    Bench- markingmaterialspropertypredictionmethods:thematbenchtestset and automatminer reference algorithm

    Dunn, A., Wang, Q., Ganose, A., Dopp, D., Jain, A., 2020. Bench- markingmaterialspropertypredictionmethods:thematbenchtestset and automatminer reference algorithm. npj Computational Materials 6, 138

    2020

  2. [2]

    Structure-based out-of-distribution (ood) materials property prediction: a benchmark study

    Omee, S.S., Fu, N., Dong, R., Hu, M., Hu, J., 2024. Structure-based out-of-distribution (ood) materials property prediction: a benchmark study. npj Computational Materials 10, 144

    2024

  3. [3]

    Benchmarking graph neural networks for materials chemistry

    Fung, V., Zhang, J., Juarez, E., Sumpter, B.G., 2021. Benchmarking graph neural networks for materials chemistry. npj Computational Materials 7, 84

    2021

  4. [4]

    Known unknowns: Out-of-distribution prop- erty prediction in materials and molecules

    Segal, N., Netanyahu, A., Greenman, K.P., Agrawal, P., Gómez- Bombarelli, R., 2025. Known unknowns: Out-of-distribution prop- erty prediction in materials and molecules. npj Computational Materials 11, 345

    2025

  5. [5]

    Bench- marking gnns for ood materials property prediction with uncertainty quantification

    Tan, L., Chen, P., Liu, M., Wang, X., Cen, J., Zou, Q., 2025. Bench- marking gnns for ood materials property prediction with uncertainty quantification. arXiv preprint arXiv:2511.11697

    arXiv 2025

  6. [6]

    Out-of-distributionmaterialprop- erty prediction using adversarial learning

    Li,Q.,Miklaucic,N.,Hu,J.,2025. Out-of-distributionmaterialprop- erty prediction using adversarial learning. The Journal of Physical Chemistry C 129, 6372–6385

    2025

  7. [7]

    Commentary: The materials project: A materials genome approach to accelerating materials innovation

    Jain, A., Ong, S.P., Hautier, G., Chen, W., Richards, W.D., Dacek, S., Cholia, S., Gunter, D., Skinner, D., Ceder, G., et al., 2013. Commentary: The materials project: A materials genome approach to accelerating materials innovation. APL materials 1

    2013

  8. [8]

    The materials application programming inter- face(api):Asimple,flexibleandefficientapiformaterialsdatabased on representational state transfer (rest) principles

    Ong, S.P., Cholia, S., Jain, A., Brafman, M., Gunter, D., Ceder, G., Persson, K.A., 2015. The materials application programming inter- face(api):Asimple,flexibleandefficientapiformaterialsdatabased on representational state transfer (rest) principles. Computational Materials Science 97, 209–215

    2015

  9. [9]

    Predictingtheband gapsofinorganicsolidsbymachinelearning

    Zhuo,Y.,MansouriTehrani,A.,Brgoch,J.,2018. Predictingtheband gapsofinorganicsolidsbymachinelearning. Thejournalofphysical chemistry letters 9, 1668–1673

    2018

  10. [10]

    Scientific data 2, 150009

    De Jong, M., Chen, W., Angsten, T., Jain, A., Notestine, R., Gamst, A.,Sluiter,M.,KrishnaAnde,C.,VanDerZwaag,S.,Plata,J.J.,etal., 2015.Chartingthecompleteelasticpropertiesofinorganiccrystalline compounds. Scientific data 2, 150009

    2015

  11. [11]

    New cubic perovskites for one-andtwo-photonwatersplittingusingthecomputationalmaterials repository

    Castelli, I.E., Landis, D.D., Thygesen, K.S., Dahl, S., Chorkendorff, I., Jaramillo, T.F., Jacobsen, K.W., 2012. New cubic perovskites for one-andtwo-photonwatersplittingusingthecomputationalmaterials repository. Energy & Environmental Science 5, 9034–9043

    2012

  12. [12]

    High- throughput screening of inorganic compounds for the discovery of novel dielectric and optical materials

    Petousis,I.,Mrdjenovich,D.,Ballouz,E.,Liu,M.,Winston,D.,Chen, W., Graf, T., Schladt, T.D., Persson, K.A., Prinz, F.B., 2017. High- throughput screening of inorganic compounds for the discovery of novel dielectric and optical materials. Scientific data 4, 160134

    2017

  13. [13]

    Atomistic line graph neural net- workforimprovedmaterialspropertypredictions

    Choudhary, K., DeCost, B., 2021. Atomistic line graph neural net- workforimprovedmaterialspropertypredictions. npjComputational Materials 7, 185

    2021

  14. [14]

    Schnet: A continuous-filter convolutional neural network for modeling quantum interactions

    Schütt, K., Kindermans, P.J., Sauceda Felix, H.E., Chmiela, S., Tkatchenko, A., Müller, K.R., 2017. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. Advances in neural information processing systems 30

    2017

  15. [15]

    The Journal of chemical physics 148

    Schütt, K.T., Sauceda, H.E., Kindermans, P.J., Tkatchenko, A., Müller,K.R.,2018.Schnet–adeeplearningarchitectureformolecules and materials. The Journal of chemical physics 148

    2018

  16. [16]

    Npj Computational Materials 7, 77

    Wang,A.Y.T.,Kauwe,S.K.,Murdock,R.J.,Sparks,T.D.,2021.Com- positionally restricted attention-based network for materials property predictions. Npj Computational Materials 7, 77

    2021

  17. [17]

    The lsst-desc 3x2pt tomography optimization challenge

    Kovacs, E., 2021. The lsst-desc 3x2pt tomography optimization challenge. The Open Journal of Astrophysics

    2021

  18. [18]

    Crystal structure representations for machine learning models of formationenergies

    Faber, F., Lindmaa, A., Von Lilienfeld, O.A., Armiento, R., 2015. Crystal structure representations for machine learning models of formationenergies. InternationalJournalofQuantumChemistry115, 1094–1101

    2015

  19. [19]

    A general-purpose machine learning framework for predicting proper- ties of inorganic materials

    Ward, L., Agrawal, A., Choudhary, A., Wolverton, C., 2016. A general-purpose machine learning framework for predicting proper- ties of inorganic materials. npj Computational Materials 2, 16028

    2016

  20. [20]

    Chemistry of Materials 31, 3564–3572

    Chen,C.,Ye,W.,Zuo,Y.,Zheng,C.,Ong,S.P.,2019.Graphnetworks asauniversalmachinelearningframeworkformoleculesandcrystals. Chemistry of Materials 31, 3564–3572

    2019

  21. [21]

    Atutorialonbayesianoptimization

    Frazier,P.I.,2018. Atutorialonbayesianoptimization. arXivpreprint arXiv:1807.02811

    Pith/arXiv arXiv 2018

  22. [22]

    Generalized gradient approximation made simple

    Perdew, J.P., Burke, K., Ernzerhof, M., 1996. Generalized gradient approximation made simple. Physical review letters 77, 3865

    1996

  23. [23]

    Van der waals density functional for general geometries

    Dion, M., Rydberg, H., Schröder, E., Langreth, D.C., Lundqvist, B.I., 2004. Van der waals density functional for general geometries. First Author et al.:Preprint submitted to ElsevierPage 15 of 16 AutoMatBench: An Automatic Optimization Toolkit Physical review letters 92, 246401

    2004

  24. [24]

    Materials design and discovery with high-throughput density func- tional theory: the open quantum materials database (oqmd)

    Saal, J.E., Kirklin, S., Aykol, M., Meredig, B., Wolverton, C., 2013. Materials design and discovery with high-throughput density func- tional theory: the open quantum materials database (oqmd). Jom 65, 1501–1509

    2013

  25. [25]

    The joint automatedrepository for various integrated simulations (jarvis) for data-driven materials design

    Choudhary, K., Garrity, K.F., Reid, A.C., DeCost, B., Biacchi, A.J., Hight Walker, A.R., Trautt, Z., Hattrick-Simpers, J., Kusne, A.G., Centrone, A.,et al., 2020. The joint automatedrepository for various integrated simulations (jarvis) for data-driven materials design. npj computational materials 6, 173

    2020

  26. [26]

    Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conferenceoncomputervisionandpatternrecognition,Ieee.pp.248– 255

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L., 2009. Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conferenceoncomputervisionandpatternrecognition,Ieee.pp.248– 255

    2009

  27. [27]

    Zhan, J., Wang, L., Gao, W., Wang, C., Li, H., Fan, F., Kang, G.,

  28. [28]

    Version 1.0 (first edition, 2025) ed., BenchCouncil Press, Hong Kong

    Evaluatology: The Science of Uncovering the Effects. Version 1.0 (first edition, 2025) ed., BenchCouncil Press, Hong Kong

    2025

  29. [29]

    Backpropagation and stochastic gradient descent method

    Amari, S.i., 1993. Backpropagation and stochastic gradient descent method. Neurocomputing 5, 185–196

    1993

  30. [30]

    Adam:Amethodforstochasticoptimiza- tion

    Kingma,D.P.,Ba,J.,2014. Adam:Amethodforstochasticoptimiza- tion. arXiv preprint arXiv:1412.6980

    Pith/arXiv arXiv 2014

  31. [31]

    Saibench:Benchmarkingaiforscience

    Li,Y.,Zhan,J.,2022. Saibench:Benchmarkingaiforscience. Bench- Council Transactions on Benchmarks, Standards and Evaluations 2, 100063

    2022

  32. [32]

    Saibench: A structural interpretation of ai for science through benchmarks

    Li, Y., Zhan, J., 2023. Saibench: A structural interpretation of ai for science through benchmarks. arXiv preprint arXiv:2311.17869

    Pith/arXiv arXiv 2023

  33. [33]

    Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties

    Xie, T., Grossman, J.C., 2018. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical review letters 120, 145301

    2018

  34. [34]

    Scalabledeepergraphneuralnetworksforhigh-performance materials property prediction

    Omee,S.S.,Louis,S.Y.,Fu,N.,Wei,L.,Dey,S.,Dong,R.,Li,Q.,Hu, J.,2022. Scalabledeepergraphneuralnetworksforhigh-performance materials property prediction. Patterns 3

    2022

  35. [35]

    Materials property prediction for limited datasets enabled by feature selection and joint learning with modnet

    De Breuck, P.P., Hautier, G., Rignanese, G.M., 2021. Materials property prediction for limited datasets enabled by feature selection and joint learning with modnet. npj computational materials 7, 83

    2021

  36. [36]

    The k-means algorithm: A comprehensive survey and performance evaluation

    Ahmed, M., Seraj, R., Islam, S.M.S., 2020. The k-means algorithm: A comprehensive survey and performance evaluation. Electronics 9, 1295

    2020

  37. [37]

    Nested cross-validation when selecting classifiers is overzealous for most practical applications

    Wainer, J., Cawley, G., 2021. Nested cross-validation when selecting classifiers is overzealous for most practical applications. Expert Systems with Applications 182, 115222

    2021

  38. [38]

    Visualizing data using t-sne

    Van der Maaten, L., Hinton, G., 2008. Visualizing data using t-sne. Journal of machine learning research 9

    2008

  39. [39]

    Onestimationofaprobabilitydensityfunctionand mode

    Parzen,E.,1962. Onestimationofaprobabilitydensityfunctionand mode. The annals of mathematical statistics 33, 1065–1076

    1962

  40. [40]

    Functional form of the generalized gradient approximation for exchange: The pbe𝛼functional

    Madsen, G.K., 2007. Functional form of the generalized gradient approximation for exchange: The pbe𝛼functional. Physical Review B—Condensed Matter and Materials Physics 75, 195108

    2007

  41. [41]

    Lehrbuchderkristallphysik(teubner,leipzig,1928)

    Voigt,W.,1908. Lehrbuchderkristallphysik(teubner,leipzig,1928). There is no corresponding record for this reference

    1908

  42. [42]

    Quantum-chemical insights from deep tensor neural net- works

    Schütt,K.T.,Arbabzadah,F.,Chmiela,S.,Müller,K.R.,Tkatchenko, A., 2017. Quantum-chemical insights from deep tensor neural net- works. Nature communications 8, 13890

    2017

  43. [43]

    Journal of the Optical Society of America B 23, 498– 505

    Wangberg,R.,Elser,J.,Narimanov,E.E.,Podolskiy,V.A.,2006.Non- magneticnanocompositesforopticalandinfrarednegative-refractive- index media. Journal of the Optical Society of America B 23, 498– 505. A. Full results of Experiment set 1, 2, and 3 This section presents the complete experimental results for Experiment sets 1, 2, and 3. AsillustratedinFigure8andF...

    2006