{"id":"ac078535-18b7-4c07-a4a1-6cfe95ca3c9a","arxiv_id":"2411.14034","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Machine learning trained on experimental data can identify known transparent conducting materials from chemical formula alone, but its discoveries are compositionally similar to its training set.","lead":"Researchers built two experimental databases, one for electrical conductivity and one for band gap, and tested whether machine learning can predict both properties from a material's chemical formula alone. They found the best model could recognize most known transparent conducting materials when held out, and could flag a few new candidate materials for testing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The family-discovery-rate is computed on only 10 families with an at-least-one-above-threshold success rule; without uncertainty intervals or a trivial baseline, the claimed CrabNet superiority is not yet established.","rationale":"The paper is a serious empirical study with an unusually honest limitations section, and the reader's CONDITIONAL verdict is appropriate. The reader identified the stoichiometry-only representation as the weakest assumption, and the GaInZnO4 failure is a real and well-documented limitation. However, the single most load-bearing concern for the paper's headline claim is the evaluation metric itself. The FDR comparison is the only quantitative evidence for the statement that CrabNet shows superior robustness in identifying novel stoichiometric combinations. That evidence is fragile: with 10 families, the difference between 90% and 40% is not accompanied by any uncertainty estimate, and the success criterion is deliberately lenient, requiring only one representative above thresholds. The paper also does not test a trivial composition-only baseline, which matters because the model's attention analysis shows it focuses on dopant elements. A baseline that simply labels doped oxides as TCMs would likely capture much of the same signal. I agree with the reader that the datasets and code should be released and that the 55-composition screen lacks experimental validation, but those concerns strengthen the CONDITIONAL verdict without changing it. The concrete test proposed here would settle whether the central FDR claim is robust or an artifact of evaluation design. If the test fails, the paper would need to be revised to temper the discovery claim; if it passes, the current CONDITIONAL status could potentially be upgraded. For now, the verdict remains CONDITIONAL, matching the reader's assessment.","tokens_in":25135,"tokens_out":4284,"duration_ms":47285,"concrete_test":"Re-run the leave-one-TCM-family-out evaluation with: (a) a per-family all-representatives success rule (e.g., both median predicted conductivity and median predicted band gap must exceed thresholds); (b) a threshold sweep with sigma in {10^1, 10^2, 10^3, 10^4} S/cm and Eg in {2.5, 3.0, 3.5} eV; (c) bootstrap 95% confidence intervals across the 10 families; and (d) a dummy baseline that labels every oxide containing Al, Ga, Sn, or In as a TCM. If CrabNet's FDR advantage over RF disappears under (a)-(c) or the dummy baseline matches its FDR, then Section 5.3 does not support the claimed superior robustness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The manuscript's central quantitative support for CrabNet's discovery ability is the leave-one-TCM-family-out FDR in Section 5.3: 90% (9/10 families) for CrabNet versus 40% (4/10) for random forest. This metric is load-bearing because the abstract and conclusions generalize from it to claims of superior robustness in identifying novel stoichiometric combinations. Three unaddressed weaknesses undermine it. First, N=10 families is very small: a single family changes FDR by 10 percentage points, and no confidence interval, bootstrap, or significance test is reported for the 90%-versus-40% gap. Second, success is defined as at least one representative of a held-out family exceeding thresholds of 10^2 S/cm and 3 eV, rather than as accurate property prediction. Because all test families are established TCMs with measured conductivities around 10^2-10^4 S/cm and band gaps around 3.6-4.2 eV, a model that systematically overpredicts conductivity and band gap for doped oxides could achieve a high FDR without genuine extrapolation. Third, no trivial baseline is compared, such as classifying every oxide containing a known dopant element (Al, Ga, Sn, In) as a TCM. If such a baseline also achieves a high FDR, the claimed advantage of CrabNet is not evidence of learned chemistry but of the evaluation rule. This concern is distinct from the acknowledged stoichiometry-only limitation: it questions whether the headline metric itself supports the paper's strongest claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a data-driven framework for accelerating the discovery of transparent conducting materials (TCMs). The authors create two curated experimental datasets for room-temperature electrical conductivity and band gap, train random forest (RF) and CrabNet models on stoichiometry-only inputs, and evaluate them with K-fold cross-validation, LOCO-CV, and a bespoke leave-one-TCM-family-out scheme. In the latter, they report a family-discovery-rate (FDR) of 90% for CrabNet versus 40% for RF across 10 known TCM families. They also screen 55 database compositions containing common TCM elements, rank them with a risk-adjusted figure of merit, and highlight several candidate materials. The central claims are that CrabNet is more robust than RF at identifying previously unseen stoichiometric combinations and that the framework can accelerate the identification of promising TCM candidates.","tokens_in":25446,"tokens_out":5247,"duration_ms":51518,"significance":"If the claims are supported, the paper would make a useful contribution by releasing curated experimental datasets and proposing a dedicated evaluation scheme for discovery-oriented ML in materials science. The attention-score analysis offers an interpretability angle that is rare in this literature. However, the headline result rests on a small-sample, threshold-based family-discovery-rate that is not accompanied by uncertainty quantification or comparison to trivial baselines, and the practical discovery section lacks a quantitative evaluation of the ranking quality. The contributions are valuable but the main quantitative claim needs strengthening before the paper can be fully credited.","major_comments":[{"comment":"The family-discovery-rate is computed on only 10 families, and the reported 90% vs. 40% difference is not accompanied by any uncertainty estimate or statistical test. With N=10, a single family changes the FDR by 10 percentage points; the absence of confidence intervals, bootstrap, or permutation testing means the claimed superiority of CrabNet over RF is not established at the reported precision. Please report per-family predictions, provide a bootstrap confidence interval, and ideally a permutation test over family labels.","section":"§5.3, Eq. (2), Table 1, Figure 6"},{"comment":"The success criterion is threshold crossing for at least one representative, with thresholds (σ>10^2 S/cm, Eg>3 eV) chosen 'drawing insights from the statistics of reported TCMs' — i.e., calibrated to the same families in Table 1 that form the test set. All ten held-out families have mean conductivities at or above ~10^2 S/cm and band gaps above 3.6 eV, so a model that systematically overpredicts conductivity and band gap for doped oxides could achieve a high FDR without genuine extrapolation. Also, the 'at least one representative' rule introduces a multiple-comparison effect: families with more representatives (up to 6 in Table 1) have more chances to exceed the threshold. The paper should report the full distribution of predictions per family, compare with trivial baselines (e.g., predicting every oxide containing Al, Ga, In, or Sn as a TCM, or a constant predictor that always exceeds both thresholds), and state how many representatives in each family exceed the thresholds, not just a binary success.","section":"§4.2, §5.3"},{"comment":"The 55-composition search is presented as evidence that the framework can highlight previously overlooked TCM candidates, but only CrabNet is applied, no baseline or quantitative evaluation of the ranking is given, and the two illustrative cases are not validated experimentally. Moreover, one of the examples discussed (GaInZnO4, Table 4 entry 51) is a known TCM whose predicted conductivity (-6.6 log10 S/cm) is over nine orders of magnitude below the measured value (2.7 log10 S/cm); the paper explains this by anti-site defects, but this failure is not factored into the assessment of the discovery pipeline. The abstract's claim that the framework 'empirically demonstrates that it can highlight material candidates that may have been previously overlooked' needs a more systematic evaluation, e.g., precision at top-k against known TCMs or comparison with random ranking.","section":"§6, §8"}],"minor_comments":[{"comment":"There is a typo in the sentence 'to accomodate φTCM within a data-driven perspective' — 'accomodate' should be 'accommodate'.","section":"§3"},{"comment":"The text refers to 'scikit learn' as 'sci-kit learn'; the correct spelling is 'scikit-learn'.","section":"§9"},{"comment":"The definition of FDR would benefit from clearer notation; the symbol N*_f is difficult to read and the equation formatting is awkward. Please define each variable explicitly.","section":"§4.2, Eq. (2)"},{"comment":"Entry 26 lists '5.038' for the ΦM value, which appears to be a typo; also consider aligning the decimal points in the numeric columns for readability.","section":"Table 4"},{"comment":"The curated experimental datasets are not made directly available; the authors only link to the original sources. Providing the curated datasets as supplementary data would greatly improve reproducibility.","section":"§10"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful contribution in its curated datasets and evaluation idea, but the headline FDR claim needs substantial strengthening with uncertainty quantification and baselines before it can be considered a reliable finding. I would encourage the editor to invite a revision that addresses the small-sample statistics and the circularity concern around the threshold choice."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the TCM screening preprint. The headline number—CrabNet reaching a 90% family-discovery-rate versus 40% for random forest—is softer than it sounds. The paper does something genuinely useful: it curates and validates an experimental conductivity dataset and a band gap dataset, and it introduces a leave-one-TCM-family-out evaluation to ask whether composition-only models can rediscover known TCM families. That evaluation design is the most reusable idea here. The data curation is described in detail, and the limitations section is honestly written—they explicitly walk through GaInZnO4, where the model predicts about 1e-7 S/cm against a measured ~500 S/cm because the real conductivity comes from anti-site defects that stoichiometry alone cannot see.\n\nNow the soft spots. The FDR is computed on just 10 families, with success defined as at least one representative above 10^2 S/cm and 3 eV. One family shifts the rate by 10 percentage points, and no confidence intervals, bootstrap, or significance test are reported. The thresholds themselves are derived from the statistics of the exact families being tested, so the metric partly rewards models that reproduce the training family statistics rather than genuinely extrapolating. I also didn't see a trivial baseline—for example, classifying every oxide containing Al, Ga, Sn, or In as a TCM. The stress-test note is right: without that baseline, the 90%-versus-40% gap isn't yet evidence of learned chemistry. That said, the paper's own conclusion is more modest than its abstract suggests: it says ML tends to find TCMs compositionally similar to the training set, and it acknowledges the mechanism limitation. So the load-bearing claim is the 'superior robustness' sentence in Section 5.3, not the whole paper.\n\nThe bigger practical issue is reproducibility. The curated datasets are the main contribution, but the data availability statement points only to raw sources; the processed, validated datasets and the code are not released. The 55-composition screen has no experimental follow-up, so it reads as a ranking exercise.\n\nWho is this for? Materials informatics researchers and the TCM community. The evaluation scheme and the data curation deserve serious peer review, and a good referee can push for the missing pieces. I'd send it to review, but I wouldn't cite it for the discovery claim until the data, a baseline, and uncertainty bounds are out.","headline":"A useful evaluation scheme and honest data curation, but the headline discovery metric is too fragile to support the paper's strongest claim on its own.","tokens_in":26033,"tokens_out":3594,"would_cite":false,"duration_ms":34779,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A composition-only ML model recovers 90% of held-out transparent-conductor families.","keywords":["transparent conducting materials","machine learning","band gap prediction","electrical conductivity prediction","composition-only models","CrabNet","materials discovery","leave-one-family-out evaluation"],"falsifier":"An independent test: compile a fresh set of, say, twenty TCM families from recent experimental reports, run the same leave-one-family-out protocol, and count how many families are recovered; a discovery rate well below 90% would show the reported result is tied to the nine chosen families. A second check: model GaInZnO$_4$ with the same stoichiometry-only input; the paper predicts roughly $10^{-7}$ S/cm against a measured value near 500 S/cm, so any claim that the pipeline finds mechanism-diverse TCMs can be refuted by a single such counterexample.","tokens_in":24943,"feed_emoji":"⚡","tokens_out":6454,"duration_ms":57307,"temperature":0.7,"pith_summary":"This paper asks whether machine learning can point to new transparent conducting materials when it is given only a chemical formula and experimental values of band gap and conductivity. To test that, the authors built two curated experimental datasets and a bespoke evaluation: hold out a whole known TCM family, train on everything else, and count whether the model still marks a representative as a TCM (conductivity above $10^2$ S/cm and band gap above 3 eV). The central result is that CrabNet, a transformer-style model, reaches a 90% family-discovery-rate on this test while a random forest reaches 40%. On a search list of 55 compositions, CrabNet ranks promising candidates, several previously overlooked as transparent conductors. The authors are explicit that the approach finds TCMs compositionally similar to the training data, not materials whose conductivity arises from defect or doping mechanisms invisible to the formula, such as GaInZnO$_4$.","feed_headline":"ML model spots 90% of held-out transparent conductor families","feed_subtitle":"Trained only on chemical formulas, CrabNet beats a random forest and flags previously overlooked candidates.","key_machinery":"The load-bearing machinery is CrabNet, a transformer-style neural network that treats the elements of a chemical formula as tokens and uses self-attention to build a composition representation; the paper finds that attention assigned to dopant-host and dopant-dopant interactions carries the conductivity signal. Around it sit three supports: two curated experimental datasets (6,592 conductivity entries and 4,767 band-gap entries), a transfer-learning step that pretrains CrabNet on a large DFT band-gap database before fine-tuning on the experimental gaps, and a bespoke evaluation protocol, leave-one-TCM-family-out, with a family-discovery-rate metric defined as the percentage of held-out TCM families whose at least one representative is correctly predicted above the $10^2$ S/cm and 3 eV thresholds.","core_discovery":"The paper's central claim, stated on its own terms, is that a machine-learning model trained on stoichiometry alone can identify most known TCM families even when each family is absent from the training set, and can usefully rank candidate materials from a database search. CrabNet achieves a family-discovery-rate of 90% under leave-one-TCM-family-out evaluation, compared with 40% for a random forest, and its attention scores concentrate on dopant-related interactions. On a list of 55 oxide compositions built from elements common in known TCMs, the model ranks doped binary oxides and several three-cation phases at the top of a risk-adjusted figure of merit, including at least one composition, Al$_{0.67}$Ga$_{1.33}$Zn$_{37}$O$_{40}$, whose band gap and conductivity had not been reported. The authors interpret this as evidence that ML can accelerate the identification of previously overlooked stoichiometric combinations, while explicitly limiting the claim to materials that resemble the training distribution in composition.","pith_inferences":["Editorial: the 90% versus 40% FDR is measured on only nine families, and all share the same handful of cations (Zn, Ga, Sn, Al, In); a wider family set could move the number substantially.","If the attention-dopant pattern is causal, a model trained with explicit structural or doping-site information should close the GaInZnO$_4$ gap; this is a testable extension the paper itself points toward.","The ranking table singles out Al$_{0.67}$Ga$_{1.33}$Zn$_{37}$O$_{40}$ as an unmeasured candidate; experimental measurement of its conductivity and band gap would be a direct check of the screening pipeline.","The deeper lesson is that this kind of ML discovery is interpolation in composition space, which is valuable for triage but should not be read as mechanistic understanding of why a material conducts."],"forward_implications":["CrabNet is a practical screening tool: feed it a candidate list of oxide formulas and it ranks them by a risk-adjusted figure of merit that trades predicted band gap against predicted conductivity.","The leave-one-family-out protocol gives researchers a sharper way to measure extrapolation than K-fold or LOCO-CV for materials-discovery claims.","The attention weights can be read as an indication of which chemical interactions the model relies on, here dopant contributions, giving a partial window into why a composition is predicted conductive.","Success on the 55-composition search is expected mainly when candidates are compositionally similar to known TCMs; mechanism-diverse candidates, like GaInZnO$_4$ with its defect-driven conductivity, will be missed.","The curated experimental datasets themselves are a contribution that lets other models be evaluated on real measured values rather than DFT-computed gaps and conductivities."],"supporting_citations":[{"why":"CrabNet architecture; the model whose predictions carry the paper's main result.","marker":"[58]"},{"why":"Source of the published experimental band-gap dataset that the authors curate and extend.","marker":"[22]"},{"why":"Experimental thermoelectric dataset supplementing the conductivity database.","marker":"[21]"},{"why":"Main source of conductivity entries and of the 55-composition search list.","marker":"[4]"},{"why":"Large DFT band-gap database used for transfer-learning pretraining of CrabNet.","marker":"[3]"},{"why":"Evaluation conventions and preprocessing for composition-only property prediction benchmarks.","marker":"[39]"},{"why":"Leave-one-cluster-out evaluation that motivates the bespoke leave-one-TCM-family-out protocol.","marker":"[67]"},{"why":"Attribution of GaInZnO4's conductivity to Ga anti-site defects, explaining why a stoichiometry-only model fails on it.","marker":"[89]"}],"fun_headline_variants":["ML trained on formulas alone spots 90% of hidden transparent conductor families","CrabNet finds 90% of unseen transparent conductor families from formulas alone","Stoichiometry-only ML finds 90% of unseen transparent conductor families","Data-driven model recovers 90% of transparent conductor families left out of training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that band gap and electrical conductivity can be predicted with useful accuracy from the bare chemical formula alone, because no crystal structure, dopant site, defect chemistry, or doping mechanism is provided to the model.","fun_headline_variants_meta":{"raw":{"variants":["ML trained on formulas alone spots 90% of hidden transparent conductor families","CrabNet finds 90% of unseen transparent conductor families from formulas alone","Stoichiometry-only ML finds 90% of unseen transparent conductor families","Data-driven model recovers 90% of transparent conductor families left out of training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001299,"raw_usage":{"total_tokens":5327,"prompt_tokens":997,"completion_tokens":4330,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":4247}},"tokens_in":613,"tokens_out":4330,"duration_ms":26490,"temperature":1.0,"reasoning_tokens":4247,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:35:47.022661+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent test: compile a fresh set of, say, twenty TCM families from recent experimental reports, run the same leave-one-family-out protocol, and count how many families are recovered; a discovery rate well below 90% would show the reported result is tied to the nine chosen families. A second check: model GaInZnO$_4$ with the same stoichiometry-only input; the paper predicts roughly $10^{-7}$ S/cm against a measured value near 500 S/cm, so any claim that the pipeline finds mechanism-diverse TCMs can be refuted by a single such counterexample.","supporting_citations":[{"cited_title":"Gaultois, Taylor D","cited_arxiv_id":null,"evidence_quote":"Experimental thermoelectric dataset supplementing the conductivity database."},{"cited_title":"The PAULING FILE Project and Materials Platform for Data Science: From Big Data Toward Materials Genome, pages 1–26","cited_arxiv_id":null,"evidence_quote":"Main source of conductivity entries and of the 55-composition search list."},{"cited_title":"Carrier generation in multicomponent wide-bandgap oxides: InGaZnO 4","cited_arxiv_id":null,"evidence_quote":"Attribution of GaInZnO4's conductivity to Ga anti-site defects, explaining why a stoichiometry-only model fails on it."}],"review_version":1}