{"id":"afa8fd3a-7148-41f6-a580-4b0e086102fd","arxiv_id":"2607.11384","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.5,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Mainstream pre-trained MLIPs show limited transferability to amorphous solids, making disordered phases a frontier challenge that needs dedicated benchmarks and fine-tuning.","lead":"Current universal machine-learned interatomic potentials, trained mostly on crystals, often fail to transfer to amorphous solids. The authors introduce a benchmark dataset and evaluation framework to measure that gap and test fine-tuning fixes.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The claim that amorphous systems form a frontier challenge for universal MLIPs rests on the unexamined representativeness of a curated reference set whose composition and metrics are not inspectable from the abstract.","rationale":"The reader’s weakest_assumption correctly isolates the representativeness of the curated amorphous set and metrics as the single load-bearing vulnerability. Because only the abstract is available, no additional technical soft spot (equation inconsistency, table error, or hidden circularity) can be identified; the concern is therefore identical. The UNVERDICTED status with low confidence remains appropriate until the concrete test above can be executed on the full materials. No adjustment to the verdict is warranted.","tokens_in":1953,"tokens_out":475,"duration_ms":14344,"concrete_test":"Once the full text, dataset, and code are available, enumerate the chemical diversity, coordination environments, and quench protocols of the reference amorphous systems and recompute the reported energy/force/RDF errors after restricting the comparison to those systems whose crystalline counterparts are already well-described by the same pre-trained models; if the amorphous-specific error inflation vanishes under that control, the frontier-challenge framing is not supported by the benchmark.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper’s central claim—that the amorphous state is a frontier challenge for universal MLIPs because current pre-trained models show transferability failures—requires that the curated “canonical” amorphous systems and the chosen structure/property metrics are representative of disordered solids more broadly. The abstract asserts a systematic evaluation that “identifies limitations” and introduces a benchmarking framework, yet supplies no inventory of systems, chemistries, preparation protocols, model list, quantitative error tables, or side-by-side crystal-versus-amorphous comparisons. Without those details it remains possible that the reported failures are artifacts of a narrow test set (e.g., only melt-quenched oxides, or systems whose crystalline counterparts already lie outside the pre-training distribution) or of metrics that penalize the absence of long-range order by construction. The generalization from “these models fail on our set” to “amorphous materials are a central challenge for the field” is therefore the least secure step in the argument.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The manuscript argues that the amorphous state is a central challenge for universal (pre-trained/foundational) machine-learned interatomic potentials (MLIPs). Early models and benchmarks have focused on ordered crystals, leaving transferability to non-crystalline solids unclear. The authors report a systematic evaluation of mainstream pre-trained models on a curated reference dataset of canonical amorphous systems, with structure and property validation; they identify transferability limitations, examine fine-tuning strategies for disordered phases, and introduce a benchmarking framework intended to guide next-generation training sets and models for amorphous functional materials.","tokens_in":2134,"tokens_out":902,"duration_ms":15103,"significance":"If the evaluation is rigorous and the reference set is representative, the work would fill a genuine gap: crystal-centric MLIP benchmarks can miss failures on disordered solids, and a reusable amorphous benchmarking framework would be of practical value to the community. Explicit investigation of fine-tuning for disordered phases is also useful. The abstract-level contribution is therefore potentially significant for materials modelling of glasses and amorphous functional materials; significance cannot be confirmed without the full methods, data inventory, and quantitative results.","major_comments":[{"comment":"Abstract (central claim): The assertion that “the amorphous state is indeed a central challenge for future universal MLIPs” is load-bearing and rests on generalization from a curated “canonical” reference set. The abstract supplies no inventory of systems, chemistries, preparation protocols (e.g., melt-quench vs. other routes), or property metrics. Without that inventory and a clear argument for representativeness of disordered solids more broadly, the step from “models fail on our set” to “amorphous materials are a frontier challenge for the field” cannot be assessed and may over-generalize from a narrow test suite.","section":null},{"comment":"Abstract (evaluation design): Transferability limitations are claimed relative to crystal-focused practice, yet the abstract does not state whether the same models were evaluated side-by-side on crystalline counterparts of the same chemistries, nor whether failures track absence of long-range order versus chemistries already outside the pre-training distribution. A crystal-versus-amorphous comparison (or an explicit statement that crystalline performance is already known and adequate) is needed for the “amorphous-specific challenge” framing to hold.","section":null},{"comment":"Abstract (results support): The abstract asserts “systematic evaluation,” “identifies limitations,” and a “benchmarking framework” without any quantitative error tables, model list, baselines, error bars, or data-exclusion criteria. Those elements are load-bearing for the central claim; the full manuscript must present them in a form that allows independent judgment of effect sizes and of whether reported failures are robust rather than metric- or protocol-specific artifacts.","section":null}],"minor_comments":[{"comment":"Abstract wording: “canonical amorphous systems” and “validation for structures and properties” are left undefined; even a brief parenthetical list of system classes (e.g., oxides, metals, semiconductors) and metric families would improve clarity for readers scanning the abstract.","section":null},{"comment":"Abstract: “mainstream models” should be named or at least scoped (architecture families / public foundational MLIPs) so the claimed coverage is checkable.","section":null},{"comment":"Abstract: Fine-tuning strategies are mentioned as investigated; a one-phrase indication of what was varied (data composition, loss terms, freeze/unfreeze policy) would help readers judge novelty relative to existing fine-tuning practice.","section":null}],"recommendation":"uncertain","confidential_remarks":"Only the abstract was available for this review (full text not provided). I therefore cannot verify methods, tables, or the representativeness of the reference set; recommendation is uncertain pending the full manuscript. The stress-test concern about unexamined representativeness of the curated set is the main load-bearing risk and should be checked first when the full text is in hand. If the full paper includes a broad multi-chemistry inventory, crystal-vs-amorphous controls, and quantitative tables, the contribution could move to minor_revision or accept; if the set is narrow or metrics penalize disorder by construction, major_revision or reject may be warranted."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a methods/benchmarking paper claiming that mainstream pre-trained MLIPs, built largely on ordered crystals, transfer poorly to amorphous solids, and that the field needs a dedicated amorphous reference set plus fine-tuning guidance. From the abstract that claim is plausible and useful; we cannot yet score the evidence.\n\nWhat is actually new is the package: a curated “canonical” amorphous reference dataset, a structure/property validation framework, a systematic sweep of current foundational models, and a look at fine-tuning for disordered phases. Crystal-centric benchmarks have been the default. Calling out the amorphous gap and shipping a reference set is the right kind of contribution if the data and protocols hold up. The design is structurally clean—external evaluation of pre-trained models on held-out amorphous references—so circularity is low by construction. The problem statement is also clean: universal MLIPs are sold as general-purpose; checking them on disordered systems is an obvious next step many groups have only done ad hoc.\n\nSoft spots, in proportion: we only have the abstract. The stress-test concern lands at this stage. The jump from “models fail on our curated set” to “the amorphous state is a frontier challenge for universal MLIPs” depends on how representative the systems, chemistries, preparation routes, and metrics are. If the set is narrow, or if metrics punish missing long-range order by construction, the headline overreaches. Soundness, baselines, error bars, model list, and data release are uncheckable without the full text. That is a limitation of this read, not a proven flaw in the paper.\n\nWho it is for: people building or using foundational MLIPs, and groups working on amorphous functional materials. It deserves a serious referee rather than a desk reject—the question is important enough and the design is non-circular. I would not bring it to a general reading group until the dataset and tables are inspectable; for an MLIP-validation group, yes. Send to peer review; insist on full inventory of systems, quantitative crystal-versus-amorphous comparisons, and open data.","headline":"Timely benchmarking pitch that amorphous solids expose transferability gaps in crystal-trained universal MLIPs—but from the abstract alone we cannot yet judge how representative the curated set is.","tokens_in":2741,"tokens_out":534,"would_cite":false,"duration_ms":13862,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Current universal machine-learned potentials struggle with amorphous solids, a frontier that crystal-focused benchmarks miss.","keywords":["machine-learned interatomic potentials","amorphous materials","transferability","foundational models","benchmarking","fine-tuning","disordered solids"],"falsifier":"A mainstream pre-trained MLIP that, without specialized fine-tuning, reproduces reference structures and key properties (for example radial distribution functions, densities, or mechanical/thermal observables) for the paper's canonical amorphous systems at the same accuracy level it achieves on crystalline benchmarks.","tokens_in":2800,"feed_emoji":"⚗️","tokens_out":494,"duration_ms":3912,"temperature":0.7,"pith_summary":"Pre-trained machine-learned interatomic potentials (MLIPs) have become standard tools for materials modelling, yet most early models and their benchmarks were built around ordered crystals. This paper argues that the amorphous state is a distinct, central challenge for the next generation of universal MLIPs. The authors evaluate mainstream pre-trained models on a curated reference set of canonical amorphous systems, checking both structures and properties, and find clear transferability limits that crystal-only tests do not reveal. They also examine fine-tuning strategies aimed at disordered phases. If the claim holds, researchers modelling glasses, amorphous semiconductors, and other disordered functional materials will need more carefully designed training data and validation protocols rather than assuming crystal-trained models will simply transfer.","feed_headline":"Universal atomistic models falter on amorphous solids","feed_subtitle":"Crystal-trained MLIPs miss transferability gaps that disordered materials expose","key_machinery":"A benchmarking framework built on a curated reference dataset of canonical amorphous systems, together with structure and property validation metrics, used to expose transferability gaps and to test fine-tuning strategies for disordered phases.","core_discovery":"Systematic evaluation of current mainstream pre-trained MLIPs on a curated set of canonical amorphous systems shows that many models fail to transfer reliably to non-crystalline solids; crystal-focused benchmarks therefore do not capture the performance limits that matter for disordered materials.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Crystal-trained MLIPs fail on amorphous solids","Universal MLIPs falter with disordered materials","Amorphous systems expose MLIP transferability limits","Pre-trained models miss gaps in non-crystalline solids","Foundational MLIPs struggle beyond crystalline order"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The chosen set of canonical amorphous systems and the structure/property metrics used for validation are representative enough of the broader space of disordered solids that the observed failures can be treated as a general frontier challenge rather than artifacts of a narrow test set.","fun_headline_variants_meta":{"raw":{"variants":["Crystal-trained MLIPs fail on amorphous solids","Universal MLIPs falter with disordered materials","Amorphous systems expose MLIP transferability limits","Pre-trained models miss gaps in non-crystalline solids","Foundational MLIPs struggle beyond crystalline order"]},"model":"grok-4.5","effort":"low","cost_usd":0.003204,"raw_usage":{"total_tokens":1044,"prompt_tokens":669,"num_sources_used":0,"completion_tokens":57,"cost_in_usd_ticks":32040000,"prompt_tokens_details":{"text_tokens":669,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":318,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":669,"tokens_out":57,"duration_ms":2965,"temperature":1.0,"reasoning_tokens":318,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T01:27:17.949773+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A mainstream pre-trained MLIP that, without specialized fine-tuning, reproduces reference structures and key properties (for example radial distribution functions, densities, or mechanical/thermal observables) for the paper's canonical amorphous systems at the same accuracy level it achieves on crystalline benchmarks.","supporting_citations":[],"review_version":1}