{"id":"1903e79d-7646-44b5-b684-06f75a97fdfa","arxiv_id":"2506.05616","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"MAPPS combines LLM workflow planning, code generation, and human intuition with machine-learned force fields to discover crystal structures, reporting high stability and novelty rates on MP-20 and Matbench.","lead":"MAPPS is a multi-agent AI system for discovering crystal structures, where a language model plans multi-step workflows, writes code to run physics-based tools, and takes guidance from human scientists. The authors report a five-fold improvement in stable, unique, and novel materials over prior generative models, but the comparison gives MAPPS access to the dataset the baselines are only trained on.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing retrieval control: MAPPS retrieves from MP-20 while baselines only train on it, so high stability and S.U.N. rates may inherit from the database; a retrieval-augmented baseline is needed before the claimed 5x improvement can be attributed to planning/physics.","rationale":"After reading the paper in good faith, I find the central empirical claim is the 5x S.U.N./stability improvement on MP-20. For that claim to stand, the comparison must distinguish a retrieval-augmented system from generative models that lack retrieval. The paper explicitly uses MP-20 as MAPPS's retrieval database and as the baselines' training set (Sec. 4.1), so the comparison mixes a database-lookup/relaxation advantage with generation ability. The absence of any retrieval-augmented baseline is the load-bearing gap. I agree with the reader's weakest_assumption and keep the REJECT verdict: the evidence as presented does not support attributing the improvement to planning or physics. Strengths: Section 4.4 honestly reports that LLMs fail without human guidance, and the limitation paragraph is candid about Level 2 vs Level 3; those are useful. But they do not repair the uncontrolled comparison. A simple retrieval-augmented control would settle the matter; if it does not close the gap, a revised paper could support the claim.","tokens_in":14247,"tokens_out":7920,"duration_ms":87928,"concrete_test":"Run a retrieval-augmented baseline for Table 1: give FlowLLM (or CrystalTextLLM) access to the same MP-20 database that MAPPS uses, allowing it to retrieve prototypes and then apply the same MLFF relaxation and DFT evaluation budget as MAPPS; compute S.U.N. and DFT stability rates. If this control reaches values comparable to MAPPS (e.g., S.U.N. near 20-25%), the claimed five-fold improvement is mostly retrieval access rather than MAPPS's planning and physics, and the verdict should be conditional or reject unless the claim is rescaled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states that MP-20 is 'used as the retrieval database of our method and the training set of the baselines.' MAPPS can therefore start from known stable or near-hull Materials Project structures, relax them with ML force fields (CHGNet/M3GNet), and submit them to DFT evaluation. Baselines such as CDVAE, DiffCSP, FlowMM, and FlowLLM are trained on MP-20 but cannot retrieve from it at inference time. The headline numbers (34.3% DFT stability, 24.9% S.U.N. vs. FlowLLM's 17.8%/4.92%) can be substantially explained by selection from a database of stable/near-stable prototypes rather than by workflow planning. Novelty is computed against the same original dataset (Appendix C), so exact copies are excluded, but a retrieved prototype that is modified or recombined can count as novel while inheriting thermodynamic stability. Section 4.2 even states that MAPPS is given training data 'to ensure a fair comparison' for CSP, but Table 1 has no retrieval-augmented baseline. Without that control, the five-fold claim over generative models is not established as an effect of MAPPS's planning, physics, or autonomy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MAPPS, a multi-agent LLM framework for materials discovery that combines a Workflow Planner, a Tool Code Generator, and a Scientific Mediator, with human scientists providing intuition and feedback. The system is designed to achieve Level 2 autonomy: the agent proposes workflows while humans provide high-level goals, domain heuristics, and step-by-step approval. Experiments are reported on crystal structure generation (MP-20, Matbench), crystal structure prediction (MP-20, MPTS-52, and a challenge set), property-guided generation (bandgap targets), and a workflow-validity study. The headline claim is a five-fold improvement in stability, uniqueness, and novelty rates over prior generative models on MP-20.","tokens_in":14475,"tokens_out":5379,"duration_ms":53757,"significance":"If the evaluation were sound, the paper would make a useful contribution by showing that LLM-driven workflow planning, combined with ML force fields and human guidance, can produce high-quality crystal candidates and competitive structure-prediction results without training a purpose-built generative model. The autonomy-level taxonomy is a helpful framing, and the inclusion of DFT-based stability evaluation and a workflow-validity analysis is a strength. However, the central quantitative claims are not currently supported because the evaluation design confounds the effect of retrieval from the MP-20 database with the effect of planning and physics; moreover, the primary LLM is not identified, and no error bars or retrieval-augmented baselines are provided. The framework is promising, but the evidence as presented is not sufficient to establish the claimed advantages.","major_comments":[{"comment":"The claim of a 'five-fold improvement in stability, uniqueness, and novelty rates' is not supported by the data in Table 1. The DFT stability rate of MAPPS is 34.3% versus 17.8% for FlowLLM, which is a factor of approximately 1.93, not five. The only metric that is roughly five-fold higher is the composite S.U.N. rate (24.9% vs 4.92%). Please report the per-component uniqueness and novelty rates separately, and rephrase the claim so that it refers specifically to the composite S.U.N. rate or to the individual components that actually show such improvement.","section":"Abstract, Section 4.1, Table 1"},{"comment":"The comparison with generative baselines is confounded by retrieval. The text states that MP-20 is 'used as the retrieval database of our method and the training set of the baselines.' Because MP-20 contains only stable or near-stable materials, MAPPS can start from known stable prototypes, relax them with CHGNet/M3GNet, and submit them to DFT, while CDVAE, DiffCSP, FlowMM, and FlowLLM must generate structures from learned distributions without retrieval. A retrieval-augmented baseline that randomly samples structures from the same database and applies the identical MLFF relaxation and DFT evaluation pipeline is needed to control for the contribution of the database itself. Without such a control, the reported S.U.N. rate and the headline improvement cannot be attributed to MAPPS's planning, physics, or autonomy.","section":"Section 4.1"},{"comment":"The same retrieval confound affects the crystal structure prediction results. The sentence 'To ensure a fair comparison, we also provide our agents with access to the corresponding training data for retrieval' indicates that only MAPPS (and not the baselines) is allowed to retrieve candidate prototypes during inference. For CSP, this means MAPPS can retrieve a structurally similar prototype for a given composition and then substitute atoms, which is a substantially easier task than generating a structure from scratch. The match-rate improvements (63.9% vs 61.3% on MP-20, 27.6% vs 23.1% on MPTS-52, and 31.0% vs 22.4% on the challenge set) may therefore reflect the retrieval mechanism rather than workflow planning. A baseline that retrieves a random or similarity-based prototype from the training data and proceeds with the same relaxation pipeline should be included.","section":"Section 4.2"},{"comment":"The main experimental results do not identify which LLM or reasoning model is used for the Workflow Planner and Tool Code Generator. The text says only 'We conduct our experiments using OpenAI API and a single NVIDIA A100 GPU.' Given that Section 4.4 shows that workflow validity varies from 0% to 100% depending on the model (GPT-4o-mini, GPT-4o, O3-mini), the Tables 1-4 results cannot be reproduced or assessed without naming the exact model and version, along with sampling parameters. Additionally, no error bars, standard deviations, or multi-seed runs are reported for any of the main metrics, so it is unclear whether the reported differences are statistically meaningful.","section":"Section 4"},{"comment":"The definition of novelty is partially circular with the retrieval setup. Appendix C states that a crystal is novel if it does not match any existing structure in the original dataset, and the same dataset is used as MAPPS's retrieval source. A structure retrieved from MP-20 and then relaxed or slightly modified can be counted as novel according to the StructureMatcher criterion while retaining the thermodynamic stability of the original prototype. This inflates the S.U.N. rate in a way that cannot be separated from retrieval. Please either compute novelty against a held-out set (for example, a newer release of the Materials Project or a temporally later test set) or provide an analysis that shows how many of the 'novel' structures originate from retrieved prototypes and how much perturbation was applied.","section":"Appendix C"}],"minor_comments":[{"comment":"The heading contains a typo: 'Planing' should be 'Planning'.","section":"Section 2.2"},{"comment":"References [30] and [32] are duplicates; both cite the same Pymatgen paper. Please remove the duplicate and renumber.","section":"References"},{"comment":"The row for CrystalTextLLM has a missing stability value and the column header for the M3GNet metric is split across lines; please format the table so all entries are complete and aligned.","section":"Table 1"},{"comment":"For the bandgap-constrained generation results, please clarify how many independent runs or random seeds were used to obtain the 74.6% and 92.2% satisfaction rates, and report confidence intervals or standard errors.","section":"Section 4.3"},{"comment":"Please add axis labels and a caption note indicating whether the histograms are over the full 500 generated structures or only the valid/unique subset, and state the bin width used.","section":"Appendix B, Figure 4"},{"comment":"The limitations paragraph acknowledges that MAPPS does not reach Level 3 autonomy and focuses only on materials tasks; this is appropriate, but the abstract and introduction should more consistently refer to 'Level 2 autonomy' rather than 'greater autonomy' in a way that might overstate the result.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a potentially interesting framework, but the main quantitative evaluation currently conflates retrieval with planning and does not provide the experimental details needed to assess reproducibility. I recommend the editor send the manuscript back for major revision with a request for a retrieval-augmented baseline, per-component metrics, identification of the LLM, and error bars. The work may fit better in a specialized AI-for-materials venue, but the core ideas are worth pursuing after these issues are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper builds a coherent Level-2 autonomy agent for materials discovery: a Workflow Planner, a Tool Code Generator, and a Scientific Mediator, with human intuition injected as a structured prompt. The Level 1/2/3 autonomy taxonomy is useful, and the Section 4.4 study is the most honest part — it shows that all tested LLMs produce zero valid workflows without human guidance, which is a real and important negative result. The agent architecture is a reasonable integration of existing pieces (LLM planning, CHGNet/M3GNet relaxation, retrieval), and the authors clearly state their limitations up front. Credit is due for that.\n\nThe soft spots are serious and concentrated in the evaluation. The headline \"five-fold improvement\" is not supported by Table 1 as stated: the five-fold gain applies only to the S.U.N. rate (24.9% vs 4.92%), while DFT stability is 34.3% vs 17.8%, roughly a two-fold gain. More importantly, the comparison is confounded: MP-20 is used as MAPPS's retrieval database and as the baselines' training set, so MAPPS can start from known stable prototypes, relax them, and then be judged for novelty against the same database. No retrieval-augmented baseline is provided, so the improvement cannot be attributed to planning or physics. The abstract overstates the result, and the specific LLM used for the headline numbers is not named — Section 4.4 mentions O3-mini, but it is unclear if that is the model behind Tables 1–4. There are also no error bars, which matters for a stochastic LLM pipeline.\n\nThese flaws are load-bearing for the central claim, not minor. That said, the framework itself is not invalid; it is a plausible design for a semi-autonomous discovery assistant, and the authors admit that human intuition is essential — which is exactly what the Level-3 study shows. The paper is worth a serious referee, but only for major revision: add a retrieval-augmented baseline, fix the five-fold claim, name the models, and report variance. If the authors can show that MAPPS beats a retriever-plus-relaxation baseline, the S.U.N. results become credible. Without that control, the empirical contribution is suggestive but not established.","headline":"A well-structured Level-2 agent framework undercut by an uncontrolled retrieval advantage; the five-fold improvement claim does not survive scrutiny, but the autonomy taxonomy and the Level-3 failure study are worth engaging.","tokens_in":704,"tokens_out":1618,"would_cite":false,"duration_ms":29004,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MAPPS is a multi-agent system that plans its own materials-discovery workflows from high-level goals and human intuition, and the paper claims it raises the rate of stable, unique, novel crystals on MP-20 roughly five-fold over prior…","keywords":["large language model agents","materials discovery","crystal structure generation","crystal structure prediction","property-guided generation","machine learning force fields","workflow planning","human-in-the-loop"],"falsifier":"Run the strongest baseline (e.g., FlowLLM) with the same retrieval database, ML-force-field relaxation, and novelty filtering that MAPPS uses; if its S.U.N. rate approaches 24.9%, the claimed five-fold improvement is mostly retrieval and relaxation, not planning. A direct check would also count how many of MAPPS's 'novel' structures are relaxed copies of retrieved database entries.","tokens_in":14036,"feed_emoji":"⚛️","tokens_out":4251,"duration_ms":37540,"temperature":0.7,"pith_summary":"The paper tries to show that large language models can be more than step-executing tools in materials discovery: given a high-level goal and human scientific intuition, an LLM-based agent can plan a multi-step discovery workflow, write the code to carry it out, incorporate physics via machine-learning force fields, and revise its own errors. If this holds, the practical consequence is that generative materials AI need not be retrained for each new task; a planning agent can steer existing tools toward stable and novel crystals, predicting structures for given compositions, and designing materials with target bandgaps. The headline evidence is a five-fold improvement in the rate of stable, unique, novel crystals generated on MP-20 compared with prior generative models, with DFT-confirmed stability at 34.3%.","feed_headline":"Self-planning AI agent finds stable crystals at 5x the rate","feed_subtitle":"MAPPS merges LLM planning with ML force fields and human feedback, beating generative baselines on MP-20.","key_machinery":"The load-bearing component is the three-agent control loop: a Workflow Planner that turns a high-level task and a scientist's intuition into a five-step plan; a Tool Code Generator that writes executable Python for each step, drawing on a physics toolbox of ML force fields (e.g. CHGNet, M3GNet), pymatgen, and space-group analysis; and a Scientific Mediator that carries intermediate results between steps, requests human approval at decision points, and triggers error-based code revision. Together they implement what the paper calls Level 2 autonomy: the agent designs its own workflow, but a human still shapes it and checks results.","core_discovery":"The paper argues that a language-agent system can move from executing fixed, human-designed steps (Level 1) to planning its own multi-step scientific workflows under human guidance (Level 2), and that this added autonomy produces markedly better materials. On MP-20, MAPPS attains a DFT stability rate of 34.3% and a stability-uniqueness-novelty (S.U.N.) rate of 24.9%, compared with 17.8% and 4.92% for FlowLLM; on crystal structure prediction it reaches 63.9% match rate on MP-20, 27.6% on MPTS-52, and 31.0% on a recent-discovery challenge set, without training any new generative model. The system's workflows start from a task description plus human intuition, are approved by a scientist, and are executed as generated Python code that calls machine-learning force fields, space-group analyzers, and structure-matching tools, with self-reflection to recover from errors.","pith_inferences":["The S.U.N. comparison is not apples-to-apples: MAPPS retrieves candidates from the full MP-20 database and relaxes them, while baselines must generate from training data, so part of the five-fold gain is attributable to retrieval plus physics relaxation rather than to planning. A retrieval-augmented baseline would settle how much.","A natural extension is to measure the contribution of each component separately—planner, self-reflection, and human feedback—by ablating them, and to test whether the same agent transfers to molecules or polymers once a suitable physics toolbox exists.","The paper's own Table 5 suggests that the human intuition input, not the language model, is the main source of workflow validity; this makes 'autonomy' largely conditional on expert guidance."],"forward_implications":["If MAPPS's numbers hold, generative materials models can be outperformed by an agent that does not train new weights but retrieves candidate prototypes, relaxes them with ML force fields, and filters by symmetry and stability.","The same framework transfers across three task families—unconditional generation, composition-to-structure prediction, and bandgap-constrained design—suggesting one agent architecture can replace task-specific pipelines.","Because the property-guided setting reaches 74.6% (high bandgap) and 92.2% (low bandgap) DFT satisfaction, agentic planning is a viable route to inverse design of electronic properties.","The paper's Level-3 study indicates current reasoning LLMs cannot yet produce valid workflows without human intuition, so human-in-the-loop remains necessary for reliability."],"supporting_citations":[{"why":"MP-20 dataset of stable crystals used as retrieval database and baseline training set.","marker":"[22]"},{"why":"DiffCSP baseline and provider of the MPTS-52 CSP benchmark.","marker":"[6]"},{"why":"FlowLLM baseline whose 17.8% stability and 4.92% S.U.N. rates are the main comparison.","marker":"[29]"},{"why":"MatLLMSearch baseline and the processing setup for Matbench evaluation.","marker":"[15]"},{"why":"CHGNet machine-learning force field used for relaxation and metastability estimation.","marker":"[26]"},{"why":"M3GNet surrogate used for metastability estimates.","marker":"[27]"},{"why":"pymatgen StructureMatcher used to define match rate, uniqueness, and novelty.","marker":"[30]"},{"why":"JARVIS-DFT dataset supplying bandgap values for property-guided generation.","marker":"[1]"}],"fun_headline_variants":["AI agent plans its own crystal-discovery pipelines","Self-planning AI finds stable crystals 5x more often","LLM-driven agent automates materials workflow planning","AI agent with human feedback beats generative crystal models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes MAPPS's edge comes from its planning agents rather than from being allowed to retrieve from the full MP-20 dataset while baselines only train on it; no retrieval-augmented baseline is tested.","fun_headline_variants_meta":{"raw":{"variants":["AI agent plans its own crystal-discovery pipelines","Self-planning AI finds stable crystals 5x more often","LLM-driven agent automates materials workflow planning","AI agent with human feedback beats generative crystal models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1493,"prompt_tokens":945,"completion_tokens":548,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":487}},"tokens_in":561,"tokens_out":548,"duration_ms":5645,"temperature":1.0,"reasoning_tokens":487,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:13:42.608749+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the strongest baseline (e.g., FlowLLM) with the same retrieval database, ML-force-field relaxation, and novelty filtering that MAPPS uses; if its S.U.N. rate approaches 24.9%, the claimed five-fold improvement is mostly retrieval and relaxation, not planning. A direct check would also count how many of MAPPS's 'novel' structures are relaxed copies of retrieved database entries.","supporting_citations":[{"cited_title":"Commentary: The Materials Project: A materials genome approach to accelerating materials innovation.APL Materials, 1(1):011002, 2013","cited_arxiv_id":null,"evidence_quote":"MP-20 dataset of stable crystals used as retrieval database and baseline training set."},{"cited_title":"Crystal structure prediction by joint equivariant diffusion.Advances in Neural Information Processing Systems, 2023","cited_arxiv_id":null,"evidence_quote":"DiffCSP baseline and provider of the MPTS-52 CSP benchmark."},{"cited_title":"Flowllm: Flow matching for material generation with large language models as base distributions.Advances in Neural Information Processing Systems, 2024","cited_arxiv_id":null,"evidence_quote":"FlowLLM baseline whose 17.8% stability and 4.92% S.U.N. rates are the main comparison."},{"cited_title":"Large language models are innate crystal structure generators","cited_arxiv_id":null,"evidence_quote":"MatLLMSearch baseline and the processing setup for Matbench evaluation."},{"cited_title":"Python Materials Genomics (pymatgen): A robust, open-source python library for materials analysis.Computational Materials Science, 68:314–319, 2013","cited_arxiv_id":null,"evidence_quote":"pymatgen StructureMatcher used to define match rate, uniqueness, and novelty."}],"review_version":1}