{"id":"74df6806-d7cf-4802-9901-b73b2f91ee0a","arxiv_id":"2608.10483","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A retrieval-based LLM agent predicts double perovskite space groups with improved Top-1 accuracy on rare space groups while keeping overall accuracy competitive with strong baselines.","lead":"This paper presents DyRIS, a framework that uses a large language model to predict the crystal space group of double perovskite compositions. It reports better accuracy on rare space-group classes than standard machine-learning baselines, which matters because rare structures often drive material function.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ordering-surrogate train/test split is never stated; if it was trained on full data, test compositions' structure-derived ordering labels leak into the prompts and could explain part of the minor-SG gains.","rationale":"The reader's weakest assumption identifies the same point I consider most load-bearing. The paper's central claim is that the reported minor-SG gains come from DyRIS's combination of diversity-enhanced retrieval, quantitative indicators, major-SG bias control, and B/B' ordering prior. Of these components, only the B/B' ordering surrogate has an ambiguous relationship to the train/test split. Every other component is explicitly described as using the training split: retrieval draws from the training pool, quantitative indicators use training-set means and standard deviations, and the re-ranker experiment even describes a leave-one-out protocol to prevent leakage. The ordering surrogate, by contrast, is described only as 'evaluated on test data' in SI S3, with no statement about its training split. Because ordering labels are inferred from the same structure files whose space groups are the prediction target, using full-data training for the surrogate would constitute a direct leakage path into the prompts. The 0.5 ablation shows that removing ordering information has mixed effects at that ratio, so the surrogate concern is not the sole explanation for the headline 3.26 percentage-point improvement; nevertheless, the absence of an explicit train/test statement for the surrogate is the clearest unverified assumption that could invalidate the causal attribution of the method's gains. The paper's broader claim that DyRIS achieves the best minor-SG metrics at all training-data ratios also depends on the high-data ordering contribution, where the leakage risk is larger. Secondary concerns about LLM stochasticity and missing code are real but less decisive: the stochasticity issue affects variance reporting, while the surrogate split affects the validity of the evidence itself. The recommended verdict remains CONDITIONAL, matching the reader's judgment.","tokens_in":28121,"tokens_out":9353,"duration_ms":88039,"concrete_test":"Obtain the code/data or have the authors run the following check: retrain the PyCaret ordering surrogate on the same 0.5 and 0.8 training splits only, regenerate ordering probabilities for the test queries, and rerun full DyRIS. Compare the resulting Table 3 rows and SI Table S3 with the published values; if Minor-SG Top-1/Top-3 metrics or the 0.8 ordering-ablation change move by more than the reported split-to-split standard deviation, the leakage concern is confirmed; if all metrics stay within noise, the central claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the unstated train/test protocol of the B/B' ordering surrogate. Section 2.4.1 says DyRIS uses a PyCaret-based surrogate to predict B/B' ordering probabilities for each query, and SI Section S3 describes the surrogate and its evaluation, but neither passage states that the surrogate was trained exclusively on the training split. The ordering labels are derived from the same CIF/POSCAR structure files whose SG labels are the prediction target, and the surrogate's probabilities are injected into every DyRIS prompt as 'ordering evidence' together with compatible-SG flags. If the surrogate saw test compositions during training, the LLM receives structural information derived from the very structures it is supposed to predict, inflating minor-SG metrics, especially the 4.24 percentage-point ordering-ablation gain reported at ratio 0.8 and potentially the ratio-0.5 headline. The main-text sentence that 'the training set was used to implement DyRIS' does not resolve this, because the surrogate construction is not explicitly tied to that split. Without code or data, the reader cannot verify this from the manuscript.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DyRIS, an LLM-agent framework for predicting space groups of double perovskites from composition. DyRIS uses diversity-enhanced dynamic few-shot retrieval in a weighted feature embedding to select in-context examples while restricting per-SG representation, and a rule-guided inference step that combines B/B′ ordering probabilities from a PyCaret surrogate, quantitative indicators (Combined score, Global fit, z-consistency), and major-SG bias control to produce ranked Top-3 SGs. The authors evaluate on 3,528 thermodynamically filtered DP entries across 19 SG classes at data ratios 0.3/0.5/0.8, comparing against PyCaret, CrabNet, and CRYSPNet. They report competitive overall accuracy, best macro-F1 and minor-SG metrics at ratio 0.5, and a 3.26 percentage-point improvement in Minor-SG Top-1 accuracy over CrabNet. Ablations and replacement experiments indicate each component contributes, and the final rule-guided inference is not trivially replaced by supervised re-rankers. The paper also transparently documents a high-data limitation in which final ranking demotes correct major-SG Top-1 predictions.","tokens_in":28312,"tokens_out":5041,"duration_ms":41920,"significance":"If the results are reproducible, the paper makes a useful contribution to composition-based space-group prediction under severe class imbalance: it combines LLM in-context learning with crystallographic priors and provides a concrete instance where LLM-based rule inference outperforms conventional imbalanced-learning baselines for minor classes. Strengths include the systematic evaluation over repeated splits, the data-ratio sensitivity analysis, the component ablations, the replacement experiments with learned re-rankers, and the candid analysis of the high-data Top-1 degradation. The method is falsifiable in the sense that the prompt template and inference rules are fully specified in the SI, and the design choices are motivated. However, the absence of code and of repeated LLM sampling, together with an underspecified surrogate training protocol, currently prevents verification of the headline numerical claims.","major_comments":[{"comment":"The construction of the B/B′ ordering surrogate does not explicitly state that the surrogate was trained only on the training split. Section S3 says the surrogate was evaluated on the test data and that ordering labels were derived from the same structure files whose SGs are the prediction target, but it does not report the surrogate's training split. If the surrogate was fit on the full dataset, the ordering probabilities injected into every DyRIS prompt would encode information derived from the test structures, which could inflate the reported minor-SG gains, particularly the 4.24 percentage-point ordering-ablation decrease at ratio 0.8 and the ratio-0.5 headline metrics. Please state unambiguously that the surrogate is trained exclusively on the training split or, if that is not the case, retrain it under the correct protocol and rerun the affected tables.","section":"2.4.1 / SI S3"},{"comment":"The evaluation does not account for LLM sampling stochasticity. Table 1 sets temperature to 1.0, yet the reported mean ± standard deviation in Figures 2 and 3 and Tables 3 and 4 appears to be computed from a single LLM response per query per split. At temperature 1.0, the rule-guided inference is stochastic, so the five-split standard deviations reflect only split variability, not model stochasticity; the 'best' claims are therefore not supported by any measure of run-to-run variability. Please either run multiple LLM samples per query and aggregate, or lower the temperature and justify the single-sample protocol.","section":"Table 1 / Section 3.1.1"},{"comment":"The 'best across all metrics' claims are not accompanied by significance tests or confidence intervals. For example, at ratio 0.5 the Overall Top-1 macro-F1 difference between DyRIS and CrabNet is about 0.02 (0.3846 vs 0.365), which may be within the noise of five splits, and several Minor-SG Top-3 differences are small. Without paired tests or effect-size reporting, the superiority claims are not statistically established. Please add appropriate significance tests or clearly label differences as not statistically significant.","section":"Section 3.1.1 / Table 3"}],"minor_comments":[{"comment":"The header of Table 3 appears corrupted: the column labels 'SG87 SG139 Minor SGs SG201' are not metric names. Please replace them with the six metric names listed in the caption or with a clear abbreviation and define the abbreviation in the table notes.","section":"Table 3"},{"comment":"The sentence 'we varied the training-data ratio from 0.3, 0.5, and 0.8' is ungrammatical; it should read 'to 0.3, 0.5, and 0.8' or 'at 0.3, 0.5, and 0.8'.","section":"Section 3.1.2"},{"comment":"The feature-block weighting procedure should state how each feature block is scaled before concatenation and before computing weighted distances. Without per-feature normalization, the 132-dimensional Magpie block may dominate the weighted Euclidean distance used for retrieval.","section":"Section 2.3 / SI S2"},{"comment":"The metric labeled 'Overall Top-1 accuracy' in Table S3 is actually an ordering-compatible coverage fraction, not a classification accuracy. Please rename the row to avoid confusion.","section":"SI Table S3"},{"comment":"The manuscript contains no code or data availability statement. Providing a repository with the feature construction, prompt templates, and evaluation scripts would substantially improve reproducibility of the LLM pipeline and the baseline comparisons.","section":"General reproducibility"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical study with an honest limitations section, but the surrogate train/test ambiguity is a central leakage risk that must be resolved in revision. For a journal audience, code release and a repeated-sampling protocol for the LLM are important; without them the quantitative claims cannot be independently verified. The paper fits the journal scope if these issues are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the central claim is plausible but conditional. The new thing here is the specific integration—diversity-constrained dynamic retrieval plus rule-guided LLM inference with explicit major-SG bias control—applied to double-perovskite space-group prediction. That combination is not in the prior work, and the paper is honest about its high-data limitation. The evaluation has real substance: five random splits, ablations, and replacement experiments showing that the rule-guided step isn't trivially replaced by a learned ranker. Credit where due: the paper does a good job of dissecting where its gains come from and where they disappear.\n\nThe main soft spot is exactly what the stress-test flags. Neither main text Section 2.4.1 nor SI Section S3 states that the PyCaret ordering surrogate was trained exclusively on the training split. It's described as evaluated on test data, and the main text's general sentence that the training set was used to implement DyRIS is not enough to rule out leakage. Since the ordering probabilities are injected into every prompt as structural evidence, full-data training would inflate the minor-SG numbers, especially the 4.24 pp ordering-ablation gain at ratio 0.8. This needs to be stated explicitly, and ideally the code/data released so it can be checked. The absence of repeated LLM sampling at temperature 1.0 is also a minor issue: the reported standard deviations only cover split variation, not LLM stochasticity. The tiny minor-SG test sets at ratio 0.8 (single digits for some classes) are disclosed but still make some comparisons noisy.\n\nWho is this for? Materials informatics people and anyone using retrieval-augmented LLMs for imbalanced classification. The method is coherent, the paper is well-written, and the authors are not overselling. I'd send it to peer review without hesitation, but I'd ask for a clear statement of the surrogate split as a condition of acceptance. If the surrogate was trained on the full data, that's a significant revision; if it was clean, the current analysis largely holds.","headline":"A solid, honest LLM-retrieval method for imbalanced space-group prediction, but the ordering-surrogate train/test split is ambiguous and needs to be resolved before the headline numbers can be fully trusted.","tokens_in":28829,"tokens_out":2478,"would_cite":true,"duration_ms":24809,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM agent with diversity-limited retrieval and rule-guided ranking improves rare space-group prediction for double perovskites, gaining 3.26 percentage points over the strongest composition-only baseline on minor space-group Top-1…","keywords":["double perovskites","space-group prediction","large language models","few-shot in-context learning","class imbalance","B/B' cation ordering","materials informatics","rule-guided inference"],"falsifier":"Retrain the PyCaret ordering surrogate under a strict training/test split and rerun DyRIS at a 0.5 training ratio, then check whether Minor-SG Top-1 accuracy still beats CrabNet by 3.26 percentage points; a second check is to randomize the ordering probabilities supplied to the LLM and measure how much Minor-SG Top-1 changes, because if the gain persists, ordering evidence is not carrying the effect.","tokens_in":27892,"feed_emoji":"🧪","tokens_out":6515,"duration_ms":55208,"temperature":0.7,"pith_summary":"DyRIS is a large-language-model agent that predicts the space group of a double perovskite from its chemical formula, and the paper's central claim is that its two design choices—diversity-restricted retrieval of in-context examples and rule-guided LLM ranking—fix a real weakness of data-driven baselines: rare space groups are buried by two majority classes, SG 14 and SG 225. At a training-data ratio of 0.5, DyRIS is roughly tied with CrabNet on overall Top-1 accuracy, beats every baseline on Overall Top-1 macro-F1, and lifts Minor-SG Top-1 accuracy by 3.26 percentage points over CrabNet while also leading Minor-SG Top-3 accuracy. Ablations attribute the gain to the retrieval examples first, with quantitative indicators, majority-class bias control, and B/B' ordering information each contributing, and the final LLM ranking step outperforms trained classifiers and rankers given the same candidate evidence. The paper also shows the method's limit: as training data grow, the final ranking step can demote correct majority-class predictions, so Top-1 accuracy does not keep improving. A sympathetic reader would take the message to be that retrieval-based LLM reasoning plus crystallographic rules is a viable complement to statistical models for imbalanced materials classification.","feed_headline":"LLM agent lifts rare space-group picks by 3.26 points","feed_subtitle":"Dynamic few-shot retrieval plus rule-guided ranking beats composition-only baselines on imbalanced double-perovskite data.","key_machinery":"The central object is DyRIS's two-stage decision loop. First, diversity-enhanced dynamic few-shot retrieval: a 161-dimensional composition embedding, built from DP-specific, site, B-site-derived, and Magpie features, with block weights tuned on training data, and each query retrieves its nearest training neighbors under the constraint that no space group contributes more than one in-context example. Second, rule-guided LLM inference: the LLM receives the retrieved examples plus B/B' ordering probabilities from a PyCaret surrogate, a Combined score $CS(s) = \\log(1+n_{\\mathrm{local}}(s)) \\cdot n_{\\mathrm{local}}(s)/(N_s+1)$, a Global fit $\\mathrm{G_{fit}}(s)$ over six standardized features, and feature-wise z-consistency metrics ($z_{\\mathrm{abs\\_mean}}$, $z_{\\mathrm{max\\_abs}}$, $z_{\\mathrm{best\\_count}}$), while explicit rules retain candidates with high local density, replace them only on converging evidence, admit rare-ordering-compatible candidates when supported, and hold SG 14 and SG 225 to a higher Top-1 bar. The machinery's work is to convert a class-imbalanced nearest-neighbor problem into an evidence-combination problem that the LLM resolves per query.","core_discovery":"On the paper's own terms, the central discovery is that space-group prediction on a heavily imbalanced double-perovskite dataset can be improved for the rare classes by making the LLM's evidence dynamic and class-balanced rather than relying on the model's static knowledge or a fixed training set. The proposed pipeline retrieves five in-context examples nearest to the query in a weighted composition embedding but enforces at most one example per space group, so the two majority classes, SG 14 and SG 225, which together account for 72% of the data, cannot dominate the prompt. It then hands the LLM a structured query containing B/B' ordering probabilities, a Combined score measuring local class density, Global fit and feature-wise z-consistency relative to each candidate's training distribution, and explicit instructions to rank the majority SGs at Top-1 only when multiple indicators agree. On five random 50/50 splits, the framework reaches 0.4266 Overall Top-1 accuracy and 0.3762 Minor-SG Top-1 accuracy, gains 3.26 percentage points over CrabNet on Minor-SG Top-1, and attains the best Overall Top-1 macro-F1 and Minor-SG metrics among the compared models. The paper's explanation for the gain is integration: retrieval narrows the candidate set, quantitative indicators stabilize decisions for classes with few samples, ordering information acts as a soft crystallographic prior that matters most when the candidate set is crowded, and bias control stops the majority classes from being overpicked.","pith_inferences":["The same retrieval-diversity plus rule-guided ranking recipe should transfer to other imbalanced structure-prediction tasks, such as simple perovskites, Heusler compounds, or mineral paragenesis classes, where majority classes dominate and fixed embeddings are less reliable; the paper demonstrates it only on double perovskites.","Because the paper leaves unclear whether the B/B' ordering surrogate is trained only on the training split, an immediate test is to retrain that surrogate under a strict split and see whether the minor-SG gains survive; if the surrogate saw test structures, the reported 3.26-point gain is an upper bound.","The class-dependent ranking diagnosis suggests a cheap improvement: trust retrieval order for majority classes while applying evidence-integrated ranking for minor classes; the paper's own hybrid experiment is a heuristic first step in that direction."],"forward_implications":["At a 0.5 training-data ratio, DyRIS matches CrabNet on overall Top-1 accuracy while improving Minor-SG Top-1 by 3.26 percentage points and beating the strongest PyCaret baseline on Minor-SG Top-3.","The retrieved in-context examples are the load-bearing component: removing them drops Overall Top-3 accuracy from 0.6994 to 0.5424 and Overall Top-1 accuracy from 0.4266 to 0.2214.","Quantitative indicators mainly stabilize minor-SG decisions: removing them decreases Minor-SG Top-1 accuracy by about 5.5 percentage points at a training-data ratio of 0.5.","B/B' ordering information becomes a useful soft prior as data grow: at a training-data ratio of 0.8, removing it lowers Minor-SG Top-1 accuracy by about 4.2 percentage points.","The final LLM ranking step outperforms conventional classifiers and learning-to-rank models trained on the same candidate evidence, with the largest gaps on Minor-SG Top-1 metrics."],"supporting_citations":[{"why":"supplies the descriptor-based SG prediction baseline (CRYSPNet) that DyRIS must beat.","marker":"[23]"},{"why":"supplies the composition-based attention baseline (CrabNet) used for the headline 3.26 percentage-point Minor-SG Top-1 comparison.","marker":"[58]"},{"why":"provides the machine-learning view of double-perovskite cation ordering on which the PyCaret ordering surrogate is built.","marker":"[54]"},{"why":"supplies the crystallographic ordering classification and ordering-to-space-group compatibility mapping used as soft structural prior.","marker":"[43]"},{"why":"justifies the class-size filtering and the separate minor-SG evaluation by documenting effects of class imbalance.","marker":"[37]"},{"why":"grounds kNN-based in-context example retrieval, which diversity-enhanced retrieval modifies.","marker":"[47]"},{"why":"defines the Magpie composition features used as one of the four embedding blocks.","marker":"[45]"}],"fun_headline_variants":["LLM with balanced few-shot aids rare space-group calls","DyRIS uses dynamic few-shot to lift minor space-group picks","Class-balanced prompts help LLM predict rare space groups","LLM agent gains 3.26 points on rare double-perovskite phases","Dynamic few-shot retrieval lifts LLM rare-space-group accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the B/B' ordering surrogate model used to supply soft ordering evidence is trained on the training split alone; if it was fitted on the full dataset, the ordering information for test queries would leak the very structures whose space groups are being predicted, and the reported minor-SG gains could be inflated.","fun_headline_variants_meta":{"raw":{"variants":["LLM with balanced few-shot aids rare space-group calls","DyRIS uses dynamic few-shot to lift minor space-group picks","Class-balanced prompts help LLM predict rare space groups","LLM agent gains 3.26 points on rare double-perovskite phases","Dynamic few-shot retrieval lifts LLM rare-space-group accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1586,"prompt_tokens":1179,"completion_tokens":407,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":795,"completion_tokens_details":{"reasoning_tokens":321}},"tokens_in":795,"tokens_out":407,"duration_ms":4300,"temperature":1.0,"reasoning_tokens":321,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:19:34.156429+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the PyCaret ordering surrogate under a strict training/test split and rerun DyRIS at a 0.5 training ratio, then check whether Minor-SG Top-1 accuracy still beats CrabNet by 3.26 percentage points; a second check is to randomize the ordering probabilities supplied to the LLM and measure how much Minor-SG Top-1 changes, because if the gain persists, ordering evidence is not carrying the effect.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the descriptor-based SG prediction baseline (CRYSPNet) that DyRIS must beat."},{"cited_title":"Y.-T., Kauwe, S","cited_arxiv_id":null,"evidence_quote":"supplies the composition-based attention baseline (CrabNet) used for the headline 3.26 percentage-point Minor-SG Top-1 comparison."},{"cited_title":"P., Shaikh, M","cited_arxiv_id":null,"evidence_quote":"provides the machine-learning view of double-perovskite cation ordering on which the PyCaret ordering surrogate is built."},{"cited_title":"& Woodward, P","cited_arxiv_id":null,"evidence_quote":"supplies the crystallographic ordering classification and ordering-to-space-group compatibility mapping used as soft structural prior."},{"cited_title":"& Herrera, F","cited_arxiv_id":null,"evidence_quote":"justifies the class-size filtering and the separate minor-SG evaluation by documenting effects of class imbalance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"grounds kNN-based in-context example retrieval, which diversity-enhanced retrieval modifies."},{"cited_title":"& Wolverton, C","cited_arxiv_id":null,"evidence_quote":"defines the Magpie composition features used as one of the four embedding blocks."}],"review_version":1}