{"id":"ee144e35-f2f1-4358-885f-061a5d772e88","arxiv_id":"2608.11283","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuned LLMs using the mof2text textual descriptor match graph-based models at detecting unreasonable metal-organic framework structures and add explainable error diagnostics.","lead":"This paper converts metal-organic framework crystal data into chemistry-aware text and shows that large language models can then flag defective structures about as accurately as graph models, while also giving human-readable reasons. It matters for database curation because downstream simulations can be badly skewed by unreasonable crystal structures.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark varies descriptor content and format together, so the claim that textualization itself is the enabling step is underdetermined; a content-controlled ablation is needed.","rationale":"The reader's conditional verdict is consistent with the evidence: the paper is a practical applied-ML study with public code and data, and the main gaps are specific and addressable. My concern is not that the results are fabricated, but that the central attribution is underdetermined by the experimental design. The descriptors differ in both content and format, so the observed accuracy gap cannot uniquely support the conclusion that textualization, rather than feature engineering, is the enabling step. This is the load-bearing assumption behind the phrase 'chemically meaningful textualization.' The proposed content-controlled ablation would separate these explanations. I partially agree with the reader's weakest_assumption: label correctness and information retention are adjacent issues, but the more precise gap is that the benchmark does not vary textualization independently of descriptor content, and mof2text may embed label-relevant rule outputs. Since this is an addressable methodological gap rather than a demonstrated error, the appropriate verdict remains CONDITIONAL, and I do not recommend changing the reader's verdict.","tokens_in":7048,"tokens_out":7253,"duration_ms":70364,"concrete_test":"Build a controlled triplet from identical underlying descriptor content: (1) the current mof2text natural-language string; (2) a structured key-value encoding of the same fields (formula, topology, building units, charge/oxidation states, connectivity summary, cell parameters) with no prose; (3) the same fields as a numeric/categorical feature vector. Fine-tune the same LLM (e.g., Qwen3-4B) on (1) and (2), and train a gradient-boosting or MLP on (3), using the paper's train/test split and labels. If (1) does not significantly beat (2) on the held-out test set, the 'linguistically learnable' claim is unsupported. Additionally, grep the mof2text strings for explicit validity-bearing phrases such as 'charge imbalance' or 'missing H'; if such phrases occur, rerun version (1) with those phrases removed to rule out label leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the enabling step for LLM-based MOF validation is chemically meaningful textualization, not the amount of structural information (abstract; Results section around Figure 2). But the nine-descriptor benchmark never isolates textualization as a variable: the descriptors differ simultaneously in which chemical features are included and whether those features are expressed as prose, key-value strings, or token sequences. For example, mof2text explicitly adds topology, building units, and charge-related information (Section S2.2), while other descriptors such as cif_p1 or atom_sequences_plusplus retain atomic positions or sequence-based data. The reported fine-tuned LLM advantage over non-text and semi-text descriptors (Figure 2; Table 1, e.g., Qwen3-mof2text Acc 0.757) can therefore be explained by feature selection alone. Without a paired comparison that fixes the feature set and varies only the linguistic form, the conclusion that 'linguistically learnable form' rather than 'chemically meaningful features' drives performance is underdetermined. A related but secondary risk is that some mof2text fields (charge-related information, connectivity summaries) may transcribe the same rule outputs used to construct the labels, which would make part of the validation task circular. Both risks are addressable by the same content-controlled ablation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the problem of validating computation-ready metal-organic framework (MOF) structures. It first shows that structurally unreasonable MOFs distort predicted properties, then benchmarks nine structural descriptors—ranging from non-text to full-text—for LLM-based binary classification of MOF reasonableness. Fine-tuned LLMs with mof2text and robocry descriptors achieve accuracy comparable to graph-based classifiers (Table 1) and also produce diagnostic rationales and four-class error-type predictions (Figures 3–4). The central claim is that chemically meaningful textualization, not raw structural information content, is the enabling step for LLM-based MOF validation.","tokens_in":7243,"tokens_out":5462,"duration_ms":47436,"significance":"If the central claim holds, the paper provides an interpretable, LLM-based alternative for MOF database curation, with practical value for high-throughput screening workflows. The authors are commendably open: code, datasets, fine-tuned models, and embeddings are released on GitHub, Zenodo, and Hugging Face, which strengthens reproducibility. The paper also makes a falsifiable claim—that representation format, not information content, drives LLM validation performance—which is testable via the content-controlled ablation proposed below. The main weaknesses are experimental-design confounds and lack of uncertainty quantification, both addressable with additional experiments.","major_comments":[{"comment":"The nine-descriptor benchmark varies chemical feature content and linguistic format simultaneously. For instance, mof2text adds topology, building units, and charge-related fields (Section S2.2) that are absent from cif_p1 and atom_sequences_plusplus. The improved fine-tuned performance of mof2text and robocry over non-text and semi-text descriptors could therefore be attributed to feature selection alone, not to 'linguistically learnable form.' A paired ablation that fixes the feature set (e.g., the same mof2text fields expressed as prose, as key-value pairs, and as token sequences) is required to isolate the textualization effect. Without this, the central conclusion that textualization—rather than chemistry-aware feature engineering—is the enabling step is underdetermined.","section":"Figure 2 and Section S2.2"},{"comment":"The main text reports only point estimates, with no confidence intervals, significance tests, or repeated-run variability. The claim that fine-tuned LLMs are 'comparable' to graph-based models rests on differences such as 0.757 (FT-Qwen3-mof2text) vs. 0.764 (MOFClassifier) and 0.807 (CGCNN). These gaps are within the range of typical noise for held-out test sets, especially if the test set is small. Please provide error bars or significance tests, and state the test-set size and the variance across random seeds or k-fold splits.","section":"Table 1 and Figure 2"},{"comment":"mof2text includes PACMAN-derived atomic charge information, while the benchmark's 'charge' defect class is defined by charge imbalance. If the charge fields directly transcribe the same rule-based output used to construct the labels, the LLM may be reading off the answer rather than learning chemistry. I recommend a label-leakage test: remove all charge-related fields from mof2text and re-run the fine-tuning. If accuracy on the 'charge' class specifically collapses, the result would indicate that the model uses the rule output directly, which would weaken the claim of chemically meaningful learning.","section":"Figure 3b and Section S2.2"}],"minor_comments":[{"comment":"The structure 'W AFKAQ' appears with a space in several places; this should be 'WAFKAQ'.","section":"Figure 1 and main text"},{"comment":"Reference 34 is incomplete: it lists 'Y ear' in place of the year and venue. Please provide the full citation with correct bibliographic details.","section":"Reference 34"},{"comment":"The caption's 'lower-left half circle' and 'upper-left half circle' is confusing; please clarify which half corresponds to pre-trained and which to fine-tuned models.","section":"Figure 2 caption"},{"comment":"The row labeled 'LLM-prop' uses footnote e to indicate mof2text, but the descriptor name is not otherwise defined; please reconcile the notation (LLM-prop vs. Qwen-3-4B) so the descriptor mapping is unambiguous.","section":"Table 1"},{"comment":"The sentence 'From LLM generation, capture not only isolated defect labels but also the broader chemical consequences' is grammatically incomplete; please revise.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is timely and the open-source release is a strength. My main concern is the confound between feature content and textual form in the benchmark; the authors can resolve this with a contained ablation. I also note that the references to Supporting Information for dataset splits and hyperparameters make the evaluation hard to verify without the SI; this should be available to reviewers. The novelty is not in doubt, but the central claim needs the proposed ablation before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one if you work on MOF curation or LLMs for materials. The new thing is mof2text, a natural-language descriptor that condenses a CIF into coordination, connectivity, and charge context, plus a nine-descriptor benchmark of LLM validity classification and a fine-tuned error-category predictor. Code, data, and models are public, and the paper compares against standard rule- and graph-based methods. Those are real strengths; the empirical core is reproducible and the writing is clear.\n\nWhat the paper does well: it shows that after fine-tuning, full-text descriptors like mof2text and robocry classify unreasonable MOFs roughly as well as graph models, and that rationales cluster into chemically plausible error types. The property-deviation analysis in Figure 1 makes a reasonable case that validation matters. The benchmark includes sensible baselines (MOFChecker, MOSAEC, CGCNN, PMTransformer) and reports accuracy, precision, recall, and F1.\n\nThe soft spot is the paper's headline claim. The data show that fine-tuned LLMs do best with mof2text/robocry, but the nine descriptors vary chemical content and linguistic format simultaneously. A descriptor like mof2text adds topology, building units, and charge information while stripping atom coordinates; comparing it against raw CIF text or atom sequences changes both what is in the representation and how it is phrased. So the conclusion that the enabling step is textualization, rather than feature selection, is underdetermined. The fix is a paired ablation: hold the feature set fixed and vary only the prose form. I would treat the current conclusion as plausible but not proven.\n\nSecond, the main text reports only point estimates in Table 1, without confidence intervals or significance tests. Dataset construction and hyperparameters live in the SI. That is acceptable only if the SI is watertight; the authors should clearly flag the split procedure and label provenance. A related worry I have not been able to dismiss from the main text is that some mof2text fields (charge-related information, connectivity summaries) may transcribe the same rule outputs used to construct the labels, which would make part of the validation circular. The paper needs to show the label-generation pipeline and the descriptor-generation pipeline side by side.\n\nThe error-rationale results are qualitative, and the authors mostly say so. That is a minor weakness, not a fatal one.\n\nWho gets value: MOF database curators and people building LLM tools for materials. It deserves a serious referee. I would not desk-reject it. I would send it back conditional on the content-controlled ablation and on making label and descriptor provenance explicit.","headline":"Useful applied-ML benchmark, but the central claim about textualization is underdetermined because the descriptors vary chemical content and linguistic format together; still worth peer review.","tokens_in":7760,"tokens_out":2617,"would_cite":true,"duration_ms":24438,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rewriting MOF structures as chemistry-aware text lets fine-tuned LLMs match graph-based validators while also explaining each flag.","keywords":["metal-organic frameworks","large language models","structural validation","chemical textualization","mof2text","robocry","interpretable machine learning","MOF database curation"],"falsifier":"Train the same fine-tuned pipeline on mof2text and on a token-identical but chemically scrambled version of mof2text (words preserved, chemical organization destroyed); if accuracy does not drop, the reported success is not caused by chemically meaningful organization, and the paper's central claim fails.","tokens_in":6826,"feed_emoji":"🧪","tokens_out":8536,"duration_ms":77143,"temperature":0.7,"pith_summary":"The paper claims that large language models (LLMs) can act as reliable validators of metal-organic framework (MOF) crystal structures, provided the crystallographic data is first rewritten as chemically meaningful natural language. Benchmarking nine structural descriptors, the authors find that simply adding more structural detail does not help: what matters is whether local coordination, framework connectivity, and chemical context are organized into a linguistically learnable form. After fine-tuning, LLMs using the specialized descriptors mof2text and robocry match graph-based classifiers in accuracy and recall, and they go beyond classification by generating textual rationales and predicting coarse error categories such as missing hydrogens, disorder, and charge imbalance. If correct, this makes general-purpose language models practical and explainable curators for computation-ready MOF databases.","feed_headline":"Fine-tuned LLMs match graph models at flagging bad MOFs","feed_subtitle":"A chemistry-aware text format turns generic language models into explainable validators for MOF databases.","key_machinery":"The load-bearing object is the chemically meaningful textualization itself, concretely realized in the mof2text descriptor, with the robocry full-text format playing the same role. mof2text converts a crystallographic information file into natural-language prose listing the chemical formula, space group, dimension, topology, lattice parameters, unique building units, interatomic distances, and atomic charges, while avoiding an atom-by-atom translation of the periodic lattice. Its job is to present the chemistry that decides validity, coordination environment, framework connectivity, and charge balance, in a form a language model can learn, and to keep that chemistry visible enough that the model can later name the defect. The paper's evidence that this is the operative mechanism is the descriptor benchmark: more detailed geometric encodings do not improve performance, and only chemically organized text separates valid from invalid MOFs in embedding space.","core_discovery":"The central discovery is that textualization, not information content, is the enabling step for LLM-based MOF validation. Benchmarking nine descriptors from bare chemical formulas to full crystallographic text, the authors show that pre-trained LLMs struggle with raw crystallographic input, while fine-tuned LLMs succeed only when the structure is encoded as chemically legible prose that keeps building units, connectivity, topology, space group, and charge-related information and deliberately drops redundant atom-by-atom periodicity. The resulting descriptor, mof2text (with robocry behaving similarly), yields validation accuracy comparable to graph neural networks and clearly above geometry-based checkers. The same descriptor separates reasonable from unreasonable structures in embedding space, and the LLM attributes defects to chemically meaningful categories, so the paper's claim is that mof2text is an enabling representation rather than an alternative encoding: it is what lets a general text model internalize crystallographic validity.","pith_inferences":["The causal claim implies a sharp, testable ablation: a token-identical but chemically scrambled version of mof2text should destroy fine-tuned accuracy if textual organization is truly the carrier, so running that ablation is the natural next experiment.","The same representation principle should transfer to other reticular solids such as covalent organic frameworks or hydrogen-bonded organic frameworks, whose validity also depends on connectivity and charge balance rather than stoichiometry alone.","Since charge-related rationales became more frequent when explicit atomic charges were added to the text, a routine mof2text variant that always includes charge annotations could improve charge-error recall in practical screening.","The t-SNE separation produced by chemically meaningful descriptors suggests such text could serve as a general learned embedding for MOF property prediction, not only for validity classification."],"forward_implications":["Fine-tuned LLMs paired with mof2text or robocry can screen MOF databases and flag unreasonable entries with accuracy comparable to graph-based classifiers, without requiring access to licensed structural databases.","Each flagged MOF arrives with a textual rationale naming likely defect types, turning a binary filter into a diagnostic tool for database curation.","On annotated data, the same models can assign coarse error categories (missing hydrogens, disorder, charge imbalance, other) with roughly 70 percent exact-match accuracy, showing that classification and explanation are jointly learnable.","Because mof2text keeps global features such as topology and space group alongside local coordination, the method integrates information that rule-based checkers handle separately, which is presumably why it generalizes across error classes."],"supporting_citations":[{"why":"Supplies the bond-order-based geometry baseline whose limited accuracy motivates a better validator.","marker":"[9]"},{"why":"Provides the graph-based MOFClassifier baseline that sets the performance level the fine-tuned LLMs are compared with.","marker":"[11]"},{"why":"Supplies the geometry-based MOFChecker baseline, which the fine-tuned LLMs clearly outperform.","marker":"[13]"},{"why":"Provides the literature accuracy values for the oxidation-state-based MOSAEC method.","marker":"[14]"},{"why":"Supplies the transformer-based PMTransformer baseline, the strongest comparison model in the evaluation.","marker":"[15]"},{"why":"Supplies crystal_text_LLM, a crystallographic text descriptor that underperforms when fed to pre-trained LLMs.","marker":"[34]"},{"why":"Supplies robocry, one of the two specialized text descriptors that enable the central result.","marker":"[35]"},{"why":"Provides the general natural-language material representation approach on which the text descriptors draw.","marker":"[36]"}],"fun_headline_variants":["Textualization, not info volume, turns LLMs into MOF validators","Chemically meaningful text lets LLMs explain MOF defects","LLMs match graph nets at flagging bad MOFs, with reasons","Explainable MOF checks: LLMs go beyond black-box screening","Chemistry-aware text converts LLMs into explainable MOF validators"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes the manually and rule-derived labels for reasonable versus unreasonable MOFs, and for the four error classes, are correct, and assumes that mof2text strips away only chemically irrelevant atom-level detail.","fun_headline_variants_meta":{"raw":{"variants":["Textualization, not info volume, turns LLMs into MOF validators","Chemically meaningful text lets LLMs explain MOF defects","LLMs match graph nets at flagging bad MOFs, with reasons","Explainable MOF checks: LLMs go beyond black-box screening","Chemistry-aware text converts LLMs into explainable MOF validators"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000628,"raw_usage":{"total_tokens":2903,"prompt_tokens":947,"completion_tokens":1956,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":1863}},"tokens_in":563,"tokens_out":1956,"duration_ms":41079,"temperature":1.0,"reasoning_tokens":1863,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:12:04.589097+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same fine-tuned pipeline on mof2text and on a token-identical but chemically scrambled version of mof2text (words preserved, chemical organization destroyed); if accuracy does not drop, the reported success is not caused by chemically meaningful organization, and the paper's central claim fails.","supporting_citations":[{"cited_title":"Daglar, H","cited_arxiv_id":null,"evidence_quote":"Supplies the bond-order-based geometry baseline whose limited accuracy motivates a better validator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the graph-based MOFClassifier baseline that sets the performance level the fine-tuned LLMs are compared with."},{"cited_title":"Gibaldi, A","cited_arxiv_id":null,"evidence_quote":"Supplies the geometry-based MOFChecker baseline, which the fine-tuned LLMs clearly outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the literature accuracy values for the oxidation-state-based MOSAEC method."},{"cited_title":"White, M","cited_arxiv_id":null,"evidence_quote":"Supplies the transformer-based PMTransformer baseline, the strongest comparison model in the evaluation."},{"cited_title":"Dybtsev, H","cited_arxiv_id":null,"evidence_quote":"Supplies crystal_text_LLM, a crystallographic text descriptor that underperforms when fed to pre-trained LLMs."},{"cited_title":"Ganose and A","cited_arxiv_id":null,"evidence_quote":"Supplies robocry, one of the two specialized text descriptors that enable the central result."}],"review_version":1}