{"id":"951d7b5e-50fb-40e8-ba09-3953d9a1ce65","arxiv_id":"2501.12919","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CLaSP aligns crystal structures with publication text via contrastive learning, enabling text-based retrieval and semantically organized material maps.","lead":"CLaSP trains a neural network to embed crystal structures and natural-language text into a shared space, using over 400,000 materials and their publication titles and abstracts as training data. A user can then query a structure database with a free-text description like \"superconductor\" or \"narrow-bandgap material\" and retrieve candidate crystals without needing property labels.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No deduplication of COD entries; near-duplicate structures across train/test likely inflate Table 1 ROC-AUC, threatening the central retrieval claim.","rationale":"The reader's weakest_assumption focused on the comprehensiveness/reliability of titles as property descriptions, citing the near-random 'ferromagnetic' retrieval. That is a real and acknowledged limitation, but it does not invalidate the central claim for concepts that are well-expressed in titles. The more load-bearing risk is dataset leakage through near-duplicate structures: if it exists, the headline numbers themselves are not trustworthy evidence for any learned structure–property mapping, regardless of label quality. The reader did list 'possible duplicate-structure leakage' among the evaluation gaps, so this is a partial agreement rather than a disagreement. I do not assert that leakage definitely exists; I assert that the paper provides no evidence against it, and the random split makes it likely given COD's redundancy. The concrete test would settle this. If the test passes, the central claim stands on much firmer ground; if it fails, the paper needs substantial revision (e.g., deduplicated splits and re-evaluation). Thus the appropriate verdict remains CONDITIONAL, with the explicit condition being a deduplication-aware evaluation.","tokens_in":17132,"tokens_out":3983,"duration_ms":45624,"concrete_test":"Compute, for each of the 40,606 test entries, the maximum cosine similarity between its CGCNN embedding (or alternative structural fingerprint) and all 324,838 training entries; also flag exact or reduced-formula + space-group matches. Remove all test entries whose nearest training neighbor exceeds a threshold (e.g., cosine similarity > 0.95, or same reduced formula and space group), then recompute the ROC-AUC and AP values in Table 1. If the pre-trained 'superconductor' ROC-AUC drops by more than 0.05 or the average drops materially, the reported metrics are leakage-inflated and the central claim requires major qualification.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central quantitative evidence for the claim that CLaSP captures property- and functionality-related structure similarities is Table 1's ROC-AUC/AP. These scores are computed on a random 8:1:1 split of 406,048 COD entries with no reported deduplication by chemical composition, space group, or structural similarity. COD is known to contain multiple entries for the same or nearly identical crystals (polymorphs, re-refinements, the same compound reported in different papers). CGCNN embeddings of near-identical structures are nearly identical, so a test structure whose near-duplicate appeared in training with a title containing 'superconductor' can be retrieved correctly by structure memorization, not by learning a general structure–property mapping. This would inflate the reported PT ROC-AUC of 0.9431 for 'superconductor' and the other high scores, directly undermining the strongest claim that text-based screening works for unannotated structures. The paper's own discussion of the poor 'ferromagnetic' result (Sec. 5.1) addresses label noise, not this leakage mechanism, so it does not mitigate the concern.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLaSP (Contrastive Language–Structure Pre-training), a two-stage contrastive learning framework that aligns crystal-structure embeddings (a CGCNN encoder) with text embeddings (a frozen SciBERT encoder followed by an MLP). Pre-training uses pairs of COD crystal structures and their publication titles; fine-tuning uses LLM-generated property/functionality keywords derived from title–abstract pairs. The authors evaluate the method by zero-shot text-based retrieval of test-set structures, reporting ROC-AUC and AP for six property keywords (e.g., superconductor, ferroelectric), and by t-SNE visualization of the learned embedding space. They also compare against a structure-only baseline (CMML) and analyze the effect of the CosFace loss margin and scale. The central claim is that bibliographic text provides sufficient supervision to learn a crossmodal space that supports intuitive, text-driven screening of unannotated materials.","tokens_in":17340,"tokens_out":3207,"duration_ms":34434,"significance":"If the reported results hold, CLaSP would be a valuable contribution to materials informatics: it offers a scalable route to property- and functionality-sensitive structure embeddings without expensive first-principles labels, and it enables interpretable, text-queryable materials retrieval. The paper is strengthened by the release of code, pretrained weights, and the constructed dataset, and by the use of a held-out test split for retrieval evaluation. The qualitative visualizations (Sec. 4.3) provide intuitive evidence of semantic organization. However, the quantitative evaluation currently has gaps—most notably the absence of train/test deduplication for the COD data, the lack of a conventional text-search baseline, and the reliance on a single training run—that make the headline ROC-AUC numbers difficult to interpret as evidence of genuine generalization to unannotated structures.","major_comments":[{"comment":"The random 8:1:1 split of 406,048 COD entries is performed without any deduplication by chemical composition, space group, or structural similarity. COD is known to contain multiple entries for the same or nearly identical crystals (polymorphs, re-refinements, the same compound reported in different papers). Because CGCNN embeddings of near-identical structures are nearly identical, a test structure whose near-duplicate appeared in training with a title containing 'superconductor' could be retrieved correctly by structural memorization rather than by learning a general structure–property mapping. This would inflate the PT ROC-AUC of 0.9431 for 'superconductor' and other high scores in Table 1. The discussion in Sec. 5.1 addresses title-label noise (e.g., the low 'ferromagnetic' score) but does not address this leakage mechanism. The authors should either deduplicate the dataset (e.g., by composition and space group, or by structural similarity) and re-evaluate, or provide a clear analysis showing that near-duplicate train/test pairs do not drive the reported performance.","section":"Sec. 7.3 and Table 1"},{"comment":"The retrieval evaluation compares CLaSP only against the CMML baseline, which is a structure-only self-supervised method with a proxy concept-embedding procedure. The paper does not include a conventional text-search baseline, such as retrieving test structures by directly matching the query keyword against titles or against TF-IDF representations of titles/abstracts. Such a baseline would quantify the added value of the learned crossmodal embedding over simply searching the existing textual metadata—the very setting the paper claims to improve upon (Sec. 1: 'conventional text-based search systems falter when textual metadata are absent'). Adding this baseline is necessary to substantiate the claim that CLaSP enables screening of structures that lack textual annotations.","section":"Sec. 4.1 and Sec. 4.4"},{"comment":"All retrieval scores are reported for a single training run with no error bars, confidence intervals, or significance tests. The number of positives in the test set is small (e.g., 20 true positives for 'thermoelectric', 24 for 'electroluminescence'), so the ROC-AUC and AP estimates are subject to high variance. The authors should report results across multiple random seeds (or at least bootstrap confidence intervals) to establish that the differences between pre-trained and fine-tuned models, and between CLaSP and the baseline, are not due to chance.","section":"Sec. 7.1 and Table 1"},{"comment":"The bandgap sanity check (Fig. 4) uses bandgaps predicted by Crystalformer rather than experimentally measured or DFT-computed values. While this is a reasonable illustration, the text in Sec. 4.1 states that 'these distributions successfully reflect the expected bandgap ranges'—this is only as valid as the Crystalformer predictions themselves. The authors should either caveat this more explicitly or, where possible, validate a small subset against external measurements. This is a supporting result, not the central claim, but it should be framed as a model-based sanity check rather than an independent validation.","section":"Sec. 4.1 and Sec. 7.3"}],"minor_comments":[{"comment":"The section heading 'Data preprocessing and spliting' contains a typo ('spliting' should be 'splitting').","section":"Sec. 7.3"},{"comment":"The sentence 'see Sec. 7.6 for for methodological details' has a duplicated 'for'.","section":"Sec. 7.6"},{"comment":"The caption says 'pre-trained (PT) and fine-tined (FT)'—'fine-tined' should be 'fine-tuned'.","section":"Table 1 caption"},{"comment":"The AP evaluation on a balanced subset with randomly downsampled negatives is non-standard; the paper should state explicitly that AP is computed on this balanced subset and note that the absolute AP values are therefore not comparable to AP computed on the full test set.","section":"Sec. 4.1"},{"comment":"The text says the model was trained for '2000 epochs' with a global batch size of 16,384; given 324,838 training pairs, this corresponds to roughly 40,000 optimizer steps, which is plausible, but the authors should confirm that 'epochs' is the intended unit rather than 'steps'.","section":"Sec. 7.1"}],"recommendation":"major_revision","confidential_remarks":"The deduplication concern is the key technical issue: if train/test near-duplicates exist, the reported ROC-AUC values in Table 1 could substantially overstate generalization. The paper's code and data release will enable a quick check, so this is fixable within the manuscript's scope. I also recommend ensuring the authors add a conventional text-search baseline, as the current comparison against CMML does not demonstrate the claimed advantage over standard keyword search. The core idea is sound and the writing is mostly clear; the revision should focus on strengthening the evaluation rather than expanding the method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core idea is a natural extension of CLIP to materials: align crystal structure embeddings with text from publication titles and abstracts, then fine-tune on LLM-generated keywords. That is genuinely useful, and the use of bibliographic records as supervision is a step beyond prior structure-feature-text models. The retrieval evaluation is a real held-out test with title-derived labels, and the qualitative t-SNE maps plus the bandgap check with Crystalformer predictions give convergent evidence that the embeddings organize materials by semantic properties. The authors also openly discuss title noise and dataset bias, which is more honest than many papers in this area.\n\nThe stress-test note flags the real soft spot: the random 8:1:1 split of 406,048 COD entries has no deduplication. COD contains multiple entries for the same or nearly identical structures (polymorphs, re-refinements, same compound in different papers). If a test structure's near-duplicate appears in training with a title containing 'superconductor', retrieval can succeed by structure memorization rather than by learning a general structure–property mapping. The paper does not address this at all. That is a genuine omission, and it directly affects the headline numbers in Table 1. A referee should ask for a similarity-based split or at least an analysis of how many test structures have near-duplicates in training. That said, the concern is not fatal: the 'ferromagnetic' near-random result and the fact that fine-tuning shifts performance indicate the model is not purely memorizing, but the quantitative claims need this caveat.\n\nOther gaps are minor relative to the dedup issue: a single training run with no error bars, no conventional text-search baseline (e.g., BM25 on titles) to benchmark against, and predicted rather than measured bandgaps for the sanity check. The 'zero-shot' label for a model pre-trained on titles is also slightly loose, though the fine-tuned model is clearly not zero-shot.\n\nOverall, this is a competent paper with an honest limitations section and a plausible central argument. It is not groundbreaking, but it is a useful contribution to materials informatics. The deduplication flaw is fixable, and the paper would be materially improved by addressing it. I would send it to peer review and ask a referee to push on the split and baselines; the core idea is sound enough to warrant that effort. I would not cite it in my own work yet, pending a tighter evaluation.","headline":"CLaSP is a sensible application of CLIP-style contrastive learning to crystals using publication texts; the missing deduplication in the train/test split is the main soft spot, but the paper deserves serious peer review.","tokens_in":17861,"tokens_out":2440,"would_cite":false,"duration_ms":27691,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A contrastive model trained on published crystal structures and their paper titles lets users search materials by plain-text descriptions like 'superconductor'.","keywords":["CLaSP","contrastive learning","crystal structure embedding","materials informatics","text-based retrieval","crossmodal representation","zero-shot screening"],"falsifier":"A controlled test would take materials with a well-known property that their publication titles and abstracts never mention, then check whether text-based retrieval for that property still ranks those materials above random; the paper's own 'ferromagnetic' result, a pre-trained ROC-AUC of 0.4281, shows such a test can fail and would settle whether title-based supervision is comprehensive.","tokens_in":16938,"feed_emoji":"🔬","tokens_out":4674,"duration_ms":45142,"temperature":0.7,"pith_summary":"This paper introduces CLaSP, a training method that aligns crystal structures with the words used to describe them in publications. It builds a shared embedding space from over 400,000 crystal structures and their paper titles, so a user can query a database with a phrase such as 'superconductor' and retrieve structures that were never labelled with that phrase. The authors show that this literature-driven supervision captures property- and functionality-level similarities that structure-only embedding methods miss. If the approach works at scale, materials researchers could search unannotated structural databases by the properties they care about, without needing hand-curated property labels.","feed_headline":"Paper titles teach AI to search crystals by plain-text queries","feed_subtitle":"CLaSP embeds crystal structures and their publication texts in one space, so unannotated materials can be screened by properties.","key_machinery":"The central object is the CLaSP crossmodal embedding space, trained by minimizing a large-margin cosine contrastive loss between a crystal encoder and a text encoder. The crystal encoder is a CGCNN graph neural network with its property-regression head replaced by a linear projection to a 768-dimensional vector; the text encoder is a frozen SciBERT model followed by a multilayer perceptron. During pre-training the positive pairs are crystal structures and their publication titles, and during fine-tuning they are structures and LLM-generated keywords distilled from titles and abstracts. The loss pulls paired structure-text embeddings together while pushing all other pairs in the batch apart.","core_discovery":"On the paper's own terms, the central discovery is that publication metadata, specifically paper titles and abstracts via LLM-generated keywords, can serve as a comprehensive source of property and functionality supervision for crystal structures. CLaSP learns a joint embedding space in which structures and texts are matched through a large-margin cosine contrastive loss. In zero-shot screening of 40,604 test structures, queries such as 'superconductor' reach a ROC-AUC of 0.9431 before fine-tuning, and keyword-based fine-tuning raises the mean ROC-AUC from 0.7121 to 0.7804 while also organizing the embedding space into semantically coherent material clusters.","pith_inferences":["Inference: the same contrastive scheme could be trained on full-text articles, citation contexts, or Wikipedia-style descriptions, which would likely fix the blind spots the paper found for terms that titles omit, such as 'ferromagnetic'.","Inference: the embedding space could be reused for open-vocabulary classification or captioning of crystal structures, where the model generates textual descriptions from structures instead of retrieving by text.","Inference: the method's success depends on how consistently the literature uses property words; a systematic audit of keyword prevalence versus known structural property labels would show where text-driven supervision saturates."],"forward_implications":["Text-based screening lets a researcher retrieve candidate materials by writing a property description, no matter whether the target structures carry any stored text labels.","Because the crystal encoder is trained without property labels, the approach can be applied to large unannotated structure databases such as newly simulated or measured structures.","The learned embedding space doubles as a materials map: clusters correspond to intuitive families like organometallic complexes, oxides, and intermetallics.","The paper's bandgap validation shows the text-driven clusters reflect quantitative physical properties, not just vocabulary.","LLM-generated keywords from abstracts improve retrieval over titles alone, indicating that richer text sources should yield further gains."],"supporting_citations":[{"why":"Supplies the contrastive language-image pre-training paradigm and crossmodal retrieval objective that CLaSP adapts to crystal structures.","marker":"[14]"},{"why":"Provides the CGCNN crystal graph encoder architecture whose output head CLaSP replaces with an embedding projection.","marker":"[3]"},{"why":"Supplies the Crystallography Open Database, the source of over 400,000 crystal structures paired with publication titles.","marker":"[23]"},{"why":"Provides the frozen SciBERT text encoder used to embed titles and keyword captions.","marker":"[25]"},{"why":"Llama 3 generates the keyword captions from title-abstract pairs used for fine-tuning.","marker":"[26]"},{"why":"Supplies the large-margin cosine loss that CLaSP uses instead of the standard CLIP loss.","marker":"[24]"},{"why":"Provides the CMML embedding baseline, which uses structure-only contrastive learning and is compared against CLaSP.","marker":"[11]"},{"why":"Crystalformer predicts bandgaps of retrieved materials to validate that text-retrieved clusters reflect real physical properties.","marker":"[8]"},{"why":"Demonstrates that materials science abstracts contain latent knowledge accessible through word embeddings, motivating literature-driven supervision.","marker":"[53]"}],"fun_headline_variants":["AI mines paper titles to match crystals with text queries","CLaSP: one embedding space for crystal structures and papers","Literature-driven AI finds crystals by natural language","Crystal search via text: CLaSP learns from publication records","From paper titles to crystal properties: CLaSP's joint space"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that paper titles and abstracts give a comprehensive and representative description of a material's properties and functionalities; if they systematically omit a property, as the paper finds for 'ferromagnetic', the learned embedding cannot recover it.","fun_headline_variants_meta":{"raw":{"variants":["AI mines paper titles to match crystals with text queries","CLaSP: one embedding space for crystal structures and papers","Literature-driven AI finds crystals by natural language","Crystal search via text: CLaSP learns from publication records","From paper titles to crystal properties: CLaSP's joint space"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1291,"prompt_tokens":856,"completion_tokens":435,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":353}},"tokens_in":472,"tokens_out":435,"duration_ms":5250,"temperature":1.0,"reasoning_tokens":353,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:37:01.612856+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled test would take materials with a well-known property that their publication titles and abstracts never mention, then check whether text-based retrieval for that property still ranks those materials above random; the paper's own 'ferromagnetic' result, a pre-trained ROC-AUC of 0.4281, shows such a test can fail and would settle whether title-based supervision is comprehensive.","supporting_citations":[{"cited_title":"InThe 38th International Conference on Machine Learning (ICML 2021), vol","cited_arxiv_id":null,"evidence_quote":"Supplies the contrastive language-image pre-training paradigm and crossmodal retrieval objective that CLaSP adapts to crystal structures."},{"cited_title":"InThe Twelfth International Conference on Learning Representations (ICLR 2024)(2024)","cited_arxiv_id":null,"evidence_quote":"Crystalformer predicts bandgaps of retrieved materials to validate that text-retrieved clusters reflect real physical properties."}],"review_version":1}