REVIEW 5 major objections 6 minor 34 references
Iterative Corpus Refinement for Materials Property Prediction Based on Scientific Texts
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper argues that an iterative loop of selecting the most diverse scientific abstracts, training a Word2Vec model, and stopping when the model's composition-property view stabilizes can predict high-performing electrocatalyst…
desk verdict An interesting and plausibly useful selection loop, but the reported comparison overclaims: the selected-corpus model misses the measured optimum in half the systems, and the paper contradicts itself on HER numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is an iterative loop of four steps. First, Doc2Vec embeds all 6,506 abstracts and PCA reduces them to two dimensions; greedy farthest-point sampling, started from the corpus centroid, orders documents by diversity in batches of 50. Second, a skip-gram Word2Vec model is trained on the selected batches. Third, a candidate composition is represented as the linearly weighted sum of its pure-element word vectors, and its cosine similarity to the words 'dielectric' and 'conductivity' forms a two-dimensional similarity space; the centroid of all candidate compositions in that space tracks model change between iterations. Fourth, when the centroid moves less than $0.03$, training stops, and Pareto optimization picks compositions that maximize conductivity similarity while minimizing dielectric similarity (ORR, HER) or the reverse (OER).
What would settle it
Train the pipeline on a held-out material system with convergence thresholds of, say, $0.01$ and $0.05$, experimentally measure the top Pareto candidates, and check whether the $0.03$ threshold yields better current densities than fully converged or later-stopped models; if later batches improve predictions, the criterion stops too early.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that document selection guided by embedding diversity can replace corpus size: a Word2Vec model trained on a converged subset of abstracts predicts electrocatalyst compositions on the Pareto front in similarity space that match or beat the full-corpus model's predictions, with experimental current densities confirming the top candidates. For ORR systems the selected-corpus model hits minimum current densities of $-0.50$, $-0.67$, and $-0.37$ mA/cm$^2$ versus measured $-0.58$, $-0.67$, and $-0.37$; for HER it matches or approaches the best measured values; for OER it reproduces the same $6.90$ mA/cm$^2$ maximum. The paper's central message is that fewer, strategically chosen documents yield a less noisy, task-focused representation without sacrificing predictive power.
Load-bearing premise
The stopping rule assumes that when the centroid of dielectric and conductivity similarities moves less than $0.03$ after adding a batch of 50 documents, additional documents will not improve predictions of high-performing catalysts; the paper applies this same threshold to all six systems without testing its link to predictive accuracy.
Editorial extensions
If this is right
- A few hundred well-chosen abstracts can replace a 6,506-document corpus for composition-property prediction, cutting training cost and noise.
- The selected-corpus model matches or exceeds the full-corpus model's ability to find the best experimentally measured catalyst compositions in five of six material systems tested, and ties in the sixth.
- The same $0.03$ convergence threshold produces converged models for all six systems, suggesting the criterion is portable across composition spaces, though the number of batches needed varies.
- Because the loop discards documents that do not shift the property centroid, it can serve as a filter against redundant or low-quality text, including repetitive AI-generated content.
- The framework is not tied to electrocatalysis: any material system with defined target property words can be screened this way.
Reading between the lines
- The convergence criterion measures embedding stability, not predictive accuracy directly; a natural extension is to tune the threshold per system or tie it to a held-out prediction error instead of a fixed $0.03$.
- The linear-superposition representation of compositions could be tested on other additive properties, and would fail gracefully if a property depends on nonlinear interactions among elements.
- Coupling the selection loop with transformer-based scientific language models could refine the chosen documents further, since those models capture longer-range context than Word2Vec.
- One omitted check is comparison against random or similarity-based document subsets of equal size; without it, the gain cannot be fully attributed to the greedy diversity selection.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an iterative corpus refinement framework for materials property prediction from scientific texts. Starting from 6,506 open-access abstracts, the authors compute Doc2Vec embeddings, apply greedy farthest-point selection in a PCA-reduced space, and train Word2Vec models on successive batches of 50 documents. They monitor the shift of a centroid defined by cosine similarities of composition embeddings to the words 'dielectric' and 'conductivity', stopping when the shift falls below a threshold of 0.03. The resulting selected-corpus model is then used to identify Pareto-optimal electrocatalyst compositions for ORR, HER, and OER, and the measured current densities of the selected compositions are compared with those from a full-corpus model. The central claim is that the selected-corpus model matches or outperforms the full-corpus model while using far fewer documents. The manuscript includes code and data availability statements and reports results for six material systems.
Significance. If the central claim were established, the framework would be a practically useful way to reduce the cost of text-based materials discovery and to filter redundant or noisy literature. The authors are transparent about code and data availability, and the idea of using centroid stability as a stopping rule is clearly presented. However, the evidence in the manuscript does not currently support the claim that selected-corpus models match or exceed full-corpus models: the selected model misses the measured optimum in three of six systems, one HER comparison is internally contradictory, and the evaluation metric is confounded by large differences in the number of retained Pareto-front entries. The framework's general value is therefore plausible but unproven at this stage.
major comments (5)
- [Section III B, Table II] The central claim that the iterative selection 'successfully predicts the highest performing compositions' is not supported by Table II. The selected-corpus model misses the measured optimum in AgPdPt (-0.50 vs -0.58 mA/cm²), AgAuPdPtRh (-1.11 vs -1.13 mA/cm²), and AgAuPdPtRu (-1.27 vs -1.49 mA/cm²). In AgAuPdPtRu, the selected model is also worse than the full-corpus model (-1.27 vs -1.41 mA/cm²), which directly contradicts the statement that performance is not compromised. Please soften the claim or provide a corrected evaluation that addresses these failures.
- [Section III C, Tables I and II] The evaluation metric is the minimum (or maximum) current density over the retained Pareto-front entries, but the number of retained entries differs by an order of magnitude across models and systems (e.g., AgPdPtRu: 4 vs 27; AgAuPdPtRu: 3 vs 23; AgPdPt: 63 vs 11). Comparing extrema over sets of very different sizes is biased: a smaller candidate set has fewer opportunities to include rare high-performing compositions, so the reported minima are not directly comparable. Please report fixed top-k metrics, rank-based measures, or full distributions over the retained sets, and justify the choice of k.
- [Section III B, HER paragraph] The HER results contain an internal contradiction. The first paragraph reports selected-corpus values of -1.13 and -1.44 mA/cm² and full-corpus values matching the measurements, while the second paragraph reports selected-corpus values of -1.11 and -1.27 mA/cm² and Figure 4 and Table II agree with the second set. Both sets of numbers cannot be correct; the discrepancy must be resolved and the affected text, figure, and table reconciled before any quantitative conclusion about HER is drawn.
- [Section II B, Convergence Criterion] The stopping rule is a heuristic threshold of 0.03 applied to the shift of a centroid defined in the same dielectric/conductivity similarity space that is later used for Pareto optimization. This makes convergence self-referential with respect to the chosen target properties: the criterion measures internal consistency of the two chosen similarities, not whether additional documents improve predictive accuracy. Moreover, the same threshold is applied to all six material systems with no sensitivity analysis. Please provide experiments varying the threshold, batch size, and Word2Vec hyperparameters, and show that the conclusions are robust to these choices.
- [Section III, all comparisons] No error bars, repeated runs, or statistical tests are reported. Word2Vec and Doc2Vec training are stochastic, and the greedy selection depends on the initial Doc2Vec embedding; a single run cannot support the claim that the selected-corpus model 'does not compromise performance'. Please include multiple independent runs with different random seeds and report the variability of the selected compositions and current densities, or formally justify why variability is negligible.
minor comments (6)
- [Section II A] There is a typo in the sentence about the Scopus API: 'Scopus and and limit ourselves' should read 'Scopus and limit ourselves'.
- [Section II B] The notation 'RA20B20 = 0.2RA + 0.8RB' is confusing because the subscript does not match the weights; a clearer notation such as R_{A_0.2 B_0.8} = 0.2 R_A + 0.8 R_B is recommended.
- [Section III B] The two consecutive HER paragraphs are largely redundant and should be merged after the numerical discrepancy is fixed.
- [Figure 3] The order of material systems on the x-axis (AgPdPt, AgPdPtRu, AgPdRu) differs from the order in the text (AgPdPt, AgPdRu, AgPdPtRu); please align the order for readability.
- [References] Reference [24], which is the basis for the reproducibility claim, lists only a title and year without a URL or persistent identifier; please provide the repository location or DOI.
- [Throughout] There are minor typos such as 'elments' in Section II B and inconsistent capitalization of 'word2vec' versus 'Word2Vec'; a careful proofreading pass would remove these distractions.
Circularity Check
No significant circularity: the central claim is validated by external experimental measurements, and the self-citation used for the Pareto objective is not the sole support for the claim.
full rationale
The paper's derivation chain is: collect 6506 documents, build Doc2Vec embeddings, greedily select diverse documents in batches of 50, train Word2Vec on each growing subset, compute composition-to-property similarities S_dielectric and S_conductivity, monitor the centroid of these similarities until it moves less than 0.03, and then use the converged model to construct a Pareto front in the same similarity space. The final 'predictions' are the Pareto-front compositions, and these are compared against experimentally measured current densities from independent datasets. No parameter is fitted to the experimental current densities and then renamed as a prediction; the experimental values are used only for validation. The convergence criterion is a heuristic defined on the same similarity space used for the Pareto front, which is self-referential in a weak sense, but it is not definitional: convergence is stability of the centroid, while prediction is the Pareto extreme of the same similarities, and the actual claim of predictive quality is tested against laboratory measurements. The Pareto optimization method is cited from the authors' prior work [26], a self-citation, but the central claim does not rest solely on that citation because the present paper includes external experimental validation across six material systems and three reactions. Issues such as the confounded comparison due to differing numbers of retained Pareto entries and the internal numeric inconsistency in the HER results are correctness and evaluation concerns, not circularity. Therefore no circular step meeting the required evidentiary standard is present.
Assumptions & free parameters
free parameters (6)
- Convergence threshold =
0.03 (Euclidean distance in centroid space)
- Batch size =
50 documents
- Word2Vec and Doc2Vec hyperparameters =
vector size 200, window 5, skip-gram, hierarchical softmax
- PCA dimensionality =
2
- Target properties =
"dielectric" and "conductivity"
- Pareto optimization directions =
maximize conductivity and minimize dielectric for ORR and HER; reverse for OER
assumptions (5)
- domain assumption Word2Vec embeddings trained on scientific abstracts capture meaningful composition-property relationships.
- ad hoc to paper A material composition can be represented by a linearly weighted sum of its element word embeddings.
- ad hoc to paper Cosine similarity to the words 'conductivity' and 'dielectric' is a valid objective for ranking electrocatalyst performance.
- domain assumption Doc2Vec embeddings, after PCA to 2D, preserve enough semantic geometry for farthest-point sampling to select diverse documents.
- ad hoc to paper The centroid stability criterion is a valid proxy for predictive convergence.
Cite this review
Pith. "Pith review of Iterative Corpus Refinement for Materials Property Prediction Based on Scientific Texts." pith.science (2026). https://pith.science/paper/UZSQWWCT
@misc{pith2026250521646,
author = {Pith},
title = {Pith review of: Iterative Corpus Refinement for Materials Property Prediction Based on Scientific Texts},
year = {2026},
howpublished = {\url{https://pith.science/paper/UZSQWWCT}},
note = {Machine review of arXiv:2505.21646}
}
read the original abstract
The discovery and optimization of materials for specific applications is hampered by the practically infinite number of possible elemental combinations and associated properties, also known as the `combinatorial explosion'. By nature of the problem, data are scarce and all possible data sources should be used. In addition to simulations and experimental results, the latent knowledge in scientific texts is not yet used to its full potential. We present an iterative framework that refines a given scientific corpus by strategic selection of the most diverse documents, training Word2Vec models, and monitoring the convergence of composition-property correlations in embedding space. Our approach is applied to predict high-performing materials for oxygen reduction (ORR), hydrogen evolution (HER), and oxygen evolution (OER) reactions for a large number of possible candidate compositions. Our method successfully predicts the highest performing compositions among a large pool of candidates, validated by experimental measurements of the electrocatalytic performance in the lab. This work demonstrates and validates the potential of iterative corpus refinement to accelerate materials discovery and optimization, offering a scalable and efficient tool for screening large compositional spaces where reliable data are scarce or non-existent.
Figures
Reference graph
Works this paper leans on
-
[26]
Y. Goldberg and O. Levy, word2vec explained: deriv- ing mikolov et al.’s negative-sampling word-embedding method (2014), arXiv:1402.3722 [cs.CL]
arXiv 2014
-
[1]
Reduced Noise: Unnecessary or very similar in- formation is left out, preventing additional fuzzi- ness of key semantic signals related to electrocat- alytic properties
-
[2]
Computational Efficiency: Fewer documents result in faster training and analysis
-
[3]
F ocus: The final Word2Vec embeddings capture terminology tailored to the studied compositions, reactions, and the properties dielectric and conduc- tivity. B. Practical Impact on Electrocatalyst Screening By identifying fewer yet relevant documents for train- ing, this approach can speed up the overall pipeline of electrocatalyst design in particular for...
-
[4]
A. Fujishima, T. N. Rao, and D. A. Tryk, Titanium diox- ide photocatalysis, Journal of Photochemistry and Pho- tobiology C: Photochemistry Reviews 1, 1 – 21 (2000), cited by: 7896
work page 2000
-
[5]
A. Geim and K. Novoselov, The rise of graphene, Nature Materials 6, 183 – 191 (2007), cited by: 36635
work page 2007
-
[6]
J.-W. Yeh, S.-K. Chen, S.-J. Lin, J.-Y. Gan, T.-S. Chin, T.-T. Shun, C.-H. Tsau, and S.-Y. Chang, Nanostruc- tured high-entropy alloys with multiple principal ele- ments: Novel alloy design concepts and outcomes, Ad- vanced Engineering Materials 6, 299 – 303 (2004), cited by: 11123
work page 2004
-
[7]
P. Ferreira, G. La O’, Y. Shao-Horn, D. Morgan, R. Makharia, S. Kocha, and H. Gasteiger, Instability of pt/c electrocatalysts in proton exchange membrane fuel cells: A mechanistic investigation, Journal of the Elec- trochemical Society 152, A2256–A2271 (2005), cited by: 1430; All Open Access, Bronze Open Access
work page 2005
Show all 34 references
-
[8]
Ezugwu, J
E. Ezugwu, J. Bonney, and Y. Yamane, An overview of the machinability of aeroengine alloys, Journal of Mate- rials Processing Technology 134, 233 – 253 (2003), cited by: 995
2003
-
[9]
Miracle and O
D. Miracle and O. Senkov, A critical review of high en- tropy alloys and related concepts, Acta Materialia 122, 448 – 511 (2017), cited by: 6137; All Open Access, Hy- brid Gold Open Access
2017
-
[10]
Zerdoumi, A
R. Zerdoumi, A. Ludwig, and W. Schuhmann, High en- tropy intermetallic compounds: A discovery platform for structure–property correlations and materials design principles in electrocatalysis, Current Opinion in Elec- trochemistry 48, 101590 (2024)
2024
-
[11]
Huang, J.-W
P.-K. Huang, J.-W. Yeh, T.-T. Shun, and S.-K. Chen, Multi-principal-element alloys with improved oxidation and wear resistance for thermal spray coating, Advanced Engineering Materials 6, 74 – 78 (2004), cited by: 695
2004
-
[12]
T. A. Batchelor, J. K. Pedersen, S. H. Winther, I. E. Castelli, K. W. Jacobsen, and J. Rossmeisl, High-entropy alloys as a discovery platform for electrocatalysis, Joule 3, 834 (2019)
2019
-
[13]
Larcher and J.-M
D. Larcher and J.-M. Tarascon, Towards greener and more sustainable batteries for electrical energy storage, Nature Chemistry 7, 19 – 29 (2015), cited by: 6265
2015
-
[14]
A. J. Ragauskas, C. K. Williams, B. H. Davison, G. Britovsek, J. Cairney, C. A. Eckert, W. J. Freder- ick Jr., J. P. Hallett, D. J. Leak, C. L. Liotta, J. R. Mielenz, R. Murphy, R. Templer, and T. Tschaplinski, The path forward for biofuels and biomaterials, Science 311, 484 –...
2006
-
[15]
A. Jain, Y. Shin, and K. A. Persson, Computational pre- dictions of energy materials using density functional the- ory, NATURE REVIEWS MATERIALS 1, 10.1038/na- trevmats.2015.4 (2016)
2016 doi
-
[16]
J. E. Saal, S. Kirklin, M. Aykol, B. Meredig, and C. Wolverton, Materials design and discovery with high- throughput density functional theory: The open quan- tum materials database (oqmd), JOM 65, 1501 – 1509 (2013), cited by: 1733
2013
-
[17]
Schmidt, T
J. Schmidt, T. F. Cerqueira, A. H. Romero, A. Loew, F. J¨ ager, H.-C. Wang, S. Botti, and M. A. Marques, Improving machine-learning models in materials science through large datasets, Materials Today Physics 48, 101560 (2024)
2024
-
[18]
Y. Liu, T. Zhao, W. Ju, and S. Shi, Materials discov- ery and design using machine learning, Journal of Ma- teriomics 3, 159 – 177 (2017), cited by: 913; All Open Access, Gold Open Access
2017
-
[19]
Raccuglia, K
P. Raccuglia, K. C. Elbert, P. D. F. Adler, C. Falk, M. B. Wenny, A. Mollo, M. Zeller, S. A. Friedler, J. Schrier, and A. J. Norquist, Machine-learning-assisted materials discovery using failed experiments, Nature 533, 73 – 76 (2016), cited by: 1193
2016
-
[20]
Tshitoyan, J
V. Tshitoyan, J. Dagdelen, L. Weston, A. Dunn, Z. Rong, O. Kononova, K. A. Persson, G. Ceder, and A. Jain, Unsupervised word embeddings capture latent knowledge from materials science literature, Nature 571, 95 – 98 (2019), cited by: 681
2019
-
[21]
Mikolov, K
T. Mikolov, K. Chen, G. Corrado, J. Dean, L. Sutskever, and G. Zweig, word2vec, URL https://code. google. com/p/word2vec 22, 795 (2013)
2013
-
[22]
Le and T
Q. Le and T. Mikolov, Distributed representations of sen- tences and documents (2014), cited by: 2114
2014
-
[23]
Shumailov, Z
I. Shumailov, Z. Shumaylov, Y. Zhao, N. Papernot, R. Anderson, and Y. Gal, Ai models collapse when trained on recursively generated data, Nature 631, 755 (2024)
2024
-
[24]
Shumailov, Z
I. Shumailov, Z. Shumaylov, Y. Zhao, N. Papernot, R. Anderson, and Y. Gal, Ai models collapse when trained on recursively generated data, Nature 631, 755 – 759 (2024), cited by: 36; All Open Access, Green Open Access, Hybrid Gold Open Access
2024
-
[25]
Bilgin and ˙I
M. Bilgin and ˙I. F. S ¸ent¨ urk, Sentiment analysis on twit- ter data with semi-supervised doc2vec, in 2017 Interna- tional Conference on Computer Science and Engineering (UBMK) (2017) pp. 661–666
2017
-
[27]
iterative corpus re- finement for materials property prediction based on sci- entific texts
L. Zhang and M. Stricker, Code for “iterative corpus re- finement for materials property prediction based on sci- entific texts” (2025)
2025
-
[28]
Zhang and M
L. Zhang and M. Stricker, Matnexus: A comprehensive text mining and analysis suite for materials discovery, 8 SoftwareX 26, 101654 (2024), cited by: 0; All Open Ac- cess, Green Open Access
2024
-
[29]
Zhang and M
L. Zhang and M. Stricker, Electrocatalyst discovery through text mining and multi-objective optimization (2025), arXiv:2502.20860 [cond-mat.mtrl-sci]
2025 arXiv
-
[30]
Banko, O
L. Banko, O. Krysiak, W. Schumann, and A. Ludwig, Electrochemical activity of several compositions in the system ag-pd-pt-ru for the oxygen reduction reaction in 0.05 m koh solution ph 12.5, 10.5281/zenodo.13992986 (2024)
2024 doi
-
[31]
Thelen, R
F. Thelen, R. Zehl, N. Limani, W. Schuhmann, and A. Ludwig, High-throughput seccm and edx data for the hydrogen evolution reaction in ag-au-pd-pt-ru and ag-au-pd-pt-rh thin-film materials libraries, 10.5281/zen- odo.14959252 (2025)
2025 doi
-
[32]
Thelen, R
F. Thelen, R. Zehl, R. Zerdoumi, J. L. B¨ urgel, W. Schuh- mann, and A. Ludwig, Dataset - accelerating combinato- rial electrocatalyst discovery with bayesian optimization: A case study in the quaternary system ni-pd-pt-ru for the oxygen evolution reaction, 10.5281/zenodo.1489...
2025 doi
-
[33]
Zhang, Y
Y. Zhang, Y. Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y. Zhang, Y. Chen, L. Wang, A. T. Luu, W. Bi, F. Shi, and S. Shi, Siren’s song in the ai ocean: A sur- vey on hallucination in large language models (2023), arXiv:2309.01219 [cs.CL]
2023 arXiv
-
[34]
Trewartha, N
A. Trewartha, N. Walker, H. Huo, S. Lee, K. Cruse, J. Dagdelen, A. Dunn, K. A. Persson, G. Ceder, and A. Jain, Quantifying the advantage of domain-specific pre-training on named entity recognition tasks in mate- rials science, Patterns 3, 100488 (2022). 9 TABLE I. Statistical ...
2022
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.