REVIEW 2 major objections 5 minor 40 references
Bridging AI and Science: Implications from a Large-Scale Literature Analysis of AI4Science
T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper claims that AI4Science research is unevenly integrated, with many scientific problems and AI methods underexplored, and that link prediction can propose new pairings.
desk verdict Useful large-scale dataset and a genuinely new semantic map of AI4Science, but the headline method-level findings are probably biased by the single-'main-method' extraction schema. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying machinery is an extraction-to-network pipeline: a large language model (GPT-4o mini) extracts scientific problems, AI methods, and usage descriptions from titles and abstracts; semantic embeddings with dimensionality reduction and density-based clustering organize them into 390 problem clusters and 355 method clusters; and the clusters become the two node types of a bipartite graph whose edges are publications. This graph makes the gap between AI and science measurable as uneven cluster sizes, hub-and-peripheral degree structure, and novel-link prediction targets.
What would settle it
Re-run the cluster-level analysis on a sample of AI-conference papers that domain experts relabel by hand, using the same embedding, clustering, and regression steps; if hand labels move clusters such as Neural Network Challenges or Urban Traffic Management below the regression line or promote current under-explored clusters to well-explored, the paper's top rankings change. A simpler check is to recompute the under-explored versus well-explored partition under an alternative AI4Science classifier and see whether the top-ten lists in Table 12 survive.
Extended reading notes
Core claim
The paper's central discovery is a landscape, not a single theorem: in the decade 2014-2024, only 7,542 of 162,656 papers from these venues qualify as AI4Science, and their distribution is markedly uneven. Under-explored scientific problem clusters include Asymmetric Synthesis, Magnetic Materials and Spintronics, Forests and Climate Change, Viral Mechanisms, and Cancer Drug Resistance; underused AI methods include Attention Mechanisms, Gradient-Based Methods, Graph Embedding, and Transformer Architectures. The bipartite graph of problem clusters and method clusters has heavy-tailed degree distributions, so a few hub problems and methods account for most connectivity. Link prediction experiments show that LLM-based models, especially paper-level generative prediction with retrieval augmentation, find novel links beyond the 683 actually observed in 2023-2024, which the authors take as evidence that the gap is bridgeable.
Load-bearing premise
The quantitative landscape depends on the LLM's binary judgment that a paper is AI4Science; on AI-conference papers that classifier has precision 0.516 and F1 0.647, and human annotators agree only at F1 0.632, so if classification errors are systematically concentrated in certain clusters the under-explored and hub findings could shift.
Editorial extensions
If this is right
- Under-explored problem clusters such as Asymmetric Synthesis, Forests and Climate Change, and Viral Mechanisms are the places where new AI applications could have the least competition.
- Underused methods such as Attention Mechanisms, Graph Embedding, Contrastive Learning, and Transformer Architectures have room to be carried into scientific domains where they are rarely applied.
- LLM-based link prediction, particularly paper-level generation with retrieval augmentation, can surface hundreds of novel problem-method pairings at small K, giving researchers a concrete shortlist.
- Because the science and AI communities prioritize different problems and methods, bridging efforts can be targeted: scientists could adopt broader general AI methods while AI researchers take up domain-specific scientific problems.
Reading between the lines
- An implication the paper leaves implicit is that its under-explored lists double as a research agenda: funding agencies or labs could use the cluster rankings to pick problems where AI has low prior penetration.
- A testable extension is to track whether AI-proposed novel links that later appear in publications are cited more or less than random problem-method pairings, which would measure whether the predicted directions are actually fruitful.
- The same pipeline could be rerun on full texts or a broader set of venues; that would test whether the under-explored labels are artifacts of restricting analysis to titles, abstracts, and top journals.
- Because the degree distribution is heavy-tailed and log-normal-like, a dynamic version of the dataset could test whether new links preferentially attach to existing hubs, which would indicate whether the imbalance is self-reinforcing.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a large-scale literature analysis of AI4Science. The authors use GPT-4o mini to extract the scientific problem, the main AI method, and the usage description from titles and abstracts of 162,656 publications in top science journals and AI conferences (2014-2024). The extracted problem and method descriptions are embedded, projected, and clustered into 390 scientific problem clusters and 355 AI method clusters, which are then used to construct a bipartite graph connecting problem clusters to method clusters. Based on this dataset, the paper reports three main findings: (F1) different subdomains show varying engagement in AI4Science, leaving many problems and methods underexplored; (F2) connectivity in the bipartite graph is highly imbalanced with hub and peripheral nodes; and (F3) science and AI communities prioritize different problems and methods. The paper then formulates link prediction between problem and method clusters and compares conventional baselines (Katz index, node2vec) with LLM-based prediction, including a gpt-3.5 leakage check. Code and data are promised on GitHub.
Significance. If the findings hold, the dataset and the quantitative landscape it enables would be a useful community resource, providing a more scalable and semantically richer alternative to keyword-based science-of-science analyses. The link prediction section is a reasonable exploratory step, with several baselines and a leakage check using gpt-3.5. The paper is clearly written and the authors are transparent about many limitations, including selection of top venues, reliance on titles/abstracts, and the ground-truth evaluation of link prediction. The value of the paper depends on the validity of the extraction and classification pipeline, which is the main source of risk.
major comments (2)
- [Section 2.2, Appendix A.2, Figure 1] The extraction schema restricts each publication to exactly one 'main AI method,' as shown in the prompt in Appendix A.2 ('a keyword or a keyphrase that summarizes the main method used in this paper'). This systematically excludes methods that appear as components, training techniques, or secondary contributions. The AlphaFold example in Figure 1 illustrates the issue: the method is labeled 'Neural network-based model' even though the work is centrally based on attention/transformer machinery. The list of 'underutilized' methods in Figure 3b and Table 12 (Attention Mechanisms, Transformer Architectures, Gradient-Based Methods, Regularization Methods, Contrastive Learning) is exactly what this single-label schema would predict, because these methods are rarely stated as the main method of a paper even when they are essential. This bias is load-bearing for F1 and F2 as they apply to AI methods: the degree distribution in Figure 4c and the hub list in Table 13 may also be distorted by the tendency to map specific architectures to generic headers such as 'Machine Learning' and 'Deep Learning Models.' I would ask the authors to either extract multiple methods per paper (e.g., all AI methods mentioned), or to re-frame the findings as being about 'main methods' only and to add a validation on a subsample where multiple methods are annotated.
- [Appendix A.2, Table 9] The AI4Science binary classifier has precision 0.516 on AI-conference papers (F1 0.647), and the human annotators themselves agree only at F1 0.632. Since AI conferences contribute 3,539 of the 7,542 AI4Science positives (Tables 6 and 7), nearly half of the positive labels from that domain could be false positives. No cluster-level precision or recall is reported, and no sensitivity analysis is provided to show that the findings in Section 3 (F1-F3) are robust to classification noise. If false positives are more common in some problem or method clusters (e.g., CS-adjacent topics such as 'Urban Traffic Management' or 'Electronic Health Data Challenges'), the under-explored rankings, hub lists, and community discrepancy results could change. I recommend reporting stratified precision/recall by cluster or venue, and re-running the main descriptive analyses either with a precision-weighted correction or on a subset where the classifier is more reliable.
minor comments (5)
- [Section 3.3] The sentence 'Similarly, AI methods frequently used by the science community are applied across a broader range of scientific challenges' is repeated verbatim in the same paragraph; one occurrence should be removed.
- [Figure 4 caption] The caption contains a typo: 'the resulting isolated noes are hidden' should read 'nodes.'
- [Table 16 caption] The caption says 'gpt-3.5-turbo-012' but the model name should be 'gpt-3.5-turbo-0125'.
- [Appendix A.2] The statement that the high recall (0.866) 'provides a guarantee of avoiding missing important AI4Science work' is too strong given the low precision; a more balanced phrasing should acknowledge that many false positives are also included.
- [Appendix D] The paper refers to an 'anonymous repository' in the appendix while the abstract and main text give a named GitHub URL; please reconcile this inconsistency.
Circularity Check
No circular derivation: the findings are descriptive statistics over the curated corpus, and the only self-referential element is the use of the authors' MASSW dataset as a data source, which is not load-bearing for the central claims.
full rationale
This paper reports descriptive, data-driven findings rather than a derivation whose outputs are recycled as inputs. The main pipeline (Sec. 2.2-2.4) uses LLM extraction, semantic embedding, clustering, and bipartite-graph analysis to produce findings F1-F3 in Sec. 3.4; the regression lines in Fig. 3 are descriptive summaries of cluster sizes versus AI4Science publication counts, not fitted parameters later renamed as predictions. Link prediction in Sec. 4 is trained on publications from 2014-2022 and evaluated against held-out publications from 2023-2024, so the evaluation is not forced by construction. The single-'main method' extraction schema and the AI-conference AI4Science classifier precision of 0.516 are measurement and validity concerns, but they are not circularity: no target finding is used to define its own evidence. The only self-referential element is that AI-venue publications are sourced from MASSW [37], a dataset on which one of the present authors is a co-author; however, MASSW is a separately released dataset and the present paper contributes new extraction, clustering, and link-prediction layers on top of it. That self-citation is minor and not load-bearing for the central findings, so the appropriate circularity score is low.
Assumptions & free parameters
free parameters (4)
- Katz index penalty alpha =
0.1
- HDBSCAN clustering parameters =
not reported
- LargeVis projection parameters =
not reported
- RAG retrieval count n =
5
assumptions (5)
- domain assumption Titles and abstracts contain sufficient information to identify the primary scientific problem and AI method of each paper
- domain assumption LLM-based AI4Science classification is accurate enough for quantitative comparison
- domain assumption The 12 selected venues represent the AI4Science landscape
- domain assumption Embedding, LargeVis, and HDBSCAN preserve semantically meaningful clusters
- domain assumption Published future links are a valid ground truth for link prediction
Cite this review
Pith. "Pith review of Bridging AI and Science: Implications from a Large-Scale Literature Analysis of AI4Science." pith.science (2026). https://pith.science/paper/J7DCYGEC
@misc{pith2026241209628,
author = {Pith},
title = {Pith review of: Bridging AI and Science: Implications from a Large-Scale Literature Analysis of AI4Science},
year = {2026},
howpublished = {\url{https://pith.science/paper/J7DCYGEC}},
note = {Machine review of arXiv:2412.09628}
}
read the original abstract
Artificial Intelligence has proven to be a transformative tool for advancing scientific research across a wide range of disciplines. However, a significant gap still exists between AI and scientific communities, limiting the full potential of AI methods in driving broad scientific discovery. Existing efforts in identifying and bridging this gap have often relied on qualitative examination of small samples of literature, offering a limited perspective on the broader AI4Science landscape. In this work, we present a large-scale analysis of the AI4Science literature, starting by using large language models to identify scientific problems and AI methods in publications from top science and AI venues. Leveraging this new dataset, we quantitatively highlight key disparities between AI methods and scientific problems, revealing substantial opportunities for deeper AI integration across scientific disciplines. Furthermore, we explore the potential and challenges of facilitating collaboration between AI and scientific communities through the lens of link prediction. Our findings and tools aim to promote more impactful interdisciplinary collaborations and accelerate scientific discovery through deeper and broader AI integration. Our code and dataset are available at: https://github.com/charles-pyj/Bridging-AI-and-Science.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Mohammad Al Hasan, Vineet Chaoji, Saeed Salem, and Mohammed Zaki. 2006. Link prediction using supervised learning. In SDM06: workshop on link analysis, counter-terrorism and security, Vol. 30. 798–805
work page 2006
-
[2]
Alexandre Blanco-Gonzalez, Alfonso Cabezon, Alejandro Seco-Gonzalez, Daniel Conde-Torres, Paula Antelo-Riveiro, Angel Pineiro, and Rebeca Garcia-Fandino
-
[3]
Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. 2013. Density-based clustering based on hierarchical density estimates. In Pacific-Asia conference on knowledge discovery and data mining . Springer, 160–172
2013
-
[4]
Minkyu Choi, Kuan Han, Xiaokai Wang, Yizhen Zhang, and Zhongming Liu
-
[5]
Suresh Dara, Swetha Dhamercherla, Surender Singh Jadav, CH Madhu Babu, and Mohamed Jawed Ahsan. 2022. Machine learning in drug discovery: a review. Artificial intelligence review 55, 3 (2022), 1947–1999
work page 2022
-
[6]
A Dual-Stream Neural Network Explains the Functional Segregation of Dorsal and Ventral Visual Pathways in Human Brains. arXiv:2310.13849 [cs.CV] https://arxiv.org/abs/2310.13849
-
[7]
Jian Gao and Dashun Wang. 2023. Quantifying the benefit of artificial intelligence for scientific research. arXiv preprint arXiv:2304.10578 (2023)
work page Pith review arXiv 2023
-
[8]
Eamon Duede, William Dolan, André Bauer, Ian Foster, and Karim Lakhani. 2024. Oil & water? diffusion of ai within and across scientific fields. arXiv preprint arXiv:2405.15828 (2024)
arXiv 2024
Show all 40 references
-
[9]
Kai Guo, Zhenze Yang, Chi-Hua Yu, and Markus J Buehler. 2021. Artificial intelligence and machine learning in design of mechanical materials. Materials Horizons 8, 4 (2021), 1153–1172
2021
-
[10]
Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining . 855–864
2016
-
[11]
Leo Katz. 1953. A new status index derived from sociometric analysis. Psychome- trika 18, 1 (1953), 39–43
1953
-
[12]
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. 2021. Highly accurate protein structure prediction with Al- phaFold. nature 596, 7873 (2021), 583–589
2021
-
[13]
Georgios Leontidis. 2024. Science in the age of AI: How artificial intelligence is changing the nature and method of scientific research. (2024)
2024
-
[14]
Kuenneth and R
C. Kuenneth and R. Ramprasad. 2023. polyBERT: a chemical language model to enable fully machine-driven ultrafast polymer informatics. Nature Communica- tions 14 (2023), 4099. doi:10.1038/s41467-023-39868-6
2023 doi
-
[15]
Jiali Li, Kaizhuo Lim, Haitao Yang, Zekun Ren, Shreyaa Raghavan, Po-Yen Chen, Tonio Buonassisi, and Xiaonan Wang. 2020. AI applications through the whole life cycle of material discovery. Matter 3, 2 (2020), 393–432
2020
-
[16]
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing...
2020
-
[17]
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81
2004
-
[18]
Junwei Li, Yanhao Lin, Thomas Meier, Zhipan Liu, Wei Yang, Ho kwang Mao, Shengcai Zhu, and Qingyang Hu. 2023. Silica-water su- perstructure and one-dimensional superionic conduit in Earth’s man- tle. Science Advances 9, 35 (2023), eadh3784. doi:10.1126/sciadv.adh3784 arXiv:htt...
2023 doi
-
[19]
Qiaozhu Mei, Xuehua Shen, and ChengXiang Zhai. 2007. Automatic labeling of multinomial topic models. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining . 490–499
2007
-
[20]
Kit-Kay Mak, Yi-Hang Wong, and Mallikarjuna Rao Pichika. 2023. Artificial intelligence in drug discovery and development. Drug Discovery and Evaluation: Safety and Pharmacokinetic Assays (2023), 1–38
2023
-
[21]
Şükrü Demir İnan Özer, Günce Keziban Orman, and Vincent Labatut. 2024. Link Prediction in Bipartite Networks. arXiv preprint arXiv:2406.06658 (2024)
2024 arXiv
-
[22]
Meller, M
A. Meller, M. Ward, J. Borowsky, M. Kshirsagar, J. M. Lotthammer, F. Oviedo, J. L. Ferres, and G. R. Bowman. 2023. Predicting locations of cryptic pockets from single protein structures using the PocketMiner graph neural network. Nature Communications 14, 1 (Mar 2023), 1177. d...
2023 doi
-
[23]
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining . 701–710
2014
-
[24]
Matjaž Perc. 2010. Growth and structure of Slovenia’s scientific collaboration network. Journal of Informetrics 4, 4 (2010), 475–482
2010
-
[25]
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. BLEURT: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696 (2020)
2020 arXiv
-
[26]
Rizwan Qureshi, Muhammad Irfan, Taimoor Muzaffar Gondal, Sheheryar Khan, Jia Wu, Muhammad Usman Hadi, John Heymach, Xiuning Le, Hong Yan, and Tanvir Alam. 2023. AI in drug discovery and its clinical relevance. Heliyon 9, 7 (2023)
2023
-
[27]
Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2024. Can LLMs Generate Novel Research Ideas? arXiv preprint arXiv:2409.04109 (2024)
2024 arXiv
-
[28]
Wuxin Sha, Yaqing Guo, Qing Yuan, Shun Tang, Xinfang Zhang, Songfeng Lu, Xin Guo, Yuan-Cheng Cao, and Shijie Cheng. 2020. Artificial intelligence to power the future of materials science and engineering. Advanced Intelligent Systems 2, 4 (2020), 1900143
2020
-
[29]
Huan Sun, Mudhakar Srivatsa, Shulong Tan, Yang Li, Lance M Kaplan, Shu Tao, and Xifeng Yan. 2014. Analyzing expert behaviors in collaborative networks. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining . 1486–1495
2014
-
[30]
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, and Tao Yu. 2022. One embedder, any task: Instruction-finetuned text embeddings. arXiv preprint arXiv:2212.09741 (2022)
2022 arXiv
-
[31]
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei
-
[32]
Jian Tang, Jingzhou Liu, Ming Zhang, and Qiaozhu Mei. 2016. Visualizing large-scale and high-dimensional data. In Proceedings of the 25th international conference on world wide web . 287–297
2016
-
[33]
Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. 2023. Scientific discovery in the age of artificial intelligence. Nature 620, 7972 (2023), 47–60
2023
-
[34]
Yan Wang, Jie Liang, and Jianguo Lu. 2014. Discover hidden web properties by random walk on bipartite graph. Information retrieval 17 (2014), 203–228
2014
-
[35]
Demival Vasques Filho and Dion RJ O’Neale. 2018. Degree distributions of bipartite networks and their projections. Physical Review E 98, 2 (2018), 022307
2018
-
[36]
Yang Yang, Ryan N Lichtenwalter, and Nitesh V Chawla. 2015. Evaluating link prediction methods. Knowledge and Information Systems 45 (2015), 751–782
2015
-
[37]
N/A" if no relevant information can be found in the paper title and abstract. {
Xingjian Zhang, Yutong Xie, Jin Huang, Jinge Ma, Zhaoying Pan, Qijia Liu, Ziyang Xiong, Tolga Ergen, Dongsub Shim, Honglak Lee, et al. 2024. MASSW: A New Dataset and Benchmark Tasks for AI-Assisted Scientific Workflows. arXiv preprint arXiv:2406.06357 (2024). Bridging AI and S...
2024 arXiv
-
[38]
Ruoxi Xu, Yingfei Sun, Mengjie Ren, Shiguang Guo, Ruotong Pan, Hongyu Lin, Le Sun, and Xianpei Han. 2024. AI for social science and social science of AI: A survey. Information Processing & Management 61, 3 (2024), 103665
2024
-
[2015]
In Proceedings of the 24th international conference on world wide web
Line: Large-scale information network embedding. In Proceedings of the 24th international conference on world wide web . 1067–1077
-
[2023]
Pharmaceuticals 16, 6 (2023), 891
The role of AI in drug discovery: challenges, opportunities, and strategies. Pharmaceuticals 16, 6 (2023), 891
2023
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.