{"id":"d4126e34-d7ec-4ee9-a4cd-de6eef3b5805","arxiv_id":"2412.09628","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Using LLM-extracted labels from 162,656 papers, this analysis shows that AI4Science connections are sparse and uneven, with a few hub areas dominating while many scientific problems and AI methods remain weakly linked.","lead":"This paper maps ten years of AI-for-science research using 162,656 papers from top science and AI venues, with large language models labeling each paper's scientific problem and AI method. It finds that these connections are uneven, with many problems and methods rarely paired, and explores whether link prediction can suggest useful new pairings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The under-explored AI-methods finding (F1) and method-hub structure (F2) may be artifacts of the single-\"main-method\"-per-paper extraction schema: methods like attention, transformers, and gradient descent are systematically undercounted because they are rarely the stated main method.","rationale":"The Reader's weakest assumption about binary classification noise is valid and well-documented (Appendix A.2, Table 9), but I identify a different, more upstream measurement issue: the single-\"main-method\" extraction protocol. This issue directly threatens the method-specific parts of the central claim, especially F1's \"underutilized AI methods\" list and F2's method-degree hubs, and it would persist even with a perfect binary classifier. The AlphaFold example itself illustrates the mechanism: a transformer-based system is labeled as a generic \"neural network,\" so the extraction cannot see the attention/transformer methods that the paper actually uses. My proposed test is concrete and feasible: a multi-label re-annotation of a modest sample would reveal whether the under-explored method findings are stable or artifacts. I therefore recommend keeping the conditional verdict, but the revision condition should include validating the extraction schema's cardinality, not only the binary classification. The Reader's concern should still be addressed, but it is not the single most load-bearing issue here.","tokens_in":24631,"tokens_out":8264,"duration_ms":81856,"concrete_test":"Re-run the extraction pipeline on a stratified random sample of 200-500 AI4Science papers (stratified by method cluster and venue) using a multi-label prompt that extracts ALL AI methods mentioned in title/abstract, plus a separate primary-method judgment. Rebuild Figure 3b and Tables 12-13 using (a) all-method edges and (b) primary-method edges only. If Attention Mechanisms, Transformer Architectures, or Gradient-Based Methods move above the regression line or enter the well-explored set under multi-label extraction, the F1 under-explored methods claim is an artifact of the single-main-method schema; if the rankings and hub structure are stable, the concern is mitigated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central findings F1 (under-explored AI methods) and F2 (method-node hubs) are built from a dataset in which each publication contributes exactly one AI method. Section 2.2 describes extracting \"The main AI method applied in the paper,\" and the actual prompt in Appendix A.2 asks for \"a keyword or a keyphrase that summarizes the main method used in this paper.\" The AlphaFold example in Figure 1 is labeled \"Neural network-based model\" even though AlphaFold is a transformer-based system, so even this flagship case does not register the attention/transformer machinery it uses. This single-label schema systematically excludes methods that appear as components, training techniques, or secondary contributions: attention mechanisms, gradient descent, regularization, and similar methods will almost never be extracted as the main method of an abstract, even when they are essential to the work. The \"underutilized\" method list in Figure 3b and Table 12 (Attention Mechanisms, Transformer Architectures, Gradient-Based Methods, Regularization Methods, Contrastive Learning) is exactly what this bias predicts. The same design may inflate the degree of generic method clusters (Machine Learning, Deep Learning Models, Neural Networks) because specific architectures are often abstracted into generic headers, contributing to the hub/periphery claim in F2. The Reader's classifier-noise concern is real, but it is downstream: even a perfect AI4Science binary classifier would not fix the systematic undercount of secondary methods. If the extraction schema were changed to capture all AI methods mentioned in an abstract, the method-level landscape and the under-explored rankings could shift substantially.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a large-scale literature analysis of AI4Science. The authors use GPT-4o mini to extract the scientific problem, the main AI method, and the usage description from titles and abstracts of 162,656 publications in top science journals and AI conferences (2014-2024). The extracted problem and method descriptions are embedded, projected, and clustered into 390 scientific problem clusters and 355 AI method clusters, which are then used to construct a bipartite graph connecting problem clusters to method clusters. Based on this dataset, the paper reports three main findings: (F1) different subdomains show varying engagement in AI4Science, leaving many problems and methods underexplored; (F2) connectivity in the bipartite graph is highly imbalanced with hub and peripheral nodes; and (F3) science and AI communities prioritize different problems and methods. The paper then formulates link prediction between problem and method clusters and compares conventional baselines (Katz index, node2vec) with LLM-based prediction, including a gpt-3.5 leakage check. Code and data are promised on GitHub.","tokens_in":24949,"tokens_out":4558,"duration_ms":41646,"significance":"If the findings hold, the dataset and the quantitative landscape it enables would be a useful community resource, providing a more scalable and semantically richer alternative to keyword-based science-of-science analyses. The link prediction section is a reasonable exploratory step, with several baselines and a leakage check using gpt-3.5. The paper is clearly written and the authors are transparent about many limitations, including selection of top venues, reliance on titles/abstracts, and the ground-truth evaluation of link prediction. The value of the paper depends on the validity of the extraction and classification pipeline, which is the main source of risk.","major_comments":[{"comment":"The extraction schema restricts each publication to exactly one 'main AI method,' as shown in the prompt in Appendix A.2 ('a keyword or a keyphrase that summarizes the main method used in this paper'). This systematically excludes methods that appear as components, training techniques, or secondary contributions. The AlphaFold example in Figure 1 illustrates the issue: the method is labeled 'Neural network-based model' even though the work is centrally based on attention/transformer machinery. The list of 'underutilized' methods in Figure 3b and Table 12 (Attention Mechanisms, Transformer Architectures, Gradient-Based Methods, Regularization Methods, Contrastive Learning) is exactly what this single-label schema would predict, because these methods are rarely stated as the main method of a paper even when they are essential. This bias is load-bearing for F1 and F2 as they apply to AI methods: the degree distribution in Figure 4c and the hub list in Table 13 may also be distorted by the tendency to map specific architectures to generic headers such as 'Machine Learning' and 'Deep Learning Models.' I would ask the authors to either extract multiple methods per paper (e.g., all AI methods mentioned), or to re-frame the findings as being about 'main methods' only and to add a validation on a subsample where multiple methods are annotated.","section":"Section 2.2, Appendix A.2, Figure 1"},{"comment":"The AI4Science binary classifier has precision 0.516 on AI-conference papers (F1 0.647), and the human annotators themselves agree only at F1 0.632. Since AI conferences contribute 3,539 of the 7,542 AI4Science positives (Tables 6 and 7), nearly half of the positive labels from that domain could be false positives. No cluster-level precision or recall is reported, and no sensitivity analysis is provided to show that the findings in Section 3 (F1-F3) are robust to classification noise. If false positives are more common in some problem or method clusters (e.g., CS-adjacent topics such as 'Urban Traffic Management' or 'Electronic Health Data Challenges'), the under-explored rankings, hub lists, and community discrepancy results could change. I recommend reporting stratified precision/recall by cluster or venue, and re-running the main descriptive analyses either with a precision-weighted correction or on a subset where the classifier is more reliable.","section":"Appendix A.2, Table 9"}],"minor_comments":[{"comment":"The sentence 'Similarly, AI methods frequently used by the science community are applied across a broader range of scientific challenges' is repeated verbatim in the same paragraph; one occurrence should be removed.","section":"Section 3.3"},{"comment":"The caption contains a typo: 'the resulting isolated noes are hidden' should read 'nodes.'","section":"Figure 4 caption"},{"comment":"The caption says 'gpt-3.5-turbo-012' but the model name should be 'gpt-3.5-turbo-0125'.","section":"Table 16 caption"},{"comment":"The statement that the high recall (0.866) 'provides a guarantee of avoiding missing important AI4Science work' is too strong given the low precision; a more balanced phrasing should acknowledge that many false positives are also included.","section":"Appendix A.2"},{"comment":"The paper refers to an 'anonymous repository' in the appendix while the abstract and main text give a named GitHub URL; please reconcile this inconsistency.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claims depend on the quality of the LLM extraction and the AI4Science classification. The two major issues above (the single-main-method schema and the low precision on AI-conference papers) are fixable with additional validation and re-analysis, so I do not recommend rejection. However, I would ask the editor to verify that the released dataset and code actually contain the full extraction outputs and human annotations, because the reproducibility of the descriptive statistics relies on the exact prompts and model outputs."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before reading this one closely. First, the dataset is real: 162,656 papers from top science and AI venues, with LLM-extracted problem and method descriptions, human-evaluated at 91% extraction accuracy, clustered into 390 problem and 355 method clusters, with code and data promised on GitHub. Second, the paper's headline finding about \"under-explored AI methods\" (F1) is probably an artifact of how methods were extracted.\n\nThe extraction prompt asks for the main method used in the paper, singular. So attention, transformers, gradient descent, regularization, and contrastive learning are almost never captured unless the paper names one as its principal contribution. The AlphaFold example in Figure 1 is telling: it is labeled \"Neural network-based model\" even though the method is transformer-based. The under-utilized method list in Table 12 is exactly what this bias predicts. The same schema likely inflates the generic hubs (Machine Learning, Deep Learning Models, Neural Networks) in F2, because specific architectures get abstracted into broad headers. This is a load-bearing flaw for F1 and partially for F2, and it is independent of the binary AI4Science classification issue the reader flagged.\n\nThat said, the classification issue is real too. The AI4Science label on AI-conference papers has precision 0.516, F1 0.647, and human annotators only agree at F1 0.632. The authors disclose this in the appendix and argue human agreement is no better, which is fair, but they do not propagate this uncertainty into any of the Section 3 statistics. No sensitivity analysis is provided. The link prediction section is more solid: they include several baselines and a leakage check with gpt-3.5, and they are upfront that gpt-4o may have seen some test-period material.\n\nWhat the paper does well: the community-discrepancy analysis (F3) is interesting and less affected by the method-extraction bias, the bipartite graph and degree-distribution characterization are competent, and the dataset itself is a contribution even if some findings need revisiting.\n\nWho is this for: science-of-science researchers, AI-for-science recommenders, and anyone mapping interdisciplinary fields. The paper deserves a serious referee, but the referee should demand a sensitivity analysis and, ideally, a multi-label method extraction or at least an explicit acknowledgment that \"under-explored\" means \"rarely the stated main method,\" not \"rarely used.\"","headline":"Useful large-scale dataset and a genuinely new semantic map of AI4Science, but the headline method-level findings are probably biased by the single-'main-method' extraction schema.","tokens_in":25508,"tokens_out":1470,"would_cite":true,"duration_ms":15162,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that AI4Science research is unevenly integrated, with many scientific problems and AI methods underexplored, and that link prediction can propose new pairings.","keywords":["AI for Science","AI4Science","Large-Scale Literature Analysis","Link Prediction","LLM Extraction","Science of Science","Bipartite Graph"],"falsifier":"Re-run the cluster-level analysis on a sample of AI-conference papers that domain experts relabel by hand, using the same embedding, clustering, and regression steps; if hand labels move clusters such as Neural Network Challenges or Urban Traffic Management below the regression line or promote current under-explored clusters to well-explored, the paper's top rankings change. A simpler check is to recompute the under-explored versus well-explored partition under an alternative AI4Science classifier and see whether the top-ten lists in Table 12 survive.","tokens_in":24441,"feed_emoji":"🔬","tokens_out":7496,"duration_ms":60055,"temperature":0.7,"pith_summary":"This paper tries to establish a quantitative, data-driven picture of AI4Science: where AI methods actually meet scientific problems and where the connections are sparse. It builds a dataset of 162,656 papers from leading science journals and AI conferences, uses large language models to extract the scientific problem and AI method of each paper, clusters them into semantic maps, and connects them in a bipartite graph. The main findings are that engagement is uneven, a small number of hubs dominate connectivity, and the science and AI communities pursue different integration strategies. If true, the under-explored clusters and method gaps are concrete targets for interdisciplinary work, and the link prediction results suggest that LLMs can propose novel problem-method pairings worth testing.","feed_headline":"AI meets science unevenly: 162,000 papers show the gaps","feed_subtitle":"A new map reveals where AI and science rarely meet and flags new pairings to try.","key_machinery":"The carrying machinery is an extraction-to-network pipeline: a large language model (GPT-4o mini) extracts scientific problems, AI methods, and usage descriptions from titles and abstracts; semantic embeddings with dimensionality reduction and density-based clustering organize them into 390 problem clusters and 355 method clusters; and the clusters become the two node types of a bipartite graph whose edges are publications. This graph makes the gap between AI and science measurable as uneven cluster sizes, hub-and-peripheral degree structure, and novel-link prediction targets.","core_discovery":"The paper's central discovery is a landscape, not a single theorem: in the decade 2014-2024, only 7,542 of 162,656 papers from these venues qualify as AI4Science, and their distribution is markedly uneven. Under-explored scientific problem clusters include Asymmetric Synthesis, Magnetic Materials and Spintronics, Forests and Climate Change, Viral Mechanisms, and Cancer Drug Resistance; underused AI methods include Attention Mechanisms, Gradient-Based Methods, Graph Embedding, and Transformer Architectures. The bipartite graph of problem clusters and method clusters has heavy-tailed degree distributions, so a few hub problems and methods account for most connectivity. Link prediction experiments show that LLM-based models, especially paper-level generative prediction with retrieval augmentation, find novel links beyond the 683 actually observed in 2023-2024, which the authors take as evidence that the gap is bridgeable.","pith_inferences":["An implication the paper leaves implicit is that its under-explored lists double as a research agenda: funding agencies or labs could use the cluster rankings to pick problems where AI has low prior penetration.","A testable extension is to track whether AI-proposed novel links that later appear in publications are cited more or less than random problem-method pairings, which would measure whether the predicted directions are actually fruitful.","The same pipeline could be rerun on full texts or a broader set of venues; that would test whether the under-explored labels are artifacts of restricting analysis to titles, abstracts, and top journals.","Because the degree distribution is heavy-tailed and log-normal-like, a dynamic version of the dataset could test whether new links preferentially attach to existing hubs, which would indicate whether the imbalance is self-reinforcing."],"forward_implications":["Under-explored problem clusters such as Asymmetric Synthesis, Forests and Climate Change, and Viral Mechanisms are the places where new AI applications could have the least competition.","Underused methods such as Attention Mechanisms, Graph Embedding, Contrastive Learning, and Transformer Architectures have room to be carried into scientific domains where they are rarely applied.","LLM-based link prediction, particularly paper-level generation with retrieval augmentation, can surface hundreds of novel problem-method pairings at small K, giving researchers a concrete shortlist.","Because the science and AI communities prioritize different problems and methods, bridging efforts can be targeted: scientists could adopt broader general AI methods while AI researchers take up domain-specific scientific problems."],"supporting_citations":[{"why":"Supplies the AI-conference corpus (AAAI, IJCAI, ICLR, ICML, NeurIPS, WWW, SIGKDD) used in dataset curation.","marker":"[37]"},{"why":"Prior Science of Science analysis of AI diffusion across fields, whose 'oil-and-water' finding this paper extends and contrasts with.","marker":"[6]"},{"why":"Prior quantitative study of AI use and citation advantage, used as the coarse keyword-taxonomy baseline this paper improves on.","marker":"[7]"},{"why":"AlphaFold serves as the motivating AI4Science exemplar for extraction and dataset construction.","marker":"[10]"},{"why":"Instructor embedding model that produces the semantic representations used for clustering problems and methods.","marker":"[28]"},{"why":"LargeVis dimensionality reduction used to project embeddings into the 2D landscape maps.","marker":"[30]"},{"why":"HDBSCAN density-based clustering that creates the 390 problem and 355 method clusters.","marker":"[3]"},{"why":"Node2vec embeddings provide the structural link prediction baseline that outperforms Katz.","marker":"[8]"},{"why":"Katz index is the link-score baseline used for cluster-level bipartite link prediction.","marker":"[11]"},{"why":"Retrieval-augmented generation technique used to improve paper-level LLM link prediction with similar examples.","marker":"[14]"}],"fun_headline_variants":["Mapping 162K papers reveals where AI and science drift apart","LLMs map AI4Science gaps across 162K publications","162K papers show AI and science are unevenly paired","Link prediction suggests untried AI-science pairings","7,542 AI4Science papers of 162K: where cooperation lags"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quantitative landscape depends on the LLM's binary judgment that a paper is AI4Science; on AI-conference papers that classifier has precision 0.516 and F1 0.647, and human annotators agree only at F1 0.632, so if classification errors are systematically concentrated in certain clusters the under-explored and hub findings could shift.","fun_headline_variants_meta":{"raw":{"variants":["Mapping 162K papers reveals where AI and science drift apart","LLMs map AI4Science gaps across 162K publications","162K papers show AI and science are unevenly paired","Link prediction suggests untried AI-science pairings","7,542 AI4Science papers of 162K: where cooperation lags"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000555,"raw_usage":{"total_tokens":2625,"prompt_tokens":912,"completion_tokens":1713,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":1626}},"tokens_in":528,"tokens_out":1713,"duration_ms":12021,"temperature":1.0,"reasoning_tokens":1626,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:39:24.030009+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the cluster-level analysis on a sample of AI-conference papers that domain experts relabel by hand, using the same embedding, clustering, and regression steps; if hand labels move clusters such as Neural Network Challenges or Urban Traffic Management below the regression line or promote current under-explored clusters to well-explored, the paper's top rankings change. A simpler check is to recompute the under-explored versus well-explored partition under an alternative AI4Science classifier and see whether the top-ten lists in Table 12 survive.","supporting_citations":[{"cited_title":"A Dual-Stream Neural Network Explains the Functional Segregation of Dorsal and Ventral Visual Pathways in Human Brains","cited_arxiv_id":"2310.13849","evidence_quote":"Prior Science of Science analysis of AI diffusion across fields, whose 'oil-and-water' finding this paper extends and contrasts with."},{"cited_title":"Quantifying the Benefit of Artificial Intelligence for Scientific Research","cited_arxiv_id":"2304.10578","evidence_quote":"Prior quantitative study of AI use and citation advantage, used as the coarse keyword-taxonomy baseline this paper improves on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Instructor embedding model that produces the semantic representations used for clustering problems and methods."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Katz index is the link-score baseline used for cluster-level bipartite link prediction."}],"review_version":1}