Pith. sign in

REVIEW 6 major objections 6 minor 121 references

Mapping Scientific Literature with Large Language Models and Topic Modeling

T0 review · 6 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An unsupervised LLM pipeline can map a 20-year scientific corpus into interpretable topics and recover editorial dual classifications from text alone.

desk verdict Useful pipeline, overclaimed independence; the abstract promises more than the body delivers. read the letter →

arxiv 2510.16152 v2 pith:YTY5XKHN submitted 2025-10-17 cs.DL cs.AIcs.CLcs.LG

Mason Smetana , Lev Khazanovich This is my paper · ORCID
classification cs.DLcs.AIcs.CLcs.LG
keywords scientificliteraturemappinglargelanguagemodelstopicmodelingunsupervisedclassificationinterdisciplinarystructurefull-textanalysisbipartitegraphPNASengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a large language model, guided only by clustering and its own generated topic descriptions, can map the thematic structure of a scientific corpus without a predefined taxonomy. It applies this to 1,519 engineering-focused articles from a major multidisciplinary journal over 2005–2024, first labeling each abstract into one of 16 topics, then classifying 18,636 full-text segments into multiple topics to expose cross-topic connections. The central demonstration is that this unsupervised pipeline largely reproduces the journal's own dual-classification scheme—biology plus engineering, for instance—without being told that scheme exists. If true, this would give bibliometric researchers an interpretable way to discover emerging fields and interdisciplinary bridges that keyword or citation counts miss. The paper also shows the LLM topics align with classical word-frequency structure, offered as evidence the topics are linguistically grounded.

What carries the argument

The engine is an iterative cluster-then-reclassify loop. K-means groups abstracts via vector embeddings; an LLM invents names and descriptions for the groups; the same LLM then reassigns each abstract to the best-matching invented topic. An agreement score measures whether the LLM's assignment matches the original cluster; clusters meeting a 60% threshold become stable topics, and the leftovers are re-clustered. This converts the LLM into a dynamic taxonomy-builder, solving the what-to-call-the-cluster problem and letting small subfields emerge over iterations. The same topic set then acts as the label space for full-text segment classification, and a bipartite graph recorded as an adjacency

What would settle it

Strip or paraphrase every phrase in the 1,519 abstracts that echoes the journal's category names ('medical sciences', 'applied biological sciences', 'biophysics', and so on), then rerun the pipeline; if the sixteen topics still align with the dual-classification scheme at the same precision and lift, the recovery is genuine, and if alignment collapses, it was driven by memorized editorial cues.

Watch

Extended reading notes

Core claim

The central claim is that a two-phase LLM-driven classification pipeline independently recovers latent topical structure. In phase one, abstract embeddings are K-means clustered; the LLM generates titles and descriptions for each cluster, then reclassifies each abstract into those categories. Clusters whose reclassification agreement reaches 60% are kept; unstable abstracts are re-clustered recursively until under 10% remain. This produces sixteen stable topics covering 90.1% of abstracts. In phase two, full-text segments are independently classified into one or more of those sixteen topics; 75% of segments receive multiple labels. The resulting asymmetric adjacency matrix between abstract t

Load-bearing premise

The load-bearing premise is that the LLM's agreement with its own cluster labels is evidence of discovered topics, rather than self-consistency or memorized editorial categories.

Editorial extensions

If this is right

  • Reapplying the pipeline to other journals or time windows should produce fresh interpretable topic maps without researchers pre-defining categories.
  • The 'Other' residual category works as an early-warning signal: when it grows, re-running the loop can surface nascent subfields such as shape-morphing materials.
  • The asymmetric abstract-to-full-text matrix reveals one-way dependencies, e.g., Material Science is foundational for Catalysis and Energy even though the reverse flow is small.
  • The method's plain-language topic titles and descriptions could make large-corpus overviews accessible to non-specialists and science communicators.
  • Because the topics derive from content rather than author keywords, the approach sidesteps keyword sparsity, recovering fields like microfluidics that appear in under 2% of abstracts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A sterner test would be to run the same pipeline on a corpus assembled after the LLM's training cutoff, or on abstracts with editorial category names paraphrased, to determine how much of the 'independent recovery' is discovery versus recall of the journal's taxonomy.
  • The asymmetric flow matrix could be compared against citation or co-authorship networks: if full-text topic flows predict downstream citations, the method becomes a content-based proxy for knowledge transfer.
  • The sixteen-topic schema could be treated as a benchmark and re-run on the next five years of publications to test whether the convergence thresholds remain stable as the corpus grows.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. This paper presents an LLM-based two-phase framework for mapping scientific literature. The first phase iteratively clusters abstract embeddings with K-means, uses GPT-4o mini to label and describe the clusters, reclassifies abstracts against those labels, and retains only clusters with a reclassification agreement score above 0.60, yielding 16 topics for 1,519 PNAS engineering articles (2005-2024). The second phase applies the same topic scheme to full-text segments, allowing multi-label assignments, and builds a bipartite/adjacency representation of primary-secondary topic flows. The authors report that the framework independently recovers PNAS's editorial dual-classification structure and, in the arXiv abstract, claim 75.9% manual validation accuracy and superiority over established topic models.

Significance. If validated, the framework would be a useful, interpretable tool for unsupervised mapping of scientific fields, and the step-by-step prompts and equations in Appendices A-E are a strength. The c-TF-IDF and BoW analyses provide useful linguistic characterization of the derived topics. However, the strongest quantitative claims are not supported by the body: the manual-validation protocol is absent, the comparison to established topic models is absent, and the "independent recovery" claim is threatened by the self-referential validation loop and by possible pretraining contamination. The contribution is potentially valuable, but the evidence as presented is insufficient for the headline claims.

major comments (6)
  1. [Abstract; Sec. 4.2; Appendix A.2-A.5] The central claim that the pipeline "independently recovers" PNAS's editorial dual-classification structure is not established. Eq. (4) measures agreement between the LLM's reclassification and the K-means cluster label for the same cluster, and those labels were generated by the LLM from the very same clusters (Appendix A.2). This is an internal-consistency score, not evidence of recovery of an external schema. The external anchor in Table 1 compares topics to PNAS dual labels, but GPT-4o mini's pretraining data are not disclosed; because PNAS is a major journal, contamination cannot be ruled out, making high lift compatible with memorization. The "without prior knowledge" phrasing should be removed unless the authors provide a contamination control (e.g., applying the same pipeline to a corpus whose editorial labels postdate the model's training cutoff, or showing that permuted or mask
  2. [Abstract; entire body] The abstract's claim of "75.9% manual validation accuracy" is unreproducible. No manual-validation protocol appears in the main text or appendices: no sample size, sampling scheme, annotator instructions, inter-annotator agreement, or confusion matrix. This number should either be fully documented (preferably in an appendix) or removed from the abstract. As it stands, the number cannot be checked.
  3. [Abstract; Sec. 5] The abstract states that a "comparative evaluation against established topic modeling methods shows higher topic diversity and lower overlap with competitive coherence metrics." No such comparison appears in the manuscript. There are no LDA, NMF, BERTopic, or other baseline results, no coherence metrics (e.g., NPMI, UMass), and no diversity/overlap measurements. Either add the comparison or delete this claim from the abstract.
  4. [Sec. 4.1; Table 4] The numerical counts are inconsistent. Sec. 4.1 reports 46,639 effective classifications; Table 4's column sums total 49,633 (not 49,639 as also stated). The "No. Class" and "Sum" rows are difficult to interpret, and the "Percent Corpus" column appears to be computed on a different denominator. Because the full-text percentages, adjacency matrix, and cross-topic claims depend on these counts, the authors must reconcile the totals and define every column.
  5. [Appendix A.6; Sec. 5.1] The stability of the final 16-topic structure depends on the user-set thresholds tau=0.60 and delta=0.10. The paper cites "benchmark evaluations of embedding-based clustering accuracy [54]" as justification for tau, but no such evaluation is reported. A sensitivity analysis varying tau and delta (e.g., tau in {0.50,0.55,0.65,0.70}, delta in {0.05,0.15,0.20}) is needed to show that the main qualitative conclusions are not artifacts of these thresholds.
  6. [Sec. 3.2; Sec. 5.2] The BoW and c-TF-IDF analyses are described as "validation" of the LLM topics, but they only show that the topics share vocabulary with the abstracts from which the topics were derived. This does not provide external validity. The framing should be changed to "post-hoc linguistic characterization," and claims of "confirming the validity" should be softened accordingly.
minor comments (6)
  1. [General] The arXiv title "Mapping Scientific Literature with Large Language Models and Topic Modeling" differs from the full-text title "PUBLICATION TREND ANALYSIS AND SYNTHESIS VIA LARGE LANGUAGE MODEL: A CASE STUDY OF ENGINEERING IN PNAS." Please align the titles.
  2. [Sec. 3.1] Typos: "intially" should be "initially"; elsewhere "prevelance" (Introduction) and "apporach" (Conclusion) need correction.
  3. [Table 4] Column headings should be defined precisely, especially "No. Class," "No. Seg.," "Exclusive," "Same," and "To Other." The reader cannot reconstruct the relationship between these columns from the current caption.
  4. [Eq. (21)] The adjacency-matrix definition is notationally unclear: the right-hand side mixes a sum over documents and segments with a set union. Please rewrite Eq. (21) with explicit indicator notation and define the dimensions.
  5. [References] References [40] and [59] are the same work (Kötter et al., ICDM 2015); duplicate entries should be merged. Also, reference [32] (PNAS website) is used as a general citation for corpus composition; a data-version citation would be more precise.
  6. [Appendix A.2] The prompts list no decoding parameters. For reproducibility, the temperature, max tokens, and any seed should be reported, or the paper should state that default decoding was used.

Circularity Check

1 steps flagged · score 2.0 of 10

Central PNAS-recovery comparison is independent of the topic-construction loop; only the internal NLP 'validation' is self-referential.

  1. self definitional [Section 3.2 and Appendix B.2]
    "In contrast to Grootendorst’s original implementation, which combines c-TF-IDF with dimensionality reduction (UMAP) and clustering (HDBSCAN), this approach uses the classification outputs from LLM-driven topic modeling as predefined groups. ... Together, these NLP results support the internal consistency and validity of the LLM-driven classification: high-frequency terms identified through BoW and c-TF-IDF models consistently aligned with the dominant themes discovered by the LLM."

    The c-TF-IDF and BoW 'validation' take the LLM-generated topic labels as the class definitions (B.2), so term frequencies are aggregated within those same classes. The resulting 'clear topical separation' (Fig. 4) is thus a property of the grouping that produced the labels, not independent confirmation. The validation restates the cluster content rather than testing it against an external reference. The only genuinely external anchor, the PNAS dual-classification lift (Table 1), is not part of this loop.

full rationale

The derivation chain for the central claim is not circular: topics are built from embeddings -> K-means clustering -> LLM-generated labels (Appendix A.2) -> LLM reclassification (Eq. 3) -> agreement filter (Eq. 4), none of which uses PNAS editorial labels. The comparison to PNAS dual classifications (Table 1, Eqs. 14-16) is an external anchor applied after topic construction, so the 'recovery' claim is not a fit to the labels by construction. Caveats exist but are not circularity: (1) the agreement score measures self-consistency with the clusters that generated the labels, so it is not evidence of external validity; (2) the BoW/c-TF-IDF checks are computed with the LLM topics as predefined classes, making them internal redescription rather than independent validation; (3) GPT-4o mini's pretraining may include PNAS categories, a training-contamination threat to 'without prior knowledge,' but that is an external-validity concern, not a reduction of the derivation to its inputs. These issues lower confidence in the strength of the claims but do not make the central derivation circular, hence the low score.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The framework's results depend on a chain of modeling choices: embedding-plus-clustering geometry (k=7 per iteration), the tau/delta/3-of-5 thresholds that define 'stable' topics, the LLM's unstated decoding defaults, and the assumption that LLM-cluster agreement measures truth. The external anchor (PNAS dual labels) is real but plausibly contaminated by training-data leakage. No invented entities are introduced.

free parameters (6)
  • k (K-means clusters per iteration) = 7
    Fixed by hand; elbow and Silhouette showed no optimum (Sec. 5.1, Fig. 8). Directly shapes the number of topics per iteration.
  • tau (agreement threshold) = 0.60
    Clusters survive only if LLM reclassification agrees with K-means on at least 60% of abstracts; justified only by a vague reference to MTEB benchmark accuracy [54] (Appendix A.6).
  • delta (termination threshold) = 0.10
    Recursion stops when <10% of abstracts remain; this directly produces the 90.1% coverage figure and sets the size of the 'Other' bucket (Appendix A.7, Sec. 3.1).
  • 3-out-of-5 consensus rule
    An abstract must match the same topic in 3 of 5 LLM runs; filters ~5% of abstracts (Fig. 10). Measures variance, not accuracy (Appendix A.4).
  • top-k representative abstracts per cluster (prompt context) = unspecified
    The number of abstracts fed to the LLM to generate each cluster's title and description (Appendix A.2) is never stated; it directly shapes the generated labels.
  • LLM decoding parameters (temperature, max tokens) = unspecified (API defaults)
    Not reported; stochasticity is the reason the 3-of-5 rule exists, so sampling parameters materially affect all classifications.
assumptions (6)
  • domain assumption text-embedding-3-small embeddings + cosine K-means group abstracts into semantically coherent clusters
    Entire pipeline starts from this; clustering quality is asserted, not measured (Appendix A.1; Fig. 8 shows no clustering optimum).
  • ad hoc to paper LLM-vs-K-means agreement >= 0.60 (Eq. 4–5) measures true topic coherence
    Stability criterion defined by this paper's own procedure; the mapping from tau=0.60 to MTEB benchmarks [54] is unstated (Appendix A.5–A.6).
  • domain assumption GPT-4o mini is a reliable classifier under 3-out-of-5 consensus
    Consensus measures variance, not accuracy; no external accuracy check is reported in the body (Appendix A.4).
  • domain assumption PNAS editorial dual-classification is valid external ground truth for interdisciplinarity
    Used as the benchmark for Table 1 lifts; the same labels may appear in GPT-4o mini's training data, undermining the 'independent recovery' claim (Appendix C; Sec. 4.2).
  • domain assumption c-TF-IDF/BoW topic separation confirms semantic validity
    Near-tautological: clusters derived from distributional embeddings will have distinct vocabularies; this validates self-consistency, not external truth (Sec. 3.2, B.2).
  • domain assumption Full-text segments inherit the abstract-derived 16-topic schema without recalibration
    Secondary classification assumes the abstract-level topic set applies to full-text content (Appendix E); no check that abstract topics are the right ontology for body text.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mapping Scientific Literature with Large Language Models and Topic Modeling." pith.science (2026). https://pith.science/paper/YTY5XKHN

@misc{pith2026251016152,
  author       = {Pith},
  title        = {Pith review of: Mapping Scientific Literature with Large Language Models and Topic Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YTY5XKHN}},
  note         = {Machine review of arXiv:2510.16152}
}
read the original abstract

Scientific literature is increasingly fragmented by disciplinary boundaries, specialized terminology, and potentially sparse keyword systems, making it difficult to capture the evolving structure of modern science. This study introduces a large language model (LLM)-driven framework for mapping scientific literature from a topic modeling perspective. The approach is demonstrated on a 20-year corpus of more than 1,500 engineering-related articles published in the Proceedings of the National Academy of Sciences (PNAS). A two-stage classification pipeline first assigns a primary thematic category to each article based on its abstract, followed by full-text analysis to identify secondary classifications that reveal latent cross-topic connections within the corpus. Unlike conventional topic models, the LLM-based framework produces semantically interpretable topics while maintaining strong quantitative performance. Comparative evaluation against established topic modeling methods shows higher topic diversity and lower overlap with competitive coherence metrics. Manual validation on a randomly sampled subset of abstracts yields an accuracy of 75.9%. Additional traditional natural language processing analyses confirm that the generated topics correspond to meaningful linguistic patterns in the corpus. A bipartite network linking primary and secondary classifications further reveals implicit thematic relationships that are not readily observable through abstracts or keyword systems alone. The findings indicate that the framework independently recovers much of the journal's editorial dual-classification structure without prior knowledge of its schema. Overall, the proposed approach offers a powerful tool for mapping science and identifying emerging cross-topic connections in research.

Figures

Figures reproduced from arXiv: 2510.16152 by the authors.

Figure 1
Figure 1. Overview of methodology. abstract groupings. More specifically, Section B describes the application of word frequency analyses using modified class-based TF-IDF (c-TF-IDF) and lemmatized BoW. Section C then compares the LLM-generated classifications to the original PNAS classification system using precision, recall, and lift to assess interdisciplinary alignment. The lower portion of the figure illustrates the secon… view at source ↗
Figure 2
Figure 2. Cumulative (a) and overlapping (b) distributions of top LLM-derived topics and classifications for engineering [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Word frequency (f) versus rank (r) for all abstracts on linear scale (a) and log-log scale with normalized frequency (b) conforming to Zipf’s law. The highlighted area under each curve represents the Pareto region, where 80% of the most frequent words occur in the first 20% of rankings. “dislocation,” “alloys,” and “strength.” Shared terms such as “mechanical” and “stiffness” suggest some overlap, though this techni… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Progressive c-TF-IDF word clouds of LLM-derived topic abstract classifications. The x-axis shows four [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Distribution of LLM-derived topic classifications for PNAS engineering abstracts and full-text segments. The [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Abstract (row) to full-text (column) normalized adjacency matrix [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Example abstract K-means clustering for iteration [PITH_FULL_IMAGE:figures/full_fig_p028_7.png]
Figure 8
Figure 8. Figure 8: Varying K-means clustering over 1,519 PNAS abstracts from [PITH_FULL_IMAGE:figures/full_fig_p029_8.png]
Figure 9
Figure 9. Figure 9: All nine iterations of primary abstract classification. Each subplot shows interim topics: the number of [PITH_FULL_IMAGE:figures/full_fig_p030_9.png]
Figure 10
Figure 10. Figure 10: All nine iterations of primary abstract classification, and cases in which the abstract satisfied the “3-out-of-5” [PITH_FULL_IMAGE:figures/full_fig_p031_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

121 extracted references · 28 canonical work pages

  1. [54]

    MTEB: Massive Text Embedding Bench- mark.arXiv preprint arXiv:2210.07316, 2023

    Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. MTEB: Massive Text Embedding Bench- mark.arXiv preprint arXiv:2210.07316, 2023. doi: 10.48550/arXiv.2210.07316

    arXiv doi:10.48550/arxiv.2210.07316 2023

  2. [1]

    Public understanding of science versus public understanding of research.Public Understanding of Science, 10(4):421–426, 2001

    Hyman Field and Patricia Powell. Public understanding of science versus public understanding of research.Public Understanding of Science, 10(4):421–426, 2001. doi: 10.3109/a036879

    work page doi:10.3109/a036879 2001

  3. [2]

    Public Communication of Science in Blogs: Recontextualizing Scientific Discourse for a Diversified Audience.Written Communication, 30(4):428–457, 2013

    María Luzón. Public Communication of Science in Blogs: Recontextualizing Scientific Discourse for a Diversified Audience.Written Communication, 30(4):428–457, 2013. doi: 10.1177/0741088313493610

    work page doi:10.1177/0741088313493610 2013

  4. [3]

    Measuring mumbo jumbo: A preliminary quantification of the use of jargon in science communication.Public Understanding of Science, 23(5):528–546, 2014

    Aviv Sharon and Ayelet Baram-Tsabari. Measuring mumbo jumbo: A preliminary quantification of the use of jargon in science communication.Public Understanding of Science, 23(5):528–546, 2014. doi: 10.1177/0963662512469916

    work page doi:10.1177/0963662512469916 2014

  5. [4]

    Personalized Jargon Identification for Enhanced Interdisciplinary Communication

    Yue Guo, Joseph Chang, Maria Antoniak, et al. Personalized Jargon Identification for Enhanced Interdisciplinary Communication.arXiv preprint arXiv:2311.09481, 2024. doi: 10.48550/arXiv.2311.09481

    work page Pith review arXiv doi:10.48550/arxiv.2311.09481 2024

  6. [5]

    Assessing what to address in science communication.Proceedings of the National Academy of Sciences, 110:14062–14068, 2013

    Wändi Bruine de Bruin and Ann Bostrom. Assessing what to address in science communication.Proceedings of the National Academy of Sciences, 110:14062–14068, 2013. doi: 10.1073/pnas.1212729110

    work page doi:10.1073/pnas.1212729110 2013

  7. [6]

    From complexity to clarity: How AI enhances perceptions of scientists and the public’s understanding of science.PNAS Nexus, 3(9):pgae387, 2024

    David Markowitz. From complexity to clarity: How AI enhances perceptions of scientists and the public’s understanding of science.PNAS Nexus, 3(9):pgae387, 2024. doi: 10.1093/pnasnexus/pgae387

    doi:10.1093/pnasnexus/pgae387 2024

  8. [7]

    Lay summaries needed to enhance science communication.Proceedings of the National Academy of Sciences, 112(12):3585–3586, 2015

    Lauren Kuehne and Julian Olden. Lay summaries needed to enhance science communication.Proceedings of the National Academy of Sciences, 112(12):3585–3586, 2015. doi: 10.1073/pnas.1500882112

    work page doi:10.1073/pnas.1500882112 2015

Show all 121 references
  1. [8]

    Improving accessibility of scientific research by artificial intelligence—An example for lay abstract generation.Digital Health, 9, 2023

    Boris Schmitz. Improving accessibility of scientific research by artificial intelligence—An example for lay abstract generation.Digital Health, 9, 2023. doi: 10.1177/20552076231186245

    2023 doi

  2. [9]

    Automated Lay Language Summarization of Biomedical Scientific Reviews.arXiv preprint arXiv:2012.12573, 2020

    Yue Guo, Wei Qiu, Yizhong Wang, and Trevor Cohen. Automated Lay Language Summarization of Biomedical Scientific Reviews.arXiv preprint arXiv:2012.12573, 2020. doi: 10.48550/arXiv.2012.12573

    2012 doi arXiv

  3. [10]

    Charting the course for PNAS.Proceedings of the National Academy of Sciences, 105(8): 2755–2756, 2008

    Randy Schekman. Charting the course for PNAS.Proceedings of the National Academy of Sciences, 105(8): 2755–2756, 2008. doi: 10.1073/pnas.0800528105

    2008 doi

  4. [11]

    Impact, not impact factor.Proceedings of the National Academy of Sciences, 112(26):7875–7876,

    Inder Verma. Impact, not impact factor.Proceedings of the National Academy of Sciences, 112(26):7875–7876,

  5. [12]

    Mapping knowledge domains.Proceedings of the National Academy of Sciences, 101:5183–5185, 2004

    Richard Shiffrin and Katy Börner. Mapping knowledge domains.Proceedings of the National Academy of Sciences, 101:5183–5185, 2004. doi: 10.1073/pnas.0307852100

    2004 doi

  6. [13]

    Science communication reconsidered.Nature Biotechnology, 27(6):514–518, 2009

    Tania Bubela, Matthew Nisbet, Rick Borchelt, et al. Science communication reconsidered.Nature Biotechnology, 27(6):514–518, 2009. doi: 10.1038/nbt0609-514

    2009 doi

  7. [14]

    Mapping research trends from 35 years of publications in Distance Education.Distance Education, 37(3):245–269, 2016

    Olaf Zawacki-Richter and Som Naidu. Mapping research trends from 35 years of publications in Distance Education.Distance Education, 37(3):245–269, 2016. doi: 10.1080/01587919.2016.1185079

    2016

  8. [15]

    Meta-research: Why research on research matters.PLoS Biology, 16(3):e2005468, 2018

    John Ioannidis. Meta-research: Why research on research matters.PLoS Biology, 16(3):e2005468, 2018. doi: 10.1371/journal.pbio.2005468

    2018 doi

  9. [16]

    Reproducible research practices, transparency, and open access data in the biomedical literature, 2015-2017.PLoS Biology, 16(11):e2006930, 2018

    Joshua Wallach, Kevin Boyack, and John Ioannidis. Reproducible research practices, transparency, and open access data in the biomedical literature, 2015-2017.PLoS Biology, 16(11):e2006930, 2018. doi: 10.1371/jour- nal.pbio.2006930

    2015 doi

  10. [17]

    Trends in US public confidence in science and op- portunities for progress.Proceedings of the National Academy of Sciences, 121(11):e2319488121, 2024

    Arthur Lupia, David Allison, Kathleen Jamieson, et al. Trends in US public confidence in science and op- portunities for progress.Proceedings of the National Academy of Sciences, 121(11):e2319488121, 2024. doi: 10.1073/pnas.2319488121

    2024 doi

  11. [18]

    Mapping topics and topic bursts in PNAS.Proceedings of the National Academy of Sciences, 101:5287–5290, 2004

    Ketan Mane and Katy Börner. Mapping topics and topic bursts in PNAS.Proceedings of the National Academy of Sciences, 101:5287–5290, 2004. doi: 10.1073/pnas.0307626100

    2004 doi

  12. [19]

    Mapping knowledge domains: Characterizing PNAS.Proceedings of the National Academy of Sciences, 101:5192–5199, 2004

    Kevin Boyack. Mapping knowledge domains: Characterizing PNAS.Proceedings of the National Academy of Sciences, 101:5192–5199, 2004. doi: 10.1073/pnas.0307509100

    2004 doi

  13. [20]

    Reconceptualizing the classification of PNAS articles

    Edoardo Airoldi, Elena Erosheva, Stephen Fienberg, et al. Reconceptualizing the classification of PNAS articles. Proceedings of the National Academy of Sciences, 107(49):20899–20904, 2010. doi: 10.1073/pnas.1013452107

    2010 doi

  14. [21]

    Disciplinary structures in Nature, Science and PNAS: journal and country levels.Scientometrics, 116(3):1817–1852, 2018

    Jielan Ding, Per Ahlgren, Liying Yang, and Ting Yue. Disciplinary structures in Nature, Science and PNAS: journal and country levels.Scientometrics, 116(3):1817–1852, 2018. doi: 10.1007/s11192-018-2812-9

    2018 doi

  15. [22]

    Feature analysis of multidisciplinary scientific collaboration patterns based on PNAS.EPJ Data Science, 7:1–17, 2018

    Zheng Xie, Miao Li, Jianping Li, Xiaojun Duan, and Zhenzheng Ouyang. Feature analysis of multidisciplinary scientific collaboration patterns based on PNAS.EPJ Data Science, 7:1–17, 2018. doi: 10.1140/epjds/s13688- 018-0134-z

    2018 doi

  16. [23]

    Nature, Science, and PNAS: disciplinary profiles and impact.Scientometrics, 123(3):1301–1315,

    Staša Milojevi´c. Nature, Science, and PNAS: disciplinary profiles and impact.Scientometrics, 123(3):1301–1315,

  17. [24]

    Scientific Large Language Models: A Survey on Biological & Chemical Domains.ACM Computing Surveys, 57(6):1–38, 2025

    Qiang Zhang, Keyan Ding, Tianwen Lv, et al. Scientific Large Language Models: A Survey on Biological & Chemical Domains.ACM Computing Surveys, 57(6):1–38, 2025. doi 10.1145/3715318

    2025 doi

  18. [25]

    ChatGPT, Bard, and Large Language Models for Biomedical Re- search: Opportunities and Pitfalls.Annals of Biomedical Engineering, 51:2647–2651, 2023

    Surendrabikram Thapa and Surabhi Adhikari. ChatGPT, Bard, and Large Language Models for Biomedical Re- search: Opportunities and Pitfalls.Annals of Biomedical Engineering, 51:2647–2651, 2023. doi: 10.1007/s10439- 023-03284-0

    2023 doi

  19. [26]

    Tracing the Influence of Large Language Models across the Most Impactful Scientific Works.Electronics, 12(24):4957, 2023

    Dana-Mihaela Petros, anu, Alexandru Pîrjan, and Alexandru T˘abus, c˘a. Tracing the Influence of Large Language Models across the Most Impactful Scientific Works.Electronics, 12(24):4957, 2023. doi: 10.3390/electron- ics12244957

    2023 doi

  20. [27]

    Cathy Shyr, Randall Grout, Nan Kennedy, et al. Leveraging artificial intelligence to summarize abstracts in lay language for increasing research accessibility and transparency.Journal of the American Medical Informatics Association, 31(10):2294–2303, 2024. doi: 10.1093/jamia/ocae186

    2024 doi

  21. [28]

    Paper Plain: Making Medical Research Papers Approachable to Healthcare Consumers with Natural Language Processing.ACM Transactions on Computer-Human Interaction, 30(5):1–38, 2023

    Tal August, Lucy Wang, Jonathan Bragg, et al. Paper Plain: Making Medical Research Papers Approachable to Healthcare Consumers with Natural Language Processing.ACM Transactions on Computer-Human Interaction, 30(5):1–38, 2023. doi: 10.1145/3589955

    2023 doi

  22. [29]

    Towards Reasoning in Large Language Models: A Survey.arXiv preprint arXiv:2212.10403, 2023

    Jie Huang and Kevin Chang. Towards Reasoning in Large Language Models: A Survey.arXiv preprint arXiv:2212.10403, 2023. doi: 10.48550/arXiv.2212.10403. doi: 10.48550/arXiv.2212.10403

    2023 doi arXiv

  23. [30]

    A study of generative large language model for medical research and healthcare.npj Digital Medicine, 6:1–17, 2023

    Cheng Peng, Xi Yang, Aokun Chen, et al. A study of generative large language model for medical research and healthcare.npj Digital Medicine, 6:1–17, 2023. doi: 10.1038/s41746-023-00958-w

    2023 doi

  24. [31]

    Structured information extraction from scientific text with large language models.Nature Communications, 15(1):1418, 2024

    John Dagdelen, Alexander Dunn, Sanghoon Lee, et al. Structured information extraction from scientific text with large language models.Nature Communications, 15(1):1418, 2024. doi: 10.1038/s41467-024-45563-x

    2024 doi

  25. [32]

    PNAS home, 2025

    National Academy of Sciences. PNAS home, 2025. Proceedings of the National Academy of Sciences Website. Available athttps://www.pnas.org/. Accessed 3 January 2025

    2025

  26. [33]

    text-embedding-3-small, 2024

    OpenAI. text-embedding-3-small, 2024. OpenAI Platform Website. Available at https://platform.openai. com/docs/models/text-embedding-3-small/. Accessed 9 February 2025

    2024

  27. [34]

    GPT-4o mini, 2024

    OpenAI. GPT-4o mini, 2024. OpenAI Platform Website. Available at https://platform.openai.com/docs/ models/gpt-4o-mini. Accessed 9 February 2025

    2024

  28. [35]

    Houghton Mifflin, Boston, MA, 1935

    George Zipf.The Psycho-Biology of Language: an Introduction to Dynamic Philology. Houghton Mifflin, Boston, MA, 1935. Reprinted by Routledge, 2014

    1935

  29. [36]

    Zipf’s word frequency law in natural language: A critical review and future directions

    Steven Piantadosi. Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic Bulletin & Review, 21:1112–1130, 2014. doi: 10.3758/s13423-014-0585-6

    2014 doi

  30. [37]

    Power laws, Pareto distributions, and Zipf’s law.Contemporary Physics, 46(5):323–351, 2005

    Mark Newman. Power laws, Pareto distributions, and Zipf’s law.Contemporary Physics, 46(5):323–351, 2005. doi: 10.1080/00107510500052444. doi: 10.1080/00107510500052444

    2005 doi

  31. [38]

    Bertopic: Neural topic modeling with a class-based tf-idf procedure.arXiv preprint arXiv:2203.05794, 2022

    Marten Grootendorst. Bertopic: Neural topic modeling with a class-based tf-idf procedure.arXiv preprint arXiv:2203.05794, 2022. doi: 10.48550/arXiv.2203.05794

    2022 doi arXiv

  32. [39]

    Multiscale Visualization and Exploration of Large Bipartite Graphs.Computer Graphics Forum, 37(3):549–560, 2018

    Nicola Pezzoti, Jean-Daniel Fekete, Thomas Höllt, et al. Multiscale Visualization and Exploration of Large Bipartite Graphs.Computer Graphics Forum, 37(3):549–560, 2018. doi: 10.1111/cgf.13441

    2018 doi

  33. [41]

    Brittle and ductile yielding in soft materials.Proceedings of the National Academy of Sciences, 121(22):e2401409121, 2024

    Krutarth Kamani and Simon Rogers. Brittle and ductile yielding in soft materials.Proceedings of the National Academy of Sciences, 121(22):e2401409121, 2024. doi: 10.1073/pnas.2401409121

    2024 doi

  34. [42]

    A quantitative discriminant method of elbow point for the optimal number of clusters in clustering algorithm.EURASIP Journal on Wireless Communications and Networking, 31, 2021

    Congming Shi, Bingtao Wei, Shoulin Wei, Wen Wang, and Jialei Liu. A quantitative discriminant method of elbow point for the optimal number of clusters in clustering algorithm.EURASIP Journal on Wireless Communications and Networking, 31, 2021. doi: 10.1186/s13638-021-01910-w

    2021 doi

  35. [43]

    Approaches for finding Optimal Number of Clusters using K-Means and Agglomerative Hierarchical Clustering Techniques

    Punyaban Patel, Borra Sivaiah, and Riyam Patel. Approaches for finding Optimal Number of Clusters using K-Means and Agglomerative Hierarchical Clustering Techniques. InProceedings of the International Conference on Intelligent Controller and Computing for Smart Power, ICICCSP,...

    2022

  36. [44]

    Improving spectral clustering with deep embed- ding, cluster estimation and metric learning.Knowledge and Information Systems, 63:675–694, 2021

    Liang Duan, Shuai Ma, Charu Aggarwal, and Sather Saket. Improving spectral clustering with deep embed- ding, cluster estimation and metric learning.Knowledge and Information Systems, 63:675–694, 2021. doi: 10.1007/s10115-020-01530-8. 17

    2021 doi

  37. [45]

    Lost in the Middle: How Language Models Use Long Contexts

    Nelson Liu, Kevin Lin, John Hewitt, et al. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024. doi: 10.1162/tacl_a_00638

    2024 doi

  38. [46]

    Attention Is All You Need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, et al. Attention Is All You Need. InProceedings of the 31st Conference on Neural Information Processing Systems, NIPS, volume 30, Long Beach, California, 2017. Advances in Neural Information Processing Systems, Curran Associates, Inc....

    2017

  39. [47]

    Defining T cell receptor repertoires using nanovial-based binding and functional screening.Proceedings of the National Academy of Sciences, 121(14):e2320442121, 2024

    Doyeon Koo, Zhiyuan Mao, Mikayo Noguchi, et al. Defining T cell receptor repertoires using nanovial-based binding and functional screening.Proceedings of the National Academy of Sciences, 121(14):e2320442121, 2024. doi: 10.1073/pnas.2320442121

    2024 doi

  40. [48]

    Detecting hallucinations in large language models using semantic entropy.Nature, 630:625–630, 2024

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy.Nature, 630:625–630, 2024. doi: 10.1038/s41586-024-07421-0

    2024 doi

  41. [49]

    Highway Construction Safety Analy- sis Using Large Language Models.Applied Sciences, 14(4):1352, 2024

    Mason Smetana, Lucio Salles de Salles, Igor Sukharev, and Lev Khazanovich. Highway Construction Safety Analy- sis Using Large Language Models.Applied Sciences, 14(4):1352, 2024. doi: https://doi.org/10.3390/app14041352

    2024 doi

  42. [50]

    Text and Code Embeddings by Contrastive Pre-Training.arXiv preprint arXiv:2201.10005, 2022

    Arvind Neelakantan, Tao Xu, Raul Puri, et al. Text and Code Embeddings by Contrastive Pre-Training.arXiv preprint arXiv:2201.10005, 2022. doi: 10.48550/arXiv.2201.10005

    2022 doi arXiv

  43. [51]

    Language Models are Few-Shot Learners

    Tom Brown, Benjamin Mann, Nick Ryder, et al. Language Models are Few-Shot Learners. InProceedings of the 34th International Conference on Neural Information Processing Systems, NIPS, volume 33, pages 1877–1907, Online, 2020. Advances in Neural Information Processing Systems

    1907

  44. [52]

    Language Models are Unsupervised Multitask Learners

    Alec Radford, Jeffrey Wu, Rewon Child, et al. Language Models are Unsupervised Multitask Learners. OpenAI preprint. Accesssed 14 November 2024, 2019. URL https://cdn.openai.com/better-language-models/ language_models_are_unsupervised_multitask_learners.pdf

    2024

  45. [53]

    Explaining How Transformers Use Context to Build Predictions

    Javier Ferrando, Gerald Gállego, Ioannis Tsiamas, and Marta Costa-jussà. Explaining How Transformers Use Context to Build Predictions. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics, ACL, volume 1, pages 5486–5513, Toronto, Canada, 20...

    2023 doi

  46. [55]

    An Overview of Bag of Words;Importance, Implementation, Applications, and Challenges

    Wisam Qader, Musa Ameen, and Bilal Ahmed. An Overview of Bag of Words;Importance, Implementation, Applications, and Challenges. InProceedings of the International Engineering Conference, IEC, pages 200–204, Erbil, Iraq, 2019. IEEE. doi: 10.1109/IEC47844.2019.8950616

    2019

  47. [56]

    On the rise of fear speech in online social media.Proceedings of the National Academy of Sciences, 120(11):e2212270120, 2023

    Punyajoy Saha, Kiran Garimella, Narla Kalyan, et al. On the rise of fear speech in online social media.Proceedings of the National Academy of Sciences, 120(11):e2212270120, 2023. doi: 10.1073/pnas.2212270120

    2023 doi

  48. [57]

    Visualizing association rules in hierarchical groups.Journal of Business Economics, 87:317–335, 2017

    Michael Hahsler and Radoslaw Karpienko. Visualizing association rules in hierarchical groups.Journal of Business Economics, 87:317–335, 2017. doi: 10.1007/s11573-016-0822-8

    2017 doi

  49. [58]

    A Review on recent research in information retrieval.Procedia Computer Science, 201:777–782, 2022

    Sara Ibrihich, Ahmed Oussous, Ouafaa Ibrihich, and Mustapha Esghir. A Review on recent research in information retrieval.Procedia Computer Science, 201:777–782, 2022. doi: 10.1016/j.procs.2022.03.106

    2022 doi

  50. [59]

    Automatic Taxonomy Extraction from Bipartite Graphs

    Tobias Kötter, Stephan Günnemann, Michael Berthold, and Christos Faloutsos. Automatic Taxonomy Extraction from Bipartite Graphs. InProceedings of the IEEE International Conference on Data Mining, pages 221–230, Atlantic City, NJ, USA, 2015. IEEE. doi: 10.1109/ICDM.2015.24

    2015 doi

  51. [60]

    Engineering

    Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE.Journal of Machine Learning Research, 9(86):2579–2605, 2008. URLhttp://jmlr.org/papers/v9/vandermaaten08a.html. 18 Appendix A Primary Abstract Classification Let X={x 1, x2, . . . , xn} be a set of abstra...

    2008

  52. [63]

    Neuroscience 2,008 2,109 1,699 1,560 7,376 9.74%

  53. [64]

    Biochemistry 2,002 1,902 1,424 1,103 6,431 8.49%

  54. [65]

    Biophysics and Computational Biology 1,611 1,571 1,248 1,155 5,585 7.37%

  55. [66]

    Medical Sciences 1,738 1,501 1,036 718 4,993 6.59%

  56. [67]

    Microbiology 1,081 1,215 1,046 1,156 4,498 5.94%

  57. [68]

    Cell Biology 1,268 1,296 906 902 4,372 5.77%

  58. [69]

    Immunology and Inflammation 1,108 1,223 902 912 4,145 5.47%

  59. [70]

    Evolution 843 813 748 690 3,094 4.09%

  60. [71]

    Genetics 878 769 587 505 2,739 3.62%

  61. [72]

    Plant Biology 564 643 609 546 2,362 3.12%

  62. [73]

    Ecology 444 480 574 612 2,110 2.79%

  63. [74]

    Developmental Biology 605 552 365 348 1,870 2.47%

  64. [75]

    Physiology 605 552 365 348 1,870 2.47%

  65. [76]

    Applied Biological Sciences 306 372 303 321 1,302 1.72%

  66. [77]

    Psychological and Cognitive Sciences 159 234 224 213 830 1.10%

  67. [78]

    Environmental Sciences 134 255 202 189 780 1.03%

  68. [79]

    Pharmacology 207 188 143 123 661 0.87%

  69. [80]

    Anthropology 110 164 134 120 528 0.70%

  70. [81]

    Systems Biology 10 192 152 114 468 0.62%

  71. [82]

    Agricultural Sciences 88 103 106 155 452 0.60%

  72. [83]

    Sustainability Science 66 96 103 114 379 0.50%

  73. [84]

    Population Biology 57 54 64 97 272 0.36% All Biological Sciences Subcategories 15,699 16,104 12,879 11,949 56,631 74.8%

  74. [85]

    Chemistry 868 984 866 944 3,662 4.83%

  75. [86]

    Earth, Atmospheric, and Planetary Sciences 224 496 633 765 2,118 2.80%

  76. [87]

    Applied Physical Sciences 294 541 588 664 2,087 2.76%

  77. [88]

    Physics 246 462 542 556 1,806 2.38%

  78. [89]

    Engineering 129 298 432 647 1,506 1.99%

  79. [90]

    Biophysics and Computational Biology – – 387 754 1,141 1.51%

  80. [91]

    Environmental Sciences 117 238 218 222 795 1.05%

  81. [92]

    Applied Mathematics 154 157 153 179 643 0.85%

  82. [93]

    Statistics 53 66 89 89 297 0.39%

  83. [94]

    Computer Sciences 42 56 67 123 288 0.38%

  84. [95]

    Mathematics 50 82 37 34 203 0.27%

  85. [96]

    Astronomy 13 38 21 30 102 0.13%

  86. [97]

    Sustainability Science – – – 65 65 0.09% All Physical Sciences Subcategories 2,190 3,418 4,033 5,072 14,713 19.4%

  87. [98]

    Psychological and Cognitive Sciences 153 377 568 559 1,657 2.19%

  88. [99]

    Sustainability Science 62 181 227 187 657 0.87%

  89. [100]

    Social Sciences 65 132 207 193 597 0.79%

  90. [101]

    Anthropology 101 139 179 143 562 0.74%

  91. [102]

    Economic Sciences 60 126 127 196 509 0.67%

  92. [103]

    Environmental Sciences 18 45 63 102 228 0.30%

  93. [104]

    Political Sciences 101 139 179 143 562 0.74%

  94. [105]

    No. Cluster

    Demography – – – 30 30 0.04% All Social Sciences Subcategories 466 1,015 1,409 1,506 4,396 5.80% 32 Table 3: Sixteen LLM-derived topics and distributions of initial clustering (“No. Cluster”) and reclassification (“No. Class”). At j represents the corresponding agreement score...

  95. [106]

    Study of tissue engineering and mechanotransduction, emphasizing the role of extracellular matrix and stem cell behavior in biomaterials

  96. [107]

    Advancements in cancer treatment utilizing nanoparticle technology for targeted drug delivery and personalized medicine

  97. [108]

    Research on catalysis and energy storage technologies, focusing on CO 2 conversion and sustainable lithium battery solutions

  98. [109]

    Exploration of cutting-edge electronic and photonic technologies, focusing on 2D materials and flexible electronics

  99. [110]

    Innovations in diagnostic technologies aimed at enhancing healthcare outcomes through real-time analysis and personalized medicine

  100. [111]

    Investigation of mechanical properties at micro and nanoscale levels, focusing on micromechanics and additive manufac- turing

  101. [112]

    Development of flexible and biointegrated technologies in biomedical engineering, focusing on wearable health monitoring solutions

  102. [113]

    Innovations in synthetic biology, emphasizing tools and methodologies for gene regulation and metabolic processes

  103. [114]

    Insights into bioinspired robotics, highlighting locomotion and collective behavior derived from natural systems

  104. [115]

    Innovations in soft robotics, emphasizing advancements in material design and actuation mechanisms for biomedical applications

  105. [116]

    Exploration of innovative materials and technologies that can change shape and function, enhancing applications in soft robotics and programmable matter

  106. [117]

    Insights into fluid dynamics, including turbulent flow and viscoelastic fluids, with applications in active systems

  107. [118]

    Exploration of cutting-edge advancements in microfluidics technology, focusing on fluid manipulation and particle analysis for various applications

  108. [119]

    Development of advanced technologies for water purification, emphasizing membrane technology and sustainable wastew- ater treatment solutions

  109. [120]

    Advancements in bioengineering and human-machine interfaces aimed at enhancing medical technology and sustainable agriculture

  110. [121]

    Transformative advances in biological imaging methods, including 3D and Raman imaging for nanoscale resolution

  111. [122]

    Same” column indicates how many segments were classified under the same parent abstract topic (uik =y i). The “To Other

    None of the other categories fit this text. 33 Table 4: Summary of secondary LLM (full-text) classification. For each topic, and therefore each abstract, due to multi-label tolerance, a significant number of segments were classified under more than one topic in S∗. The “Same” ...

  112. [2015]

    doi: 10.1073/pnas.1509912112

    doi

  113. [2020]

    doi: 10.1007/s11192-020-03441-5. 16

    doi

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.