Pith. sign in

REVIEW 4 major objections 1 minor 5 references

AI publications are 5.5 to 10.2 percentage points more likely than non-AI ones to rank in the top creativity decile.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-28 03:27 UTC pith:3GSSTD7M

load-bearing objection AI papers show higher top-decile creativity in OpenAlex data with mode-specific novelty patterns, but the result hinges on untested classification steps. the 4 major comments →

arxiv 2606.05118 v1 pith:3GSSTD7M submitted 2026-06-03 cs.CY

Does Artificial Intelligence Advance Science?

Liangping Ding , Cornelia Lawson , Philip Shapira This is my paper
classification cs.CY
keywords artificial intelligencescientific creativityrecombinant noveltyobject noveltyresearch modescitation impactOpenAlex
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether adopting artificial intelligence increases scientific creativity and finds that it does, but not uniformly. Using more than one million OpenAlex publications, the authors measure creativity through recombinant novelty, object novelty, and citation impact, then compare AI and non-AI papers. They report that AI papers reach the top creativity decile at substantially higher rates. The gains differ by research mode: applying existing AI tools to new domains produces the strongest recombinant-novelty boost, while adapting AI models to domain problems produces the strongest object-novelty boost. The results imply that AI advances science through separate creative routes rather than a single mechanism.

Core claim

AI publications are significantly more likely to achieve top-decile creativity relative to non-AI publications, with 5.5 to 10.2 percentage point higher likelihood to rank in the top creativity decile. Tool-oriented AI research is associated with the largest gains in recombinant-based creativity, while adaptation-oriented AI research is associated with relatively higher object-based creativity. These findings show that AI does not advance science through a single mechanism but through structurally distinct creative pathways that depend on how AI is incorporated into the research process.

What carries the argument

The distinction between tool-oriented AI research (applying existing models) and adaptation-oriented AI research (modifying models for domain problems) and their separate links to recombinant novelty versus object novelty.

Load-bearing premise

The rules used to label publications as AI-related and to score recombinant novelty, object novelty, and citation impact in OpenAlex correctly capture real AI use and genuine creativity without large measurement error or bias.

What would settle it

Repeating the decile-ranking analysis after replacing the OpenAlex AI labels or the novelty/impact formulas with independent alternative definitions and finding that the 5.5–10.2 point gap disappears.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Tool-oriented AI use produces the largest increase in recombinant novelty.
  • Adaptation-oriented AI use produces the largest increase in object novelty.
  • Different ways of folding AI into research generate different types of scientific contribution.
  • Assessment frameworks should separate recombinant from conceptual forms of creativity when evaluating AI-assisted work.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Policy that funds only one mode of AI adoption may miss gains available from the other mode.
  • Citation-based impact alone may understate the creativity contribution of adaptation-oriented AI work.
  • Future studies could test whether the same pattern holds when novelty is measured by expert panels instead of text-based indicators.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 1 minor

Summary. The manuscript analyzes over one million OpenAlex publications to assess whether AI adoption advances scientific creativity. It reports that AI papers have a 5.5 to 10.2 percentage point higher probability of ranking in the top creativity decile across measures of recombinant novelty, object novelty, and citation impact. The analysis also identifies heterogeneity by AI research mode, with tool-oriented approaches linked to recombinant creativity gains and adaptation-oriented approaches to object novelty gains.

Significance. Should the measurement of AI adoption and creativity dimensions prove robust, the results would suggest that AI contributes to science via multiple distinct pathways rather than a uniform mechanism. This has potential implications for research assessment frameworks and science policy, emphasizing the need to differentiate between forms of creativity. The large-scale data approach is a positive if accompanied by transparent methods.

major comments (4)
  1. [Methods] Methods section: The classification of publications as AI or non-AI is central to the main claim but the abstract provides no information on the OpenAlex query, keyword set, or machine learning classifier employed; this must be detailed to allow assessment of potential selection bias or field-specific effects.
  2. [Methods] Methods section: Recombinant novelty and object novelty metrics are not described (e.g., whether recombinant novelty uses new reference combinations or concept co-occurrences); without this, it is impossible to evaluate whether the 5.5-10.2 pp difference reflects genuine creativity or measurement artifacts.
  3. [Results] Results section: The split into tool-oriented vs. adaptation-oriented AI research requires explicit, operational definitions and robustness checks; the differential associations with creativity types are load-bearing for the heterogeneity conclusion but vulnerable to classification error.
  4. [Discussion] Discussion section: Potential confounders such as subfield, team size, or data availability are not mentioned as controlled; if these correlate with AI adoption, they could explain the observed differences without implying a causal advance in creativity.
minor comments (1)
  1. [Abstract] Abstract: The abstract mentions 'over one million publications' but does not specify the time period or exact sample construction criteria.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for these constructive comments, which help improve the transparency and robustness of our analysis. We address each major point below and have revised the manuscript to incorporate additional methodological detail and controls.

read point-by-point responses

  1. Referee: [Methods] Methods section: The classification of publications as AI or non-AI is central to the main claim but the abstract provides no information on the OpenAlex query, keyword set, or machine learning classifier employed; this must be detailed to allow assessment of potential selection bias or field-specific effects.

    Authors: We agree that full details on the AI/non-AI classification are essential. The Methods section of the original manuscript contained a high-level description, but we have now expanded it substantially to report the precise OpenAlex query, the keyword list used for initial filtering, the machine-learning classifier architecture and training procedure, and out-of-sample performance metrics. These additions allow direct evaluation of selection bias and field coverage. revision: yes

  2. Referee: [Methods] Methods section: Recombinant novelty and object novelty metrics are not described (e.g., whether recombinant novelty uses new reference combinations or concept co-occurrences); without this, it is impossible to evaluate whether the 5.5-10.2 pp difference reflects genuine creativity or measurement artifacts.

    Authors: We accept that the operational definitions were insufficiently explicit. We have added a dedicated subsection in Methods that specifies recombinant novelty as the share of previously unobserved reference-pair combinations (following the standard combinatorial approach) and object novelty as the introduction of previously unseen scientific concepts or entities. We also include the exact formulas, data sources, and validation checks against alternative concept-based measures to demonstrate that the reported differences are not artifacts of a single operationalization. revision: yes

  3. Referee: [Results] Results section: The split into tool-oriented vs. adaptation-oriented AI research requires explicit, operational definitions and robustness checks; the differential associations with creativity types are load-bearing for the heterogeneity conclusion but vulnerable to classification error.

    Authors: We have inserted explicit, replicable definitions in the revised Results section: tool-oriented papers are those that apply off-the-shelf AI models without modification, while adaptation-oriented papers are those that fine-tune, extend, or re-architect AI components for the domain problem. We further report two alternative classification schemes (keyword-based and embedding-based) together with robustness tables showing that the differential associations with recombinant versus object novelty remain stable across these specifications. revision: yes

  4. Referee: [Discussion] Discussion section: Potential confounders such as subfield, team size, or data availability are not mentioned as controlled; if these correlate with AI adoption, they could explain the observed differences without implying a causal advance in creativity.

    Authors: Subfield fixed effects were already included in all main specifications; we have now made this explicit in both the Methods and Discussion sections. In the revision we additionally control for team size (log number of authors) and include a proxy for data availability (whether the paper mentions a public dataset). After these controls the AI coefficient remains positive and significant, though we acknowledge that observational data cannot fully rule out residual confounding and have updated the Discussion to reflect this limitation. revision: partial

Circularity Check

0 steps flagged

No circularity: observational analysis on external data

full rationale

The paper reports empirical associations from over one million OpenAlex publications comparing AI vs. non-AI papers on pre-defined novelty and impact metrics. No equations, fitted parameters renamed as predictions, self-citation load-bearing derivations, or ansatzes appear in the provided text or abstract. Classifications and decile rankings are operational choices applied to external data rather than self-referential constructions. The central claim is a statistical comparison, not a derivation that reduces to its inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the validity of bibliometric proxies for creativity and the accurate categorization of AI research modes from OpenAlex metadata, which are treated as given without independent validation in the abstract.

axioms (1)
  • domain assumption OpenAlex database provides accurate and complete metadata for identifying AI publications and computing novelty and citation metrics.
    The entire comparison depends on the quality of this external database for both exposure and outcome variables.

pith-pipeline@v0.9.1-grok · 5766 in / 1267 out tokens · 54764 ms · 2026-06-28T03:27:01.547238+00:00 · methodology

0 comments
read the original abstract

This paper examines whether and how artificial intelligence (AI) advances scientific creativity. Drawing on scientific publications, the primary output of researchers, we analyze over one million publications from OpenAlex to investigate the relationship between AI adoption and multiple dimensions of scientific creativity, including novelty (recombinant novelty and object novelty) and impact (3-year short-run citation impact and 10-year long-run citation impact). We find that AI publications are significantly more likely to achieve top-decile creativity relative to non-AI publications, with 5.5 to 10.2 percentage point higher likelihood to rank in the top creativity decile. Critically, we uncover substantial heterogeneity across AI research modes. Tool-oriented AI research, which applies existing AI models to domain tasks, is associated with the largest gains in recombinant-based creativity, while Adaptation-oriented AI research, modifying AI models for domain-specific problems, is associated with relatively higher object-based creativity. These findings reveal that AI does not advance science through a single mechanism but through structurally distinct creative pathways that depend on how AI is incorporated into the research process. Our results contribute to ongoing debates about AI's role in science and carry direct implications for research evaluation and science policy, highlighting the need for assessment frameworks that can distinguish between recombinant and conceptual forms of creativity and that recognize how different modes of AI adoption produce fundamentally different types of scientific contribution.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

5 extracted references · 4 canonical work pages

  1. [1]

    S., & Goldfarb, A

    Agrawal, A., Gans, J. S., & Goldfarb, A. (2019). Artificial intelligence: The ambiguous labor market impact of automating prediction. Journal of Economic Perspectives, 33(2), 31–50. Amabile, T. M. (2018). Creativity in context: Update to the social psychology of creativity. Routledge. Arts, S., Melluso, N., & Veugelers, R. (2025). Beyond Citations: Measur...

    work page doi:10.1162/rest_a_01561 2019

  2. [2]

    Death of the Renaissance Man

    https://doi.org/10.1038/s41467-022-28865-w Brynjolfsson, E., Li, D., & Raymond, L. (2025). Generative AI at Work. The Quarterly Journal of Economics, qjae044. https://doi.org/10.1093/qje/qjae044 Cockburn, I. M., Henderson, R., & Stern, S. (2018). The impact of artificial intelligence on innovation: An exploratory analysis. In The economics of artificial i...

    work page doi:10.1038/s41467-022-28865-w 2025

  3. [3]

    Leahey, E., Lee, J., & Funk, R. J. (2023). What types of novelty are most disruptive? American Sociological Review, 88(3), 562–597. Lee, Y.-N., Walsh, J. P., & Wang, J. (2015). Creativity in scientific teams: Unpacking novelty and impact. Research Policy, 44(3), 684–697. https://doi.org/10.1016/j.respol.2014.10.007 March, J. G. (1991). Exploration and exp...

    work page doi:10.1016/j.respol.2014.10.007 2023

  4. [4]

    (Gordon), & Agarwal, R

    https://doi.org/10.1016/j.respol.2017.06.006 Wang, W., Gao, G. (Gordon), & Agarwal, R. (2023). Friend or Foe? Teaming Between Artificial Intelligence and Workers with Variation in Experience. Management Science, mnsc.2021.00588. https://doi.org/10.1287/mnsc.2021.00588 34 TABLES Table 1 Summary Statistics N Mean SD Min Max Novelty Recomb. 1127716 0.172 0.3...

    work page doi:10.1016/j.respol.2017.06.006 2017

  5. [5]

    Mean Differences between AI and non-AI Publications Non-AI AI Adaptation Foundational Tool Novelty Recomb. 0.096 0.218 a 0.170 a 0.152 ai 0.270 ahx Novelty Object 0.224 0.206 b 0.234 a 0.234 a 0.179 biy Impact 3-yr 0.283 0.322 a 0.343 a 0.363 ah 0.297 aiy Impact 10-yr 0.261 0.302 a 0.308 a 0.333 ah 0.291 aiy Creativity Recomb. 3yr 0.070 0.135 a 0.106 a 0....

    2015