Pith. sign in

REVIEW 4 major objections 5 minor 47 references

Using Large Language Models to Study Mathematical Practice

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read This paper reports that a large language model can scan 5,000 mathematics papers, recognize genuine discussions of explanation, and produce an annotated corpus large enough to ground empirically driven philosophy of mathematics.

desk verdict First real LLM-corpus study in PMP; the dataset and honest methodology make it worth refereeing, but the quantitative claims need external validation. read the letter →

arxiv 2507.02873 v1 pith:6M6K766X submitted 2025-06-16 math.HO cs.AI

classification math.HOcs.AI MSC 00A3000A35
keywords mathematicalexplanationphilosophyofpracticelargelanguagemodelscorpusanalysisword-conceptproblemexplanatoryproofsubfieldvariationLLMannotation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that current large language models can do reliable, large-scale corpus analysis for the philosophy of mathematical practice, focused on when mathematicians discuss explanation. It uses Gemini 2.5 Pro to read 5,000 randomly sampled mathematics papers, prompted with a philosophical survey that defines the target concept, and produces a dataset of hundreds of annotated examples. The point is to move past cherry-picked case studies and ambiguous keyword counting, letting the model reason about meaning rather than just count words like 'explain'. On the empirical side, the paper reports that clear explanation talk appears in at least a few percent of recent papers, that its frequency varies by subfield, and that the model's analysis favors unificationist and ontic theories supplemented with epistemic and pragmatic elements. It concludes that LLM-assisted corpus work is feasible and valuable, while acknowledging that the dataset is imperfect.

What carries the argument

The machinery is a large language model used as a semantic annotator: Gemini 2.5 Pro, whose one-million-token context window and integrated chain-of-thought reasoning let it process batches of 25 papers per query. The prompt embeds a roughly 5,000-word excerpt from an encyclopedia survey on mathematical explanation, which defines the target concept and supplies examples, along with instructions to quote sources and avoid hallucination. A Python script written by the model itself automated 200 runs over about 24 hours, and a second, stricter filtering prompt removed 50–60% of candidates, leaving the final annotated dataset. The defining move is replacing keyword counting with a prompt that asks the model to judge whether the concept of explanation is genuinely in play in each passage, regardless of the specific words used.

What would settle it

Take a random sample of about 200 of the 5,000 corpus papers, have a panel of mathematicians and philosophers independently mark every passage that clearly discusses mathematical explanation, and compare their annotations with Gemini's flags on the same papers; if agreement is near chance or the model misses most passages experts agree on, the claim that the model can do accurate corpus work fails.

Watch

Extended reading notes

Core claim

The central discovery is that a current frontier LLM, prompted with a nuanced definition of mathematical explanation and run over a 5,000-paper random sample of the mathematics preprint archive, can produce useful annotations at a scale impossible by manual reading, thereby sidestepping the word-concept problem that plagued keyword-counting corpus methods. The author reports that Gemini 2.5 Pro identified roughly 1,250 candidate instances from about 735 distinct papers, with roughly 20% high-quality cases, 20% low-quality cases, and 60% borderline cases by his estimate. On substantive empirical questions, the paper claims that at least about 3% of papers contain clear explanation claims and about 12% contain borderline-or-better cases; that explanatory practice varies by subfield, with coefficients of explanatory richness ranging from 0.77 for probability and statistics to 1.32 for logic and set theory, with combinatorics at 1.19; and that when asked to adjudicate philosophical theories, the model argues for a combination of unificationism and ontic structure-revealing accounts, supplemented by epistemic or pragmatic elements to handle reproofs, heuristic explanations, and demystifications. A further reported finding is the model's observation that obstructions, or reasons why a strategy fails, often function as explanatory concerns, which the author identifies as a promising new research direction.

Load-bearing premise

The whole empirical picture rests on trusting that Gemini 2.5 Pro reads mathematics papers accurately enough to label explanations, since the author checked its outputs only informally and even he estimates that a fifth of the final dataset is low quality and another sixty percent is borderline.

Editorial extensions

If this is right

  • The same pipeline can be scaled to much larger corpora, including the full repository of roughly 80,000 mathematics preprints, making corpus philosophy of mathematics a practical research program.
  • The resulting dataset gives philosophers hundreds of clear and borderline examples of mathematical explanation that were not selected for their convenience or theory-friendliness.
  • Explanatory practice is not uniform across mathematics: subfield-specific richness coefficients imply that combinatorics and logic-and-set-theory communities invoke explanation more than average, while probability and statistics invoke it less.
  • The rarity debate about mathematical explanation is sharpened: at least about 3–12% of recent preprint papers engage explanation talk, enough to count as a settled practice without being ubiquitous.
  • LLMs can act as interpretative assistants, not just annotators: querying the dataset yields targeted examples, such as tradeoffs between explanatory and other proof virtues, and can propose new philosophical projects like the study of obstructions as explanations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If LLM annotation is validated against human gold standards, the prompting-plus-filtering pattern could be exported to other philosophically loaded concepts such as beauty, simplicity, depth, and naturalness, giving empirical philosophy a general tool rather than a one-off study.
  • The subfield variation suggests a testable sociological hypothesis beyond the paper's table: explanatory talk clusters where mathematical objects admit multiple representational perspectives, such as algebra, geometry, topology, and combinatorics, and thins where a single analytic or computational framework dominates.
  • The model's obstruction observation points to a concrete coding extension: future studies could explicitly tag explanations of failure, such as why a theorem is false, why a method fails, or why a construction is obstructed, to see whether that category forms a substantial share of mathematical explanation.
  • One could push the paper's quality-filtering idea further by using the model's chain-of-thought traces to build self-scored confidence labels for each example, reducing the need for human spot checks in larger follow-up studies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports a large-scale corpus study of mathematical explanation in 5,000 arXiv mathematics papers, using Google's Gemini 2.5 Pro as an automated annotator. The author describes a pipeline in which the model is prompted with a long excerpt from the Stanford Encyclopedia of Philosophy article on mathematical explanation, processes papers in batches of 25, produces candidate annotated examples, and then applies a second strict filtering prompt. The resulting dataset contains roughly 1,250 examples from about 735 papers. From this dataset the paper reports subfield-level 'coefficients of explanatory richness' (Table 1), an estimate that at least ~3% of papers contain clear explanation claims and ~12% contain borderline-or-better cases, several targeted queries of the dataset, and an extended experiment in which Gemini adjudicates between philosophical theories of explanation and proposes a novel theory ('Explanatory Resonance Theory'). The paper frames itself as a proof of concept, arguing that current LLMs can perform corpus work that is 'sophisticated, accurate and interesting' on a scale impossible by other means.

Significance. If the central claim holds, the paper would establish a genuinely new methodological avenue for the philosophy of mathematical practice, moving beyond keyword-count corpus methods to concept-sensitive annotation at scale. The paper is transparent about its pipeline: the full prompt is quoted, the filtering instruction is quoted, and the dataset is made available as a 500+-page document. It also contains honest and explicit limitation statements, including the author's own estimate that roughly 20% of the filtered dataset is low-quality and 60% is borderline. These strengths are real. However, the paper's quantitative findings—the subfield coefficients and the prevalence estimates—rest on the reliability of Gemini's annotations, and that reliability is asserted rather than demonstrated. The absence of any gold-standard validation, inter-annotator agreement, or sensitivity analysis is a load-bearing gap, not a cosmetic one. The philosophical-adjudication experiment in §3.4 is also weakened by circularity: the same model that generated and filtered the dataset is asked to assess which theory best fits it.

major comments (4)
  1. [§2 (Methods) and §3.1 (Table 1)] The central claim that Gemini can do 'sophisticated, accurate and interesting corpus work' (§1) is not backed by systematic validation. The paper's own quality estimate at the end of §2 is that the filtered dataset is roughly 20% high-quality, 20% low-quality, and 60% borderline, with inclusion of borderline cases 'in part a matter of taste.' Yet Table 1 reports coefficients of explanatory richness (e.g., 1.32 for logic and set theory, 0.77 for probability and statistics) as though they are robust measurements. Because the filtering decision is made by the same model that created the examples, and because no gold standard, inter-annotator reliability measure, or human-expert audit is reported, the observed variation across subfields could easily shift under reasonable alternative coding decisions. A sensitivity analysis (e.g., recalculating coefficients under strict-only, borderline-included, and permissive inclusion policies) is needed before these numbers can support any empirical conclusion.
  2. [§3.2 (Prevalence of explanatory concerns)] The estimate that 'at least ~150 out of 5000 research papers (around 3%)' contain clear explanation claims relies on two unexamined assumptions: that the model's false-negative rate is negligible, and that example-level quality percentages transfer to paper-level counts. The paper explicitly says 'I expect it not to have missed large numbers of high-quality cases,' but recall is never measured. Since the filtering prompt instructed the model to exclude at least 50–60% of examples, the false-negative rate could be substantial, and the 'at least' claim would then be misleadingly low. The paper should report a recall-oriented validation (e.g., on a random subset of papers that human experts annotate independently) and present the prevalence estimate with confidence intervals or as a range across coding policies.
  3. [§3.4 (Gemini as philosophical assistant)] The adjudication experiment is circular in a way that limits its evidentiary value. The same model (Gemini 2.5 Pro) generated the initial candidate examples, applied the filtering prompt, and was then asked to evaluate which philosophical theory of explanation best fits the filtered dataset. Its assessment is therefore shaped by its own earlier filtering decisions and by the SEP excerpt embedded in the original prompt, which already presupposes a particular taxonomy of explanatory concepts. While the discussion is interesting as an illustration of what LLMs can produce, any claim that it 'helps settle debates between rival theories' (§1) requires at least a blind comparison with human expert judgments on the same dataset. As it stands, the model's pluralist/epistemic conclusion is better described as a hypothesis generated by the method than as evidence for that hypothesis.
  4. [§2 (Filtering prompt)] The filtering prompt includes the instruction 'You MUST exclude at least 50-60% of the original examples' and repeatedly urges the model to be 'ruthless.' This makes the filtering threshold a free parameter that directly influences all downstream quantitative claims, but no justification is given for choosing this threshold, and the paper reports that repeated applications of the filter produced no further changes. This is not merely a technical detail: the 3% and 12% prevalence estimates in §3.2 are computed from the filtered dataset, so the quantitative conclusions are partly determined by an arbitrarily chosen exclusion target. The paper should either justify the threshold empirically or present results as a function of filtering strictness.
minor comments (5)
  1. [§3.1] There is a typo: 'algebra, topology and and combinatorics' should read 'algebra, topology and combinatorics.'
  2. [§2] The prompt and filtering instructions are reproduced in full, which is excellent for reproducibility, but the exact model version used for each stage (e.g., 2.5 Pro Experimental vs. Preview) is not always specified in the results; this matters for replication.
  3. [§3.4] The sentence 'In general, It's difficult to see what such a broad and ill-defined construct might add' has an unnecessarily capitalised 'It's' after the period.
  4. [References] Reference [D'Alessandro 2025] is listed with 'DOI: XXXX,' which should be completed before publication.
  5. [§2] The paper states that the dataset is available at a URL, but a persistent identifier (e.g., a DOI or a stable repository link) would be more appropriate for a dataset that is central to the paper's claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the LLM pipeline is openly described, and the empirical claims are estimates from the produced dataset rather than reductions to the prompt.

full rationale

The paper does not derive any output from its inputs by construction. The prompt defines a target concept operationally and instructs the model to find and then filter examples; the resulting counts and ratios are descriptive statistics of that filtered dataset, not quantities fitted to reproduce a target answer. The article repeatedly discloses the dependence of the dataset on model judgment and on the author's own quality estimates, including the roughly 20% low-quality and 60% borderline cases, and it presents the philosophical adjudication in section 3.4 as preliminary. Self-citations such as D'Alessandro 2025 are used only to motivate the methodology, not to supply a uniqueness theorem or to forbid alternatives. The main concerns raised by the paper—same-model annotation and adjudication, no inter-annotator agreement, no sensitivity analysis—are validity and robustness issues for an empirical study, not circularity in the derivation chain.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper does not introduce new mathematical or physical entities. Its central claim rests on assumptions about the LLM's reliability, the adequacy of the operational definition of explanation, and the representativeness of the corpus. The free parameters are the author's subjective quality estimates and the ad hoc filtering threshold.

free parameters (2)
  • Quality estimate percentages = 20% high-quality, 20% low-quality, 60% borderline
    The author estimates these proportions by inspection of the filtered dataset; they are not measured with any quantitative method.
  • Filtering exclusion threshold = 50-60% excluded
    The filtering prompt instructs the model to exclude at least 50-60% of examples. This hand-chosen threshold affects the size and composition of the final dataset.
assumptions (3)
  • domain assumption Gemini 2.5 Pro produces accurate analyses of mathematical texts on long inputs.
    This is the central tooling assumption, supported only by the author's informal spot checks and claims of accuracy in the paper (Section 2).
  • domain assumption The Stanford Encyclopedia excerpt (Mancosu et al. 2023) provides an unbiased characterization of mathematical explanation.
    The prompt relies on this excerpt to define the target concept. Different or more contestable definitions could lead to different annotation results.
  • domain assumption The random sample of 5000 arXiv papers is representative enough of mathematical practice for the study's purposes.
    The author acknowledges arXiv is a recent and specific slice of mathematics. Representativeness is assumed but not established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Using Large Language Models to Study Mathematical Practice." pith.science (2026). https://pith.science/paper/6M6K766X

@misc{pith2026250702873,
  author       = {Pith},
  title        = {Pith review of: Using Large Language Models to Study Mathematical Practice},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6M6K766X}},
  note         = {Machine review of arXiv:2507.02873}
}
read the original abstract

The philosophy of mathematical practice (PMP) looks to evidence from working mathematics to help settle philosophical questions. One prominent program under the PMP banner is the study of explanation in mathematics, which aims to understand what sorts of proofs mathematicians consider explanatory and what role the pursuit of explanation plays in mathematical practice. In an effort to address worries about cherry-picked examples and file-drawer problems in PMP, a handful of authors have recently turned to corpus analysis methods as a promising alternative to small-scale case studies. This paper reports the results from such a corpus study facilitated by Google's Gemini 2.5 Pro, a model whose reasoning capabilities, advances in hallucination control and large context window allow for the accurate analysis of hundreds of pages of text per query. Based on a sample of 5000 mathematics papers from arXiv.org, the experiments yielded a dataset of hundreds of useful annotated examples. Its aim was to gain insight on questions like the following: How often do mathematicians make claims about explanation in the relevant sense? Do mathematicians' explanatory practices vary in any noticeable way by subject matter? Which philosophical theories of explanation are most consistent with a large body of non-cherry-picked examples? How might philosophers make further use of AI tools to gain insights from large datasets of this kind? As the first PMP study making extensive use of LLM methods, it also seeks to begin a conversation about these methods as research tools in practice-oriented philosophy and to evaluate the strengths and weaknesses of current models for such work.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 46 canonical work pages

  1. [1]

    Alekseev, Anton, Eckhard Meinrenken and Chris Woodward. 2000. ``Formulas of Verlinde type for non-simply connected groups.'' arXiv:math/0005047

  2. [2]

    The Complex Hyperbolic Geometry of the Moduli Space of Cubic Surfaces

    Allcock, Daniel, James A. Carlson and Domingo Toledo. 2000. ``The complex hyperbolic geometry of the moduli space of cubic surfaces.'' arXiv:math/0007048

  3. [3]

    Altman, Sam. 2025. ``The gentle singularity.'' Sam Altman, June 10 2025 (accessed June 13 2025), URL = https://blog.samaltman.com/the-gentle-singularity

  4. [4]

    Baron, Sam and Mark Colyvan. 2020. ``Explanation impossible.'' Philosophical Studies 178, 559--576

  5. [5]

    Bessis, David, Francois Digne and Jean Michel. 2000. ``Springer theory in braid groups and the Birman-Ko-Lee monoid.'' arXiv:math/0010254

  6. [6]

    Bressoud, David M. 2000. ``Three alternating sign matrix identities in search of bijective proofs.'' arXiv:math/0007114

  7. [7]

    Bryant, Robert L. 2001. ``Rigidity and quasi-rigidity of extremal cycles in Hermitian symmetric spaces.'' arXiv:math/0006186. english

  8. [8]

    Chartrand, Louis. 2022. ``Modeling and corpus methods in experimental philosophy.'' Philosophy Compass 17 (6), 2022. american

Show all 47 references
  1. [9]

    english Cherednik, Ivan. 2000. ``One-dimensional double Hecke algebras and Gaussians.'' arXiv:math/0003017

  2. [10]

    english Cohn, Henry, Noam Elkies and James Propp. 2000. ``Local statistics for random domino tilings of the Aztec diamond.'' arXiv:math/0008243

  3. [11]

    english D'Alessandro, William. 2020. ``Mathematical explanation beyond explanatory proof.'' British Journal for the Philosophy of Science 71, 581--603

  4. [12]

    english D'Alessandro, William. 2021. ``Proving quadratic reciprocity: Explanation, disagreement, transparency and depth.'' Synthese 198, 8621--8664

  5. [13]

    english D'Alessandro, William. 2023. ``Unrealistic models in mathematics.'' Philosophers' Imprint 23, DOI: 10.3998/phimp.1712

  6. [14]

    D'Alessandro, William. 2025. ``Toward a methodology for the philosophy of mathematical practice.'' Philosophy of Science, DOI: XXXX

  7. [15]

    D'Alessandro, William and Ellen Lehet. 2024. ``A noetic account of explanation in mathematics.'' Philosophical Quarterly, DOI: 10.1093/pq/pqae137

  8. [16]

    and Jonghae Keum

    Dolgachev, Igor V. and Jonghae Keum. 2001. ``Birational automorphisms of quartic Hessian surfaces.'' arXiv:math/0103186

  9. [17]

    Fill, James Allen and Svante Janson. 2000. ``Smoothness and decay properties of the limiting Quicksort density function.'' arXiv:math/0005235

  10. [18]

    Gannon, Terry. 2002. ``Modular data: The algebraic combinatorics of conformal field theory.'' arXiv:math/0103044

  11. [19]

    Guillemin, Victor and Catalin Zara. 2000. ``Combinatorial formulas for products of Thom classes.'' arXiv:math/0007166. english

  12. [20]

    Inglis, Matthew and Andrew Aberdein. 2015. ``Beauty is not simplicity: An analysis of mathematicians' proof appraisals.'' Philosophia Mathematica 23, 87-109

  13. [21]

    Inglis, Matthew and Andrew Aberdein. 2016. ``Diversity in proof appraisal.'' In Brendan Larvor (ed.), Mathematical Cultures, Birkhäuser: Switzerland, 163--179. american

  14. [22]

    english Johansen, Mikkel Willum, Morten Misfeldt and Josefine Lomholt Pallavicini. 2018. ``A typology of english mathematical diagrams.'' In Peter Chapman, Gem Stapleton, Amirouche Moktefi, Sarah Perez-Kriz english & Francesco Bellucci (eds.), Diagrammatic Representation and I...

  15. [23]

    english Kaloshin, Vadim. 2000. ``A geometric proof of the existence of Whitney stratifications.'' arXiv:math/0010144. english

  16. [24]

    Lange, Marc. 2017. ``Mathematical explanations that are not proofs.'' Erkenntnis, DOI: 10.1007/s10670-017-9941-z

  17. [25]

    Lehet, Ellen. 2021. ``Mathematical explanation in practice.'' Axiomathes 31, 553--574. american

  18. [26]

    english Li, Jun. 2001. ``A degeneration of stable morphisms and relative stable morphisms.'' arXiv:math/0009097

  19. [27]

    english Liedtke, Christian. 2013. ``Algebraic surfaces in positive characteristic.'' In Fedor Bogomolov, Brendan Hassett and Yuri Tschinkel (eds.), Birational Geometry, Rational Curves, and Arithmetic, Springer: New York, 229--292

  20. [28]

    and Erik Talvila

    english Loeb, Peter A. and Erik Talvila. 2001. ``Covering theorems and Lebesgue integration.'' arXiv:math/0101014

  21. [29]

    english Mancosu, Paolo, Francesca Poggiolesi and Christopher Pincock. 2023. ``Mathematical explanation.'' In Edward N. Zalta and Uri Nodelman (eds.), The Stanford Encyclopedia of Philosophy (Fall 2023 Edition), URL = <https://plato.stanford.edu/archives/fall2023/entries/mathem...

  22. [30]

    english Massey, David B. 2001. ``A little microlocal Morse theory.'' arXiv:math/0006185

  23. [31]

    english McDuff, Dusa. 2004. ``Symplectomorphism groups and almost complex structures.'' arXiv:math/0010274

  24. [32]

    english Mejía Ramos, Juan Pablo, Lara Alcock, Kristen Lew, Paolo Rago, Chris Sangwin and Matthew Inglis. 2019. ``Using corpus linguistics to investigate mathematical explanation.'' In Eugen Fischer and Mark Curtis (eds.), Methodological Advances in Experimental Philosophy, Lon...

  25. [33]

    Mejía Ramos, Juan Pablo, Tanya Evans, Colin Rittberg and Matthew Inglis. 2021. ``Mathematicians' assessments of the explanatory value of proofs.'' Axiomathes 31, 575--599. american

  26. [34]

    english Milson, Robert. 2000. ``Composition sum identities related to the distribution of coordinate values in a discrete simplex.'' arXiv:math/0003126. english

  27. [35]

    Mizrahi, Moti. 2020. ``Proof, explanation, and justification in mathematical practice.'' Journal for General Philosophy of Science 51, 551-568. american

  28. [36]

    english Pease, Alison, Andrew Aberdein and Ursula Martin. 2019. ``Explanation in mathematical conversations: An empirical investigation.'' Philosophical Transactions of the Royal Society A 377: 20180159. DOI: 0.1098/rsta.2018.0159

  29. [37]

    english Pisier, Gilles. 2000. ``Remarks on the similarity degree of an operator algebra.'' arXiv:math/0009052

  30. [38]

    english Ramakrishnan, Dinakar. 2001. ``Modularity of solvable Artin representations of GO (4 ) -type.'' arXiv:math/0102231. english

  31. [39]

    and David Kushner

    Resnik, Michael D. and David Kushner. 1987. ``Explanation, independence and realism in mathematics.'' British Journal for the Philosophy of Science 38, 141-158. american

  32. [40]

    english Rice, Collin. 2016. ``Factive scientific understanding without accurate representation.'' Biology and Philosophy 31, 81--102. english

  33. [41]

    Rittberg, Colin Jakob. 2019. ``On the contemporary practice of philosophy of mathematics.'' Acta Baltica Historiae et Philosophiae Scientiarum 7, 5--26. american

  34. [42]

    english Sa, Rentuya, Lara Alcock, Matthew Inglis and Fenner Stanley Tanswell. 2024. ``Do mathematicians agree about mathematical beauty?'' Review of Philosophy and Psychology 15, 299--325. english

  35. [43]

    Steiner, Mark. 1978. ``Mathematical explanation.'' Philosophical Studies 34, 135-151. american

  36. [44]

    english Sullivan, Emily and Kareem Khalifa. 2019. ``Idealizations and understanding: Much ado about nothing?'' Australasian Journal of Philosophy 97, 673--689

  37. [45]

    english Tao, Terence. 2000. ``Global regularity of wave maps I. Small critical Sobolev norm in high dimension.'' arXiv:math/0010068

  38. [46]

    english Thas, Koen. 2011. ``Order in building theory.'' In Robin Chapman (ed.), Surveys in Combinatorics 2011, Cambridge: Cambridge University Press, 235--332

  39. [47]

    english van der Kallen, Wilberd. 2000. ``From Mennicke symbols to Euler class groups.'' arXiv:math/0010226

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.