Pith. sign in

REVIEW 3 major objections 7 minor 31 references

Astro-HEP-BERT: A bidirectional language model for studying the meanings of concepts in astrophysics and high energy physics

T0 review · 3 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a general BERT model given three extra epochs of training on 21.84 million paragraphs of astrophysics and high-energy physics text produces contextualized word embeddings comparable to physics-specific models…

desk verdict Useful new domain-adapted BERT and corpus for physics text, but the main performance claim is unevidenced in this preprint and rests on a single-term companion study. read the letter →

arxiv 2411.14877 v1 pith:NCZBVNQA submitted 2024-11-22 cs.CL physics.hist-ph

Arno Simons This is my paper · ORCID
classification cs.CLphysics.hist-ph
keywords contextualizedwordembeddingsdomainadaptationBERTastrophysicshighenergyphysicssensedisambiguationlexicalsemanticchangehistoryphilosophyandsociologyofscience
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents Astro-HEP-BERT, a transformer language model made by taking a general-purpose BERT model and running three additional training passes over 21.84 million paragraphs from more than 600,000 astrophysics and high-energy physics articles. The author's central claim is that the resulting contextualized word embeddings perform about as well as domain-specific BERT models trained from scratch on larger physics corpora, at least for disambiguating and tracing word meanings such as the term "Planck". If that claim holds, researchers in the history, philosophy, and sociology of science could build custom tools for studying how scientific concepts shift in meaning using only open code, open data, and a single laptop, without the cost of training a model from scratch.

What carries the argument

The machinery is continued pretraining of a general bidirectional transformer with three modifications to the original BERT protocol: masked language modeling without the next-sentence prediction objective, whole-word masking, and a "full-paragraphs format" in which each training sequence is a complete paragraph rather than a sentence or document. Training batches are organized to hold about 8,192 tokens with limited padding, so compute is spent on real text rather than placeholder tokens. The paragraph format is the distinctive design choice: the author argues that the paragraph, not the sentence, is the basic unit of meaning in academic writing, so embedding each paragraph as a unit should improve semantic coherence in the model's contextualized embeddings.

What would settle it

A controlled ablation that trains Astro-HEP-BERT's identical setup on sentence-level sequences instead of full paragraphs, evaluated on the same word sense disambiguation and semantic change tasks, would settle whether the paragraph format actually helps; if the two versions perform equally, the assumed advantage of paragraphs over sentences is absent.

Watch

Extended reading notes

Core claim

Astro-HEP-BERT is a BERT model given three more epochs of masked-language-model training on the Astro-HEP Corpus, 21.84 million paragraphs drawn from more than 600,000 astrophysics and high-energy physics articles published between 1986 and 2022. The paper's central claim is that this modest continued pretraining, performed with freely available tools on one laptop, yields contextualized word embeddings that a companion evaluation finds comparable to physics-specific BERT models trained from scratch on larger corpora. The author frames this as evidence that domain adaptation, rather than from-scratch training, is a viable and affordable route for studying the meanings of scientific concepts.

Load-bearing premise

The full-paragraphs format carries the argument: the paper assumes that a paragraph is a better unit of meaning than a sentence or document for academic writing, and offers no ablation to test that assumption.

Editorial extensions

If this is right

  • Researchers in the history, philosophy, and sociology of science can build domain-adapted language models for new fields on a single laptop, using openly available code, weights, and text.
  • Astro-HEP-BERT can disambiguate and trace the meanings of terms such as "Planck" across the 1986–2022 corpus, with shifts tied to events like the Planck space mission.
  • The Astro-HEP Corpus provides a reusable dataset of 21.84 million paragraphs with article-level metadata for studying concept change in astrophysics and high-energy physics.
  • The decreasing masked-language-model loss over three epochs indicates that continued pretraining on physics text does capture domain-specific language, supporting the idea that from-scratch training is not required for useful domain embeddings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is to apply the same continued-pretraining recipe to other scientific literatures; if the laptop-scale result generalizes, domain-adapted transformers could become a standard tool for conceptual history across many fields.
  • The full-paragraphs format, if confirmed by ablation, would imply that academic paragraphs are a better semantic unit than sentences for language-model pretraining, a principle that could inform future model designs beyond this corpus.
  • Because the base model is uncased, case-only distinctions between terms (such as names that are also ordinary words) may be flattened; a cased or symbol-aware variant might improve fine-grained semantic analysis.
  • The comparability claim currently rests on the single test term "Planck"; a multi-term benchmark of homographs and polysemous words would show whether the result is a general property of continued pretraining or specific to that case.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper presents Astro-HEP-BERT, a BERT-base model further pretrained for three epochs of masked language modeling on a newly curated corpus of 21.84 million paragraphs from more than 600,000 arXiv articles in astrophysics and high-energy physics. It describes the corpus construction pipeline, the model training configuration (whole-word masking, no NSP, paragraph-level sequences, dynamic batch sizing), and the feasibility of training on a single M2 MacBook over 48 days. The stated contribution is twofold: a reusable model and corpus for the history, philosophy, and sociology of science, and evidence that cost-effective domain adaptation can achieve performance comparable to domain-adapted BERT models trained from scratch. The abstract and conclusion make this comparative performance claim, but Section 4 defers all evaluation to a companion paper by the same author and contains no metrics, baselines, or error analysis.

Significance. If the comparative performance claim is borne out, this is a useful contribution: it would demonstrate that continued pretraining of a general BERT model on a modest compute budget can provide domain-appropriate contextualized embeddings for historical, philosophical, and sociological analyses of scientific concepts. The manuscript has concrete strengths: the model and corpus are publicly released, the corpus pipeline is described in unusual detail, filtering decisions are illustrated with distributions, and the training recipe is concrete enough to reproduce. The main significance is therefore conditional on evidence that is not present in this manuscript.

major comments (3)
  1. [Section 4 (Evaluation), Abstract, and Section 5 (Conclusion)] The load-bearing comparative claim is not supported by evidence in this manuscript. Section 4 states only that evaluation appears in Simons (2024) and reports no quantitative results for word sense disambiguation, induction, or semantic change detection. The abstract's claim that Astro-HEP-BERT's CWEs 'perform comparably to domain-adapted BERT models trained from scratch on larger datasets' and the conclusion's claim that Astro-HEP-BERT 'performs comparably with four leading BERT models' therefore cannot be checked, reproduced, or falsified from this preprint. Figure 4's decreasing training loss is evidence of mask-prediction optimization, not of semantic quality or downstream task performance. To make the paper self-contained, either include the evaluation data (task setups, metrics, baselines, and error analyses) or remove the comparative claim from the abstract and conclusion and re-scope the paper as a model and corpus resource description.
  2. [Section 4 (Evaluation) and Abstract] The supporting evaluation is described as a single-term case study. Section 4 says the companion study uses 'the polysemous physics term "Planck" as a test study' over a 30-year period. Even if the companion results were included or summarized, one term is too narrow a basis for the abstract's general claim about 'domain-specific word sense disambiguation and induction and related semantic change analyses.' The manuscript should either report results on multiple concepts or explicitly qualify the claim as a single-case demonstration.
  3. [Section 3 (The Astro-HEP-BERT Model)] The claimed benefit of the full-paragraphs format is an untested design assumption. Section 3 states that this format 'recognizes the paragraph as the basic unit of meaning in academic writing' and that the author 'anticipate[s] even stronger semantic coherence,' but no ablation compares paragraph-level training against sentence-level or document-level training. Because this format is a distinctive feature of Astro-HEP-BERT relative to the comparators, the comparative evaluation in Simons (2024) cannot isolate its contribution. I would like to see a small ablation (even on a subset) or a reduced claim that does not attribute performance to paragraph-level coherence.
minor comments (7)
  1. [Section 3 and Section 5] The word 'fine-tuned' is used for continued masked-language-model pretraining (e.g., 'fine-tuned with the newly developed Astro-HEP Corpus'); this is task-specific fine-tuning terminology and should be replaced with 'domain-adapted' or 'further pretrained.'
  2. [Section 2, Figures 2 and 3] The chosen thresholds (250 characters; whitespace rates 0.1 and 0.2) are described as arising from frequency analysis and manual inspection, but the figures do not mark these cutoffs; please add reference lines and state how sensitive the corpus composition is to these thresholds.
  3. [Section 1, Introduction] The sentence 'which could refer to weight in the particle is "light"' is ungrammatical and should be rewritten, for example as 'which could refer to the property of low mass in "the particle is light" or to the electromagnetic phenomenon in "light is a particle."
  4. [Section 4, first paragraph] The phrase 'test study .' contains a stray space before the period; please remove it.
  5. [Section 3, paragraph-length range] The corpus filtering removes paragraphs under 250 characters, yet the model section reports paragraph lengths ranging from 48 to 510 subwords; clarify the relationship between these numbers (characters versus subwords) to avoid an apparent inconsistency.
  6. [Section 3, citation] The phrase 'the document-sentence input format proposed by Liu et al. (2019)' attributes a training-data format to RoBERTa; please cite the relevant analysis more precisely or rephrase to avoid implying that the RoBERTa paper proposed this format.
  7. [Section 1, Introduction] The term 'colexification' is used where 'polysemy' or 'homonymy' seems intended; colexification usually refers to a single word form covering multiple senses across languages, not to context-dependent meaning variation within one language.

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation is circular; the headline performance claim is deferred to the author's companion paper, a self-citation that is not definitionally forced but leaves the central comparison unevidenced in this manuscript.

full rationale

The manuscript contains no equations, no fitted parameters, and no prediction generated from a fit; it is a model and corpus resource report. The only load-bearing empirical claim is in the abstract: 'Astro-HEP-BERT's CWEs perform comparably to domain-adapted BERT models trained from scratch on larger datasets for domain-specific word sense disambiguation and induction and related semantic change analyses.' Section 4 explicitly defers all evaluation to the author's companion paper: 'In Simons (2024), I evaluate the performance of five BERT-based models, including my own Astro-HEP-BERT... using the polysemous physics term "Planck" as a test study.' This is a same-author self-citation and, as presented, the comparison is not checkable or reproducible from this preprint; the training-loss curve in Figure 4 only shows masked-token prediction improving on the training corpus, which is not a semantic evaluation. However, the companion is described as benchmarking Astro-HEP-BERT against independently trained PhysBERT, astroBERT, SciBERT, and BERT on the term 'Planck', so the performance claim is not equivalent to an input by construction and is externally falsifiable in principle. The full-paragraphs design choice is asserted without an ablation, but that is an untested assumption, not a circular reduction. Accordingly, no circular step is exhibited; the score reflects the minor self-citation and deferred-evidence issue rather than any definitional circularity.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The paper involves no mathematical derivation and introduces no new theoretical entities. The free parameters are hyperparameters and curation thresholds chosen by hand; they affect model behavior but are not fitted to a target result. The key assumptions are the paragraph-as-meaning-unit premise and the representativeness of the arXiv-based corpus.

free parameters (5)
  • Paragraph length cutoff = 250 characters
    Chosen from a frequency distribution of paragraph lengths (Figure 2) to filter non-paragraph artifacts. It is a manual curation choice, not fitted to any performance metric.
  • Whitespace rate thresholds = 0.1 and 0.2
    Chosen via frequency analysis and manual sample inspection (Figure 3) to remove paragraphs with atypical whitespace. The thresholds are subjective and not validated against downstream performance.
  • Number of training epochs = 3
    Chosen by the author without reported ablation. It affects model quality but is not optimized against any evaluation set.
  • Target batch token count = 8192 tokens
    Chosen to minimize padding while staying within 5 percent of the target. It is an efficiency choice, not a fitted parameter.
  • Maximum padding share = 20 percent
    Set as a bound to reduce padding overhead. It does not come from data fitting.
assumptions (3)
  • domain assumption A paragraph is the basic unit of meaning in academic writing
    Section 3 introduces the full-paragraphs format based on this assumption, and the author states he anticipates stronger semantic coherence, but no experiment tests this assumption against sentence-level or document-level training.
  • domain assumption arXiv papers in astro-ph and hep categories from 1986 to 2022 form a representative corpus for studying concept meanings in these fields
    The Astro-HEP Corpus is built exclusively from arXiv metadata and source files. The paper assumes that this coverage is sufficient for historical and sociological analysis of concepts, though other publication venues and non-arXiv discourse are excluded.
  • standard math Removing the next-sentence prediction objective and using Masked Language Modeling alone improves domain adaptation
    This follows results from Liu et al. (2019) on RoBERTa and Mickus et al. (2020), as cited in Section 3. The paper adopts this as a design principle without further testing on this specific corpus.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Astro-HEP-BERT: A bidirectional language model for studying the meanings of concepts in astrophysics and high energy physics." pith.science (2026). https://pith.science/paper/NCZBVNQA

@misc{pith2026241114877,
  author       = {Pith},
  title        = {Pith review of: Astro-HEP-BERT: A bidirectional language model for studying the meanings of concepts in astrophysics and high energy physics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NCZBVNQA}},
  note         = {Machine review of arXiv:2411.14877}
}
read the original abstract

I present Astro-HEP-BERT, a transformer-based language model specifically designed for generating contextualized word embeddings (CWEs) to study the meanings of concepts in astrophysics and high-energy physics. Built on a general pretrained BERT model, Astro-HEP-BERT underwent further training over three epochs using the Astro-HEP Corpus, a dataset I curated from 21.84 million paragraphs extracted from more than 600,000 scholarly articles on arXiv, all belonging to at least one of these two scientific domains. The project demonstrates both the effectiveness and feasibility of adapting a bidirectional transformer for applications in the history, philosophy, and sociology of science (HPSS). The entire training process was conducted using freely available code, pretrained weights, and text inputs, completed on a single MacBook Pro Laptop (M2/96GB). Preliminary evaluations indicate that Astro-HEP-BERT's CWEs perform comparably to domain-adapted BERT models trained from scratch on larger datasets for domain-specific word sense disambiguation and induction and related semantic change analyses. This suggests that retraining general language models for specific scientific domains can be a cost-effective and efficient strategy for HPSS researchers, enabling high performance without the need for extensive training from scratch.

Figures

Figures reproduced from arXiv: 2411.14877 by the authors.

Figure 1
Figure 1. The Astro-HEP Corpus: 21.84M paragraphs found in 0.61M articles on astrophysics (ASTRO) and/or high [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Distribution of paragraph length before filtering out short paragraphs—35.38M paragraphs found in 0.61M [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Distribution of whitespace rate before filtering out paragraphs with a rate of less than 0.1 or more than [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Decreasing cross-entropy loss during the extended pretraining of [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 22 canonical work pages

  1. [1]

    Beltagy, I., Lo, K., and Cohan, A. (2019). SciBERT : A pretrained language model for scientific text. arXiv :1903.10676

    arXiv 2019

  2. [2]

    Callon, M., Law, J., and Rip, A. (1986). Qualitative scientometrics. In Callon, M., Law, J., and Rip, A., editors, Mapping the Dynamics of Science and Technology. Sociology of Science in the Real World , pages 103--123. Macmillan

    work page 1986

  3. [3]

    Chang, H. (2007). Inventing Temperature: Measurement and Scientific Progress . Oxford University Press

    work page 2007

  4. [4]

    Devlin, J. (2019). bert repository (https://github.com/google-research/bert)

    work page 2019

  5. [5]

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). BERT : Pre-training of deep bidirectional transformers for language understanding. arXiv :1810.04805

    arXiv 2018

  6. [6]

    Fleck, L. (1979). Genesis and development of a scientific fact . University of Chicago Press

    work page 1979

  7. [7]

    Improving astroBERT using Semantic Textual Similarity

    Grezes, F., Allen, T., Blanco-Cuaresma, S., Accomazzi, A., Kurtz, M. J., Shapurian, G., Henneken, E., Grant, C. S., Thompson, D. M., Hostetler, T. W., Templeton, M. R., Lockhart, K. E., Chen, S., Koch, J., Jacovich, T., and Protopapas, P. (2022). Improving astroBERT using semantic textual similarity. arXiv :2212.00744

    work page Pith review arXiv 2022

  8. [8]

    J., Shapurian, G., Henneken, E., Grant, C

    Grezes, F., Blanco-Cuaresma, S., Accomazzi, A., Kurtz, M. J., Shapurian, G., Henneken, E., Grant, C. S., Thompson, D. M., Chyla, R., McDonald , S., Hostetler, T. W., Templeton, M. R., Lockhart, K. E., Martinovic, N., Chen, S., Tanner, C., and Protopapas, P. (2021). Building astroBERT , a language model for astronomy & astrophysics. arXiv :2112.00590

    arXiv 2021

Show all 31 references
  1. [9]

    Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., and Poon, H. (2021). Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare ( HEALTH ) , 3(1):1--23

    2021

  2. [10]

    Hacking, I. (1975). The emergence of probability: A philosophical study of early ideas about probability, induction and statistical inference . Cambridge University Press

    1975

  3. [11]

    Hellert, T., Montenegro, J., and Pollastro, A. (2024). PhysBERT : A Text Embedding Model for Physics Scientific Literature . arXiv :2408.09574

    2024 arXiv

  4. [12]

    Kleymann, R., Niekler, A., and Burghardt, M. (2022). Conceptual forays: A corpus-based study of “theory” in digital humanities journals. Journal of Cultural Analytics , 7(4)

    2022

  5. [13]

    D., Maienschein, J., and Renn, J

    Laubichler, M. D., Maienschein, J., and Renn, J. (2019). Computational history of knowledge: Challenges and opportunities. Isis , 110(3):502--512

    2019

  6. [14]

    M., Rivelli, L., and Pence, C

    Lean, O. M., Rivelli, L., and Pence, C. H. (2023). Digital literature analysis for empirical philosophy of science. The British Journal for the Philosophy of Science , 74(4):875--898

    2023

  7. [15]

    H., and Kang, J

    Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., and Kang, J. (2020). BioBERT : a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics , 36(4):1234--1240

    2020

  8. [16]

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa : A robustly optimized BERT pretraining approach. arXiv :1907.11692

    2019 arXiv

  9. [17]

    T., and Camacho-Collados, J

    Loureiro, D., Rezaee, K., Pilehvar, M. T., and Camacho-Collados, J. (2020). Language models and word sense disambiguation: An overview and analysis. arXiv :2008.11608

    2020 arXiv

  10. [18]

    and Léonard, M

    Malaterre, C. and Léonard, M. (2024). Epistemic markers in the scientific discourse. Philosophy of Science , 91(1):151--174

    2024

  11. [19]

    Mickus, T., Paperno, D., Constant, M., and van Deemter, K. (2020). What do you mean, BERT ? assessing BERT as a distributional semantics model. arXiv :1911.05758

    2020 arXiv

  12. [20]

    Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv :1301.3781

    2013 arXiv

  13. [21]

    Overton, J. A. (2013). “explain” in scientific discourse. Synthese , 190(8):1383--1405

    2013

  14. [22]

    Pence, C. H. and Ramsey, G. (2018). How to do digital philosophy of science. Philosophy of Science , 85(5):930--941

    2018

  15. [23]

    and Montanelli, S

    Periti, F. and Montanelli, S. (2024). Lexical semantic change through large language models: a survey. ACM Computing Surveys , 56(11):282:1--282:38

    2024

  16. [24]

    Pickering, A. (1999). Constructing Quarks: A Sociological History of Particle Physics . University of Chicago Press

    1999

  17. [25]

    Rogers, A., Kovaleva, O., and Rumshisky, A. (2020). A primer in BERTology : What we know about how BERT works. arXiv :2002.12327

    2020 arXiv

  18. [26]

    Simons, A. (2024). Meaning at the planck scale? contextualized word embeddings for doing history, philosophy, and sociology of science. arXiv :2411.14073

    2024 arXiv

  19. [27]

    Steinle, F. (2016). Exploratory Experiments: Ampère, Faraday, and the Origins of Electrodynamics . University of Pittsburgh Press

    2016

  20. [28]

    and Platos, J

    Sun, Y. and Platos, J. (2023). A method for constructing word sense embeddings based on word sense induction. Scientific Reports , 13(1):12945

    2023

  21. [29]

    and Koolen, M

    Wevers, M. and Koolen, M. (2020). Digital begriffsgeschichte: Tracing semantic change using word embeddings. Historical Methods: A Journal of Quantitative and Interdisciplinary History , 53(4):226--243

    2020

  22. [30]

    Wiedemann, G., Remus, S., Chawla, A., and Biemann, C. (2019). Does BERT make any sense? interpretable word sense disambiguation with contextualized embeddings. arXiv :1909.10430

    2019 arXiv

  23. [31]

    and Wüthrich, A

    Zichert, M. and Wüthrich, A. (2024). Tracing the development of the virtual particle concept using semantic change detection. arXiv :2410.16855

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.