{"id":"55a023f6-7185-4092-85dd-6175efba066d","arxiv_id":"2411.14073","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Domain-adapted BERT models distinguish senses of 'Planck' better than general models, and reveal the rise of the Planck mission meaning in physics papers.","lead":"This paper tests whether BERT-style language models, especially ones trained on physics papers, can tell apart different meanings of the word 'Planck' (the physicist, Planck units, the Planck space mission) in scientific texts. It also tracks how the space mission meaning grew in the astrophysics literature between 1990 and 2022.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"In-sample prototype evaluation and pretraining overlap with the test corpus, not label noise, are the main threat to the claimed domain-adapted superiority.","rationale":"The paper's central claim is the comparative one: domain-adapted models outperform general-purpose models at WSD/WSI for Planck. That claim rests on Section 5's F1 scores and Section 6's purity scores. Section 5 uses an in-sample prototype evaluation: prototypes are averages of all labeled occurrences, including the very occurrences being scored. This guarantees a self-similarity advantage for every model and makes the absolute F1 values uninterpretable as generalization performance. The comparison between models is also vulnerable because Astro-HEP-BERT was pretrained on the same corpus from which the Astro-HEP-Planck test paragraphs were drawn; no overlap removal is reported. This means the model may have seen the exact test sentences, which selectively inflates its performance (and, to a lesser extent, other domain models trained on arXiv physics) over BERT. The reader's weakest assumption was label quality; I agree that single-annotator labels need scrutiny, but the in-sample/leakage problem is more foundational: even with a perfect gold standard, the reported scores would not support the headline claim. The diachronic finding (mission sense rising around 1996-1998 and 2013) is well aligned with known ESA milestones and gives qualitative plausibility, but it does not rescue the quantitative model-comparison claim. A held-out paragraph-level evaluation with deduplication against the pretraining corpus is a decisive, low-cost check.","tokens_in":17418,"tokens_out":5128,"duration_ms":49825,"concrete_test":"Run the Section 5 1NN evaluation under paragraph-level 5-fold cross-validation: build prototypes from the training folds only, score the held-out fold, and compute bootstrap confidence intervals. For the Astro-HEP-Planck corpus, additionally remove from the evaluation any paragraph that appears in the Astro-HEP-BERT pretraining corpus (or retrain Astro-HEP-BERT on the corpus with those paragraphs held out). If the held-out F1 gap between domain-adapted models and BERT falls below 0.05 or reverses, the headline claim of domain-adapted superiority rests on leakage, not on generalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 5, sense prototypes are 'generated by averaging the CWEs of all labeled Planck occurrences corresponding to each sense,' and F1 is then computed on those same labeled occurrences. Each test vector is therefore a component of its own correct prototype, inflating nearest-prototype accuracy in a way that a held-out evaluation would not. This in-sample design affects all models, but it is the central evidence for the paper's strongest claim (domain-adapted models outperform general-purpose models). Independently, Astro-HEP-BERT was further pretrained on the Astro-HEP Corpus (Section 4), while the Astro-HEP-Planck Corpus is a sample of paragraphs from that same corpus (Section 3); no de-duplication or exclusion is reported. Thus the superior F1 scores of Astro-HEP-BERT (and to a lesser degree PhysBERT/astroBERT) on the Astro-HEP-Planck Corpus may reflect memorized paragraphs rather than better sense disambiguation. The Wikipedia-physics comparison is less affected by this particular leakage but still shares the in-sample prototype problem. Without a held-out split or leakage check, the quantitative ranking of models is not established; the qualitative diachronic alignment with Planck mission milestones is credible independent evidence, but it does not validate the comparative WSD/WSI claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes contextualized word embeddings (CWEs) as a scalable tool for the history, philosophy, and sociology of science (HPSS), using the term \"Planck\" as a test case. It compares five BERT-based models (BERT, SciBERT, astroBERT, PhysBERT, and the author's Astro-HEP-BERT) on supervised word sense prediction, unsupervised clustering purity, cluster separation/cohesion, and lexical semantic change. The central claim is that domain-adapted models outperform general-purpose models in disambiguating and clustering senses of \"Planck,\" and that domain-adapted CWEs can track the emergence of the Planck mission sense over time.","tokens_in":17609,"tokens_out":5248,"duration_ms":48015,"significance":"If the empirical claims were established, this would be a valuable contribution to computational HPSS, offering a reusable pipeline and releasing two labeled corpora and a domain-adapted model. The diachronic finding that the Planck mission sense rose to dominance in the 2010s is plausible and historically meaningful. However, the current evaluation design undermines the comparative model-ranking claim, so the paper's significance is conditional on a re-analysis with proper held-out evaluation and leakage checks. The author's effort to release data and models is a clear strength that should be credited regardless of the revisions.","major_comments":[{"comment":"The 1NN sense prediction in Section 5 is an in-sample evaluation: the sense prototypes are computed by averaging the CWEs of all labeled \"Planck\" occurrences (second paragraph of Section 5), and the F-1 scores are then computed on those same labeled occurrences. Each test vector is therefore a component of its own correct prototype, which mechanically inflates nearest-prototype accuracy. Since this evaluation is the primary evidence for the abstract's claim that domain-adapted models outperform general-purpose ones, the claim is not established by the reported F-1 scores; a held-out split or leave-one-out evaluation is required.","section":"§5"},{"comment":"Astro-HEP-BERT was further pretrained on the Astro-HEP Corpus (Section 4), and the Astro-HEP-Planck Corpus is a sample of paragraphs from that same Astro-HEP Corpus (Section 3). The paper does not report any de-duplication or exclusion of evaluation paragraphs (or near-duplicates) from the continued pretraining data. Consequently, the superior F-1 and purity scores of Astro-HEP-BERT on the Astro-HEP-Planck Corpus might reflect memorized or highly similar paragraphs rather than better sense disambiguation. The author should check for overlap between the labeled paragraphs and the pretraining data, and ideally retrain the model with the evaluation paragraphs held out.","section":"§3–§4"},{"comment":"The ground-truth labels for both corpora were created by a single researcher with no inter-annotator agreement or adjudication (Section 3, paragraph 2). Because every quantitative result in Sections 5–8 is computed against these labels, any inconsistency or noise in the annotation scheme (21 predefined labels, with very low-frequency labels such as PERSON in the Astro-HEP-Planck Corpus) propagates into all F-1, purity, JSD, and CDPT values. The paper should provide an annotation guideline, report agreement on a sample of doubly-annotated occurrences, and discuss how borderline cases are handled.","section":"§3"},{"comment":"The diachronic analysis in Section 8 is presented primarily through the qualitative interpretability of PhysBERT's and BERT's five-cluster solutions (Figure 5), and the claim that domain-specific models show 'greater sensitivity' to semantic change is not backed by quantitative comparisons across all five models in the text. Although Figure 6 appears to plot JSD and CDPT for each model, no numerical values, statistical tests, or separate figures are provided to substantiate the relative-sensitivity claim. This weakens the conclusion that domain adaptation improves lexical semantic change tracking.","section":"§8"},{"comment":"The 'novel purity indicator' introduced in Section 6 is used as the sole measure of cluster quality for the WSI comparison, but the paper does not validate it against standard metrics (e.g., cluster purity, adjusted Rand index, normalized mutual information) or on synthetic data with known cluster structure. Without such validation, it is difficult to assess whether the reported differences (e.g., Astro-HEP-BERT above 0.95 vs BERT at 0.92) are meaningful or an artifact of the metric's dependence on label frequency distributions.","section":"§6"}],"minor_comments":[{"comment":"In the paragraph on domain adaptation, 'they have found to underperform' should be 'they have been found to underperform.'","section":"§2"},{"comment":"In the Astro-HEP-BERT description, 'using on my Astro-HEP Corpus' should be 'using my Astro-HEP Corpus.'","section":"§4"},{"comment":"The reference to Simons (2024) gives an empty arXiv identifier; please provide the full citation or a working URL.","section":"References"},{"comment":"The caption says 'two to six labels in the Astro-HEP-Planck Corpus' but the surrounding text mentions subsets with up to seven labels; please make the range descriptions consistent.","section":"Figure 2 caption"},{"comment":"The PERSON label has only 4 occurrences in the Astro-HEP-Planck Corpus; consider reporting results with and without such rare labels to assess robustness.","section":"Table 1"},{"comment":"The isotropy analysis in the supplement is clearly presented and adds useful context; consider moving a brief summary of it into the main text, since Section 7 relies on it.","section":"Supplement S1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern raised by the reader is well-founded: the in-sample prototype evaluation and the pretraining/evaluation overlap are independent, load-bearing problems. The qualitative diachronic alignment with Planck mission milestones is interesting but does not rescue the quantitative comparative claim. I recommend requiring a held-out evaluation and a leakage check before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a well-written, useful methodological demonstration for HPSS, but its headline quantitative claims are undercut by an in-sample evaluation and possible corpus leakage. The diachronic story is the best part.\n\nWhat's new: two labeled datasets (though not released), a custom domain-adapted BERT (Astro-HEP-BERT) trained on a large astro-HEP corpus, and the empirical finding that the 'Planck mission' sense rose sharply after 2013 in arXiv abstracts. The paper also proposes a CV-based purity indicator for cluster evaluation. The writing and framing for HPSS readers is clear, and the comparison across five models is thoughtfully designed.\n\nWhere it wobbles: the WSD evaluation in Section 5 averages prototypes from the same labeled instances it then classifies, so the reported F1 is a self-similarity measure. That's a real flaw, and it directly undermines the abstract's claim that domain-adapted models 'outperform' general-purpose ones. Second, Astro-HEP-BERT was further pretrained on the same corpus that supplies the test paragraphs, with no de-duplication; so its advantage on the Astro-HEP-Planck Corpus could partly be memorized text. The Wikipedia comparison is less affected, but the prototype issue applies there too. Also, labels come from a single annotator with no agreement check; that's a limitation but not necessarily fatal for a proof-of-concept. There are no error bars or significance tests, and no non-BERT baseline (e.g., static embeddings) to calibrate whether the complexity is needed.\n\nWhat holds up: the diachronic analysis in Section 8 clusters unlabeled data and finds a plausible rise of the mission sense tied to actual ESA mission milestones. That result is robust to the prototype critique because it doesn't use the labeled data for evaluation, and it's historically coherent.\n\nVerdict: the paper deserves a serious referee. It's a credible contribution to the HPSS methods literature, but the quantitative model-ranking claims need a held-out evaluation, leakage checks, and label release. I'd send it to a venue that values methodological experimentation; the authors should be pushed to fix the evaluation before publication.","headline":"A clear and useful HPSS methods experiment whose headline model-comparison claims are undercut by in-sample prototype evaluation and corpus overlap, while the diachronic Planck-mission result is solid and interesting.","tokens_in":18206,"tokens_out":1665,"would_cite":false,"duration_ms":16819,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Domain-adapted BERT models map how 'Planck' changed meaning in physics texts.","keywords":["contextualized word embeddings","word sense disambiguation","word sense induction","lexical semantic change","domain-adapted language models","history and philosophy of science","scientific concepts","Planck"],"falsifier":"Have two or more independent annotators label the same 'Planck' occurrences and compute inter-annotator agreement; if agreement is low, or if a non-contextual baseline such as static word2vec embeddings or bag-of-words classifiers matches the domain-adapted BERT scores on sense prediction, the claim that contextualized, domain-adapted embeddings are necessary for this kind of HPSS analysis would be undercut.","tokens_in":17153,"feed_emoji":"🔭","tokens_out":5044,"duration_ms":43235,"temperature":0.7,"pith_summary":"This paper argues that contextualized word embeddings, vectors assigned to each word based on its surrounding sentence, can serve as a scalable tool for the history, philosophy, and sociology of science. Using 'Planck' as a test case, it compares five BERT-style language models and finds that models adapted to astrophysics and high-energy physics outperform general-purpose BERT at telling apart the term's senses, such as Max Planck the physicist, Planck units, and the Planck space mission. The same embeddings reveal a measurable semantic shift: references to the Planck mission rise sharply after the mission's approval in 1996 and again after data releases in 2013. The paper concludes that domain-adapted embeddings offer a cost-effective way to trace conceptual change in scientific discourse.","feed_headline":"Domain-trained BERT maps how 'Planck' changed meaning in physics","feed_subtitle":"Domain-adapted embeddings disambiguate 'Planck' and reveal the Planck mission overtaking Planck units.","key_machinery":"The central object is the contextualized word embedding (CWE): a vector for a single occurrence of a word produced by averaging the last four hidden layers of a BERT-style transformer. The paper combines three operations on these vectors, cosine similarity between occurrences, prototypical embeddings averaged over labeled groups, and K-means clustering, to perform word sense disambiguation, word sense induction, and lexical semantic change analysis. A novel purity indicator based on the coefficient of variation of label distributions within clusters measures how well each clustering separates known senses.","core_discovery":"The central claim is that contextualized word embeddings from BERT provide a practical, transferable method for studying how scientific concepts acquire and change meanings. On the test term 'Planck', domain-adapted models (SciBERT, astroBERT, PhysBERT, and the author's Astro-HEP-BERT) beat the general-purpose BERT baseline in supervised sense prediction, unsupervised sense clustering, and cluster quality; Astro-HEP-BERT reaches parity with or exceeds from-scratch domain models despite being a cheaper continue-pretraining adaptation. Applied to over 300,000 unlabeled occurrences in the Astro-HEP Corpus, the embeddings show the Planck mission sense overtaking Planck units as the dominant meaning around 2012 to 2013, while the two semantic-change indicators spike in 1997 and 2013, matching the mission's renaming and public data releases.","pith_inferences":["The success on 'Planck' suggests the same pipeline could detect paradigm shifts or 'trading zone' borrowings in other polysemous scientific terms, though the paper does not test that extension.","Because the sense labels come from a single annotator, the reported F-1 and purity scores should be treated as provisional; an independent second annotation set could change the model ranking.","A direct comparison against non-contextual embeddings such as word2vec would clarify whether the contextual component or the domain adaptation is doing the heavy lifting in these tasks.","The diachronic analysis assumes sense clusters remain stable over time; allowing clusters to split or merge across years might yield a different picture of how the mission sense emerged."],"forward_implications":["Domain-adapted BERT models can automatically disambiguate and induce the senses of scientific terms, reducing the need for manual close reading in large corpora.","The method transfers to other scientific concepts, because it requires only a small labeled set and an off-the-shelf model adapted with continue-pretraining.","Lexical semantic change indicators JSD and CDPT align with known historical milestones, suggesting embeddings can date when a new meaning becomes dominant.","Astro-HEP-BERT demonstrates that reusing general-purpose pretrained weights is a cost-effective alternative to training a domain model from scratch.","The two labeled 'Planck' corpora provide reusable evaluation datasets for future computational studies of scientific language."],"supporting_citations":[{"why":"Defines BERT and establishes the standard practice of extracting embeddings from the last four hidden layers.","marker":"Devlin et al. (2018)"},{"why":"Supplies the 1-nearest-neighbor prototype method used for supervised word sense prediction.","marker":"Loureiro et al. (2020)"},{"why":"Provides the survey framework and the JSD and CDPT indicators used for lexical semantic change analysis.","marker":"Periti and Montanelli (2024)"},{"why":"Introduces SciBERT, a from-scratch scientific baseline model in the comparison.","marker":"Beltagy et al. (2019)"},{"why":"Introduces astroBERT, an astronomy-specific from-scratch model in the comparison.","marker":"Grezes et al. (2022)"},{"why":"Introduces PhysBERT, a physics-specific from-scratch model and sentence-embedding reference.","marker":"Hellert et al. (2024)"},{"why":"Introduces Astro-HEP-BERT, the author's domain-adapted model that is central to the comparison.","marker":"Simons (2024)"}],"fun_headline_variants":["BERT embeddings trace 'Planck' meaning shift in physics","How 'Planck' came to mean mission, not just units","Astro-HEP-BERT reveals Planck mission dominance in papers","Word embeddings show Planck mission overtaking units","Domain-specific BERT decodes Planck's semantic evolution"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire evaluation rests on a single researcher's manual annotation of 2,900 plus 1,186 occurrences of 'Planck' into 21 sense labels, with no second annotator to confirm the labels; if those labels are inconsistent or incomplete, every performance comparison inherits that error.","fun_headline_variants_meta":{"raw":{"variants":["BERT embeddings trace 'Planck' meaning shift in physics","How 'Planck' came to mean mission, not just units","Astro-HEP-BERT reveals Planck mission dominance in papers","Word embeddings show Planck mission overtaking units","Domain-specific BERT decodes Planck's semantic evolution"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000657,"raw_usage":{"total_tokens":3053,"prompt_tokens":1039,"completion_tokens":2014,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":655,"completion_tokens_details":{"reasoning_tokens":1947}},"tokens_in":655,"tokens_out":2014,"duration_ms":13694,"temperature":1.0,"reasoning_tokens":1947,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:33:40.417206+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two or more independent annotators label the same 'Planck' occurrences and compute inter-annotator agreement; if agreement is low, or if a non-contextual baseline such as static word2vec embeddings or bag-of-words classifiers matches the domain-adapted BERT scores on sense prediction, the claim that contextualized, domain-adapted embeddings are necessary for this kind of HPSS analysis would be undercut.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Astro-HEP-BERT, the author's domain-adapted model that is central to the comparison."}],"review_version":1}