Pith. sign in

REVIEW 3 major objections 6 minor 51 references

Domain Pre-training Impact on Representations

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pre-training on a small, specialized corpus can be as effective as pre-training on a large generic one when the specialized text is close to the target task, and the benefit of combining both is predictable from distributional similarity.

desk verdict Useful empirical study, but the headline comparison is confounded and the correlation analysis needs statistical support. read the letter →

arxiv 2505.24455 v1 pith:GLLQJZOT submitted 2025-05-30 cs.CL

Cesar Gonzalez-Gutierrez , Ariadna Quattoni This is my paper
classification cs.CL
keywords pre-trainingdomainadaptationtransformerrepresentationsrepresentationqualityprobingdistributionalsimilaritylow-annotationlearningBERT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what the choice of pre-training corpus does to a transformer's representations, before any fine-tuning. Pre-training a fixed BERT-size model from scratch on seven domain-specific corpora and evaluating on nine tasks with three representation-quality metrics, it finds that a small, specialized corpus can match a much larger generic corpus whenever the specialized text is close to the target and big enough. It further finds that domain-adaptive pre-training, continuing a generic model on a specialized corpus, helps or hurts depending on how similar the specialized corpus is to the target task. Across all tasks the paper reports strong correlations between n-gram overlap of pre-training and target distributions and the measured gains, which suggests that relevance can matter as much as scale.

What carries the argument

The load-bearing machinery is a three-metric evaluation suite applied without any supervised fine-tuning: low-annotation probing, which measures the area under the accuracy curve of simple classifiers trained on 100-1000 examples (ALC); task hierarchical alignment, which scores how purely task labels separate into agglomerative clusters (THAS); and data-dependent complexity, an eigenvalue-based measure of label-representation compatibility restricted to binary tasks (DDC). These metrics compare in-domain, general-domain, and domain-adaptive pre-training on a single BERT architecture. To explain the observed gains, the paper introduces two asymmetric n-gram distribution similarity measures, n-gram coverage and expected L1 accuracy, and correlates them with the measured quality differences.

What would settle it

Fine-tune every pre-trained representation from the three scenarios on the nine target tasks with standard supervised data, then compare fine-tuned performance to the ALC, THAS, and DDC scores. If representations that score higher on these metrics do not also perform better after fine-tuning, or if domain-adaptive pre-training with low n-gram similarity beats high-similarity setups, the paper's central claim is contradicted.

Watch

Extended reading notes

Core claim

The paper's central claim is that representation quality induced solely by pre-training is governed by distributional relevance to the target, not by corpus size alone. In-domain pre-training on small corpora of roughly 1.8M to 100M words reaches representation quality comparable to a general-domain model pre-trained on about 3.3 billion words, with TREC as the one clear failure because its pre-training corpus is extremely small. Domain-adaptive pre-training produces the largest observed gains on domain-matched tasks, but it can also degrade representations when the specialized corpus is far from the target. The paper substantiates this dependence by correlating two asymmetric distributional similarity scores with the improvements over the general-domain baseline, reporting Spearman correlations mostly between 55% and 96% across probing, task alignment, and data-dependent complexity.

Load-bearing premise

The load-bearing premise is that the three representation-quality metrics — low-annotation probing, THAS, and DDC — correctly measure whether a representation will be useful on real downstream tasks, since no fine-tuning is performed anywhere in the study.

Editorial extensions

If this is right

  • Practitioners can pre-train from scratch on a domain corpus of a few million words and expect representation quality competitive with a much larger generic pre-training run, provided the corpus is not extremely small.
  • Domain-adaptive pre-training is not automatically a win: its benefit can be estimated in advance by measuring distributional similarity between the candidate specialized corpus and the target text.
  • The largest domain-matched representation gains come from continuing a generic model on in-domain data, not from either corpus alone.
  • The conclusions hold across different layer and token pooling choices, so the improvement is not an artifact of one feature-extraction strategy.
  • Because a failure case (TREC) is explained by corpus size, the paper implies a minimum scale below which specialized pre-training loses its advantage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the correlation results suggest a cheap screening heuristic the authors do not spell out — compute n-gram coverage between unlabeled target text and a candidate specialized corpus before spending compute on continued pre-training, and skip the corpus if coverage is low.
  • My inference: since the practical recommendation depends on whether these representation metrics predict real fine-tuning performance, a natural next experiment is to fine-tune all representations on the nine tasks and check that the ranking survives.
  • My inference: because one similarity score (expected L1 accuracy) clusters near 99.6-100% in the appendix while n-gram coverage spreads across 0-100%, the n-gram coverage measure is likely carrying most of the predictive signal, which suggests only one metric may be needed in practice.
  • My inference: the paper implies a relevance-over-scale trade-off that could be mapped explicitly — hold the domain gap fixed, vary corpus size, and identify where in-domain quality saturates; the authors do not run this sweep.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper reports an empirical study of how the choice of pre-training corpus affects the quality of transformer representations, measured without any supervised fine-tuning. The authors compare three scenarios: in-domain pre-training from scratch on a small specialized corpus (ID), a general-domain baseline (GD, the released BERT model), and domain-adaptive pre-training that continues pre-training BERT on the specialized corpus (DA). Representation quality is assessed with three metrics: low-annotation probing (ALC), task hierarchical alignment (THAS), and data-dependent complexity (DDC). The central claims are that a small, domain-matched specialized corpus can produce representations competitive with a much larger generic corpus, and that the benefit of domain-adaptive pre-training is predicted by the distributional similarity between the specialized corpus and the target task, as quantified by n-gram coverage and an expected-L1-accuracy measure.

Significance. If the claims hold, the paper has a useful practical message: for domain-specific applications, expensive generic pre-training may be partly replaceable by cheaper, targeted pre-training when the domain match is strong, and the success of domain-adaptive pre-training can be anticipated with simple distributional similarity measures. The study has notable strengths: it spans seven specialized corpora and nine evaluation tasks, uses three complementary representation-quality metrics, averages over five seeds, and includes a broad appendix of feature-extraction ablations. The correlation analysis with two similarity measures is a valuable attempt to move from qualitative observations to a testable predictor. However, the central ID-versus-GD comparison is confounded by the use of an off-the-shelf BERT baseline, and the correlation evidence lacks significance testing and has a very narrow dynamic range for one of the two similarity measures.

major comments (3)
  1. [§2.1, Table 4, Figure 1] The general-domain baseline is not matched to the in-domain and domain-adaptive training setups. Section 2.1 states that the authors omit the next-sentence prediction objective following Liu et al. (2019b), and Table 4 specifies the training recipe (AdamW, learning rate 5e-5, batch size 96, fp16). The GD baseline, however, is the released BERT model pre-trained on BookCorpus plus English Wikipedia with the original NSP objective and the original BERT hyperparameters. No GD model is pre-trained with the Table 4 recipe. Consequently, the observation in Figure 1 that ID models match or exceed the GD baseline could be caused by the removal of NSP, the different optimizer and learning rate, the different batch size, or the shorter training schedule, rather than by the relevance of the specialized corpus. This directly undermines the headline conclusion in §4 that 'the relevance of the PT data may be more important than its size.' The authors should either train a GD model under the same MLM-only recipe and comparable compute, or substantially weaken the ID-versus-GD claims.
  2. [§2.2, Table 2, Figures 5 and 6] The correlation evidence for the central similarity hypothesis is statistically under-supported. The paper reports Spearman correlations of 55.39–96.00% in Table 2, but gives no confidence intervals, p-values, or per-seed correlation variability, even though each experiment was repeated with five seeds. More importantly, the E[accL1] similarity measure varies only between 99.6 and 100 in the scatter plots of Figures 5 and 6, so the strong correlations involving this measure are computed against an almost constant regressor and may be unstable or driven by a few points. The authors should report per-seed or bootstrap correlations with significance tests, and should consider a similarity measure with a larger dynamic range, or at least demonstrate that the E[accL1] correlations are robust to the narrow range of observed values.
  3. [Limitations and §4] The paper explicitly states that it 'focuses solely on the changes induced by pre-training, without any supervised learning,' yet the conclusions in §4 include practical recommendations about developing 'high-quality models' with domain-specific data. The three metrics (probing, THAS, DDC) are, as the authors implicitly acknowledge, proxies for downstream usefulness, but the manuscript does not validate that improvements on these metrics translate to fine-tuning performance. Since the practical recommendation is load-bearing, the authors should either add a small fine-tuning validation on a subset of the tasks or explicitly restrict the conclusions to representation-level quality and remove the practical 'high-quality models' framing.
minor comments (6)
  1. [§4] The phrase 'a through analysis' should read 'a thorough analysis.'
  2. [Figure 1 and Tables 5–7] Although the experiments are repeated with five random seeds, the main matrix figures and the appendix tables report only averages, with no error bars or variance measures; adding standard deviations would help the reader judge whether the reported improvements are stable.
  3. [Table 3] The AG-News row is garbled: '41/|Y|38' should presumably read '|Y|=4, prior 1/|Y|=0.25' or similar; please correct the formatting.
  4. [§2.1 and A.1] The exact BERT checkpoint used as the GD baseline should be specified (e.g., bert-base-uncased versus bert-base-cased), and the vocabulary size of 30,522 suggests a specific tokenizer; this should be identified precisely for reproducibility.
  5. [A.1] The reproducibility statement says the authors 'will release the weights of the domain pre-trained models after publication,' but the paper should also release or specify the training code, exact hyperparameters per dataset, seed values, and evaluation scripts, since the experiments are central to all claims.
  6. [Figures 2 and 4] The x-axis is labeled 'pre-training steps,' but the caption and appendix refer to epochs (e.g., BERT_IMDb-80); please clarify the relationship between epochs and update steps in the captions or main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the main claims are empirical correlations computed from independently defined inputs, and the only self-citation (THAS) is a published metric that is not load-bearing by itself.

full rationale

The paper's central claims are empirical rather than derivational. The correlation claim is tested by computing two independent quantities: representation-quality deltas (probe ALC, THAS, and DDC, measured on held-out test partitions) and text-distribution similarity (ncvg and E[accL1] over n-grams). Neither quantity is defined in terms of the other, so the correlation in Table 2 is an empirical finding rather than a by-construction identity. The claim that small specialized corpora can match generic pre-training is likewise an experimental comparison against an external BERT baseline, not a fitted prediction. The only self-citation is the THAS metric (Gonzalez-Gutierrez et al., 2023), which is a published, externally defined measure used alongside two non-self-cited metrics (probing and DDC); the conclusions are not forced by it. The Limitations section explicitly disclaims supervised fine-tuning and other architectures, which limits generality but does not make the argument circular. The off-the-shelf GD baseline differs from the trained ID/DA models in NSP objective and hyperparameters; that is a confound affecting experimental validity, but it is not an equivalence between input and output and therefore does not count as circularity under the stated criteria. No specific reduction from input to output is exhibited anywhere in the paper.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on hand-chosen training budgets, an unspecified n-gram order, and the assumption that three representation metrics proxy downstream usefulness. No new entities are introduced, and the correlation analysis is empirical rather than derived.

free parameters (2)
  • n-gram order n = unspecified
    The similarity metrics ncvg and E[accL1] are defined over 'n-grams (up to size n)' but the value of n is never stated in the paper, so the correlation results depend on an undisclosed choice.
  • Pre-training steps/epochs per corpus = varies (e.g., IMDb 80 epochs, S140 20 epochs, Wiki Toxic 40 epochs)
    Each domain model is trained for a hand-chosen number of epochs, and the reported performance at the final epoch is used for comparisons. The conclusions about parity with the general-domain model depend on this training budget.
assumptions (4)
  • domain assumption The n-gram distribution of the target test set represents the target task domain.
    Similarity between pre-training and target domains is computed on the held-out test partitions; if test-set n-grams are not representative of the true target distribution, the correlation analysis is biased.
  • domain assumption BERT-base is a representative transformer architecture for studying pre-training effects.
    The authors state in Limitations they have not compared other architectures, yet the conclusions are phrased generally about transformer representations.
  • domain assumption The three chosen metrics capture representation quality relevant to transfer learning.
    The paper never fine-tunes on downstream tasks; it assumes ALC, THAS, and DDC reflect the usefulness of representations for subsequent supervised learning.
  • domain assumption The unlabeled text of classification datasets constitutes a valid specialized domain corpus.
    Seven classification datasets are used as pre-training corpora; their text distributions are assumed to represent their respective domains, e.g., IMDb represents movie reviews.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Domain Pre-training Impact on Representations." pith.science (2026). https://pith.science/paper/GLLQJZOT

@misc{pith2026250524455,
  author       = {Pith},
  title        = {Pith review of: Domain Pre-training Impact on Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GLLQJZOT}},
  note         = {Machine review of arXiv:2505.24455}
}
read the original abstract

This empirical study analyzes the effects of the pre-training corpus on the quality of learned transformer representations. We focus on the representation quality induced solely through pre-training. Our experiments show that pre-training on a small, specialized corpus can yield effective representations, and that the success of combining a generic and a specialized corpus depends on the distributional similarity between the target task and the specialized corpus.

Figures

Figures reproduced from arXiv: 2505.24455 by the authors.

Figure 1
Figure 1. Cross-domain representation performance. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Probe performance as a function of PT steps. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. shows the pre-training dynamic curves using the task alignment metric, as in §2.2. A.3 Performance Improvements Correlation To complement the correlation study presented in §2.2, this section presents scatter plots between the distribution distance measurements and repre￾sentation quality improvements. This comparison only considers the representations obtained in the 1 For reproducibility, we will release the weigh… view at source ↗
Figures from the paper (3 more)
Figure 3
Figure 3. Figure 3: shows the data-dependent complexity ma￾trices corresponding to the analysis performed in §2.2. Data-Dependent Complexity IMDb S140 WTx CvC BERT IMDb S140 WTx CvC Pre-training Corpus IMDb S140 WTx CvC Evaluation Task 4.7 +11.6 +14.2 +8.8 64.3 19.8 +8.0 1.3 1.1 +9.5 +1.7…
Figure 6
Figure 6. Figure 6: Correlation between distribution similarity [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Representation performance curves for different datasets using BERT embeddings generated from the last [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 12 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. https://arxiv.org/abs/1608.04207 Fine-grained analysis of sentence embeddings using auxiliary prediction tasks . Preprint, arXiv:1608.04207

    arXiv 2017

  4. [4]

    Roee Aharoni and Yoav Goldberg. 2020. https://doi.org/10.18653/v1/2020.acl-main.692 Unsupervised domain clusters in pretrained language models . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7747--7763, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.692 2020

  5. [5]

    Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael L...

    arXiv 2024

  6. [6]

    Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. https://doi.org/10.18653/v1/D19-1371 S ci BERT : A pretrained language model for scientific text . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615--3620, Hong Kong, China...

    doi:10.18653/v1/d19-1371 2019

  7. [7]

    Terra Blevins, Hila Gonen, and Luke Zettlemoyer. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.234 Analyzing the mono- and cross-lingual pretraining dynamics of multilingual language models . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3575--3590, Abu Dhabi, United Arab Emirates. Association for Com...

    doi:10.18653/v1/2022.emnlp-main.234 2022

  8. [8]

    Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. https://doi.org/10.1145/3308560.3317593 Nuanced metrics for measuring unintended bias with real data for text classification . In Companion Proceedings of The 2019 World Wide Web Conference, WWW '19, page 491–500, New York, NY, USA. Association for Computing Machinery

    arXiv 2019

Show all 51 references
  1. [9]

    Cheng-Han Chiang, Sung-Feng Huang, and Hung-yi Lee. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.553 P retrained language model embryology: T he birth of ALBERT . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6813...

    2020 doi

  2. [10]

    Leshem Choshen, Guy Hacohen, Daphna Weinshall, and Omri Abend. 2022. https://doi.org/10.18653/v1/2022.acl-long.568 The grammar-learning trajectories of neural language models . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1...

    2022 doi

  3. [11]

    Alexandra Chronopoulou, Matthew Peters, and Jesse Dodge. 2022. https://doi.org/10.18653/v1/2022.naacl-main.96 Efficient hierarchical domain adaptation for pretrained language models . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Co...

    2022 doi

  4. [12]

    Alexis Conneau, German Kruszewski, Guillaume Lample, Lo \"i c Barrault, and Marco Baroni. 2018. https://doi.org/10.18653/v1/P18-1198 What you can cram into a single \ & ! \# * vector: Probing sentence embeddings for linguistic properties . In Proceedings of the 56th Annual Mee...

    2018 doi

  5. [13]

    Yuyang Deng, Junyuan Hong, Jiayu Zhou, and Mehrdad Mahdavi. 2024. On the Generalization Ability of Unsupervised Pretraining . Proceedings of machine learning research, 238:4519--4527

    2024

  6. [14]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  7. [15]

    Kakade, Jason D

    Simon Shaolei Du, Wei Hu, Sham M. Kakade, Jason D. Lee, and Qi Lei. 2021. https://openreview.net/forum?id=pW2Q2xLwIMD Few-shot learning via learning the representation, provably . In International Conference on Learning Representations

    2021

  8. [16]

    Kawin Ethayarajh. 2019. https://doi.org/10.18653/v1/D19-1006 How contextual are contextualized word representations? C omparing the geometry of BERT , ELM o, and GPT -2 embeddings . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and t...

    2019 doi

  9. [17]

    Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. 2016. https://doi.org/10.18653/v1/W16-2524 Probing for semantic evidence of composition by means of simple classification tasks . In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP , pages ...

    2016 doi

  10. [18]

    Jiawei Ge, Shange Tang, Jianqing Fan, and Chi Jin. 2024. https://openreview.net/forum?id=rmXXKxQpOR On the provable advantage of unsupervised pretraining . In The Twelfth International Conference on Learning Representations

    2024

  11. [19]

    Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter Sentiment Classification using Distant Supervision . CS224N project report, Stanford, 1(12):6

    2009

  12. [20]

    Cesar Gonzalez-Gutierrez, Audi Primadhanty, Francesco Cazzaro, and Ariadna Quattoni. 2023. https://doi.org/10.18653/v1/2023.acl-short.7 Analyzing text representations by measuring task alignment . In Proceedings of the 61st Annual Meeting of the Association for Computational L...

    2023 doi

  13. [21]

    David Grangier and Dan Iter. 2022. https://doi.org/10.18653/v1/2022.acl-long.264 The trade-offs of domain adaptation for neural language models . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3802--381...

    2022 doi

  14. [22]

    Suchin Gururangan, Ana Marasovi \'c , Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. https://doi.org/10.18653/v1/2020.acl-main.740 Don ' t stop pretraining: Adapt language models to domains and tasks . In Proceedings of the 58th Annual Meeting o...

    2020 doi

  15. [23]

    Xiaochuang Han and Jacob Eisenstein. 2019. https://doi.org/10.18653/v1/D19-1433 Unsupervised domain adaptation of contextualized embeddings for sequence labeling . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internation...

    2019 doi

  16. [24]

    Harris, K

    Charles R. Harris, K. Jarrod Millman, St \' e fan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jai...

    2020

  17. [25]

    William Hersh, Chris Buckley, T. J. Leone, and David Hickam. 1994. Ohsumed: An interactive retrieval evaluation and new large test collection for research. In SIGIR '94, pages 192--201, London. Springer London

    1994

  18. [26]

    John Hewitt and Christopher D. Manning. 2019. https://doi.org/10.18653/v1/N19-1419 A structural probe for finding syntax in word representations . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Langu...

    2019 doi

  19. [27]

    Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin-Yew Lin, and Deepak Ravichandran. 2001. https://aclanthology.org/H01-1069/ Toward semantics-based answer pinpointing . In Proceedings of the First International Conference on Human Language Technology Research

    2001

  20. [28]

    Kundan Krishna, Saurabh Garg, Jeffrey Bigham, and Zachary Lipton. 2023. https://doi.org/10.18653/v1/2023.acl-long.682 Downstream datasets make surprisingly good pretraining corpora . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vo...

    2023 doi

  21. [29]

    Kullback and R

    S. Kullback and R. A. Leibler. 1951. https://doi.org/10.1214/aoms/1177729694 On Information and Sufficiency . The Annals of Mathematical Statistics, 22(1):79 -- 86

    1951

  22. [30]

    Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. https://doi.org/10.1093/bioinformatics/btz682 Biobert: a pre-trained biomedical language representation model for biomedical text mining . Bioinformatics, 36(4):1234–1240

    2019 doi

  23. [31]

    Xin Li and Dan Roth. 2002. https://aclanthology.org/C02-1150/ Learning question classifiers . In COLING 2002: The 19th International Conference on Computational Linguistics

    2002

  24. [32]

    Liu, Matt Gardner, Yonatan Belinkov, Matthew E

    Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019 a . https://doi.org/10.18653/v1/N19-1112 Linguistic knowledge and transferability of contextual representations . In Proceedings of the 2019 Conference of the North A merican Chapter of t...

    2019 doi

  25. [33]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 b . https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . Preprint, arXiv:1907.11692

    2019 arXiv

  26. [34]

    Zeyu Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A. Smith. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.71 Probing across time: What does R o BERT a know and when? In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 820--8...

    2021 doi

  27. [35]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. https://aclanthology.org/P11-1015/ Learning word vectors for sentiment analysis . In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: H...

    2011

  28. [36]

    Fabian Pedregosa, Ga\" e l Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and \' E douard ...

    2011

  29. [37]

    Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

    Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. https://doi.org/10.18653/v1/N18-1202 Deep contextualized word representations . In Proceedings of the 2018 Conference of the North A merican Chapter of the As...

    2018 doi

  30. [38]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. https://openai.com/research/language-unsupervised Improving Language Understanding by Generative Pre - Training

    2018

  31. [39]

    Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...

    2019 doi

  32. [40]

    Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. https://doi.org/10.1162/tacl_a_00349 A primer in BERT ology: What we know about how BERT works . Transactions of the Association for Computational Linguistics, 8:842--866

    2020 doi

  33. [41]

    Naomi Saphra and Adam Lopez. 2019. https://doi.org/10.18653/v1/N19-1329 Understanding learning dynamics of language models with SVCCA . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technol...

    2019 doi

  34. [42]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua ...

    2024

  35. [43]

    Nilesh Tripuraneni, Michael Jordan, and Chi Jin. 2020. On the theory of transfer learning: The importance of task diversity. Advances in neural information processing systems, 33:7852--7862

    2020

  36. [44]

    Hetong Wang, Pasquale Minervini, and Edoardo Ponti. 2024. https://doi.org/10.18653/v1/2024.findings-acl.724 Probing the emergence of cross-lingual alignment during LLM training . In Findings of the Association for Computational Linguistics: ACL 2024, pages 12159--12173, Bangko...

    2024 doi

  37. [45]

    Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.16 Learning which features matter: R o BERT a acquires a preference for linguistic generalizations (eventually) . In Proceedings of the 2020 Conference ...

    2020 doi

  38. [46]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  39. [47]

    Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. https://doi.org/10.1145/3038912.3052591 Ex machina: Personal attacks seen at scale . In Proceedings of the 26th International Conference on World Wide Web, WWW '17, page 1391–1399, Republic and Canton of Geneva, CHE. Interna...

    2017

  40. [48]

    Gregory Yauney and David Mimno. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.449 Comparing text representations: A theory-driven approach . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5527--5539, Online and Punta Cana, ...

    2021 doi

  41. [49]

    Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS'15, page 649–657, Cambridge, MA, USA. MIT Press

    2015

  42. [50]

    Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. 2021. https://doi.org/10.18653/v1/2021.acl-long.90 When do you need billions of words of pretraining data? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int...

    2021 doi

  43. [51]

    Yichu Zhou and Vivek Srikumar. 2021. https://doi.org/10.18653/v1/2021.naacl-main.401 D irect P robe: Studying representations without classifiers . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu...

    2021 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.