{"id":"77688d1b-3cfd-447a-8a86-fbd26cadfbf1","arxiv_id":"2505.24455","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Pre-training on small, domain-specific corpora can match generic pre-training, and n-gram similarity between pre-training and target data predicts whether combining corpora helps.","lead":"This paper tests whether pre-training a language model on a small, specialized text corpus can match or beat training on a huge generic corpus, and when mixing the two helps. It finds that specialized pre-training works when the corpus is similar to the target task, and that a simple n-gram overlap measure predicts the benefit.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GD baseline is off-the-shelf BERT trained with NSP and different hyperparameters, while ID/DA models are trained MLM-only with Table 4 recipe, conflating corpus choice with training objective and setup.","rationale":"The reader's weakest assumption—that ALC/THAS/DDC are valid proxies for downstream usefulness—is plausible and important for the practical recommendation, and the paper itself acknowledges the lack of fine-tuning in Limitations. However, the most immediately load-bearing issue for the paper's central empirical claim is the uncontrolled GD baseline. The paper's ID and DA models are trained with an MLM-only objective and a specific optimization recipe (Table 4), whereas the GD baseline is the original BERT, trained with NSP and different hyperparameters. Since the headline claim 'pre-training on a small, specialized corpus can yield effective representations' is established by comparing ID models to this GD baseline, any observed parity could be due to the recipe change rather than corpus relevance. This is a design confound, not a consensus disagreement. The correlation claim for DA is less affected because all DA runs start from the same base, but the magnitude of improvements and the motivating ID results are invalid as presented. The proposed test—pre-training a matched GD baseline with the same recipe—would directly resolve whether the effect persists. If it does, the paper's qualitative conclusions stand (assuming the metrics-proxy concern is also addressed); if it does not, the central claim fails. Because the check is feasible and the flaw is addressable, a conditional verdict is appropriate, but the condition should include this matched-baseline experiment in addition to the statistical and specification issues raised by the reader.","tokens_in":16988,"tokens_out":7318,"duration_ms":82444,"concrete_test":"Pre-train a BERT-base model from scratch on the generic corpus (BookCorpus+Wikipedia) using the exact recipe in Table 4 (MLM-only, AdamW, lr 5e-5, batch 96, fp16, the same total number of updates as the ID runs for each dataset, e.g., 50k steps). Then recompute the ID curves and matrices in Figures 1 and 2 against this matched GD baseline. If the specialized corpus no longer matches or beats the generic corpus on the in-domain diagonal, the corpus-relevance claim is an artifact of the training-recipe confound. If the parity persists, the confound is not responsible.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central comparison between small specialized corpora (ID) and a large generic corpus (GD) is confounded. In §2.1, the authors state that their models omit NSP and use MLM only, and Table 4 lists a specific recipe (AdamW, lr 5e-5, batch 96, fp16). But the GD baseline is not trained with this recipe: it is the released BERT model pre-trained on BookCorpus+Wikipedia with the original NSP objective and different hyperparameters (Devlin et al., 2019). The paper does not report pre-training a GD model with Table 4's recipe. Therefore, the observation that ID models match or exceed the GD baseline could be caused by the removal of NSP, the different optimizer/learning rate/batch size, or the shorter training schedule, rather than by the relevance of the specialized corpus. The DA experiments also start from this same off-the-shelf BERT, so DA improvements are measured against a mismatched base. This directly undermines the headline conclusion that 'the relevance of the PT data may be more important than its size' (§4). The correlation analysis in Table 2 is less affected, since all DA runs share the same GD baseline, but the absolute improvement magnitudes and the ID-vs-GD comparisons that motivate the DA hypothesis are not valid as reported.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study of how the choice of pre-training corpus affects the quality of transformer representations, measured without any supervised fine-tuning. The authors compare three scenarios: in-domain pre-training from scratch on a small specialized corpus (ID), a general-domain baseline (GD, the released BERT model), and domain-adaptive pre-training that continues pre-training BERT on the specialized corpus (DA). Representation quality is assessed with three metrics: low-annotation probing (ALC), task hierarchical alignment (THAS), and data-dependent complexity (DDC). The central claims are that a small, domain-matched specialized corpus can produce representations competitive with a much larger generic corpus, and that the benefit of domain-adaptive pre-training is predicted by the distributional similarity between the specialized corpus and the target task, as quantified by n-gram coverage and an expected-L1-accuracy measure.","tokens_in":17256,"tokens_out":3436,"duration_ms":44639,"significance":"If the claims hold, the paper has a useful practical message: for domain-specific applications, expensive generic pre-training may be partly replaceable by cheaper, targeted pre-training when the domain match is strong, and the success of domain-adaptive pre-training can be anticipated with simple distributional similarity measures. The study has notable strengths: it spans seven specialized corpora and nine evaluation tasks, uses three complementary representation-quality metrics, averages over five seeds, and includes a broad appendix of feature-extraction ablations. The correlation analysis with two similarity measures is a valuable attempt to move from qualitative observations to a testable predictor. However, the central ID-versus-GD comparison is confounded by the use of an off-the-shelf BERT baseline, and the correlation evidence lacks significance testing and has a very narrow dynamic range for one of the two similarity measures.","major_comments":[{"comment":"The general-domain baseline is not matched to the in-domain and domain-adaptive training setups. Section 2.1 states that the authors omit the next-sentence prediction objective following Liu et al. (2019b), and Table 4 specifies the training recipe (AdamW, learning rate 5e-5, batch size 96, fp16). The GD baseline, however, is the released BERT model pre-trained on BookCorpus plus English Wikipedia with the original NSP objective and the original BERT hyperparameters. No GD model is pre-trained with the Table 4 recipe. Consequently, the observation in Figure 1 that ID models match or exceed the GD baseline could be caused by the removal of NSP, the different optimizer and learning rate, the different batch size, or the shorter training schedule, rather than by the relevance of the specialized corpus. This directly undermines the headline conclusion in §4 that 'the relevance of the PT data may be more important than its size.' The authors should either train a GD model under the same MLM-only recipe and comparable compute, or substantially weaken the ID-versus-GD claims.","section":"§2.1, Table 4, Figure 1"},{"comment":"The correlation evidence for the central similarity hypothesis is statistically under-supported. The paper reports Spearman correlations of 55.39–96.00% in Table 2, but gives no confidence intervals, p-values, or per-seed correlation variability, even though each experiment was repeated with five seeds. More importantly, the E[accL1] similarity measure varies only between 99.6 and 100 in the scatter plots of Figures 5 and 6, so the strong correlations involving this measure are computed against an almost constant regressor and may be unstable or driven by a few points. The authors should report per-seed or bootstrap correlations with significance tests, and should consider a similarity measure with a larger dynamic range, or at least demonstrate that the E[accL1] correlations are robust to the narrow range of observed values.","section":"§2.2, Table 2, Figures 5 and 6"},{"comment":"The paper explicitly states that it 'focuses solely on the changes induced by pre-training, without any supervised learning,' yet the conclusions in §4 include practical recommendations about developing 'high-quality models' with domain-specific data. The three metrics (probing, THAS, DDC) are, as the authors implicitly acknowledge, proxies for downstream usefulness, but the manuscript does not validate that improvements on these metrics translate to fine-tuning performance. Since the practical recommendation is load-bearing, the authors should either add a small fine-tuning validation on a subset of the tasks or explicitly restrict the conclusions to representation-level quality and remove the practical 'high-quality models' framing.","section":"Limitations and §4"}],"minor_comments":[{"comment":"The phrase 'a through analysis' should read 'a thorough analysis.'","section":"§4"},{"comment":"Although the experiments are repeated with five random seeds, the main matrix figures and the appendix tables report only averages, with no error bars or variance measures; adding standard deviations would help the reader judge whether the reported improvements are stable.","section":"Figure 1 and Tables 5–7"},{"comment":"The AG-News row is garbled: '41/|Y|38' should presumably read '|Y|=4, prior 1/|Y|=0.25' or similar; please correct the formatting.","section":"Table 3"},{"comment":"The exact BERT checkpoint used as the GD baseline should be specified (e.g., bert-base-uncased versus bert-base-cased), and the vocabulary size of 30,522 suggests a specific tokenizer; this should be identified precisely for reproducibility.","section":"§2.1 and A.1"},{"comment":"The reproducibility statement says the authors 'will release the weights of the domain pre-trained models after publication,' but the paper should also release or specify the training code, exact hyperparameters per dataset, seed values, and evaluation scripts, since the experiments are central to all claims.","section":"A.1"},{"comment":"The x-axis is labeled 'pre-training steps,' but the caption and appendix refer to epochs (e.g., BERT_IMDb-80); please clarify the relationship between epochs and update steps in the captions or main text.","section":"Figures 2 and 4"}],"recommendation":"major_revision","confidential_remarks":"The main blocker is the unmatched GD baseline. This is fixable in principle by pre-training a GD model with the same MLM-only recipe, but that is a substantial additional experiment. If the authors can provide that matched baseline and strengthen the correlation statistics, the paper would be a solid empirical contribution; in its current form, the central ID-versus-GD comparison is not valid as reported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one if you care about corpus selection, but don't trust the headline as it stands. The paper compares three pre-training scenarios (in-domain from scratch, generic baseline, domain-adaptive) using three representation-quality metrics, and finds that small specialized corpora can match a large generic one, and that domain-adaptive gains correlate with n-gram similarity. That is a useful empirical question, and the experiment is more systematic than most: seven corpora, nine tasks, multiple seeds, three metrics, and an appendix that checks different layers and pooling functions. I credit them for being explicit about scope limits (no fine-tuning, one architecture, DDC binary only).\n\nThe soft spot is the one the stress-test note names, and it is real. The GD baseline is the released BERT, trained with NSP and Devlin et al.'s recipe; the ID and DA models are trained MLM-only with Table 4's recipe (AdamW, lr 5e-5, batch 96, fp16, far fewer steps). So the ID-vs-GD comparison, the one that motivates the \"relevance beats size\" conclusion, conflates corpus choice with training objective and hyperparameters. The DA improvements are also measured against that same mismatched baseline, so the correlation in Table 2 is not as clean as claimed—it could partly reflect the recipe change, not the corpus similarity. That is the load-bearing flaw, and it is fixable: either train a GD model with the same recipe, or clearly reframe the result as \"vs off-the-shelf BERT\" and soften the conclusion. Without that, the paper overstates.\n\nThe statistical reporting is also thin: five seeds but no error bars or significance tests, and the E[accL1] similarity values sit in 99.6–100%, so correlation coefficients over that range are fragile. The n-gram order for both similarity measures is never specified. Given the paper's own limitations, the conclusion that small specialized data \"can be highly competitive\" is reasonable as a hypothesis but not as strongly evidenced as the abstract suggests, especially without fine-tuning experiments.\n\nWho is this for? Practitioners choosing pre-training corpora, and people working on probing and representation analysis. The citation pattern is fine; the THAS self-citation is a published external measure. It deserves a serious referee, but with a request for major revision: a matched GD baseline or reframed claims, error bars, and specified n-gram orders.","headline":"Useful empirical study, but the headline comparison is confounded and the correlation analysis needs statistical support.","tokens_in":17763,"tokens_out":3039,"would_cite":false,"duration_ms":37158,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-training on a small, specialized corpus can be as effective as pre-training on a large generic one when the specialized text is close to the target task, and the benefit of combining both is predictable from distributional similarity.","keywords":["pre-training","domain adaptation","transformer representations","representation quality","probing","distributional similarity","low-annotation learning","BERT"],"falsifier":"Fine-tune every pre-trained representation from the three scenarios on the nine target tasks with standard supervised data, then compare fine-tuned performance to the ALC, THAS, and DDC scores. If representations that score higher on these metrics do not also perform better after fine-tuning, or if domain-adaptive pre-training with low n-gram similarity beats high-similarity setups, the paper's central claim is contradicted.","tokens_in":16791,"feed_emoji":"📊","tokens_out":6696,"duration_ms":80451,"temperature":0.7,"pith_summary":"This paper asks what the choice of pre-training corpus does to a transformer's representations, before any fine-tuning. Pre-training a fixed BERT-size model from scratch on seven domain-specific corpora and evaluating on nine tasks with three representation-quality metrics, it finds that a small, specialized corpus can match a much larger generic corpus whenever the specialized text is close to the target and big enough. It further finds that domain-adaptive pre-training, continuing a generic model on a specialized corpus, helps or hurts depending on how similar the specialized corpus is to the target task. Across all tasks the paper reports strong correlations between n-gram overlap of pre-training and target distributions and the measured gains, which suggests that relevance can matter as much as scale.","feed_headline":"Small domain corpora can match giant pretraining sets","feed_subtitle":"A seven-corpus study shows target relevance, not raw size, decides whether small pretraining corpora win.","key_machinery":"The load-bearing machinery is a three-metric evaluation suite applied without any supervised fine-tuning: low-annotation probing, which measures the area under the accuracy curve of simple classifiers trained on 100-1000 examples (ALC); task hierarchical alignment, which scores how purely task labels separate into agglomerative clusters (THAS); and data-dependent complexity, an eigenvalue-based measure of label-representation compatibility restricted to binary tasks (DDC). These metrics compare in-domain, general-domain, and domain-adaptive pre-training on a single BERT architecture. To explain the observed gains, the paper introduces two asymmetric n-gram distribution similarity measures, n-gram coverage and expected L1 accuracy, and correlates them with the measured quality differences.","core_discovery":"The paper's central claim is that representation quality induced solely by pre-training is governed by distributional relevance to the target, not by corpus size alone. In-domain pre-training on small corpora of roughly 1.8M to 100M words reaches representation quality comparable to a general-domain model pre-trained on about 3.3 billion words, with TREC as the one clear failure because its pre-training corpus is extremely small. Domain-adaptive pre-training produces the largest observed gains on domain-matched tasks, but it can also degrade representations when the specialized corpus is far from the target. The paper substantiates this dependence by correlating two asymmetric distributional similarity scores with the improvements over the general-domain baseline, reporting Spearman correlations mostly between 55% and 96% across probing, task alignment, and data-dependent complexity.","pith_inferences":["My inference: the correlation results suggest a cheap screening heuristic the authors do not spell out — compute n-gram coverage between unlabeled target text and a candidate specialized corpus before spending compute on continued pre-training, and skip the corpus if coverage is low.","My inference: since the practical recommendation depends on whether these representation metrics predict real fine-tuning performance, a natural next experiment is to fine-tune all representations on the nine tasks and check that the ranking survives.","My inference: because one similarity score (expected L1 accuracy) clusters near 99.6-100% in the appendix while n-gram coverage spreads across 0-100%, the n-gram coverage measure is likely carrying most of the predictive signal, which suggests only one metric may be needed in practice.","My inference: the paper implies a relevance-over-scale trade-off that could be mapped explicitly — hold the domain gap fixed, vary corpus size, and identify where in-domain quality saturates; the authors do not run this sweep."],"forward_implications":["Practitioners can pre-train from scratch on a domain corpus of a few million words and expect representation quality competitive with a much larger generic pre-training run, provided the corpus is not extremely small.","Domain-adaptive pre-training is not automatically a win: its benefit can be estimated in advance by measuring distributional similarity between the candidate specialized corpus and the target text.","The largest domain-matched representation gains come from continuing a generic model on in-domain data, not from either corpus alone.","The conclusions hold across different layer and token pooling choices, so the improvement is not an artifact of one feature-extraction strategy.","Because a failure case (TREC) is explained by corpus size, the paper implies a minimum scale below which specialized pre-training loses its advantage."],"supporting_citations":[{"why":"Defines domain-adaptive pre-training, the generic-plus-specialized scenario whose success the paper explains by distributional similarity.","marker":"Gururangan et al., 2020"},{"why":"Supplies the BERT architecture and the generic corpus (BookCorpus plus Wikipedia) used as the general-domain baseline G.","marker":"Devlin et al., 2019"},{"why":"Provides the probing technique that becomes the low-annotation ALC metric.","marker":"Ettinger et al., 2016"},{"why":"Extends probing to auxiliary classification tasks, the approach behind the low-annotation probes.","marker":"Adi et al., 2017"},{"why":"Defines the task hierarchical alignment score (THAS) used to measure label-representation alignment.","marker":"Gonzalez-Gutierrez et al., 2023"},{"why":"Defines data-dependent complexity (DDC), the third representation-quality metric.","marker":"Yauney and Mimno, 2021"},{"why":"Motivates training without next-sentence prediction, the MLM-only setup used for all pre-training.","marker":"Liu et al., 2019b"}],"fun_headline_variants":["Pretraining quality depends on relevance, not size","Small in-domain corpora rival billions-scale pretraining","Domain-adaptive pretraining wins when distribution matches","Corpus relevance beats corpus size in pretraining","Pretraining: small and domain-matched beats giant generic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the three representation-quality metrics — low-annotation probing, THAS, and DDC — correctly measure whether a representation will be useful on real downstream tasks, since no fine-tuning is performed anywhere in the study.","fun_headline_variants_meta":{"raw":{"variants":["Pretraining quality depends on relevance, not size","Small in-domain corpora rival billions-scale pretraining","Domain-adaptive pretraining wins when distribution matches","Corpus relevance beats corpus size in pretraining","Pretraining: small and domain-matched beats giant generic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000264,"raw_usage":{"total_tokens":1502,"prompt_tokens":743,"completion_tokens":759,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":359,"completion_tokens_details":{"reasoning_tokens":686}},"tokens_in":359,"tokens_out":759,"duration_ms":8919,"temperature":1.0,"reasoning_tokens":686,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:21:28.798999+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune every pre-trained representation from the three scenarios on the nine target tasks with standard supervised data, then compare fine-tuned performance to the ALC, THAS, and DDC scores. If representations that score higher on these metrics do not also perform better after fine-tuning, or if domain-adaptive pre-training with low n-gram similarity beats high-similarity setups, the paper's central claim is contradicted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the task hierarchical alignment score (THAS) used to measure label-representation alignment."}],"review_version":1}