{"id":"32ffeec6-821b-4d09-8434-91e5a097bd07","arxiv_id":"2506.16912","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Language models trained on the same data differ most on rare facts; larger models and LLaMA architectures appear more sample-efficient at factual recall.","lead":"This paper measures how many times a fact appears in a model's training text and checks whether the model can answer a question about that fact. It finds that larger models and LLaMA-style architectures are better at learning rare facts, while all models learn very common facts equally well.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline finding depends on unvalidated co-occurrence frequency counts; if the x-axis is miscalibrated the alpha-based sample-efficiency ranking is not trustworthy.","rationale":"I agree with the reader's weakest assumption. The x-axis frequency estimate is upstream of both proposed metrics and is explicitly acknowledged as approximate. A validation step is necessary before accepting the ranking. I also considered the absence of confidence intervals and the questionable separate reporting of L0=0.00 and x0=0.88, since only their sum enters F(x); both are real but secondary. No amount of uncertainty quantification would repair an unvalidated independent variable, while even perfect frequencies would not justify a ranking without repeated seeds or interval estimates. The paper is otherwise a controlled comparison with a public repository and shared training data, so a conditional verdict requiring frequency validation and uncertainty reporting is appropriate.","tokens_in":17690,"tokens_out":9950,"duration_ms":113609,"concrete_test":"Take a stratified sample of ~200 BEAR facts across all frequency buckets. For each, use an entity linker (or manual annotation) to find true mentions of the subject/object in the Wikipedia dump and label whether the BEAR relation is actually expressed in each sentence. Compute precision and recall of FactMatcherSimple counts against this gold standard, then refit L0, x0, and alpha_m on the corrected counts and compare with Table 1. If the LLaMA-first ranking or the size ordering changes, or if alpha differences fall within bootstrap confidence intervals, the central claim fails this check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is in Section 3.1: a fact is counted whenever the subject and object strings co-occur in a sentence, with no check that the BEAR relation is expressed and no entity disambiguation. These counts are the x-axis for every result, including the MLE of F(x)=1-(L0+x0)/(1+x)^alpha_m in Section 3.3.2 and the W_ASB buckets. The authors assert the heuristic is 'sufficiently accurate to predict the relative frequencies' but provide no precision/recall measurement; the Limitations section itself concedes 'possible mismappings of entities due to likely ambiguity or relation misidentification.' Because the model-specific alpha is fit against this shared but noisy covariate, systematic false positives (e.g., entity aliases co-occurring in unrelated sentences) and false negatives (unmatched surface forms) can reorder or compress true frequencies. That would change the shape of the fitted curve and the alpha differences that drive the ranking: small alphas differ by only ~0.002-0.003 (GPT2 0.084 vs XLSTM 0.086 vs MAMBA 0.087), so even modest x-axis noise could flip the ordering. The absence of confidence intervals or repeated seeds means the LLaMA-first, size-monotone conclusion is currently supported only by this unvalidated x-axis and a single run per configuration.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a framework for measuring how efficiently language models learn relational facts as a function of fact frequency in the pre-training corpus. The authors estimate fact frequencies by a co-occurrence heuristic applied to a Wikipedia dump, pre-train eight small-to-medium models (GPT-2, LLaMA, xLSTM, Mamba-2) on the same five-billion-token corpus, and propose two sample-efficiency metrics: a weighted accuracy over frequency buckets (W_ASB) and a fitted power-law curve F(x) = 1 - (L0 + x0)/(1+x)^alpha_m whose per-model slope alpha_m is interpreted as sample efficiency. The empirical results indicate that larger models are more sample-efficient, that LLaMA models achieve the highest scores, and that most models differ mainly on low-frequency facts. The paper also tracks metrics across training checkpoints and reports robustness experiments on frequency-split test sets.","tokens_in":18000,"tokens_out":2821,"duration_ms":30837,"significance":"If the proposed framework is valid, it would provide a useful, controlled methodology for comparing sample efficiency across architectures and sizes, and the released fact-frequency counts would be a reusable resource. The study has real strengths: all models are trained on the same corpus with the same tokenizer and comparable parameter counts, intermediate checkpoints are probed, and code and frequency statistics are released. The central qualitative trend, that accuracy increases with fact frequency and that model differences concentrate on low-frequency facts, is visible in the raw bucket accuracies and is not dependent on the parametric fit alone. However, the paper's headline ranking of architectures rests on the validity of the unvalidated co-occurrence frequency estimates and on the fitted alpha_m values, which are reported without uncertainty quantification. The significance of the contribution is therefore conditional on addressing these load-bearing methodological gaps.","major_comments":[{"comment":"The fact-frequency estimates that form the x-axis of the entire analysis are produced by a co-occurrence heuristic that counts a fact whenever the subject and object strings appear in the same sentence, with no verification that the relation is expressed and no entity disambiguation. The authors state that this heuristic is 'sufficiently accurate to predict the relative frequencies' but provide no precision/recall evaluation, no comparison against entity-linked counts, and no manual validation sample. The Limitations section itself concedes 'possible mismappings of entities due to likely ambiguity or relation misidentification.' Because every alpha_m and W_ASB value is computed from these frequency counts, systematic false positives (for example, alias co-occurrence in unrelated sentences) or false negatives can reorder or compress the true frequency axis and change the shape of the fitted curve. This is not a peripheral concern: the differences between non-LLaMA models in Table 1 are tiny (alpha_m = 0.084, 0.086, 0.087), so even modest x-axis noise could alter the reported ranking. I request a validation of the heuristic against a manually checked or entity-linked sample, and a sensitivity analysis showing that the main conclusions are stable under plausible frequency-count perturbations.","section":"Section 3.1"},{"comment":"The maximum-likelihood procedure fits L0, x0, and the model-specific alpha_m jointly on the same data that are then used to rank the models, and the paper reports no confidence intervals, bootstrap estimates, or repeated-seed variance for alpha_m. The conclusion that LLaMA is most sample-efficient and that larger models are more sample-efficient is essentially an ordering of fitted alpha_m values, yet the reported differences between several models are extremely small (for example, small GPT-2 at 0.084 versus small xLSTM at 0.086 versus small Mamba-2 at 0.087 in Table 1). Without uncertainty quantification or at least multiple training runs, it is impossible to determine whether these differences are statistically meaningful. I ask the authors to report profile-likelihood or bootstrap confidence intervals for alpha_m, and to state whether the ranking is stable across random seeds or replicated training runs.","section":"Section 3.3.2"},{"comment":"The weighting function in the W_ASB metric depends on an arbitrarily chosen decay parameter lambda = 0.05, and the paper does not report how the metric or the resulting model ranking changes as lambda varies. Since W_ASB is presented as one of the two proposed sample-efficiency metrics and is used to support the architecture-level conclusions, the choice of lambda should be justified either by a sensitivity analysis over a range of lambda values or by an explicit criterion for setting it. The current text states the value, but does not show that the ranking of models is robust to this choice.","section":"Section 3.3.1"},{"comment":"The metric-robustness experiments vary the frequency composition of the test split, but they do not test robustness to errors in the frequency estimates themselves, which is the more load-bearing uncertainty given the unvalidated heuristic in Section 3.1. In addition, the claim that alpha is robust across splits assumes that the sampled facts are 'on average equally hard,' an assumption that is stated but not tested. I recommend either directly validating the frequency estimates or adding a perturbation analysis that adds noise to the fact counts and measures the resulting change in alpha_m and W_ASB rankings.","section":"Section 4.3"}],"minor_comments":[{"comment":"There is a typo in the first sentence: 'wether' should be 'whether.' Also, 'repostory' in the Contributions section should be 'repository.'","section":"Section 3.1"},{"comment":"Figures 6(a) and 6(b) appear to display the same correlation matrix in the manuscript text; please check whether the two panels are intended to be different, and if so, label them clearly.","section":"Figure 6"},{"comment":"The model name 'LL AMA' is written with a space throughout the paper; this is likely a formatting artifact and should be corrected to 'LLaMA.'","section":"Throughout"},{"comment":"The sentence reporting L0 = 0.00 and x0 = 0.88 as optimized values would benefit from a brief description of the uncertainty in these estimates, especially since the interpretation of x0 as reflecting the heuristic's underestimation is central to the discussion.","section":"Section 4.1.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and addresses a timely question. The main risk is that the headline ranking depends on an unvalidated frequency estimate and on fitted parameters with no uncertainty quantification. I believe the required additional analyses (frequency-heuristic validation or sensitivity analysis, and confidence intervals for alpha_m) are feasible within a revision and would substantially strengthen the paper. I do not see grounds for rejection, as the raw bucket accuracies provide an independent check on the qualitative trend."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, clearly written empirical paper that adds a concrete way to measure how many times facts appear during pre-training and how model accuracy scales with that count. The controlled setup (same data, same tokenizer, same training hyperparameters, four architectures at two sizes) is genuinely nice—this is the right way to compare sample efficiency. The two metrics, W_ASB and the power-law alpha, are simple but reasonable, and the raw bucket accuracy plots are a good sanity check.\n\nWhat's actually new: frequency-annotated BEAR facts on a Wikipedia dump, released code and counts, and the finding that LLaMA models show the steepest accuracy-per-occurrence curve. The observation that scale helps most on rare facts is not new (they cite Kandpal and Lu), but the explicit framework for quantifying it is.\n\nThe soft spots are real but fixable. First and most importantly, the x-axis—the fact frequency counts—comes from a co-occurrence heuristic that is never validated. A fact is counted when subject and object strings appear in the same sentence, with no relation check and no entity disambiguation. The authors say it's 'sufficiently accurate' and then concede in the limitations that mismappings may occur. That's an assertion, not a measurement. If the heuristic compresses or reorders true frequencies, the fitted alpha values—and the LLaMA-first ranking—could change. The alpha differences between GPT-2, XLSTM, and Mamba are tiny (0.084 vs 0.086 vs 0.087), so even modest noise in the x-axis could flip them. This is the load-bearing issue.\n\nSecond, there are no confidence intervals, no repeated seeds, and the L0/x0/alpha parameters are fitted on the same data used to evaluate the models. The paper is honest about the fitting, and the raw bucket accuracies provide some independent support, but the headline ordering is essentially an ordering of fitted values with no uncertainty attached.\n\nThird, the lambda=0.05 weighting in W_ASB is arbitrary, though they do show the metric is somewhat robust across frequency splits.\n\nThe citation pattern looks fine. The related work on long-tail knowledge and scaling laws is appropriately cited, and the self-citations are to the BEAR probe which they build on.\n\nWho is this for: researchers working on factual knowledge evaluation or LM sample efficiency, especially those comparing architectures with limited compute. It deserves a serious referee. The main revision should be validating the frequency heuristic against a small hand-labeled or entity-linked set, and reporting repeated seeds or bootstrap confidence intervals. If the ranking survives that, it's a useful contribution. As it stands, I'd treat the LLaMA-first conclusion as suggestive, not established.","headline":"Useful framework, honest limitations, but the headline ranking rests on an unvalidated frequency heuristic and no uncertainty quantification.","tokens_in":18481,"tokens_out":2390,"would_cite":false,"duration_ms":23854,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a language model's factual sample efficiency can be measured by fitting a power-law curve to its accuracy as a function of how often a fact appears in pre-training data, with the fitted slope $\\alpha_m$ serving as a…","keywords":["sample efficiency","language models","factual knowledge","fact frequency","power-law scaling","LLaMA","knowledge probing","long-tail distribution"],"falsifier":"Take a random sample of BEAR facts, recompute their frequencies with an entity-linking and relation-extraction pipeline (or manual annotation), and re-fit $\\alpha_m$; if the relative frequencies change enough to reorder the $\\alpha_m$ values across the eight models, the claim that $\\alpha_m$ measures sample efficiency is not supported.","tokens_in":17500,"feed_emoji":"📈","tokens_out":4150,"duration_ms":38023,"temperature":0.7,"pith_summary":"This paper tries to define and measure sample efficiency for language models: how much exposure to a fact in the training corpus a model needs before it can answer questions about that fact. The authors count how often each relational fact appears in a Wikipedia pre-training corpus, train eight models of three architectures at two sizes on the same data, and probe them with multiple-choice factual questions. They fit a power-law curve to each model's accuracy as a function of fact frequency, and the fitted slope $\\alpha_m$ becomes a single number summarizing that model's sample efficiency. They report that larger models, and LLaMA models in particular, have higher $\\alpha_m$ values, meaning they gain more accuracy per additional exposure, especially on rare facts. The contribution is a concrete, reproducible way to compare architectures on learning efficiency rather than final accuracy alone.","feed_headline":"One fitted slope ranks how fast language models learn facts","feed_subtitle":"A new metric ties accuracy to fact frequency in pre-training data, separating LLaMA from GPT2, xLSTM and Mamba2.","key_machinery":"The central object is the power scaling function $F(x) = 1 - \\frac{L_0 + x_0}{(1+x)^{\\alpha_m}}$, fitted by maximum likelihood over all BEAR probe items and models, with $\\alpha_m$ as the model-specific slope that the paper treats as the sample-efficiency score. It is paired with a weighted accuracy score over frequency buckets, $W_{ASB}$, which gives a more direct accuracy view, and both metrics rest on frequency counts produced by a co-occurrence heuristic that counts a fact whenever its subject and object appear in the same sentence. The function does the load-bearing work: it replaces a binary 'learned or not' threshold with a continuous curve, so sample efficiency becomes how quickly accuracy grows with exposure and can be compared across models with a single number.","core_discovery":"The paper's central claim is that a language model's factual sample efficiency can be quantified by fitting the function $F(x) = 1 - \\frac{L_0 + x_0}{(1+x)^{\\alpha_m}}$, where $x$ is the number of times a subject–relation–object fact co-occurs in a sentence in the pre-training corpus and $\\alpha_m$ is a per-model parameter. A higher $\\alpha_m$ means the probability of answering a factual question correctly rises faster per additional observed sample, so $\\alpha_m$ ranks models by sample efficiency. Trained on identical five-billion-token Wikipedia data, the LLaMA models achieved the highest $\\alpha$ values (0.103 for the small model, 0.120 for the medium), followed by Mamba 2, xLSTM, and GPT2, and in every architecture the medium model beat the small one. The paper also finds that models differ most on low-frequency facts (fewer than about 1024 occurrences), while high-frequency facts are learned to similar accuracy by all models, and that $\\alpha$ values are robust across test splits with different frequency profiles.","pith_inferences":["The co-occurrence heuristic may inflate frequencies when two entities appear in a sentence without actually expressing the target relation; recomputing counts with entity linking or manual annotation could reorder the $\\alpha_m$ rankings across the eight models.","The fitted $\\alpha_m$ could be connected to neural scaling laws: if it scales predictably with model size or compute, it could become a planning tool for estimating how much data a given architecture needs to reach a target fact-recall level.","The method could be applied to other corpora or languages to test whether the sample-efficiency rankings persist across data distributions, since the paper only uses one English Wikipedia dump."],"forward_implications":["If $\\alpha_m$ is a valid sample-efficiency measure, model comparisons no longer need matched final accuracies; one can rank architectures by how fast they convert exposures into knowledge.","The finding that size helps mainly on low-frequency facts implies that training larger models is a direct route to better long-tail knowledge, while smaller models can match them on common facts.","The reported robustness of $\\alpha$ across test splits with different frequency profiles means a non-representative probe set could still yield a reliable efficiency ranking.","Because $\\alpha$ values stabilize early in training, the metric measured at an intermediate checkpoint could predict final sample efficiency without completing full training."],"supporting_citations":[{"why":"Supplies the BEAR probe, the multiple-choice factual question set used to evaluate all models' knowledge.","marker":"(Wiland et al., 2024)"},{"why":"Provides the distant-supervision assumption that co-occurrence of two entities in a sentence indicates a relational fact, the basis of the frequency-counting heuristic.","marker":"(Mintz et al., 2009)"},{"why":"Contributes the power-law scaling form that the paper adapts into the probability function F(x).","marker":"(Kaplan et al., 2020)"},{"why":"Prior work on scaling laws for fact memorization that the paper extends and whose observation about high-frequency facts it corroborates.","marker":"(Lu et al., 2024)"},{"why":"Motivates the long-tail knowledge problem that sample efficiency is meant to address.","marker":"(Kandpal et al., 2023)"},{"why":"The English Wikipedia dump that serves as both the pre-training corpus and the source of fact frequency counts.","marker":"(Wikimedia Foundation, 2023)"}],"fun_headline_variants":["Slope parameter ranks language models' fact-learning speed","Fact frequency reveals which LLMs learn rare facts best","Rare facts separate models; slope quantifies learning speed","One fitted slope ranks LLMs by fact-learning efficiency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole analysis rests on the assumption that counting sentences where the subject and object merely appear together gives accurate relative frequencies of how often a fact is actually expressed in the training text.","fun_headline_variants_meta":{"raw":{"variants":["Slope parameter ranks language models' fact-learning speed","Fact frequency reveals which LLMs learn rare facts best","Rare facts separate models; slope quantifies learning speed","One fitted slope ranks LLMs by fact-learning efficiency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000689,"raw_usage":{"total_tokens":3100,"prompt_tokens":899,"completion_tokens":2201,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":2137}},"tokens_in":515,"tokens_out":2201,"duration_ms":15137,"temperature":1.0,"reasoning_tokens":2137,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:16:04.354716+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of BEAR facts, recompute their frequencies with an entity-linking and relation-extraction pipeline (or manual annotation), and re-fit $\\alpha_m$; if the relative frequencies change enough to reorder the $\\alpha_m$ values across the eight models, the claim that $\\alpha_m$ measures sample efficiency is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the distant-supervision assumption that co-occurrence of two entities in a sentence indicates a relational fact, the basis of the frequency-counting heuristic."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The English Wikipedia dump that serves as both the pre-training corpus and the source of fact frequency counts."}],"review_version":2}