Pith. sign in

REVIEW 3 major objections 5 minor 16 references

Scaling Down Semantic Leakage: Investigating Associative Bias in Smaller Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Within the Qwen2.5 family, smaller models leak less semantic association—but not linearly, as the 3B model sometimes outleaks the 7B.

desk verdict A well-documented study whose central scaling claim is undercut by a confound the author herself identifies: low Leak-Rate in small models may mean prompt insensitivity, not less semantic leakage. read the letter →

arxiv 2501.06638 v1 pith:3J7SZEDF submitted 2025-01-11 cs.CL

Veronika Smilga This is my paper · ORCID
classification cs.CL
keywords semanticleakagelanguagemodelscalingQwen2.5associativebiascolorpromptsMeanLeak-Ratesmallmodelsinstruction-tuned
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether smaller language models are less prone to semantic leakage, the unexpected surfacing of training-data associations in generated text. Testing four Qwen2.5-Instruct models from 0.5B to 7B parameters, it finds that leakage generally increases with model size, but the trend is not monotonic: the 3B model sometimes surpasses the 7B. To measure this, the author builds a new color-focused prompt dataset alongside the original semantic-leakage dataset. If correct, the result means model size alone is not a reliable predictor of associative bias, and even small models carry some leaking associations.

What carries the argument

The central object is the Mean Leak-Rate metric, which scores each prompt as 100, 0, or 50 depending on whether the similarity between a leakage-triggering concept and a test generation is higher, lower, or equal to the similarity with a control generation. The other load-bearing components are the Qwen2.5-Instruct family spanning 0.5B, 1.5B, 3B, and 7B parameters (the last in GPTQ-Int4 quantized form) and a new dataset of 660 color-template prompts plus 60 manually crafted prompts with color-related names and set expressions. BERT-score and SentenceBERT supply the similarity measures that drive the metric.

What would settle it

Run the same semantic-leakage prompts on the full-precision Qwen2.5-7B-Instruct with identical decoding settings and check whether its Mean Leak-Rate exceeds that of the 3B model; if it does, the non-monotonic size trend is an artifact of the quantized 7B checkpoint.

Watch

Extended reading notes

Core claim

The author claims that within the Qwen2.5-Instruct family, models under 7 billion parameters tend to exhibit more semantic leakage as their size increases, with the 0.5B model showing the least leakage and the 3B model occasionally surpassing the 7B. This non-linear pattern appears on both the original prompt set and the new color-prompt dataset, using BERT-score and SentenceBERT to compute Mean Leak-Rate. All models leak above the 50% random baseline, and Qwen2.5 models appear more leak-prone than larger Llama and OpenAI models previously reported. The color dataset further shows that prompts requiring a non-color concept leak most on average, contrary to the hypothesis that color-to-color associations would leak more.

Load-bearing premise

The study treats a quantized, compressed version of the 7B model as equivalent to the full-precision model in leakage behavior, so the size-comparison conclusions hinge on quantization not changing leakage.

Editorial extensions

If this is right

  • If size predicts leakage only loosely, then statements that equate larger models with more associative leakage should be qualified.
  • The 3B model's higher leakage is largely driven by its tendency to repeat the prompt's color concept in the generation, a behavior the other models do not exhibit.
  • On the color dataset, leakage is strongest when the expected generation is a non-color concept, not when both prompt and generation are colors.
  • Smaller models' lower leakage coincides with less context-sensitive and less diverse generations, suggesting leakage is tied to associative knowledge rather than being purely harmful.
  • Extending the same evaluation to Qwen2.5-14B, -32B, and -72B would test whether the size trend reverses or continues beyond 7B.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 3B-versus-7B comparison may be confounded because the 7B model is a quantized variant; if quantization reduces leakage, the non-monotonic result might be an artifact.
  • The metric may conflate genuine associative leakage with a copying or repetition failure mode, as the paper's own observation about the 3B model suggests.
  • The color dataset's category imbalance (330/330/60) makes conclusions about the smallest category less reliable, which the author acknowledges.
  • A natural next test is to run the full-precision 7B model and models with different pretraining distributions to see whether the size trend is specific to Qwen2.5 or more general.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies semantic leakage in Qwen2.5-Instruct models of 0.5B, 1.5B, 3B, and 7B parameters. It applies the Mean Leak-Rate metric from Gonen et al. (2024) to the original 109-prompt dataset and to a new color-focused dataset of 720 prompts (660 template-based prompts for two color categories and 60 manually crafted prompts for a third category). Leak-Rate is computed with BERT-score and SentenceBERT. The reported results show that the 0.5B model has the lowest or near-lowest Mean Leak-Rate on both datasets and that the 3B model sometimes exceeds the 7B model, leading to the paper's claim that semantic leakage increases with model size in the 0.5B-to-7B range but not strictly linearly. The paper releases the dataset, model generations, and evaluation code.

Significance. If the descriptive pattern were robust, the paper would be a useful contribution to our understanding of how associative biases scale with model size and would inform model selection for applications where semantic leakage matters. The paper's strengths are its public release of data, generations, and code, its use of two similarity metrics, and its candid acknowledgment of several limitations. However, the central conclusion is currently not established: the paper's own Section 4 raises a prompt-insensitivity mechanism that can depress Leak-Rate for small models without indicating an absence of learned associations, and the 7B model is quantized rather than full-precision. These issues make the headline claim a plausible but unsupported interpretation of the observed numbers.

major comments (3)
  1. [Section 4 and Eq. (1)] Section 4 states that Qwen2.5-0.5B-Instruct and Qwen2.5-1.5B-Instruct 'tend to generate the same continuations to the prompts regardless of the preceding context.' Under the Leak-Rate definition in Eq. (1), if a model's test and control generations coincide, the per-prompt score is at the 50% chance level; more generally, reduced sensitivity to the prompt's concept will compress Leak-Rate toward 50. The paper's central evidence that smaller models leak less is therefore compatible with the alternative explanation that small models simply ignore the leakage-triggering concept. Because the paper itself raises this mechanism but still concludes 'smaller models exhibit less semantic leakage overall' in the Abstract and Section 3, the conclusion is not supported as stated. A control analysis that separates prompts with and without differential test-versus-control generations, or a measure of conditional dependence on the concept, is needed.
  2. [Section 2.1, footnote 5] The 7B data point is Qwen2.5-7B-Instruct-GPTQ-Int4 while the other three models are unquantized. The non-monotonic comparisons in Table 2 (for example, BERT-score 83.03 for 3B versus 74.77 for 7B on the original dataset) assume that GPTQ-Int4 quantization does not materially change leakage behavior. This is a load-bearing assumption for the 'not strictly linear' part of the claim, and the paper should either validate it by running the full-precision 7B model on at least a subset, or restrict the size-scaling conclusion to the 0.5B-to-3B range.
  3. [Section 3, Table 2] The Mean Leak-Rate values are point estimates from a single run with temperature 0.5. With stochastic decoding and no repeated sampling or bootstrap intervals, differences of a few points (for example, SentenceBERT 77.52 versus 80.64 for the 1.5B and 7B models on the original dataset) cannot be distinguished from sampling noise. The ranking claims need confidence intervals, multiple seeds, or a significance test.
minor comments (5)
  1. [Section 2.1, footnote 5] The footnote contains a typo: 'Dut to computational constraints' should be 'Due to computational constraints'.
  2. [Section 2.3] The paper states that four handcrafted templates were used for category 3, but only two examples are given. To make the dataset fully reproducible, all four templates and their corresponding control prompts should be listed, not just the color-related names and set expressions.
  3. [Section 3 and Table 3] The cross-category comparisons in Table 3 are based on unbalanced category sizes (330, 330, and 60 prompts). The paper acknowledges this imbalance in Section 4, but the acknowledgment should also be repeated when Table 3 is discussed so that readers do not overinterpret the category-level differences.
  4. [Section 3] The comparison with Gonen et al. (2024) results (for example, 'Qwen2.5-3B-Instruct surpasses Llama-3-8B-Instruct') should be treated cautiously because the original code is not available and the implementation of the metric may differ. The paper notes the code unavailability but still presents these cross-paper differences as directly comparable.
  5. [Section 2.2] The paper should specify which BERT-score variant is used (for example, F1) and whether the same tokenization settings were applied across models; this would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper applies an externally defined metric and dataset; acknowledged confounds are validity issues, not circular derivation.

full rationale

The paper's derivation chain is empirical rather than definitional. Semantic leakage is operationalized via Mean Leak-Rate from Gonen et al. (2024), and the paper applies that external metric to externally specified or newly constructed prompts without fitting any parameter to the outcome. The central claim that smaller Qwen2.5 models leak less is a generalization from Table 2, not a consequence of the metric's definition: the metric compares similarity of the concept to test and control generations, and the scores are not constrained by construction to order model sizes. The limitations acknowledged in Section 4 — that the smaller models' lower scores may be explained by generating the same continuations regardless of context, and that the 7B model is GPTQ-Int4 quantized (footnote 5) — are threats to construct validity and comparability, not circularity. No load-bearing step reduces to its own inputs, and there are no self-citations carrying the argument. The measurement-choice issues could make the conclusions less reliable, but they do not make the paper circular.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

There are no fitted numerical parameters in the paper: the Leak-Rate formula, similarity models, and the 50% baseline all come from prior work or definition. The central claim rests on domain assumptions about the validity of the metric, the comparability of the quantized 7B model, the representativeness of single generations, and the quality of the handcrafted dataset. These assumptions are mostly stated or acknowledged, but they are not independently verified within the paper.

assumptions (6)
  • domain assumption The Leak-Rate metric, with BERT-score and SentenceBERT similarities, validly measures semantic leakage.
    Used throughout Section 2.2 and the results; if the similarity models are biased toward lexical repetition, repetition examples like 'green house' inflate leakage scores, a limitation the paper itself notes in Section 4.
  • domain assumption GPTQ-Int4 quantization of the 7B model leaves its leakage behavior comparable to a full-precision 7B model.
    Section 2.1, footnote 5 states the 7B model is quantized for compute reasons; Tables 2 and 3 treat this model as the 7B data point in size comparisons.
  • domain assumption A single generation per prompt at temperature 0.5 is representative of the model's leakage behavior.
    Section 2.1 sets temperature and max tokens; no repeated sampling or seed control is reported, so small score differences may not be stable.
  • ad hoc to paper The handcrafted color templates and control prompts isolate the prompt concept as the only varying factor.
    Section 2.3 describes slot-filling 20 templates and manually built category 3 prompts; if templates carry unintended associations, per-category comparisons are affected.
  • domain assumption Qwen2.5-Instruct models represent smaller language models generally.
    Sections 1 and 2.1 generalize from one open-weight family; the paper does not test other families below 7B.
  • domain assumption Manual identification of the concept and control item for each Gonen et al. prompt is accurate.
    Section 2.3 states 'I manually identify the concept and the corresponding control item for each test prompt'; manual labeling can introduce subjectivity into the original-dataset scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Down Semantic Leakage: Investigating Associative Bias in Smaller Language Models." pith.science (2026). https://pith.science/paper/3J7SZEDF

@misc{pith2026250106638,
  author       = {Pith},
  title        = {Pith review of: Scaling Down Semantic Leakage: Investigating Associative Bias in Smaller Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3J7SZEDF}},
  note         = {Machine review of arXiv:2501.06638}
}
read the original abstract

Semantic leakage is a phenomenon recently introduced by Gonen et al. (2024). It refers to a situation in which associations learnt from the training data emerge in language model generations in an unexpected and sometimes undesired way. Prior work has focused on leakage in large language models (7B+ parameters). In this study, I use Qwen2.5 model family to explore whether smaller models, ranging from 500M to 7B parameters, demonstrate less semantic leakage due to their limited capacity for capturing complex associations. Building on the previous dataset from Gonen et al. (2024), I introduce a new dataset of color-focused prompts, categorized into specific types of semantic associations, to systematically evaluate the models' performance. Results indicate that smaller models exhibit less semantic leakage overall, although this trend is not strictly linear, with medium-sized models sometimes surpassing larger ones in leaking behavior. The dataset, the model generations, and the evaluation code are publicly available at https://github.com/smilni/semantic_leakage_project.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 2 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Brent Berlin and Paul Kay. 1991. Basic color terms: Their universality and evolution. Univ of California Press

    work page 1991

  4. [4]

    Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. http://arxiv.org/abs/1607.06520 Man is to computer programmer as woman is to homemaker? debiasing word embeddings

    arXiv 2016

  5. [5]

    Gonzalez, and Ion Stoica

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. http://arxiv.org/abs/2403.04132 Chatbot arena: An open platform for evaluating llms by human preference

    arXiv 2024

  6. [6]

    Omer Dahary, Or Patashnik, Kfir Aberman, and Daniel Cohen-Or. 2024. http://arxiv.org/abs/2403.16990 Be yourself: Bounded attention for multi-subject text-to-image generation

    arXiv 2024

  7. [7]

    Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. 2023. http://arxiv.org/abs/2212.05032 Training-free structured diffusion guidance for compositional text-to-image synthesis

    arXiv 2023

  8. [8]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. http://arxiv.org/abs/2210.17323 Gptq: Accurate post-training quantization for generative pre-trained transformers

    arXiv 2023

Show all 16 references
  1. [9]

    Hila Gonen, Terra Blevins, Alisa Liu, Luke Zettlemoyer, and Noah A Smith. 2024. Does liking yellow imply driving a school bus? semantic leakage in language models. arXiv preprint arXiv:2408.06518

    2024 arXiv

  2. [10]

    Li Lucy and David Bamman. 2021. https://doi.org/10.18653/v1/2021.nuse-1.5 Gender and representation bias in GPT -3 generated stories . In Proceedings of the Third Workshop on Narrative Understanding, pages 48--55, Virtual. Association for Computational Linguistics

    2021 doi

  3. [11]

    Roberto Navigli, Simone Conia, and Bj\" o rn Ross. 2023. https://doi.org/10.1145/3597307 Biases in large language models: Origins, inventory, and discussion . J. Data and Information Quality, 15(2)

    2023 doi

  4. [12]

    Royi Rassin, Shauli Ravfogel, and Yoav Goldberg. 2022. http://arxiv.org/abs/2210.10606 Dalle-2 is seeing double: Flaws in word-to-concept mapping in text2image models

    2022 arXiv

  5. [13]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing

    2019

  6. [14]

    Rothkopf, and Kristian Kersting

    Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A. Rothkopf, and Kristian Kersting. 2021. http://arxiv.org/abs/2103.11790 Language models have a moral dimension . CoRR, abs/2103.11790

    2021 arXiv

  7. [15]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  8. [16]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In Proceedings of International Conference on Learning Representations

    2020

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.