{"id":"71d3ba40-9add-4afa-afcd-d76cfd3fceba","arxiv_id":"2501.06638","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Tests four Qwen2.5 models and finds the 0.5B model shows the least semantic leakage, but the size trend is non-monotonic.","lead":"This paper measures a phenomenon called semantic leakage, where language models copy or repeat associations from their training data in odd ways, in small-to-medium AI models from 500 million to 7 billion parameters. It finds that the smallest model leaks least often, but the relationship is not smooth, and it releases a new color-focused test set and all model outputs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Small models' low leak rates may reflect prompt insensitivity, not less semantic leakage — a confound the paper itself acknowledges.","rationale":"The reader's weakest_assumption focuses on the quantized 7B model, which threatens the non-linear pattern (3B sometimes surpassing 7B). However, the more fundamental threat is to the primary conclusion that smaller models leak less. The paper itself provides the key evidence for this threat: small models generate the same continuations regardless of prompt context. If confirmed, this means the Leak-Rate for small models is not measuring semantic leakage as defined (concept-driven association) but rather a failure to condition on the prompt, forcing the metric toward its 50% baseline. This makes the headline conclusion misleading and gives a mechanistic alternative explanation for the size trend. The reader's concern about quantization is valid for the secondary claim, but the prompt-insensitivity confound is more load-bearing because it invalidates the interpretation of the smallest model's low scores, which is the core evidence. I keep the verdict at CONDITIONAL (unchanged) because the paper is transparent about this limitation and provides public data/code, so a revised analysis could salvage parts of the claim; however, the stated conclusion needs substantial reworking or additional control analyses before the scaling claim can be accepted. My proposed concrete test directly measures test-control similarity and generation diversity, which would decisively show whether the small models are simply ignoring the prompt.","tokens_in":7525,"tokens_out":6275,"duration_ms":60085,"concrete_test":"For each model, compute the mean BERTScore F1 similarity between test and control generations across all prompts in both datasets. If the 0.5B and 1.5B models show near-identical test/control outputs (mean F1 > 0.85) while the 3B and 7B models show clearly lower similarity, the low Leak-Rate for small models is an artifact of prompt insensitivity. Additionally, within each template, count distinct generations across the 11 inserted colors; if 0.5B produces only 1-2 distinct outputs per template, the model is not conditioning on the concept, so the Leak-Rate cannot measure associative leakage for that model. Publishing these distributions alongside the mean Leak-Rates would settle whether the scaling conclusion is interpretable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'smaller models exhibit less semantic leakage overall' is undermined by the paper's own observation in Section 4: 'the lower Mean Leak-Rate of Qwen2.5-0.5B-Instruct and Qwen2.5-1.5B-Instruct may be explained by the fact that these models tend to generate the same continuations to the prompts regardless of the preceding context.' If a model ignores the concept in the prompt, its test generation will resemble its control generation, forcing Leak-Rate toward the 50% chance baseline. Thus a low Leak-Rate for small models can indicate prompt insensitivity rather than an absence of learned associations. This is not a peripheral caveat: the 0.5B model's consistently lowest score is the primary evidence for the scaling claim. The discussion's admission of this mechanism is internally inconsistent with the paper's conclusion. While the quantized 7B model noted by the reader is a real confound, it bears only on the secondary non-monotonic pattern; this prompt-insensitivity confound strikes at the main size-leakage relationship itself.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies semantic leakage in Qwen2.5-Instruct models of 0.5B, 1.5B, 3B, and 7B parameters. It applies the Mean Leak-Rate metric from Gonen et al. (2024) to the original 109-prompt dataset and to a new color-focused dataset of 720 prompts (660 template-based prompts for two color categories and 60 manually crafted prompts for a third category). Leak-Rate is computed with BERT-score and SentenceBERT. The reported results show that the 0.5B model has the lowest or near-lowest Mean Leak-Rate on both datasets and that the 3B model sometimes exceeds the 7B model, leading to the paper's claim that semantic leakage increases with model size in the 0.5B-to-7B range but not strictly linearly. The paper releases the dataset, model generations, and evaluation code.","tokens_in":7682,"tokens_out":6238,"duration_ms":60856,"significance":"If the descriptive pattern were robust, the paper would be a useful contribution to our understanding of how associative biases scale with model size and would inform model selection for applications where semantic leakage matters. The paper's strengths are its public release of data, generations, and code, its use of two similarity metrics, and its candid acknowledgment of several limitations. However, the central conclusion is currently not established: the paper's own Section 4 raises a prompt-insensitivity mechanism that can depress Leak-Rate for small models without indicating an absence of learned associations, and the 7B model is quantized rather than full-precision. These issues make the headline claim a plausible but unsupported interpretation of the observed numbers.","major_comments":[{"comment":"Section 4 states that Qwen2.5-0.5B-Instruct and Qwen2.5-1.5B-Instruct 'tend to generate the same continuations to the prompts regardless of the preceding context.' Under the Leak-Rate definition in Eq. (1), if a model's test and control generations coincide, the per-prompt score is at the 50% chance level; more generally, reduced sensitivity to the prompt's concept will compress Leak-Rate toward 50. The paper's central evidence that smaller models leak less is therefore compatible with the alternative explanation that small models simply ignore the leakage-triggering concept. Because the paper itself raises this mechanism but still concludes 'smaller models exhibit less semantic leakage overall' in the Abstract and Section 3, the conclusion is not supported as stated. A control analysis that separates prompts with and without differential test-versus-control generations, or a measure of conditional dependence on the concept, is needed.","section":"Section 4 and Eq. (1)"},{"comment":"The 7B data point is Qwen2.5-7B-Instruct-GPTQ-Int4 while the other three models are unquantized. The non-monotonic comparisons in Table 2 (for example, BERT-score 83.03 for 3B versus 74.77 for 7B on the original dataset) assume that GPTQ-Int4 quantization does not materially change leakage behavior. This is a load-bearing assumption for the 'not strictly linear' part of the claim, and the paper should either validate it by running the full-precision 7B model on at least a subset, or restrict the size-scaling conclusion to the 0.5B-to-3B range.","section":"Section 2.1, footnote 5"},{"comment":"The Mean Leak-Rate values are point estimates from a single run with temperature 0.5. With stochastic decoding and no repeated sampling or bootstrap intervals, differences of a few points (for example, SentenceBERT 77.52 versus 80.64 for the 1.5B and 7B models on the original dataset) cannot be distinguished from sampling noise. The ranking claims need confidence intervals, multiple seeds, or a significance test.","section":"Section 3, Table 2"}],"minor_comments":[{"comment":"The footnote contains a typo: 'Dut to computational constraints' should be 'Due to computational constraints'.","section":"Section 2.1, footnote 5"},{"comment":"The paper states that four handcrafted templates were used for category 3, but only two examples are given. To make the dataset fully reproducible, all four templates and their corresponding control prompts should be listed, not just the color-related names and set expressions.","section":"Section 2.3"},{"comment":"The cross-category comparisons in Table 3 are based on unbalanced category sizes (330, 330, and 60 prompts). The paper acknowledges this imbalance in Section 4, but the acknowledgment should also be repeated when Table 3 is discussed so that readers do not overinterpret the category-level differences.","section":"Section 3 and Table 3"},{"comment":"The comparison with Gonen et al. (2024) results (for example, 'Qwen2.5-3B-Instruct surpasses Llama-3-8B-Instruct') should be treated cautiously because the original code is not available and the implementation of the metric may differ. The paper notes the code unavailability but still presents these cross-paper differences as directly comparable.","section":"Section 3"},{"comment":"The paper should specify which BERT-score variant is used (for example, F1) and whether the same tokenization settings were applied across models; this would improve reproducibility.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The main issue is internal consistency: Section 4's prompt-insensitivity explanation directly undermines the headline scaling claim, and this needs to be resolved in the analysis rather than only mentioned as a caveat. If the author can add a sensitivity-controlled analysis and address the quantized 7B model, the paper could become a solid descriptive study; as it stands, the central claim is not fully supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a careful, well-documented study whose main conclusion is undermined by a confound the author states in her own discussion. The claim that smaller models leak less semantic information is not supported by the data as analyzed.\n\nThe genuinely useful parts: a new 660-prompt color-focused dataset (with control prompts), the first systematic look at leakage in Qwen2.5 models below 7B, and public release of generations and evaluation code. The writing is clear and the limitations section is unusually honest. The citation pattern is fine; Gonen et al. is the obvious comparator and is used appropriately.\n\nThe central problem: Section 4 says the lower Mean Leak-Rate of the 0.5B and 1.5B models may be because those models generate the same continuation regardless of preceding context. If the test generation matches the control generation, Leak-Rate is forced toward the 50% 'no leakage' baseline. So a low Leak-Rate can mean the model is ignoring the prompt, not that it lacks learned associations. The paper treats this as a caveat but still concludes 'smaller models exhibit less semantic leakage overall.' That conclusion does not follow. The non-monotonic pattern (3B sometimes exceeding 7B) also gets murky because the 7B model is a GPTQ-Int4 quantized checkpoint, so you're comparing different implementations.\n\nMinor issues: no repeated sampling or confidence intervals, so a 73.85 vs 77.52 difference is hard to interpret; category 3 has only 60 prompts versus 330 each for the other two. These are fixable.\n\nWho benefits: anyone building or evaluating LM bias metrics should read this, mostly as a cautionary example. If you need a color-prompt dataset, the released resource is usable regardless of the scaling claim.\n\nRecommendation: worth a serious referee, but the review should ask for a response condition: recompute or reinterpret once prompt sensitivity is controlled, or explicitly reframe the paper as a study of metric behavior rather than model scaling. It should not be accepted as-is.","headline":"A well-documented study whose central scaling claim is undercut by a confound the author herself identifies: low Leak-Rate in small models may mean prompt insensitivity, not less semantic leakage.","tokens_in":8231,"tokens_out":2915,"would_cite":true,"duration_ms":26603,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Within the Qwen2.5 family, smaller models leak less semantic association—but not linearly, as the 3B model sometimes outleaks the 7B.","keywords":["semantic leakage","language model scaling","Qwen2.5","associative bias","color prompts","Mean Leak-Rate","small language models","instruction-tuned models"],"falsifier":"Run the same semantic-leakage prompts on the full-precision Qwen2.5-7B-Instruct with identical decoding settings and check whether its Mean Leak-Rate exceeds that of the 3B model; if it does, the non-monotonic size trend is an artifact of the quantized 7B checkpoint.","tokens_in":7277,"feed_emoji":"🎨","tokens_out":11232,"duration_ms":79852,"temperature":0.7,"pith_summary":"This paper asks whether smaller language models are less prone to semantic leakage, the unexpected surfacing of training-data associations in generated text. Testing four Qwen2.5-Instruct models from 0.5B to 7B parameters, it finds that leakage generally increases with model size, but the trend is not monotonic: the 3B model sometimes surpasses the 7B. To measure this, the author builds a new color-focused prompt dataset alongside the original semantic-leakage dataset. If correct, the result means model size alone is not a reliable predictor of associative bias, and even small models carry some leaking associations.","feed_headline":"0.5B model leaks least; 3B outleaks 7B in Qwen2.5","feed_subtitle":"Leakage grows with model size until the mid-size 3B beats the largest tested, so size alone is not a reliable predictor.","key_machinery":"The central object is the Mean Leak-Rate metric, which scores each prompt as 100, 0, or 50 depending on whether the similarity between a leakage-triggering concept and a test generation is higher, lower, or equal to the similarity with a control generation. The other load-bearing components are the Qwen2.5-Instruct family spanning 0.5B, 1.5B, 3B, and 7B parameters (the last in GPTQ-Int4 quantized form) and a new dataset of 660 color-template prompts plus 60 manually crafted prompts with color-related names and set expressions. BERT-score and SentenceBERT supply the similarity measures that drive the metric.","core_discovery":"The author claims that within the Qwen2.5-Instruct family, models under 7 billion parameters tend to exhibit more semantic leakage as their size increases, with the 0.5B model showing the least leakage and the 3B model occasionally surpassing the 7B. This non-linear pattern appears on both the original prompt set and the new color-prompt dataset, using BERT-score and SentenceBERT to compute Mean Leak-Rate. All models leak above the 50% random baseline, and Qwen2.5 models appear more leak-prone than larger Llama and OpenAI models previously reported. The color dataset further shows that prompts requiring a non-color concept leak most on average, contrary to the hypothesis that color-to-color associations would leak more.","pith_inferences":["The 3B-versus-7B comparison may be confounded because the 7B model is a quantized variant; if quantization reduces leakage, the non-monotonic result might be an artifact.","The metric may conflate genuine associative leakage with a copying or repetition failure mode, as the paper's own observation about the 3B model suggests.","The color dataset's category imbalance (330/330/60) makes conclusions about the smallest category less reliable, which the author acknowledges.","A natural next test is to run the full-precision 7B model and models with different pretraining distributions to see whether the size trend is specific to Qwen2.5 or more general."],"forward_implications":["If size predicts leakage only loosely, then statements that equate larger models with more associative leakage should be qualified.","The 3B model's higher leakage is largely driven by its tendency to repeat the prompt's color concept in the generation, a behavior the other models do not exhibit.","On the color dataset, leakage is strongest when the expected generation is a non-color concept, not when both prompt and generation are colors.","Smaller models' lower leakage coincides with less context-sensitive and less diverse generations, suggesting leakage is tied to associative knowledge rather than being purely harmful.","Extending the same evaluation to Qwen2.5-14B, -32B, and -72B would test whether the size trend reverses or continues beyond 7B."],"supporting_citations":[{"why":"Defines semantic leakage, provides the original 109-prompt dataset and Mean Leak-Rate metric, and reports leakage scores for larger Llama and OpenAI models that this study compares against.","marker":"Gonen et al. (2024)"},{"why":"Supplies the Qwen2.5 model family and the exact checkpoints used in all experiments.","marker":"Qwen Team (2024)"},{"why":"Describes GPTQ quantization, the technique used to fit the 7B model on a single GPU.","marker":"Frantar et al. (2023)"},{"why":"Provides BERT-score, one of the two similarity measures used to compute Leak-Rate.","marker":"Zhang et al. (2020)"},{"why":"Provides SentenceBERT, the second similarity measure used to compute Leak-Rate.","marker":"Reimers and Gurevych (2019)"},{"why":"Defines the basic color terms used to construct the new color-prompt dataset.","marker":"Berlin and Kay (1991)"}],"fun_headline_variants":["Smaller LLMs leak less, but 3B outleaks 7B in Qwen2.5","Qwen2.5 leakage: 0.5B least, 3B can outleak 7B","Leakage not strictly size-driven: 3B outleaks 7B in Qwen2.5","Tiny LLMs leak less, but 3B beats 7B in Qwen2.5"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The study treats a quantized, compressed version of the 7B model as equivalent to the full-precision model in leakage behavior, so the size-comparison conclusions hinge on quantization not changing leakage.","fun_headline_variants_meta":{"raw":{"variants":["Smaller LLMs leak less, but 3B outleaks 7B in Qwen2.5","Qwen2.5 leakage: 0.5B least, 3B can outleak 7B","Leakage not strictly size-driven: 3B outleaks 7B in Qwen2.5","Tiny LLMs leak less, but 3B beats 7B in Qwen2.5"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000536,"raw_usage":{"total_tokens":2548,"prompt_tokens":891,"completion_tokens":1657,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":1543}},"tokens_in":507,"tokens_out":1657,"duration_ms":10213,"temperature":1.0,"reasoning_tokens":1543,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:55:36.778145+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same semantic-leakage prompts on the full-precision Qwen2.5-7B-Instruct with identical decoding settings and check whether its Mean Leak-Rate exceeds that of the 3B model; if it does, the non-monotonic size trend is an artifact of the quantized 7B checkpoint.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides BERT-score, one of the two similarity measures used to compute Leak-Rate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the basic color terms used to construct the new color-prompt dataset."}],"review_version":1}