{"id":"37d4605a-6e4e-405b-8525-a61c2ff55cb1","arxiv_id":"2606.13993","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Frequency and predictability shift the internal representation of “up” inside V+up phrases away from standalone “up” in text and audio models, a pattern interpreted as holistic storage.","lead":"Researchers tested whether language models store common verb + “up” phrases like “pick up” as whole chunks. They found that higher-frequency and more predictable phrases make the word “up” look less like standalone “up” inside the model, in both text and speech models, which they read as evidence for holistic phrase storage.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Probe may measure contextual predictability, not holistic storage; missing control invalidates core inference.","rationale":"The reader identified the logistic probe's validity as the weakest assumption, and I agree. The V+up frequency/predictability effects could be explained by the fact that the hidden state of 'up' is modulated by the preceding verb's predictiveness, independent of any holistic storage. This is a more fundamental challenge than the secondary issue that BabyLMs show inconsistent predictability effects (Experiment 2), because it threatens both the frequency and predictability conclusions across all models. The proposed test directly isolates contextual predictability by applying the same probe to standalone 'up' in varied predictability contexts; if the probe logit drops with predictability there, the central inference is invalid. Since the concern is addressable with a simple validation, the verdict remains CONDITIONAL: accept only if the probe is shown to be insensitive to context predictability (or the analysis controls for it). The reader's overstatement point about BabyLM predictability is also valid but secondary; it would only weaken part of the claim, not the core mechanism. Overall, the paper is valuable and methodologically clear, but the lack of a probe validation makes the conclusion premature without the suggested check.","tokens_in":15045,"tokens_out":6778,"duration_ms":69881,"concrete_test":"Compute, for held-out standalone 'up' occurrences in C4, the conditional log-probability of 'up' given the preceding token. Train the same logistic probe as in Experiment 1, then regress probe logit on this predictability measure. A significant negative relationship would indicate that the probe's drop in high-predictability V+up phrases is a context-predictability artifact, not holistic storage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that lower probe logits for high-frequency/predictability V+up phrases indicate holistic storage. This rests on the assumption that the logistic probe's logit is a valid measure of resemblance to standalone 'up' that is not confounded by contextual predictability. In a transformer, the hidden state of a token is a function of its context; when 'up' is strongly predicted by the preceding verb, the model may encode it with lower surprisal, causing its hidden state to be less similar to a standalone 'up' representation—even without any holistic storage. The random intercept for verb type controls for the verb's identity but not for the conditional probability of 'up' given that verb, which is exactly the predictability predictor. No control condition or validation is provided: e.g., the probe is never tested on standalone 'up' items in high- vs low-predictability contexts, and the authors do not show that the effect is specific to V+up phrases as opposed to any highly predictable 'up' occurrence. Additionally, the by-layer GAM assumes the classifier is equally calibrated across layers; if the logit scale shifts with layer, the layer trends may be artifacts.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether transformer language models and an ASR model store V+up phrasal verbs holistically. The authors train three BabyLM-size OPT models, and also use OLMo-3 7B and Whisper-small. They train logistic regression probes on embeddings of standalone 'up', then apply the probes to embeddings of 'up' in V+up phrases. Lower probe logits for high-frequency and high-predictability phrases are interpreted as evidence that these phrases have holistic representations that diverge from the compositional 'up' representation. Experiment 1 uses a probe trained only on standalone 'up'; Experiment 2 broadens positive training examples to 'up' as a subword; Experiment 3 applies the standalone-up probe to Whisper's encoder and decoder. The main reported result is negative frequency and predictability effects on probe logits, with by-layer analyses suggesting earlier emergence in larger models.","tokens_in":15325,"tokens_out":4814,"duration_ms":52474,"significance":"If the central inference were valid, this would be a useful extension of usage-based accounts of holistic storage to neural language models, including a cross-modal ASR result and models trained on human-comparable data. The paper has concrete strengths: the three BabyLM models are released, corpus counts and predictor definitions are explicit, Bayesian mixed-effects models provide credible intervals, and the item-level statistics are reported in appendices. However, the core claim depends on an unvalidated interpretive step: that a decrease in probe logit specifically indexes holistic storage rather than a generic contextual-predictability effect. Experiment 2 also contains predictability coefficients for the BabyLM models that are positive or non-significant, which is inconsistent with the abstract's across-the-board claim. These issues are load-bearing for the paper's main conclusion, so the manuscript needs substantive revision.","major_comments":[{"comment":"The inference from lower probe logits to 'holistic storage' is not uniquely identified. A transformer's representation of a token is a function of its context, and when 'up' is highly predictable given the preceding verb (Eqn. 2), the model may encode it with lower surprisal even if the phrase is not stored as a unit. The random intercept for verb_up in Eqn. (3) does not control for this, because log-predictability is exactly a property of the verb–up pair. The paper does not offer a control condition in which predictability is manipulated outside V+up contexts, nor any behavioral or generation-based validation that lower logits correspond to holistic phrase storage rather than contextual modulation. Without such a control, the central claim is underdetermined.","section":"§3.1.2 and Conclusion"},{"comment":"The abstract and conclusion state that all models show holistic storage driven by frequency and predictability, but the Experiment 2 final-layer predictability coefficients for the BabyLM models are absent or positive: BabyLM 125M has -0.09 (95% CI [-0.25, 0.06]), BabyLM 350M has +0.25 [0.11, 0.39], and BabyLM 1.3B has +0.14 [0.04, 0.24]. Positive coefficients mean high-predictability phrases yield higher 'up-like' logits, the opposite of the prediction. The text in §4.2 acknowledges 'absent or positive effects,' but the abstract and conclusion do not qualify the claim. This is a load-bearing inconsistency for the 'all models' claim and for the claim that predictability effects strengthen with scale.","section":"Experiment 2, Table 8"},{"comment":"The by-layer GAM comparisons assume the probe logit is comparable across layers, but the logistic probes are fit separately at each layer with no layer-wise calibration. If the scale or class separation of logits differs across layers, the tensor-product smooth over predictor and layer can produce spurious 'early versus late emergence' patterns. The conclusion that larger models show effects in earlier layers depends on this assumption. The paper should report per-layer classifier calibration or use a layer-invariant measure (e.g., rank-based or standardized scores) before interpreting layer trajectories.","section":"§3.1.2, Figures 2/4/6"}],"minor_comments":[{"comment":"The text reports that 1,426 V+up types were retained for Whisper, but Table 3 lists 1,724 items for both encoder and decoder. The text also gives a median frequency of 243,412, while Table 3 reports 169,679. These numerical discrepancies should be reconciled.","section":"§5.1.1 vs Table 3"},{"comment":"Typo: 'with a a random intercept' should read 'with a random intercept.'","section":"§3.1.2, Eqn. (4)"},{"comment":"The superscript '4' after 'preposition' appears to be a footnote marker, but no corresponding footnote text is present.","section":"§3.1.1"},{"comment":"Spacing issue: 'upembedded' should be 'up embedded.'","section":"§4.1.1"},{"comment":"The limitations section mentions construction, language, and checkpoint coverage, but should also acknowledge the probe-validity concern raised above, and note that no direct validation against behavioral or mechanistic criteria was performed.","section":"§7 Limitations"}],"recommendation":"major_revision","confidential_remarks":"Editor: The paper is potentially publishable after substantial revision. The main issue is that the probe-logit measure is the sole evidence for holistic storage, and the manuscript does not validate that measure against any independent criterion. The inconsistency between Experiment 2's positive predictability coefficients and the abstract's global claim also needs to be addressed head-on. I would also note that the manuscript leans heavily on the first author's prior work (Houghton 2025a,b; Houghton and Morgan 2023/2024; Houghton et al. 2024/2025; Pluth et al. 2026) for theoretical framing; some independent grounding would strengthen the contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The main thing you should know: this paper has a solid, reproducible empirical core — the first test of V+up representational divergence in an ASR model and in models trained on human-scale data. The frequency effect is large, consistent across BabyLM sizes, OLMo-3, and Whisper encoder/decoder, and reported with credible intervals. The methods are clearly described; the per-layer GAMs and the subword probe in Experiment 2 are thoughtful attempts to rule out alternative explanations like semantic bleaching. That is real value.\n\nThe soft spot is the interpretive leap from probe logits to 'holistic storage.' The probe is trained on standalone 'up' and tested on in-context 'up', but transformer hidden states are inherently contextual. High predictability of 'up' given the verb could cause its representation to diverge from standalone 'up' even without phrase-level storage — the model may simply encode the prediction differently. The random intercept for verb does not remove this confound, because predictability is precisely the conditional probability given the verb. A control with matched high-predictability contexts that are not V+up phrases, or a demonstration of V+up specificity, is missing. This does not sink the paper if the claim is reframed as 'contextual divergence consistent with usage-based accounts,' but the current wording overreaches.\n\nThe predictability story is also shakier than the abstract suggests. In Experiment 2, BabyLMs show absent or even positive predictability coefficients, so the claim that 'all models show evidence driven by frequency and predictability' is only true for the frequency half. The scale-dependent interpretation is plausible, but the abstract should not overstate it.\n\nMinor issues: corpus frequencies for OLMo and Whisper use Dolma v1.7 rather than the actual training distribution (v3 for OLMo), and the claimed model release has no artifacts or link in the paper. Self-citation is heavy, though the cited work is directly relevant.\n\nWho is this for? Usage-based linguists and people working on multiword representation in LMs will find the empirical pattern useful. The Whisper result is novel and worth knowing. I would accept this for peer review — the question is important, the frequency effect is solid, and the methods are transparent. But the authors need to add a control or substantially soften the 'holistic storage' interpretation before publication.","headline":"Robust frequency-driven probe divergence across LM scales and Whisper, but the 'holistic storage' interpretation is undercut by a missing contextual control; predictability effects are inconsistent with the abstract's blanket claim.","tokens_in":15795,"tokens_out":2991,"would_cite":true,"duration_ms":36696,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Language models store frequent verb+up phrases as whole units, not word-by-word.","keywords":["holistic storage","multi-word units","phrasal verbs","language models","speech recognition","probing classifiers","frequency effects","predictability"],"falsifier":"If the same frequency-driven logit drop appears for a control word that cannot form a stored phrase—for example, 'up' in a frequent bigram that is not a verb+particle construction, or for a particle with identical frequency but no phrasal predictability—then the effect would be contextual rather than holistic. A more direct test would ask whether the divergence predicts behavior: if models are no faster or more accurate at using high-frequency verb+up phrases than compositional controls, the representational difference is not storage in any processing sense.","tokens_in":14979,"feed_emoji":"","tokens_out":8197,"duration_ms":79368,"temperature":0.7,"pith_summary":"The paper asks whether language models store common multi-word phrases as single units rather than composing them from parts each time, the way usage-based linguistics says humans do. Using a probe that learns what the word 'up' looks like inside a model, it measures how much 'up' drifts when it appears inside verb+up phrases that vary in frequency and predictability. Across four text-based models of different sizes and in the encoder and decoder of an audio speech-recognition model, high-frequency and high-predictability phrases show the largest drift. The same pattern appears even when the probe is trained to recognize 'up' as a subword rather than only as a preposition, and it appears in models trained on a human-comparable amount of text. The authors read this as evidence that holistic storage emerges naturally from distributional learning, with no dedicated mechanism required.","feed_headline":"Frequent verb+up phrases get stored whole in language models","feed_subtitle":"Frequent, predictable verb phrases make 'up' drift from its standalone form, a sign common phrases are stored whole.","key_machinery":"The central instrument is a per-layer logistic probe trained to distinguish hidden-layer embeddings of standalone 'up' from other tokens in the same sentence. Applied to held-out verb+up phrases, the probe's logit measures how similar that phrase's 'up' representation is to standalone 'up'; a lower logit is taken as evidence that the phrase has been stored holistically. The paper analyzes these logits with Bayesian mixed-effects regressions and generalized additive models over frequency, predictability, and layer index.","core_discovery":"The paper's central claim is that all tested models—three transformer language models trained from scratch on a 150-million-token corpus, a 7-billion-parameter text model, and a small encoder-decoder speech recognizer—show a frequency- and predictability-driven divergence: the internal representation of 'up' inside a verb+up phrase becomes progressively less like the representation of standalone 'up' as the phrase's frequency and predictability rise. The authors interpret this divergence as holistic storage of the phrase, paralleling human listening studies in which participants detect 'up' more slowly in high-frequency phrases. They also find the divergence deepens over layers, appears earl","pith_inferences":["The probe-logit gradient is never validated against behavior, so a natural test would pair it with generation or recognition tasks: if high-frequency verb+up phrases are also produced faster or recognized more accurately, the representational divergence would be tied to a real processing advantage.","Because larger models likely also saw more data, a controlled experiment training one architecture at multiple scales on identical token budgets would isolate whether exposure or capacity drives the predictability effect.","The same protocol could be applied to languages with different particle systems, such as German separable prefixes or Chinese resultative compounds, to test whether frequency-driven holistic storage is a general property of usage rather than an artifact of English orthography.","The layer-by-layer pattern suggests a mechanistic prediction: interventions on later layers should selectively damage high-frequency phrase representations while leaving low-frequency compositional phrases intact, giving a causal handle on where holistic units live."],"forward_implications":["Holistic storage is a byproduct of exposure to distributional patterns; no separate memory or rule mechanism is needed to explain it.","Human-scale training data, roughly half the tokens a college-aged person has experienced, is enough for frequency-driven holistic storage to emerge.","Sensitivity to predictability, rather than raw frequency alone, strengthens with model size, suggesting that encoding conditional statistics requires greater representational capacity.","The effect generalizes across modalities: text-based models and the encoder and decoder of a speech recognizer show the same gradient, so holistic storage is not tied to written tokenization.","Storage is a graded, layer-dependent phenomenon rather than an all-or-nothing property: divergence from standalone 'up' accumulates across layers and appears earlier in larger models."],"fun_headline_variants":["Frequency makes models store 'up' as part of a phrase","LLMs store common verb+up phrases as whole chunks","Predictable verb+up phrases get holistic storage in AI","Models show usage-based storage: frequent 'up' phrases go holistic","Common 'up' phrases drift from standalone meaning in LLMs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a lower probe score for 'up' inside a frequent phrase means the whole phrase is stored, rather than that the word's meaning or sound is simply adjusted by context; if contextual modulation can produce the same drop, the holistic-storage reading loses its footing.","fun_headline_variants_meta":{"raw":{"variants":["Frequency makes models store 'up' as part of a phrase","LLMs store common verb+up phrases as whole chunks","Predictable verb+up phrases get holistic storage in AI","Models show usage-based storage: frequent 'up' phrases go holistic","Common 'up' phrases drift from standalone meaning in LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000179,"raw_usage":{"total_tokens":1068,"prompt_tokens":610,"completion_tokens":458,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":354,"completion_tokens_details":{"reasoning_tokens":387}},"tokens_in":354,"tokens_out":458,"duration_ms":5317,"temperature":1.0,"reasoning_tokens":387,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T04:43:21.721684+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If the same frequency-driven logit drop appears for a control word that cannot form a stored phrase—for example, 'up' in a frequent bigram that is not a verb+particle construction, or for a particle with identical frequency but no phrasal predictability—then the effect would be contextual rather than holistic. A more direct test would ask whether the divergence predicts behavior: if models are no faster or more accurate at using high-frequency verb+up phrases than compositional controls, the representational difference is not storage in any processing sense.","supporting_citations":[],"review_version":1}