Pith. sign in

REVIEW 3 major objections 6 minor 4 cited by

This paper argues that symbol grounding in language models emerges as a localized, causal mechanism: middle-layer 'aggregate' attention heads carry information from environmental tokens to linguistic predictions, and ablating them breaks th

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 09:40 UTC pith:OLJAIBVA

load-bearing objection Solid, well-scoped mechanistic study of grounding in toy LMs, but the main behavioral measure is partly confounded by fixed template position and the co-occurrence control does not rule out a positional n-gram strategy. the 3 major comments →

arxiv 2510.13796 v3 pith:OLJAIBVA submitted 2025-10-15 cs.CL cs.CV

The Mechanistic Emergence of Symbol Grounding in Language Models

Shuyu Wu , Ziqiao Ma , Xiaoxi Luo , Yidong Huang , Josue Torres-Fonseca , Freda Shi , Joyce Chai This is my paper
classification cs.CL cs.CV
keywords symbol groundingmechanistic interpretabilityattention headsgather-and-aggregateemergent abilitiescausal interventionvision-language modelsstate-space models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper aims to show that symbol grounding—the link between a linguistic word and the environmental situation it refers to—is an emergent, mechanistically local property of autoregressive language models, not a byproduct of token co-occurrence alone. Training Transformers, Mamba-2, and unidirectional LSTMs from scratch on child-directed speech and dialogue data, with environmental and linguistic tokens kept as distinct surface forms, the authors measure a grounding information gain: matched environmental context lowers surprisal for the corresponding word, while mismatched context does not. Saliency and layer-wise probing place this effect in middle layers, and causal ablation shows that zeroing 'aggregate' attention heads—heads that route information from the environmental token to the position predicting the word—raises surprisal relative to random-head controls. The pattern repeats in caption-grounded and image-grounded dialogue and in a large vision-language model, but is absent in LSTMs. A sympathetic reader would care because the result identifies a specific, editable circuit that carries meaning, with implications for detecting and controlling when a model is actually grounded.

Core claim

The paper's central claim is that symbol grounding in autoregressive LMs is implemented by a sparse set of middle-layer attention heads that act as an 'aggregate' mechanism: after shallow-layer 'gather' heads compress contextual evidence into environmental tokens, mid-layer aggregate heads broadcast that environmental token to the position immediately preceding the linguistic form, making the grounded word predictable. The evidence is threefold. Behaviorally, grounding information gain Gθ(v), the expected log-likelihood ratio of matched versus mismatched environmental context, rises over training in Transformers and Mamba-2 and stays flat in unidirectional LSTMs. Mechanistically, gradient-we

What carries the argument

The load-bearing object is the gather-and-aggregate attention-head circuit. Gather heads are shallow-layer heads whose saliency (gradient × attention contribution to the loss) flows from prior tokens into the environmental ground token; aggregate heads are middle-layer heads whose saliency flows from that environmental token to the token immediately preceding the linguistic target. The paper defines an aggregate head operationally (≥30% of saliency from the environmental ground to the predicting position) and validates it causally by zeroing the head's output and comparing surprisal against random-head controls. The behavioral measure that ties the circuit to grounding is the grounding infor

Load-bearing premise

The load-bearing premise is that the contrastive surprisal gap measures a general mapping between environmental evidence and linguistic form, rather than a template-specific positional regularity: in every test template the environmental filler is the last environmental token before the linguistic prompt, and the co-occurrence control counts 512-token chunks, so a purely local completion rule could in principle reproduce the reported gap without any general grounding mechanis

What would settle it

A concrete check: re-run the evaluation with the environmental filler moved earlier in the template, separated from the linguistic prompt by several unrelated tokens. If the grounding information gain Gθ(v) collapses to zero or to chance under this rearrangement, the reported effect is a template-completion artifact rather than a general environmental-to-linguistic mapping; if the gain survives, the grounding claim is robust. A complementary check is to test a simple n-gram or position-aware baseline on the same templates and compare its matched-versus-mismatched surprisal gap to the models'.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Symbol grounding can emerge from next-token prediction alone in architectures that support content-addressable retrieval; unidirectional LSTMs, which compress sequence state without such retrieval, do not ground.
  • Grounding is causally attributable to a sparse set of middle-layer aggregate heads, so the behavior can be turned on and off by targeted head interventions.
  • The mechanism is modality-general: the same aggregate pattern appears whether the ground is child-directed speech annotations, captions, or image-patch embeddings.
  • Aggregate heads multiply and move deeper over training, so grounding is a gradual developmental outcome rather than a sudden capability.
  • Monitoring aggregate-head saliency before generation could serve as a reliability signal for grounded production, a direct corollary of the causality evidence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the account is to vary the position of the environmental filler relative to the prompt; the current templates always place it as the last environmental token, so if the surprisal gap depends on that adjacency, the mechanism may be a positional completion rule rather than the general word–referent mapping claimed. This is our inference, not a paper claim.
  • If aggregate heads are the causal locus, decoding-time steering of those heads should selectively increase or decrease grounded prediction, offering a concrete extension beyond the paper's observational monitoring discussion.
  • The LSTM null result suggests a sharper architectural boundary test: memory-augmented recurrent models that add content-addressable retrieval should recover grounding if the paper's retrieval-based explanation is correct.
  • Since the paper's co-occurrence control uses 512-token chunks, a stricter control conditioning on exact template position would tell whether the reported divergence from co-occurrence statistics survives at the local level.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a minimal controlled testbed in which each noun is split into an environmental token (⟨ENV⟩) and a distinct linguistic token (⟨LAN⟩), trains small Transformers, Mamba-2, and LSTM models from scratch on CHILDES-style data and on Visual Dialog data, and measures a grounding information gain Gθ(v), defined as the mean surprisal difference between matched and mismatched environmental contexts. The paper reports that Transformers and Mamba-2 exhibit a stable match/mismatch surprisal gap, that this gap is not fully explained by chunk-level co-occurrence statistics, and that it is absent in unidirectional LSTMs. Mechanistically, saliency and tuned-lens analyses localize the effect to middle layers, and causal head-ablation experiments are used to argue that aggregate attention heads—heads that route environmental-token information to the linguistic prediction position—implement the grounding mechanism. The findings are extended to a VLM setting with DINOv2 image tokens and to a large-scale LLaVA case study.

Significance. If the main claims hold, the paper would provide a tractable model of how symbol grounding can emerge mechanistically in autoregressive language models, with concrete implications for interpreting and potentially controlling generation reliability. The multi-seed, multi-architecture design, the use of distinct token forms to prevent trivial token-identity shortcuts, the causal intervention protocol, and the availability of code are clear strengths. However, the behavioral and mechanistic evidence is currently not strong enough to support the strongest interpretation: the central co-occurrence control does not exclude a position-specific template-memorization strategy, and the causal head selection is performed on the same contexts used for evaluation, which weakens the causal attribution. These issues are fixable with additional controls, but they are load-bearing for the paper's central claims.

major comments (3)
  1. [§4.2, Eq. (2)] The 'beyond co-occurrence' control does not exclude the most natural shallow alternative. In the evaluation templates, the filler ENV token always occupies the same syntactic slot—immediately before the role token that introduces the fixed LAN prompt—and the target LAN token is predicted at a fixed later position. A model can therefore produce the match/mismatch surprisal gap by a purely positional rule, e.g., 'if the token immediately before the prompt is X_ENV, the prompt-final word is X_LAN', without acquiring a general ground-to-symbol mapping. The §4.2 control counts co-occurrences inside 512-token chunks, which is insensitive to this exact-position regularity; a bigram/trigram model could have low R² against the chunk-level statistic while still reproducing the Gθ gap. Please add an explicit n-gram baseline, or compare with ENV fillers inserted at varying syntactic positions, and r
  2. [§5.3, Table 2] The causal intervention is not fully controlled for selection bias. Aggregate heads are selected by ENV→LAN saliency on the same contexts where the intervention effect is measured, and the control is an equal number of random heads per layer. Because the selected heads have high saliency and therefore generally contribute more to the loss, the intervention effect may reflect overall head importance rather than the specific aggregate pathway. A stronger control would compare against equally salient heads that do not carry ENV→LAN flow, or would select heads on one subset of templates and evaluate on held-out templates. Without such a control, the causal claim that these heads specifically implement grounding is not established.
  3. [§4.1, Figure 2d] The LSTM negative result is presented as an architectural boundary, but the comparison is not matched for capacity, optimization, or residual structure. LSTMs are trained with the same hidden size and steps but have a fundamentally different parameterization and are generally harder to optimize; the lack of a match/mismatch gap could reflect underfitting rather than the absence of content-addressable retrieval. A stronger test would include an LSTM with residual connections or a matched-parameter comparison, and would verify that the LSTM reaches comparable training perplexity to the Transformer and Mamba models before attributing the negative result to an architectural limitation.
minor comments (6)
  1. [Figures 2–4, 9–10] The paper states that all experiments are repeated with five seeds, but the figures show only point curves with no error bars or confidence intervals. Adding variance shading or per-seed ranges would make the consistency across seeds visible.
  2. [§4.2, Figure 3] The definition of the reported R² statistic is incomplete. Please state what is regressed on what (e.g., log co-occurrence counts of words against their Gθ values), whether R² is averaged over seeds, and how the significance of the regression is assessed.
  3. [§5.3, Table 2] The significance testing is not described. Please specify the test used (e.g., paired permutation test), the unit of observation (contexts, template instances, or seeds), and how the control and intervention conditions were paired.
  4. [Eq. (2) and §3.2] The blank/prediction position is not visibly marked in Eq. (2) or Eq. (1). Please make explicit, perhaps with an underline or a placeholder token, where the target ⟨LAN⟩ token is predicted, as this is central to the evaluation protocol.
  5. [Appendix A.1] There is a typo ('genrated') in the CHILDES prompt description. Also, in the visual dialogue caption templates, clarify whether the [FILLER] in the ENV segment is the environmental token and whether the predicted token is the same word's LAN form after the ⟨A⟩ token.
  6. [§5.3–5.4] The gather/aggregate head classifications rely on arbitrary thresholds (30% and 70%/90%). A sensitivity analysis over these thresholds would help confirm that the reported causal effects are not an artifact of the specific cutoff.

Circularity Check

0 steps flagged

No significant circularity: Gθ is a directly measured log-likelihood ratio and the mechanistic claims rest on causal interventions with random-head controls.

full rationale

The central behavioral measure Gθ(v) is defined as an expected log-likelihood ratio between matched and mismatched contexts, and the paper does not force this ratio to be positive by construction—a model that failed to learn any environmental-to-linguistic mapping would yield Gθ≈0. Reporting that Gθ increases during training is an empirical observation, not a restatement of the definition. The mechanistic analysis selects gather/aggregate heads using gradient-based saliency and then tests them with zeroing-out interventions against random-head controls; although the saliency and the ablation both use the same loss, the intervention result is not logically entailed by the selection criterion, and the random-head control gives the finding independent content. The §4.2 co-occurrence control is computed at the 512-token chunk level and does not exactly match the local positional statistic inherent in the test templates; this is a validity limitation of the control rather than a circular reduction of the paper's own equations. There is no load-bearing self-citation chain: the key mechanistic references (Bick et al. 2025, Wang et al. 2023, Belrose et al. 2023) are external, and the paper invokes no uniqueness theorem or imported ansatz that pre-determines its conclusions. Overall, the derivation chain is self-contained and empirically testable; the main risks are construct validity and confound control, not circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claims rest on the operationalization of grounding as a surprisal gap on template contexts, on standard interpretability approximations (gradient-weighted saliency, tuned lens), and on a head-ablation causal framework; the only hand-fitted quantities are the head-identification thresholds and the evaluation/template sizes. No new entities are posited.

free parameters (4)
  • saliency threshold for gather/aggregate head identification (text) = 30%
    Heads are classified as gather/aggregate if ≥30% of saliency matches the defined flow; this threshold is chosen by the authors and changes the set of heads (§5.3, Table 2).
  • saliency threshold for aggregate head identification (VLM) = 70% and 90%
    Two thresholds are used in the image-grounded setting; results differ between them (§5.4, Figure 7).
  • evaluation vocabulary size and template count = V=100 nouns; N=10 templates per word
    The grounding metric averages over 100 target words and 10 hand/generated prompts per word; the choice of these supports the statistical power but is arbitrary (Appendix A).
  • co-occurrence chunk size = 512 tokens
    The correlation between Gθ and co-occurrence is computed over 512-token training chunks; the value affects R² but is a design choice (§4.2).
axioms (5)
  • domain assumption Contrastive surprisal (Gθ) is a valid measure of symbol grounding; a positive Gθ indicates the model grounds symbols in environments.
    The central behavioral evidence (§4.1) relies on this equivalence between a next-token surprisal gap and grounding; it is an operationalization rather than a proven equivalence.
  • domain assumption CHILDES-style environmental annotations and MSCOCO captions/images are adequate proxies for sensorimotor experience in studying grounding.
    The testbed (§3.1) replaces real perceptual experience with text/ViT-token contexts; whether this captures Harnad-style grounding is assumed.
  • domain assumption Gradient-weighted attention saliency (Wang et al., 2023) faithfully measures causal information flow.
    The head identification (§5.1, §5.3) uses Iℓ = |Σ_h A_h ⊙ ∂L/∂A_h|; this is standard but only an approximation of influence.
  • domain assumption Zeroing out attention-head outputs is a valid causal intervention that leaves other model behavior intact.
    The causal claims (§5.3, Table 2) depend on head-ablation preserving the model's overall function except for the targeted mechanism.
  • domain assumption The tuned lens affine projectors faithfully map intermediate representations to the output space.
    Layer-wise surprisal conclusions (§5.1, Figure 5b) rely on the tuned lens approximation from Belrose et al. (2023).

pith-pipeline@v1.3.0-alltime-deepseek · 19321 in / 14024 out tokens · 113509 ms · 2026-08-04T09:40:32.848834+00:00 · methodology

0 comments
read the original abstract

Symbol grounding (Harnad, 1990) describes how symbols such as words acquire their meanings by connecting to real-world sensorimotor experiences. Recent work has shown preliminary evidence that grounding may emerge in (vision-)language models trained at scale without using explicit grounding objectives. Yet, the specific loci of this emergence and the mechanisms that drive it remain largely unexplored. To address this problem, we introduce a controlled evaluation framework that systematically traces how symbol grounding arises within the internal computations through mechanistic and causal analysis. Our findings show that grounding concentrates in middle-layer computations and is implemented through the aggregate mechanism, where attention heads aggregate the environmental ground to support the prediction of linguistic forms. This phenomenon replicates in multimodal dialogue and across architectures (Transformers and state-space models), but not in unidirectional LSTMs. Our results provide behavioral and mechanistic evidence that symbol grounding can emerge in language models, with practical implications for predicting and potentially controlling the reliability of generation.

Figures

Figures reproduced from arXiv: 2510.13796 by Freda Shi, Josue Torres-Fonseca, Joyce Chai, Shuyu Wu, Xiaoxi Luo, Yidong Huang, Ziqiao Ma.

Figure 1
Figure 1. Figure 1: Illustration of the symbol grounding mechanism through information aggregation. Lighter [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Average surprisal of the experimental and control conditions over training steps. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Grounding information gain and its correlation to the co-occurrence of linguistic and [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Average surprisal of the experimental and control conditions in caption- and image-grounded [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Overtime mechanistic analysis on GPT-CHILDES. Saliency flow. For each layer ℓ, we compute a saliency matrix fol￾lowing Wang et al. (2023): Iℓ = [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Examples of gather and aggregate heads identified in GPT-CHILDES. L: layer; H: head. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Mechanistic analysis in the image-grounded visual dialogue setting. Left: Causal interven [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Average surprisal of the experimental and control conditions over training steps. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Grounding information gain and its correlation to the co-occurrence of linguistic and [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 8
Figure 8. Figure 8: Gather-and-aggregate overtime. We show the complete behavioral evidence for all models in [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. EasyLens: A Training-Free Plug-and-Play Subtle-Lesion Representation Amplifier for Medical Vision-Language Models

    cs.CV 2026-06 conditional novelty 7.0

    Prior-minimal multi-agent RL agents develop indexical encoding, persistent self-state, and an echo-mismatch self-monitoring circuit that vanishes when the echo affordance is removed during training.

  2. Input Pathways Shape Few-Shot, Not Zero-Shot, Binding in Tiny Transformers: A Fully-Enumerable Study

    cs.LG 2026-07 accept novelty 6.0

    In information-matched tiny transformers, zero-shot compositional binding fails for every route, while few-shot efficiency is governed by input-pathway sharing and code readability.

  3. EasyLens: A Training-Free Plug-and-Play Subtle-Lesion Representation Amplifier for Medical Vision-Language Models

    cs.CV 2026-06 unverdicted novelty 6.0

    EasyLens introduces a plug-and-play amplifier that uses pathology-anatomy prototypes and morphology-guided residual enhancement to boost subtle-lesion cues in frozen medical VLMs.

  4. Emergent Language as an Approach to Conscious AI

    cs.CL 2026-06 unverdicted novelty 4.0

    Agents in a minimal multi-agent RL setup develop self-referential communication and an echo-mismatch detection circuit that emerges from environmental affordances rather than task structure or architecture.

Reference graph

Works this paper leans on

83 extracted references · 6 linked inside Pith · cited by 3 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    The claude 3 model family: Opus, sonnet, haiku, March 2024

    Anthropic. The claude 3 model family: Opus, sonnet, haiku, March 2024. URL https://www.anthropic.com/news/claude-3-family

    2024

  3. [3]

    Mechanistic evaluation of transformers and state space models

    Aryaman Arora, Neil Rathi, Nikil Roashan Selvam, Róbert Csórdas, Dan Jurafsky, and Christopher Potts. Mechanistic evaluation of transformers and state space models. arXiv preprint arXiv:2505.15105, 2025

    arXiv 2025

  4. [4]

    Eliciting latent predictions from transformers with the tuned lens

    Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Xing, and Albert Gu

    Aviv Bick, Eric P. Xing, and Albert Gu. Understanding the skill gap in recurrent models: The role of the gather-and-aggregate mechanism. In Forty-second International Conference on Machine Learning, 2025

    2025

  6. [6]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp.\ 2397--2430. PMLR, 2023

    2023

  7. [7]

    Birth of a transformer: A memory viewpoint

    Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of a transformer: A memory viewpoint. Advances in Neural Information Processing Systems, 2023

    2023

  8. [8]

    Analyzing the mono-and cross-lingual pretraining dynamics of multilingual language models

    Terra Blevins, Hila Gonen, and Luke Zettlemoyer. Analyzing the mono-and cross-lingual pretraining dynamics of multilingual language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.\ 3575--3590, 2022

    2022

  9. [9]

    Grounding everything: Emerging localization properties in vision-language transformers

    Walid Bousselham, Felix Petersen, Vittorio Ferrari, and Hilde Kuehne. Grounding everything: Emerging localization properties in vision-language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 3828--3837, 2024

    2024

  10. [10]

    Emerging pixel grounding in large multimodal models without grounding supervision

    Shengcao Cao, Liang-Yan Gui, and Yu-Xiong Wang. Emerging pixel grounding in large multimodal models without grounding supervision. In International Conference on Machine Learning, 2025

    2025

  11. [11]

    Word acquisition in neural language models

    Tyler A Chang and Benjamin K Bergen. Word acquisition in neural language models. Transactions of the Association for Computational Linguistics, 10: 0 1--16, 2022

    2022

  12. [12]

    Characterizing learning curves during language model pre-training: Learning, forgetting, and stability

    Tyler A Chang, Zhuowen Tu, and Benjamin K Bergen. Characterizing learning curves during language model pre-training: Learning, forgetting, and stability. Transactions of the Association for Computational Linguistics, 12: 0 1346--1362, 2024

    2024

  13. [13]

    Revisiting referring expression comprehension evaluation in the era of large multimodal models

    Jierun Chen, Fangyun Wei, Jinjing Zhao, Sizhe Song, Bohuai Wu, Zhuoxuan Peng, S-H Gary Chan, and Hongyang Zhang. Revisiting referring expression comprehension evaluation in the era of large multimodal models. arXiv preprint arXiv:2406.16866, 2024 a

    Pith/arXiv arXiv 2024

  14. [14]

    Shikra: Unleashing multimodal llm's referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm's referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Multi-object hallucination in vision language models

    Xuweiyi Chen, Ziqiao Ma, Xuejun Zhang, Sihan Xu, Shengyi Qian, Jianing Yang, David Fouhey, and Joyce Chai. Multi-object hallucination in vision language models. Advances in Neural Information Processing Systems, 37: 0 44393--44418, 2024 b

    2024

  16. [16]

    The lexicon in acquisition

    Eve V Clark. The lexicon in acquisition. Number 65. Cambridge University Press, 1995

    1995

  17. [18]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning, pp.\ 10041--10071. PMLR, 2024

    2024

  18. [19]

    Vision transformers need registers

    Timoth \'e e Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024

    2024

  19. [20]

    Visual dialog

    Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jos \'e MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 326--335, 2017

    2017

  20. [21]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020

    2020

  21. [22]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A...

    2021

  22. [23]

    Linnea Evanson, Yair Lakretz, and Jean-R \'e mi King. Language acquisition: do children and language models follow similar learning stages? In Findings of the Association for Computational Linguistics: ACL 2023, pp.\ 12205--12218, 2023

    2023

  23. [24]

    A probabilistic computational model of cross-situational word learning

    Afsaneh Fazly, Afra Alishahi, and Suzanne Stevenson. A probabilistic computational model of cross-situational word learning. Cognitive Science, 34 0 (6): 0 1017--1063, 2010

    2010

  24. [25]

    Macarthur-bates communicative development inventories

    Larry Fenson, Virginia A Marchman, Donna J Thal, Phillip S Dale, J Steven Reznick, and Elizabeth Bates. Macarthur-bates communicative development inventories. PsycTESTS Dataset, 2006

    2006

  25. [26]

    The acquisition of the lexicon

    Lila R Gleitman and Barbara Landau. The acquisition of the lexicon. MIT Press, 1994

    1994

  26. [27]

    A bayesian framework for cross-situational word-learning

    Noah Goodman, Joshua Tenenbaum, and Michael Black. A bayesian framework for cross-situational word-learning. Advances in neural information processing systems, 20, 2007

    2007

  27. [28]

    Pragmatic norms are all you need--why the symbol grounding problem does not apply to llms

    Reto Gubelmann. Pragmatic norms are all you need--why the symbol grounding problem does not apply to llms. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 11663--11678, 2024

    2024

  28. [29]

    Machine psychology: Investigating emergent capabilities and behavior in large language models using psychological methods

    Thilo Hagendorff. Machine psychology: Investigating emergent capabilities and behavior in large language models using psychological methods. arXiv preprint arXiv:2303.13988, 2023

    Pith/arXiv arXiv 2023

  29. [30]

    The symbol grounding problem

    Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42 0 (1-3): 0 335--346, 1990

    1990

  30. [31]

    Long short-term memory

    Sepp Hochreiter and J \"u rgen Schmidhuber. Long short-term memory. Neural computation, 9 0 (8): 0 1735--1780, 1997

    1997

  31. [32]

    Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation

    Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 13418--13427, 2024

    2024

  32. [33]

    Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens

    Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 25004--25014, 2025

    2025

  33. [34]

    Hidden breakthroughs in language model training

    Sara Kangaslahti, Elan Rosenfeld, and Naomi Saphra. Hidden breakthroughs in language model training. arXiv preprint arXiv:2506.15872, 2025

    arXiv 2025

  34. [35]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 10965--10975, 2022

    2022

  35. [36]

    Does circuit analysis interpretability scale? E vidence from multiple choice capabilities in chinchilla

    Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? E vidence from multiple choice capabilities in chinchilla. arXiv preprint arXiv:2307.09458, 2023

    Pith/arXiv arXiv 2023

  36. [37]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp.\ 740--755. Springer, 2014

    2014

  37. [38]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in neural information processing systems, volume 36, pp.\ 34892--34916, 2023

    2023

  38. [39]

    Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, and Iryna Gurevych. Are emergent abilities in large language models just in-context learning? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 5098--5139, 2024

    2024

  39. [40]

    World-to-words: Grounded open vocabulary acquisition through fast mapping in vision-language models

    Ziqiao Ma, Jiayi Pan, and Joyce Chai. World-to-words: Grounded open vocabulary acquisition through fast mapping in vision-language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 524--544, 2023

    2023

  40. [41]

    Babysit a language model from scratch: Interactive language learning by trials and demonstrations

    Ziqiao Ma, Zekun Wang, and Joyce Chai. Babysit a language model from scratch: Interactive language learning by trials and demonstrations. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp.\ 991--1010, 2025

    2025

  41. [42]

    The childes project: Tools for analyzing talk: Volume i: Transcription format and programs, volume ii: The database, 2000

    Brian MacWhinney. The childes project: Tools for analyzing talk: Volume i: Transcription format and programs, volume ii: The database, 2000

    2000

  42. [43]

    Tenenbaum, and Jiajun Wu

    Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The neuro-symbolic concept learner: Interpreting scenes, words, sentences from natural supervision. International Conference on Learning Representations (ICLR), 2019

    2019

  43. [44]

    Shi, Jiajun Wu, Roger P

    Jiayuan Mao, Freda H. Shi, Jiajun Wu, Roger P. Levy, and Joshua B. Tenenbaum. Grammar-based grounded lexicon learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021

    2021

  44. [45]

    Locating and editing factual associations in GPT

    Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT . In Advances in Neural Information Processing Systems, 2022

    2022

  45. [46]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, a...

    2022

  46. [47]

    Hello gpt-4o, May 2024

    OpenAI. Hello gpt-4o, May 2024. URL https://openai.com/index/hello-gpt-4o/

    2024

  47. [48]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth \'e e Darcet, Th \'e o Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, pp.\ 1--31, 2024

    2024

  48. [49]

    Grounding multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Qixiang Ye, and Furu Wei. Grounding multimodal large language models to the world. In The Twelfth International Conference on Learning Representations, 2024

    2024

  49. [50]

    Grounded situation recognition

    Sarah Pratt, Mark Yatskar, Luca Weihs, Ali Farhadi, and Aniruddha Kembhavi. Grounded situation recognition. In European Conference on Computer Vision, pp.\ 314--332. Springer, 2020

    2020

  50. [51]

    Context-based word acquisition for situated dialogue in a virtual world

    Shaolin Qu and Joyce Yue Chai. Context-based word acquisition for situated dialogue in a virtual world. Journal of Artificial Intelligence Research, 37: 0 247--277, 2010

    2010

  51. [52]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  52. [53]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PmLR, 2021

    2021

  53. [54]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    2024

  54. [55]

    The emergence of words: Attentional learning in form and meaning

    Terry Regier. The emergence of words: Attentional learning in form and meaning. Cognitive science, 29 0 (6): 0 819--865, 2005

    2005

  55. [56]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \"o rn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 10684--10695, 2022

    2022

  56. [57]

    Learning words from sights and sounds: A computational model

    Deb K Roy and Alex P Pentland. Learning words from sights and sounds: A computational model. Cognitive science, 26 0 (1): 0 113--146, 2002

    2002

  57. [58]

    Simalign: High quality word alignments without parallel training data using static and contextualized embeddings

    Masoud Jalili Sabet, Philipp Dufter, Fran c ois Yvon, and Hinrich Sch \"u tze. Simalign: High quality word alignments without parallel training data using static and contextualized embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2020

    2020

  58. [59]

    Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36, 2023

    Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36, 2023

    2023

  59. [60]

    It's a (blind) match! T owards vision-language correspondence without parallel data

    Dominik Schnaus, Nikita Araslanov, and Daniel Cremers. It's a (blind) match! T owards vision-language correspondence without parallel data. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.\ 24983--24992, 2025

    2025

  60. [61]

    The multiberts: Bert reproductions for robustness analysis

    Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander D'Amour, Tal Linzen, Jasmijn Bastings, Iulia Raluca Turc, Jacob Eisenstein, et al. The multiberts: Bert reproductions for robustness analysis. In International Conference on Learning Representations, 2021

    2021

  61. [62]

    Haoyue Shi, Luke Zettlemoyer, and Sida I. Wang. Bilingual lexicon induction via unsupervised bitext construction and word alignment. In ACL, 2021

    2021

  62. [63]

    A computational study of cross-situational techniques for learning word-to-meaning mappings

    Jeffrey Mark Siskind. A computational study of cross-situational techniques for learning word-to-meaning mappings. Cognition, 61 0 (1-2): 0 39--91, 1996

    1996

  63. [64]

    Polypythias: Stability and outliers across fifty language model pre-training runs

    Oskar van der Wal, Pietro Lesci, Max M \"u ller-Eberstein, Naomi Saphra, Hailey Schoelkopf, Willem Zuidema, and Stella Biderman. Polypythias: Stability and outliers across fifty language model pre-training runs. In Proceedings of the Thirteenth International Conference on Learning Representations (ICLR 2025), pp.\ 1--25, 2025

    2025

  64. [65]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  65. [66]

    Label words are anchors: An information flow perspective for understanding in-context learning

    Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 9840--9855, 2023

    2023

  66. [67]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems, 37: 0 121475--121499, 2024

    2024

  67. [68]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022

    2022

  68. [69]

    Answer, assemble, ace: Understanding how LM s answer multiple choice questions

    Sarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannaneh Hajishirzi, and Ashish Sabharwal. Answer, assemble, ace: Understanding how LM s answer multiple choice questions. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  69. [70]

    Retrieval head mechanistically explains long-context factuality

    Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long-context factuality. In The Thirteenth International Conference on Learning Representations, 2025 a

    2025

  70. [71]

    The semantic hub hypothesis: Language models share semantic representations across languages and modalities

    Zhaofeng Wu, Dani Yogatama, Jiasen Lu, and Yoon Kim. The semantic hub hypothesis: Language models share semantic representations across languages and modalities. In ICML, 2025 b

    2025

  71. [72]

    Training trajectories of language models across scales

    Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. Training trajectories of language models across scales. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 13711--13738, 2023

    2023

  72. [73]

    Gsva: Generalized segmentation via multimodal large language models

    Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    2024

  73. [74]

    Word learning as bayesian inference

    Fei Xu and Joshua B Tenenbaum. Word learning as bayesian inference. Psychological review, 114 0 (2): 0 245, 2007

    2007

  74. [75]

    Ferret: Refer and ground anything anywhere at any granularity

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. In The Twelfth International Conference on Learning Representations, 2024

    2024

  75. [76]

    The emergence of links between lexical acquisition and object categorization: A computational study

    Chen Yu. The emergence of links between lexical acquisition and object categorization: A computational study. Connection science, 17 0 (3-4): 0 381--397, 2005

    2005

  76. [77]

    A unified model of early word learning: Integrating statistical and social cues

    Chen Yu and Dana H Ballard. A unified model of early word learning: Integrating statistical and social cues. Neurocomputing, 70 0 (13-15): 0 2149--2165, 2007

    2007

  77. [78]

    Grounded language learning from video described with sentences

    Haonan Yu and Jeffrey Mark Siskind. Grounded language learning from video described with sentences. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 53--63, 2013

    2013

  78. [79]

    Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding

    Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. Advances in neural information processing systems, 37: 0 71737--71767, 2024 a

    2024

  79. [80]

    Groundhog: Grounding large language models to holistic segmentation

    Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai. Groundhog: Grounding large language models to holistic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024 b

    2024

  80. [81]

    Rosie Zhao, Naomi Saphra, and Sham M. Kakade. Distributional scaling laws for emergent capabilities. In NeurIPS 2024 Workshop on Scientific Methods for Understanding Deep Learning, 2024

    2024

Showing first 80 references.