Pith. sign in

REVIEW 4 major objections 4 minor 25 references

Why Are Parsing Actions for Understanding Message Hierarchies Not Random?

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that random-branching parsing is not a plausible model of human language understanding: on nested Dyck-k meanings its communication accuracy drops, and with a surprisal objective it shows high cognitive load on unseen…

desk verdict A fair, incremental follow-up: the Dyck-k result is solid, but the cognitive-load claim in Experiment II is a bridge too far without validation. read the letter →

arxiv 2506.22366 v1 pith:J4LY7K7O submitted 2025-06-27 cs.CL

Daichi Kato , Ryo Ueda , Yusuke Miyao This is my paper · ORCID
classification cs.CL
keywords emergentcommunicationrandom-branchingparsingDyck-kStackLSTMsurprisalcognitiveloadsignalinggamehierarchicalstructure
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks why human parsing strategies are not random, given that earlier emergent-communication experiments found random-branching agents reaching high communication accuracy. It tests two modifications to that setup: a meaning space with genuine nested hierarchy (Dyck-k) and a surprisal-based objective borrowed from beta-VAE. Under the hierarchical meaning space, random-branching accuracy drops relative to learned and left-branching parsers when k>1. Under the surprisal objective, random-branching's learned message prior improves on training messages but worsens on test messages, which the authors read as high cognitive load on unseen meanings. The upshot is that random-branching is implausible as a model of human language understanding, and hierarchical meaning spaces matter for eliciting human-like parsing.

What carries the argument

The central objects are the Stack LSTM receiver, an LSTM controller over a differentiable neural stack whose pop, push, and read strengths determine how a message is parsed, and random-branching, a baseline that samples those strengths uniformly at each timestep. Dyck-k provides the hierarchical meaning space: strings of properly nested parentheses of k types, so that k>1 requires remembering both depth and type order. The second experiment adds a $\beta$-VAE-style objective, pairing reconstruction with a KL term that penalizes divergence between the sender's message distribution and the receiver's learned prior $\log P^{\mathrm{prior}}_{\theta}(M)$, which is what operationalizes surprisal.

What would settle it

Measure human reading times or cloze probabilities on messages produced by random-branching versus left-branching receivers from Experiment II; if messages for unseen meanings do not reliably elicit higher human surprisal, the paper's cognitive-load interpretation is unsupported.

Watch

Extended reading notes

Core claim

The central claim is that the high communication accuracy previously reported for random parsing strategies is an artifact of too-simple meaning spaces and of ignoring surprisal. In Experiment I, using Dyck-k with k>1, random-branching parsers score lower communication accuracy than the Stack LSTM controller and the left-branching baseline, because the receiver must track both nesting depth and the order of parenthesis types. In Experiment II, with the receiver's objective augmented by a message prior so that surprisal enters training, random-branching improves the prior over training messages while degrading it on test messages; the authors interpret this divergence as high cognitive load for unseen meanings. Therefore random-branching is not a plausible model of human parsing.

Load-bearing premise

The load-bearing premise is that the receiver's learned probability of a message is a faithful stand-in for human surprisal, so the paper's high-cognitive-load reading would fail if that probability just reflects neural-stack training artifacts.

Editorial extensions

If this is right

  • Random-branching's high training-time message predictability combined with test-time degradation indicates that it memorizes message statistics rather than learning reusable hierarchical structure.
  • Emergent-communication studies should not treat high communication accuracy under random parsing as evidence that hierarchy is unnecessary.
  • Hierarchical meaning spaces such as Dyck-k with k>1 are a better test bed than attribute-value spaces for distinguishing human-like from random parsers.
  • Surprisal-aware objectives reveal that random-branching generalizes poorly to unseen meanings, measured as a degraded message prior on test data.
  • Models that aim to match human language processing should include both hierarchical structure and surprisal pressure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension would test whether the learned message prior actually tracks human surprisal by collecting reading times or cloze probabilities on messages from the Experiment II receivers.
  • The same Dyck-k manipulation could be applied to non-stack architectures, such as transformers or plain recurrent networks, to see whether the random-branching penalty is specific to stack-based parsing or general across models.
  • If the cognitive-load interpretation holds, emergent languages should be evaluated not only on communication accuracy but also on test-time predictability of messages, which may predict learnability by human-like learners.
  • The random-branching overfitting pattern suggests a testable extension: varying the strength of the KL term should widen or shrink the train/test gap in message predictability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper investigates why random parsing actions are not observed in human language understanding, using emergent communication simulations. Building on Kato et al.'s finding that a random-branching receiver can achieve high communication accuracy, the authors run two modifications: (I) a Dyck-k meaning space with nested hierarchical structure, and (II) a beta-VAE objective that adds a message-prior/surprisal term. In Experiment I, random-branching ComAcc is lower than learned and left-branching parsers for Dyck-k with k>1. In Experiment II, random-branching's ComAcc remains competitive, but the receiver's learned log-probability of messages improves on training messages while degrading on test messages, which the authors interpret as high cognitive load for unseen meanings and hence non-human-like processing. The paper concludes that random-branching is implausible as a model of human language understanding and that hierarchically structured meaning spaces may be important for eliciting human-like parsing in emergent communication.

Significance. If the claims hold, the paper makes a useful contribution to emergent communication by showing that an untrained random-parsing baseline, previously reported as surprisingly strong, is sensitive to the structure of the meaning space and exhibits an unfavorable train/test gap in a message-prior objective. The Dyck-k manipulation is a clean way to test the role of hierarchical structure, and the use of 24 seeds and a Stack LSTM implementation helps reproducibility. However, the central 'cognitive load' interpretation in Experiment II rests on an unvalidated proxy for human surprisal, and the ComAcc claims are presented without error bars or significance tests. The paper is therefore a promising empirical study whose main interpretive claim needs additional support or reframing.

major comments (4)
  1. [Section 4.2, Figure 2c] The paper's central conclusion that random-branching is not human-like because it incurs 'high cognitive cost' rests on interpreting log P_priorθ(M) as a measure of surprisal/cognitive load. This proxy is not validated against any human processing data or against an established surprisal estimate; P_priorθ is a neural density model trained jointly with the communication objective, so the train/test divergence in Figure 2c could be a standard generalization gap of that density estimator rather than a property of random parsing. Since ComAcc does not decline for random-branching in Experiment II (Figure 2b), this is the only evidence supporting the paper's main claim. Please either provide a validation of the proxy, or substantially weaken the cognitive-load interpretation and instead report the result as an overfitting/generalization phenomenon.
  2. [Section 4.1, Figure 2a] The ComAcc comparisons for Experiment I are reported only as aggregated curves with no error bars, confidence intervals, or significance tests, despite the use of 24 seeds. The claim that random-branching 'tends to be lower' for k>1 is therefore not quantitatively supported; please report per-seed distributions, confidence intervals, and preferably a paired test across the shared seeds (e.g., Wilcoxon signed-rank) for the final ComAcc values.
  3. [Section 4.2, Appendix A.2] The paper excludes runs where the β coefficient did not reach 0.95 by the final iteration, but reports neither the number of excluded runs nor whether exclusions differ by condition. If random-branching's β annealing is slower, this exclusion could selectively remove the worst random-branching runs. Please report the exclusion counts per condition and a sensitivity analysis using a different β threshold or using all runs.
  4. [Section 5] The conclusion states that Experiment I observed 'a tendency for the ComAcc gap between random agents and other models to diminish,' which contradicts Section 4.1, where random-branching's ComAcc is lower than other models for larger k (i.e., the gap is larger). This is likely a typo, but as written it inverts the paper's key Experiment I result and must be corrected.
minor comments (4)
  1. [Appendix A.1] The statement that the message space has 'the number of symbols, including EOS, is 4' should clarify that this is the sender's message alphabet, distinct from the Dyck-k meaning alphabet, to avoid an apparent inconsistency with k=4 and k=9.
  2. [Figure 2] The panels do not show per-seed variation or confidence bands; adding shaded intervals or individual seed curves would make the reported trends easier to assess.
  3. [Section 3.2] The switch back to attribute-value meaning spaces for Experiment II deserves a sentence of justification, given the paper's emphasis on hierarchical meaning spaces in Experiment I and in the title.
  4. [References] Several references contain typographical artifacts such as 'V ancouver' (e.g., references [6], [10], [24]); these should be checked against the published versions.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the paper's claims are empirical comparisons against untrained baselines, and the cognitive-load reading is an interpretive step, not a construction.

full rationale

The paper is an empirical comparison, not a derivation chain. Experiment I defines random-branching as an untrained baseline ('The random-branching baseline samples the strengths independently from a uniform distribution over the same ranges as the controller') and measures held-out ComAcc; the finding that ComAcc drops for Dyck-k with k>1 is an observed outcome, not an output forced by a fitted parameter. Experiment II uses P_prior, a receiver-trained density model, and reports a train/test gap ('as message predictability improves on the training data, it deteriorates on the test data'). The step from this gap to 'high cognitive load' is an interpretive mapping to surprisal theory, not a circular reduction: no equation identifies cognitive load with P_prior by construction, and the gap itself is an empirical generalization result. The paper's self-citations ([17], [21]) supply the baseline convention and the beta-VAE objective, but the conclusions do not reduce to those citations: the motivating prior result is independently reproduced in Appendix B, and the beta-VAE framework is used as a modeling tool rather than as evidence for the random-branching conclusion. I find no fitted parameter renamed as a prediction, no uniqueness theorem imported from same-author work, and no definition that makes the claimed result true by construction. The unvalidated surprisal proxy is a construct-validity concern, not circularity; the modest score reflects same-author prior work in the setup, not a load-bearing circular step.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on several domain assumptions about the Stack LSTM, the difficulty of Dyck-k, and the surprisal proxy. The free parameters are experimental design choices rather than fitted quantities in a predictive model, and the paper introduces no new theoretical entities.

free parameters (4)
  • Stack operation strength maxima ku=kd=kr=2 = 2
    Chosen by hand in App A.1; defines the sampling range for random-branching and the allowed controller output range, directly shaping the random baseline.
  • Meaning space max length lmax (18, 8, 6 for k=1,4,9) = 18/8/6
    Chosen to constrain meaning space size (App A.1); controls task difficulty across the three Dyck-k configurations.
  • beta-VAE annealing beta initial value = 0.001
    Initial KL weight in REWO annealing (App A.2), inherited from Ueda et al.; affects the surprisal objective and was post hoc filtered by beta>=0.95.
  • Batch size 8192, LR 1e-4, entropy coeff 0.5, 15k/10k iterations = as listed
    Standard optimization hyperparameters; not fitted to the conclusion but could affect the ranking of baselines across configurations.
assumptions (4)
  • domain assumption The differentiable Neural Stack faithfully implements hierarchical parsing, so differences in stack action strengths isolate the effect of parse strategy.
    Section 3.1: only pop/push/read strengths differ among models; if Stack LSTM does not actually use stack memory, the random vs learned comparison may not be about hierarchy.
  • domain assumption Dyck-k with k>1 requires tracking the order of opened parenthesis types, making it a harder hierarchical meaning space than attribute-value sets.
    Section 4.1 explanation of why random-branching drops; this is a property of the task, not a measured quantity.
  • ad hoc to paper The receiver's learned prior log P_prior(theta)(M) is a valid measure of surprisal and cognitive load.
    Section 4.2 and 5 interpret test-set log-probability divergence as cognitive load; the proxy is not externally validated.
  • standard math Ueda et al.'s reinterpretation of signaling games as beta-VAE with REWO annealing is accepted as given.
    Section 2.2: the objective J_ec-vae is adopted from prior work; its validity is not re-derived in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Why Are Parsing Actions for Understanding Message Hierarchies Not Random?." pith.science (2026). https://pith.science/paper/J4LY7K7O

@misc{pith2026250622366,
  author       = {Pith},
  title        = {Pith review of: Why Are Parsing Actions for Understanding Message Hierarchies Not Random?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J4LY7K7O}},
  note         = {Machine review of arXiv:2506.22366}
}
abstract

If humans understood language by randomly selecting parsing actions, it might have been necessary to construct a robust symbolic system capable of being interpreted under any hierarchical structure. However, human parsing strategies do not seem to follow such a random pattern. Why is that the case? In fact, a previous study on emergent communication using models with hierarchical biases have reported that agents adopting random parsing strategies$\unicode{x2013}$ones that deviate significantly from human language comprehension$\unicode{x2013}$can achieve high communication accuracy. In this study, we investigate this issue by making two simple and natural modifications to the experimental setup: (I) we use more complex inputs that have hierarchical structures, such that random parsing makes semantic interpretation more difficult, and (II) we incorporate a surprisal-related term, which is known to influence the order of words and characters in natural language, into the objective function. With these changes, we evaluate whether agents employing random parsing strategies still maintain high communication accuracy.

Figures

Figures reproduced from arXiv: 2506.22366 by the authors.

Figure 1
Figure 1. Overview of the Stack RNN. Due to the discrete nature of these operations, standard backpropagation cannot be applied, making it generally difficult to incorporate stacks into neural network architectures. Grefenstette et al. [19] addressed this issue by treating the pop and push operations as continuous values and introducing intermediate stack states, thereby making the entire stack computation differen￾tiable. At… view at source ↗
Figure 2
Figure 2. Experimental results 4.2 Experiment II As seen in Figure 2b, introducing a surprisal-related term did not significantly affect the overall trend that the ComAcc of the random-branching condition tends to be higher. However, as shown in Figure 2c, in our Stack LSTM-based model as well as the left-branching model, the predictability of messages during training tends to improve along with their predictability on the te… view at source ↗
Figure 3
Figure 3. ComAcc results of the preliminary experiment [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 24 canonical work pages

  1. [1]

    Emergent multi-agent communication in the deep learning era

    Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. CoRR, abs/2006.02419, 2020

    arXiv 2006

  2. [2]

    A review of the applications of deep learning-based emergent communication

    Brendon Boldt and David R Mortensen. A review of the applications of deep learning-based emergent communication. Transactions on Machine Learning Research, 2024

    work page 2024

  3. [3]

    Emergent language: a survey and taxonomy

    Jannik Peters, Constantin Waubert de Puiseau, Hasan Tercan, Arya Gopikrishnan, Gus- tavo Adolpho Lucas de Carvalho, Christian Bitter, and Tobias Meisen. Emergent language: a survey and taxonomy. Autonomous Agents and Multi-Agent Systems , 39(1):18, mar 2025

    work page 2025

  4. [4]

    David K. Lewis. Convention: A Philosophical Study . Wiley-Blackwell, 1969

    work page 1969

  5. [5]

    Anti-efficient encoding in emergent communication

    Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient encoding in emergent communication. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems...

    work page 2019

  6. [6]

    Compositionality and generalization in emergent languages

    Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020 , page...

    work page 2020

  7. [7]

    Emergent communication: Generalization and overfitting in lewis games

    Mathieu Rita, Corentin Tallec, Paul Michel, Jean-Bastien Grill, Olivier Pietquin, Emmanuel Dupoux, and Florian Strub. Emergent communication: Generalization and overfitting in lewis games. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neura...

    work page 2022

  8. [8]

    On the word boundaries of emergent languages based on harris’s articulation scheme

    Ryo Ueda, Taiga Ishii, and Yusuke Miyao. On the word boundaries of emergent languages based on harris’s articulation scheme. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023

    work page 2023

Show all 25 references
  1. [9]

    Emergence of language with multi-agent games: Learning to communicate with sequences of symbols

    Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Garnett, editors,Advances in Neu...

    2017

  2. [10]

    Emergence of linguistic communication from referential games with symbolic and pixel input

    Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In 6th Interna- tional Conference on Learning Representations, ICLR 2018, V ancouver , BC, Canada, April 30 - May...

    2018

  3. [11]

    Long short-term memory.Neural Comput., 9(8):1735– 1780, 1997

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory.Neural Comput., 9(8):1735– 1780, 1997. 6

    1997

  4. [12]

    Learning phrase representations using RNN encoder- decoder for statistical machine translation

    Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, ed...

    2014

  5. [13]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Garnett,...

    2017

  6. [14]

    Findings of the babylm challenge: Sample-efficient pretraining on developmentally plausible corpora

    Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, et al. Findings of the babylm challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of...

    2023

  7. [15]

    Syntactic Structures

    Noam Chomsky. Syntactic Structures. De Gruyter Mouton, Berlin, Boston, 1957

    1957

  8. [16]

    Neurophysi- ological dynamics of phrase-structure building during sentence processing

    Matthew J Nelson, Imen El Karoui, Kristof Giber, Xiaofang Yang, Laurent Cohen, Hilda Koopman, Sydney S Cash, Lionel Naccache, John T Hale, Christophe Pallier, et al. Neurophysi- ological dynamics of phrase-structure building during sentence processing. Proceedings of the Natio...

    2017

  9. [17]

    Emergent communication with stack-based agents

    Daichi Kato, Ryo Ueda, Jason Naradowsky, and Yusuke Miyao. Emergent communication with stack-based agents. In Proceedings of the 46th Annual Meeting of the Cognitive Science Society , 2024

    2024

  10. [18]

    Expectation-based syntactic comprehension

    Roger Levy. Expectation-based syntactic comprehension. Cognition, 106(3):1126–1177, 2008

    2008

  11. [19]

    Learning to transduce with unbounded memory

    Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. Learning to transduce with unbounded memory. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: An...

    2015

  12. [20]

    Finding syntactic representations in neural stacks

    William Merrill, Lenny Khazan, Noah Amsel, Yiding Hao, Simon Mendelsohn, and Robert Frank. Finding syntactic representations in neural stacks. CoRR, abs/1906.01594, 2019

    1906 arXiv

  13. [21]

    Lewis’s signaling game as beta-vae for natural word lengths and segments

    Ryo Ueda and Tadahiro Taniguchi. Lewis’s signaling game as beta-vae for natural word lengths and segments. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024

    2024

  14. [22]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014

    2014

  15. [23]

    Burgess, Xavier Glorot, Matthew M

    Irina Higgins, Loïc Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In 5th International Conference on Learning Rep- resen...

    2017

  16. [24]

    Learn- ing hierarchical priors in vaes

    Alexej Klushyn, Nutan Chen, Richard Kurle, Botond Cseke, and Patrick van der Smagt. Learn- ing hierarchical priors in vaes. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Infor- mation...

    2019

  17. [25]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. 7 Stack LSTM left...

    2015

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.