{"id":"0ac2e021-8808-465d-a7e3-a61f8ac52439","arxiv_id":"2506.22366","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Random-branching parsing agents lose their communication-accuracy advantage on Dyck-k hierarchical meaning spaces and show high surprisal on unseen messages, suggesting they are poor models of human language understanding.","lead":"This paper tests whether agents that parse message hierarchies at random still communicate accurately on harder, structured tasks. It finds that on Dyck-k hierarchical meaning spaces and with a surprisal-sensitive objective, random parsing loses its advantage and behaves unlike human language processing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central claim that random-branching incurs high cognitive load rests on an unvalidated proxy: log P_prior as human surprisal, so the train/test divergence in Figure 2c may be a neural density-model artifact rather than a property of random parsing.","rationale":"I agree with the reader's identification of the weakest assumption. The paper's final conclusion that random agents are inappropriate as models of human communication rests primarily on Experiment II's cognitive-load interpretation, which depends on log P_prior as a surrogate for human surprisal. The paper itself introduces the prior through the beta-VAE formulation but treats its negative log-likelihood as 'surprisal' without any calibration to human processing data. The train/test gap in Figure 2c could equally arise from the receiver's density model overfitting to training messages, especially under random-branching where read vectors are stochastic; this is a known failure mode of learned generative models and does not require invoking cognitive cost. The concern is addressable with a control using a fixed, non-parametric prior, so it does not change the reader's conditional verdict; it reinforces that the paper should not be accepted until this proxy is validated or the claim is weakened.","tokens_in":7050,"tokens_out":4698,"duration_ms":53416,"concrete_test":"Re-run Experiment II with the learned neural prior replaced by a fixed k-gram prior (e.g., trigram with Kneser-Ney smoothing) estimated only from training messages, and compute log P_prior on both train and test messages for each branching condition. If random-branching no longer exhibits the train/test divergence seen in Figure 2c, then the effect is an artifact of the learned density model, and the cognitive-load interpretation in Section 4.2 is unsupported. If the divergence persists under the fixed prior, it is a property of the message distribution, but a further human reading-time or cloze study would still be needed to validate the surprisal-to-cognitive-load mapping.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The decisive step is in Section 4.2: Figure 2c shows that random-branching improves log P_prior on training messages but worsens on test messages, and the authors conclude that such receivers 'interpret messages with greater surprise' and suffer 'high cognitive cost' (Section 5). This inference requires that the receiver's learned prior P_prior(theta)(M), defined in Sections 2.2 and 3.2, is a valid proxy for human processing difficulty. The paper offers no external validation of this mapping. Because P_prior is a neural language model trained with a beta-VAE objective on the receiver's own read vectors, its train/test gap can be driven by the capacity and optimization of that density model—for example, memorizing training messages or exploiting the stochastic read vectors of random-branching—rather than by any cognitive property of random parsing. If so, the observed divergence is just a standard generalization gap, and the conclusion that random-branching is 'not human-like' does not follow. The claim is load-bearing because it is the only evidence in Experiment II that random-branching is implausible as a model of human language understanding (ComAcc itself does not decline in Experiment II).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates why random parsing actions are not observed in human language understanding, using emergent communication simulations. Building on Kato et al.'s finding that a random-branching receiver can achieve high communication accuracy, the authors run two modifications: (I) a Dyck-k meaning space with nested hierarchical structure, and (II) a beta-VAE objective that adds a message-prior/surprisal term. In Experiment I, random-branching ComAcc is lower than learned and left-branching parsers for Dyck-k with k>1. In Experiment II, random-branching's ComAcc remains competitive, but the receiver's learned log-probability of messages improves on training messages while degrading on test messages, which the authors interpret as high cognitive load for unseen meanings and hence non-human-like processing. The paper concludes that random-branching is implausible as a model of human language understanding and that hierarchically structured meaning spaces may be important for eliciting human-like parsing in emergent communication.","tokens_in":7321,"tokens_out":5988,"duration_ms":64901,"significance":"If the claims hold, the paper makes a useful contribution to emergent communication by showing that an untrained random-parsing baseline, previously reported as surprisingly strong, is sensitive to the structure of the meaning space and exhibits an unfavorable train/test gap in a message-prior objective. The Dyck-k manipulation is a clean way to test the role of hierarchical structure, and the use of 24 seeds and a Stack LSTM implementation helps reproducibility. However, the central 'cognitive load' interpretation in Experiment II rests on an unvalidated proxy for human surprisal, and the ComAcc claims are presented without error bars or significance tests. The paper is therefore a promising empirical study whose main interpretive claim needs additional support or reframing.","major_comments":[{"comment":"The paper's central conclusion that random-branching is not human-like because it incurs 'high cognitive cost' rests on interpreting log P_priorθ(M) as a measure of surprisal/cognitive load. This proxy is not validated against any human processing data or against an established surprisal estimate; P_priorθ is a neural density model trained jointly with the communication objective, so the train/test divergence in Figure 2c could be a standard generalization gap of that density estimator rather than a property of random parsing. Since ComAcc does not decline for random-branching in Experiment II (Figure 2b), this is the only evidence supporting the paper's main claim. Please either provide a validation of the proxy, or substantially weaken the cognitive-load interpretation and instead report the result as an overfitting/generalization phenomenon.","section":"Section 4.2, Figure 2c"},{"comment":"The ComAcc comparisons for Experiment I are reported only as aggregated curves with no error bars, confidence intervals, or significance tests, despite the use of 24 seeds. The claim that random-branching 'tends to be lower' for k>1 is therefore not quantitatively supported; please report per-seed distributions, confidence intervals, and preferably a paired test across the shared seeds (e.g., Wilcoxon signed-rank) for the final ComAcc values.","section":"Section 4.1, Figure 2a"},{"comment":"The paper excludes runs where the β coefficient did not reach 0.95 by the final iteration, but reports neither the number of excluded runs nor whether exclusions differ by condition. If random-branching's β annealing is slower, this exclusion could selectively remove the worst random-branching runs. Please report the exclusion counts per condition and a sensitivity analysis using a different β threshold or using all runs.","section":"Section 4.2, Appendix A.2"},{"comment":"The conclusion states that Experiment I observed 'a tendency for the ComAcc gap between random agents and other models to diminish,' which contradicts Section 4.1, where random-branching's ComAcc is lower than other models for larger k (i.e., the gap is larger). This is likely a typo, but as written it inverts the paper's key Experiment I result and must be corrected.","section":"Section 5"}],"minor_comments":[{"comment":"The statement that the message space has 'the number of symbols, including EOS, is 4' should clarify that this is the sender's message alphabet, distinct from the Dyck-k meaning alphabet, to avoid an apparent inconsistency with k=4 and k=9.","section":"Appendix A.1"},{"comment":"The panels do not show per-seed variation or confidence bands; adding shaded intervals or individual seed curves would make the reported trends easier to assess.","section":"Figure 2"},{"comment":"The switch back to attribute-value meaning spaces for Experiment II deserves a sentence of justification, given the paper's emphasis on hierarchical meaning spaces in Experiment I and in the title.","section":"Section 3.2"},{"comment":"Several references contain typographical artifacts such as 'V ancouver' (e.g., references [6], [10], [24]); these should be checked against the published versions.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for cs.CL and the experimental setup is clearly described. The main concern is that the 'cognitive load' conclusion in Experiment II is an interpretation of a neural density model's generalization gap, and the paper would be stronger if the authors either validated this proxy or adopted a more cautious framing. The missing error bars and the undocumented run exclusions are also important for a journal-level assessment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: the paper earns its keep with Experiment I and overreaches in Experiment II. The authors show that random-branching parsers—which previously matched or beat learned parsers on attribute-value meanings—lose that edge on Dyck-k meaning spaces with k>1, where recognizing the order of different parenthesis types matters. That is a clean, reproducible result and it does real work: it explains the earlier anomaly as an artifact of too-simple meaning spaces. Good.\n\nThey also add a beta-VAE objective to bring in surprisal, and report that under random-branching, the receiver's learned message prior gets better on training messages but worse on test messages. They interpret this as high cognitive load for unseen meanings, making random-branching implausible as a model of human parsing. That interpretation is not earned. The stress-test note is right: log P_prior is a neural density model's predictive probability, and a train/test divergence in that quantity is exactly what you'd expect from any model that overfits the training messages, especially one with stochastic stack operations. The authors themselves call it 'overfitting' in Section 4.2, then pivot to 'cognitive cost' in Section 5 without any external validation that this proxy tracks human surprisal. That is the softest part of the paper, and it would need a lot more support to be load-bearing.\n\nOther soft spots: no error bars or significance tests on any of the ComAcc comparisons, even with 24 seeds; and Experiment II excludes runs where beta never reached 0.95, with no sensitivity analysis on that threshold. Both are minor and fixable.\n\nWho benefits: anyone working on emergent communication or on stack-based inductive biases in neural sequence models. The Dyck-k result is worth citing even if the cognitive-load framing falls away. I'd send it to a serious referee, with the request that Experiment II be repositioned as hypothesis-generating rather than evidence about human cognition. The paper is honest and clearly written; it just needs to say less about what the surprisal numbers mean.\n\nRecommendation: engage with it in review, not with a desk reject.","headline":"A fair, incremental follow-up: the Dyck-k result is solid, but the cognitive-load claim in Experiment II is a bridge too far without validation.","tokens_in":7868,"tokens_out":2708,"would_cite":true,"duration_ms":29147,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that random-branching parsing is not a plausible model of human language understanding: on nested Dyck-k meanings its communication accuracy drops, and with a surprisal objective it shows high cognitive load on unseen…","keywords":["emergent communication","random-branching parsing","Dyck-k","Stack LSTM","surprisal","cognitive load","signaling game","hierarchical structure"],"falsifier":"Measure human reading times or cloze probabilities on messages produced by random-branching versus left-branching receivers from Experiment II; if messages for unseen meanings do not reliably elicit higher human surprisal, the paper's cognitive-load interpretation is unsupported.","tokens_in":6801,"feed_emoji":"🧠","tokens_out":5894,"duration_ms":61060,"temperature":0.7,"pith_summary":"The paper asks why human parsing strategies are not random, given that earlier emergent-communication experiments found random-branching agents reaching high communication accuracy. It tests two modifications to that setup: a meaning space with genuine nested hierarchy (Dyck-k) and a surprisal-based objective borrowed from beta-VAE. Under the hierarchical meaning space, random-branching accuracy drops relative to learned and left-branching parsers when k>1. Under the surprisal objective, random-branching's learned message prior improves on training messages but worsens on test messages, which the authors read as high cognitive load on unseen meanings. The upshot is that random-branching is implausible as a model of human language understanding, and hierarchical meaning spaces matter for eliciting human-like parsing.","feed_headline":"Random parsing fails on nested meanings and looks costly for humans","feed_subtitle":"Two experiments: random-branching parsers lose accuracy on Dyck-k hierarchies and overfit their message priors.","key_machinery":"The central objects are the Stack LSTM receiver, an LSTM controller over a differentiable neural stack whose pop, push, and read strengths determine how a message is parsed, and random-branching, a baseline that samples those strengths uniformly at each timestep. Dyck-k provides the hierarchical meaning space: strings of properly nested parentheses of k types, so that k>1 requires remembering both depth and type order. The second experiment adds a $\\beta$-VAE-style objective, pairing reconstruction with a KL term that penalizes divergence between the sender's message distribution and the receiver's learned prior $\\log P^{\\mathrm{prior}}_{\\theta}(M)$, which is what operationalizes surprisal.","core_discovery":"The central claim is that the high communication accuracy previously reported for random parsing strategies is an artifact of too-simple meaning spaces and of ignoring surprisal. In Experiment I, using Dyck-k with k>1, random-branching parsers score lower communication accuracy than the Stack LSTM controller and the left-branching baseline, because the receiver must track both nesting depth and the order of parenthesis types. In Experiment II, with the receiver's objective augmented by a message prior so that surprisal enters training, random-branching improves the prior over training messages while degrading it on test messages; the authors interpret this divergence as high cognitive load for unseen meanings. Therefore random-branching is not a plausible model of human parsing.","pith_inferences":["A direct extension would test whether the learned message prior actually tracks human surprisal by collecting reading times or cloze probabilities on messages from the Experiment II receivers.","The same Dyck-k manipulation could be applied to non-stack architectures, such as transformers or plain recurrent networks, to see whether the random-branching penalty is specific to stack-based parsing or general across models.","If the cognitive-load interpretation holds, emergent languages should be evaluated not only on communication accuracy but also on test-time predictability of messages, which may predict learnability by human-like learners.","The random-branching overfitting pattern suggests a testable extension: varying the strength of the KL term should widen or shrink the train/test gap in message predictability."],"forward_implications":["Random-branching's high training-time message predictability combined with test-time degradation indicates that it memorizes message statistics rather than learning reusable hierarchical structure.","Emergent-communication studies should not treat high communication accuracy under random parsing as evidence that hierarchy is unnecessary.","Hierarchical meaning spaces such as Dyck-k with k>1 are a better test bed than attribute-value spaces for distinguishing human-like from random parsers.","Surprisal-aware objectives reveal that random-branching generalizes poorly to unseen meanings, measured as a degraded message prior on test data.","Models that aim to match human language processing should include both hierarchical structure and surprisal pressure."],"supporting_citations":[{"why":"Supplies the original finding that random-branching achieves high communication accuracy and defines the baselines this paper challenges.","marker":"[17]"},{"why":"Provides the reinterpretation of the signaling game as a beta-VAE, including the message prior and surprisal-related KL term used in Experiment II.","marker":"[21]"},{"why":"Introduces the Neural Stack and Stack LSTM architecture whose pop, push, and read strengths define the left-branching and random-branching baselines.","marker":"[19]"},{"why":"Supplies the Stack LSTM variant used here, which allows pop, push, and read values to exceed 1.","marker":"[20]"},{"why":"Provides the surprisal concept that the paper maps onto the receiver's learned message prior.","marker":"[18]"},{"why":"Supplies the beta-weighted objective underlying the signaling-game-as-VAE formulation.","marker":"[23]"}],"fun_headline_variants":["Random parsing loses on Dyck-k hierarchies and overfits message priors","Nested meanings break random parsing and reveal surprisal costs","Random parsing fails on hierarchical inputs and shows cognitive load","Random parsing costs on unseen meanings due to surprisal","Why random parsing is not how humans understand message hierarchies"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the receiver's learned probability of a message is a faithful stand-in for human surprisal, so the paper's high-cognitive-load reading would fail if that probability just reflects neural-stack training artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Random parsing loses on Dyck-k hierarchies and overfits message priors","Nested meanings break random parsing and reveal surprisal costs","Random parsing fails on hierarchical inputs and shows cognitive load","Random parsing costs on unseen meanings due to surprisal","Why random parsing is not how humans understand message hierarchies"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000763,"raw_usage":{"total_tokens":3340,"prompt_tokens":854,"completion_tokens":2486,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":2402}},"tokens_in":470,"tokens_out":2486,"duration_ms":18149,"temperature":1.0,"reasoning_tokens":2402,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:05:01.129872+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure human reading times or cloze probabilities on messages produced by random-branching versus left-branching receivers from Experiment II; if messages for unseen meanings do not reliably elicit higher human surprisal, the paper's cognitive-load interpretation is unsupported.","supporting_citations":[{"cited_title":"Emergent communication with stack-based agents","cited_arxiv_id":null,"evidence_quote":"Supplies the original finding that random-branching achieves high communication accuracy and defines the baselines this paper challenges."},{"cited_title":"Lewis’s signaling game as beta-vae for natural word lengths and segments","cited_arxiv_id":null,"evidence_quote":"Provides the reinterpretation of the signaling game as a beta-VAE, including the message prior and surprisal-related KL term used in Experiment II."},{"cited_title":"Learning to transduce with unbounded memory","cited_arxiv_id":null,"evidence_quote":"Introduces the Neural Stack and Stack LSTM architecture whose pop, push, and read strengths define the left-branching and random-branching baselines."},{"cited_title":"Finding Syntactic Representations in Neural Stacks","cited_arxiv_id":"1906.01594","evidence_quote":"Supplies the Stack LSTM variant used here, which allows pop, push, and read values to exceed 1."},{"cited_title":"Expectation-based syntactic comprehension","cited_arxiv_id":null,"evidence_quote":"Provides the surprisal concept that the paper maps onto the receiver's learned message prior."},{"cited_title":"Burgess, Xavier Glorot, Matthew M","cited_arxiv_id":null,"evidence_quote":"Supplies the beta-weighted objective underlying the signaling-game-as-VAE formulation."}],"review_version":1}