{"id":"c68d2ab2-688d-4028-a78c-f4baada32931","arxiv_id":"2505.05970","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A language-only question-answering reference game gives language models a communicative reward signal, but fine-tuning with it changes speaker behavior without improving grammatical ability.","lead":"This paper tests whether a speaker language model can learn from a listener's success at answering questions about its summaries, instead of from next-word prediction alone. It finds the reward signal is usable in principle but does not yet improve the model's grammar, and reports mostly negative training results.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reward signal is not shown to isolate grammaticality from content loss; §5's perturbations confound syntax with semantics, so the abstract's grammaticality claim is under-supported.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: ROUGE-L against ground-truth answers in a frozen QA listener is not validated as a proxy for human communicative success, and the perturbations in the feasibility study conflate grammaticality with general text quality. My stress test agrees with that assessment and contributes a concrete minimal-pair-style test that would separate the two. The paper is honestly framed as a proof of concept and reports a negative main result, so the conditional verdict remains appropriate. No further verdict adjustment is warranted. The primary check should be added as a condition: demonstrate that the reward isolates grammaticality from content preservation, or soften the abstract's claim accordingly.","tokens_in":16162,"tokens_out":2895,"duration_ms":34651,"concrete_test":"Run a controlled reward audit on held-out SQuAD 2.0 dev passages: for each passage, construct matched grammatical and ungrammatical summaries with the same content words (e.g., function-word deletion, subject-verb agreement errors, local word-order swaps within clauses that preserve noun-phrase meaning). Score both variants with the exact reward pipeline from §4 (UnifiedQA listener + ROUGE-L), using ≥100 samples per condition and bootstrap confidence intervals. If the grammatical condition does not significantly outperform the ungrammatical condition after controlling for length and content-word overlap, the §5 feasibility result is explained by content loss rather than grammaticality, and the grammaticality-signal claim requires revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central feasibility claim—that the reward provides an 'indirect signal about grammaticality' (Abstract, §5)—rests on showing that listener ROUGE-L degrades when the context is corrupted. However, every corruption type in Figure 2 (truncation, deletion, scrambling) simultaneously removes or rearranges semantic content. The only syntax-targeted manipulation, stop-word removal, yields only a 'minor loss,' suggesting the reward gradient is driven mainly by content preservation, not grammaticality. Moreover, Figure 2 reports macro-averages without error bars, and the paper does not test whether the reward can distinguish grammatical from ungrammatical variants with matched content. This matters because the actual speaker outputs are T5 summaries, which may not fall in the corrupted-input distribution tested; the reward may therefore provide no usable gradient for grammar during training. If the reward tracks general text quality rather than grammar specifically, the proof-of-concept for grammaticality fails, even though the bottleneck experiments do show interpretable behavioral changes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes the 'abstract reference game,' a language-only variant of Lewis-style reference games in which a speaker T5 model summarizes a passage, a frozen UnifiedQA listener answers a question based on the summary, and the speaker receives a reward based on ROUGE-L between the listener's answer and a ground-truth answer. The authors present a feasibility study (Section 5) showing that listener QA performance degrades under truncation, deletion, and scrambling of the context, which they interpret as evidence that the reward is an indirect signal for grammaticality. They then apply PPO to fine-tune T5 under length- or surprisal-based bottlenecks (Section 7) and find that a length bottleneck yields telegraphic, function-word-depleted output while a surprisal bottleneck preserves grammatical structure more; no improvement on BLiMP or LanguageTool is observed. Training from scratch (Section 6) fails to increase reward. The paper is honest about its negative results and frames the contribution as a proof of concept.","tokens_in":16349,"tokens_out":5480,"duration_ms":53672,"significance":"The abstract reference game is a novel and reasonably clean operationalization of communicative feedback for language acquisition research, and the use of a frozen listener to avoid semantic drift is well motivated. The paper reports controlled experiments with three seeds and multiple metrics, and it explicitly acknowledges the limitations of its training budget and benchmarks. If the feasibility claim were established, the paradigm would be a useful testbed for studying interaction as a learning signal in developmentally plausible LMs. However, the current evidence does not isolate grammaticality from content preservation, so the central claim is not yet supported. The observed bottleneck-dependent behavioral changes are interesting and reproducible, but their connection to grammatical learning remains speculative.","major_comments":[{"comment":"The feasibility study confounds grammaticality with content preservation. Truncation, deletion, and word-order scrambling all remove or rearrange semantic information, so the observed degradation in listener performance is expected even if the listener is indifferent to grammar. The only syntax-specific manipulation, stop-word removal, is reported as a 'minor loss' with no error bars or significance test. To support the abstract's claim that the reward is an 'indirect signal about grammaticality,' the authors should test matched-content minimal pairs (e.g., grammatical vs. ungrammatical paraphrases with identical propositions) and show that the reward separates them. As it stands, the gradient in Figure 2 may reflect general text quality rather than grammar specifically.","section":"Section 5, Figure 2"},{"comment":"The figure reports macro-averages over 17 datasets without error bars, confidence intervals, or statistical tests. Claims such as 'minor loss' and 'inversely correlated' rest on visual inspection of single points. Because the feasibility study is the primary evidence for the central claim, the authors should provide variation across datasets and/or repeated perturbation draws, or at least a table of per-condition values with standard deviations.","section":"Section 5, Figure 2"},{"comment":"The reward is ROUGE-L between the listener's answer and the ground truth. ROUGE-L is a lexical overlap metric that is not validated as a proxy for human communicative success or for grammaticality. The paper should at least discuss this limitation, and ideally provide a small human evaluation or a correlation with an alternative metric (e.g., accuracy on multiple-choice QA subsets) to justify the operationalization.","section":"Section 4, Equations for SCORElength and SCOREsurprisal"}],"minor_comments":[{"comment":"The phrase 'GPTs, Llamas, and Pythias' should read 'GPTs, Llamas, and Pythia' (or 'Pythia models').","section":"Section 2.1"},{"comment":"The citation 'V ong et al.' contains a spacing artifact and should be 'Vong et al.'.","section":"Section 3"},{"comment":"Several references contain spacing artifacts, e.g., 'F orm and Function' for Bloom (1970). These should be cleaned up.","section":"References"},{"comment":"The caption says 'T5 QA Performance', but the listener is UnifiedQA; rename to 'QA listener performance' to avoid confusion.","section":"Figure 2"},{"comment":"The specific values of the perturbation parameters (c, m, s, d) for the low/medium/high conditions are not provided, which makes the feasibility study not fully reproducible; please include them in the appendix.","section":"Section 5"},{"comment":"The fine-tuning experiments use only 240 PPO steps on a 540k-token SQuAD 2.0 validation split. The absence of grammatical improvements is therefore unsurprising and should not be overinterpreted; the authors' discussion already acknowledges this, but the point could be stated even more explicitly in the abstract.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a proof of concept with a modest central claim; the experiments are honest and the writing is generally clear. The main risk is that the feasibility claim—the load-bearing pillar for the abstract—is not currently supported because the perturbations confound syntax with semantics and the results lack error bars. This is fixable with additional controls and more careful reporting, so I recommend major revision rather than rejection. The novelty is moderate but reasonable for a venue interested in cognitive modeling and interactive learning."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the headline: this is a genuinely new instantiation of a reference game in a language-only setting, and the paper is admirably honest about what doesn't work. Second, the abstract's claim that the reward is an 'indirect signal about grammaticality' is real but under-supported: the feasibility study doesn't isolate grammar from content, and the training results show no grammatical improvement.\n\nWhat the paper does well: it operationalizes communicative success via a summarization-plus-QA game, uses a frozen listener to avoid semantic drift, and carefully documents how length and surprisal bottlenecks alter speaker output. The finding that a length penalty drives telegraphic, function-word-poor summaries while a surprisal penalty preserves more grammatical structure is credible, backed by three seeds and multiple metrics. The authors also deserve credit for reporting null results on BLiMP and LanguageTool, and for citing the independent work of Giulianelli and Sepulveda.\n\nThe main soft spot is exactly what the stress-test note flags: the feasibility study in Section 5 does not separate grammaticality from semantic content. Truncation, scrambling, and deletion all co-remove or rearrange meaning, so the observed reward degradation could be driven by content loss. The closest syntax-targeted manipulation, stop-word removal, yields only a minor loss, which suggests the reward is not particularly sensitive to grammar specifically. That said, the paper explicitly frames the claim as 'indirect' and the experiments as a feasibility study, so this is a limitation of the evidence rather than a misrepresentation. Still, a referee should ask for a cleaner test — for example, minimal pairs that match content but differ in grammaticality — before the grammaticality claim is taken as established.\n\nOther issues are minor but worth naming: Figure 2 reports macro-averages without error bars; ROUGE-L is a coarse reward proxy; fine-tuning runs only 240 steps on a small SQuAD subset; and no code or data are released, which hampers reproduction of the behavioral findings.\n\nOverall, this is a useful proof-of-concept for a research direction, not a demonstrated method improvement. It deserves peer review and publication after revision. I would bring it to a reading group focused on language acquisition or interactive learning; it will generate good discussion about how to design learning signals that isolate syntax from semantics.\n\nRecommended for peer review, conditional on the grammaticality claim being tightened and code/data release.","headline":"An honest proof-of-concept for communicative-success rewards in a language-only reference game, with solid behavioral findings under bottlenecks but a grammaticality claim that is only partially supported.","tokens_in":16846,"tokens_out":2374,"would_cite":true,"duration_ms":27514,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that communicative success in a language-only question-answering reference game can serve as a learning signal for language models, that the signal is at least indirectly about grammaticality, and that channel…","keywords":["communicative success","reinforcement learning","language acquisition","reference game","summarization","grammaticality","surprisal","question answering"],"falsifier":"Prepare pairs of summaries that use exactly the same words but differ only in word order—one grammatical, one scrambled—and measure the ROUGE-L reward the listener returns; if the reward does not drop when grammar is destroyed, the signal carries no grammatical information. A complementary experiment is to train the speaker with a listener that ignores word order entirely and check whether BLiMP scores still fail to move.","tokens_in":15978,"feed_emoji":"🗣️","tokens_out":9994,"duration_ms":90234,"temperature":0.7,"pith_summary":"The paper proposes that an artificial language model can learn from the simple fact of being understood, the way a child does. In the authors' 'abstract reference game,' a speaker summarizes a passage, a frozen question-answering listener answers a question from that summary, and the speaker is rewarded by how well the listener's answer matches a ground-truth answer. A feasibility study shows that corrupting the passage—truncating it, deleting stop words, or scrambling word order—degrades the listener's answers, so the reward carries at least an indirect signal about text quality and plausibly about grammaticality. Reinforcement-learning fine-tuning with this reward changes behavior: a length bottleneck drives summaries toward telegraphic, function-word-poor speech, while a surprisal bottleneck produces output closer to the original model's grammar. The paper reports no improvement in grammatical ability on the BLiMP and LanguageTool benchmarks, and training from scratch degenerates into nonsense.","feed_headline":"Being understood reshapes how language models talk, not their grammar","feed_subtitle":"A question-answering reference game gives models a reward signal from communicative success, but grammar benchmarks stay flat","key_machinery":"The mechanism is the abstract reference game: a speaker sees a passage, produces a summary, a frozen pretrained question-answering listener reads the summary and answers a question, and the reward is the ROUGE-L F1 score between the listener's answer and the ground truth. To keep the speaker from simply copying the passage, the score subtracts a penalty proportional either to the summary's length relative to the passage or to the listener's surprisal (negative log probability) of the summary relative to the passage. The speaker is trained with Proximal Policy Optimization, and the balance between reward and penalty is controlled by a single hyperparameter $\\lambda$ through $\\text{score} = (1-\\lambda)\\cdot\\text{reward} - \\lambda\\cdot\\text{penalty}$. This design converts the abstract goal of 'being understood' into a concrete, optimizable numerical signal, and lets the authors attribute changes in speaker behavior to specific communicative costs.","core_discovery":"The central claim is that communicative success in a language-only question-answering reference game is a usable learning signal for language models, and specifically 'an indirect signal about grammaticality.' The authors make the case with a feasibility study: when the listener is fed ungrammatical or incomplete summaries, its answer quality degrades smoothly, so a speaker optimizing that reward has some incentive to produce grammatical, complete text. They then fine-tune a pretrained speaker with reinforcement learning, with a penalty that either rewards brevity or rewards low listener surprisal. The result is that the bottleneck changes what the speaker says in interpretable ways—copying under no penalty, telegraphic speech under a length penalty, and more conservative grammatical output under a surprisal penalty. None of these conditions improves grammatical knowledge on BLiMP or LanguageTool, and the paper presents that negative result as an open challenge rather than a refutation of the method.","pith_inferences":["Because the feasibility study's corruptions change syntax and semantics together, the same evidence is consistent with the reward tracking general text quality rather than grammaticality specifically—a distinction the paper does not resolve.","A direct test this paper does not run would replace the frozen listener with a bag-of-words or order-insensitive reader; if grammaticality still influences the reward, word order is not the carrier, and if it stops, the signal is about syntax.","The 240 training steps and roughly 500,000-token fine-tuning budget are smaller than the scale at which interactive rewards might be expected to show grammatical gains, so the negative BLiMP result is more a ceiling on this configuration than on the method.","A curriculum that starts with a tight length bottleneck and gradually relaxes it could turn the observed telegraphic drift into a developmental trajectory, mirroring children's progression from one-word utterances to full sentences."],"forward_implications":["If the reward is a valid learning signal, language models can in principle improve their output with no explicit grammatical feedback, purely from observing whether a listener understood them.","A length bottleneck reliably pushes outputs toward telegraphic summaries that drop function words, which could be used to model the one- and two-word utterance phases of child speech.","A surprisal bottleneck keeps summaries closer to the grammatical style of the base model, indicating that information-density pressure is more compatible with grammatical output than raw brevity is.","The lack of any BLiMP or LanguageTool improvement means the current recipe is missing at least one ingredient—longer training, a larger question-answering data budget, a different model, or a richer notion of communicative success.","Training from scratch with this reward fails outright, so the method cannot yet explain how a learner with no prior language bootstraps from interaction alone."],"supporting_citations":[{"why":"Supplies the UnifiedQA listener; the listener's generated answers define communicative success and determine the reward.","marker":"Khashabi et al. (2020)"},{"why":"Provides the 17 QA datasets used in the feasibility study and the QA training data for the speaker.","marker":"Khashabi et al. (2022)"},{"why":"Supplies T5-small, the speaker model, the pretrained weights for fine-tuning, and the C4 subset for from-scratch bootstrapping.","marker":"Raffel et al. (2023)"},{"why":"Supplies the SQuAD 2.0 validation set used as the fine-tuning task data for the summarization game.","marker":"Rajpurkar et al. (2018)"},{"why":"Defines ROUGE-L, the overlap metric that computes the reward from the listener's answer against the ground truth.","marker":"Lin (2004)"},{"why":"Supplies PPO, the reinforcement-learning algorithm that updates the speaker from the reward signal.","marker":"Schulman et al. (2017)"},{"why":"Supplies BLiMP, the minimal-pair benchmark used to test whether fine-tuning improved grammatical knowledge.","marker":"Warstadt et al. (2020)"},{"why":"Motivates the surprisal bottleneck by treating a word's negative log probability as comprehension difficulty.","marker":"Levy (2008)"}],"fun_headline_variants":["Being understood changes talk, not grammar","Communicative reward shifts LM style, grammar flat","Understanding reward changes LM output, not grammar","Grammar stays flat as LMs learn to be understood"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the frozen question-answering listener's answer quality, scored by ROUGE-L against a ground-truth answer, is a faithful proxy for human communicative success and, in particular, for grammaticality rather than for general text quality.","fun_headline_variants_meta":{"raw":{"variants":["Being understood changes talk, not grammar","Communicative reward shifts LM style, grammar flat","Understanding reward changes LM output, not grammar","Grammar stays flat as LMs learn to be understood"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001041,"raw_usage":{"total_tokens":4349,"prompt_tokens":889,"completion_tokens":3460,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":3403}},"tokens_in":505,"tokens_out":3460,"duration_ms":23213,"temperature":1.0,"reasoning_tokens":3403,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:51:43.111065+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Prepare pairs of summaries that use exactly the same words but differ only in word order—one grammatical, one scrambled—and measure the ROUGE-L reward the listener returns; if the reward does not drop when grammar is destroyed, the signal carries no grammatical information. A complementary experiment is to train the speaker with a listener that ignores word order entirely and check whether BLiMP scores still fail to move.","supporting_citations":[{"cited_title":"Unifiedqa: Crossing format boundaries with a single qa system, 2020","cited_arxiv_id":null,"evidence_quote":"Supplies the UnifiedQA listener; the listener's generated answers define communicative success and determine the reward."},{"cited_title":"Unifiedqa-v2: Stronger generalization via broader cross-format training, 2022","cited_arxiv_id":null,"evidence_quote":"Provides the 17 QA datasets used in the feasibility study and the QA training data for the speaker."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies BLiMP, the minimal-pair benchmark used to test whether fine-tuning improved grammatical knowledge."}],"review_version":1}