{"id":"92073296-8d6e-469d-90da-4be9d4afd105","arxiv_id":"1908.04660","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Two neural agents can play a log(4)-questions sentence-guessing game with about 80 percent accuracy, but question meaningfulness and game accuracy are in tension.","lead":"The paper introduces a two-agent guessing game over four sentences, where one agent asks yes/no questions and another answers, so the asker must identify the hidden sentence. A trained system reaches about 80 percent accuracy, but its questions are only human-meaningful when trained with an extra signal that reduces accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The learned A-Bot can encode the target index in its two yes/no answers, so Table 1's game accuracy does not by itself show that the Q-Bot reasons over sentence semantics; a fixed-oracle retraining is needed.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the jointly trained A-Bot means game success does not cleanly demonstrate semantic reasoning. I agree, and the concern can be sharpened: for N=4, two answer bits are exactly sufficient to transmit the target index, so a private code is not just possible but information-theoretically as efficient as the intended binary-search strategy. The paper's qualitative examples and SW grounding results are suggestive, but they do not rule out such a code because the only automatic 'meaningfulness' signal is the SW heuristic, which is also the target of the secondary loss; the reported trade-off between game accuracy and SW prediction is therefore partly built into the training objective. The manuscript also contains a code placeholder, so reproducibility of the exact numbers cannot currently be checked. These are addressable issues: a fixed-oracle or human-answerer evaluation, a separate measure of question meaningfulness, released code, and variance estimates would materially strengthen the claims. Since the reader's conditional verdict already requires these changes, my stress-test pass does not move the verdict: it remains CONDITIONAL.","tokens_in":5694,"tokens_out":7059,"duration_ms":82045,"concrete_test":"Using the same data splits, Q-Bot architecture, and game loss as Section 3, replace the learned A-Bot with a frozen oracle for the QL=1 setting: the oracle answers yes iff the single question token appears in the target sentence, and no otherwise. Retrain the Q-Bot end-to-end and report held-out game accuracy with the same protocol as Table 1. If accuracy remains in the 80-85% range, the Q-Bot can play against a truthful lexical answerer and the joint-training-code explanation is not the sole driver. If accuracy falls substantially toward the 25% chance level or below the roughly 70% achieved by SW-grounded models, then the original high accuracy depended on the learned A-Bot's encoded responses, and Table 1 cannot be read as evidence of sentence reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the intended game, the answerer is truthful and knows only the target sentence. In Section 3, however, the A-Bot is a learned neural responder trained jointly with the Q-Bot, with no constraint enforcing truthful answers. With N=4 and two yes/no questions, the two response bits have exactly enough capacity to transmit the target's identity: the Q-Bot can emit two distinguishable tokens, the A-Bot can map its target-sentence encoding to the two bits, and the Q-Bot can decode them. Under this strategy, high game accuracy reflects coordination, not the ability to ask questions that group candidates by shared attributes. The pretraining variant (marked *) pretrains the A-Bot to answer whether a question token is in the target, but joint fine-tuning continues afterward, and the paper reports no measure of A-Bot truthfulness at test time. The 80.0% accuracy on the non-SW test set does not resolve this, because a jointly learned code can transfer to unseen sentence sets through the sentence encoder's regularities. Consequently, the central claim that the Q-Bot narrows down a target sentence by asking meaningful questions is not established by the reported game accuracy alone; the evaluation conflates a cooperative referential game with the described truthful-answerer game.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a cooperative two-agent game in which a questioner (Q-Bot) must identify one target sentence among N candidates using log(N) yes/no questions, and introduces an end-to-end differentiable model in which both agents are trained jointly via Gumbel-Softmax discrete communication and a straight-through binary responder. The authors construct a 108k-set corpus from STS/NLI passages, split by the presence of splitting words (words occurring in exactly half of the candidate sentences), and compare training with game-only loss, a secondary splitting-word grounding loss, and pretraining of the answerer. The main experimental claims are that the agents reach roughly 80% game accuracy, that game-only training does not produce splitting-word questions, and that grounding on splitting words improves question interpretability at a measurable accuracy cost, which pretraining partly recovers.","tokens_in":5938,"tokens_out":5752,"duration_ms":59575,"significance":"The result is potentially significant for goal-oriented dialogue and interpretable QA: it demonstrates a tractable setup in which discrete communication can be learned end-to-end over sentence sets, and it makes a concrete, falsifiable claim about a trade-off between task accuracy and human-interpretable questions. The corpus construction and the splitting-word heuristic are useful resources for future work. However, the significance is conditional: the main evidence that the system reasons over sentence semantics would require an answerer whose answers are truthful by construction (or a human answerer), because the reported game accuracy alone could reflect a learned code between the two agents rather than semantic grouping.","major_comments":[{"comment":"Section 3 specifies that the A-Bot is a learned neural responder trained jointly with the Q-Bot, and §3.1 states that the training signal is the Q-Bot's guess accuracy plus optional auxiliary losses. This does not match the game definition in §1 and Figure 1, where the answerer is truthful and knows only the target sentence. With N=4 and two one-bit answers, the pair has enough channel capacity to transmit the target's identity through the question tokens and/or responses, so the 80–85% game accuracies in Table 1 could be achieved by coordination rather than by splitting candidate sets along semantic attributes. The paper should evaluate at least one condition with a fixed truthful oracle (e.g., a token-membership rule) or with human answers, and, for the learned A-Bot setting, report a held-out truthfulness measure for the A-Bot responses. Without this, the central claim about reasoning over sentence semantics is not established.","section":"§3, 'The Answerer (A-Bot)'; §3.1"},{"comment":"All numbers in Table 1 are single runs on a single 80/10/10 split, with no error bars, confidence intervals, or significance tests. This matters because the trade-off claim rests on small differences between training variants (e.g., 69.8%, 74.7%, and 79.1%), and because a single unlucky seed could change the ordering of the question-length settings (82.2%, 84.7%, and 77.2%). The authors should report means and standard deviations over at least 3–5 seeds, and a paired statistical comparison for the key rows, together with the number of test game instances used to compute each percentage.","section":"§4, Table 1"},{"comment":"The 'SW Pred' column is not an independent evaluation metric for the rows trained with loss=sw,game: in those settings, the secondary cross-entropy loss explicitly optimizes the Q-Bot's first-round question to contain a splitting word, so high SW Pred partly measures fit to the training objective rather than an emergent property. The qualitative example in Figure 4 is helpful but is only one sentence set. I recommend adding a held-out human evaluation of question meaningfulness, or a grounding metric not used in any training loss (for example, whether the predicted first question separates the remaining candidates under a fixed oracle), and showing more than one qualitative instance.","section":"§3.1, 'Training'; §4, Table 1"}],"minor_comments":[{"comment":"There are several typos (e.g., 'prediciton' in the Introduction), and the final row of Table 1 does not state its game setting; the caption should specify that it uses QL=1, loss=game.","section":"§1 and Table 1"},{"comment":"The recursive sampling procedure should state whether duplicate sentences are allowed or rejected; if a passage can appear more than once in a set, the notion of a single target sentence becomes ambiguous.","section":"§2, Dataset"},{"comment":"The re-normalization of w_i when r=1 is not fully specified; please clarify whether the sentence representations e_i^{t+1} are re-normalized and how the offset gamma is chosen or tuned.","section":"§3, 'Sentence Gater'"},{"comment":"The sentence 'Our code is available online *PLACEHOLDER*' contains a placeholder; either provide a permanent link or remove the sentence.","section":"§4, paragraph 1"},{"comment":"The qualitative example would be much easier to evaluate if the figure marked the target sentence and displayed the exact questions asked by both systems for each of the four game instances, since this example is central to the interpretability claim.","section":"§4.2 and Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads like an early-stage workshop paper: the code placeholder, single-seed results, and narrow evaluation make it premature in its current form. The central issue for acceptance is the truthful-answerer evaluation; I would not recommend rejection because the fix is straightforward and the setup itself is a useful contribution. Please also consider whether the journal requires a stronger set of baselines for discrimination among four candidate sentences."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper deserves a referee, but the central claim — that the Q-Bot reasons over sentence semantics — is not established by the reported game accuracy, because the A-Bot is trained jointly and can act as a learned code rather than a truthful oracle.\n\nWhat's genuinely new: the log(N)-questions game for sentences, a 108k-set corpus built by recursive STS/NLI sampling, and a clean empirical demonstration of a trade-off between task success and questions that use splitting words. The architecture is standard (BLSTM, memory network, Gumbel softmax), but applying it to this setup is reasonable. The qualitative example is honest: without SW grounding the questions are semantically meaningless.\n\nThe soft spots are load-bearing. First, with N=4 and two yes/no answers, two bits can transmit the target's identity. The A-Bot, trained end-to-end, can map its target encoding to those bits; the Q-Bot can decode them. High game accuracy can therefore reflect coordinated communication, not the ability to group sentences by attributes. The paper never tests with a fixed truthful oracle or a human answerer, and the pretraining variant still fine-tunes jointly. Second, the SW metric is directly in the loss for the grounded models, so SW prediction partly measures fitting; the trade-off is real but the interpretability measure is not independent. Third, there are no error bars or significance tests in Table 1, and the code is a placeholder. These are all addressable.\n\nThe paper does not overclaim much — the abstract says 'difficult trade-off,' which holds — but the reasoning claim in the conclusion goes beyond the evidence. The dataset and game could be useful to others studying question generation and hypothesis tracking.\n\nMy take: send it to review, but the referee should demand a fixed-oracle baseline or a human-subject experiment, plus code and data release. If the authors can show the game can be played with a truthful answerer, the result becomes solid. Until then, read the accuracy numbers as measuring coordination, not semantics.","headline":"A clever game and dataset, but the learned answerer means the accuracy numbers may show coordination, not semantic reasoning — worth peer review with heavy revision.","tokens_in":6460,"tokens_out":2377,"would_cite":false,"duration_ms":24947,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two neural agents can learn to play a $\\log(N)$-question game over sentences, and the training signal determines whether their questions are meaningful.","keywords":["two-agent game","log(N)-questions","sentence guessing","goal-oriented dialogue","splitting words","Gumbel Softmax","interpretability","end-to-end training"],"falsifier":"Fix the answerer to a non-learned oracle (for example, a human or a lexically defined rule that answers \\u201cyes\\u201d iff the question is true of the target) and rerun the trained questioner; if accuracy drops to near chance or the first-round words no longer split the candidate sentences, the original high accuracy was an artifact of the two learned agents fitting each other.","tokens_in":5483,"feed_emoji":"❓","tokens_out":7495,"duration_ms":72271,"temperature":0.7,"pith_summary":"This paper introduces a two-agent game in which a questioner has $\\log(N)$ yes/no questions to identify which of $N$ sentences an answerer has secretly chosen, and trains both agents end-to-end. The central claim is that the agents can learn to play the game from the game signal alone, reaching 82\\u201385% accuracy on sets of four sentences while producing questions that are almost never the \\u201csplitting words\\u201d a human would use. Adding a grounding loss that rewards splitting words makes the questions meaningful but lowers accuracy, and pretraining the answerer recovers part of that loss. If correct, the work provides a testbed for whether neural agents can learn information-seeking behavior under an explicit communication budget, and for what such agents actually mean by a \\u201cgood question.\\u201d","feed_headline":"Neural agents learn to play a sentence-guessing yes/no game","feed_subtitle":"Trained with only game success they win 80%+ of rounds; adding meaningful questions costs points.","key_machinery":"The apparatus is a pair of differentiable agents speaking over a discrete channel. The questioner (Q-Bot) encodes each candidate sentence with a bidirectional LSTM, fuses the representations with a memory network that doubles as the guess distribution, and emits a natural-language question token-by-token through a decoder trained with the Gumbel Softmax estimator. On receiving the answerer\\u2019s single bit, the Q-Bot updates both the fused game state (combiner adjustor) and the per-sentence vector magnitudes (sentence gater, which scales each sentence by the probability the response assigns to it). The answerer (A-Bot) encodes only the target sentence, encodes the question, and converts their concatenation into a binary response through a straight-through estimator. The whole system is trained on the cross-entropy of the final guess, optionally with an auxiliary loss that rewards emitting a splitting word in the first round.","core_discovery":"On its own terms, the paper\\u2019s discovery is that a questioner trained with no supervision other than whether its final guess is correct can adapt its word choices to the answerer\\u2019s bit responses and win the game on over 80% of held-out four-sentence instances. The same questioner, however, does not ask questions that line up with the splitting-word heuristic: it emits such words in only 0.01% of first rounds. When a second loss term explicitly rewards splitting words, the questioner\\u2019s first question becomes interpretable (70.7\\u201372.7% splitting-word prediction), but game accuracy drops by roughly 7\\u201315 percentage points; training a model with the splitting-word loss down-weighted narrows the drop to about 3 points at the cost of lower interpretability. The paper also reports 80.0% accuracy on test sets without splitting words, suggesting the learned strategy is not purely lexical, and a word-embedding baseline that finds splitting words via entropy minimization reaches 84% accuracy.","pith_inferences":["The absence of a fixed truthful oracle is the main blind spot: with a learned answerer, high game accuracy may measure the questioner\\u2019s ability to exploit the answerer\\u2019s learned regularities. Testing the same questioner against a fixed oracle or human respondent would settle whether the questions are genuinely informative.","The game could be reused as a self-supervised probe for sentence representations: a word that reliably splits a candidate set is an interpretable feature of the encoder\\u2019s notion of semantic difference, potentially giving a diagnostic for what LSTM or memory-network encoders capture.","The same architecture generalizes in principle to larger $N$ by scaling to $\\log(N)$ rounds and to other discrete channels; the main practical requirement is a trustworthy answerer, since the joint-training shortcut becomes more tempting as the communication channel narrows.","The entropy-minimizing word-embedding baseline in Section 4.1 suggests a cheap, deterministic alternative to trained questioners: pick the vocabulary token whose embedding distribution best separates the candidates into two equal groups, which could anchor future evaluations of what makes a question meaningful."],"forward_implications":["Game success alone is insufficient to measure whether a questioner has learned semantically meaningful information-seeking; a 84.7%-accurate model can emit essentially random-looking single tokens (0.01% splitting words).","Rewarding a simple human strategy (splitting words) during training produces interpretable questions, with a cost in accuracy that depends on how the two losses are weighted.","Pretraining the answerer on question-token-in-target answers reduces the accuracy cost of interpretable questioning, improving splitting-word prediction to 72.7% while keeping game accuracy at 74.7%.","The trained agents transfer to sentence sets that contain no splitting words with 80.0% accuracy, suggesting the game encourages a strategy that is not limited to surface word matching.","A pure word-embedding heuristic that selects the token best separating two sentence pairs reaches 84% splitting-word accuracy, marking a strong non-neural baseline for the game\\u2019s first round."],"supporting_citations":[{"why":"Supplies the semantic-textual-similarity passages used to sample pairs of related sentences for building the game corpus.","marker":"(Agirre et al., 2012)"},{"why":"Supplies the natural-language-inference passages used to sample pairs of sentences for building the game corpus.","marker":"(Bowman et al., 2015)"},{"why":"Provides the Gumbel Softmax estimator that allows discrete question tokens to be trained by backpropagation.","marker":"(Jang et al., 2017)"},{"why":"Provides the straight-through estimator used to backpropagate through the answerer\\u2019s discrete yes/no response.","marker":"(Bengio et al., 2013)"},{"why":"Defines the memory network used as the questioner\\u2019s sentence combiner and guess distribution.","marker":"(Sukhbaatar et al., 2015)"},{"why":"Initializes the word embeddings and fixes the shared 20k-token vocabulary used by both agents.","marker":"(Pennington et al., 2014)"},{"why":"Defines the bidirectional LSTM architecture used to encode sentences and questions.","marker":"(Graves and Schmidhuber, 2005)"}],"fun_headline_variants":["Questioner wins sentence game but asks few splitting words","Training goal trade-off: game accuracy vs question interpretability","Neural questioner: winning over 80% but questions lack logic","Sentence-guessing game: interpretable questions cost accuracy","Questioner learns to win game, not to ask meaningful questions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole evaluation assumes the learned answerer answers truthfully on the basis of the target sentence; if the questioner learns to exploit the answerer\\u2019s regularities instead, the accuracy numbers do not measure reasoning about sentence meaning.","fun_headline_variants_meta":{"raw":{"variants":["Questioner wins sentence game but asks few splitting words","Training goal trade-off: game accuracy vs question interpretability","Neural questioner: winning over 80% but questions lack logic","Sentence-guessing game: interpretable questions cost accuracy","Questioner learns to win game, not to ask meaningful questions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000178,"raw_usage":{"total_tokens":1240,"prompt_tokens":834,"completion_tokens":406,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":323}},"tokens_in":450,"tokens_out":406,"duration_ms":3805,"temperature":1.0,"reasoning_tokens":323,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:35:16.178389+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix the answerer to a non-learned oracle (for example, a human or a lexically defined rule that answers \\u201cyes\\u201d iff the question is true of the target) and rerun the trained questioner; if accuracy drops to near chance or the first-round words no longer split the candidate sentences, the original high accuracy was an artifact of the two learned agents fitting each other.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the semantic-textual-similarity passages used to sample pairs of related sentences for building the game corpus."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Gumbel Softmax estimator that allows discrete question tokens to be trained by backpropagation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the memory network used as the questioner\\u2019s sentence combiner and guess distribution."}],"review_version":1}