Pith. sign in

REVIEW 3 major objections 5 minor 19 references

Playing log(N)-Questions over Sentences

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Two neural agents can learn to play a $\log(N)$-question game over sentences, and the training signal determines whether their questions are meaningful.

desk verdict A clever game and dataset, but the learned answerer means the accuracy numbers may show coordination, not semantic reasoning — worth peer review with heavy revision. read the letter →

arxiv 1908.04660 v1 pith:F52XWQD7 submitted 2019-08-13 cs.CL

Peter Potash , Kaheer Suleman This is my paper · ORCID
classification cs.CL
keywords two-agentgamelog(N)-questionssentenceguessinggoal-orienteddialoguesplittingwordsGumbelSoftmaxinterpretabilityend-to-endtraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces a two-agent game in which a questioner has $\log(N)$ yes/no questions to identify which of $N$ sentences an answerer has secretly chosen, and trains both agents end-to-end. The central claim is that the agents can learn to play the game from the game signal alone, reaching 82\u201385% accuracy on sets of four sentences while producing questions that are almost never the \u201csplitting words\u201d a human would use. Adding a grounding loss that rewards splitting words makes the questions meaningful but lowers accuracy, and pretraining the answerer recovers part of that loss. If correct, the work provides a testbed for whether neural agents can learn information-seeking behavior under an explicit communication budget, and for what such agents actually mean by a \u201cgood question.\u201d

What carries the argument

The apparatus is a pair of differentiable agents speaking over a discrete channel. The questioner (Q-Bot) encodes each candidate sentence with a bidirectional LSTM, fuses the representations with a memory network that doubles as the guess distribution, and emits a natural-language question token-by-token through a decoder trained with the Gumbel Softmax estimator. On receiving the answerer\u2019s single bit, the Q-Bot updates both the fused game state (combiner adjustor) and the per-sentence vector magnitudes (sentence gater, which scales each sentence by the probability the response assigns to it). The answerer (A-Bot) encodes only the target sentence, encodes the question, and converts their concatenation into a binary response through a straight-through estimator. The whole system is trained on the cross-entropy of the final guess, optionally with an auxiliary loss that rewards emitting a splitting word in the first round.

What would settle it

Fix the answerer to a non-learned oracle (for example, a human or a lexically defined rule that answers \u201cyes\u201d iff the question is true of the target) and rerun the trained questioner; if accuracy drops to near chance or the first-round words no longer split the candidate sentences, the original high accuracy was an artifact of the two learned agents fitting each other.

Watch

Extended reading notes

Core claim

On its own terms, the paper\u2019s discovery is that a questioner trained with no supervision other than whether its final guess is correct can adapt its word choices to the answerer\u2019s bit responses and win the game on over 80% of held-out four-sentence instances. The same questioner, however, does not ask questions that line up with the splitting-word heuristic: it emits such words in only 0.01% of first rounds. When a second loss term explicitly rewards splitting words, the questioner\u2019s first question becomes interpretable (70.7\u201372.7% splitting-word prediction), but game accuracy drops by roughly 7\u201315 percentage points; training a model with the splitting-word loss down-weighted narrows the drop to about 3 points at the cost of lower interpretability. The paper also reports 80.0% accuracy on test sets without splitting words, suggesting the learned strategy is not purely lexical, and a word-embedding baseline that finds splitting words via entropy minimization reaches 84% accuracy.

Load-bearing premise

The whole evaluation assumes the learned answerer answers truthfully on the basis of the target sentence; if the questioner learns to exploit the answerer\u2019s regularities instead, the accuracy numbers do not measure reasoning about sentence meaning.

Editorial extensions

If this is right

  • Game success alone is insufficient to measure whether a questioner has learned semantically meaningful information-seeking; a 84.7%-accurate model can emit essentially random-looking single tokens (0.01% splitting words).
  • Rewarding a simple human strategy (splitting words) during training produces interpretable questions, with a cost in accuracy that depends on how the two losses are weighted.
  • Pretraining the answerer on question-token-in-target answers reduces the accuracy cost of interpretable questioning, improving splitting-word prediction to 72.7% while keeping game accuracy at 74.7%.
  • The trained agents transfer to sentence sets that contain no splitting words with 80.0% accuracy, suggesting the game encourages a strategy that is not limited to surface word matching.
  • A pure word-embedding heuristic that selects the token best separating two sentence pairs reaches 84% splitting-word accuracy, marking a strong non-neural baseline for the game\u2019s first round.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The absence of a fixed truthful oracle is the main blind spot: with a learned answerer, high game accuracy may measure the questioner\u2019s ability to exploit the answerer\u2019s learned regularities. Testing the same questioner against a fixed oracle or human respondent would settle whether the questions are genuinely informative.
  • The game could be reused as a self-supervised probe for sentence representations: a word that reliably splits a candidate set is an interpretable feature of the encoder\u2019s notion of semantic difference, potentially giving a diagnostic for what LSTM or memory-network encoders capture.
  • The same architecture generalizes in principle to larger $N$ by scaling to $\log(N)$ rounds and to other discrete channels; the main practical requirement is a trustworthy answerer, since the joint-training shortcut becomes more tempting as the communication channel narrows.
  • The entropy-minimizing word-embedding baseline in Section 4.1 suggests a cheap, deterministic alternative to trained questioners: pick the vocabulary token whose embedding distribution best separates the candidates into two equal groups, which could anchor future evaluations of what makes a question meaningful.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a cooperative two-agent game in which a questioner (Q-Bot) must identify one target sentence among N candidates using log(N) yes/no questions, and introduces an end-to-end differentiable model in which both agents are trained jointly via Gumbel-Softmax discrete communication and a straight-through binary responder. The authors construct a 108k-set corpus from STS/NLI passages, split by the presence of splitting words (words occurring in exactly half of the candidate sentences), and compare training with game-only loss, a secondary splitting-word grounding loss, and pretraining of the answerer. The main experimental claims are that the agents reach roughly 80% game accuracy, that game-only training does not produce splitting-word questions, and that grounding on splitting words improves question interpretability at a measurable accuracy cost, which pretraining partly recovers.

Significance. The result is potentially significant for goal-oriented dialogue and interpretable QA: it demonstrates a tractable setup in which discrete communication can be learned end-to-end over sentence sets, and it makes a concrete, falsifiable claim about a trade-off between task accuracy and human-interpretable questions. The corpus construction and the splitting-word heuristic are useful resources for future work. However, the significance is conditional: the main evidence that the system reasons over sentence semantics would require an answerer whose answers are truthful by construction (or a human answerer), because the reported game accuracy alone could reflect a learned code between the two agents rather than semantic grouping.

major comments (3)
  1. [§3, 'The Answerer (A-Bot)'; §3.1] Section 3 specifies that the A-Bot is a learned neural responder trained jointly with the Q-Bot, and §3.1 states that the training signal is the Q-Bot's guess accuracy plus optional auxiliary losses. This does not match the game definition in §1 and Figure 1, where the answerer is truthful and knows only the target sentence. With N=4 and two one-bit answers, the pair has enough channel capacity to transmit the target's identity through the question tokens and/or responses, so the 80–85% game accuracies in Table 1 could be achieved by coordination rather than by splitting candidate sets along semantic attributes. The paper should evaluate at least one condition with a fixed truthful oracle (e.g., a token-membership rule) or with human answers, and, for the learned A-Bot setting, report a held-out truthfulness measure for the A-Bot responses. Without this, the central claim about reasoning over sentence semantics is not established.
  2. [§4, Table 1] All numbers in Table 1 are single runs on a single 80/10/10 split, with no error bars, confidence intervals, or significance tests. This matters because the trade-off claim rests on small differences between training variants (e.g., 69.8%, 74.7%, and 79.1%), and because a single unlucky seed could change the ordering of the question-length settings (82.2%, 84.7%, and 77.2%). The authors should report means and standard deviations over at least 3–5 seeds, and a paired statistical comparison for the key rows, together with the number of test game instances used to compute each percentage.
  3. [§3.1, 'Training'; §4, Table 1] The 'SW Pred' column is not an independent evaluation metric for the rows trained with loss=sw,game: in those settings, the secondary cross-entropy loss explicitly optimizes the Q-Bot's first-round question to contain a splitting word, so high SW Pred partly measures fit to the training objective rather than an emergent property. The qualitative example in Figure 4 is helpful but is only one sentence set. I recommend adding a held-out human evaluation of question meaningfulness, or a grounding metric not used in any training loss (for example, whether the predicted first question separates the remaining candidates under a fixed oracle), and showing more than one qualitative instance.
minor comments (5)
  1. [§1 and Table 1] There are several typos (e.g., 'prediciton' in the Introduction), and the final row of Table 1 does not state its game setting; the caption should specify that it uses QL=1, loss=game.
  2. [§2, Dataset] The recursive sampling procedure should state whether duplicate sentences are allowed or rejected; if a passage can appear more than once in a set, the notion of a single target sentence becomes ambiguous.
  3. [§3, 'Sentence Gater'] The re-normalization of w_i when r=1 is not fully specified; please clarify whether the sentence representations e_i^{t+1} are re-normalized and how the offset gamma is chosen or tuned.
  4. [§4, paragraph 1] The sentence 'Our code is available online *PLACEHOLDER*' contains a placeholder; either provide a permanent link or remove the sentence.
  5. [§4.2 and Figure 4] The qualitative example would be much easier to evaluate if the figure marked the target sentence and displayed the exact questions asked by both systems for each of the four game instances, since this example is central to the interpretability claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the game loss, the SW-training signal, and the held-out evaluation are distinct, and no load-bearing step reduces to its own inputs by construction.

full rationale

The reported game accuracy is not circular: the Q-Bot is trained with categorical cross-entropy against the true answer sentence (Section 3.1, “The loss is the categorical cross-entropy over all sentences, with the target class being the answer sentence”), and held-out accuracy is measured on new sentence sets with each sentence used as the target (Section 4). The SW-prediction result is likewise an explicitly supervised quantity: the loss=sw,game setting adds a cross-entropy loss whose target is a SW, so the model’s SW output on held-out sets is a generalization measurement of that training signal, not a “prediction” derived from the game loss alone. The paper does not claim SWs emerge from game-only training; indeed the game-only SW prediction is 0.01% (Table 1). No load-bearing result is justified by self-citation: the references (GloVe, BLSTM, Gumbel-Softmax, Memory Networks, etc.) are standard external methods, and the paper contains no self-citations by the authors. The concern that the jointly trained A-Bot could coordinate two response bits to encode the target identity is a validity threat to the “reasoning” interpretation, but it is not an equation-level circularity in the paper’s derivation chain, because the final guess is still scored against the true target and the paper’s equations do not define the target in terms of the model output or vice versa. Derivation of the empirical claims is therefore self-contained, and no circular step can be exhibited from the stated equations or training objectives.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper does not introduce new physical or conceptual entities beyond the game itself; the main assumptions are about dataset utility, answerer reliability, and the validity of the SW proxy. The free parameters are the question length, the gating offset, and the SW loss weighting, none of which are given precise values or justified independently.

free parameters (3)
  • Question length L = 1, 5, 10 (tested)
    Chosen by experiment; affects game accuracy, with no principled selection criterion stated.
  • Gating offset gamma
    Introduced in Equation 4 to control the magnitude update of sentence representations; the value is not reported.
  • Secondary loss weighting for SW grounding
    The trade-off between game loss and SW loss is adjusted via a weighting, but the weights are not specified.
assumptions (3)
  • domain assumption The recursively sampled STS/NLI pairs yield sentence sets of four where a target sentence can be identified by yes/no questions.
    Dataset construction in Section 2 uses transitive sampling from semantic similarity and NLI pairings; there is no validation that these sets are game-suitable.
  • domain assumption The jointly trained A-Bot provides a reliable oracle that answers according to target-sentence semantics.
    Section 3 defines the A-Bot as giving a single bit; because both agents are trained simultaneously, the A-Bot may be gamed rather than truthful.
  • domain assumption Splitting Words are a valid proxy for meaningful questions.
    Section 2 states 'we use SWs as a heuristic for a simple human strategy'; the central interpretability conclusion relies on this proxy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Playing log(N)-Questions over Sentences." pith.science (2026). https://pith.science/paper/F52XWQD7

@misc{pith2026190804660,
  author       = {Pith},
  title        = {Pith review of: Playing log(N)-Questions over Sentences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F52XWQD7}},
  note         = {Machine review of arXiv:1908.04660}
}
read the original abstract

We propose a two-agent game wherein a questioner must be able to conjure discerning questions between sentences, incorporate responses from an answerer, and keep track of a hypothesis state. The questioner must be able to understand the information required to make its final guess, while also being able to reason over the game's text environment based on the answerer's responses. We experiment with an end-to-end model where both agents can learn simultaneously to play the game, showing that simultaneously achieving high game accuracy and producing meaningful questions can be a difficult trade-off.

Figures

Figures reproduced from arXiv: 1908.04660 by the authors.

Figure 1
Figure 1. This is an example of two humans playing our [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Inside the Q-Bot. cot is defined in Equation 1, k 1,...,L t in Equation 2, cat in Equation 3, and ht+1 in Equation 6 [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Inside the Q-Bot’s memory cell, which holds [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Given sentences at the bottom of the Figure, [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 11 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Eneko Agirre, Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre. 2012. Semeval-2012 task 6: A pilot on semantic textual similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Ev...

    work page 2012

  4. [4]

    Eneko Agirre, Aitor Gonzalez-Agirre, Inigo Lopez-Gazpio, Montse Maritxalar, German Rigau, and Larraitz Uria. 2016. Semeval-2016 task 2: Interpretable semantic textual similarity. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 512--524

    work page 2016

  5. [5]

    Yoshua Bengio, Nicholas L \'e onard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432

    arXiv 2013

  6. [6]

    Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632--642

    2015

  7. [7]

    Abhishek Das, Satwik Kottur, Jos \'e MF Moura, Stefan Lee, and Dhruv Batra. 2017. Learning cooperative visual dialog agents with deep reinforcement learning. In Computer Vision (ICCV), 2017 IEEE International Conference on, pages 2970--2979. IEEE

    work page 2017

  8. [8]

    Alex Graves and J \"u rgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks, 18(5-6):602--610

    2005

Show all 19 references
  1. [9]

    Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparametrization with gumble-softmax. In International Conference on Learning Representations 2017

    2017

  2. [10]

    Harsh Jhamtani and Taylor Berg-Kirkpatrick. 2018. Learning to describe differences between pairs of similar images. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4024--4034

    2018

  3. [11]

    Yining Li, Chen Huang, Xiaoou Tang, and Chen Change Loy. 2017. Learning to disambiguate by asking discriminative questions. In Proceedings of the IEEE International Conference on Computer Vision, pages 3419--3428

    2017

  4. [12]

    Zhongxin Liu, Xin Xia, Ahmed E Hassan, David Lo, Zhenchang Xing, and Xinyu Wang. 2018. Neural-machine-translation-based commit message generation: how far are we? In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pages 373--384. ACM

    2018

  5. [13]

    I \ n igo Lopez-Gazpio, Montse Maritxalar, Aitor Gonzalez-Agirre, German Rigau, Larraitz Uria, and Eneko Agirre. 2017. Interpretable semantic textual similarity: Finding and explaining differences between sentences. Knowledge-Based Systems, 119:186--199

    2017

  6. [14]

    Charles Sanders Peirce. 1901. On the logic of drawing history from ancient documents, especially from testimonies. The Essential Peirce, 1893-1913, 2:75--114

    1901

  7. [15]

    Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532--1543

    2014

  8. [16]

    Mrinmaya Sachan and Eric Xing. 2018. Self-training for jointly learning to ask and answer questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages...

    2018

  9. [17]

    Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in neural information processing systems, pages 2440--2448

    2015

  10. [18]

    Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association of Computational Linguistics, 6:287--302

    2018

  11. [19]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA : A dataset for diverse, explainable multi-hop question answering. In Proceedings of the Conference on Empirical Methods in Natural Language ...

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.