Pith. sign in

REVIEW 4 major objections 4 minor 37 references

Let's Ask Again: Refine Network for Automatic Question Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Refining a draft question in a second decoding pass, with attention to both the passage and the draft, makes generated questions more fluent, complete, and answerable across three question-generation benchmarks.

desk verdict RefNet is a useful two-pass QG architecture with real empirical support, but the training-time handling of the draft is under-specified and the abstract overstates the passage-level gains. read the letter →

arxiv 1909.05355 v1 pith:NRA2U37S submitted 2019-08-31 cs.CL cs.AI

Preksha Nema , Akash Kumar Mohankumar , Mitesh M. Khapra , Balaji Vasan Srinivasan , Balaraman Ravindran This is my paper · ORCID
classification cs.CLcs.AI
keywords automaticquestiongenerationsequence-to-sequencetwo-passdecodingdualattentionanswer-awareREINFORCEfluencyandanswerabilityrewards
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that automatic question generation should not be a single-pass process: the model should first write a rough draft and then revise it, much like a human writer. It proposes RefNet, a sequence-to-sequence model with two decoders, where the second decoder reads the first decoder's output and attends to both the original passage and the draft before emitting the final question. Across SQuAD, HOTPOT-QA, and DROP, RefNet reports BLEU-4 gains over single-pass models. A variant, Reward-RefNet, trains the second decoder to make drafts more fluent, more answerable, or more original.

What carries the argument

The load-bearing mechanism is the dual attention network in the refinement decoder. After the preliminary decoder produces a draft, the refinement decoder conditions each output word on three things: the answer-aware passage representation, an attention-weighted summary of the draft's word embeddings, and the answer representation. This lets the second decoder revisit the passage for missing context while also seeing where the draft went wrong, and the shared output projection keeps the two passes aligned in vocabulary space.

What would settle it

Run a controlled experiment where the refinement decoder is trained with reference questions as its draft input and then applied to first-decoder drafts at test time; if BLEU-4 collapses relative to a model trained on first-decoder drafts, the reported RefNet gain is inflated by train-test mismatch.

Watch

Extended reading notes

Core claim

The central claim is that a second-pass refinement step, guided by dual attention over the passage and the draft, repairs the incompleteness and grammar errors that single-pass encoders leave in generated questions. On BLEU-4, RefNet beats the prior state of the art by 12.30% on SQuAD sentence-level question generation, 9.74% on HOTPOT-QA, 17.48% on DROP, and 3.71% on SQuAD passage-level generation, and it also beats its own single-decoder baseline by 7.83%, 7.57%, 15.25%, and 3.85% in those same settings. Human evaluation reported by the paper confirms that the refinement decoder produces more fluent, complete, and answerable questions in most pairwise comparisons.

Load-bearing premise

The refinement decoder's reported gains depend on it being trained on drafts that look like what the preliminary decoder actually produces; if training instead uses reference questions, the model could be learning to polish a perfect input rather than repair a draft.

Editorial extensions

If this is right

  • If the refinement gain is real, adding a second decoder to an existing sequence-to-sequence question generator is a cheap architectural upgrade: keep the single-pass model as the first decoder, reuse the encoder, and let the second decoder revise the draft.
  • The first decoder also improves when the second decoder is present, even before the dual attention path is added, which suggests that gradient feedback through the shared encoder and output layer is part of the benefit.
  • Reward signals can be targeted: rewarding improvements in fluency raises BLEU-4, rewarding answerability raises the QBLEU score, and rewarding low n-gram overlap with the passage pushes generated questions toward originality.
  • The gains on HOTPOT-QA and DROP suggest the refinement idea helps beyond simple span-based SQuAD questions, including multi-hop and discrete-reasoning settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test the paper leaves open is whether the refinement decoder is trained on drafts produced by the first decoder or on reference questions; if training uses reference questions, part of the reported gain could come from polishing an already-correct input rather than repairing a draft.
  • The draft-and-revise structure could transfer to other text-generation tasks, such as summarization, simplification, or dialogue response generation, where single-pass outputs also tend to be incomplete drafts.
  • Refining over the draft's word embeddings has a ceiling: it can repair wording and reorder content, but it cannot introduce facts that are absent from both the passage and the draft, so the reported originality gains are limited to rephrasing.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes RefNet, a two-decoder sequence-to-sequence model for answer-aware question generation. A preliminary decoder generates an initial draft question, and a refinement decoder conditions on that draft through a dual attention mechanism over the passage/answer representation and the draft, producing a revised question. A variant, Reward-RefNet, fine-tunes the refinement decoder with REINFORCE, using the difference between the rewards of the refined and preliminary outputs as the learning signal. The authors evaluate on SQuAD (sentence and passage level), HOTPOT-QA, and DROP, reporting improvements over their EAD baseline and prior work across BLEU, ROUGE-L, METEOR, and QBLEU, and they supplement the automatic scores with human evaluations on fluency, completeness, and answerability. They also analyze the interaction between the two decoders and introduce an originality-focused reward.

Significance. If the refinement mechanism is genuinely responsible for the observed gains, this paper makes a useful contribution: it gives a simple architectural recipe, two decoders plus dual attention, that consistently improves over a single-pass baseline on several datasets, and it provides an ablation showing that the direct attention path to the preliminary decoder (A3) matters. The release of code, the multi-dataset evaluation, and the inclusion of human preference judgments are clear strengths. The main caveats are that the training-time data generation for the refinement decoder is underspecified, which is load-bearing for the central 'refinement' claim, and that some abstract-level claims exceed the numbers actually reported in Section 4.1. The QBLEU metric's shared provenance with the reward also weakens the automatic answerability evaluation, although human judgments partially mitigate this concern.

major comments (4)
  1. [Section 2.2, Section 3.2] Section 2.2 defines the refinement decoder's input as the initial draft Q-tilde, but the paper never states whether, at training time, Q-tilde is produced by the preliminary decoder or is the reference question under teacher forcing. Section 3.2 only says that the models are trained for 10 epochs with cross-entropy loss. If the refinement decoder was trained on reference questions but applied to preliminary-decoder outputs at inference, there is a train/test distribution mismatch, and the gains in Table 2 over EAD could reflect extra decoder capacity or the shared parameters shown in Table 4 rather than learning to repair drafts. The paper should specify the training-time generation of Q-tilde explicitly; if teacher forcing was used, the authors should verify the refinement claim by training on sampled drafts or by reporting an oracle/self-repair experiment.
  2. [Abstract, Section 4.1] The abstract claims that RefNet outperforms existing state-of-the-art methods by 7-16% on all datasets, but Section 4.1 reports a gain of only 3.71% over the previous state of the art on SQuAD (passage level) and 3.85% over EAD on the same setting. The abstract should be corrected to match the numbers in Table 2, or the claim should be qualified so that it does not overstate the passage-level result.
  3. [Section 2.3, Section 3.3, Table 6] The answerability reward used in Reward-RefNet and the QBLEU-4 metric used for evaluation are both taken from Nema and Khapra (2018), which shares authors with this submission. Because the same score is both optimized and reported as evidence of answerability improvement, the automatic result is not an independent confirmation of the reward's effect. The human preference data in Table 6 helps, but the paper should either report an independently defined answerability metric or present the QBLEU numbers as an in-family measure and rely on human evaluation for the answerability claim.
  4. [Section 4.1, Table 2, Table 4] The central comparative claim rests on single-point BLEU-4 differences, several of which are small, for example 16.99 versus 16.36 on SQuAD (passage level). No confidence intervals, bootstrap intervals, or significance tests are reported for the automatic metrics, and Table 4 similarly reports gaps of about half a BLEU point without any uncertainty estimate. Because the contribution is framed as a clear improvement over prior state of the art, the authors should report statistical reliability for at least the key comparisons, or clearly state the evaluation size so readers can judge the stability of these differences.
minor comments (4)
  1. [Section 2.4] Section 2.4 says the pointer-network and coverage mechanism from See et al. (2017) is adopted, but the paper does not explain how the pointer/copy distribution is integrated into the output equations of the two decoders; please add the relevant equations or a precise reference to an appendix.
  2. [Section 2.2] In the refinement decoder description, the context vector c_t from the passage and the draft attention vector g_t are both introduced in a way that makes it hard to tell which attention network, A2 or A3, computes which; the notation should be made consistent.
  3. [Section 3.1] For HOTPOT-QA and DROP, the preprocessing is described only at a high level; please state how supporting facts are concatenated into the passage, how the validation split is made, and how non-span answers such as yes/no are handled.
  4. [Section 4.2] The human evaluation reports majority labels but no inter-annotator agreement measure such as Fleiss' kappa; reporting agreement would strengthen the reliability of the preference percentages in Section 4.2 and Table 6.

Circularity Check

1 steps flagged · score 2.0 of 10

The main RefNet-versus-baseline claim rests on BLEU, ROUGE, METEOR, and human evaluation, so it is not circular; the only self-citation loop is the use of the authors' own QBLEU4 score both as the answerability reward in Reward-RefNet and as the answerability evaluation metric.

  1. other [Section 2.3 'Reward-RefNet' and Section 3.3 'Evaluation']
    "We use these fluency and answerability scores proposed by (Nema and Khapra, 2018) as reward signals. ... As mentioned, this reward r(.) can be the fluency score or answerability score as defined by (Nema and Khapra, 2018). ... We also quantify the answerability of our models using QBLEU4 (Nema and Khapra, 2018)."

    The answerability reward optimized by Reward-RefNet is the same QBLEU4-style score that is then reported as the answerability improvement, so the QBLEU4 gain is in part an optimized objective rather than an independent measurement. This is a mild self-referential loop, not a full circularity: the central RefNet claim is validated by BLEU, ROUGE, METEOR, and human preference, and the Reward-RefNet answerability claim is separately confirmed by annotators (70% preference). The QBLEU4 number alone should not be treated as independent evidence for the answerability reward.

full rationale

The derivation chain for RefNet is self-contained: the preliminary decoder and refinement decoder are trained with a standard cross-entropy loss on SQuAD, HOTPOT-QA, and DROP, and the main comparison against EAD and prior systems is made with BLEU, ROUGE-L, METEOR, and human evaluation. Those metrics do not depend on any parameter fitted by the model, and the human evaluation independently prefers RefNet over EAD on fluency, completeness, and answerability. The only overlap with the authors' prior work is QBLEU4 (Nema and Khapra, 2018), which is used both as the answerability reward for Reward-RefNet and as the answerability evaluation metric. That is a mild optimization/evaluation loop for the answerability variant, but it is not load-bearing for the paper's central claim, and the human preference results provide external support. The paper's silence on whether the refinement decoder is trained on drafts produced by the preliminary decoder or on reference questions is a reproducibility and validity concern about the refinement mechanism, but it is not a circularity: the claim could have failed empirically, and the paper's equations do not define the refinement metric in terms of the training signal. Overall, the central derivation is independent, and the only self-citation issue is minor and non-load-bearing.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No ad hoc fitted constants or new physical entities are introduced. The model relies on standard architectural choices and domain assumptions about the evaluation metrics and the effectiveness of attention mechanisms for refinement.

assumptions (3)
  • domain assumption BLEU, ROUGE, METEOR and QBLEU are valid proxies for question quality.
    The paper uses these as primary evaluation metrics. BLEU's validity for fluency is contested, and QBLEU is a heuristic from the same group as the paper.
  • domain assumption The dual attention over passage and initial draft is sufficient for the refinement decoder to correct draft errors.
    This is the architectural premise of RefNet; it is tested empirically but not derived from first principles.
  • domain assumption The answer-aware passage representation and answer encodings are sufficient to condition answer-specific question generation.
    Standard in AQG models; the paper does not formally justify that this representation captures all needed answer-question dependency.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Let's Ask Again: Refine Network for Automatic Question Generation." pith.science (2026). https://pith.science/paper/NRA2U37S

@misc{pith2026190905355,
  author       = {Pith},
  title        = {Pith review of: Let's Ask Again: Refine Network for Automatic Question Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NRA2U37S}},
  note         = {Machine review of arXiv:1909.05355}
}
read the original abstract

In this work, we focus on the task of Automatic Question Generation (AQG) where given a passage and an answer the task is to generate the corresponding question. It is desired that the generated question should be (i) grammatically correct (ii) answerable from the passage and (iii) specific to the given answer. An analysis of existing AQG models shows that they produce questions which do not adhere to one or more of {the above-mentioned qualities}. In particular, the generated questions look like an incomplete draft of the desired question with a clear scope for refinement. {To alleviate this shortcoming}, we propose a method which tries to mimic the human process of generating questions by first creating an initial draft and then refining it. More specifically, we propose Refine Network (RefNet) which contains two decoders. The second decoder uses a dual attention network which pays attention to both (i) the original passage and (ii) the question (initial draft) generated by the first decoder. In effect, it refines the question generated by the first decoder, thereby making it more correct and complete. We evaluate RefNet on three datasets, \textit{viz.}, SQuAD, HOTPOT-QA, and DROP, and show that it outperforms existing state-of-the-art methods by 7-16\% on all of these datasets. Lastly, we show that we can improve the quality of the second decoder on specific metrics, such as, fluency and answerability by explicitly rewarding revisions that improve on the corresponding metric during training. The code has been made publicly available \footnote{https://github.com/PrekshaNema25/RefNet-QG}

Figures

Figures reproduced from arXiv: 1909.05355 by the authors.

Figure 1
Figure 1. Our RefNet model with Preliminary and Refinement Decoder. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Generated Question Length Distribution for [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Originality Analysis: Plot of Q-BLEU score [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Attention plots for a) A1, b) A2, c) A3 re￾spectively Initial Generated Question: “What is the name of the oncogenic virus?” Refined Generated Question: “What is the name of the organism that causes cervical cancer?” Passage: “The antigens expressed by tumors have sev￾…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 29 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473

    arXiv 2014

  4. [4]

    Xinya Du and Claire Cardie. 2017. Identifying where to focus in reading comprehension for neural question generation. In EMNLP , pages 2067--2073. Association for Computational Linguistics

    work page 2017

  5. [5]

    Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. In ACL (1) , pages 1342--1352. Association for Computational Linguistics

    work page 2017

  6. [6]

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP : A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACL

    2019

  7. [7]

    Michael Heilman and Noah A. Smith. 2010. Good question! statistical ranking for question generation. In HLT-NAACL , pages 609--617. The Association for Computational Linguistics

    work page 2010

  8. [8]

    Sepp Hochreiter and J\" u rgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735--1780

    work page 1997

Show all 37 references
  1. [9]

    Unnat Jain, Ziyu Zhang, and Alexander G. Schwing. 2017. Creativity: Generating diverse questions using variational autoencoders. In CVPR , pages 5415--5424. IEEE Computer Society

    2017

  2. [10]

    Yanghoon Kim, Hwanhee Lee, Joongbo Shin, and Kyomin Jung. 2019. Improving neural question generation using answer separation. CoRR, abs/1809.02393

    2019 arXiv

  3. [11]

    Igor Labutov, Sumit Basu, and Lucy Vanderwende. 2015. Deep questions without deep understanding. In ACL (1) , pages 889--898. The Association for Computer Linguistics

    2015

  4. [12]

    Denkowski

    Alon Lavie and Michael J. Denkowski. 2009. The meteor metric for automatic evaluation of machine translation. Machine Translation, 23(2-3):105--115

    2009

  5. [13]

    Yikang Li, Nan Duan, Bolei Zhou, Xiao Chu, Wanli Ouyang, and Xiaogang Wang. 2017. Visual question generation as dual task of visual question answering. CoRR, abs/1709.07192

    2017 arXiv

  6. [14]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proc. ACL workshop on Text Summarization Branches Out, page 10

    2004

  7. [15]

    Nesbit, and Philip H

    David Lindberg, Fred Popowich, John C. Nesbit, and Philip H. Winne. 2013. Generating natural language questions to support learning on-line. In ENLG , pages 105--114. The Association for Computer Linguistics

    2013

  8. [16]

    Jack Mostow and Wei Chen. 2009. Generating instruction automatically for the reading strategy of self-questioning. In AIED , volume 200 of Frontiers in Artificial Intelligence and Applications, pages 465--472. IOS Press

    2009

  9. [17]

    Preksha Nema and Mitesh M. Khapra. 2018. Towards a better metric for evaluating question generation systems. CoRR, abs/1808.10192

    2018 arXiv

  10. [18]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei - Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL , pages 311--318. ACL

    2002

  11. [19]

    Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. CoRR, abs/1705.04304

    2018 arXiv

  12. [20]

    Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532--1543

    2014

  13. [21]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016 ...

    2016

  14. [22]

    Khapra, and Sachindra Josh

    Sathish Reddy, Dinesh Raghu, Mitesh M. Khapra, and Sachindra Josh. 2017. Generating natural language question-answer pairs from a knowledge graph using a RNN based question generation model. In EACL (1) , pages 376--385. Association for Computational Linguistics

    2017

  15. [23]

    Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel

    Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1179--1195

    2017

  16. [24]

    Liu, and Christopher D

    Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL

    2017

  17. [25]

    Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. CoRR, abs/1611.01603

    2016 arXiv

  18. [26]

    Courville, and Yoshua Bengio

    Iulian Vlad Serban, Alberto Garc \' a - Dur \' a n, C aglar G \" u l c ehre, Sungjin Ahn, Sarath Chandar, Aaron C. Courville, and Yoshua Bengio. 2016. Generating factoid questions with recurrent neural networks: The 30m factoid question-answer corpus. In ACL (1) . The Associat...

    2016

  19. [27]

    Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. CoRR, abs/1706.09799

    2017 arXiv

  20. [28]

    Heung-yeung Shum, Xiao-dong He, and Di Li. 2018. From eliza to xiaoice: challenges and opportunities with social chatbots. Frontiers of Information Technology & Electronic Engineering , 19(1):10--26

    2018

  21. [29]

    Linfeng Song, Zhiguo Wang, and Wael Hamza. 2017. A unified query-based generative model for question generation and question answering. CoRR, abs/1709.01058

    2017 arXiv

  22. [30]

    Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma, and Shi Wang. 2018. Answer-focused and position-aware neural question generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3930--3939, Brussels, Belgium. Association fo...

    2018

  23. [31]

    Williams

    Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8(3-4):229--256

    1992

  24. [32]

    Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu. 2017. Deliberation networks: Sequence generation beyond one-pass decoding. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neu...

    2017

  25. [33]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA : A dataset for diverse, explainable multi-hop question answering. In Proceedings of the Conference on Empirical Methods in Natural Language ...

    2018

  26. [34]

    Kaichun Yao, Libo Zhang, Tiejian Luo, Lili Tao, and Yanjun Wu. 2018. Teaching machines to ask questions. In IJCAI

    2018

  27. [35]

    Xingdi Yuan, Tong Wang, C aglar G \" u l c ehre, Alessandro Sordoni, Philip Bachman, Saizheng Zhang, Sandeep Subramanian, and Adam Trischler. 2017. Machine comprehension by text-to-text neural question generation. In Rep4NLP@ACL, pages 15--25. Association for Computational Linguistics

    2017

  28. [36]

    Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. 2018. Paragraph-level neural question generation with maxout pointer and gated self-attention networks. In EMNLP

    2018

  29. [37]

    Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. 2018. Sequential copying networks. In AAAI

    2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.