{"id":"27a450a0-981d-44b1-a202-89e63db6c81c","arxiv_id":"1909.05355","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A two-pass refinement decoder with dual attention improves automatic question generation over single-pass models on SQuAD, HOTPOT-QA, and DROP.","lead":"The paper introduces RefNet, a two-decoder neural model for question generation that first drafts a question and then refines it using dual attention over the passage and the draft. It reports improved BLEU and human-judged quality over single-pass models on three reading comprehension datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 2.2 never specifies whether the refinement decoder is trained on drafts produced by the preliminary decoder or on reference questions; the central refinement claim depends on this, and standard teacher forcing would create a train/test mismatch that could explain the gains.","rationale":"I adopt the reader's weakest_assumption as the most load-bearing concern. The paper's central contribution is that a second pass repairs an imperfect draft; the equations in Section 2.2 define \\tilde{Q} only for inference, and Section 3.2 gives no training procedure for the two decoders. If the refinement decoder is trained with reference questions as its 'draft,' the model never learns to correct the specific errors of the preliminary decoder, so the reported 3.7-17.5% BLEU gains cannot be attributed to refinement. This is checkable because the code is public. I did not elevate the abstract's 7-16% overclaim (SQuAD passage is 3.71%) to the primary concern: it is a reporting error and does not invalidate the direction of the result. Similarly, the use of reimplemented baselines on HOTPOT/DROP is a weakness, but the central mechanism still rests on the draft-conditioning distribution. The reader's CONDITIONAL verdict is appropriate; I would keep it conditional pending the code check or ablation.","tokens_in":12238,"tokens_out":7566,"duration_ms":113074,"concrete_test":"Open the released training code (https://github.com/PrekshaNema25/RefNet-QG) and identify the sequence fed to the refinement decoder for each training example. If it is the reference question rather than the preliminary decoder's output, the concern lands and the paper must be revised to describe draft sampling. If it is the draft, run a controlled ablation on SQuAD (sentence-level): train a variant of RefNet with the refinement input replaced by the reference question and compare BLEU-4 against the released, draft-conditioned model. A drop of more than one BLEU-4 point in the teacher-forced variant would confirm that draft-conditioned training is load-bearing; a negligible difference would indicate the refinement decoder's contribution is not sensitive to the training draft distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2 defines the initial draft \\tilde{Q} with an arg-max over the preliminary decoder and then conditions the refinement decoder on \\tilde{Q}, but no training-time loss or data-generation procedure for the refinement decoder is given. Section 3.2 only says 'train our models for 10 epochs using cross entropy loss.' If the implementation uses standard teacher forcing, the refinement decoder sees the reference question as its source during training and a predicted draft at inference. That distribution shift means the model has not been trained to repair the specific errors the preliminary decoder makes, so the improvement of RefNet over EAD cannot be attributed to the refinement mechanism; it could come from the extra capacity of a second decoder or from the shared parameters noted in Table 4, where the initial draft also improves (17.59 without A3, 17.89 with A3 versus EAD's 16.84). Since the paper's central claim is that the second pass refines a draft, this missing specification is the load-bearing point. The code is public, so the question is empirically checkable rather than hypothetical.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RefNet, a two-decoder sequence-to-sequence model for answer-aware question generation. A preliminary decoder generates an initial draft question, and a refinement decoder conditions on that draft through a dual attention mechanism over the passage/answer representation and the draft, producing a revised question. A variant, Reward-RefNet, fine-tunes the refinement decoder with REINFORCE, using the difference between the rewards of the refined and preliminary outputs as the learning signal. The authors evaluate on SQuAD (sentence and passage level), HOTPOT-QA, and DROP, reporting improvements over their EAD baseline and prior work across BLEU, ROUGE-L, METEOR, and QBLEU, and they supplement the automatic scores with human evaluations on fluency, completeness, and answerability. They also analyze the interaction between the two decoders and introduce an originality-focused reward.","tokens_in":12456,"tokens_out":5574,"duration_ms":63264,"significance":"If the refinement mechanism is genuinely responsible for the observed gains, this paper makes a useful contribution: it gives a simple architectural recipe, two decoders plus dual attention, that consistently improves over a single-pass baseline on several datasets, and it provides an ablation showing that the direct attention path to the preliminary decoder (A3) matters. The release of code, the multi-dataset evaluation, and the inclusion of human preference judgments are clear strengths. The main caveats are that the training-time data generation for the refinement decoder is underspecified, which is load-bearing for the central 'refinement' claim, and that some abstract-level claims exceed the numbers actually reported in Section 4.1. The QBLEU metric's shared provenance with the reward also weakens the automatic answerability evaluation, although human judgments partially mitigate this concern.","major_comments":[{"comment":"Section 2.2 defines the refinement decoder's input as the initial draft Q-tilde, but the paper never states whether, at training time, Q-tilde is produced by the preliminary decoder or is the reference question under teacher forcing. Section 3.2 only says that the models are trained for 10 epochs with cross-entropy loss. If the refinement decoder was trained on reference questions but applied to preliminary-decoder outputs at inference, there is a train/test distribution mismatch, and the gains in Table 2 over EAD could reflect extra decoder capacity or the shared parameters shown in Table 4 rather than learning to repair drafts. The paper should specify the training-time generation of Q-tilde explicitly; if teacher forcing was used, the authors should verify the refinement claim by training on sampled drafts or by reporting an oracle/self-repair experiment.","section":"Section 2.2, Section 3.2"},{"comment":"The abstract claims that RefNet outperforms existing state-of-the-art methods by 7-16% on all datasets, but Section 4.1 reports a gain of only 3.71% over the previous state of the art on SQuAD (passage level) and 3.85% over EAD on the same setting. The abstract should be corrected to match the numbers in Table 2, or the claim should be qualified so that it does not overstate the passage-level result.","section":"Abstract, Section 4.1"},{"comment":"The answerability reward used in Reward-RefNet and the QBLEU-4 metric used for evaluation are both taken from Nema and Khapra (2018), which shares authors with this submission. Because the same score is both optimized and reported as evidence of answerability improvement, the automatic result is not an independent confirmation of the reward's effect. The human preference data in Table 6 helps, but the paper should either report an independently defined answerability metric or present the QBLEU numbers as an in-family measure and rely on human evaluation for the answerability claim.","section":"Section 2.3, Section 3.3, Table 6"},{"comment":"The central comparative claim rests on single-point BLEU-4 differences, several of which are small, for example 16.99 versus 16.36 on SQuAD (passage level). No confidence intervals, bootstrap intervals, or significance tests are reported for the automatic metrics, and Table 4 similarly reports gaps of about half a BLEU point without any uncertainty estimate. Because the contribution is framed as a clear improvement over prior state of the art, the authors should report statistical reliability for at least the key comparisons, or clearly state the evaluation size so readers can judge the stability of these differences.","section":"Section 4.1, Table 2, Table 4"}],"minor_comments":[{"comment":"Section 2.4 says the pointer-network and coverage mechanism from See et al. (2017) is adopted, but the paper does not explain how the pointer/copy distribution is integrated into the output equations of the two decoders; please add the relevant equations or a precise reference to an appendix.","section":"Section 2.4"},{"comment":"In the refinement decoder description, the context vector c_t from the passage and the draft attention vector g_t are both introduced in a way that makes it hard to tell which attention network, A2 or A3, computes which; the notation should be made consistent.","section":"Section 2.2"},{"comment":"For HOTPOT-QA and DROP, the preprocessing is described only at a high level; please state how supporting facts are concatenated into the passage, how the validation split is made, and how non-span answers such as yes/no are handled.","section":"Section 3.1"},{"comment":"The human evaluation reports majority labels but no inter-annotator agreement measure such as Fleiss' kappa; reporting agreement would strengthen the reliability of the preference percentages in Section 4.2 and Table 6.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the venue and the central idea is worth pursuing, but the underspecified training procedure for the refinement decoder is a serious reproducibility and validity issue that should be resolved before acceptance. The abstract-level overstatement of the passage-level result should also be fixed. No concerns about novelty or authorship beyond the QBLEU provenance already noted in the major comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"RefNet is a useful two-pass architecture for question generation, and it deserves a serious referee. The central idea—generate a draft with a first decoder, then refine it with a second decoder that attends to both the passage and the draft—is a clean adaptation of Xia et al.'s deliberation networks to QG. The paper does a lot of things right: it evaluates on three datasets (SQuAD, HOTPOT-QA, DROP), reports human evaluations, includes an ablation on the direct attention path, and releases code. The REINFORCE baseline trick (using the first decoder's reward as the baseline for the second) is a nice, transferable idea.\n\nThe biggest soft spot is that Section 2.2 never says how the refinement decoder is trained. If it is standard teacher forcing, the model sees the reference question as the draft during training and a predicted draft at inference—a distribution shift. Then the gains you see could come from the extra capacity of a second decoder, not from actually repairing imperfect drafts. Table 4 makes this concrete: the preliminary decoder improves even without the direct attention path (A3), from 16.84 to 17.59 BLEU-4, which suggests joint training alone helps. The code is public, so this is checkable, and the paper should state the training procedure explicitly. This is the load-bearing question, not just a detail.\n\nSecond, the abstract claims 7-16% gains on all datasets, but the SQuAD passage setting is only a 3.71% relative BLEU-4 improvement. That overstatement should be fixed. Third, no variances or significance tests are reported; some deltas are small, especially on passage-level SQuAD. Fourth, the answerability reward and QBLEU metric both come from the same group's prior work. Not disqualifying—human evaluations back the reward results—but it does soften the independence of those numbers.\n\nOverall, the core RefNet claim likely holds for the sentence-level setting, where human evaluation shows a clear preference. The paper is for researchers working on QG or multi-pass decoding; it would be a useful reading-group discussion and a reasonable citation. I would accept it for peer review, but I'd ask for clarification of the training-time draft handling and a corrected abstract before publication.","headline":"RefNet is a useful two-pass QG architecture with real empirical support, but the training-time handling of the draft is under-specified and the abstract overstates the passage-level gains.","tokens_in":12999,"tokens_out":4518,"would_cite":true,"duration_ms":40926,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Refining a draft question in a second decoding pass, with attention to both the passage and the draft, makes generated questions more fluent, complete, and answerable across three question-generation benchmarks.","keywords":["automatic question generation","sequence-to-sequence","two-pass decoding","dual attention","answer-aware generation","REINFORCE","fluency and answerability rewards"],"falsifier":"Run a controlled experiment where the refinement decoder is trained with reference questions as its draft input and then applied to first-decoder drafts at test time; if BLEU-4 collapses relative to a model trained on first-decoder drafts, the reported RefNet gain is inflated by train-test mismatch.","tokens_in":12061,"feed_emoji":"❓","tokens_out":6396,"duration_ms":55178,"temperature":0.7,"pith_summary":"This paper claims that automatic question generation should not be a single-pass process: the model should first write a rough draft and then revise it, much like a human writer. It proposes RefNet, a sequence-to-sequence model with two decoders, where the second decoder reads the first decoder's output and attends to both the original passage and the draft before emitting the final question. Across SQuAD, HOTPOT-QA, and DROP, RefNet reports BLEU-4 gains over single-pass models. A variant, Reward-RefNet, trains the second decoder to make drafts more fluent, more answerable, or more original.","feed_headline":"Second-pass question refinement beats single-pass generation","feed_subtitle":"RefNet's decoder revisits the passage and its own draft, lifting BLEU-4 by 3.7–17.5% on three datasets.","key_machinery":"The load-bearing mechanism is the dual attention network in the refinement decoder. After the preliminary decoder produces a draft, the refinement decoder conditions each output word on three things: the answer-aware passage representation, an attention-weighted summary of the draft's word embeddings, and the answer representation. This lets the second decoder revisit the passage for missing context while also seeing where the draft went wrong, and the shared output projection keeps the two passes aligned in vocabulary space.","core_discovery":"The central claim is that a second-pass refinement step, guided by dual attention over the passage and the draft, repairs the incompleteness and grammar errors that single-pass encoders leave in generated questions. On BLEU-4, RefNet beats the prior state of the art by 12.30% on SQuAD sentence-level question generation, 9.74% on HOTPOT-QA, 17.48% on DROP, and 3.71% on SQuAD passage-level generation, and it also beats its own single-decoder baseline by 7.83%, 7.57%, 15.25%, and 3.85% in those same settings. Human evaluation reported by the paper confirms that the refinement decoder produces more fluent, complete, and answerable questions in most pairwise comparisons.","pith_inferences":["A natural test the paper leaves open is whether the refinement decoder is trained on drafts produced by the first decoder or on reference questions; if training uses reference questions, part of the reported gain could come from polishing an already-correct input rather than repairing a draft.","The draft-and-revise structure could transfer to other text-generation tasks, such as summarization, simplification, or dialogue response generation, where single-pass outputs also tend to be incomplete drafts.","Refining over the draft's word embeddings has a ceiling: it can repair wording and reorder content, but it cannot introduce facts that are absent from both the passage and the draft, so the reported originality gains are limited to rephrasing."],"forward_implications":["If the refinement gain is real, adding a second decoder to an existing sequence-to-sequence question generator is a cheap architectural upgrade: keep the single-pass model as the first decoder, reuse the encoder, and let the second decoder revise the draft.","The first decoder also improves when the second decoder is present, even before the dual attention path is added, which suggests that gradient feedback through the shared encoder and output layer is part of the benefit.","Reward signals can be targeted: rewarding improvements in fluency raises BLEU-4, rewarding answerability raises the QBLEU score, and rewarding low n-gram overlap with the passage pushes generated questions toward originality.","The gains on HOTPOT-QA and DROP suggest the refinement idea helps beyond simple span-based SQuAD questions, including multi-hop and discrete-reasoning settings."],"supporting_citations":[{"why":"Supplies the SQuAD question-answer dataset used for the sentence- and passage-level evaluation.","marker":"(Rajpurkar et al., 2016)"},{"why":"Provides the paragraph-level single-pass baseline and position-aware encoding idea that RefNet is compared against as state of the art.","marker":"(Zhao et al., 2018)"},{"why":"Provides the pointer-generator and coverage mechanism RefNet adopts to handle out-of-vocabulary words and repetition.","marker":"(See et al., 2017)"},{"why":"Defines the fluency and answerability scores used as reward signals and as the QBLEU evaluation metric.","marker":"(Nema and Khapra, 2018)"},{"why":"Introduces the deliberation-network idea of two-pass decoding that RefNet carries over to question generation.","marker":"(Xia et al., 2017)"},{"why":"Supplies the REINFORCE-with-baseline rule used to train Reward-RefNet.","marker":"(Williams, 1992)"},{"why":"Provides the HOTPOT-QA multi-hop dataset for one of the three evaluation settings.","marker":"(Yang et al., 2018)"},{"why":"Provides the DROP discrete-reasoning dataset for the third evaluation setting.","marker":"(Dua et al., 2019)"}],"fun_headline_variants":["Draft-and-refine boosts question generation","Two-pass decoder sharpens question accuracy","RefNet: second pass fixes draft questions","Dual attention refines questions beyond baseline","Refinement decoder lifts QG BLEU by up to 17%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The refinement decoder's reported gains depend on it being trained on drafts that look like what the preliminary decoder actually produces; if training instead uses reference questions, the model could be learning to polish a perfect input rather than repair a draft.","fun_headline_variants_meta":{"raw":{"variants":["Draft-and-refine boosts question generation","Two-pass decoder sharpens question accuracy","RefNet: second pass fixes draft questions","Dual attention refines questions beyond baseline","Refinement decoder lifts QG BLEU by up to 17%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000179,"raw_usage":{"total_tokens":1342,"prompt_tokens":1026,"completion_tokens":316,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":244}},"tokens_in":642,"tokens_out":316,"duration_ms":4939,"temperature":1.0,"reasoning_tokens":244,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T06:00:55.839823+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled experiment where the refinement decoder is trained with reference questions as its draft input and then applied to first-decoder drafts at test time; if BLEU-4 collapses relative to a model trained on first-decoder drafts, the reported RefNet gain is inflated by train-test mismatch.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SQuAD question-answer dataset used for the sentence- and passage-level evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the paragraph-level single-pass baseline and position-aware encoding idea that RefNet is compared against as state of the art."},{"cited_title":"Liu, and Christopher D","cited_arxiv_id":null,"evidence_quote":"Provides the pointer-generator and coverage mechanism RefNet adopts to handle out-of-vocabulary words and repetition."},{"cited_title":"Towards a Better Metric for Evaluating Question Generation Systems","cited_arxiv_id":"1808.10192","evidence_quote":"Defines the fluency and answerability scores used as reward signals and as the QBLEU evaluation metric."}],"review_version":1}