{"id":"381360ca-39be-4c19-a0b0-acbfa0a9191c","arxiv_id":"2507.11875","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A weighted training scheme with adaptive reward scaling improves cloze-test distractor ranking, reaching 37.84% P@1 on MCQ versus 31.66% for the strongest baseline.","lead":"This paper introduces DualReward, a training method that gives different reward weights to human-written and machine-generated distractor candidates to improve cloze-test wrong-answer generation. It matters because it reports consistent ranking improvements, with the largest gains on diverse multiple-choice questions, while the method is simple enough to reuse in other generation tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'RL' objective in Sec. 3.2 is a fixed-label weighted cross-entropy, and the adaptive 'reward scale' is a single global multiplier; with AdamW this scale is largely absorbed, so the claimed adaptive-reward RL mechanism does not do the work attributed to it.","rationale":"The reader's weakest assumption correctly identifies that the update in Section 3.2 is a weighted maximum-likelihood objective over fixed labels rather than a genuine policy-gradient RL update. I agree with that diagnosis. My stress-test goes one step further: even within the weighted-CE interpretation, the 'adaptive' component is a single global multiplier that cancels out of the relative gold-versus-generated reward ratio and is largely absorbed by the AdamW optimizer. This makes the mechanism's claimed role in the reported gains especially fragile, and it is the most load-bearing point because the paper's contribution is framed as an adaptive reward-scaling RL framework. If this point holds, the empirical results may still stand as evidence for a supervised reweighting method with an ad hoc learning-rate schedule, but the conceptual contribution and the explanation of the MCQ improvements would need substantial revision. The paper does provide useful ablations and honest limitation statements, and the core empirical claims are not obviously fabricated, so a conditional verdict remains appropriate rather than outright rejection. The concrete test above would settle whether the adaptive scale is merely a learning-rate schedule and whether the RL machinery is necessary at all.","tokens_in":1003,"tokens_out":1010,"duration_ms":99459,"concrete_test":"Run a single controlled experiment on MCQ: implement the training objective exactly as -Σ_i w_i log p(y_i|s_i) over the fixed 10-label set, with w_i = 1 for gold labels and w_i = 0.9 * confidence for generated labels, and no other RL machinery. Compare two variants: (a) the paper's adaptive reward scale as a global multiplier, and (b) a constant scale combined with an explicit AdamW learning-rate scheduler that follows the same average-loss curve. If (a) and (b) reproduce the same P@1/MRR/NDCG values as Table 4, the adaptive 'reward scaling' is equivalent to a learning-rate schedule and the RL formulation is not load-bearing; if they diverge, the scale has a genuine per-step effect that must be characterized.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 fixes, per instance, 10 candidate labels: 3 gold distractors and 7 candidates generated once by the CSG/BERT module. The update in Section 3.2 is written as L_RL = -E_{(s,a)~pi_theta}[reward(s,a) log pi_theta(a|s)], but no on-policy sampling, no replay from the current policy, no importance-weighting correction, and no baseline/advantage term are described. The actual loss is therefore -Σ_i reward_i log p(y_i|s_i) over the precompiled multiset: a weighted maximum-likelihood objective, not a policy-gradient update. This matters because the paper's central novelty is presented as reward-driven RL exploration; under the described algorithm, the 'exploration' is confined to the initial static candidate generation, and the rewards only reweight those fixed labels. A second, more specific problem compounds the first: the 'reward scale' is a single scalar computed from the average loss of a training phase. It multiplies every label reward, so it cancels in the ratio reward_gold/reward_gen and cannot alter the intended dual-reward hierarchy. It only scales the total gradient. With the AdamW optimizer used in Section 4.3, a common multiplicative factor on all gradients is largely normalized away (Adam's m_hat/sqrt(v_hat) ratio is approximately invariant to a global gradient scale). Consequently, the reported 3.48-3.86% P@1 gains from 'adaptive vs. constant reward' would need to be explained as a learning-rate schedule driven by average loss, not as an adaptive reward mechanism. The paper does not report variance or significance testing, and the CLOTH-F P@1 improvement over the best baseline is only 0.4%, so the core mechanism is not yet empirically pinned down.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DualReward, a framework for distractor generation in cloze tests. It builds a fixed set of 10 candidate labels per instance (3 gold distractors from human references and 7 candidates generated once by a BERT-based CSG module), and trains a T5-base model with a loss that weights each label by a reward: gold labels receive reward 1 times a scale, generated labels receive 0.9 times the same scale times a static BERT confidence score. The scale is computed adaptively from the average training loss via a sigmoid. The authors evaluate on CLOTH-F and MCQ, reporting P@1 improvements over baselines, and ablations showing benefits of dual over uniform reward and adaptive over constant scale.","tokens_in":11621,"tokens_out":6627,"duration_ms":73023,"significance":"If the reported improvements are real, the dual-reward weighting of gold versus model-generated distractors is a practically useful idea for distractor ranking, especially on cross-domain MCQ data. The paper includes a code link, ablation studies, and qualitative examples, which are strengths. However, the 'reinforcement learning' framing is not substantiated: the objective is a weighted maximum-likelihood update over fixed labels, and the adaptive scale is a global multiplier that is approximately cancelled by the AdamW optimizer, so the central mechanism described is not what the experiments claim to test. The paper's own limitations section appropriately acknowledges that the evaluation matches human distractors rather than measuring pedagogical effectiveness. The significance of the empirical results therefore depends on re-interpreting the method as supervised reweighting and on providing evidence that the adaptive component has any measurable effect.","major_comments":[{"comment":"The loss L_RL = -E_{(s,a)~pi_theta}[reward(s,a) log pi_theta(a|s)] is presented as a policy-gradient RL objective, but the training set is fixed: for each instance, Section 3.1 precompiles exactly 10 labels (3 gold plus 7 generated by the BERT CSG module), and the expectation is taken over this fixed multiset. There is no on-policy sampling, no replay from the current policy, no importance-weighting, and no baseline or advantage term. The implemented update is therefore -Sigma_i reward_i log p(y_i|s_i), i.e., weighted maximum likelihood or weighted cross-entropy, not a policy-gradient update. This is not merely a presentational issue: the paper's claim to 'dynamic reinforcement learning' and its interpretation of the dual-reward ablation as 'reward-driven exploration' depend on the RL framing. The authors should either re-describe the method as reward-weighted supervised learning or introduce an actual sampling-based RL objective.","section":"Section 3.2, Eq. (2)"},{"comment":"The adaptive reward scale is a single global scalar that multiplies both reward_gold and reward_gen. It therefore cannot change the relative weighting between gold and generated distractors, because reward_gold/reward_gen = 1/(0.9*confidence) is independent of the scale. Furthermore, with the AdamW optimizer, a global multiplicative factor on all gradients is approximately normalized away by the bias-corrected moment estimates (m_hat/sqrt(v_hat) is invariant to a constant scaling of gradients). The reported differences between adaptive and constant rewards in Table 4 (0.21-0.46% on CLOTH-F and 3.48-3.86% on MCQ) cannot, under the described algorithm, be attributed to the adaptive mechanism. The authors need to either (a) provide direct evidence that the adaptive scale changes the optimization trajectory (e.g., gradient norms, parameter updates, or comparisons with SGD), or (b) revise the claims and ablation to reflect that the adaptive component is not the source of the observed gains.","section":"Section 3.2, Eq. (3) and Section 4.3 (AdamW)"},{"comment":"The MCQ test set contains only 258 examples (Table 2). A 3.48-3.86% P@1 difference corresponds to roughly 9-10 questions. The paper reports no variance, confidence intervals, or significance tests across random seeds, so it is unclear whether the adaptive-versus-constant differences in Table 4 are beyond noise. This is particularly relevant given the theoretical concern in the previous comment. Please report multiple seeds and statistical significance, or temper the claims accordingly.","section":"Table 4 and Table 2 (MCQ test set size)"}],"minor_comments":[{"comment":"The 'discount factor' of 0.9 is a constant reward weight, not a discount factor in the reinforcement-learning sense; please rename it to avoid confusion.","section":"Section 3.2"},{"comment":"Some baseline cells are '—' (R@1 for CSG+DS, MRR@3/NDCG@3 for CDGP). Report the values or explicitly state that they are unavailable.","section":"Table 3"},{"comment":"Figure 2 is referenced in the text but no plot appears in the submitted manuscript; please include the figure or remove the reference.","section":"Section 5.2"},{"comment":"The CLOTH-F row shows '6500' for the All column but the column headings are ambiguous; please align the columns and ensure the totals match the sum of train, dev, and test splits.","section":"Section 4.1, Table 2"},{"comment":"The qualitative examples include apparent truncations or typos such as 'Undrate' and 'eva'; please correct them or mark them as model outputs in a way that does not confuse the reader.","section":"Table 5"},{"comment":"The statement that adaptive scaling provides 'stronger reinforcement signals when the model struggles' should be made precise by connecting it to Eq. (3); as written, the direction (high loss leads to high scale) is correct but the explanation is informal.","section":"Section 4.3"},{"comment":"The description of the T5 multi-task baseline mentions 'pseudo Kullback-Leibler divergence regulation' without a citation; please add the appropriate reference.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The central technique as described is not a policy-gradient RL update, and the adaptive reward scaling appears to be a no-op under AdamW because it is a global multiplier on all gradients. If the authors cannot provide evidence that the adaptive scale changes the optimization trajectory, the paper should be substantially reframed. I recommend major revision rather than rejection because the dual-reward reweighting of gold versus generated distractors is a potentially useful contribution if presented as reward-weighted supervised learning and if the adaptive claims are either properly evidenced or removed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The empirical results are plausibly real, but the paper's central framing is wrong. The update in Sec. 3.2 is written as an RL objective, but what they actually optimize is a weighted cross-entropy over a fixed set of 10 precompiled labels per instance (3 gold + 7 generated). There is no on-policy sampling, no importance weighting, no advantage term. This matters because the claimed novelty is reward-driven RL exploration; in fact, exploration ends at the static candidate-generation step, and the dual rewards only reweight fixed labels. That is a reasonable idea—asymmetric weighting of gold vs. generated distractors—but it is not reinforcement learning.\n\nThe adaptive reward scale compounds the problem. It is a single scalar computed from the average loss, multiplying every reward. It cancels in the gold/gen reward ratio, so it does not affect the dual-reward hierarchy; it only scales the total gradient. With AdamW, a constant scale would be largely normalized away, but a time-varying one acts as a loss-dependent learning rate. So the reported gains from adaptive vs. constant reward are likely a learning-rate schedule effect, not an adaptive reward mechanism. The stress-test note is right on this.\n\nWhat is genuinely new is applying differential rewards to gold vs. model-generated distractors for cloze ADG, together with a loss-based scaling. The ingredients are known, but the integration is new and the ablations are consistent. The qualitative examples show real improvements in contextual appropriateness. The limitations section is honest about not measuring pedagogical outcomes.\n\nSoft spots beyond the framing: no variance or significance testing. On CLOTH-F, the P@1 gain over the best baseline is 0.4%, within noise. The MCQ gains are larger, but without confidence intervals they are hard to trust. The hyperparameters alpha and threshold came from preliminary experiments, with no sensitivity analysis. The code link has no commit hash.\n\nAudience: people building cloze-test distractor generators. For that niche, the method is worth trying, but not as RL. I would send it to review—the empirical work deserves referee time—but it needs major revision, either re-framing as weighted supervised learning or implementing an actual policy-gradient update.","headline":"Plausible gains, but the 'RL' objective in Sec. 3.2 is weighted cross-entropy over fixed labels, and the adaptive reward scale is a global loss-dependent multiplier that acts like a learning-rate schedule.","tokens_in":12214,"tokens_out":3994,"would_cite":false,"duration_ms":44055,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DualReward claims that separating rewards for human and machine distractors lifts cloze-test ranking quality.","keywords":["distractor generation","cloze test","reinforcement learning","adaptive reward scaling","dual reward","ranking quality","T5"],"falsifier":"Train the same T5 model with the same ten labels per instance using standard weighted cross-entropy where gold distractors receive weight 1 and generated distractors receive weight 0.9 × confidence, with no adaptive scaling or reinforcement-learning framing. If P@1 on MCQ remains at or above 37.84%, the adaptive dual-reward mechanism is not responsible for the gain; if it falls below the reported number, the claim survives this test.","tokens_in":11071,"feed_emoji":"📝","tokens_out":4064,"duration_ms":43204,"temperature":0.7,"pith_summary":"This paper tries to establish that a distractor-generation model for cloze tests improves when the training signal distinguishes human-authored gold distractors from model-generated ones and scales that signal adaptively as training loss falls. The authors propose DualReward, which gives gold distractors a fixed reward and generated candidates a reward discounted by model confidence, with the overall scale adjusted by a sigmoid of the running average loss. Evaluated on a passage-level benchmark (CLOTH-F) and a cross-domain sentence-level benchmark (MCQ), the method reports state-of-the-art precision at rank one, with the largest gains on MCQ (37.84% P@1 versus 31.66% for the strongest baseline). The intended significance is that automated test construction can learn from the full spectrum of imperfect candidates instead of discarding them, and that adaptive reward intensity helps most when question domains vary.","feed_headline":"Dual rewards lift cloze-test distractor ranking by 6.18 points","feed_subtitle":"Separating gold from model-generated distractors and scaling rewards by loss boosts P@1 on cross-domain MCQ to 37.84%.","key_machinery":"The load-bearing object is a weighted maximum-likelihood objective written as $L_{RL} = -\\mathbb{E}_{(s,a)\\sim\\pi_\\theta}[\\mathrm{reward}(s,a)\\log\\pi_\\theta(a|s)]$, trained on a fixed set of ten labels per instance: three gold distractors and seven model-generated candidates. Rewards are set by the dual structure ($1$ for gold, $0.9 \\times$ BERT confidence for generated) and scaled by $\\mathrm{reward\\ scale} = \\mathrm{base} + (\\mathrm{max}-\\mathrm{base})\\,\\sigma(\\alpha(\\mathrm{avg\\ loss}-\\mathrm{threshold}))$. The confidence score is the softmax probability from BERT's masked-language-model head at the blank position. This machinery converts the quality distinction between human and generated candidates into different gradient magnitudes.","core_discovery":"The central claim is that a dual reward structure — gold distractors scored at 1×scale, generated candidates at 0.9×scale×confidence — combined with a loss-based adaptive scale yields better distractor ranking than uniform rewards, constant reward scales, and existing baselines, on both passage-level and sentence-level cloze benchmarks. On CLOTH-F the reported P@1 is 28.87% with larger gains in ranking metrics; on MCQ with retrieval-augmented pretraining the reported P@1 is 37.84%, exceeding the RAP-PT5 baseline by 6.18 percentage points. The authors attribute the cross-domain advantage to the adaptive scaling mechanism, which provides stronger signals when loss is high and gentler signals as performance stabilizes.","pith_inferences":["The reported objective is formally identical to reward-weighted maximum likelihood over a static set of labels; whether the gains come from reward-driven exploration or from reweighting the training distribution is not settled by the experiments, since no online rollouts are used.","A testable extension would be to compare against plain weighted cross-entropy with the same weights; if gains persist, the reinforcement-learning framing is not needed to explain them.","The dual reward hierarchy could be reinterpreted as a regularizer that prevents the model from overfitting to its own generations, which would predict larger benefits on small, diverse datasets like MCQ.","The adaptive scaling schedule depends only on training loss; using validation-based or confidence-calibration-based triggers could make the mechanism more robust across domains."],"forward_implications":["Distractor ranking models can be trained without discarding low-confidence candidates, using them as controlled negative examples with discounted reward.","The adaptive scale offers a way to reduce sensitivity to the choice of reward magnitude, since the same base, max, and threshold settings work across homogeneous and cross-domain data.","The ranking-metric gains (MRR@3, NDCG@3) imply that even when top-1 precision moves little, the ordered list of distractors becomes more useful for assembling real multiple-choice items.","On diverse sentence-level data, pretraining plus dual reward appears complementary, with the MCQ gains coming largely from the reward mechanism rather than from pretraining alone.","The framework transfers to other text-generation tasks where training examples have graded reliability, not just cloze distractors."],"supporting_citations":[{"why":"Supplies the Candidate Set Generation module and the BERT masked-language confidence scores used to score generated distractors.","marker":"Chiang et al. (2022)"},{"why":"Provides the CLOTH-F dataset and the T5 multi-task baselines that DualReward is compared against.","marker":"Wang et al. (2023)"},{"why":"Provides the MCQ dataset and the CSG+DS baseline for cross-domain sentence-level evaluation.","marker":"Ren and Zhu (2021)"},{"why":"Provides the RAP-PT5 baseline and the retrieval-augmented pretraining procedure on Sciq-all used before DualReward training on MCQ.","marker":"Yu et al. (2024)"},{"why":"Motivates the sigmoid-based adaptive reward scaling as an analogue of temperature adaptation in soft actor-critic.","marker":"Haarnoja et al. (2018)"},{"why":"Motivates the dynamic adjustment of learning signal intensity based on model performance, similar to reverse curriculum generation.","marker":"Florensa et al. (2018)"},{"why":"Provides the policy-gradient foundation and the bounded-scaling idea behind the choice of base and max reward values.","marker":"Schulman et al. (2017)"}],"fun_headline_variants":["Adaptive reward scaling boosts cloze distractor P@1 by 6.18%","Dynamic dual rewards improve cross-domain cloze distractor ranking","Reward-shaping for distractors yields 6.18% P@1 gain on MCQ","Adaptive scaling in RL boosts cloze distractor quality on diverse data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The premise that a single fixed set of ten labels per instance with confidence-based rewards is a genuine policy-gradient training signal; if the objective is just weighted cross-entropy, the reported gains may come from reweighting rather than from reinforcement learning.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive reward scaling boosts cloze distractor P@1 by 6.18%","Dynamic dual rewards improve cross-domain cloze distractor ranking","Reward-shaping for distractors yields 6.18% P@1 gain on MCQ","Adaptive scaling in RL boosts cloze distractor quality on diverse data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000412,"raw_usage":{"total_tokens":2099,"prompt_tokens":881,"completion_tokens":1218,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":1133}},"tokens_in":497,"tokens_out":1218,"duration_ms":13747,"temperature":1.0,"reasoning_tokens":1133,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:59:22.470932+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same T5 model with the same ten labels per instance using standard weighted cross-entropy where gold distractors receive weight 1 and generated distractors receive weight 0.9 × confidence, with no adaptive scaling or reinforcement-learning framing. If P@1 on MCQ remains at or above 37.84%, the adaptive dual-reward mechanism is not responsible for the gain; if it falls below the reported number, the claim survives this test.","supporting_citations":[],"review_version":1}