{"id":"c5e1c9da-6a27-47f0-a91c-5badcc99a468","arxiv_id":"2501.11463","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding a prediction-error curiosity reward, masked to non-top-k tokens, improves output diversity of RLHF-trained LLMs while keeping reward-model judged quality roughly unchanged.","lead":"The paper adds a curiosity-driven intrinsic reward to standard RLHF, rewarding the model for exploring less likely tokens, and reports better output diversity with similar alignment quality. It matters because LLM alignment often reduces output variety, and this gives a practical way to counter that collapse during training.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The alignment claim is supported only by the same reward model used as the PPO training signal, so parity in RM scores does not establish that CD-RLHF preserves human-preferred quality.","rationale":"The reader's weakest assumption concerns whether ICM prediction error is a valid novelty signal. That is a real concern, but it mostly affects the paper's mechanistic framing: even if the ICM signal were replaced by a simple token-rarety bonus, the empirical observation of higher diversity might persist. The more load-bearing vulnerability is the alignment evaluation. The paper's headline claim is that CD-RLHF maintains alignment with human preferences, yet the principal evidence for that is the RM score, and that RM is the same function used as the PPO reward. Because both policies are trained against this RM, equal RM scores are a weak, partially circular witness for human-preferred quality. The paper is transparent about this in Section 4.1, but the implication is not addressed in the main narrative. The independent GPT-4 check in Appendix C is limited to one model and shows only a tie, and MT-Bench is a general capability proxy rather than a preference-alignment measure. A held-out preference judge is the minimal experiment that would convert the conditional acceptance into a firm result. This does not change the reader's CONDITIONAL verdict; it sharpens the condition under which the central claim should be accepted.","tokens_in":21150,"tokens_out":10263,"duration_ms":121676,"concrete_test":"Score the final RLHF and CD-RLHF policies from all four model sizes on 2,000 held-out prompts from each dataset using a reward model trained on a disjoint split of the preference data (or a public preference judge such as PairRM), and run the same comparison with GPT-4 pairwise preference judgments. If CD-RLHF's win rate against vanilla RLHF on this held-out judge is below 45% (or consistently outside the confidence interval around 50%), then the 'maintaining alignment comparable to vanilla RLHF' claim is not supported and the observed diversity gains are partly an artifact of measuring with the training reward.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two parts: improved diversity and maintained alignment. The diversity gain is mechanistically plausible because the top-1 mask directly rewards non-greedy token choices. The alignment half, however, is not independently evidenced. In Section 4.1 the paper states that 'RM scores evaluate the alignment quality using the trained reward model, which is identical to the one used in PPO training.' Every main-table alignment number (Table 1) is therefore a measurement on the very objective both policies were optimized against. Comparable RM scores can occur even if CD-RLHF is worse on true human preferences, especially because the intrinsic reward upweights low-probability tokens, which are precisely the outputs a reward model may overvalue relative to human raters. The only independent quality check, Table 6, covers a single model (Llama-3.2-3B) and yields a statistical tie on TL;DR (27 win / 23 loss) and on UltraFeedback (21 win / 22 loss / 7 tie), which does not confirm parity across the four model sizes. The qualitative evidence amplifies the risk: the UltraFeedback case study in Table 9 shows a CD-RLHF response asserting a cause-effect relationship ('large-scale manufacturing plant in Tokyo can lead to a decrease in demand for steel products') that contradicts the background paragraph, illustrating that the additional diversity can include factual errors that the training RM may not penalize.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CD-RLHF, an extension of PPO-based RLHF in which a token-level intrinsic reward, computed as the prediction error of an Intrinsic Curiosity Module (ICM), is added to the extrinsic reward-model score and KL penalty. The intrinsic reward is gated by a top-k mask so that it only applies when the sampled token is not among the k most probable tokens (k=1 in the main experiments). The authors evaluate the method on TL;DR summarization and UltraFeedback instruction following using Gemma-2B, Gemma-7B, Llama-3.2-1B, and Llama-3.2-3B, reporting consistent improvements on lexical and semantic diversity metrics, positive GPT-4 and human diversity win rates, and RM scores comparable to vanilla RLHF. They additionally report MT-Bench and story-generation results. The central claim is that CD-RLHF improves output diversity while maintaining alignment quality comparable to standard RLHF.","tokens_in":21475,"tokens_out":5719,"duration_ms":70697,"significance":"If the central claim holds, the method is a simple, practical addition to PPO-style RLHF that could mitigate the well-documented diversity collapse of aligned LLMs. The paper has clear strengths: it targets an important problem; it evaluates across four base models and two tasks; it includes multiple diversity metrics spanning lexical and semantic aspects; it provides GPT-4 and human diversity judgments; it includes ablations on the top-k gate and on reward frequency; and it releases code. These are substantial assets. However, the evidence as presented is not yet sufficient to support the strength of the abstract's claims. All headline numbers come from single training runs without error bars or significance tests, and the alignment-quality evidence relies on the same reward model used as the PPO training signal. The paper's own Limitations section concedes that the diversity-alignment trade-off still persists and that CD-RLHF remains below SFT in diversity, which tempers the framing but does not contradict the stated contribution.","major_comments":[{"comment":"All results in Table 1 and throughout the paper are single-run point estimates: no error bars, confidence intervals, multiple seeds, or significance tests are reported for any diversity metric or RM score. The abstract's phrase \"significant gains\" is therefore not statistically established. Please provide at least three independent training runs per configuration and report means with standard deviations and appropriate paired tests for the headline comparisons, or else temper the language to \"consistent improvements\" in the current runs.","section":"§4.1, Table 1"},{"comment":"The RM scores used to support the alignment-parity claim are produced by the same reward model that provides the PPO training signal for both RLHF and CD-RLHF (as stated in §4.1 and in the evaluation section). Parity on this metric does not establish that CD-RLHF preserves human-preferred quality, since the intrinsic reward upweights low-probability tokens that the training reward model may overvalue. Table 6 is the only independent quality check, and it covers a single model (Llama-3.2-3B) with results that are statistical ties on both datasets (27 win / 23 loss on TL;DR; 21 win / 22 loss / 7 tie on UltraFeedback). Please add per-model independent quality evaluation, for example with a held-out reward model or human preference judgments, and report the complete results rather than only the single-model table.","section":"§4.1 evaluation description, Table 1, Table 6"},{"comment":"The top-k mask directly rewards non-greedy token choices, so the reported diversity gains could in principle arise from this anti-greedy upweighting rather than from the prediction-error novelty signal. No control is provided in which the same top-k mask is combined with a non-curiosity intrinsic reward (for example, a constant bonus, a uniform random bonus, or an entropy bonus restricted to non-top-k tokens). Without such a control, it is not established that the ICM's prediction error is the operative mechanism or that the \"curiosity\" component adds value over simply penalizing greedy choices. Please add such a baseline or clearly reinterpret the contribution as an anti-greedy exploration bonus.","section":"§3.2.2, Figure 5, Table 7"},{"comment":"The MT-Bench results are reported as evidence of strong generalization, but Figure 7 shows that CD-RLHF's win rate against RLHF is only 22-32% with the majority of evaluations ending in ties; the text should be framed accordingly. More importantly, the case study in Table 9 contains a CD-RLHF response asserting that \"a large-scale manufacturing plant in Tokyo can lead to a decrease in demand for steel products,\" which contradicts the background paragraph's stated causal relationship and the story. This example illustrates that the additional diversity can include factual errors that the training reward model may not penalize. The paper should discuss this risk explicitly and, ideally, add a factual-consistency or correctness metric to the alignment evaluation.","section":"§4.5, Table 2, Table 9"}],"minor_comments":[{"comment":"The GPT-4 and human win-rate figures are based on 50 and 25 sampled instances per dataset, respectively, but no confidence intervals or exact counts are shown for Figure 3; please report the raw counts and intervals alongside the percentages.","section":"§4.1, A.3"},{"comment":"The reward whitening formula r(i) = (r(i) − µ)/σ^2 uses σ^2 without stating whether σ denotes standard deviation or variance; please define the notation precisely.","section":"§3.2.2"},{"comment":"The human evaluation protocol does not state the number of annotators, their qualifications, or inter-annotator agreement; these details are needed to assess the reliability of the human diversity judgments.","section":"Appendix B.2"},{"comment":"The phrase \"win rates ... over RLHF\" is misleading when CD-RLHF wins in only about a quarter to a third of comparisons and ties in the majority; please report the full win/tie/loss distribution and avoid the \"win rate over\" wording.","section":"§4.5, Figure 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for the journal and the code release is a genuine asset. My main concern is that the central claim has two parts, and only the diversity part is consistently evidenced; the alignment part currently rests on the training reward model and one tied single-model GPT-4 evaluation. I would want multi-seed statistics and an independent alignment evaluation before acceptance. There is also an unresolved mechanistic question about whether the curiosity signal itself matters beyond the top-k anti-greedy bonus, which I would ask the authors to address with a control experiment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"If you work on the RLHF diversity trade-off, this is worth a careful read. The diversity result looks real; the alignment-parity claim does not yet have the evidence to back it.\n\nWhat is actually new: this is the first application of forward-dynamics prediction-error curiosity to the RLHF policy-optimization stage with the explicit goal of output diversity. The top-1 masking that restricts intrinsic rewards to non-greedy token choices is a simple adaptation that makes the mechanism transparent: it directly rewards tokens outside the top-1, which is a credible way to increase diversity. The code is public, the experiments span four base models and two datasets, and the diversity metrics (lexical, semantic, GPT-4, and human win rates) consistently favor CD-RLHF. The ablations on reward frequency and top-k are a nice touch.\n\nSoft spots are real but addressable. All main numbers are single-run point estimates; there are no error bars or significance tests, so the abstract's \"significant gains\" is not statistically established. The RM score used as the alignment metric is the same RM used in PPO training, so comparable RM scores between RLHF and CD-RLHF do not establish preserved human-preferred quality; the only independent GPT-4 quality check covers one model and yields a statistical tie. The case study in Table 9 shows a CD-RLHF response asserting that the Tokyo explosion leads to a decrease in demand for steel products—flipping the relationship in the background—so the added diversity can include factual errors that the training RM apparently does not penalize. The ICM's feature-space alignment (reference-model states, actor embeddings) is plausible but not validated; a small analysis of what high prediction error actually captures would strengthen the paper. Finally, eta is hand-tuned per model, and the authors acknowledge the intrinsic reward scale gap; that is minor but worth stating.\n\nOn balance, the diversity claim is mechanically credible and consistently observed across models and datasets. The alignment-parity claim is not yet established. The paper is honest, cites relevant work, and its limitations section is straightforward. I would send it to peer review, with a request for multiple seeds and an independent quality evaluation on more than one model. A revised version along those lines would be a solid contribution to the RLHF diversity literature.","headline":"The diversity half of the paper is plausible; the alignment half is under-evidenced, but the study is honest, broad, and worth a serious referee.","tokens_in":21939,"tokens_out":2727,"would_cite":true,"duration_ms":32091,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding an intrinsic curiosity reward—the prediction error of a small forward model on token choices outside the top-$k$—lets RLHF produce more diverse outputs while keeping reward-model-judged alignment comparable to standard RLHF.","keywords":["RLHF","output diversity","intrinsic reward","curiosity-driven exploration","prediction error","large language models","alignment","text generation"],"falsifier":"Replace the ICM prediction error with random noise of the same scale, keeping the top-$k$ gate and reward whitening identical; if diversity rises as much as with the real signal, prediction error is not the active ingredient. A complementary check is to measure factual accuracy of the diverse outputs on a ground-truth task such as summarization factuality or closed-book QA, since useful diversity should not systematically increase errors.","tokens_in":20967,"feed_emoji":"🎲","tokens_out":8944,"duration_ms":83285,"temperature":0.7,"pith_summary":"Reinforcement learning from human feedback (RLHF) aligns language models with preferences but tends to shrink the variety of outputs they produce. This paper asks whether an intrinsic curiosity reward, borrowed from exploration in reinforcement learning, can undo that shrinkage at the RLHF stage without sacrificing alignment. The proposed method, CD-RLHF, adds a reward proportional to how poorly a small forward model predicts the next state, but only when the policy chose a token outside its top-$k$ choices ($k=1$ in the experiments). Across summarization and instruction-following tasks, the paper reports higher lexical and semantic diversity on every model tested, with reward-model scores essentially level against vanilla RLHF, and says the gains carry over to out-of-distribution questions and story writing. If right, it means output diversity can be treated as a training-time objective alongside preference alignment, not only as a post-hoc trade-off.","feed_headline":"Curiosity reward widens RLHF output with no alignment loss","feed_subtitle":"Adding an intrinsic reward for unpredictable token choices lifts diversity by up to 40 percent while reward-model scores hold steady.","key_machinery":"The load-bearing object is the Intrinsic Curiosity Module (ICM), a two-layer MLP feature encoder $\\phi$ plus a two-layer MLP forward model $f$ that predicts $\\phi(s_{t+1})$ from $\\phi(s_t)$ concatenated with the action embedding $\\psi(a_t)$. Its squared prediction error is the curiosity reward; a top-$k$ gate zeros that reward for probable continuations, and reward whitening rescales it along the trajectory. The module is trained separately from the policy by minimizing the same squared error, so familiar states become less rewarding as prediction improves. The mechanism's job is to turn each RLHF step into a per-token novelty signal that can be added to the sparse reward-model score and the KL penalty without changing the alignment objective itself.","core_discovery":"CD-RLHF extends the standard RLHF reward $r^{(e)} = R - \\beta D_{\\mathrm{KL}}(\\pi_{\\mathrm{policy}} \\| \\pi_{\\mathrm{ref}})$ by adding an intrinsic term $\\eta r^{(i)}$. The Intrinsic Curiosity Module (ICM) encodes the current state and action, predicts the next state's features with a small forward model, and takes the squared prediction error $\\frac{1}{2}\\|\\hat{\\phi}(s_{t+1}) - \\phi(s_{t+1})\\|_2^2$ as the curiosity signal. This reward is zeroed whenever the chosen token falls in the top-$k$ of the policy's output distribution, so exploration is aimed at less probable continuations. State features are produced from the reference model's last hidden layers and action features from the policy model's token embeddings, giving the prediction problem one coherent feature space. With this reward added to PPO, the paper reports consistent diversity gains over vanilla RLHF on TL;DR summarization and UltraFeedback instruction following, comparable reward-model scores, and better judge-model win rates on the out-of-distribution MT-Bench benchmark.","pith_inferences":["Because the intrinsic reward is gated to non-top-$k$ tokens, part of the diversity gain may come from upweighting low-probability tokens rather than from genuinely useful novelty; a direct way to test this is to compare CD-RLHF against a version with a random noise reward of matched scale.","The curiosity signal resembles an epistemic-uncertainty estimate, so the same ICM reward could double as a calibration measure; one could check whether the more diverse completions are also more accurate on factuality benchmarks.","The mechanism is not tied to PPO: the same reward could be plugged into DPO-style objectives or used at decoding time, which the paper leaves untested."],"forward_implications":["Output diversity can be improved during RLHF training itself, rather than through decoding-time tricks or auxiliary sentence-level rewards.","The diversity gains survive outside the training distribution: on MT-Bench, CD-RLHF scores higher than RLHF by an LLM judge and still produces more diverse outputs.","A single hyperparameter $\\eta$ and the top-$k$ gate give a practical dial: fewer intrinsic rewards (larger $k$) reduce diversity but keep alignment, while more frequent rewards eventually hurt alignment.","The alignment-diversity trade-off is shifted but not eliminated: CD-RLHF models remain less diverse than the SFT models they start from."],"supporting_citations":[{"why":"Supplies the ICM design: intrinsic reward as forward-dynamics prediction error.","marker":"Pathak et al., 2017"},{"why":"Shows latent-space prediction errors generalize better than raw observations, justifying hidden-state features.","marker":"Burda et al., 2019a"},{"why":"Supports prediction-error intrinsic rewards and their use as exploration bonuses.","marker":"Burda et al., 2019b"},{"why":"Documents the RLHF diversity-alignment trade-off that the paper targets.","marker":"Kirk et al., 2024"},{"why":"Provides the Sent-Rewards baseline and the notion of diversity-oriented RLHF rewards.","marker":"Hong et al., 2024"},{"why":"Supplies the TL;DR summarization dataset and preference-based reward setup.","marker":"Stiennon et al., 2020"},{"why":"Provides the PPO/RLHF training recipe with KL penalty that CD-RLHF extends.","marker":"Ouyang et al., 2022"},{"why":"Supplies MT-Bench, the out-of-distribution benchmark used for generalization gains.","marker":"Zheng et al., 2024"}],"fun_headline_variants":["Curiosity bonus boosts text diversity without hurting alignment","Intrinsic novelty reward diversifies RLHF outputs, alignment intact","Curiosity-driven RLHF lifts diversity 40% while holding quality","Novel-state bonus in RLHF improves diversity, preserves alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a small predictor's failure to guess the next hidden state marks a continuation that is worth exploring, rather than merely unpredictable or noisy.","fun_headline_variants_meta":{"raw":{"variants":["Curiosity bonus boosts text diversity without hurting alignment","Intrinsic novelty reward diversifies RLHF outputs, alignment intact","Curiosity-driven RLHF lifts diversity 40% while holding quality","Novel-state bonus in RLHF improves diversity, preserves alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000517,"raw_usage":{"total_tokens":2494,"prompt_tokens":917,"completion_tokens":1577,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":1508}},"tokens_in":533,"tokens_out":1577,"duration_ms":12307,"temperature":1.0,"reasoning_tokens":1508,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:14:09.837243+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the ICM prediction error with random noise of the same scale, keeping the top-$k$ gate and reward whitening identical; if diversity rises as much as with the real signal, prediction error is not the active ingredient. A complementary check is to measure factual accuracy of the diverse outputs on a ground-truth task such as summarization factuality or closed-book QA, since useful diversity should not systematically increase errors.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ICM design: intrinsic reward as forward-dynamics prediction error."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the RLHF diversity-alignment trade-off that the paper targets."},{"cited_title":"Glass, Akash Srivastava, and Pulkit Agrawal","cited_arxiv_id":null,"evidence_quote":"Provides the Sent-Rewards baseline and the notion of diversity-oriented RLHF rewards."}],"review_version":1}