{"id":"f0d00a38-e3e9-4866-8319-f24ca9c10fad","arxiv_id":"1909.01214","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A reward function learned from human ratings, with no reference summaries, guides RL summarizers to produce summaries rated higher by humans than ROUGE-trained systems.","lead":"The authors train a reward model on human ratings of news summaries, then use that reward to train summarization systems without needing reference summaries. These systems produce summaries that human raters prefer over those trained with ROUGE rewards.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Learned reward's out-of-distribution validity is the load-bearing assumption; the human evaluation in §7 is too small to establish it.","rationale":"The paper's internal correlation study (§6) is competently done and supports the narrow claim that BERT+MLP+Pref correlates better than ROUGE with human ratings on summaries from the four training systems. The load-bearing step is the transfer to RL: the reward is trained on a narrow distribution of summaries and then used as an objective that actively searches for high-reward outputs. The only evidence for transfer is the human evaluation in §7, which is small (30 and 20 documents) and does not measure reward-human correlation on the RL systems' own outputs. This is a correctness risk rather than a disagreement with consensus; it could be resolved by the proposed out-of-distribution rating study. I therefore agree with the reader's weakest-assumption identification and see no reason to change the conditional verdict: the method is plausible and promising, but the abstract's strong claim should not be accepted as established until the reward's validity on RL-generated summaries is directly demonstrated. I am not flagging any issue of author conduct or internal inconsistency; the paper is clearly written and the reported experiments are meaningful, just insufficient for the strongest claim.","tokens_in":13290,"tokens_out":5569,"duration_ms":52232,"concrete_test":"Run a held-out human rating study: sample 100 CNN/DailyMail test documents; generate summaries with NeuralTD+LearnedReward and ExtAbsRL+LearnedReward; have crowdworkers rate each summary using the §3 Likert protocol (at least 5 ratings per summary). Compute the Spearman correlation between BERT+MLP+Pref scores and mean human ratings for these RL-generated summaries, separately for extractive and abstractive outputs. Compare it against (a) ROUGE-1/ROUGE-L correlations on the same summaries and (b) the in-distribution correlation from Table 2. If the learned reward's out-of-distribution correlation is not significantly higher than ROUGE, or falls well below the Table 2 value, the central transfer claim fails. Report confidence intervals for the human ratings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires the reward model (BERT+MLP+Pref, Table 2) to stay aligned with human quality when used as an RL objective. The model was trained on 2,500 summaries from four fixed systems: See et al. (2017) seq2seq/pointer and Paulus et al. (2018) ml/ml+rl (§3). RL training then optimizes this reward over outputs that are deliberately different from those training summaries. Any artifact in the reward—system-specific stylistic cues, length preferences, n-gram overlap with the source, or avoidance of certain constructions—can be amplified by the RL agent, a form of reward hacking. The paper's evidence that this does not happen consists of human ratings on 30 articles with 5 raters (§7.2) and 20 articles with 3 raters (§7.3). These evaluations compare average system-level ratings; they do not estimate the reward's correlation with human judgments on RL-generated summaries, and their sample sizes are too small to detect a modest drop in correlation. Even in-distribution, G-Pre/G-Rec for BERT+MLP+Pref are only .556/.608 (Table 2), leaving substantial room for the reward to be gamed. Without a direct test of reward validity on the RL output distribution, the claim that better rewards yield better summaries rests on an untested extrapolation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a reference-free reward function for reinforcement-learning-based document summarization. The reward is learned from 2,500 human ratings of summaries produced by four fixed systems on 500 CNN/DailyMail articles, using either a regression loss or a preference (cross-entropy) loss over summary pairs, with several text encoders (CNN-RNN, PMeans-RNN, BERT) and two reward architectures (MLP and SimRed). The authors report that BERT+MLP trained with the preference loss has significantly higher summary-level correlation with human judgments than ROUGE and other reference-based metrics. They then use this learned reward to train an extractive RL system (NeuralTD) and an abstractive system (ExtAbsRL), and report human evaluations on 30 and 20 test documents, respectively, in which summaries trained with the learned reward receive higher average human ratings than comparisons trained with ROUGE-based rewards.","tokens_in":13520,"tokens_out":5066,"duration_ms":52320,"significance":"If the central claim holds, the paper would show that a reward function learned from a small set of human ratings (2,500 summaries) can replace ROUGE as an RL training signal and yield summaries that humans prefer, without requiring reference summaries at either training or inference time. The strengths of the paper include a clean problem formulation, a careful cross-validated evaluation of the reward itself, an explicit comparison of regression versus preference learning, the release of code and models, and the inclusion of both extractive and abstractive RL systems. The authors also directly document the poor summary-level correlation of ROUGE, which is a useful negative result. However, the final human evaluations are small, the comparisons are partly confounded by architecture differences, and the out-of-distribution validity of the learned reward on RL-generated outputs is not directly tested; these issues limit the strength of the conclusions that can currently be drawn.","major_comments":[{"comment":"The reward model is trained on human ratings of summaries produced by four fixed systems (See et al. 2017; Paulus et al. 2018), and Table 2 evaluates it only on that same distribution via 5-fold cross-validation. The RL agents trained in §7 optimize this reward over outputs that are deliberately different from those training summaries, so the central claim that \"better rewards yield better summaries\" requires the reward to remain aligned with human quality on RL-generated outputs. This alignment is not directly tested: no correlation or agreement analysis is reported on the actual outputs of NeuralTD or ExtAbsRL. I request either a direct evaluation of the learned reward against fresh human ratings on RL-generated summaries, or a clearly stated limitation with appropriately weakened claims.","section":"§3 and §6, Table 2"},{"comment":"The extractive human evaluation compares NeuralTD+LearnedReward with Refresh and ExtAbsRL, which differ from NeuralTD not only in reward but also in architecture, training procedure, and extractor design. Because a NeuralTD model trained with ROUGE as the reward is not included, the observed human-rating differences cannot be attributed solely to the reward function. In addition, the abstract claims superiority over \"the state-of-the-art supervised-learning systems,\" but none of the supervised systems listed in Table 3 is included in the human evaluation in §7.2 or §7.3. Please add the missing controlled comparison (e.g., NeuralTD with ROUGE reward) or revise the claims to match the experiments actually performed.","section":"§7.2, Tables 3-4"},{"comment":"The human evaluation is based on only 30 documents with 5 raters (extractive) and 20 documents with 3 raters (abstractive). The reported t-tests are not corrected for multiple comparisons and are applied to ordinal Likert-scale ratings. Consequently, the reported differences (e.g., 2.52 vs. 2.27 and 2.20 vs. 1.75) have wide confidence intervals, and the \"Best%\" and \"Pref%\" proportions are based on very small counts. Please report confidence intervals (bootstrap or Bayesian), effect sizes, and the number of statistical tests performed, and interpret the results accordingly.","section":"§7.2 and §7.3, Tables 4-5"}],"minor_comments":[{"comment":"\"2,500 human ratings on 500 summaries\" should read \"2,500 summaries from 500 articles\" or \"2,500 human ratings of summaries for 500 articles.\"","section":"§8, Conclusion"},{"comment":"The statement that \"p-values between each pair of metrics/rewards can be found in the supplementary material\" is not verifiable from the arXiv version; please include the full pairwise significance table in the paper or in an accessible appendix.","section":"§6"},{"comment":"The sliding-window description gives an offset size of 128 tokens but not the window length; please specify both the window size and the stride for reproducibility.","section":"§5.1, BERT encoder"},{"comment":"The choice of α = 0.85 is reported without a sensitivity analysis; a small study over α values would increase confidence that the SimRed results are not tuned to a single fold.","section":"§5.2, Eq. (4)"},{"comment":"The lead-sentence bonus of 0.5 and the length factor of 1.2 are motivated by pilot studies, but the range of values tried and the selection criterion are not reported; please add these details for reproducibility.","section":"§7.1"},{"comment":"The phrase \"full-length ROUGE F-scores\" could be misread; clarify that these are standard ROUGE F1 scores computed on the complete generated summaries.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does the thing it claims: learn a no-reference reward from human ratings and use it as an RL reward for summarization, with human evals showing clear preference over ROUGE-reward systems. I think the central result holds up, with the usual caveats about small eval sizes and an abstract that overstates the comparison to supervised systems.\n\nWhat's new: learning rewards from human feedback isn't new, but this specific combination—a no-reference learnable reward used as an RL training signal for both extractive and abstractive systems, with human evaluation of the resulting outputs—is a real contribution. Preference loss over pairwise comparisons beats regression, BERT+MLP is the strongest reward model, and the paper's summary-level correlation study nicely documents ROUGE's weakness at distinguishing good from mediocre summaries.\n\nThe reward model is cross-validated, and the final systems are evaluated by human raters on new documents, so the central evaluation is out-of-sample. Effect sizes are large: extractive mean rating 2.52 vs 2.27 for Refresh, abstractive 2.20 vs 1.75 with 75% preferences. These are not subtle differences. The paper is also honest that their systems score lower on ROUGE.\n\nSoft spots: (1) Human evals use only 30 and 20 documents. Enough to detect large differences, but confidence intervals are wide and no multiple-comparison correction is applied. (2) The abstract says \"state-of-the-art supervised-learning systems\" are outperformed; the human-eval baselines are Refresh and ExtAbsRL, both RL-based. Pure supervised systems appear only in ROUGE tables, never in human evals. That's an overstatement. (3) \"Without reference summaries\" is qualified: ExtAbsRL uses reference length for stopping and a pre-trained abstractor, so the claim should be \"no references at reward time.\" (4) The out-of-distribution worry is real but not fatal: the reward was trained on four fixed systems and then optimized by RL agents. The human eval on RL outputs is the right test, and it passes, but it doesn't directly measure reward-human correlation on those outputs, so a modest drop could go undetected. That's a limitation, not a flaw.\n\nWho it's for: anyone working on summarization evaluation or RL for NLG. It deserves a serious referee. I'd send it out, with requested revisions to temper the abstract, add confidence intervals, and release evaluation code/data.","headline":"Learning a reference-free reward from human ratings genuinely works for RL summarization, though the human evals are small and the abstract oversells the supervised comparison.","tokens_in":14084,"tokens_out":2731,"would_cite":true,"duration_ms":26428,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Human-rated rewards beat ROUGE for training summarisers.","keywords":["reinforcement learning","text summarization","reward learning","human evaluation","reference-free evaluation","ROUGE","BERT","preference learning"],"falsifier":"Take a held-out set of articles, generate summaries with a policy trained on the learned reward, collect human ratings for those summaries, and measure the reward's Spearman correlation with the ratings; if it is no better than ROUGE's correlation, the central claim that the learned reward is a better training signal than ROUGE fails.","tokens_in":13082,"feed_emoji":"📝","tokens_out":4853,"duration_ms":43775,"temperature":0.7,"pith_summary":"This paper argues that the standard practice of using ROUGE as the reward signal in reinforcement-learning summarisers is flawed, because ROUGE's correlation with human judgement at the level of individual summaries is weak. The authors learn a reward function from human ratings on 2,500 summaries, taking only the source document and the generated summary as input, so it never needs a reference summary. They report that this learned reward correlates significantly better with human ratings than ROUGE and other metrics, and that both an extractive and an abstractive RL summariser trained with it produce summaries that human readers rate higher than those produced by state-of-the-art systems. If this holds, reference-free reward learning can replace ROUGE as the training objective for summarisation, using far fewer human annotations than the document-reference pairs currently required.","feed_headline":"Human-rated rewards beat ROUGE for training summarisers","feed_subtitle":"A learned reward from 2,500 ratings guides RL summarisers to summaries people prefer, no references needed.","key_machinery":"The central object is a learned reward function $R(x,y;\\theta)$ that scores a summary $y$ for document $x$ without any reference summary. Its key component is a pairwise preference loss: for every pair of summaries of the same document, the model is trained to prefer the one with the higher human rating, using a softmax over reward scores. The best-performing implementation encodes the document and summary with BERT, applied with a sliding window for long texts, and feeds the concatenated embeddings into a multilayer perceptron; a second architecture, SimRed, explicitly computes sentence-similarity and redundancy matrices. The same reward is then used both as the optimisation target for RL training and as the scoring function during inference.","core_discovery":"On its own terms, the paper claims that a reward function $R(x,y;\\theta)$ trained on 2,500 human overall-quality ratings of summaries for 500 CNN/DailyMail articles can stand in for reference-based metrics in RL summarisation. The best configuration, BERT+MLP with a pairwise preference loss, reaches a Spearman correlation of $0.505$ with human ratings, substantially above ROUGE-1's $0.290$, and improves the precision and recall of identifying 'good' summaries. In human evaluation, the extractive NeuralTD system trained on the learned reward scores $2.52$ on a 3-point scale, versus $2.27$ for Refresh and $1.66$ for ExtAbsRL, and the abstractive ExtAbsRL system rises from $1.75$ with ROUGE rewards to $2.20$ with the learned reward. The authors conclude that a small set of human ratings can replace the two orders of magnitude more document-reference pairs used by current systems, and that the idea extends naturally to other natural-language generation tasks.","pith_inferences":["If the reward generalises beyond CNN/DailyMail, the same recipe could replace ROUGE in other generation tasks such as machine translation and dialogue, where references are also scarce; the authors leave this as future work.","A natural next test is iterative training: use the reward to train a policy, collect new human ratings on that policy's summaries, retrain the reward, and see whether correlation with human judgement and downstream quality keep improving.","Because the reward is trained on summaries from only four fixed systems, it may be biased towards their error patterns; an out-of-distribution check on summaries from a very different summariser would clarify its robustness."],"forward_implications":["Summarisation systems can be trained end-to-end without any reference summaries, as long as a reward function has been learned from human ratings.","Human ratings collected once on a few thousand summaries can be reused to train many different summarisers, since the reward model is independent of the policy.","ROUGE should not be trusted as a summary-level reward for RL; the paper's correlation study shows it can barely distinguish good from bad summaries.","Both extractive and abstractive RL frameworks benefit from swapping in the learned reward, suggesting the result is not tied to one architecture."],"supporting_citations":[{"why":"Supplies the 2,500 human ratings over 500 articles that form the training data for the learned reward function.","marker":"Chaganty et al. (2018)"},{"why":"Defines ROUGE, the reference-based metric that the paper studies as a reward and compares against.","marker":"Lin (2004b)"},{"why":"Provides the CNN/DailyMail dataset and the pointer-generator systems whose summaries were rated by humans.","marker":"See et al. (2017)"},{"why":"Its ml and ml+rl systems generated rated summaries and serve as ROUGE-reward RL baselines.","marker":"Paulus et al. (2018)"},{"why":"Refresh is the ROUGE-trained extractive RL baseline that the learned-reward system is compared against in human evaluation.","marker":"Narayan et al. (2018b)"},{"why":"ExtAbsRL is the abstractive RL framework that the paper retrains with its learned reward.","marker":"Chen and Bansal (2018)"},{"why":"NeuralTD, the input-specific RL extractive system, is the architecture the paper combines with its learned reward.","marker":"Gao et al. (2019)"},{"why":"Provides the feature-rich learned-metric baseline that the paper outperforms without using reference summaries.","marker":"Peyrard and Gurevych (2018)"}],"fun_headline_variants":["Learned human-ratings reward beats ROUGE for summarizers","Train summarizers with learned rewards, no references","Human ratings as reward: higher-quality RL summaries","RL summarization improves using human-rated rewards"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a reward function trained on human ratings of summaries produced by four fixed systems will also give accurate scores to the different kinds of summaries an RL agent generates when optimising against it; the paper only tests this indirectly on 30 and 20 new articles in two human evaluations.","fun_headline_variants_meta":{"raw":{"variants":["Learned human-ratings reward beats ROUGE for summarizers","Train summarizers with learned rewards, no references","Human ratings as reward: higher-quality RL summaries","RL summarization improves using human-rated rewards"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000466,"raw_usage":{"total_tokens":2324,"prompt_tokens":944,"completion_tokens":1380,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":1317}},"tokens_in":560,"tokens_out":1380,"duration_ms":10569,"temperature":1.0,"reasoning_tokens":1317,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:24:20.445536+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of articles, generate summaries with a policy trained on the learned reward, collect human ratings for those summaries, and measure the reward's Spearman correlation with the ratings; if it is no better than ROUGE's correlation, the central claim that the learned reward is a better training signal than ROUGE fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 2,500 human ratings over 500 articles that form the training data for the learned reward function."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ExtAbsRL is the abstractive RL framework that the paper retrains with its learned reward."}],"review_version":1}