{"id":"894b4674-1eef-48bc-b39b-de0d98c72a1c","arxiv_id":"2501.11828","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"FPG combines history-aware attention and contrastive learning to generate personalized news headlines that preserve article facts, reporting improved FactCC and ROUGE scores on PENS.","lead":"This paper introduces FPG, a transformer-based model for personalized news headline generation that weighs facts from the article according to a reader's click history and uses contrastive training to keep headlines factually faithful. The authors report gains over existing systems on the PENS benchmark, but the evaluation leans on the same metric used to build training data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Factual-consistency claim rests on an unvalidated, potentially circular FactCC metric; FPG's personalization gains over BART are also marginal, so the 'balance' is not established.","rationale":"The reader's weakest assumption correctly flags the FactCC evaluation as the Achilles heel. I agree that the absence of an independent factual-consistency oracle makes the main contribution unverifiable. I would add two nuances. First, the paper's D* construction (Section IV-D) is ambiguous: 'high factual accuracy scores' is never defined, so the claimed circularity is plausible but not proven; this ambiguity is itself a reason to demand disclosure. Second, Table II reveals that FPG's PC scores are close to BART, so on the paper's own personalization metric FPG is only marginally more personalized than a non-personalized model. The high ROUGE against user-specific references provides some evidence of personalization, but it does not rescue the factuality side. The proposed concrete test—an independent QA-based or human factual consistency evaluation—would settle the central claim. If FPG's advantage over BART under such a metric is not significant, the 'fact-preserved' contribution collapses. Therefore the appropriate verdict remains CONDITIONAL, requiring the disclosed scorer, independent evaluation, and variance reporting.","tokens_in":9463,"tokens_out":9759,"duration_ms":100671,"concrete_test":"Instruct the authors to (1) disclose the exact scorer used to construct D* and whether it is FactCC; (2) evaluate FPG-GRU, BART, and PENS-NAML on the PENS test set using an independent factual consistency metric (e.g., QAFactEval or manual annotation of a random sample of 200 headlines, with inter-annotator agreement); (3) report bootstrap confidence intervals for FactCC, ROUGE, and PC. If FPG-GRU's FactCC advantage over BART disappears or is not significant under the independent metric, the factual-consistency improvement and the 'balance' claim are unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that FPG improves factual consistency while preserving personalization—is supported almost entirely by FactCC (Table II), a weakly supervised model-based scorer whose judgments are not ground truth. Section IV-D constructs the contrastive training set D* using 'high factual accuracy scores' without specifying the scorer; if the same FactCC is used to select positives and to evaluate, the reported FactCC gains (89.26 vs 86.67 for BART) are a training-to-metric artifact. Even if D* simply uses original headlines, no independent human or QA-based factual consistency evaluation is provided, so the model could be exploiting FactCC's known bias toward extractive overlap. Furthermore, the personalization component of the 'balance' is weak: FPG-GRU's PC(avg/max) (2.88/17.27) are nearly indistinguishable from non-personalized BART (2.72/17.13) and far below personalized baselines (PENS-NAML: 3.93/22.73). The high ROUGE against user-specific references partially offsets this, but the factuality side remains unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FPG (Fact-Preserved Personalized News Headline Generation), an encoder-decoder framework for personalized news headline generation. FPG augments a BART-initialized Transformer with a GRU-based history encoder, a personalized news encoder that uses history-cross attention between the candidate news body and the user's clicked headlines, and a user-guided decoder that replaces the [BOS] token with a fact-aware global user embedding. A contrastive fact-enhanced training stage is added to improve factual consistency. Experiments on the PENS benchmark report that FPG-GRU outperforms personalized and non-personalized baselines on ROUGE-1/2/L, FactCC, and the paper's personalization metric PC(avg)/PC(max), with a case study illustrating the desired behavior.","tokens_in":9689,"tokens_out":4287,"duration_ms":44821,"significance":"The task is timely and practically important: personalized headline generation must balance user appeal with faithfulness to the source article. The proposed architecture is clean and the use of a public benchmark (PENS) is a strength. If the factual-consistency gains were independently verified, the paper would make a useful contribution. However, the central claim currently rests on FactCC, a weakly supervised model-based scorer whose validity for this task is not established, and the personalization gains over the non-personalized BART baseline are numerically small. The paper also reports no significance tests despite claiming 'significantly outperforming.' These issues limit the strength of the empirical conclusions but are addressable with additional evaluation.","major_comments":[{"comment":"The factual-consistency evaluation relies entirely on FactCC, a weakly supervised model-based scorer, with no validation of its reliability for generated news headlines. The paper provides no correlation of FactCC with human judgments, no human annotation of outputs, and no independent entailment- or QA-based metric. Since the central claim is improved factual consistency, an independent evaluation is required; otherwise the reported FactCC advantage of FPG-GRU (89.26 vs. 86.67 for BART) cannot be interpreted as greater faithfulness.","section":"Section V-C and Table II"},{"comment":"The contrastive training set D* is constructed by selecting 'prominently ranked headline samples with high factual accuracy scores' without specifying the scoring method, while the evaluation uses FactCC, a model-based consistency scorer. If the same scorer family is used both to select positive examples and to score generated outputs, the reported FactCC gain partly reflects training to the metric. Please specify the exact scoring procedure used to build D*, and either use gold/original headlines as positives or verify the positive and negative examples with human judgments.","section":"Section IV-D and Section V-C"},{"comment":"The text claims that FPG-GRU 'significantly outperforms' baselines, but only single-run results are reported, with no error bars, no multiple seeds, and no significance tests. This is particularly important for the small gaps in ROUGE-1 (27.33 vs. 26.27 for BART) and FactCC (89.26 vs. 86.67). Please report variance across at least three runs and provide paired significance tests, or soften the significance claim accordingly.","section":"Table II and Section VI-A"},{"comment":"The personalization metric PC(avg)/PC(max) measures cosine similarity between the generated headline and the user's historically clicked headlines. This metric can reward overlap with generic or repeated vocabulary and does not directly measure user-specific preference. More importantly, FPG-GRU's PC(avg)/PC(max) values (2.88/17.27) are close to the non-personalized BART baseline (2.72/17.13) and much lower than the personalized PENS-NAML baseline (3.93/22.73). The claim of striking a 'balance' needs additional supporting evidence, such as human evaluation of personalization or agreement with annotator-written personalized headlines, since on the reported metric FPG appears closer to BART than to personalized baselines.","section":"Section V-C, Eq. (9), and Table II"}],"minor_comments":[{"comment":"The line 'Froze ξ to train the history encoder' is ambiguous because the subsequent update step minimizes Eq. (7) over θ; please state explicitly which parameter groups are updated in each of the four stages.","section":"Algorithm 1"},{"comment":"The rule-based methods used to create negative examples are not enumerated; please specify them (e.g., entity substitution, numeric changes, negation) and state how many negatives are paired with each positive.","section":"Section IV-D"},{"comment":"The text says 'The epoch number for the pre-trained phrase is 5'; this should read 'pre-trained phase,' and the list of learning rates should be explicitly mapped to the four training stages.","section":"Section V-D"},{"comment":"The training sets D10 and D3 appear in Table I but are not described in the text; please clarify how the limit l=10 and l=3 are used and why D5 is preferred for the main experiments.","section":"Section V-A and Table I"},{"comment":"The notation D* is used in Algorithm 1 before it is defined in Section IV-D; consider reordering or adding a forward reference.","section":"Algorithm 1 and Section IV-D"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant problem and the architecture is intuitive, but the factual-consistency evidence is the main risk. I would ask the authors to add an independent factuality evaluation (human or QA/entailment-based), report statistical significance, and provide a more direct analysis of the personalization-factuality trade-off. Without these, the central claim of achieving a 'balance' is not fully supported by the reported data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a clear, well-described system paper for a narrow but real task, and the architecture is the best part. The new bit is combining history-candidate cross-attention to build a fact-aware user embedding with a contrastive training stage for factual consistency. That is a sensible design and the paper explains it well. I also give credit for comparing against a full set of baselines, including BART, which is the right bar, and for showing a concrete case study.\n\nThe soft spots are in the evaluation, and they are load-bearing. FactCC is doing double duty: the contrastive positives are selected using \"high factual accuracy scores\" without naming the scorer, and factual consistency is later measured with FactCC. If the same scorer is used in both places, the reported FactCC gain (89.26 vs 86.67 over BART) is partly training to the metric. Even if it is not the same scorer, there is no independent oracle here — no human annotation, no QA-based consistency check, nothing to show the model is genuinely more faithful rather than better at matching FactCC's surface-overlap biases. That is the main thing to fix.\n\nThe personalization side is also weaker than the paper claims. FPG-GRU gets PC(avg/max) of 2.88/17.27, barely above non-personalized BART at 2.72/17.13, and far below personalized baselines like PENS-NAML at 3.93/22.73. So the \"balance between personalization and factual consistency\" is not demonstrated by the personalization numbers; the ROUGE gains are real but ROUGE against user-specific references is a weak personalization signal by itself.\n\nMinor but worth noting: Table II reports single runs with no error bars or significance tests despite the text saying \"significantly outperforming,\" and no code or data is released. The D5 filter is also a free parameter that deserves a sensitivity analysis.\n\nThe circularity concern is real but not a sign of incoherence — the model itself is sensible. It is a measurement problem. This is a revise-and-resubmit, not a reject. Who is it for? People working on personalized headline generation or news summarization will get value from the architecture and the cautionary tale about evaluation. I would bring it to reading group and would send it to review, but the authors need to release artifacts and add an independent factuality evaluation before the central claim should be trusted. I would not cite the headline numbers as evidence until that happens.","headline":"Useful architecture, shaky evaluation: the factual-consistency claim is built on an unvalidated FactCC loop and the personalization gains over BART are too small to call a balance.","tokens_in":10231,"tokens_out":2172,"would_cite":false,"duration_ms":25007,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Personalized news headlines can stay faithful to the facts, not just to the reader's tastes, if user interests are tied to the candidate article and trained with contrastive examples.","keywords":["personalized news headline generation","factual consistency","user embedding","contrastive learning","encoder-decoder","PENS benchmark","abstractive summarization","attention"],"falsifier":"A human-annotation study on a sample of generated headlines could settle the claim: take roughly 200 headlines from FPG and BART on the PENS test set, have three annotators mark factual errors, and compare. If FPG does not beat BART on human-judged faithfulness, the FactCC improvement does not reflect genuine factual consistency.","tokens_in":9209,"feed_emoji":"📰","tokens_out":3294,"duration_ms":36069,"temperature":0.7,"pith_summary":"The paper argues that personalized news headline generation does not have to sacrifice factual consistency for user appeal. It proposes FPG, which uses the similarity between a candidate news article and a user's clicked history to weight which facts matter, producing a fact-aware user embedding that guides decoding. A contrastive training stage then pulls generated headlines toward factually consistent examples and away from deliberately distorted ones. On the PENS benchmark, FPG reports the highest coverage, factuality, and personalization scores among the compared systems, suggesting the tension assumed by earlier work is not forced.","feed_headline":"A fact-aware user embedding keeps personalized headlines honest","feed_subtitle":"New training scheme beats prior personalized headline systems on consistency and coverage without losing personalization.","key_machinery":"The central object is the fact-aware global user embedding: a weighted sum of historical clicked-headline representations, where the weights come from a history-cross attention sub-layer that measures similarity between the candidate news body and each clicked headline. This embedding replaces the standard beginning-of-sentence token in the decoder, so it steers every decoding step toward user-relevant facts without injecting arbitrary user preferences. The second piece of machinery is a fact-enhanced contrastive training loss that uses FactCC-ranked positive examples and rule-based negative examples to teach the model to prefer factually consistent continuations.","core_discovery":"The paper claims that blindly injecting a global user interest embedding into a headline decoder causes factual drift, especially when none of the user's historical clicks relate to the candidate news. FPG replaces that with a fact-aware global user embedding computed as a similarity-weighted sum of clicked-news representations, and it adds a contrastive learning phase that trains the generator toward factually consistent headlines and away from rule-based factual corruptions. In the reported experiments, FPG-GRU attains a FactCC score of 89.26, ROUGE-1 of 27.33, and personalization score of 2.88, compared with BART's 86.67, 26.27, and 2.72, which the paper reads as evidence that personalization and factual consistency can be pursued together.","pith_inferences":["If FactCC can be gamed by surface overlap, part of the reported consistency gain may be metric optimization rather than genuine faithfulness; a human-annotation or question-answering factuality probe would separate the two.","A clean ablation would replace the learned similarity weights with uniform or random weights while keeping the contrastive stage; if FactCC stays high, the contrastive loss may be carrying most of the consistency improvement.","The same fact-aware user embedding idea could transfer to personalized summarization of financial or medical content, where a false entity association is costlier than in entertainment news."],"forward_implications":["Factual consistency can be injected at the decoder's input side and through training objectives, rather than only through post-hoc filtering or reranking.","A user-conditioned headline generator can preserve source facts while still producing distinct headlines for different readers, because the user signal is aligned to the candidate article's content.","The contrastive training recipe could generalize to other user-conditioned text generation tasks where global user vectors are known to contaminate factual content.","The similarity-weighted attention gives the model a natural way to down-weight irrelevant historical clicks, reducing the risk of injecting misinformation from unrelated interests."],"supporting_citations":[{"why":"Defines the PENS benchmark, the personalized headline generation task, and supplies the personalized baselines and test protocol.","marker":"[11]"},{"why":"Provides the pre-trained BART encoder-decoder that initializes FPG and serves as the strongest non-personalized baseline.","marker":"[25]"},{"why":"Supplies the Transformer backbone on which the encoder, decoder, and attention sub-layers of FPG are built.","marker":"[13]"},{"why":"Motivates the contrastive fact-enhanced training procedure for improving factual consistency in abstractive generation.","marker":"[20]"},{"why":"Supplies FactCC, the weakly supervised model-based scorer used both to select positive contrastive examples and to evaluate factual consistency.","marker":"[45]"}],"fun_headline_variants":["Fact-aware user embedding personalizes without distorting facts","Similarity-weighted facts keep personalized headlines truthful","New framework balances personalization and factual consistency","Contrastive training enhances fact consistency in personalized headlines","Fact-preserved personalization: no more distortion in user-targeted headlines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the automatic factuality score it uses, FactCC, genuinely measures whether a headline states the article's facts rather than merely resembling the source text; this same score picks the positive training examples and is the reported consistency metric, so if it can be fooled by word overlap, the consistency gains may be an artifact of training to the metric.","fun_headline_variants_meta":{"raw":{"variants":["Fact-aware user embedding personalizes without distorting facts","Similarity-weighted facts keep personalized headlines truthful","New framework balances personalization and factual consistency","Contrastive training enhances fact consistency in personalized headlines","Fact-preserved personalization: no more distortion in user-targeted headlines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000337,"raw_usage":{"total_tokens":1821,"prompt_tokens":855,"completion_tokens":966,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":890}},"tokens_in":471,"tokens_out":966,"duration_ms":7409,"temperature":1.0,"reasoning_tokens":890,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:48:07.979050+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A human-annotation study on a sample of generated headlines could settle the claim: take roughly 200 headlines from FPG and BART on the PENS test set, have three annotators mark factual errors, and compare. If FPG does not beat BART on human-judged faithfulness, the FactCC improvement does not reflect genuine factual consistency.","supporting_citations":[{"cited_title":"PENS: A dataset and generic framework for personalized news headline generation,","cited_arxiv_id":null,"evidence_quote":"Defines the PENS benchmark, the personalized headline generation task, and supplies the personalized baselines and test protocol."},{"cited_title":"BART: Denoising sequence-to- sequence pre-training for natural language generation, translation, and comprehension,","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained BART encoder-decoder that initializes FPG and serves as the strongest non-personalized baseline."},{"cited_title":"Attention is all you need,","cited_arxiv_id":null,"evidence_quote":"Supplies the Transformer backbone on which the encoder, decoder, and attention sub-layers of FPG are built."},{"cited_title":"Improving factual consistency of abstractive summarization via question answering,","cited_arxiv_id":null,"evidence_quote":"Motivates the contrastive fact-enhanced training procedure for improving factual consistency in abstractive generation."},{"cited_title":"Evaluating the factual consistency of abstractive text summarization,","cited_arxiv_id":null,"evidence_quote":"Supplies FactCC, the weakly supervised model-based scorer used both to select positive contrastive examples and to evaluate factual consistency."}],"review_version":1}