{"id":"61c95ff8-5c02-4189-9593-f20580143392","arxiv_id":"2502.10482","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"CAGSR uses cross-attention coverage, focus, and repetition penalties as a self-supervised reward to fine-tune LLMs with PPO, claiming gains over no-RL baselines.","lead":"This paper proposes a reinforcement learning method that uses a language model's own cross-attention patterns to build a training reward, avoiding human preference labels. If valid, it would lower the cost of aligning LLMs with user instructions, though the authors report it still trails human-supervised RLHF.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The §4 policy-gradient estimator omits the dependence of R on θ: since R is built from the current model's cross-attention, the update is biased unless a frozen reference/stop-gradient is used, which the paper never states.","rationale":"The central claim is empirical, but its mechanism is a policy-gradient optimization of a self-supervised reward. The most load-bearing element is therefore the correctness of that optimization. The reader flagged the reward-independence assumption as a second issue; I agree and elevate it because it is an internal inconsistency, not a contestable modeling choice. If R depends on θ and no frozen reference is used, the gradient estimator in Section 4 is not the gradient of J(θ), and the method's stated convergence/improvement rationale collapses. The missing error bars and absence of code compound but do not replace this concern. The attention-as-proxy debate is real but secondary: even if attention were a perfect proxy, the biased update would still not implement the claimed RL objective. A re-derivation plus the three-condition experiment would settle whether the flaw is formal or also empirically damaging.","tokens_in":9554,"tokens_out":8204,"duration_ms":84738,"concrete_test":"Re-derive ∇_θ E_{x,y∼π_θ}[R_θ(x,y)] for the R defined in §3.2, including the ∂R/∂θ term, and compare with the estimator in §4. Then run the paper's Synthetic QA protocol in three conditions: (A) reward from a frozen initial model's attention, (B) reward from the live policy with gradients detached, (C) the corrected gradient including ∇_θ R. If final prompt-relevance or ROUGE-L between A/B/C differs by more than the paper's reported No-RL-to-CAGSR gap (0.11 relevance), the method as published is not optimizing the objective it claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 constructs R(x,y) from cross-attention vectors A_t^(ℓ) of the model being trained, and Section 4 derives the policy-gradient estimator by differentiating log π_θ while holding R(x,y) constant. That derivation is valid only if R is independent of θ. The paper never says that a frozen reference model or stop-gradient is used. If attention is taken from the policy that generated the rollout, then R depends on θ, and the true gradient of J(θ)=E_{x,y∼π_θ}[R_θ(x,y)] contains an additional term E[∇_θ R_θ(x,y)]. Omitting this term means the update is not a policy-gradient estimate of ∇J(θ). If the reward is instead detached, the optimization target changes every iteration because the attention maps and therefore R change as θ moves, so standard policy-gradient guarantees do not apply. Section 6 explicitly acknowledges 'reward hacking' through artificially spiky attention patterns, which is precisely the pathology this bias invites. As written, the theoretical foundation of CAGSR does not support the claim that it is maximizing the stated self-supervised objective.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CAGSR, a reinforcement-learning fine-tuning method for large language models that replaces human preference rewards with a self-supervised reward derived from the model's own cross-attention distributions. The reward combines a prompt-coverage term, a negative-entropy focus term, and a repetition penalty. Section 4 presents a policy-gradient/PPO update for this reward, and Section 7 reports improvements over a no-RL baseline and a synthetic-preference RL baseline on an instruction-following dataset, with a limited human-preference RL method as an upper bound. Section 8 claims novelty in using internal attention signals as the alignment signal without external human annotations.","tokens_in":9765,"tokens_out":4627,"duration_ms":47013,"significance":"If the proposed method worked as claimed, it would offer a scalable, low-label route to LLM alignment, which is a meaningful research direction. The paper is also honest about practical difficulties, explicitly discussing reward hacking and the possible value of hybrid human/self-supervised signals in Section 6. However, the central theoretical derivation is invalid as written because the reward is computed from the very parameters being optimized, so the stated policy-gradient objective is not well defined. The empirical section reports only point estimates with no variance, significance tests, or sufficient reproducibility details. The contribution therefore does not currently provide a sound foundation for the claimed results.","major_comments":[{"comment":"The reward R(x,y) is defined in Section 3.2 from the current model's cross-attention vectors A_t^(ℓ), which are a function of the policy parameters θ. In the policy-gradient derivation in Section 4, however, R(x,y) is differentiated only through log π_θ and is treated as a constant. The true gradient of J(θ) = E_{x,y∼π_θ}[R_θ(x,y)] contains an additional term E[∇_θ R_θ(x,y)]. Omitting this term means the update is not an unbiased estimate of ∇J(θ). If the reward is instead detached with a stop-gradient, the objective becomes non-stationary because R changes as θ moves, so the policy-gradient and PPO guarantees stated in Section 4 do not apply. The reward-hacking failure mode acknowledged in Section 6, artificially spiky attention patterns, is precisely the pathology this biased objective invites. This is a load-bearing flaw in the central claim that CAGSR maximizes the stated self-supervised objective.","section":null},{"comment":"Section 3.1 states that the method applies to decoder-only Transformers such as GPT-style models because they 'inherently utilize cross-attention in the decoder module.' Standard decoder-only architectures do not have a separate cross-attention module attending to an encoded prompt; they use self-attention over the full context, including the prompt tokens. The formal setup with cross-attention vectors A_t^(ℓ) is therefore not defined for the claimed decoder-only case. Since the experiments in Section 7.2 use an encoder-decoder model, this error does not invalidate the empirical setup, but it undermines the stated generality of the method.","section":null},{"comment":"The load-bearing assumption of the method is that high prompt coverage and low-entropy attention distributions correlate with better alignment and output quality. The paper cites 'Attention is not explanation' and 'Is attention interpretable?' as evidence of an ongoing debate, but provides no direct evidence for the positive claim that cross-attention quality, as defined by the reward components, tracks human judgments of response quality. The external metrics in Section 7.4 are used only for evaluation, not for validating the reward. A concrete test would be to measure, on a held-out set before training, the correlation between each reward component (coverage, negative entropy, repetition penalty) and human or BERT-based quality scores. Without such evidence, the reward is a heuristic whose alignment with the stated goal is unverified.","section":null},{"comment":"The central empirical claim of 'significant gains' rests on a single table of point estimates with no standard deviations, confidence intervals, statistical significance tests, or number of evaluation samples. The text reports one relevance value, one ROUGE-L value, and one human score per method, so the reader cannot assess whether the differences between CAGSR and the baselines are meaningful or stable. The ablation studies in Section 7.6 are described only in prose, with no quantitative results. Given that the theoretical foundation is also problematic, this level of experimental detail is insufficient to support the paper's conclusions.","section":null}],"minor_comments":[{"comment":"The caption says 'Figure X illustrates the complete pipeline,' which is a placeholder; the figure itself is not included in the manuscript.","section":null},{"comment":"Reference [5] lists 'X. Ouyang, J. Wu, X. Jiang, et al.' for the InstructGPT paper, but the actual author list is Long Ouyang, Jeff Wu, Xu Jiang, et al.; the abbreviated first names and the ordering should be corrected. Reference [7] also appears to have incorrect initials.","section":null},{"comment":"The sentence 'the LLM can be instantiated as either a decoder-only Transformer, encoder only (e.g. BERT) or an encoder-decoder Transformer' mentions encoder-only BERT, but BERT has no decoder and would not generate responses; this is confusing and should be clarified.","section":null},{"comment":"The reward hyperparameters are said to be 'calibrated via empirical experimentation,' but Section 7.2 fixes α=1.0, β=0.5, and γ=1.0 with no sensitivity analysis or description of the grid search; a brief summary of the search or a robustness check would help.","section":null}],"recommendation":"reject","confidential_remarks":"The manuscript reads like an early draft: it contains placeholder figure text, incomplete reference details, and no accompanying code or data. The gradient-dependence problem in Section 4 is not a presentational issue; it affects the validity of the paper's central derivation. If the authors can reformulate the method as an explicit non-stationary self-training objective with a detached reward and provide rigorous empirical validation, a future submission on this direction could be worth considering, but the current version is not ready for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's central gradient derivation is wrong because the reward is a function of the parameters being optimized, and the experiments are too thin to compensate. That said, the specific reward composition is new and the authors are honest about the gap to human RLHF.\n\nWhat's actually new: combining prompt coverage (IDF-weighted attention), negative-entropy focus, and a repetition penalty into one RL reward, with no human labels. I haven't seen that exact combination in the cited literature. The paper is also written clearly, and it explicitly acknowledges reward hacking and the possibility of spiky attention (Section 6), which is more candor than most.\n\nThe soft spot is not minor. Section 3.2 defines R(x,y) using cross-attention A_t^(ℓ) of the model being trained. Section 4 then derives the policy gradient by differentiating log π_θ while holding R constant. That step is valid only if R does not depend on θ. If attention is taken from the rollout policy, the gradient of E[R_θ] has an extra term E[∇_θ R_θ] that is dropped, so the update is biased. If the authors intend to detach attention, they never say so, and then the optimization target shifts every iteration because the attention maps (and therefore R) change as θ moves. Either way, the paper does not actually maximize the stated objective. Section 6's own admission of reward hacking confirms this is not a theoretical curiosity.\n\nSecondary issues: the paper cites Jain & Wallace and Serrano & Smith, which question attention as explanation, yet the whole reward rests on attention as a quality signal. No code, no data, no error bars; Table 1 reports single numbers. Ablation results are described qualitatively. The hyperparameters α, β, γ and L' are free parameters with a single grid search.\n\nBottom line: the idea is a reasonable hypothesis, and with a frozen reference model for the reward, plus real experiments, it might become a short workshop paper. As it stands, the central derivation is invalid and the evidence doesn't support the claims. I would not send this to peer review. If it lands on my desk, I'd reject with an invitation to resubmit after fixing the reward to be θ-independent and reporting uncertainty.","headline":"The CAGSR reward is built from the very policy being optimized, so the Section 4 gradient is biased; the empirical claims are too thin to save it, but the reward composition is new.","tokens_in":10304,"tokens_out":2527,"would_cite":false,"duration_ms":22755,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cross-attention scores fine-tune LLMs without human feedback","keywords":["self-supervised reinforcement learning","cross-attention","LLM fine-tuning","reward function design","prompt coverage","attention entropy","alignment","PPO"],"falsifier":"Train CAGSR on a set of prompts whose correct answers are known, then compare the model's reward $R(x,y)$ with human relevance ratings on a held-out set; if the reward does not rank-order the human ratings, the central premise that attention patterns track alignment is wrong. A sharper version: keep generated text fixed but rewire the last-layer attention maps at inference time to maximize $R$; if human relevance stays flat while $R$ jumps, the reward is measuring attention artifacts rather than output quality.","tokens_in":9334,"feed_emoji":"🎯","tokens_out":8532,"duration_ms":74163,"temperature":0.7,"pith_summary":"The paper proposes CAGSR, a reinforcement-learning fine-tuning method whose reward is computed entirely from the model's own cross-attention signals instead of human annotations. The reward rewards covering important prompt tokens, keeping attention distributions concentrated (negative entropy), and penalizing repetition. A policy-gradient/PPO loop then pushes the model toward outputs that score higher on these internal measures. On a 1.3B encoder-decoder model, CAGSR improves prompt relevance, ROUGE-L, and human ratings over a no-RL baseline and over RL with synthetic preference labels, while staying below a limited human-preference RL baseline. This is offered as a scalable path to alignment that consumes little or no human labeling.","feed_headline":"Cross-attention scores fine-tune LLMs without human feedback","feed_subtitle":"A 1.3B model trained with a coverage-and-focus reward reaches 0.83 prompt relevance, close to 0.87 with human labels.","key_machinery":"The central object is the cross-attention distribution $A_t^{(\\ell)}$ over prompt tokens at decoding step $t$ and decoder layer $\\ell$, aggregated over the last $L'$ layers (typically 1 or 3). The mechanism is the composite reward $R(x,y) = \\alpha\\,\\mathrm{coverage}(x,y) + \\beta\\,\\mathrm{focus}(x,y) - \\gamma\\,\\mathrm{repeatPenalty}(y)$, with coverage tracking attention on salient tokens $I_x$, focus given by negative entropy, and repeatPenalty penalizing repeated n-grams. This scalar reward enters the PPO objective through the advantage $A(x,y) \\approx R(x,y) - V_{\\theta_{\\mathrm{old}}}(x)$, so the model is trained to make its own attention patterns healthier by its own metric. The load-bearing step is using these stored attention vectors as ground truth for quality, closing the RL loop without external labels.","core_discovery":"The paper's central claim is that the cross-attention distributions a Transformer decoder already computes during generation can be turned into a reward signal that guides fine-tuning in the same direction as human preference data. The reward is $R(x,y) = \\alpha\\,\\mathrm{coverage}(x,y) + \\beta\\,\\mathrm{focus}(x,y) - \\gamma\\,\\mathrm{repeatPenalty}(y)$, where coverage sums the attention mass placed on a heuristic set of important prompt tokens, focus is the negative average entropy of the attention distribution, and repeatPenalty discourages n-gram repetition. With weights $\\alpha=1.0$, $\\beta=0.5$, $\\gamma=1.0$ on the Instruction dataset, CAGSR reaches prompt relevance $0.83$, ROUGE-L $0.39$, and a human score $3.7$, compared with $0.72/0.33/3.2$ for the no-RL baseline and $0.78/0.37/3.5$ for synthetic-preference RL; a limited human-preference RL baseline scores $0.87/0.44/4.0$. The paper reads this as evidence that internal attention health is a usable proxy for alignment, not yet a full replacement for human judgment.","pith_inferences":["Editorial inference: the gradient derivation in Section 4 treats $R$ as constant in $\\theta$, but the attention weights that define $R$ are computed by the policy being trained, so the update is biased; using a frozen reference model to supply the attention signals would make the estimator honest and might change the reported gains.","Editorial inference: the paper's own reward-hacking discussion suggests a stronger failure mode: the model could learn to produce artificially spiky attention maps that satisfy the reward without improving content; an entropy lower bound may not be enough if the model shifts off-topic tokens into $I_x$ via the heuristic token selection.","Editorial inference: one could test the proxy premise cheaply by computing $R$ on an existing LLM's outputs and correlating it with human quality ratings; the paper reports end-to-end training results but not this direct validation of the reward.","Editorial inference: for decoder-only models, cross-attention is unavailable, but attention over the prompt prefix in causal self-attention could supply an analogous signal, extending the idea to the most widely used LLM family."],"forward_implications":["If CAGSR is right, large-scale alignment fine-tuning can begin without collecting human preference data, and human labels can be reserved for final polishing rather than the main training signal.","The same reward pipeline applies to any decoder-only or encoder-decoder Transformer that exposes cross-attention, so the method transfers across architectures without new annotation pipelines.","Because the reward is defined by the model's own attention rather than a separately trained reward model, adapting the method to a new domain does not require collecting new preference labels; the same self-supervised signal transfers.","CAGSR can serve as a warm start for RLHF: a policy pre-trained with cross-attention rewards should require less human feedback to reach a given quality level than a policy trained from scratch.","The reported gains over synthetic-preference RL indicate that internal attention signals can be a stronger reward than automatically generated preference labels, at least on the tested instruction tasks."],"supporting_citations":[{"why":"Supplies the Transformer architecture and the cross-attention mechanism whose per-token distributions the reward is built from.","marker":"[11]"},{"why":"Raises the core objection that attention weights may not faithfully explain model decisions, the premise CAGSR depends on.","marker":"[3]"},{"why":"Second work questioning attention interpretability; it defines the risk the reward design must survive.","marker":"[9]"},{"why":"Provides Proximal Policy Optimization, the update rule that converts the self-supervised reward into policy changes.","marker":"[8]"},{"why":"Establishes the human-feedback RL fine-tuning paradigm that CAGSR compares against and aims to approximate.","marker":"[13]"},{"why":"Demonstrates the RLHF success that motivates scalable alignment without human labels.","marker":"[5]"},{"why":"Documents the cost and scalability bottleneck of human preference data, the problem CAGSR targets.","marker":"[1]"}],"fun_headline_variants":["Cross-attention scores replace human feedback in LLM fine-tuning","Attention-based reward fine-tunes LLMs without human labels","Self-supervised RL uses attention to align LLMs","Attention health as reward: LLM fine-tuning sans humans"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method stands or falls on whether the way the model distributes attention over the prompt is a trustworthy sign of how well the response follows the prompt; if attention can be sharp or diffuse for reasons unrelated to true relevance, the reward will train toward the wrong outputs.","fun_headline_variants_meta":{"raw":{"variants":["Cross-attention scores replace human feedback in LLM fine-tuning","Attention-based reward fine-tunes LLMs without human labels","Self-supervised RL uses attention to align LLMs","Attention health as reward: LLM fine-tuning sans humans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1515,"prompt_tokens":991,"completion_tokens":524,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":457}},"tokens_in":607,"tokens_out":524,"duration_ms":5314,"temperature":1.0,"reasoning_tokens":457,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T20:16:11.004478+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train CAGSR on a set of prompts whose correct answers are known, then compare the model's reward $R(x,y)$ with human relevance ratings on a held-out set; if the reward does not rank-order the human ratings, the central premise that attention patterns track alignment is wrong. A sharper version: keep generated text fixed but rewire the last-layer attention maps at inference time to maximize $R$; if human relevance stays flat while $R$ jumps, the reward is measuring attention artifacts rather than output quality.","supporting_citations":[],"review_version":1}