{"id":"92cdfc05-b809-4c64-9df7-27f16095bf74","arxiv_id":"2505.17910","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A reward-feedback fine-tuning framework trains a face reward model and uses its gradient plus structural and regularization losses to improve diffusion face restoration models.","lead":"DiffusionReward uses a face-specific reward model to fine-tune diffusion-based face restoration models, improving facial detail and identity consistency. It brings reward feedback learning, previously used for text-to-image generation, into the face restoration setting.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The FaceReward metric in Table 1 is circular: it uses the authors' own FRM, which is the optimization target, and the dynamic update makes it self-fulfilling, so it cannot support the SOTA claim.","rationale":"The paper's central claim is that DiffusionReward significantly enhances perceptual quality and identity consistency and outperforms SOTA. The most direct threat to the evidence for this claim is the circular evaluation metric: the authors use their own FRM, which is part of the optimization loop, as a headline evaluation metric. This is not merely a methodological nicety; it could fully explain the large FaceReward gaps (e.g., 0.4389 vs. 0.2864 for the next best method) without any genuine perceptual advantage. The reader's weakest assumption focuses on the dynamic update rule (Eqs. 7-8) as a potential failure mode for anti-reward-hacking. That is a legitimate and related concern, but it is more about the theoretical validity of a proposed mechanism. The circular evaluation is more load-bearing because it directly undermines the evidence used to support the SOTA claim. That said, the paper does report improvements on several external metrics (LPIPS, LMD, MUSIQ, FID, aesthetic), which are not optimized by the method and therefore provide some independent support. The absence of error bars and the inconsistent Table 3 further weaken confidence. Since the core issue is addressable by re-evaluation with non-circular metrics or human studies, the appropriate verdict remains CONDITIONAL, matching the reader's position. My read does not change the verdict; it sharpens the reason for conditionality.","tokens_in":17764,"tokens_out":7200,"duration_ms":80298,"concrete_test":"On a held-out set of 200 CelebA-Test images, generate outputs from OSEDiff(+ours), DiffBIR(+ours), CodeFormer, GFPGAN, and the corresponding base models. Run a forced-choice human preference study (e.g., 5 independent annotators per pair, majority vote) comparing each proposed output against each baseline, and compute win rates. Then correlate these human win rates with the FaceReward differences from Table 1. Separately, re-rank all Table 1 methods after removing the FaceReward column; check whether the proposed methods still occupy a majority of top-two positions across the remaining 10 metrics. If human win rates do not track FaceReward, or if the re-ranking substantially changes the SOTA conclusion, the circular metric is inflating the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central conclusion ('outperforming existing state-of-the-art methods') is supported in part by the 'FaceReward' column of Table 1, where OSEDiff(+ours) and DiffBIR(+ours) obtain the highest scores. But FaceReward is the output of the paper's own Face Reward Model (FRM), which is (i) fine-tuned on preference data labeled by an SVM trained on only 3,600 human annotations; (ii) used as the reward signal in L_reward = -R(\\hat{I}_HQ, T) (Eq. 2), i.e., the very objective the restoration network is optimized against; and (iii) further updated during training to prefer the ground-truth image over the current generator output (Eqs. 7-8). Hence, high FaceReward values for the proposed method are expected by construction and are not independent evidence of perceptual superiority. The dynamic update rule makes the circularity sharper: it re-trains the FRM to push the generator's outputs toward GT, so the FRM becomes a moving target aligned with the generator's objective. The paper provides no calibration of FaceReward against human judgment on the test set, so the reader cannot tell how much of the reported gain is genuine. This concern is load-bearing because the conclusion explicitly cites FaceReward among the metrics where the method achieves top rankings.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DiffusionReward, a reward-feedback learning framework for blind face restoration. It fine-tunes diffusion-based restorers (OSEDiff and DiffBIR) with a Face Reward Model (FRM) that scores restored images, combined with a structural consistency loss (LPIPS and DWT low-frequency) and a weight regularization term. The FRM is trained on a preference dataset with human and SVM-generated labels, and it is dynamically updated during training so that ground-truth images are always preferred over current restorations, with the stated goal of avoiding reward hacking. Experiments on synthetic and wild face datasets report improvements over the base models and prior SOTA across most of the 11 Table-1 metrics, including a new FaceReward score.","tokens_in":18119,"tokens_out":9496,"duration_ms":92429,"significance":"If the results hold, the paper is one of the first to transfer reward-feedback learning to blind face restoration, and the dynamic FRM update is a potentially instructive way to combat reward hacking. The paper is also transparent in releasing code, data, and models, and its ablation study decomposes the contribution of each loss term. However, the as-stated conclusion that the method 'outperforms existing state-of-the-art methods' is currently overreaching: the headline FaceReward metric is produced by the very model used as the training reward, several quantitative claims are internally inconsistent, and one loss expression is not well defined. These issues are fixable but require a revised evaluation and a careful rewriting of the affected passages.","major_comments":[{"comment":"The FaceReward column in Table 1 is not an independent evaluation metric. FaceReward is the output of the FRM, and the same FRM provides the training reward in Eq. (2) (L_reward = -R(restored, T)) and is periodically updated to prefer the ground truth over the current generator output (Eqs. 7-8). Therefore, high FaceReward values for the proposed method are expected by construction, and the Table-1 claim that the method achieves top rankings on FaceReward cannot be used as evidence of superiority over prior methods. The external validation of 87.78% vs. 63.05% on 360 pairs is a useful sanity check but is too small to calibrate the metric on the test sets and does not break the circularity. I recommend removing FaceReward from the comparison tables, or supplementing it with a human-preference study on the actual test images and reporting the SVM auto-labeler's accuracy.","section":"Sec. 4.2 (Table 1), Sec. 3.3 (Eq. 2), Sec. 3.3 (Eqs. 7-8)"},{"comment":"The weight regularization term is written as D_KL(theta || theta_base), but KL divergence is defined between probability distributions, not between parameter vectors, and no distribution over theta is specified. As written, Eq. (5) is not a computable loss. If the intent is a Gaussian KL with fixed covariance, it reduces to a scaled L2 distance and should be stated as such; if a distributional interpretation is intended, the distributions need to be defined. This matters because L_reg is one of the three components of the final objective in Eq. (6).","section":"Sec. 3.3, Eq. (5)"},{"comment":"The text states that 'our FRM obviously outperfoms HPS v2' on the basis of Table 3, but the table is mixed: FRM is better on MANIQA (0.6535 vs. 0.6630) and FID (42.59 vs. 48.94), but worse on MUSIQ (69.78 vs. 74.82). The claim of obvious superiority is not supported by the table as presented. Please specify the primary metric, report statistical significance, or qualify the conclusion.","section":"Sec. 4.3, Table 3"},{"comment":"The anti-reward-hacking effect of the dynamic FRM update is central to the method's novelty, but the current evidence is limited to one qualitative comparison and one row in Table 4. The assumption that always preferring the ground-truth image over the current restoration keeps the reward model on the real-face manifold is not demonstrated; an alternative possibility is that the FRM simply learns to reject the current generator's outputs, which could cause optimization to oscillate or drift. I would ask for direct quantitative evidence, such as FID or precision/recall of the restored images during training with and without the dynamic update, or a plot of FRM accuracy on held-out human preference pairs across updates.","section":"Sec. 3.3 and Sec. 4.3 (Table 4, Fig. 6(c))"}],"minor_comments":[{"comment":"The sentence 'we used five reference metrics' is followed by a list of six (SSIM, PSNR, LPIPS, CLIP Score, Deg., and LMD), and the sentence 'four non-reference metrics' is followed by a list of three (MUSIQ, MANIQA, and FID). Please correct the counts.","section":"Sec. 4.1"},{"comment":"The GFPGAN method is cited as [1] in the comparison list, but reference [1] is GLEAN; the GFP-GAN citation in Section 2 is [40]. Please fix the citation.","section":"Sec. 2 and Sec. 4.1"},{"comment":"The sentence that OSEDiff (+ours) and DiffBIR (+ours) 'achieve top rankings in the majority of metrics, such as Deg., LMD, Aesthetic, and FaceReward' is ambiguous: for Deg. and LMD, only DiffBIR (+ours) is the top method, while OSEDiff (+ours) is not. Please state which variant is top for each metric.","section":"Sec. 4.2"},{"comment":"The sentence 'we manually annotate 360 pairs' should clarify that these 360 pairs are a separate evaluation set from the 3,600 training pairs used to train the SVM, since the preceding paragraph describes 3,600 manually annotated pairs.","section":"Sec. 4.3"}],"recommendation":"major_revision","confidential_remarks":"The central issue for the editor is the circular evaluation: the FaceReward metric is produced by the same model that is optimized during training, so part of the reported improvement is expected by construction. I would not reject the paper on that basis, because the authors can remove FaceReward from the tables and still have a contribution, but the current version overstates the evidence. I also think the dynamic-update claim needs a quantitative check before publication. The paper is a reasonable fit for the journal, and the code/data/model release is a positive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something genuinely new: it brings reward feedback learning into blind face restoration, with a face-specific reward model, a dynamic update rule, and two auxiliary constraints. The framing is clear, the adaptation to off-the-shelf diffusion restorers (OSEDiff and DiffBIR) is sensible, and the reported gains on external metrics like LPIPS, LMD, FID, MUSIQ, and aesthetic score are consistent and non-trivial. That is real evidence the method improves perceptual quality and identity preservation.\n\nThe ablation study is also useful, especially the demonstration that the dynamic FRM update suppresses reward hacking artifacts. The small human-preference check (87.78% vs 63.05%) gives some independent validation that the reward model is capturing human judgment.\n\nNow the soft spots, in proportion. The biggest issue is the FaceReward metric in Table 1: it is the output of the same FRM used as the training reward, and the dynamic update makes the evaluator a moving target aligned with the generator's objective. That column cannot support the \"outperforming SOTA\" conclusion. The authors do not rely on it exclusively, but they do cite it among the top-ranked metrics, and that part of the claim is circular. They should either drop it or calibrate it against human labels on the test set.\n\nThe KL regularizer in Eq. (5) is ill-defined: KL divergence on parameter vectors needs a distributional interpretation, and none is given. Table 3 also does not clearly support the text claim that FRM \"obviously outperforms\" HPS v2, since MUSIQ is worse (69.78 vs 74.82), though FID is better. No error bars or significance tests appear anywhere, so we do not know how stable the improvements are.\n\nThe dynamic update rule (labeling every current restoration as the loser against GT) is a plausible anti-reward-hacking measure, but it could also just teach the reward model to reject the generator's outputs without guiding it toward better behavior. The qualitative results and the ablation partially allay this concern, but the mechanism deserves a clearer analysis.\n\nOverall, the central idea holds up. The circular metric is a blemish, not a fatal flaw, because the external metrics carry most of the weight. This paper is aimed at people working on face restoration or on reward-based fine-tuning of diffusion models. It deserves a serious referee. I would engage with it, and I would ask for the circular metric to be excised or recalibrated, the KL term to be defined, Table 3 to be reconciled, and error bars to be added.","headline":"A solid first application of reward feedback learning to blind face restoration, with real gains on external metrics, but the FaceReward evaluation column is circular and should not support the SOTA claim.","tokens_in":18594,"tokens_out":1829,"would_cite":true,"duration_ms":19847,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Blind face restoration can be improved by reward feedback fine-tuning of diffusion restorers, yielding better perceptual quality, richer detail, and stronger identity consistency than prior state-of-the-art methods.","keywords":["blind face restoration","reward feedback learning","diffusion models","face reward model","reward hacking","perceptual quality","identity consistency","fine-tuning"],"falsifier":"Run the same DiffusionReward training with the Face Reward Model frozen and every other component identical. If the frozen-reward version matches the dynamic-update version on perceptual quality, identity metrics, and artifact-free appearance on an unseen wild test set, then the dynamic update is not the load-bearing anti-reward-hacking mechanism. A second check: monitor the FRM score of restored images relative to real faces during training; if the score of restored images rises above the score of real faces, the claim that the dynamic update keeps outputs on the real-face manifold fails.","tokens_in":17558,"feed_emoji":"🖼️","tokens_out":6847,"duration_ms":61771,"temperature":0.7,"pith_summary":"This paper argues that blind face restoration, which must reconstruct a plausible high-quality face from a heavily degraded input, can be improved by fine-tuning the diffusion restoration model itself with reward feedback. The authors propose DiffusionReward, a training loop in which a Face Reward Model scores the restored image against a text description of the original face, and the resulting gradient is pushed through the last denoising step of an off-the-shelf diffusion restorer. Two auxiliary terms keep the fine-tune honest: a structural consistency loss preserves facial fidelity and identity, and a weight regularizer keeps the model close to its pre-trained state to retain generative diversity. During training the reward model is periodically re-trained to prefer the ground-truth face over the current restoration, which the authors argue prevents reward hacking. The reported result is that both a single-step restorer and a multi-step restorer, upgraded this way, beat the prior state of the art on synthetic and real-world face benchmarks.","feed_headline":"Reward feedback sharpens blind face restoration","feed_subtitle":"Fine-tuning diffusion restorers with a face reward model improves texture and identity consistency.","key_machinery":"The load-bearing mechanism is the parameterized denoising process: a latent diffusion restorer $g_\\theta$ is viewed as a generator, and gradients from the reward loss are backpropagated through the final denoising step, with the truncation window set to one step. The guiding signal combines the Face Reward Model score $R(\\hat{I}_{HQ},T)$, a structural consistency loss made of LPIPS plus a DWT low-frequency term, and a KL weight regularization toward the initial parameters. The anti-reward-hacking device is the dynamic Face Reward Model update, which after every $n$ generator iterations re-trains the scorer to prefer the ground-truth image over the current restoration using the softmax preference probability in Eqs. (7)-(8).","core_discovery":"On its own terms, the central discovery is that the denoising trajectory of a diffusion-based face restoration model can be treated as a differentiable generator and steered by a face-specific reward model, and that this steering materially improves restoration quality. Trained on pairwise human preferences distilled into a CLIP-style scorer, the Face Reward Model supplies the main gradient; the LPIPS and DWT low-frequency losses anchor the output to the original identity; and the KL weight regularizer prevents the fine-tune from destroying the base model's generative prior. The dynamic update of the reward model, declaring the current restoration the loser against its ground-truth image and re-optimizing the scorer accordingly, is presented as the mechanism that keeps the restored faces on the real-face manifold instead of drifting into high-scoring but unnatural artifacts. With DiffBIR and OSEDiff as base models, the paper reports consistent gains in no-reference quality, aesthetic score, identity distance, and its own FaceReward metric, and a sharp drop in the reward-hacking artifacts shown in its ablation.","pith_inferences":["If the dynamic Face Reward Model update genuinely prevents reward hacking, the same two-loop scheme could be applied to other conditional diffusion restoration tasks, such as super-resolution of text, hands, or medical images, wherever a domain-specific preference scorer can be trained.","The hybrid annotation strategy, human labels plus an SVM trained on six image-quality metrics, means the reward model's ceiling is partly set by those six metrics; a preference scorer trained on richer or more diverse human judgments could plausibly yield larger gains than the ones reported.","The weight-regularization result suggests that reward fine-tuning is most effective as a small, local adjustment to a strong pre-trained restorer, rather than as a full retraining recipe.","Because the same base model is used before and after fine-tuning, the reported comparisons may understate the practical value: the method converts an existing deployment into a better restorer at no additional inference cost."],"forward_implications":["State-of-the-art diffusion restorers can be upgraded by a post-training reward-feedback pass without changing their inference-time architecture or sampling cost.","Reward feedback learning transfers from text-to-image generation to restoration tasks, so any restoration domain with a reliable preference signal becomes a candidate for the same treatment.","Dynamic reward updates plus weight regularization give a working countermeasure to reward hacking as it appears in restoration, suppressing stylized painterly outputs and repetitive blemish artifacts.","The framework applies to both single-step and multi-step diffusion restoration models, suggesting the mechanism is tied to the denoising parameterization rather than to one specific base model."],"supporting_citations":[{"why":"Establishes the reward feedback learning paradigm for text-to-image diffusion models, from which the paper's gradient-flow optimization is adapted.","marker":"[48]"},{"why":"Shows direct fine-tuning of diffusion models on differentiable rewards and motivates truncated backpropagation and the reward-hacking concern.","marker":"[5]"},{"why":"Supplies the pre-trained human-preference scoring model used to initialize the Face Reward Model and to define the dynamic-update preference probability.","marker":"[46]"},{"why":"DiffBIR, the multi-step diffusion restoration base model on which the method is trained and evaluated.","marker":"[26]"},{"why":"OSEDiff, the one-step diffusion restoration base model on which the method is trained and evaluated.","marker":"[45]"},{"why":"Provides the latent diffusion backbone and autoencoder used by both base models, defining the latent space in which gradients flow.","marker":"[33]"},{"why":"The CLIP architecture underlying the Face Reward Model, with its image and text encoders and cosine-similarity scoring.","marker":"[32]"},{"why":"Characterizes reward gaming, the failure mode that the dynamic reward-update strategy is designed to prevent.","marker":"[36]"}],"fun_headline_variants":["Reward feedback guides diffusion to restore faces better","Face reward model steers restoration to keep identity","Dynamic reward tuning sharpens blind face regeneration","DiffusionReward improves face details via reward learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dynamic reward-model update assumes that labeling every current restoration as a loser against its own ground-truth image teaches the reward model to stay on the real-face manifold; if that update only makes the reward model reject the current generator's outputs without guiding it toward better behavior, the paper's main anti-reward-hacking claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Reward feedback guides diffusion to restore faces better","Face reward model steers restoration to keep identity","Dynamic reward tuning sharpens blind face regeneration","DiffusionReward improves face details via reward learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1257,"prompt_tokens":1014,"completion_tokens":243,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":184}},"tokens_in":630,"tokens_out":243,"duration_ms":3241,"temperature":1.0,"reasoning_tokens":184,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:37:59.405991+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same DiffusionReward training with the Face Reward Model frozen and every other component identical. If the frozen-reward version matches the dynamic-update version on perceptual quality, identity metrics, and artifact-free appearance on an unseen wild test set, then the dynamic update is not the load-bearing anti-reward-hacking mechanism. A second check: monitor the FRM score of restored images relative to real faces during training; if the score of restored images rises above the score of real faces, the claim that the dynamic update keeps outputs on the real-face manifold fails.","supporting_citations":[{"cited_title":"Imagereward: learning and evaluating human preferences for text-to-image generation","cited_arxiv_id":null,"evidence_quote":"Establishes the reward feedback learning paradigm for text-to-image diffusion models, from which the paper's gradient-flow optimization is adapted."},{"cited_title":"Diffbir: Toward blind image restoration with generative diffusion prior","cited_arxiv_id":null,"evidence_quote":"DiffBIR, the multi-step diffusion restoration base model on which the method is trained and evaluated."},{"cited_title":"One-step effective diffusion network for real-world image super-resolution.Advances in Neural Information Processing Systems (NeurIPS), 2024","cited_arxiv_id":null,"evidence_quote":"OSEDiff, the one-step diffusion restoration base model on which the method is trained and evaluated."},{"cited_title":"High- resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Provides the latent diffusion backbone and autoencoder used by both base models, defining the latent space in which gradients flow."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"The CLIP architecture underlying the Face Reward Model, with its image and text encoders and cosine-similarity scoring."},{"cited_title":"Defining and characterizing reward gaming.Advances in Neural Information Processing Systems, 35:9460– 9471, 2022","cited_arxiv_id":null,"evidence_quote":"Characterizes reward gaming, the failure mode that the dynamic reward-update strategy is designed to prevent."}],"review_version":1}