{"id":"47bcb494-6873-4fe4-8d86-69a9d0cbbe9b","arxiv_id":"2411.18824","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"FaithDiff restores high-resolution images by jointly fine-tuning the VAE encoder, an alignment module, and the SDXL diffusion model, and reports improved perceptual metrics and OCR accuracy over state-of-the-art SR methods.","lead":"FaithDiff is a new image super-resolution method that fine-tunes a text-to-image diffusion model, instead of keeping it frozen, while using a small alignment module to feed low-quality image features into the diffusion process. It reports better perceptual quality and OCR readability than several recent diffusion-based methods on synthetic and real benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Real-world faithfulness claim is untested: evidence is limited to no-reference metrics on 238 real images and an OCR test on synthetic degradations, so the 'faithful structure' claim for real photos is not established.","rationale":"The reader's weakest assumption identifies the synthetic-to-real degradation gap, and I agree that this is the most load-bearing point: the paper's abstract and §1 promise faithful restoration of real-world images, but the evidence for that is methodologically weak. I considered two other candidates: (i) the inconsistency between Eq. 2 and Eq. 3 in how the alignment features are passed to the denoising network, and (ii) the risk that fine-tuning the VAE encoder makes the frozen VAE decoder incompatible with the drifted latent distribution. On inspection, (i) appears to be a notation shorthand, and (ii) depends on whether the HQ encoding path uses a frozen original encoder or the fine-tuned one; the paper is ambiguous but a separate LQ encoder copy would avoid the issue. The degradation concern is more clearly load-bearing because it directly undermines the 'faithful' real-world claim regardless of architecture details. The proposed test—evaluating with paired real-world references or a real-world OCR benchmark—would settle whether the fine-tuning strategy actually preserves structure under degradations it was not trained on. If the gains vanish, the conclusion should be restricted to the synthetic degradation distribution. Thus the reader's CONDITIONAL verdict remains appropriate.","tokens_in":13641,"tokens_out":9311,"duration_ms":81679,"concrete_test":"Run FaithDiff on real-world SR datasets with reference ground truth (e.g., RealSR or DRealSR, or a newly collected set of real LQ/HR pairs from old photos and film stills) and report PSNR, SSIM, and a structural correspondence metric in addition to MUSIQ. Alternatively, rerun the OCR experiment (§4.3) on real-world text images with known transcripts rather than synthetic degradations; if the accuracy advantage over SeeSR/SUPIR disappears, the fidelity claim is an artifact of the training degradation distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—faithful, structure-preserving SR that outperforms SOTA on real-world images—depends on the assumption in §4.1 that training on synthetic LQ images 'following the same configuration as [43]' transfers to old photos, film stills, and social-media images. The real-world evaluation in Table 2 uses only no-reference metrics (MUSIQ, CLIPIQA+) on RealPhoto60 and a self-collected RealDeg set of 238 images, and the OCR recognition test in §4.3 is generated with the same synthetic degradation model used in training. No-reference metrics can reward plausible texture that is not faithful, and the OCR test does not exercise the real degradation distribution the paper claims to handle. Moreover, on the synthetic benchmarks (Table 1), the method's PSNR/SSIM are systematically lower than GAN-based baselines, so the 'faithful' claim rests entirely on perceptual metrics and a single synthetic OCR result. Without a reference-based or human-evaluated fidelity test on genuine real-world degradations, the headline conclusion is unsupported for its stated target domain.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents FaithDiff, a latent diffusion model (LDM) based super-resolution method that, unlike prior work freezing the diffusion prior, jointly fine-tunes the VAE encoder, a proposed alignment module, and the diffusion model. The alignment module injects penultimate-layer LQ encoder features into the noisy latent via transformer blocks and a linear projection. Experiments are reported on synthetic DIV2K and LSDIR benchmarks, the real-world RealPhoto60 and a self-collected RealDeg set, plus an OCR recognition test. The claimed contribution is that unleashing the diffusion prior through unified optimization yields faithful, structure-preserving SR results that outperform state-of-the-art methods.","tokens_in":13844,"tokens_out":4654,"duration_ms":36869,"significance":"If the central claim is substantiated, the paper makes a useful contribution by showing that fine-tuning, rather than freezing, the diffusion prior can improve fidelity in LDM-based SR, and the simple alignment module plus joint optimization recipe is plausible and well ablated. The ablations in Tables 5 and 6 are honest comparisons with the same training settings, and the run-time advantage in Table 4 is a concrete strength. However, the evidence for the real-world faithfulness claim is incomplete: no-reference metrics and a synthetic-degradation OCR test do not establish structure preservation on genuine real-world degradations. The contribution is therefore potentially significant but requires additional validation.","major_comments":[{"comment":"The claim in the abstract and Section 3.3 that FaithDiff 'outperforms state-of-the-art methods' is not uniformly supported by Table 1. On DIV2K-Val Level-I, SUPIR achieves lower LPIPS (0.2139 vs. 0.2187) and DreamClear achieves higher CLIPIQA+ (0.5478 vs. 0.5432); on LSDIR-Val Level-I, SUPIR achieves higher PSNR (21.58 vs. 21.20) and SSIM (0.5961 vs. 0.5760). Please restrict the claim to the perceptual metrics and severe-degradation settings where the advantage holds, or add statistical significance testing to support a broad claim.","section":"4.2, Table 1"},{"comment":"The 'faithful structures' conclusion for real-world images is not established. The real-world evaluation relies solely on no-reference metrics (MUSIQ, CLIPIQA+) over RealPhoto60 and the 238-image RealDeg set, which can reward plausible textures that are not faithful. The OCR test in Section 4.3 generates LQ images with the same synthetic degradation model used in training (Section 4.1), so it does not test generalization to real-world degradations. Please add a reference-based evaluation on a real-world SR benchmark with ground truth, or a human fidelity study, to support the faithfulness claim.","section":"4.2, Table 2 and 4.3"},{"comment":"The ablations are informative but report only point estimates of MUSIQ and LPIPS without variance or significance tests. For instance, in Table 6 the jointly optimized model's MUSIQ gain over 'FT EN & DM (SP)' is 2.80 on RealPhoto60, while the gap to 'Fix EN & FT DM' is only 1.63. Since training is stochastic, please report standard deviations over repeated runs or otherwise justify that the observed differences are not noise.","section":"5, Tables 5-6"}],"minor_comments":[{"comment":"The text 'following the same method as described in Seciton 4.1' contains a typo: 'Seciton' should be 'Section'.","section":"4.3"},{"comment":"The header 'DreamClear [41]' is incorrect; DreamClear is reference [2], while [41] is SeeSR.","section":"Table 4"},{"comment":"The entry 'StabeSR' should be spelled 'StableSR'.","section":"Table 2"},{"comment":"The phrase 'high viusal quality' should be corrected to 'high visual quality'.","section":"Figure 2 caption"},{"comment":"The word 'latant' in 'the noisy latant of the diffusion model' should be 'latent'.","section":"5.1"},{"comment":"The degradation configuration is only cited as '[43]'; please specify the exact degradation model and parameter ranges so that the training setup is reproducible.","section":"4.1"},{"comment":"Equation (3) has inconsistent spacing and the norm delimiters are not clearly closed in the formatted text; please fix the LaTeX rendering.","section":"3.3, Eq. (3)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is technically sound in its architecture and ablations, but the evaluation of the headline claim needs strengthening before publication. The authors should address the uneven Table 1 results and the lack of reference-based real-world fidelity evaluation. The paper is otherwise within the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something real and shows it with decent ablations, but the headline claim about faithful real-world restoration goes beyond what the evidence supports. The method—jointly fine-tuning the SDXL UNet and VAE encoder, plus a small transformer alignment module—is a sensible departure from the freeze-the-diffusion trend in DiffBIR, SeeSR, and SUPIR. The ablations in Tables 5 and 6 genuinely support the design choices: the alignment module helps, penultimate features help, and joint fine-tuning beats freezing either half. That is the strongest part of the paper.\n\nThe soft spots are where the claims outrun the measurements. Table 1 does not uniformly put FaithDiff ahead: SUPIR wins on LPIPS on one benchmark, DreamClear on CLIPIQA+ on another, and PSNR/SSIM are consistently below GAN baselines. The authors frame this as perceptual superiority, but \"outperforms state-of-the-art\" is too broad. More importantly, the real-world \"faithful structure\" conclusion rests on no-reference metrics (MUSIQ, CLIPIQA+) on RealPhoto60 and a 238-image self-collected set, plus an OCR test. The OCR test is generated with the same synthetic degradation pipeline used in training, so it does not test transfer to real degradations. No-reference metrics can reward plausible texture without measuring structural fidelity. There are no error bars, and no code is released. These are addressable, but as is, the central real-world claim is not established.\n\nThe novelty claim is also a bit overstated: Rombach et al. already fine-tuned an LDM for SR, so \"unleashing\" a frozen prior is a new combination, not a new idea. Still, the paper is clearly written, the comparisons are mostly fair, and the runtime advantage (2.55s vs 7-46s) is a tangible benefit.\n\nMy verdict: a solid, incremental contribution that deserves a serious referee. I would ask for stronger real-world fidelity evidence—a human study or a reference-based test on genuinely real degradations—and code release before accepting.","headline":"A plausible diffusion-SR recipe with honest component ablations, but the real-world faithfulness claim outruns the evidence.","tokens_in":14378,"tokens_out":2051,"would_cite":true,"duration_ms":18814,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Diffusion super-resolution is more faithful when the diffusion prior is fine-tuned jointly with the encoder, not frozen.","keywords":["latent diffusion model","image super-resolution","faithful restoration","joint fine-tuning","alignment module","diffusion prior","structural fidelity","real-world SR"],"falsifier":"Evaluate FaithDiff on real-world degraded images whose degradations lie far outside the synthetic training distribution—for example, spatially varying motion blur combined with heavy JPEG compression and sensor noise—and compare perceptual scores and OCR accuracy against SeeSR and SUPIR. If the MUSIQ/CLIPIQA+ lead shrinks to noise or reverses on such out-of-distribution inputs, the claim that joint fine-tuning yields faithful real-world structures would be unsupported.","tokens_in":13435,"feed_emoji":"🖼️","tokens_out":7716,"duration_ms":60024,"temperature":0.7,"pith_summary":"FaithDiff claims that diffusion-based image super-resolution is held back when the diffusion model is frozen, because a pre-trained denoiser treats errors in the low-quality image features as real structures and amplifies them. The paper's remedy is to fine-tune the diffusion model together with the encoder that maps the degraded input into latent space, connected by a lightweight alignment module that merges encoder features with the noisy latent at every denoising step. If the claim is correct, the prevailing recipe of keeping the diffusion prior fixed and adding stronger adaptors is backwards: letting the prior adapt to degraded inputs is what allows faithful recovery of fine structures while keeping results realistic. The paper backs this with better perceptual scores than state-of-the-art diffusion SR methods and with an OCR test showing that restored text stays readable.","feed_headline":"Fine-tuning, not freezing, unlocks faithful image super-resolution","feed_subtitle":"FaithDiff jointly fine-tunes the VAE encoder and diffusion model, beating state-of-the-art perceptual SR.","key_machinery":"The load-bearing mechanism is the joint fine-tuning loop. The alignment module computes $f^a_t = \\mathrm{Linear}(\\mathrm{Trans}(\\mathrm{Concat}(\\mathrm{Conv}(x^{HQ}_t), \\mathrm{Conv}(f^{LQ}))) + \\mathrm{Conv}(x^{HQ}_t))$ and injects it into each denoising step, while the whole trainable stack—LQ encoder, alignment module, and denoiser—is optimized on $\\lVert \\epsilon - \\hat{\\epsilon}_\\theta(\\sqrt{\\bar\\alpha_t}x^{HQ}_0 + \\sqrt{1-\\bar\\alpha_t}\\epsilon, f^{LQ}, c, t)\\rVert_1$. Because the noisy latent becomes clearer as denoising progresses while the LQ features stay fixed, the alignment module keeps the two streams compatible at every step; the unified optimization is what prevents the denoiser from amplifying encoder mistakes into false structures.","core_discovery":"In latent-diffusion super-resolution, the encoder that turns a low-quality image into latent features and the denoising network should be optimized as one system, not deployed as a frozen prior. FaithDiff takes the SDXL VAE encoder's penultimate-layer features (512 channels, richer than the 8-channel last layer used by DiffBIR and SUPIR), pushes them together with the noisy latent through an alignment module built from two Transformer blocks, and optimizes the encoder, alignment module, and denoiser jointly on an L1 noise-prediction loss. The paper demonstrates through ablations that freezing either module, or fine-tuning them separately, lowers perceptual quality (MUSIQ 69.7–71.1 versus 72.7 on RealPhoto60) and produces structures inconsistent with the input. Joint optimization is what lets the network separate degradation effects from genuine image content, producing state-of-the-art results on synthetic benchmarks and on real photos, film stills, and social-media images.","pith_inferences":["If joint fine-tuning is what unlocks faithfulness, the same principle likely transfers to other latent-diffusion inverse problems such as deblurring, inpainting, and face restoration, where frozen-prior adaptor designs still dominate; re-running FaithDiff's FT EN & Fix DM ablation in those tasks would test this directly.","The OCR-as-faithfulness evaluation offers a cheap, objective proxy for structural fidelity; adopting it as a standard benchmark would help the SR field distinguish 'realistic-looking' from 'content-preserving'.","Fine-tuning the VAE encoder could drift its latent space away from what the frozen VAE decoder expects; measuring FaithDiff's encoder-decoder reconstruction error on clean high-quality inputs would reveal whether that drift is paid for elsewhere."],"forward_implications":["Fine-tuning the diffusion model instead of freezing it suppresses the failure mode that makes DiffBIR, SeeSR, and SUPIR produce wrong characters and spurious structures.","Jointly optimizing the encoder and the diffusion model beats optimizing either one alone: separate fine-tuning scores 69.94 MUSIQ on RealPhoto60 versus 72.74 for joint optimization.","FaithDiff restores text more faithfully: OCR precision and recall on restored road signs reach 36.45% and 46.74%, compared with 24.32% and 25.14% for SeeSR and 22.45% and 23.50% for SUPIR.","The alignment module removes the need for a ControlNet-style adaptor, cutting diffusion runtime to 2.55 seconds per 1024×1024 image, the fastest among the compared diffusion SR methods.","Taking features from the penultimate encoder layer rather than the compressed last layer improves LPIPS on DIV2K-Val by 0.0222."],"supporting_citations":[{"why":"SDXL serves as the base latent diffusion model; its VAE encoder becomes the LQ encoder and its denoiser is the module being fine-tuned.","marker":"[26]"},{"why":"DiffBIR is the representative frozen-prior two-stage method whose degradation-removal mistakes FaithDiff's analysis identifies as the failure mode.","marker":"[21]"},{"why":"PASD supplies the degradation-removal-module paradigm and the exact synthetic degradation configuration FaithDiff uses to build training LQ images.","marker":"[43]"},{"why":"SUPIR is the scaled-up frozen-prior baseline and the source of the RealPhoto60 benchmark used to show FaithDiff's real-world gains.","marker":"[44]"},{"why":"DAAM attribution maps are used to show that frozen-prior methods attend weakly to text-relevant structures while FaithDiff's attention is sharper.","marker":"[33]"},{"why":"ControlNet is the alternative LQ-conditioning mechanism used in the ablation that freezes the diffusion model, isolating the effect of joint fine-tuning.","marker":"[47]"},{"why":"Classifier-free guidance is applied at inference with a 20% text-dropout training schedule, the mechanism that makes the text-conditioned sampling in Eq. (2) controllable.","marker":"[8]"},{"why":"The Occluded RoadText 2024 OCR benchmark provides the text-recognition test whose accuracy measures whether restored structures are faithful.","marker":"[35]"}],"fun_headline_variants":["Joint encoder-denoise tuning, not freezing, produces faithful SR","FaithDiff's alignment module and joint tuning beat frozen priors","Stop freezing diffusion priors: fine-tune everything together","Penultimate-layer features and joint tuning give faithful SR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the synthetic degradation model used to create training low-quality images—the same configuration as PASD—is representative enough of real-world degradations that FaithDiff's gains on old photos, film stills, and social-media images are genuine, even though only 238 real images are used for validation.","fun_headline_variants_meta":{"raw":{"variants":["Joint encoder-denoise tuning, not freezing, produces faithful SR","FaithDiff's alignment module and joint tuning beat frozen priors","Stop freezing diffusion priors: fine-tune everything together","Penultimate-layer features and joint tuning give faithful SR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000839,"raw_usage":{"total_tokens":3650,"prompt_tokens":932,"completion_tokens":2718,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":2649}},"tokens_in":548,"tokens_out":2718,"duration_ms":23460,"temperature":1.0,"reasoning_tokens":2649,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:50:34.506246+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate FaithDiff on real-world degraded images whose degradations lie far outside the synthetic training distribution—for example, spatially varying motion blur combined with heavy JPEG compression and sensor noise—and compare perceptual scores and OCR accuracy against SeeSR and SUPIR. If the MUSIQ/CLIPIQA+ lead shrinks to noise or reverses on such out-of-distribution inputs, the claim that joint fine-tuning yields faithful real-world structures would be unsupported.","supporting_citations":[{"cited_title":"Sdxl: Improving latent diffusion models for high-resolution image synthesis","cited_arxiv_id":null,"evidence_quote":"SDXL serves as the base latent diffusion model; its VAE encoder becomes the LQ encoder and its denoiser is the module being fine-tuned."},{"cited_title":"Scaling up to excellence: Practicing model scaling for photo- realistic image restoration in the wild","cited_arxiv_id":null,"evidence_quote":"SUPIR is the scaled-up frozen-prior baseline and the source of the RealPhoto60 benchmark used to show FaithDiff's real-world gains."},{"cited_title":"Adding conditional control to text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"ControlNet is the alternative LQ-conditioning mechanism used in the ablation that freezes the diffusion model, isolating the effect of joint fine-tuning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Occluded RoadText 2024 OCR benchmark provides the text-recognition test whose accuracy measures whether restored structures are faithful."}],"review_version":1}