{"id":"f4119435-2ba2-4632-b178-c58d4c69bb40","arxiv_id":"2412.05043","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ReF-LDM restores degraded face images by conditioning a latent diffusion model on reference photos of the same identity, using a one-pass KV cache and a diffusion-aware identity loss.","lead":"This paper presents ReF-LDM, a latent diffusion model that restores a low-quality face photo using a few high-quality photos of the same person as references. It introduces a fast CacheKV mechanism for injecting reference images and a timestep-weighted identity loss, plus a new FFHQ-Ref benchmark of over 20,000 face images with matched references.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline identity gain is measured by ArcFace IDS, and ArcFace is also used to construct FFHQ-Ref (Sec. 4.1), to supervise training (Sec. 3.2), and to define the metric (Sec. 5.1.2); without an independent identity metric the reported gains are unverified.","rationale":"The reader's weakest assumption and my concern are the same root issue: ArcFace appears in dataset construction, objective, and metric. I do not see a fatal internal inconsistency in the method; CacheKV is a reasonable approach, the ablations in Tables 2-5 support the design choices (with the caveat that spatial-concatenation actually achieves higher IDS than CacheKV at much higher cost), and the timestep-scaling result is intuitive. The problem is evidential rather than logical: the headline identity-fidelity comparison is not independently testable without either code/weights/dataset release or an evaluation using a different identity representation. The paper itself notes the competing methods have data leakage on FFHQ-Ref and DMDNet fails on many severe inputs, which further complicates the comparison. Therefore a CONDITIONAL verdict is exactly right: accept the architecture and dataset as a plausible contribution, but withhold the quantitative claim until the identity metric is validated independently. No change to the reader's verdict is needed.","tokens_in":12690,"tokens_out":3306,"duration_ms":33717,"concrete_test":"Recompute Table 7 with a held-out face-recognition model not used anywhere in training or dataset construction—e.g., AdaFace, FaceNet, or a different-architecture ArcFace backbone trained on a disjoint dataset—and additionally collect human identity-match judgments on a random 100-image subset of the severe benchmark. If ReF-LDM retains a clear lead in an independent embedding and in human ratings, the claim survives; if IDS gains shrink toward baselines, the headline result is an artifact of optimizing and evaluating in the same ArcFace space.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim is that ReF-LDM 'outperforms recent state-of-the-art methods with a significant improvement in face identity similarity' (Sec. 1). Every quantity used to support that claim is computed in the embedding space of the same ArcFace model: FFHQ-Ref identities and reference pairs are defined by an ArcFace cosine threshold r=0.4 (Sec. 4.1); the proposed L_time_ID minimizes ArcFace cosine distance between generated and target images (Eq. 1, Sec. 3.2); and the reported IDS metric is ArcFace cosine similarity (Sec. 5.1.2). This closed loop means the network is explicitly trained to maximize the very score on which it is then evaluated, and the evaluation pairs themselves were selected to be close in that same embedding. A model could therefore show large IDS gains by latching onto ArcFace-specific discriminative cues rather than by restoring the person's true identity. The problem is compounded by the footnote in Table 7: DMDNet's scores are computed only on the images it successfully processed (214/857, 29/857, and 488/2533 failures), and no confidence intervals are reported for any entry. The core quantitative comparison is thus plausible but not yet independently grounded.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ReF-LDM, a latent diffusion model for reference-based face image restoration. The model conditions a denoising U-Net on a low-quality image and multiple high-quality reference images, using a CacheKV mechanism that extracts keys and values from reference latents in a single U-Net pass and reuses them at every denoising timestep. A timestep-scaled identity loss, proportional to sqrt(alpha_t), is used to avoid the quality degradation caused by applying an ArcFace identity loss at noisy timesteps. The authors also introduce FFHQ-Ref, an identity-grouped dataset derived from FFHQ, with automatic ArcFace-based identity labeling and a manually verified test split. Experiments report IDS, fLPIPS, LPIPS, FID, and NIQE, comparing ReF-LDM with CodeFormer, VQFR, DAEFR, LDM, and DMDNet, and claim significant improvements in identity similarity, especially on severely degraded inputs.","tokens_in":12901,"tokens_out":3629,"duration_ms":39569,"significance":"If the identity-preservation claim is valid, ReF-LDM makes a useful contribution to reference-based face restoration: the CacheKV mechanism is a plausible and efficient way to inject multiple unaligned reference images into a latent diffusion model, and the FFHQ-Ref dataset with identity-based splitting and manual test verification is a valuable resource for the community. The paper includes systematic ablations of the reference-conditioning mechanism, the identity loss scaling, and the number of references, and it reports inference-time and memory numbers that support the efficiency advantage of CacheKV over spatial concatenation. The central limitation is that the headline identity metric (ArcFace IDS) is also the objective used for dataset construction and training, so the current evidence does not independently establish that the restored faces preserve true identity rather than optimize a closed-loop ArcFace score.","major_comments":[{"comment":"The identity evaluation is circular. FFHQ-Ref identities and reference pairs are defined by an ArcFace cosine threshold r=0.4 (Sec. 4.1); the training loss minimizes ArcFace cosine distance (Eq. (1), Sec. 3.2.1); and the reported IDS metric is ArcFace cosine similarity (Sec. 5.1.2). Because the model is explicitly trained to maximize the same score on which it is evaluated, and because the test pairs themselves were selected to be close in that embedding, the large IDS gains in Table 7 may reflect optimization of the metric rather than genuine identity preservation. I would like to see IDS computed with a face recognition model that was not used in training or dataset construction (e.g., AdaFace or FaceNet), plus a small human study or manual verification of whether the restored faces are judged to be the same person as the reference.","section":"Sec. 4.1, Eq. (1), Sec. 5.1.2"},{"comment":"The DMDNet comparison is computed only on the images that DMDNet successfully processed: 214/857, 29/857, and 488/2,533 failures on the three benchmarks are excluded. This is especially problematic on FFHQ-Ref-Severe, where DMDNet fails on about a quarter of the test set; the remaining images are likely easier, so the aggregate IDS comparison is biased in ReF-LDM's favor. No confidence intervals or paired statistics are reported. The comparison should be redone on a common subset where all methods succeed, and the failure rate itself should be reported as a separate criterion; ideally the per-image paired IDS differences should be reported with confidence intervals.","section":"Table 7, footnote"},{"comment":"The FFHQ-Ref training and validation identity labels are produced automatically by ArcFace with a cosine distance threshold of r=0.4, and no manual verification is reported for those splits; only the test split is manually checked. If the automatic labeling is noisy, then the training targets, the reference pairing, and the test-set construction are all affected in the same direction. The paper should quantify the label noise, for example by manually checking a random sample of training pairs and reporting precision, or by comparing against a second face recognition model.","section":"Sec. 4.1 and Sec. 4.3"}],"minor_comments":[{"comment":"The sentence 'detect facial components (i.e., eyes, nose, and mouse)' contains a typo: 'mouse' should be 'mouth'.","section":"Sec. 2"},{"comment":"The word 'utlized' should be 'utilized'.","section":"Fig. 2 caption"},{"comment":"The table title contains the typo 'numbrs'; it should be 'numbers'.","section":"Table 5"},{"comment":"The name 'CodeFomer' should be 'CodeFormer'.","section":"Sec. 5.3.2"},{"comment":"The FID column is present for the two FFHQ-Ref benchmarks but omitted for CelebA-Test-Ref; the paper should state why FID is not reported there, or add it for consistency.","section":"Table 7"},{"comment":"Spatial-concatenation achieves a higher IDS (0.69) than CacheKV (0.65) while CacheKV has better NIQE and much lower compute; the text should frame CacheKV as an efficiency-accuracy trade-off rather than implying it is uniformly superior to spatial concatenation.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The core technical contribution is plausible and the dataset is potentially useful, but the identity-similarity claim is currently supported only by a metric that is circular with the training objective and dataset construction. If the authors cannot provide an independent identity metric or a human evaluation, the claimed 'significant improvement in face identity similarity' should be substantially weakened. The DMDNet subset issue also needs to be fixed before the comparison can be trusted. This is addressable within the scope of the paper, hence major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has a genuinely useful idea (CacheKV) and a real dataset contribution, but the central quantitative claim is not independently verified. The authors train with an ArcFace identity loss, build FFHQ-Ref pairs with an ArcFace threshold, and then evaluate with ArcFace cosine similarity. Every IDS number in Table 7 is computed in the exact embedding space the model was optimized for. That doesn't make the method wrong, but it does mean \"significant improvement in identity similarity\" is, at this point, an in-distribution optimization result, not an established fact about preserving identity.\n\nWhat's actually new: the CacheKV mechanism is clever. Extracting reference keys/values once through the U-Net and reusing them across all denoising steps is a practical efficiency win, and the comparison against channel-, cross-, and spatial-concatenation is fairly done. The ablation shows it achieves similar IDS to spatial-concatenation at a fraction of the cost. The timestep-scaled identity loss is reasonable: scaling by sqrt(alpha_bar_t) to avoid feeding noisy latents to ArcFace is a sensible fix, and the experiments show it avoids the quality collapse of naive identity loss. The FFHQ-Ref dataset, if released with identity-disjoint splits, is a contribution future work can build on.\n\nThe soft spots are real but not fatal. The ArcFace circularity is the main one; it affects the headlined claim directly. The authors do manually check the test split (Sec. 4.3), but the reference selection is still driven by the same embedding. The DMDNet footnote is also a problem: evaluating only on the images it succeeded on (e.g., 29/857 failures on moderate) makes its IDS number optimistic, so the comparison against ReF-LDM is not apples-to-apples. No error bars anywhere, and no code, weights, or dataset release, which makes it impossible to check. The background LPIPS issue they concede in Appendix D is honestly reported, which is to their credit.\n\nThis is a solid paper for the face-restoration subfield. It has a mechanistically interesting contribution, a clean ablation story, and a new benchmark. I'd send it to review, but I'd expect referees to push for an independent identity metric (e.g., a different face matcher like FaceNet or a human evaluation) and for release of the assets before the identity claim is taken at face value.","headline":"Useful machinery and a usable dataset, but the headline identity claim rides entirely on the same ArcFace embedding used to build and train on the test set.","tokens_in":13541,"tokens_out":2788,"would_cite":false,"duration_ms":27733,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A latent diffusion model conditioned on multiple high-quality reference images restores low-quality faces while preserving the real person's identity, reaching identity similarity of 0.676, 0.840, and 0.779 on three benchmarks.","keywords":["reference-based face restoration","latent diffusion model","identity similarity","CacheKV","timestep-scaled identity loss","FFHQ-Ref","blind face restoration"],"falsifier":"Take a random sample of FFHQ-Ref test pairs, have human judges decide whether each restoration shows the same person as the reference, and compare human agreement with the ArcFace-based IDS ranking; alternatively, recompute IDS with a face recognition model not used in training and check whether ReF-LDM's advantage over CodeFormer persists.","tokens_in":1532,"feed_emoji":"🧑","tokens_out":4786,"duration_ms":256112,"temperature":0.7,"pith_summary":"The paper proposes ReF-LDM, a latent diffusion model that restores a low-quality face image by also taking several high-quality photos of the same person as reference. The central claim is that reference conditioning, implemented by a CacheKV mechanism that runs the denoising network over the reference images only once and reuses their attention keys and values at every denoising step, substantially improves facial identity fidelity. A timestep-scaled identity loss lets the model learn identity without sacrificing image quality. The authors also build FFHQ-Ref, a 20,405-image dataset with identity-disjoint splits, to train and evaluate reference-based face restoration models. If the claim holds, restored faces can look like the actual person rather than a plausible but different face, even under severe degradation.","feed_headline":"Reference photos restore faces as the same person","feed_subtitle":"ReF-LDM feeds several HQ reference images into a latent diffusion model and beats prior methods on identity similarity.","key_machinery":"The central mechanism is CacheKV: the encoded latent of each reference image is passed through the same denoising U-net once, and the keys and values from each self-attention layer are cached; during each of the T denoising steps, these cached reference keys and values are concatenated with the main tokens so the self-attention layers can attend to reference appearance. The second load-bearing component is the timestep-scaled identity loss, which multiplies the cosine-distance identity loss by $\\sqrt{\\bar{\\alpha}_t}$ so that identity supervision weakens at noisy timesteps and strengthens as the latent approaches a clean face. The FFHQ-Ref dataset, built by grouping FFHQ images with ArcFace cosine distance below 0.4 and splitting by connected components, supplies the training and evaluation pairs.","core_discovery":"The authors establish that LDM-based face restoration can be made reference-conditioned without the computational cost of passing reference images through the network at every timestep. By extracting keys and values from the denoising U-net once at timestep zero and caching them, then concatenating these cached tokens with the main self-attention keys and values during denoising, the model gains access to reference appearance while avoiding redundant network passes. They further show that scaling an ArcFace identity loss by $\\sqrt{\\bar{\\alpha}_t}$, the diffusion noise schedule value, improves identity similarity without the quality collapse caused by naive identity loss. On the severe FFHQ-Ref benchmark, ReF-LDM reaches an identity similarity score of 0.676, compared with 0.185 for DMDNet and 0.323 for CodeFormer, and it also leads on moderate degradation and on the CelebA-Test-Ref subset.","pith_inferences":["Beyond the paper, the cache-and-reuse KV mechanism could apply to any diffusion task with unaligned conditioning inputs, such as personalized text-to-image generation or video editing where references change pose or viewpoint.","Since the identity loss, the dataset grouping, and the evaluation metric all use the same ArcFace embedding space, the reported identity gains could partly reflect the model learning to satisfy ArcFace rather than human identity perception; a human-judged identity test or an evaluation with a different recognition model would clarify this.","The dataset construction suggests an inexpensive recipe for retrofitting large existing face datasets with identity groups: threshold a recognition embedding and split by connected components, which could transfer to other domains such as person re-identification or animal face datasets."],"forward_implications":["Restoration can use one to several reference images without landmark detection, so severely degraded inputs that break landmark-based reference methods no longer cause failure.","The caching design makes multi-reference conditioning practical: about one-fifth the inference time and 39 percent of the GPU memory of spatial concatenation with comparable identity gain.","The timestep-scaled identity loss demonstrates that diffusion identity supervision should be down-weighted at noisy timesteps, a principle transferable to other diffusion-based face generation tasks.","FFHQ-Ref gives the community an identity-disjoint, licensed benchmark with twice the images and six times the identities of the previous reference-restoration dataset.","Identity similarity can be improved substantially on the same image-quality metrics because the identity loss is scheduled to act mainly when the predicted face is already close to natural."],"supporting_citations":[{"why":"Base latent diffusion model architecture and denoising U-net that CacheKV modifies.","marker":"[22]"},{"why":"ArcFace provides the face recognition embeddings used for identity loss, dataset grouping, and the IDS metric.","marker":"[3]"},{"why":"DMDNet is the prior multi-reference landmark-based method that ReF-LDM compares against.","marker":"[16]"},{"why":"VQFR is a non-reference baseline and a source of the identity similarity metric.","marker":"[5]"},{"why":"CodeFormer is the non-reference state-of-the-art baseline on all three benchmarks.","marker":"[32]"},{"why":"Supplies the degradation model used to synthesize low-quality inputs.","marker":"[28]"},{"why":"FFHQ is the source dataset for FFHQ-Ref and the target distribution for FID.","marker":"[12]"},{"why":"Defines the noise schedule and sqrt(alpha_bar_t) used by the timestep-scaled identity loss.","marker":"[8]"}],"fun_headline_variants":["Reference photos keep faces looking like the same person","Cache reference faces to restore identity in diffusion models","Diffusion model uses reference photos for accurate face ID","Better face restoration by caching reference images in LDM","Reference-aware diffusion beats prior face restoration methods"],"cache_read_input_tokens":15488,"weakest_assumption_plain":"The identity-fidelity claim rests on automatic grouping of faces by ArcFace embeddings with a cosine-distance threshold of 0.4; if these groupings are wrong, the training references, the evaluation pairs, and the identity metric are all affected in the same direction.","fun_headline_variants_meta":{"raw":{"variants":["Reference photos keep faces looking like the same person","Cache reference faces to restore identity in diffusion models","Diffusion model uses reference photos for accurate face ID","Better face restoration by caching reference images in LDM","Reference-aware diffusion beats prior face restoration methods"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00055,"raw_usage":{"total_tokens":2605,"prompt_tokens":905,"completion_tokens":1700,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":1628}},"tokens_in":521,"tokens_out":1700,"duration_ms":29039,"temperature":1.0,"reasoning_tokens":1628,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:57:20.842290+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of FFHQ-Ref test pairs, have human judges decide whether each restoration shows the same person as the reference, and compare human agreement with the ArcFace-based IDS ranking; alternatively, recompute IDS with a face recognition model not used in training and check whether ReF-LDM's advantage over CodeFormer persists.","supporting_citations":[{"cited_title":"Rombach, A","cited_arxiv_id":null,"evidence_quote":"Base latent diffusion model architecture and denoising U-net that CacheKV modifies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DMDNet is the prior multi-reference landmark-based method that ReF-LDM compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"VQFR is a non-reference baseline and a source of the identity similarity metric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CodeFormer is the non-reference state-of-the-art baseline on all three benchmarks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the degradation model used to synthesize low-quality inputs."},{"cited_title":"Karras, S","cited_arxiv_id":null,"evidence_quote":"FFHQ is the source dataset for FFHQ-Ref and the target distribution for FID."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the noise schedule and sqrt(alpha_bar_t) used by the timestep-scaled identity loss."}],"review_version":1}