{"id":"7c988b5b-26c0-4d73-a118-ace2a4911600","arxiv_id":"2501.12382","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"DiffDoctor uses a pixel-level artifact detector to provide per-pixel feedback that reduces artifacts in text-to-image diffusion models.","lead":"The paper trains an artifact detector to find defects in AI-generated images, then uses it to fine-tune image diffusion models so they produce fewer artifacts. The claimed payoff is a general post-training recipe that could make text-to-image models more reliable without losing quality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline artifact-frequency drop is measured with the same detector used to train the model (Eq. (1) vs. Tab. 4), and Sec. 4.2 admits confidences are driven low; the reported magnitude needs independent human or second-detector verification.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: the artifact detector is both the objective and the evaluation metric, so the headline artifact-frequency drop is vulnerable to reward hacking. Our stress-test adds two concrete supporting details from the paper, namely Sec. 4.2's admission that 'the artifact confidences are all optimized to low values' and the absence of any external validation of the detector beyond the authors' own 771-image benchmark. The independent metrics (ImageReward, CLIP-T, 24-user study) are insufficient to establish the claimed magnitude because they are either global scores insensitive to localization or a small study without significance testing. The proposed check--re-measuring artifact frequency with human raters or a second detector not used in training--directly settles whether the concern lands. Since the reader already issued a CONDITIONAL verdict that appropriately requires such external validation, our read does not change the verdict; it reinforces the need for the stated conditions rather than moving to a stronger or weaker recommendation.","tokens_in":12500,"tokens_out":2816,"duration_ms":33453,"concrete_test":"Re-evaluate the before/after models on the same 100 unseen prompts and seeds, but measure artifact frequency by independent human raters (e.g., 3 raters label each image as artifact-present or not, with majority vote and reported Cohen's kappa) or by a second detector trained on RichHF/PAL4VST labels and never used in L_pixel. Report the thresholded artifact frequency for FLUX.1, SDXL, and Kolors under this independent measure. If the post-treatment drop is substantially smaller or not statistically significant, the headline 82.66% to 22.00% result is an artifact of the training/evaluation loop rather than evidence of genuine artifact reduction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim--82.66% to 22.00% artifact frequency for FLUX.1, with similarly large drops for SDXL and Kolors--rests on the artifact detector's own confidence maps as both the training signal and the evaluation metric. In Eq. (1), L_pixel minimizes the detector confidence M*C(pi_theta(z_T)) over the synthesized image; in Tab. 4, 'Mean Artifact Freq.' is the fraction of images whose max detector confidence exceeds 0.5. The diffusion model is therefore optimized to suppress exactly the signal used to measure improvement. This is not merely a theoretical risk: Sec. 4.2 states that during treating 'the artifact confidences are all optimized to low values,' so the evaluation detector is operating on outputs it has been tuned to score low. The drop could reflect reward hacking--for example, the model producing slightly blurry or out-of-distribution images that the detector no longer recognizes--rather than the removal of human-visible artifacts. The independent metrics do not resolve the magnitude: ImageReward and CLIP-T are global quality/text-alignment scores that can stay flat or improve even when artifact localization fails, and the 24-user study has no significance testing and no inter-annotator agreement, making a 60% artifact-win rate suggestive but not decisive. Because the detector itself is only validated on a 771-image benchmark constructed by the authors (the RichHF test set is rejected), the entire chain lacks an external anchor for 'artifact' as perceived by humans.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DiffDoctor, a two-stage pipeline for reducing artifacts in text-to-image diffusion models. In the first stage, the authors train an artifact detector (SegFormer-b5) on a large dataset of flawed synthesized images, using a class-balancing strategy, human-in-the-loop labeling of hard cases, and semi-supervised pseudo-labeling. In the second stage, the trained detector is used as a differentiable pixel-level reward: the model is fine-tuned with a loss that minimizes detector confidence on artifact-prone pixels, combined with an offline diffusion-loss regularization term. Experiments on FLUX.1, SDXL, and Kolors report large reductions in 'mean artifact frequency' (e.g., FLUX.1 from 82.66% to 22.00%) and small improvements in ImageReward and CLIP-T scores, plus a 24-user study asking participants to choose between images before and after treating.","tokens_in":12863,"tokens_out":3801,"duration_ms":38725,"significance":"If the reported artifact reductions are genuine, DiffDoctor introduces a useful and novel 'diagnose-then-treat' paradigm, and it is, to my knowledge, the first method to use pixel-level artifact feedback for fine-tuning diffusion models. The paper's contribution also includes a large-scale human-in-the-loop dataset collection effort and a careful discussion of data imbalance in existing artifact annotations. The method is demonstrated on multiple backbones and on a downstream DreamBooth task, and the qualitative results show plausible artifact suppression on unseen prompts. However, the central quantitative evidence for the headline artifact-frequency drop is undermined by a circularity between the training signal and the evaluation metric, and the independent metrics (ImageReward, CLIP-T, and the small user study) do not by themselves establish the claimed magnitude. The contribution is promising, but the current evidence needs substantial strengthening before the central claim can be accepted.","major_comments":[{"comment":"The primary outcome 'Mean Artifact Freq.' is computed by thresholding the max artifact confidence of the same detector that is minimized by L_pixel in Eq. (1). Because L_pixel directly suppresses the detector's confidence values, and Sec. 4.2 states that 'the artifact confidences are all optimized to low values,' the large drops in Table 4 (e.g., FLUX.1 from 82.66% to 22.00%) are expected by construction and do not establish the removal of human-visible artifacts. The authors should re-evaluate artifact frequency using an independent artifact detector, human annotations on a held-out set, or an artifact-specific perceptual metric that is not used in training.","section":"Sec. 4.2/4.3, Eq. (1), Tab. 4"},{"comment":"The artifact detector is validated only on a self-constructed benchmark of 771 images, and the RichHF test set is rejected because its annotations are fixed-radius circles. Without an external benchmark, there is no independent anchor for the detector's accuracy. Since the detector is the core supervisory signal for treating, I recommend reporting agreement with human annotators (e.g., pixel-level IoU or AUC) on a sample of images and, if feasible, also providing results on the RichHF test set as a secondary benchmark despite its annotation limitations.","section":"Sec. 4.1, detector benchmark"},{"comment":"The user study has only 24 participants and reports artifact-winning rates of 63.64%, 68.18%, and 60.98% for DiffDoctor over the three backbones, but no significance testing, confidence intervals, or inter-annotator agreement are provided. With a forced binary choice and n=24, a 60% win rate is not statistically significant; the study needs formal testing (e.g., a binomial test or exact confidence intervals) and ideally more participants to support the claim that DiffDoctor reduces artifacts as judged by humans.","section":"Sec. 4.3, Tab. 5"}],"minor_comments":[{"comment":"The sentence 'We conduct most experiments o FLUX.1 Schnell' contains a typo: 'o' should be 'on'.","section":"Sec. 4.1"},{"comment":"The y-axis label 'Mas Artifact Confidence' should be 'Max Artifact Confidence'.","section":"Fig. 4"},{"comment":"Line 16 'Lpixel aggregate ← − − − −C(x)' is unclear; the aggregation over selected pixels should be written with a proper sum or an explicit symbol (e.g., Σ_{selected} C(x)[i,j]).","section":"Algorithm 1"},{"comment":"The row 'Threshold + Best Detector' in Table 2 reports the same ImageReward/CLIP-T values as 'FLUX.1 + DiffDoctor' in Table 4; the text should clarify whether these are the same checkpoint and whether the offline regularization is included in both.","section":"Sec. 4.2, Tab. 2 vs. Tab. 4"},{"comment":"The selection of the 2k hard cases via 'thresholding the images detected with extremely high or low artifact confidence' is described briefly; please specify the thresholds and the number of active-learning rounds used in practice.","section":"Sec. 3.2"},{"comment":"The paper does not state whether the collected dataset, the artifact detector weights, or the code will be released; providing this information would improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me cut to the chase. The core idea is real: using a dense artifact detector to provide pixel-level feedback for fine-tuning diffusion models is a legitimate new application, and the detector training recipe (real photos as negatives, hard-case selection, pseudo-label scaling) is a solid engineering contribution. The paper is well-written and the ablations show the authors understand the failure modes, e.g., naive detector causing collapse.\n\nBut the headline numbers—FLUX artifact frequency from 82.66% to 22.00%—are not trustworthy as reported. The evaluator is the same detector used as the training loss in Eq. (1), and Tab. 4 thresholds that detector's confidence. The paper itself admits in Sec. 4.2 that artifact confidences are 'all optimized to low values' during treating, making the metric 'not feasible' in the ablation. That is a textbook reward-hacking setup. The drop could mean the model learned to suppress detector responses rather than remove human-visible artifacts.\n\nThe independent evidence is much weaker: ImageReward changes are within noise (+0.004 to +0.034), CLIP-T slightly improves, and the user study has 24 participants with no significance testing. These results are suggestive and consistent with a modest real improvement, but they do not support the claimed magnitude.\n\nA second issue: the artifact detector is validated only on a self-constructed 771-image benchmark, and the RichHF test set is rejected. Without an external anchor for what counts as an artifact, the whole evaluation chain is self-referential. No code or data is released, so reproduction is not possible from the preprint.\n\nI want to be fair: the method is plausible, the direction is the right one, and the authors are transparent about the metric problem, which is more than most. The fix is not difficult—evaluate with an independent second detector, run a larger human study with significance testing, and release the detector and data. The DreamBooth experiment is a nice touch.\n\nFor a reader: if you work on diffusion alignment, this is worth reading as a new approach, but don't quote the artifact-frequency numbers. It deserves peer review, not desk rejection, but it needs major revisions to substantiate the central claim.","headline":"Genuinely new pixel-level feedback approach, but the headline artifact-frequency numbers are circular and need independent verification.","tokens_in":13370,"tokens_out":2258,"would_cite":true,"duration_ms":23261,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DiffDoctor claims that locating artifacts first, then feeding pixel-level artifact confidence back into a text-to-image diffusion model, cuts artifact frequency sharply—FLUX.1 from 82.66% to 22.00%—while preserving image-reward and…","keywords":["text-to-image diffusion models","artifact detection","pixel-level feedback","diffusion model fine-tuning","class-balanced data","human-in-the-loop annotation","artifact frequency reduction","diagnose-then-treat"],"falsifier":"Measure artifact frequency on the same unseen prompts with an independently trained artifact detector or human labels; if the reported drops (for example, FLUX.1 from 82.66% to 22.00%) shrink substantially or vanish, the central claim is not supported and the treated model may simply be fooling the training detector.","tokens_in":12319,"feed_emoji":"🩺","tokens_out":6807,"duration_ms":59140,"temperature":0.7,"pith_summary":"DiffDoctor is a two-stage recipe for cleaning up text-to-image diffusion models. First it trains an artifact detector, a pixel-level segmenter that outputs a confidence map for three artifact classes (shape distortions, unreasonable content, watermarks), on a deliberately class-balanced, human-in-the-loop dataset of over a million images. Then it uses the detector's confidence maps as a per-pixel loss to fine-tune the diffusion model via LoRA, so the model learns to avoid generating those localized defects. The paper reports that on FLUX.1, SDXL, and Kolors the frequency of artifacts on unseen prompts falls from 82.66% to 22.00%, from 55.33% to 27.50%, and from 65.31% to 29.33%, respectively, while ImageReward and CLIP-T scores are preserved or slightly improved. The central thesis is that diagnosing where defects are, before treating the model, beats image-level reward feedback.","feed_headline":"Pixel-level feedback cuts diffusion artifacts by up to 60 points","feed_subtitle":"Tuning FLUX.1, SDXL, and Kolors with an artifact detector's pixel maps removes most defects without hurting quality.","key_machinery":"The artifact map, a per-pixel confidence map produced by a SegFormer-b5 binary segmenter trained with mean-squared error on over a million images (starting from RichHF and PAL4VST annotations, augmented with real photos, hard cases, and pseudo-labels). Treating uses the pixel-level loss $L_{\\text{pixel}} = \\frac{1}{N_{\\text{aggr}}} \\sum_{i,j} M \\circ C(\\pi_\\theta(z_T))[i,j]$, back-propagated through the detector into the diffusion denoiser with gradient truncation on the last 25% of steps, plus an optional rectified-flow offline regularization $\\gamma L_{\\text{offline}}$ with $\\gamma = 0.25$ to delay collapse.","core_discovery":"The paper's central discovery is that pixel-level artifact feedback is a viable and strong training signal for diffusion models. The authors build a robust artifact detector and show that the detector's false-positive behavior is decisive: when the detector has shortcut-like high confidence on faces and limbs, treating collapses the model into blurriness, while a balanced detector lowers artifact confidence smoothly. Using the artifact map with a threshold mask (pixels above 0.1) gives the best image quality, better than using all pixels or only the maximum-confidence pixel. They further show that diagnose-then-treat beats image-level reward fine-tuning with HPSv2 on artifact frequency, and that the pipeline transfers to DreamBooth-style customization.","pith_inferences":["Editorial inference: the same diagnose-then-treat loop should transfer to other generative models with localized defects, such as video diffusion or 3D generation, as long as a pixel-level artifact detector exists for that modality.","Editorial inference: the artifact maps could also serve as editable masks for targeted regeneration or inpainting, rather than only as a training loss, potentially avoiding the layout shift that full-chain backpropagation can introduce.","Editorial inference: because the detector is trained on three artifact types (distortion, unreasonable content, and watermarks), the method's reach is bounded by the detector's vocabulary; extending to semantic or reasoning-dependent artifacts would require a detector that can reason, a limitation the paper acknowledges."],"forward_implications":["If the central claim holds, pixel-level artifact feedback can be used as a post-training step for any diffusion model that produces localized defects, without requiring paired reference images.","The reported artifact frequencies fall from 82.66% to 22.00% on FLUX.1, from 55.33% to 27.50% on SDXL, and from 65.31% to 29.33% on Kolors, while ImageReward and CLIP-T scores stay flat or improve.","Class-balancing the detector data is necessary: a detector trained on imbalanced annotations drives treating into collapse, so the diagnosis stage is what makes the treatment work.","The pipeline transfers to DreamBooth-style customization, reducing artifacts on unseen prompts while preserving subject identity."],"supporting_citations":[{"why":"Provides the initial pixel-level artifact annotations that the detector training starts from.","marker":"[15]"},{"why":"Supplies concurrent dense artifact annotations and a baseline for comparison.","marker":"[39]"},{"why":"The language model used to generate the diverse prompts for artifact-prone categories and for the treating dataset.","marker":"[2]"},{"why":"Direct reward backpropagation technique that DiffDoctor extends from image-level rewards to pixel-level artifact maps.","marker":"[7]"},{"why":"AlignProp, a direct backpropagation method used as a baseline and technical precursor for gradient-truncated reward fine-tuning.","marker":"[20]"},{"why":"HPSv2, the global human-preference reward model that DiffDoctor compares against in treating experiments.","marker":"[29]"},{"why":"ImageReward, the metric and reward model used for evaluating image quality and for ReFL-style feedback baselines.","marker":"[31]"},{"why":"SegFormer-b5, the backbone architecture of the artifact detector.","marker":"[30]"}],"fun_headline_variants":["Pixel maps cut diffusion artifacts","Balanced detector avoids blurry diffusion","Diagnose-then-treat diffusion flaws","Pixel feedback beats image-level tuning","Artifact detector sharpens diffusion output"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the artifact detector's confidence maps are an unbiased proxy for human-perceived artifacts; the same detector supplies the training signal and the primary evaluation metric, and the paper's own ablation notes that treating drives these confidences to near zero, so the reported artifact-frequency drop may reflect the model suppressing detector responses rather than genuinely removing defects.","fun_headline_variants_meta":{"raw":{"variants":["Pixel maps cut diffusion artifacts","Balanced detector avoids blurry diffusion","Diagnose-then-treat diffusion flaws","Pixel feedback beats image-level tuning","Artifact detector sharpens diffusion output"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000343,"raw_usage":{"total_tokens":1843,"prompt_tokens":863,"completion_tokens":980,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":921}},"tokens_in":479,"tokens_out":980,"duration_ms":9347,"temperature":1.0,"reasoning_tokens":921,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:12:25.716437+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure artifact frequency on the same unseen prompts with an independently trained artifact detector or human labels; if the reported drops (for example, FLUX.1 from 82.66% to 22.00%) shrink substantially or vanish, the central claim is not supported and the treated model may simply be fooling the training detector.","supporting_citations":[{"cited_title":"Rich human feedback for text-to-image generation","cited_arxiv_id":null,"evidence_quote":"Provides the initial pixel-level artifact annotations that the detector training starts from."},{"cited_title":"Perceptual artifacts local- ization for image synthesis tasks","cited_arxiv_id":null,"evidence_quote":"Supplies concurrent dense artifact annotations and a baseline for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Direct reward backpropagation technique that DiffDoctor extends from image-level rewards to pixel-level artifact maps."},{"cited_title":"Imagere- ward: Learning and evaluating human preferences for text- to-image generation","cited_arxiv_id":null,"evidence_quote":"ImageReward, the metric and reward model used for evaluating image quality and for ReFL-style feedback baselines."},{"cited_title":"Segformer: Simple and ef- ficient design for semantic segmentation with transformers","cited_arxiv_id":null,"evidence_quote":"SegFormer-b5, the backbone architecture of the artifact detector."}],"review_version":1}