{"id":"6d5b4893-12f5-471e-8db9-bffbceb8a617","arxiv_id":"2412.17109","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The similarity between consecutive denoised images during diffusion sampling correlates with artifact presence, enabling a classifier trained on 680 images to detect flawed outputs with 72.35% accuracy.","lead":"This paper shows that a pattern in how a diffusion model's denoised images change from one sampling step to the next, called the Similarity Trajectory, is correlated with whether the final image contains visible artifacts. The authors train a classifier on just 680 labeled images and reach 72.35% accuracy, suggesting a data-efficient way to detect flawed generated images.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Core accuracy may be inflated by same-prompt leakage: the 10-fold CV splits images, not prompts, so the 72.35% figure could reflect prompt-level trajectory similarity rather than artifact presence; prompt-grouped CV is needed.","rationale":"The paper's central claim is that a drop or low similarity in the trajectory signals artifacts, supported by max-decline statistics and the classifier accuracy. The classifier number is the most load-bearing piece of evidence: if 72.35% does not survive proper splitting, the claim reduces to a weak statistical difference on a hand-filtered set. The reader's concern about the filtered, extreme-only dataset is real and limits external validity, but my concern is internal to the reported experiment: the train/test split structure is not prompt-aware. Since each prompt contributes up to nine images and trajectories carry prompt-conditioned semantic content, random image-level folds very likely share prompts between train and test. That allows the random forest to memorize prompt-level trajectory patterns and inflate the correlation with artifact labels. This is a concrete, testable leakage mode that would directly overstate the central claim. I do not see it as a rejection of the paper: the human evaluation uses within-prompt pairs and shows 58.1% agreement vs 21.7% disagreement, which is independent evidence that the trajectory carries artifact-related signal. Also, the authors' own limitations section appropriately notes that accuracy is imperfect and artifact sources are diverse. My recommendation is therefore to keep the conditional verdict, with prompt-grouped cross-validation as a mandatory revision check rather than a reason to reject outright.","tokens_in":12771,"tokens_out":4939,"duration_ms":50556,"concrete_test":"Repeat the 10-fold CV from Sec. 7.2 using prompt-grouped folds: for each prompt, keep all generated/selected images from that prompt in the same fold, and train and test only on disjoint prompt sets. Report the resulting accuracy and compare it with the 72.35% image-level result. If the grouped accuracy drops substantially (e.g., toward 50% or below the 58.1% human-agreement rate), the headline accuracy is at least partly due to same-prompt leakage and the central claim needs re-benchmarking; if grouped accuracy remains near 72%, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest evidence for the central claim is the 72.35% 10-fold CV accuracy (Sec. 7.2) and the max-decline gap (Sec. 7.1). Both are computed on images drawn from 250 prompts with 9 generated images per prompt (Sec. 6.1). The balanced training set (255 natural + 255 artifact) is split by image at random; the paper never states that all images from the same prompt are kept in the same fold. If they are not, the same prompt can appear in both training and test folds. Because the Similarity Trajectory is built from denoised predictions x0^(t) that carry strong prompt/semantic content, and because DreamSim is a semantic similarity metric, trajectories from the same prompt are likely to be more similar to each other than to trajectories from different prompts, independent of artifact status. A random forest can then exploit prompt identity as a shortcut, inflating both the reported accuracy and the apparent trajectory-artifact correlation. The 'first 255 natural images' selection (Sec. 7.2) may also make prompt distributions differ between classes, further allowing prompt rather than artifact to drive classification. The human evaluation pairs images from the same prompt, so it avoids this particular leakage and gives independent support, but it selects extreme-probability images and has no raw-image baseline; it does not fully rescue the quantitative claim. Therefore, the classifier evidence must be shown to survive prompt-conditioned splitting before the central claim is accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Similarity Trajectory, defined as the sequence of DreamSim similarities between consecutive denoised predictions x0^(t) during diffusion sampling. The central claim is that a drop or low similarity in this trajectory signals the presence of artifacts in the final generated image. The authors build a random forest classifier on statistical features of the trajectory (including Haar transform coefficients and k-NN probabilities) trained on 680 hand-filtered images from Stable Diffusion 2, report 72.35% accuracy in 10-fold cross-validation, and support the claim with a max-decline analysis, a human evaluation, and a preliminary model-assessment study on EDM2 checkpoints. The paper positions this as enabling artifact detection with orders-of-magnitude less labeled data than prior work.","tokens_in":13075,"tokens_out":5735,"duration_ms":51672,"significance":"The idea of using the sampling process rather than the final image for artifact detection is novel and potentially valuable, especially if it truly enables low-data training. The Similarity Trajectory is a simple, generalizable concept that could be applied to any denoising-based generative model. However, the significance is currently limited by evaluation weaknesses: the cross-validation likely suffers from prompt-level leakage, the dataset is filtered to exclude borderline cases, there is no raw-image baseline, and the human evaluation lacks a chance baseline. If the authors can show that the result survives prompt-grouped cross-validation and that trajectory features outperform raw-image features under the same data budget, the contribution would be solid. The model-evaluation part (Section 7.4) is explicitly preliminary and would benefit from more extensive validation.","major_comments":[{"comment":"The 10-fold cross-validation splits images, not prompts. Given that the dataset contains 250 prompts with 9 images per prompt, and that DreamSim is a semantic similarity metric, trajectories from the same prompt are likely to be more similar to each other than to trajectories from different prompts regardless of artifact status. This makes the reported 72.35% accuracy potentially inflated by prompt identity leakage. Please report the accuracy under prompt-conditioned CV (e.g., grouped k-fold by prompt or leave-one-prompt-out) and describe how the 255 natural images were selected; if they are the first 255 in the dataset order, the prompt distribution may differ between classes, further enabling prompt-driven shortcuts.","section":"Section 7.2, Section 6.1"},{"comment":"The dataset is manually filtered to 425 natural-looking and 255 artifact-exhibiting images, discarding all images that fall between these categories. This means the classifier and the max-decline analysis are evaluated on a distribution free of borderline cases. Since real deployment includes images across the full severity spectrum, the reported 72.35% accuracy and the 0.017 vs 0.027 max-decline gap may not transfer. Please report performance on an unfiltered random sample or provide a principled discussion of how the filtering affects the claimed correlation and the classifier's expected real-world performance.","section":"Section 6.1"},{"comment":"The paper claims that the Similarity Trajectory enables classification with only 680 images, which is 0.1% of the data used in prior work. However, there is no ablation comparing trajectory features to raw-image features (or final-image features) under the same data budget and evaluation protocol. A classifier trained on final-image features with the same 680 images could potentially achieve similar accuracy, which would undermine the specific advantage of the trajectory. Please add such a baseline (e.g., a small CNN or a classifier on pixel statistics) to demonstrate that the trajectory, not merely the small dataset, is what enables the result.","section":"Section 7.2, Section 5.2"},{"comment":"The human evaluation reports 58.1% agreement, 21.7% disagreement, and 20.2% draws, but no chance baseline or statistical test is provided. With three possible outcomes and draws handled as a separate category, the agreement rate must be compared to a chance model (e.g., random selection among non-draw responses) and accompanied by confidence intervals or a significance test. In addition, the comparison to prior work's 71.4% agreement (Section 7.3) is not apples-to-apples because the pairing protocol differs: prior work pairs best-of-100 vs random, while this paper pairs highest vs lowest probability according to the classifier.","section":"Section 7.3"},{"comment":"The maximum-decline analysis is restricted to time steps 13-34, a window selected based on Gini impurity computed on the same dataset that is later used for evaluation (Figure 4). This data-driven selection can inflate the apparent difference between artifact and natural images. Please validate the selected window on a held-out subset or report sensitivity of the max-decline difference to the choice of time-step range.","section":"Section 7.1"}],"minor_comments":[{"comment":"The notation in Eq. (5) is confusing: {z_t}^1_{t=T-1} should be {z_t}_{t=1}^{T-1}. Similarly, the definition of D_max has reversed index bounds; it should read '1 ≤ s < e ≤ T-1' for a strictly decreasing subsequence, and the condition 'zs > zs+1 > ... > ze' should be made explicit.","section":"Section 4, Eq. (5) and Section 5.1"},{"comment":"The definitions of mean crossings and zero crossings assume an ordering on the elements of set S, but S is defined as a set. Please clarify that these features are computed on the ordered sequence rather than an unordered set.","section":"Section 5.2, Eqs. (8)-(9)"},{"comment":"The model evaluation uses RMSE between denoised latents as a dissimilarity metric, whereas the classifier experiments use DreamSim on decoded images. Please clarify the relationship between these two similarity measures and whether the RMSE-based evaluation supports the same conclusions as the DreamSim-based trajectory analysis.","section":"Section 6.2 and Section 7.4"},{"comment":"The text states that stronger models exhibit 'higher similarity' between adjacent latents, but the y-axis of Figure 5 is RMSE (where larger values indicate more dissimilarity). Please make the axis labeling and the accompanying explanation consistent.","section":"Section 7.4, Figure 5"},{"comment":"There are several typos and formatting issues, including 'theSimilarity Trajectory' appearing without a space in multiple places (Introduction, Section 5.2), and 'Similarity' used where 'Similarly' is intended. Please proofread the manuscript.","section":"Throughout"},{"comment":"Please clarify how the 10 human participants were split across the 100 test pairs: does each of the 50 pairs in a group get evaluated by all 5 participants in that group, or do participants evaluate disjoint subsets? This affects how the agreement percentage is computed and interpreted.","section":"Section 7.3"},{"comment":"Reference [27] is cited as prior work showing that the middle section of the sampling process has the greatest influence on the final image. Please ensure this citation is appropriate, as ShiftDDPMs focuses on shifting diffusion trajectories rather than on time-step importance for artifact formation.","section":"Section 7.1"}],"recommendation":"major_revision","confidential_remarks":"The strongest concern is the prompt-level leakage in cross-validation, which, if confirmed, would invalidate the headline accuracy and the central claim. The absence of a raw-image baseline is another key omission. The paper's idea is interesting and the experiments are relatively easy to fix, so I would not reject it outright, but the current version is not acceptable without major revisions. The paper might be a better fit for a workshop or a shorter venue, but with additional experiments it could reach the bar for this journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know about this paper because it introduces a genuinely new representation for artifact detection in diffusion images: the Similarity Trajectory, i.e., the sequence of perceptual similarities between consecutive denoised predictions during sampling. That idea is not in the cited prior work, and it is the right kind of idea: it uses the sampling dynamics rather than the final image, and it leads to a classifier that needs only 680 labeled images, two orders of magnitude fewer than preference-based methods. The paper does that part well.\n\nThe central claim—that a drop in consecutive-step similarity correlates with artifacts—is plausible and supported by several independent but imperfect pieces of evidence. The max-decline gap (0.017 vs 0.027) is in the right direction, the EDM2 training/model-size experiments show the expected ordering, and the human evaluation agrees with the classifier more than it disagrees (58.1% vs 21.7%, 20.2% draws). Those are real contributions.\n\nBut the quantitative headline, the 72.35% 10-fold accuracy, is softer than it looks. The data set is hand-filtered to extremes—425 natural-looking and 255 obvious-artifact images, with all borderline cases discarded—so the reported accuracy is on a distribution intentionally free of ambiguity. The middle time-window (steps 13-34) is chosen post hoc on the same data, using Gini importance from the very classifier being evaluated. And the most serious issue, which the stress-test note correctly identifies: the 10-fold CV splits images, not prompts, and the 510-image balanced set is drawn from only 250 prompts (9 images each). Since trajectories are built from semantic similarity (DreamSim) between denoised images, trajectories from the same prompt are likely more similar to each other than to those from other prompts, independent of artifact status. A random forest can exploit that. Prompt-grouped CV is needed, and the paper doesn't report it. The 'first 255 natural images' selection (Sec 7.2) may also make prompt distributions differ between classes.\n\nThe human evaluation pairs same-prompt images, so it avoids this leakage, but it selects extreme-probability images and lacks a chance baseline; it supports the trend but doesn't pin down the accuracy.\n\nOverall, this is a promising empirical observation, not yet a proven detection method. The right next step is to redo the CV with prompt-based folds and compare trajectory features against raw-image features under the same 680-image budget. If that holds, the paper becomes a solid subfield contribution.\n\nVerdict for you: worth sending to a serious referee, but the referee should ask for the prompt-grouped CV before accepting the core number. I'd take the central correlation as plausible, not established. Bring it to the reading group if you want to discuss data-splitting pitfalls in practical CV.","headline":"A genuinely new representation for data-efficient artifact detection, but the headline accuracy needs prompt-conditioned CV before it can be trusted.","tokens_in":13604,"tokens_out":2758,"would_cite":false,"duration_ms":22619,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The pattern of similarity between consecutive denoised images during the diffusion sampling process predicts whether the final image contains artifacts.","keywords":["Similarity Trajectory","diffusion models","artifact detection","sampling process","DreamSim","Haar transform","random forest","data-efficient classification"],"falsifier":"Generate a large uncurated batch of images from the same SD2/DDIM setting and obtain human ratings of artifact severity for every image, including the previously discarded borderline ones; then test whether trajectory features and the max-decline statistic sort these images by severity. If accuracy on the unfiltered set is near chance, the link between trajectory drops and artifacts is an artifact of extreme-case selection.","tokens_in":12556,"feed_emoji":"🖼️","tokens_out":5113,"duration_ms":40186,"temperature":0.7,"pith_summary":"The paper tries to show that how a diffusion model samples matters for image quality: the sequence of similarities between denoised images at neighboring time steps, which the authors call the Similarity Trajectory, contains a signal about artifacts in the final image. If true, this would let artifact detection run on a few hundred labeled examples instead of the million-plus images used by prior approaches. The authors measure a larger maximum similarity drop in images with obvious artifacts than in natural-looking ones, and a random forest trained on trajectory features reaches 72.35% accuracy in 10-fold cross-validation. They also present evidence that averaged trajectories track generator quality, improving with training progress and model size.","feed_headline":"Similarity drops in the sampling path expose diffusion artifacts","feed_subtitle":"A random forest on these trajectory features hits 72.35% accuracy using only 680 labeled images, 0.1% of prior data.","key_machinery":"The central object is the Similarity Trajectory, the time series $z_t = d(x_0^{(t)}, x_0^{(t-1)})$ of similarities between denoised images predicted at consecutive sampling steps, using DreamSim as the human-aligned metric $d$. The paper also defines the maximum decline $D_{\\max}$ of a trajectory and uses the Haar transform plus bag-of-statistics features (percentiles, mean crossings, entropy) to turn each trajectory into a vector for a random forest classifier. These components carry the argument: the trajectory is the evidence that links sampling dynamics to artifacts, and the classifier demonstrates the claim is actionable with little labeled data.","core_discovery":"Diffusion sampling produces a sequence of denoised images; the paper's central discovery is that the similarity between consecutive denoised images, monitored across the whole sampling run, is correlated with the severity of artifacts in the final generated image. Concretely, the authors define the Similarity Trajectory $\\{z_t\\}$ with $z_t = d(x_0^{(t)}, x_0^{(t-1)})$, and a maximal decline in that trajectory is statistically larger for images with obvious artifacts than for natural-looking ones. Because the trajectory compresses the sampling process into a compact signature, a random forest classifier trained on Haar-transform coefficients and statistical features of the trajectory can predict artifact presence from only 680 labeled images, and the predictions agree with human judgments more often than they disagree on a paired real-world test. The same signature, averaged over many generated images, shifts in a direction consistent with model improvement as training time and model capacity increase.","pith_inferences":["We infer that the same trajectory signal could support real-time feedback during sampling, such as rejecting or restarting a run when a large similarity drop is detected, something the paper does not test.","The claim is established on a deliberately filtered distribution (only extreme artifact and clearly natural images); we infer that extending the approach to continuous artifact severity will require either richer labels or trajectory features tailored to borderline cases.","We infer that the connection between trajectory smoothness and artifact presence may transfer to other iterative generative paradigms, such as flow matching or consistency models, that also produce a denoising path, though the paper only studies DDIM and EDM2/Heun samplers."],"forward_implications":["Artifact detection for diffusion models can be data-efficient: trajectory features reduce the annotated-data requirement to a few hundred images, roughly 0.1% of what prior image-based methods need.","A drop or sustained decline in the Similarity Trajectory is a usable red flag for artifacts, since the average maximum decline is 0.027 for artifact-exhibiting images versus 0.017 for natural-looking ones.","The averaged Similarity Trajectory can serve as a complementary model-evaluation signal to FID: adjacent-latent similarity increases with both training progress and model capacity in the EDM2 framework.","Human evaluation on paired images shows the trajectory-based classifier's artifact picks align with human judgment about 2.7 times more often than they contradict it."],"supporting_citations":[{"why":"Defines DreamSim, the human-aligned similarity metric used to construct the trajectory $z_t$.","marker":"[5]"},{"why":"Supplies the DDIM sampler and the formula for the denoised images $x_0^{(t)}$ used to compute trajectories.","marker":"[22]"},{"why":"Provides the Pick-a-Pic dataset used for the real-world paired human evaluation and the data-scale baseline the paper compares against.","marker":"[13]"},{"why":"The prior artifact-detection work with rich annotations that motivates the task and illustrates the large labeled-data requirement.","marker":"[15]"},{"why":"The random forest classifier that turns trajectory features into artifact predictions.","marker":"[1]"},{"why":"The EDM2 training framework and checkpoints used to show averaged trajectories track training progress and model size.","marker":"[10]"},{"why":"The Haar transform used to extract fluctuation features from the trajectory.","marker":"[23]"}],"fun_headline_variants":["Similarity dip in sampling path flags diffusion artifacts","Tracking denoised-image similarity catches diffusion glitches","Fewer labels spot artifacts via similarity trajectories","Sampling similarity curve predicts diffusion image flaws","680 images enough to spot artifacts via similarity dips"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The labeled dataset used for the max-decline statistics and classifier training was manually filtered to keep only images with obvious artifacts or clear natural quality, discarding every ambiguous case, so the measured separation and 72.35% accuracy may not hold on unfiltered, real-world images where most artifacts are borderline.","fun_headline_variants_meta":{"raw":{"variants":["Similarity dip in sampling path flags diffusion artifacts","Tracking denoised-image similarity catches diffusion glitches","Fewer labels spot artifacts via similarity trajectories","Sampling similarity curve predicts diffusion image flaws","680 images enough to spot artifacts via similarity dips"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000366,"raw_usage":{"total_tokens":1960,"prompt_tokens":931,"completion_tokens":1029,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":959}},"tokens_in":547,"tokens_out":1029,"duration_ms":7287,"temperature":1.0,"reasoning_tokens":959,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:47:01.006545+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate a large uncurated batch of images from the same SD2/DDIM setting and obtain human ratings of artifact severity for every image, including the previously discarded borderline ones; then test whether trajectory features and the max-decline statistic sort these images by severity. If accuracy on the unfiltered set is near chance, the link between trajectory drops and artifacts is an artifact of extreme-case selection.","supporting_citations":[{"cited_title":"Dreamsim: Learning new dimensions of human visual simi- larity using synthetic data","cited_arxiv_id":null,"evidence_quote":"Defines DreamSim, the human-aligned similarity metric used to construct the trajectory $z_t$."},{"cited_title":"Pick-a-pic: An open dataset of user preferences for text-to-image generation","cited_arxiv_id":null,"evidence_quote":"Provides the Pick-a-Pic dataset used for the real-world paired human evaluation and the data-scale baseline the paper compares against."},{"cited_title":"Collins, Yiwen Luo, Yang Li, Kai J Kohlhoff, Deepak Ramachandran, and Vidhya Naval- pakkam","cited_arxiv_id":null,"evidence_quote":"The prior artifact-detection work with rich annotations that motivates the task and illustrates the large labeled-data requirement."},{"cited_title":"Random forests","cited_arxiv_id":null,"evidence_quote":"The random forest classifier that turns trajectory features into artifact predictions."},{"cited_title":"Analyzing and improving the training dynamics of diffusion models","cited_arxiv_id":null,"evidence_quote":"The EDM2 training framework and checkpoints used to show averaged trajectories track training progress and model size."},{"cited_title":"Stankovi ´c and Bogdan J","cited_arxiv_id":null,"evidence_quote":"The Haar transform used to extract fluctuation features from the trajectory."}],"review_version":1}