{"id":"f59f3975-1d4a-47bb-b1c4-f89e6856618f","arxiv_id":"2507.18667","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A small empirical study finds that a baseline Stable Diffusion model generates police sketches that score higher on SSIM, PSNR, CLIP score, and LPIPS than versions augmented with CLIP or LoRA-fine-tuned CLIP.","lead":"This paper compares three Stable Diffusion pipelines for generating police sketches from text descriptions: a plain image-to-image model, the same model with CLIP, and a version with a LoRA-fine-tuned CLIP. The plain baseline wins on all reported similarity metrics, while the fine-tuned CLIP adds little.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline ranking of Model 1 over Models 2 and 3 rests on point estimates with no variance, sample-size, or significance testing; a 1.61% CLIP-score gap over a 59-image held-out set would be within noise.","rationale":"The reader's REJECT verdict is well supported by the paper's lack of a defensible central claim. My read does not change that verdict: the absence of error bars, significance tests, and a clearly specified evaluation split is a direct threat to the internal validity of the claimed ranking, not merely an external-representativeness concern. The reader mentioned 'no error bars' in the rationale but selected dataset representativeness as the weakest assumption; I would place the statistical grounding of the comparison first because it is more immediate and can be settled with the existing data. The proposed check is feasible if per-pair scores are released, and it would decisively show whether the reported margins are real or noise.","tokens_in":4362,"tokens_out":5649,"duration_ms":64936,"concrete_test":"Re-run the evaluation per image pair on the held-out 20% of CUFSF (the split described in Section III-D), recording SSIM, PSNR, CLIP score, and LPIPS for each test pair for all three models. Compute paired differences (Model 1 minus Model 3, Model 1 minus Model 2) and report 95% bootstrap confidence intervals and a Wilcoxon signed-rank p-value for each metric over the five iterations. Also state explicitly whether the curves in Figures 4-5 used all 295 pairs or only the held-out split. If any headline metric's confidence interval includes zero, the claim that Model 1 'consistently' outperforms the CLIP-augmented models is not supported by the reported experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is an empirical ranking: Model 1 beats Model 2 and Model 3 on SSIM, PSNR, CLIP score, and LPIPS. For that claim to be supported, the measured differences must be distinguishable from image-to-image and run-to-run noise. The paper does not provide this. Section IV reports curves, and the Figures 4 and 5 captions quote percentage advantages (e.g., 20.84% SSIM, 1.61% CLIP-score advantage), but there are no standard deviations, confidence intervals, per-pair scatter, or paired significance tests anywhere. The 295-pair CUFSF set is split 80/20 for fine-tuning, so if evaluation is on the held-out 20%, n=59; a 1.61% gap in CLIP score over 59 pairs is plausibly within noise. If evaluation is on all 295 pairs, then Model 3 was evaluated on its own training pairs, which biases the comparison and still cannot justify the absence of variance estimates. Because the paper's only contribution is this empirical comparison, the unquantified point estimates are load-bearing: the headline ordering may be an artifact of a single run or a few outlier pairs.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript describes an empirical comparison of three pipelines for generating police-style face sketches: (1) Stable Diffusion applied as an image-to-image model; (2) Stable Diffusion integrated with a pre-trained CLIP model; and (3) Stable Diffusion integrated with a CLIP model fine-tuned via LoRA on self- and cross-attention layers. Using 295 (description, sketch) pairs from CUFSF with ChatGPT-generated template descriptions, the authors report SSIM, PSNR, CLIP score, and LPIPS over five iterative refinement steps. The paper's central claims are that Model 1 (baseline) achieves the highest SSIM (0.72) and PSNR (25 dB), that Model 3 improves over Model 2 in CLIP score and LPIPS, and that fine-tuning both attention layers of CLIP yields the best text-sketch alignment.","tokens_in":4640,"tokens_out":8467,"duration_ms":81199,"significance":"Should the claims be substantiated, the work would be a modest applied contribution to automated suspect sketching, and the ablation of LoRA placement in CLIP attention layers could be of interest to the multimodal generation community. The authors use a real face-sketch dataset, multiple standard metrics, and a structured text-description protocol, which are appropriate choices. However, the central empirical ranking is not supported by statistical evidence, the evaluation of Model 3 is entangled with the fine-tuned scoring model, and the paper's own Sections V and VII contradict the claimed novelty of Model 3. The paper also provides no code, data, or reproducibility artifacts. The potential significance is therefore currently unrealized.","major_comments":[{"comment":"The headline ranking is based on point estimates without any uncertainty quantification. The Figure 4 caption reports that Model 1 outperforms Model 2 by 20.84% in SSIM and Model 3 by 19.72%, and Figure 5 reports a 1.61% CLIP-score advantage over Model 3, but the text does not state whether evaluation is on the held-out 20% (about 59 pairs) or on the full 295 pairs, and no standard deviations, confidence intervals, per-pair scatter, or paired significance tests are provided. A 1.61% CLIP-score difference over 59 pairs is well within plausible image-to-image and run-to-run noise, so the central claim that Model 1 outperforms Models 2 and 3 is not established. Please report the evaluation split size and include bootstrap or paired statistical tests for all metrics.","section":"Section IV, Figures 4 and 5"},{"comment":"The evaluation of Model 3 is partly circular. Model 3 is produced by fine-tuning CLIP with a contrastive loss (Section III-D), and the CLIP score reported in Figure 5 is computed with the same CLIP model family. The fine-tuned CLIP is also used inside the iterative-refinement loop to update embeddings and prompts (Section III-C). Consequently, the higher CLIP score of Model 3 relative to Model 2 may reflect the scorer's adaptation to Model 3's outputs rather than better text-image alignment. The authors should report CLIP score computed with a fixed pre-trained CLIP model for all three models, or use an independent text-image alignment metric as a safeguard.","section":"Section III-D and Section V"},{"comment":"It is unclear whether Model 1, the baseline, receives the same text description as Models 2 and 3. Model 1 is described in Section II as an image-to-image model that generates sketches directly from input sketches, with no mention of text conditioning, while Models 2 and 3 explicitly use CLIP text embeddings. If Model 1 is not text-conditioned, the comparison is confounded by the input modality, and the conclusion that the baseline outperforms the multimodal models is not a fair test of the CLIP component. Please specify the exact inputs to each model and, if Model 1 lacks text conditioning, include a text-conditioned Stable Diffusion baseline without CLIP.","section":"Section II and Section III-C"},{"comment":"The manuscript's stated contribution is not supported by its own results. The abstract and Section VI call Model 3 a novel approach that sets a new benchmark, but Section V states that Model 3 still trailed behind Model 1 across all metrics, and Section VII concedes that Model 1 currently outperforms the other two models. The ablation in Figure 2 only compares layer-wise fine-tuning choices and does not test the value of the full Model 3 pipeline. As written, the experimental evidence contradicts the claimed novelty; either the claims should be revised to match the reported results or additional experiments demonstrating a task-specific benefit of Model 3 are needed.","section":"Section V and Section VII"}],"minor_comments":[{"comment":"Reference [7] cites an 'OpenAI CLIP Bit Base Patch32' model at a URL for openai/clip-bit-base-patch32, which does not appear to be a valid model; the correct identifier is likely openai/clip-vit-base-patch32 as used in Section II. Please correct the reference and ensure that all model identifiers are consistent.","section":"References"},{"comment":"In Section VIII, 'Kandinsky' is misspelled as 'Kandin-sky' and the phrase 'into implement into the SD model' is ungrammatical.","section":"Section VIII"},{"comment":"Section III-A states that descriptions were generated using ChatGPT-4 with a template, but no example prompt or description is shown; including one would improve reproducibility.","section":"Section III-A"},{"comment":"The figures are referenced only by captions, with no in-text interpretation of the qualitative differences shown in Figure 6; a brief discussion would help the reader understand the visual claims.","section":"Figures"},{"comment":"Section III-C reports hyperparameters (strength 0.3, guidance scale 7.5) but does not state whether they are shared by all models; if they are shared, say so explicitly.","section":"Section III-C"}],"recommendation":"reject","confidential_remarks":"This manuscript reads as an undergraduate project report rather than a completed research paper. The experimental design has merit, but the claims are far stronger than the evidence, and the internal contradictions between the abstract and the results (Sections V and VII) cannot be resolved by minor edits. The paper is also very short and lacks the depth expected of a journal article; I would not recommend publication in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is an honest, clearly written undergraduate study comparing three Stable Diffusion pipelines for police sketch generation. The headline finding—that the plain baseline beats the CLIP-augmented variants—is plausible but not statistically established, and the evaluation has a couple of load-bearing gaps. That said, the paper is not a waste of time: it reports an ablation, shows training curves, and openly concludes that its own 'novel' model underperforms. That transparency deserves credit.\n\nWhat's actually new? Applying LoRA fine-tuning of CLIP's attention layers to sketch generation, plus an iterative refinement loop. The novelty is modest—the components are standard—but the application domain is underexplored, and the negative result is worth reporting if it holds.\n\nNow the soft spots, in proportion. The central claim is an empirical ranking (Model 1 > Model 2 > Model 3). The paper gives no error bars, confidence intervals, or significance tests. The 80/20 split is mentioned but not whether evaluation is on the held-out 20% (n=59) or all 295 pairs. If the latter, Model 3 is evaluated on its own training data, which biases the comparison. If the former, the 1.61% CLIP-score advantage is likely within noise. Either way, the headline ordering is not supported. The CLIP score is also entangled: Model 3 is fine-tuned to maximize CLIP alignment, then evaluated with a CLIP-based metric, so part of the comparison risks circularity. There is also no comparison to any prior sketch-generation method or forensic composite system, which limits the practical claim. Minor points: reference [7] names a 'CLIP Bit Base Patch32' model that does not exist (the actual model is clip-vit-base-patch32), and the 'novel' label appears before the model is shown to be useful.\n\nRead the paper if you're planning to do generative-model evaluation; it's a useful case study in how easy it is to draw conclusions from point estimates. But I wouldn't cite it, and I wouldn't send it to peer review in its current form. The evaluation needs a clean held-out protocol, error bars, and ideally a comparison to a simple text-oriented baseline. As a workshop paper or extended abstract with those fixes, it could be a worthwhile cautionary note. For now, desk reject is the right call.","headline":"Honest negative result about Stable Diffusion vs. CLIP-augmented variants, but the evaluation lacks the statistical support to make the ranking real.","tokens_in":5159,"tokens_out":3239,"would_cite":false,"duration_ms":32305,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a plain Stable Diffusion pipeline beats both CLIP-augmented and LoRA-fine-tuned CLIP pipelines for generating police sketches, with the baseline model achieving the highest SSIM (0.72), PSNR (25 dB), CLIP score, and…","keywords":["police sketch generation","Stable Diffusion","CLIP","LoRA fine-tuning","text-to-image","face sketch","multimodal alignment","iterative refinement"],"falsifier":"A field study in which witnesses describe a stranger and the descriptions are run through all three pipelines, followed by blinded identification of the suspect from the sketches, would settle whether the baseline's metric lead holds for realistic, non-template text; if the LoRA-fine-tuned CLIP model produces sketches that are identified more reliably, the paper's conclusion is false.","tokens_in":4175,"feed_emoji":"🎨","tokens_out":5179,"duration_ms":47928,"temperature":0.7,"pith_summary":"This paper asks whether adding CLIP-based text guidance to a Stable Diffusion sketch generator improves police sketch quality. The authors compare a plain image-to-image Stable Diffusion model against the same model augmented with a pre-trained CLIP model and against a version in which CLIP is fine-tuned with LoRA on its attention layers. Across structural, pixel-level, and perceptual metrics on 295 face-sketch pairs, the plain baseline wins: highest SSIM (0.72), PSNR (25 dB), CLIP score, and lowest LPIPS. The fine-tuned CLIP version beats the pre-trained CLIP version on some metrics, and an ablation shows that fine-tuning both self- and cross-attention layers gives the best text-sketch alignment, but neither CLIP-augmented model overtakes the baseline. The result matters for law enforcement applications because it suggests simple pipelines may be more reliable than complex multimodal ones for this task.","feed_headline":"Plain Stable Diffusion beats CLIP-guided models for AI police sketches","feed_subtitle":"Adding CLIP text guidance, even LoRA-tuned, lowered SSIM, PSNR, and perceptual scores across 295 sketch pairs.","key_machinery":"The central mechanism is a three-way comparison pipeline built on Stable Diffusion v1-5 as the image generator, with CLIP text-image embeddings injected as guidance. The novel variant uses LoRA (low-rank adaptation) to fine-tune CLIP's self-attention and cross-attention layers with a contrastive objective, and then feeds the refined CLIP embeddings into Stable Diffusion. Iterative refinement re-encodes each generated image into updated latents and combines them with updated text prompts over five passes. The ablation of attention layers is what carries the claim about fine-tuning: only when both self- and cross-attention are fine-tuned does the CLIP model align text and sketches well, and this configuration is then integrated into Model 3.","core_discovery":"On the paper's own terms, the central discovery is that the naive Stable Diffusion pipeline is the most effective of the three tested approaches for generating suspect sketches. Model 1, which maps an input sketch directly through Stable Diffusion without any CLIP text alignment, achieved SSIM of 0.72 and PSNR of 25 dB, outperforming Model 2 (pre-trained CLIP) by roughly 20% on both and Model 3 (LoRA-fine-tuned CLIP) by about 20% and 25% respectively. Model 1 also kept the highest CLIP score across all iterative refinement steps and the lowest LPIPS, indicating closest perceptual resemblance to ground truth. Model 3 improved on Model 2 in CLIP score and LPIPS but still trailed Model 1. The authors further claim, from an ablation study, that fine-tuning both self- and cross-attention layers of CLIP yields the best alignment between text and sketches, and they treat this as evidence that their LoRA-tuning strategy works even though the integrated model fails to beat the baseline.","pith_inferences":["Inference: Template-based ChatGPT-4 descriptions may not exercise the kind of fine-grained, ambiguous language witnesses actually use; a dataset of real transcribed witness statements could change the ranking of the models.","Inference: The SSIM/PSNR/CLIP/LPIPS metrics measure image fidelity and embedding alignment, not whether a witness or investigator can identify the suspect from the sketch; a human identification test could overturn the paper's 'best model' conclusion even if metric scores hold.","Inference: The 77-token truncation of prompts, which the paper flags as a limitation, likely harms the CLIP versions more than the baseline because the added text channel is starved; extending token capacity might reverse the ranking."],"forward_implications":["If the claim holds, law enforcement sketch tools should not assume that adding text-guidance modules improves output; the simplest diffusion baseline may be more robust.","The CLIP integration and LoRA fine-tuning pipeline as implemented adds computational cost without metric benefit, so future multimodal sketch systems need a different integration strategy.","Fine-tuning both self- and cross-attention layers is the right target for CLIP alignment in this domain, but the gain does not translate to better end-to-end sketch scores.","Iterative refinement improves perceptual similarity for the CLIP-augmented models, especially Model 3, indicating that feedback loops may help even when the static pipeline does not.","The 77-token truncation of prompts, which the paper identifies as a limitation, likely constrains the CLIP-based models more than the baseline, pointing to a concrete target for future work."],"supporting_citations":[{"why":"Provides the latent diffusion Stable Diffusion baseline model that is Model 1 and the foundation for Models 2 and 3.","marker":"[8]"},{"why":"Supplies the pre-trained CLIP model used for text-image alignment in Models 2 and 3.","marker":"[2]"},{"why":"Introduces LoRA, the low-rank adaptation method used to fine-tune CLIP attention layers in Model 3.","marker":"[3]"},{"why":"Identifies the specific Stable Diffusion v1.5 model checkpoint used in the experiments.","marker":"[9]"},{"why":"Provides background on the SSIM metric used to evaluate structural similarity of the generated sketches.","marker":"[5]"}],"fun_headline_variants":["Naive Stable Diffusion tops CLIP-tuned models in sketch tests","For AI police sketches, simple beats smart: SD outperforms CLIP","Plain SD wins over CLIP and LoRA: a baseline surprise","No CLIP needed: Stable Diffusion alone leads in sketch fidelity","In suspect sketches, baseline SD beats CLIP-guided variants"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire comparison rests on the assumption that 295 (description, sketch) pairs from CUFSF with ChatGPT-4 template descriptions represent real suspect descriptions, and that SSIM, PSNR, CLIP score, and LPIPS measure sketch quality in a way that matters for police work.","fun_headline_variants_meta":{"raw":{"variants":["Naive Stable Diffusion tops CLIP-tuned models in sketch tests","For AI police sketches, simple beats smart: SD outperforms CLIP","Plain SD wins over CLIP and LoRA: a baseline surprise","No CLIP needed: Stable Diffusion alone leads in sketch fidelity","In suspect sketches, baseline SD beats CLIP-guided variants"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1556,"prompt_tokens":946,"completion_tokens":610,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":521}},"tokens_in":562,"tokens_out":610,"duration_ms":5887,"temperature":1.0,"reasoning_tokens":521,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:37:50.860080+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A field study in which witnesses describe a stranger and the descriptions are run through all three pipelines, followed by blinded identification of the suspect from the sketches, would settle whether the baseline's metric lead holds for realistic, non-template text; if the LoRA-fine-tuned CLIP model produces sketches that are identified more reliably, the paper's conclusion is false.","supporting_citations":[{"cited_title":"Stable Diffusion v1.5 Model,","cited_arxiv_id":null,"evidence_quote":"Identifies the specific Stable Diffusion v1.5 model checkpoint used in the experiments."},{"cited_title":"Understanding SSIM,","cited_arxiv_id":null,"evidence_quote":"Provides background on the SSIM metric used to evaluate structural similarity of the generated sketches."}],"review_version":1}