Gen-AI Police Sketches with Stable Diffusion

Aaron Contreras; Johnny Ni; Katherine Harvey; Nicholas Fidalgo

REVIEW 4 major objections 5 minor 10 references

Gen-AI Police Sketches with Stable Diffusion

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a plain Stable Diffusion pipeline beats both CLIP-augmented and LoRA-fine-tuned CLIP pipelines for generating police sketches, with the baseline model achieving the highest SSIM (0.72), PSNR (25 dB), CLIP score, and…

desk verdict Honest negative result about Stable Diffusion vs. CLIP-augmented variants, but the evaluation lacks the statistical support to make the ranking real. read the letter →

arxiv 2507.18667 v1 pith:7A23SED5 submitted 2025-07-24 cs.CV cs.AI

Nicholas Fidalgo , Aaron Contreras , Katherine Harvey , Johnny Ni This is my paper · ORCID

classification cs.CVcs.AI

keywords policesketchgenerationStableDiffusionCLIPLoRAfine-tuningtext-to-imagefacemultimodalalignmentiterativerefinement

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether adding CLIP-based text guidance to a Stable Diffusion sketch generator improves police sketch quality. The authors compare a plain image-to-image Stable Diffusion model against the same model augmented with a pre-trained CLIP model and against a version in which CLIP is fine-tuned with LoRA on its attention layers. Across structural, pixel-level, and perceptual metrics on 295 face-sketch pairs, the plain baseline wins: highest SSIM (0.72), PSNR (25 dB), CLIP score, and lowest LPIPS. The fine-tuned CLIP version beats the pre-trained CLIP version on some metrics, and an ablation shows that fine-tuning both self- and cross-attention layers gives the best text-sketch alignment, but neither CLIP-augmented model overtakes the baseline. The result matters for law enforcement applications because it suggests simple pipelines may be more reliable than complex multimodal ones for this task.

What carries the argument

The central mechanism is a three-way comparison pipeline built on Stable Diffusion v1-5 as the image generator, with CLIP text-image embeddings injected as guidance. The novel variant uses LoRA (low-rank adaptation) to fine-tune CLIP's self-attention and cross-attention layers with a contrastive objective, and then feeds the refined CLIP embeddings into Stable Diffusion. Iterative refinement re-encodes each generated image into updated latents and combines them with updated text prompts over five passes. The ablation of attention layers is what carries the claim about fine-tuning: only when both self- and cross-attention are fine-tuned does the CLIP model align text and sketches well, and this configuration is then integrated into Model 3.

What would settle it

A field study in which witnesses describe a stranger and the descriptions are run through all three pipelines, followed by blinded identification of the suspect from the sketches, would settle whether the baseline's metric lead holds for realistic, non-template text; if the LoRA-fine-tuned CLIP model produces sketches that are identified more reliably, the paper's conclusion is false.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the naive Stable Diffusion pipeline is the most effective of the three tested approaches for generating suspect sketches. Model 1, which maps an input sketch directly through Stable Diffusion without any CLIP text alignment, achieved SSIM of 0.72 and PSNR of 25 dB, outperforming Model 2 (pre-trained CLIP) by roughly 20% on both and Model 3 (LoRA-fine-tuned CLIP) by about 20% and 25% respectively. Model 1 also kept the highest CLIP score across all iterative refinement steps and the lowest LPIPS, indicating closest perceptual resemblance to ground truth. Model 3 improved on Model 2 in CLIP score and LPIPS but still trailed Model 1. The authors further claim, from an ablation study, that fine-tuning both self- and cross-attention layers of CLIP yields the best alignment between text and sketches, and they treat this as evidence that their LoRA-tuning strategy works even though the integrated model fails to beat the baseline.

Load-bearing premise

The entire comparison rests on the assumption that 295 (description, sketch) pairs from CUFSF with ChatGPT-4 template descriptions represent real suspect descriptions, and that SSIM, PSNR, CLIP score, and LPIPS measure sketch quality in a way that matters for police work.

Editorial extensions

If this is right

If the claim holds, law enforcement sketch tools should not assume that adding text-guidance modules improves output; the simplest diffusion baseline may be more robust.
The CLIP integration and LoRA fine-tuning pipeline as implemented adds computational cost without metric benefit, so future multimodal sketch systems need a different integration strategy.
Fine-tuning both self- and cross-attention layers is the right target for CLIP alignment in this domain, but the gain does not translate to better end-to-end sketch scores.
Iterative refinement improves perceptual similarity for the CLIP-augmented models, especially Model 3, indicating that feedback loops may help even when the static pipeline does not.
The 77-token truncation of prompts, which the paper identifies as a limitation, likely constrains the CLIP-based models more than the baseline, pointing to a concrete target for future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: Template-based ChatGPT-4 descriptions may not exercise the kind of fine-grained, ambiguous language witnesses actually use; a dataset of real transcribed witness statements could change the ranking of the models.
Inference: The SSIM/PSNR/CLIP/LPIPS metrics measure image fidelity and embedding alignment, not whether a witness or investigator can identify the suspect from the sketch; a human identification test could overturn the paper's 'best model' conclusion even if metric scores hold.
Inference: The 77-token truncation of prompts, which the paper flags as a limitation, likely harms the CLIP versions more than the baseline because the added text channel is starved; extending token capacity might reverse the ranking.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

Quick take: this is an honest, clearly written undergraduate study comparing three Stable Diffusion pipelines for police sketch generation. The headline finding—that the plain baseline beats the CLIP-augmented variants—is plausible but not statistically established, and the evaluation has a couple of load-bearing gaps. That said, the paper is not a waste of time: it reports an ablation, shows training curves, and openly concludes that its own 'novel' model underperforms. That transparency deserves credit.

What's actually new? Applying LoRA fine-tuning of CLIP's attention layers to sketch generation, plus an iterative refinement loop. The novelty is modest—the components are standard—but the application domain is underexplored, and the negative result is worth reporting if it holds.

Now the soft spots, in proportion. The central claim is an empirical ranking (Model 1 > Model 2 > Model 3). The paper gives no error bars, confidence intervals, or significance tests. The 80/20 split is mentioned but not whether evaluation is on the held-out 20% (n=59) or all 295 pairs. If the latter, Model 3 is evaluated on its own training data, which biases the comparison. If the former, the 1.61% CLIP-score advantage is likely within noise. Either way, the headline ordering is not supported. The CLIP score is also entangled: Model 3 is fine-tuned to maximize CLIP alignment, then evaluated with a CLIP-based metric, so part of the comparison risks circularity. There is also no comparison to any prior sketch-generation method or forensic composite system, which limits the practical claim. Minor points: reference [7] names a 'CLIP Bit Base Patch32' model that does not exist (the actual model is clip-vit-base-patch32), and the 'novel' label appears before the model is shown to be useful.

Read the paper if you're planning to do generative-model evaluation; it's a useful case study in how easy it is to draw conclusions from point estimates. But I wouldn't cite it, and I wouldn't send it to peer review in its current form. The evaluation needs a clean held-out protocol, error bars, and ideally a comparison to a simple text-oriented baseline. As a workshop paper or extended abstract with those fixes, it could be a worthwhile cautionary note. For now, desk reject is the right call.

Referee Report

4 major / 5 minor

Summary. This manuscript describes an empirical comparison of three pipelines for generating police-style face sketches: (1) Stable Diffusion applied as an image-to-image model; (2) Stable Diffusion integrated with a pre-trained CLIP model; and (3) Stable Diffusion integrated with a CLIP model fine-tuned via LoRA on self- and cross-attention layers. Using 295 (description, sketch) pairs from CUFSF with ChatGPT-generated template descriptions, the authors report SSIM, PSNR, CLIP score, and LPIPS over five iterative refinement steps. The paper's central claims are that Model 1 (baseline) achieves the highest SSIM (0.72) and PSNR (25 dB), that Model 3 improves over Model 2 in CLIP score and LPIPS, and that fine-tuning both attention layers of CLIP yields the best text-sketch alignment.

Significance. Should the claims be substantiated, the work would be a modest applied contribution to automated suspect sketching, and the ablation of LoRA placement in CLIP attention layers could be of interest to the multimodal generation community. The authors use a real face-sketch dataset, multiple standard metrics, and a structured text-description protocol, which are appropriate choices. However, the central empirical ranking is not supported by statistical evidence, the evaluation of Model 3 is entangled with the fine-tuned scoring model, and the paper's own Sections V and VII contradict the claimed novelty of Model 3. The paper also provides no code, data, or reproducibility artifacts. The potential significance is therefore currently unrealized.

major comments (4)

[Section IV, Figures 4 and 5] The headline ranking is based on point estimates without any uncertainty quantification. The Figure 4 caption reports that Model 1 outperforms Model 2 by 20.84% in SSIM and Model 3 by 19.72%, and Figure 5 reports a 1.61% CLIP-score advantage over Model 3, but the text does not state whether evaluation is on the held-out 20% (about 59 pairs) or on the full 295 pairs, and no standard deviations, confidence intervals, per-pair scatter, or paired significance tests are provided. A 1.61% CLIP-score difference over 59 pairs is well within plausible image-to-image and run-to-run noise, so the central claim that Model 1 outperforms Models 2 and 3 is not established. Please report the evaluation split size and include bootstrap or paired statistical tests for all metrics.
[Section III-D and Section V] The evaluation of Model 3 is partly circular. Model 3 is produced by fine-tuning CLIP with a contrastive loss (Section III-D), and the CLIP score reported in Figure 5 is computed with the same CLIP model family. The fine-tuned CLIP is also used inside the iterative-refinement loop to update embeddings and prompts (Section III-C). Consequently, the higher CLIP score of Model 3 relative to Model 2 may reflect the scorer's adaptation to Model 3's outputs rather than better text-image alignment. The authors should report CLIP score computed with a fixed pre-trained CLIP model for all three models, or use an independent text-image alignment metric as a safeguard.
[Section II and Section III-C] It is unclear whether Model 1, the baseline, receives the same text description as Models 2 and 3. Model 1 is described in Section II as an image-to-image model that generates sketches directly from input sketches, with no mention of text conditioning, while Models 2 and 3 explicitly use CLIP text embeddings. If Model 1 is not text-conditioned, the comparison is confounded by the input modality, and the conclusion that the baseline outperforms the multimodal models is not a fair test of the CLIP component. Please specify the exact inputs to each model and, if Model 1 lacks text conditioning, include a text-conditioned Stable Diffusion baseline without CLIP.
[Section V and Section VII] The manuscript's stated contribution is not supported by its own results. The abstract and Section VI call Model 3 a novel approach that sets a new benchmark, but Section V states that Model 3 still trailed behind Model 1 across all metrics, and Section VII concedes that Model 1 currently outperforms the other two models. The ablation in Figure 2 only compares layer-wise fine-tuning choices and does not test the value of the full Model 3 pipeline. As written, the experimental evidence contradicts the claimed novelty; either the claims should be revised to match the reported results or additional experiments demonstrating a task-specific benefit of Model 3 are needed.

minor comments (5)

[References] Reference [7] cites an 'OpenAI CLIP Bit Base Patch32' model at a URL for openai/clip-bit-base-patch32, which does not appear to be a valid model; the correct identifier is likely openai/clip-vit-base-patch32 as used in Section II. Please correct the reference and ensure that all model identifiers are consistent.
[Section VIII] In Section VIII, 'Kandinsky' is misspelled as 'Kandin-sky' and the phrase 'into implement into the SD model' is ungrammatical.
[Section III-A] Section III-A states that descriptions were generated using ChatGPT-4 with a template, but no example prompt or description is shown; including one would improve reproducibility.
[Figures] The figures are referenced only by captions, with no in-text interpretation of the qualitative differences shown in Figure 6; a brief discussion would help the reader understand the visual claims.
[Section III-C] Section III-C reports hyperparameters (strength 0.3, guidance scale 7.5) but does not state whether they are shared by all models; if they are shared, say so explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline ranking is empirical and CLIP-independent on its principal metrics.

full rationale

This paper is an empirical comparison of three Stable Diffusion pipelines rather than a derivation, and the central ordering (Model 1 > Model 2, Model 3) is not forced by any fitted parameter. The headline metrics SSIM (0.72), PSNR (25 dB), and LPIPS are computed against ground-truth sketches and do not depend on the CLIP embeddings that Models 2 and 3 use for generation; Model 1 does not use CLIP at all. The only possible self-reference is the CLIP Score metric in Section III-E ('Assesses similarity between generated images and text descriptions using the CLIP model'), which could overlap with the CLIP model that Model 3 fine-tunes and that the iterative refinement loop uses. However, the paper never states that the scoring CLIP is the fine-tuned model, and the reported CLIP Score ranks the no-CLIP baseline highest, so this metric does not drive the paper's conclusion. The ablation study (Section III-D, Figure 3) uses training loss and top-k retrieval accuracy to select attention layers; that is a standard model-selection procedure, not a quantity derived from the fitted weights by construction. There are no self-citations, no imported uniqueness theorems, and no renamed known result. The absence of variance/confidence intervals and the possible use of training pairs for final evaluation are statistical-support concerns, not circularity.

Assumptions & free parameters 7 free parameters · 3 assumptions · 0 invented entities

The central comparison rests on unverified assumptions about data representativeness and metric validity, plus hand-chosen hyperparameters that are not justified.

free parameters (7)

guidance_scale = 7.5
Chosen by hand in Section III-C; affects all models and is standard for Stable Diffusion but not justified for this task.
strength = 0.3
Chosen by hand in Section III-C to control image deviation; underlies all iterative refinements.
num_iterations = 5
Set to 5 for all models; arbitrary choice that affects LPIPS claim.
LoRA_rank
Not reported in Section III-D; the fine-tuning configuration is absent, so the central comparison is under-specified.
LoRA_alpha
Not reported.
learning_rate
Not reported.
training_epochs = ~15 (from Figure 3)
The figure suggests convergence around epoch 15, but exact count is not stated.

assumptions (3)

domain assumption Pre-trained Stable Diffusion v1.5, CLIP ViT-B/32, and LoRA perform as documented and transfer to sketch generation without modification.
Invoked throughout Sections II and III; if these models do not transfer, the comparison is invalid.
domain assumption The CUFSF dataset and ChatGPT-4-generated templated descriptions represent realistic suspect descriptions.
Dataset construction in Section III-A; no evidence that templated descriptions match the variety of real witness statements.
domain assumption SSIM, PSNR, LPIPS, and CLIP score are valid measures of sketch quality and text alignment for this task.
Metrics defined in Section III-E; no human evaluation is performed, so the relevance of these metrics to law enforcement is unverified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Gen-AI Police Sketches with Stable Diffusion." pith.science (2026). https://pith.science/paper/7A23SED5

@misc{pith2026250718667,
  author       = {Pith},
  title        = {Pith review of: Gen-AI Police Sketches with Stable Diffusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7A23SED5}},
  note         = {Machine review of arXiv:2507.18667}
}

read the original abstract

This project investigates the use of multimodal AI-driven approaches to automate and enhance suspect sketching. Three pipelines were developed and evaluated: (1) baseline image-to-image Stable Diffusion model, (2) same model integrated with a pre-trained CLIP model for text-image alignment, and (3) novel approach incorporating LoRA fine-tuning of the CLIP model, applied to self-attention and cross-attention layers, and integrated with Stable Diffusion. An ablation study confirmed that fine-tuning both self- and cross-attention layers yielded the best alignment between text descriptions and sketches. Performance testing revealed that Model 1 achieved the highest structural similarity (SSIM) of 0.72 and a peak signal-to-noise ratio (PSNR) of 25 dB, outperforming Model 2 and Model 3. Iterative refinement enhanced perceptual similarity (LPIPS), with Model 3 showing improvement over Model 2 but still trailing Model 1. Qualitatively, sketches generated by Model 1 demonstrated the clearest facial features, highlighting its robustness as a baseline despite its simplicity.

Figures

Figures reproduced from arXiv: 2507.18667 by the authors.

**Figure 2.** Results of fine-tuning on self-attention layers only [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Left: Validation accuracy (Top-1, Top-5, Top-10, and [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Left: SSIM over iterative refinement. Right: PSNR over [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]

**Figure 5.** Figure 5: Left: CLIP score over iterative refinement. Right: [PITH_FULL_IMAGE:figures/full_fig_p003_5.png]

**Figure 6.** Figure 6: Top: Model (1). Middle: Model (2). Bottom: Model (3). [PITH_FULL_IMAGE:figures/full_fig_p003_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 5 canonical work pages

[7]

OpenAI CLIP Bit Base Patch32 Model,

OpenAI, “OpenAI CLIP Bit Base Patch32 Model,” Hugging Face, 2023. [Online]. Available: https://huggingface.co/openai/clip-bit-base-patch32

work page 2023
[1]

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models,

A. Nichol, P. Dhariwal, A. Ramesh, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, “GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models,” arXiv, 2021. [Online]. Available: https://arxiv.org/abs/2112.10741. Accessed: Nov. 10, 2024

arXiv 2021
[2]

Learning Transferable Visual Models From Natural Lan- guage Supervision,

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, et al., “Learning Transferable Visual Models From Natural Lan- guage Supervision,” ICML, pp. 8748–8763, 2021. [Online]. Available: https://arxiv.org/abs/2103.00020. Accessed: Nov. 10, 2024

arXiv 2021
[3]

LoRA: Low-Rank Adaptation of Large Language Models,

E. J. Hu, D. Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, et al., “LoRA: Low-Rank Adaptation of Large Language Models,” NeurIPS, 2021. [Online]. Available: https://arxiv.org/abs/2106.09685. Accessed: Nov. 10, 2024

arXiv 2021
[4]

[Online]

Hugging Face, ”LoRA: Low-Rank Adaptation of Large Language Models,” Hugging Face Blog. [Online]. Available: https://huggingface.co/blog/lora. Accessed: Nov. 10, 2024

work page 2024
[5]

Understanding SSIM,

J. Nilsson and T. Akenine-M ¨oller, “Understanding SSIM,” NVIDIA

work page
[6]

[Online]

OpenAI, ”CLIP (Contrastive Language–Image Pretraining),” OpenAI. [Online]. Available: https://openai.com/index/clip/. Accessed: Nov. 10, 2024

work page 2024
[8]

High-Resolution Image Synthesis with Latent Diffusion Models,

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Om- mer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, pp. 10684–10694, 2022. [Online]. Available: https://arxiv.org/abs/2112.10752. Accessed: Nov. 10, 2024

arXiv 2022

Show all 10 references

[9]

Stable Diffusion v1.5 Model,

Stability AI, “Stable Diffusion v1.5 Model,” Hugging Face, 2022. [Online]. Available: https://huggingface.co/stable-diffusion-v1-5/stable- diffusion-v1-5

2022
[2020]

Available: arXiv, https://doi.org/10.48550/arXiv.2006.13846

2006 doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [7]

OpenAI CLIP Bit Base Patch32 Model,

OpenAI, “OpenAI CLIP Bit Base Patch32 Model,” Hugging Face, 2023. [Online]. Available: https://huggingface.co/openai/clip-bit-base-patch32

work page 2023

[2] [1]

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models,

A. Nichol, P. Dhariwal, A. Ramesh, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, “GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models,” arXiv, 2021. [Online]. Available: https://arxiv.org/abs/2112.10741. Accessed: Nov. 10, 2024

arXiv 2021

[3] [2]

Learning Transferable Visual Models From Natural Lan- guage Supervision,

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, et al., “Learning Transferable Visual Models From Natural Lan- guage Supervision,” ICML, pp. 8748–8763, 2021. [Online]. Available: https://arxiv.org/abs/2103.00020. Accessed: Nov. 10, 2024

arXiv 2021

[4] [3]

LoRA: Low-Rank Adaptation of Large Language Models,

E. J. Hu, D. Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, et al., “LoRA: Low-Rank Adaptation of Large Language Models,” NeurIPS, 2021. [Online]. Available: https://arxiv.org/abs/2106.09685. Accessed: Nov. 10, 2024

arXiv 2021

[5] [4]

[Online]

Hugging Face, ”LoRA: Low-Rank Adaptation of Large Language Models,” Hugging Face Blog. [Online]. Available: https://huggingface.co/blog/lora. Accessed: Nov. 10, 2024

work page 2024

[6] [5]

Understanding SSIM,

J. Nilsson and T. Akenine-M ¨oller, “Understanding SSIM,” NVIDIA

work page

[7] [6]

[Online]

OpenAI, ”CLIP (Contrastive Language–Image Pretraining),” OpenAI. [Online]. Available: https://openai.com/index/clip/. Accessed: Nov. 10, 2024

work page 2024

[8] [8]

High-Resolution Image Synthesis with Latent Diffusion Models,

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Om- mer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, pp. 10684–10694, 2022. [Online]. Available: https://arxiv.org/abs/2112.10752. Accessed: Nov. 10, 2024

arXiv 2022