REVIEW 5 cited by
On the De-duplication of LAION-2B
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Generative models, such as DALL-E, Midjourney, and Stable Diffusion, have societal implications that extend beyond the field of computer science. These models require large image databases like LAION-2B, which contain two billion images. At this scale, manual inspection is difficult and automated analysis is challenging. In addition, recent studies show that duplicated images pose copyright problems for models trained on LAION2B, which hinders its usability. This paper proposes an algorithmic chain that runs with modest compute, that compresses CLIP features to enable efficient duplicate detection, even for vast image volumes. Our approach demonstrates that roughly 700 million images, or about 30\%, of LAION-2B's images are likely duplicated. Our method also provides the histograms of duplication on this dataset, which we use to reveal more examples of verbatim copies by Stable Diffusion and further justify the approach. The current version of the de-duplicated set will be distributed online.
Forward citations
Cited by 5 Pith papers
-
Finding DoRI: Discovery of Retained Images in Diffusion Models
Adversarially optimized text embeddings re-trigger supposedly removed memorized images in pruned diffusion models, showing memorization is distributed rather than local.
-
Dynamic Cluster Data Sampling for Efficient and Long-Tail-Aware Vision-Language Pre-training
DynamiCS dynamically scales semantic clusters per training epoch to reduce VLM pre-training compute while improving accuracy on long-tail concepts compared to static or flattening baselines.
-
FPAN: Mitigating Replication in Diffusion Models through the Fine-Grained Probabilistic Addition of Noise to Token Embeddings
Probabilistically adding high-intensity noise to individual token embeddings during fine-tuning reduces replication in Stable Diffusion by up to 28.78% in the paper's experiments, with unchanged or improved FID.
-
EVOLVE: Efficient Learned Volume Compression with Variable-Rate Encoding on a Cross-Domain Database
A single autoencoder trained on 6,376 simulation volumes compresses unseen scientific volumes at 2,000–10,000× compression with higher reported fidelity than prior compressors, with adjustable rate at inference.
-
Structured Captions Improve Prompt Adherence in Text-to-Image Models (Re-LAION-Caption 19M)
Structured four-field captions produced small but consistent gains in VQA-based text-image alignment over shuffled versions of the same captions when fine-tuning PixArt-Sigma and Stable Diffusion 2.
Discussion (0). Sign in to comment.