Separating Knowledge and Perception with Procedural Data

Adri\'an Rodr\'iguez-Mu\~noz; Antonio Torralba; Manel Baradad; Phillip Isola

REVIEW 4 major objections 5 minor 9 references

Separating Knowledge and Perception with Procedural Data

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that training an embedding model only on procedural shader images, then storing all real images in an explicit memory database, gives near-parity with real-data models and makes every real image removable and auditable.

desk verdict Clean separation of perception and knowledge, with an honest but unquantified limitation on part-defined objects. read the letter →

arxiv 2508.11697 v1 pith:NTUR6X65 submitted 2025-08-12 cs.CV cs.AI

Adrián Rodríguez-Muñoz , Manel Baradad , Phillip Isola , Antonio Torralba This is my paper · ORCID

classification cs.CVcs.AI

keywords proceduraldatavisualmemoryself-supervisedlearningDINOk-nearestneighborsunlearningprivacyvisiontransformer

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Modern vision models bake the images they train on into their weights, which makes removing or auditing data expensive. This paper tries to break that link: it trains the embedding model exclusively on procedurally generated shader images, then stores all real-world knowledge in an explicit visual memory of reference embeddings used for k-nearest-neighbor retrieval. The claim is that this fully separates perception from knowledge, so no real image influences gradient updates and unlearning or privacy audits reduce to adding or deleting rows from the memory. The empirical backbone is that the best procedural model reaches 82.4% agreement with humans on NIGHTS, beats a Places-trained model by 15.7 points on Flowers102 and 8.0 points on CUB200, and comes within about 10 points on ImageNet-1K. The paper also identifies where the approach falls short: procedural models fail to bind visually different parts of the same real object, which explains the remaining gap.

What carries the argument

The system has two coupled parts. A vision transformer (ViT-S) is trained with the DINO local-to-global self-supervised objective on procedural images, forcing the model to make representations of local crops consistent with a global view; on real data this objective groups object parts, while on shader data it only groups abstract shapes and textures. Around it sits a visual memory: an explicit database of reference-image embeddings queried by k-nearest neighbors, so labels come from a majority vote of retrieved neighbors rather than a classifier readout. The new data process, Shaders KML Mixup, derives mixing masks from KMeans clustering of shader images and then applies Mixup interpolatio

What would settle it

Train the same pipeline from scratch on a strictly audited corpus of generated shader code with no real images, then run membership-inference or canary tests: if any real image measurably changes the embedding weights, or if a single real image appears in the training corpus, the compartmentalization claim is false. Separately, a NIGHTS-style two-alternative forced-choice block where humans and the procedural model systematically disagree on same-object-part pairs would falsify the transfer claim at the point the paper itself names as the gap.

Watch

Extended reading notes

Core claim

The central claim is full compartmentalization with respect to all real-world images: the gradient-based encoder is trained only on abstract OpenGL shader images, so knowledge of real entities lives entirely in the memory database and never in the weights. Memory retrieval, rather than a learned classifier, produces all outputs. The paper reports that this is not just a privacy convenience at the cost of usefulness: on NIGHTS the best procedural model scores 82.4% human agreement, within 0.9 points of Places; on fine-grained datasets it surpasses Places (75.2 vs 59.5 on Flowers102, 27.1 vs 19.1 on CUB200, 48.7 vs 47.8 on Food101); on ImageNet-1K it reaches 37.9% top-1 versus 47.3%; and on CO

Load-bearing premise

The load-bearing premise is that DINO-style self-supervised learning on abstract shader images produces an embedding space whose cosine similarity on real photographs tracks human and semantic similarity closely enough for k-nearest-neighbor retrieval to be useful; the paper tests this transfer empirically but cannot derive it from first principles.

Editorial extensions

If this is right

If correct, unlearning a real image (a face, a medical scan, a copyrighted photo) is one database deletion, with no retraining or weight editing; a legally requested removal is complete by construction.
Privacy guarantees become computable: differential privacy with respect to any real image can be checked by comparing predictions with and without that image in memory, and the paper finds under 0.6% of ImageNet samples are non-private in this sense.
Memory-based classification should transfer to new datasets or new knowledge at database speed: add embeddings to the store rather than fine-tune the encoder.
Storing all current real images is cheap relative to training compute: the paper estimates the full ImageNet embedding store at about 2 GiB, making the memory approach viable at scale.
Sensitive domains like medical imaging can be served by a procedural encoder plus a reference store, matching or beating published baselines on 7 of 10 MedMNIST tasks without training on the medical distribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the cross-domain transfer holds, the DINO objective can be read as teaching a domain-agnostic form of perceptual grouping, and the remaining object-part gap is a failure of a specific grouping rule, not a general capacity limit; a procedural generator that builds in part coherence might close most of the gap.
The paper's linear relationship between privacy and accuracy suggests a knob: one could deliberately include more or fewer non-private samples in memory, but the paper does not explore using that knob to trade privacy against performance.
A testable extension would train the procedural encoder with an auxiliary objective that pulls together parts of the same procedurally generated object, directly attacking the documented failure mode of dissimilar parts within a real object.
The gestalt results imply that neither real nor procedural unsupervised models group scenes the way humans do; adding gestalt priors to the self-supervised objective could push the same memory architecture toward human-like grouping without any real-data training.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper delivers what it promises—a system where no real image ever influences the embedding, all real-world knowledge sits in an explicit memory, and unlearning or privacy auditing is a database lookup. The two new procedural processes (Shaders KML and Shaders KML Mixup) are genuine additions, and the evaluation is broad and honestly conducted.

Strengths: strict protocol (one ViT-S per dataset, ImageNet hyperparameters reused for all, no fine-tuning), external benchmarks only, and the authors' own analysis section identifies the main failure mode: procedural embeddings cluster by visual resemblance rather than object membership, so parts of the same object get dissimilar codes and KNN retrieves the wrong class. They show this clearly in Figures 8 and 18. That is the right kind of self-criticism.

Soft spots: first, the gap they identify is never quantified. The ImageNet shortfall (37.9 vs 47.3 for Places) might be concentrated in part-defined classes; if so, 'strong performance' is uneven. The stress-test concern is valid but not fatal—the paper already points at it. Second, everything is single seed; no error bars on classification numbers that are sometimes within a few points of each other. Third, no code or data release for the new shader processes, so the numbers can't be checked. Fourth, the Gestalt section overreads Table 4: several R2 values are above 0.5, so 'all models do not have gestalt perception' is too sweeping. Minor: calling the setup 'full compartmentalization' is fair for gradient updates, but the DINO hyperparameters were tuned on ImageNet, so the separation is slightly less absolute than the abstract implies.

The NIGHTS nuance: the best procedural similarity model (S. KML, 0.8244) is within a point of Places (0.8331) and the z-test treats them as equivalent; S. KML Mixup, the best classifier, is lower (0.8105). So the similarity and classification leaders are not the same model.

Bottom line: the central claim holds with caveats. It is a useful paper for anyone working on memory-based vision, synthetic data, or privacy. It deserves a serious referee. I would ask for multi-seed results, code, and a per-class breakdown of the ImageNet gap to test the part-coherence explanation.

Referee Report

4 major / 5 minor

Summary. This paper proposes training a ViT-S embedding with the DINO self-supervised objective on procedural images (new Shaders KML and Shaders KML Mixup processes) and then performing classification, similarity, and segmentation via a non-parametric visual memory database of real-image embeddings, with no fine-tuning. The authors claim full compartmentalization: no real image is used to update the embedding weights. Headline results include 82.4% on NIGHTS (within ~1% of the Places-trained model), KNN accuracies on Flowers102/CUB200/Food101 that beat Places, 37.9% ImageNet-1K top-1 vs 47.3% for Places, COCO segmentation R2 within 10% of real-data models, and competitive MedMNIST results. The paper also analyzes why procedural models fail on some object classes: parts of the same object receive dissimilar representations, leading to wrong nearest neighbors.

Significance. If the claims hold, the contribution is significant: it provides a concrete recipe for separating perceptual feature learning from real-world knowledge, with inexpensive unlearning and privacy auditing. The use of external benchmarks (NIGHTS, fine-grained datasets, ImageNet, COCO, ADE20K, Pascal, MedMNIST) and the strict protocol of reusing ImageNet-tuned hyperparameters without fine-tuning strengthen the evaluation. The new S. KML Mixup process is a useful advance over prior procedural data. However, the performance claim is moderated by the paper's own structural analysis: the embedding geometry lacks object-part coherence, and the paper does not quantify how far this limits 'strong performance' on general object recognition.

major comments (4)

[§3-4, Tables 1/5] All models are trained once, with no seed variance reported, and the KNN evaluation protocol (memory split, number of neighbors, distance/normalization) is not specified. The headline margins (e.g., 'beats Places by 8%' on CUB, 'within 10%' on ImageNet) are therefore point estimates without uncertainty. Given DINO's stochasticity and the close NIGHTS margin, please report multiple seeds or bootstrapped confidence intervals and state the full KNN protocol.
[§5, Figures 8/18] The paper correctly identifies lack of part-to-object invariance as the reason for wrong nearest neighbors, but this explanation is qualitative. The abstract states the analysis explains the remaining performance gap; to support that, quantify the effect: e.g., break down ImageNet/COCO errors by classes whose identity depends on functional parts vs appearance, or measure part-consistency in the embedding and correlate it with KNN accuracy. Without this, the central 'strong performance' claim is not fully grounded.
[§3, Appendix B.1, Table 5] A z-test cannot establish equivalence; if claiming NIGHTS equivalence, use an equivalence test (TOST) or report confidence intervals. Also, Table 5 shows S. KML Mixup at 0.8105 ± 0.0095, 2.3% below Places; the 'within 1%' statement applies to S. KML only. Please make the headline model consistent across the abstract, Section 3, and Table 5.
[§6, Definition 6.1] The privacy argument checks whether removing a memory sample changes predictions on a fixed test set. This is a sensitivity audit for that test set, not an eps-DP guarantee for the mechanism over arbitrary queries. If the paper claims differential privacy for the memory-based classifier, it needs an argument covering all possible outputs/query sets and a stated epsilon; otherwise use 'sample sensitivity audit' or similar terminology.

minor comments (5)

[§8] Typographical issues: 'wether' should be 'whether'; 'ascribing' is misspelled; figure captions have 'neareast' instead of 'nearest'.
[Appendix B.1] State which pairs were tested and whether a correction for multiple comparisons was applied.
[§8, Table 4] Define the gestalt segmentation protocol and the random baseline; currently the reader cannot interpret the R2 values.
[References] The reference list contains corrupted URL placeholders in the provided text; verify all URLs before production.
[§3.1] The sentence 'S. KML obtains comparable performance to Shaders Mixup' is imprecise given Table 1; specify that it is comparable on some benchmarks but not others.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: headlining results come from external benchmarks and the compartmentalization claim is a design guarantee, not a fitted prediction.

full rationale

The paper's derivation chain is empirical rather than circular: train a DINO embedding on procedural data, freeze it, store real reference embeddings in an explicit memory, and perform KNN retrieval. All headline quantitative claims (NIGHTS, CUB200, Flowers102, Food101, ImageNet-1K, COCO, ADE20k, Pascal, MedMNIST, CelebA) are evaluated on external benchmarks that are not used to fit the embedding or the new Shaders KML processes. The new procedural process is introduced by a concrete mechanism (KMeans-derived mixing masks) and compared against prior baselines (Baradad et al. 2021/2022) rather than reverse-engineered from the target results. Self-citations to Baradad et al. and the NIGHTS benchmark are published prior work with independent content; they provide baselines and a human-judgment benchmark, and do not define the paper's reported accuracies. The 'full compartmentalization' claim is not a circular derivation but a direct property of the stated training procedure (procedural data only), and the paper candidly analyzes the residual gap in Section 5, attributing it to the absence of object-part invariance in procedural embeddings. No equation or fitted parameter is reused as a prediction. Therefore the paper does not exhibit the circularity patterns described in the taxonomy.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No new physical entities are introduced. The new procedural processes (Shaders KML, Shaders KML Mixup) are data-generation methods, not entities. The ledger's free parameters are under-reported hyperparameters rather than results fitted to evaluation benchmarks: none of the numbers in Tables 1-3 are fitted constants derived from the data being predicted.

free parameters (2)

KMeans cluster count for mixing masks = not reported (likely 3)
Used to derive the mixing mask m in Shaders KML (Figure 6). The number of clusters shapes dataset diversity and is not swept or reported, so it is an unreported hand-chosen hyperparameter.
k for KNN classification/segmentation = not reported
KNN memory retrieval uses an unstated k in Tables 1 and 3 and in KNN segmentation (Figure 10). Results may shift with k; the paper does not say how k was chosen.

assumptions (5)

domain assumption The DINO local-to-global objective yields useful part-level similarity when trained on procedural images
Section 3 transfers the DINO objective from real to procedural data without re-deriving why it should produce transferable perceptual features. All downstream results depend on this.
domain assumption Procedural images contain negligible real-world knowledge and privacy risk
Section 6 and the Conclusion rely on this for the privacy/unlearning guarantees. The paper cites prior work for the idea but does not formally bound the risk.
domain assumption NIGHTS 2AFC accuracy proxies human visual similarity
Section 3 interprets NIGHTS alignment as measuring perception quality; this is a benchmark assumption inherited from Fu et al., not proven inside the paper.
domain assumption R^2 of PCA features versus human labels proxies segmentation quality
Section 4 and Table 2 use this metric to claim 'zero-shot segmentation' ability. The metric is a correlation measure, not an end-to-end segmentation evaluation.
standard math Cosine similarity is a valid retrieval metric in the embedding space
Underpins every KNN memory operation in Sections 3-7; standard for DINO embeddings but still an assumption about the metric structure of the learned space.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Separating Knowledge and Perception with Procedural Data." pith.science (2026). https://pith.science/paper/NTUR6X65

@misc{pith2026250811697,
  author       = {Pith},
  title        = {Pith review of: Separating Knowledge and Perception with Procedural Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NTUR6X65}},
  note         = {Machine review of arXiv:2508.11697}
}

abstract

We train representation models with procedural data only, and apply them on visual similarity, classification, and semantic segmentation tasks without further training by using visual memory -- an explicit database of reference image embeddings. Unlike prior work on visual memory, our approach achieves full compartmentalization with respect to all real-world images while retaining strong performance. Compared to a model trained on Places, our procedural model performs within $1\%$ on NIGHTS visual similarity, outperforms by $8\%$ and $15\%$ on CUB200 and Flowers102 fine-grained classification, and is within $10\%$ on ImageNet-1K classification. It also demonstrates strong zero-shot segmentation, achieving an $R^2$ on COCO within $10\%$ of the models trained on real data. Finally, we analyze procedural versus real data models, showing that parts of the same object have dissimilar representations in procedural models, resulting in incorrect searches in memory and explaining the remaining performance gap.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 4 canonical work pages

[7]

Sitawarin, C

URL �� . Sitawarin, C. and Wagner, D. On the robustness of deep k-nearest neighbors. In 2019 IEEE Security and Privacy Workshops (SPW), pp. 1–7, 2019. doi: 10.1109/SPW. 2019.00014. Sun, D., Vlasic, D., Herrmann, C., Jampani, V ., Krainin, M., Chang, H., Zabih, R., Freeman, W. T., and Liu, C. 12 Separating Kno...

work page arXiv 2019
[9]

Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A

URL �� . Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Torralba, A. Places: A 10 Million Image Datab...

arXiv 2017
[131]

Bergman, L., Cohen, N., and Hoshen, Y

URL �� . Bergman, L., Cohen, N., and Hoshen, Y . Deep Near- est Neighbor Anomaly Detection, February 2020. URL �� . arXiv:2002.10445 [cs]. Bossard, L., Guillaumin, M., and Van Gool, L. Food- 101 – Mining Discriminative Components with Ran- dom Forests. In Fleet, D., Pajdla, T., Schie...

arXiv 2020
[218]

Dwork, C

URL �� . Dwork, C. A firm foundation for private data analysis. Com- mun. ACM, 54(1):86–95, January 2011. ISSN 0001-0782. doi: 10.1145/1866739.1866758. URL �� . Dwork, C. and Roth, A. The Algorithmic Foundations of Differential Privacy. Foundations and Trends® in T...

arXiv 2011
[644]

URL ��

doi: 10.1109/TGRS.2019.2959838. URL �� . Conference Name: IEEE Transactions on Geoscience and Remote Sensing. Zhang, H., Ciss ´e, M., Dauphin, Y . N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, B...

work page arXiv 2019
[1992]

Lee, R., Clarke, J., Agogino, A., and Giannakopoulou, D

ISBN 978-0-87140-218-9. Lee, R., Clarke, J., Agogino, A., and Giannakopoulou, D. Improving Trust in Deep Neural Networks with Nearest Neighbors. In AIAA Scitech 2020 Forum, Or- lando, FL, January 2020. American Institute of Aeronau- tics and Astronautics. ISBN 978-1-62410-595-1. doi: 10.2514/6.2020-2098. URL �� ....

arXiv 2020
[2018]

Rajani, N

URL �� . Rajani, N. F., Krause, B., Yin, W., Niu, T., Socher, R., and Xiong, C. Explaining and Improving Model Be- havior with k Nearest Neighbor Representations, Oc- tober 2020. URL �� . arXiv:2010.09030 [cs]. Russakovsky, O...

arXiv 2020
[2022]

Iscen, A., Caron, M., Fathi, A., and Schmid, C

URL �� . Iscen, A., Caron, M., Fathi, A., and Schmid, C. Retrieval- enhanced contrastive vision-text models. In The Twelfth International Conference on Learning Representations,

work page

Show all 9 references

[2024]

Johnson, J., Douze, M., and J´egou, H

URL �� . Johnson, J., Douze, M., and J´egou, H. Billion-scale similar- ity search with GPUs. IEEE Transactions on Big Data, 7 (3):535–547, 2019. Kataoka, H., Okayasu, K., Matsumoto, A., Yamagata, E., Yamada, R., Inoue, N., Nakamura, A.,...

2019

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [7]

Sitawarin, C

URL �� . Sitawarin, C. and Wagner, D. On the robustness of deep k-nearest neighbors. In 2019 IEEE Security and Privacy Workshops (SPW), pp. 1–7, 2019. doi: 10.1109/SPW. 2019.00014. Sun, D., Vlasic, D., Herrmann, C., Jampani, V ., Krainin, M., Chang, H., Zabih, R., Freeman, W. T., and Liu, C. 12 Separating Kno...

work page arXiv 2019

[2] [9]

Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A

URL �� . Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Torralba, A. Places: A 10 Million Image Datab...

arXiv 2017

[3] [131]

Bergman, L., Cohen, N., and Hoshen, Y

URL �� . Bergman, L., Cohen, N., and Hoshen, Y . Deep Near- est Neighbor Anomaly Detection, February 2020. URL �� . arXiv:2002.10445 [cs]. Bossard, L., Guillaumin, M., and Van Gool, L. Food- 101 – Mining Discriminative Components with Ran- dom Forests. In Fleet, D., Pajdla, T., Schie...

arXiv 2020

[4] [218]

Dwork, C

URL �� . Dwork, C. A firm foundation for private data analysis. Com- mun. ACM, 54(1):86–95, January 2011. ISSN 0001-0782. doi: 10.1145/1866739.1866758. URL �� . Dwork, C. and Roth, A. The Algorithmic Foundations of Differential Privacy. Foundations and Trends® in T...

arXiv 2011

[5] [644]

URL ��

doi: 10.1109/TGRS.2019.2959838. URL �� . Conference Name: IEEE Transactions on Geoscience and Remote Sensing. Zhang, H., Ciss ´e, M., Dauphin, Y . N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, B...

work page arXiv 2019

[6] [1992]

Lee, R., Clarke, J., Agogino, A., and Giannakopoulou, D

ISBN 978-0-87140-218-9. Lee, R., Clarke, J., Agogino, A., and Giannakopoulou, D. Improving Trust in Deep Neural Networks with Nearest Neighbors. In AIAA Scitech 2020 Forum, Or- lando, FL, January 2020. American Institute of Aeronau- tics and Astronautics. ISBN 978-1-62410-595-1. doi: 10.2514/6.2020-2098. URL �� ....

arXiv 2020

[7] [2018]

Rajani, N

URL �� . Rajani, N. F., Krause, B., Yin, W., Niu, T., Socher, R., and Xiong, C. Explaining and Improving Model Be- havior with k Nearest Neighbor Representations, Oc- tober 2020. URL �� . arXiv:2010.09030 [cs]. Russakovsky, O...

arXiv 2020

[8] [2022]

Iscen, A., Caron, M., Fathi, A., and Schmid, C

URL �� . Iscen, A., Caron, M., Fathi, A., and Schmid, C. Retrieval- enhanced contrastive vision-text models. In The Twelfth International Conference on Learning Representations,

work page