Pith. sign in

REVIEW 3 major objections 5 minor 61 references

Beginning with You: Perceptual-Initialization Improves Vision-Language Representation and Alignment

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Seeding a CLIP vision encoder with human similarity judgments before web-scale contrastive pretraining yields better zero-shot classification on 23 of 29 benchmarks and better retrieval on Flickr30k and MS-COCO.

desk verdict Genuinely new idea—initializing a CLIP vision encoder with NIGHTS human triplet embeddings before contrastive pretraining—with consistent but unproven gains due to a missing warm-start control and single-seed runs. read the letter →

arxiv 2505.14204 v1 pith:DE3PD5LE submitted 2025-05-20 cs.CV q-bio.NC

Yang Hu , Runchen Wang , Stephen Chong Zhao , Xuhui Zhan , Do Hun Kim , Mark Wallace , David A. Tovar This is my paper
classification cs.CVq-bio.NC
keywords perceptualinitializationhumansimilarityCLIPzero-shottransfervision-languagepretrainingNIGHTSdatasettripletcontrastivelossYFCC15Mrepresentationlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that where human perceptual knowledge enters a vision-language model matters as much as whether it enters. Instead of aligning a trained CLIP model to human similarity judgments as a fine-tuning step, the authors first train the vision encoder on 20,000 human-annotated NIGHTS image triplets—roughly 3 GPU-hours—and only then run standard contrastive pretraining on 15 million image–text pairs from YFCC15M. Against an otherwise identical randomly initialized baseline, this perceptually initialized model improves zero-shot top-1 accuracy on 23 of 29 classification datasets and improves image–text retrieval recall on both Flickr30k and MS-COCO, with no task-specific fine-tuning. The authors also show that the same perceptual supervision applied as post-hoc fine-tuning collapses alignment to near-random levels. The claim, if true, recasts human behavioral data as a principled initialization prior rather than a corrective afterthought.

What carries the argument

The load-bearing object is the Perceptual-Initialization schedule: a two-stage pipeline in which a ViT-B/32 vision encoder is first trained on NIGHTS human triplet judgments with a margin-based triplet contrastive loss, then unfrozen and trained jointly with a text encoder using the standard symmetric InfoNCE objective on YFCC15M image–text pairs. The NIGHTS embeddings supply a human perceptual geometry for the weight initialization; because deep networks are path-dependent, this geometry is amplified by early gradient updates rather than washed out, and it anchors the representation space while web-scale contrastive learning scales it. The margin $m=0.05$ triplet loss converts 2AFC human choices into a distance constraint on the embedding space, and this is what distinguishes the initialization from a random seed.

What would settle it

Train a control model whose Stage 1 uses the same NIGHTS triplet schedule but with randomly permuted human labels (or a non-human objective such as image reconstruction), then run identical Stage 2; if zero-shot accuracy matches the perceptual-initialization model, the human content is not causal. Alternatively, run the proposed pipeline and the random-start baseline across at least ten seeds and check whether the 23-of-29 improvement pattern persists outside the reported single seed.

Watch

Extended reading notes

Core claim

Starting from human perception rather than random weights changes what a contrastively trained vision–language model learns. The paper's central claim is that pre-biasing a CLIP ViT-B/32 vision encoder with NIGHTS triplet judgments—a two-alternative forced-choice similarity loss with margin 0.05—before CLIP-style InfoNCE pretraining on YFCC15M produces representations that generalize better zero-shot than the same pipeline with random initialization. The gains are reported across five benchmark families (ImageNet, ImageNet-OOD, VTAB, Fine-grained & Specialty, Domain & Small), with top-1 lifts of +2.4 to +4.0 percentage points by family and 23 of 29 datasets better at top-1; retrieval R@1 and R@5 improve on both Flickr30k and MS-COCO in both directions. The improvement appears early in pretraining (around 15 epochs on ImageNet) and the scaling exponents are higher, meaning the gap widens with more data. The paper further claims that the identical perceptual loss applied after pretraining, as in prior fine-tuning work, disrupts image–text alignment (COCO I→T R@1 falls from 14.2% to 1.3%), so the initialization stage—not the perceptual signal per se—is what preserves alignment.

Load-bearing premise

The gains are attributed to the human perceptual content of the NIGHTS warm start, but the baseline has no equivalent warm start with a non-human objective and only one run per condition is reported; if extra pretraining alone or seed luck explains the gap, the central claim collapses.

Editorial extensions

If this is right

  • If the central claim holds, human perceptual datasets can be used as cheap initialization priors (about 3 GPU-hours) instead of expensive post-hoc alignment, and the resulting models need no task-specific fine-tuning to improve zero-shot transfer.
  • Because gains emerge early and scaling exponents are higher, perceptually initialized models reach a given accuracy with fewer web-scale training samples, translating into compute savings in large pretraining runs.
  • The same perceptual-first schedule should transfer to other backbones (e.g., ResNet-style CNNs) and multimodal settings, where seed-induced variance is larger and initialization effects may be stronger.
  • The dependence of gains on semantic overlap between the behavioral dataset and target domain implies that curated, coverage-balanced perceptual datasets could yield larger and more uniform improvements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's baseline receives no equivalent warm start with a non-human objective, so a control experiment using a synthetic or self-supervised triplet objective (same data volume, same schedule) is needed to isolate whether it is the human content of NIGHTS—rather than the extra training itself—that causes the gains.
  • Because only one run per condition is reported and the paper itself notes seed-induced variance, multi-seed repetitions could change the size or direction of some per-dataset deltas; the family-level averages are more likely to be stable than single-dataset numbers.
  • If the mechanism is genuinely path-dependence through initialization, the same idea could be tested with brain-derived embeddings (fMRI/MEG) as initialization priors, not just behavioral judgments, and with non-visual modalities where perceptual judgments are scarce.
  • The catastrophic collapse from post-hoc fine-tuning hints that contrastive text supervision and human perceptual geometry conflict after training; perceptual initialization may work precisely because it shapes the geometry before text supervision locks it in, a testable ordering hypothesis.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Perceptual-Initialization (PI), a two-stage procedure in which a CLIP ViT-B/32 image encoder is first trained on 20k human 2AFC triplet judgments from NIGHTS using the margin loss in Eq. (2), and then the full CLIP model is trained on YFCC15M with the standard InfoNCE loss in Eq. (3). The authors compare this model against a randomly initialized CLIP baseline and a post-hoc NIGHTS fine-tuned variant, reporting higher zero-shot top-1/top-5 accuracy on 23 of 29 classification benchmarks and higher retrieval R@1/R@5 on Flickr30k and MS-COCO. They also report power-law scaling curves and claim that embedding human perceptual structure at initialization improves vision-language alignment.

Significance. If the causal claim were established, this would be a useful and somewhat provocative result: it would show that small human behavioral datasets can shape large-scale contrastive representations when applied at initialization, and it would challenge the prevailing post-hoc alignment paradigm. The paper has real strengths: the evaluation is external to the training objective (NIGHTS labels are not derived from the evaluation benchmarks), the compute budget is reported transparently, and the post-hoc fine-tuning collapse provides a useful point of comparison. However, the experimental design as presented cannot separate the effect of human perceptual content from the effect of additional pretraining, and the single-seed design conflicts with the paper's own discussion of seed-induced variance. The significance is therefore conditional on additional control experiments.

major comments (3)
  1. [§3.3, Tables 1–2] The central claim that human perceptual structure causes the reported improvements is not supported by the current comparison. The PI pipeline differs from the baseline in two ways at once: it receives an additional Stage-1 optimization on 20k NIGHTS triplets for 32 epochs, and those triplets carry human labels. A random-start baseline that receives no Stage-1 training cannot rule out the possibility that any reasonable warm start—for example, training on the same images with pseudo-labels, a self-supervised objective, or even random labels—would produce similar or identical gains by providing better low-level features or additional gradient steps. I request a control condition that matches the Stage-1 data and compute budget but replaces human labels with a non-human objective (e.g., shuffled or random triplet labels, or an unsupervised contrastive warm start on the same images), followed by the same Stage-2 YFCC15M training. Without such a control, the phrase "human perceptual structure" in the abstract and Section 1 is an attribution, not a demonstrated mechanism. The post-hoc fine-tuning collapse in Section 4.2 also shows that the same NIGHTS objective can severely distort a trained CLIP space, which makes it especially important to show that the initialization-stage benefit is specific to human labels rather than a generic property of the NIGHTS objective.
  2. [§1, Tables 1–2] Each condition is reported from a single training run, yet the paper's own introduction (Section 1, Refs. [1–3,6]) emphasizes that random seed can change ImageNet accuracy by several percentage points. The reported deltas range from +0.2 pp (Stanford Cars) to +14.6 pp (STL-10), with five losses and one tie; absent error bars or multiple seeds, the reader cannot distinguish signal from seed luck. This is not a cosmetic issue because the paper's headline claim is "consistently" higher performance across benchmarks. I ask for at least three seeds for the core comparison (PI vs. baseline) on the key ImageNet and retrieval benchmarks, or a statistically principled alternative (e.g., paired bootstrap over evaluation subsets if compute is the constraint).
  3. [§4.2] The comparison that motivates the "initialization vs. fine-tuning" framing is not controlled. Stage 1 trains the full vision encoder for 32 epochs on NIGHTS, while the post-hoc variant trains only QKV projections for 8 epochs on the same data, after a different initialization (YFCC-trained weights) and without matching the Stage-1 learning-rate schedule. The observed collapse therefore does not establish that perceptual supervision is inherently harmful at the fine-tuning stage; it may be an artifact of the particular fine-tuning recipe. To support the timing claim, the authors should either match the two conditions (e.g., full-encoder fine-tuning for 32 epochs with the same optimizer settings) or soften the conclusion to "the specific post-hoc recipe we tried collapsed."
minor comments (5)
  1. [§3.1, §3.2] Equations (2) and (3) contain unresolved citation placeholders "[?]" for the InfoNCE loss and AdamW optimizer; these should be completed before publication.
  2. [§4.2] The post-hoc fine-tuning collapse is reported only in prose; a small table with the collapsed classification and retrieval numbers would make the claim easier to verify, especially the COCO I→T R@1 drop from 14.2% to 1.3%.
  3. [Figures 3–4] The power-law exponents β are reported without confidence intervals or a description of the fitting procedure; since these exponents are used to argue for "steeper" scaling, their uncertainty should be quantified or the claim softened.
  4. [Table 1] The column headings "Ours@1 Base@1∆@1" are hard to read, and the "—" entries for top-5 accuracy (KITTI-CVD, PCAM, Rendered-SST2) are not explained in the caption.
  5. [§5] The Discussion proposes a sub-sampling study and architecture extensions but does not report them; the language should clearly separate completed experiments from future work.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported zero-shot gains are external measurements, and no equation or fitted parameter reduces the benchmark scores to the NIGHTS inputs.

full rationale

The paper's derivation chain is: Stage 1 trains a ViT-B/32 vision encoder on NIGHTS human triplet judgments using the perceptual loss of Eq. 2; Stage 2 trains the full CLIP model on YFCC15M with the contrastive InfoNCE loss of Eq. 3; zero-shot classification and retrieval are then measured on external benchmarks such as ImageNet, VTAB, Flickr30k, and MS-COCO. None of these benchmark labels appear in either training objective, and no parameter is fitted to the reported test numbers. The power-law exponents beta in Figures 3 and 4 are descriptive fits to the models' own training curves rather than predictions derived from the NIGHTS objective, so they are not a fitted input renamed as a prediction. The only self-citation to the authors' prior work, reference [16] (Zhao et al.), appears in the related-work and discussion sections as support for post-hoc perceptual alignment and is not load-bearing for the central claim. The triplet loss and margin are attributed to the external prior work [12], not to a self-citation chain or a uniqueness theorem. The paper's own observation in Section 5 that gains are largest on benchmarks overlapping with NIGHTS content is an empirical correlation, not a definitional equivalence. The absence of a warm-start control that uses non-human labels is a genuine experimental confound that weakens causal attribution to human perceptual structure, and single-seed runs raise variance concerns, but these are correctness and ablation issues, not circularity: they do not make the reported zero-shot scores equivalent to the NIGHTS inputs by construction. No circular step is exhibited, so the score is 0.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

No new physical or architectural entities are introduced. The method is a training schedule; all functional components (CLIP, ViT-B/32, triplet loss) are from prior work.

free parameters (7)
  • Perceptual margin m = 0.05
    Margin in the NIGHTS triplet loss (Eq. 2), taken from Sundaram et al. [12] without a sweep; directly shapes Stage 1 embeddings.
  • Stage 1 NIGHTS training epochs = 32
    Chosen by hand; controls how strongly the perceptual prior is embedded before YFCC15M.
  • Stage 2 YFCC15M training epochs = 32
    Training length for both PI and baseline; all comparisons are at this endpoint.
  • CLIP logit scale initialization = ln(100)
    Standard CLIP initialization for the temperature parameter, optimized during Stage 2.
  • Batch size = 30,720
    Batch size for Stage 2 and baseline; affects contrastive learning dynamics and was not varied.
  • Unreported AdamW hyperparameters = not disclosed
    Learning rate, weight decay, betas, warmup, and schedule are not given, so several effective free choices are undocumented.
  • Power-law exponents beta = e.g., 0.649 (I->T R@1), 0.514 (I->T R@5), 0.703 (T->I R@1), 0.546 (T->I R@5)
    Fitted to retrieval learning curves and used to argue that PI has steeper scaling; no confidence intervals are reported.
assumptions (5)
  • domain assumption NIGHTS 2AFC triplet judgments capture useful human perceptual structure for vision.
    Stage 1 loss (Eq. 2) treats these judgments as ground-truth similarity; no independent validation that this transfers to broader vision.
  • domain assumption InfoNCE contrastive learning on YFCC15M yields generalizable zero-shot representations.
    Stage 2 relies on the standard CLIP objective; the paper does not test other objectives.
  • domain assumption The PI and baseline training differ only in initialization.
    Section 3.4 claims identical hyperparameters and data, but this is asserted, not shown, and no code is released.
  • ad hoc to paper A single training run per condition is adequate for reliable comparisons.
    Paper cites seed-induced variance of several ImageNet points yet reports one seed; no error bars or repeated runs.
  • domain assumption Zero-shot top-1/top-5 and retrieval recall are valid proxies for vision-language alignment.
    Evaluations are all zero-shot classification and retrieval; the method is not tested on other alignment metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beginning with You: Perceptual-Initialization Improves Vision-Language Representation and Alignment." pith.science (2026). https://pith.science/paper/DE3PD5LE

@misc{pith2026250514204,
  author       = {Pith},
  title        = {Pith review of: Beginning with You: Perceptual-Initialization Improves Vision-Language Representation and Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DE3PD5LE}},
  note         = {Machine review of arXiv:2505.14204}
}
read the original abstract

We introduce Perceptual-Initialization (PI), a paradigm shift in visual representation learning that incorporates human perceptual structure during the initialization phase rather than as a downstream fine-tuning step. By integrating human-derived triplet embeddings from the NIGHTS dataset to initialize a CLIP vision encoder, followed by self-supervised learning on YFCC15M, our approach demonstrates significant zero-shot performance improvements, without any task-specific fine-tuning, across 29 zero shot classification and 2 retrieval benchmarks. On ImageNet-1K, zero-shot gains emerge after approximately 15 epochs of pretraining. Benefits are observed across datasets of various scales, with improvements manifesting at different stages of the pretraining process depending on dataset characteristics. Our approach consistently enhances zero-shot top-1 accuracy, top-5 accuracy, and retrieval recall (e.g., R@1, R@5) across these diverse evaluation tasks, without requiring any adaptation to target domains. These findings challenge the conventional wisdom of using human-perceptual data primarily for fine-tuning and demonstrate that embedding human perceptual structure during early representation learning yields more capable and vision-language aligned systems that generalize immediately to unseen tasks. Our work shows that "beginning with you", starting with human perception, provides a stronger foundation for general-purpose vision-language intelligence.

Figures

Figures reproduced from arXiv: 2505.14204 by the authors.

Figure 1
Figure 1. Perceptual-Initialization (PI) yields faster, stronger zero-shot performance. Model initialization. The image encoder is pre-biased with human triplet-similarity judgment from the NIGHTS dataset, while a control model is fully random-initialized. Model Training. Both models are then trained with the same image–text contrastive objective on YFCC15M. Zero-shot evaluation. Without any task-specific fine-tuning, the per… view at source ↗
Figure 2
Figure 2. Perceptual-Initialization yields consistent zero-shot gains across all benchmark families. (a) Mean Top-1 accuracy and (b) mean Top-5 accuracy after 32 epochs of YFCC15M pre-training. Perceptual-Initialization surpasses the web-only baseline for every family—ImageNet, ImageNet-OOD, VTAB, Fine-grained & Specialty, and Domain & Small. Numbers above the bars denote the average lift in percentage points (pp). Overall, P… view at source ↗
Figure 3
Figure 3. Zero-shot classification scaling results. Top-1 accuracy (top row) and Top-5 accuracy (bottom row) are shown for five benchmark families—ImageNet, ImageNet OOD, VTAB, Fine-grained & Specialty, and Misc./Domain & Small—plotted against the log-scale of training samples seen (10 M → 300 M) over total of 32 training epochs. The blue curve denotes our Perceptual-Initialization pipeline (NIGHTS20k → YFCC15M) and the orang… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Retrieval Tasks Scaling Results. Recall@1 and Recall@5 are plotted (log-scale, number of image–text pairs seen) over successive epochs on YFCC15M for two retrieval directions: (a) Image → Text R@1, (b) Image → Text R@5, (c) Text → Image R@1, and (d) Text → Image R@5. T…
Figure 5
Figure 5. Figure 5: Qualitative comparison of zero-shot retrieval. (a) Image→Text: For two query images, we list the ground-truth captions (left) and the top-5 captions returned by each model, together with their cosine similarity scores (higher is better). Ground-truth matches are highli…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 38 canonical work pages

  1. [12]

    URL https://arxiv.org/abs/2410.10817.2410.10817

    Sundaram, S.et al.When does perceptual alignment benefit vision representations? (2024). URL https://arxiv.org/abs/2410.10817.2410.10817

    arXiv 2024

  2. [1]

    J., Kriegeskorte, N

    Mehrer, J., Spoerer, C. J., Kriegeskorte, N. & Kietzmann, T. C. Individual differences among deep neural network models.Nature Communications11, 5725 (2020). URL https://www.nature. com/articles/s41467-020-19632-w

    work page 2020

  3. [2]

    & Jain, R

    Madhyastha, P. & Jain, R. On model stability as a function of random seed, arxiv.arXiv preprint arXiv:1909.10447(2019)

    arXiv 2019

  4. [3]

    Picard, D. Torch. manual_seed (3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision.arXiv preprint arXiv:2109.08203(2021)

    arXiv 2021

  5. [4]

    ImageNet-1k benchmark

    Russakovsky, O.et al.Imagenet large scale visual recognition challenge.IJCV211–252 (2015). ImageNet-1k benchmark

    work page 2015

  6. [5]

    The Goldilocks zone: Towards better understanding of neural network loss landscapes

    Fort, S. & Scherlis, A. The goldilocks zone: Towards better understanding of neural network loss landscapes (2018). URLhttps://arxiv.org/abs/1807.02581.1807.02581

    work page Pith review arXiv 2018

  7. [6]

    URL https://arxiv.org/abs/2103.03098.2103.03098

    Bouthillier, X.et al.Accounting for variance in machine learning benchmarks (2021). URL https://arxiv.org/abs/2103.03098.2103.03098

    arXiv 2021

  8. [7]

    On the variance of neural network training with respect to test sets and distributions

    Jordan, K. On the variance of neural network training with respect to test sets and distributions. arXiv preprint arXiv:2304.01910(2023)

    arXiv 2023

Show all 61 references
  1. [8]

    N.et al.Things: A database of 1,854 object concepts and more than 26,000 naturalistic object images.PloS one14, e0223792 (2019)

    Hebart, M. N.et al.Things: A database of 1,854 object concepts and more than 26,000 naturalistic object images.PloS one14, e0223792 (2019)

    2019

  2. [9]

    N., Zheng, C

    Hebart, M. N., Zheng, C. Y ., Pereira, F. & Baker, C. I. Revealing the multidimensional mental representations of natural objects underlying human similarity judgements.Nature human behaviour 4, 1173–1185 (2020)

    2020

  3. [10]

    URLhttps://arxiv.org/abs/2306.09344.2306.09344

    Fu, S.et al.Dreamsim: Learning new dimensions of human visual similarity using synthetic data (2023). URLhttps://arxiv.org/abs/2306.09344.2306.09344

    2023 arXiv

  4. [11]

    A., Shechtman, E

    Zhang, R., Isola, P., Efros, A. A., Shechtman, E. & Wang, O. The unreasonable effectiveness of deep features as a perceptual metric (2018). URL https://arxiv.org/abs/1801.03924. 1801.03924

    2018 arXiv

  5. [13]

    URLhttps://arxiv.org/abs/2306.04507.2306.04507

    Muttenthaler, L.et al.Improving neural network representations using human similarity judgments (2023). URLhttps://arxiv.org/abs/2306.04507.2306.04507

    2023 arXiv

  6. [14]

    URLhttps://arxiv.org/abs/2409.06509.2409.06509

    Muttenthaler, L.et al.Aligning machine and human visual representations across abstraction levels (2024). URLhttps://arxiv.org/abs/2409.06509.2409.06509

    2024 arXiv

  7. [16]

    C.et al.Shifting attention to you: Personalized brain-inspired ai models.arXiv preprint arXiv:2502.04658(2025)

    Zhao, S. C.et al.Shifting attention to you: Personalized brain-inspired ai models.arXiv preprint arXiv:2502.04658(2025)

    2025 arXiv

  8. [17]

    & Philbin, J

    Schroff, F., Kalenichenko, D. & Philbin, J. Facenet: A unified embedding for face recognition and clustering. In2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 815–823 (IEEE, 2015). URLhttp://dx.doi.org/10.1109/CVPR.2015.7298682

    2015

  9. [18]

    URLhttps://arxiv.org/abs/2010.11929.2010.11929

    Dosovitskiy, A.et al.An image is worth 16x16 words: Transformers for image recognition at scale (2021). URLhttps://arxiv.org/abs/2010.11929.2010.11929

    2021 arXiv

  10. [19]

    & Hinton, G

    Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A simple framework for contrastive learning of visual representations.arXiv preprint arXiv:2002.05709(2020). URL https://arxiv.org/ abs/2002.05709. 11

    2020 arXiv

  11. [20]

    URLhttp://dx.doi.org/10.1145/2812802

    Thomee, B.et al.Yfcc100m: the new data in multimedia research.Communications of the ACM 59, 64–73 (2016). URLhttp://dx.doi.org/10.1145/2812802

    2016 doi

  12. [21]

    URLhttps://arxiv.org/abs/2103.00020.2103.00020

    Radford, A.et al.Learning transferable visual models from natural language supervision (2021). URLhttps://arxiv.org/abs/2103.00020.2103.00020

    2021 arXiv

  13. [22]

    InICML(2021)

    Jia, C.et al.Scaling up visual and vision-language representation learning with noisy text supervi- sion. InICML(2021)

    2021

  14. [23]

    & Beyer, L

    Zhai, X., Kolesnikov, A., Houlsby, N. & Beyer, L. Lit: Zero-shot transfer with locked-image text tuning.arXiv preprint arXiv:2201.03545(2022)

    2022 arXiv

  15. [24]

    URL https://arxiv.org/abs/2111.12710.2111.12710

    Dong, X.et al.Peco: Perceptual codebook for bert pre-training of vision transformers (2022). URL https://arxiv.org/abs/2111.12710.2111.12710

    2022 arXiv

  16. [25]

    D., Croce, F

    Schlarmann, C., Singh, N. D., Croce, F. & Hein, M. Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models (2024). URL https: //arxiv.org/abs/2402.12336.2402.12336

    2024 arXiv

  17. [26]

    Croce, F., Schlarmann, C., Singh, N. D. & Hein, M. Adversarially robust clip models can induce better (robust) perceptual metrics.arXiv preprint arXiv:2502.11725(2025)

    2025 arXiv

  18. [27]

    Gu, T.et al.Rwkv-clip: A robust vision-language representation learner (2024).2406.06973

    2024 arXiv

  19. [28]

    R.et al.Deep neural networks and brain alignment: Brain encoding and decoding (survey) (2024)

    Oota, S. R.et al.Deep neural networks and brain alignment: Brain encoding and decoding (survey) (2024). URLhttps://arxiv.org/abs/2307.10246.2307.10246

    2024 arXiv

  20. [29]

    Y .et al.Datacomp: In search of the next generation of multimodal datasets (2023)

    Gadre, S. Y .et al.Datacomp: In search of the next generation of multimodal datasets (2023). URL https://arxiv.org/abs/2304.14108.2304.14108

    2023 arXiv

  21. [30]

    & Song, D

    Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J. & Song, D. Natural adversarial examples. In CVPR(2021). Introduces ImageNet-A and ImageNet-O

    2021

  22. [31]

    & Song, D

    Hendrycks, D., Burns, C., Fei, A. & Song, D. The many faces of robustness: A critical analysis of out-of-distribution generalization. InICCV(2021). Releases ImageNet-R

    2021

  23. [32]

    Wang, H., Meghawat, A., Lipton, Z. C. & Xing, E. P. Learning robust global representations by penalizing local predictive power.arXiv:1908.09912(2019). Contains ImageNet-Sketch

    2019 arXiv

  24. [33]

    & Shankar, V

    Recht, B., Roelofs, R., Schmidt, L. & Shankar, V . Do imagenet classifiers generalize to imagenet? InICML(2019). Introduces ImageNet-V2

    2019

  25. [34]

    In Wallach, H.et al.(eds.)Advances in Neural Information Processing Systems, vol

    Barbu, A.et al.Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Wallach, H.et al.(eds.)Advances in Neural Information Processing Systems, vol. 32 (Curran Associates, Inc., 2019). URL https://proceedings.neurips.cc/ paper_...

    2019

  26. [35]

    & Hinton, G

    Krizhevsky, A. & Hinton, G. Learning multiple layers of features from tiny images. Tech. Rep. 0, University of Toronto, Toronto, Ontario (2009). URL https://www.cs.toronto.edu/~kriz/ learning-features-2009-TR.pdf

    2009

  27. [36]

    & Perona, P

    Fei-Fei, L., Fergus, R. & Perona, P. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. InProceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW’04) Volume 12 - ...

    2004

  28. [37]

    URLhttps://arxiv.org/abs/1612.06890.1612.06890

    Johnson, J.et al.Clevr: A diagnostic dataset for compositional language and elementary visual reasoning (2016). URLhttps://arxiv.org/abs/1612.06890.1612.06890

    2016 arXiv

  29. [38]

    & Urtasun, R

    Geiger, A., Lenz, P. & Urtasun, R. Are we ready for autonomous driving? the kitti vision benchmark suite. InConference on Computer Vision and Pattern Recognition (CVPR)(2012)

    2012

  30. [39]

    & Vedaldi, A

    Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S. & Vedaldi, A. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)(2014). 12

    2014

  31. [40]

    & Borth, D

    Helber, P., Bischke, B., Dengel, A. & Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing12, 2217–2226 (2019)

    2019

  32. [41]

    & Zisserman, A

    Nilsback, M.-E. & Zisserman, A. Automated flower classification over a large number of classes. InIndian Conference on Computer Vision, Graphics and Image Processing(2008)

    2008

  33. [42]

    M., Vedaldi, A., Zisserman, A

    Parkhi, O. M., Vedaldi, A., Zisserman, A. & Jawahar, C. V . Cats and dogs. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)(2012)

    2012

  34. [43]

    Cheng, G., Han, J. & Lu, X. Remote sensing image scene classification: Benchmark and state of the art.IEEE Transactions on Geoscience and Remote Sensing55, 3093–3105 (2017). URL https://doi.org/10.1109/tgrs.2017.2650986

    2017

  35. [44]

    InNIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011(2011)

    Netzer, Y .et al.Reading digits in natural images with unsupervised feature learning. InNIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011(2011). URL http: //ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf

    2011

  36. [45]

    S., Linmans, J., Winkens, J., Cohen, T

    Veeling, B. S., Linmans, J., Winkens, J., Cohen, T. & Welling, M. Rotation equivariant cnns for digital pathology. InInternational Conference on Medical image computing and computer-assisted intervention, 210–218 (Springer, 2018)

    2018

  37. [46]

    & Fei-Fei, L

    Krause, J., Stark, M., Deng, J. & Fei-Fei, L. 3d object representations for fine-grained categorization. In4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13)(Sydney, Australia, 2013)

    2013

  38. [47]

    & Van Gool, L

    Bossard, L., Guillaumin, M. & Van Gool, L. Food-101 – mining discriminative components with random forests. InEuropean Conference on Computer Vision(2014)

    2014

  39. [48]

    & Vedaldi, A

    Maji, S., Rahtu, E., Kannala, J., Blaschko, M. & Vedaldi, A. Fine-grained visual classification of aircraft (2013). URLhttps://arxiv.org/abs/1306.5151.1306.5151

    2013 arXiv

  40. [49]

    K., Winn, J

    Everingham, M., Gool, L., Williams, C. K., Winn, J. & Zisserman, A. The pascal visual object classes (voc) challenge.Int. J. Comput. Vision88, 303–338 (2010). URL https://doi.org/10. 1007/s11263-009-0275-4

    2010

  41. [50]

    & Sim, J

    Weyand, T., Araujo, A., Cao, B. & Sim, J. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 11041–11050 (2020)

    2020

  42. [51]

    & Igel, C

    Stallkamp, J., Schlipsing, M., Salmen, J. & Igel, C. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition.Neural Networks32, 323–332 (2012)

    2012

  43. [52]

    & Haffner, P

    LeCun, Y ., Bottou, L., Bengio, Y . & Haffner, P. Gradient-based learning applied to document recognition.Proceedings of the IEEE86, 2278–2324 (1998)

    1998

  44. [53]

    In Yarowsky, D., Baldwin, T., Korhonen, A., Livescu, K

    Socher, R.et al.Recursive deep models for semantic compositionality over a sentiment treebank. In Yarowsky, D., Baldwin, T., Korhonen, A., Livescu, K. & Bethard, S. (eds.)Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 1631–1642 (Associa...

    2013

  45. [54]

    & Lee, H

    Coates, A., Ng, A. & Lee, H. An analysis of single-layer networks in unsupervised feature learning. In Gordon, G., Dunson, D. & Dudík, M. (eds.)Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, vol. 15 ofProceedings of Machine Le...

    2011

  46. [55]

    URL https://arxiv.org/ abs/1405.0312.1405.0312

    Lin, T.-Y .et al.Microsoft coco: Common objects in context (2015). URL https://arxiv.org/ abs/1405.0312.1405.0312

    2015 arXiv

  47. [56]

    & Hockenmaier, J

    Young, P., Lai, A., Hodosh, M. & Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.Transactions of the As- sociation for Computational Linguistics2, 67–78 (2014). URL https://aclanthology.org/ Q...

    2014

  48. [57]

    M., Khosla, A., Pantazis, D., Torralba, A

    Cichy, R. M., Khosla, A., Pantazis, D., Torralba, A. & Oliva, A. Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence.Scientific Reports6, 27755 (2016)

    2016

  49. [58]

    URL https://www.biorxiv.org/content/early/2018/09/05/ 407007.https://www.biorxiv.org/content/early/2018/09/05/407007.full.pdf

    Schrimpf, M.et al.Brain-score: Which artificial neural network for object recognition is most brain-like?bioRxiv(2018). URL https://www.biorxiv.org/content/early/2018/09/05/ 407007.https://www.biorxiv.org/content/early/2018/09/05/407007.full.pdf

    2018

  50. [59]

    & Hebart, M

    Kaniuth, P. & Hebart, M. N. Feature-reweighted representational similarity analysis: A method for improving the fit between computational models, brains, and behavior.NeuroImage257, 119294 (2022)

    2022

  51. [60]

    & Wang, Q

    Liu, Y ., Zhang, X., Wang, Y . & Wang, Q. A deep learning approach for predicting odor perception from molecular structure. In2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 1433–1436 (IEEE, 2022). URL https://ieeexplore....

    2022

  52. [61]

    URLhttps://www.nature.com/articles/s41598-024-71693-9

    Li, Z.et al.Structure-based prediction of the odor perception of molecules.Scientific Reports14, 9406 (2024). URLhttps://www.nature.com/articles/s41598-024-71693-9

    2024

  53. [62]

    URL https: //www.nature.com/articles/s41540-024-00401-0

    Li, Y .et al.A deep position-encoding model for predicting olfactory perception from molecular structures and electrostatics.npj Systems Biology and Applications10, 23 (2024). URL https: //www.nature.com/articles/s41540-024-00401-0. 14

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.