REVIEW 4 cited by
InfoNCE: Identifying the Gap Between Theory and Practice
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Prior theory work on Contrastive Learning via the InfoNCE loss showed that, under certain assumptions, the learned representations recover the ground-truth latent factors. We argue that these theories overlook crucial aspects of how CL is deployed in practice. Specifically, they either assume equal variance across all latents or that certain latents are kept invariant. However, in practice, positive pairs are often generated using augmentations such as strong cropping to just a few pixels. Hence, a more realistic assumption is that all latent factors change with a continuum of variability across all factors. We introduce AnInfoNCE, a generalization of InfoNCE that can provably uncover the latent factors in this anisotropic setting, broadly generalizing previous identifiability results in CL. We validate our identifiability results in controlled experiments and show that AnInfoNCE increases the recovery of previously collapsed information in CIFAR10 and ImageNet, albeit at the cost of downstream accuracy. Finally, we discuss the remaining mismatches between theoretical assumptions and practical implementations.
Forward citations
Cited by 4 Pith papers
-
Estimating the Empowerment of Language Model Agents
EELMA estimates the mutual information between an LM agent's actions and future text states, and this 'empowerment' is shown to correlate with task performance across toy games and WebArena.
-
Contrastive Self-Supervised Network Intrusion Detection using Augmented Negative Pairs
CLAN clusters genuine benign network flows while repelling augmented copies, then classifies new flows by distance to the cluster centroid; on Lycos2017 it reports the highest mean AUROC among compared SSL and anomaly...
-
Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learning
Contrastive Successor Features recover ground-truth RL states up to a linear map whenever the skill-conditioned transition differences follow a von Mises-Fisher distribution and policies are diverse.
-
Zero Shot Composed Image Retrieval
Fine-tuning BLIP-2 with a Q-Former raises FashionIQ validation Recall@10 to roughly 45%, but the zero-shot framing is inaccurate and the DPO variant omits the reference image.
Discussion (0). Sign in to comment.