MatMMExtract pipeline creates MatSciFig dataset of 391k annotated materials science figure panels and MaterialScope detection dataset with high accuracy.
super hub Mixed citations
Representation Learning with Contrastive Predictive Coding
Mixed citation behavior. Most common role is background (52%).
597
Pith papers citing it
14
external citations · Pith
Background
52% of classified citations
abstract
While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence. In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models. We use a probabilistic contrastive loss which induces the latent space to capture information that is maximally useful to predict future samples. It also makes the model tractable by using negative sampling. While most prior work has focused on evaluating representations for a particular modality, we demonstrate that our approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.
hub tools
citation-role summary
background 30
method 23
dataset 2
baseline 1
citation-polarity summary
claims ledger
- abstract While supervised learning has enabled great progress in many applications, unsupervised learning has not seen such widespread adoption, and remains an important and challenging endeavor for artificial intelligence. In this work, we propose a universal unsupervised learning approach to extract useful representations from high-dimensional data, which we call Contrastive Predictive Coding. The key insight of our model is to learn such representations by predicting the future in latent space by using powerful autoregressive models. We use a probabilistic contrastive loss which induces the latent s
authors
co-cited works
representative citing papers
LeJEPA achieves linear identifiability of latent variables uniquely when the latents are Gaussian in worlds with stationary additive-noise transitions.
Dimension d = O(m^{-2} log n) nearly achieves the optimal margin m^rd(+∞, A) for retrieval embeddings, with matching lower bounds showing d = O(k log(n/k)) suffices and is necessary for m = Θ(k^{-1/2}) on k-sparse query matrices.
Weighted InfoNCE objectives realize specific target geometries in embedding space, with SupCon producing size-dependent inter-class similarities under imbalance while Soft SupCon and certain continuous variants preserve regular simplex or unique optima.
Task structure is identifiable across time steps and task-relevant representations are identifiable within steps in a nonparametric setting under sparsity regularization.
CLAD is the first deep learning framework for log anomaly detection that operates directly on compressed byte streams using a dilated convolutional encoder, hybrid Transformer-mLSTM, and two-stage training, achieving 0.9909 average F1-score across five datasets.
FashionMV introduces product-level multi-view CIR, a 127K-product dataset built via automated LMM pipeline, and a 0.8B ProCIR model that beats larger baselines on three fashion benchmarks.
Masked autoencoders with asymmetric encoder-decoder and 75% masking ratio enable scalable self-supervised pre-training of vision transformers, achieving 87.8% ImageNet-1K accuracy with ViT-Huge using only unlabeled data.
CLIPScore uses a web-pretrained CLIP model to evaluate image captions without references and achieves higher human correlation than CIDEr or SPICE.
SCOREBED isolates EIG double intractability in a policy-independent score-matching stage, then trains design policies with a singly intractable gradient estimator, enabling cheap multi-policy selection.
For stationary image datasets and standard augmentations, the optimal contrastive representation is partial whitening of DFT power, implemented by a CNN with sinusoidal first-layer filters and a waterfilling weight allocation.
CPF-GCD enforces low-rank compositional structure on vision backbone features via spatial primitive fields so that novel categories emerge as new activation patterns over a shared vocabulary of reusable visual primitives.
BrainFIBRE pretrains a five-expert Mixture-of-Experts model on NODDI-derived microstructural maps and outperforms prior deep models on age, sex, cerebrovascular, neurodegenerative, and cognitive prediction.
PromptGNN-sim uses GAT-based semantically aware neighborhood selection and structure-aware LLM prompts with bi-directional contrastive alignment to outperform prior GNN, LLM, and fusion methods on text-attributed graph datasets.
Theoretical characterization of the inlier-memorization effect in simple autoencoders, deriving its emergence, strength, and persistence from data distribution and initialization, plus guidelines achieving SOTA on ADBench.
OvESyn is the first text-conditioned 3D CT synthesis framework for abdomino-pelvic oncologic imaging, constructing evidence-based text from metadata to adapt a latent diffusion model across the domain gap from chest CT pretraining.
OctoSense supplies a large multimodal robotics dataset and a late-fusion masked autoencoder that runs fast and outperforms image-only models on optical flow, depth, segmentation, and ego-motion tasks while remaining robust under sensor degradation.
DT² trains digital twins to preserve pairwise policy rankings from fitted Q-evaluation on offline data rather than minimizing one-step transition errors, improving policy ranking and reducing decision regret.
DREAM enables training of dense retrieval embeddings using autoregressive next-token prediction from LLMs by modulating attention with retriever scores.
HeRA aligns least-aligned attention heads in MLLMs using an MKNN-based contrastive objective to preserve cross-modal topological structure, yielding gains on vision-centric tasks and reduced hallucinations across 18 benchmarks.
Closed-form expression for conditional mutual information in linear Gaussian DAGs, constructed from AD primitives, enables gradient-based optimization of multi-terminal wireless rate regions.
Introduces MultiMem as the first metric for memorization in multi-modal contrastive learning, identifies cross-modal misalignment (text dominant) as key driver, and shows targeted augmentations reduce it while improving performance.
UNIEGO creates a unified egocentric encoder by using proxy models to mediate multi-teacher distillation from ego-exo, multi-modal, and foundation-model sources, with selective distillation and convex initialization, achieving SOTA on action recognition, retrieval, and segmentation.
S-JEPA uses soft GMM posteriors in a JEPA framework for self-supervised speech learning, achieving lowest WER below 90M parameters without offline re-clustering.
citing papers explorer
-
Unlocking the Visual Record of Materials Science: A Large-Scale Multimodal Dataset from Scientific Literature
MatMMExtract pipeline creates MatSciFig dataset of 391k annotated materials science figure panels and MaterialScope detection dataset with high accuracy.
-
When Does LeJEPA Learn a World Model?
LeJEPA achieves linear identifiability of latent variables uniquely when the latents are Gaussian in worlds with stationary additive-noise transitions.
-
Is Dimensionality a Barrier for Retrieval Models?
Dimension d = O(m^{-2} log n) nearly achieves the optimal margin m^rd(+∞, A) for retrieval embeddings, with matching lower bounds showing d = O(k log(n/k)) suffices and is necessary for m = Θ(k^{-1/2}) on k-sparse query matrices.
-
A Unified Geometric Framework for Weighted Contrastive Learning
Weighted InfoNCE objectives realize specific target geometries in embedding space, with SupCon producing size-dependent inter-class similarities under imbalance while Soft SupCon and certain continuous variants preserve regular simplex or unique optima.
-
From Generalist to Specialist Representation
Task structure is identifiable across time steps and task-relevant representations are identifiable within steps in a nonparametric setting under sparsity regularization.
-
CLAD: Efficient Log Anomaly Detection Directly on Compressed Representations
CLAD is the first deep learning framework for log anomaly detection that operates directly on compressed byte streams using a dilated convolutional encoder, hybrid Transformer-mLSTM, and two-stage training, achieving 0.9909 average F1-score across five datasets.
-
FashionMV: Product-Level Composed Image Retrieval with Multi-View Fashion Data
FashionMV introduces product-level multi-view CIR, a 127K-product dataset built via automated LMM pipeline, and a 0.8B ProCIR model that beats larger baselines on three fashion benchmarks.
-
Masked Autoencoders Are Scalable Vision Learners
Masked autoencoders with asymmetric encoder-decoder and 75% masking ratio enable scalable self-supervised pre-training of vision transformers, achieving 87.8% ImageNet-1K accuracy with ViT-Huge using only unlabeled data.
-
CLIPScore: A Reference-free Evaluation Metric for Image Captioning
CLIPScore uses a web-pretrained CLIP model to evaluate image captions without references and achieves higher human correlation than CIDEr or SPICE.
-
Bayesian Experimental Design via Score Matching
SCOREBED isolates EIG double intractability in a policy-independent score-matching stage, then trains design policies with a singly intractable gradient estimator, enabling cheap multi-policy selection.
-
A Theory of Contrastive Learning with Natural Images
For stationary image datasets and standard augmentations, the optimal contrastive representation is partial whitening of DFT power, implemented by a CNN with sinusoidal first-layer filters and a waterfilling weight allocation.
-
Identifying Latent Concepts and Structures for Generalized Category Discovery
CPF-GCD enforces low-rank compositional structure on vision backbone features via spatial primitive fields so that novel categories emerge as new activation patterns over a shared vocabulary of reusable visual primitives.
-
BrainFIBRE: A Foundation Model via Information Decomposition for Brain Microstructure
BrainFIBRE pretrains a five-expert Mixture-of-Experts model on NODDI-derived microstructural maps and outperforms prior deep models on age, sex, cerebrovascular, neurodegenerative, and cognitive prediction.
-
PromptGNN-sim: Deep Fusion and Alignment of GNN and LLMs for Text-Attributed Graph Learning
PromptGNN-sim uses GAT-based semantically aware neighborhood selection and structure-aware LLM prompts with bi-directional contrastive alignment to outperform prior GNN, LLM, and fusion methods on text-attributed graph datasets.
-
What Drives the Inlier-Memorization Effect? A Theory of Outlier Detection via Early Training Dynamics
Theoretical characterization of the inlier-memorization effect in simple autoencoders, deriving its emergence, strength, and persistence from data distribution and initialization, plus guidelines achieving SOTA on ADBench.
-
Evidence-Based Text-Conditioned 3D CT Synthesis for Ovarian Cancer
OvESyn is the first text-conditioned 3D CT synthesis framework for abdomino-pelvic oncologic imaging, constructing evidence-based text from metadata to adapt a latent diffusion model across the domain gap from chest CT pretraining.
-
OctoSense: Self-Supervised Learning for Multimodal Robot Perception
OctoSense supplies a large multimodal robotics dataset and a late-fusion masked autoencoder that runs fast and outperforms image-only models on optical flow, depth, segmentation, and ego-motion tasks while remaining robust under sensor degradation.
-
$\text{DT}^2$: Decision-Targeted Digital Twins
DT² trains digital twins to preserve pairwise policy rankings from fitted Q-evaluation on offline data rather than minimizing one-step transition errors, improving policy ranking and reducing decision regret.
-
DREAM: Dense Retrieval Embeddings via Autoregressive Modeling
DREAM enables training of dense retrieval embeddings using autoregressive next-token prediction from LLMs by modulating attention with retriever scores.
-
Mind the Heads: Topological Representation Alignment for Multimodal LLMs
HeRA aligns least-aligned attention heads in MLLMs using an MKNN-based contrastive objective to preserve cross-modal topological structure, yielding gains on vision-centric tasks and reduced hallucinations across 18 benchmarks.
-
Differentiable Conditional Mutual Information for Multi-Terminal Linear Gaussian Wireless Networks
Closed-form expression for conditional mutual information in linear Gaussian DAGs, constructed from AD primitives, enables gradient-based optimization of multi-terminal wireless rate regions.
-
MultiMem: Measuring and Mitigating Memorization in Multi-Modal Contrastive Learning
Introduces MultiMem as the first metric for memorization in multi-modal contrastive learning, identifies cross-modal misalignment (text dominant) as key driver, and shows targeted augmentations reduce it while improving performance.
-
UNIEGO: Proxies as Mediators for Unified Egocentric Video Representation Learning
UNIEGO creates a unified egocentric encoder by using proxy models to mediate multi-teacher distillation from ego-exo, multi-modal, and foundation-model sources, with selective distillation and convex initialization, achieving SOTA on action recognition, retrieval, and segmentation.
-
S-JEPA : Soft Clustering Anchors for Self-Supervised Speech Representation Learning
S-JEPA uses soft GMM posteriors in a JEPA framework for self-supervised speech learning, achieving lowest WER below 90M parameters without offline re-clustering.
-
Contextualizing Biological Language Models across Modalities via Logit-Space Contrastive Alignment
LOGICA adds context to pretrained biological LMs via logit-space contrastive alignment with gated adapters, improving AUC on held-out drug-resistance mutation ranking from ~0.55 to ~0.65 while preserving token likelihoods.
-
Non-negative Elastic Net Decoding for Information Retrieval
NNN decoding selects documents via non-negative elastic net reconstruction of the query embedding, with a theorem showing it strictly dominates dense retrieval on correlated corpora and experiments showing gains over inner-product baselines.
-
Where Should Action Generation Begin? A Learnable Source Prior for Generative Robot Policies
LeaP introduces a learnable proprioception-conditioned diagonal Gaussian source prior for generative robot policies, raising average success rates on 15 RoboTwin tasks from baselines by 6.5-25.5 points.
-
Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins
OR3 converts OR clips to action-driven digital twins, uses LLM imagination for hypothetical ActDTs, and achieves 57.6 R@1 and 77.3 R@5 on 276 implicit queries from 386 robotic knee procedure clips, outperforming baselines.
-
Learning Object Manipulation from Scratch via Contrastive Interaction
IWR improves CRL sample efficiency and performance in interaction-rich manipulation by interaction-aware resampling that preserves mode boundaries, yielding 19.8% average gains and a real-world air-hockey agent.
-
Unsupervised Style Representation Learning for AI-Text Detection via Paraphrase Inversion
Unsupervised style representations learned via paraphrase inversion enable competitive few-shot and zero-shot AI-text detection with better generalization to unseen LLMs than supervised baselines.
-
PrivCode++: Latent-Conditioned Differentially Private Code Generation for Comprehensive Guarantees
PrivCode++ introduces the first DP code generation method protecting both prompts and code via latent-conditioned two-stage training, claiming higher utility and stronger privacy than prior baselines.
-
CRANE: Knowledge Editing for Reasoning MLLMs
CRANE uses dual-library retrieval plus two-phase training (SFT then GRPO with cognitive routing reward) to reach 96.9% grounded success on conflict edits in reasoning MLLMs.
-
Contrast encodes inductive bias: separating slow noise from dynamics in predictive representation learning
Cross-trajectory negative sampling in contrastive predictive objectives causes encoding of slow noise over dynamics; intra-trajectory sampling eliminates the shortcut and recovers dynamical variables even under strong noise.
-
Mutual Information Optimization via K-Recursion and Automatic Differentiation for Linear Gaussian Wireless Networks
K-recursion plus automatic differentiation yields a topology-agnostic optimizer for end-to-end mutual information in linear Gaussian wireless DAGs that recovers water-filling optima where known.
-
BCG-FM: A Foundation Model for Ambient Cardiac Health Sensing
BCG-FM, the first foundation model for ambient BCG, achieves 3.26-year MAE on biological age estimation and discriminates 15 health conditions using frozen embeddings from participant-level contrastive pretraining on the largest raw biosignal corpus reported.
-
A Vision-language Framework for Comparative Reasoning in Radiology
Introduces MedReCo-DB dataset of 690k+ images and entity-aware models MedReCo/MedReCo-VLM that improve reference retrieval and comparative change interpretation in radiology across multiple centers and modalities.
-
CausalPOI: Spatio-Temporal Graph-Based Causal Modeling for Cold-Start POI Check-in Forecasting
CausalPOI proposes a spatio-temporal graph causal learning method for cold-start POI check-in forecasting that builds functional interaction graphs and treatment-control pairs to outperform baselines on SafeGraph data.
-
Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain
SAS reveals that medical images retain richer structural information than paired clinical reports in VLMs, an asymmetry hidden from symmetric metrics, with strongest correlation to retrieval performance.
-
Argus-Retriever: Vision-LLM Late-Interaction Retrieval with Region-Aware Query-Conditioned MoE for Visual Document Retrieval
Argus achieves the highest reported NDCG scores among open late-interaction models on ViDoRe V1 and combined V1+V2 by introducing query-dependent document representations via a region-aware MoE on Qwen3.5-VL, trained on 9% of public data with a 1024-dim head.
-
Are Common Substructures Transferable? Riemannian Graph Foundation Model with Neural Vector Bundles
GAUGE is a pretrainable Riemannian graph model with neural vector bundles and a Dirichlet loss that captures transferable intrinsic geometry, validated on zero-shot link prediction and graph isomorphism.
-
PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation
PixelRAG shows that operating RAG entirely over web screenshots outperforms text-based retrieval on NQ, SimpleQA, MMSearch, LiveVQA, and MoNaCo, with up to 18.1% accuracy gains and 3x token savings via image compression.
-
Cross-modal linkage risk in clinical vision-language models
Clinical VLMs enable image-to-report retrieval far above chance (15-50x at N=100-10k), persisting beyond disease labels, with targeted DP on projection heads cutting Recall@1 by 61.8% and preserving AUROC.
-
Generalization Limits in Vehicle Re-Identification
Standard vehicle re-ID benchmarks allow memorization of seen vehicle types; a new train/test split by vehicle type and view shows that state-of-the-art methods fail to generalize to unseen vehicles.
-
When Hard Negatives Hurt: Bridging the Generative-Discriminative Gap in Hard Negative Synthesis for Retrieval
Identifies the generative-discriminative gap in LLM hard negative synthesis for retrieval and proposes CausalNeg using CoT counterfactual perturbation plus query-view entropy maximization to generate more effective negatives.
-
Chameleon: Style-Content Disentangled Framework for Cross-Domain Object Compositing
Chameleon proposes the first large-scale cross-domain compositing dataset and a disentangled encoder plus gated diffusion transformer that outperforms prior in-domain and cross-domain methods on plausibility and fidelity.
-
Learn from your own latents and not from tokens: A sample-complexity theory
Latent prediction SSL recovers latent trees from PCFG data with sample complexity constant in hierarchy depth L (up to logs), unlike exponential for token-level or supervised methods.
-
From Item-Only to Query-Item: Query-Conditioned Generative Search with QGS in Quark
QGS introduces query-item pair encoding and query-conditioned prediction with a linear HSTU encoder and HFG-Attention to reduce noise from query switches in generative search ranking, reporting online gains in a commercial system.
-
Harmony in Diversity: Multi-domain Contrastive Policy Optimization for Large Reasoning Models
MCPO applies contrastive learning to GRPO-style RL by treating cross-domain correct rollouts as positives and incorrect ones as negatives to improve multi-domain reasoning performance in LRMs.
-
UWM-JEPA: Predictive World Models That Imagine in Belief Space
UWM-JEPA uses a density-matrix latent and unitary predictor in JEPA to preserve joint-state spectrum during blind rollouts, achieving 0.77 accuracy on a five-step hidden-velocity task versus 0.53 for an LSTM baseline.
-
Contrast to Detect: Dynamic Graph Contrastive Regularization for Unsupervised Anomaly Detection in Multivariate Time Series
ContrastAD achieves highest mean F1 on all five MTS benchmarks and highest AUC on three by building DTW-based sparse graph snapshots and contrasting divergent pairs with a stable anchor instead of enforcing invariance.