REVIEW 30 cited by
Contrastive Representation Distillation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Contrastive Representation Distillation
read the original abstract
Often we wish to transfer representational knowledge from one neural network to another. Examples include distilling a large network into a smaller one, transferring knowledge from one sensory modality to a second, or ensembling a collection of models into a single estimator. Knowledge distillation, the standard approach to these problems, minimizes the KL divergence between the probabilistic outputs of a teacher and student network. We demonstrate that this objective ignores important structural knowledge of the teacher network. This motivates an alternative objective by which we train a student to capture significantly more information in the teacher's representation of the data. We formulate this objective as contrastive learning. Experiments demonstrate that our resulting new objective outperforms knowledge distillation and other cutting-edge distillers on a variety of knowledge transfer tasks, including single model compression, ensemble distillation, and cross-modal transfer. Our method sets a new state-of-the-art in many transfer tasks, and sometimes even outperforms the teacher network when combined with knowledge distillation. Code: http://github.com/HobbitLong/RepDistiller.
Forward citations
Cited by 30 Pith papers
-
Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge
LaViD distills LLM conceptual knowledge to vision models via LLM-generated MCQ soft labels, outperforming vision-language distillation baselines on fine-grained benchmarks while improving robustness on spurious correl...
-
CoDMD: Copula-aware Distribution Matching Distillation for Fast Video Generation
CoDMD adds a copula-matching regularizer to DMD for distilling 50-step video diffusion models to 4 steps, reporting VBench scores of 84.46/84.87 on 1.3B/14B Wan-2.1-T2V models.
-
Toward Calibrated, Fair, and accurate Deepfake Detection
Face-Feature Tuning is a label-free logit remapping method that reduces FPR/TPR gaps across groups in deepfake detection while preserving overall accuracy.
-
On the Generalization of Knowledge Distillation: An Information-Theoretic View
Knowledge distillation generalization bounds are derived via a new distillation divergence measuring teacher-student kernel difference, with tighter bounds from teacher loss flatness.
-
On the Generalization of Knowledge Distillation: An Information-Theoretic View
Derives upper and lower generalization bounds for the student relative to the teacher using a new distillation divergence, plus a loss-sharpness-aware bound and a bias-variance-rank decomposition in the linear Gaussian case.
-
Training a Student Expert via Semi-Supervised Foundation Model Distillation
A semi-supervised framework distills vision foundation models into compact instance segmentation experts that outperform their teachers by up to 11.9 AP on Cityscapes and 8.6 AP on ADE20K while being 11 times smaller.
-
VLM-Aware Meta-Optic Front-End Design for Frozen Vision-Language Models
Optimizing a constrained meta-optic density for frozen CLIP cross-entropy, not focusing, raises ImageNet-100 zero-shot accuracy by about 12 points over a focus baseline and transfers across models and datasets.
-
Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations
Student-conditioned cyclic-shift perturbations produce diverse single-teacher distillation views in one stage, matching two-stage methods with far fewer parameters.
-
VLM-Aware Meta-Optic Front-End Design for Frozen Vision-Language Models
CODA optimizes continuous-density meta-optics via adjoint gradients on Maxwell simulations to boost frozen CLIP zero-shot accuracy on ImageNet-100 from 53.75% to 65.41%, with transfer to other models.
-
Moebius: 0.2B Lightweight Image Inpainting Framework with 10B-Level Performance
Moebius introduces a compressed diffusion inpainting model using Local-λ Mix Interaction blocks and latent-space multi-granularity distillation to reach 10B-level quality with 0.22B parameters.
-
Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning
Hidden-Align adds an auxiliary loss to align hidden states of correct reasoning paths at the pre-answer token in RLVR, improving pass@1 by 3.8-6.2 points over DAPO on eight math benchmarks for Qwen3 models of 1.7B-14B scale.
-
Lightweight SAR Ship Detection via Contrastive Distillation
Proposes SURGE, a contrastive relational distillation method for lightweight SAR ship detection that reports up to 6.2 mAP gains on SSDD and HRSID benchmarks while claiming to be the first transformer-based KD framewo...
-
AURORA: Contextual Orthogonalization for Geometric Representation Learning in Healthcare Foundation Models
AURORA is a representation learning framework that uses contextual orthogonalization and relational alignment to create disentangled, geometrically interpretable latent spaces in healthcare foundation models.
-
STARS: Spike Tail-Aware Relational Synthesis for ANN-to-SNN Data-Free Knowledge Distillation
STARS augments BN-guided data synthesis with relational consistency alignment and tail-aware regularization for ANN-to-SNN data-free KD, reporting gains up to 4.6% on CIFAR-10 and 6.7% on CIFAR-100.
-
Event Fields: Learning Latent Event Structure for Waveform Foundation Models
Event-centric waveform foundation models are learned via self-supervised consistency on latent event structures and interactions, yielding improved performance and label efficiency over sequence-based baselines on phy...
-
Displacement Preserving Relational Distillation for Robust Medical Segmentation
ROI-masked pairwise displacement alignment lets a tiny nnU-Net student match or exceed MedNeXt Dice and HD95 on AMOS and ISLES with ~5% parameters.
-
Rationale-Guided Knowledge Distillation for Cross-Lingual Stance Detection
Distilling LLM-generated reasoning rationales into mBERT via dual-path contrastive distillation improves cross-lingual stance detection by 1–3% accuracy on three benchmarks.
-
Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations
SAKD generates multiple virtual teacher views from a single student-conditioned perturbation expanded by parameter-free cyclic shifts, beating random-noise augmentation and matching two-stage KD methods.
-
MorphologyFM: A Foundation Model for Morphology-Aware Representation Learning from ECG and Pulse Oximetry Waveforms
Morphology-aware masking plus cross-modal ECG–SpO2 pretraining on MIMIC yields stronger transfer than MAE, contrastive, Barlow Twins, and JEPA on several clinical prediction tasks.
-
SFKD: Spatial--Frequency Joint-Aware Heterogeneous Knowledge Distillation via Multi-Level Wavelet Spectral Interaction
SFKD uses multi-level discrete wavelet transform plus dual-stream refinement and Gaussian-filtered frequency loss to transfer spatial and global information across heterogeneous models.
-
Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions
Mixup applied only to the student during KD induces independent linearity acquisition that reduces overconfidence by an order of magnitude while improving accuracy, with calibration transferring separately from accuracy.
-
WISTERIA: Learning Clinical Representations from Noisy Supervision via Multi-View Consistency in Electronic Health Records
WISTERIA learns robust clinical representations from noisy EHR labels by enforcing consistency across multiple weak supervision views plus ontology regularization.
-
Deep Reprogramming Distillation for Medical Foundation Models
DRD introduces a reprogramming module and CKA-based distillation to enable efficient, robust adaptation of medical foundation models to downstream 2D/3D classification and segmentation tasks, outperforming prior PEFT ...
-
Fast Text-to-Audio Generation with One-Step Sampling via Energy-Scoring and Auxiliary Contextual Representation Distillation
A one-step text-to-audio model using energy-distance training and contextual distillation outperforms prior fast baselines on AudioCaps and achieves up to 8.5x faster inference than the multi-step IMPACT system with c...
-
Benchmarking Federated Learning and Knowledge Distillation for Point Cloud Classification
Benchmark of federated learning plus knowledge distillation for point cloud classification reveals that label-free distillation objectives are required for student accuracy to reflect federated teacher quality rather ...
-
Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression
The prune-quantize-distill ordering produces a better accuracy-size-latency frontier on CIFAR-10/100 than any single technique or other orderings, with INT8 QAT providing the main runtime gain.
-
Exploring Vision Neural Network Pruning via Screening Methodology
A unified F-statistic screening and weighted evaluation method prunes both unstructured and structured parameters in FNNs and CNNs, claiming order-of-magnitude size reduction with competitive accuracy on vision datasets.
-
Streaming Chain
Prune then INT8 QAT then KD yields a better measured accuracy–size–latency trade-off on CIFAR CNNs than any single stage, and stage order itself changes accuracy.
-
Logit Distillation on Manifolds: Mapping by Learning
Presents a layer- and point-wise projection mapping for manifold-based logit distillation combined with LoRA to enable low-parameter student training with reported WER gains.
-
Platonic Projection Structures: Operator-Induced Observability in Representation Learning
The paper introduces 'Platonic Projection Structures,' a reformulation of standard PSD operator theory applied to representation learning, with experiments that verify definitions rather than test predictions.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.