REVIEW 17 cited by
A-JEPA: Joint-Embedding Predictive Architecture Can Listen
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
A-JEPA: Joint-Embedding Predictive Architecture Can Listen
read the original abstract
This paper presents that the masked-modeling principle driving the success of large foundational vision models can be effectively applied to audio by making predictions in a latent space. We introduce Audio-based Joint-Embedding Predictive Architecture (A-JEPA), a simple extension method for self-supervised learning from the audio spectrum. Following the design of I-JEPA, our A-JEPA encodes visible audio spectrogram patches with a curriculum masking strategy via context encoder, and predicts the representations of regions sampled at well-designed locations. The target representations of those regions are extracted by the exponential moving average of context encoder, \emph{i.e.}, target encoder, on the whole spectrogram. We find it beneficial to transfer random block masking into time-frequency aware masking in a curriculum manner, considering the complexity of highly correlated in local time and frequency in audio spectrograms. To enhance contextual semantic understanding and robustness, we fine-tune the encoder with a regularized masking on target datasets, instead of input dropping or zero. Empirically, when built with Vision Transformers structure, we find A-JEPA to be highly scalable and sets new state-of-the-art performance on multiple audio and speech classification tasks, outperforming other recent models that use externally supervised pre-training.
Forward citations
Cited by 17 Pith papers
-
PairAlign: A Framework for Sequence Tokenization via Self-Alignment with Applications to Audio Tokenization
PairAlign learns compact audio token sequences via self-alignment of paired content views using an autoregressive decoder, achieving strong cross-view consistency and edit-distance preservation while reducing token co...
-
PairAlign: A Framework for Sequence Tokenization via Self-Alignment with Applications to Audio Tokenization
PairAlign learns compact variable-length token sequences for audio via self-alignment on paired content-preserving views, achieving 55% fewer archive tokens than VQ while preserving edit-distance retrieval at 12.71 tokens/s.
-
Recurrent Video Masked Autoencoders
RVM uses recurrent computation inside a masked autoencoder to learn video representations that match or exceed prior video and image models on classification, tracking, and dense spatial tasks with up to 30x better pa...
-
The JEPA Paradox in Language: The Geometry of Linguistic Alternatives
Pure squared-error JEPA on masked text saturates mutual information and keeps high conditional variance, then collapses in rank and cosine similarity and transfers poorly, unlike matched I-JEPA on images.
-
Music-JEPA: Learning a World Model of Sound from Action
An action-conditioned JEPA trained on paired piano audio and MIDI learns latent sound dynamics that support MIR tasks and transcription-style planning.
-
Joint-Embedding Predictive Architecture for Sensor-based Activity Recognition
JEPA-based self-supervised pre-training on inertial sensor data improves recognition of rare transitional human activities compared to supervised learning, with gains mostly on transition classes.
-
Frequency-Aware Self-Supervised Music Representation Learning
PupuJEPA applies a visual JEPA framework to 2D spectrograms with music-specific adaptations and outperforms 1D SSL models on the MARBLE benchmark for multiple MIR tasks.
-
Frequency-Aware Self-Supervised Music Representation Learning
PupuJEPA applies a 2D joint-embedding predictive architecture to music spectrograms and outperforms 1D SSL models on the MARBLE benchmark.
-
MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning
MJ EPA applies a single shared ViT encoder and one predictive objective within and across audio-visual modalities, reporting >6.8 mAP gains on AudioSet-20K and competitive video results with 10x less data.
-
AudioMosaic: Contrastive Masked Audio Representation Learning
AudioMosaic learns general-purpose audio representations through contrastive pre-training with structured spectrogram masking, reaching state-of-the-art results on standard benchmarks and improving audio-language tasks.
-
AeroJEPA: Learning Semantic Latent Representations for Scalable 3D Aerodynamic Field Modeling
AeroJEPA applies joint-embedding predictive learning to produce scalable, semantically organized latent representations for 3D aerodynamic fields that support both field reconstruction and downstream design tasks.
-
Self-Distillation of Hidden Layers for Self-Supervised Representation Learning
Predicting the outputs of several hidden layers of an EMA teacher, rather than only the final layer or pixels, substantially improves self-supervised ViT representations on ImageNet and downstream tasks.
-
Joint-Embedding Predictive Architecture for Solar PV Panel Fault Classification
JEFFNet fuses StoP-JEPA semantic embeddings with EfficientNetV2-S features for thermal IR PV fault classification, beating GEPFNet on F1 for multiclass and binary tasks with 47% fewer parameters.
-
Scalable and Efficient Joint Spiking Embedding Predictive Architecture for Large-Scale Dynamic Graphs
SG-JEPA applies joint-embedding predictive learning to dynamic graphs, using spiking-neuron context encoders to predict future node embeddings without edge reconstruction or graph augmentation.
-
AGE: Adaptive-masking for Graph Embedding in Graph Retrieval-Augmented Generation
AGE applies adaptive masking via a learnable sampler in Transformer-based SSL to align graph and text embeddings, yielding higher accuracy on four GraphQA benchmarks for non-parametric GraphRAG.
-
Stylistic-STORM (ST-STORM) : Perceiving the Semantic Nature of Appearance
ST-STORM introduces a dual-branch SSL framework that disentangles semantic content from stylistic appearance using gated latent streams, JEPA for content invariance, and adversarial constraints for style capture.
-
World Action Models: A Survey
A survey that clarifies boundaries and organizes World Action Models by generation requirements and predictive substrates, identifying a trend toward generating less of the future.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.