{"id":"3714ec9f-6800-4cab-85e7-64bccf36e1eb","arxiv_id":"2602.01456","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Matching JEPA features to a rectified generalized Gaussian target produces sparse, non-negative representations with accuracy close to dense baselines.","lead":"This paper introduces a regularizer that makes self-supervised image representations sparse and non-negative by matching them to a \"rectified\" bell-shaped distribution with a pile of probability at zero. On ImageNet-100 and CIFAR-100, the resulting Rectified LpJEPA keeps accuracy close to dense baselines while zeroing most feature entries.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"High-sparsity ℓ0 control is not established: finite-batch SWD plus plain ReLU can yield dead units, and Table 1's RGN2(−2.5) empirical ℓ0 is 3.6× the theoretical target; Fig. 3b lacks error bars and batch-size analysis.","rationale":"The mathematical core—RGG definitions, moment formulas, and Rényi dimension—is internally consistent and re-implementable; Proposition 3.5 and Theorem 3.6 are valid for the target distribution, not for the learned features. The load-bearing step is the unstated assumption that optimizing the finite-sample SWD loss makes learned features track the target's ℓ0, especially where the target's positive component is rare and the invariance term has comparable or larger scale. This is exactly the condition on which 'controllable sparsity' rests, and it is the least secure: no theorem, no error bars, and a visible discrepancy in Table 1. The proposed batch-size and dead-unit experiment would distinguish genuine target control from dead-ReLU collapse. Because the concern is addressable and the method may survive the check, I keep the reader's CONDITIONAL verdict but sharpen the condition under which acceptance would be justified.","tokens_in":52406,"tokens_out":15854,"duration_ms":157490,"concrete_test":"Run RGN2(−2.5,σGN) and RGN2(−3,σGN) on ImageNet-100 with batch sizes B=64, 128, 256, 512, keeping all other hyperparameters fixed and training to the usual plateau. Report: (i) normalized ℓ0 of projector features versus the theoretical E[ℓ0]/D from Eq. 8; (ii) fraction of dead ReLU units (pre-activation ≤0 for all samples in the batch) and the participation ratio (TrΣ)²/TrΣ² of the feature covariance; (iii) encoder and projector linear-probe accuracy. If normalized ℓ0 moves substantially toward the theoretical value as B grows, or if the participation ratio is O(1) while the dead-unit fraction is ≈1−E[ℓ0]/D, the high-sparsity tracking is an artifact of finite-batch SWD and dead ReLU units rather than target control. If ℓ0 is stable across B and the participation ratio remains O(D), the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that learned feature ℓ0 track E[ℓ0] of the RGG target (Prop. 3.5, Fig. 3b). This is not a consequence of the loss: RDMReg (Eq. 16) is a finite-sample, finite-projection sliced 2-Wasserstein loss and never directly penalizes ℓ0. At high sparsity (µ<0 with σ=σGN), the target has a large Dirac mass at zero and a rare positive tail; for RGN2(−3,σGN) the per-coordinate post-rectification variance is ≈2e−4 (Fig. 6d). The SWD gradient is then small relative to the invariance term, and because features use plain ReLU, any unit with negative pre-activation across a batch receives zero gradient and remains dead. Observed sparsity may therefore reflect dead-ReLU collapse rather than controlled matching of the RGG tail. The only direct evidence for tracking is Fig. 3b, which has no error bars, and Table 1 already shows RGN2(−2.5,σGN) with normalized ℓ0=0.0224 versus the Proposition 3.5 value 0.0062 (a 3.6× discrepancy). Thus the 'controllable sparsity at extreme sparsity' component of the central claim is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Rectified LpJEPA, a JEPA variant whose features are passed through ReLU and regularized by RDMReg, a sliced two-sample distribution-matching loss that aligns the feature distribution to a Rectified Generalized Gaussian (RGG) target. The RGG family is a mixture of a Dirac at zero and a truncated Generalized Gaussian, parameterized by (μ, σ, p). The authors derive closed-form moments, an expected ℓ0 formula, maximum-entropy characterizations for the truncated family, and a Rényi information-dimension characterization for the rectified family. They argue that RDMReg strictly generalizes LeJEPA, induces controllable sparsity, preserves task-relevant information, and achieves favorable sparsity–performance trade-offs. Empirical results on CIFAR-100 and ImageNet-100 compare Rectified LpJEPA against dense and sparse baselines using linear probing and transfer evaluations.","tokens_in":52796,"tokens_out":6034,"duration_ms":69635,"significance":"If the main claims hold, the paper makes a useful contribution to distribution-based collapse prevention in JEPAs by adding a principled sparsity and non-negativity axis to the existing Gaussian-matching framework. The theoretical appendices are a clear strength: Proposition B.9 gives closed-form moments, Proposition 3.3 gives a maximum-entropy derivation for truncated Generalized Gaussians, and Theorem 3.6 gives a careful Rényi information-dimension treatment of the rectified family. The claim that Rectified LpJEPA strictly generalizes LeJEPA is credible. The significance is currently limited by the empirical support: the central sparsity-transfer claim rests on single-seed runs, a partially circular figure, and a large discrepancy at extreme sparsity. With multi-seed results, error bars, and a sharper analysis of the high-sparsity regime, the paper would be a solid contribution.","major_comments":[{"comment":"The central 'controllable sparsity' claim is not established at high sparsity. The only direct evidence is Fig. 3b, which has no error bars and plots the empirical ℓ0 against the target value from Proposition 3.5. Because the training objective (Eq. 16) matches the feature distribution to the RGG target, this agreement is a check that the regularizer worked, not an independent prediction. More concretely, Table 1 reports RGN2(−2.5, σGN) with normalized ℓ0 = 0.0224, whereas Proposition 3.5 gives Φ(μ/σ) ≈ 0.0062, a 3.6× discrepancy. The visual closeness in Fig. 3b therefore overstates agreement at extreme sparsity. Please report per-seed error bars, a quantitative mismatch curve, and an analysis of dependence on batch size and number of projections.","section":"§5.4, Fig. 3b, Table 1"},{"comment":"There is no argument that the joint objective has non-degenerate optima with the target sparsity. For RGN2(−3, σGN), the per-coordinate variance after rectification is about 2×10⁻⁴ (Fig. 6d), while the RDMReg term uses B=128 samples and N=8192 projections and is not an ℓ0 penalty. In this regime, the sliced 2-Wasserstein gradient can be small relative to the invariance term, and because the features use plain ReLU, high measured sparsity could reflect dead units rather than controlled matching of the RGG tail. Section 4.5 only asserts 'σ > ε ... to prevent collapse'; no theorem or diagnostic supports non-collapse. Please add a collapse/dead-unit analysis (effective rank, dead-unit fraction, gradient norms of the two loss terms) and a batch-size/projection sweep in the high-sparsity regime.","section":"§4.5, Eqs. (14)–(16), Fig. 6d"},{"comment":"All ImageNet-100 pretraining results are reported from a single run (seed 5). The headline comparisons show small gaps — e.g., RGN2(0, σGN) encoder accuracy 85.08 vs. LeJEPA 84.80 and VICReg 84.18 — and no variance or significance is reported. This is insufficient support for the abstract's and Section 6's claims of 'favorable sparsity–performance trade-offs' and 'competitive downstream performance.' Please report means and standard deviations over at least 3–5 seeds for the main configurations, and state whether the differences are significant.","section":"§5.8, Table 1, Appendix L.8"},{"comment":"The wording in Section 3.5 — 'rectifications still preserve the maximal entropy property ... up to rescaling' — is not supported by a variational statement. Proposition 3.3 maximizes differential entropy over continuous densities on a support S under an E∥x∥_p^p constraint. Theorem 3.6 only computes the Rényi information dimension and d(ξ)-dimensional entropy of the RGG family; it does not prove that RGG maximizes any entropy among distributions with a given expected ℓ0 or a given Dirac-mixing probability. To make the claimed maximum-entropy property precise, the authors should either state and prove a maximization result over a well-defined class of mixtures, or temper the language.","section":"§3.5, Theorem 3.6"}],"minor_comments":[{"comment":"The objective is written without weights in Eqs. (14)–(15), but Appendix L.5 reports λ_sim = 25.0 and λ_dist = 125.0. The exact weighted objective should appear in the main text for reproducibility.","section":"§4.2, Appendix L.5"},{"comment":"The notation 'z+', 'p z+', and 'p z' in the figure and text is not defined before use. Please define z+ := ReLU(z) and the target-distribution shorthand in the caption or in Section 4.","section":"Fig. 3a, §5.2"},{"comment":"The bisection search assumes initial bounds σ_L and σ_U with f(σ_L) < 0 and f(σ_U) > 0, but the existence and construction of such bounds are not discussed. A brief justification or a practical initialization would help.","section":"Algorithm 2"},{"comment":"There is a typo: 'Genealized' should be 'Generalized' in the section heading.","section":"Appendix D.1"},{"comment":"The figure legend says results span ResNet, ViT, and ConvNeXt, but no per-backbone error bars or per-backbone breakdown is shown. It would be clearer to plot each backbone separately or report across-backbone variance.","section":"§5.4, Fig. 3b"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competent extension of LeJEPA with careful probability-theory appendices. The main concern is not the derivations but the empirical support for the headline sparsity-transfer claim: single-seed runs, no error bars, and a 3.6× discrepancy at one of the advertised sparse settings. I do not see a novelty or disclosure problem, and the scope fits the journal. If the authors add multi-seed results, address the dead-ReLU/collapse concern with diagnostics, and sharpen the maximum-entropy wording, I would be willing to support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the RGG target family plus the sliced two-sample matching loss, and the probability work is the strongest part: the moments, the max-entropy derivation for truncated generalized Gaussians, and the Rényi-dimension entropy all look careful and correct, and the paper is honest that p=2 recovers the known rectified Gaussian and that LeJEPA is a special case. If the empirical claims hold up, this gives JEPA users a practical dial for sparsity, which is a useful tool even if not a field reshaper. I believe the method is re-implementable from the text.\n\nThe soft spots are real but not fatal. Appendix L.8 says all ImageNet-100 runs are single-seed, so the key comparisons in Table 1 and the sparsity–accuracy trade-off curves have no error bars. More specifically, the empirical ℓ0 tracking in Fig. 3b is presented as the central validation of 'controllable sparsity,' but Table 1 already shows RGN₂(−2.5) with normalized ℓ0 of 0.0224 versus the Proposition 3.5 value of 0.0062 — a 3.6× discrepancy — and Fig. 3b has no uncertainty. The stress-test concern about dead ReLU units at high sparsity is plausible: the loss is a finite-sample sliced Wasserstein distance, not a direct ℓ0 penalty, and with µ very negative the target variance is tiny and the invariance term can dominate. The authors should rule this out with batch-size ablations or gradient statistics; right now the tracking claim is weaker than the abstract implies. Also, the max-entropy theorem is about the RGG target distribution, not about the learned representation, so 'prevents collapse' overstates what is proven; the paper's own Section 4.5 only asserts σ > ε as a heuristic.\n\nThe other issues are more minor. The Fig. 3c Pareto plot only shows their own points, so the 'favorable trade-off' framing would be stronger with baseline points on the same axes. And the circuitousness point is fair: aligning features to an RGG and then showing features match the RGG's expected ℓ0 confirms the regularizer works, but it is not an independent prediction. None of this sinks the paper. The math is careful, the citations are honest, and the limitations are mostly acknowledged if you read the appendices.\n\nI would send this to peer review. It deserves referee time. The requested revisions should be straightforward: multi-seed runs with error bars, a direct check for dead ReLU collapse at high sparsity, and either softening the collapse-prevention claim or proving something about the joint objective.","headline":"Useful regularizer with careful math, but the headline sparsity-tracking claim rests on single-seed runs and a few loose ends; deserves a thorough referee.","tokens_in":53282,"tokens_out":2018,"would_cite":true,"duration_ms":25745,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","62B10","94A17"],"pacs":[],"model":"deepseek-v4-flash","headline":"Rectified LpJEPA matches JEPA representations to a Rectified Generalized Gaussian target, giving users direct control over the expected fraction of zero entries while preserving maximum-entropy structure and, empirically, downstream accurac","keywords":["self-supervised learning","joint-embedding predictive architectures","sparse representations","distribution matching","rectified generalized Gaussian","maximum entropy","collapse prevention","ℓ0 sparsity"],"falsifier":"Train Rectified LpJEPA at µ=−3, p=2 on a standard benchmark across several seeds and measure per-coordinate variance and the fraction of nonzero entries: if the empirical ℓ0 norm departs systematically from d·Φ(µ/σ), or if the representation collapses (variance→0) while the RDMReg loss is near zero, the sparsity-transfer claim fails.","tokens_in":52296,"feed_emoji":"🎛️","tokens_out":6507,"duration_ms":68803,"temperature":0.7,"pith_summary":"Rectified LpJEPA claims that the right way to make self-supervised representations sparse is to change the distribution you regularize toward, not to add a sparsity penalty. The paper defines a rectified generalized Gaussian target — a point mass at zero plus a truncated generalized Gaussian — whose expected fraction of zero entries is fixed by three dials (µ, σ, p), and matches the learned features to it with a sliced two-sample loss. If the claim is right, a user can choose a sparsity level up front, and the encoder will produce non-negative features with about that many zeros while keeping enough task-relevant information to remain competitive on image classification until roughly 95% of entries are zero. The construction also preserves a maximum-entropy guarantee in Rényi information dimension, so the sparse code is not just empty — it is the least-committal code given the sparsity budget.","feed_headline":"Sparse features on demand: JEPA keeps accuracy until 95% zeros","feed_subtitle":"A rectified target distribution lets users pick the fraction of zero features, with accuracy intact until extreme sparsity.","key_machinery":"Rectified Distribution Matching Regularization (RDMReg): a sliced two-sample loss, instantiated as sliced 2-Wasserstein distance, that matches one-dimensional projections of rectified features to projections of samples from a Rectified Generalized Gaussian (RGG) distribution. RGG is a mixture of a point mass at zero and a truncated generalized Gaussian on (0,∞), with parameters (µ, σ, p); its expected ℓ0 norm is d·Φ(µ/σ), so the target's sparsity is chosen in advance. Because RGG is not closed under linear projections, the Cramér–Wold device cannot reduce the match to a parametric family per projection, which is why the loss is nonparametric and two-sample.","core_discovery":"Central claim: sparsity and non-negativity can be imposed on JEPA representations by changing the target distribution rather than adding explicit ℓ0 or ℓ1 penalties. The paper introduces the Rectified Generalized Gaussian (RGG)—a Dirac mass at zero mixed with a truncated generalized Gaussian on the positive reals—and proves it is maximum-entropy under an expected ℓp-norm constraint on the positive orthant. Rectified LpJEPA matches projected feature marginals to RGG samples through sliced two-sample distribution matching (RDMReg). The paper argues this strictly generalizes Gaussian-based JEPA regularization and reports that learned features track the target's expected ℓ0 norm, with accuracy d","pith_inferences":["Because the RGG family is not closed under linear projections, the method's reliance on nonparametric two-sample matching suggests that other sliced divergences (e.g., energy-based or kernel-based) could substitute for the sliced 2-Wasserstein loss; the paper's theory does not single out one metric.","The maximum-entropy derivation extends beyond the exponential family: any truncated exponential-family density on the positive reals mixed with a Dirac mass at zero would give controllable ℓ0 sparsity with an analogous entropy guarantee, so the rectified generalized Gaussian is one member of a broader design space.","At extreme sparsity the paper's own qualitative analysis shows retrieval flips between competing scene factors rather than losing information; a testable consequence is that sparsity acts as a feature-selection prior, and combining it with explicit decorrelation at tiny µ might preserve accuracy further.","The fixed-σGN choice makes target per-coordinate variance shrink sharply as µ becomes very negative; an editor's reading is that a variance-normalized target or a feature-whitening step could push the Pareto frontier past roughly 95% sparsity, but the paper does not claim this."],"forward_implications":["Sparsity becomes a selectable property: choosing (µ, σ, p) for the target RGG sets the expected fraction of nonzero output entries, and the learned features track that fraction across backbones.","Gaussian-based JEPA regularization is recovered as the dense edge case (p=2, no rectification), so the method is a strict generalization rather than an alternative.","Sparse, non-negative features can be obtained without negative pairs, reconstruction, or explicit ℓ0 penalties, retaining competitive linear-probe accuracy until roughly 95% of entries are zero.","Matching to RGG implicitly drives the feature covariance toward a scaled identity, so second-order collapse is controlled with a linear number of projections.","Representation sparsity varies systematically across downstream datasets after one pretraining, providing a potential signal for out-of-distribution detection."],"fun_headline_variants":["Sparse JEPA without penalties: pick your zero fraction","Target distribution shapes sparsity in JEPA","No penalties needed: JEPA matches rectified Gaussian","JEPA goes sparse: choose zero ratio, keep accuracy"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The empirical claim that features track the target's expected ℓ0 norm at extreme sparsity assumes the RDMReg term continues to dominate the invariance term even when the target's per-coordinate variance is tiny (about 0.0002 at µ=−3), a regime where no theorem guarantees non-collapsed optima.","fun_headline_variants_meta":{"raw":{"variants":["Sparse JEPA without penalties: pick your zero fraction","Target distribution shapes sparsity in JEPA","No penalties needed: JEPA matches rectified Gaussian","JEPA goes sparse: choose zero ratio, keep accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000149,"raw_usage":{"total_tokens":1016,"prompt_tokens":720,"completion_tokens":296,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":233}},"tokens_in":464,"tokens_out":296,"duration_ms":3539,"temperature":1.0,"reasoning_tokens":233,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:39:03.311241+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train Rectified LpJEPA at µ=−3, p=2 on a standard benchmark across several seeds and measure per-coordinate variance and the fraction of nonzero entries: if the empirical ℓ0 norm departs systematically from d·Φ(µ/σ), or if the representation collapses (variance→0) while the RDMReg loss is near zero, the sparsity-transfer claim fails.","supporting_citations":[],"review_version":1}