{"id":"c4ac8267-2ea9-4a09-90f8-1bde626860c2","arxiv_id":"2504.16667","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The MINC loss converts the Spectral Contrastive loss into a non-contrastive form via power iteration and a Hebbian orthogonality update, preventing collapse and improving ImageNet representation accuracy over the contrastive baseline.","lead":"This paper introduces MINC, a new training objective that turns a contrastive self-supervised learning loss into a non-contrastive one, removing the need to compare every image with every other image. On ImageNet, MINC consistently improves over the Spectral Contrastive loss baseline and matches BYOL with a linear predictor, though not the original BYOL.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Figure 3 ablation shows the MINC objective collapses without the GHA lower-triangular term, contradicting the abstract's claim that the mutual-information formulation prevents collapse; the anti-collapse mechanism is an unproven GHA heuristic.","rationale":"Agreed with the reader's weakest-assumption location. The most load-bearing issue is that the non-collapse guarantee advertised in the abstract is contradicted by the paper's own Figure 3, and the mechanism invoked to fix it (GHA) is used outside its proven regime. This does not make the empirical result worthless: the method does learn on ImageNet, and the power-iteration / BYOL connection is a useful framing. But the conceptual claim that one can 'keep the mutual information formulation preventing collapse' is not supported as stated, and the main empirical comparison rests on single-seed runs with no error bars. These are addressable via re-ablation and multi-seed evaluation, so CONDITIONAL remains the right verdict; no change to the reader's judgment.","tokens_in":13427,"tokens_out":10083,"duration_ms":97287,"concrete_test":"At batch size 1408, run the Figure 3 'None' condition with the target network included but GHA removed (use full Lambda, not LT[Lambda], in Eq. (21)), and monitor the effective rank and variance of the embedding Gram matrix across training. If the representation collapses or degrades, the anti-collapse behavior is due to GHA rather than the MI objective. As a complementary check, replace GHA with exact QR orthogonalization of psi after each power-iteration step; if that method fails while the GHA version succeeds, the empirical success is tied to the specific unproven Sanger update, further weakening the MI-based rationale.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Sec. 3 claim that MINC 'retains the non-collapse guarantees enjoyed by f-MI contrastive losses.' The derivation replaces the exact orthogonalization in power-iteration step (10c) with the Generalized Hebbian Algorithm (Sanger, 1989) via the lower-triangular LT[Lambda] term in Eq. (14). GHA's asymptotic-orthogonality result is established for a single-layer linear network with stationary input; here it is applied to a deep non-linear ResNet trained jointly with an EMA Lambda, and the paper supplies no proof or analysis for this regime. The load-bearing problem is not just the missing proof: Figure 3 shows that a variant with the same MI objective and auxiliary EMA but without GHA 'essentially collapses to a constant,' and the text states 'GHA is successfully preventing the collapse.' Thus, empirically, the non-collapse property is supplied by the GHA modification, not by the mutual-information formulation. Since the paper's central contribution is precisely that the MI formulation can prevent collapse while removing pairwise comparisons, this internal evidence directly undermines the central claim. The single-seed, roughly 0.01-margin results in Table 1 do not independently test the mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MINC, a non-contrastive reformulation of the Spectral Contrastive loss, obtained by introducing an auxiliary covariance matrix Λ and a power-iteration viewpoint, and by replacing the exact orthogonalization step with a Generalized Hebbian Algorithm (GHA) update. The resulting objective avoids pairwise comparisons, uses an EMA updated Λ, and optionally a target network. The authors evaluate MINC on ImageNet with ResNet-50 and ResNet-50 2x at batch sizes 512, 1408, and 4096, reporting consistent top-1 accuracy improvements over a re-run Spectral Contrastive baseline, and they ablate the choice of α-divergence, the GHA transformation, and the auxiliary EMA. The paper also draws connections to BYOL with a linear predictor.","tokens_in":13690,"tokens_out":10124,"duration_ms":91521,"significance":"If the empirical improvement over Spectral Contrastive is confirmed, the power-iteration perspective is a useful conceptual bridge between spectral contrastive methods and non-contrastive objectives such as Linear BYOL, and the ablation study provides a clear decomposition of the algorithmic components. However, the paper's central stated claim that the mutual-information formulation itself prevents collapse is directly contradicted by its own Figure 3, where the objective collapses without the GHA transformation; the non-collapse property is empirically supplied by a heuristic whose theoretical validity is asserted only by analogy to single-layer linear networks. The main empirical claim also rests entirely on single-seed runs with margins near 1 percentage point. The paper is valuable as a heuristic recipe and a conceptual framework, but as written it does not establish its headline claims.","major_comments":[{"comment":"The abstract and §1 state that MINC 'keeps the mutual information formulation of the contrastive method preventing collapse', and §3 says it 'retains the non-collapse guarantees enjoyed by f-MI contrastive losses'. This is directly contradicted by the ablation in §4.2 (Figure 3): with the same MINC objective and auxiliary EMA=0.8 but without the GHA lower-triangular transformation, the representation 'essentially collapses to a constant', and the text states 'GHA is successfully preventing the collapse'. The non-collapse property is therefore supplied by the GHA modification, not by the mutual-information formulation, and no theorem establishes a non-collapse guarantee for the proposed non-contrastive objective. This is a central claim of the paper and must be corrected or supported by analysis before publication.","section":"Abstract, §1, §3, §4.2"},{"comment":"The substitution of the Generalized Hebbian Algorithm for exact orthogonalization in (10c) is load-bearing, because the GHA update is the only mechanism that prevents collapse in Figure 3. Sanger's asymptotic-orthogonality result is established for a single-layer linear network with stationary input; here it is applied to a deep non-linear ResNet trained jointly with an EMA auxiliary matrix and a target network. No convergence or approximation analysis is provided for this regime, and the derivation from (10a)-(10c) to (14) is therefore heuristic. Please either provide a formal statement for a tractable setting (e.g., a linear embedding with whitened input) or present substantially more empirical evidence, such as eigenvalue diagnostics of the learned embedding covariance, that the asymptotic orthogonality actually holds in the deep non-linear setting.","section":"§3.3, Eq. (14)"},{"comment":"The central empirical claim is that MINC 'consistently outperforms the Spectral Contrastive baseline across all batch sizes and for both network sizes' (Table 1), but all results are from a single seed. The margins are roughly 0.9-1.1 percentage points on ImageNet, which can be within run-to-run variation for 300-epoch self-supervised training. Please report multiple seeds with standard deviations or confidence intervals (at least for the smallest batch size and for the 2x network), or otherwise demonstrate that the improvements are not noise.","section":"§4.1, Table 1"},{"comment":"The statement that (12) and (14) form an 'asymptotic non-contrastive approximation of gradient descent on the Spectral Contrastive loss' is not made rigorous: there is no bound on the approximation error introduced by replacing E[φφ^T] with LT[Λ_{t+1}], by the EMA, or by the target network, and no convergence result for the resulting coupled update. If this equivalence is intended as a theoretical contribution, it needs a precise statement and proof; otherwise the text should be weakened to a heuristic motivation.","section":"§3.3, after Eq. (14)"}],"minor_comments":[{"comment":"The text 'raining becomes noticeably unstable' should read 'training becomes noticeably unstable'.","section":"§4.2"},{"comment":"The word 'eigen-deomposition' is a typo and should read 'eigen-decomposition'.","section":"§3.3"},{"comment":"The displayed gradient of the Spectral Contrastive loss has both terms with a minus sign; for the maximization objective in (3) the gradient should be 2(E[φ^T∇φφ'] - E[(φ^Tφ')φ^T∇φφ']). Please check the sign convention (gradient ascent vs. descent) or clarify whether the 'loss' is the negative objective.","section":"§3.3"},{"comment":"The paper says representations are trained on both the train and validation splits and evaluated on the ImageNet test split; please clarify the exact data-split protocol, since public ImageNet test labels are not generally available.","section":"§4.1"},{"comment":"The introduction of the target network in (21)-(22) is stated without derivation; please explain how the target-network modification is compatible with the power-iteration derivation from (10a)-(14), or justify it as a stability heuristic analogous to BYOL.","section":"§3.5, Eq. (21)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's own Figure 3 is the strongest piece of evidence against the abstract's central claim, and I would ask the authors to reframe the contribution or add a rigorous analysis of the GHA-based anti-collapse mechanism. The paper is closely related to the authors' prior work (SpecCon, f-MICL), but the empirical comparison is against a re-run baseline, so I do not see a novelty disclosure problem. The main concern is whether the headline claims can be supported within the scope of a revision; if the authors can provide multi-seed evidence and a corrected statement of the non-collapse mechanism, I would be willing to reconsider."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuine, useful piece of work. The MINC conversion is new, the empirical comparison against a re-run SpecCon baseline is honest, and the gains, while small, are consistent. But the paper's headline claim about collapse prevention is overstated, and the paper's own Figure 3 is the evidence: without the GHA term, the objective collapses to a constant. The MI formulation does not prevent collapse; the GHA lower-triangular trick does. That is a load-bearing qualification, not a nitpick.\n\nWhat's new: reformulating Spectral Contrastive via power iteration, replacing exact orthogonalization with GHA, and the connection to linear BYOL. The derivation is readable and the practical recipe is concrete. The ablations over alpha, EMA, and the target network are exactly what you want to see. Credit where due: the ImageNet results use a re-run SpecCon baseline, not just published numbers, and MINC does beat that baseline at all three batch sizes and both widths. That is a real measurement.\n\nSoft spots, in proportion. First: single-seed runs. The margins are roughly +0.01 top-1 accuracy with no error bars, and that might not survive re-seeding, especially at batch size 512 where the gap is smallest. The authors admit the cost; still, for a paper whose central empirical claim rests on those margins, I'd want at least two or three seeds or a clear statement of variance. Second: the GHA step is applied to a deep nonlinear network with nonstationary inputs—Sanger's analysis does not cover that regime, and the paper supplies no alternative analysis. That would be fine as an empirical heuristic if it were framed that way, and in the ablation it basically is. But the abstract and Section 3 claim non-collapse guarantees from the MI formulation. Those guarantees are not delivered; the anti-collapse mechanism is the added GHA term. Third: no code release. Given the tricks involved (EMA beta, lower-triangular masking, target EMA), reproducibility would be a lot easier with code.\n\nBottom line: the paper deserves a serious referee. It is a clear, honest contribution to the contrastive-to-non-contrastive conversion story, with a fresh theoretical connection. I would recommend acceptance conditional on fixing the collapse-prevention claim (rewrite the abstract and theory section to attribute collapse avoidance to GHA), adding at least a variance check or more seeds, and releasing code. I would cite it if I worked on non-contrastive SSL.","headline":"Genuine and mostly honest conversion of SpecCon to a non-contrastive loss, but the anti-collapse story is carried by a GHA heuristic, not the MI objective, and the empirical margins rest on single-seed runs.","tokens_in":14243,"tokens_out":1929,"would_cite":true,"duration_ms":17691,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Non-contrastive MINC beats its contrastive source on ImageNet","keywords":["self-supervised learning","representation learning","non-contrastive loss","spectral contrastive loss","mutual information","power iteration","Generalized Hebbian Algorithm","ImageNet"],"falsifier":"Run MINC and the Spectral Contrastive baseline with the same code and hyperparameters but multiple seeds (the paper reports one seed per setting in Table 1) and, separately, replace the $\\operatorname{LT}[\\Lambda]$ update with exact batch orthogonalization; if the multi-seed gaps shrink to zero or the exact-orthogonalization run trains as well as the GHA version, both the stability of the claimed improvement and the paper's explanation of why GHA matters are in question.","tokens_in":13201,"feed_emoji":"🖼️","tokens_out":10970,"duration_ms":90505,"temperature":0.7,"pith_summary":"The paper claims that the Spectral Contrastive loss, a contrastive objective that compares every embedding with every other and so needs large batches, can be rewritten as a non-contrastive objective, MINC, that keeps the mutual-information core and its non-collapse property while dropping pairwise comparisons. The rewriting interprets the loss as an eigen-decomposition and runs power iteration, replacing exact orthogonalization with a Generalized Hebbian Algorithm update and summarizing all other points by a moving-average matrix $\\Lambda$. On ImageNet, MINC improves top-1 accuracy over the Spectral Contrastive baseline by roughly one point at every tested batch size (512, 1408, 4096) and for both ResNet-50 and a doubled-width ResNet-50. Because a non-contrastive objective removes the quadratic pairwise cost, this is a concrete recipe for getting contrastive-style non-collapse at non-contrastive variance and memory cost.","feed_headline":"Non-contrastive MINC beats its contrastive source on ImageNet","feed_subtitle":"Rewriting spectral contrastive loss as power iteration lets MINC train with smaller batches and no collapse.","key_machinery":"The load-bearing mechanism is a power-iteration reformulation of the Spectral Contrastive loss. In the eigen-decomposition view, the optimal embedding satisfies a fixed-point equation, and the embedding covariance at the solution is a diagonal eigenvalue matrix $\\Lambda$. Power iteration alternates an update of $\\Lambda$, approximated as an exponential moving average, with a gradient update of the embedding that replaces the full covariance $\\Lambda_{t+1}$ by its lower-triangular part $\\operatorname{LT}[\\Lambda_{t+1}]$. That lower-triangular substitution is the Generalized Hebbian Algorithm's asymptotic orthogonalization, which removes the need for explicit Gram-Schmidt and is what keeps the representation from collapsing; the paper adds a slow target network for stability and an inner scale $s$ so that the normalized embeddings are compared as scaled cosines.","core_discovery":"The paper's central claim is that the quadratic pairwise dependence of the Spectral Contrastive loss is not essential to its mutual-information objective: the loss can be rearranged into a constrained form with an auxiliary matrix $\\Lambda$ that summarizes the embedding statistics, and then iterated with power iteration. The resulting MINC objective (Algorithm 1) treats $\\Lambda$ as a running exponential moving average of outer products of target embeddings, and updates the embedding network with a gradient that uses the lower-triangular part of $\\Lambda$ ($\\operatorname{LT}[\\Lambda]$) as an asymptotic orthogonalization step drawn from the Generalized Hebbian Algorithm. Empirically, on ImageNet, MINC consistently outperforms the Spectral Contrastive baseline across batch sizes 512, 1408, and 4096 and for both ResNet-50 and a doubled-width ResNet-50, and it behaves like linear-predictor BYOL while not matching the full non-linear-predictor BYOL. The paper also generalizes the construction to $\\alpha$-divergences and shows that the non-linear-predictor gap suggests a direction for better contrastive foundations.","pith_inferences":["A direct test of the mechanism would replace the GHA lower-triangular update with exact batch orthogonalization and compare performance and feature rank; if exact orthogonalization does at least as well, then Hebbian asymptotic orthogonalization is not the crucial ingredient and only the power-iteration framing matters.","The strong dependence on $\\alpha$ in MINC, in contrast to the small $\\alpha$ differences reported for $f$-MICL, suggests that the scalar transform $t_\\alpha$ changes the geometry of the objective substantially; sweeping $t_\\alpha$ independently of $\\alpha$ could isolate where the $\\alpha=2$ advantage comes from.","Because MINC avoids negative pairs and large batches, it could apply to settings where negative sampling is unnatural or costly, such as graph-level or reinforcement-learning representation learning; the paper stops at images, but the power-iteration derivation does not use image-specific structure."],"forward_implications":["MINC offers a non-contrastive objective that keeps the spectral and mutual-information foundation of the Spectral Contrastive loss, so the non-collapse behavior is inherited from the contrastive formulation rather than engineered through tricks.","Because MINC does not need pairwise comparisons, it trains effectively at batch size 512, and its reported ImageNet top-1 accuracy at that small batch size is higher than the Spectral Contrastive baseline at batch size 4096.","The connection derived in Section 3.6 places BYOL with a linear predictor inside the same power-iteration picture, so insights from BYOL's predictor design can be applied to spectral objectives.","For $\\alpha$-divergences other than $\\alpha=2$, the construction yields a family of non-contrastive objectives, with $\\alpha=2$ (the $\\chi^2$ case) giving the best and most stable performance in the paper's ImageNet ablations."],"supporting_citations":[{"why":"Defines the Spectral Contrastive Loss, the MI-based contrastive objective that MINC converts into non-contrastive form.","marker":"HaoChen et al., 2021"},{"why":"Supplies the $f$-mutual-information and $\\alpha$-divergence framework used to state MINC and its generalization.","marker":"Lu et al., 2024"},{"why":"Provides the Generalized Hebbian Algorithm whose lower-triangular update gives MINC its asymptotic orthogonalization.","marker":"Sanger, 1989"},{"why":"Supplies the power-iteration viewpoint and convergence motivation used to derive the fixed-point updates.","marker":"Golub and Van Loan, 2013"},{"why":"Defines BYOL, the representative non-contrastive method that MINC is compared with and connected to via a linear predictor.","marker":"Grill et al., 2020"},{"why":"Sets the SimCLR ImageNet training and evaluation protocol that MINC follows, and gives the SimCLR baseline for comparison.","marker":"Chen et al., 2020"},{"why":"Documents orthonormalization effects in BYOL, the spectral view that the paper cites in relating BYOL updates to power iteration.","marker":"Richemond et al., 2023"},{"why":"Provides the doubly stochastic gradient view that supports applying GHA-style updates in the non-linear setting.","marker":"Xie et al., 2015"}],"fun_headline_variants":["MINC loss: non-contrastive, no collapse, beats baseline","Single-batch non-contrastive loss outperforms spectral contrastive","Rewriting contrastive loss kills pairwise, boosts ImageNet","MINC: mutual info without pairwise comparisons, lower variance","Non-contrastive MINC: better than its contrastive source"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Generalized Hebbian Algorithm's lower-triangular update, designed for single-layer linear networks, keeps the deep non-linear ResNet embeddings asymptotically orthogonal during training; Figure 3 shows that without this update MINC collapses to a constant representation, so if the premise fails the non-collapse claim does too.","fun_headline_variants_meta":{"raw":{"variants":["MINC loss: non-contrastive, no collapse, beats baseline","Single-batch non-contrastive loss outperforms spectral contrastive","Rewriting contrastive loss kills pairwise, boosts ImageNet","MINC: mutual info without pairwise comparisons, lower variance","Non-contrastive MINC: better than its contrastive source"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000224,"raw_usage":{"total_tokens":1536,"prompt_tokens":1099,"completion_tokens":437,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":715,"completion_tokens_details":{"reasoning_tokens":346}},"tokens_in":715,"tokens_out":437,"duration_ms":4244,"temperature":1.0,"reasoning_tokens":346,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:57:18.504245+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MINC and the Spectral Contrastive baseline with the same code and hyperparameters but multiple seeds (the paper reports one seed per setting in Table 1) and, separately, replace the $\\operatorname{LT}[\\Lambda]$ update with exact batch orthogonalization; if the multi-seed gaps shrink to zero or the exact-orthogonalization run trains as well as the GHA version, both the stability of the claimed improvement and the paper's explanation of why GHA matters are in question.","supporting_citations":[],"review_version":1}