{"id":"2dc3e9f1-6ee2-4574-95ea-8c9738ee5a4a","arxiv_id":"2501.03469","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"IMSVD softly discretizes latent variables and uses a cross-joint entropy loss to learn transform-invariant, non-collapsed, redundancy-minimized image representations without labels.","lead":"IMSVD is a self-supervised learning method that splits each latent feature into small one-hot-like switches and trains a network to maximize information across those switches. On ImageNet it reaches 73.6% top-1 accuracy for linear probing at 1000 epochs, closely matching or slightly beating Barlow Twins and VICReg while using less time and memory.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1 is false for the finite batches used: with N=2048 and D_M=80, exact uniform marginals and pairwise-independent joints are unattainable, so the central provable claim as stated cannot hold.","rationale":"Both the reader's circular-derivation concern and my concern point to the theory being incomplete, but the finite-N issue is more load-bearing because it targets Theorem 1 itself, not the route by which the loss was motivated. The reader's strongest_claim is exactly the theorem; if the theorem is false for the batch sizes used, the paper's headline theoretical contribution ('provably produces transform-invariant, non-collapsed, pairwise-independent discrete representations') is unsupported. The empirical results may still stand, and the issue is fixable by restating the theorem as an asymptotic statement with a convergence rate, or by deriving a finite-N bound tying the loss value to the deviation from uniformity/independence. I therefore keep the reader's CONDITIONAL verdict: the method is promising and reproducible, but the theoretical claim as written is not correct. I would not move to REJECT because the experiments and code provide substantial evidence of the method's practical value, and the theoretical gap appears repairable. I partially agree with the reader: the derivation of Eq. (11) from Eq. (9) indeed assumes the conclusion q'=q'', but that concern does not touch the theorem's internal validity; the finite-N obstruction does.","tokens_in":17458,"tokens_out":9559,"duration_ms":87109,"concrete_test":"Solve the exact minimizer of Eq. (11) for a small instance with N=3, D_M=2, M=1 (or M=2) over one-hot q'=q''. Enumerate all code assignments; the theorem predicts p(m,d)=1/2 (and P=1/4 for M=2). The actual minimizer has fractional counts such as p=(2/3,1/3), so the theorem's conclusion fails. As a complementary check, run the same exhaustive enumeration for N=6400, D_M=80, M=2; if a uniform joint 1/6400 is achievable and is the minimizer, the theorem holds only in that divisible case, confirming the finite-N obstruction. This settles whether Theorem 1 is a statement about the empirical loss actually optimized.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is Theorem 1: minimizing Eq. (11) yields q'_i=q''_i one-hot, p(m,d)=1/D_M, and P(m1,m2;d1,d2)=1/D_M^2 for m1≠m2. However, Eq. (11) is defined with empirical batch estimates P^c = (1/N)Σ_i q'_i(m1,d1)q''_i(m2,d2). For a finite batch, these equalities are integer-count constraints. With the paper's own settings (N=2048, D_M=80, so D_M^2=6400), neither 2048/80 nor 2048/6400 is an integer; indeed 2048<6400. Thus no assignment of one-hot codes can make p(m,d)=1/80 or P=1/6400 exactly. The global minimum of the empirical loss is therefore not a point where the theorem's conclusions hold; the theorem is false as stated. A correct statement would require an infinite-sample limit or a finite-N bound quantifying the approximation (as in the empirical Fig. 3(a), which shows approximate uniformity). This is independent of the circular derivation of Eq. (11) from Eq. (9): even taking Eq. (11) as given, its minimizer over discrete codes is constrained by finite-N integer counts. The missing Appendix-II proof cannot repair this unless it explicitly changes the loss to a population version, which is not what is minimized in training.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IMSVD, a self-supervised representation-learning method in which the projector output is split into M segments, each softly discretized by a softmax over DM units. Marginal and joint distributions of these discrete variables are estimated over the batch, and the training objective (Eqs. (4) and (9)) maximizes cross-view agreement, per-variable entropy, and joint entropy, with r=2 in the implemented loss. The authors derive a cross-joint entropy loss (Eq. (11)) and state Theorem 1, which claims that minimizing this loss drives the two views to identical one-hot codes, uniform per-variable marginals, uniform pairwise joint distributions, and zero pairwise mutual information. The paper reports ImageNet linear and k-NN classification, transfer learning, efficiency, and ablations, plus visualizations of the learned discrete variables, and it makes code publicly available.","tokens_in":17835,"tokens_out":4229,"duration_ms":39650,"significance":"If the theoretical claim were established, IMSVD would be a notable contribution: a non-contrastive SSL objective directly based on information measures, with stronger redundancy reduction than linear decorrelation, and with interpretable, near-discrete embeddings obtained without a hard quantization step. The empirical results are broadly consistent with that narrative, and the paper contains useful strengths: reproducible public code, careful comparisons with Barlow Twins and VICReg, and ablations of batch size, projector depth, feature dimension, and the DM hyperparameter. However, the central theorem as stated is not defensible against the finite-batch loss that is actually optimized, and the derivation of the implemented loss from the information-theoretic objective contains a circular step. The contribution is therefore promising but requires a corrected theoretical statement and verifiable proofs before it can be accepted.","major_comments":[{"comment":"The derivation of the cross-joint entropy loss equates P^c(m1,m2;d1,d2) with the self-joint probability P(m1,m2;d1,d2) by asserting that the embedding features are optimized to be transform-invariant, i.e., q'_i = q''_i. That equality is the intended outcome of the optimization, not a premise available when the loss is derived. During training the two views can and do differ, so Eq. (11) is a surrogate for Eq. (9) rather than an equivalent loss. The information-theoretic interpretation of the actual minimized loss is therefore incomplete. The paper should state explicitly in what sense Eq. (11) approximates or bounds Eq. (9), or prove the desired minimizer properties directly for the cross-joint loss without assuming q'_i = q''_i.","section":"Section III-D, Eqs. (10)-(11)"},{"comment":"Theorem 1 is false as stated for the finite batches actually optimized. The loss in Eq. (11) uses empirical batch probabilities P^c = (1/N) sum_i q'_i(m1,d1) q''_i(m2,d2). With the paper's default settings N=2048 and DM=80, the conclusions p(m,d)=1/80 and P(m1,m2;d1,d2)=1/6400 cannot hold for any one-hot assignment, because they would require per-unit counts of 2048/80=25.6 and 2048/6400=0.32 respectively. Hence the global minimum of the empirical loss does not satisfy the equalities stated in Theorem 1. The theorem should be restated for the population loss, or the paper should provide finite-N discrepancy bounds. Figure 3(a) already shows only approximate uniformity, which is consistent with this limitation rather than with the exact statement of the theorem.","section":"Theorem 1, Eq. (11), and Section IV implementation settings"},{"comment":"The derivation of Eq. (11) from Eq. (9) and the proof of Theorem 1 are relegated to appendices that are not present in the reviewed manuscript. Because these arguments are load-bearing for the central claim, the review cannot verify them. The paper should include the full derivation and proof, or the supplementary material containing them should be part of the version under review.","section":"Appendices I, II, and III"}],"minor_comments":[{"comment":"The word \"non-travail\" in the abstract appears to be a typo for \"non-collapsed\"; please correct it.","section":"Abstract and Introduction"},{"comment":"The text says \"where pi(m,d) denotes the probability...\" but the quantity defined is p(m,d); the subscript i is not introduced and should be removed.","section":"Section III-B, Eq. (2)"},{"comment":"The column headers \"DE+OE+TIC\" and \"DE+OE+TI\" do not match the abbreviations DE, OE, TIC, and TI defined in the text; please make the table headers consistent with the defined terms.","section":"Section V-B, Table VIII"},{"comment":"Reference [10] is cited as \"Unsupervised representation learning by predicting image rotations,\" but the method being compared is VICReg; the citation title and venue should be corrected.","section":"References"},{"comment":"The sentence \"IMSVD statically assigns instances\" should likely read \"statistically\" or \"stochastically\"; as written, \"statically\" is confusing.","section":"Section III-E"},{"comment":"The cross-reference to \"Sec. V-B3\" does not match the numbered items in Section V-B, and the description of the 400x400 matrix in Fig. 3(a) should clarify which variables and dataset split are used.","section":"Section V-C and cross-references"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical contribution is real and the public code is a plus, but the theoretical claim is currently over-stated: the finite-batch issue alone invalidates Theorem 1 as written, and the circular step in the loss derivation weakens the information-theoretic justification. I would encourage the authors to restate the theorem for the population loss or to provide finite-N bounds, and to put the appendix proofs into the reviewed version. If those fixes are clean, the manuscript could become acceptable; I am not recommending rejection because the core methodology and experiments appear sound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: IMSVD is a real contribution to the non-contrastive SSL line, and the empirical package is solid. The main theorem as written is overclaimed, and the loss derivation has a circular step, but the method itself looks salvageable and worth refereeing.\n\nWhat's new: instead of adding covariance penalties, the authors soft-discretize each latent segment with softmax and directly maximize an entropy objective over the batch-level joint distribution. That's a different mechanism from Barlow Twins/VICReg, and it gives one-hot, interpretable features without Gumbel temperature scheduling. The claim that minimizing the loss statistically separates instances (each pair of variables can encode 6400 samples, batch 2048) is a nice way of connecting to contrastive learning. The experiments are honestly reported: IMSVD beats BT and VICReg by small margins at every epoch count, uses a 2-layer projector instead of 3, and runs faster. The ablation study is thorough, including the effect of D_M. Code is public. The citation pattern is fine; they engage with the relevant SSL and discretization literature.\n\nSoft spots, in order of severity. First, Theorem 1 is not true for the finite batches actually used. The loss in Eq. (11) is computed on empirical batch probabilities. With N=2048 and D_M=80, no assignment of one-hot codes can make p(m,d)=1/80 exactly, and pairwise uniformity 1/6400 is even further out of reach. So the theorem as stated would require a population limit or a finite-N approximation; as written it's false. Second, the derivation of Eq. (11) from Eq. (9) in Sec. III-D says \"since features are optimized to be transform-invariant, q'_i = q''_i\" -- that's the conclusion the loss is supposed to produce, so the cross-joint entropy is a surrogate, not an equivalence. The final minimizer of the surrogate may still have the intended properties, but the paper doesn't prove it directly. Third, the proof of the theorem is in the supplementary appendix, which wasn't available, so the core theoretical claims are unverified.\n\nNone of these are fatal to the empirical method. The gains over baselines are modest (0.2-0.4 top-1) but consistent, and the efficiency story is real. The explainability visualizations (texture/shape units) are a genuine extra.\n\nWho should read it: SSL researchers working on non-contrastive or information-theoretic objectives. The method is worth trying. Recommendation: send to peer review, but require a corrected theorem with finite-N analysis or a population version, and the full proof.","headline":"Genuinely new non-contrastive objective with solid experiments, but the main theorem is overclaimed for finite batches and the loss derivation has a circular step; method is salvageable and worth refereeing.","tokens_in":18319,"tokens_out":3036,"would_cite":true,"duration_ms":29120,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A self-supervised learning objective that first turns each latent variable into a soft one-hot code can be optimized directly by information measures; the paper proves that its minimizer is a transform-invariant, non-collapsed…","keywords":["self-supervised learning","image representation learning","variable discretization","information theory","cross-joint entropy","redundancy reduction","non-contrastive learning","linear classification"],"falsifier":"Train an IMSVD model to convergence on ImageNet and compute the empirical marginal $p(m,d)$ and pairwise joint $P(m_1,m_2,d_1,d_2)$ over the training set. If $p(m,d)$ is not close to $1/D_M$ for every variable and $P(m_1,m_2,d_1,d_2)$ is not close to $1/D_M^2$ for every variable pair, or if the empirical mutual information between any two variables is not near zero, then the claimed optimum is not the one reached. A direct check on held-out augmentation pairs, whether $q'_i$ and $q''_i$ are both one-hot and equal, would also settle the mechanism.","tokens_in":17282,"feed_emoji":"🧩","tokens_out":9461,"duration_ms":80207,"temperature":0.7,"pith_summary":"The paper aims to show that self-supervised representation learning can be driven directly by information measures rather than by similarity heuristics or linear decorrelation, if the latent space is first re-encoded as a set of discrete variables. It introduces IMSVD, which applies a softmax to each block of the feature vector so that each block behaves like a categorical variable. The proposed cross-joint entropy loss maximizes the information carried by each variable while forcing different views of the same image to share the same discrete code. The central result is Theorem 1: at the minimizer, every variable is one-hot, every variable is uniformly distributed, every pair of variables is jointly uniform and independent, and the mutual information between any two variables is zero. If true, this gives a non-contrastive SSL objective with proven invariance, collapse avoidance, and redundancy reduction, and it provides variable-level explainability.","feed_headline":"Loss that discretizes features provably kills collapse and redundancy","feed_subtitle":"Softly discretized variables let a cross-entropy loss maximize information directly, with proofs and ImageNet gains.","key_machinery":"The central device is soft variable discretization: the projector output is split into $M$ contiguous blocks of size $D_M$, and each block is normalized by a softmax to form a categorical sub-vector $q_i(m,:)$. This turns a continuous latent vector into a collection of categorical variables whose marginal and joint distributions can be estimated from a batch, making entropy and mutual information computable. The training loss is the cross-joint entropy in Eq. (11): its first term is the average log inner product between the two views' discrete codes, and its second term is the negative entropy of the empirical cross-joint distribution over all variable pairs, masked to exclude diagonal elements inside each self-block. Theorem 1 shows that at the minimum of this loss the codes are exactly one-hot, marginals are uniform, all variable pairs are jointly uniform, and all pairwise mutual information vanishes.","core_discovery":"The paper's central claim is that softly discretizing each latent variable makes information measures directly computable from a training batch, so the SSL objective can be stated and optimized in information-theoretic terms. The author proves Theorem 1: minimizing the cross-joint entropy loss of Eq. (11) drives the two-view features $q'_i$ and $q''_i$ to identical one-hot vectors for every sample and variable, each variable to the uniform marginal $p(m,d)=1/D_M$, every pair of variables to the uniform joint distribution $1/D_M^2$, and the mutual information between any two variables to zero. This is a stronger redundancy guarantee than the pairwise linear decorrelation used by existing non-contrastive methods, because it removes arbitrary nonlinear dependence. The same optimal solution gives transform invariance without collapse and, because each variable pair can encode $(D_M)^2$ distinct samples, the method statistically performs instance discrimination even though it never uses negative pairs.","pith_inferences":["The equivalence between the implementable cross-joint loss and the information-theoretic objective rests on assuming the two views already produce identical discrete codes; a variant that estimates the joint distribution symmetrically from both views, or adds an explicit invariance term before forming the cross-joint term, would place the derivation on firmer ground without changing the final theo","Theorem 1 describes the global minimizer over distributions, but achieving it requires the batch to be large enough to represent $1/D_M^2$ combinations for every variable pair; very small batches may not reach the stated optimum, and testing at batch sizes where $D_M^2$ exceeds the batch could reveal a different fixed point or a partial collapse.","Because the discretized variables are categorical and interpretable, the same loss could be adapted to hierarchical clustering, multi-modal alignment, or generative modeling by treating each variable as a discrete token; the paper notes this potential but does not demonstrate it.","The claim that a non-contrastive loss statistically performs contrastive learning suggests a broader principle: any method that maximizes the entropy of a factorization of the latent space may implicitly separate instances, which could guide the design of other non-contrastive objectives."],"forward_implications":["Minimizing the cross-joint entropy loss provably drives the representation to a configuration in which every variable is one-hot, every variable is uniformly distributed, and every variable pair is independent, so the learned code is simultaneously transform-invariant, non-collapsed, and redundancy-free.","Because redundancy is removed for arbitrary nonlinear dependence rather than only linear correlation, IMSVD can use a shorter embedding and a two-layer projector while matching or beating methods that require 8,192-dimensional features and three-layer projectors.","The optimal discrete code statistically performs instance discrimination: with $D_M=80$, each variable pair can encode 6,400 distinct samples, so different images receive distinct codes even though no negative pairs are used.","The discrete variables are interpretable: units of the first variable capture texture patterns and units of the second variable capture shape patterns, and these can be visualized and localized with Grad-CAM.","The method's performance degrades only mildly with smaller batch sizes and transfers to object detection, instance segmentation, and scene classification, suggesting it does not depend on a large bank of negative samples."],"supporting_citations":[{"why":"Supplies the MultiView assumption that shared information between different views is sufficient for downstream tasks, which motivates the information-theoretic objective.","marker":"[14]"},{"why":"Defines average entropy and total correlation, which the IMSVD objective respectively maximizes and minimizes.","marker":"[51]"},{"why":"Provides the covariance-based non-contrastive baseline whose linear redundancy reduction IMSVD claims to surpass.","marker":"[19]"},{"why":"Provides the twin-architecture baseline with invariance, variance, and covariance terms; IMSVD compares against it and uses a shallower projector and shorter embedding.","marker":"[10]"},{"why":"Represents an earlier information-theoretic SSL method using log-determinant mutual information, which is limited to linear/Gaussian dependence and is contrasted with direct discrete estimation.","marker":"[16]"},{"why":"Shows that contrastive InfoNCE maximizes a lower bound on mutual information between views, the route that IMSVD replaces with direct information computation.","marker":"[18]"}],"fun_headline_variants":["Discretized features provably end redundancy in SSL","Soft discretization yields provable info-theoretic SSL","Discretize latents, prove zero redundancy","Explainable SSL via information-maximized discretization","Non-contrastive SSL with a provable information loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The loss derivation assumes the two augmented views already give identical discrete codes, even though making them identical is precisely the goal of training; if the views differ during optimization, the loss is a surrogate rather than the exact information-theoretic objective.","fun_headline_variants_meta":{"raw":{"variants":["Discretized features provably end redundancy in SSL","Soft discretization yields provable info-theoretic SSL","Discretize latents, prove zero redundancy","Explainable SSL via information-maximized discretization","Non-contrastive SSL with a provable information loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0009,"raw_usage":{"total_tokens":3893,"prompt_tokens":982,"completion_tokens":2911,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":2834}},"tokens_in":598,"tokens_out":2911,"duration_ms":21405,"temperature":1.0,"reasoning_tokens":2834,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:52:56.133823+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an IMSVD model to convergence on ImageNet and compute the empirical marginal $p(m,d)$ and pairwise joint $P(m_1,m_2,d_1,d_2)$ over the training set. If $p(m,d)$ is not close to $1/D_M$ for every variable and $P(m_1,m_2,d_1,d_2)$ is not close to $1/D_M^2$ for every variable pair, or if the empirical mutual information between any two variables is not near zero, then the claimed optimum is not the one reached. A direct check on held-out augmentation pairs, whether $q'_i$ and $q''_i$ are both one-hot and equal, would also settle the mechanism.","supporting_citations":[{"cited_title":"Information theoretical analysis of multivariate correla- tion,","cited_arxiv_id":null,"evidence_quote":"Defines average entropy and total correlation, which the IMSVD objective respectively maximizes and minimizes."},{"cited_title":"Barlow twins: Self-supervised learning via redundancy reduction,","cited_arxiv_id":null,"evidence_quote":"Provides the covariance-based non-contrastive baseline whose linear redundancy reduction IMSVD claims to surpass."},{"cited_title":"Unsupervised representation learning by predicting image rotations,","cited_arxiv_id":null,"evidence_quote":"Provides the twin-architecture baseline with invariance, variance, and covariance terms; IMSVD compares against it and uses a shallower projector and shorter embedding."},{"cited_title":"Self- supervised learning with an information maximization criterion,","cited_arxiv_id":null,"evidence_quote":"Represents an earlier information-theoretic SSL method using log-determinant mutual information, which is limited to linear/Gaussian dependence and is contrasted with direct discrete estimation."},{"cited_title":"Data-efficient image recognition with contrastive predictive coding,","cited_arxiv_id":null,"evidence_quote":"Shows that contrastive InfoNCE maximizes a lower bound on mutual information between views, the route that IMSVD replaces with direct information computation."}],"review_version":1}