{"id":"faa9a55f-f358-4dec-8e5a-6e35324183dd","arxiv_id":"2412.08222","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Structured IB augments the information bottleneck objective with auxiliary encoders and an independence penalty, improving accuracy and estimated mutual information on MNIST and CIFAR-10.","lead":"This paper adds extra, separately trained feature extractors to the information bottleneck method for deep networks, and reports higher accuracy and better information preservation on MNIST and CIFAR-10. The approach matters because information bottleneck is used in semantic communication and robust deep learning, where better compression trade-offs could improve real systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's assumption (13) already forces I(Z+Z',Y)=H(Y), making the claimed inequality trivial and independent of the Gaussian machinery; the proof's Eq. (16) is algebraically wrong, so the theoretical support for SIB's information gain collapses.","rationale":"Good-faith reading: SIB is a plausible engineering heuristic, and the empirical plots show consistent small gains on MNIST and CIFAR-10, with code and network details in the appendix. What would have to be true for the central claim is that the auxiliary feature actually increases I(Z,Y) at fixed compression, and that the comparison is fair. The only theoretical argument for the information gain is Theorem 1. That theorem is not load-bearing in the intended way: condition (13) is equivalent to perfect prediction by the combined feature, making the inequality follow from H(Y)>=I(Z,Y), and no experiment or argument shows the trained decoder meets (13). The proof's Eq. (16) uses a false identity, so the stated determinant condition and Gaussian derivation do not support the inequality. This is an internal correctness issue, not a disagreement with consensus. The empirical results may still be true, but the manuscript as written overstates the theoretical justification. The reader's verdict of CONDITIONAL already captures this; my read does not move it. I also note the paper explicitly leaves the I(X,Z_hat) compression side for future work, which reinforces the conditional status.","tokens_in":12665,"tokens_out":9546,"duration_ms":97351,"concrete_test":"Audit Theorem 1 in one pass: re-derive Eq. (16) with I(Z,Z+Z')=H(Z+Z')-H(Z'), and check whether the corrected difference is H(Z') with condition (14) cancelling; then run the released SVIB code on MNIST and measure the error rate of the decoder's one-hot output h(Y'). If Eq. (16) is wrong or the error rate is nonzero, the theorem's stated conditions do not support the reported I(Z,Y) gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SIB increases I(Z,Y) at fixed compression rests on Theorem 1. Condition (13), I(h(Y'),Y)=H(Y), says the one-hot decoder output h(Y') determines Y. If (13) holds, the chain Z+Z' -> Y' -> h(Y') -> Y gives I(Z+Z',Y)=H(Y) by the data processing inequality, and since I(Z,Y) <= H(Y), conclusion (12) is immediate. Gaussianity, independence, and condition (14) are not needed. The paper never shows the trained decoder satisfies (13); the reported test accuracies (about 98-99% on MNIST) leave nonzero error, so the theorem does not apply to the experiments. The proof also has an algebraic error: equation (16) replaces I(Z,Z+Z') with H(Z), but for independent Z,Z' the correct identity is I(Z,Z+Z')=H(Z+Z')-H(Z'). With that correction, the difference I(Z+Z',Y')-I(Z,Y') equals H(Z') for an invertible linear decoder, making condition (14) irrelevant. The paper also explicitly defers analysis of I(X,Z_hat), so the compression side of the central claim has no theoretical support. Thus the theoretical justification is either trivial under an unverified perfect-prediction assumption or invalid as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Structured IB (SIB), an extension of IB Lagrangian methods in which a main encoder is trained with the standard IB objective and K auxiliary encoders are trained sequentially to maximize I(Z_i,Y) - beta I(X,Z_i) while penalizing dependence on previous features. Features are combined by weighted summation and the weights are tuned with the IB Lagrangian. The authors claim that SIB achieves higher accuracy and I(Z,Y) at the same or lower I(X,Z) than VIB, sqVIB, and NIB, even with fewer parameters, and they provide Theorem 1 as a theoretical justification for weighted summation under Gaussianity and an independence/determinant condition. Experiments on MNIST and CIFAR-10 compare SVIB, sqSVIB, SNIB, and structured versions of MINE/KNIFE against their base algorithms. The paper includes code and an appendix with implementation details.","tokens_in":13045,"tokens_out":10880,"duration_ms":106031,"significance":"If the reported empirical gains are robust, SIB could be a simple drop-in enhancement for IB-Lagrangian representation learning; the code release and the breadth of comparisons (VIB, sqVIB, NIB, MINE, KNIFE) are strengths. However, the theoretical support as written does not hold: the displayed proof of Theorem 1 contains an algebraic error, and the main assumption (13) already makes the conclusion a direct consequence of the data processing inequality, so the Gaussian calculation is superfluous. The experiments also lack repeated runs or error bars, which matters because the reported I(Z,Y) gains are small. The paper's contribution would be substantially weakened if the theorem is reduced to a trivial DPI observation, but the empirical direction remains worth evaluating in a revision.","major_comments":[{"comment":"The displayed equality (16) is algebraically incorrect. From (15) and the identity I(Z;Z+Z')=H(Z)+H(Z+Z')-H(Z,Z+Z'), the correct expression is H(Z+Z',Z)-H(Z), not H(Z+Z')-H(Z). As printed, the proof replaces I(Z;Z+Z') with H(Z), which is false in general. The subsequent block-determinant calculation actually evaluates H(Z+Z',Z)-H(Z), so the proof can be repaired locally, but the theorem statement and the role of condition (14) need to be revisited.","section":"Appendix, Proof of Theorem 1, Eqs. (15)-(16)"},{"comment":"Condition (13) alone already implies I(Z+Z',Y) >= I(Z,Y), because Y is a deterministic function of h(Y'), hence of Y'=W(Z+Z'), hence of Z+Z'; the data processing inequality applied to the chain Z -> Z+Z' -> Y gives the result. Gaussianity, independence of Z and Z', and condition (14) play no role. Moreover, the paper does not verify (13): the claim that achieving global optimality ensures I(h(Y'),Y)=H(Y) is unsupported, and the reported test accuracies (about 98-99% on MNIST) leave nonzero error, so the theorem does not apply to the evaluated models.","section":"Theorem 1, condition (13)"},{"comment":"The abstract and introduction claim that SIB achieves the same or lower compression I(X,Z), but the manuscript explicitly defers the theoretical analysis of I(X,\\hat Z) with the sentence 'More effort on the theoretical analysis of I(X, ˆZ) will be put on.' Figures 3 and 4 show only empirical estimates without error bars, so the claim of a favorable IB trade-off is not established beyond the specific runs shown.","section":"Justification of f(·) and compression claims"},{"comment":"The main empirical claims rest on single-run comparisons. The reported gains in I(Z,Y) are small (e.g., roughly 2.16 to 2.24 nats on MNIST), and the I(X,Z) differences are also small; without multiple seeds, confidence intervals, or significance tests, it is not possible to tell whether the structured variants genuinely dominate the baselines on the IB plane.","section":"Experiments, Figures 3 and 4"}],"minor_comments":[{"comment":"The heading 'Relative Work' should be 'Related Work', and the appendix title contains the typo 'Supplimentary'.","section":"Title and headings"},{"comment":"Equation (9) has a malformed expression 'Z+i−1Xj=1Zj' caused by missing braces or delimiters in the LaTeX; please fix the rendering.","section":"Equation (9)"},{"comment":"The discriminator objective is written with opposite signs from the one in Eq. (10); clarify the convention for d's output so the minimax game matches the stated KL objective.","section":"Algorithm 1, lines 10 and 12"},{"comment":"The caption's description of left and right figures is ambiguous because the figure contains eight panels; please label the panels explicitly as MNIST and CIFAR-10.","section":"Figure 3"},{"comment":"Belghazi et al. 2018a and 2018b are the same paper and should be merged or clearly distinguished; the MINE citation is duplicated.","section":"References"},{"comment":"The sentence stating that the covariance matrix Σ can be treated as a hyperparameter when using NIB is unclear; specify how the covariance is set in the experiments.","section":"Justification of f(·), NIB covariance"}],"recommendation":"major_revision","confidential_remarks":"The paper's main theoretical result needs substantial revision before publication. If condition (13) is retained, the theorem should be stated as a DPI corollary and the Gaussian machinery removed; otherwise the authors need a nontrivial theorem that does not assume a perfect predictor. The current framing is likely too ambitious for cs.IT; after adding proper statistical validation, the empirical study could fit an applied ML venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this for the empirical idea, not the theory. The SIB architecture—main encoder trained with the IB Lagrangian, auxiliary encoders trained sequentially to capture complementary information while being pushed toward independence, a fixed decoder, and a learned weighted sum—is a reasonable engineering contribution. The experiments on MNIST and CIFAR-10 show small but consistent gains in accuracy and I(Z,Y) over VIB/sqVIB/NIB, sometimes with fewer parameters, and the code is linked. The encoder dropout results are a nice sanity check that the auxiliary encoders matter.\n\nThe theory, though, does not support the weight. Theorem 1's assumption (13), I(h(Y'),Y)=H(Y), already forces the combined feature to be a perfect predictor of Y, so I(Z+Z',Y) ≥ I(Z,Y) follows immediately from the data processing inequality; the Gaussian entropy machinery is unnecessary. Condition (14) is irrelevant once (13) holds. Worse, the proof of (16) is algebraically wrong: for independent Z,Z', I(Z, Z+Z') = H(Z+Z') - H(Z'), not H(Z). With the corrected identity, the difference term becomes H(Z') for an invertible decoder, which again makes (14) decoration. The paper also never shows that the trained decoder actually satisfies (13); the reported test accuracies leave a small error floor. And the compression side, I(X,Z_hat), is explicitly deferred, so the central claim 'more I(Z,Y) at the same or lower I(X,Z)' has no theoretical backing.\n\nAre the experiments enough? They are suggestive, not conclusive. No error bars or multiple seeds, the gains are a few tenths of a percent in accuracy and a few hundredths of a bit in MI, and the baselines are not always matched in architecture scale per encoder (they divide hidden dims by two, which is fair but should be stated more carefully). Still, the pattern is consistent across two datasets and three IB variants, and the encoder dropout experiment reinforces the mechanism.\n\nMy take: treat this as an empirical paper with a flawed theoretical wrapper. A serious referee should look at it because the idea is real and the comparison is useful, but the theorem should be either fixed or removed and the empirical claims need error bars. If the authors reframe as 'a heuristic that works in practice,' it could be a decent workshop paper; with a corrected theory or a larger empirical study, it could be a solid conference paper. Don't let the current Theorem 1 gatekeep the contribution—but don't cite it as proof either.","headline":"A plausible empirical trick for IB Lagrangian training, but Theorem 1 is broken and should not be cited as theory.","tokens_in":13483,"tokens_out":2997,"would_cite":false,"duration_ms":29121,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["94A17","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding small auxiliary encoders to the IB Lagrangian yields higher accuracy and more retained label information at equal or lower compression.","keywords":["information bottleneck","structured features","auxiliary encoders","mutual information","representation learning","IB Lagrangian","variational information bottleneck","feature compression"],"falsifier":"Compute $H(Y \\mid h(W(Z+Z')))$ on the test set for a trained Structured IB model. If it is far from zero while $I(Z'+Z,Y)\\ge I(Z,Y)$ is still observed, the theorem's stated premise is not what drives the gain. Separately, check the proof's step (16): for independent Gaussian $Z,Z'$, $I(Z,Z+Z')$ should equal $H(Z+Z')-H(Z'\\mid Z)$; if it is not $H(Z)$, the displayed derivation needs repair.","tokens_in":12478,"feed_emoji":"📈","tokens_out":8494,"duration_ms":82560,"temperature":0.7,"pith_summary":"The paper tries to establish that the information bottleneck trade-off is not a fixed wall: a representation learned by a single IB-Lagrangian encoder leaves task-relevant information behind, and that information can be recovered by training a few small auxiliary encoders alongside the main one. The proposed Structured IB pipeline trains the main encoder with the usual IB Lagrangian, adds auxiliary encoders that each minimize the same Lagrangian plus a penalty on mutual information with previously collected features, and then tunes scalar weights on the summed features. On MNIST and CIFAR-10, the structured versions of VIB, square-VIB, and NIB report higher test accuracy and higher $I(Z,Y)$ at equal or lower $I(X,Z)$, even when per-encoder hidden widths are halved. The theoretical anchor is a Gaussian theorem asserting that adding an independent feature $Z'$ cannot decrease label information, provided the combined decoder output determines the label and the auxiliary covariance is not too small.","feed_headline":"Structured IB lifts accuracy and label info at same compression","feed_subtitle":"Auxiliary encoders recover information the main encoder misses, improving prediction without raising compression.","key_machinery":"The mechanism is additive structured features: the final representation is $\\hat{Z}=w_0Z+\\sum_{i=1}^K w_i Z_i$, where $Z$ comes from the main encoder and each $Z_i$ from an auxiliary encoder trained sequentially. Auxiliary training minimizes the IB Lagrangian plus a density-ratio estimate of $I(Z_i, Z+\\sum_{j<i}Z_j)$, using a discriminator to encourage independence from the existing features. Theorem 1 is the load-bearing identity: for independent Gaussian $Z,Z'$, $H(Z'+Z,Z)-H(Z)=\\frac{D}{2}\\log(2\\pi e)+\\frac{1}{2}\\log\\det(\\Sigma')$, so the added feature enlarges the entropy of the joint representation when $\\det(\\Sigma')\\ge(2\\pi e)^{-D}$; combined with the perfect-prediction condition on the decoder output, this yields $I(Z'+Z,Y)\\ge I(Z,Y)$. The paper invokes variational encoders to justify Gaussianity and a fixed one-layer linear decoder to justify the one-hot condition.","core_discovery":"The central claim is that the representation itself can be structured instead of monolithic. Theorem 1 states that for independent Gaussian features $Z,Z'$ with a full-rank linear decoder and one-hot outputs, $I(Z'+Z, Y) \\ge I(Z,Y)$ holds when $I(h(Y'),Y)=H(Y)$ and $\\det(\\Sigma') \\ge (2\\pi e)^{-D}$. The paper reports IB-plane comparisons where structured variants sit above the corresponding single-encoder curves, with reduced parameter counts, and encoder-dropout experiments show accuracy and $I(Z,Y)$ fall as auxiliary branches are removed while $I(X,Z)$ rises.","pith_inferences":["A testable extension is to replace the one-hot classification condition with a regression counterpart, but the entropy comparison would then need a continuous distortion measure; the paper does not make this move.","The paper does not explain why $I(X,Z)$ tends to fall as $K$ grows; if that pattern is real, auxiliary branches may act as a compression regularizer, and the mechanism could be studied by tracking per-encoder gradients.","Condition (13) alone may already imply the inequality through data processing, which would make the Gaussian entropy calculation an illustration of the core requirement rather than the essential driver.","The natural next experiment is to vary auxiliary encoder capacity and measure the marginal gain per branch, which would locate the diminishing-returns point the paper observes."],"forward_implications":["Single-encoder IB curves are not an upper limit for a fixed architecture budget; the same Lagrangian can be wrapped with auxiliary branches to move up in the IB plane.","Because encoder widths can be halved without losing to the full-width baseline, structured training is a parameter-efficiency strategy, not just an accuracy strategy.","The encoder-dropout trend implies auxiliary branches contribute to compression: removing branches raises $I(X,Z)$.","Any IB-style objective satisfying the theorem's two conditions can in principle be run in structured mode; the paper demonstrates this for VIB, square-VIB, NIB, MINE, and KNIFE.","The main encoder naturally receives the largest combination weight without explicit constraints, suggesting the auxiliary branches act as targeted corrections rather than replacements."],"supporting_citations":[{"why":"Defines the IB problem and Lagrangian that Structured IB modifies.","marker":"Tishby, Pereira, and Bialek (2000)"},{"why":"Supplies the variational bounds (VIB) used to train the main encoder and the Gaussian encoder assumption.","marker":"Alemi et al. (2016)"},{"why":"Provides the convex IB Lagrangian and square-IB variant used as baselines and for the beta-range convention.","marker":"Rodríguez Gálvez, Thobaben, and Skoglund (2020)"},{"why":"Cited to justify that global optimality yields the perfect-prediction condition (13).","marker":"Pan et al. (2021)"},{"why":"Supplies the shuffling scheme and density-ratio estimation used in the independence penalty of Eq. (10).","marker":"Belghazi et al. (2018a)"},{"why":"Provides the Monte Carlo estimator used to report I(X,Z) on the IB plane.","marker":"Goldfeld et al. (2018)"},{"why":"The MNIST benchmark used for all main experiments.","marker":"LeCun et al. (1998)"},{"why":"The CIFAR-10 benchmark used for all main experiments.","marker":"Krizhevsky, Hinton et al. (2009)"}],"fun_headline_variants":["Structured IB: more label info, same compression, fewer params","Auxiliary encoders enrich IB features, boosting accuracy without extra rate","Structured IB lifts IB tradeoff, shrinks model size","Structured IB: richer features for the same compression cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is condition (13), that the one-hot output of the fixed linear decoder on the combined feature determines the label $Y$; without that perfect-prediction state, the theorem does not say the auxiliary feature helps.","fun_headline_variants_meta":{"raw":{"variants":["Structured IB: more label info, same compression, fewer params","Auxiliary encoders enrich IB features, boosting accuracy without extra rate","Structured IB lifts IB tradeoff, shrinks model size","Structured IB: richer features for the same compression cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00078,"raw_usage":{"total_tokens":3373,"prompt_tokens":795,"completion_tokens":2578,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":411,"completion_tokens_details":{"reasoning_tokens":2505}},"tokens_in":411,"tokens_out":2578,"duration_ms":21323,"temperature":1.0,"reasoning_tokens":2505,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:05:43.362001+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute $H(Y \\mid h(W(Z+Z')))$ on the test set for a trained Structured IB model. If it is far from zero while $I(Z'+Z,Y)\\ge I(Z,Y)$ is still observed, the theorem's stated premise is not what drives the gain. Separately, check the proof's step (16): for independent Gaussian $Z,Z'$, $I(Z,Z+Z')$ should equal $H(Z+Z')-H(Z'\\mid Z)$; if it is not $H(Z)$, the displayed derivation needs repair.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cited to justify that global optimality yields the perfect-prediction condition (13)."}],"review_version":1}