{"id":"3118e6cd-e3f0-4331-b6cf-d871dc08127f","arxiv_id":"2504.12939","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A CNN surgery method identifies polysemantic channels and rewires each into two concept-specific channels plus a residual channel that preserves the original output exactly.","lead":"This paper presents an algorithm that splits a single CNN channel responding to two unrelated concepts into two separate channels, one per concept. The method rewires the network without changing its output, which could make feature visualizations and circuit analysis of image classifiers much clearer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The method's edge-splitting rule assumes concepts are separable in layer l−1; with shared or distributed l−1 channels the 'disentangled' channels still co-activate, and Table 1 is rho-tuned on the same objective.","rationale":"The paper's central claim has two parts: (1) the edited network preserves the original channel activation, and (2) each disentangled channel activates for only one concept. Part (1) is algebraically secure if the inserted layer l′ is linear: with c1 + c2 − residual equal to the original pre-activation, the residual construction exactly reconstructs the input to layer l. I therefore do not base the objection on output preservation, despite the absence of accuracy numbers. Part (2) is the genuinely load-bearing empirical claim, and it depends on concepts being separable in the previous layer. The assignment rule in Ineq. (5) is a hard partition: every l−1 channel is either t1-specific, t2-specific, or shared. If the true concept directions overlap, the shared edges remain active in both disentangled channels, so neither channel is monosemantic; the residual channel then encodes a real part of the concept signal rather than a negligible remainder. The paper's own Figure 1 caption concedes residual cross-concept pattern visibility, and the Conclusion restricts the method to settings where concepts are semantically distinct in later layers. Table 1 is suggestive but not decisive: rho is tuned on the training split using Eq. (6), which maximizes the same separation ratio that Table 1 measures on validation images, and the reported means lack error bars, per-channel variance, and any baseline. The reader's weakest assumption identifies exactly this separability condition, and the reader's CONDITIONAL verdict already asks for stronger validation. My stress-test therefore does not move the verdict; it sharpens the precise test that would settle the concern.","tokens_in":8021,"tokens_out":14909,"duration_ms":166661,"concrete_test":"For every γ-polysemantic channel c, compute the shared ARV mass S_c = Σ_i min(|ARV(l,c,t1)_i|, |ARV(l,c,t2)_i|) / Σ_i max(|ARV(l,c,t1)_i|, |ARV(l,c,t2)_i|). Then report, per channel, the mean relative activation of c1 on t2 images and of c2 on t1 images against S_c. Also rerun Table 1 with ρ=1 (no tuning) and with a random edge-split baseline under the same evaluation protocol. If cross-activation increases with S_c, or if high-S_c channels show positive cross-class activation, the disentangled channels are not monosemantic for those channels and the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that each disentangled channel activates for only one concept. The construction that is supposed to deliver this is Ineq. (5), which assigns every previous-layer channel i wholly to concept t1, wholly to concept t2, or leaves it shared. This partition is only valid if the two concepts are separable at the level of individual channels in l−1. If the concepts recruit overlapping or distributed directions — for example, if the same l−1 channel is relevant to both classes, or if each concept is spread across many of the same channels — then the shared set is nonempty. Shared edges are copied into both disentangled channels, so channel c1 still receives class-t2 signal through those edges, and c2 still receives class-t1 signal; the residual channel then carries a nontrivial part of the concept signal instead of being close to zero. The paper implicitly concedes this in Fig. 1, where 'some cauliflower patterns are still lightly visible' in the first disentangled channel, and in the Conclusion, where the authors note the method 'may work better in later layers.' Table 1 does not resolve the concern: rho is selected on training images by maximizing the very activation-ratio objective (Eq. 6) that Table 1 then reports on validation images, and the table gives no error bars, no per-channel shared-mass statistics, and no comparison to a random edge-split or PURE baseline. The monosemanticity claim therefore rests on an unvalidated separability condition, not on a demonstrated property of the edited network.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a post-hoc method to identify and explicitly disentangle polysemantic channels in a pretrained CNN. A channel is flagged as γ-polysemantic when the average Input×Gradient relevance vectors (ARVs) from the previous layer differ between two classes (cosine similarity below γ, Definition 3.1). The disentanglement inserts an auxiliary layer l′ in which channel c is replaced by two concept channels plus a residual channel; the residual is connected to c with weight −1 so that c1 + c2 − residual linearly reconstructs the original channel's pre-activation. Experiments on the penultimate layer of an ImageNet-trained ResNet-50 show, for a case study (channel #1660, digital clock/cauliflower) and aggregated over polysemantic channels, that the two new channels activate almost exclusively for one class each while the residual stays near zero.","tokens_in":8340,"tokens_out":18607,"duration_ms":192363,"significance":"The contribution is genuinely useful if the claims hold. The explicit weight-space rewiring is a clean alternative to PURE's virtual disentanglement, and the c1 + c2 − residual construction is an elegant linear device that provably recovers the original pre-activation; the authors ship code and define γ-polysemanticity crisply. The case study is convincing: the activation densities in Fig. 3 and the MACO visualizations in Fig. 1 show clear separation, and the WordNet-similarity heuristic for choosing γ (Fig. 4) is a reasonable empirical calibration. Three caveats temper the result: the monosemanticity claim is only partial because shared edges remain in both channels (as the authors concede in the Fig. 1 discussion); Table 1 evaluates essentially the same objective used to select ρ; and the asserted output preservation is never measured empirically. None of these flaws is fatal, but the claims currently outrun the evidence.","major_comments":[{"comment":"The central claim that each disentangled channel 'only activates for one of the respective class concepts' (Sec. 3.2) is not guaranteed by Ineq. (5): edges for which neither ARV(l,c,t2)_i·ρ > ARV(l,c,t1)_i nor the symmetric condition holds are copied into both c1 and c2, so class-t2 signal continues to reach c1 through those shared edges. The manuscript itself concedes residual cross-concept signal in the Fig. 1 discussion ('some cauliflower patterns are still lightly visible'). Table 1 quantifies mean cross-activations of −9%/−21% but reports no error bars, no number of aggregated channels or class pairs, and no statistic on the mass of shared edges. I ask for per-channel distributions and confidence intervals, the number of channels and class pairs aggregated, a shared-edge mass or activation-purity metric, and a wording change from 'only activating for one concept' to a qualified claim such as 'mainly activating for one concept'.","section":"Sec. 3.2, Ineq. (5), Table 1"},{"comment":"ρ is selected by maximizing, on training images, the ratio of |c′1| to |c′2| activations for the two classes (Eq. 6); Table 1 then reports essentially the same relative-activation quantities on validation images. The main quantitative evaluation therefore re-measures the selection objective, which limits its confirmatory value for the claim of effective disentanglement. I suggest adding at least one outcome that is not the selection objective, such as classification accuracy before and after rewiring, logit reconstruction error, or a MACO-based concept purity measured on channels held out from ρ selection, and a sensitivity analysis over the ρ grid.","section":"Appendix A, Eq. (6); Table 1"},{"comment":"The claim that rewiring 'does not affect the output or predictive performance' rests on pre-activation(c) = c1 + c2 − residual. This identity holds only if the inserted layer l′ is purely linear (no BatchNorm or ReLU) and if the two masked edge sets are disjoint, which is true for positive ARVs and ρ ≤ 1 but can fail with negative ARVs. The paper does not state whether l′ contains normalization or an activation, nor how the insertion interacts with the residual skip connections of ResNet-50, and no empirical output check is reported. Please specify the implementation of l′, report the reconstruction error of c1 + c2 − residual versus the original channel on the validation set (maximum and quantiles), and report top-1 accuracy or logit outputs before and after rewiring.","section":"Sec. 3.2, output-preservation claim"},{"comment":"The experiments contain no baseline comparison. Since PURE [5] is the closest prior work and is discussed at length in Sec. 2, a direct comparison (e.g., per-concept activation purity) would substantiate the positioning of the explicit approach. In addition, a random edge-split baseline for Ineq. (5) would show whether the ARV-based masking, rather than the mere act of splitting the channel into two, is responsible for the observed separation. Without such comparisons, the claim that the proposed algorithm 'is effective' (Sec. 4) is supported only by the method's own numbers.","section":"Sec. 4"}],"minor_comments":[{"comment":"The text refers twice to 'Theorem 3.1' (in the qualitative analysis of Sec. 4 and in the quantitative analysis of Appendix A); Definition 3.1 is a definition, not a theorem.","section":"Sec. 4; Appendix A"},{"comment":"Please report the number of polysemantic channels and class pairs aggregated in Table 1, and clarify how channels relevant to three or more classes are reduced to a two-concept disentanglement; the paper only states that this 'can be broken down' into two-concept cases without specifying the pairing rule.","section":"Table 1"},{"comment":"Please specify the discretization and range of the ρ grid used to solve Eq. (6), and report the selected ρ values; additionally, state whether ARVs are nonnegative in practice, since for negative attribution values the scaled comparison in Ineq. (5) changes its meaning and can even make the two masking sets overlap, breaking the exact reconstruction.","section":"Appendix A, Eq. (6); Ineq. (5)"},{"comment":"The quantitative evaluation is restricted to validation images whose relative attribution exceeds τ, but this filter is described only in Appendix A; please state it in the main text, since Table 1 otherwise appears to average over all images of the two classes.","section":"Appendix A, quantitative analysis"},{"comment":"The paper states that τ, γ, and p are handpicked; given that the aggregate analysis depends on γ through Definition 3.1, a small sensitivity analysis over γ (and ideally τ and p) would help establish that the reported results are not brittle to these choices.","section":"Appendix A, hyperparameters"},{"comment":"Both channel identification (Ineq. 2) and edge assignment (Ineq. 5) rely on Input×Gradient attributions, and the paper states only that this choice 'works sufficiently well'; a small faithfulness sanity check (e.g., comparing ARVs from Input×Gradient with LRP or Integrated Gradients on the case-study channel) would increase confidence in the whole pipeline.","section":"Sec. 2; Ineqs. (2), (5)"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the circularity between the ρ-selection objective (Eq. 6) and the quantitative evaluation in Table 1; I would ask for an independent evaluation (accuracy preservation, a baseline comparison, error bars) before the aggregate claims are accepted. The single-case qualitative evidence (Figs. 1 and 3) is strong and the linear construction is elegant, so I believe the paper is salvageable within the scope of a revision. The repeated 'Theorem 3.1' mislabeling and the unstated image filter in the quantitative evaluation suggest that the manuscript would benefit from a careful proofread. I have no concerns about novelty disclosure or citation practices."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2504.12939. The core construction is a genuinely neat trick: insert a new layer before the last convolution, split the polysemantic channel into two concept channels plus a residual channel connected back with weight −1, and the identity c1 + c2 − residual recovers the original activation exactly. Because the split is done in weight space, you can run MACO or any feature visualization on the disentangled channels directly, which is something PURE's virtual disentanglement cannot do. The digital clock/cauliflower example is convincing, and the activation density plots in Fig. 3 show real separation between the two channels.\n\nThe genuinely new bit is the explicit, output-preserving rewrite. Sparse autoencoders approximate the network in a larger space; this edits the network so its outputs are unchanged by construction. The γ-polysemanticity criterion (low cosine similarity between average relevance vectors in the previous layer, validated against WordNet) is a reasonable, cheap heuristic. Related work is handled fairly — PURE, sparse autoencoders, CAVs, and concept whitening are all cited and clearly differentiated. Code is released, which for this kind of work matters.\n\nThe soft spots are in the evaluation, not the idea. Table 1 aggregates across channels with no error bars, no per-channel spread, no random-split baseline, and no comparison to PURE. The ρ threshold is fitted on training images by maximizing essentially the same activation-ratio objective the table reports on validation. That is not leakage — the split is honest, train versus validation — but it is a fitted hyperparameter, and without error bars the headline numbers overstate the generality. The claim that each disentangled channel 'only activates for one concept' is also stronger than what is shown; the paper itself concedes light cauliflower leakage in Fig. 1, which is exactly what you expect when the two concepts share or distribute over the same previous-layer channels. That separability condition is the real scope limit. The residual channel absorbs shared edges, but if the shared mass is large the concept channels still co-activate. I read the stress-test note as claiming this is unvalidated; I would soften that — it is under-validated, and mostly acknowledged in the conclusion. Two minor gaps: normalization layers are never discussed, and output preservation is argued by construction but never verified numerically on task accuracy, a cheap check that would settle it.\n\nThis is a workshop-scale proof of concept for CNN mechanistic interpretability. The construction is correct, the idea is new, and with error bars, a random-split baseline, and a PURE comparison it becomes a solid technique paper. I would send it to peer review and accept after revision.","headline":"A neat workshop-scale paper: explicit residual-corrected splitting of polysemantic channels that preserves outputs and makes MACO visualizations interpretable; the flagship example convinces, the quantitative evaluation does not.","tokens_in":8852,"tokens_out":9548,"would_cite":true,"duration_ms":97132,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A pretrained CNN's polysemantic channels can be explicitly split into single-concept channels by rewiring the weights through a new intermediate layer, without changing the network's predictions.","keywords":["mechanistic interpretability","polysemantic channels","channel disentanglement","feature visualization","input-gradient attribution","gamma-polysemanticity","ResNet-50","ImageNet"],"falsifier":"Take a channel flagged as $\\gamma$-polysemantic, apply the split with the paper's automatic $\\rho$ selection, and feed held-out class images through the edited network. If the residual channel's mean absolute activation is comparable to the original channel's activation rather than near zero, or if the concept-matched disentangled channel responds weakly on its own class or strongly on the other, the 'only activates for one concept' claim is falsified. Comparing the original and edited logits also settles the recovery guarantee: any material change contradicts the claim that the original activation is exactly recoverable.","tokens_in":7824,"feed_emoji":"🧩","tokens_out":8771,"duration_ms":79564,"temperature":0.7,"pith_summary":"The paper tries to establish that polysemantic channels in convolutional neural networks—channels that fire for several unrelated concepts, such as 'digital clock' and 'cauliflower'—are not an unavoidable property of the trained model. It proposes a weight-rewiring procedure that splits such a channel into two channels, each responding to one concept, plus a residual channel that exactly restores the original activation. Because the disentanglement is explicit, standard tools such as feature visualization can be applied to each concept separately instead of producing a blended image. If the claim holds, pretrained CNNs can be made more interpretable without retraining and without changing their predictive behavior.","feed_headline":"Polysemantic CNN channels split into single-concept channels","feed_subtitle":"A weight-rewiring algorithm separates a channel's mixed concepts into dedicated channels without changing predictions.","key_machinery":"The machinery is the average relevance vector (ARV): for a channel $c$ in layer $l$ and a class $t$, the vector of Input×Gradient attributions from every channel in layer $l-1$ to $c$, averaged over class-$t$ images. Cosine similarity between the two class ARVs defines $\\gamma$-polysemanticity (below threshold $\\gamma$), and the scaled comparison $\\mathrm{ARV}(l,c,t_2)_i\\cdot\\rho > \\mathrm{ARV}(l,c,t_1)_i$ assigns each previous-layer channel $i$ to concept 2, allowing the incoming edges of $c$ to be partitioned into concept-1, concept-2, and shared parts. The residual channel, wired to $c$ with weight $-1$, removes the double encoding of shared features and recovers the original activation exactly. This three-way weight partition is what makes the disentanglement explicit rather than virtual.","core_discovery":"The central claim is that a channel $c$ in layer $l$ that is $\\gamma$-polysemantic for two classes $t_1$ and $t_2$ can be replaced by three channels in a new layer between $l-1$ and $l$: one channel that keeps only the incoming edges mainly responsible for $t_1$, one that keeps only those mainly responsible for $t_2$, and a residual channel that keeps the shared edges and connects to the original channel $c$ with weight $-1$. Summing the three recovers $c$'s original activation, so the edit is prediction-preserving. On a ResNet-50 trained on ImageNet, the paper reports that for penultimate-layer channels with $\\gamma=0.5$, the concept-matched disentangled channels retain roughly 114% and 125% of the original activation on their own class while staying near zero on the other, and the residual channel stays near zero. This is presented as evidence that each disentangled channel 'only activates for one of the respective class concepts' and that the residual does not need to be explained.","pith_inferences":["Beyond the paper's own evidence, the residual channel's activation level could be used as a general diagnostic: if the residual stays small across many splits, the network's late-layer representations are closer to a privileged basis than superposition theorizing suggests; if it grows, the concepts are distributed and edge-masking alone cannot isolate them.","The same 'two concept channels plus a negative residual' wiring is a generic identity for decomposing any mixed signal into components plus a correction term, so the construction may transfer to transformers or other architectures where a separation criterion analogous to the ARV inequality can be defined.","Because polysemanticity is defined through class labels, concepts that appear in many classes or none, such as textures or shapes, may evade detection; an unsupervised variant based on clustering activation patterns, rather than classes, would test whether the class-based definition is essential or merely convenient.","The near-zero residual in the paper's experiments suggests a testable claim: for penultimate-layer channels at gamma=0.5, pairwise concept separation is almost complete, which could be checked by applying the procedure to all flagged channels and measuring the distribution of residual activations."],"forward_implications":["Feature visualizations such as MACO applied to the disentangled channels should show one concept each rather than a blend; the paper demonstrates this for the 'digital clock' and 'cauliflower' channel #1660.","The edited network's outputs are unchanged by construction, so the procedure can be applied to deployed pretrained models without retraining or fine-tuning.","gamma-polysemanticity gives a quantitative criterion for when a channel is worth splitting, and the relative-activation table gives a quantitative check of whether the split succeeded.","The construction is not tied to ResNet-50: any CNN layer where Input×Gradient attributions can be computed is eligible, so the method can be transferred to earlier layers or other architectures.","Multi-concept channels beyond two concepts can be handled by recursive or repeated application, since the paper decomposes multi-concept polysemanticity into pairwise cases."],"supporting_citations":[{"why":"Supplies the superposition and non-privileged-basis account of why channels become polysemantic, motivating the entire approach.","marker":"[6]"},{"why":"Closest prior work that performs a virtual disentanglement; the paper's explicit rewrite is designed to go beyond it.","marker":"[5]"},{"why":"Introduces feature visualization by optimization, the explanatory technique that the explicit disentanglement is meant to improve.","marker":"[7]"},{"why":"The Input×Gradient attribution rule used to define the average relevance vectors that drive both detection and splitting.","marker":"[22]"},{"why":"Defines the ResNet-50 architecture on which the method is implemented and evaluated.","marker":"[9]"},{"why":"Provides the ImageNet dataset used for computing class statistics and for evaluation.","marker":"[3]"},{"why":"Provides the MACO feature-visualization method used to produce qualitative evidence of disentangled concepts.","marker":"[8]"},{"why":"Provides a semantic-similarity measure used to choose the polysemanticity threshold gamma=0.5.","marker":"[4]"}],"fun_headline_variants":["CNN channels get a divorce: mixed concepts split cleanly","One concept per channel: new algorithm splits CNN features","Split CNN channels without changing predictions","Giving every concept its own channel in CNNs","Polysemantic channels: now each concept gets its own"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that the two concepts are cleanly separable in the immediately preceding layer: each previous-layer channel must be attributable mostly to one concept, so cutting edges by the scaled ARV ($\\mathrm{ARV}(l,c,t_2)_i\\cdot\\rho > \\mathrm{ARV}(l,c,t_1)_i$) comparison isolates the concepts rather than removing shared or distributed features that the residual channel cannot compensate for.","fun_headline_variants_meta":{"raw":{"variants":["CNN channels get a divorce: mixed concepts split cleanly","One concept per channel: new algorithm splits CNN features","Split CNN channels without changing predictions","Giving every concept its own channel in CNNs","Polysemantic channels: now each concept gets its own"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000582,"raw_usage":{"total_tokens":2707,"prompt_tokens":881,"completion_tokens":1826,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":1751}},"tokens_in":497,"tokens_out":1826,"duration_ms":13860,"temperature":1.0,"reasoning_tokens":1751,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:19:03.209651+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a channel flagged as $\\gamma$-polysemantic, apply the split with the paper's automatic $\\rho$ selection, and feed held-out class images through the edited network. If the residual channel's mean absolute activation is comparable to the original channel's activation rather than near zero, or if the concept-matched disentangled channel responds weakly on its own class or strongly on the other, the 'only activates for one concept' claim is falsified. Comparing the original and edited logits also settles the recovery guarantee: any material change contradicts the claim that the original activation is exactly recoverable.","supporting_citations":[{"cited_title":"Toy models of superposition","cited_arxiv_id":null,"evidence_quote":"Supplies the superposition and non-privileged-basis account of why channels become polysemantic, motivating the entire approach."},{"cited_title":"PURE: Turning polysemantic neurons into pure features by identifying rele- vant circuits","cited_arxiv_id":null,"evidence_quote":"Closest prior work that performs a virtual disentanglement; the paper's explicit rewrite is designed to go beyond it."},{"cited_title":"Bengio, Aaron Courville, and Pascal Vin- cent","cited_arxiv_id":null,"evidence_quote":"Introduces feature visualization by optimization, the explanatory technique that the explicit disentanglement is meant to improve."},{"cited_title":"Learning important features through propagating activation differences","cited_arxiv_id":null,"evidence_quote":"The Input×Gradient attribution rule used to define the average relevance vectors that drive both detection and splitting."},{"cited_title":"Deep residual learning for image recognition","cited_arxiv_id":null,"evidence_quote":"Defines the ResNet-50 architecture on which the method is implemented and evaluated."},{"cited_title":"ImageNet: A large-scale hierarchical image database","cited_arxiv_id":null,"evidence_quote":"Provides the ImageNet dataset used for computing class statistics and for evaluation."},{"cited_title":"Unlocking feature visualization for deep net- work with magnitude constrained optimization","cited_arxiv_id":null,"evidence_quote":"Provides the MACO feature-visualization method used to produce qualitative evidence of disentangled concepts."},{"cited_title":"Visual and semantic similarity in ImageNet","cited_arxiv_id":null,"evidence_quote":"Provides a semantic-similarity measure used to choose the polysemanticity threshold gamma=0.5."}],"review_version":1}