Pith. sign in

REVIEW 4 major objections 5 minor 40 references

Expanding a trained network's neuron count — while keeping its non-zero parameter count fixed — improves accuracy by reducing feature interference, whether the expansion is structured or random.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 11:28 UTC pith:QRJCZMZ3

load-bearing objection Nice idea and plausible theory, but the headline accuracy gains are confounded by unequal training budgets; needs a same-epoch control before the central claim is credible. the 4 major comments →

arxiv 2510.04500 v3 pith:QRJCZMZ3 submitted 2025-10-06 cs.LG

Expand Neurons, Not Parameters

Linghao Kong , Inimai Subramanian , Yonadav Shavit , Micah Adler , Dan Alistarh , Nir Shavit This is my paper
classification cs.LG
keywords polysemanticitysuperpositionneuron splittingFixed Parameter Expansionsparse networksfeature interferencefeature capacityGram matrix
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that a network's performance bottleneck is often not its parameter count but feature collisions: when several features are forced to share one neuron, they interfere. Fixed Parameter Expansion (FPE) replaces each trained neuron with several sub-neurons that inherit disjoint slices of its incoming weights, then re-prunes the output layer so the total non-zero parameter count is unchanged. Across Boolean satisfiability tasks, classifiers over fixed pretrained embeddings, and deeper multilayer networks, the widened-sparse model consistently outscores the dense baseline, and measured polysemanticity drops. Even random weight splits capture most of the gain, pointing to reduced collisions rather than precise feature assignment as the driver.

Core claim

The central claim is that increasing the number of neurons at a fixed non-zero parameter budget improves test accuracy by reducing interference between features that previously shared neurons. The paper demonstrates this with FPE: each dense neuron is duplicated α times and its incoming weight vector is partitioned into α disjoint masks, so no two sub-neurons share an input connection; the wider output layer is re-sparsified by pruning smallest-magnitude weights. On 4-literal monotone read-once DNF Boolean tasks, clause-aligned splits raise accuracy from 78.7% to 99.4% (random splits to 88.7%) at the same parameter count, with feature capacity rising and neuron cosine similarity falling. The

What carries the argument

Fixed Parameter Expansion (FPE): replacing a trained dense neuron with α sub-neurons that inherit disjoint subsets of its incoming weights, then pruning the smallest-magnitude weights in the duplicated output layer to restore the original non-zero parameter budget. The supporting theoretical analysis shows that a randomly sparsified FPE layer keeps clause coverage with high probability (miss probability bounded by C·e^{-r/α^{k-1}}) while cutting expected clause-collisions by roughly α^{-(2k-1)}. The Gram matrix of the first layer, feature capacity, and neuron cosine similarity serve as the empirical measures of interference.

Load-bearing premise

The headline comparisons assume a fair training budget: dense baselines are trained for pre-training epochs only, while FPE models get those same epochs plus additional post-split training, so the reported gains may partly reflect extra training time rather than neuron splitting alone.

What would settle it

Train a dense baseline for the full combined epoch count (pre-train plus post-split epochs) and compare against FPE on the same tasks; if accuracy matches or exceeds the FPE model, the central claim fails. Alternatively, split neurons but then re-merge them into the dense architecture (keeping the same weights) and verify whether the accuracy gain disappears, isolating the width-plus-sparsity structure as the cause.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • At fixed non-zero parameters, wider-sparse networks can match or exceed the accuracy of narrower dense ones; in one CIFAR-100 experiment, a 32-neuron FPE model matched a dense model with ≥1.2× parameters.
  • Reducing polysemanticity is not merely an interpretability nicety — it is a measurable performance mechanism, since accuracy gains track feature-capacity and orthogonality improvements.
  • Because gains concentrate in high-interference regimes, FPE is most useful for small, overstuffed models and becomes a smaller lever as width grows.
  • FPE composes with practical deployment techniques: dynamic mask rewiring adds further gains, and 2:4 structured sparsity performs comparably to unstructured random splits on CIFAR-100.
  • The benefits appear without knowing true feature structure — random splits approximate clause-aligned ones — simplifying application to settings where features are unknown.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the interference story is right, the same split operation should transfer to attention heads or residual-stream channels in transformers, where superposition-like sharing also occurs — a testable extension the paper does not run.
  • The training-budget asymmetry (dense baseline trained for fewer total epochs than the pretrain-then-split FPE model) means the headline gains could shrink under an equal-total-epoch control; the paper's Section 3.4 claim that even a late split wins at equal training lacks a detailed protocol.
  • FPE suggests a cheap finetune-time intervention on already-trained compact models: a deployable accuracy lift with no added memory, which could complement quantization or distillation pipelines.
  • The plateau past roughly 16 clauses hints at a capacity ceiling for splitting; identifying when expansion stops helping (perhaps when sub-neuron degree falls below feature size) could turn FPE into a principled width-selection rule.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Fixed Parameter Expansion (FPE), a procedure that replaces each hidden neuron in a trained dense network with α sub-neurons inheriting disjoint subsets of the parent's incoming weights, and then resparsifies the output layer to preserve the total number of non-zero weights. The authors claim that increasing neuron count at a fixed non-zero parameter count improves test accuracy, and that this improvement is associated with reduced feature interference. They support this with experiments on Boolean DNF satisfiability, classifiers over CLIP embeddings (FashionMNIST, CIFAR-100, ImageNet-100/1k), and deeper MLPs, plus a theoretical analysis of random sparse networks.

Significance. If the central claims were established, the paper would be a significant contribution: it would provide a simple, interpretability-motivated mechanism to improve accuracy without increasing parameter count, with potential hardware efficiency benefits. The theoretical calculation that random FPE reduces expected clause collisions is a nice formal intuition, and the appendix includes reproducible algorithmic details and multiple real-world benchmarks. However, the current empirical support is undermined by a training-budget confound in the headline comparisons and by an inconsistency in the parameter-count accounting. These issues are load-bearing for the paper's abstract claim.

major comments (4)
  1. [§3.1, §3.3 (Fig. 5), §3.4] The headline comparisons give FPE strictly more training than the dense baseline. In §3.1 the dense model trains for 1000 epochs; the FPE model pre-trains for 1000 epochs and then continues for another 1000 after splitting. Similarly, Figure 5 reports that the baseline is pre-trained for 20/25 epochs, but the FPE model is then trained further after splitting; no same-total-epoch dense control is shown. The single sentence in §3.4 claiming 'even the later split outperforms the baseline given the same amount of training' lacks protocol details (exact epochs, LR schedule, comparison points). Without a dense baseline trained for the same total number of steps, the reported accuracy gains could reflect extra optimization rather than neuron splitting. This confound directly affects the central claim, so same-total-epoch controls must be supplied for the main experiments.
  2. [§2.3, Algorithm A1] The FPE procedure as described does not actually keep the total number of non-zero parameters fixed. The bias vector b1 is copied to all αh sub-neurons, increasing the bias parameter count from h to αh. The pruning step in Algorithm A1 prunes only W1 and W2; no bias pruning is mentioned. Thus the total non-zero parameter count after FPE is larger than the dense baseline by (α−1)h. This contradicts the abstract's claim of 'without increasing its total number of non-zero parameters' and also affects the 'test accuracy per parameter' plots. The authors should either modify the procedure to keep biases fixed (e.g., share one bias across sub-neurons, or prune biases) or explicitly state that the parameter count excludes biases and adjust the claim accordingly.
  3. [§2.3 vs. Algorithm A1] The text states that expanding W2 'results in (α−1)C excess parameters,' but Algorithm A1 computes Δ ← (α−1)h·C. If W2 ∈ R^{C×h} is expanded to R^{C×(αh)}, the excess is indeed (α−1)hC, not (α−1)C. These two statements are inconsistent by a factor of h. This is not a mere typo: it determines how many weights are pruned to preserve the parameter budget. The authors must reconcile the text with the pseudocode and verify that the reported 'per-parameter' numbers are computed with the correct excess.
  4. [§A.5] The theoretical justification analyzes a randomly constructed sparse network with αr neurons each connecting to d=m/α randomly chosen literals (allowing replacement), and shows coverage probability and collision-reduction ratios. This is not the same object as the FPE procedure, which starts from a trained dense network and splits each neuron's existing weights into disjoint subsets, then fine-tunes. The random-split experiments in the paper use disjoint partitions of the trained weights, not independent random selections from all m literals. Consequently, the theory does not directly apply to the empirical FPE pipeline and should not be cited as its justification without additional arguments or a modified model. At minimum, the mismatch should be acknowledged and the scope of the theoretical claim restricted.
minor comments (5)
  1. [§3.2 (feature capacity formula)] The definition 'C_i = (W·,i·W·,i)^2 / Σ_j (W·,i·W·,j)^2' uses W for activations, but the text then asserts 'in our Boolean setting this is the same as the weight matrix.' For a ReLU network, activations are not literally the weight matrix. Please clarify what is being computed and how it maps onto the stated formula.
  2. [Figure 4c] The caption says 'coefficients of determination indicated,' but the main text does not report the R² values. Please provide the numerical values in the text or caption so readers can assess the strength of the claimed correlations.
  3. [§3.3, Figure 5] For the vision tasks, the number of training epochs after the split is not given in the main text or the appendix. Please specify the full training protocol (pre-training epochs, post-split epochs, learning-rate schedule) for each dataset.
  4. [§3.4] The statement 'even the later split outperforms the baseline given the same amount of training' is not backed by a pointer to a specific figure, table, or protocol. Please provide the supporting experiment and describe how the equal-budget comparison was performed.
  5. [Throughout] The paper uses 'non-zero parameters' and 'non-zero weights' inconsistently. Since biases are handled differently, please define exactly which quantities are counted in the parameter budget and in the 'accuracy per parameter' metrics.

Circularity Check

1 steps flagged

FPE accuracy gains are empirical, but the paper's interference-reduction evidence is partly forced by the disjoint-mask construction.

specific steps
  1. self definitional [Section 2.3 (FPE construction) vs. Section 3.2 (Direct Evidence of Feature Disentanglement) and Figure 4]
    "we duplicate w_i across α sub-neurons ... The input dimension is partitioned into α disjoint masks ... and each mask is applied to one sub-neuron. This ensures no sub-neurons share input features ... The mean cosine similarity is calculated by averaging the cosine similarity between all pairs of distinct neuron weight vectors. Lower similarity is indicative of lower polysemanticity."

    The FPE construction forces sibling sub-neurons to have disjoint input supports, making their inner product and cosine similarity identically zero; cross-parent pairs inherit exactly the mask-imposed overlap fraction (roughly 1/α). The reported drop in mean cosine similarity, the block-diagonal Gram structure, and the associated feature-capacity changes are therefore direct restatements of the disjoint-mask construction rather than independent, emergent properties of the trained network. Presenting these metrics as 'direct evidence of feature disentanglement' measures the intervention itself. The accuracy gain remains an empirical result, but the proposed interference mechanism is largely validated by construction, not by an independent test.

full rationale

The paper's main accuracy result—FPE improves test accuracy over dense baselines at fixed non-zero parameter count—is self-contained and not circular: it is a set of empirical comparisons, and the A.5 combinatorial analysis of random FPE is a valid consequence of the stated mask model, not a fitted quantity. However, the paper's mechanistic claim that FPE 'reduces interference' is partially self-definitional. The cosine-similarity, Gram-matrix, and feature-capacity metrics used as evidence are computed on networks whose sparsity masks already impose disjoint input partitions and therefore suppress cross-feature overlaps by construction. This makes the measured 'disentanglement' partly a restatement of the method, though the performance improvements themselves are not. The self-citations (Adler et al.) are background motivation rather than load-bearing proof, and the unequal training-budget issue is a confound rather than a circularity. Overall, the circularity is partial and localized to the mechanism evidence, not to the central empirical claim.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central empirical claims rest mostly on hand-set experimental hyperparameters (α, pretraining epochs) and on assumptions about what is necessary for a ReLU to compute a clause and about what the dataset generator actually produces. The theoretical section adds no free parameters but uses a with-replacement approximation and an interference proxy. There are no new physical or architectural entities beyond FPE itself.

free parameters (5)
  • expansion factor α = 2, 4, 8
    Chosen per experiment; controls the width/parameter trade-off that the central claim depends on.
  • pre-training epochs W before FPE = 1000 (Boolean), 20/25 (vision)
    Determines how much extra training FPE receives relative to the dense baseline; central to the fairness of the comparison.
  • Boolean positive-sample active-bit range = minOnes=floor(m/4), maxOnes=floor(m/4)+floor(m/8)
    Hand-set in Algorithm A2; controls positive-example sparsity and hence task difficulty.
  • Boolean continuous input ranges = true bits in [3,3.5], false bits in [0,0.5]
    Ad hoc choice to make ReLU training stable; changes input geometry and separability.
  • regularization strengths λ1, λ2 = 1e-7, 1e-5 (Boolean); λorth when used
    Hand-set hyperparameters that affect learned representations and the reported metrics.
axioms (5)
  • domain assumption A neuron can compute a k-literal DNF clause only if it receives all k literals; coverage is necessary for learning the clause.
    Used in A.5 to define coverage probability and collision reduction; the random-split results appear inconsistent with this premise in small Boolean setups.
  • domain assumption Interference, defined as a neuron covering multiple clauses, is a proxy for optimization difficulty and is the primary cause of dense-network underperformance.
    Inherited from the superposition/feature-channel literature (Adler & Shavit; Adler et al.); not independently established here.
  • ad hoc to paper The random FPE construction can be analyzed by sampling each neuron's d literals independently with replacement (A.5: 'allow for replacement').
    Simplifies the coverage/collision calculation but does not match the disjoint-partition construction actually used in experiments.
  • domain assumption Negative examples in the Boolean dataset violate every clause, as stated in §2.2.1.
    The appendix Algorithm A2 heads branch does not implement this; the task may be easier than described.
  • domain assumption The Gram-matrix clustering of W1 in vision tasks recovers feature groups that should stay together.
    Authors later acknowledge it 'likely did not recover the true underlying feature structure' (§3.3), so this assumption is weak but not load-bearing for random-split results.

pith-pipeline@v1.3.0-alltime-deepseek · 15053 in / 23007 out tokens · 185373 ms · 2026-08-04T11:28:26.513535+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Expand Neurons, Not Parameters." pith.science (2026). https://pith.science/paper/QRJCZMZ3

@misc{pith2026251004500,
  author       = {Pith},
  title        = {Pith review of: Expand Neurons, Not Parameters},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QRJCZMZ3}},
  note         = {Machine review of arXiv:2510.04500}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This work demonstrates how increasing the number of neurons in a network without increasing its total number of non-zero parameters improves performance. We show that this gain corresponds with a decrease in interference between multiple features that would otherwise share the same neurons. On symbolic Boolean tasks, splitting each neuron into sparser sub-neurons with knowledge of the clauses systematically reduces polysemanticity metrics and yields higher task accuracy. Notably, even random splits of neuron weights approximate these gains, indicating that reduced collisions, not precise assignment, are a primary driver. Consistent with the superposition hypothesis, the benefits of this framework grow with increasing interference: when polysemantic load is high, accuracy improvements are the largest. Transferring these insights to more realistic models, including classifiers over CLIP embeddings, convolutional neural networks, and deeper multilayer networks, we find that widening networks while maintaining a constant non-zero parameter count consistently increases accuracy. These results identify an interpretability-grounded mechanism to leverage width against superposition, improving performance without increasing the number of non-zero parameters. Such a direction is well matched to modern accelerators, where memory movement of non-zero parameters, rather than raw compute, is often a dominant bottleneck.

Figures

Figures reproduced from arXiv: 2510.04500 by Dan Alistarh, Inimai Subramanian, Linghao Kong, Micah Adler, Nir Shavit, Yonadav Shavit.

Figure 1
Figure 1. Figure 1: Paradigms of parameter efficiency in training and inference. “Train-then-sparsify” min [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Dense and clause-split FPE model Gram and [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Trends of performance under superposition in neurons and clauses. (a) Relative improve [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Changes in feature interference metrics for varying neurons and clauses. (a) Feature [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Fixed Parameter Expansion helps on real datasets like (a) FashionMNIST, (b) CLIP [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: More extensive experiments on CIFAR-100 with random splitting. (a) An FPE model with [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

40 extracted references · 16 linked inside Pith

  1. [1]

    On the complexity of neural computation in superposition

    Micah Adler and Nir Shavit. On the complexity of neural computation in superposition. arXiv preprint arXiv:2409.15318, 2024

    arXiv 2024

  2. [2]

    Towards combinatorial interpretability of neural computation

    Micah Adler, Dan Alistarh, and Nir Shavit. Towards combinatorial interpretability of neural computation. arXiv preprint arXiv:2504.08842, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    The lottery ticket hypothesis for pre-trained bert networks

    Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 33: 0 15834--15846, 2020

    2020

  4. [4]

    The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models

    Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 16306--16316, 2021

    2021

  5. [5]

    Net2net: Accelerating learning via knowledge transfer

    Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.05641, 2015

    Pith/arXiv arXiv 2015

  6. [6]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009

    2009

  7. [7]

    Pure: Turning polysemantic neurons into pure features by identifying relevant circuits

    Maximilian Dreyer, Erblina Purelku, Johanna Vielhaben, Wojciech Samek, and Sebastian Lapuschkin. Pure: Turning polysemantic neurons into pure features by identifying relevant circuits. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 8212--8217, 2024

    2024

  8. [8]

    Toy models of superposition

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022

    Pith/arXiv arXiv 2022

  9. [9]

    Rigging the lottery: Making all tickets winners

    Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International conference on machine learning, pp.\ 2943--2952. PMLR, 2020

    2020

  10. [10]

    Maskllm: Learnable semi-structured sparsity for large language models

    Gongfan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, and Xinchao Wang. Maskllm: Learnable semi-structured sparsity for large language models. arXiv preprint arXiv:2409.17481, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018

    Pith/arXiv arXiv 2018

  12. [12]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp.\ 10323--10337. PMLR, 2023

    2023

  13. [13]

    Multimodal neurons in artificial neural networks

    Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 6 0 (3): 0 e30, 2021

    2021

  14. [14]

    Are wider nets better given the same number of parameters? arXiv preprint arXiv:2010.14495, 2020

    Anna Golubeva, Behnam Neyshabur, and Guy Gur-Ari. Are wider nets better given the same number of parameters? arXiv preprint arXiv:2010.14495, 2020

    Pith/arXiv arXiv 2010

  15. [15]

    Universal neurons in gpt2 language models

    Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons in gpt2 language models. arXiv preprint arXiv:2401.12181, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Learning both weights and connections for efficient neural network

    Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015

    2015

  17. [17]

    Dynamic neural networks: A survey

    Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44 0 (11): 0 7436--7456, 2021

    2021

  18. [18]

    Mathematical models of computation in superposition

    Kaarel H \"a nni, Jake Mendel, Dmitry Vaintrob, and Lawrence Chan. Mathematical models of computation in superposition. arXiv preprint arXiv:2408.05451, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Engineering monosemanticity in toy models

    Adam S Jermyn, Nicholas Schiefer, and Evan Hubinger. Engineering monosemanticity in toy models. arXiv preprint arXiv:2211.09169, 2022

    Pith/arXiv arXiv 2022

  20. [20]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  21. [21]

    What causes polysemanticity? an alternative origin story of mixed selectivity from incidental causes

    Victor Lecomte, Kushal Thaman, Rylan Schaeffer, Naomi Bashkansky, Trevor Chow, and Sanmi Koyejo. What causes polysemanticity? an alternative origin story of mixed selectivity from incidental causes. arXiv preprint arXiv:2312.03096, 2023

    Pith/arXiv arXiv 2023

  22. [22]

    A survey of lottery ticket hypothesis

    Bohan Liu, Zijie Zhang, Peixiong He, Zhensen Wang, Yang Xiao, Ruimeng Ye, Yang Zhou, Wei-Shinn Ku, and Bo Hui. A survey of lottery ticket hypothesis. arXiv preprint arXiv:2403.04861, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Superposition yields robust neural scaling, 2025

    Yizhou Liu, Ziming Liu, and Jeff Gore. Superposition yields robust neural scaling, 2025. URL https://arxiv.org/abs/2505.10465

    Pith/arXiv arXiv 2025

  24. [24]

    Proving the lottery ticket hypothesis: Pruning is all you need

    Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In International Conference on Machine Learning, pp.\ 6682--6691. PMLR, 2020

    2020

  25. [25]

    Analysis of boolean functions

    Ryan O'Donnell. Analysis of boolean functions. Cambridge University Press, 2014

    2014

  26. [26]

    Zoom in: An introduction to circuits

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5 0 (3): 0 e00024--001, 2020

    2020

  27. [27]

    Mixturegrowth: Growing neural networks by recombining learned parameters

    Chau Pham, Piotr Teterwak, Soren Nelson, and Bryan A Plummer. Mixturegrowth: Growing neural networks by recombining learned parameters. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.\ 2800--2809, 2024

    2024

  28. [28]

    Accelerating inference with sparsity using the nvidia ampere architecture and nvidia tensorrt

    Jeff Pool, Abhishek Sawarkar, and Jay Rodge. Accelerating inference with sparsity using the nvidia ampere architecture and nvidia tensorrt. NVIDIA Developer Technical Blog, https://developer. nvidia. com/blog/accelerating-inference-with-sparsityusing-ampere-and-tensorrt, 2021

    2021

  29. [29]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PmLR, 2021

    2021

  30. [30]

    Polysemanticity and capacity in neural networks

    Adam Scherlis, Kshitij Sachan, Adam S Jermyn, Joe Benton, and Buck Shlegeris. Polysemanticity and capacity in neural networks. arXiv preprint arXiv:2210.01892, 2022

    Pith/arXiv arXiv 2022

  31. [31]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    Firefly neural architecture descent: a general approach for growing neural networks

    Lemeng Wu, Bo Liu, Peter Stone, and Qiang Liu. Firefly neural architecture descent: a general approach for growing neural networks. Advances in neural information processing systems, 33: 0 22373--22383, 2020

    2020

  33. [33]

    Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms

    Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017

    Pith/arXiv arXiv 2017

  34. [34]

    An effective gram matrix characterizes generalization in deep networks

    Rubing Yang and Pratik Chaudhari. An effective gram matrix characterizes generalization in deep networks. arXiv preprint arXiv:2504.16450, 2025

    arXiv 2025

  35. [35]

    Growing efficient deep networks by structured continuous sparsification

    Xin Yuan, Pedro Savarese, and Michael Maire. Growing efficient deep networks by structured continuous sparsification. arXiv preprint arXiv:2007.15353, 2020

    Pith/arXiv arXiv 2007

  36. [36]

    Deconstructing lottery tickets: Zeros, signs, and the supermask

    Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. Advances in neural information processing systems, 32, 2019

    2019

  37. [37]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  38. [38]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  39. [39]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  40. [40]

    I D ocٙ\^9;g=;;3g DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD suDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD䘘 TDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD &&

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv 2080