REVIEW 3 major objections 4 minor 42 references
An open dataset of neural networks for hypernetwork research
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper releases 10,000 trained LeNet-5 binary classifiers, grouped by the ImageNette class they recognize, and shows that a Naive Bayes classifier can identify a network's target class from its weight vector with 72.0% accuracy.
desk verdict Useful public dataset of 10k LeNet-5 weight vectors, but the paper's central description and generative-infrastructure claims substantially outrun what Table 1 and the experiments support. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the class-labeled collection of weight vectors: 10 groups of 1,000 flattened parameter tensors from LeNet-5, a small convolutional network with 91,481 parameters per model, where the label is the ImageNette class the network was trained to detect. The mechanism that gives the dataset its structure is the one-versus-all training protocol: every network uses the same architecture and the same image source, and only the target class and the random draw of 'other' images vary, so any systematic difference in weights is attributable to the classification problem the network solves. The classification baseline treats each 91,481-dimensional weight vector as a feature vector; the measured 72.0% accuracy of a Naive Bayes classifier is the evidence the paper offers that these class-conditional distributions are distinguishable.
What would settle it
Train a simple generative model on the released weight vectors and load its sampled outputs into the LeNet-5 architecture; if those generated networks classify their intended ImageNette class at or near chance while the stored 10,000 weights remain separable at 72%, the claim that the distribution is learnable for generation would be refuted.
Extended reading notes
Core claim
The dataset consists of $10^4$ trained LeNet-5 binary classifiers, each recognizing one of ten Imagenette V2 classes (tench, English springer, cassette player, chain saw, church, French horn, garbage truck, gas pump, golf ball, parachute) against randomly sampled images from all other classes. Each network has 91,481 trainable parameters, trained for 25 epochs to an average accuracy of about 91.5%, and every network differs because the images of the 'other' class are drawn randomly. The weights are released in two arrangements: modelwise, as flattened per-model tensors, and layerwise, grouped by class and layer. The paper's headline empirical result is that a classifier applied to the flattened weights can predict which of the 10 classes a network was trained to recognize with 72.0% accuracy ($p < 10^{-5}$), which the authors read as evidence that the class-conditional weight distributions are distinct and learnable, and therefore suitable for training generative hypernetworks.
Load-bearing premise
The load-bearing premise is that 72% separability of stored weight vectors by a simple classifier implies a generative model can learn and sample the class-conditional weight distributions; if that inference is false, the dataset remains useful but the paper's central motivation for it weakens.
Editorial extensions
If this is right
- Hypernetwork and GAN researchers can use the 10,000 weight vectors as training examples, with the 72.0% Naive Bayes result serving as a baseline discriminator that future generative models should improve upon.
- Because every model shares the same architecture, image source, training length, and balanced class structure, differences in weights can be studied as a function of the target class and the random subsampling of negative images.
- The modelwise and layerwise formats support both whole-vector generation and layer-by-layer generative modeling, matching how hypernetworks often predict weights in parts.
- The dataset's open release and generation code allow the same protocol to be extended to new ImageNette classes, giving a route to larger or more fine-grained hypernetwork training sets.
- A classifier accuracy far above chance is presented as evidence that a neural-network discriminator is feasible, which is a necessary component of any GAN that generates network weights.
Reading between the lines
- The paper does not train a generator, so its central motivation rests on an extrapolation: separability by a flat classifier does not by itself prove that a hypernetwork can sample valid weights; the decisive test would be whether generated weights, loaded into LeNet-5, classify the intended class above chance.
- The paper's own Table 1 points to a complication it does not discuss: average recall ranges from 0.068 for chain saw to 0.587 for tench, so a generative model may need class-conditional capacity or rebalancing to cover rare classes.
- A testable extension would be to ablate the 72% classifier: if accuracy is driven mainly by a few layers or by simple statistics such as the sparsity of near-zero weights, then the weight distributions may be easier for a generator to match than the full 91,481-dimensional geometry suggests.
- Because the dataset is limited to one small architecture, a hypernetwork trained on it would likely need retraining or adaptation before generating weights for deeper networks such as ResNet, so the resource is best read as a controlled starting point rather than a general weight-space corpus.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a publicly released dataset of 10,000 LeNet-5 networks trained for binary one-vs-all image classification on the ImageNette V2 dataset, organized into 10 classes of 1,000 networks each. The authors report that a Naive Bayes classifier can identify the target class of a network from its 91,481 flattened weights with 72% accuracy, and they argue that this separability indicates the dataset can support hypernetwork research for generating neural networks. The dataset and code are publicly available. The central claims are that each class contains networks that can identify the corresponding ImageNette class and that the weight-space classification result demonstrates the dataset's utility for generative modeling of network weights.
Significance. If the dataset were as described, it would be a valuable public resource for weight-space learning and hypernetwork research, reflecting substantial computational investment. However, the paper's own Table 1 undermines the dataset's defining property: many of the networks do not actually identify their target class, as evidenced by extremely low recall values. The main experimental result (72% Naive Bayes accuracy) is presented without controlling for obvious confounds and does not establish that the class labels correspond to the networks' functional behavior. The paper's significance therefore rests on a claim that the data themselves contradict.
major comments (3)
- [Section 3 / Table 1] The abstract and Section 3 state that each class contains 1,000 neural networks that can identify the images of that class, but Table 1 reports average recall of only 0.324 overall, with chainsaw at 0.068, French horn at 0.186, and gas pump at 0.151, and per-class minimum recall as low as 0.008. Because each binary training set contains only 9–10% target-class images, a network that always predicts the majority class achieves roughly 90% accuracy with zero recall; the reported values therefore imply that many dataset instances are degenerate classifiers that never identify the target class. This directly contradicts the paper's central description of the dataset and calls into question whether the class labels are meaningful for the stated hypernetwork purpose.
- [Section 5.2 / Table 4] The 72% Naive Bayes accuracy on 91,481-dimensional weight vectors is not shown to reflect the networks' ability to identify the target class. The paper does not control for confounds such as final training loss, random training-set composition, initialization, or amount of optimization, and the observation that Logistic Regression attains exactly 10% (chance) while Naive Bayes attains 72% suggests the signal may reside in per-feature variance rather than in a usable class-conditional structure. A permutation test, a prediction of held-out accuracy from weights, or a classifier trained only on networks that pass a recall threshold would be needed to support the claim that the classification result reflects class-related information.
- [Section 6] The Discussion infers that separability of weight vectors by a flat classifier indicates that the class-conditional weight distributions 'can be produced by generative AI' for hypernetworks. No generative model is trained or evaluated, and separability of point clouds does not imply sampleability by a generator, particularly in a 91,481-dimensional space. This is a logical leap that overstates the dataset's readiness for hypernetwork research and should be either removed or supported by explicit generative experiments.
minor comments (4)
- [Section 6] The statement that 'Weights in a neural network are independent values' is incorrect; weights of a trained network are strongly correlated due to the optimization process. This claim should be corrected or removed.
- [Table 4] Logistic Regression is reported at 10% accuracy, exactly the chance level for a balanced 10-class problem, which is suspicious and should be checked for an implementation or scaling issue.
- [Abstract and Section 3] The notation '10 4' should be rendered as '10^4' for clarity.
- [Acknowledgments] The acknowledgments thank 'the three reviewers,' which appears to be an artifact of the submission process and should not be included in a published manuscript.
Circularity Check
No circularity: the dataset construction and the 72% classification benchmark are self-contained; the generative-AI inference is an extrapolation, not a result that reduces to its inputs.
full rationale
The paper's load-bearing empirical claim is the 72.0% Naive Bayes accuracy on flattened LeNet-5 weight vectors (Section 5.2, Table 4). This is a measured out-of-sample classification result: 70% of the 10,000 weight vectors were used to train the classifier and the rest for testing (Section 5.1), so the accuracy is not a fitted parameter masquerading as a prediction. The class labels are defined by the ImageNette target class used in training each model (Section 3), an external input, not by the classifier's own outputs. The Discussion's statement that separability 'provides an indication' that the distributions 'can be produced by generative AI' is an extrapolation to a capability that is not trained or evaluated, and the claim in Section 6 that 'weights in a neural network are independent values' is an unsupported assumption; both are validity/correctness concerns, not circular reductions. The paper also explicitly acknowledges limitations (Section 7: small size, single architecture), and its self-citations (Shamir et al. 2008; Eze and Shamir 2024) appear only in a list of benchmark datasets and are not load-bearing. Table 1's low per-class recall contradicts the abstract's description of models that 'can identify' the target class, but this is a dataset-quality issue independent of whether the classification benchmark is circular. No equation, fitted value, or citation chain connects the output back to the input by construction, so no circular step is present.
Assumptions & free parameters
free parameters (3)
- Dataset training epochs =
25
- Positive-class sampling ratio =
9-10%
- DNN baseline hyperparameters =
3 layers [256,128,64], batch norm, ReLU, dropout 0.6
assumptions (4)
- domain assumption Imagenette V2 labels and images provide reliable one-vs-all class definitions.
- ad hoc to paper A high-accuracy model on a 9-10% positive binary task is a network that identifies the target class.
- ad hoc to paper Separability of weight vectors by a flat classifier implies a hypernetwork or GAN can learn and sample the class-conditional weight distribution.
- ad hoc to paper Weights of a trained neural network are independent values.
Cite this review
Pith. "Pith review of An open dataset of neural networks for hypernetwork research." pith.science (2026). https://pith.science/paper/XUEJNPTS
@misc{pith2026250715869,
author = {Pith},
title = {Pith review of: An open dataset of neural networks for hypernetwork research},
year = {2026},
howpublished = {\url{https://pith.science/paper/XUEJNPTS}},
note = {Machine review of arXiv:2507.15869}
}
abstract
Despite the transformative potential of AI, the concept of neural networks that can produce other neural networks by generating model weights (hypernetworks) has been largely understudied. One of the possible reasons is the lack of available research resources that can be used for the purpose of hypernetwork research. Here we describe a dataset of neural networks, designed for the purpose of hypernetworks research. The dataset includes $10^4$ LeNet-5 neural networks trained for binary image classification separated into 10 classes, such that each class contains 1,000 different neural networks that can identify a certain ImageNette V2 class from all other classes. A computing cluster of over $10^4$ cores was used to generate the dataset. Basic classification results show that the neural networks can be classified with accuracy of 72.0%, indicating that the differences between the neural networks can be identified by supervised machine learning algorithms. The ultimate purpose of the dataset is to enable hypernetworks research. The dataset and the code that generates it are open and accessible to the public.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
K., Zhou, J., Lu, P., Molaei, S., and Clifton, D
Chauhan, V. K., Zhou, J., Lu, P., Molaei, S., and Clifton, D. A. (2024). A brief review of hypernetworks in deep learning. Artificial Intelligence Review , 57(9):1--29
work page 2024
-
[2]
Cohen, G., Afshar, S., Tapson, J., and Van Schaik, A. (2017). EMNIST : Extending MNIST to handwritten letters. In International Joint Conference on Neural Networks , pages 2921--2926. IEEE
work page 2017
-
[3]
Cui, C., Wang, W., Zhang, M., Chen, G., Luo, Z., and Ooi, B. C. (2021). Alphaevolve: A learning framework to discover novel alphas in quantitative investment. In Proceedings of the 2021 International Conference on Management of Data , pages 2208--2216
work page 2021
-
[4]
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition , pages 248--255. IEEE
work page 2009
-
[5]
Dueben, P. D., Schultz, M. G., Chantry, M., Gagne, D. J., Hall, D. M., and McGovern, A. (2022). Challenges and benchmark datasets for machine learning in the atmospheric sciences: Definition, status, and outlook. Artificial Intelligence for the Earth Systems , 1(3):e210002
work page 2022
-
[6]
Continual Learning in Recurrent Neural Networks
Ehret, B., Henning, C., Cervera, M. R., Meulemans, A., von Oswald, J., and Grewe, B. F. (2021). Continual learning in recurrent neural networks. arXiv:2006.12109
work page Pith review arXiv 2021
-
[7]
Eilertsen, G., J \"o nsson, D., Ropinski, T., Unger, J., and Ynnerman, A. (2020). Classifying the classifier: dissecting the weight space of neural networks. In 24th European Conference on Artificial Intelligence , pages 1119--1126. IOS Press
work page 2020
-
[8]
Eze, C. S. and Shamir, L. (2024). Analysis and prevention of ai-based phishing email attacks. Electronics , 13(10):1839
work page 2024
Show all 42 references
-
[9]
Friedman, N., Geiger, D., and Goldszmidt, M. (1997). Bayesian network classifiers. Machine learning , 29:131--163
1997
-
[10]
Gong, Y., Liu, G., Xue, Y., Li, R., and Meng, L. (2023). A survey on dataset quality in machine learning. Information and Software Technology , 162:107268
2023
-
[11]
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2020). Generative adversarial networks. Communications of the ACM , 63(11):139--144
2020
-
[12]
J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Advances in Neural Information Processing Systems , 27
2014
-
[13]
Ha, D., Dai, A., and Le, Q. V. (2016). Hypernetworks. arXiv:1609.09106
2016 arXiv
-
[14]
Huang, Y., Xie, K., Bharadhwaj, H., and Shkurti, F. (2021). Continual model-based reinforcement learning with hypernetworks. IEEE International Conference on Robotics and Automation
2021
-
[15]
Keynan, S., Sarafian, E., and Kraus, S. (2021). Recomposing the reinforcement learning building blocks with hypernetworks. Proceedings of the 38 th International Conference on Machine Learning , 139:9301--9312
2021
-
[16]
S., Beigpour, S., Van de Weijer, J., and Felsberg, M
Khan, F. S., Beigpour, S., Van de Weijer, J., and Felsberg, M. (2014). Painting-91: a large scale database for computational painting categorization. Machine vision and applications , 25:1385--1397
2014
-
[17]
and Yang, Y
Klimt, B. and Yang, Y. (2004). The enron corpus: A new dataset for email classification research. In European Conference on Machine Learning , pages 217--226. Springer
2004
-
[18]
W., and Romero Soriano, A
Knyazev, B., Drozdzal, M., Taylor, G. W., and Romero Soriano, A. (2021). Parameter prediction for unseen deep architectures. Advances in Neural Information Processing Systems , 34:29433--29448
2021
-
[19]
Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images
2009
-
[20]
Krueger, D., Huang, C.-W., Islam, R., Turner, R., Lacoste, A., and Courville, A. (2017). Bayesian hypernetworks. arXiv:1710.04759
2017 arXiv
-
[21]
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278--2324
1998
-
[22]
Li, H., Gao, Q., and Zhang, S. (2023). Assessing and improving dataset and evaluation methodology in deep learning for code clone detection. In 34th International Symposium on Software Reliability Engineering , pages 497--508. IEEE
2023
-
[23]
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. (2014). Microsoft coco: Common objects in context. In 13th European Conference on Computer Vision , pages 740--755. Springer
2014
-
[24]
P., and Lanckriet, G
McFee, B., Bertin-Mahieux, T., Ellis, D. P., and Lanckriet, G. R. (2012). The million song dataset challenge. In Proceedings of the 21st International Conference on World Wide Web , pages 909--916
2012
-
[25]
Moscato, V., Picariello, A., and Sperl \' , G. (2021). A benchmark of machine learning approaches for credit score prediction. Expert Systems with Applications , 165:113986
2021
-
[26]
Navon, A., Shamsian, A., Achituve, I., Fetaya, E., Chechik, G., and Maron, H. (2023). Equivariant architectures for learning in deep weight spaces. In International Conference on Machine Learning , pages 25790--25816. PMLR
2023
-
[27]
J., Wechsler, H., Huang, J., and Rauss, P
Phillips, P. J., Wechsler, H., Huang, J., and Rauss, P. J. (1998). The FERET database and evaluation procedure for face-recognition algorithms. Image and Vision Computing , 16(5):295--306
1998
-
[28]
Samaria, F. S. (1994). Face recognition using hidden Markov models . PhD thesis, PhD Dissertation, University of Cambridge Cambridge, UK
1994
-
[29]
Sch \"u rholt, K., Knyazev, B., Gir \'o -i Nieto, X., and Borth, D. (2022a). Hyper-representations as generative models: Sampling unseen neural network weights. Advances in Neural Information Processing Systems , 35:27906--27920
2022
-
[30]
Sch \"u rholt, K., Kostadinov, D., and Borth, D. (2021). Self-supervised representation learning on neural network weights for model characteristic prediction. Advances in Neural Information Processing Systems , 34:16481--16493
2021
-
[31]
Sch \"u rholt, K., Taskiran, D., Knyazev, B., Gir \'o -i Nieto, X., and Borth, D. (2022b). Model zoos: A dataset of diverse populations of neural network models. Advances in Neural Information Processing Systems , 35:38134--38148
2022
-
[32]
J., and Goldberg, I
Shamir, L., Orlov, N., Mark Eckley, D., Macura, T. J., and Goldberg, I. G. (2008). IICBU 2008: a proposed benchmark suite for biological image analysis. Medical & Biological Engineering & Computing , 46:943--947
2008
-
[33]
J., Liang, J
Singh, V., Pencina, M., Einstein, A. J., Liang, J. X., Berman, D. S., and Slomka, P. (2021). Impact of train/test sample regimen on performance estimate stability of machine learning in cardiovascular imaging. Scientific Reports , 11(1):14490
2021
-
[34]
Sinka, M. P. and Corne, D. W. (2002). A large benchmark dataset for web document clustering. Soft Computing Systems: Design, Management and Applications , 87:881--890
2002
-
[35]
Thiyagalingam, J., Shankar, M., Fox, G., and Hey, T. (2022). Scientific machine learning benchmarks. Nature Reviews Physics , 4(6):413--420
2022
-
[36]
Tschalzev, A., Purucker, L., L \"u dtke, S., Hutter, F., Bartelt, C., and Stuckenschmidt, H. (2025). Unreflected use of tabular data repositories can undermine research quality. arXiv:2503.09159
2025 arXiv
-
[37]
Unterthiner, T., Keysers, D., Gelly, S., Bousquet, O., and Tolstikhin, I. (2020). Predicting neural network accuracy from weights. arXiv:2002.11448
2020 arXiv
-
[38]
F., and Sacramento, J
von Oswald, J., Henning, C., Grewe, B. F., and Sacramento, J. (2022). Continual learning with hypernetworks. arXiv:1906.00695
2022 arXiv
-
[39]
N., Gomes, J., Geniesse, C., Pappu, A
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., Leswing, K., and Pande, V. (2018). Moleculenet: a benchmark for molecular machine learning. Chemical science , 9(2):513--530
2018
-
[40]
H., Cha, J., Kim, C., and Kim, T
Yoo, Y. H., Cha, J., Kim, C., and Kim, T. (2024). Hyper-cl: Conditioning sentence representations with hypernetworks. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics , 1:700--711
2024
-
[41]
Zhang, C., Ren, M., and Urtasun, R. (2019). Graph hypernetworks for neural architecture search. International Conference on Learning Representations , page 871
2019
-
[42]
Zhang, C., Ren, M., and Urtasun, R. (2020). Graph hypernetworks for neural architecture search
2020
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.