{"id":"54eea402-6ba8-4dea-b39d-511d40f4dac3","arxiv_id":"2606.04583","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"HalfNet learns low-rank geometry for random weight distributions to match full MLP performance with fewer parameters on image tasks.","lead":"HalfNet learns a low-rank factorization of covariance Σ to draw random neural network weights from N(0, Σ) instead of standard normal. This matches fully trained MLP accuracy on MNIST and CIFAR-10 while using substantially fewer parameters by capturing weight-space geometry from data.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Learned low-rank factorization of Σ may be an artifact of supervised optimization on the target task rather than revealing intrinsic weight-space geometry","rationale":"The reader’s weakest_assumption already isolates the precise risk that the learned geometry could be an artifact of the training procedure or dataset. The full-text description of HalfNet as a “data-dependent random embedding” that admits a supervised-metric-learning view confirms rather than removes this risk. Because the manuscript supplies no ablation that decouples the supervised learning of Σ from the subsequent random embedding, the UNVERDICTED verdict with high correctness_risk is appropriate and requires no adjustment.","tokens_in":1689,"tokens_out":415,"duration_ms":44749,"concrete_test":"Reproduce the MNIST/CIFAR-10 HalfNet experiments but replace the learned low-rank factors of Σ with the top-r principal components of the weight matrices taken from an independently trained standard MLP on the identical dataset and architecture; if test accuracy drops by more than 3–5 points relative to the end-to-end learned-Σ version while keeping the same head and sampling procedure, the claim that geometry alone suffices weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that sampling fixed weights from N(0, Σ) with data-learned low-rank Σ, plus a small trainable head, recovers full-MLP performance because the geometry itself (not the precise weight values) carries the predictive power. Because Σ is obtained by learning a low-rank factorization directly from the labeled training data (explicitly noted as interpretable via supervised metric learning), the optimization of the factors can embed task-specific directions that a post-hoc spectral analysis of trained networks would also exhibit. This creates a circularity: the geometry is not shown to be an independent property of neural weight spaces; it is co-optimized with the embedding. Consequently the spectral analysis and the scaling-with-rank observation do not yet separate “geometry matters” from “the particular supervised procedure used to obtain the geometry matters.”","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proposes HalfNet, in which a subset of network weights are drawn randomly from N(0, Σ) where Σ admits a learned low-rank factorization obtained from the labeled training data. Experiments on MNIST and CIFAR-10 are reported to show that the resulting model matches the accuracy of fully trained MLPs while using substantially fewer trainable parameters. Spectral analysis is presented to argue that predictive power resides primarily in the geometry of the weight-space distribution rather than in the precise values of individual weights, with accuracy observed to scale smoothly with the rank of the factorization. The construction is also interpreted as a data-dependent random embedding, a supervised metric-learning procedure, or a random-feature model.","tokens_in":1850,"tokens_out":664,"duration_ms":22773,"significance":"If the central empirical and spectral claims hold after the circularity concern is addressed, the work would provide concrete evidence that a learned low-dimensional subspace geometry can substitute for full weight optimization in MLPs. This would strengthen the random-feature and kernel perspectives on neural networks and offer a practical route to parameter-efficient training. The smooth rank-accuracy scaling and the explicit low-rank factorization are potentially falsifiable predictions that could be tested on additional architectures and datasets.","major_comments":[{"comment":"§3 (learning procedure for Σ): the low-rank factors of Σ are obtained by supervised optimization on the target labeled data. This procedure can embed task-specific directions into the geometry, so the subsequent claim that 'much of the predictive power lies in the geometry rather than the precise values' risks circularity; the geometry is co-optimized with the embedding. A control experiment that learns Σ without labels (or from a disjoint task) and then freezes it is required to separate intrinsic geometry from supervised artifact.","section":"§3"},{"comment":"Experiments section and Table 1/2: the manuscript states that HalfNet 'matches' fully trained MLPs on MNIST and CIFAR-10, yet supplies no numerical accuracies, standard deviations across runs, baseline architectures with matched parameter counts, or exclusion criteria. Without these quantities the scaling-with-rank observation cannot be assessed for statistical reliability or compared to the reader's weakest assumption.","section":"Experiments"},{"comment":"Spectral analysis (presumably §4): the claim that geometry dominates is supported only by post-hoc analysis of the learned Σ. Because Σ is itself the output of the same supervised procedure, the spectral decomposition may simply recover the directions already favored by gradient descent rather than revealing an independent property of neural weight spaces.","section":"§4"}],"minor_comments":[{"comment":"Abstract: the phrase 'substantially fewer parameters' is used without a concrete ratio or reference to the corresponding table/figure.","section":"Abstract"},{"comment":"Notation: the precise definition of the low-rank factorization (e.g., Σ = AAᵀ with A ∈ ℝ^{d×r}) and how the random weights are sampled and frozen should be stated once in a single equation block for reproducibility.","section":"§2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed and constructive report. We address each major comment below and indicate where revisions to the manuscript are planned.","responses":[{"response":"We acknowledge the validity of the circularity concern. The supervised optimization of the low-rank factors of Σ does incorporate task-specific information. To separate the contribution of the learned geometry from supervised artifacts, we will add a control experiment in the revised manuscript in which Σ is learned either from unlabeled data statistics or from a disjoint task and then held fixed while sampling the random weights for HalfNet. This addition will directly test whether the geometry carries predictive power independent of the specific supervised optimization used in the main experiments.","revision_made":"yes","referee_comment":"[§3] §3 (learning procedure for Σ): the low-rank factors of Σ are obtained by supervised optimization on the target labeled data. This procedure can embed task-specific directions into the geometry, so the subsequent claim that 'much of the predictive power lies in the geometry rather than the precise values' risks circularity; the geometry is co-optimized with the embedding. A control experiment that learns Σ without labels (or from a disjoint task) and then freezes it is required to separate intrinsic geometry from supervised artifact."},{"response":"We agree that the experimental reporting requires greater detail for proper evaluation. Although the manuscript contains the underlying results, the presentation in the Experiments section and Tables 1/2 will be revised to include explicit mean accuracies, standard deviations computed over multiple independent runs, additional baseline models whose total parameter counts are matched to those of the HalfNet variants, and any relevant exclusion criteria. These changes will allow readers to assess the statistical reliability of the rank-accuracy scaling.","revision_made":"yes","referee_comment":"[Experiments] Experiments section and Table 1/2: the manuscript states that HalfNet 'matches' fully trained MLPs on MNIST and CIFAR-10, yet supplies no numerical accuracies, standard deviations across runs, baseline architectures with matched parameter counts, or exclusion criteria. Without these quantities the scaling-with-rank observation cannot be assessed for statistical reliability or compared to the reader's weakest assumption."},{"response":"The spectral analysis is indeed performed after Σ has been learned via the supervised procedure. Nevertheless, the central empirical support for the geometry claim is the observed smooth scaling of accuracy with the rank of the factorization, which holds even though only the subspace geometry (not the precise weight values) is retained. We will expand the discussion in §4 to clarify this distinction and to note the limitation that the recovered directions may align with those favored by gradient descent. We will also add a brief comparison against low-rank factorizations initialized without supervision to provide further context, though the primary scaling result remains unchanged.","revision_made":"partial","referee_comment":"[§4] Spectral analysis (presumably §4): the claim that geometry dominates is supported only by post-hoc analysis of the learned Σ. Because Σ is itself the output of the same supervised procedure, the spectral decomposition may simply recover the directions already favored by gradient descent rather than revealing an independent property of neural weight spaces."}],"tokens_in":1449,"tokens_out":671,"duration_ms":41539,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that this paper takes randomized-weight networks and learns a low-rank factorization of the covariance Σ from data so that fixed random weights plus a small trainable head can reach the accuracy of a fully trained MLP. Experiments are reported on MNIST and CIFAR-10, with an added spectral analysis and the note that accuracy scales with rank.\n\nWhat is actually new is the explicit data-driven low-rank structure on the sampling distribution rather than using a fixed isotropic Gaussian. The multiple framings (random features, supervised metric learning, kernel view) are a reasonable way to situate the method. The rank-scaling observation is concrete and could be practically useful.\n\nThe paper does the basic job of showing the idea on standard small datasets. If the full text supplies the exact learning procedure for the factors, the baselines used, and the quantitative gaps to fully trained models, that would be the useful part.\n\nThe soft spot is the evidence for the stronger claim that geometry carries the predictive power. Because Σ is fit directly on labeled data, the learned directions could simply be task-specific rather than revealing something general about neural weight spaces. The stress-test concern about circularity holds up on the abstract: the spectral analysis does not yet show that the geometry is independent of the supervised procedure that produced it. No error bars or exclusion details are mentioned either.\n\nThis is for people working on parameter-efficient or randomized networks. A reader already following that literature might extract the rank-scaling result or the embedding perspective.\n\nIt deserves a serious referee because the proposal is straightforward to test and the datasets are accessible, even if the analysis section will need tightening.","headline":"HalfNet learns low-rank covariance for random weights to match full MLPs on MNIST/CIFAR with fewer trained params, but the geometry claim risks being circular with the supervised fitting of Σ.","tokens_in":2308,"tokens_out":415,"would_cite":false,"duration_ms":22948,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"HalfNet draws random weights from a learned low-rank covariance and matches fully trained networks with far fewer parameters.","keywords":["randomized neural networks","low-rank factorization","subspace geometry","random embedding","MNIST","CIFAR-10","metric learning"],"falsifier":"An experiment on a new dataset or architecture in which HalfNet with the learned Σ fails to reach the accuracy of a fully trained network of comparable size would falsify the claim that the learned geometry captures the essential structure.","tokens_in":2575,"feed_emoji":"🧠","tokens_out":646,"duration_ms":20124,"temperature":0.7,"pith_summary":"The paper proposes that random weights drawn from a normal distribution whose covariance matrix has a low-rank factorization learned from data can recover the performance of standard multilayer perceptrons. Experiments on MNIST and CIFAR-10 show comparable accuracy while training only a small head instead of all weights. Spectral analysis supports the claim that geometry of the weight space, rather than precise individual values, carries most of the predictive power. Accuracy increases smoothly as the rank of the factorization grows. The construction is presented as a data-dependent random embedding interpretable also as supervised metric learning or random features.","feed_headline":"Learned covariance lets random-weight nets match trained accuracy","feed_subtitle":"HalfNet draws weights from a data-learned distribution and trains only a small head, matching full performance on image tasks with far fewer","key_machinery":"The low-rank factorization of the covariance Σ that defines the distribution from which random weights are drawn, producing a data-dependent random embedding.","core_discovery":"HalfNet draws random weights from N(0, Σ) where Σ has a learned low-rank factorization. This yields networks that match the accuracy of fully trained multilayer perceptrons on MNIST and CIFAR-10 while using substantially fewer parameters. Spectral analysis indicates that predictive power resides primarily in the geometry of the weight space rather than in the exact values of individual parameters, and accuracy scales smoothly with rank.","pith_inferences":["The approach could be tested on sequence or graph tasks to check whether the same learned-geometry principle transfers beyond image data.","If the low-rank Σ can be estimated once and reused across related tasks, it would imply a form of transferable random embedding.","The smooth rank-accuracy curve suggests an inexpensive way to trade parameter count for performance without retraining the entire network."],"forward_implications":["Accuracy scales smoothly with the rank of the learned factorization.","The method can be viewed through supervised metric learning or random-feature and kernel perspectives.","Much of the predictive power of neural networks lies in the geometry of their weight space rather than precise parameter values."],"fun_headline_variants":["Learned covariance in random-weight nets matches trained MLP accuracy","HalfNet draws weights from learned low-rank distribution for fewer params","Spectral analysis shows weight geometry holds predictive power in nets","Accuracy scales with rank in networks using learned weight space geometry"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"That a low-rank factorization of Σ learned from data produces a random embedding whose fixed weights, when combined with a small trainable head, recover full network performance without the learned geometry being an artifact of the training procedure or dataset choice.","fun_headline_variants_meta":{"raw":{"variants":["Learned covariance in random-weight nets matches trained MLP accuracy","HalfNet draws weights from learned low-rank distribution for fewer params","Spectral analysis shows weight geometry holds predictive power in nets","Accuracy scales with rank in networks using learned weight space geometry"]},"model":"grok-4.3","cost_usd":0.004462,"raw_usage":{"total_tokens":2195,"prompt_tokens":605,"num_sources_used":0,"completion_tokens":66,"cost_in_usd_ticks":44624500,"prompt_tokens_details":{"text_tokens":605,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1524,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":605,"tokens_out":66,"duration_ms":12855,"temperature":1.0,"reasoning_tokens":1524,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T07:31:30.287393+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An experiment on a new dataset or architecture in which HalfNet with the learned Σ fails to reach the accuracy of a fully trained network of comparable size would falsify the claim that the learned geometry captures the essential structure.","supporting_citations":[],"review_version":1}