{"id":"dfaf05d1-f920-4e41-a667-a2cb36d53a27","arxiv_id":"2412.16168","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Least-confidence active learning appeared to increase feature overlap (superposition) in ResNet18 on CIFAR-10 and Tiny ImageNet, contrary to the authors' expectation, though the evidence is indirect and statistically unverified.","lead":"This paper tested whether training a ResNet18 model with active learning, which selects the most uncertain images, changes how neurons pack multiple features together, a phenomenon known as superposition. The authors report that the active learning model showed more feature overlap, not less, but the experiment lacks error bars, accuracy numbers, and matched training protocols.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The baseline and active-learning models are trained on unmatched data volumes and update counts, so the reported superposition difference may be an underfitting artifact rather than an effect of active learning.","rationale":"The reader's rejection is well founded, and the paper indeed lacks direct feature-level superposition measurements. However, the single most load-bearing flaw is the unmatched training schedule: the paper attributes a difference to active learning when the two models were trained with different data volumes and numbers of gradient steps. This confound alone invalidates the central comparison, even before considering whether cosine similarity and clustering metrics are valid proxies for superposition. My concrete test would settle this by adding a random-sampling control with the same loop schedule. If the random control reproduces the active-learning metrics, the paper's conclusion is not supported; if it does not, the schedule concern is answered and the remaining proxy-validity question becomes the primary issue. Either way, the reader's REJECT verdict stands, so I recommend no change.","tokens_in":5139,"tokens_out":2176,"duration_ms":25199,"concrete_test":"Run a controlled comparison on CIFAR-10 with identical ResNet-18 initialization and optimizer settings: (a) baseline trained for 20 full epochs; (b) active-learning model with the paper's 10-loop schedule; (c) a matched control trained on 10 randomly sampled 5000-image loops (same data volume and update count as active learning, but no uncertainty selection); and (d) active learning trained for 20 epochs by repeating the selected loop budget. Report test accuracy and the Section 4 metrics (cosine similarity mean/std, silhouette score, Davies-Bouldin index) over at least five seeds. If (c) matches (b) within seed noise, the reported superposition difference is a training-budget artifact, not an effect of active learning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical contrast in Sections 4–5 compares a baseline trained for 20 epochs on the full CIFAR-10 or Tiny ImageNet training set with an active-learning model trained in 10 loops on 5000 or 10,000 images per loop, respectively (Section 2, Training Details). These training budgets are not matched: the baseline sees the entire dataset 20 times, while the active-learning model sees each selected image at most once, giving far fewer total gradient steps. The active-learning model's higher cosine similarity, lower silhouette score, and worse t-SNE separation could therefore be explained by reduced training and overfitting to a small, repeatedly selected subset, rather than by uncertainty sampling causing superposition. The claim in Section 5 that active learning 'clearly performs worse overall and seems to have more Superposition' is load-bearing on this comparison, but no control equalizes data volume, epochs, or compute. Without matched training budgets, the observed difference cannot be attributed to the active-learning strategy itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether uncertainty-based active learning influences superposition (neuron polysemanticity) in a ResNet-18 trained on CIFAR-10 and Tiny ImageNet. The authors compare a baseline trained for 20 epochs on the full training set with an active-learning model trained in 10 loops on 5,000 (CIFAR-10) or 10,000 (Tiny ImageNet) images per loop, using least-confidence sampling. Superposition is assessed indirectly through t-SNE visualizations, cosine similarity histograms and statistics, Silhouette scores, Davies-Bouldin indices, and activation maximization images. The paper concludes that the active-learning model performs worse and exhibits more superposition than the baseline, contrary to the authors' initial intuition.","tokens_in":5301,"tokens_out":4147,"duration_ms":40771,"significance":"The question of whether data-selection strategies can control superposition is interesting and largely unexplored, so the paper addresses a worthwhile topic. The experimental design uses standard datasets (CIFAR-10, Tiny ImageNet) and a common architecture (ResNet-18), and it applies multiple visualization and clustering metrics. If the reported effect were robust and properly controlled, it could motivate further study of interpretability-aware data curation. However, the current significance is limited by severe methodological gaps: the central comparison is confounded by unmatched training budgets, no accuracy numbers or statistical tests are reported, and the superposition metrics are unvalidated proxies. The paper also honestly acknowledges some limitations (e.g., non-comparable activation maximization units), which is commendable, but these acknowledgments further weaken the evidence for the main claim.","major_comments":[{"comment":"The baseline and active-learning models are trained under fundamentally unmatched budgets. The baseline is trained for 20 epochs over the full dataset, seeing each image 20 times, while the active-learning model trains in 10 loops on 5,000 (CIFAR-10) or 10,000 (Tiny ImageNet) images per loop, with each selected image seen at most once. This means the two models receive very different amounts of data and gradient updates. Consequently, the higher cosine similarity, lower silhouette score, and poorer t-SNE separation of the active-learning model could simply reflect underfitting or overfitting to a small, repeatedly selected subset, rather than an effect of the active-learning strategy on superposition. The claim in Section 5 that active learning 'clearly performs worse overall and seems to have more Superposition' is not supported without a control that matches data volume, number of optimization steps, or compute. This is a load-bearing issue for the paper's central empirical claim.","section":"Section 2, Training Details; Section 5"},{"comment":"The paper never reports classification accuracy values or any quantitative measure of task performance, despite stating in the Abstract that the active-learning model 'did not significantly outperform' and in Section 5 that it 'clearly performs worse overall.' No error bars, multiple seeds, or statistical tests are provided; all conclusions rest on single runs. Without these, the empirical comparisons are not quantitatively supported, and the claim of 'worse overall performance' cannot be evaluated.","section":"Section 4; Section 5"},{"comment":"The metrics used to infer superposition—cosine similarity histograms, silhouette scores, Davies-Bouldin indices, and t-SNE overlap—are indirect and unvalidated proxies. Section 3 explicitly says these approaches 'help us guess the amount of superposition,' which falls short of a rigorous operationalization. No direct feature-level test (e.g., measuring the number of features per neuron, interference, or sparse probing) is performed. Moreover, the direction of the metrics is inconsistent across datasets: on CIFAR-10 the active-learning model has higher mean cosine similarity (0.927 vs. 0.885), but on Tiny ImageNet it has lower mean cosine similarity (0.254 vs. 0.310), yet both are interpreted as evidence of more superposition. This inconsistency undermines the interpretive claim and suggests the metrics do not track a single underlying phenomenon.","section":"Section 3; Section 4"},{"comment":"The activation maximization comparison is explicitly acknowledged by the authors to be non-comparable: 'these units are not reacting to the same features as they are from different models, so we can not directly compare them.' Nevertheless, the text uses these visualizations to conclude that the active-learning model shows a 'more crowded pattern' and 'might indicate more superposition.' This is an internal contradiction; the evidence should either be discarded or replaced with a comparison that matches units or uses a quantitative feature-level analysis.","section":"Section 4, Figure 6"}],"minor_comments":[{"comment":"The manuscript contains numerous grammatical errors and awkward phrasings (e.g., 'the most intricately beautiful blockers'), which impede readability and require careful editing.","section":"Throughout"},{"comment":"Reference [2] is cited as 'A mathematical framework for transformer circuits,' but the text discusses 'Toy Models of Superposition'; the reference is mismatched. Reference [11] is not the correct citation for t-SNE; the standard source is van der Maaten and Hinton (2008).","section":"References"},{"comment":"Key experimental details are omitted: optimizer, learning rate, batch size, weight decay, data augmentation, and the specific layer from which feature vectors are extracted for t-SNE and cosine similarity. These details are necessary for reproducibility.","section":"Section 2, Training Details"},{"comment":"Figure captions are minimal and do not specify which layer or feature space the visualizations correspond to, nor how the t-SNE plots were generated (e.g., perplexity, number of points).","section":"Figures"}],"recommendation":"major_revision","confidential_remarks":"The paper is very preliminary and reads like a workshop submission. The central question is worth asking, but the current evidence is too weak to support the conclusion. The unmatched training budgets and absence of any statistical rigor would require a substantial experimental overhaul, not just minor edits. I would not rule out the possibility of a publishable paper after such a revision, but the current version is far from that. The lack of code and incomplete experimental details also hinder reproducibility assessment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper asks whether least-confidence active learning changes superposition in a small CNN, and reports that the active-learning model looks more polysemantic on clustering metrics. The question is a reasonable new combination, and the paper is open about the result being the opposite of what it expected. It does not overclaim by much in the abstract — it says 'suggests' — but the conclusion in Section 5 is stronger than the evidence.\n\nWhat is actually new: applying active learning as a probe for superposition is not in the prior literature, and the paper tries several inspection methods (t-SNE, cosine similarity histograms, silhouette, Davies-Bouldin, activation maximization). That is worth something as a pilot.\n\nThe soft spots are substantial. The baseline is trained 20 epochs over the full dataset; the active-learning model gets 10 loops of 5000/10000 images, with each image selected once. Those budgets are not matched, so the active-learning model receives far fewer gradient updates and less data. The higher mean cosine similarity, lower silhouette score, and overlapping t-SNE clusters could simply be underfitting or overfitting to a small selected subset. There is no control, no repeated seeds, no error bars, and no accuracy numbers — the paper says 'overall accuracy' but never reports the values. On Tiny ImageNet, the active-learning model actually has lower mean cosine similarity (0.254 vs 0.310), which undercuts the 'more superposition' interpretation; the paper mentions it but doesn't reconcile it.\n\nThe second issue is the proxy assumption. Cosine similarity and cluster quality are at best indirect indicators of neuron polysemanticity. Without a feature-level test (e.g., how many distinct features load onto individual neurons), the inference from clustering metrics to superposition is a leap. The paper itself notes the activation-maximization units are from different models and can't be directly compared, which is honest but means that evidence is weak.\n\nIn short, this is an honest pilot with a confounded design. The core claim is not supported as it stands. I wouldn't send this version to serious peer review; desk reject with an invitation to resubmit after matching training budgets, adding seeds and accuracy numbers, and using more direct superposition measures. The idea might be worth a workshop paper once the experiment is cleaned up. For a reading group, it's a fine case study of why training-schedule confounds matter in interpretability experiments, but not as a citable result.","headline":"A well-intentioned pilot that asks the right question but compares unequal training budgets, so the 'active learning increases superposition' conclusion is not supported.","tokens_in":5786,"tokens_out":2926,"would_cite":false,"duration_ms":28578,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a ResNet-18 with least-confidence active learning produced higher feature overlap and worse class separation than baseline training, indicating the sampling strategy increases rather than reduces neuron superposition.","keywords":["superposition","neuron polysemanticity","active learning","uncertainty sampling","interpretability","CIFAR-10","Tiny ImageNet","feature overlap"],"falsifier":"A direct feature-level measurement on the trained models: identify individual neurons in the final convolutional layer, count how many unrelated object classes each neuron responds to (for example, through class-conditional activation statistics or causal ablations), and compare the distribution of per-neuron feature counts between baseline and active learning models. If the baseline has equal or more polysemantic neurons, the paper's conclusion is falsified.","tokens_in":4939,"feed_emoji":"🧠","tokens_out":5859,"duration_ms":57621,"temperature":0.7,"pith_summary":"The paper asks whether neuron superposition, the phenomenon where a single neuron responds to multiple unrelated features, can be reduced by choosing training data through active learning. It trains ResNet-18 on CIFAR-10 and Tiny ImageNet under a least-confidence uncertainty-sampling schedule and compares the resulting feature spaces with a baseline trained on the same data. The paper reports that the active learning model does not separate classes better; by several overlap measures it appears to have more superposition, not less. The point of the exercise is to test a possible lever on superposition: if data selection changes how features are packed into neurons, then superposition is not fixed by architecture alone.","feed_headline":"Uncertainty sampling increased neuron entanglement","feed_subtitle":"On CIFAR-10 and Tiny ImageNet, active learning showed more feature overlap and worse clusters than baseline training.","key_machinery":"The load-bearing object is the least-confidence uncertainty sampling loop: at each of 10 rounds, the model scores the remaining unlabeled training images by the rule $s_{LC} = \\arg\\max_x (1 - P(\\hat{y}|x))$, selects the least confident batch (5000 images per round on CIFAR-10, 10,000 on Tiny ImageNet), retrains, and repeats. This intervention is intended to change which data the model sees and therefore how features are packed into neurons. The paper measures the result with feature-space overlap proxies: t-SNE cluster overlap, cosine similarity histograms and statistics, Silhouette scores, Davies-Bouldin indices, and visual inspection of activation maximization images. The comparison between baseline and active learning models on all four criteria is what carries the conclusion.","core_discovery":"The paper's central claim is that, under least-confidence uncertainty sampling, active learning produces a model with more superposition than a standard baseline, contrary to the intuition that focusing on uncertain samples should sharpen feature representations. On CIFAR-10, the active learning model had a higher mean cosine similarity between features (0.927 versus 0.885), a lower Silhouette score (0.189 versus 0.212), and more overlapping t-SNE clusters than the baseline. On Tiny ImageNet, the active learning model also had more overlap and worse clustering, with lower Silhouette scores and a higher Davies-Bouldin index. The intended mechanism was that preferentially training on least-confident samples would force the model to build more distinct features, but instead the model appears to have reinforced existing class overlaps or created new ones. The paper concludes that active learning, at least in this least-confidence form, is not a reliable way to decode or reduce superposition.","pith_inferences":["An implication the author leaves implicit is that the sampling distribution itself may be the cause: least-confident examples sit near class boundaries, so repeatedly training on them could force neurons to encode boundary-blending features, making superposition worse rather than better.","A testable extension, not run here, would be to keep the same architecture and training budget but compare least-confidence sampling with entropy-based, margin-based, and diversity-based selection, then measure per-neuron feature counts; this would isolate whether the effect comes from active learning in general or from the specific query rule.","If these clustering proxies are validated against direct neuron-level probes, the paper's cheap comparison procedure could become a screening tool for superposition across training regimes, which the author does not claim."],"forward_implications":["If the result holds, least-confidence uncertainty sampling is not a way to reduce superposition, and may actively increase feature entanglement.","Superposition is influenced by training-data selection, not only by model capacity and feature sparsity, so data curation is a lever worth studying.","Cluster-separability metrics can serve as cheap, scalable proxies for comparing superposition across training regimes, even if they still need direct validation.","Stronger active learning strategies, such as diversity-based or core-set selection, are the natural next test because this paper's failure suggests the sampling rule matters.","Interpretability analyses may need to treat actively trained models as potentially more polysemantic than standard baselines."],"supporting_citations":[{"why":"Introduces the notion of superposition and polysemantic neurons that the paper builds on.","marker":"[1]"},{"why":"Supplies the toy-model superposition hypothesis and the open question about controlling superposition that motivates the experiment.","marker":"[2]"},{"why":"Defines active learning's goal of choosing data points to label, the framework the paper applies to feature-space training.","marker":"[3]"},{"why":"Supplies the confidence-based active-learning sampling method that the paper adapts as least-confidence selection.","marker":"[7]"},{"why":"Provides the active-learning survey that frames uncertainty sampling and its expected benefits.","marker":"[8]"},{"why":"Provides the activation-maximization visualization approach the paper adapts to inspect neurons.","marker":"[9]"},{"why":"Provides the visualization tools used to generate the activation maximization images.","marker":"[10]"},{"why":"Supplies t-SNE, the dimensionality-reduction method used to visualize class-cluster overlap.","marker":"[11]"}],"fun_headline_variants":["Uncertainty sampling makes neural nets more tangled","Active learning boosts neuron overlap, not separation","Least-confidence training worsens feature clustering","Uncertainty sampling fails to untangle superposition","Active learning deepens superposition in ResNet18"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's conclusion rests on treating clustering and similarity statistics (cosine similarity, Silhouette score, Davies-Bouldin index, and t-SNE overlap) as valid proxies for neuron superposition; if those metrics do not actually reflect how many distinct features are packed into individual neurons, the reported differences do not establish more superposition.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty sampling makes neural nets more tangled","Active learning boosts neuron overlap, not separation","Least-confidence training worsens feature clustering","Uncertainty sampling fails to untangle superposition","Active learning deepens superposition in ResNet18"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000136,"raw_usage":{"total_tokens":1139,"prompt_tokens":927,"completion_tokens":212,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":144}},"tokens_in":543,"tokens_out":212,"duration_ms":2796,"temperature":1.0,"reasoning_tokens":144,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:18:46.624978+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct feature-level measurement on the trained models: identify individual neurons in the final convolutional layer, count how many unrelated object classes each neuron responds to (for example, through class-conditional activation statistics or causal ablations), and compare the distribution of per-neuron feature counts between baseline and active learning models. If the baseline has equal or more polysemantic neurons, the paper's conclusion is falsified.","supporting_citations":[{"cited_title":"Zoom in: An introduction to circuits","cited_arxiv_id":null,"evidence_quote":"Introduces the notion of superposition and polysemantic neurons that the paper builds on."},{"cited_title":"A mathematical framework for transformer circuits","cited_arxiv_id":null,"evidence_quote":"Supplies the toy-model superposition hypothesis and the open question about controlling superposition that motivates the experiment."},{"cited_title":"Confidence-based active learning","cited_arxiv_id":null,"evidence_quote":"Supplies the confidence-based active-learning sampling method that the paper adapts as least-confidence selection."},{"cited_title":"Active learning literature survey","cited_arxiv_id":null,"evidence_quote":"Provides the active-learning survey that frames uncertainty sampling and its expected benefits."},{"cited_title":"Openai microscope, 2020","cited_arxiv_id":null,"evidence_quote":"Provides the activation-maximization visualization approach the paper adapts to inspect neurons."},{"cited_title":"Lucent: A neural network visualization library, 2020","cited_arxiv_id":null,"evidence_quote":"Provides the visualization tools used to generate the activation maximization images."}],"review_version":1}