{"id":"b9438986-ffe9-43e8-85f6-669b8966eb9e","arxiv_id":"2504.18929","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"In controlled toy experiments, Transformers converge to lower-entropy distributions than the target, an effect driven by the FFN and linked to dynamic sparsity patterns.","lead":"Using a tiny artificial language where the true probability distribution is known exactly, the paper finds that Transformers learn distributions that are even lower-entropy than the real one, especially as models get larger. It links this behavior to feedforward modules and to a growing preference for skipping attention and activating fewer neurons.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported entropy undershoot may be a support-normalization artifact: no entropy renormalized to the 80-sequence target support is reported, so the claim that Transformers learn more deterministic distributions is not yet established.","rationale":"The reader's weakest assumption concerns whether the sparsity principle makes the task representative. I agree that representativeness is a concern, but the more immediate and testable problem is that the paper's headline entropy comparison mixes two distinct effects: (1) assigning low probability to sequences that the target already gives zero probability, which is correct learning, and (2) learning more deterministic conditionals on the support, which would be the claimed bias. The paper explicitly states that larger Transformers are better at learning zero probabilities in the sparse part, yet still counts the resulting entropy decrease as evidence of a low-entropy bias. This is internally inconsistent unless the support-conditional entropy is shown to be lower than the target entropy. The renormalized entropy test separates these effects cleanly. If the test passes, Observations 1 and 2 gain real support; if it fails, the main claim reduces to a much weaker statement about zeros. This is load-bearing because the central claim is precisely that Transformers impose an extra information-compressing prior beyond matching ptgt. The paper does have independent strengths: exact entropy computation over all 3125 sequences, a fully known target, and support-constrained experiments are all well suited to running the proposed check. I also note that the paper itself flags Eq. (1) as intuitive and leaves rigorous theory for future work, so the empirical decomposition in Figure 3 is the sole support for the mechanism; that decomposition is currently insufficient. The RNN depth confound the reader raises is real but secondary to the normalization issue, since even the Transformer-only results are not yet diagnosed correctly. Therefore the appropriate verdict remains CONDITIONAL, with the support-renormalized entropy check as a required addition rather than a reason for rejection.","tokens_in":14881,"tokens_out":10417,"duration_ms":121160,"concrete_test":"Restrict evaluation to the 80 sequences in the target support. For each trained model, compute q = Σ_{s in supp(ptgt)} pθ(s) and the renormalized distribution r(s) = pθ(s)/q, then compute H(r) = -Σ_{s in supp} r(s) log r(s). Also compute the prefix-weighted conditional entropy C = Σ_i E_{s_{<i} ~ ptgt} H(pθ(·|s_{<i})) using the true target prefix frequencies rather than pθ's own prefix distribution. If, for d=64 Transformers, H(r) ≥ H(ptgt) (or C ≈ H(ptgt)) while H(pθ) < H(ptgt), then the low-entropy effect is entirely attributable to placing mass away from target-zero sequences, and the 'more deterministic distributions' version of Observation 1 fails. If H(r) < H(ptgt) persists, the bias claim is supported on the actual support.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"In the controlled setup, the target distribution has support of only 80 sequences out of |V|^n = 3125; the remaining 3045 sequences have zero target probability and contribute exactly zero to H(ptgt)=3.571. The learned pθ, however, is a normalized product over the full vocabulary and assigns nonzero probability to many of those sparse sequences, so H(pθ) includes contributions from regions of sequence space never observed in training. The paper's Figure 3 reports raw entropy contributions of the sparse and non-sparse parts. A decrease in the sparse-part entropy reflects learning to assign low probability to target-zero sequences, which moves pθ closer to ptgt, not farther away. Meanwhile, a decrease in the raw non-sparse entropy can be driven simply by an increase in the total probability mass assigned to the support, without any sharpening of the conditional distributions. Therefore Observation 1's interpretation that larger Transformers 'produce more deterministic distributions' and the proposed implicit regularizer αH(pθ) in Eq. (1) do not follow from the reported numbers. The RNN comparison is similarly confounded, because RNNs and Transformers may extrapolate very differently on target-zero prefixes, and the paper never isolates the supported-sequence conditional entropy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies a controlled next-character prediction task with a known target distribution over sequences of length 5 from a vocabulary of size 5, where the target has only 80 non-zero probability sequences. It reports three sets of findings. First, from measurements of the entropy H(pθ) and KL divergence of trained Transformers of increasing size, it claims that Transformers have an inductive bias to learn distributions with entropy below H(ptgt) (Observation 1), that this bias is driven by the FFN module (Observation 2), and it formalizes the bias as an implicit regularizer αH(pθ) in Eq. (1). Second, with a modified attention module that learns routing weights over attention heads and a residual path, it claims that larger Transformers prefer residual connections, bypassing attention computation (Observation 3). Third, by enumerating all 3,125 possible inputs, it studies FFN neuron activation counts and reports that larger models have a lower proportion of active neurons and more dead neurons, that sparsity increases in a jump-like manner during training, and that loss spikes coincide with sudden increases in dead neurons (Observations 4 and 5). The paper concludes that Transformers do not merely approximate the target distribution but further compress its information content, and that parameter redundancy enables compression through dynamic sparsity.","tokens_in":15086,"tokens_out":10465,"duration_ms":98386,"significance":"The controlled setup with a known target distribution and tractable entropy is a useful methodological contribution, and the exhaustive enumeration of all possible inputs for the neuron-activation analysis is a strength that goes beyond typical corpus-based studies. If the low-entropy bias is confirmed, it would connect Transformer architecture to implicit regularization in a falsifiable way. However, the paper currently does not establish that the entropy undershoot is a property of the conditional distributions on the target support rather than an artifact of probability mass assigned to zero-probability sequences. The supporting claims about attention routing and FFN attribution are also based on modified architectures, and the experimental evidence lacks multiple seeds and error bars. For these reasons the central claims need substantial additional analysis before they can be accepted.","major_comments":[{"comment":"The claim that larger Transformers learn more deterministic distributions is not yet established because the reported entropies are not renormalized to the support of the target distribution. The target has support only 80 out of 3,125 sequences (Appendix B), so a decrease in the raw entropy of the non-sparse part can be produced by an increase in the total probability mass pθ(A) assigned to the support while the conditional distribution on the support remains flat; for example, a uniform distribution over the 80 supported sequences with total mass 0.7 has non-sparse entropy ≈ 3.32, which is below H(ptgt) = 3.571. The paper does not report pθ(A) or the renormalized entropy H(pθ|A) = -Σ_{s∈A} (pθ(s)/pθ(A)) log(pθ(s)/pθ(A)), nor the conditional entropies per prefix. Without these, Observation 1 and the implicit regularizer in Eq. (1) do not follow from the reported numbers, and the RNN comparison is similarly confounded because RNNs may assign different total mass to the support.","section":"Section 3.1, Fig. 3, Eq. (1)"},{"comment":"Observation 3 is based on a modified architecture in which a learned routing network f(x) = Softmax(W2σ(W1x)) explicitly parameterizes the weight of each attention head and of the residual connection. This routing mechanism is not part of the standard Transformer, and the observed preference for residual paths could be an artifact of how the routing parameters are initialized or trained, especially in larger models with more capacity in the routing MLP. The manuscript does not show that standard Transformers, without the added routing, bypass attention computations. The claim should either be restricted to the specific routing-augmented model or be corroborated by measurements on the original architecture, such as effective rank analysis of attention outputs or layer-wise gradient attribution.","section":"Section 4.1, Eq. (2), Observation 3"},{"comment":"The empirical support for the scaling claims is weakened by the absence of multiple random seeds and error bars: every curve and scatter point appears to come from a single run. Moreover, the paper explicitly excludes d = 8 cases from the main scatter plots 'for clear presentation' because they are outliers. Excluding the smallest models changes the apparent trend, and the statement that the low-entropy preference 'gradually emerges in larger Transformers' is supported by removing exactly the points that behave differently. Please report full results including d = 8 with multiple seeds and variance estimates, and justify any exclusion in the main text.","section":"Appendix B, Figures 2 and 4"},{"comment":"The ablation that attributes the low-entropy bias to the FFN module is confounded by capacity and depth: Attention-only retains only attention layers, FFN-main retains all FFN layers plus one attention layer, and Attention-main is an intermediate configuration. These variants differ in parameter count and depth, and the lower entropy of FFN-main could be due to the larger number of FFN parameters rather than to a specific inductive bias of the FFN module. A capacity-matched control is needed, for example an attention-only transformer with width or depth increased to match the parameter count of FFN-main, or a standard transformer with the FFN replaced by an identity mapping, before assigning the causal role to the FFN.","section":"Section 3.2, Fig. 4, Observation 2"},{"comment":"The claimed strong correlation between loss spikes and sudden increases in dead neurons is based on a single run and is presented visually without a quantitative correlation coefficient or a significance test. In addition, the optimizer comparison in Fig. 8 varies not only the optimizer but also hyperparameters such as learning rate and weight decay (Appendix B), so the differences in dead-neuron dynamics and instability cannot be attributed solely to second-order gradient information. The conclusions in Observation 5 should be supported by multiple seeds and controlled hyperparameter choices.","section":"Section 4.2, Figs. 7 and 8"}],"minor_comments":[{"comment":"The phrase 'cneter right' should read 'center right'.","section":"Section 4.2"},{"comment":"The author name 'V oita' should be 'Voita'.","section":"References"},{"comment":"In the sentence before Eq. (2), 'The entropy of the learned ptgt' should refer to 'the learned pθ'.","section":"Section 2"},{"comment":"The smoothing of entropy and KL curves with a window of size 3 is described in the text, but the figure captions for Figures 2, 4, and the appendix figures do not state this; please disclose the smoothing in the captions.","section":"Appendix B"},{"comment":"The paper does not state whether a single sampled training set is reused across all model configurations; if so, this should be stated, since it affects the comparability of runs.","section":"Section 2"},{"comment":"In Eq. (1), the entropy H(pθ) is defined over the full sequence distribution, but the notation could be clarified to distinguish the sequence-level entropy from per-token conditional entropies used in the training loss.","section":"Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an interesting question and the controlled setup is a good idea, but the main entropy claim is currently vulnerable to the support-normalization artifact. If the authors can provide renormalized entropies on the target support, multiple seeds, and a capacity-matched FFN ablation, the paper could become a solid contribution. The attention-routing section should be reframed as a study of a modified architecture rather than of standard Transformers. I would not support acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core observation here is real and worth attention: in a small controlled setting with a known target distribution, Transformers learn distributions whose entropy falls below the target's, while RNNs sit near it. That is a new empirical result, and the controlled setup with exact entropy computation is a good idea. The FFN attribution via ablations is also a clean way to localize the effect, and the exhaustive dead-neuron counts over all possible inputs are a nice improvement over the usual corpus-sampling approach.\n\nThat said, the paper has several soft spots. Most importantly, the central claim rests on a single set of runs; there are no error bars or multiple seeds reported. For a phenomenon that is inherently about optimization behavior, that is a real weakness. The exclusion of d=8 cases from the main scatter plots as outliers is also a bit uncomfortable, even if the authors explain it. The attention-routing result (Observation 3) comes from a modified architecture with an added routing function, so it is not directly about standard Transformers. And the loss-spike/dead-neuron correlation is qualitative—plots suggest it, but no quantitative measure is given.\n\nThe stress-test worry about support normalization does not, in my reading, land. The raw non-sparse entropy is computed as a sum over the 80 supported sequences of -p(s) log p(s). If a model merely increased the total mass assigned to that support, scaling all probabilities up by a constant, each term would increase (since the probabilities are well below 1/e), not decrease. So the observed decrease does point to sharpening of the conditional distributions, not an artifact. Still, the paper would be stronger if it reported the normalized conditional entropy on supported prefixes, which would directly address the concern. The RNN comparison also conflates architecture with depth—single-layer RNNs versus five-layer Transformers—and the varied-layer sweeps help but do not fully resolve the confound.\n\nEq. (1) is explicitly a post-hoc interpretation, which is fine as a hypothesis, but it should not be presented as if it were derived. The authors are honest about it being intuition, so I do not flag this as a major flaw.\n\nOverall, this is a paper that deserves serious refereeing. The controlled setting is a real contribution, and the main observation is plausibly true. But the lack of repeated runs and the unmodified-architecture caveat mean the current form would need substantial revision. I would send it to review and ask for multiple seeds, a clearer treatment of the d=8 cases, and a quantitative correlation for the loss-spike claim.","headline":"A controlled setting reveals a genuinely interesting Transformer entropy undershoot, but the paper overreaches from toy to real LLMs and needs robustness fixes before it's publishable.","tokens_in":15609,"tokens_out":3071,"would_cite":true,"duration_ms":33934,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Under controlled language tasks, Transformers learn distributions whose entropy falls below the target, with the FFN as driver.","keywords":["transformers","inductive bias","entropy","dynamic sparsity","feed-forward networks","dead neurons","loss spikes","language modeling"],"falsifier":"The direct check is to rerun the same protocol on a non-sparse target distribution, for example assigning every character probability $1/5$ at each prefix; if a $d=64$ Transformer trained on that target ends with $H(p_\\theta) \\geq H(p_{\\mathrm{tgt}})$ while the sparse-target run still undershoots, the claimed bias is specific to the task's sparsity rather than a general architectural property.","tokens_in":14654,"feed_emoji":"📉","tokens_out":11913,"duration_ms":103963,"temperature":0.7,"pith_summary":"In a controlled language-modeling setup with a known target distribution, the paper reports that Transformers do more than approximate that distribution: they systematically produce a learned distribution whose entropy is lower than the target's, and the gap grows with model size. This inductive bias means larger Transformers trade perfect alignment with the target for extra compression, ending with a larger KL divergence than RNNs of similar size. The feed-forward (FFN) module is identified as the driver: variants with FFN layers reproduce the low-entropy behavior, while attention-only variants do not. In the same setting, larger Transformers also show stronger dynamic sparsity, favoring residual connections over attention heads and activating fewer FFN neurons, and training loss spikes align with sudden jumps in dead neurons. If the bias generalizes, larger Transformers are not just approximating the data distribution but imposing an additional information-compressing prior.","feed_headline":"Bigger Transformers learn lower-entropy distributions than data","feed_subtitle":"In a controlled language task, the entropy gap grows with size and the FFN drives it.","key_machinery":"The enabling object is the fully known target distribution over $|V|^5 = 3125$ sequences, produced by giving each prefix only two nonzero continuation probabilities, 0.8 and 0.2. Because both $p_{\\mathrm{tgt}}$ and $p_\\theta$ are enumerable, the paper can compute $H(p_{\\mathrm{tgt}})$, $H(p_\\theta)$, and $\\mathrm{KL}(p_{\\mathrm{tgt}}\\|p_\\theta)$ exactly and track them per epoch, which is impossible at real language-modeling scale. The conceptual machinery is the reading of Eq. (1) as cross-entropy plus an implicit entropy regularizer $\\alpha H(p_\\theta)$ induced by the architecture, plus two sparsity lenses: attention heads and residual connections treated as routable forward paths, and FFN neurons treated as key-value memories whose activations can be counted across all 15,625 tokens.","core_discovery":"The central discovery is an architectural inductive bias: trained by maximum likelihood on a known sparse language, Transformers converge to distributions $p_\\theta$ with $H(p_\\theta) < H(p_{\\mathrm{tgt}})$, and the undershoot grows with width and depth. RNN controls converge near $H(p_{\\mathrm{tgt}})$, so this is not a generic effect of capacity. Ablations attribute the bias to the FFN module: keeping only FFN layers reproduces the low-entropy landscape, while attention-only models hover at or above the target entropy. The same controlled setup also shows that routing weights concentrate on residual connections as models grow, and FFN neurons become dynamically sparse (over 25% of neurons fire on at most 5% of the maximum possible inputs, and nearly 10% never fire when $d=64$), with the sparsity sharpening in jumps that coincide with loss spikes.","pith_inferences":["The authors leave implicit that the entropy undershoot predicts overconfident predictions in larger models; this can be tested on real language models by comparing predictive entropy with calibration error.","Because the FFN drives the bias, an experiment that varies FFN width while freezing attention would isolate how sharply the undershoot scales with FFN capacity.","The jump-like neuron death suggests an intervention: if the second-moment estimate spikes at the same time as neuron death, clipping or resetting it could separate cause from symptom in the instability."],"forward_implications":["As Transformers scale, their learned distributions should drift further below the true entropy, so larger models may show higher KL divergence even while their cross-entropy loss keeps decreasing.","The FFN is the causally relevant component for the low-entropy bias: attention-only stacks remove it, while FFN-dominated stacks retain it.","Larger Transformers will route an increasing share of forward computation through residual connections, so much of the attention computation is effectively bypassed on typical inputs.","Dynamic sparsity in FFN neurons is pervasive, including neurons that never activate on any input, which means the sparsity is a property of the architecture, not of a particular sampled corpus.","Training instability in larger models is coupled to step-like neuron death, so the fraction of dead neurons is a dynamically meaningful training signal rather than a static property."],"supporting_citations":[{"why":"Frames language modeling as compression, the premise this paper tests and extends by tracking the entropy of the learned distribution.","marker":"Delétang et al., 2023"},{"why":"Supplies the decoder-only Transformer architecture used in all controlled experiments.","marker":"Vaswani et al., 2017"},{"why":"Defines the LSTM baseline whose entropy stays near the target, providing the comparison that exposes the Transformer bias.","marker":"Hochreiter and Schmidhuber, 1997"},{"why":"Defines the GRU baseline used in the capacity-matched RNN comparison.","marker":"Chung et al., 2014"},{"why":"Gives the rank-collapse and path view of attention heads versus residual connections that motivates the routing analysis.","marker":"Dong et al., 2021"},{"why":"Documents dead and rarely activated neurons in real LLMs, the pattern this paper verifies exhaustively in the controlled setting.","marker":"Voita et al., 2023"},{"why":"Connects loss spikes in large-scale training to Adam's second-order state, which the optimizer experiments build on.","marker":"Molybog et al., 2023"},{"why":"Treats FFN layers as key-value memories, the lens used to count neuron activations and define dynamic sparsity.","marker":"Geva et al., 2020"},{"why":"Supplies the linear compression-to-performance relationship that motivates comparing KL divergence and loss in the controlled setup.","marker":"Huang et al., 2024"}],"fun_headline_variants":["Transformers secretly prefer lower-entropy data","FFN drives transformers' low-entropy bias, study finds","Larger transformers learn sparser, lower-entropy models","Why transformers compress beyond the target distribution","Transformer entropy gap: size matters, FFN responsible"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand or fall on whether the toy five-character language, where every prefix admits only two possible next characters with probabilities 0.8 and 0.2, is representative enough of real language for the entropy undershoot to count as a general architectural bias rather than a quirk of this unusually sparse task.","fun_headline_variants_meta":{"raw":{"variants":["Transformers secretly prefer lower-entropy data","FFN drives transformers' low-entropy bias, study finds","Larger transformers learn sparser, lower-entropy models","Why transformers compress beyond the target distribution","Transformer entropy gap: size matters, FFN responsible"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1548,"prompt_tokens":1006,"completion_tokens":542,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":468}},"tokens_in":622,"tokens_out":542,"duration_ms":5257,"temperature":1.0,"reasoning_tokens":468,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:06:01.118879+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The direct check is to rerun the same protocol on a non-sparse target distribution, for example assigning every character probability $1/5$ at each prefix; if a $d=64$ Transformer trained on that target ends with $H(p_\\theta) \\geq H(p_{\\mathrm{tgt}})$ while the sparse-target run still undershoots, the claimed bias is specific to the task's sparsity rather than a general architectural property.","supporting_citations":[],"review_version":1}