{"id":"0b512d01-05f3-4cc7-8800-46b361b356bc","arxiv_id":"2504.12700","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Learning in deep networks is proposed to consist of a fast memorization phase followed by a slow compression phase that drives generalization, with mutual information between hidden layers and inputs as a progress measure.","lead":"This paper proposes that deep neural networks learn in two phases: a fast curve-fitting phase followed by a slower compression or coarse-graining phase, and argues that the same two phases explain grokking, double descent, and the information bottleneck. It reports aligned timescales for these phenomena in two transformer training runs, one on modular division and one on a permutation group dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central alignment claim rests entirely on under-specified mutual information estimates from single runs; without estimator details the claimed time-scale alignment is not independently checkable.","rationale":"The reader's weakest_assumption is precisely the under-specified mutual information estimation, and I agree that this is the most load-bearing weakness. My stress-test pass did not find an alternative concern that would change the verdict: the redefinition of double descent as a training-time phenomenon is a conceptual stretch, but the paper's own Figures 3c and 4c do show a second dip after the interpolation threshold, and the paper explicitly flags the analogy as a proposal. The absence of code and seeds is a reproducibility gap, not a demonstrated error, so it supports CONDITIONAL rather than REJECT. I also considered whether the circuit-based analysis (Section 4.4) contradicts the use of GELU in the experiments, but the paper explicitly states that the circuit framework is only conceptual for smooth activations and that the LMN can still be evaluated, so this is a limitation rather than an inconsistency. I therefore recommend keeping the CONDITIONAL verdict unchanged.","tokens_in":10348,"tokens_out":1379,"duration_ms":12471,"concrete_test":"Obtain the code, checkpoints, or a complete training-log release for the division-mod-96 and S5 experiments; re-estimate I(T;X) and I(T;Y) using a stated estimator (e.g., Kraskov-Stögbauer-Grassberger with k=5, or a fixed binning with stated bin counts), with at least 3 seeds and bootstrap or seeded-run error bars. Then recompute the time of the 'right-most point' (the apparent compression onset) and compare it to the grokking onset (training accuracy reaching 1.0) and to the onset of the second descent in test error. If the disagreement between these three time scales exceeds the seed-to-seed spread, the alignment claim fails; if it replicates within noise, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that the end of the curve-fitting phase aligns with the onset of double descent's second descent and with the 'right-most point' of the information-plane trajectory (Section 3.2, Figures 3 and 4). The paper never states the estimator used for I(T;X) and I(T;Y), the binning or partitioning scheme, the number of samples used for estimation, or the number of seeds (Section 3.1). Mutual information between continuous, high-dimensional transformer activations and inputs is extremely sensitive to estimation bias; a systematically biased or high-variance estimator can shift the apparent 'right-most point' and the apparent compression onset by orders of magnitude in training time. Because Figure 5a and 5b also report I(T;X) values that appear identical in range for different layers and datasets, a single estimation artifact could corrupt the claimed alignment of all three time scales. The abstract and Section 4.1 elevate this alignment to the evidence for the two-phase hypothesis, so the claim is unsupported without estimator details or code. The paper also does not provide the checkpoint schedule, so even the visual time-scale identification cannot be reproduced.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes that learning in deep neural networks proceeds in two phases: a fast curve-fitting phase followed by a slower compression or coarse-graining phase. It argues that three phenomena—grokking, double descent, and the information bottleneck—share a common temporal structure, and it presents numerical experiments on modular division modulo 96 and the multiplication table of S5 using a two-layer transformer. The experiments plot training and test accuracy, test error, and information-plane trajectories, and the paper claims that the end of the fitting phase, the onset of the second descent, and the right-most point of the information-plane trajectory all align. It then discusses this two-phase picture in terms of circuit complexity, a renormalization-group analogy, and three geometric mechanisms for reducing the number of linear regions.","tokens_in":10542,"tokens_out":2748,"duration_ms":30474,"significance":"If the time-scale alignment claim is correct, the paper would provide a unifying perspective on grokking, double descent, and the information bottleneck, and it would identify I(T;X) as a useful progress measure for deep learning. A strength of the paper is that it makes a concrete, falsifiable empirical claim and does not fit parameters to confirm a pre-chosen trend. However, the significance is currently conditional: the central alignment claim rests on under-specified mutual-information estimates from single runs, and without the estimator details, multiple seeds, and quantitative time-scale definitions, the claim cannot be independently checked or built upon. The theoretical RG analogy is exploratory, but the empirical part is the load-bearing component.","major_comments":[{"comment":"The mutual information estimator used for I(T;X) and I(T;Y) is never specified. Please state the estimator, the binning or partitioning scheme, the number of samples used for estimation, the network layers over which T is defined, and the checkpoint schedule. Without these details, the information-plane trajectories in Figures 3-5 cannot be reproduced, and the claimed alignment of the three time scales is not independently checkable.","section":"Section 3.1 and Figures 3-5"},{"comment":"The identification of the relevant time scales is made by visual inspection of single runs, with qualitative language such as \"midway between 10^3 and 10^4 steps\" and \"around 10^5 steps.\" Please provide a quantitative definition of each time scale (for example, a threshold or change-point criterion), error bars over multiple seeds, and the number of seeds used. The central claim is that three time scales align, which requires a quantitative comparison rather than visual coincidence.","section":"Section 3.2"},{"comment":"The claim that I(T;X) serves as a progress measure of learning depends on the decrease of I(T;X) being a real dynamical feature rather than an artifact of a biased or variance-heavy mutual-information estimator. Because the estimator is not specified in Section 3.1, this part of the two-phase interpretation is currently unsupported; please provide estimator validation, for example by comparing against a known baseline or by reporting the estimates as a function of sample size.","section":"Section 4.2"},{"comment":"The two-phase structure of the information-plane trajectory is introduced as established fact, citing Refs. [16,17], and is then used to label the observed trajectories in Figures 3-5. The paper should clearly separate this inherited assumption from the new empirical evidence, which concerns the alignment of time scales rather than the existence of the two phases. A precise operational definition of the \"onset of compression\" or \"right-most point\" would help make the test falsifiable.","section":"Section 2.3"}],"minor_comments":[{"comment":"The paper alternates between \"progress measure\" and \"process measure\"; please use one term consistently and define it.","section":"Section 4.2"},{"comment":"The learning rate is written as \"10 −3\" with a stray space; please format as 10^-3.","section":"Section 3.1"},{"comment":"\"division modulo 96\" should be phrased as \"modular division with modulus 96\" for clarity.","section":"Section 3.1"},{"comment":"The reference to \"Figure 2 of [6]\" is not self-contained; either reproduce the figure or describe the relevant observations in the text.","section":"Section 4.4"},{"comment":"No code or data availability statement is included. Making the experimental code and checkpoints available would substantially improve reproducibility, especially given the missing estimator details.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The empirical core of this hep-th submission is an applied deep-learning experiment, and the missing experimental details are unusual for a physics journal. The editor may wish to confirm that the manuscript fits the journal's scope. Additionally, the RG discussion leans heavily on Refs. [21-23], two of which are self-citations; the editor may want to check that this does not reflect unnecessary self-citation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the empirical claim that the onset of generalization (grokking), the second descent in double descent, and the turn toward compression in the information plane all happen at the same training time, in two different transformer settings. The two-phase picture itself is not new — the paper's own citations already contain it — but the alignment of all three time scales in the same runs is something I have not seen stated before. If it holds up, it is a useful unification and makes I(T;X) a plausible monitoring tool for early stopping. The paper also writes clearly and the RG discussion is honest about being analogical, building on prior work by the same group.\n\nThat said, the evidence for the central claim is thinner than the paper admits. Figures 3 and 4 are single runs, read by eye. No seeds, no error bars, no checkpoint schedule, and — critically — no description of the mutual information estimator, let alone binning or sample counts. Mutual information between high-dimensional activations and inputs is notoriously fragile; a biased estimate can shift the apparent compression onset by orders of magnitude. Since the abstract and Section 4.1 lean on this alignment as the main support for the two-phase hypothesis, the missing estimator details are a load-bearing gap, not a minor omission. I also think the treatment of double descent as a training-time phenomenon (test error during a single run) needs explicit justification, since the standard double descent curve is about model complexity, not training time. Those are interpretive and empirical issues, not fatal ones.\n\nI disagree with any suggestion that the paper is circular: the alignment observation is new data, not fitted parameters. And the self-citations on RG are fine, since they are prior work and the paper does not pretend otherwise.\n\nBottom line: the hypothesis is interesting and worth taking seriously, but the empirical support currently does not carry the weight. The weaknesses are addressable with more seeds, an explicit estimator, error bars, and released code. I would send this to peer review rather than desk reject, because a serious referee could push it toward becoming a much stronger paper. If the authors can show the alignment survives across seeds and a described MI estimator, this could be a genuinely useful contribution. As it stands, I would not cite it in my own work yet; I would want the reproducibility details first.","headline":"A plausible two-phase synthesis of grokking, double descent, and information bottleneck, but the central alignment claim rests on under-specified single-run mutual information estimates.","tokens_in":679,"tokens_out":1142,"would_cite":false,"duration_ms":25685,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that grokking, double descent, and the information bottleneck are the same two-phase learning process, and that mutual information I(T;X) is the progress measure that tracks it.","keywords":["grokking","double descent","information bottleneck","two-phase learning","mutual information","coarse graining","renormalization group","deep learning dynamics"],"falsifier":"Train the same two transformer architectures on a task where grokking occurs, but compute mutual information with a known-accurate estimator (or with analytically tractable low-dimensional representations); if the decrease in I(T;X) does not begin at the same step as the second descent of test error, the claimed alignment fails. Alternatively, if a training algorithm that directly minimizes I(T;X) does not shorten the delay to generalization, the claim that phase two is incidental and compressible is undercut.","tokens_in":10121,"feed_emoji":"📉","tokens_out":4520,"duration_ms":40401,"temperature":0.7,"pith_summary":"The paper proposes that learning in deep neural networks is not a single smooth process but unfolds in two phases: a rapid curve-fitting phase, in which the network memorizes training data and drives training error to zero, followed by a slower compression or coarse-graining phase in which the network discards input details irrelevant to the task and thereby acquires generalization. It claims that three well-known phenomena—grokking, double descent, and the information bottleneck—are not separate curiosities but the same two-phase dynamics seen from different angles. The paper supports this by showing in two transformer settings (modular division and multiplication in S5) that the timescales align: the moment test error starts to drop matches the onset of compression in the information plane. If true, a single progress measure—the mutual information I(T;X) between a hidden layer and the input—tracks learning, and the slow second phase is not a fundamental prerequisite but an incidental side effect that tailored algorithms could shorten.","feed_headline":"Grokking, double descent, and the information bottleneck are one process","feed_subtitle":"Two experiments align the three delay phenomena; mutual information becomes the natural progress measure.","key_machinery":"The central object is the information plane trajectory of a hidden layer, with coordinates (I(T;X), I(T;Y)) — mutual information between the layer's representation and the input, and between the representation and the labels. The paper also uses the circuit framework for ReLU/GeLU networks: linear-region partitions of input space, local complexity, and the Linear Mapping Number as complexity measures. The argument's load-bearing move is the empirical alignment of three timescales: end of fitting, onset of second descent, and onset of compression.","core_discovery":"The paper's central claim is that learning in a deep neural network proceeds through two distinct phases. Phase one is curve fitting: the loss is actively minimized, training accuracy rises to perfect, and I(T;Y) grows as the network absorbs task-relevant information; this is the ascending limb before the interpolation threshold. Phase two is coarse graining: training loss stays at zero while the network reorganizes its linear regions, reduces local complexity and the linear mapping number, and gradually decreases I(T;X) while keeping I(T;Y) roughly constant. The paper's empirical claim is that the timescales of grokking, double descent, and the information bottleneck coincide: the point where training accuracy reaches perfect marks the end of fitting; the onset of the second descent in test error and the rightmost point of the information-plane trajectory both occur at the same training step; and the end of generalization corresponds to the information-plane fixed point. Grokking is therefore the visible face of this two-phase structure, and I(T;X) is proposed as a natural process measure of learning.","pith_inferences":["If the timescale alignment holds across more architectures and datasets, one could predict grokking's onset from information-plane measurements alone, without waiting for test accuracy to rise.","The claim that compression is incidental suggests a testable recipe: modifying the loss to penalize I(T;X) directly, or adding explicit compression pressure, should shorten phase two; the paper does not run this experiment.","The renormalization-group analogy implies the existence of marginal and relevant 'circuit parameters' that survive coarse graining; identifying them in trained networks could connect compression to generalization bounds.","The layer-depth insensitivity of phase-one end hints that the fitting-to-compression transition is governed by the global task structure rather than by individual layer architecture."],"forward_implications":["Grokking, double descent, and the information bottleneck are manifestations of the same two-phase learning dynamics; observations in one setting can be transferred to the others.","Mutual information I(T;X) serves as a progress measure whose decrease during the second phase signals movement toward generalization, and whose flattening marks the end of learning.","Because the second phase is not explicitly optimized by standard training algorithms, it can be unnecessarily prolonged; algorithms that deliberately promote compression could accelerate generalization.","The renormalization-group analogy suggests the second phase is a principled form of forgetting: irrelevant details, like irrelevant parameters in RG flow, are discarded, and this forgetting is critical for generalization.","The three geometric mechanisms—alignment, disappearance, and merging of zero-locus lines—provide a starting point for a dynamical theory of the compression phase."],"supporting_citations":[{"why":"Introduces grokking and the delayed generalization after zero training error.","marker":"[1]"},{"why":"Supplies the claim that grokking is widespread and motivates the circuit-based local complexity view.","marker":"[4]"},{"why":"Provides the layerwise visualization of circuit formation after grokking used in the coarse-graining discussion.","marker":"[6]"},{"why":"Establishes progress measures for grokking via mechanistic interpretability, motivating the search for a scalar progress measure.","marker":"[8]"},{"why":"Defines the Linear Mapping Number used as a complexity measure that decreases during the second phase.","marker":"[11]"},{"why":"Source for the double descent phenomenon and its interpolation threshold timescale.","marker":"[12]"},{"why":"Defines the information bottleneck objective and the trade-off between compression and prediction.","marker":"[15]"},{"why":"Proposes the two-phase interpretation of deep learning within the information bottleneck framework.","marker":"[16]"},{"why":"Provides empirical information-plane trajectories that the paper compares against.","marker":"[17]"},{"why":"Supplies the renormalization-group formalism underlying the analogy between coarse graining and learning.","marker":"[18]"}],"fun_headline_variants":["Two phases unify grokking, double descent, info bottleneck","Deep learning's two phases: fit then forget","Grokking and double descent share a single timescale","Learning's late phase is principled forgetting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole alignment rests on the reliability of the reported mutual information estimates I(T;X) and I(T;Y); if those numbers are biased by the estimator or binning, the coincident timescales could be an artifact.","fun_headline_variants_meta":{"raw":{"variants":["Two phases unify grokking, double descent, info bottleneck","Deep learning's two phases: fit then forget","Grokking and double descent share a single timescale","Learning's late phase is principled forgetting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1384,"prompt_tokens":886,"completion_tokens":498,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":434}},"tokens_in":502,"tokens_out":498,"duration_ms":5282,"temperature":1.0,"reasoning_tokens":434,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:24:14.607653+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same two transformer architectures on a task where grokking occurs, but compute mutual information with a known-accurate estimator (or with analytically tractable low-dimensional representations); if the decrease in I(T;X) does not begin at the same step as the second descent of test error, the claimed alignment fails. Alternatively, if a training algorithm that directly minimizes I(T;X) does not shorten the delay to generalization, the claim that phase two is incidental and compressible is undercut.","supporting_citations":[{"cited_title":"Grokking and the Geometry of Circuit Formation,","cited_arxiv_id":null,"evidence_quote":"Provides the layerwise visualization of circuit formation after grokking used in the coarse-graining discussion."},{"cited_title":"Deep double descent: Where bigger models and more data hurt,","cited_arxiv_id":null,"evidence_quote":"Source for the double descent phenomenon and its interpolation threshold timescale."},{"cited_title":"Deep learning and the information bottleneck principle,","cited_arxiv_id":null,"evidence_quote":"Proposes the two-phase interpretation of deep learning within the information bottleneck framework."},{"cited_title":"The renormalization group: Critical phenomena and the Kondo problem,","cited_arxiv_id":null,"evidence_quote":"Supplies the renormalization-group formalism underlying the analogy between coarse graining and learning."}],"review_version":1}