{"id":"4f87b761-bf87-4f0f-bd15-2b827e13a742","arxiv_id":"2505.11643","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Training GPT-2 small on an easy-to-hard curriculum speeds early learning at moderate thresholds but yields worse final accuracy and relies on unreliable head-count metrics.","lead":"A small GPT-2 model was trained on a four-stage easy-to-hard curriculum and compared with a single-phase baseline. The curriculum reaches moderate accuracy faster and shifts attention patterns, but final answer accuracy is about 30% lower.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline transparency claim rests on head counts that exceed the model's physical head count, so the central metric is internally inconsistent as reported.","rationale":"The reader identified the gradient-saliency probe as the weakest assumption, and the full text confirms this with a stronger form: the reported counts are not merely an imperfect proxy but are numerically impossible as counts of heads. GPT-2 small has 144 attention heads; an average of 6,814 specialized heads per checkpoint cannot be a count of heads unless the paper is using 'head' in a nonstandard sense that is never defined. The paper's own limitation statement explicitly warns that accumulation inflates counts, yet the headline metric is precisely an accumulated/cumulative number. This makes the central transparency claim rest on a metric that contradicts the model's architecture. The 24-layer figure in Section 3.1 contradicts Section 2.1's 12-layer description, further undermining the layer-redistribution result. The sample-efficiency claim is also weaker than advertised: the curriculum reaches 0.25-0.30 success in 500 vs 1,000 steps, but the baseline is trained for 60k steps while the curriculum stops at ~10k, so the comparison is not at matched optimization budgets for the headline 'identical compute budget' statement; and final success is 32% worse. However, the head-count inconsistency is the most load-bearing because it invalidates the primary mechanistic evidence for 'reasoning transparency,' which is half of the paper's central contribution. The paper deserves credit for releasing code, running ablations, and candidly listing limitations, but those strengths do not rescue a headline result built on an internally inconsistent metric. The reader's REJECT verdict is appropriate; my independent analysis does not change it.","tokens_in":12526,"tokens_out":2616,"duration_ms":28306,"concrete_test":"Clone the released repository, inspect config.json for n_layer and n_head, and rerun the RepresentationTracker on the final curriculum checkpoint to compute the set of unique (layer, head) pairs above the saliency threshold. If the number of unique pairs exceeds 144 (or 288 if the config actually has 24 layers), the probe is not counting distinct heads and the headline head-count comparisons in Table 1 and Figures 2-3 cannot be interpreted as stated. As a second check, replot Figure 3 with the actual n_layer from config.json; if the 24-layer axis persists alongside a 12-layer config, the architectural inconsistency is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1 states the model is GPT-2 small with 12 transformer layers and 12 attention heads per layer, i.e. 144 heads total. Yet Section 3.1 and Table 1 report averages of 6,814 specialized heads per checkpoint, a final total of 4,126 heads, and stage-wise cumulative unions of 4,355 heads; Figure 3 and the layer-group tables are drawn over 24 layers with 288 heads. These numbers cannot be literal counts of distinct attention heads, since they exceed the entire head budget by more than an order of magnitude. The paper's own Limitations section concedes that the gradient-saliency detector 'can inflate counts when accumulated across checkpoints' and 'fails altogether in Stage 4,' which is exactly the aggregation regime used for the headline 'order-of-magnitude more gradient-salient reasoning heads' claim. Because the central assertion of improved reasoning transparency is operationalized by these counts, and because the counts are not physically interpretable as heads, the transparency result is not supported as stated. The 12-layer/24-layer discrepancy is not a minor typo: it changes the denominator of every per-layer statistic and makes Figure 3's 'layers 12–23' nonsensical for a 12-layer model. This is an internal inconsistency, not merely a disagreement with external consensus. While the sample-efficiency and attention analyses are reported separately, the paper frames the head-count gain as the primary mechanistic evidence for curriculum-induced transparency, so the invalidity of that metric undermines the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Cognivolve, a four-stage easy-to-hard curriculum for GPT-2 small (claimed 124M parameters, 12 layers, 12 heads per layer), and compares it against a single-phase baseline trained on the same aggregated data. The authors report three main benefits of the curriculum: roughly two-fold faster attainment of moderate accuracy thresholds, an order-of-magnitude increase in the number of gradient-salient \"specialized reasoning heads,\" and a redistribution of those heads toward deeper layers accompanied by higher-entropy attention. The paper also reports mixed final-task results: the curriculum improves step-by-step alignment by 2.9% but lags the baseline by 31.8% on final-answer success. The manuscript includes a Limitations section that candidly acknowledges the saliency probe is lossy, inflates counts under checkpoint accumulation, and fails entirely in Stage 4, and that the final accuracy gap remains unresolved.","tokens_in":12757,"tokens_out":4357,"duration_ms":45049,"significance":"If the central claims were valid, the result would be significant: a training curriculum that halves compute while producing more interpretable reasoning components in a small language model would be of broad interest to the efficient-NLP and interpretability communities. The paper ships code, data splits, and checkpoints, and the authors are transparent about known weaknesses. However, the quantitative backbone of the interpretability claim is internally inconsistent to the point of impossibility: reported head counts exceed the physical number of attention heads in the model, and the attention statistics are computed over 24 layers although the model is stated to have 12. These are not cosmetic issues; they invalidate the headline mechanistic evidence. The sample-efficiency comparison is also compromised by contradictory statements about the baseline's compute budget and by a final accuracy that is lower than thresholds the curriculum is claimed to have reached early in training. Because the main positive claims rest on these numbers, the significance of the paper as written is not established.","major_comments":[{"comment":"The model is specified in §2.1 as GPT-2 small with 12 transformer layers and 12 attention heads per layer, which gives 144 heads total. Yet §3.1 and Table 1 report 6,814 average specialized heads per checkpoint, 4,126 final total heads, and 4,355 cumulative heads, and Figure 3 and the layer-group tables are drawn over 24 layers with 288 heads. These numbers cannot be literal counts of attention heads because they exceed the model's entire head budget by more than an order of magnitude. The 12-layer/24-layer discrepancy also changes the denominator of every per-layer statistic and makes the 'layers 12–23' analysis in Figure 3 and Table 3 meaningless for the stated architecture. This is an internal inconsistency in the central metric, not a presentation issue.","section":"§2.1, §3.1, Table 1, Figure 3"},{"comment":"The headline transparency claim rests on counts of 'gradient-salient reasoning heads' obtained by accumulating detections across checkpoints. The Limitations section concedes that the saliency probe is lossy, 'can inflate counts when accumulated across checkpoints,' and 'fails altogether in Stage 4.' Stage 4 is exactly where the paper reports the cumulative union freezing and live heads dropping to zero. The reported order-of-magnitude gain in specialized heads is therefore an artifact-prone aggregation of a probe that the authors themselves say is not a census of distinct functional modules. Because the central claim of improved reasoning transparency is operationalized by these numbers, the transparency result is not supported as stated.","section":"§3.1, §3.5, Limitations"},{"comment":"Section 2.5 states the baseline has 'identical parameter count, compute budget, and total number of optimisation steps,' but Section 3.2 reports that the curriculum run terminates at roughly 10k steps while the baseline continues to 60k. These statements are contradictory. If the baseline actually trains for six times as many updates, the claimed 2x speed-up to reach 0.25 and 0.30 accuracy is not a controlled comparison of equal compute budgets; at minimum, the manuscript must clarify how 'identical compute budget' is compatible with a 60k-step baseline.","section":"§2.5, §3.2, Table 2"},{"comment":"There is a direct numerical inconsistency in the sample-efficiency claim. Table 2 reports that the curriculum reaches a 0.30 success rate at 500 updates, but Table 4 reports an end-of-training success rate of 0.21 for the curriculum. If the curriculum truly attained 0.30 early in training and then ended at 0.21, the claim of 'reaching target accuracy in half the optimization steps' is misleading without an explanation of the subsequent collapse. If instead the numbers come from different evaluation protocols or checkpoint averaging, that needs to be stated explicitly. As written, the two tables cannot both describe the same evaluation metric.","section":"Table 2, Table 4, §3.2"},{"comment":"The attention-pattern analysis in Table 3 and Appendix B consistently aggregates over 'all 24×12 heads' and reports layer groups including layers 12–23, again for a model stated in §2.1 to have 12 layers and 12 heads per layer. This makes the layer-wise entropy, sparsity, local-focus, and average-distance statistics undefined for the stated architecture. Since Section 3.3 interprets these statistics as mechanistic evidence for the curriculum effect, this inconsistency further undermines the paper's central quantitative claims.","section":"Table 3, §3.3, Appendix B"}],"minor_comments":[{"comment":"Figure 2 says the shaded region denotes one standard deviation across three seeds, while Section 2.6 says all results are averaged over five random seeds. These numbers should be reconciled.","section":"Figure 2 caption"},{"comment":"Table 4 reports a baseline final success rate of 0.32, but Section 3.2 states that the baseline 'eventually nudges success above 0.4.' These are inconsistent descriptions of the same experiment.","section":"Table 4 vs. §3.2"},{"comment":"Section 3.2 says the curriculum terminates at roughly 10k steps, but Section 3.7 places the onset of the final complex-reasoning stage at approximately 11,500 updates. These numbers cannot both be correct for the same training run.","section":"§3.2 vs. §3.7"},{"comment":"The sentence 'These cumulative tallies reach 4040, 4145 and 4355 for stages 1–3' is unclear because Table 5's 'Cumulative' column and 'Live Heads' column are not distinguished in the text; please clarify whether these are per-checkpoint live counts or cumulative unions at each stage boundary.","section":"§3.5, Table 5"}],"recommendation":"reject","confidential_remarks":"The paper has multiple load-bearing internal contradictions: impossible attention-head counts, a 12-layer model analyzed as 24 layers, contradictory statements about the baseline compute budget, and inconsistent accuracy numbers across tables. These are not fixable by copyediting because they concern the reported data and the central quantitative claims. The candid Limitations section is a strength, but it confirms rather than mitigates the fragility of the main metric. I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things before spending time on this. The empirical setup is genuine: a four-stage easy-to-hard curriculum on GPT-2 small, with cleaned data, released code and checkpoints, plus some controls (shuffled order, optimizer resets). The sample-efficiency comparison is the strongest part—it is an external behavioral metric, and the 2x speed-up to reach 0.25–0.30 accuracy is a clean, falsifiable result. But the central transparency claim is not supported as stated. The paper reports averages of 6,814 specialized heads per checkpoint and cumulative unions of 4,355 heads, while the model it describes has 12 layers and 12 heads per layer, i.e., 144 heads total. One table even says the cumulative pool is 15% of the 288-head budget—but the architecture section says 12 layers. That is not a typo; it changes every per-layer denominator and makes the 24-layer figures in the text and appendix meaningless. The paper's own Limitations section concedes the gradient-saliency probe 'can inflate counts when accumulated across checkpoints' and 'fails altogether in Stage 4.' That is exactly the aggregation regime used for the headline claim, so the 'order-of-magnitude' gain is not physically interpretable. The behavioral side is also mixed: the curriculum is 31.8% worse on final-answer accuracy, and the step-by-step advantage is a thin 2.9%. The speed-up claim is limited to a narrow accuracy band; both runs cross the lowest thresholds at the first checkpoint, and the curriculum stops at ~10k steps while the baseline runs to 60k, so part of the apparent gain is just a shorter run. Where the paper deserves credit: it is honest about these limitations, the baseline is matched in compute budget for the steps both runs share, and the attention statistics (entropy, Gini, distance) are reproducible. A reader interested in curriculum learning will find a useful cautionary example, but not a valid demonstration of improved transparency. This paper deserves a serious referee because the experiment is real and the data/code are released, but my own verdict is that the main claim fails. A revision that fixes the architecture inconsistency, defines head counts as saliency events rather than physical heads, and withholds the transparency conclusion could be reconsidered.","headline":"The curriculum setup is real and the sample-efficiency comparison is a legitimate external result, but the headline transparency numbers cannot be literal head counts and the paper's own limitations concede the probe's collapse.","tokens_in":13321,"tokens_out":2240,"would_cite":false,"duration_ms":22845,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an ordered four-stage curriculum trains a 124-million-parameter GPT-2 to reach target reasoning accuracy in half the optimization steps of a single-phase baseline, while activating roughly an order of magnitude more…","keywords":["curriculum learning","small language models","GPT-2","reasoning transparency","sample efficiency","attention heads","gradient saliency","chain-of-thought"],"falsifier":"Run the paper's own leakage diagnostic—randomly interleave 10% of Stage-4 problems into each earlier epoch and check whether validation accuracy remains unchanged; if it does, the curriculum's gains do not depend on strict progression. Separately, causally ablate the heads the saliency probe flags and see whether step-by-step accuracy drops; if it does not, the flagged 'reasoning heads' are not load-bearing.","tokens_in":12249,"feed_emoji":"🧠","tokens_out":5785,"duration_ms":55677,"temperature":0.7,"pith_summary":"This paper argues that the order in which a small language model meets training data can substitute for scale. It trains a 124-million-parameter GPT-2 on a four-stage syllabus that moves from one-step lexical problems to multi-step symbolic inference, and claims the staged model reaches target accuracy in half the optimizer updates of a single-phase baseline, activates roughly an order of magnitude more gradient-salient reasoning heads, and moves those heads into deeper layers. If true, easy-to-hard curricula would be a cheap way to make small models reason more transparently and efficiently. The same curriculum applied out of order, or with optimizer resets, does not reproduce the gains, which the paper takes as evidence that progression itself—not extra compute—does the work. The paper also concedes that the curriculum run's final-answer success lags the baseline by about 30% and that its saliency probe detects no specialized heads in the hardest stage.","feed_headline":"Easy-to-hard curriculum halves reasoning-training steps","feed_subtitle":"A 124M GPT-2 also lights up 8x more attention heads in deep layers—though final accuracy trails by 30 percent.","key_machinery":"The load-bearing mechanism is the ordered syllabus itself: four difficulty tiers (simple, basic, intermediate, complex reasoning) built by a logistic classifier over operator density, sentence count, and step delimiters, presented strictly in increasing difficulty with no optimizer resets so that weights, token embeddings, and scheduler state carry over. The paper's evidence for transparency is a gradient-based saliency probe that labels an attention head 'specialized' when its saliency on a held-out reasoning probe exceeds the 95th percentile of a random-head null distribution; these counts are used to quantify component growth, layer-wise redistribution, retention between stages, and emergence speed. Attention maps supply supporting measures—Gini sparsity, Shannon entropy, local focus within a ±2-token window, and mean key-query distance—that describe how the curriculum changes information flow.","core_discovery":"The paper's central claim is that a developmentally ordered curriculum can unlock specialized reasoning circuitry in a small transformer without adding parameters. In Cognivolve, the same GPT-2 weights are trained continuously across four difficulty tiers—simple reasoning, basic reasoning, intermediate reasoning, complex reasoning—with per-stage learning-rate ceilings and no task-specific layers. Against a matched single-phase baseline that sees the same corpus and compute budget, the curriculum run reaches success-rate thresholds of 0.25 and 0.30 in 500 updates versus 1,000, ends with an average of 6,814 gradient-salient specialized heads per checkpoint versus 873, fills previously empty late layers (12–23) with up to 193 heads each, and raises attention entropy by about 2%. The paper interprets these as signs of broader, more balanced information flow and of reasoning circuitry shifting toward deep layers, and argues this constitutes a transparency gain: the curriculum model's step-by-step reasoning traces align with gold rationales 2.9% better than baseline. It also reports that final-answer success remains 31.8% below baseline, which it attributes to termination policy and loss weighting rather than to the curriculum itself, and that the saliency probe under-detects verbal-knowledge heads in Stage 4.","pith_inferences":["The 7.8x gap in head counts should be read cautiously: the paper's own limitation statement says the saliency probe is lossy, can inflate counts when accumulated across checkpoints, and finds zero heads in Stage 4, so a causal test—ablating or silencing the flagged heads and checking whether step-by-step accuracy drops—would settle whether these are functional reasoning circuits.","A reader could test the paper's leakage concern directly by randomly interleaving 10% of Stage-4 problems into each earlier epoch; if validation accuracy stays flat, the curriculum's gains are not due to memorizing stage boundaries.","A natural extension is a brief mixed-stage fine-tune after Stage 4, which the paper leaves open; this could determine whether the 30% final-accuracy gap is a fixable termination artifact or a fundamental trade-off between transparency and answer accuracy.","If the same four-stage ordering were applied to larger GPT-2 variants, the saturation point of curriculum gains would reveal how far ordering can substitute for scale before model capacity becomes the bottleneck."],"forward_implications":["Curriculum ordering can substitute for compute: a small model reaches moderate reasoning accuracy in half the optimizer updates at equal parameter count and wall-clock budget.","Small models can be trained to use deeper layers for reasoning; the baseline leaves layers 12–23 with zero specialized heads, while the curriculum fills them.","Progression, not extra data or compute, drives the gains: shuffling the stage order or resetting the optimizer at stage boundaries fails to reproduce the benefits.","Transparency and final accuracy can decouple: the curriculum improves step-by-step reasoning alignment while final-answer success lags, suggesting these objectives need separate tuning.","Order-of-magnitude more specialized reasoning heads means interpretability tools can study reasoning circuits in small models, but the probe's Stage-4 blind spot limits where that analysis applies."],"supporting_citations":[{"why":"Supplies the Natural Reasoning corpus that is cleaned and split into the four curriculum difficulty tiers.","marker":"Yuan et al., 2025"},{"why":"Establishes curriculum learning, the paradigm the four-stage syllabus applies to language modeling.","marker":"Bengio et al., 2009"},{"why":"Provides the few-shot in-context learning and scaling baseline that the paper contrasts with its small-model curriculum approach.","marker":"Brown et al., 2020"},{"why":"Defines the chain-of-thought behavior the paper measures through its step-by-step alignment metric.","marker":"Wei et al., 2022"},{"why":"Supplies the AdamW optimizer configuration used for both curriculum and baseline runs.","marker":"Loshchilov and Hutter, 2019"},{"why":"Motivates the developmental ordering rationale by connecting human cognitive development to curriculum learning for language models.","marker":"Campos, 2021"}],"fun_headline_variants":["Easy-to-hard curriculum halves reasoning-training steps","Curriculum order activates 10x more deep reasoning heads in GPT-2","Ordered curriculum: faster reasoning, but 30% accuracy lag","Four-stage syllabus sparks reasoning in 124M model, cuts training 2x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central transparency claim rests on the assumption that the gradient-based saliency probe accurately counts distinct, functional reasoning heads, but the paper's own limitations state the probe is lossy, can inflate counts when accumulated across checkpoints, and detects zero heads in the hardest stage.","fun_headline_variants_meta":{"raw":{"variants":["Easy-to-hard curriculum halves reasoning-training steps","Curriculum order activates 10x more deep reasoning heads in GPT-2","Ordered curriculum: faster reasoning, but 30% accuracy lag","Four-stage syllabus sparks reasoning in 124M model, cuts training 2x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001191,"raw_usage":{"total_tokens":4937,"prompt_tokens":993,"completion_tokens":3944,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":3870}},"tokens_in":609,"tokens_out":3944,"duration_ms":27971,"temperature":1.0,"reasoning_tokens":3870,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:50:10.998503+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's own leakage diagnostic—randomly interleave 10% of Stage-4 problems into each earlier epoch and check whether validation accuracy remains unchanged; if it does, the curriculum's gains do not depend on strict progression. Separately, causally ablate the heads the saliency probe flags and see whether step-by-step accuracy drops; if it does not, the flagged 'reasoning heads' are not load-bearing.","supporting_citations":[{"cited_title":"Louradour, R","cited_arxiv_id":null,"evidence_quote":"Establishes curriculum learning, the paradigm the four-stage syllabus applies to language modeling."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the chain-of-thought behavior the paper measures through its step-by-step alignment metric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the developmental ordering rationale by connecting human cognitive development to curriculum learning for language models."}],"review_version":1}