{"id":"aac20e64-0106-4a3d-a601-4b6061ee22f0","arxiv_id":"2601.06649","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper's finding that training efficiency declines monotonically with token count is guaranteed by its efficiency metric, which divides by token count and power consumption.","lead":"Training a 1.1B-parameter TinyLlama model on 500K, 1M, or 2M tokens produces a lower score on the paper's own energy-aware efficiency metric as token count rises. A generalist should read this to see why the main result is built into the metric's denominator rather than being discovered from the data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim rests on an average-efficiency metric whose denominator contains token count; monotonic decline is a near-tautology, not evidence about marginal energy inefficiency.","rationale":"The reader's weakest assumption centers on RMS power not being multiplied by time, so the metric measures average power rather than total energy. That is a valid and related concern: it contributes to the metric's inability to capture marginal energy cost. However, the more fundamental issue is the average-versus-marginal distinction. Even if total energy were measured perfectly, Eq. (3) would still divide by total token count, making the monotonic decline unavoidable for any saturating learning curve. The ANOVA and pairwise tests only confirm that the denominator changes as designed. The paper's abstract and conclusion explicitly invoke 'marginal performance improvements' and 'energetically inefficient,' but the analysis never isolates the marginal contribution of additional tokens. Thus the central claim is not supported by the reported evidence. I agree with the reader's overall REJECT verdict, but the specific load-bearing concern is the mismatch between the cumulative metric and the marginal claim, not solely the power-proxy issue. The proposed concrete test—computing marginal efficiency across adjacent token intervals—would directly settle whether the additional tokens are genuinely inefficient or whether the reported decline is an artifact of averaging.","tokens_in":5854,"tokens_out":5558,"duration_ms":53834,"concrete_test":"Reanalyze the raw trial data to compute, for each adjacent token-count interval, the marginal efficiency: ME = (mean invPPL at higher count − mean invPPL at lower count) / (mean total energy at higher count − mean total energy at lower count), where total energy = RMS(W) × duration or, equivalently, use TT_tokens as the denominator. Compare the two intervals. If ME for 1M→2M is not significantly lower than ME for 500K→1M, the claimed monotonic decline in training efficiency is an artifact of the cumulative denominator in Eq. (3), and the central claim is unsupported. If the raw data are unavailable, the authors should at least report the means and standard deviations necessary for this computation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that increasing token count is energetically inefficient even when marginal performance gains occur—requires measuring the marginal energy cost of additional tokens. Eq. (3) instead defines PE as a cumulative average efficiency: PE = K·invPPL·CS·TT_scale / (MS_params·TT_tokens·RMS(W)). Total token count TT_tokens appears in the denominator, and RMS(W) is a power average, not energy. If invPPL grows sublinearly with tokens (as it must for any saturating learning curve), PE automatically declines even if the marginal cost per additional token is constant. The reported F(2,98)=24,268.23 with η²_g=.997 is a near-deterministic consequence of this algebraic structure, not an empirical discovery. The separate finding that RMS power increases with token count is plausible but does not rescue the inference: even if RMS power were constant, the PE decline would persist. To support the claim, the paper must compare marginal efficiency across intervals—e.g., Δ(invPPL)/Δ(energy) for the 500K→1M versus 1M→2M increments—and show that the latter is worse. No such marginal analysis is presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a repeated-measures experiment in which a TinyLlama 1.1B model is trained on 500K/1M/2M true tokens, with 50 fresh trials per condition on a fixed AWS A10G GPU, while RMS GPU power is sampled every 60 seconds. The central claim is that an energy-aware parameter-efficiency metric, PE = K·invPPL·CS_TFLOPS·TT_scale / (MS_params·TT_tokens·RMS(W)), declines strictly monotonically as token count increases, and that this indicates 'increases in training token counts may be energetically inefficient even when marginal performance improvements are observed.' A repeated-measures ANOVA on PE is reported as F(2,98)=24,268.23, p<.001, η²_g=.997, with all Bonferroni-corrected pairwise comparisons significant. A secondary analysis reports that mean RMS power also increases monotonically with token count.","tokens_in":6122,"tokens_out":3491,"duration_ms":36520,"significance":"The study has genuine strengths: a controlled setup with fixed architecture, optimizer, hardware, and epoch count is a good basis for studying scale effects; a 150-trial design is reasonably sized; and direct power monitoring is a step beyond purely performance-centric scaling studies. If the inference were valid, the result could be practically relevant for energy-aware deployment decisions. However, the central inference is not valid as presented. The PE metric in Eq. (3) places total token count and RMS power in the denominator, so the reported monotonic decline is a mathematical consequence of the metric rather than an empirical discovery. The separate power analysis shows only that RMS power draw increases with token count, not that total energy per unit of performance worsens. The statistical design is also misaligned with the repeated-measures label. The paper's value is limited to a useful reminder that power draw should be reported and considered in scaling evaluations; it does not support the stated conclusion.","major_comments":[{"comment":"The central claim is forced by the metric's definition. PE = K·invPPL·CS_TFLOPS·TT_scale / (MS_params·TT_tokens·RMS(W)). Total token count TT_tokens appears in the denominator, and RMS(W) is reported to increase with token count (§4). Unless invPPL grows faster than linearly in tokens, PF must decline; inverse perplexity saturates for typical learning curves. The reported F(2,98)=24,268.23 with η²_g=.997 is therefore a near-deterministic consequence of the algebraic structure rather than evidence about marginal energy inefficiency. To support the claim, the paper must compare marginal efficiency across intervals, e.g., Δ(invPPL)/Δ(energy) for 500K→1M versus 1M→2M, and show that the latter is worse. No such marginal analysis is presented.","section":"§1, Eq. (3)"},{"comment":"The energy proxy is not energy. Eq. (4) defines RMS(W), an average power, and the metric never multiplies by time. The paper mentions power sampling counts as a proxy for duration (§4), but total energy consumption is never computed. If longer runs simply sustain the same average power for more time, the PE decline would persist even when marginal energy per token is constant. Thus the conclusion that token scaling is 'energetically inefficient' is unsupported. The power analysis shows only that mean RMS power differs across conditions, not that total energy per token or per unit of performance worsens.","section":"§3 Methods, Eq. (4)"},{"comment":"The repeated-measures ANOVA is misapplied. The design uses 50 fresh model/dataset initializations per token-count condition; there is no repeated measurement on the same unit across conditions. Calling the 50 trials per condition 'within-subject' observations conflates trial-to-trial variability with a repeated-measures design. The appropriate analysis would be between-subjects or a mixed model with trials as a random factor. This does not affect the algebraic inevitability of the PE decline, but it affects the statistical reporting and the claim of a 'repeated-measures' experimental design.","section":"§3 Methods, §4 Results"}],"minor_comments":[{"comment":"Raw p-values are reported as 0.00. They should be reported as p < .001.","section":"§4, Tables 1 and 3"},{"comment":"There are formatting glitches such as 'ANOV A' and missing spaces (e.g., 'under representing,' 'PEdissertation'). These should be corrected.","section":"Abstract/throughout"},{"comment":"The 'stability threshold' for batch skipping is not described in enough detail to assess reproducibility; please provide the exact criterion.","section":"§3 Methods"},{"comment":"Several references are incomplete or self-referential, e.g., '(2026)' and 'Dwyer, 2026' in the Methods text. These should be completed or removed.","section":"§3 Methods"},{"comment":"The limitations section restricts generalizability to other architectures/hardware but does not acknowledge that the core metric's denominator makes the main decline at least partially definitional. This should be addressed directly.","section":"§6 Limitations"}],"recommendation":"reject","confidential_remarks":"The paper's central result is a tautology created by the definition of the efficiency metric in Eq. (3). The enormous F-statistic and η²_g are expected under that definition and do not constitute empirical evidence. The power analysis is also not a correct measure of energy. Because the main claim cannot be rescued without redefining the metric and performing new marginal analyses, the manuscript is not suitable for publication in its current form. The controlled setup and large number of trials are commendable, but the contribution as presented does not meet the standard for an empirical discovery in this venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: the central claim doesn't survive contact with its own equation. Eq. (3) puts total tokens and RMS power in the denominator, so the monotonic decline in PE is a mathematical requirement, not an empirical finding. The F statistic of 24,268 is exactly what you'd expect from a near-deterministic ratio. The paper's own wording calls it a confirmatory replication, which is fair — but it confirms a tautology.\n\nWhat's worth credit: the controlled setup is real. Same architecture, hardware, optimizer, three token counts, 50 fresh trials per condition. The power sampling is a sensible addition, and the secondary result that RMS power draw increases with token count is plausible and internally consistent. The limitations section is honest about single-model/single-GPU scope.\n\nThe soft spots are not minor. First, the metric is a cumulative average, so it says nothing about marginal efficiency. The actual question — does each additional token cost more energy per unit of performance than the previous interval? — is never asked. Second, Eq. (4) is RMS power, not energy. Without multiplying by duration, the 'energy-aware' label is misleading; longer runs at constant average power would show the same decline because tokens are in the denominator. Third, the repeated-measures ANOVA pairs 50 fresh trials per condition without any pairing mechanism; that's a design error. Fourth, there is no code or data, and the prior work that defines the metric and supplies the data is an unpublished dissertation and a Zenodo record that isn't detailed. The heavy self-citation is a symptom of that reliance.\n\nSo the paper is a clear example of a metric that guarantees its own conclusion. The raw power measurements might still be useful as a small systems datapoint, but the stated takeaway is unsupported. I would not send it to peer review as-is; desk rejection with an invitation to resubmit a marginal-efficiency analysis with code and data is the right call. Not a paper I'd cite.","headline":"The central result is a tautology of its own metric; controlled setup but load-bearing design errors.","tokens_in":6596,"tokens_out":2955,"would_cite":false,"duration_ms":29715,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that, at fixed hardware and architecture, increasing training tokens lowers energy-aware parameter efficiency monotonically even when inverse perplexity changes little.","keywords":["LLM training","parameter efficiency","token scaling","power consumption","energy-aware evaluation","repeated measures ANOVA","compute overhead"],"falsifier":"Compute total energy as the time integral of sampled power for each trial and divide by tokens processed. If energy per token is flat or lower at 1M and 2M tokens than at 500K, the paper's central claim fails; this is directly checkable by logging wall-clock time alongside the power samples.","tokens_in":5688,"feed_emoji":"⚡","tokens_out":6467,"duration_ms":62475,"temperature":0.7,"pith_summary":"This paper asks whether adding training tokens is worth the energy it costs. Holding hardware, architecture, optimizer, and epochs fixed, it trains a 1.1-billion-parameter TinyLlama model on 500K, 1M, and 2M tokens and reports a strictly monotonic decline in an energy-aware parameter-efficiency metric as tokens increase. The driver is not a collapse in model quality—inverse perplexity moves only modestly—but a systematic rise in average GPU power consumption and longer runs. The conclusion is that token-scale increases can be energetically inefficient even when conventional performance metrics show only diminishing or inconsistent gains. The paper positions this as a confirmatory extension that treats energy as a first-class evaluation variable.","feed_headline":"Token scaling raises power draw faster than perplexity improves","feed_subtitle":"A 1.1B-parameter model run on 500K to 2M tokens shows a monotonic drop in energy-aware efficiency.","key_machinery":"The central object is the energy-aware parameter efficiency ratio PE = K · invPPL · C_S / (TFLOPS · M_params · T_tokens · RMS(W)), where invPPL is inverse perplexity, M_params is the parameter count, T_tokens is the total tokens processed (target tokens times epochs), and RMS(W) is the root-mean-square GPU power draw defined by Eq. (4). Token count enters the denominator, and the paper's own power ANOVA shows that RMS(W) also rises with token count, so the metric is structured to decline unless inverse perplexity grows faster than the product of tokens and power. The fixed TFLOPS term acts as a throughput normalization constant; the paper uses this ratio to convert training runs into a singl","core_discovery":"On its own terms, the paper establishes that under fixed hardware and training conditions, moving from 500K to 1M to 2M training tokens produces a statistically significant drop in energy-aware parameter efficiency, with token count explaining nearly all within-subject variance (repeated-measures ANOVA F(2,98)=24,268.23, p<.001, generalized eta-squared=.997). A separate repeated-measures ANOVA on RMS GPU power shows power consumption increasing monotonically with token count (F(2,98)=3143.21, p<.001, eta-squared=.969). Because inverse perplexity varies only modestly across conditions, the authors attribute the efficiency collapse primarily to rising energy use and duration rather than to deg","pith_inferences":["A direct test the paper does not run: compute time-integrated energy (watts × seconds) per million tokens. The reported metric uses RMS power only, so the monotonic decline could be driven by more tokens in the denominator; if total energy per token is flat, the central conclusion would need softening.","The monotonically rising RMS power across longer runs may indicate that the GPU stays at higher utilization for longer rather than that each additional token costs more energy. Separating those requires higher-frequency sampling and a wall-clock time log.","The framework could be extended to other training choices—batch size, learning rate, data order—to rank them by energy per unit of perplexity gain instead of by loss alone, giving practitioners a sustainability-oriented selection tool."],"forward_implications":["At fixed architecture and hardware, adding training tokens will reduce this energy-aware efficiency metric because power draw rises and tokens enter the denominator.","Evaluations that report only loss or perplexity can miss the energy burden; including power can change which training configuration is judged better.","The repeated-measures design shows the effect is consistent within a single GPU instance and holds after Bonferroni correction, making the metric usable as a monitoring tool in fixed-hardware settings.","Because the metric formulation is not tied to a particular architecture or device, the same measurement can be ported to other model families and hardware to test whether the efficiency decline persists.","Efficiency-aware analysis reframes diminishing returns in neural scaling as partly an energy cascade: performance may plateau while energy cost keeps rising."],"fun_headline_variants":["Training on more tokens burns energy without proportional gains","Token scaling: more compute, less energy efficiency","Energy-aware efficiency drops as token count rises","More tokens: more power, no better perplexity","Token increases cut energy efficiency despite marginal gains"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole conclusion depends on treating the average GPU power draw, sampled every 60 seconds, as the energy cost; if longer runs simply draw about the same power for more time, the reported decline is an artifact of dividing by more tokens rather than evidence that efficiency worsened.","fun_headline_variants_meta":{"raw":{"variants":["Training on more tokens burns energy without proportional gains","Token scaling: more compute, less energy efficiency","Energy-aware efficiency drops as token count rises","More tokens: more power, no better perplexity","Token increases cut energy efficiency despite marginal gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000738,"raw_usage":{"total_tokens":3149,"prompt_tokens":775,"completion_tokens":2374,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":2304}},"tokens_in":519,"tokens_out":2374,"duration_ms":16176,"temperature":1.0,"reasoning_tokens":2304,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T11:20:20.943247+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute total energy as the time integral of sampled power for each trial and divide by tokens processed. If energy per token is flat or lower at 1M and 2M tokens than at 500K, the paper's central claim fails; this is directly checkable by logging wall-clock time alongside the power samples.","supporting_citations":[],"review_version":1}