{"id":"1596fc13-781b-42c1-9f9f-110c88e26d3d","arxiv_id":"2509.10537","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A naive gradient step-up function with critical-phase detection can, for selected hyperparameters, make large-batch training match or beat small-batch training accuracy at equal iteration counts, though the proposed teacher-model version is not evaluated.","lead":"This paper proposes a heuristic for large-batch federated learning: scale up gradients by a factor X outside 'critical' training phases to mimic small-batch behavior, and reports up to 32.33% higher test accuracy than small-batch training on ResNet50 for equal iteration counts. The central technique it advocates, a teacher model that maps large-batch gradients to small-batch gradients, is described but never implemented.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 32.33% accuracy gain is confounded by sample count: large-batch runs see 32x more data per iteration, so equal-iteration comparison does not isolate gradient scaling.","rationale":"The reader's verdict is REJECT, and my analysis supports that verdict. The reader's weakest_assumption focuses on the unvalidated modeling assumption in Eq. (7) that scaled large-batch gradients approximate small-batch gradients. That is a real concern, but I find an even more load-bearing problem earlier in the argument: the headline comparison at equal iteration counts does not equalize the number of training samples seen. Since the large-batch runs process 32x (ResNet50) or 16x (VGG11) more samples per iteration, the reported accuracy advantage is fully confounded with increased data exposure. This alone invalidates the central claim as stated, independent of whether Eq. (7) holds. The reader's rationale mentions the same sample-count issue, so agreement is partial: the reader identified it in the rationale but not as the weakest assumption. My concrete test would settle the concern by matching samples processed; if the advantage disappears, the paper's central claim is unsupported. If it survives, the scaling idea would deserve a more positive reading. No change to the reader's verdict is needed.","tokens_in":11753,"tokens_out":1960,"duration_ms":17957,"concrete_test":"Train ResNet50 on CIFAR10 under two conditions: (a) B=1024 with X=16 as in the paper, and (b) B=32 with 32x more iterations so both see the same total number of samples (same epochs). Plot test accuracy vs samples processed. If (a) no longer beats (b) by a large margin, the scaling benefit is an artifact of unequal sample counts. Also run B=1024 with X=1 under the same sample-matched protocol to isolate the step-function effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim compares test accuracy at the same number of iterations without holding the number of training samples constant. For ResNet50, the B=1024 run with X=16 uses 1024 samples per iteration, while the B=32 baseline uses 32; after T iterations the large-batch model has seen 32T samples and the small-batch model only T samples. Thus the reported 32.33% 'improvement' may simply reflect 32x more epochs over the training set, not any benefit of the proposed step-function scaling. The step function itself (Eq. 6-7) is equivalent to multiplying the update by X, i.e., scaling the learning rate by X outside critical phases; any LR-scaling heuristic could produce such gains when the comparison is confounded. The threshold and X are also selected per model after seeing results, so without equal-sample/equal-epoch comparisons or multiple seeds the headline claim is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses large-batch training in federated learning. It introduces a memory-estimation model and a compute-time model to predict batch-size bounds, and proposes a \"gradient mapping\" approach in which a teacher model maps large-batch gradients to small-batch-like gradients. As a naive substitute for the teacher model, it evaluates a step function that scales gradients by a factor X except in \"critical training phases\" detected via a gradient-change metric. Experiments on ResNet50/CIFAR10 and VGG11/CIFAR100 report that with this scaling, large-batch training (B=1024) achieves higher test accuracy than small-batch training for the same number of iterations, up to 32.33% higher for ResNet50 and 3.74% higher for VGG11, with respect to small-batch baselines at B=32 and B=64 respectively. The proposed teacher model itself is not implemented; only the step-function surrogate is evaluated.","tokens_in":11916,"tokens_out":4171,"duration_ms":36105,"significance":"If the headline result were valid, it would be significant for federated learning: a simple gradient-scaling heuristic with critical-phase detection could reduce the large-batch generalization gap while preserving the parallel efficiency of large batches. The paper also provides useful empirical timing and memory measurements across several models. However, the central empirical claim is confounded by the number of training samples seen per iteration, and the reported improvements depend on per-model, post hoc selection of the scaling factor and threshold. Without an equal-sample/equal-epoch comparison, multiple seeds, or any validation of the underlying gradient-equivalence assumption, the evidence does not establish the claimed effect.","major_comments":[{"comment":"The headline comparison is confounded by sample count. Reporting test accuracy after the same number of iterations with B=1024 versus B=32 does not hold the number of training samples constant: after T iterations the large-batch run has processed 32T samples while the small-batch run has processed only T samples. The claimed 32.33% improvement for ResNet50 could therefore reflect the larger number of epochs over the training data rather than any benefit of the proposed step-function scaling. An equal-epoch or equal-sample-count comparison, or a baseline that trains the small-batch model for 32 times more iterations, is necessary to isolate the effect of gradient scaling.","section":"Section III.B, Figs. 7-8"},{"comment":"The modeling assumption G(bsmall) ≈ X·G(blarge) is unvalidated. No experiment compares scaled large-batch gradients to true small-batch gradients, and the gradient noise term γ in Eq. (4) is never measured. The step-function method is presented as a naive substitute for the teacher model A(·) of Eq. (5), but A(·) is never implemented or tested. Thus the claimed mechanism for improving large-batch generalization — that scaling by X reproduces small-batch updates outside critical phases — has no direct empirical support.","section":"Section III.B, Eq. (7)"},{"comment":"The reported successes depend on post hoc selection of the scaling factor X and the gradient-change threshold per model: X=16 with threshold 0.5 for ResNet50, and X=8 with threshold 0.8 for VGG11. The paper itself states that the ideal threshold varies with model and configuration, and it reports degraded accuracy for VGG11 at X=8 and X=16 under threshold 0.5. No seeds, error bars, or confidence intervals are reported. Consequently, the statement that \"naively scaling up the gradient updates improves model quality and diminishes generalization gap even at larger batches\" is not supported as a general claim.","section":"Section III.B, Figs. 7-8 and Summary"},{"comment":"The accuracy claims for the linear compute-time and memory models are based on in-sample evaluation. The linear model is fitted to compute times at batch sizes such as 32, 128, 512, and 1024, and the reported error percentages are computed at those same fitted points. This does not demonstrate predictive accuracy for unseen batch sizes. A held-out evaluation or a cross-validation procedure is required to substantiate the claim that the models can \"accurately predict\" batch-memory or compute time for a given configuration.","section":"Section III.A, Figs. 4-5"}],"minor_comments":[{"comment":"The caption says \"When B increases, generalization improves as models converge over flatter minima,\" which contradicts the figure and the surrounding text (where larger batches achieve lower test accuracy). This should read \"worsens\" or \"degrades.\"","section":"Figure 3 caption"},{"comment":"The notation for the step function is inconsistent: Eq. (6a) defines U(X), while Eqs. (6b) and (6c) use U(X, G). Please use a consistent function signature, e.g., U(G; X).","section":"Equations (6a)-(6c)"},{"comment":"There is a typo in \"hetereogeneous\" (should be \"heterogeneous\") in the paragraph on heterogeneity under federated learning.","section":"Section III.B"},{"comment":"Reference [25] is incomplete: it lists only a title, without authors, venue, or year. Please provide full bibliographic information.","section":"References"},{"comment":"The phrase \"explore new directions\" in the abstract should be \"explores new directions\" to agree with the subject \"this work.\"","section":"Abstract"}],"recommendation":"reject","confidential_remarks":"The manuscript reads as a preliminary vision paper, but it is submitted as a full research article. The central empirical claim is not supported as presented: the equal-iteration comparison confounds batch size with the total number of samples processed, the scaling factor and threshold are tuned per model using the test results, and the proposed teacher model is not tested. The useful parts — the timing/memory measurements and the gradient-mapping idea — could be the basis of a future study, but the current evidence does not meet the bar for publication in this venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe punchline: this paper's headline result is not supported by its own experiments. The 32.33% \"improvement\" over B=32 comes from comparing at equal iteration counts without equalizing samples seen. After T iterations, the B=1024 run has processed 32T samples and the B=32 run only T, so the gain could simply be more epochs, not the proposed gradient scaling. The stress-test concern holds.\n\nWhat's actually new: the paper proposes a teacher model that maps large-batch gradients to approximate small-batch gradients. That's an interesting idea in intent, but it is never implemented or evaluated. What is evaluated is a step function that multiplies gradients by X outside critical phases — essentially learning-rate scaling conditioned on the gradient-change metric from the author's earlier work. The paper does a decent job surveying the large-batch literature, and it openly reports that the trick hurts VGG11 for large X. The memory and compute-time linear fits are reasonable but are evaluated in-sample, with no held-out prediction.\n\nThe soft spots are substantial. There are no seeds or error bars. The values X=16 for ResNet50 and X=8 with threshold 0.8 for VGG11 are picked after seeing the results, so the reported wins are fitted, not predicted. The teacher model, the actual contribution, is absent. And the equal-iteration comparison conflates batch size with sample count, so the central claim about closing the generalization gap is not established. A fairer comparison is the one against baseline large-batch at the same batch size (X=1 vs X=16), which shows a 4.89% gain — but even that is post hoc.\n\nThe paper reads honestly: it calls its approach naive, says the ideal threshold varies, and discusses limitations. But it overclaims by stating that \"naively scaling up gradient updates improves model quality and diminishes generalization gap\" and by putting the small-batch accuracy gap in the abstract.\n\nI would not send this to a serious referee in its current form. It's a position paper with preliminary results that need to be re-run with equal-epoch comparisons, multiple seeds, and a held-out threshold selection. If the teacher model gets implemented and evaluated, then it could be worth a look. For now, it's a useful example of a confounding comparison, but not a paper whose accuracy claims should be trusted.\n\nMy take: do not send to peer review as is; if it comes back with the actual teacher model and proper controls, reconsider.","headline":"The headline accuracy claim is confounded by unequal sample counts, and the evaluated method is a tuned learning-rate scaling trick; the actual teacher model is never implemented.","tokens_in":12454,"tokens_out":3145,"would_cite":false,"duration_ms":28432,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In federated learning, multiplying large-batch gradients by a fixed $X$—outside critical phases—raises test accuracy over small-batch training by up to 32.33% at the same iteration count.","keywords":["federated learning","large-batch training","generalization gap","gradient scaling","critical training phases","gradient noise","gradient compression","batch-size estimation"],"falsifier":"Train ResNet50 on CIFAR10 with batch 1024, $X=16$, and threshold 0.5 for the same iteration count as batch 32; if the reported 32.33% test-accuracy advantage does not reproduce, the claim fails. A more direct check would record true batch-32 gradients and scaled batch-1024 gradients at identical weights and compute their cosine similarity or Euclidean distance.","tokens_in":11484,"feed_emoji":"📈","tokens_out":13588,"duration_ms":102710,"temperature":0.7,"pith_summary":"The paper tries to establish that the test-accuracy penalty normally associated with large-batch federated learning can be removed by a very simple operation: multiply the large-batch gradient by a fixed factor $X$ except during critical phases of training. Under the same iteration budget, this scaled large-batch training is reported to beat ordinary small-batch training by up to 32.33% on ResNet50 (batch 1024 at $X=16$ versus batch 32) and by 3.74% on VGG11 (batch 1024 at $X=8$ versus batch 64). The practical stakes are that federated systems could keep the parallel speed and communication savings of large global batches while recovering small-batch generalization. The paper also proposes, but does not implement, a trained teacher model that would map large-batch gradients to small-batch equivalents, with the step function serving as a naive substitute.","feed_headline":"Scaling gradients closes the large-batch gap in federated learning","feed_subtitle":"For equal iterations, scaled large batches beat small batches on ResNet50 by 32.33% and on VGG11 by 3.74%.","key_machinery":"The load-bearing object is the step-function surrogate $U(X)$ defined by $U(X)=1.0$ in critical training phases and $U(X)=X$ otherwise; the update becomes $\\tilde{G}=X\\odot G(b_{\\text{large}})$ outside critical phases and $\\tilde{G}=G(b_{\\text{large}})$ inside them. Critical phases are detected from the normalized gradient-change metric $\\triangle(G^{(b)}_{(i)})=\\left|\\,|G^{(b)}_{(i)}|^2-|G^{(b)}_{(i-1)}|^2\\,\\right|/|G^{(b)}_{(i-1)}|^2$, with thresholds 0.5 and 0.8 tested. This machinery implements the paper's assumed identity $G(b_{\\text{small}})\\approx X\\odot G(b_{\\text{large}})$, i.e., the added term $(X-1)G(b_{\\text{large}})$ plays the role of small-batch gradient noise. The paper pairs this statistical mechanism with two linear prediction models—one for activation and batch memory, one for compute time—so that the largest permissible and fastest batch size can be estimated for a given device.","core_discovery":"The central claim, stated in the paper's own terms, is that 'naively scaling up the gradient updates improves model quality and diminishes generalization gap even at larger batches.' Concretely, for the same number of training iterations, the step-function surrogate $U(X)$—which multiplies gradients by $X$ when the normalized gradient change is above threshold and leaves them at $1\\times$ in critical phases—yields up to 32.33% higher test accuracy than small-batch training for ResNet50 on CIFAR10 and 3.74% higher for VGG11 on CIFAR100. The paper presents this as preliminary evidence that a large-batch update scaled by $X$ approximates a small-batch update, and that switching back to unscaled updates during sensitive periods prevents divergence. It then conjectures that replacing the step function with a learned teacher model would improve the estimate further.","pith_inferences":["If $X\\odot G(b_{\\text{large}})\\approx G(b_{\\text{small}})$ is a real relationship, the same idea could apply to local-SGD and asynchronous federated variants, with each client's scaling factor chosen from its own batch size rather than one global $X$.","Because the best $X$ and threshold are found per model, a natural extension is an online calibration rule that estimates gradient noise during training and sets $X$ adaptively, eliminating grid search.","The success of the naive step function suggests the small-batch advantage may come mainly from injecting update noise of the right scale; if so, any cheap noise-injection scheme with similar magnitude might work at large batches."],"forward_implications":["Synchronous federated training can use large global batches without automatically accepting worse test accuracy, because outside critical phases the update is simply re-scaled rather than recomputed at a smaller batch.","A client can predict, before training, both the largest batch that fits in memory and the batch that minimizes per-epoch time, making large-batch FL practical on heterogeneous devices.","Large-batch gradients are already close to compressed sparse updates, so scaled large-batch FL can tolerate aggressive gradient compression and save communication without proportional loss in accuracy.","The choice of $X$ and of the gradient-change threshold changes the outcome; at threshold 0.5, VGG11 degrades at $X=8,16$, while raising the threshold to 0.8 markedly improves $X=8$ (though not above $X=1$), so critical-phase detection carries the method.","If the step function is a valid stand-in, a teacher model trained to output small-batch gradients from large-batch inputs should extend the benefit to settings where a single fixed $X$ is not optimal."],"supporting_citations":[{"why":"Supplies the empirical gradient-noise model that frames a small-batch gradient as a large-batch gradient plus noise.","marker":"[5]"},{"why":"Provides the empirical large-batch training and performance model behind the memory and compute-time predictions.","marker":"[15]"},{"why":"Documents the large-batch generalization gap that the paper's scaling method is meant to close.","marker":"[9]"},{"why":"Attributes that gap to convergence to sharp minima, the failure mode the step function is designed to avoid.","marker":"[11]"},{"why":"Connects the gap to a larger Hessian spectrum, supporting the sharp-minima reading.","marker":"[12]"},{"why":"Supplies the gradient-change metric and threshold used to identify critical training phases.","marker":"[2]"},{"why":"Identifies critical learning periods, motivating the switch to unscaled gradients in sensitive phases.","marker":"[32]"},{"why":"Shows the early phase of training is distinct, supporting phase-dependent gradient scaling.","marker":"[33]"}],"fun_headline_variants":["Gradient scaling closes the large-batch accuracy gap in FL","Step-scaled gradients mimic small-batch training in FL","Scaled large batches: 32% accuracy gain in federated learning","Large-batch FL accuracy restored with gradient upscaling","Federated learning: scaled gradients make large batches act small"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that multiplying a large-batch gradient by a fixed constant $X$ gives a valid stand-in for a small-batch gradient (Equation (7)), and that the gradient-change threshold reliably marks the phases where the model must not be scaled; the paper never directly compares scaled large-batch gradients with true small-batch gradients, and the best $X$ and threshold are chosen per model.","fun_headline_variants_meta":{"raw":{"variants":["Gradient scaling closes the large-batch accuracy gap in FL","Step-scaled gradients mimic small-batch training in FL","Scaled large batches: 32% accuracy gain in federated learning","Large-batch FL accuracy restored with gradient upscaling","Federated learning: scaled gradients make large batches act small"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001059,"raw_usage":{"total_tokens":4446,"prompt_tokens":953,"completion_tokens":3493,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":3406}},"tokens_in":569,"tokens_out":3493,"duration_ms":23948,"temperature":1.0,"reasoning_tokens":3406,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:23:43.900375+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ResNet50 on CIFAR10 with batch 1024, $X=16$, and threshold 0.5 for the same iteration count as batch 32; if the reported 32.33% test-accuracy advantage does not reproduce, the claim fails. A more direct check would record true batch-32 gradients and scaled batch-1024 gradients at identical weights and compute their cosine similarity or Euclidean distance.","supporting_citations":[{"cited_title":"Scavenger: A Cloud Service For Optimizing Cost and Performance of ML Training","cited_arxiv_id":null,"evidence_quote":"Provides the empirical large-batch training and performance model behind the memory and compute-time predictions."},{"cited_title":"Train longer, generalize better: closing the general- ization gap in large batch training of neural networks","cited_arxiv_id":null,"evidence_quote":"Documents the large-batch generalization gap that the paper's scaling method is meant to close."},{"cited_title":"On Large-Batch Training for Deep Learn- ing: Generalization Gap and Sharp Minima","cited_arxiv_id":null,"evidence_quote":"Attributes that gap to convergence to sharp minima, the failure mode the step function is designed to avoid."},{"cited_title":"Hessian-based Analysis of Large Batch Training and Robustness to Adversaries","cited_arxiv_id":null,"evidence_quote":"Connects the gap to a larger Hessian spectrum, supporting the sharp-minima reading."},{"cited_title":"Accelerating Distributed ML Training via Selective Synchronization","cited_arxiv_id":null,"evidence_quote":"Supplies the gradient-change metric and threshold used to identify critical training phases."}],"review_version":1}