{"id":"cda2358a-5c09-4bfb-a0fa-eb3706f1a027","arxiv_id":"2412.11777","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"FSG trains a fast MLP and a slow Mamba-based hypernetwork on current and historical gradients to approximate sign-function derivatives, reporting higher CIFAR accuracy than several baselines.","lead":"This paper introduces a training method for binary neural networks in which two learned networks generate gradient approximations, one from the current gradient and one from a stored history of gradients. The method reports higher accuracy on CIFAR-10 and CIFAR-100 than several existing binary-network optimizers, but the gains are often within one standard deviation and the code is not verifiably available.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Empirical support for the headline gains is undermined by duplicated baseline entries and absent significance testing; the CIFAR-100 advantage over IR-Net is within one standard deviation, so the central 'outperforms' claim is not yet established.","rationale":"The paper's central claim is empirical: FSG converges faster and reaches lower loss and higher accuracy than existing BNN optimization baselines. The appropriate standard of proof is therefore reliable, reproducible baseline comparisons. The reader's verdict correctly identified missing code, duplicated baseline numbers, and absent significance tests as reasons for a conditional verdict. My stress-test focuses on the most load-bearing of these: the duplicated FCGrad/LSTMFC entries across different backbones and the within-one-standard-deviation CIFAR-100 margin over IR-Net. If those baseline numbers are not accurate, the central claim is simply unsupported, regardless of how the method is motivated. A secondary concern is that Theorem 1 assumes the slow-net produces unbiased momentum (E_k(Delta_s^k)=0), which is exactly the behavior the learned network is supposed to acquire, so the proof does not cover the actual trained slow-net; however, the empirical baseline issue is more directly load-bearing for the paper's headline claim. The recommended verdict is unchanged: conditional acceptance pending code release, corrected tables, and significance testing. This is not an objection to the idea of using historical gradients or sequence models for BNN optimization; it is a concrete request that the evidence be made verifiable before the outperformance claim is accepted.","tokens_in":17835,"tokens_out":3218,"duration_ms":31849,"concrete_test":"Obtain the authors' training code and configuration files, then rerun Table 4 on ResNet-20/44 (CIFAR-10) and ResNet-56 (CIFAR-100) using FSG plus each baseline (DoReFa, ReSTE, LSTMFC, IR-Net, RBNN) under identical data splits, learning-rate schedules, and seeds. Record per-seed test accuracies and apply a paired Wilcoxon signed-rank test between FSG and IR-Net on CIFAR-100. If the corrected baseline values differ materially from the duplicated entries in Tables 1 and 2, or if the FSG-vs-IR-Net difference is not significant at p<0.05, the 'outperforming existing baselines' claim fails as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (Abstract, Table 4) is that FSG 'achieves faster convergence and lower loss values, outperforming existing baselines.' The load-bearing condition is that the baseline numbers in Tables 1, 2, and 4 are accurate, independently measured values recorded under a common protocol. That condition is not secure. In Table 1, FCGrad and LSTMFC test accuracies are exactly identical across ResNet-32 and ResNet-44 under SGD (89.93 and 90.40), which is implausible as independent measurements for different backbones. In Table 2, FCGrad and LSTMFC train accuracy are identical on ResNet-56 SGD (73.934). Table 5 labels a row 'ReSTE' with slow-net Mamba, which does not match ReSTE's published method and suggests labeling or table-construction errors. The headline CIFAR-100 advantage over IR-Net is 69.48 vs 68.94, a margin of 0.54 percentage points, while FSG's reported standard deviation is 0.979 and IR-Net's is 0.967, so the difference is well within one standard deviation; no paired significance test is reported. Additionally, hyperparameters beta (0.3) and l (6) are selected on CIFAR-100 and then reused for the CIFAR-100 SOTA table, so the reported margin is inflated by selection on the same benchmark used for the final comparison. If the duplicated baseline numbers are copy errors, or if the baseline values do not reproduce under a shared training setup, the central claim has no reliable empirical support. This is a correctness risk in the reported evidence, not a dispute with community consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FSG (Fast and Slow Gradient Generation), a hypernetwork-based method for training binary neural networks. It introduces a Historical Gradient Storage (HGS) module that keeps the last l flattened per-layer gradients, a slow-net (Mamba or LSTM) that consumes this history to produce a momentum-like gradient, a fast-net (MLP) that produces a current-gradient term, and Layer Recognition Embeddings (LRE) to make the shared slow-net layer-aware. The update rule is W^{t+1}_i = W^t_i - α M_f(g^t_{W_i}, \\hat W^t_i) + β M_s(h^t_i). The authors report experiments on CIFAR-10 and CIFAR-100 with ResNet backbones, claiming faster convergence and lower loss than STE, FCGrad, LSTMFC, IR-Net, RBNN, and ReSTE, and include a convergence theorem with an appendix proof.","tokens_in":18216,"tokens_out":3685,"duration_ms":36052,"significance":"If the empirical claims are reliable, FSG would be a practical learned-gradient alternative to hand-designed straight-through estimators for BNNs, and the use of a sequence model over historical gradients is a reasonable idea worth exploring. The paper also makes a useful conceptual distinction between fast current-gradient generation and slow momentum generation, and it includes an ablation on the choice of slow-net and on the hyperparameters β and l. However, the significance is currently bounded by weaknesses in the evidence: duplicated baseline entries, a mislabeled overhead table, absent significance testing, and hyperparameter selection on the same benchmark used for the final comparison. The theoretical result is a conditional consistency statement rather than a convergence guarantee for the actual BNN training problem. These issues must be resolved before the central 'outperforms existing baselines' claim can be accepted.","major_comments":[{"comment":"The baseline entries contain implausible duplications. In Table 1, FCGrad and LSTMFC report exactly the same test accuracies on ResNet-32 and ResNet-44 under SGD (89.93 and 90.40, respectively), and in Table 2, FCGrad and LSTMFC report identical train accuracy (73.934) on ResNet-56 under SGD. Independent runs on different backbones would not produce identical values to three decimal places. This strongly suggests that rows were copied or mislabeled, and it undermines the reliability of the baseline numbers against which FSG is compared. The authors should provide the original run logs or correct the table entries and confirm that all baselines were measured under the same protocol.","section":"Table 1 and Table 2"},{"comment":"The headline CIFAR-100 comparison is not statistically supported. FSG reports 69.48 ± 0.979 while IR-Net reports 68.94 ± 0.967, a difference of 0.54 percentage points that is well within one standard deviation. The paper does not report paired significance tests, and the Reproducibility Checklist explicitly answers 'no' to the question about appropriate statistical tests. Without significance testing or a larger number of runs, the claim that FSG 'outperforms existing baselines' on CIFAR-100 is not established.","section":"Table 4"},{"comment":"The hyperparameters β = 0.3 and l = 6 are selected on CIFAR-100 (Fig. 2c and 2d) and then reused for the CIFAR-100 comparison in Table 4. This means the reported CIFAR-100 margin over IR-Net is inflated by selection on the same benchmark used for the final evaluation. The authors should either report the selection procedure explicitly, evaluate on a held-out configuration, or correct for multiple testing. At minimum, they should state that the SOTA table configuration was chosen on the evaluation dataset.","section":"Ablation Experiment (Influence of β and l)"},{"comment":"The convergence theorem assumes E_k[ M_s(ℏ_k) - (x_k - x_{k-1}) ] = 0, i.e., that the slow-net momentum equals the true momentum in expectation. This is precisely the desired behavior of the learned slow-net, so the theorem is a consistency result conditional on the slow-net being correct, not a guarantee that FSG converges for the BNN training objective. In addition, the assumptions of convex f_i, bounded iterates, and a bounded positive square-matrix fast-net do not match the nonconvex, binarized setting of the experiments. The theorem should be reframed as a conditional consistency property and the mismatch with the experimental setting should be acknowledged.","section":"Theorem 1 and Appendix proof"},{"comment":"The computational overhead table labels a row 'ReSTE' with Fast-net 'Multi MLP', Slow-net 'Mamba', and test accuracy 69.48. This does not match ReSTE, which is a rectified straight-through estimator rather than a hypernetwork-based method, and the reported 69.48 is the accuracy attributed to FSG elsewhere in the paper. The row appears mislabeled and the overhead comparison is therefore invalid. The authors should correct the row labels and clarify which method each runtime corresponds to.","section":"Table 5"}],"minor_comments":[{"comment":"The Abstract states the code is available at github.com/FSG, while the text at the end of the Abstract and the introduction give http://github.com/two-tiger/FSG. This is inconsistent and the shorter URL appears to be a placeholder.","section":"Abstract and code URL"},{"comment":"The historical gradient index is inconsistent: Eq. (8) uses h^t_i = Concat[g^{t-l+1}_{w_i}, ..., g^t_{w_i}], while Eq. (10) writes the input as [t_i; g^{t-l}_{w_i} W_a, ..., g^t_{w_i} W_a]. The two equations describe different historical windows; please align them.","section":"Equations (8) and (10)"},{"comment":"Eq. (12) writes the forward update as A(W^t_i - α M_f(...) ∂A/∂W + β M_s(ℏ^t_i)), while Algorithm 1 line 4 writes Q{A[ W^t_i - α M_f(...) ∂A/∂W + β M_s(ℏ^t)] }. The placement of the parentheses differs and it is unclear whether the quantization is applied before or after the A(·) preprocessing. Please make the formulas consistent.","section":"Equation (12) and Algorithm 1"},{"comment":"The proof uses ϕ_f and ϕ_s without clearly defining them; in some lines ϕ_f appears to denote the fast-net output and in others a scalar factor or linear map. This makes the proof difficult to follow. Please introduce notation explicitly and ensure that every symbol is defined before use.","section":"Theorem 1 proof"},{"comment":"Table 3 reports ResNet-56 CIFAR-100 accuracy of 68.04 ± 0.948 for the Mamba slow-net, while Table 2 and Table 4 report 69.48 ± 0.979 for FSG on the same backbone and dataset. The paper does not explain the discrepancy; please clarify the configuration difference (e.g., β, l, or optimizer settings) or correct the table.","section":"Table 3 vs Table 4"}],"recommendation":"major_revision","confidential_remarks":"I share the reader's concern about the duplicated baseline entries and the mislabeled Table 5 row. These appear to be data-construction or copy-paste errors rather than deliberate misreporting, but they must be corrected and the raw logs supplied. If the baseline numbers cannot be reproduced under a common protocol, the central empirical claim is not supported. Please ask the authors to provide the original logs and to address the statistical-testing and benchmark-selection issues before the paper is considered further."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new idea here is a hypernetwork that generates gradients for binary networks by combining a fast MLP on the current gradient with a slow Mamba/LSTM on a stored history of gradients, plus per-layer embeddings. That combination is new, and the motivation—momentum in SGD uses history, so a learned gradient generator should too—is sensible. The paper does a reasonable job of situating itself relative to MetaQuant and LSTMFC, includes ablations on the slow-net choice and on the history length, and is upfront in the Limitation section that it only tests ResNet-style backbones on CIFAR.\n\nWhat I trust: the method is clearly described, the algorithm is reproducible in principle, and the small ablations (Mamba vs LSTM, β, l) are useful empirical data. The convergence theorem is a standard convex bound with momentum; it's honest enough to assume the slow-net's momentum error is unbiased in expectation, so it functions as a consistency result rather than a free guarantee. That is not fatal—it's worth saying explicitly.\n\nThe soft spots are in the empirical reporting. Table 1 shows FCGrad and LSTMFC with identical test accuracies across ResNet-32 and ResNet-44 under SGD (89.93 and 90.40), and Table 2 has the same train accuracy (73.934) for both on ResNet-56. Those are implausible as independent runs and look like copy-paste errors; they dent confidence in the rest of the table. Table 5 labels a row 'ReSTE' that uses a Mamba slow-net, which doesn't match ReSTE's published method. The headline CIFAR-100 advantage over IR-Net is 0.54 points with standard deviations near 1, so it's well within noise; no significance test is reported. And β and l are chosen on CIFAR-100 and then reused for the final CIFAR-100 comparison, which inflates the apparent gain.\n\nNone of this makes the core idea wrong. But the current evidence doesn't establish the 'outperforms existing baselines' claim as stated. The paper is for people working on learned-gradient BNN optimization; it deserves a serious referee and a major revision that fixes the tables, adds significance tests, and tempers the theorem's interpretation. I'd engage with it on those terms.","headline":"A plausible incremental idea—hypernetwork gradient generation with memory—that currently overclaims its empirical support due to table errors and weak statistical evidence.","tokens_in":18777,"tokens_out":4063,"would_cite":false,"duration_ms":36631,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that combining a fast current-gradient MLP with a Mamba-generated momentum term read from recent per-layer gradients makes binary neural networks converge faster and reach higher accuracy on CIFAR-10 and CIFAR-100.","keywords":["Binary Neural Networks","network quantization","gradient approximation","hypernetwork","straight-through estimator","momentum","Mamba","CIFAR benchmarks"],"falsifier":"Take the same binary ResNet-56 on CIFAR-100 and replace the slow-net output with the classical momentum term computed from true weight gradients, keeping the fast-net and all other settings; if this classical-momentum baseline reaches or exceeds 69.48% test accuracy, the learned slow-net is not what supplies the gain. A second check is to record the cosine similarity between the slow-net output and the classical momentum direction during training; if the similarity is near zero while accuracy still rises, the paper's interpretation of slow-net as a momentum generator would not be supported.","tokens_in":17551,"feed_emoji":"🧠","tokens_out":9929,"duration_ms":83198,"temperature":0.7,"pith_summary":"This paper argues that straight-through estimators and earlier hypernetwork gradient generators leave information on the table because they use only the current gradient, while classical momentum shows that past gradients carry useful direction. To close that gap, it introduces a Historical Gradient Storage module that keeps the last several flattened weight gradients of each layer as a time series; a shared Mamba block, called the slow-net, reads that sequence plus a per-layer embedding and emits a momentum-like gradient, while a small MLP, called the fast-net, processes the current gradient. The two terms are combined and inserted into the forward pass so both hypernetworks are trained by the task loss. On binary ResNets, the method reports faster convergence, lower loss, and higher test accuracy than the compared baselines, including 92.78% on ResNet-44/CIFAR-10 and 69.48% on ResNet-56/CIFAR-100. If these results hold, FSG offers a practical learned-gradient alternative to hand-designed straight-through estimators for binary network training.","feed_headline":"Feeding past gradients to a Mamba net lifts binary ResNet accuracy","feed_subtitle":"FSG reaches 92.78% on CIFAR-10 ResNet-44 and 69.48% on CIFAR-100 ResNet-56, beating prior gradient estimators.","key_machinery":"The central object is the two-branch gradient generator: a Historical Gradient Storage module that stores the most recent $l$ flattened per-layer gradients as a sequence, a shared Mamba-based slow-net with Layer Recognition Embeddings that maps that sequence to a momentum gradient, and a fast MLP that maps the current gradient and previous full-precision weights to a current gradient. The load-bearing identity is the update $W^{t+1}_i = W^t_i - \\alpha M_f(g^t_W, \\hat W^t_i) + \\beta M_s(h^t_i)$, whose generated gradient is registered in the optimizer and also inserted into the forward pass so that both hypernetworks receive gradient signal from the task loss. The Mamba block does the work of filtering noise in the historical gradient sequence and producing the momentum term, while the Layer Recognition Embedding gives the shared slow-net a per-layer tag so that gradients from different layers do not interfere.","core_discovery":"The central claim is that the non-differentiable gradient of the binarization function can be approximated better by a learned combination of a current-gradient term and a historical-gradient momentum term than by current-gradient-only hypernetworks or hand-designed straight-through estimators. Concretely, the update for layer $i$ is $W^{t+1}_i = W^t_i - \\alpha M_f(g^t_W, \\hat W^t_i) + \\beta M_s(h^t_i)$, where $M_f$ is a fast MLP and $M_s$ is a shared Mamba block consuming the last $l$ flattened gradients, and this generated gradient is registered in the optimizer and inserted into the forward pass so both hypernetworks are optimized against the task loss. The claimed consequence is that the generated gradient carries momentum information and layer-specific context, reducing accumulated gradient mismatch and producing faster convergence and lower loss. The authors support this with comparisons to STE, FCGrad, LSTMFC, and several SOTA binary-network methods across ResNet-20/32/44 on CIFAR-10 and ResNet-56/110 on CIFAR-100, and they supplement the empirical results with a convex convergence analysis giving an $O(1/\\sqrt{t+1})$ expected-error bound.","pith_inferences":["The authors explicitly scope the experiments to ResNet-style convolutional networks; a direct next test is whether the same HGS and slow-net construction transfers to Transformer or LLM quantization, where gradient mismatch also appears.","The reported optimum of $\\beta=0.3$ implies the fast current-gradient branch should dominate; a sharper test would sweep $\\beta$ jointly with history length $l$ and optimizer choice, since the paper fixes one hyperparameter at a time on a single dataset.","Because the slow-net is shared across layers and only distinguished by a learned embedding, one could test whether the learned embeddings encode meaningful layer structure, for example by clustering the embeddings and comparing the clusters to layer depth."],"forward_implications":["Binary ResNet training can reach test accuracy within about one percentage point of full-precision training on small datasets: 92.78% versus 93.55% on ResNet-44/CIFAR-10, and 91.42% versus 92.13% on ResNet-32/CIFAR-10.","The reported loss curves imply the learned momentum term helps most early in training, so FSG reaches a given loss in fewer epochs than STE, FCGrad, and LSTMFC.","The extra training cost is confined to training: adding the slow-net raises per-epoch time by about 3.2 times in the overhead measurement, while at inference the binary network runs without either hypernetwork.","Under the paper's convexity assumptions, the update rule is proved to converge at $O(1/\\sqrt{t+1})$, matching the standard stochastic-subgradient rate, so the practical gains are claimed to come from better constants and better descent directions, not from an improved worst-case rate."],"supporting_citations":[{"why":"Introduces the straight-through estimator, the baseline the paper argues is handicapped by gradient mismatch.","marker":"Courbariaux, Bengio, and David 2015"},{"why":"MetaQuant is the hypernetwork gradient-generation method FSG extends; its experimental setting and FCGrad/LSTMFC baselines are reused.","marker":"Chen, Wang, and Pan 2019"},{"why":"Supplies the Mamba block that serves as the slow-net momentum generator in the proposed method.","marker":"Gu and Dao 2023"},{"why":"DoReFa is the preprocessing and quantization scheme used to binarize weights in all experiments.","marker":"Zhou et al. 2016"},{"why":"Provides the LSTM alternative slow-net used in the sequence-model ablation.","marker":"Hochreiter and Schmidhuber 1997"},{"why":"IR-Net is the strongest SOTA baseline on CIFAR-100 and contributes the Error Decay Estimator and experimental setup.","marker":"Qin et al. 2020"},{"why":"RBNN is a SOTA straight-through-style baseline that FSG is compared against in Table 4.","marker":"Lin et al. 2020"},{"why":"ReSTE is a recent rectified straight-through estimator used as a SOTA comparison baseline.","marker":"Wu et al. 2023"}],"fun_headline_variants":["Past gradients sharpen binary neural network training","Mamba net improves gradient estimation for binary nets","Historical gradients speed up binary ResNet convergence","Layer-aware gradient generation boosts binary networks","FSG: fast and slow gradients for binary neural nets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gains depend on a single shared sequence model being able to turn the recent history of each layer's flattened gradients into a useful momentum, and on injecting that learned momentum into the forward pass without steering the optimizer into a bad direction.","fun_headline_variants_meta":{"raw":{"variants":["Past gradients sharpen binary neural network training","Mamba net improves gradient estimation for binary nets","Historical gradients speed up binary ResNet convergence","Layer-aware gradient generation boosts binary networks","FSG: fast and slow gradients for binary neural nets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1507,"prompt_tokens":1070,"completion_tokens":437,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":686,"completion_tokens_details":{"reasoning_tokens":369}},"tokens_in":686,"tokens_out":437,"duration_ms":4239,"temperature":1.0,"reasoning_tokens":369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:36:27.517120+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same binary ResNet-56 on CIFAR-100 and replace the slow-net output with the classical momentum term computed from true weight gradients, keeping the fast-net and all other settings; if this classical-momentum baseline reaches or exceeds 69.48% test accuracy, the learned slow-net is not what supplies the gain. A second check is to record the cosine similarity between the slow-net output and the classical momentum direction during training; if the similarity is near zero while accuracy still rises, the paper's interpretation of slow-net as a momentum generator would not be supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MetaQuant is the hypernetwork gradient-generation method FSG extends; its experimental setting and FCGrad/LSTMFC baselines are reused."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"IR-Net is the strongest SOTA baseline on CIFAR-100 and contributes the Error Decay Estimator and experimental setup."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RBNN is a SOTA straight-through-style baseline that FSG is compared against in Table 4."}],"review_version":1}