Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Perturbation-efficient Zeroth-order Optimization for Hardware-friendly On-device Training

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PeZO claims that zeroth-order optimization becomes feasible for on-device training once Gaussian per-weight perturbations are replaced by a small reused pool of uniform random numbers rescaled to the expected Gaussian modulus.

desk verdict A real hardware bottleneck, sensible reuse ideas, and actual FPGA synthesis, but the core scaling LUT contradicts the paper's own equation and the performance claims are softer than the abstract suggests. read the letter →

arxiv 2504.20314 v2 pith:Z7IYTXNE submitted 2025-04-28 cs.LG cs.AI

Qitao Tan , Sung-En Chang , Rui Xia , Huidong Ji , Chence Yang , Ci Zhang , Jun Liu , Zheng Zhan
show 5 more authors
Zhenman Fang Zhou Zou Yanzhi Wang Jin Lu Geng Yuan
This is my paper · ORCID
classification cs.LGcs.AI
keywords zeroth-orderoptimizationon-devicetrainingFPGAimplementationrandomnumbergenerationGaussianperturbationuniformlanguagemodelfine-tuninghardware-efficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Zeroth-order (ZO) optimization estimates gradients from function values, which promises memory savings for training on devices, but it carries a hidden hardware cost: every weight needs a fresh Gaussian random number at every step. This paper argues that this cost, rather than the forward passes, is what makes ZO training infeasible on FPGAs and ASICs, and proposes PeZO to remove it. PeZO replaces costly per-weight Gaussian perturbations with a small pool of reused uniform random numbers, adaptively rescaled to the modulus a Gaussian perturbation would have. On few-shot NLP fine-tuning benchmarks, the authors report accuracy competitive with MeZO, the representative ZO baseline, while cutting random-number-generation logic (LUTs and flip-flops) by 48.6% and 12.7%, and power by up to 86%. If that empirical equivalence holds, ZO becomes a realistic memory-efficient training path for edge hardware.

What carries the argument

PeZO's load-bearing mechanism is random-number reuse plus adaptive modulus scaling. Reuse works by concatenating numbers from a pre-generated pool (size $2^{12}$) or from a small array of uniform RNGs, with circular-shift operators that vary the ordering so perturbations do not fall into regular patterns; this cuts the unique random numbers needed per clock cycle from roughly the parameter count to $2^{12}$ or $2^5$. Adaptive modulus scaling replaces Gaussian draws with uniform draws by rescaling each perturbation $u_i$ to the expected Gaussian modulus $\mathbb{E}\|\hat{u}_i\|_2$, with the scaling factor stored in a look-up table and rounded to a power of two so the scaling itself is just bitwise shifts. Together these remove Gaussian random-number generators from the hardware design, while the low intrinsic dimensionality of language-model fine-tuning provides the intuition for why heavily reused perturbations can still find good updates.

What would settle it

Run PeZO on a non-NLP task, such as few-shot image classification on CIFAR-10 with a standard convolutional model, and compare final accuracy against the same optimizer using fresh Gaussian perturbations; a drop materially larger than the Gaussian baseline's run-to-run variation would show the equivalence does not generalize. A cheaper probe is to compute the mean-squared error of PeZO's rescaled-uniform gradient estimate versus the Gaussian estimate on one fixed model and batch, since large bias would explain any task class that fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the two obstacles to hardware ZO — the large quantity of random numbers and the Gaussian distribution — can be dissolved without hurting accuracy. With pre-generation reuse, as few as $2^{12}$ pre-stored random numbers suffice; with on-the-fly reuse, 32 uniform random-number generators suffice. The adaptive modulus scaling step rescales each uniform perturbation to the expected Euclidean norm of a same-dimensional Gaussian perturbation, with the factor computed in advance and looked up from a small table so hardware only performs bit-shifts. Across RoBERTa-base/large, OPT-1.3B/2.7B, and Llama3-1B/3B on standard few-shot classification tasks, the accuracy stays competitive with MeZO's ideal Gaussian perturbation, and in the $k=256$ regime the gap narrows to within 0.5%. Hardware synthesis on a ZCU102 FPGA shows the random-number-generation block dropping from 133,120 LUTs and 4.474 W to 32 LUTs and 0.608–2.104 W, which the paper presents as making ZO optimization feasible for on-device training.

Load-bearing premise

The load-bearing premise is that a small set of reused uniform random numbers, rescaled to the expected Gaussian modulus, produces gradient estimates statistically close enough to fresh Gaussian perturbations to preserve accuracy across diverse models and tasks; the paper supports this only empirically on a few few-shot NLP benchmarks and provides no theoretical analysis.

Editorial extensions

If this is right

  • ZO fine-tuning can be implemented on FPGAs and ASICs with on-chip BRAM and a few dozen uniform generators, making memory-efficient on-device training concrete.
  • The two reuse strategies give a designer a hardware-accuracy trade-off: on-the-fly reuse is better on small-data tasks, pre-generation reuse is better on larger-data, harder settings.
  • RNG bit-width can stay low — 8 bits for RoBERTa-class models and 14 bits for OPT-class models — with RNG shifting extending the effective randomness cycle.
  • PeZO plugs into any ZO method that uses the standard two-point perturbation estimator, so future ZO improvements can inherit the hardware savings.
  • With random-number generation cut to 32 LUTs and 449–512 FFs at 0.61–2.10 W, the perturbation block is no longer the bottleneck; inference compute becomes the limiting cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An unstated test is outside NLP: the paper's own limitation section concedes that few-shot computer-vision ZO lacks the prompt-style augmentation that NLP provides, so if reused uniform perturbations fail on vision, the hardware gains apply only to text-domain fine-tuning.
  • The results suggest a cost-geometry principle worth testing: what matters for ZO gradient estimation may be the modulus and symmetry of the perturbation distribution rather than its exact Gaussian shape; a rigorous bias-variance comparison of the rescaled-uniform estimator against the Gaussian estimator would settle this.
  • Reuse could compound with parameter-efficient fine-tuning: updating fewer weights needs fewer perturbations, which would cut the random-number demand even further; the paper notes LoRA-style methods but does not quantify this combination.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes PeZO, a perturbation-efficient zeroth-order optimization framework for on-device training. It argues that ZO training, although memory-efficient, is impractical on FPGA/ASIC because generating many Gaussian random numbers is resource-intensive. PeZO replaces Gaussian perturbations with uniformly distributed perturbations rescaled by an adaptive modulus-scaling factor, and introduces two random-number reuse strategies: pre-generation (a fixed pool of random numbers) and on-the-fly generation (a small array of uniform RNGs). The authors report accuracy results fine-tuning RoBERTa, OPT, and Llama models on few-shot NLP benchmarks, comparing against MeZO, and report FPGA resource and power numbers showing large reductions in LUTs, FFs, and power for the random-number-generation subsystem. The conclusion is that PeZO enables feasible ZO on-device training without compromising accuracy.

Significance. If the hardware implementation actually matches the evaluated algorithm, the paper identifies a real and underappreciated bottleneck in ZO training and offers a practical route around it. The reuse strategies and the empirical characterization of pool size, RNG count, and bit-width are useful design guidance. The paper is primarily empirical rather than theoretical, and no code or formal analysis is provided, but the breadth of models, datasets, and the concrete FPGA implementation are strengths. The central value depends on validating that the hardware scaling mechanism genuinely realizes Eq. (3) and on tempering the accuracy claims to match the reported tables.

major comments (3)
  1. [§3.2, Eq. (3), Figure 2] The on-the-fly scaling implementation as described cannot implement Eq. (3). Eq. (3) scales the full perturbation vector u_i by a factor s_i = E||u_hat_i||_2 / ||u_i||_2, whose value depends on the L2 norm of the entire n-dimensional vector produced in one clock cycle. With n RNGs each outputting b bits, the number of distinct cycle-level moduli is (2^b)^n, not 2^b, and the modulus is not determined by any single RNG output. The text says there are "2^b possible combinations" and uses the current output of the RNG pointed to by the pointer as the LUT address. A 2^b-entry LUT can encode scale factors for one scalar output, but not for the n-tuple that appears in Eq. (3), unless n=1 or all RNGs emit identical values. The resource counts in Table 6 (32 LUTs, 449 or 512 FFs, 1 BRAM) may therefore be for a design that does not evaluate the algorithm whose accuracy appears in Tables 4–5. Please provide the actual hardware scaling structure (e.g., a tree of sum-of-squares accumulators over all RNG outputs, or an explicitly justified approximation), re-validate accuracy with that exact design, and update resource/power numbers accordingly.
  2. [§4.2, Table 4] The statement that with k=256 "the accuracy gap between PeZO and the baseline still narrows to within 0.5% across all tasks" is not supported by the data. For RoBERTa-base on-the-fly generation, RTE is 60.3 vs. MeZO's 65.7 and TREC is 78.4 vs. 81.4; for RoBERTa-large on-the-fly, RTE is 66.4 vs. 72.2 and TREC is 84.2 vs. 90.8. These gaps are 3.0–6.6 points. The abstract's claim "without compromising training performance" is therefore too strong for several on-the-fly configurations. Please qualify the central claim by specifying which reuse strategy and which k settings are meant, and revise the abstract and conclusion accordingly. The conclusion that PeZO is competitive in many settings may still be defensible, but the current blanket wording is contradicted by the paper's own tables.
  3. [Table 4, RoBERTa-base k=16] The BP-based and MeZO rows for RoBERTa-base with k=16 are identical in all five columns (88.1, 40.3, 55.7, 56.6, 59.6). This appears to be a copy/paste or reporting error, since the same table and Table 5 show nontrivial differences between BP-based and MeZO elsewhere. Because MeZO is the central baseline, these numbers should be corrected and the affected comparisons re-checked.
minor comments (4)
  1. [Table 4 caption and Table 5] The caption says all reported numbers are averaged accuracy with standard deviation, but Table 5 does not include standard deviations for OPT/Llama models. Please either add standard deviations or state that they are omitted for those models.
  2. [§3.1 and Figure 1] The phrase "yield(ing) 2^b unique random numbers per cycle" is imprecise: each RNG produces one b-bit value, so the RNG array produces n b-bit values per cycle; the number of possible output combinations is (2^b)^n, not 2^b. Please clarify the wording to avoid confusion with the LUT-size argument in §3.2.
  3. [Table 6] The pre-generation row lists "Ours Pre-gen - 16 8 2.104 700" with no LUT value, while the on-the-fly rows report 32 LUTs but different FF counts (449 for RoBERTa, 512 for OPT). Please clarify what the dash means and why the FF counts differ between the two on-the-fly rows.
  4. [§2.3] The latency comparison for LLaMA2-7B would be more informative if the number of random numbers per attention layer update (4×4096×4096) were explicitly derived or a pointer to the layer configuration given, since the reader cannot verify the factor of four from the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: PeZO's accuracy and hardware claims are empirically measured against external baselines, and the self-citations are not load-bearing.

full rationale

PeZO's derivation chain is not circular. The central claims are (i) that reusing a small pool of uniform random numbers with an adaptive modulus-scaling rule can fine-tune language models at accuracy close to MeZO, and (ii) that the corresponding random-number-generation hardware is much cheaper. Claim (i) is established by direct, held-out comparison against MeZO, an external baseline, in Tables 4 and 5. Equation (3) defines the scaling rule (match the uniform perturbation's modulus to the expected Gaussian modulus); it does not assert, and is not used to derive, the accuracy result. The accuracy outcomes are measured empirically. The choices of 2^12 pre-generated numbers, 2^5 RNGs, and 8/14-bit widths come from the controlled sweeps in Figures 3 and 4; these are ordinary hyperparameter selections, not fitted quantities renamed as predictions. The two self-citations are not load-bearing: reference [32] appears only in the limitation/future-work discussion as a compatible recent method, and reference [40] supports the general hardware observation that many parallel URNGs have non-negligible cost; neither justifies PeZO's accuracy or resource savings. The paper's Section 5.2 limitation explicitly acknowledges that only NLP tasks were evaluated; that is a scope restriction, not a circular step. No quantity in the paper is defined in terms of the target result, no load-bearing premise is justified solely by the authors' prior work, and no 'prediction' reduces by construction to a fitted input. A skeptical reviewer's concern about the 2^b-entry LUT not being able to represent the full-dimension modulus of Eq. (3) is a hardware-fidelity/correctness issue, not evidence of circularity in the derivation chain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's central claim depends on the empirical equivalence of scaled uniform perturbations to Gaussian ones, on the low intrinsic dimensionality of the models, and on a specific hardware tiling assumption. The main free parameters are the pool size, RNG count, and bit widths, all selected from the authors' own experiments. No new physical entities are introduced.

free parameters (4)
  • pre_generation_pool_size = 2^12 (4096)
    Chosen from experiments in Figure 3 as the pool size beyond which accuracy saturates; used as the default for the pre-generation strategy.
  • on_the_fly_rng_count = 2^5 (32)
    Chosen from experiments in Figure 3 as the number of RNGs giving peak accuracy for on-the-fly generation.
  • rng_bit_width_roberta = 8 bits
    Chosen from experiments in Figure 4 as the bit width beyond which no further loss reduction is observed for RoBERTa models.
  • rng_bit_width_opt = 14 bits
    Chosen from experiments in Figure 4 as the bit width needed to fully exploit OPT model performance.
assumptions (4)
  • domain assumption Language models have low intrinsic dimensionality, so low-dimensional perturbations suffice for fine-tuning.
    Used in Section 3.1 to justify reusing a small pool of random numbers; cited from [1] but not derived or tested for the specific models.
  • ad hoc to paper A uniform random vector scaled to the expected Gaussian modulus behaves like a Gaussian perturbation for zeroth-order gradient estimation.
    The paper provides no proof; the claim rests on empirical results in Tables 4 and 5 for a few NLP tasks.
  • domain assumption The hardware baseline with 1024 Gaussian random number generators is representative of MeZO's random number needs.
    Section 4.2 assumes a tiling size of 1024; the resource comparison in Table 6 depends on this assumption.
  • standard math The expected modulus of a d-dimensional standard Gaussian is given by Eq. 4 using the Gamma function.
    Standard statistical formula; the reformulation in Eq. 5 is for numerical stability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Perturbation-efficient Zeroth-order Optimization for Hardware-friendly On-device Training." pith.science (2026). https://pith.science/paper/Z7IYTXNE

@misc{pith2026250420314,
  author       = {Pith},
  title        = {Pith review of: Perturbation-efficient Zeroth-order Optimization for Hardware-friendly On-device Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z7IYTXNE}},
  note         = {Machine review of arXiv:2504.20314}
}
read the original abstract

Zeroth-order (ZO) optimization is an emerging deep neural network (DNN) training paradigm that offers computational simplicity and memory savings. However, this seemingly promising approach faces a significant and long-ignored challenge. ZO requires generating a substantial number of Gaussian random numbers, which poses significant difficulties and even makes it infeasible for hardware platforms, such as FPGAs and ASICs. In this paper, we identify this critical issue, which arises from the mismatch between algorithm and hardware designers. To address this issue, we proposed PeZO, a perturbation-efficient ZO framework. Specifically, we design random number reuse strategies to significantly reduce the demand for random number generation and introduce a hardware-friendly adaptive scaling method to replace the costly Gaussian distribution with a uniform distribution. Our experiments show that PeZO reduces the required LUTs and FFs for random number generation by 48.6\% and 12.7\%, and saves at maximum 86\% power consumption, all without compromising training performance, making ZO optimization feasible for on-device training. To the best of our knowledge, we are the first to explore the potential of on-device ZO optimization, providing valuable insights for future research.

Figures

Figures reproduced from arXiv: 2504.20314 by the authors.

Figure 1
Figure 1. Illustration of two random number reuse strategies for pre-generation and on-the-fly generation settings. Therefore, random numbers or RNGs are set to powers of two minus one (i.e., 2 𝑛 − 1), but denote them as powers of two for simplicity in the following sections. To further enhance the irregularity and diversity of generated perturbations, we implement different shift operators on the ran￾dom number pool and RNGs… view at source ↗
Figure 2
Figure 2. Illustration of using the random number generated by the RNG pointed to by the pointer to efficiently query the scaling factor from the look-up table. For E∥𝑢ˆ𝑖 ∥2, it is a fixed statistical value with respect to the num￾ber of RNGs used. We compute this value before training and then store it in the memory, avoiding repeated computation and saving resources. For ∥𝑢𝑖 ∥2, as each RNG generates one random number per c… view at source ↗
Figure 3
Figure 3. Experiments of pre-generation strategy and on-the-fly generation strategy with different numbers of pre-generated random numbers/RNGs. OPT-series models collapse on SST2 with 2 2 RNGs, thus there is no data point [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Experiment on bit-width effectiveness. We train 1K and 10K steps for RoBERTa and OPT model, respectively. reaches 8 bits, further increase has no noticeable performance im￾provement. This suggests that an 8-bit RNG is sufficient for the RoBERTa series models. For large…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memory

    cs.CL 2025-08 reject novelty 5.0 of 10

    RCR-Router selects role-relevant memory slices under token budgets for multi-agent LLM systems, reporting token savings and quality gains on HotPotQA, MuSiQue, and 2WikiMultihop, but several experimental claims are un...

Reference graph

Works this paper leans on

48 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [14]

    Jiaqi Gu, Chenghao Feng, Zheng Zhao, Zhoufeng Ying, Ray T Chen, and David Z Pan. 2021. Efficient on-chip learning for optical neural networks through power- aware sparse zeroth-order optimization. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 7583–7591

    work page 2021

  2. [1]

    Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. 2020. Intrinsic dimen- sionality explains the effectiveness of language model fine-tuning.arXiv preprint arXiv:2012.13255 (2020)

    arXiv 2020

  3. [2]

    Seyedarmin Azizi, Souvik Kundu, and Massoud Pedram. 2024. Lamda: Large model fine-tuning via spectrally decomposed low-dimensional adaptation. arXiv preprint arXiv:2406.12832 (2024)

    arXiv 2024

  4. [3]

    Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising textual entail- ment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment, Vol. 1. Citeseer

    work page 2006

  5. [4]

    Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The Fifth PASCAL Recognizing Textual Entailment Challenge. TAC (2009)

    work page 2009

  6. [5]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044 (2019)

    arXiv 2019

  7. [6]

    Leonard Colavito and Dennis Silage. 2009. Efficient PGA LFSR implemen- tation whitens pseudorandom numbers. In 2009 international conference on Reconfigurable Computing and FPGAs. IEEE, 308–313

    work page 2009

  8. [7]

    Jonas Crols, Guilherme Paim, Shirui Zhao, and Marian Verhelst. 2024. TreeGRNG: Binary Tree Gaussian Random Number Generator for Efficient Probabilistic AI Hardware. In DATE

    work page 2024

Show all 48 references
  1. [8]

    Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. InMachine learning challenges workshop. Springer, 177–190

    2005

  2. [9]

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems 35 (2022), 30318–30332

    2022

  3. [10]

    Amit Dhurandhar, Tejaswini Pedapati, Avinash Balakrishnan, Pin-Yu Chen, Karthikeyan Shanmugam, and Ruchir Puri. 2019. Model agnostic contrastive explanations for structured data. arXiv preprint arXiv:1906.00117 (2019)

    2019 arXiv

  4. [11]

    Jianwei Feng and Dong Huang. 2021. Optimal gradient checkpoint search for arbitrary computation graphs. In CVPR. 11433–11442

    2021

  5. [12]

    Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723 (2020)

    2020 arXiv

  6. [13]

    Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing

    2007

  7. [15]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

    2021 arXiv

  8. [16]

    Jin Hee Kim, Brett Grady, Ruolong Lian, John Brothers, and Jason H Anderson

  9. [17]

    D-U Lee, John D Villasenor, Wayne Luk, and Philip Heng Wai Leong. 2006. A hardware Gaussian noise generator using the Box-Muller method and its error analysis. IEEE transactions on computers (2006)

    2006

  10. [18]

    Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning

    2012

  11. [19]

    Jie Li, Dingjiang Yan, Fangzhou He, Zhicheng Dong, and Mingfei Jiang. 2024. A Mixed-Precision Transformer Accelerator With Vector Tiling Systolic Array for License Plate Recognition in Unconstrained Scenarios. IEEE Transactions on Intelligent Transportation Systems (2024), 1–1...

    2024

  12. [20]

    Sijia Liu, Bhavya Kailkhura, Pin-Yu Chen, Paishun Ting, Shiyu Chang, and Lisa Amini. 2018. Zeroth-order stochastic variance reduction for nonconvex optimiza- tion. Advances in Neural Information Processing Systems 31 (2018)

    2018

  13. [21]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR abs/1907.11692 (2019). http://arxiv.org/abs/1907.11692

    2019 arXiv

  14. [22]

    Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You

  15. [23]

    Yufei Ma, Yu Cao, Sarma Vrudhula, and Jae-Sun Seo. 2019. Performance modeling for CNN inference accelerators on FPGA.IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 39, 4 (2019), 843–856

    2019

  16. [24]

    Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. 2023. Fine-tuning language models with just forward passes. Nerips (2023)

    2023

  17. [25]

    Taesik Na, Jong Hwan Ko, Jaeha Kung, and Saibal Mukhopadhyay. 2017. On-chip training of recurrent neural networks with limited numerical precision. InIJCNN. IEEE, 3716–3723

    2017

  18. [26]

    Mohammad Taher Pilehvar and Jose Camacho-Collados. 2018. WiC: the word-in- context dataset for evaluating context-sensitive meaning representations. arXiv preprint arXiv:1808.09121 (2018)

    2018 arXiv

  19. [27]

    Francesco Restuccia, Alessandro Biondi, Mauro Marinoni, Giorgiomaria Cicero, and Giorgio Buttazzo. 2020. AXI HyperConnect: A Predictable, Hypervisor-level Interconnect for Hardware Accelerators in FPGA SoC. In 2020 57th ACM/IEEE Design Automation Conference (DAC). 1–6. https:/...

    2020

  20. [28]

    Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI spring symposium series

    2011

  21. [29]

    Ozan Sener and Vladlen Koltun. 2020. Learning to Guide Random Search. CoRR abs/2004.12214 (2020). https://arxiv.org/abs/2004.12214

    2020 arXiv

  22. [30]

    Ohad Shamir. 2017. An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. Journal of Machine Learning Research (2017)

    2017

  23. [31]

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP

    2013

  24. [32]

    Qitao Tan, Jun Liu, Zheng Zhan, Caiwei Ding, Yanzhi Wang, Jin Lu, and Geng Yuan. 2025. Harmony in Divergence: Towards Fast, Accurate, and Memory- efficient Zeroth-order LLM Fine-tuning. arXiv preprint arXiv:2502.03304 (2025)

    2025

  25. [33]

    David B Thomas. 2014. FPGA gaussian random number generators with guar- anteed statistical accuracy. In 2014 IEEE 22nd Annual International Symposium on Field-Programmable Custom Computing Machines. IEEE, 149–156

    2014

  26. [34]

    David B Thomas. 2015. The table-Hadamard GRNG: An area-efficient FPGA Gaussian random number generator. TRETS 8, 4 (2015), 1–22

    2015

  27. [35]

    Astha Verma, Siddhesh Bangar, A Venkata Subramanyam, Naman Lal, Rajiv Ratn Shah, and Shin’ichi Satoh. 2023. Certified Zeroth-order Black-Box Defense with Robust UNet Denoiser. arXiv preprint arXiv:2304.06430 (2023)

    2023 arXiv

  28. [36]

    Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test col- lection. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval

    2000

  29. [37]

    Xiaoxing Wang, Wenxuan Guo, Jianlin Su, Xiaokang Yang, and Junchi Yan. 2022. Zarts: On zero-order optimization for neural architecture search. Advances in Neural Information Processing Systems 35 (2022), 12868–12880

    2022

  30. [38]

    John Wishart. 1928. The generalised product moment distribution in samples from a normal multivariate population. Biometrika (1928)

    1928

  31. [39]

    Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. 2020. Pyhes- sian: Neural networks through the lens of the hessian. In Big data. IEEE

    2020

  32. [40]

    Geng Yuan, Sung-En Chang, Qing Jin, Alec Lu, Yanyu Li, Yushu Wu, Zhenglun Kong, Yanyue Xie, Peiyan Dong, Minghai Qin, et al . 2022. You Already Have It: A Generator-Free Low-Precision DNN Training Framework Using Stochastic Rounding. In European Conference on Computer Vision. Springer

    2022

  33. [41]

    Shulin Zeng, Jun Liu, Guohao Dai, Xinhao Yang, Tianyu Fu, Hongyi Wang, Wen- heng Ma, Hanbo Sun, Shiyao Li, Zixiao Huang, et al. 2024. Flightllm: Efficient large language model inference with a complete mapping flow on fpgas. InProceedings of the 2024 ACM/SIGDA International Sy...

    2024

  34. [42]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022)

    2022 arXiv

  35. [43]

    Yihua Zhang et al . 2024. Revisiting zeroth-order optimization for memory- efficient llm fine-tuning: A benchmark. arXiv preprint arXiv:2402.11592 (2024)

    2024 arXiv

  36. [44]

    Yuxin Zhang, Lirui Zhao, Mingbao Lin, Yunyun Sun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. 2023. Dynamic sparse no training: Training-free fine-tuning for sparse llms.arXiv preprint arXiv:2310.08915 (2023)

    2023 arXiv

  37. [45]

    Yanjun Zhao, Sizhe Dang, Haishan Ye, Guang Dai, Yi Qian, and Ivor W Tsang. 2024. Second-order fine-tuning without pain for llms: A hessian informed zeroth-order optimizer. arXiv preprint arXiv:2402.15173 (2024)

    2024 arXiv

  38. [46]

    Hongyu Zhu et al . 2022. ROLLER: Fast and Efficient Tensor Compilation for Deep Learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). Carlsbad, CA, 233–248

    2022

  39. [2017]

    In 2017 30th IEEE International System-on-Chip Conference (SOCC)

    FPGA-based CNN inference accelerator synthesized from multi-threaded C software. In 2017 30th IEEE International System-on-Chip Conference (SOCC). IEEE, 268–273

    2017

  40. [2024]

    arXiv preprint arXiv:2402.15751 (2024)

    Sparse mezo: Less parameters for better performance in zeroth-order llm fine-tuning. arXiv preprint arXiv:2402.15751 (2024)

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.