Pith. sign in

REVIEW 1 major objections 6 minor 60 references

Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimization

T0 review · 1 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that tensor-compressed transformer training can run end-to-end on an FPGA with all parameters and gradients held in on-chip memory, cutting memory 30–51x and energy up to 4x per epoch versus GPU baselines.

desk verdict First FPGA accelerator for end-to-end tensor-compressed transformer training; the hardware numbers are estimates, but the BTT contraction idea is real and worth refereeing. read the letter →

arxiv 2501.06663 v2 pith:SNKAEL6Q submitted 2025-01-11 cs.LG cs.ARcs.CL

Jiayi Tian , Jinming Lu , Hai Li , Xiangwei Wang , Cong Hao , Ian Young , Zheng Zhang This is my paper · ORCID
classification cs.LGcs.ARcs.CL
keywords transformertrainingFPGAacceleratortensor-traindecompositionbidirectionalcontractionon-chipmemoryedgelow-rankcompressionenergy-efficient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that full end-to-end transformer training can run on an FPGA with all model parameters, gradients, and most activations held in on-chip memory, by first compressing every weight matrix and embedding table into low-rank tensor-train form. It introduces a bidirectional contraction flow that shortens the sequential contraction chain and removes the dependence of most intermediate tensors on batch size and sequence length, which is what makes the compressed training loop small enough to fit on chip. On transformers of 36.7 to 93.5 MB (FP32) trained on the ATIS dataset, the claimed result is a 30x to 51x reduction in computing memory versus uncompressed GPU training and up to 4.0x lower energy per epoch than tensor-compressed GPU training. The authors present the FPGA design as a proof of concept, with custom kernels for forward, backward, and update passes, task scheduling, operation fusion, and BRAM/URAM grouping; they note that no pruning or quantization is used.

What carries the argument

The load-bearing object is the bidirectional tensor-train contraction (BTT), a reordering of the tensor-network contraction for a TT-linear layer. Instead of contracting the $2d$ TT cores sequentially from right to left, BTT contracts the left and right halves in parallel toward the middle, shortening the critical path from $2d$ to $d+1$ steps and keeping the large factor $K$ (batch size times sequence length) out of all but the final contractions. Around this, the paper builds three supporting mechanisms: fine-grained contraction fusion that replaces large intermediate buffers with $\mathcal{O}(r)$-sized subtensors, task rescheduling that reuses multiplier kernels across the Q, K, and V projections, and tensor-core grouping that packs many small TT cores into BRAM/URAM blocks to approach the theoretical memory limit.

What would settle it

Run the synthesized bitstream on a physical Alveo U50 at the stated 100 MHz and measure one training epoch's wall-clock latency, full-board power, and on-chip memory use for the 6-encoder FP32 ATIS transformer; if the design exceeds the 28.4-MB BRAM plus URAM budget, or if its measured energy per epoch does not beat the RTX 3090 tensor-compressed baseline, the central claim is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that tensor-compressed transformer training stops being memory-infeasible once the tensor-network contractions are scheduled from both ends toward the middle. Standard right-to-left tensor-train (TT) contraction forces every intermediate tensor to carry the batch-times-sequence factor $K$ and creates $2d$ sequential stages; the proposed bidirectional TT (BTT) flow contracts the left $d$ cores and right $d$ cores independently, reducing the number of stages to $d+1$ and making all but the final contraction independent of $K$. Combined with tensor-train-matrix (TTM) compression of embedding tables, the authors show that full forward propagation, backward propagation, and parameter update for transformers with 2 to 6 encoder layers can be executed with less than 6 MB of BRAM and 22.5 MB of URAM on an AMD Alveo U50 FPGA, at the same accuracy as PyTorch training. On the ATIS benchmark, tensor-compressed models match matrix-format test accuracy while shrinking model size 30.5x to 52.0x; the accelerator reports 30x to 51x lower computing memory than uncompressed GPU training and up to 4.0x lower energy per epoch than tensor-compressed GPU training.

Load-bearing premise

The FPGA's energy, latency, power, and on-chip memory numbers are taken from high-level synthesis, placement, and routing reports rather than from executing the design on the physical Alveo U50 board, so the claimed reductions assume those reports accurately predict on-board behavior.

Editorial extensions

If this is right

  • Full-model transformer training can be performed on a single FPGA without storing weights or gradients off chip, so the memory wall that usually forces edge training into fine-tuning is removed for models up to roughly 100 MB uncompressed.
  • Tensor-compressed training becomes more attractive as sequence length grows, because BTT's advantage over sequential TT and over matrix multiplication increases with $K$.
  • The 30x to 51x memory reduction means model sizes that currently require a workstation GPU become trainable on an edge device, at the cost of higher per-epoch latency from the FPGA's lower clock rate.
  • The accelerator's energy per epoch beats both matrix-format and tensor-compressed GPU training, making repeated on-device retraining for domain adaptation cheaper in energy terms.
  • The proof-of-concept leaves DSP and LUT utilization at roughly 40 to 67 percent, so the same architecture has headroom to absorb larger ranks or deeper encoders before resources saturate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The BTT contraction ordering is a contraction-order principle, not an FPGA-specific trick: the same bidirectional, $K$-independent intermediate schedule could be implemented as fused GPU kernels or in tensor-compilation stacks, where intermediate activation memory is the main bottleneck.
  • Because the paper uses only FP32 arithmetic and no pruning or quantization, the memory and energy gains are conservative lower bounds; combining the same tensorized training with low-precision arithmetic or structured sparsity could push edge-trainable transformer sizes further, a testable extension.
  • The claimed 4.0x energy advantage is most sensitive to how the FPGA's real board-level power, including static power, is measured against the GPU at a comparable batch size; a deployment-oriented reproduction should measure full-system power on the physical board rather than from synthesis reports.
  • The BRAM/URAM grouping strategy suggests a general memory-packing rule for tensor accelerators: choose memory block width and depth per group of small tensor cores to minimize rounding waste, a step that any tensor-core storage scheme could adopt.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. The paper presents an FPGA accelerator for end-to-end training of tensor-compressed transformers. The algorithm side introduces a bidirectional tensor-train (BTT) contraction flow that reduces computational and intra-layer memory costs relative to standard right-to-left TT contraction, and a tensor-grouping method for efficient BRAM/URAM utilization. The hardware side is implemented in C++/HLS for the AMD Alveo U50 and evaluated on the ATIS dataset with 2-, 4-, and 6-encoder transformers. The central claims are that the accelerator fits model, activations, and gradients in under 6 MB BRAM plus 22.5 MB URAM, achieves 30x to 51x memory reduction versus uncompressed GPU training, and consumes up to 4.0x less energy per epoch than tensor-compressed training on an RTX 3090. The functional correctness is verified by comparing training curves with a PyTorch GPU implementation, but all FPGA power, latency, and on-chip memory figures come from HLS/RTL implementation reports rather than board-level measurements.

Significance. If the hardware claims were validated on the physical board, this would be the first demonstration of end-to-end tensor-compressed transformer training on an FPGA with model parameters and gradients held entirely in on-chip memory. The BTT contraction scheme and the tensor-grouping BRAM management strategy are useful and novel contributions, and the analytical complexity comparison (Table I, Eqs. 18-21) provides a valuable framework for future tensor-training accelerators. The paper also includes an HLS-based functional evaluation that matches PyTorch training accuracy on ATIS, which is a credible algorithmic check. However, the central energy and memory claims currently rest on synthesis/implementation reports rather than measurements, and one reported memory figure (34.5 MB for the 6-encoder model) is inconsistent with the physical on-chip capacity of the board. With board-level validation or a clearly qualified re-framing of the estimates, the work would be a strong proof-of-concept.

major comments (1)
  1. [Section VI-C-1 and Section VI-A] The functionality evaluation is described as an 'HLS FPGA implementation' but it is not clear whether the training curves come from C/RTL co-simulation, post-synthesis RTL simulation, or a board execution. The experimental setup mentions only synthesis, placement, and routing, not running the bitstream. Since the abstract says the accelerator 'could conduct' training, this conditional language is appropriate, but the paper should explicitly state the validation method for the training curves and, if it is simulation, note that no physical execution was performed.
minor comments (6)
  1. [Abstract] The board name is spelled 'Alevo' in the abstract but 'Alveo' throughout the rest of the paper and in the experimental setup; please make this consistent.
  2. [Section VI-D-2] The text says 'over 3.6x and 3.4x lower energy consumption than TT and BTT training on GPU, respectively,' while the abstract and Fig. 1 report 'up to 4.0x.' The 4.0x figure comes from the GPU-TT ratio (20.2/5.1 = 3.96), so the abstract should specify 'versus sequential TT-based training' or the text should consistently report the 4.0x value.
  3. [Section II-C] There is a typo: 'weight matirx' should be 'weight matrix.'
  4. [Section VI-A / Table IV] The caption of Table IV lists 'Power (W) Dynamic Static Total,' but it is not stated whether these are from the Vitis HLS power report or the RTL implementation report; please state the tool and methodology, and include the ambient/board conditions for any power estimation.
  5. [Section IV-B, Table I] The TTM FLOPs entry 'O(Knd+1((d-2)r^2+2r))' has inconsistent dimensionality (K times n^{d+1} versus other terms scaling as n^d); please check and correct the exponent, or clarify the notation.
  6. [References] Reference [48] contains a typo: 'Workship' should be 'Workshop.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the BTT complexity analysis and accuracy results are derived or benchmarked independently, and the FPGA performance claims rest on RTL estimates rather than on a construction that reduces to the paper's own inputs.

full rationale

The paper's main derivation chain is not circular. The bidirectional tensor-train contraction cost model (Eqs. 18-21 and Table I) is an analytical expression derived from the stated contraction order and compared against matrix, TTM, and right-to-left TT baselines; it is not fitted to the quantities it claims to explain. The algorithmic accuracy claim (Table III and Fig. 13) is validated by matching HLS FPGA training curves against PyTorch GPU training on the same ATIS task, so the correctness of the tensorized training is benchmarked externally rather than assumed. The FPGA memory and energy figures in Tables IV-V are taken from Vitis HLS synthesis and RTL implementation reports rather than from board-level measurement; this is an evidence-strength and correctness-risk issue, not a logical circularity, because the reported numbers are not obtained by renaming a fitted parameter or by invoking an assumption that already contains the result. The paper does cite the authors' earlier tensor-training work, notably CoMERA [56], for TTM embedding compression and low-rank training, but these citations are used as algorithmic building blocks and background, not as the support for the novel FPGA accelerator claims. There is no self-citation chain that uniquely forces the central result, no uniqueness theorem imported from the authors' prior work, and no fitted input that is later relabeled as a prediction. The main performance claims therefore stand or fall on the quality of the HLS/RTL estimates and on fair GPU baselines, which are separate from circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claims rest on standard tensor-decomposition theory plus two significant domain assumptions: that the chosen TT/TTM structure preserves accuracy, and that HLS/RTL reports predict real hardware performance. No new entities are introduced.

free parameters (3)
  • TT rank (r=12) = 12
    Chosen for all TT layers; determines compression ratios (30-52x) and every FLOP/memory estimate in the paper.
  • TTM rank (r=30) = 30
    Chosen for the embedding table; controls the embedding compression and accuracy.
  • Learning rate = 4e-3
    Set for SGD optimization in the ATIS experiments; one hyperparameter, not tuned across runs.
assumptions (3)
  • domain assumption Tensor-train and tensor-train-matrix decompositions can represent the weight matrices and embedding tables with acceptable accuracy loss for this task.
    The entire approach assumes TT/TTM is a valid low-rank structure for transformer weights (Section II-C); validated empirically on ATIS but not for other tasks.
  • domain assumption Uniform-rank and square-dimension simplification in complexity analysis.
    Table I assumes m=n and a single rank r to derive closed-form FLOP/memory expressions; the actual hardware uses mixed dimensions but the trends are expected to hold.
  • domain assumption HLS synthesis and RTL implementation reports faithfully represent on-board performance.
    All FPGA power, latency, and energy figures come from reports (Section VI-C, Table IV), not from running the design on the Alveo U50.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimization." pith.science (2026). https://pith.science/paper/SNKAEL6Q

@misc{pith2026250106663,
  author       = {Pith},
  title        = {Pith review of: Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SNKAEL6Q}},
  note         = {Machine review of arXiv:2501.06663}
}
abstract

Transformer models have achieved state-of-the-art performance across a wide range of machine learning tasks. There is growing interest in training transformers on resource-constrained edge devices due to considerations such as privacy, domain adaptation, and on-device scientific machine learning. However, the significant computational and memory demands required for transformer training often exceed the capabilities of an edge device. Leveraging low-rank tensor compression, this paper presents the first on-FPGA accelerator for end-to-end transformer training. On the algorithm side, we present a bi-directional contraction flow for tensorized transformer training, significantly reducing the computational FLOPS and intra-layer memory costs compared to existing tensor operations. On the hardware side, we store all highly compressed model parameters and gradient information on chip, creating an on-chip-memory-only framework for each stage in training. This reduces off-chip communication and minimizes latency and energy costs. Additionally, we implement custom computing kernels for each training stage and employ intra-layer parallelism and pipe-lining to further enhance run-time and memory efficiency. Through experiments on transformer models within $36.7$ to $93.5$ MB using FP-32 data formats on the ATIS dataset, our tensorized FPGA accelerator could conduct single-batch end-to-end training on the AMD Alevo U50 FPGA, with a memory budget of less than $6$-MB BRAM and $22.5$-MB URAM. Compared to uncompressed training on the NVIDIA RTX 3090 GPU, our on-FPGA training achieves a memory reduction of $30\times$ to $51\times$. Our FPGA accelerator also achieves up to $3.6\times$ less energy cost per epoch compared with tensor Transformer training on an NVIDIA RTX 3090 GPU.

Figures

Figures reproduced from arXiv: 2501.06663 by the authors.

Figure 1
Figure 1. Comparison of computing memory consumption (top) and energy costs (bottom) between PyTorch-based GPU training and our accelerator; blue bars represent reserved CUDA memory without additional overhead. ory efficiency of tensor-compressed forward and backward propagation. This contraction flow surpasses the sequential computation flows employed in previous inference acceler￾ators [41], [42]. In addition, we develop a … view at source ↗
Figure 2
Figure 2. Transformer structure for classification tasks. Inter-layer activation is represented using yellow blocks, embedding tables and linear layer weights are [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Tensor graph representation for tensors, tensor contraction, and tensor [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Tensor graph representations for the TT-format (a) forward propagation, (b) gradient computation w.r.t. activations and (c) gradient computation w.r.t. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Comparison of the computing flow of the TT-format and our modified [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Comparison of the computation and memory costs of MM, TTM, TT [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Computational and memory costs of TTM-based contraction, TT [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: (a): Overall architecture of the tensorized transformer training accelerator. (b) and (c): The detailed interactions between bottom-level computing [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 10
Figure 10. Figure 10: Schedule viewer of BTT-format back propagation before and after tensor fusion. Left: Computational graph of the BTT-linear layer during back￾propagation. Right: Timeline-based execution schedule illustrating how tensor fusion enables buffer reuse and reduces buffer si…
Figure 11
Figure 11. Figure 11: Configurations of BRAM 36K. Number of BRAM of one array. [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: Comparison of the BRAM utilization efficiency using different [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 13
Figure 13. Figure 13: Training accuracy and loss of the accelerator (2-ENC) compared with [PITH_FULL_IMAGE:figures/full_fig_p011_13.png]
Figure 14
Figure 14. Figure 14: BRAM usage for all tensor factors using different memory manage [PITH_FULL_IMAGE:figures/full_fig_p011_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 50 canonical work pages

  1. [1]

    When machine learning meets privacy: A survey and outlook,

    B. Liu, M. Ding, S. Shaham, W. Rahayu, F. Farokhi, and Z. Lin, “When machine learning meets privacy: A survey and outlook,” ACM Computing Surveys (CSUR) , vol. 54, no. 2, pp. 1–36, 2021

    work page 2021

  2. [2]

    Sok: Security and privacy in machine learning,

    N. Papernot, P. McDaniel, A. Sinha, and M. P. Wellman, “Sok: Security and privacy in machine learning,” in European symposium on security and privacy, 2018, pp. 399–414

    work page 2018

  3. [3]

    DeepReach: a deep learning approach to high-dimensional reachability,

    S. Bansal and C. J. Tomlin, “DeepReach: a deep learning approach to high-dimensional reachability,” in 2021 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2021, pp. 1817–1824

    work page 2021

  4. [4]

    A neural network approach applied to multi-agent optimal control,

    D. Onken, L. Nurbekyan, X. Li, S. Wu Fung, S. Osher, and L. Ruthotto, “A neural network approach applied to multi-agent optimal control,” in European Control Conference, 2021. 13

    work page 2021

  5. [5]

    Learning certified control using contraction metric,

    D. Sun, S. Jha, and C. Fan, “Learning certified control using contraction metric,” in Conference on Robot Learning . PMLR, 2021, pp. 1519– 1539

    work page 2021

  6. [6]

    Learning in the wild: When, how, and what to learn for on-device dataset adaptation,

    S. Lee and S. Nirjon, “Learning in the wild: When, how, and what to learn for on-device dataset adaptation,” in Proceedings of the 2nd International Workshop on Challenges in Artificial Intelligence and Machine Learning for Internet of Things , 2020, pp. 34–40

    work page 2020

  7. [7]

    EF-train: Enable efficient on-device CNN training on FPGA through data reshaping for online adaptation or personalization,

    Y . Tang, X. Zhang, P. Zhou, and J. Hu, “EF-train: Enable efficient on-device CNN training on FPGA through data reshaping for online adaptation or personalization,” ACM Transactions on Design Automation of Electronic Systems (TODAES) , vol. 27, no. 5, pp. 1–36, 2022

    work page 2022

  8. [8]

    BOOST: block minifloat-based on-device CNN training accelerator with transfer learning,

    C. Guo, B. Lou, X. Liu, D. Boland, P. H. Leong, and C. Zhuo, “BOOST: block minifloat-based on-device CNN training accelerator with transfer learning,” in Intl. Conf. Computer Aided Design , 2023, pp. 1–9

    work page 2023

Show all 60 references
  1. [9]

    ETA: an efficient training accelerator for DNNs based on hardware-algorithm co-optimization,

    J. Lu, C. Ni, and Z. Wang, “ETA: an efficient training accelerator for DNNs based on hardware-algorithm co-optimization,” IEEE Transac- tions on Neural Networks and Learning Systems , 2022

    2022

  2. [10]

    Training deep neural networks in low-precision with high accuracy using FPGAs,

    S. Fox, J. Faraone, D. Boland, K. Vissers, and P. H. Leong, “Training deep neural networks in low-precision with high accuracy using FPGAs,” in Intl. Conf. Field-Programmable Technology, 2019, pp. 1–9

    2019

  3. [11]

    FAST: DNN training under variable precision block floating point with stochastic rounding,

    S. Q. Zhang, B. McDanel, and H. Kung, “FAST: DNN training under variable precision block floating point with stochastic rounding,” in Int. Symp. High-Performance Computer Architecture , 2022, pp. 846–860

    2022

  4. [12]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017

    2017

  5. [13]

    BERT: Pre-training of deep bidirectional transformers for language understanding,

    J. D. M.-W. C. Kenton and L. K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of naacL-HLT, vol. 1, 2019, p. 2

    2019

  6. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020

    2010 arXiv

  7. [15]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Loet al., “Segment anything,” in Proc. CVPR, 2023, pp. 4015–4026

    2023

  8. [16]

    PINNsFormer: A transformer- based framework for physics-informed neural networks,

    Z. Zhao, X. Ding, and B. A. Prakash, “PINNsFormer: A transformer- based framework for physics-informed neural networks,” in Int. Conf. Learning Representations, 2024

    2024

  9. [17]

    Federated learning in asr: Not as easy as you think,

    W. Yu, J. Freiwald, S. Tewes, F. Huennemeyer, and D. Kolossa, “Federated learning in asr: Not as easy as you think,” in Speech Communication; 14th ITG Conference . VDE, 2021, pp. 1–5

    2021

  10. [18]

    Federated domain adaptation for asr with full self-supervision,

    J. Jia, J. Mahadeokar, W. Zheng, Y . Shangguan, O. Kalinli, and F. Seide, “Federated domain adaptation for asr with full self-supervision,” arXiv preprint arXiv:2203.15966, 2022

    2022 arXiv

  11. [19]

    BEBERT: efficient and robust binary ensemble BERT,

    J. Tian, C. Fang, H. Wang, and Z. Wang, “BEBERT: efficient and robust binary ensemble BERT,” in Intl. Conf. Acoustics, Speech and Signal Processing, 2023, pp. 1–5

    2023

  12. [20]

    BETA: binarized energy-efficient trans- former accelerator at the edge,

    Y . Ji, C. Fang, and Z. Wang, “BETA: binarized energy-efficient trans- former accelerator at the edge,” arXiv preprint arXiv:2401.11851, 2024

    2024 arXiv

  13. [21]

    Training deep neural networks with 8-bit floating point numbers,

    N. Wang, J. Choi, D. Brand, C.-Y . Chen, and K. Gopalakrishnan, “Training deep neural networks with 8-bit floating point numbers,” in NIPS, 2018, pp. 7675–7684

    2018

  14. [22]

    Quantization- aware and tensor-compressed training of transformers for natural lan- guage understanding,

    Z. Yang, S. Choudhary, S. Kunzmann, and Z. Zhang, “Quantization- aware and tensor-compressed training of transformers for natural lan- guage understanding,” arXiv preprint arXiv:2306.01076 , 2023

    2023 arXiv

  15. [23]

    LoRETTA: low-rank eco- nomic tensor-train adaptation for ultra-low-parameter fine-tuning of large language models,

    Y . Yang, J. Zhou, N. Wong, and Z. Zhang, “LoRETTA: low-rank eco- nomic tensor-train adaptation for ultra-low-parameter fine-tuning of large language models,” in Proc. North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2024, ...

    2024

  16. [24]

    AdaZeta: adaptive zeroth-order tensor-train adaption for memory- efficient large language models fine-tuning,

    Y . Yang, K. Zhen, E. Banijamal, A. Mouchtaris, and Z. Zhang, “AdaZeta: adaptive zeroth-order tensor-train adaption for memory- efficient large language models fine-tuning,” in Proceedings of the Empirical Methods in Natural Language Processing , 2024

    2024

  17. [25]

    An algorithm–hardware co-optimized framework for accelerating N: M sparse transformers,

    C. Fang, A. Zhou, and Z. Wang, “An algorithm–hardware co-optimized framework for accelerating N: M sparse transformers,” IEEE Trans. Very Large Scale Integration Systems , vol. 30, no. 11, pp. 1573–1586, 2022

    2022

  18. [26]

    Efficient N: M sparse DNN training using algorithm, architecture, and dataflow co-design,

    C. Fang, W. Sun, A. Zhou, and Z. Wang, “Efficient N: M sparse DNN training using algorithm, architecture, and dataflow co-design,” IEEE Trans. CAD of Integrated Circuits and Systems , 2023

    2023

  19. [27]

    MINILM: Deep self-attention distillation for task-agnostic compression of pre- trained transformers,

    W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, “MINILM: Deep self-attention distillation for task-agnostic compression of pre- trained transformers,” Advances in Neural Information Processing Sys- tems, vol. 33, pp. 5776–5788, 2020

    2020

  20. [28]

    Wanda++: Pruning large language models via regional gradients,

    Y . Yang, K. Zhen, B. Ganesh, A. Galstyan, G. Huybrechts, M. M ¨uller, J. M. K ¨ubler, R. V . Swaminathan, A. Mouchtaris, S. B. Bodapati et al., “Wanda++: Pruning large language models via regional gradients,” in Sparsity in LLMs (SLLM): Deep Dive into Mixture of Experts, Quan...

  21. [29]

    Compressing dma engine: Leveraging activation sparsity for training deep neural networks,

    M. Rhu, M. O’Connor, N. Chatterjee, J. Pool, Y . Kwon, and S. W. Keckler, “Compressing dma engine: Leveraging activation sparsity for training deep neural networks,” in 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2018, pp. 78–91

    2018

  22. [30]

    Quantized neural networks: Training neural networks with low pre- cision weights and activations,

    I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y . Bengio, “Quantized neural networks: Training neural networks with low pre- cision weights and activations,” The Journal of Machine Learning Research, vol. 18, no. 1, pp. 6869–6898, 2017

    2017

  23. [31]

    Deep learning with limited numerical precision,

    S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deep learning with limited numerical precision,” in International Conference on Machine Learning , 2015, pp. 1737–1746

    2015

  24. [32]

    Ultra- low precision 4-bit training of deep neural networks,

    X. Sun, N. Wang, C.-Y . Chen, J. Ni, A. Agrawal, X. Cui, S. Venkatara- mani, K. El Maghraoui, V . V . Srinivasan, and K. Gopalakrishnan, “Ultra- low precision 4-bit training of deep neural networks,” NIPS, vol. 33, 2020

    2020

  25. [33]

    Tensor decompositions and applications,

    T. G. Kolda and B. W. Bader, “Tensor decompositions and applications,” SIAM review, vol. 51, no. 3, pp. 455–500, 2009

    2009

  26. [34]

    Speeding-up convolutional neural networks using fine-tuned cp- decomposition,

    V . Lebedev, Y . Ganin, M. Rakhuba, I. Oseledets, and V . Lempit- sky, “Speeding-up convolutional neural networks using fine-tuned cp- decomposition,” in 3rd International Conference on Learning Represen- tations, ICLR 2015-Conference Track Proceedings , 2015

    2015

  27. [35]

    Compression of deep convolutional neural networks for fast and low power mobile applications,

    Y .-D. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and D. Shin, “Compression of deep convolutional neural networks for fast and low power mobile applications,” arXiv preprint arXiv:1511.06530 , 2015

    2015 arXiv

  28. [36]

    Fast video facial expression recognition by deeply tensor-compressed LSTM neural network on mobile device,

    P. Zhen, B. Liu, Y . Cheng, H.-B. Chen, and H. Yu, “Fast video facial expression recognition by deeply tensor-compressed LSTM neural network on mobile device,” in Proceedings of the 4th ACM/IEEE Symposium on Edge Computing , 2019, pp. 298–300

    2019

  29. [37]

    Tensorizing neural networks,

    A. Novikov, D. Podoprikhin, A. Osokin, and D. P. Vetrov, “Tensorizing neural networks,” Advances in neural information processing systems , vol. 28, 2015

    2015

  30. [38]

    Compression and interpretability of deep neural networks via Tucker tensor layer,

    G. G. Calvi, A. Moniri, M. Mahfouz, Q. Zhao, and D. P. Mandic, “Compression and interpretability of deep neural networks via Tucker tensor layer,” arXiv:1903.06133, 2019

    1903 arXiv

  31. [39]

    Tensorized embedding layers,

    O. Hrinchuk, V . Khrulkov, L. Mirvakhabova, E. Orlova, and I. Oseledets, “Tensorized embedding layers,” in Findings of the Association for Computational Linguistics: EMNLP 2020 , 2020, pp. 4847–4860

    2020

  32. [40]

    A tensorized transformer for language modeling,

    X. Ma, P. Zhang, S. Zhang, N. Duan, Y . Hou, M. Zhou, and D. Song, “A tensorized transformer for language modeling,” Advances in neural information processing systems , vol. 32, 2019

    2019

  33. [41]

    TIE: energy- efficient tensor train-based inference engine for deep neural network,

    C. Deng, F. Sun, X. Qian, J. Lin, Z. Wang, and B. Yuan, “TIE: energy- efficient tensor train-based inference engine for deep neural network,” in Proc. Int. Symp. Computer Architecture , 2019, pp. 264–278

    2019

  34. [42]

    ETTE: Efficient tensor-train-based computing engine for deep neural networks,

    Y . Gong, M. Yin, L. Huang, J. Xiao, Y . Sui, C. Deng, and B. Yuan, “ETTE: Efficient tensor-train-based computing engine for deep neural networks,” in Proc. Intl. Symp. Computer Architecture , 2023, pp. 1–13

    2023

  35. [43]

    TT-CIM: Tensor train decomposition for neural network in rram-based compute-in-memory systems,

    F.-H. Meng, Y . Wu, Z. Zhang, and W. D. Lu, “TT-CIM: Tensor train decomposition for neural network in rram-based compute-in-memory systems,” IEEE Trans. Circuits and Systems I: Regular Papers , 2023

    2023

  36. [44]

    15.4 a 5.99-to-691.1 TOPS/W tensor-train in- memory-computing processor using bit-level-sparsity-based optimiza- tion and variable-precision quantization,

    R. Guo, Z. Yue, X. Si, T. Hu, H. Li, L. Tang, Y . Wang, L. Liu, M.- F. Chang, Q. Li et al. , “15.4 a 5.99-to-691.1 TOPS/W tensor-train in- memory-computing processor using bit-level-sparsity-based optimiza- tion and variable-precision quantization,” in 2021 IEEE International ...

    2021

  37. [45]

    Tensor- compressed back-propagation-free training for (physics-informed) neural networks,

    Y . Zhao, X. Yu, Z. Chen, Z. Liu, S. Liu, and Z. Zhang, “Tensor- compressed back-propagation-free training for (physics-informed) neural networks,” arXiv preprint arXiv:2308.09858 , 2023

    2023 arXiv

  38. [46]

    Tensorized optical multimodal fusion network,

    Y . Zhao, X. Xiao, G. Kurczveil, R. G. Beausoleil, and Z. Zhang, “Tensorized optical multimodal fusion network,” in 2023 Conference on Lasers and Electro-Optics (CLEO) . IEEE, 2023, pp. 1–2

    2023

  39. [47]

    Real-time fj/mac pde solvers via tensorized, back- propagation-free optical pinn training,

    Y . Zhao, X. Xiao, X. Yu, Z. Liu, Z. Chen, G. Kurczveil, R. G. Beausoleil, and Z. Zhang, “Real-time fj/mac pde solvers via tensorized, back- propagation-free optical pinn training,” in Machine Learning with New Compute Paradigms

  40. [48]

    The ATIS spoken language systems pilot corpus,

    C. T. Hemphill, J. J. Godfrey, and G. R. Doddington, “The ATIS spoken language systems pilot corpus,” in Workship on Speech and Natural Language, 1990

    1990

  41. [49]

    Tensor-train decomposition,

    I. V . Oseledets, “Tensor-train decomposition,” SIAM Journal on Scien- tific Computing, vol. 33, no. 5, pp. 2295–2317, 2011

    2011

  42. [50]

    Tensor-train recurrent neural net- works for video classification,

    Y . Yang, D. Krompass, and V . Tresp, “Tensor-train recurrent neural net- works for video classification,” in International Conference on Machine Learning. PMLR, 2017, pp. 3891–3900

    2017

  43. [51]

    Learning compact recurrent neural networks with block-term tensor decompo- sition,

    J. Ye, L. Wang, G. Li, D. Chen, S. Zhe, X. Chu, and Z. Xu, “Learning compact recurrent neural networks with block-term tensor decompo- sition,” in Proc. Computer Vision and Pattern Recognition , 2018, pp. 9378–9387. 14

    2018

  44. [52]

    Towards compact neural networks via end-to-end training: A Bayesian tensor approach with automatic rank determination,

    C. Hawkins, X. Liu, and Z. Zhang, “Towards compact neural networks via end-to-end training: A Bayesian tensor approach with automatic rank determination,” SIAM Journal on Mathematics of Data Science , vol. 4, no. 1, pp. 46–71, 2022

    2022

  45. [53]

    Bayesian tensorized neural networks with automatic rank selection,

    C. Hawkins and Z. Zhang, “Bayesian tensorized neural networks with automatic rank selection,” Neurocomputing, vol. 453, pp. 172–180, 2021

    2021

  46. [54]

    Compressing 3D CNNs based on tensor train decomposition,

    D. Wang, G. Zhao, G. Li, L. Deng, and Y . Wu, “Compressing 3D CNNs based on tensor train decomposition,” Neural Networks , vol. 131, pp. 215–230, 2020

    2020

  47. [55]

    Nonlinear tensor train format for deep neural network compression,

    D. Wang, G. Zhao, H. Chen, Z. Liu, L. Deng, and G. Li, “Nonlinear tensor train format for deep neural network compression,” Neural Networks, vol. 144, pp. 320–333, 2021

    2021

  48. [56]

    CoMERA: Computing-and memory-efficient training via rank-adaptive tensor optimization,

    Z. Yang, Z. Liu, S. Choudhary, X. Xie, C. Gao, S. Kunzmann, and Z. Zhang, “CoMERA: Computing-and memory-efficient training via rank-adaptive tensor optimization,” in Annual Conference on Neural Information Processing Systems , 2024

    2024

  49. [57]

    An FPGA-based processor for training convolutional neural networks,

    Z. Liu, Y . Dou, J. Jiang, Q. Wang, and P. Chow, “An FPGA-based processor for training convolutional neural networks,” in Intl. Conf. on Field Programmable Technology. IEEE, 2017, pp. 207–210

    2017

  50. [58]

    Automatic compiler based FPGA accelerator for CNN training,

    S. K. Venkataramanaiah, Y . Ma, S. Yin, E. Nurvithadhi, A. Dasu, Y . Cao, and J.-S. Seo, “Automatic compiler based FPGA accelerator for CNN training,” in International Conference on Field Programmable Logic and Applications (FPL). IEEE, 2019, pp. 166–172

    2019

  51. [59]

    FPGA-based low-batch training accelerator for modern CNNs featuring high bandwidth memory,

    S. K. Venkataramanaiah, H.-S. Suh, S. Yin, E. Nurvitadhi, A. Dasu, Y . Cao, and J.-s. Seo, “FPGA-based low-batch training accelerator for modern CNNs featuring high bandwidth memory,” inProc. International Conference on Computer-Aided Design , 2020, pp. 1–8. VIII. B IOGRAPHY S...

    2020

  52. [1983]

    for leadership in research and development on circuits and processes for the evolution of microprocessors

    Currently, he is responsible for defining future circuit directions with emerging novel devices and identifying leading options to manufacture energy efficient integrated circuits for computing in the beyond-CMOS era. He has authored or coauthored more than 300 technical artic...

    2009

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.