{"id":"49c52f78-40d5-43c4-bbfa-7f33b1d17263","arxiv_id":"2501.19135","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Tensor-train compression of LLM linear layers plus a systolic FPGA accelerator reduces first-token latency by 1.45x to 1.57x, with whole-network compression of 1.94x and 1.60x and modest accuracy loss.","lead":"Researchers compressed the linear layers inside two large language models using tensor-train decomposition and mapped the compressed models onto an FPGA accelerator. The design cuts the time to first token by about 1.45x to 1.57x compared with the same accelerator running the uncompressed, quantized models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLaMA2-7B first-token delay table is internally inconsistent: 19 TTD blocks at 413.5us plus 13 uncompressed blocks at 735.1us gives 17.76ms, not the reported 15.20ms; the 1.57x speedup is therefore not yet substantiated.","rationale":"The reader's weakest assumption was that Vivado emulation timings may mis-model HBM bandwidth or buffer contention. My concern is stronger and more elementary: the LLaMA2-7B timing tables do not add up even on their own terms. This is not a manufacturing issue; it is a consistency check any reviewer can do from the tables. The ChatGLM3-6B numbers reconstruct well (15*313.11 + 13*686.6 + 701.18 = 14.32 ms vs reported 14.34 ms), which suggests the authors did compute first-token delay as a sum, making the LLaMA discrepancy (19*413.46 + 13*735.1 + 349.16 = 17.76 ms vs reported 15.20 ms) hard to dismiss as a definitional difference. The whole-network CR check reinforces the suspicion that the LLaMA row has a miscounted block count or misreported per-block latency. If the emulation rerun confirms 17.76 ms, the headline 1.57x speedup is overstated by about 15%. This does not invalidate the TTD-plus-GVSA idea, but it means the central quantitative claim needs correction before acceptance. I therefore keep the reader's CONDITIONAL verdict; the condition should include a corrected and internally consistent LLaMA timing table.","tokens_in":11628,"tokens_out":13806,"duration_ms":122860,"concrete_test":"Recompute the LLaMA2-7B first-token delay using the stated numbers: total blocks = 32, TTD blocks = 19, TTD block latency = 413.46 us (sum of Table IV), non-TTD block latency = 735.1 us (Fig. 8 baseline), output layer = 349.16 us. If the sum is not 15.20 ms, rerun the accelerator emulation and supply a corrected Table IV, Fig. 8, and block-count statement; also re-derive the whole-network CR from the per-layer sizes in Table I to check whether 19 compressed blocks can yield 1.60x.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is internal arithmetic inconsistency in the headline LLaMA2-7B timing result, independent of whether emulation matches silicon. Section V-B and Table I state that 19 of 32 blocks are TTD-compressed, while the remaining blocks are only INT4-quantized. Summing Table IV's per-operation latencies gives 413.46 us for a TTD-compressed block, and Fig. 8 gives 735.1 us for an uncompressed block; the output layer adds 349.16 us. The implied first-token delay is 19*413.46 + 13*735.1 + 349.16 = 17,761 us = 17.76 ms, not the 15.20 ms reported in Table IV. Conversely, 15.20 ms would require about 27 TTD-compressed blocks, contradicting the stated 19. The whole-network CR is also inconsistent: 19 blocks at single-block CR 4.01 would give a network CR of about 1.80, not 1.60 (16 blocks would give 1.60). Since the 1.57x speedup is computed from the reported 15.20 ms, the central claim would drop to roughly 1.34x if the reconstructed 17.76 ms is correct. This must be resolved before the speedup claim is accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes applying tensor-train decomposition (TTD) to selected linear layers of ChatGLM3-6B and LLaMA2-7B, and maps the resulting TT-core contractions onto an FPGA group-vector systolic array (GVSA) with FP16 x INT4 DSP-shared processing elements. Whole-network compression ratios of 1.94x and 1.60x are reported, with a C-Eval score decrease of 4.21 and a C4 perplexity increase of 2.62. Against the same accelerator without TTD, the authors report 1.45x and 1.57x first-token delay reductions. The TTD mathematics is standard, and the hardware mapping (ping-pong reordering, dataflow scheduling on GVSA, DSP packing) is described in detail. However, the headline LLaMA2-7B timing result is internally inconsistent: the per-block latencies in Table IV and Fig. 8, combined with the stated 19 compressed blocks, yield a first-token delay of about 17.76 ms rather than the reported 15.20 ms; the single-block and whole-network compression ratios likewise imply about 16 compressed blocks, not 19.","tokens_in":1551,"tokens_out":3143,"duration_ms":119200,"significance":"If the results are corrected and validated, the paper would make a useful contribution: it demonstrates an end-to-end path from TT-compressed LLM weights to a systolic-array implementation, with concrete mechanisms for reorder-free TTD inference and DSP sharing. The controlled comparison against the same GVSA without TTD is a legitimate way to isolate the effect of compression, and the resource and per-operation delay tables are informative. At present, the load-bearing LLaMA2-7B speedup and whole-network compression ratio are not internally consistent, and the absence of board-level measurements leaves the emulation-to-silicon mapping unverified. The results therefore must be treated as conditional until these points are resolved.","major_comments":[{"comment":"Section V-B and Table IV report 19 TTD-compressed blocks out of 32 for LLaMA2-7B. Summing the per-operation delays in Table IV gives 413.46 us per TTD block; Fig. 8 gives 735.1 us for an uncompressed block; the output layer contributes 349.16 us. The first-token delay is therefore 19 x 413.46 + 13 x 735.1 + 349.16 = 17,761 us = 17.76 ms, not the reported 15.20 ms. The speedup against the approximately 23.87 ms baseline would be about 1.34x, not 1.57x. Please correct the table, the figure, or the stated block count, and recompute all downstream speedups.","section":"V-B, Table IV, Fig. 8"},{"comment":"Table I lists a single-block compression ratio of 4.01 for LLaMA2-7B and a whole-network compression ratio of 1.60. With 32 blocks and 19 compressed at CR 4.01 while 13 remain at CR 1, the whole-network CR is 32 / (19/4.01 + 13) = 1.80, not 1.60. The reported 1.60 corresponds to about 16 compressed blocks. The block count and the compression ratio must be reconciled.","section":"V-B, Table I"},{"comment":"The performance numbers are obtained from 'accelerator system emulation for V80' rather than from measurements on the physical FPGA, while the abstract and conclusion describe an FPGA implementation. The delay and speedup claims depend on how the emulation models HBM/DDR bandwidth, AXI contention, and the ping-pong buffer reordering between TTD stages. Please state explicitly which results are post-implementation timing reports and which are emulation, and provide validation or justification for the emulation timings, for example by comparing against a small on-board measurement or a known reference design.","section":"V-A"},{"comment":"The criterion for selecting which blocks to compress is not stated ('we choose specific blocks to perform TTD compression'). This choice directly determines the accuracy/speedup trade-off and is needed for reproducibility. Please specify the selection rule, such as a sensitivity analysis or a resource budget, and ideally report the sensitivity of accuracy and speedup to different block selections.","section":"V-A"},{"comment":"The accuracy results (C-Eval decrease of 4.21 and C4 perplexity increase of 2.62) are reported as single numbers with no variance, number of runs, or evaluation protocol details. Since these metrics are part of the paper's central claim, please report standard deviations or intervals and describe the evaluation setup (prompting, decoding parameters, dataset size).","section":"V-B"}],"minor_comments":[{"comment":"The text says the AMD Alveo V80 has 12574K LUTs, while Table II lists 2574K LUTs as available; this appears to be a typo and should be corrected.","section":"V-A, Table II"},{"comment":"The definitions of 'CR for Linear Layer', 'CR for Single Block', and 'CR for Whole Network' are not given explicitly, and the role of INT4 quantization in these ratios is unclear. Please define each quantity and state whether quantization is included in the reported compression ratios.","section":"Table I"},{"comment":"The bar values in Fig. 8 are difficult to read from the typeset text; providing the exact values in a table or in the caption would help verify the reported speedups.","section":"Fig. 8"},{"comment":"The text refers to BatchNorm (BN) in the fused operators, but ChatGLM3-6B and LLaMA2-7B use LayerNorm rather than BatchNorm; please clarify the terminology.","section":"III-A"},{"comment":"The Factorization(N, d) function is not defined; the choice of tensorization shapes (n_k, m_k) in Table I appears ad hoc and should be explained or referenced.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The citation pattern is heavily self-referential: references [5], [7], [8], [12], [16], [17], [18], [21], [22], [23], and [24] share authors with this submission. This is not itself a defect, but the novelty relative to the earlier GVSA paper [21] should be stated more crisply. The paper also claims an FPGA implementation while reporting system emulation; this should be clarified before acceptance because it affects how the contribution is positioned. The topic fits the scope of cs.AR, but the accuracy evaluation is thin relative to the ML benchmarks used."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a legitimate engineering effort: tensor-train decomposition applied to the linear layers of ChatGLM3-6B and LLaMA2-7B, with the resulting inference mapped onto a group-vector systolic array using DSP-shared FP16xINT4 MACs. The mapping, including ping-pong buffer reordering between TTD stages, is described in enough detail to be useful. Second, the headline LLaMA2-7B speedup is not substantiated by the paper's own numbers.\n\nWhat is genuinely new is the combination of TTD-compressed LLM inference on FPGA with a systolic array built for TTD's multi-stage contractions. Per-layer compression ratios (481-1446x) are high, and the accuracy impact is modest: 4.21 C-Eval drop for ChatGLM3-6B and 2.62 PPL increase for LLaMA2-7B. Comparing against the same accelerator without TTD is a fair controlled experiment, which is more than many papers do.\n\nNow the soft spots, and they are load-bearing. The LLaMA2-7B timing arithmetic does not close. Table IV's per-operation latencies sum to 413.46us for a TTD-compressed block. With 19 TTD blocks, 13 uncompressed blocks, and 349.16us for the output layer, the first-token delay should be roughly 19*413.46 + 13*1070.9 + 349.16 = 22.1ms, not 15.20ms. The stress-test note's own calculation used 735.1us as the uncompressed block delay, but that is the MLP-only figure from Fig. 8; the full uncompressed block is about 335.8+735.1us. Either way, 15.20ms does not reproduce. Even if all 32 blocks were TTD-compressed, the sum would be 13.58ms, so adding 13 uncompressed blocks can only push the total higher. Similarly, 19 blocks at single-block CR 4.01 give a whole-network CR around 1.80, not 1.60; 16 blocks would give 1.60. One of these numbers is wrong. Because the 1.57x speedup is computed from 15.20ms, the central claim currently rests on a number that does not come out of the paper.\n\nBeyond that, the FPGA results come from Vivado system emulation rather than board measurements; accuracy is reported from a single run with no variance; and the criteria for choosing which blocks to TTD-compress are not stated. The GPU and FlightLLM comparisons are also underspecified, since FlightLLM runs on a different FPGA (U280 vs V80).\n\nThat said, the core approach is sound and the flaws look fixable. This paper deserves a serious referee if the authors are willing to redo the timing tables and get board-level measurements. I would not desk-reject it, but I would send it back with major revision and a request to make the arithmetic internally consistent.","headline":"Solid TTD-on-FPGA engineering, but the LLaMA2-7B timing and CR tables don't add up and the speedup claim needs board measurements.","tokens_in":12491,"tokens_out":9811,"would_cite":false,"duration_ms":78257,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tensor-train decomposition of linear layers, paired with a group-vector systolic array and DSP-shared FP16 x INT4 PEs, lets ChatGLM3-6B and LLaMA2-7B fit on an FPGA and cuts first-token delay by 1.45x and 1.57x.","keywords":["tensor-train decomposition","LLM compression","FPGA accelerator","systolic array","linear layer compression","first-token latency","ChatGLM3","LLaMA2"],"falsifier":"Run the same TTD-compressed ChatGLM3-6B and LLaMA2-7B designs on a physical V80 FPGA and compare end-to-end first-token delays against the emulated 14.34ms and 15.20ms; if the no-TTD baseline is not slower by the claimed 1.45x/1.57x, or if the TTD blocks become memory-bound under real HBM contention, the central speedup claim fails.","tokens_in":11419,"feed_emoji":"⚡","tokens_out":9875,"duration_ms":81634,"temperature":0.7,"pith_summary":"The paper claims that tensor-train decomposition (TTD) can compress the linear layers of 6B- and 7B-scale LLMs enough for FPGA deployment, and that a systolic array built for TTD inference makes the compressed model faster than the same accelerator running the uncompressed network. Applying TTD to selected transformer blocks gives whole-network compression ratios of 1.94x for ChatGLM3-6B and 1.60x for LLaMA2-7B, with C-Eval score down 4.21 and C4 perplexity up 2.62. On the group-vector systolic array with DSP-shared FP16 x INT4 processing elements, first-token delay falls by 1.45x and 1.57x relative to the same accelerator without TTD inference. This matters because it offers a concrete path to billion-parameter LLM inference on resource-constrained hardware without relying on sparsity or aggressive quantization alone.","feed_headline":"Tensor-train compression of LLM layers cuts FPGA first-token delay 1.57x","feed_subtitle":"Whole-network compression of 1.94x and 1.60x for ChatGLM3-6B and LLaMA2-7B costs 4.21 C-Eval points and 2.62 PPL.","key_machinery":"The load-bearing object is the tensor-train decomposition itself: the compressed form is a chain of low-rank cores $G_k$, and TTD inference multiplies the tensorized input through those cores one at a time, summing over each input dimension in sequence. The group-vector systolic array (GVSA) executes these contractions, using DSP-shared parallel vector processing elements that pack two INT4 weights with one FP16 feature into a single 27x18 DSP multiplier, so the compute runs as FP16 x INT4 without dedicated FP16 multipliers. The reordering needed between TTD stages is hidden by writing each intermediate $P_k$ into ping-pong buffers along a block dimension and an address dimension, then reading across blocks so that the next stage sees the data already in the required layout.","core_discovery":"The central discovery is that tensor-train decomposition of linear-layer weight matrices is compatible with high-throughput systolic-array inference, and the combination produces measurable end-to-end latency gains on an FPGA-class accelerator. The paper tensorizes each linear weight $W$ into a $d$-dimensional tensor, factorizes it into rank-16 cores $G_1,\\ldots,G_d$, and computes output $Y$ by sequential contractions $P_k[t_{k-1},j_k]=\\sum_{i_k} G_k[i_k,j_k]P_{k-1}[i_k,t_{k-1}]$, reordering the intermediate $P_k$ in ping-pong buffers between stages. With TTD applied to 15 blocks of ChatGLM3-6B and 19 blocks of LLaMA2-7B while attention Q/K/V layers remain uncompressed but INT4-quantized, the whole networks shrink by 1.94x and 1.60x. The deployed accelerator reaches first-token delays of 14.34ms and 15.20ms, which the paper reports as 1.45x and 1.57x reductions versus the same accelerator without TTD inference, with MLP speedups of 3.22x and 3.88x.","pith_inferences":["The ping-pong reordering trick is not specific to tensor-train: any decomposition that reshapes an intermediate tensor between contractions could hide its transpose or reshape in buffer addressing, so the same accelerator style may carry CP, Tucker, or mixed decompositions.","Because only attention Q/K/V and output layers were left uncompressed, and per-block compression ratios are 10.72x and 4.01x, applying the same pipeline to more blocks or to larger 13B/70B models is a plausible extension whose accuracy cost would need re-measuring.","A physical-silicon measurement is the natural next experiment: if HBM/DDR bandwidth and the ping-pong reorder behave as modeled, the claimed 1.45x/1.57x first-token gain should reproduce on real hardware and persist for long generations because of the KV cache."],"forward_implications":["On the target FPGA, TTD inference is 3.22x and 3.88x faster in MLP evaluation and 2.19x and 1.78x faster per transformer block for ChatGLM3-6B and LLaMA2-7B, respectively.","Whole-network compression of 1.94x and 1.60x means both models deploy on a 32GB-HBM FPGA, so TTD can remove part of the weight-memory pressure that normally limits linear-layer accelerators.","The accuracy cost of the combined TTD plus INT4 pipeline is bounded: 4.21 C-Eval points for ChatGLM3-6B and 2.62 perplexity points for LLaMA2-7B on C4.","The same accelerator runs both transformer families by chaining only three operation types, TTD-linear, linear, and nonlinear, which suggests the mapping is portable across LLM architectures.","For both models the speed advantage over the no-TTD baseline persists across the decode phase with a KV cache, not just at the first token."],"supporting_citations":[{"why":"It defines tensor-train decomposition, the factorization algorithm and core notation used throughout the compression pipeline.","marker":"[14]"},{"why":"It demonstrates TTD's very high compression ratios for neural-network linear layers, motivating its application to LLMs.","marker":"[15]"},{"why":"It identifies the ChatGLM3-6B transformer architecture whose blocks are compressed and deployed.","marker":"[19]"},{"why":"It identifies the LLaMA2-7B transformer architecture whose blocks are compressed and deployed.","marker":"[20]"},{"why":"It provides the group-vector systolic array architecture reused as the compute core for TTD inference.","marker":"[21]"},{"why":"It supplies the system-emulation methodology and the throughput baseline compared in the hardware evaluation.","marker":"[25]"},{"why":"It supplies the C-Eval benchmark used to measure the 4.21 accuracy decrease for ChatGLM3-6B.","marker":"[26]"},{"why":"It supplies the C4 corpus used to measure the 2.62 perplexity increase for LLaMA2-7B.","marker":"[27]"}],"fun_headline_variants":["Tensor-train FPGA cuts LLM first-token delay 1.57x","TTD on FPGA: 1.94x smaller, 1.57x faster first token","FPGA tensor-train: 1.94x compression, 1.57x latency drop","Systolic array plus tensor-train: 1.94x smaller, 1.57x faster","TTD on FPGA: 1.94x model shrink, 1.57x first-token cut"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed 1.45x and 1.57x first-token delay reductions come from system emulation for the V80 platform rather than from measurements on a physical FPGA; if the emulated HBM/DDR bandwidth, AXI contention, or ping-pong buffer reordering is optimistic, the real speedups would be smaller.","fun_headline_variants_meta":{"raw":{"variants":["Tensor-train FPGA cuts LLM first-token delay 1.57x","TTD on FPGA: 1.94x smaller, 1.57x faster first token","FPGA tensor-train: 1.94x compression, 1.57x latency drop","Systolic array plus tensor-train: 1.94x smaller, 1.57x faster","TTD on FPGA: 1.94x model shrink, 1.57x first-token cut"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000631,"raw_usage":{"total_tokens":2958,"prompt_tokens":1034,"completion_tokens":1924,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":650,"completion_tokens_details":{"reasoning_tokens":1798}},"tokens_in":650,"tokens_out":1924,"duration_ms":17269,"temperature":1.0,"reasoning_tokens":1798,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T21:10:52.791341+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same TTD-compressed ChatGLM3-6B and LLaMA2-7B designs on a physical V80 FPGA and compare end-to-end first-token delays against the emulated 14.34ms and 15.20ms; if the no-TTD baseline is not slower by the claimed 1.45x/1.57x, or if the TTD blocks become memory-bound under real HBM contention, the central speedup claim fails.","supporting_citations":[{"cited_title":"C-eval: A multi-level multi- discipline chinese evaluation suite for foundation models,","cited_arxiv_id":null,"evidence_quote":"It supplies the C-Eval benchmark used to measure the 4.21 accuracy decrease for ChatGLM3-6B."},{"cited_title":"Exploring the limits of transfer learning with a unified text-to-text transformer,","cited_arxiv_id":null,"evidence_quote":"It supplies the C4 corpus used to measure the 2.62 perplexity increase for LLaMA2-7B."}],"review_version":1}