{"id":"169ec678-c898-477a-80d4-3d1488d0df56","arxiv_id":"2411.15982","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Anda stores LLM activations in a group-shared-exponent format with per-module mantissa widths and uses bit-serial hardware that skips unused bits, reporting roughly 2.4x speedup, 4.0x area efficiency, and 3.1x energy efficiency over a GPU-like baseline.","lead":"The paper introduces Anda, a new data format that makes the floating-point activations in large language models cheaper to store and compute by sharing one exponent across groups of values and letting each part of the model use fewer mantissa bits. It also designs hardware that skips the skipped bits, reporting about 2.4x speedup, 4.0x area efficiency, and 3.1x energy efficiency over a GPU-like baseline for OPT, LLaMA, and LLaMA-2 models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central accuracy-efficiency claim rests solely on perplexity, with no downstream-task validation and a few Table II entries already exceeding the stated PPL tolerance.","rationale":"The reader's weakest assumption correctly identifies that the accuracy side of the central claim is validated only by perplexity on three language-modeling datasets, with no downstream-task evaluation. My stress-test agrees with this and sharpens it with two additional observations: (1) even within the reported metric, a few Table II entries exceed the stated 1% tolerance (LLaMA2-7B 1.07%, LLaMA-13B 1.16% on WikiText2), so the tolerance is already a soft constraint rather than a guarantee; (2) because the precision search is calibrated and evaluated on the same perplexity metric, any mismatch between perplexity sensitivity and task-level sensitivity directly undermines the claimed accuracy-efficiency balance. The paper has independent support on the hardware side: RTL synthesis at 16nm, power analysis from VCD files, and a cycle-accurate simulator, with sensitivity studies that motivate the variable-length format. No fatal internal inconsistency is evident, and the concern is addressable by a downstream-task evaluation rather than a flaw in the architecture or algorithms. Therefore the reader's CONDITIONAL verdict remains appropriate; no verdict change is warranted.","tokens_in":30365,"tokens_out":5207,"duration_ms":49503,"concrete_test":"Take the Fig. 14 precision combinations found under the 1% tolerance on WikiText2 for a representative subset (e.g., OPT-1.3B, OPT-6.7B, LLaMA-7B, LLaMA2-7B) and evaluate the Anda-quantized W4A16g128 models on a downstream benchmark suite including MMLU (5-shot), HellaSwag, BoolQ, and ARC-Easy, using the same Omniquant weight-only baseline as the reference. Compute the relative task-accuracy drop between the Anda model and the Omniquant baseline. If any model shows an average downstream-accuracy drop exceeding 1% while its validation PPL drop is within 1%, the central accuracy claim is not supported and the efficiency gains would need to be re-qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim pairs large efficiency gains (2.4x/4.0x/3.1x over the FP-FP baseline) with accuracy kept within user-set tolerances of 0.1% or 1%. The accuracy side is established only through perplexity on WikiText2, PTB, and C4 (Sec. V-A/V-B, Table II), and the adaptive precision search itself is calibrated on the same perplexity metric (Algorithm 1). This makes the claimed accuracy guarantee as strong as the assumption that perplexity is a sufficient proxy for task-level LLM quality. Two concrete weaknesses follow. First, the tolerance is applied to calibration-data perplexity, but validation perplexity occasionally exceeds it: for example, LLaMA2-7B shows a 1.07% loss and LLaMA-13B a 1.16% loss on WikiText2 under the 1% constraint (Table II), which the paper dismisses as normal calibration/validation drift. This means the 'within tolerance' statement is not a hard guarantee even for the reported metric. Second, no downstream-task accuracy (e.g., MMLU, HellaSwag, BoolQ, ARC) is reported anywhere in the paper. If the precision combinations that preserve perplexity within 1% nonetheless degrade reasoning or knowledge tasks by more than the user's tolerance, the headline accuracy-efficiency trade-off would be overstated. Since the efficiency numbers are tied to these perplexity-derived precision combinations, the missing task-level validation is the most load-bearing gap in the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Anda, a variable-length grouped block-floating-point activation format for weight-only quantized LLMs, together with a post-training adaptive precision search that selects per-module mantissa widths under a user-specified perplexity-loss tolerance, and a hardware architecture (bit-plane memory layout, bit-serial processing units, runtime bit-plane compressor) to exploit the format. Evaluations on nine OPT/LLaMA/LLaMA-2 models and three datasets report perplexity close to the Omniquant weight-only baseline while cutting bit operations, and RTL/cycle-accurate hardware evaluation claims average 2.4x speedup, 4.0x area efficiency, and 3.1x energy efficiency over a GPU-like FP-FP accelerator.","tokens_in":30683,"tokens_out":8997,"duration_ms":86535,"significance":"If the accuracy-efficiency balance holds, this is a significant contribution to efficient LLM inference: it targets the FP-activation bottleneck that weight-only quantization leaves behind, offers a practical training-free search over four module-wise precisions, and backs the proposal with a coherent hardware design and broad model/dataset coverage. The paper honestly discloses occasional tolerance breaches and provides sensitivity analyses across group sizes, models, and modules. The main weakness is that the accuracy side of the central trade-off is validated only through perplexity, with no downstream-task accuracy, and the reported tolerance semantics are not fully pinned down.","major_comments":[{"comment":"The central accuracy claim is supported only by perplexity on WikiText2, PTB, and C4. The precision combinations in Fig. 14 are selected to keep PPL within tolerance, and the hardware efficiency numbers inherit those mantissa widths. If these PPL-preserving precisions degrade downstream task accuracy (e.g., reasoning or knowledge benchmarks) by more than the user tolerance, the headline accuracy-efficiency trade-off is overstated. Please report at least a few downstream-task benchmarks (e.g., HellaSwag, ARC, BoolQ, MMLU) for a representative subset of models and precisions, or explicitly restrict the claim and the search objective to language-modeling perplexity.","section":"Sec. V-A/V-B, Table II"},{"comment":"The user-set tolerance is not a guarantee on the reported validation metric. Under the 1% constraint, Table II shows LLaMA2-7B at 1.07% loss on WikiText2, LLaMA-13B at 1.16% loss on WikiText2, and OPT-6.7B at 1.01% loss on C4. The text attributes this to calibration/validation drift, but Algorithm 1 enforces the tolerance on the calibration data. If the stated tolerance is meant to hold during actual inference, the search should validate on held-out data or apply a margin; otherwise the abstract's 'within user-set tolerances' should be qualified as calibration-only.","section":"Table II and Sec. V-B"},{"comment":"The sign convention and definition of the red 'accuracy drop' percentages are internally inconsistent. For example, on WikiText2 OPT-1.3B, Omniquant gives 14.88 PPL and Ours(1%) gives 14.99 PPL (a 0.74% increase), but the table reports -0.74%; the text in Sec. V-B says '0.74% accuracy loss' for the same case. If the intended formula is (PPL_ref - PPL_method)/PPL_ref, the signs should be positive for these values; if the intended formula is (PPL_ref/PPL_method - 1), the label should be 'relative accuracy' rather than 'accuracy drop'. The current presentation prevents readers from verifying tolerance compliance.","section":"Table II"},{"comment":"The baseline used as fpacc in Algorithm 1 is ambiguous. If L is the weight-only quantized model, then the tolerance is relative to Omniquant and this should be stated explicitly; if fpacc is evaluated on the original FP16 model, then the red percentages in Table II, which are relative to Omniquant, are not on the same baseline as the search constraint. One clarifying sentence is needed to ensure that the reported 0.1%/1% values correspond to the same reference used in the search.","section":"Algorithm 1, line 4 and Sec. V-A"}],"minor_comments":[{"comment":"The text says the search 'efficiently finds the global optimum within 10 iterations,' but the greedy relaxation strategy is acknowledged later to possibly miss the global optimum; please rephrase to 'near-optimal solution' or provide exhaustive-search evidence for the specific case.","section":"Sec. III-C and Fig. 9"},{"comment":"The claim that the search 'operates approximately twice as fast as Omniquant and ten times faster than GPTQ' is not accompanied by measured search-time data; please add timing measurements or remove the quantitative comparison.","section":"Sec. III-D"},{"comment":"The statement that all hardware baselines are configured with 'equivalent peak throughput' needs a concrete architectural specification (PE counts, array dimensions, dataflow assumptions) so that the system-level comparisons in Fig. 16 are reproducible.","section":"Sec. V-A"},{"comment":"The abstract reports a single 2.4x average speedup, while Sec. V-D reports 2.14x at 0.1% loss and 2.49x at 1% loss; please clarify whether 2.4x is the geometric mean over the two tolerances or the average over all settings.","section":"Abstract and Sec. V-D"},{"comment":"There is a typo in line 1: 'P riorityQueue' should read 'PriorityQueue'.","section":"Algorithm 1"},{"comment":"Reference [6] spells the vendor name as 'Candence'; it should be 'Cadence'.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a solid systems/architecture paper with a well-integrated algorithm-hardware co-design and unusually broad model coverage for the hardware evaluation. The main risk is the accuracy-validation gap: the claim that efficiency is obtained 'while keeping accuracy within user-set tolerances' currently rests on perplexity alone and on calibration-set tolerances that are occasionally exceeded on the validation sets. These issues are fixable with additional experiments and careful rewording, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Anda is a serious hardware-algorithm co-design paper worth taking seriously. The core idea is simple and practical: represent FP activations in a grouped BFP format with a variable mantissa length that can differ per module, and build the datapath around bit-serial computation on bit-plane storage, so shorter mantissas directly cut compute and memory traffic. That combination is genuinely new relative to prior formats—VS-Quant, FIGNA, FAST, DaCapo, FlexBlock all use fixed or coarsely stepped mantissa lengths. The module-wise offline precision search reusing the weight-quantization calibration data is a pragmatic touch, and the sensitivity analysis (group size, per-module effects) is well done and reproducible in spirit. The hardware story is coherent: bit-plane layout, bit-serial APU with shared exponents, and a runtime compressor that converts FP16 outputs back to Anda on the fly. They synthesize at 16nm, report area/power from VCDs, and compare against several baselines including iFPU and FIGNA. That's solid engineering evidence, not hand-waving.\n\nNow the soft spots, in proportion. The accuracy side of the central claim—2.4x/4.0x/3.1x over FP-FP while keeping within 0.1% or 1% loss—is validated only by perplexity on WikiText2, PTB, and C4. No downstream-task accuracy (MMLU, HellaSwag, BoolQ, ARC) appears anywhere. The precision search is calibrated on the same perplexity metric, so the trade-off is only as strong as the assumption that perplexity tracks task quality. That assumption is common in quantized-LM papers, but it is more load-bearing here because the whole efficiency story depends on the chosen mantissa widths. And the tolerance is not a hard guarantee even on perplexity: a few Table II entries, e.g., LLaMA2-7B at 1.07% and LLaMA-13B at 1.16% on WikiText2 under the 1% constraint, exceed the stated bound, which the text dismisses as calibration/validation drift. That is honest but weakens the headline. Also, no code or artifacts are released, and the hardware comparisons are simulation-based with custom baselines rather than measurements on real silicon.\n\nNone of this is disqualifying for an architecture paper. The mechanism is sound, the format is well motivated, and the efficiency numbers follow from the design choices rather than from circular reasoning. The missing downstream-task validation is a clear revision request, not a reason to reject. The heuristic search without optimality guarantees is acceptable for a 4-tuple space and is handled honestly.\n\nI would send this to peer review. The paper deserves a serious referee, and the authors should be asked to add task-level accuracy and report calibration/validation spread. For anyone working on efficient LLM inference or FP-INT hardware, this is a relevant and citable contribution.","headline":"Anda is a serious hardware-algorithm co-design paper with a genuinely new variable-length BFP format and matching bit-serial architecture; the main soft spot is that accuracy rests on perplexity alone, but that is a fixable gap, not a fatal one.","tokens_in":31220,"tokens_out":1851,"would_cite":true,"duration_ms":19554,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Anda shows that replacing FP16 activations with a variable-length, group-shared-exponent format gives 2.4x faster LLM inference, 4.0x better area efficiency, and 3.1x better energy efficiency at near-unchanged perplexity.","keywords":["LLM inference","activation quantization","block floating point","FP-INT GeMM","bit-serial processing","post-training quantization","hardware accelerator","perplexity"],"falsifier":"Measure the exact precision combinations found by the search (e.g., [7,7,6,5] for OPT-125M) on a downstream benchmark such as question answering or reasoning; if task accuracy drops by more than the tolerance while perplexity stays within it, the claimed accuracy-efficiency balance does not transfer. Alternatively, build or simulate the Anda accelerator with the stated HBM2 memory model and verify whether the 2.4x speedup and 3.1x energy efficiency hold at the system level, since the paper's numbers come from a cycle-accurate simulator.","tokens_in":30193,"feed_emoji":"⚡","tokens_out":7357,"duration_ms":61652,"temperature":0.7,"pith_summary":"This paper aims to show that the floating-point activations in weight-only quantized large language models are the real efficiency bottleneck, and that they can be compressed and accelerated without retraining. It proposes Anda, a block-floating-point data format in which a group of activations shares one exponent and each activation tensor chooses its own mantissa width, ranging from 1 to 16 bits. A training-free search over just four tensor types (Aqkv, Ao, Au, Ad) finds precision combinations that keep perplexity within a user-set tolerance (0.1% or 1%) while minimizing bit operations. The paper reports that on OPT, LLaMA, and LLaMA-2 models this yields a 2.4x average speedup, 4.0x area efficiency, and 3.1x energy efficiency over a GPU-like FP-FP baseline, with the same models and tolerances. The significance, if correct, is that weight-only quantized LLMs can be deployed faster and cheaper on custom hardware without the costly retraining that earlier block-floating-point methods required.","feed_headline":"2.4x faster LLM inference from variable-length activations","feed_subtitle":"Anda assigns each module its own mantissa width, cutting energy 3.1x and area 4x without retraining.","key_machinery":"The central object is the Anda data type, a block-floating-point (BFP) format with a sign bit, a group-shared exponent (group size 64), and a variable-length mantissa selectable per tensor from 1 to 16 bits. The argument is carried by three pieces working together: (1) the adaptive precision combination search, which reuses the calibration data of post-training weight quantization to choose mantissa widths for the four FP-INT GeMM activation tensors; (2) a bit-plane data layout that stores mantissa bits of the same significance across 64 values together, keeping memory access regular despite variable lengths; and (3) an Anda-enhanced bit-serial processing unit and a runtime bit-plane compressor that convert FP16 outputs to the compressed format on the fly. The shared exponent removes exponent alignment and normalization inside a group, turning FP-INT dot products into integer operations with a single FP32 accumulation across groups.","core_discovery":"The paper's central claim is that FP16 activations in W4A16 LLMs can be replaced by variable-length, group-shared-exponent activations with only a small, controlled perplexity change, and that this replacement is what unlocks large efficiency gains in FP-INT GeMM operations. The underlying empirical finding is a sensitivity pattern: different LLMs and different modules within a model tolerate different amounts of mantissa truncation, with Aqkv consistently the most sensitive and the feed-forward down-projection Ad often the least. On this basis the paper defines the Anda format and an iterative module-wise search over the 4-tuple [Mqkv, Mo, Mu, Md] that maximizes BOPs reduction subject to an accuracy-loss tolerance. The claim is not that all activations can be aggressively quantized, but that precision can be assigned per module, and that the resulting format, together with bit-plane memory layout and bit-serial processing, delivers the reported system-level gains.","pith_inferences":["If the module-sensitivity pattern generalizes beyond the nine tested models, the 4-tuple search could be replaced by a learned or heuristic prior (Aqkv high precision, Ad low), making deployment even faster.","Anda's gains depend on the system actually exploiting variable precision; on a workload where all tensors need full mantissas, the bit-serial design is less efficient than bit-parallel FIGNA at fixed width, as the paper itself notes.","Combining Anda activation compression with KV-cache quantization, which the paper mentions as future work, could extend the same variable-length idea to long-context inference where FP-INT GeMMs are no longer the sole bottleneck.","The same format and search could be transferred to other transformer workloads such as vision transformers or encoder-only models, but the sensitivity ranking of the four tensor types would need re-measuring."],"forward_implications":["FP-INT GeMMs, which make up over 90% of operations in sub-4K-token weight-only LLM inference, can be executed as integer dot products with a shared exponent, cutting out per-element exponent alignment.","A user can choose the accuracy-efficiency operating point after training: relaxing the tolerance from 0.1% to 5% raises the reported speedup from 1.73x to 2.74x and energy efficiency from 2.95x to 3.22x for LLaMA-13B.","Because the search reuses the calibration data already used for weight-only quantization and runs in at most 32 iterations, Anda slots into existing post-training deployment pipelines without retraining.","Compared with FIGNA's fixed 14-bit mantissa conversion, Anda cuts bit operations by 1.46x to 2.69x at similar perplexity loss, because different tensors get different mantissa widths.","Anda's bit-plane storage and on-the-fly compressor reduce SRAM and DRAM access energy by roughly 2.2x and 2.0x over FIGNA, so memory, not just arithmetic, shares in the gain."],"supporting_citations":[{"why":"Supplies the main accuracy-preserving BFP conversion baseline and the dedicated FP-INT accelerator that Anda's hardware and BOPs savings are compared against.","marker":"[32]"},{"why":"Baseline accelerator that dynamically converts FP activations to BFP with extended mantissas; the comparison target for energy and area efficiency.","marker":"[42]"},{"why":"Weight-only PTQ baseline (W4A16g128) that provides the starting model, the calibration data, and the accuracy reference for Anda's perplexity comparisons.","marker":"[66]"},{"why":"Post-training weight-only quantization baseline whose calibration workflow and runtime are compared with the Anda precision search.","marker":"[24]"},{"why":"Aggressive 4-bit BFP baseline that requires retraining; used to show the accuracy loss when such a format is applied without retraining.","marker":"[12]"},{"why":"Introduces block floating point with group-shared exponents, the representational idea that Anda extends to variable mantissa lengths.","marker":"[19]"},{"why":"Prior multi-length BFP format with chunk-serial compute; Anda compares against its precision flexibility and bit-serial aligner.","marker":"[85]"},{"why":"Source of the bit-operations (BOPs) metric that the precision search uses to estimate computational cost of each candidate combination.","marker":"[1]"}],"fun_headline_variants":["Adaptive mantissa widths speed LLM inference 2.4x","Per-module mantissas: 2.4x speed, 3.1x less energy","Variable-length activations: 2.4x faster, 3.1x savings","No retraining: adaptive activation bits speed LLM 2.4x","Anda: per-module mantissas deliver 2.4x faster LLM inference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that perplexity on WikiText2, PTB, and C4 is a faithful enough measure of model quality that keeping it within 0.1% or 1% of the baseline guarantees the same tolerance for real downstream tasks.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive mantissa widths speed LLM inference 2.4x","Per-module mantissas: 2.4x speed, 3.1x less energy","Variable-length activations: 2.4x faster, 3.1x savings","No retraining: adaptive activation bits speed LLM 2.4x","Anda: per-module mantissas deliver 2.4x faster LLM inference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002156,"raw_usage":{"total_tokens":8425,"prompt_tokens":1073,"completion_tokens":7352,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":689,"completion_tokens_details":{"reasoning_tokens":7239}},"tokens_in":689,"tokens_out":7352,"duration_ms":45237,"temperature":1.0,"reasoning_tokens":7239,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:40:29.152541+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the exact precision combinations found by the search (e.g., [7,7,6,5] for OPT-125M) on a downstream benchmark such as question answering or reasoning; if task accuracy drops by more than the tolerance while perplexity stays within it, the claimed accuracy-efficiency balance does not transfer. Alternatively, build or simulate the Anda accelerator with the stated HBM2 memory model and verify whether the 2.4x speedup and 3.1x energy efficiency hold at the system level, since the paper's numbers come from a cycle-accurate simulator.","supporting_citations":[{"cited_title":"Figna: Integer unit-based accel- erator design for fp-int gemm preserving numerical accuracy,","cited_arxiv_id":null,"evidence_quote":"Supplies the main accuracy-preserving BFP conversion baseline and the dedicated FP-INT accelerator that Anda's hardware and BOPs savings are compared against."},{"cited_title":"Winning both the accuracy of floating point activation and the simplicity of integer arithmetic,","cited_arxiv_id":null,"evidence_quote":"Baseline accelerator that dynamically converts FP activations to BFP with extended mantissas; the comparison target for energy and area efficiency."},{"cited_title":"Omniquant: Omnidirectionally calibrated quantization for large language models,","cited_arxiv_id":null,"evidence_quote":"Weight-only PTQ baseline (W4A16g128) that provides the starting model, the calibration data, and the accuracy reference for Anda's perplexity comparisons."},{"cited_title":"Optq: Accurate quantization for generative pre-trained transformers,","cited_arxiv_id":null,"evidence_quote":"Post-training weight-only quantization baseline whose calibration workflow and runtime are compared with the Anda precision search."},{"cited_title":"Vs-quant: Per-vector scaled quantization for accurate low-precision neural network inference,","cited_arxiv_id":null,"evidence_quote":"Aggressive 4-bit BFP baseline that requires retraining; used to show the accuracy loss when such a format is applied without retraining."},{"cited_title":"Training dnns with hybrid block floating point,","cited_arxiv_id":null,"evidence_quote":"Introduces block floating point with group-shared exponents, the representational idea that Anda extends to variable mantissa lengths."},{"cited_title":"Fast: Dnn training under variable precision block floating point with stochastic rounding,","cited_arxiv_id":null,"evidence_quote":"Prior multi-length BFP format with chunk-serial compute; Anda compares against its precision flexibility and bit-serial aligner."}],"review_version":1}