{"id":"afede9bd-d375-4e16-b435-63d3340c10ce","arxiv_id":"2507.23035","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"OASIS enables efficient LLM inference with non-uniform 4-bit weights and activations via precomputed Cartesian product lookup tables and a parallel outlier-compensation branch, at a reported 1.94-2.05% average accuracy drop.","lead":"This paper proposes OASIS, a hardware design that runs large language models with both weights and activations stored in compact non-uniform 4-bit formats, using lookup tables so no dequantization is needed during matrix multiplication. It reports accuracy within about 2 percentage points of full-precision models on standard benchmarks while claiming 3x speedup over a prior lookup-table accelerator in simulation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-output-channel weight scales and per-token activation scales (Sec. III-A) are never applied in the LUT-GEMM pipeline (Sec. III-B, IV-A); a single 256-entry Cartesian-product LUT of raw centroid products cannot produce correct GEMM outputs unless scaling is folded in, which is not described.","rationale":"The reader's weakest assumption was that offline-learned activation centroids generalize to arbitrary online activations. The paper does provide some supporting evidence (Fig. 5 shows centroid RMSE of 0.01 for one layer; Fig. 17 shows robustness across C4/PTB calibration datasets and sample counts for LLaMA-3-8B). That concern is real but partially mitigated. The scaling-factor omission I identify is a more direct threat to the central claim: it questions whether the described LUT-GEMM computation is correct at all, independent of empirical generalization. If the LUT truly stores raw centroid products, per-output-channel and per-token scales must be applied somewhere, but the algorithm section and the hardware pipeline omit any such step. This is an internal consistency gap that can be settled with a small numerical test, and it affects the headline claims of LUT size reduction and dequantization-free execution. I therefore keep the reader's CONDITIONAL verdict: the idea is promising but the presentation leaves a load-bearing implementation detail unresolved. The proposed test would determine whether the issue is merely a missing description (fixable) or a fundamental inconsistency in the design (required rework of the hardware flow).","tokens_in":25326,"tokens_out":12462,"duration_ms":151518,"concrete_test":"Implement a numerical check on a real (or random) layer: quantize weights with shared 4-bit centroids plus per-output-channel scale factors, and activations with shared 4-bit centroids plus a per-token scale, exactly as described in Sec. III-A. Compute the GEMM two ways: (a) using a single 256-entry Cartesian-product LUT of raw centroid products with no scaling, as Fig. 6 and Fig. 8 appear to describe; (b) using exact FP16 GEMM on dequantized values. Compute the relative error between the two outputs. If the error is large (e.g., >5%), the missing scaling is confirmed. If the paper intends a post-reduction scaling step, repeat the check with that step included and verify the outputs match within numerical precision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of a 64x smaller precomputed Cartesian Product LUT and dequantization-free GEMM depends on the LUT storing true multiplication results between quantized weights and activations. However, Sec. III-A specifies per-output-channel weight scaling factors and per-token activation scaling factors, while Sec. III-B defines the LUT as the Cartesian product of shared weight and activation centroids. If each output channel has a distinct weight scale and each token a distinct activation scale, the actual product is scale_w(output_channel) * scale_a(token) * centroid_product, which varies across output channels and tokens. The hardware pipeline in Fig. 8 (steps 1-5) and the example in Fig. 6 contain no scaling operation. With the literal interpretation, the LUT-GEMM output is missing these scale factors, producing incorrect results beyond quantization noise. If instead the LUT is intended to contain scaled products, it would need to be replicated per output channel (and possibly per token), destroying the claimed 64x LUT size reduction and the offline-precomputation advantage. The paper never explains where these scale factors are applied, making this a load-bearing gap in the description of the core computation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"OASIS proposes a lookup-table (LUT) based GEMM scheme for non-uniformly quantized LLM inference, combining a precomputed Cartesian-product LUT of weight and activation centroids with a parallel outlier-compensation branch. The main branch avoids dequantization by counting concatenated weight/activation index pairs and reducing to a weighted sum of LUT entries; the outlier branch detects top-k activation outliers and adds their residuals in FP16. The paper also contributes Orizuru, a tree-based top-k detector, and an accelerator architecture with 16 PE lines, index counters, clustering units, and a 256-entry LUT. Evaluations report roughly 2% average zero-shot accuracy drop versus FP16 and 3.00x throughput / 1.44x energy efficiency over the FIGLUT accelerator in simulation.","tokens_in":25592,"tokens_out":7832,"duration_ms":92627,"significance":"If the described computation is correct, this is a significant algorithm-architecture co-design result. The core mathematical decomposition—look-ahead GEMM on quantized activations plus residual error compensation for outliers—is exact by construction, and the claimed 256-entry Cartesian-product LUT is a genuine improvement over the group-size-dependent inner-product LUTs of prior WOQ LUT-GEMM designs. The LUT-size and FLOP arithmetic in Table I and Fig. 16 is internally consistent, and the evaluation is broad, covering 11 model configurations, perplexity, and six zero-shot tasks. The Orizuru detector's comparison count of 1.5N + 2k log2(N) is low and the hardware study uses cycle-accurate simulation with 28 nm synthesis-based area/power numbers. These strengths make the work worth serious consideration, provided the central scaling-factor pipeline gap and the inconsistent accuracy reporting are resolved.","major_comments":[{"comment":"Section III-A specifies per-output-channel weight scaling factors and per-token activation scaling factors, but Section III-B and the hardware pipeline in Fig. 8 compute the LUT-GEMM result as a weighted sum of raw Cartesian products of shared centroids, with no step that applies those scale factors. For a token t and output channel o, the true quantized product is s_a(t) * s_w(o) * C_w[idx_w] * C_a[idx_a]; the described LUT-GEMM computes only C_w * C_a, so the outputs are missing the scale product. If the scales are meant to be applied after the LUT reduction, that step is never described in the five-step pipeline of Fig. 8 or in the example of Fig. 6. If instead the LUT is meant to store scaled products, it would have to be replicated per output channel and, because s_a(t) varies per token, would depend on runtime activations, contradicting the claimed offline precomputation and 256-entry LUT size. This is a load-bearing gap in the description of the core computation and must be clarified or corrected.","section":"Section III-A, Section III-B, Fig. 6, Fig. 8"},{"comment":"The reported accuracy improvement is inconsistent across the front matter and the evaluation. The abstract states an average accuracy drop of 1.98% and a 5.18% lower drop than Atom; the Introduction states 1.94% and 6.34%; Section V-B states a 2.05% average drop at W4A4 and 5.90% at W4A3; and the zero-shot averages in Table IV for LLaMA-2-7B, LLaMA-3-8B, and Mistral at W4A4 give (1.94 + 3.03 + 1.19)/3 = 2.05%. Please reconcile these numbers and state precisely the metric and the model set over which the headline accuracy drop is computed.","section":"Abstract, Section I, Section V-B, Table IV"},{"comment":"The hardware speedup claim against FIGLUT is not precision-matched: Fig. 11 compares OASIS-A4/A3 (W4A4/W4A3) against FIGLUT evaluated at W4A16. A substantial portion of the reported 3.00x throughput and 1.44x energy improvement may therefore be due to the reduced activation precision rather than to the LUT-based architecture itself. Please provide a W4A4-capable FIGLUT baseline, or an ablation that separates the precision contribution from the architectural contribution, so that the 'over FIGLUT' claim is not overstated.","section":"Section V-C, Fig. 11"}],"minor_comments":[{"comment":"Fig. 3 shows low similarity between online and offline outlier thresholds (RMSE 0.32-0.38), yet the static variant OASIS-S in Table III performs only slightly worse than dynamic OASIS. Please explain this apparent tension, since it bears on the value of the dynamic detector.","section":"Section III-A, Fig. 3, Table III"},{"comment":"The LUT is reported as 2 KB per chip, but a 256-entry FP16 Cartesian-product table occupies 512 bytes; please clarify what additional content (e.g., codebooks, tags) is stored in the LUT.","section":"Section V-A, Table II"},{"comment":"The Orizuru comparison count is stated as 1.5N + 2k log2(N), but the max-tree initialization requires N-1 comparisons and the min tree reuses the last level, giving 1.5N-2 comparisons before popping; the stated formula slightly overcounts.","section":"Section IV-D"},{"comment":"The LUT-size formula in Table I is hard to parse in the current formatting; please typeset it unambiguously to make the 64x reduction argument clear.","section":"Table I"},{"comment":"The '1024x higher computational parallelism' claim is configuration-specific (K=4096 versus group size 4); please state this dependence explicitly wherever the number is quoted.","section":"Section II-B"}],"recommendation":"major_revision","confidential_remarks":"The scaling-factor gap is the key technical risk. If the per-output-channel and per-token scales are not applied either inside the LUT or in a post-reduction step, the core LUT-GEMM as described is numerically incorrect. The inconsistency in the headline accuracy numbers (1.94%, 1.98%, 2.05%) also needs editorial reconciliation. These issues are fixable within the manuscript's scope, but they are central enough that I cannot recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is the Cartesian-product LUT for dual-side non-uniform quantization: because both operand codebooks are known offline, the LUT can be precomputed and is independent of reduction length. That is a substantive step past FIGLUT, LUT Tensor Core, and LUT-GEMM, which all need streaming-activation-dependent inner-product LUTs. The 64x LUT size reduction and 1024x parallelism follow cleanly from that design, and the look-ahead error compensation and Orizuru top-k engine are sensible additions. The accuracy evaluation is broad: eleven models across next-word prediction and zero-shot tasks, with OASIS consistently beating Atom and QuaRot by several points at W4A4/W4A3. Given the results, the algorithm-side story is convincing.\n\nThe problem is on the hardware side, and it is load-bearing. Section III-A says weights have per-output-channel scaling factors and activations have per-token scaling factors. Section III-B defines the LUT as the Cartesian product of the shared centroids. The product of two quantized values is not the centroid product; it is scale_w * scale_a * centroid_w * centroid_a. The paper never says where those scale factors enter the LUT-GEMM pipeline. The architecture description in Section IV-A and the example in Figure 6 show no scaling operation anywhere. With a single 256-entry LUT, you cannot store scaled products for every output channel and token. Any fix that replicates the LUT per channel would shatter the 64x size reduction and the offline-precomputation claim. This is not a nitpick about implementation details; it is a missing explanation of how the claimed computation is even mathematically correct.\n\nThere are also smaller issues. The accuracy-drop numbers are inconsistent across abstract, introduction, and evaluation (1.98%, 1.94%, 2.05%). The hardware results rely on an unreleased simulator, and the resource balance is hand-tuned to the 1% outlier sweet spot, so the 3x speedup and 1.44x energy numbers are plausible but not independently checkable. The offline-centroid generalization test covers only one layer, though the across-model results do mitigate that concern.\n\nWho gets value from this paper? Anyone working on hardware accelerators for low-precision LLM inference, especially the LUT-based GEMM line of work. The core idea deserves serious referee time, but the scaling gap must be resolved before the architecture can be assessed. I would send this to review with a clear request for the authors to either explain where scale factors are applied or release a functional correctness check for the LUT-GEMM data path. As written, the central hardware claim is unverified.","headline":"OASIS has a genuinely new LUT-based GEMM idea for non-uniform W4A4, but a load-bearing gap in how per-channel/per-token scales are handled makes the core hardware claim unverifiable as written.","tokens_in":26179,"tokens_out":3513,"would_cite":false,"duration_ms":40014,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OASIS claims non-uniform 4-bit weights and activations can be multiplied directly via precomputed Cartesian-product LUTs, eliminating dequantization, with ~2% accuracy loss and 3.00x speedup.","keywords":["large language models","non-uniform quantization","learned codebook","lookup table GEMM","W4A4 quantization","activation outliers","hardware acceleration","low-bit inference"],"falsifier":"Run the W4A4 pipeline as specified, with 16 C4 calibration samples and the top and bottom 0.5% of activations kept in FP16, on a held-out corpus from a domain far from C4, such as code or biomedical text, and compare its accuracy to the FP16 baseline; if the gap systematically exceeds the reported ~2% average, the offline-centroid transferability assumption is broken. On the hardware side, an implementation that cannot reproduce the paper's simulated cycle counts for a 1-4096-4096 GEMM at 500 MHz would falsify the claimed 3.00x speedup over FIGLUT.","tokens_in":25127,"feed_emoji":"⚡","tokens_out":11337,"duration_ms":119110,"temperature":0.7,"pith_summary":"The paper tries to establish that non-uniform weight-and-activation quantization, the kind where 4-bit indices point into learned lists of representable values, does not have to cost accuracy or require dequantizing back to high precision before every matrix multiplication. The proposed scheme, OASIS, stores all products of the weight and activation codebook values in a lookup table, so a general matrix multiplication (GEMM) between quantized operands becomes a count-and-weight operation, with a parallel branch correcting the few activation outliers that would otherwise cause large errors. If the paper is right, the standard trade-off between integer weight-and-activation quantization, which is fast but lossy, and non-uniform quantization, which is accurate but slow, is broken, making 4-bit and even 3-bit activation inference practical. The paper reports an average accuracy drop of about 2% versus FP16 at W4A4, and, in simulation, a 3.00x throughput gain and 1.44x energy efficiency gain over a prior LUT-based accelerator.","feed_headline":"Lookup-table GEMM keeps 4-bit LLMs near FP16 accuracy","feed_subtitle":"Precomputed Cartesian-product LUTs remove dequantization and hide outlier detection, reaching a simulated 3.00x speedup.","key_machinery":"The load-bearing object is the Cartesian Product LUT: a precomputed table of $2^{n_W+n_A}$ values, one for every pair of weight-centroid and activation-centroid. Its size depends only on bit widths, not on GEMM reduction length $K$, so an entire $K$-length inner product can be replaced by a histogram of concatenated indices and a weighted sum of LUT entries; this is what makes the claimed 64x LUT-size reduction and 1024x parallelism increase possible. The second mechanism is look-ahead outlier compensation, which splits computation into a main branch that quantizes every activation and a parallel branch that fetches the top and bottom 0.5% activation outliers, computes their residuals in FP16, multiplies each residual by the corresponding dequantized weight row, and adds the correction, hiding detection latency. The third is Orizuru, a top-$k$ detection engine built from two complete binary trees with shared leaves that extracts the $k$ largest and $k$ smallest elements of an $N$-value activation token in $1.5N + 2k \\log_2 N$ comparisons, several times fewer than the comparison count of a prior top-$k$ engine.","core_discovery":"On its own terms, OASIS claims to execute the GEMM between learned-codebook-quantized weights and activations with no dequantization anywhere in the main path. During offline preparation it builds a Cartesian Product LUT containing every product $w_i a_j$ of a weight centroid and an activation centroid; with $n_W = n_A = 4$ this is only $2^{8} = 256$ entries. At runtime, the accelerator concatenates the 4-bit weight index and 4-bit activation index for each element, counts the occurrences of each concatenated index along the reduction dimension $K$, and forms the output as a weighted sum of the corresponding LUT entries, so the number of additions per output channel falls from $K$ to $2^{n_W+n_A}$. Because both codebooks are learned offline, the LUT is independent of both streaming activations and reduction length, which removes on-the-fly LUT generation and permits much larger compute granularity than prior weight-only LUT GEMMs. To keep accuracy under aggressive activation quantization, OASIS marks the top and bottom 0.5% of each token's activations as outliers, preserves them in FP16, and runs a concurrent error-compensation branch whose residual products with the dequantized weight rows are added to the main branch; the paper states this yields mathematically identical results to a conventional high-precision outlier path without putting outlier detection on the critical path. Across OPT, LLaMA, LLaMA-2/3, and Mistral models, the reported W4A4 accuracy drop relative to FP16 is around 1.94-2.05% on average.","pith_inferences":["A natural extension, not made by the paper, is to apply the same Cartesian-product LUT trick to other learned-codebook tensors, such as KV caches, embedding tables, or intermediate activations, wherever both the codebook and the value distribution are known ahead of time; each would become a precomputable table rather than a dequantization path.","The reported accuracy gap between OASIS-S, which reuses offline outlier thresholds, and OASIS, which detects outliers dynamically, suggests dynamic detection will matter more on inputs that drift from the calibration corpus; a testable prediction is that OASIS-S degrades faster than OASIS on out-of-distribution text.","If Orizuru's comparison count holds at scale, top-$k$ outlier selection is cheap enough to become a standard per-token hardware primitive that could benefit quantization schemes beyond this specific accelerator; the paper does not claim this generalization."],"forward_implications":["W4A4 and W4A3 learned-codebook quantization become directly executable, so LLM inference can keep non-uniform quantization's accuracy without paying the FP16 dequantization cost that dominates weight-only quantization.","Dynamic outlier detection, which prior work found more accurate than static outlier channels, can be used without lengthening the GEMM critical path.","Because the Cartesian-product LUT size does not grow with model width, the throughput advantage should become larger on very wide GEMMs, such as those in 70B-class models, rather than evaporating at scale.","In the low-batch decode regime where edge LLM serving operates, the architecture is claimed to sustain 3.00x throughput and 1.44x energy efficiency over the FIGLUT accelerator in cycle-accurate simulation."],"supporting_citations":[{"why":"FIGLUT, the prior LUT-based accelerator whose inner-product LUT is the main comparison baseline for OASIS's 3.00x speedup, 1.44x energy gain, 64x LUT-size reduction, and higher parallelism.","marker":"[42]"},{"why":"LUT Tensor Core, the existing WOQ LUT-GEMM design that motivates the group-size-4 inner-product LUT and against which OASIS compares LUT size and reduction FLOPs.","marker":"[37]"},{"why":"LUT-GEMM, another WOQ LUT-GEMM baseline used for LUT-size and reduction-FLOP comparisons.","marker":"[43]"},{"why":"Atom, the integer W4A4 baseline whose average accuracy OASIS claims to beat by 6.34 percentage points and whose group-wise outlier handling is a reference point.","marker":"[62]"},{"why":"QuaRot, the INT-WAQ rotation method used as an accuracy baseline and as the W4A4 A100 GEMM kernel baseline for throughput comparisons.","marker":"[3]"},{"why":"The prior dynamic outlier detection approach whose workflow OASIS's look-ahead scheme removes from the critical path; it is the comparison point for the claim that outlier detection can be hidden.","marker":"[19]"},{"why":"K-Means, the learned-codebook quantization procedure OASIS adopts; it supplies the centroid-index representation the LUT is built on.","marker":"[34]"},{"why":"The tournament-tree sorting technique that Orizuru's two-tree shared-leaf top-k engine builds on.","marker":"[49]"},{"why":"C4, the calibration corpus used offline to learn the activation centroids and outlier thresholds whose transferability carries the accuracy claim.","marker":"[9]"},{"why":"DnnWeaver, the cycle-accurate simulator OASIS modifies to obtain its hardware performance figures.","marker":"[48]"}],"fun_headline_variants":["Outlier-aware LUT GEMM cuts dequantization, boosts LLM inference","64x smaller LUTs speed LLM GEMM with 1.98% accuracy drop","No-dequant LUT GEMM: 3x faster LLM inference at 4-bit","OASIS: LUT GEMM with outlier compensation hits 3x speedup","Dual-side quantized LUT GEMM keeps accuracy, accelerates inference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy result rests on activation centroids learned offline from only 16 C4 samples continuing to match the activations encountered at inference time, so the reported ~2% drop may not hold if real-world input distributions shift.","fun_headline_variants_meta":{"raw":{"variants":["Outlier-aware LUT GEMM cuts dequantization, boosts LLM inference","64x smaller LUTs speed LLM GEMM with 1.98% accuracy drop","No-dequant LUT GEMM: 3x faster LLM inference at 4-bit","OASIS: LUT GEMM with outlier compensation hits 3x speedup","Dual-side quantized LUT GEMM keeps accuracy, accelerates inference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000469,"raw_usage":{"total_tokens":2463,"prompt_tokens":1201,"completion_tokens":1262,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":817,"completion_tokens_details":{"reasoning_tokens":1151}},"tokens_in":817,"tokens_out":1262,"duration_ms":10836,"temperature":1.0,"reasoning_tokens":1151,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:07:16.799263+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the W4A4 pipeline as specified, with 16 C4 calibration samples and the top and bottom 0.5% of activations kept in FP16, on a held-out corpus from a domain far from C4, such as code or biomedical text, and compare its accuracy to the FP16 baseline; if the gap systematically exceeds the reported ~2% average, the offline-centroid transferability assumption is broken. On the hardware side, an implementation that cannot reproduce the paper's simulated cycle counts for a 1-4096-4096 GEMM at 500 MHz would falsify the claimed 3.00x speedup over FIGLUT.","supporting_citations":[{"cited_title":"Some methods for classification and analysis of mul- tivariate observations,","cited_arxiv_id":null,"evidence_quote":"K-Means, the learned-codebook quantization procedure OASIS adopts; it supplies the centroid-index representation the LUT is built on."},{"cited_title":"Using tournament trees to sort,","cited_arxiv_id":null,"evidence_quote":"The tournament-tree sorting technique that Orizuru's two-tree shared-leaf top-k engine builds on."},{"cited_title":"From high-level deep neural models to fpgas,","cited_arxiv_id":null,"evidence_quote":"DnnWeaver, the cycle-accurate simulator OASIS modifies to obtain its hardware performance figures."}],"review_version":1}