{"id":"98d75c64-42b2-462c-8afb-a3972f6ad9d2","arxiv_id":"2501.01144","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"BlockDialect assigns one of 16 FP4 'dialect' formats to each 32-64 element block of weights and activations, achieving near-full-precision accuracy with roughly 4-bit storage and integer-friendly MACs.","lead":"A new quantization scheme, BlockDialect, picks the best of 16 four-bit number formats for every small block of weights and activations in an LLM, cutting accuracy loss versus the standard MXFP4 format. On 7-8B models it lands within a few percentage points of full precision while using low-precision integer arithmetic.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gains (10.78% vs MXFP4, 5.45% below FP16) come from block sizes tuned on the same benchmarks and are reported without seeds, error bars, or public code; the central accuracy claim is not yet independently verifiable, and only prefill-style evaluation is reported.","rationale":"I read BlockDialect as a serious empirical quantization paper: the method is clearly specified, the experiments cover multiple LLM families and both linear and attention paths, the two-stage heuristic is compared against an MSE oracle, and the hardware synthesis gives concrete area/power estimates. The central accuracy claim is plausible and consistently supported across LLaMA2, LLaMA3, Mistral, and OPT. In good faith, the strongest remaining risk is not the magnitude-importance assumption highlighted by the reader. The selection heuristic is directly tested by Table 2: exact MSE selection is barely better, so the heuristic's large-magnitude focus does not lose much. The remaining formatbook-design question (whether varying lower values would help) affects optimality, not the validity of the reported gains. What is load-bearing is the verifiability and scope of the reported numbers: the headline configuration was chosen from ablations on the same benchmark, no variance is reported, no public code is accessible, and the evaluation is prefill-style. If an independent held-out evaluation with a frozen configuration reproduced the margins, the claim would be secure; if not, CONDITIONAL acceptance would be too generous. My assessment therefore keeps the reader's CONDITIONAL verdict but for a different primary reason. Agreement is 'partial' because the reader listed code/seed/selection concerns in the rationale but identified a different weakest assumption.","tokens_in":31298,"tokens_out":40803,"duration_ms":360873,"concrete_test":"Freeze the configuration exactly as stated for the headline (block size 64 with dn/Q/K at 16 for the 'all' scope; also run the untuned BlockDialect-32) and evaluate on a model and task suite not used in any ablation, e.g., Qwen2.5-7B-Instruct or LLaMA3.1-8B, using the same lm-eval-harness version for the seven 0-shot tasks plus GSM8K/MMLU, with five random seeds. Require release of the emulation code with a commit hash so the rerun is exact. If the mean gain over MXFP4-32 at comparable bitwidth falls below ~5% (reported 9.73% for LLaMA3 all-scope) or the mean gap to FP16 exceeds ~7%, the headline is inflated by configuration search; if margins reproduce within ~1%, the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: BlockDialect 'achieves 10.78% (7.48%) accuracy gain ... while being only 5.45% (2.69%) below full precision even when quantizing full-path matrix multiplication.' For that claim to be load-bearing, the reported numbers must be a faithful estimate of the method, not a product of configuration search on the evaluation set. Three features make this insecure. (1) The headline all-scope numbers use the configuration '64 (dn, Q, K: 16)', i.e., block size 64 with 16-blocks on down-projection, query, and key, selected after inspecting the block-size ablation (Table 3) on the same models and the same seven-task suite. No seeds, error bars, or per-task variance are reported, so the 5.45% and 10.78% margins are unbracketed. (2) The 'lower bit usage' comparison is against MXFP4-16 (4.31 bits); against the stronger MXFP4-32 (4.16 bits), the all-scope LLaMA3 gain is 9.73% and BlockDialect-32 uses more bits (4.28), so the headline framing is the most favorable of several defensible comparisons. (3) The emulation code is a private Stanford GitLab URL with no commit hash, so an independent re-run is not currently possible, and the evaluation is prefill-style (lm-eval-harness log-likelihood scoring), leaving decode-phase attention with the residual-FP16 KV cache (Appendix C) and per-token dialect selection untested. I do not find the reader's magnitude-importance assumption the load-bearing element: Table 2 compares the two-stage heuristic against exact MSE selection and the gap is only ~0.6%, so the selection heuristic is directly validated; the untested formatbook lower values would affect optimality, not the truth of the reported gains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BlockDialect, a block-wise mixed-format quantization technique for 4-bit weight-and-activation LLM inference. Instead of applying a single number format per tensor, BlockDialect selects, for each fine-grained block, one dialect from a fixed formatbook of FP4-like representations (DialectFP4, 16 dialects by default). A two-stage online selection mechanism picks the dialect for activation blocks without full MSE computation, and representative values are chosen with 0.5 granularity so that MAC operations reduce to 4-bit unsigned integer arithmetic. Experiments compare against MXFP4, LLM-FP4, and QuaRot on LLaMA-2-7B, LLaMA-3-8B, Mistral-7B, OPT-6.7B, and several smaller models, reporting WikiText2 perplexity and seven-task zero-shot accuracy under two scopes (linear-only and full-path including attention). Hardware synthesis at 45nm and 130nm is used to claim MAC units comparable in area and power to FP4, with small overhead for on-the-fly quantization. The central claim is that full-path 4-bit quantization is only 5.45% (LLaMA3) and 2.69% (LLaMA2) below full precision while outperforming MXFP4 by 10.78% and 7.48%, respectively, with lower effective bit usage.","tokens_in":31498,"tokens_out":5928,"duration_ms":54087,"significance":"If the empirical claims hold, the work is a substantive contribution: it reframes 4-bit quantization as a representation-selection problem rather than a scaling-only problem, and it provides a hardware-friendly integer path for mixed-format activation quantization. The paper has real strengths: wide model coverage (three 7-8B models plus OPT-6.7B, Phi-2.7B, LLaMA3-1B, GPT2-1.5B, MobileLLM-125M), full-path attention quantization in addition to linear layers, a two-stage selection method that is shown in Table 2 to closely match exact MSE selection, and synthesis-based hardware cost estimates. These are useful and falsifiable results. The main weaknesses are empirical verification issues: no seeds or error bars, configuration choices made after inspecting ablations on the same evaluation suite, a private code repository without a commit hash, prefill-only evaluation despite the decode-phase being the practically important regime, and a comparison framing that is favorable to the headline bit-usage claim. These issues do not invalidate the approach, but they do mean the headline numbers are not yet independently verifiable.","major_comments":[{"comment":"The headline all-scope numbers in Table 1 (LLaMA3: 69.00%, 5.45% below FP16; LLaMA2: 68.25%, 2.69% below FP16) are produced by the configuration “64 (dn,Q,K:16)”, which Section 4.2 selects after examining the block-size ablation in Table 3 on the same models and the same seven-task suite. No seeds, error bars, or per-task standard errors are reported, so the 10.78% and 5.45% margins are unbracketed and may reflect selection on the evaluation set. Please report multiple seeds or at least standard errors for the headline numbers, and describe a protocol that separates configuration selection from evaluation.","section":"Section 4.2, Tables 1 and 3"},{"comment":"The implementation footnote points to a private Stanford GitLab repository with no commit hash, so the central empirical claim cannot currently be independently re-run. Moreover, all Table 1 evaluations are prefill-style log-likelihood scores from lm-eval-harness; Appendix C describes a residual-FP16 KV-cache tail and per-token quantization for the decode phase, but the actual decode-phase configuration is never evaluated. The “full-path” claim should either be explicitly scoped to the prefill phase in the abstract and conclusion, or supplemented by decode-phase results (e.g., generative perplexity, throughput, or energy).","section":"Section 4.1, Appendix C"},{"comment":"The “lower bit usage per data” claim in the abstract is baseline-dependent. In the all-scope rows of Table 1, BlockDialect-64 (dn,Q,K:16) uses W:4.25/A:4.21 on LLaMA3, which is lower than MXFP4-16 (4.31 bits) but higher than MXFP4-32 (4.16 bits). Against MXFP4-32 the LLaMA3 accuracy gain is 10.16% rather than 10.78%, and for LLaMA2 the gain is 8.49% with higher bit usage. Please report comparisons against both MXFP4-16 and MXFP4-32 with the exact effective bitwidths for the reported configurations, and qualify the “lower bit usage” statement accordingly.","section":"Abstract and Table 1"},{"comment":"The DialectFP4 formatbook, including the number of dialects and the specific large-magnitude values, is designed from block-level profiling on WikiText2 (Section 3.1, Figures 2 and 7), and Table 1 then reports WikiText2 perplexity as an evaluation metric. For the perplexity numbers this is circular. Please state this limitation explicitly, and either design the formatbook on held-out calibration data or show that the formatbook and the dialect count are not sensitive to the calibration text.","section":"Section 3.1 and Section 4.1"}],"minor_comments":[{"comment":"The column header “A VG.” appears to be a typo for “Avg.”; please correct it in Tables 1, 12, 13, and 14.","section":"Table 1"},{"comment":"The LLM-FP4 baseline description reports search interval (60,2) without stating whether this setting matches the original paper's defaults or how the baselines' own variance was handled; please specify the configuration and report standard deviations for the baselines as well.","section":"Section 4.1"},{"comment":"The SmoothQuant alpha is selected as the value with the lowest perplexity on the same evaluation data, which is a form of calibration on the test metric; this should be acknowledged, and the comparison in Table 11 should state how much of the apparent gain comes from this selection.","section":"Appendix H"},{"comment":"The sentence “Note that BlockDialect’s full-path low-precision matrix multiplication is significantly more efficient during the prefill phase” qualifies the main full-path claim and should appear in the main text near the headline results, rather than only in the appendix.","section":"Appendix C"},{"comment":"Figure 4 lists only magnitude values; since a sign bit is stored separately, the figure caption should state explicitly that the 4-bit index encodes magnitude only and that the sign bit is handled at the MAC level.","section":"Figure 4 and Section 3.3"},{"comment":"The effective-bitwidth calculation assumes a sequence length of 2048; because attention operand bit costs depend on sequence length, please state how the bit-count comparison changes at shorter or longer sequence lengths.","section":"Appendix I"}],"recommendation":"major_revision","confidential_remarks":"The conceptual core of the paper is sound and the experiments are extensive, but the verification standards are the main risk. In a quantization/hardware venue, the headline accuracy comparisons are often reused by later work, so the absence of seeds, error bars, and a public commit hash is a serious issue, not a cosmetic one. I would ask the editor to make code release with a versioned commit a condition of acceptance, and to require the authors to clarify the configuration-selection protocol. The paper is within scope and deserves a major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious look. The core idea is genuinely new: instead of one scaling factor per block, pick a number format (dialect) per block from a fixed 16-entry FP4 formatbook, and do it cheaply on the fly with a two-stage logical selector. The selector is the part that makes it work; Table 2 shows it lands within about 0.6% accuracy of exhaustive MSE-based selection, at a fraction of the hardware cost. That result is the load-bearing evidence, and it holds up.\n\nThe paper also does the right kind of work around the idea: consistent gains over MXFP4 across LLaMA2-7B, LLaMA3-8B, Mistral-7B, OPT-6.7B and a few small models; ablations on block size, dialect count, block shape; and real synthesis numbers showing the dialect-MAC is comparable to FP4 and the selection/quantization logic is cheap. The integer-compatible 0.5-granularity design is a concrete, sensible engineering choice. Credit where due: this is a well-scoped contribution, and the comparison to concurrent NxFP, while qualitative, shows the authors know the surrounding work.\n\nSoft spots, in order of size. First, the headline numbers are not bracketed. No seeds, no error bars, and the configuration used for the headline all-scope results (block size 64 with 16-blocks on down-projection, Q, K) was chosen after looking at the block-size ablation on the same models and same seven-task suite. That is selection on the test set, and it makes the 10.78% gain and 5.45% gap optimistic as point estimates. The effect may not be large, but it is unquantified. Second, the emulation code is on a private Stanford GitLab without a commit hash, so nobody can reproduce the numbers independently. Third, the abstract presents percentage-point differences as percentages ('10.78% accuracy gain' is really 10.78 points), which overstates the relative improvement. Fourth, the evaluation is prefill-style; the paper itself notes in Appendix G that lm-eval processes multiple tokens in parallel. Decode-phase attention, with the residual FP16 KV cache described in Appendix C, is untested. That is a real gap for any energy-efficiency claim about inference.\n\nThe circularity concern about designing DialectFP4 on WikiText2 and then reporting WikiText2 perplexity is real but minor; the headline tasks are held-out, and the design generalizes across model families. I would not make the magnitude-importance assumption the sticking point, because Table 2 directly validates the selection heuristic against exact MSE.\n\nVerdict: this is a solid paper with one unquantified risk (test-set selection) and a few presentation problems. It deserves a serious referee; conditional acceptance with requests for variance, code release, and decode-phase evaluation is the right outcome.","headline":"BlockDialect is a solid, well-engineered mixed-format quantization paper whose headline numbers are probably right but that reports no variance, tunes its block sizes on the same benchmarks it reports, and only tests prefill-style evaluation.","tokens_in":32276,"tokens_out":2573,"would_cite":true,"duration_ms":23303,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Each 32-value block picks its own 4-bit format from a 16-entry formatbook, and BlockDialect stays within 5.45% of full precision on LLaMA3-8B even with every matrix multiplication quantized.","keywords":["BlockDialect","DialectFP4","mixed format quantization","4-bit weight-activation quantization","post-training quantization","full-path LLM inference","MXFP4 comparison","energy-efficient inference"],"falsifier":"Build a mirrored 16-dialect formatbook that instead prioritizes the smallest magnitudes (same $0.5$ granularity, same sharing structure) and rerun the full-path LLaMA3-8B experiment: if the small-priority book matches or beats the large-priority one, the importance premise fails. A finer probe is to instrument the selector and count, per block, how often the two-stage rule picks the dialect with the larger true MSE; a disagreement rate near chance on blocks whose maximum is an isolated outlier would show the heuristic is not tracking what the paper says it tracks.","tokens_in":30891,"feed_emoji":"🧮","tokens_out":12746,"duration_ms":99896,"temperature":0.7,"pith_summary":"BlockDialect is a post-training quantization scheme built on a different question from most of the field: not 'how should each block be scaled down to 4 bits?' but 'which 4-bit number format should each block use?' The paper assembles a formatbook of 16 FP4 variants, called DialectFP4, that differ in their largest representable magnitudes, assigns each 32-value block the variant that best fits its data, and reports that this recovers nearly all the accuracy lost by the uniform MXFP4 format at the same or lower bit cost. On LLaMA3-8B and LLaMA2-7B, the method lands 5.45% and 2.69% below full precision even when every matrix multiplication, including the attention-path ones, is quantized. The reason to care is energy: all representable values are multiples of $0.5$, so the data path runs on 4-bit integer multiply-accumulate hardware whose synthesized area and power match plain FP4 units.","feed_headline":"Per-block FP4 dialects keep 4-bit LLMs within 5.45%","feed_subtitle":"BlockDialect assigns each block its own FP4 format, beating MXFP4 by 10.78% on LLaMA3-8B with integer arithmetic.","key_machinery":"The load-bearing object is DialectFP4, a formatbook of 16 FP4 variants built to match three profiled facts about blocks in LLMs: block maxima spread fairly evenly across the representable range, many blocks deviate from the matrix-wide distribution near their upper edge, and power-of-two shared exponents waste or underestimate range. Each dialect is a set of eight magnitude values with $0.5$ granularity; it keeps the six smallest values of base FP4 E2M1 and varies only its two largest values, so 16 dialects cover different dynamic ranges and different large-magnitude shapes while sharing most of their code space. The companion mechanism is a two-stage online selector: stage one truncates the block maximum to pick the pair of dialects sharing that maximum, and stage two counts, using AND/OR logic on 5-bit truncations, how many block elements fall into each dialect's 'beneficial range' — the interval where the differing large value actually reduces quantization error. Because every representable magnitude is a multiple of $0.5$, quantized operands become 4-bit unsigned integers before multiplication, and the MAC unit, shifter, and accumulator stay in integer arithmetic until partial sums are promoted to FP16.","core_discovery":"Uniform 4-bit formats lose accuracy because one shared format cannot represent the idiosyncratic spread of values inside each small block of an LLM tensor. The paper's central claim is that the residual error is mostly a representation problem rather than a scaling problem: if each block may choose among 16 FP4 'dialects' that differ only in how they cover the larger magnitudes, block-level distributions that standard FP4 handles poorly become representable. Concretely, BlockDialect with DialectFP4 beats MXFP4 by 10.78% average zero-shot accuracy on LLaMA3-8B (7.48% on LLaMA2-7B) at a lower effective bitwidth, stays within 1.76% and 1.20% of full precision when only linear layers are quantized, and within 5.45% and 2.69% for full-path quantization that includes attention activation-activation multiplies. The paper further claims a practical online two-stage dialect-selection heuristic that matches exhaustive MSE-based selection in accuracy while costing a tiny fraction of its hardware, and integer-only MACs whose area and power are essentially those of FP4 units.","pith_inferences":["A natural untested combination is BlockDialect with rotation-based outlier suppression, the Hadamard-transform family: the paper tests only SmoothQuant and notes the two axes are not fully orthogonal, but rotations attack the same outlier problem from a complementary direction that could compound with per-block format choice.","Because 15 of 16 dialects share six of eight values, the dialect tag carries almost no dequantization cost; carrying tags in registers for streaming blocks could push the realized per-data overhead below the paper's 9-bit-per-block accounting.","The chosen-dialect histogram per layer could double as a free distribution-drift monitor: if deployment-time inputs shift dialect selections away from calibration-time profiles, that is a signal the hardware already produces without extra instrumentation.","Extending the formatbook idea to other precisions (6-bit dialects, or per-block choice spanning FP4 and INT4) is the obvious next axis, and the paper's block-profiling procedure already gives the recipe for deciding which variants to include."],"forward_implications":["Full-path 4-bit weight-and-activation inference becomes accurate enough on 7-8B models that high-precision queries and attention-score paths can be dropped, with average zero-shot accuracy losses of 2.69-5.45%.","Effective bitwidth stays at or near MXFP4's (BlockDialect-32 uses 4.28 bits versus MXFP4-16's 4.31) while average zero-shot accuracy rises by 10.78 points on LLaMA3-8B, so a 4-bit dialect tag per 32-value block is cheap for the accuracy it buys.","Online activation format selection is implementable at inference speed: the two-stage logic meets timing at 250 MHz in 130 nm synthesis, while the MSE-based alternative fails at 100 MHz and uses roughly 10x the power and area.","The gains reproduce across LLaMA2-7B, LLaMA3-8B, Mistral-7B, OPT-6.7B, and smaller models, so the representational mismatch the formatbook targets looks like a general property of LLM blocks rather than a single-model artifact.","Combining BlockDialect with SmoothQuant-style scaling migration adds a small but consistent accuracy improvement, indicating representation choice and scaling choice are at least partially independent axes that can be tuned together."],"supporting_citations":[{"why":"Defines the Microscaling (MX) formats; its MXFP4 is the primary baseline BlockDialect must beat and the source of the hardware-supported fine-grained scaling idea.","marker":"(Rouhani et al., 2023b)"},{"why":"SmoothQuant is the main representative of the 'how to scale' approach BlockDialect contrasts with, and the method used in the combination experiments.","marker":"(Xiao et al., 2023)"},{"why":"QuaRot supplies the strongest full-path W4A4KV4 baseline, which keeps queries and attention scores in FP16 and which BlockDialect outperforms.","marker":"(Ashkboos et al., 2024)"},{"why":"LLM-FP4 is the matrix-wise mixed-format baseline whose coarse granularity motivates BlockDialect's per-block format choice.","marker":"(Liu et al., 2023)"},{"why":"NxFP is the concurrent work with similar observations about MX format limits; the paper positions its block-level dialects against NxFP's weight-only, MSE-based selection in Appendix D.","marker":"(Lo et al., 2024)"},{"why":"Source of the outlier observations and of the claim that large-magnitude elements are more important, which the formatbook design leans on.","marker":"(Dettmers et al., 2022)"},{"why":"AWQ independently supports the large-magnitude importance premise the dialect priority ordering is built on.","marker":"(Lin et al., 2024)"},{"why":"VS-Quant establishes the per-vector hardware-supported scaling approach that precedes and motivates fine-grained block-wise formats like MXFP4.","marker":"(Dai et al., 2021)"}],"fun_headline_variants":["Per-block FP4 dialects beat MXFP4 by 10.78%","16 FP4 dialects per block keep LLMs within 5.45%","BlockDialect: choose an FP4 dialect per block for accuracy","DialectFP4: per-block formatbook boosts 4-bit LLM accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole design rests on the assumption that a block's larger-magnitude values are more important than its smaller ones, so the formatbook spends its scarce 4-bit entries on the large end and both selection stages reward capturing the largest values accurately; if small-magnitude elements mattered equally, the selector could pick the wrong dialect and BlockDialect's gains over MXFP4 would shrink.","fun_headline_variants_meta":{"raw":{"variants":["Per-block FP4 dialects beat MXFP4 by 10.78%","16 FP4 dialects per block keep LLMs within 5.45%","BlockDialect: choose an FP4 dialect per block for accuracy","DialectFP4: per-block formatbook boosts 4-bit LLM accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000789,"raw_usage":{"total_tokens":3524,"prompt_tokens":1034,"completion_tokens":2490,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":650,"completion_tokens_details":{"reasoning_tokens":2408}},"tokens_in":650,"tokens_out":2490,"duration_ms":18092,"temperature":1.0,"reasoning_tokens":2408,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:34:48.838342+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a mirrored 16-dialect formatbook that instead prioritizes the smallest magnitudes (same $0.5$ granularity, same sharing structure) and rerun the full-path LLaMA3-8B experiment: if the small-priority book matches or beats the large-priority one, the importance premise fails. A finer probe is to instrument the selector and count, per block, how often the two-stage rule picks the dialect with the larger true MSE; a disagreement rate near chance on blocks whose maximum is an isolated outlier would show the heuristic is not tracking what the paper says it tracks.","supporting_citations":[{"cited_title":"These approaches often dequantize data to FP16 before performing multiplications, which limits computational efficiency","cited_arxiv_id":null,"evidence_quote":"QuaRot supplies the strongest full-path W4A4KV4 baseline, which keeps queries and attention scores in FP16 and which BlockDialect outperforms."}],"review_version":1}