{"id":"1703c4fc-f055-4547-933c-b60fe278b048","arxiv_id":"2411.09909","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AMXFP4 is an asymmetric 4-bit microscaling format that improves calibration-free LLM inference accuracy over MXFP4 and rotation-based methods.","lead":"Researchers propose AMXFP4, a 4-bit number format that gives positive and negative values separate shared scales, letting language models run without the lengthy calibration step other fast-4-bit methods need. Tests show it beats the standard MXFP4 format on question answering and long-context tasks while adding about 10% to a single hardware multiply unit.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The accuracy and hardware claims rest on two disconnected AMXFP4 implementations with no bit-exact check, and Table 9's numbers appear inconsistent with the claimed 10% overhead.","rationale":"The reader's weakest_assumption identifies the same gap I consider most load-bearing: the accuracy emulator and the synthesized hardware are never shown to agree bit-exactly, and the accuracy improvements therefore cannot be attached to the hardware whose cost is claimed. I agree with that reading and add a concrete observation that strengthens it: Table 9's displayed numbers imply AMXFP4 has lower area-memory, power-area, and power-area-memory products than MXFP4, which contradicts the sentence that AMXFP4 adds about 10% overhead. Either the row labels are swapped, the metric is not what the text says, or the '10% overhead' claim is unsupported by the table as printed. This is an independent, checkable reporting inconsistency in the same hardware section. I did not find a fundamental flaw in the format design itself: the asymmetry analysis in Sections 3-4 is plausible, the empirical gains over MXFP4 are consistent across many benchmarks, and the manuscript includes code and extensive ablations. The reason the verdict stays CONDITIONAL rather than ACCEPT is precisely that the two halves of the central claim—accuracy and hardware cost—are not yet connected, and one of the two hardware numbers appears self-contradictory. A bit-exact co-simulation test and a corrected Table 9 would settle both issues; until then, adoption as reported is premature.","tokens_in":27889,"tokens_out":10343,"duration_ms":103917,"concrete_test":"Build an RTL testbench around the synthesized AMXFP4 MAC from Section 5.5, then feed it representative LLaMA2-7B layer-5 QKV activation and weight groups identical to those used in Table 1. Compare every MAC output bit-for-bit with the software quantize_mx_op path (Algorithm 1 plus Eq. 1), covering positive-only, negative-only, mixed-sign, zero, and max-magnitude/clamping cases. Require exact equality, or quantify the first divergent case and rerun the Table 1 and Table 3 evaluations with the hardware-accurate emulation. Separately, re-report Table 9 from the synthesis logs with a single baseline and state whether AMXFP4 is actually larger or smaller than MXFP4; if the claimed 10% overhead is real, the corrected table should show it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim couples accuracy with cost: AMXFP4 must both deliver the reported 4-bit inference accuracy and cost only ~10% more than MXFP4 in hardware. The accuracy results in Tables 1, 3, and 10 are produced by the software quantize_mx_op emulator (Appendix B.3, Algorithm 1 extended with the asymmetric FP8 E5M2 scales of Eq. 1), while the 10% overhead is measured on a separately synthesized custom MAC (Section 5.5). No experiment feeds the same tensors through both implementations and compares outputs. The format's defining operation is sign-dependent selection between positive and negative shared scales, plus exponent-floor/round and clamp decisions in Algorithm 1; a divergence in any of these—different rounding of the FP4 element, different treatment of zero, or a different mixed-sign scale-substitution rule—would propagate directly into the quantized matmul results and change the reported accuracies. Moreover, Table 9 itself is internally inconsistent with the text: AMXFP4 is shown as 8.32x vs MXFP4's 9.23x on 'Area-Memory' (and lower on the other two metrics), which implies AMXFP4 is about 10% cheaper, not 10% more expensive. As written, the hardware cost claim is not supported by the displayed numbers; the rows or units need correction. This is a validation and reporting gap, not a flaw in the format concept.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AMXFP4, a 4-bit asymmetric microscaling floating-point format for LLM inference. The format keeps FP4 (E2M1) elements but uses separate FP8 E5M2 shared scales for positive and negative values, with a sign-dependent scale-selection rule during multiplication. The authors argue that microscaling suppresses activation outliers but increases group-wise asymmetry, and that the asymmetric shared scale addresses this without calibration. They evaluate AMXFP4 on Wikitext-2 perplexity, MT-Bench, visual question answering, LongBench-E, MMLU, CSQA, and attention-only settings, reporting consistent improvements over MXFP4 and competitive or better accuracy than rotation-based methods such as QuaRot and SpinQuant. They also implement a custom AMXFP4 MAC unit and claim roughly 10% hardware overhead over an MX-compatible MAC. The paper includes ablations on shared-scale format, group size, rotation interaction, QAT, 3-bit extension, pruning, and a 70B model.","tokens_in":28151,"tokens_out":3868,"duration_ms":38521,"significance":"If the claims hold, AMXFP4 is a practically useful format: it enables calibration-free direct-cast 4-bit inference with fully quantized attention, which is a gap relative to rotation-based methods that require calibration and leave softmax outputs in FP16. The paper is creditable for releasing code, evaluating across a broad set of tasks and model families (including a 70B model and an encoder-decoder model), and for providing extensive ablations, including the interaction of rotation with microscaling and the choice of shared-scale encoding. The central accuracy comparisons (Tables 1, 3, 4, 10, 12) support the qualitative claim that asymmetric shared scales improve over symmetric MXFP4 and often match or exceed rotation-based methods. However, the hardware-cost claim is currently not supported by the reported numbers, and the accuracy results come from a software emulator that is not shown to be bit-exact with the synthesized hardware. These gaps are load-bearing for the paper's central accuracy-plus-cost claim.","major_comments":[{"comment":"The hardware-cost claim is not supported by the displayed numbers: Table 9 reports AMXFP4 at 8.32x versus MXFP4 at 9.23x on Area-Memory, and AMXFP4 is also lower on Power-Area and Power-Area-Memory. This implies AMXFP4 is roughly 10% cheaper, not 10% more expensive, as the text states in Section 4.3 and Section 5.5 (\"adds only 10% overhead\"). Please correct the rows, units, or the baseline for the overhead statement, and re-run the synthesis analysis if necessary, because the claimed cost is a central part of the contribution.","section":"§5.5, Table 9"},{"comment":"The accuracy results are produced by a software emulation path (quantize_mx_op with fp4_e2m1_asym and scale_mode=152 in Appendix B.3), while the hardware cost is measured on a separately synthesized AMXFP4 MAC unit in Section 5.5. The paper provides no bit-exact comparison between these two implementations for the same tensors, including the sign-dependent selection between the positive and negative FP8 E5M2 shared scales, the exponent-rounding rule, and the clamping behavior. Please add such a check, or characterize the numerical divergence if the implementations are not bit-identical, so that the reported accuracy improvements can be attributed to the hardware design whose cost is claimed.","section":"§5.5 vs. Appendix B.3"},{"comment":"The abstract's quantitative claims exceed what the tables show. \"Outperforms MXFP4 by 3% on VQA\" does not match Table 3, where the gains are 1.25 points on VQA-T, 2.72 on DocVQA, 0.50 on OCRBench, and 3.28 on ChartQA; there is no single 3% improvement across the VQA benchmarks. Similarly, \"exceeds rotation-based methods by 1.6% on CSQA\" is not directly shown: Table 10 reports ARC-Challenge and WinoGrande, and Table 4 compares against NVFP4 rather than rotation-based methods. Please revise the abstract to cite the exact table and metric, or add the missing CSQA comparison.","section":"Abstract and Tables 3, 4, 10"},{"comment":"The choice of shared-scale format (E5M2 vs. E4M3 vs. PoT with floor or round) is made by minimizing Wikitext-2 perplexity on LLaMA2-7B (Fig. 5, Table 18), and Wikitext-2 perplexity is also a headline evaluation metric throughout the paper (Tables 1, 12, 14). The paper should either report whether the selected configuration remains optimal on a held-out task or model that was not used for selection, or explicitly discuss the potential selection bias. As written, part of the reported advantage is tuned to the evaluation objective, which weakens the claim that E5M2 is intrinsically the best shared-scale choice.","section":"§4.2, Fig. 5, Table 18"}],"minor_comments":[{"comment":"The text refers to \"Appendix 5.5\" when describing the hardware evaluation; this should be Section 5.5.","section":"§4.3"},{"comment":"Algorithm 1 still describes the original floor-based MX quantization, but Section 4.2 introduces a modified rounding rule for the PoT scale; please present the updated algorithm or state explicitly that the emulator implements the rounding variant.","section":"Appendix B.1, Algorithm 1"},{"comment":"The caption lists two subfigures labeled \"(d)\" and the subfigure letters do not match the order in which they are discussed in the text; please renumber the subfigures.","section":"Fig. 1"},{"comment":"The column headings \"Area-Memory\", \"Power-Area\", and \"Power-Area-Memory\" need explicit definitions (e.g., whether these are products of normalized ratios) so the reader can interpret the reported multipliers.","section":"Table 9"},{"comment":"The term \"MXFP4-PoT\" is used before it is defined in Section 4.2; please define it at first use.","section":"§5.2"},{"comment":"The paper does not report the number of random seeds or trials for the rotation-based comparisons; adding this information would improve the reliability of the overfitting analysis.","section":"§5.1, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The accuracy story is reasonably well supported across multiple benchmarks, but the hardware section is the weakest link. If Table 9 is a typo and AMXFP4 is actually cheaper than MXFP4, the overhead claim is inverted; if AMXFP4 is more expensive, the table must be corrected. Either way, the emulator/hardware equivalence needs evidence before the cost claim can be accepted. The abstract overstatements should also be aligned with the tables before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a legitimate contribution, not a wrapper job. The core insight—that MX grouping buys outlier suppression at the price of group-wise asymmetry, and that sign-dependent shared scales fix that—is clearly demonstrated and the resulting format behaves well across a wide range of models and tasks. The evaluation is unusually broad: language modeling, MT-Bench, VQA, LongBench, 70B, BART, and a real MAC synthesis. The numbers consistently favor AMXFP4 over MXFP4, and the effect is not marginal.\n\nWhat I'd push back on: the abstract takes liberties. '3% on VQA' is really ChartQA (3.28), while DocVQA is 2.72 and VQA-T is 1.25. '1.6% on CSQA' is not directly backed; Table 10's best QuaRot+GPTQ WinoGrande is 66.54, AMXFP4 67.32, so ~0.8 on that axis. These are fixable wording issues, but they matter for a hardware/format paper where readers will skim.\n\nMissing DuQuant comparison: the authors call DuQuant state-of-the-art but never include it in any table. That's a gap. If the claim is 'matches or beats rotation methods', you need the strongest one in the table.\n\nThe emulator/hardware gap is real. Accuracy comes from quantize_mx_op in software; the 10% overhead comes from a separately synthesized MAC. There is no bit-exactness check between the two. If the hardware handles rounding or sign-selection differently, the accuracy numbers may not transfer. This is a validation gap, common in this line of work, but it should be addressed (at least by a statement of intent, ideally by an RTL simulation comparison).\n\nThe stress-test's Table 9 concern is a misread. Those are improvement factors relative to FP16 (higher is better), so MXFP4 at 9.23x vs AMXFP4 at 8.32x on Area-Memory means AMXFP4 is about 10% worse—consistent with '10% overhead'. No inconsistency there.\n\nSelection bias: the shared-scale format was chosen by minimizing Wikitext-2 perplexity on LLaMA2-7B, and that same metric is a headline result. The fact that gains hold across many other tasks and models mitigates, but it's worth flagging.\n\nWho's it for: hardware architects and quantization researchers. It deserves peer review. I'd send it out.\n\nRecommendation: accept after minor revisions if the abstract is fixed, DuQuant is either added or its absence is justified, and the emulator/hardware consistency is at least discussed.","headline":"Solid format paper with a real insight; abstract overclaims and the emulator/hardware gap needs a check.","tokens_in":28783,"tokens_out":4366,"would_cite":true,"duration_ms":40878,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AMXFP4, an asymmetric microscaling 4-bit floating-point format that gives positive and negative elements their own shared scales, enables calibration-free direct-cast 4-bit LLM inference that matches or beats rotation-based INT4 pipelines…","keywords":["4-bit quantization","microscaling","asymmetric floating-point","LLM inference","activation outliers","calibration-free quantization","rotation-based quantization","MXFP4"],"falsifier":"Feed the same random and real activation tensors through the software emulation from Appendix B and a detailed simulation of the synthesized AMXFP4 MAC unit, and compare outputs bit by bit; any divergence in scale-pair selection, clamping, or rounding would show that the reported accuracy gains may not hold on the hardware whose cost is claimed.","tokens_in":27610,"feed_emoji":"🧠","tokens_out":9685,"duration_ms":82135,"temperature":0.7,"pith_summary":"AMXFP4 is a 4-bit floating-point format for LLM inference that splits each quantization group into two halves, each with its own shared scale: one for positive values and one for negative values. The paper's central argument is that standard microscaling formats such as MXFP4 suppress activation outliers by grouping elements into blocks of 32, but that this grouping makes each block's value distribution asymmetric, so a symmetric zero-centered representation is no longer optimal. By using asymmetric shared FP8 scales, AMXFP4 keeps the outlier suppression of microscaling while quantizing closer to the ideal Lloyd-Max limit, and it does this through direct casting without calibration. Reported results show AMXFP4 beating MXFP4 across language modeling, chatbot, long-context, and visual question-answering tasks, and matching or exceeding rotation-based INT4 pipelines such as QuaRot with GPTQ and SpinQuant while also quantizing attention matmuls. If correct, this would remove the multi-hour calibration step from accurate 4-bit LLM inference and make fully quantized attention practical at long context.","feed_headline":"AMXFP4 makes calibration-free 4-bit LLM inference accurate","feed_subtitle":"One shared scale per sign matches rotated-INT4 accuracy without calibration.","key_machinery":"The load-bearing mechanism is the pair of asymmetric shared scales. For each group of 32 elements, AMXFP4 stores a positive shared scale $2^{e_{sp}}\\hat{M}_p$ and a negative shared scale $2^{e_{sn}}\\hat{M}_n$, both FP8 with 5 exponent bits and 2 mantissa bits, and encodes each element as an FP4 value with sign, 2-bit exponent, and 1-bit mantissa. During multiplication the product of two elements takes one of four scale combinations, $S_{Xp}S_{Wp}$, $S_{Xp}S_{Wn}$, $S_{Xn}S_{Wp}$, or $S_{Xn}S_{Wn}$, selected by a small lookup on the two operand signs; because the scale mantissa is only 2 bits and the same scale serves the whole group, this sign-aware scaling is nearly free in hardware. The design also replaces MX's floor-based power-of-two scale decision with rounding, avoiding the clamping error that otherwise grows as group size shrinks. Together these choices give the format an asymmetric grid of representable values that matches the lopsided group distributions that microscaling itself creates.","core_discovery":"The central discovery is a trade-off hidden in microscaling: shrinking the quantization group to 32 elements tames activation outliers (kurtosis falls nearly to zero), but it scatters the group means, meaning each group is more asymmetric the finer the group granularity gets. MXFP4's symmetric representation therefore leaves error on the table, and data rotation, which helps at row-level group sizes, actively hurts when combined with microscaling because it adds still more asymmetry. AMXFP4 counters this with two shared FP8 (E5M2) scales per group, one for positive and one for negative elements, with the scale chosen at multiply time from the signs of the two operands. On Wikitext-2 this lowers perplexity from 6.49 to 6.22 for LLaMA2-7B relative to MXFP4, lifts ChartQA from 46.20 to 49.48, and in Table 10 reaches a WinoGrande accuracy of 67.32 against a best rotation-baseline value of 66.22, without any calibration. The format also narrows the gap to the 16-bit baseline enough that MT-Bench conversational scores recover close to baseline, and a synthesized MAC unit implementing the sign-dependent scale selection is reported at only about 10% area overhead over a compatible MX MAC.","pith_inferences":["The paper's own group-size sweeps show AMXFP4's advantage over symmetric MXFP4 widening as groups shrink, so a natural extension is to push the asymmetric shared-scale idea to even smaller groups (for example, 16 or 8 elements) or to per-channel granularity.","Because the mechanism targets distribution shape rather than LLM-specific structure, the same asymmetric shared-scale design could plausibly transfer to other outlier-heavy workloads such as vision transformers, diffusion models, or training-side rescaling, though the paper does not test these.","The emulator-versus-hardware gap could be closed by a bit-exact co-verification harness; without that artifact, the accuracy story and the 10% overhead story remain two claims about two different implementations.","A testable prediction of the paper's analysis is that combining row-level rotation with fine-grained asymmetric microscaling would recover the row-level rotation benefit without the destructive interaction seen at group size 32, something the paper's Table 5 suggests but does not directly explore."],"forward_implications":["All attention matrix multiplications, including the softmax-output and query-key products that rotation methods leave in FP16, can be run in 4-bit, which matters most as context length grows because attention FLOPs scale quadratically.","Models can be deployed at 4-bit by direct casting with no calibration run, removing the multi-hour overhead and the calibration-set overfitting that rotation-based pipelines exhibit.","A 3-bit variant, AMXFP3, degrades Wikitext-2 perplexity by only about 1.7 on LLaMA2-7B, whereas QuaRot with GPTQ degrades by more than 30, suggesting the approach extends below 4 bits.","AMXFP4 is compatible with other compression methods: applied to a 20%-pruned LLaMA-7B model it recovers most of the pruning accuracy drop, so its benefits are additive.","An asymmetric version of the recently deployed commercial MXFP4 variant NVFP4, called ANVFP4, also beats the commercial format, particularly at group size 16, indicating that per-sign scales are a generally useful addition to MX-family formats."],"supporting_citations":[{"why":"Defines the MX specification, including the MXFP4 element format and the 8-bit shared-scale scheme that AMXFP4 modifies.","marker":"(Rouhani et al., 2023a)"},{"why":"Supplies the MX quantization procedure and the previous MXFP4 empirical results that AMXFP4 improves upon.","marker":"(Rouhani et al., 2023b)"},{"why":"QuaRot is the rotation-based INT4 baseline whose accuracy and calibration overhead AMXFP4 is compared against.","marker":"(Ashkboos et al., 2024)"},{"why":"SpinQuant provides the learned-rotation baseline and the calibration-overfitting comparison points.","marker":"(Liu et al., 2024b)"},{"why":"Source of the MX MAC unit design that the AMXFP4 hardware extension builds on and synthesizes.","marker":"(Darvish Rouhani et al., 2023)"},{"why":"Source of the AsymFP element-wise format that AMXFP4 adapts into the shared-scale setting.","marker":"(Zhang et al., 2024b)"},{"why":"Documents the activation-outlier phenomenon that motivates both the microscaling and asymmetric-representation choices.","marker":"(Dettmers et al., 2022)"},{"why":"GPTQ is the calibration-based weight quantization used in the QuaRot+GPTQ baseline that AMXFP4 matches or beats.","marker":"(Frantar et al., 2022)"}],"fun_headline_variants":["AMXFP4: dual-scale FP fixes MX outlier trade-off in 4-bit LLMs","Calibration-free AMXFP4 matches rotated INT4 accuracy in 4-bit LLM","Asymmetric microscaling: one scale per sign beats MX in 4-bit LLMs","AMXFP4 uses sign-dependent dual scales to cut MX asymmetry in 4-bit","AMXFP4: two shared scales per group tame outliers without calibration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the software emulator used for all accuracy measurements behaves exactly like the separately synthesized hardware MAC unit, including the sign-dependent choice of shared scale and the treatment of clamping and rounding; the paper reports the two implementations separately and does not show bit-level agreement between them.","fun_headline_variants_meta":{"raw":{"variants":["AMXFP4: dual-scale FP fixes MX outlier trade-off in 4-bit LLMs","Calibration-free AMXFP4 matches rotated INT4 accuracy in 4-bit LLM","Asymmetric microscaling: one scale per sign beats MX in 4-bit LLMs","AMXFP4 uses sign-dependent dual scales to cut MX asymmetry in 4-bit","AMXFP4: two shared scales per group tame outliers without calibration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000744,"raw_usage":{"total_tokens":3374,"prompt_tokens":1058,"completion_tokens":2316,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":674,"completion_tokens_details":{"reasoning_tokens":2205}},"tokens_in":674,"tokens_out":2316,"duration_ms":18303,"temperature":1.0,"reasoning_tokens":2205,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:10:37.748257+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed the same random and real activation tensors through the software emulation from Appendix B and a detailed simulation of the synthesized AMXFP4 MAC unit, and compare outputs bit by bit; any divergence in scale-pair selection, clamping, or rounding would show that the reported accuracy gains may not hold on the hardware whose cost is claimed.","supporting_citations":[],"review_version":1}