{"id":"276434c3-6b53-4ce1-993f-58fdb9cf1eb1","arxiv_id":"2506.11543","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new Fisher-information-based reconstruction loss, DPLR-FIM, improves low-bit post-training quantization accuracy for Vision Transformers without specialized quantizers.","lead":"FIMA-Q is a new post-training quantization method for Vision Transformers that replaces the usual squared-gradient loss with a Fisher Information Matrix approximation combining diagonal and low-rank terms. It reports large accuracy gains over prior PTQ methods at 3-bit and 4-bit quantization, using only a standard uniform quantizer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.2 is stated as exact but its proof is a second-order Taylor expansion, so the DPLR-FIM loss is not verified as a KL surrogate at 3-bit; the theoretical basis of the low-bit gains is unproven.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing gap: Theorem 3.2 is presented as exact but proven only via a second-order Taylor expansion, leaving the small-perturbation condition unverified for 3-bit quantization. I agree this is the most important concern because every derived loss used in the method inherits this approximation, and the paper's central explanatory claim is that the DPLR-FIM loss better captures the true KL/Hessian structure. The released code and extensive ablations are real evidence that the method may work, but they do not test whether the theoretical equality holds under the operating conditions where the gains are claimed. The unspecified α in Eq. (21) and the difficulty of reproducing the 5.31% average from Table 1 are additional reproducibility problems, but they are secondary to the correctness of the core theoretical statement. Since the reader already assigned CONDITIONAL, my read does not move the verdict; it reinforces that the paper needs either a corrected approximate theorem with a quantified remainder or an empirical validation that the surrogate tracks the true KL in the low-bit regime.","tokens_in":16531,"tokens_out":12407,"duration_ms":116870,"concrete_test":"On ViT-S and DeiT-S at W3/A3, select the first, middle, and last Transformer blocks. For each block, compute the true KL divergence between the full-precision and quantized block-output distributions over the calibration set, and compute the quadratic surrogate 1/2 Δz^T F Δz using the empirical FIM (mean outer product of score gradients). Report the relative error |KL - surrogate|/KL and the Spearman rank correlation across blocks and calibration samples. If the relative error exceeds, say, 50% or the rank correlation is low, Theorem 3.2's exact claim fails in the operating regime, and the paper should restate the theorem as an approximation with a validated error bound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. 3.2 Theorem 3.2 asserts L_KL(Δz^(b)) = 1/2 Δz^(b)T F(z^(b)) Δz^(b) as an exact equality, and Eq. (10) plus all subsequent DPLR losses (Eqs. 14, 16, 18, 21) rely on it. The proof in Sec. A.3 is a second-order Taylor expansion of log p(x; z+Δz); the remainder is dropped without a bound or even an acknowledgement that the equality is approximate. Hence the theorem holds only up to O(||Δz||^3) under a small-perturbation assumption. At W3/A3, the quantization perturbation is not infinitesimal, and this is precisely the regime where the paper reports its largest gains (23.04% and 18.18% over MSE on ViT-S and DeiT-S). If the quadratic surrogate is inaccurate, the DPLR-FIM loss is not a faithful KL-divergence loss, and the paper's explanation for these gains—that the proposed loss better approximates the FIM/Hessian—loses support. The empirical numbers may still be reproducible from the released code, but no experiment measures the size of the omitted Taylor remainder or directly compares the DPLR-FIM loss against the true KL for quantized blocks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FIMA-Q, a post-training quantization (PTQ) method for Vision Transformers. It replaces the Hessian-guided loss of prior work with a loss based on an approximation of the Fisher Information Matrix (FIM), arguing that FIM is linearly proportional to the gradient of the KL divergence rather than to the squared gradient. The method computes a diagonal approximation, a low-rank approximation, and combines them into a \"diagonal plus low-rank\" (DPLR) FIM loss. The loss is used in a block-wise reconstruction framework built on QDrop. Experiments on ImageNet classification and COCO detection/segmentation report strong results, especially at 3-bit quantization, while using only a standard uniform quantizer.","tokens_in":16789,"tokens_out":5744,"duration_ms":55247,"significance":"If the central claim is correct, the method is practically significant: it improves low-bit ViT accuracy substantially over prior PTQ methods using a simple uniform quantizer, avoiding specialized quantizers that are harder to implement in hardware. The empirical results in Tables 1 and 2 are extensive and the improvements at W3/A3 are large. The paper also provides code, which aids reproducibility. However, the theoretical foundation for the proposed loss has gaps: Theorem 3.2 is stated as exact but is proven via a truncated Taylor expansion, the combination weight alpha in Eq. (20) is never specified, and the low-rank FIM is not guaranteed to be positive semidefinite. These issues bear directly on whether the loss is a faithful surrogate for the KL divergence, which is the stated explanation for the empirical gains.","major_comments":[{"comment":"Theorem 3.2 asserts the exact equality L_KL(Delta z) = 1/2 Delta z^T F(z) Delta z, but the proof in Sec. A.3 is a second-order Taylor expansion in which the remainder is dropped without a bound. The equality therefore holds only up to O(||Delta z||^3) and requires the perturbation Delta z to be sufficiently small. The paper's largest reported gains occur at 3-bit quantization, where this small-perturbation assumption is not verified. Since Eq. (9) is the basis for all subsequent losses (Eqs. 14, 16, 18, 21), the authors should either restate Theorem 3.2 as an approximation with an explicit remainder term, or provide an experiment that measures the omitted remainder or directly compares the proposed loss with the true KL divergence for quantized blocks.","section":"Sec. 3.2 / Theorem 3.2, Sec. A.3"},{"comment":"The scalar alpha in the DPLR-FIM combination F_DPLR = alpha F_rank-k + (1-alpha) F_diag is never specified. Eq. (21) defines the final loss as alpha L_rank-k + (1-alpha) L_diag, and the paper later fixes the rank k = 15 but never gives the value of alpha or any ablation over it. Because alpha directly controls the relative contribution of the low-rank and diagonal terms, the final loss is underspecified and the results cannot be reproduced from the text alone. The authors should state the value used and, ideally, report sensitivity to it.","section":"Sec. 3.3, Eq. (20)"},{"comment":"The low-rank FIM F = grad L * Delta z^+ is not guaranteed to be symmetric or positive semidefinite. The paper explicitly relaxes the symmetry constraint, but the quadratic form in Eq. (18) is then not a principled FIM or Hessian surrogate: if the symmetric part is indefinite, the loss can be negative and can be decreased without bound when viewed as a function of Delta z. The authors should either enforce symmetry and positive semidefiniteness (e.g., by symmetrization and projection) or explain why the indefinite form still behaves as a useful objective in their optimization, with supporting evidence.","section":"Sec. 3.3, Eqs. (17)-(19)"},{"comment":"The notation for Delta z^(b) is inconsistent between a vector and a matrix. In Eqs. (9)-(11) Delta z^(b) is an a-dimensional vector, and Eq. (13) writes it as a sum of vectors. But the low-rank construction in Eqs. (17)-(19) and the progressive rank increase described in the text require Delta z^(b) to be an a x k matrix of k linearly independent perturbations, and the Moore-Penrose inverse formula only makes sense in that interpretation. This ambiguity obscures the rank-k loss and should be clarified by defining the matrix explicitly and adjusting Eq. (13) or its interpretation.","section":"Sec. 3.3, Eqs. (13)-(19), Algorithm 1"}],"minor_comments":[{"comment":"The phrase \"promoting the accuracy of MSE by 23.04% and 18.18%\" should say \"by 23.04 and 18.18 percentage points,\" since it refers to absolute differences in top-1 accuracy (e.g., 64.09 - 41.05 = 23.04).","section":"Sec. 4.3"},{"comment":"The statement that FIMA-Q surpasses the second-best approach by 5.31% on average at 3-bit quantization should specify the averaging procedure and confirm that the difference is measured in percentage points.","section":"Sec. 4.2"},{"comment":"There are minor typos: \"diagnal\" in Sec. 3.3, \"computer\" in Sec. 3.3, and \"Nvidia RX 4090\" in Table 4 should be \"RTX 4090.\"","section":"Sec. 4.1 / Table 4"},{"comment":"Eq. (8) writes the KL divergence as a sum over x, while the proof in Sec. A.3 uses an integral over R. The notation should be made consistent, since p(y; z) is a distribution and the domain of y is not specified.","section":"Sec. A.3, Eq. (8)"},{"comment":"Table A in the supplementary material is referenced in Sec. B but is not labeled with a number in the main text; adding a reference such as \"see Table A in the supplementary\" would improve readability.","section":"Supplementary, Table A"}],"recommendation":"major_revision","confidential_remarks":"The empirical results are strong and the method appears practically useful, but the theoretical presentation overclaims an exact FIM-KL identity that is only approximate, and the final loss depends on an unspecified parameter alpha. These are fixable within the scope of a revision, so I recommend major revision rather than rejection. I do not see a circularity problem: the method is evaluated on external accuracy benchmarks, and the only self-citation (AdaLog) is used as a baseline, not as a load-bearing input."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: FIMA-Q is a genuinely useful PTQ method for ViTs, with strong empirical results and released code, but the paper's central theorem is overclaimed and one hyperparameter is missing, so it should be accepted with revisions rather than as-is.\n\nThe new thing here is the DPLR-FIM reconstruction loss, a diagonal-plus-low-rank approximation to the Fisher information matrix, built on the observation that the gradient of the KL divergence w.r.t. the quantization perturbation is F times the perturbation rather than the squared gradient. That shift in the loss construction is real and appears to pay off. The experiments are broad: ImageNet classification across ViT, DeiT, Swin at 6/4/3 bits, plus COCO detection and segmentation, all with a standard uniform quantizer. The 3-bit numbers are striking—over 20 points above MSE on some small ViTs—and they beat methods with specialized quantizers. With code released, this is a checkable contribution.\n\nThe soft spots are in the framing. Theorem 3.2 says L_KL(Δz) = 1/2 Δz^T F Δz as an exact equality, but the proof in Sec. A.3 is a second-order Taylor expansion; the remainder is dropped with no bound. So the loss is an approximation, and at 3-bit the perturbation isn't small. That's exactly where the biggest gains are, so the theoretical explanation for those gains is unproven. The empirical numbers may hold anyway—this is a heuristic that works—but the paper shouldn't claim exactness. Relatedly, alpha in Eq. (20) is never specified. They fix rank k=15 but not alpha, so you can't reproduce exactly without the code. Minor point: the low-rank FIM from Eq. (17) need not be positive semidefinite, though the loss seems to tolerate that.\n\nNone of this is fatal. The core contribution is the loss and the experiments, and those look solid. The theory needs a careful rewrite: either weaken the theorem to an approximation with a bound, or add experiments checking the Taylor remainder. And they should state alpha.\n\nWho's this for? Anyone working on low-bit ViT deployment. It deserves a serious referee and probably acceptance with minor-to-moderate revisions.","headline":"Strong empirical PTQ paper with an overclaimed theoretical centerpiece; the DPLR-FIM loss works in practice, but Theorem 3.2 is an approximation stated as exact and a key hyperparameter is missing.","tokens_in":17359,"tokens_out":3225,"would_cite":true,"duration_ms":30650,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that the Fisher Information Matrix, approximated as a diagonal-plus-low-rank matrix and built from the KL-divergence gradient, is a better quantization-reconstruction loss than squared-gradient Hessian…","keywords":["post-training quantization","vision transformers","Fisher information matrix","KL divergence","Hessian approximation","diagonal plus low-rank","low-bit quantization","block-wise reconstruction"],"falsifier":"Compute, for a block-wise quantized ViT at W3/A3, the exact KL divergence $D_{\\mathrm{KL}}(p(y;z^{(b)})\\,\\|\\,p(y;z^{(b)}+\\Delta z^{(b)}))$ by sampling over the calibration set, and compare it to the DPLR-FIM quadratic form $\\tfrac{1}{2}\\Delta z^{(b)\\top} F_{\\mathrm{DPLR}} \\Delta z^{(b)}$. If the ratio of the two quantities deviates substantially from 1 across blocks, or if the higher-order remainder of the Taylor expansion is not negligible, the surrogate is not faithful and the reported gains cannot be attributed to the FIM mechanism.","tokens_in":16324,"feed_emoji":"📉","tokens_out":7719,"duration_ms":65700,"temperature":0.7,"pith_summary":"Post-training quantization of vision transformers collapses at 3-bit precision when the reconstruction loss is built from a Hessian whose diagonal is approximated by squared gradients; the paper argues this surrogate is wrong because Fisher Information is linearly proportional to the KL-divergence gradient, not to its square. It establishes the exact identity $L_{\\mathrm{KL}}(\\Delta z^{(b)}) = \\tfrac{1}{2}\\Delta z^{(b)\\top} F(z^{(b)}) \\Delta z^{(b)}$ and exploits it to design a diagonal-plus-low-rank FIM loss (DPLR-FIM) that keeps both individual output sensitivities and inter-token correlations. Plugging this loss into the standard QDrop block-reconstruction pipeline, with only a plain uniform quantizer, the method reports average gains of 5.31% over the second-best approach at W3/A3 on ImageNet, and lifts accuracy over the MSE baseline by up to 23.04% on ViT-S. The claim matters because it suggests the reconstruction loss, not quantizer flexibility, is the main bottleneck for low-bit ViT deployment.","feed_headline":"Fisher-information loss lifts 3-bit ViT accuracy by up to 23 points","feed_subtitle":"A diagonal-plus-low-rank Fisher loss beats specialized quantizers with a plain uniform quantizer.","key_machinery":"The load-bearing object is the DPLR-FIM loss $L_{\\mathrm{DPLR}} = \\alpha L_{\\mathrm{rank}-k} + (1-\\alpha) L_{\\mathrm{diag}}$, a diagonal-plus-low-rank surrogate for the Fisher Information Matrix. The identity that carries the argument is Theorem 3.2, $L_{\\mathrm{KL}}(\\Delta z^{(b)}) = \\tfrac{1}{2}\\Delta z^{(b)\\top} F(z^{(b)}) \\Delta z^{(b)}$, which makes the FIM linearly recoverable from the KL-divergence gradient via $\\nabla L_{\\mathrm{KL}} = F \\Delta z^{(b)}$. The diagonal term is built from the per-element ratio of gradient to perturbation; the rank-$k$ term is built from $\\nabla L_{\\mathrm{KL}}$ times the Moore-Penrose inverse of the perturbation matrix, capturing inter-token correlations at $O(ak)$ cost; a progressive strategy increases $k$ during reconstruction.","core_discovery":"On the paper's own terms, the central discovery is that the prevailing Hessian-guided quantization loss in post-training quantization has been using the wrong approximation: replacing the Fisher Information Matrix with squared gradients discards both the variance of the score and all off-diagonal structure. The paper proves, via a second-order Taylor expansion of the KL divergence under the regularity condition, that the quantization loss equals $\\frac{1}{2}\\Delta z^\\top F(z)\\Delta z$, and therefore the gradient of the KL loss is $F(z)\\Delta z$; FIM is thus recovered from the ratio of the KL gradient to the perturbation, linearly, not as a squared gradient. From that relation it builds three progressively richer surrogates — diagonal, rank-one/low-rank, and their diagonal-plus-low-rank combination — and shows empirically that the DPLR-FIM loss, integrated into QDrop-style block reconstruction with a uniform quantizer, pushes 3-bit and 4-bit accuracy above methods that rely on specialized quantizers. The paper reports consistent gains across ViT, DeiT, and Swin architectures on ImageNet and COCO, with the largest margins at 3-bit quantization.","pith_inferences":["A natural extension the paper leaves implicit: the same DPLR-FIM loss could be applied to post-training quantization of large language models or diffusion models, where block-wise reconstruction with Hessian surrogates is also common.","The results suggest that the reconstruction loss, rather than quantizer flexibility, is the main bottleneck for low-bit ViTs; if so, further gains should come from loss design rather than new quantizers.","The fixed rank $k=15$ could be made adaptive per block based on the estimated FIM spectrum, trading accuracy for reconstruction time on simple blocks; this is testable with the released code.","The supplementary sample-size ablation indicates reconstruction is more calibration-hungry than FIM estimation, so data-efficient reconstruction techniques could widen FIMA-Q's applicability to smaller calibration sets."],"forward_implications":["At W3/A3 on ImageNet, FIMA-Q reports 64.09% top-1 on ViT-S and 69.13% on DeiT-S with a plain uniform quantizer, surpassing the second-best optimization-based method by 5.31% on average across architectures.","Swapping the squared-gradient diagonal Hessian for the DPLR-FIM loss improves top-1 accuracy over the MSE baseline by 23.04% (ViT-S) and 18.18% (DeiT-S) at W3/A3, with an 8.74% average improvement.","Because only a standard uniform quantizer is needed, the accuracy gains do not depend on hardware-unfriendly specialized quantizers such as log or twin-uniform quantizers.","In object detection and instance segmentation on COCO, using RPN classification outputs as the task signal, the method achieves the best W4/A4 results in most Mask R-CNN and Cascade Mask R-CNN settings with Swin backbones.","Reconstruction time grows linearly with rank $k$ but stays under 480 GPU minutes on a single RTX 4090 for the largest tested model, so the method is practical for deployment."],"supporting_citations":[{"why":"BRECQ supplies the block-reconstruction framework and the diagonal squared-gradient FIM approximation that the paper identifies as inaccurate and replaces.","marker":"[16]"},{"why":"QDrop provides the block-wise reconstruction pipeline and MSE baseline that FIMA-Q builds on and improves.","marker":"[30]"},{"why":"AdaRound contributes the adaptive rounding weight update used inside the reconstruction loop.","marker":"[26]"},{"why":"Fisher (1922) is the source of the FIM definition and the regularity condition that underpin Theorems 3.1 and 3.2.","marker":"[11]"},{"why":"PTQ4ViT is a representative ViT-specific PTQ method with a twin uniform quantizer that FIMA-Q compares against as a specialized-quantizer baseline.","marker":"[33]"}],"fun_headline_variants":["FIMA-Q Fisher loss wins 3-bit ViT quantization by 23 points","Diagonal-plus-low-rank Fisher loss boosts 3-bit ViT accuracy","FIMA-Q: Better Fisher approximation improves low-bit ViTs","Fisher info loss outperforms Hessian in ViT quantization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central identity is obtained by a second-order Taylor expansion of the KL divergence, so it holds only when the quantization perturbation $\\Delta z^{(b)}$ is small enough that higher-order terms can be dropped; the paper does not directly verify this small-perturbation condition at aggressive 3-bit quantization, where the reported gains are largest.","fun_headline_variants_meta":{"raw":{"variants":["FIMA-Q Fisher loss wins 3-bit ViT quantization by 23 points","Diagonal-plus-low-rank Fisher loss boosts 3-bit ViT accuracy","FIMA-Q: Better Fisher approximation improves low-bit ViTs","Fisher info loss outperforms Hessian in ViT quantization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000247,"raw_usage":{"total_tokens":1563,"prompt_tokens":985,"completion_tokens":578,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":503}},"tokens_in":601,"tokens_out":578,"duration_ms":6744,"temperature":1.0,"reasoning_tokens":503,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:04:16.894284+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, for a block-wise quantized ViT at W3/A3, the exact KL divergence $D_{\\mathrm{KL}}(p(y;z^{(b)})\\,\\|\\,p(y;z^{(b)}+\\Delta z^{(b)}))$ by sampling over the calibration set, and compare it to the DPLR-FIM quadratic form $\\tfrac{1}{2}\\Delta z^{(b)\\top} F_{\\mathrm{DPLR}} \\Delta z^{(b)}$. If the ratio of the two quantities deviates substantially from 1 across blocks, or if the higher-order remainder of the Taylor expansion is not negligible, the surrogate is not faithful and the reported gains cannot be attributed to the FIM mechanism.","supporting_citations":[{"cited_title":"BRECQ: pushing the limit of post-training quantization by block reconstruction","cited_arxiv_id":null,"evidence_quote":"BRECQ supplies the block-reconstruction framework and the diagonal squared-gradient FIM approximation that the paper identifies as inaccurate and replaces."},{"cited_title":"Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization","cited_arxiv_id":null,"evidence_quote":"QDrop provides the block-wise reconstruction pipeline and MSE baseline that FIMA-Q builds on and improves."},{"cited_title":"Up or down? adaptive rounding for post-training quantization","cited_arxiv_id":null,"evidence_quote":"AdaRound contributes the adaptive rounding weight update used inside the reconstruction loop."},{"cited_title":"On the mathematical foundations of theoretical statistics.Philosophical Transactions of the Royal Society of London","cited_arxiv_id":null,"evidence_quote":"Fisher (1922) is the source of the FIM definition and the regularity condition that underpin Theorems 3.1 and 3.2."},{"cited_title":"Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization","cited_arxiv_id":null,"evidence_quote":"PTQ4ViT is a representative ViT-specific PTQ method with a twin uniform quantizer that FIMA-Q compares against as a specialized-quantizer baseline."}],"review_version":1}