REVIEW 3 major objections 5 minor 3 references
iFairy: the First 2-bit Complex LLM with All Parameters in $\{\pm1, \pm i\}$
T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper argues that the way past the accuracy ceiling of 2-bit quantized LLMs is a complex-valued full-precision backbone whose weights quantize to the two-bit codebook ±1, ±i — entropy-optimal and multiplication-free at inference.
desk verdict The method is real and the code is out, but the headline 'ceiling' claim is built on a parameter-count sleight of hand: a 700M complex model is 1.4B real scalars, so the comparison to BitNet is not apples-to-apples. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
PhaseQuant, the map $P(w) = i^{\lfloor 2\theta/\pi + 1/2\rfloor}$ for $\theta = \mathrm{Arg}(w) \in [-\pi, \pi]$, projects each full-precision complex weight onto the fourth roots of unity $\{\pm1, \pm i\}$; dequantization uses per-component scales $\gamma_{\mathrm{re}}$, $\gamma_{\mathrm{im}}$, and gradients pass through the non-differentiable step by Straight-Through Estimation. It sits in a complex-valued LLaMA-style backbone: ComplexLinear via the Hermitian inner product, attention scores as $\mathrm{Re}(QK^\top)$ (the Euclidean angle in $\mathbb{R}^{2n}$), dual-channel real/imaginary embeddings and LM head, component-wise ReLU², and complex rotary positions $e^{im\theta_j}$. Computation
What would settle it
Train, on the same 100B-token corpus, a real LLaMA whose hidden width is doubled so its real parameter count matches the 700M complex model (which holds about twice the real values in full precision), and a real LLaMA at the same nominal 700M width; compare their validation perplexity against the full-precision complex backbone's 10.08. If the doubled-width real model matches or beats it, the 'ceiling-raising' result is a parameter-count artifact rather than a complex-domain advantage. A second check: benchmark an actual inference kernel with 2-bit weights and INT8 activations to confirm the a
Extended reading notes
Core claim
The discovery: full use of the 2-bit space requires complex weights. PhaseQuant maps each weight $W = W_{\mathrm{re}} + iW_{\mathrm{im}}$ to one of $\{\pm1, \pm i\}$ by its phase $\theta = \mathrm{Arg}(w)$, with per-component scale factors $\gamma_{\mathrm{re}}$, $\gamma_{\mathrm{im}}$. The four codewords realize the full $\log_2 4 = 2$ bits under uniform usage, beyond the ternary set's ~1.58 bits, and each has a zero real or imaginary part, so a weight-activation product is a sign flip and a swap. The paper's claim: this complex backbone raises the full-precision ceiling, and the 2-bit models at 700M and 1.3B then exceed the 2-bit baseline ceiling on perplexity and downstream tasks — the 1.
Load-bearing premise
The load-bearing premise is that the complex domain itself, not the doubled real parameter footprint of writing each weight as $W = W_{\mathrm{re}} + iW_{\mathrm{im}}$, is what raises the full-precision 'ceiling'; if a same-total-real-parameter real model matches the complex backbone, the paradigm's foundation slides.
Editorial extensions
If this is right
- If the ceiling-raising approach holds, quantization research gains a second lever: improving the full-precision backbone's accuracy instead of only shrinking quantization error, since the 2-bit projection now sits on a higher base.
- With all four codewords used near-uniformly across every weight matrix (Section 4.4, Appendix D.1), the full 2 bits per complex weight are genuinely deployed, so the measured gains over the ~1.58-bit ternary baseline are not a storage loophole.
- At 1.3B scale, the quantized model's 46.52 average zero-shot accuracy edges past the FP16 LLaMA baseline's 46.21, so at this scale extreme 2-bit compression need not cost commonsense reasoning accuracy.
- The speedups are properties of the quantized operation, not the current implementation: the paper trains in BF16 with complex attention recast onto real FlashAttention kernels, and its limitations section (Section 5) states that realizing the addition-only, look-up-table path needs hardware or kernels not yet built for complex, multiplication-free arithmetic.
Reading between the lines
- The paradigm's headline comparison is not parameter-matched: each complex weight stores two real matrices, so the full-precision 700M complex model carries roughly twice the real values of the 700M real LLaMA it is compared against; a same-real-parameter real baseline would settle how much of the perplexity gain is the complex domain versus a doubled footprint.
- 'Surpassing the ceiling' is demonstrated against the reported and reproduced BitNet b1.58 and an in-house LLaMA, not against all conceivable 2-bit schemes; the ceiling language is a claim about the published baseline landscape, and a stronger real-valued 2-bit quantizer would reset it.
- The multiplication-free advantage is asymptotic: training and inference today run complex layers as ordinary real kernels on concatenated real/imaginary matrices, and the paper's own limitations section concedes that wall-clock gains need hardware or kernels built for complex, addition-based arithmetic.
- PhaseQuant extends naturally to other roots of unity — 8th roots for 3-bit complex weights — at the cost of codewords with both parts nonzero, which lose the zero-real/zero-imaginary sparsity that buys multiplication-freeness; testing that trade-off, or a learned unified scale in place of separate $\gamma_{\mathrm{re}}$, $\gamma_{\mathrm{im}}$, are open directions the paper does not pursue.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes iFairy, a complex-valued Transformer trained with QAT, in which complex weights are quantized to the fourth roots of unity {±1, ±i}. The architecture extends LLaMA-style blocks with dual-channel embeddings, complex linear layers using conjugate/Hermitian products, real-part attention scores, complex RoPE, and component-wise ReLU². PhaseQuant maps each complex weight by phase and applies separate per-tensor scaling factors. The authors claim that the complex domain raises the full-precision accuracy ceiling and that the 2-bit quantized model beats BitNet b1.58 and even FP16 LLaMA on perplexity and downstream tasks at 700M and 1.3B scales, while keeping storage and multiply-free compute comparable to BitNet. They release code and two model checkpoints.
Significance. The idea of using complex-valued representation to break the 'accuracy ceiling' of low-bit QAT is original and potentially valuable. The quantized codebook is elegantly simple, the inference pattern is addition-only, and the release of code/checkpoints is a plus. The paper also offers a useful analysis of codebook utilization and layer-wise norm stability. Nevertheless, the central comparison is not yet established: because every complex weight is two real trainable matrices, the full-precision '700M' iFairy has about twice the real parameters of the FP16 LLaMA and BitNet baselines, and the 2-bit quantized iFairy uses only 1 bit per real scalar. The reported gains are therefore consistent with a parameter-count effect rather than with any intrinsic advantage of complex-valued quantization. A controlled experiment at equal real-parameter count or equal bits per real parameter is required before the ceiling-raising claim can be accepted.
major comments (3)
- [3.2, Tables 3-4, §4.3] Each complex weight is defined as W = Wre + iWim, so a nominal 700M complex model contains ~1.4B real trainable scalars in its linear layers, and the dual-channel embedding and the concatenated LM head add further real parameters. Tables 3 and 4 compare this model with FP16 LLaMA and BitNet b1.58 at the same nominal size, and Section 4.3 interprets the PPL gain as 'representational power' of the complex domain. Quantized iFairy stores 2 bits per complex weight, i.e., 1 bit per real scalar; under the same bit budget it is equivalent to a 1.4B-parameter binary real model. The reported improvements over BitNet may therefore be entirely a parameter-count effect. Please add a real-valued control with the same number of real parameters and the same bit budget (e.g., a 1.4B binary real model or a 700M real model with a four-state 2-bit codebook), trained under the same 100B-token setting, and r
- [Title/Abstract vs §3.2, Algorithm 2, Fig. 7] The title and abstract claim that 'all parameters' are in {±1, ±i}. In the implemented architecture, PhaseQuant is applied to the backbone linear projections, but the dual-channel embedding layers and the language-model head are described as ordinary learned real matrices/concatenated projections (Section 3.2), and Figure 7 plots continuous-valued token embeddings and LM-head weights. Thus the 'all parameters' claim is not supported by the described system. The authors should either apply a {±1, ±i} quantizer to the embedding and head as well, or explicitly qualify the claim and report how many parameters remain unquantized and at what precision.
- [Appendix C.4] The statement that the quaternary complex codebook is 'information-theoretically optimal' because it reaches log2(4)=2 bits is not unique to complex values: any real 2-bit codebook with four values also reaches 2 bits. The meaningful comparison is between a 1.58-bit ternary codebook and a 2-bit quaternary codebook, not between complex and real representations. Please rephrase or benchmark against a real quaternary 2-bit QAT baseline; otherwise the optimality claim overstates the role of complex numbers.
minor comments (5)
- [Table 1 / §3.5] The multiplication table appears to contain algebra errors: standard complex multiplication gives (xre + i xim)·i = −xim + i xre and (xre + i xim)·(−i) = xim − i xre. The rows as printed ('+i x im + ixre' and '−i −xim − ixre') are inconsistent or mis-transcribed; please correct and double-check the conjugate convention used in the ComplexLinear operation.
- [Appendix C.4 / §3.5] The optimality claim based on log2(4)=2 bits should be stated relative to the bit budget actually used for storage. BitNet b1.58 is often described as 1.58 bits per weight in effective entropy but is deployed with 2-bit containers; the paper should specify the comparison convention and, if relevant, compare with a real four-state 2-bit quantizer.
- [Tables 3-4 / §4.2] The downstream accuracy differences between iFairy and the reproduced/results baselines are a few points on 0-100 accuracy with no variance or significance tests. Given the parameter-count confound, I would ask for confidence intervals or at least multiple-seed results for the headline comparisons.
- [Appendix D.2] Figure 9 lists '(i) V projection' and then '(j) V projection' again; one should be the K projection (or the caption should match the plot).
- [§3.5] The 'compute efficiency' claims are analytical only; no FLOP counts per token or wall-clock/energy numbers are given. Since activation channels are doubled, a table with measured CPU/GPU throughput would strengthen the strict-efficiency claim.
Circularity Check
No significant circularity: the paper's central claims are empirical comparisons against external baselines; PhaseQuant's scaling factors are computed from weight statistics rather than fitted to targets, and no load-bearing self-citation or imported uniqueness theorem is present.
full rationale
The paper's derivation chain is not circular. The motivating identity Accuracy_quant = Accuracy_full-precision − Error_quant is a definition of quantization error, not a predictive model; the paper uses it only to frame full-precision accuracy as a ceiling, and the claim of surpassing that ceiling is an empirical result, not a consequence of the identity. PhaseQuant's scaling factors γ_re and γ_im are computed from conditional expectations of the current weight magnitudes, not fitted to any target PPL or downstream score, so no fitted input is relabeled as a prediction. The 4-state codebook {±1, ±i} achieving 2 bits is a standard information-theoretic fact (log2 4 = 2) and is not derived from the experimental results. The paper's comparisons are made against external baselines (FP16 LLaMA, BitNet b1.58) under a unified training-from-scratch setting, so the central claims are empirical and self-contained. There are no load-bearing self-citations: the references to BitNet, ReLU2, and Scharnhorst are external sources, not the authors' own prior work. No uniqueness theorem is imported from the authors, and no ansatz is smuggled in via citation; architectural choices such as Re(QK^T) are explicitly justified. The main validity concern—that the full-precision complex model contains two real matrices per complex weight, so nominal 700M/1.3B parameter counts correspond to more real scalars than the LLaMA baselines—is a potential confound in the 'ceiling-raising' comparison, but it is a benchmark-design issue, not a circular reduction. The Limitations section acknowledges under-explored attention formulation, scaling-factor magnitude preservation, and hardware mismatch; none of these admissions implies that any result is true by construction. Therefore the circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption The complex-valued architecture has greater representational power than a real-valued architecture at the same nominal parameter count.
- standard math Accuracy(quant) = Accuracy(full-precision) - Error(quant) is a meaningful upper-bound relation.
- standard math The four states {±1, ±i} are information-theoretically optimal for 2-bit representation.
Cite this review
Pith. "Pith review of iFairy: the First 2-bit Complex LLM with All Parameters in $\{\pm1, \pm i\}$." pith.science (2026). https://pith.science/paper/HSISQUDB
@misc{pith2026250805571,
author = {Pith},
title = {Pith review of: iFairy: the First 2-bit Complex LLM with All Parameters in $\\pm1, \pm i\$},
year = {2026},
howpublished = {\url{https://pith.science/paper/HSISQUDB}},
note = {Machine review of arXiv:2508.05571}
}
abstract
Quantization-Aware Training (QAT) integrates quantization into the training loop, enabling LLMs to learn robust low-bit representations, and is widely recognized as one of the most promising research directions. All current QAT research focuses on minimizing quantization error on full-precision models, where the full-precision accuracy acts as an upper bound (accuracy ceiling). No existing method has even attempted to surpass this ceiling. To break this ceiling, we propose a new paradigm: raising the ceiling (full-precision model), and then still quantizing it efficiently into 2 bits. We propose Fairy$\pm i$, the first 2-bit quantization framework for complex-valued LLMs. Specifically, our method leverages the representational advantages of the complex domain to boost full-precision accuracy. We map weights to the fourth roots of unity $\{\pm1, \pm i\}$, forming a perfectly symmetric and information-theoretically optimal 2-bit representation. Importantly, each quantized weight has either a zero real or imaginary part, enabling multiplication-free inference using only additions and element swaps. Experimental results show that Fairy$\pm i$ outperforms the ceiling of existing 2-bit quantization approaches in terms of both PPL and downstream tasks, while maintaining strict storage and compute efficiency. This work opens a new direction for building highly accurate and practical LLMs under extremely low-bit constraints.
Reference graph
Works this paper leans on
-
[2]
In European conference on com- puter vision, 525–542
Xnor-net: Imagenet classification using binary convo- lutional neural networks. In European conference on com- puter vision, 525–542. Springer. Sakaguchi, K.; Bras, R. L.; Bhagavatula, C.; and Choi, Y
-
[2016]
arXiv preprint arXiv:1609.07843
Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843. Miao, X.; Oliaro, G.; Zhang, Z.; Cheng, X.; Jin, H.; Chen, T.; and Jia, Z. 2023. Towards efficient generative large language model serving: A survey from algorithms to systems. arXiv preprint arXiv:2312.15234. Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y .;...
arXiv 2023
-
[2021]
Communications of the ACM, 64(9): 99–106
Winogrande: An adversarial winograd schema chal- lenge at scale. Communications of the ACM, 64(9): 99–106. Scharnhorst, K. 2001. Angles in complex vector spaces. Acta Appl. Math., 69: 95–103. Team, M.; Xiao, C.; Li, Y .; Han, X.; Bai, Y .; Cai, J.; Chen, H.; Chen, W.; Cong, X.; Cui, G.; et al. 2025. MiniCPM4: Ultra-Efficient LLMs on End Devices. arXiv pre...
arXiv 2001
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.