Irrational Complex Rotations Empower Low-bit Optimizers

Ji-Rong Wen; Wayne Xin Zhao; Zhen Tian

REVIEW 5 major objections 5 minor 41 references

Irrational Complex Rotations Empower Low-bit Optimizers

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that any pair of parameters inside a disk of radius 2 can be stored as a single rotation angle, and that quantizing these angles to 3.32 bits preserves full optimizer accuracy.

desk verdict The paper's central theorem is false: e^{iθ}+e^{iπθ} traces a dense 1D curve, not the whole disk, so the lossless 3.32-bit compression guarantee collapses, even though the empirical method may work as a heuristic. read the letter →

arxiv 2501.12896 v1 pith:XK24KFLQ submitted 2025-01-22 cs.LG

Zhen Tian , Wayne Xin Zhao , Ji-Rong Wen This is my paper · ORCID

classification cs.LG

keywords optimizerstatecompressionlow-bitoptimizersirrationalcomplexrotationsrotationanglequantizationmemory-efficienttrainingAdamnon-uniformpi-Quant

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

$\pi$-Quant proposes to compress optimizer states — the momentum tensors kept by Adam and similar optimizers — by storing two real parameters as a single rotation angle. The paper claims a mathematical guarantee: for any pair $(x,y)$ inside the disk of radius 2, there is a real angle $\theta$ such that $x+iy=e^{i\theta}+e^{i\bar\pi\theta}$, so the pair can be reconstructed exactly from $\theta$. A system of geometric equations computes $\theta$ in linear time, and each angle is then quantized to a few decimal digits; at $\lambda=1$ the effective storage cost is about 3.32 bits per parameter. In experiments on a 1.1B-parameter language model and several other tasks, the method reports accuracy equal to or better than full-precision Adam while cutting GPU memory by about 40%. If the identity is exact as claimed, this would give a theoretically grounded way to halve optimizer-state size and push low-bit training below 4 bits without search-based quantization kernels.

What carries the argument

The load-bearing object is the complex rotation identity $x+iy=e^{i\theta}+e^{i\bar\pi\theta}$, where $\bar\pi$ is constructed from $\pi$ by shifting its digits so that $\bar\pi=10^{-\lambda}+10^{-2\lambda}\cdot 0.3589793238\ldots$; this digit layout lets one stored integer encode both the branch index $m$ and the fractional part $g$ of the angle. To solve for $\theta$, the paper gives a geometric system (Lemma 3.2) that forms the triangle of the two unit vectors and derives $\alpha=\arctan(y/x)$, $\beta=\arccos(\sqrt{x^2+y^2}/2)$, and then reduces the branch search to matching the fractional part of $m\bar\pi$ against $\Omega=(\alpha(1-\bar\pi)+\beta(1+\bar\pi))/(2\pi)$. The quantization step stores $\lfloor(\theta/2\pi)\cdot 10^\lambda\rfloor$, so the first $\lambda$ digits carry $m$ and the last $\lambda$ digits carry the remainder; the error analysis bounds the coordinate error by $\Delta\theta$ in each axis and the average error by $2(1+\bar\pi)\,10^{-\lambda}/\pi$.

What would settle it

Sample a large set of random points from the disk $x^2+y^2\le 4$, run the paper's Algorithm 1 with a large $\lambda$ (e.g., $\lambda=8$), reconstruct each point via Eq. (11), and measure the maximum Euclidean error between original and reconstructed values. If the error does not shrink toward zero as $\lambda$ grows for points that are not on the curve, the exact-representation claim fails; if it reaches machine precision for the sampled points, that claim passes this test.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is the identity $z=e^{i\theta}+e^{i\bar\pi\theta}$: with an irrational coefficient $\bar\pi$ in the second rotation, the curve traced by $\theta\mapsto e^{i\theta}+e^{i\bar\pi\theta}$ is non-periodic, and the paper claims it passes through every point of the disk $\|z\|\le 2$, so every two-dimensional pair $(x,y)$ can be assigned a unique angle $\theta$ with $x+iy=e^{i\theta}+e^{i\bar\pi\theta}$. From this identity the authors derive that a pair of real numbers can be stored as one angle, halving the element count of optimizer state tensors, and that the angle can be quantized to $\lambda$ decimal digits with a provable error bound, giving an effective bit-width of $3.32\lambda$ bits. The paper further claims that with $\lambda=1$ (a 3.32-bit state) training accuracy is fully preserved on language modeling and several downstream tasks, with memory savings of roughly 40% relative to full-precision Adam.

Load-bearing premise

The method assumes that every parameter pair stored in an optimizer state lies exactly on the curve traced by the two rotating unit vectors, and that the branch relations used to compute the angle recover the same pair when the angle is inverted; pairs not on that curve, or lying on a different branch, are not reconstructed without error.

Editorial extensions

If this is right

Optimizer state tensors drop to half their element count (two reals stored as one angle), and with the 3.32-bit angle format this yields roughly a 75% reduction in parameter scale.
At $\lambda=1$ the paper reports accuracy at or above full-precision Adam on language modeling and on ARC-Challenge, Hellaswag, Lambada, and PIQA, while GPU memory falls by about 40% (from 19.47 GB to 11.32 GB for a 1.1B-parameter model).
Because the method is arithmetic rather than search-based, it needs no custom GPU kernels and can set any bit-width simply by changing $\lambda$.
The same angle representation applies to any tensor that can be split into two equal halves, so the authors suggest it can compress model weights and, as future work, KV caches in transformers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Read as a compression codec rather than an exact bijection, the scheme is a member of a larger family: any two real values can be mapped to an angle on a dense curve, with the bit budget set by $\lambda$; this suggests applying the same idea to weight tensors, activations, or KV cache entries, not just momentum states.
The digit-shifting construction of $\bar\pi$ is one of infinitely many 'almost-rational' coefficients; choosing a different digit layout changes both the matching error for $m$ and the effective non-uniformity of the quantizer, so the coefficient itself becomes a tunable hyperparameter.
A testable extension is to measure the reconstruction error floor on random float pairs with large $\lambda$; the paper's error analysis assumes the pair lies on the curve, so random pairs would reveal whether the scheme behaves as exact compression or as a bounded-error approximate codec in practice.
The non-uniform error profile the authors plot (small errors near zero, larger at the periphery) matches a curve whose density is higher near the origin; this could be exploited by pairing the codec with per-tensor scales that push the value distribution toward the high-density region.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this paper's headline claim is wrong. Theorem 3.1 asserts that every point in the disk |z| ≤ 2 can be written uniquely as e^{iθ}+e^{iπθ}. That map is a continuous one-dimensional curve: it is dense when π is irrational, but its image has empty interior, so most points have no exact representation. The appendix 'proof' jumps from 'non-periodic' to 'traverses every point,' which is precisely the gap between dense and surjective. Lemma 3.2 also has a branch error: the relations θ = α−β and πθ = α+β only hold when cos(((π−1)/2)θ) > 0; when that cosine is negative the roles swap. The method never tracks the branch, so the reconstruction in Eq. (11) can output the wrong pair. What is genuinely new is the idea of using an irrational rotation coefficient to pack two real values into one angle. That is not present in the cited optimizer-quantization literature, and the masked-digit construction for approximating m is clever. The empirical evaluation on TinyLlama and other tasks follows a standard setup, and the reported numbers look competitive if the method is treated purely as a heuristic quantizer with a fixed codebook. But the paper frames everything around exact lossless compression, and without that the contribution is much weaker. The bit-width formula (3.32λ) only makes sense once you assume exact recovery, and the error bound in Eq. (13) misuses the triangle inequality. The experimental section lacks error bars, and the code link is unverified, but those are minor relative to the mathematical failure. My recommendation is clear: reject. The load-bearing theorem is false, and a serious referee would likely catch it quickly. I would desk reject and, if the authors resubmit, urge them to drop the exactness claim and present the method as an approximate quantization scheme with measured error bars.

Referee Report

5 major / 5 minor

Summary. The paper proposes π-Quant, a training-oriented optimizer-state compression method. It claims a new theorem that every complex number z with ||z||≤2 can be written uniquely as z=e^{iθ}+e^{iπ̄θ}; based on this it represents each pair of optimizer-state entries by one rotation angle θ, computes θ through a "geometric solution" (Lemma 3.2), quantizes θ to λ decimal digits, and reports 3.32-bit optimizer states with full accuracy on TinyLlama and several benchmark tasks. The appendix attempts to prove Theorem 3.1, Lemma 3.2, and the quantization-error bounds. The paper also provides code and a linear-complexity pipeline.

Significance. If the central theorem were true, representing two real parameters by one angle would halve the optimizer-state tensor size and could enable low-bit training, which would be an important result. The paper provides code, covers relevant baselines, and tests on a range of tasks. However, the central mathematical premise is false: the trajectory of f(θ)=e^{iθ}+e^{iπθ} is a dense one-dimensional curve, not the full disk, so exact pair-to-angle compression is impossible; the geometric branch used in Lemma 3.2 is wrong for a large fraction of representable points; and the proof in Appendix A contains an invalid inference. The experiments lack error bars and use a small number of training steps, so they cannot rescue the unsupported lossless and full-accuracy claims. The paper may describe a workable approximate quantizer, but the advertised theoretical guarantee and the 3.32-bit lossless claim require a fundamentally different argument.

major comments (5)

[Theorem 3.1 and Appendix A] The surjectivity claim of Theorem 3.1 is false. Writing f(θ)=e^{iθ}+e^{iπθ} in polar form gives f(θ)=2cos(((1−π)θ)/2)exp(i((1+π)θ)/2). The image is a real-analytic curve with empty interior; when π is irrational it is dense in the disk, but density does not imply surjectivity, and most points of the disk have no preimage. The proof in Appendix A explicitly uses the invalid inference that the trajectory "will traverse every point within the domain's circle, since it is non-closed and continuous." A standard counterexample is a dense curve on the torus, which is non-closed and continuous but not surjective. Consequently Eq. (3) and the reconstruction in Eq. (11) cannot guarantee exact recovery of an arbitrary pair (x,y), and the lossless 3.32-bit claim collapses.
[Lemma 3.2, Eq. (4), Appendix B] The system in Eq. (4) uses the wrong branch of the geometry. From the polar form, the argument of f(θ) is ((1+π)θ)/2 only when cos(((1−π)θ)/2)>0 and the arctangent branch is chosen appropriately; otherwise the argument is shifted by π. Correspondingly, the identities α−β=θ and α+β=πθ in Eq. (4) hold only on the branch where cos(((1−π)θ)/2)>0; on the branch where this cosine is negative, the roles are interchanged, giving α−β=πθ and α+β=θ (mod 2π). The method never tests or stores this branch, so Algorithm 1 and the reconstruction in Eq. (11) can return the wrong pair even for points that do lie on the curve. This is load-bearing because m and g in Eqs. (8) and (10) are computed from α−β under the false branch assumption.
[Appendix A, proof of Lemma A.1] The proof of Lemma A.1 is invalid at the step "Subtract Eq. (16) from Eq. (15)": Eq. (15) contains the original functions cosθ and sinθ, while Eq. (16) contains their derivatives, so subtracting the two systems does not produce Eq. (17). Moreover, even if f were non-periodic, non-periodicity of a bounded continuous curve does not imply surjectivity; the visualization in Figure 6 only shows dense-looking coverage over long θ-ranges and provides no evidence of exact coverage of every point. Since the proof of Theorem 3.1 rests on this inference, the central existence and uniqueness statement is unproved and in fact false.
[Section 3.3, Eqs. (12) and (13)] The claimed quantization-error advantage is arithmetically false. The text states "2 · (1 + ¯π) < π", but for ¯π≈π we have 2(1+π)≈8.28 > π, so the bound in Eq. (13) is larger than 10^{-λ}, not smaller. Additionally, the step "Since ||y′||<1 (See Eq. (9))" in the proof of Eq. (12) is wrong: y′=sinθ′+sinπ̄θ′ is a sum of two unit-magnitude terms and can be as large as 2, while Eq. (9) normalizes the stored parameters, not the trigonometric terms in Eq. (24). Therefore Eqs. (12) and (13) do not establish that the method has lower quantization error than traditional uniform quantization.
[Section 3.2, Eq. (10) and Table 3] The claimed 3.32-bit setting is not a quantization bit-width in the usual sense. Eq. (10) stores m and g as two separate λ-digit decimal numbers inside one θ, so the total storage is 2λ decimal digits per θ, i.e., 3.32λ bits per parameter because one θ covers two parameters. The choice λ=1 gives 3.32 bits by construction, not by an information-theoretic analysis of the state distribution. The memory comparison in Table 3 also mixes this decimal-digit representation with FP32/FP16 baselines; the paper should report the actual byte-level layout before claiming a 41.9% memory reduction from the 3.32-bit setting.

minor comments (5)

[Eq. (12)] The notation ||∆x|| for a scalar quantity is misleading; use |∆x| throughout the error-bound discussion.
[Appendix D and Table 4] There are typos in the appendix and table headers: "Appedix D" should be "Appendix D" and "belu scores" should be "BLEU scores."
[Section 2] The sentence "It is represented by multiplying a complex number of the form i.e., cos θ + i sin θ" contains a duplicated "i.e.," and should be rephrased.
[Algorithm 2] The initialization m0=Quant(0) and v0=Quant(0) is underspecified: Algorithm 1 expects a tensor with nontrivial X and Y splits, and the scaling step w=max(|X,Y|) is undefined for a zero tensor; specify how zero states are quantized.
[Eqs. (5)-(8)] The fractional-part notation {·} is used before being defined; define it explicitly when it first appears in Eq. (5).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation chain is self-contained; its central failure is mathematical invalidity, not definitional or self-citational circularity.

full rationale

Walking the derivation chain: Theorem 3.1 (Eq. 3) is a mathematical surjectivity claim proved in Appendix A. Its premises (continuity, boundedness, non-periodicity) are not stated in terms of its conclusion, and the proof's jump from 'non-closed and continuous' to 'traverse every point' is an invalid inference, not an equivalence-by-definition. Lemma 3.2 (Eq. 4) derives alpha-beta and alpha+beta from the geometry of two unit phasors, and Eq. (11) then inverts Eq. (3) as the intended forward/inverse pair; this is not a predicted quantity forced by a fitted parameter. The 3.32-bit figure is arithmetic from the representation: lambda decimal digits of m plus lambda decimal digits of g stored per two parameters gives lambda*log2(10) ≈ 3.32*lambda bits (Eq. 10); it is not an empirical target that the method was fitted to. Section 5 cites the authors' earlier rotation work (Tian et al., 2023, 2024a,b) as related work only; no load-bearing argument is imported from those citations. The paper's demonstrated defect is the mathematical invalidity of the surjectivity proof and the branch assumptions in Lemma 3.2, which is a correctness issue rather than a circularity issue. I find no step where an input is defined in terms of an output, no fitted parameter renamed as a prediction, and no self-citation chain that forces the central claim.

Assumptions & free parameters 1 free parameters · 4 assumptions · 1 invented entities

The central claim rests on two unsupported premises: the false surjectivity of the irrational rotation curve and the fixed branch assignment in Lemma 3.2. The free parameter λ controls the bit-width but is not fitted to the evaluation targets. No external evidence supports the invented π̄ construction.

free parameters (1)

λ = 1, 2, 4 (bit-widths 3.32, 6.64, 13.28)
Quantization precision hyperparameter that sets how many decimal digits are kept for the rotation angle; chosen by hand, not fitted, but it directly determines the claimed bit-width and the average quantization error bound.

assumptions (4)

ad hoc to paper The curve f(θ)=e^{iθ}+e^{iπθ} is surjective onto the disk because it is continuous and non-periodic (Appendix A).
Denseness does not imply surjectivity for a 1D curve in a 2D disk; this is the central false premise used to justify Theorem 3.1.
ad hoc to paper The system θ=α−β and πθ=α+β holds for all representable points (Lemma 3.2).
The correct assignment depends on the branch; one of the two angles is α+β and the other α−β. The lemma fixes the branch incorrectly.
domain assumption Parameter and momentum distributions are approximately Gaussian centered at zero, justifying the non-uniform precision allocation (Section 3.3).
The claimed advantage of non-uniform error near zero relies on this empirical distributional assumption for the tasks tested.
standard math Euler's formula and standard trigonometric identities.
Used throughout the derivation and accepted as background.

invented entities (1)

π̄, the masked-digit irrational number derived from π (Eq. 7)
purpose: Make the fractional-part equation {mπ̄}={Ω} solvable by reading the first λ digits of the target, enabling linear-time angle computation.
This is an internal construction with no falsifiable handle outside the method. It is not a physical entity, but it is a newly introduced mathematical object on which the compression scheme depends.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Irrational Complex Rotations Empower Low-bit Optimizers." pith.science (2026). https://pith.science/paper/XK24KFLQ

@misc{pith2026250112896,
  author       = {Pith},
  title        = {Pith review of: Irrational Complex Rotations Empower Low-bit Optimizers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XK24KFLQ}},
  note         = {Machine review of arXiv:2501.12896}
}

abstract

In this paper, we propose a novel optimizer state compression algorithm, namely $\pi$-Quant, which leverages the properties of irrational numbers (e.g., $\pi$) for memory-efficient training. The core idea is based on our mathematical findings, which show that a pair of parameters can be represented by a single rotation angle using the complex rotation scheme. Building on this insight, we map the parameters into a complex space and perform quantization using the corresponding rotation angles. To efficiently integrate it into optimization process, we develop an efficient system of geometric equations that computes the precise rotation angles with linear complexity. We evaluate $\pi$-Quant on a wide range of tasks. Our experiments show that it can reduce the bit-width of parameters to 3.32-bit, achieving a 75% reduction in parameter scale and a 40% decrease in GPU memory usage, all while maintaining full accuracy.

Figures

Figures reproduced from arXiv: 2501.12896 by the authors.

**Figure 1.** The overall framework of π-Quant. 3.1. Parameter Compression with Irrational Rotations The key point of our approach is a novel data compression theorem, which enables the precise representation of two variables using a univariate rotation equation with an irrational number as the coefficient. We first present the mathematical foundation behind this theorem, followed by the instantiation of our idea. Data Compressi… view at source ↗

**Figure 2.** The error distribution (top) and precision distribution (bottom) between our method and traditional methods [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Loss comparison between FP32 and our approach. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Ablation Study of the π setting in our approach. typically shorter than SGD’s, demonstrating how Adam’s momentum accelerates convergence. Our method achieves the shortest paths across all initialization points, particularly at start point 4 where it successfully finds …

**Figure 5.** Figure 5: Visualizing the gradient descent process. 5. Related Work Model Quantification. Large language models (Zhao et al., 2023) have achieved outstanding performance across various tasks. However, the extensive parameters in LLMs results in substantial memory usage, signifi…

**Figure 6.** Figure 6: Visualization of the trajectory of Function Eq. (14). B. Proof of Lemma 3.2. 𝒆 𝒊𝜽 𝒆 𝒊𝝅𝜽 (𝒙, 𝒚) 𝜽 𝝅𝜽 𝜷 𝜶 𝑋-axis 𝑌-axis 𝒍𝟏 𝒍𝟐 [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

**Figure 7.** Figure 7: Geometric relationship in x + iy = e ıθ + e iπθ As shown in [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 13 canonical work pages

[1]

\ TensorFlow \ : a system for \ Large-Scale \ machine learning

Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. \ TensorFlow \ : a system for \ Large-Scale \ machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pp.\ 265--283, 2016

work page 2016
[2]

Skyformer: Remodel self-attention with gaussian kernel and nystr " om method

Chen, Y., Zeng, Q., Ji, H., and Yang, Y. Skyformer: Remodel self-attention with gaussian kernel and nystr " om method. Advances in Neural Information Processing Systems, 34: 0 2122--2135, 2021

work page 2021
[3]

8-bit approximations for parallelism in deep learning

Dettmers, T. 8-bit approximations for parallelism in deep learning. arXiv preprint arXiv:1511.04561, 2015

arXiv 2015
[4]

8-bit optimizers via block-wise quantization

Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, L. 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:2110.02861, 2021

arXiv 2021
[5]

Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 0 30318--30332, 2022

2022
[6]

Spqr: A sparse-quantized representation for near-lossless llm weight compression

Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078, 2023

arXiv 2023
[7]

Gptq: Accurate post-training quantization for generative pre-trained transformers

Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022

arXiv 2022
[8]

A framework for few-shot language model evaluation, 07 2024

Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL http...

arXiv 2024

Show all 41 references

[9]

W., and Keutzer, K

Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., and Keutzer, K. A survey of quantization methods for efficient neural network inference. In Low-Power Computer Vision, pp.\ 291--326. Chapman and Hall/CRC, 2022

2022
[10]

Samsum corpus: A human-annotated dialogue dataset for abstractive summarization

Gliwa, B., Mochol, I., Biesek, M., and Wawer, A. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. arXiv preprint arXiv:1911.12237, 2019

1911 arXiv
[11]

Aptq: Attention-aware post-training mixed-precision quantization for large language models

Guan, Z., Huang, H., Su, Y., Huang, H., Wong, N., and Yu, H. Aptq: Attention-aware post-training mixed-precision quantization for large language models. In Proceedings of the 61st ACM/IEEE Design Automation Conference, pp.\ 1--6, 2024

2024
[12]

Harper, F. M. and Konstan, J. A. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5 0 (4): 0 1--19, 2015

2015
[13]

Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1412.6980

2015 arXiv
[14]

Learning multiple layers of features from tiny images

Krizhevsky, A. Learning multiple layers of features from tiny images. pp.\ 32--33, 2009. URL https://www.cs.toronto.edu/ kriz/learning-features-2009-TR.pdf

2009
[15]

Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models

Lee, C., Jin, J., Kim, T., Kim, H., and Park, E. Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 13355--13364, 2024

2024
[16]

Memory efficient optimizers with 4-bit states

Li, B., Chen, J., and Zhu, J. Memory efficient optimizers with 4-bit states. Advances in Neural Information Processing Systems, 36: 0 15136--15171, 2023

2023
[17]

and Li, J

Li, X. and Li, J. Angle-optimized text embeddings. arXiv preprint arXiv:2309.12871, 2023

2023 arXiv
[18]

Awq: Activation-aware weight quantization for on-device llm compression and acceleration

Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 0 87--100, 2024

2024
[19]

Learning long-range spatial dependencies with horizontal gated-recurrent units, 2019

Linsley, D., Kim, J., Veerabadran, V., and Serre, T. Learning long-range spatial dependencies with horizontal gated-recurrent units, 2019. URL https://arxiv.org/abs/1805.08315

2019 arXiv
[20]

Llm-qat: Data-free quantization aware training for large language models

Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., and Chandra, V. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888, 2023

2023 arXiv
[21]

L., Daly, R

Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C. Learning word vectors for sentiment analysis. In Lin, D., Matsumoto, Y., and Mihalcea, R. (eds.), Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language ...

2011
[22]

and Bowman, S

Nangia, N. and Bowman, S. R. Listops: A diagnostic dataset for latent tree learning, 2018. URL https://arxiv.org/abs/1804.06028

2018 arXiv
[23]

Pytorch: An imperative style, high-performance deep learning library

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019

2019
[24]

Yarn: Efficient context window extension of large language models

Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023

2023 arXiv
[25]

R., Muthukrishnan, P., and Qazvinian, V

Radev, D. R., Muthukrishnan, P., and Qazvinian, V. The ACL A nthology network corpus. In Kan, M.-Y. and Teufel, S. (eds.), Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries ( NLPIR 4 DL ) , pp.\ 54--61, Suntec City, Singapore, Augus...

2009
[26]

W., Potapenko, A., Jayakumar, S

Rae, J. W., Potapenko, A., Jayakumar, S. M., Hillier, C., and Lillicrap, T. P. Compressive transformers for long-range sequence modelling. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020

2020
[27]

Pb-llm: Partially binarized large language models

Shang, Y., Yuan, Z., Wu, Q., and Dong, Z. Pb-llm: Partially binarized large language models. arXiv preprint arXiv:2310.00034, 2023

2023 arXiv
[28]

Roformer: Enhanced transformer with rotary position embedding

Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

2024
[29]

A length-extrapolatable transformer

Sun, Y., Dong, L., Patra, B., Ma, S., Huang, S., Benhaim, A., Chaudhary, V., Song, X., and Wei, F. A length-extrapolatable transformer. arXiv preprint arXiv:2212.10554, 2022

2022 arXiv
[30]

An image patch is a wave: Phase-aware vision mlp

Tang, Y., Han, K., Guo, J., Xu, C., Li, Y., Xu, C., and Wang, Y. An image patch is a wave: Phase-aware vision mlp. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 10935--10944, 2022

2022
[31]

Long range arena: A benchmark for efficient transformers, 2020

Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., and Metzler, D. Long range arena: A benchmark for efficient transformers, 2020. URL https://arxiv.org/abs/2011.04006

2020 arXiv
[32]

X., Wen, J.-R., and Cao, Z

Tian, Z., Bai, T., Zhao, W. X., Wen, J.-R., and Cao, Z. Eulernet: Adaptive feature interaction learning via euler's formula for ctr prediction. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.\ 1376--1385, 2023

2023
[33]

X., and Wen, J.-R

Tian, Z., Shi, Y., Wu, X., Zhao, W. X., and Wen, J.-R. Rotative factorization machines. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp.\ 2912--2923, 2024 a

2024
[34]

X., Zhang, C., Zhao, X., Ma, Z., and Wen, J.-R

Tian, Z., Zhao, W. X., Zhang, C., Zhao, X., Ma, Z., and Wen, J.-R. Eulerformer: Sequential user behavior modeling with complex vector attention. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.\ 1619--1628, 2024 b

2024
[35]

Z., et al

Vanhoucke, V., Senior, A., Mao, M. Z., et al. Improving the speed of neural networks on cpus. In Proc. deep learning and unsupervised feature learning NIPS workshop, volume 1, pp.\ 4, 2011

2011
[36]

Bitnet: Scaling 1-bit transformers for large language models

Wang, H., Ma, S., Dong, L., Huang, S., Wang, H., Ma, L., Yang, F., Wang, R., Wu, Y., and Wei, F. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.11453, 2023

2023 arXiv
[37]

Smoothquant: Accurate and efficient post-training quantization for large language models

Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp.\ 38087--38099. PMLR, 2023

2023
[38]

Zeroquant: Efficient and affordable post-training quantization for large-scale transformers

Yao, Z., Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., and He, Y. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35: 0 27168--27183, 2022

2022
[39]

Tinyllama: An open-source small language model

Zhang, P., Zeng, G., Wang, T., and Lu, W. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024

2024 arXiv
[40]

X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al

Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023

2023 arXiv
[41]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

\ TensorFlow \ : a system for \ Large-Scale \ machine learning

Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. \ TensorFlow \ : a system for \ Large-Scale \ machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pp.\ 265--283, 2016

work page 2016

[2] [2]

Skyformer: Remodel self-attention with gaussian kernel and nystr " om method

Chen, Y., Zeng, Q., Ji, H., and Yang, Y. Skyformer: Remodel self-attention with gaussian kernel and nystr " om method. Advances in Neural Information Processing Systems, 34: 0 2122--2135, 2021

work page 2021

[3] [3]

8-bit approximations for parallelism in deep learning

Dettmers, T. 8-bit approximations for parallelism in deep learning. arXiv preprint arXiv:1511.04561, 2015

arXiv 2015

[4] [4]

8-bit optimizers via block-wise quantization

Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, L. 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:2110.02861, 2021

arXiv 2021

[5] [5]

Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 0 30318--30332, 2022

2022

[6] [6]

Spqr: A sparse-quantized representation for near-lossless llm weight compression

Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078, 2023

arXiv 2023

[7] [7]

Gptq: Accurate post-training quantization for generative pre-trained transformers

Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022

arXiv 2022

[8] [8]

A framework for few-shot language model evaluation, 07 2024

Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL http...

arXiv 2024