Pith. sign in

REVIEW 3 major objections 6 minor 75 references

Information-computation trade-offs in non-linear transforms

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Applying the LZ78 transform to any zero-order probability model yields a universal predictor that matches finite-state compressibility on every individual sequence.

desk verdict A well-written survey with a plausible but unproven LZ78 universality theorem; the appendix proof has a load-bearing gap that needs repair before the central claim can be trusted. read the letter →

arxiv 2506.15948 v1 pith:CRFWW37E submitted 2025-06-19 cs.IT eess.IVmath.IT

Connor Ding , Abhiram Rao Gorle , Jiwon Jeong , Naomi Sagan , Tsachy Weissman This is my paper
classification cs.ITeess.IVmath.IT MSC 94A1768P3094A15
keywords Non-lineartransformsrate-distortiontheoryimplicitneuralrepresentationstextualLempel-Zivuniversalitycompression-computationtrade-offmodelpruninggenerativeAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper makes the case that non-linear transforms are best understood by what they cost in computation and what they buy in information. Its central result is a 'LZ78 transform': take any sequential probability assignment that is only good enough to capture zero-order statistics, run a fresh copy of it at every node of the LZ78 parse tree, and the resulting assignment becomes universal—on every individual sequence it asymptotically attains the optimal finite-state log loss $\lambda(x)=\mu(x)$. If that holds, a model as simple as the add-gamma empirical distribution (3.1) becomes a universal predictor at $O(n)$ time and $O(n/\log n)$ memory. The paper also compares two image-domain non-linear transforms and proposes text-and-sketch guided denoising, presenting all three as points on a common information-computation trade-off.

What carries the argument

The LZ78 transform is a wrapper around a base sequential probability assignment. It builds one copy of the base SPA per node of the LZ78 prefix tree, and each copy only ever operates on the subsequence of symbols whose LZ78 context is that node; conditioning on the LZ context is what injects long memory into a zero-order model. The load-bearing structural facts are that every phrase is a previously seen phrase plus one symbol, that the number of phrases is sublinear—$C(x^n)=O(n/\log n)$—and that the average LZ78 context length grows without bound, so the transformed model effectively conditions on increasingly long prefixes.

What would settle it

Look for a sequence whose LZ78 parse contains a phrase with more than $M$ symbols at nodes whose visit count is below $M$ for some threshold $M$; if such a sequence exists, the $\epsilon/3$ decomposition in Lemma 3.12 fails as stated and the printed proof of Theorem 3.11 collapses.

Watch

Extended reading notes

Core claim

The central object is the transform $T_{LZ}\{q\} = \hat{q}$ with $\hat{q}(x_t \mid x^{t-1}) = q(x_t \mid Y\{x^{t-1}, z_c(x^{t-1})\})$, where $Y\{x^{t-1}, z\}$ is the subsequence of past symbols that were parsed at LZ78 node $z$. Theorem 3.11 states that if the base SPA $q$ satisfies $\limsup_n \max_{x^n} ( \frac{1}{n}\log \frac{1}{q(x^n)} - \mu_0(x^n) ) \le 0$, then the transformed SPA is universal: for every individual sequence, $\limsup_n \frac{1}{n} \log \frac{1}{\hat{q}(x^n)} \le \lambda(x) = \mu(x)$, the finite-state compressibility. The proof routes through two lemmas: Lemma 3.12 bounds the transformed log loss by the length-weighted zero-order costs at the LZ78 nodes, and Lemma 3.13 shows that weighted sum is asymptotically no larger than the optimal Markov log loss. The concrete instantiation is the add-gamma empirical SPA of (3.1), a Dirichlet Bayesian mixture that satisfies the hypothesis.

Load-bearing premise

The theorem's stated hypothesis is only that the base model handles zero-order statistics, but the proof as printed also depends on an unproven combinatorial fact about the LZ78 parse tree—that few symbols of any phrase sit at lightly visited nodes—and the appendix garbles the definition of that visit count, so the flagship claim cannot yet be checked from the text alone.

Editorial extensions

If this is right

  • Any base SPA satisfying the zero-order condition, including the add-gamma empirical model of (3.1), produces a universal SPA under the LZ78 transform.
  • Via arithmetic coding, the transformed SPA becomes a lossless compressor whose asymptotic rate on every individual sequence is no worse than the sequence's finite-state compressibility $\lambda(x)$.
  • When the base model has $O(n)$ time and $O(1)$ memory, the transformed model has $O(n)$ time and $O(n/\log n)$ memory.
  • The transform carries over to concrete tasks: classification by per-class log loss, universal discrete filtering with causal, delayed, and look-ahead estimators, and lightweight symbolic-music generation, with the empirical results reported in the paper.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: if the proof gap in Lemma 3.12 is repaired, the same wrapper argument may extend to base models that satisfy the zero-order condition only on a restricted source class, giving source-specific universality at even lower cost.
  • Inference: the complexity bound $O(n/\log n)$ memory treats the tree as static; an online setting where new phrases keep being added would need to account for the memory of all node-local SPA copies, and quantifying that drift is a natural next step.
  • Inference: the textual-denoising pipeline suggests a testable extension: measure the perception-distortion frontier of text-and-JPEG-conditioned generation against classical denoisers on the three noise models, using the trade-off cited in the paper rather than pixel error alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper is a broad, three-part study of nonlinear transforms in compression and sequential modeling. Sections 1 and 2 present exploratory empirical work: a comparison of implicit neural representations (INRs) versus 2D Gaussian Splatting for image compression, and a textual transform for ultra-low-rate compression and text-guided denoising using off-the-shelf text-to-image models. Section 3 contains the paper's central theoretical contribution: the definition of an LZ78 transform T_LZ that maps any sequential probability assignment (SPA) satisfying a zero-order optimality condition into a universal SPA. Theorem 3.11 claims that if a base SPA asymptotically achieves zero-order empirical entropy on every sequence, then its LZ78 transform achieves the finite-state compressibility on every individual sequence. The paper also gives complexity bounds, connections to a known Dirichlet-mixture family, and applications to classification, universal filtering, and symbolic music generation (LZMidi).

Significance. If Theorem 3.11 is correct, it is a valuable and clean generalization of LZ78-based universal SPAs: it identifies a precise, parameter-free hypothesis on the base SPA, yields an explicit transform, and gives concrete O(n) time and O(n/log n) memory bounds for simple base models. The disclosure that the Dirichlet-mixture special case coincides with the SPA family of [48] is useful and appropriately credited. The complexity analysis and the breadth of applications (genomics classification, universal filtering, music generation) give the paper a wide potential audience. However, the proof of the flagship theorem has load-bearing gaps in Appendix A, and the proof of Theorem 4.5 in Appendix B contains an unsupported count. The empirical sections are largely descriptive and preliminary, so the main value of the paper rests on the theoretical Section 3.

major comments (3)
  1. [Appendix A, Lemma 3.12] The inequality direction in part A is reversed. The text reads 'for m_zt > M, xi(m_zt) > eps/3', but the definition of xi and the choice delta = eps/3 give xi(m) < eps/3 for all sufficiently large m. The subsequent bound on the T_A sum uses the wrong direction. In addition, part B defines m_z as 'the number of nodes in the sub-tree for which z is the root', which contradicts Definition 3.4 and the notation m_z = |Y{x^n,z}| used earlier in the same proof. These local errors prevent the eps/3 decomposition from being verified as printed, although the intended argument appears repairable.
  2. [Appendix A, Lemma 3.13] Equality (a) is not justified by Remark 3.8. The subsequence v^{m_ell}_ell consists of symbols whose LZ78 context has length ell, but when computing mu_ell(v), the length-ell contexts are the preceding ell symbols inside v, not the original LZ78 contexts of x^n. For example, for x^n=010100 the LZ78 phrases are 0,1,01,00; with ell=1 the left side of (a) equals 2*H(1/2)=2, while the right side is 0 because v=(1,0) has mu_1(v)=0 under the paper's convention that the first k symbols are free. The subsequent Jensen step therefore upper-bounds a quantity different from (*), and the claimed o(1) argument for the finite-state compressibility mu(x) does not follow from the text. Since this is the step connecting the LZ78 context-weighted zero-order term to mu(x), the proof of Theorem 3.11 is incomplete as printed.
  3. [Appendix B, Fact B.5] The claim that ell(z^m_t,m,t) = C_m(Y^{t-1}) + O(1) is not established and is in general false. Node z^m_t is visited only when Y^{t-1} occurs at a phrase boundary of the LZ78 parse, not at every occurrence of Y^{t-1} in the raw sequence; for i.i.d. sequences these counts can differ by a factor that grows with n. Consequently Fact B.5 and Corollary B.7, which support Theorem 4.5, need a different proof. The theorem may be true, but the argument as written does not prove it.
minor comments (6)
  1. [Section 2(c)(ii)] The word 'noie' should be 'noise' in the sentence describing the generated images.
  2. [Section 1(b)(iv)] The caption 'Quantizing Guassian splat parameters' contains a typo: 'Guassian' should be 'Gaussian'.
  3. [Section 4(c)(ii)] Several typos appear in this subsection: 'micture' should be 'mixture', 'or training' should be 'For training', and 'the the precise theorem' should be 'the precise theorem'. Also, 'training time hers' should be 'training time here'.
  4. [Section 4(c)(ii)] In the discussion of Table 5, the baseline is called 'D3PM' in the text but 'ASD3PM' in the table and elsewhere; the notation should be made consistent.
  5. [References] References [57] and [73] are the same Ziv-Merhav paper and should be merged or cross-referenced; the duplicated entry should be removed.
  6. [Section 4(c)] The FLOP comparison mixes a measured profile of the diffusion baseline with an analytical worst-case bound for LZMidi; the text should state clearly that the two numbers are not obtained on the same basis.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the flagship theorem's zero-order hypothesis does not contain its finite-state universality conclusion, and the companion-work self-citations are background or example material, not load-bearing.

full rationale

The paper's central derivation is not circular. Theorem 3.11 assumes only that the base SPA asymptotically matches zero-order empirical entropy on every sequence; the conclusion is finite-state universality of the LZ78-transformed SPA, and no definition or equation identifies the two. Lemma 3.12 applies the assumption node-wise. Lemma 3.13 is meant to show the context-weighted zero-order entropy of the LZ78 parse is bounded by μ(x); this is the substantive LZ78-optimality step. The proof of Lemma 3.13 has a serious apparent gap — equality (a) would require the subsequence v^{m_ℓ}_ℓ to have the original LZ contexts as its own length-ℓ contexts, which is false in general — but a false or missing derivation is a correctness/verifiability problem, not a circular reduction. Citations to the authors' companion paper [48] for λ=μ equivalence, for Dirichlet-mixture SPAs satisfying the hypothesis, and for the LZ78-SPA family are self-citations, but they are background facts or examples, not assumptions of Theorem 3.11. The proof's remaining ingredients (C(x^n)=o(n), phrase counts) come from the external LZ78 paper [38]. The empirical sections report experiments from [51,52,53] rather than deriving their results from this paper's theorems. I therefore find no step in which a claimed prediction is equivalent by construction to its inputs; the score reflects only the minor, non-load-bearing presence of companion-work citations.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The LZ78 transform theorem is parameter-free in itself: its only real input is the hypothesis that the base SPA attains zero-order empirical entropy uniformly, and the background facts (sublinear phrase counts, lambda = mu) are standard cited results. The free parameters listed here enter the application sections rather than the theorem: gamma, temperature, and top-K are fitted by sweeps in the LZMidi and genomics experiments, w0 is a hand-set SIREN constant, and the denoising controls are hand-chosen. The textual denoising pipeline rests on an unvalidated ad hoc assumption about caption-to-image fidelity. No invented entities (particles, forces, dimensions) are postulated.

free parameters (4)
  • Dirichlet parameter gamma for the LZ78-SPA = 5e-5 (LZMidi); grid {0.1, 0.33, 0.5, 0.75, 1, 3, 5} (genomics)
    In Section 4.c(ii) gamma is chosen by an Optuna sweep minimizing Wasserstein distance; in Section 4.a(i) it is chosen by validation log loss. The core Theorem 3.11 holds for any gamma > 0, so this free parameter affects only the application claims.
  • Sampling heuristics for LZMidi = Temperature 0.8, top-K 8
    Selected via Optuna (Section 4.c(ii)); these heuristics shape the generated sequences and enter all reported FAD, WD, and KL comparisons, so the quality claims are conditional on them.
  • SIREN frequency scaling w0 = 30
    Set following prior work [4] in Section 1.a(i); it controls the spectral bandwidth of the SIREN and thus the depth-versus-PSNR and pruning-sensitivity findings.
  • Denoising pipeline controls = Prompt strength 8, canny control strength 0.5, JPEG quality 0-95
    Hand-chosen in Section 2.c(ii)-(iii); the qualitative denoising results depend on these settings, on the unspecified text-to-image model version, and on an unspecified same random seed.
assumptions (6)
  • standard math C(x^n)/n -> 0: the number of LZ78 phrases is sublinear in the sequence length, uniformly over individual sequences.
    Invoked in Lemma 3.13 ('[38] states that... C(x^n)/n = o(1)') and in the complexity bound (3.2); standard result of Ziv and Lempel [38].
  • standard math lambda(x) = mu(x): finite-state log loss equals the optimal Markov log loss for every individual sequence.
    Stated in Definitions 3.6 to 3.7 and Remark 3.9, citing [48]; this equivalence defines the universality target in Definition 3.10.
  • domain assumption The base SPA q satisfies lim sup max_{x^n} (1/n log(1/q(x^n)) - mu_0(x^n)) <= 0.
    Hypothesis of Theorem 3.11; satisfied by the add-gamma empirical SPA (3.1) and, per [49,48], by Dirichlet mixtures under regularity conditions. If this fails, universality is not claimed.
  • domain assumption Each LZ78 tree node's SPA converges to the source's conditional probabilities (Fact B.5, Corollary B.7).
    Used in Appendix B for Theorem 4.5: node-level empirical frequencies converge almost surely for i.i.d. training sequences; the theorem's validity for non-i.i.d. sources is not established.
  • ad hoc to paper LLM captions identify noise, and text-to-image models regenerate the clean image from filtered text plus a JPEG-compressed conditioning image.
    Section 2.c: the denoising pipeline presupposes this fidelity; the paper provides only qualitative figures and defers quantitative evaluation, so the premise is unvalidated.
  • domain assumption The observation channel is a discrete memoryless channel with invertible transition matrix Pi.
    Section 4.b(i): the Bayes response F(P, Pi, z) and the excess-loss bounds (Theorems 4.2-4.4) require invertibility; carried over from the companion paper [52].

how reviews work

0 comments
Cite this review

Pith. "Pith review of Information-computation trade-offs in non-linear transforms." pith.science (2026). https://pith.science/paper/CRFWW37E

@misc{pith2026250615948,
  author       = {Pith},
  title        = {Pith review of: Information-computation trade-offs in non-linear transforms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CRFWW37E}},
  note         = {Machine review of arXiv:2506.15948}
}
read the original abstract

In this work, we explore the interplay between information and computation in non-linear transform-based compression for broad classes of modern information-processing tasks. We first investigate two emerging nonlinear data transformation frameworks for image compression: Implicit Neural Representations (INRs) and 2D Gaussian Splatting (GS). We analyze their representational properties, behavior under lossy compression, and convergence dynamics. Our results highlight key trade-offs between INR's compact, resolution-flexible neural field representations and GS's highly parallelizable, spatially interpretable fitting, providing insights for future hybrid and compression-aware frameworks. Next, we introduce the textual transform that enables efficient compression at ultra-low bitrate regimes and simultaneously enhances human perceptual satisfaction. When combined with the concept of denoising via lossy compression, the textual transform becomes a powerful tool for denoising tasks. Finally, we present a Lempel-Ziv (LZ78) "transform", a universal method that, when applied to any member of a broad compressor family, produces new compressors that retain the asymptotic universality guarantees of the LZ78 algorithm. Collectively, these three transforms illuminate the fundamental trade-offs between coding efficiency and computational cost. We discuss how these insights extend beyond compression to tasks such as classification, denoising, and generative AI, suggesting new pathways for using non-linear transformations to balance resource constraints and performance.

Figures

Figures reproduced from arXiv: 2506.15948 by the authors.

Figure 1
Figure 1. Example Kodak image 1 (kodim01) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Reconstructed images with INR of different layer sizes [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 5
Figure 5. Convergence plot [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figures from the paper (17 more)
Figure 6
Figure 6. Figure 6: Training and Inference time images, effectively functioning as a nonlinear form of transform coding into what we may refer to as the neural domain. Once an image has been fit into this neural representation, conventional neural network compression techniques—such as qu…
Figure 7
Figure 7. Figure 7: Replicating COIN [4]: Original vs. Reconstructed (32 bits) vs. Quantized (16 bits) (iii) Improving COIN–Beyond Basic Quantization While the COIN framework [4] introduced a novel paradigm for image compression by transforming data from the pixel domain into the neural d…
Figure 8
Figure 8. Figure 8: Original Image, Full Model Reconstruction, Quantization Magnitude Pruning, and SuRP at two different stages encoding time and providing interpretability of the transformation. Building upon 3D Gaussian Splatting developed for real-time neural rendering [12], Zhang et a…
Figure 9
Figure 9. Figure 9: Example Splats with the same position but different covariance matrices and color coefficients We executed the scripts from [2] on an example image in the Kodak dataset and observe the following [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Fitted image with different numbers of Gaussian Splats [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 12
Figure 12. Figure 12: Convergence plot across different iterations of Gaussians [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 13
Figure 13. Figure 13: Encoding and Decoding time of Gaussian splats, as a function of number of Gaussians. percentage of the 20000 Gaussians fitted to an image with no further fine-tuning or adjustment, as shown in [PITH_FULL_IMAGE:figures/full_fig_p010_13.png]
Figure 14
Figure 14. Figure 14: Pruning Different % of Gaussians No Quantization (32 bits) 4-bits for all 8-bits for all 12-bits for all 16-bits for all 16 bits for position, 4 bits for rest 16 bits for position, 8 bits for rest 16 bits for position, 12 bits for rest Image fitted with 16384 Gaussian…
Figure 16
Figure 16. Figure 16: Image captions for noisy images generated by ChatGPT 4o. Red colored text, marked by human, describes the noise of the images. of noise on image - Gaussian, Poisson, and salt-and-pepper noise. For Gaussian noise, which is a typical and widely studied type in image den…
Figure 17
Figure 17. Figure 17: Results of denoising guided by text. Three columns on the right shows the denoised image generated by different text-to-image generation models. strength of 8 to keep the image realistic while maintaining the semantics from the image caption. All images from Titan Ima…
Figure 18
Figure 18. Figure 18: Images compressed using JPEG with different image quality. image is preserved. This preservation of global structure suggests that JPEG-compressed images can serve as effective conditioning images for text-to-image generation models. Building on our previous experimen…
Figure 19
Figure 19. Figure 19: Results of denoising guided by text and lossily compressed sketch. Three columns on the right show the denoised image generated with different conditioning images, which are the noisy images compressed using JPEG with the indicated image quality. Signal-to-Noise Ratio…
Figure 20
Figure 20. Figure 20: LZ78-based Classifier via training an LZ78-SPA on Labeled Data (diagram from [51]) In [48], preliminary experiments are conducted across both image and text domains, using the LZ78 transform of a Bayesian mixture SPA under the Dirichlet(0.1, . . . , 0.1) prior (see [4…
Figure 21
Figure 21. Figure 21: Radar Plot of LZ-Transform-based Genomics classification, compared with DNABERT-2 (figure from [51]) 128× faster inference. Even on the three tasks where DNABERT-2 retains an edge (two promoter￾detection variants and splice site detection), LZ78’s accuracy deficit (13…
Figure 22
Figure 22. Figure 22: MSE loss of universal filter, increasing the number of Monte Carlo steps (figure from [52]) [PITH_FULL_IMAGE:figures/full_fig_p030_22.png]
Figure 23
Figure 23. Figure 23: Sample Midi File from the Lakh MIDI Dataset Baseline Model: The ASD3PM (Absorbing State Denoising Diffusion Probabilistic Model) from [65], a recent state-of-the-art model for generation of symbolic music is adopted as the benchmark. ASD3PM has been shown to outperfor…
Figure 24
Figure 24. Figure 24: MIDI plots for Generated Samples using the LZMidi approach [PITH_FULL_IMAGE:figures/full_fig_p034_24.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

75 extracted references · 61 canonical work pages

  1. [48]

    2024 A Family of LZ78-based Universal Sequential Probability Assignments

    Sagan N, Weissman T. 2024 A Family of LZ78-based Universal Sequential Probability Assignments

    work page 2024

  2. [1]

    2020 Implicit Neural Representations with Periodic Activation Functions

    Sitzmann V , Martel JNP , Bergman AW, Lindell DB, Wetzstein G. 2020 Implicit Neural Representations with Periodic Activation Functions

    work page 2020

  3. [2]

    2024 GaussianImage: 1000 FPS Image Representation and Compression by 2D Gaussian Splatting

    Zhang X, Ge X, Xu T, He D, Wang Y, Qin H, Lu G, Geng J, Zhang J. 2024 GaussianImage: 1000 FPS Image Representation and Compression by 2D Gaussian Splatting. InEuropean Conference on Computer Vision

    work page 2024

  4. [3]

    1991 Kodak Dataset.http://r0k.us/graphics/kodak/

    Kodak. 1991 Kodak Dataset.http://r0k.us/graphics/kodak/. Accessed: 2025-06-09

    work page 1991

  5. [4]

    2021 COIN: COmpression with Implicit Neural representations

    Dupont E, Goli ´ nski A, Alizadeh M, Teh YW, Doucet A. 2021 COIN: COmpression with Implicit Neural representations

    work page 2021

  6. [5]

    2022 COIN++: Neural Compression Across Modalities

    Dupont E, Loya H, Alizadeh M, Goli ´ nski A, Teh YW, Doucet A. 2022 COIN++: Neural Compression Across Modalities

    work page 2022

  7. [6]

    2022 Implicit Neural Representations for Image Compression

    Strümpler Y, Postels J, Yang R, van Gool L, Tombari F. 2022 Implicit Neural Representations for Image Compression

    work page 2022

  8. [7]

    2023 COOL-CHIC: Coordinate-based Low Complexity Hierarchical Image Codec

    Ladune T, Philippe P , Henry F, Clare G, Leguay T. 2023 COOL-CHIC: Coordinate-based Low Complexity Hierarchical Image Codec

    work page 2023

Show all 75 references
  1. [8]

    2023 C3: High-performance and low- complexity neural compression from a single image or video

    Kim H, Bauer M, Theis L, Schwarz JR, Dupont E. 2023 C3: High-performance and low- complexity neural compression from a single image or video

    2023

  2. [9]

    2023 Compression with Bayesian Implicit Neural Representations

    Guo Z, Flamich G, He J, Chen Z, Hernández-Lobato JM. 2023 Compression with Bayesian Implicit Neural Representations

    2023

  3. [10]

    2024 RECOMBINER: Robust and Enhanced Compression with Bayesian Implicit Neural Representations

    He J, Flamich G, Guo Z, Hernández-Lobato JM. 2024 RECOMBINER: Robust and Enhanced Compression with Bayesian Implicit Neural Representations. InThe Twelfth International Conference on Learning Representations

    2024

  4. [11]

    2022 An Information-Theoretic Justification for Model Pruning

    Isik B, Weissman T, No A. 2022 An Information-Theoretic Justification for Model Pruning

    2022

  5. [12]

    2023 3D Gaussian Splatting for Real-Time Radiance Field Rendering

    Kerbl B, Kopanas G, Leimkühler T, Drettakis G. 2023 3D Gaussian Splatting for Real-Time Radiance Field Rendering

    2023

  6. [13]

    2025 A Survey on 3D Gaussian Splatting

    Chen G, Wang W. 2025 A Survey on 3D Gaussian Splatting

    2025

  7. [14]

    2025 3D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunities.IEEE Transactions on Circuits and Systems for Video Technologypp

    Bao Y, Ding T, Huo J, Liu Y, Li Y, Li W, Gao Y, Luo J. 2025 3D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunities.IEEE Transactions on Circuits and Systems for Video Technologypp. 1–1. (10.1109/TCSVT.2025.3538684)

    2025

  8. [15]

    2025 Large Images are Gaussians: High- Quality Large Image Representation with Levels of 2D Gaussian Splatting.arXiv preprint arXiv:2502.09039

    Zhu L, Lin G, Chen J, Zhang X, Jin Z, Wang Z, Yu L. 2025 Large Images are Gaussians: High- Quality Large Image Representation with Levels of 2D Gaussian Splatting.arXiv preprint arXiv:2502.09039

    2025 arXiv

  9. [16]

    2025 EigenGS Representation: From Eigenspace to Gaussian Image Space

    Tai LW, Li CE, Chen CL, Tsai CJ, Chen HT, Liu TL. 2025 EigenGS Representation: From Eigenspace to Gaussian Image Space

    2025

  10. [17]

    2025 Neural Video Compression using 2D Gaussian Splatting

    Gupta L, Junejo IN. 2025 Neural Video Compression using 2D Gaussian Splatting

    2025

  11. [18]

    2025 GSVC: Efficient Video Representation and Compression Through 2D Gaussian Splatting

    Wang L, Shi Y, Ooi WT. 2025 GSVC: Efficient Video Representation and Compression Through 2D Gaussian Splatting

    2025

  12. [19]

    2023 Toward textual transform coding.IEEE BITS the Information Theory Magazine 3, 32–40

    Weissman T. 2023 Toward textual transform coding.IEEE BITS the Information Theory Magazine 3, 32–40

    2023

  13. [20]

    2023 Text+ Sketch: Image Compression at Ultra Low Rates

    Lei E, Uslu YB, Hassani H, Saeedi Bidokhti S. 2023 Text+ Sketch: Image Compression at Ultra Low Rates. InICML Workshop on Neural Compression. ICML

    2023

  14. [21]

    InThe Twelfth International Conference on Learning Representations

    Careil M, Muckley MJ, Verbeek J, Lathuilière S Towards image compression with perfect realism at ultra-low bitrates. InThe Twelfth International Conference on Learning Representations

  15. [22]

    2024 Semantic Image Compression Using Textual Transforms

    Arikan L, Weissman T. 2024 Semantic Image Compression Using Textual Transforms. In2024 IEEE International Symposium on Information Theory Workshops (ISIT-W)pp. 1–6. (10.1109/ISIT- W61686.2024.10591763)

    2024

  16. [23]

    2022 Txt2vid: Ultra-low bitrate compression of talking-head videos via text.IEEE Journal on Selected Areas in Communications41, 107–118

    Tandon P , Chandak S, Pataranutaporn P , Liu Y, Mapuranga AM, Maes P , Weissman T, Sra M. 2022 Txt2vid: Ultra-low bitrate compression of talking-head videos via text.IEEE Journal on Selected Areas in Communications41, 107–118

    2022

  17. [24]

    1995 Filtering random noise from deterministic signals via data compression

    Natarajan B. 1995 Filtering random noise from deterministic signals via data compression. IEEE Transactions on Signal Processing43, 2595–2605. (10.1109/78.482110)

    1995 doi

  18. [25]

    2005 The empirical distribution of rate-constrained source codes

    Weissman T, Ordentlich E. 2005 The empirical distribution of rate-constrained source codes. IEEE transactions on information theory51, 3718–3733

    2005

  19. [26]

    2002 The Kolmogorov Sampler

    Donoho DL. 2002 The Kolmogorov Sampler

    2002

  20. [27]

    2012 Denoising via MCMC-Based Lossy Compression.IEEE Transactions on Signal Processing60, 3092–3100

    Jalali S, Weissman T. 2012 Denoising via MCMC-Based Lossy Compression.IEEE Transactions on Signal Processing60, 3092–3100. (10.1109/TSP .2012.2190597)

    2012

  21. [28]

    2016 Effect of lossy compression of quality scores on variant calling.Briefings in Bioinformatics18, 183–194

    Ochoa I, Hernaez M, Goldfeder R, Weissman T, Ashley E. 2016 Effect of lossy compression of quality scores on variant calling.Briefings in Bioinformatics18, 183–194. (10.1093/bib/bbw011) 39royalsocietypublishing.org/journal/rsta Phil. Trans. R. Soc. A 0000000

    2016 doi

  22. [29]

    1992 The JPEG still picture compression standard.IEEE Transactions on Consumer Electronics38, xviii–xxxiv

    Wallace G. 1992 The JPEG still picture compression standard.IEEE Transactions on Consumer Electronics38, xviii–xxxiv. (10.1109/30.125072)

    1992 doi

  23. [30]

    1994 Ideal spatial adaptation by wavelet shrinkage.Biometrika81, 425–455

    Donoho DL, Johnstone IM. 1994 Ideal spatial adaptation by wavelet shrinkage.Biometrika81, 425–455. (10.1093/biomet/81.3.425)

    1994 doi

  24. [31]

    1995 Adapting to Unknown Smoothness via Wavelet Shrinkage

    Donoho DL, Johnstone IM. 1995 Adapting to Unknown Smoothness via Wavelet Shrinkage. Journal of the American Statistical Association90, 1200–1224. (10.1080/01621459.1995.10476626)

    1995

  25. [32]

    1995 De-noising by soft-thresholding.IEEE Transactions on Information Theory41, 613–627

    Donoho DL. 1995 De-noising by soft-thresholding.IEEE Transactions on Information Theory41, 613–627. (10.1109/18.382009)

    1995 doi

  26. [33]

    2002 JPEG2000: standard for interactive imaging.Proceedings of the IEEE90, 1336–1357

    Taubman D, Marcellin M. 2002 JPEG2000: standard for interactive imaging.Proceedings of the IEEE90, 1336–1357. (10.1109/JPROC.2002.800725)

    2002

  27. [34]

    2023 Image Denoising: The Deep Learning Revolution and Beyond—A Survey Paper.SIAM Journal on Imaging Sciences16, 1594–1654

    Elad M, Kawar B, Vaksman G. 2023 Image Denoising: The Deep Learning Revolution and Beyond—A Survey Paper.SIAM Journal on Imaging Sciences16, 1594–1654. (10.1137/23M1545859)

    2023 doi

  28. [35]

    2018 The perception-distortion tradeoff

    Blau Y, Michaeli T. 2018 The perception-distortion tradeoff. InProceedings of the IEEE conference on computer vision and pattern recognitionpp. 6228–6237

    2018

  29. [36]

    Completely Blind

    Mittal A, Soundararajan R, Bovik AC. 2013 Making a “Completely Blind” Image Quality Analyzer.IEEE Signal Processing Letters20, 209–212. (10.1109/LSP .2012.2227726)

    2013

  30. [37]

    1977 A universal algorithm for sequential data compression.IEEE Transactions on Information Theory23, 337–343

    Ziv J, Lempel A. 1977 A universal algorithm for sequential data compression.IEEE Transactions on Information Theory23, 337–343. (10.1109/TIT.1977.1055714)

    1977

  31. [38]

    1978 Compression of individual sequences via variable-rate coding.IEEE Transactions on Information Theory24, 530–536

    Ziv J, Lempel A. 1978 Compression of individual sequences via variable-rate coding.IEEE Transactions on Information Theory24, 530–536. (10.1109/TIT.1978.1055934)

    1978

  32. [39]

    2006Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing)

    Cover TM, Thomas JA. 2006Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). USA: Wiley-Interscience

  33. [40]

    1979 Arithmetic Coding.IBM Journal of Research and Development23, 149–162

    Rissanen J, Langdon GG. 1979 Arithmetic Coding.IBM Journal of Research and Development23, 149–162. (10.1147/rd.232.0149)

    1979 doi

  34. [41]

    2000 Text categorization using compression models

    Frank E, Chui C, Witten I. 2000 Text categorization using compression models. InProceedings DCC 2000. Data Compression Conferencepp. 555–. (10.1109/DCC.2000.838202)

    2000

  35. [42]

    2004 Clustering by compression

    Cilibrasi R, Vitanyi P . 2004 Clustering by compression

    2004

  36. [43]

    2024 Language Modeling Is Compression

    Delétang G, Others. 2024 Language Modeling Is Compression. InProceedings of ICLR 2024

    2024

  37. [44]

    1983 A note on the Ziv - Lempel model for compressing individual sequences (Corresp.).IEEE Transactions on Information Theory29, 284–287

    Langdon G. 1983 A note on the Ziv - Lempel model for compressing individual sequences (Corresp.).IEEE Transactions on Information Theory29, 284–287. (10.1109/TIT.1983.1056645)

    1983

  38. [45]

    1991 Gambling using a finite state machine.IEEE Transactions on Information Theory 37, 1459–1465

    Feder M. 1991 Gambling using a finite state machine.IEEE Transactions on Information Theory 37, 1459–1465. (10.1109/18.133269)

    1991 doi

  39. [46]

    1992 Universal prediction of individual sequences.IEEE Transactions on Information Theory38, 1258–1270

    Feder M, Merhav N, Gutman M. 1992 Universal prediction of individual sequences.IEEE Transactions on Information Theory38, 1258–1270. (10.1109/18.144706)

    1992 doi

  40. [47]

    2007 Universal Filtering Via Prediction.IEEE Transactions on Information Theory53, 1253–1264

    Weissman T, Ordentlich E, Weinberger MJ, Somekh-Baruch A, Merhav N. 2007 Universal Filtering Via Prediction.IEEE Transactions on Information Theory53, 1253–1264. (10.1109/TIT.2007.892782)

    2007

  41. [49]

    1972 Admissibility properties or Gilbert’s encoding for unknown source probabilities (Corresp.).IEEE Transactions on Information Theory18, 216–217

    Cover T. 1972 Admissibility properties or Gilbert’s encoding for unknown source probabilities (Corresp.).IEEE Transactions on Information Theory18, 216–217. (10.1109/TIT.1972.1054738)

    1972

  42. [50]

    1995 The context-tree weighting method: basic properties

    Willems F, Shtarkov Y, Tjalkens T. 1995 The context-tree weighting method: basic properties. IEEE Transactions on Information Theory41, 653–664. (10.1109/18.382012)

    1995 doi

  43. [51]

    2025 Genomic Data Classification via Universal Compression

    Omri Y, Sagan N, Min E et al.. 2025 Genomic Data Classification via Universal Compression. https://doi.org/10.21203/rs.3.rs-6363017/v1. Preprint (Version 1) available at Research Square (10.21203/rs.3.rs-6363017/v1)

    2025 doi

  44. [52]

    2025 Universal Discrete Filtering with Lookahead or Delay

    Yan P , Jeong J, Sagan N, Weissman T. 2025 Universal Discrete Filtering with Lookahead or Delay

    2025

  45. [53]

    2025 LZMidi: Compression-Based Symbolic Music Generation

    Ding C, Gorle A, Bhattacharya S, Hasteer D, Sagan N, Weissman T. 2025 LZMidi: Compression-Based Symbolic Music Generation

    2025

  46. [54]

    2024 Deep learning approaches in genomic analysis: A review of DNA sequence classification techniques.International Journal of Scientific Research and Engineering Trends

    Kimbahune DV , Nerkar V . 2024 Deep learning approaches in genomic analysis: A review of DNA sequence classification techniques.International Journal of Scientific Research and Engineering Trends

    2024

  47. [55]

    2019 Pharmacogenomics.Lancet394, 521–532

    Roden DM, McLeod HL, Relling MV , Williams MS, Mensah GA, Peterson JF, Driest SLV . 2019 Pharmacogenomics.Lancet394, 521–532

    2019

  48. [56]

    2024 Genomic language models: Opportunities and challenges

    Benegas G, Ye C, Albors C, Li JC, Song YS. 2024 Genomic language models: Opportunities and challenges. Preprint. 40royalsocietypublishing.org/journal/rsta Phil. Trans. R. Soc. A 0000000

    2024

  49. [57]

    1993 A measure of relative entropy between individual sequences with application to universal classification.IEEE Transactions on Information Theory39, 1270–1279

    Ziv J, Merhav N. 1993 A measure of relative entropy between individual sequences with application to universal classification.IEEE Transactions on Information Theory39, 1270–1279

    1993

  50. [58]

    2022 Genezip.https://github.com/SharonLab/GeneZip

    Itai Sharon Group. 2022 Genezip.https://github.com/SharonLab/GeneZip. ISBRA 2022

    2022

  51. [59]

    2025 Using lossless compression algorithms to improve metagenomics binning and accelerate genome taxonomic classification

    Leibovich O, Meir Y, Sharon I. 2025 Using lossless compression algorithms to improve metagenomics binning and accelerate genome taxonomic classification. To appear at RECOMB 2025

    2025

  52. [60]

    2004 On prediction using variable order Markov models

    Begleiter R, El-Yaniv R, Yona G. 2004 On prediction using variable order Markov models. Journal of Artificial Intelligence Research22, 385–421

    2004

  53. [61]

    2024 DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome

    Zhou Z, Ji Y, Li W, Dutta P , Davuluri R, Liu H. 2024 DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome

    2024

  54. [62]

    2015 char-RNN.https://github.com/karpathy/char-rnn

    Karpathy A. 2015 char-RNN.https://github.com/karpathy/char-rnn. Accessed: 2025-06-14

    2015

  55. [63]

    2021 Documenting large webtext corpora: A case study on the colossal clean crawled corpus

    Dodge J, Sap M, Marasovi´ c A, Agnew W, Ilharco G, Groeneveld D, Mitchell M, Gardner M. 2021 Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.08758

    2021 arXiv

  56. [64]

    2023 TinyStories: How small can language models be and still speak coherent English?.arXiv preprint arXiv:2305.07759

    Eldan R, Li Y. 2023 TinyStories: How small can language models be and still speak coherent English?.arXiv preprint arXiv:2305.07759

    2023 arXiv

  57. [65]

    2023 Discrete Diffusion Probabilistic Models for Symbolic Music Generation

    Plasser M, Peter S, Widmer G. 2023 Discrete Diffusion Probabilistic Models for Symbolic Music Generation

    2023

  58. [66]

    2020 Encoding Musical Style with Transformer Autoencoders

    Choi K, Hawthorne C, Simon I, Dinculescu M, Engel J. 2020 Encoding Musical Style with Transformer Autoencoders

    2020

  59. [67]

    2021 Symbolic Music Generation with Diffusion Models

    Mittal G, Engel J, Hawthorne C, Simon I. 2021 Symbolic Music Generation with Diffusion Models

    2021

  60. [68]

    2018 GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

    Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S. 2018 GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

    2018

  61. [69]

    2017 Audio Set: An ontology and human-labeled dataset for audio events

    Gemmeke JF, Ellis DPW, Freedman D, Jansen A, Lawrence W, Moore RC, Plakal M, Ritter M. 2017 Audio Set: An ontology and human-labeled dataset for audio events. In2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)pp. 776–780. (10.1109/ICASSP ....

    2017

  62. [70]

    2017 Wasserstein GAN

    Arjovsky M, Chintala S, Bottou L. 2017 Wasserstein GAN. InProceedings of the 34th International Conference on Machine Learning. PMLR

    2017

  63. [71]

    2017 Improved Training of Wasserstein GANs

    Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V , Courville AC. 2017 Improved Training of Wasserstein GANs. InAdvances in Neural Information Processing Systems

    2017

  64. [72]

    2019 Optuna: A Next-generation Hyperparameter Optimization Framework

    Akiba T, Sano S, Yanase T, Ohta T, Koyama M. 2019 Optuna: A Next-generation Hyperparameter Optimization Framework

    2019

  65. [73]

    1993 A measure of relative entropy between individual sequences with application to universal classification.IEEE Transactions on Information Theory39, 1270–1279

    Ziv J, Merhav N. 1993 A measure of relative entropy between individual sequences with application to universal classification.IEEE Transactions on Information Theory39, 1270–1279. (10.1109/18.243444)

    1993 doi

  66. [74]

    long enough

    Seroussi G. 2004 Universal Types and Simulation of Individual Sequences. In Farach-Colton M, editor,LATIN 2004: Theoretical Informaticsvol. 2976Lecture Notes in Computer Sciencepp. 312–321. Springer, Berlin, Heidelberg. (10.1007/978-3-540-24698-5_35) 41royalsocietypublishing.o...

    2004 doi

  67. [75]

    states that, for any individual sequence, C(x n) n =o(1), so (∗)≤ nX ℓ=k+1 mℓ n µk(vmℓ ℓ ) +o(1). As pointwise minimization is concave, Jensen’s inequality provides µk(xn) = min q∈Mk 1 n nX t=k+1 log 1 q(xt|xt−1) ≥min q∈Mk 1 n nX ℓ=k+1 mℓ n 1 mℓ log 1 q(vmℓ ℓ ) ! ≥ nX ℓ=k+1 mℓ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.