Pith. sign in

REVIEW 4 major objections 5 minor 74 references

Understanding Rate-Distortion Performance in Distributed Transformer Inference

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Transformer intermediate representations become harder to compress as they go deeper, and two measurable complexity trends—covariance determinant and Rademacher complexity—drive the rise in bitrate.

desk verdict The empirical finding — deeper transformer representations are harder to compress — is solid and useful; the theoretical explanation cancels itself out. read the letter →

arxiv 2601.22002 v5 pith:6WJVDM3V submitted 2026-01-29 cs.LG cs.ITmath.IT

Anderson de Andrade , Alon Harell , Ivan V. Bajić This is my paper · ORCID
classification cs.LGcs.ITmath.IT MSC 94A3494A1568T0768P30
keywords rate-distortiontheorylearnedcompressiontransformerinferenceintermediaterepresentationscodingformachinesV-entropyRademachercomplexitysplitcomputing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that for transformer networks, the rate-distortion performance of learned compression on intermediate representations gets worse as the split point moves deeper, even though the data processing inequality guarantees the representation's Shannon entropy is non-increasing. It argues the quantity that actually controls the achievable rate is the V-entropy gap—the shortfall between what a restricted entropy model can achieve and true entropy—plus the generalization error of estimating that gap. Two measurable complexity properties of the representation, the log determinant of its covariance and its Rademacher complexity, are shown to track the achieved bitrate across split points for language and vision transformers, and the same measures still track bitrate for a convolutional ResNet even where the rate trend reverses. If the paper is right, rate-distortion performance in distributed transformer inference is predictable from these two quantities, and the usual intuition that later layers should compress better is inverted for transformers.

What carries the argument

The central object is the V-entropy gap, G_V(Y|W) = |H_V(Y|W) − H_Ω(Y|W)|: the difference between the best cross-entropy a restricted hypothesis family V can achieve and the true conditional entropy of the representation given side information. V-entropy is the infimum of expected negative log-likelihood over V. Theorem 2 bounds V-entropy by (1/2) log det Cov(Y) plus constants and the gap itself, Theorem 3 adds a Lipschitz dilation term, and Theorem 4 bounds the generalization error of the gap by the Rademacher complexity of the samples times the Lipschitz constant of the entropy model. These theorems convert "usable information" into two computable diagnostics—the covariance determinant and

What would settle it

Compute the true log determinant of Cov(Y) for GPT-2 intermediate representations at split points 3, 6, and 9—using a randomized trace-log-det estimator, or a small transformer where the full covariance is tractable—and compare it against the Arnoldi-based estimate. If the full log determinant does not increase with depth while the achieved bitrate does, the first proposed mechanism is an artifact of eigenvalue truncation.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that deeper transformer representations are harder to compress: at equal task distortion, coding the output of a later block costs more bits than coding an earlier block, for GPT-2, Pythia, and ViT, while a ResNet shows the opposite trend. The paper attributes the transformer behavior to two mechanisms. First, the V-entropy of a representation—the best rate a restricted family of entropy models can achieve—is upper-bounded by half the log determinant of its covariance, and this determinant grows with depth; this raises the rate floor. Second, the generalization error of the learned entropy estimate is upper-bounded by the Rademacher complexity of th

Load-bearing premise

The empirical support for the first mechanism depends on estimating the log determinant of the covariance from only the largest of roughly 393,000 eigenvalues (about 1,000 Arnoldi iterations), an estimate the paper admits is biased and describes only as 'adequate for assessing trends'; if that truncated estimate does not track the true determinant, the depth trend in V-entropy loses its measured basis.

Editorial extensions

If this is right

  • Early split points are the efficient ones for distributed transformer inference: the same task accuracy costs substantially fewer bits when the split is closer to the input, favoring edge devices that can only run a few layers.
  • Entropy is not a meaningful predictor of learned-codec rate on transformer representations; the V-entropy gap and its generalization error are the quantities that set the achievable rate.
  • Enlarging an entropy model can backfire, since added complexity inflates the generalization-error term; this is the paper's explanation for why its simplest standalone hyper-prior codec beats more complex entropy models by at least 10.7% in BD-rate.
  • The rate-constrained representations are themselves highly compressible with generic lossless codecs, and the proposed learned codec can transmit at a small fraction of the raw 16-bit rate (about 292 vs 12,288 bits per token on a GPU) while being faster than GPU baselines.
  • The same covariance and Rademacher terms correlate with bitrate even for a ResNet whose rate falls with depth, so the mechanism is general; the transformer-specific fact is that both quantities rise with depth.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested consequence of the covariance bound is that whitening or re-scaling intermediate representations between transformer modules—pushing the covariance determinant toward one—should lower the rate floor; this is a concrete codec modification the paper does not try.
  • Because the paper's rate data come from a single coded representation between two modules, the same depth-dependent difficulty should appear when compressing KV caches at different layers in disaggregated LLM serving, making layer choice part of the compression design.
  • The generalization-error mechanism suggests a bias-variance trade-off in learned codecs that may carry beyond transformer inference: at modest sample sizes, simpler entropy models can outperform richer ones in any learned compression setting, not just language modeling.
  • A testable follow-up is to train transformer blocks with a mild Lipschitz constraint (well below the extreme constants that the paper notes prevent training) and check whether the covariance determinant and Rademacher complexity trends, and with them the rate-versus-depth curve, flatten.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents a learned lossy codec for compressing intermediate transformer representations in distributed inference, using a hyperprior-based entropy model, and studies how rate-distortion performance varies with network depth. The authors report that, at comparable distortion, deeper transformer representations require higher bitrate, contrary to what the data-processing inequality would suggest for Shannon entropy, while a ResNet baseline shows the opposite trend. They introduce a 'V-entropy gap' between achievable rate and Shannon entropy, prove several inequalities involving covariance determinants, Lipschitz constants, and Rademacher complexity, and report strong correlations between the measured rate and estimates of the covariance determinant and Rademacher complexity. The conclusion is that the rate increase for transformers is caused by increased representation complexity and by worse generalization of the learned entropy model.

Significance. If the explanatory claims were established, the paper would contribute both a practical codec and a new theoretical lens on why learned entropy models perform poorly on deep transformer representations. The reported rate savings of the proposed codec are substantial, and the paper makes its code available, which are genuine strengths. The direct rate-distortion measurements showing that depth increases rate for transformers are interesting and clearly presented. However, the theoretical explanation is not currently supported: the covariance-based bounds cancel the V-entropy gap, and the Rademacher bound controls generalization error rather than the rate itself. The empirical correlations are based on an unvalidated truncated determinant estimate and on only three split points per model. The measured depth effect is the strongest part of the paper; the causal interpretation needs substantial revision.

major comments (4)
  1. [Section III-B / Theorem 2] In Theorem 2 (and its proof in Suppl. S-I.E), substitute the definition of G_V. Since V⊆Ω, G_V(Y|W)=H_V(Y|W)-H(Y|W) (Eq. 52 of the proof). The statement H_V≤(1/2)logdetΣ+(D/2)log(2πe)+G_V-logΔ then collapses to H(Y|W)≤(1/2)logdetΣ+(D/2)log(2πe)-logΔ, the standard Gaussian max-entropy bound. There is no V-dependent term left, so the covariance determinant cannot theoretically explain why the V-entropy or the achieved rate increases with depth. The same cancellation occurs in Theorem 3. The Section IV-D interpretation that the covariance 'provides one reason why deeper layers can exhibit higher rate' therefore has no support from these theorems.
  2. [Section III-C / Theorem 4] Theorem 4 bounds R_{V,D}(Y|W), the generalization error of the V-entropy gap, i.e. the distance between the true gap and its empirical estimate. It does not bound the optimized rate r_y+r_w that is reported in Fig. 6d. The measured rate is the training objective at convergence; an upper bound on the estimation error of the gap does not imply that a larger Rademacher complexity produces a larger rate. The correlation in Fig. 6d is therefore not a test of Theorem 4. To support the abstract's second mechanism, the paper must either derive a bound directly on the expected rate in terms of Rad(D), or explicitly downgrade this to an exploratory empirical observation.
  3. [Section IV-D.1 / Fig. 6a-b] The log-determinant estimate uses eigenvalues of a 1000×1000 Hessenberg matrix from 1,000 Arnoldi iterations on a target of dimension T·E≈393k. The omitted small eigenvalues can dominate log det Σ, so the across-layer trend—and hence the correlation in Fig. 6b—may be an artifact of truncation. The text admits the estimate is biased and asserts it 'appears adequate for assessing trends' without any validation. I ask for a concrete validation: exact/randomized log-det estimates on a smaller tractable proxy, or a stochastic trace estimator with error bars, and a sensitivity check of the Pearson correlation to the number of Arnoldi eigenvalues. This is load-bearing because, once Theorem 2 is set aside, this correlation is the only support for the first mechanism.
  4. [Section IV-D.3 / Fig. 6b, 6d] Each reported Pearson correlation is computed from three split points per model. With n=3, correlations of 0.89–1.00 are not statistically significant (for n=3, |r|>0.997 is needed for p<0.05), and no confidence intervals or multiplicity corrections are given. The claim of 'strong correlations' is therefore overstated. The normalized axes also hide the absolute rate scale. Please report per-model scatter with more split points, or at minimum bootstrap confidence intervals and p-values, before using these correlations as evidence for the explanatory mechanism.
minor comments (5)
  1. [Title] The heading says 'Rate-Distortion Optimization for Transformer Inference,' while the arXiv/abstract title refers to 'Understanding Rate-Distortion Performance in Distributed Transformer Inference.' Please align the title across versions.
  2. [Abstract] Typos: 'generaliztion' and missing spaces in 'theLipschitzconstant'.
  3. [Eq. (8)] The double-max notation is hard to parse. Write \max_{i,j} |\sum_k a_k D_k|_{i,j} and state explicitly that this estimates the Rademacher complexity of the coordinate-projection class, not of the representation as a whole.
  4. [Tables III and IV] For GPT-2, the split-3 row uses λ=0.010 while split-6/9 rows use λ=0.001. Cross-row rate comparisons at fixed λ are not apples-to-apples; the RD-curve comparison in Fig. 4b is the appropriate evidence.
  5. [Fig. 5a] The paper notes that missing curve overlap makes BD-rate for LAMBADA unreliable, but still reports the number. Please avoid quoting BD-rate for that case and rely on the curves.

Circularity Check

2 steps flagged · score 6.0 of 10

Theoretical explanation is partly self-definitional: substituting G_V cancels H_V in Theorems 2–3, so covariance/dilation bounds do not actually constrain the V-entropy or rate; the empirical correlations are not circular but are the only remaining support.

  1. self definitional [Section III-A Definition 1; Section III-B Theorem 2; Supplementary Eq. (52)]
    "G_V(Y|W)≜|H_V(Y|W)−H_Ω(Y|W)|. ... H_V(Y|W)=H(Y|W)+G_V(Y|W)(52) ... H_V(Y|W)≤ 1/2log det Σ + D/2log(2πe)+G_V(Y|W)−log Δ as Δ→0."

    Since V⊆Ω, Lemma 1 removes the absolute value: G_V(Y|W)=H_V(Y|W)−H(Y|W). Substituting this into Theorem 2 cancels H_V(Y|W) from both sides, leaving H(Y|W)≤ 1/2 log det Σ + D/2 log(2πe) − log Δ, the standard Gaussian max-entropy bound for the true conditional entropy. No V-dependent term survives, so covariance determinant cannot theoretically explain an increase in H_V or in the achieved rate. The paper later calls the covariance determinant a ‘part of the V-entropy bound’ and a reason deeper layers have higher rate; that is the bound's input restated, not a prediction.

  2. self definitional [Section III-B Theorem 3; Supplementary Eqs. (56)–(58)]
    "H_V(Y|W)≤h(X)+D(L_f−1)+G_V(Y|W)−log Δ as Δ→0."

    The same identity G_V=H_V−H(Y|W) cancels H_V, leaving H(Y|W)≤ h(X)+D(L_f−1)−log Δ. Thus the Lipschitz/dilation term bounds only the Shannon entropy of the processing chain, which the paper itself notes decreases with depth (data processing inequality). It provides no formal bound on the V-entropy gap or on the achievable rate of the restricted predictive family; the claimed ‘expansion’ mechanism is therefore the definition of G_V restated as a bound.

full rationale

The main circularity is in the theoretical derivation offered for the paper's central claim. Theorem 2 and Theorem 3 add G_V to standard differential-entropy bounds, but because G_V is defined as H_V−H_Ω, the H_V terms cancel identically; the theorems reduce to bounds on H(Y|W), which by the paper's own data-processing statement decreases with depth. They therefore cannot explain why transformer rates increase with depth. The empirical correlations between rate and covariance/Rademacher estimates (Fig. 6) are not forced by construction and remain independent evidence, though the covariance estimate is admittedly biased and unvalidated for trend detection (a correctness risk rather than circularity), and the correlations are not predictive tests. Theorem 1 is essentially a restatement of the definition of G_V plus the fact that the rate is a member of the predictive family; it does not add independent explanatory content. The Rademacher generalization bound (Theorem 4) is not circular. Self-citations are present but not load-bearing. Overall, the theory's load-bearing bounds reduce by construction, yielding partial circularity.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

No physical entities are invented. The central theoretical construct (V-entropy gap) is a definition; the main assumptions are distributional hypotheses in Theorem 4, the fixed-family assumption contradicted by measured Lipschitz constants, and the unvalidated covariance estimator.

free parameters (1)
  • λ (rate-distortion weight) = 0.001, 0.0025, 0.0075, 0.01 (Table II lists 0.025/0.075, likely typos)
    Chosen per operating point, not fitted to theory; for split-point comparisons λ is selected so distortion is comparable across split points.
assumptions (6)
  • domain assumption V-entropy theory of Xu et al. [9] applies; predictive family V can be restricted to Gaussian diagonal-covariance entropy models, and H_Ω is attainable by some ω∈Ω.
    Used in Definitions 1–2 and Theorems 1–4; the paper extends [9] without re-deriving its core estimation bounds.
  • domain assumption The conditional distribution P_{Y|W} in Theorem 4 is Lipschitz-governed by a multivariate normal with covariance αI satisfying α ≤ exp(2B/D−1)(2π)^{-1}.
    Required for the β term in the Rademacher bound; no empirical check of Gaussianity or α is provided.
  • domain assumption The entropy model architecture is fixed across split points, so Lip(V_r) acts as a constant in Theorem 4.
    Stated in §IV-D, but Table III shows measured Lip(v) varies (e.g., 6.62, 4.21, 5.04 for GPT-2).
  • ad hoc to paper The leading-eigenvalue Arnoldi estimate of log det Cov(Y) is adequate for trend assessment.
    Log determinant is dominated by smallest eigenvalues; truncation bias is acknowledged but unvalidated.
  • standard math Quantization with Δ=1 supports the continuous/discrete entropy relation H(q(Y')) ≈ h(Y') − log Δ used in Theorems 2 and 3.
    Standard approximation from Cover & Thomas, also used in the learned-compression literature.
  • ad hoc to paper The empirical Rademacher estimate (Eq. 8) with N=1000, M=10000 approximates the true Rademacher complexity of the target representation.
    No convergence certificate; authors note low variance but only compare correlations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Rate-Distortion Performance in Distributed Transformer Inference." pith.science (2026). https://pith.science/paper/6WJVDM3V

@misc{pith2026260122002,
  author       = {Pith},
  title        = {Pith review of: Understanding Rate-Distortion Performance in Distributed Transformer Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6WJVDM3V}},
  note         = {Machine review of arXiv:2601.22002}
}
read the original abstract

Transformers achieve superior performance on many tasks, but impose heavy compute and memory requirements during inference. This inference can be made more efficient by partitioning the process across multiple devices, which, in turn, requires compressing its intermediate representations. We study compressibility of transformer's intermediate representations via learned compression through the lens of rate-distortion and the theory of usable information. Our study reveals that, unlike convolutional models, deeper representations in transformers become more difficult to compress. The reason for this behavior is twofold: first, the complexity of representation increases as we move deeper into the transformer, necessitating a higher rate; and second, the higher complexity of representations worsens the generaliztion bound for learned entropy estimates, which further compromises compression performance. Through a combination of experiments and theory, we characterize and analyze the compressibility of transformer representations, derive bounds on the achievable rate of learned codecs applied to these representations, and offer a unified lens for understanding rate-distortion performance in representation coding.

Figures

Figures reproduced from arXiv: 2601.22002 by the authors.

Figure 1
Figure 1. Architecture diagrams for distributed transformer inference of lan [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Architecture overview of the proposed codec. The AE and AD blocks correspond to arithmetic encoders and decoders, respectively. They use the [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Architecture diagram of the different entropy models for the target representation [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Rate-distortion performance for GPT-2. The rate is measured in bits-per-token (BPT). Perplexity is the exponent of the classification cross-entropy loss, [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Rate-performance for GPT-2 evaluated on the the LAMBADA language task. The rate is measured in bits-per-token (BPT). [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Rate, covariance determinant, and Rademacher complexity estimates at different split points, for GPT-2 Small, Pythia 160M, ViT B/16, and ResNet [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 1
Figure 1. Figure 1: Estimates of the Lipschitz constant at different split points and corresponding bitrates, for GPT-2 Small, Pythia 160M, ViT B/16 and ResNet 34. The [PITH_FULL_IMAGE:figures/full_fig_p021_1.png]

Review history (3 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

74 extracted references · 9 linked inside Pith

  1. [1]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” inNeurIPS, 2017

    2017

  2. [2]

    Optimizing DNN training with pipeline model parallelism for enhanced performance in embedded systems,

    M. A. Maruf, A. Azim, N. Auluck, and M. Sahi, “Optimizing DNN training with pipeline model parallelism for enhanced performance in embedded systems,”JPDC, 2024

    2024

  3. [3]

    PISeL: Pipelining DNN inference for serverless computing,

    M. R. Jafari, J. Su, Y . Zhang, O. Wang, and W. Zhang, “PISeL: Pipelining DNN inference for serverless computing,” inACM CIKM, 2024

    2024

  4. [4]

    Split computing and early exiting for deep learning applications: Survey and research challenges,

    Y . Matsubara, M. Levorato, and F. Restuccia, “Split computing and early exiting for deep learning applications: Survey and research challenges,” ACM Computing Surveys, 2023

    2023

  5. [5]

    Dynamic split computing framework in distributed serverless edge clouds,

    H. Ko, H. Jeong, D. Jung, and S. Pack, “Dynamic split computing framework in distributed serverless edge clouds,”IEEE IoTJ, 2024

    2024

  6. [6]

    Variational image compression with a scale hyperprior,

    J. Ball ´e, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, “Variational image compression with a scale hyperprior,” inICLR, 2018

    2018

  7. [7]

    T. M. Cover and J. A. Thomas,Elements of information theory (Second edition). Wiley, 2006

    2006

  8. [8]

    Scalable image coding for humans and machines,

    H. Choi and I. V . Bajic, “Scalable image coding for humans and machines,”IEEE TIP, 2022

    2022

Show all 74 references
  1. [9]

    A theory of usable information under computational constraints,

    Y . Xu, S. Zhao, J. Song, R. Stewart, and S. Ermon, “A theory of usable information under computational constraints,” inICLR, 2020

    2020

  2. [10]

    Shalev-Shwartz and S

    S. Shalev-Shwartz and S. Ben-David,Understanding machine learning - from theory to algorithms. Cambridge University Press, 2014

    2014

  3. [11]

    Benyamini and J

    Y . Benyamini and J. Lindenstrauss,Geometric Nonlinear Functional Analysis. American Mathematical Society, 2000. SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 11

    2000

  4. [12]

    A survey of resource-efficient LLM and multimodal foundation models,

    M. Xu, W. Yin, D. Cai, R. Yi, D. Xu, Q. Wang, B. Wu, Y . Zhao, C. Yang, S. Wang, Q. Zhang, Z. Lu, L. Zhang, S. Wang, Y . Li, Y . Liu, X. Jin, and X. Liu, “A survey of resource-efficient LLM and multimodal foundation models,”ArXiv, vol. 2401.08092, 2024

    2024 arXiv

  5. [13]

    Orca: A distributed serving system for transformer-based generative models,

    G. Yu, J. S. Jeong, G. Kim, S. Kim, and B. Chun, “Orca: A distributed serving system for transformer-based generative models,” inUSENIX OSDI, 2022

    2022

  6. [14]

    Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve,

    A. Agrawal, N. Kedia, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, A. Tumanov, and R. Ramjee, “Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve,” inUSENIX OSDI, 2024

    2024

  7. [15]

    FlexGen: High-throughput generative inference of large language models with a single GPU,

    Y . Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. R ´e, I. Stoica, and C. Zhang, “FlexGen: High-throughput generative inference of large language models with a single GPU,” inICML, 2023

    2023

  8. [16]

    Fastdecode: High-throughput GPU-efficient LLM serving using heterogeneous pipelines,

    J. He and J. Zhai, “Fastdecode: High-throughput GPU-efficient LLM serving using heterogeneous pipelines,”ArXiv, vol. 2403.11421, 2024

    2024 arXiv

  9. [17]

    Splitwise: Efficient generative LLM inference using phase splitting,

    P. Patel, E. Choukse, C. Zhang, A. Shah, ´I. Goiri, S. Maleki, and R. Bianchini, “Splitwise: Efficient generative LLM inference using phase splitting,” inISCA, 2024

    2024

  10. [18]

    LoRAPrune: Structured pruning meets low-rank parameter-efficient fine-tuning,

    M. Zhang, H. Chen, C. Shen, Z. Yang, L. Ou, X. Yu, and B. Zhuang, “LoRAPrune: Structured pruning meets low-rank parameter-efficient fine-tuning,” inACL, 2024

    2024

  11. [19]

    AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, W. Chen, W. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration,” inMLSys, 2024

    2024

  12. [20]

    LLMLingua: Com- pressing prompts for accelerated inference of large language models,

    H. Jiang, Q. Wu, C. Lin, Y . Yang, and L. Qiu, “LLMLingua: Com- pressing prompts for accelerated inference of large language models,” inEMNLP, 2023

    2023

  13. [21]

    Longformer: The long- document transformer,

    I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long- document transformer,”ArXiv, vol. 2004.05150, 2020

    2004 arXiv

  14. [22]

    End-to-end optimized image compression,

    J. Ball ´e, V . Laparra, and E. P. Simoncelli, “End-to-end optimized image compression,” inICLR, 2017

    2017

  15. [23]

    ELIC: efficient learned image compression with unevenly grouped space- channel contextual adaptive coding,

    D. He, Z. Yang, W. Peng, R. Ma, H. Qin, and Y . Wang, “ELIC: efficient learned image compression with unevenly grouped space- channel contextual adaptive coding,” inCVPR, 2022

    2022

  16. [24]

    The devil is in the details: Window- based attention for image compression,

    R. Zou, C. Song, and Z. Zhang, “The devil is in the details: Window- based attention for image compression,” inCVPR, 2022

    2022

  17. [25]

    MLIC++: Linear complexity multi-reference entropy modeling for learned image com- pression,

    W. Jiang, J. Yang, Y . Zhai, F. Gao, and R. Wang, “MLIC++: Linear complexity multi-reference entropy modeling for learned image com- pression,”ACM TMCCA, 2025

    2025

  18. [26]

    Rate-distortion in image coding for machines,

    A. Harell, A. de Andrade, and I. V . Bajic, “Rate-distortion in image coding for machines,” inPCS, 2022

    2022

  19. [27]

    The information bottleneck method,

    N. Tishby, F. C. N. Pereira, and W. Bialek, “The information bottleneck method,” inAllerton Conference, 1999

    1999

  20. [28]

    Multivariate information bottleneck,

    N. Friedman, O. Mosenzon, N. Slonim, and N. Tishby, “Multivariate information bottleneck,” inUAI, 2001

    2001

  21. [29]

    Mutual information neural estimation,

    M. I. Belghazi, A. Baratin, S. Rajeswar, S. Ozair, Y . Bengio, R. D. Hjelm, and A. C. Courville, “Mutual information neural estimation,” in ICML, 2018

    2018

  22. [30]

    MLIC: Multi- reference entropy model for learned image compression,

    W. Jiang, J. Yang, Y . Zhai, P. Ning, F. Gao, and R. Wang, “MLIC: Multi- reference entropy model for learned image compression,” inACM MM, 2023

    2023

  23. [31]

    Frequency-aware transformer for learned image compression,

    H. Li, S. Li, W. Dai, C. Li, J. Zou, and H. Xiong, “Frequency-aware transformer for learned image compression,” inICLR, 2024

    2024

  24. [32]

    MambaVC: Learned visual compression with selective state spaces,

    S.-Y . Qin, J. Wang, Y . Zhou, B. Chen, T. Luo, B. An, T. Dai, S.-T. Xia, and Y . Wang, “MambaVC: Learned visual compression with selective state spaces,”ArXiv, vol. 2405.15413, 2024

    2024 arXiv

  25. [33]

    MambaIC: State space models for high-performance learned image compression,

    F. Zeng, H. Tang, Y . Shao, S. Chen, L. Shao, and Y . Wang, “MambaIC: State space models for high-performance learned image compression,” inCVPR, 2025

    2025

  26. [34]

    Fourier basis density model,

    A. D. la Fuente, S. Singh, and J. Ball ´e, “Fourier basis density model,” inPCS, 2024

    2024

  27. [35]

    Rate-accuracy bounds in visual coding for machines,

    I. V . Baji ´c, “Rate-accuracy bounds in visual coding for machines,” in IEEE MIPR, 2025

    2025

  28. [36]

    Towards task-compatible compressible representations,

    A. de Andrade and I. V . Bajic, “Towards task-compatible compressible representations,” inICME Workshops, 2024

    2024

  29. [37]

    Rate-distortion theory in coding for machines and its applications,

    A. Harell, Y . Foroutan, N. A. Ahuja, P. Datta, B. Kanzariya, V . S. Somayazulu, O. Tickoo, A. de Andrade, and I. V . Bajic, “Rate-distortion theory in coding for machines and its applications,”IEEE TPAMI, 2025

    2025

  30. [38]

    A theory of the learnable,

    L. G. Valiant, “A theory of the learnable,”ACM, 1984

    1984

  31. [39]

    Generating Wikipedia by summarizing long sequences,

    P. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, L. Kaiser, and N. Shazeer, “Generating Wikipedia by summarizing long sequences,” in ICLR, 2018

    2018

  32. [40]

    Improving language understand- ing by generative pre-training,

    A. Radford and K. Narasimhan, “Improving language understand- ing by generative pre-training,” https://openai.com/index/language- unsupervised, 2018

    2018

  33. [41]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. R. Stone, P. Albert, A. Almahairi, Y . Babaei, N. lay Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. M. Bikel, L. Blecher, C. tian Cant ´on Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V . Goswami, N. Goyal...

    2023 arXiv

  34. [42]

    DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving,

    Y . Zhong, S. Liu, J. Chen, J. Hu, Y . Zhu, X. Liu, X. Jin, and H. Zhang, “DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving,” inUSENIX OSDI, 2024

    2024

  35. [43]

    Universally quantized neural compression,

    E. Agustsson and L. Theis, “Universally quantized neural compression,” inNeurIPS, 2020

    2020

  36. [44]

    Lossy image compression with compressive autoencoders,

    L. Theis, W. Shi, A. Cunningham, and F. Husz ´ar, “Lossy image compression with compressive autoencoders,” inICLR, 2017

    2017

  37. [45]

    End-to-end optimization of nonlinear transform codes for perceptual quality,

    J. Ball ´e, V . Laparra, and E. P. Simoncelli, “End-to-end optimization of nonlinear transform codes for perceptual quality,” inPCS, 2016

    2016

  38. [46]

    NanoGPT,

    A. Karpathy, “NanoGPT,” https://github.com/karpathy/nanoGPT, 2022

    2022

  39. [47]

    Release strategies and the social impacts of language models,

    I. Solaiman, M. Brundage, J. Clark, A. Askell, A. Herbert-V oss, J. Wu, A. Radford, and J. Wang, “Release strategies and the social impacts of language models,”ArXiv, vol. 1908.09203, 2019

    1908 arXiv

  40. [48]

    OpenWebText Corpus,

    A. Gokaslan, V . Cohen, E. Pavlick, and S. Tellex, “OpenWebText Corpus,” https://Skylion007.github.io/OpenWebTextCorpus, 2019

    2019

  41. [49]

    Calculation of average PSNR differences between RD- curves,

    G. Bjontegaard, “Calculation of average PSNR differences between RD- curves,”ITU-T SC16/Q6 VCEG-M33, 2001

    2001

  42. [50]

    The LAMBADA dataset: Word prediction requiring a broad discourse context,

    D. Paperno, G. Kruszewski, A. Lazaridou, Q. N. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, and R. Fern ´andez, “The LAMBADA dataset: Word prediction requiring a broad discourse context,” inACL, 2016

    2016

  43. [51]

    DEFLATE compressed data format specification version 1.3,

    P. Deutsch, “DEFLATE compressed data format specification version 1.3,”RFC, 1996

    1996

  44. [52]

    Zstandard compression and the application/zstd media type,

    Y . Collet and M. S. Kucherawy, “Zstandard compression and the application/zstd media type,”RFC, 2018

    2018

  45. [53]

    Accelerating load times for DirectX games and apps with GDeflate for DirectStorage,

    Y . Uralsky, “Accelerating load times for DirectX games and apps with GDeflate for DirectStorage,” https://developer.nvidia.com/blog/ accelerating-load-times-for-directx-games-and-apps-with-gdeflate-for- directstorage, 2022

    2022

  46. [54]

    Asymmetric numeral systems: entropy coding combining speed of huffman coding with compression rate of arithmetic coding,

    J. Duda, “Asymmetric numeral systems: entropy coding combining speed of huffman coding with compression rate of arithmetic coding,” ArXiv, vol. 1311.2540, 2013

    2013 arXiv

  47. [55]

    Protocol overhead in IP/ATM networks,

    J. D. Cavanaugh, “Protocol overhead in IP/ATM networks,” inMinnesota Supercomputer Center, 1994

    1994

  48. [56]

    Pythia: A suite for analyzing large language models across training and scaling,

    S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raffet al., “Pythia: A suite for analyzing large language models across training and scaling,” inICML, 2023

    2023

  49. [57]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” inICLR, 2021

    2021

  50. [58]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inCVPR, 2016

    2016

  51. [59]

    A Krylov-Schur algorithm for large eigenproblems,

    G. W. Stewart, “A Krylov-Schur algorithm for large eigenproblems,” SIAM JMAA, 2002

    2002

  52. [60]

    R. B. Lehoucq, D. C. Sorensen, and C. Yang,ARPACK users’ guide: solution of large-scale eigenvalue problems with implicitly restarted Arnoldi methods. SIAM, 1998

    1998

  53. [61]

    Nocedal and S

    J. Nocedal and S. J. Wright,Numerical optimization. Springer, 2006

    2006

  54. [62]

    C. R. Johnson and R. A. Horn,Matrix analysis. Cambridge university press Cambridge, 1985

    1985

  55. [63]

    Localized Rademacher complexities,

    P. L. Bartlett, O. Bousquet, and S. Mendelson, “Localized Rademacher complexities,” inCOLT, 2002

    2002

  56. [64]

    Training transformers with enforced lipschitz constants,

    L. Newhouse, M. Csail, R. P. Hess, M. Bcs, F. L. Cesista, I. A. Zahorodnii, J. Bernstein, and P. Isola, “Training transformers with enforced lipschitz constants,”ArXiv, vol. 2507.13338, 2025

    2025 arXiv

  57. [65]

    Hiriart-Urruty and C

    J.-B. Hiriart-Urruty and C. Lemar ´echal,Convex analysis and minimiza- tion algorithms II: Advanced theory and bundle methods. Springer Berlin, Heidelberg, 1993

    1993

  58. [66]

    On the information loss in memoryless systems: The multivariate case,

    B. C. Geiger and G. Kubin, “On the information loss in memoryless systems: The multivariate case,”ArXiv, vol. 1109.4856, 2011

    2011 arXiv

  59. [67]

    Lecture notes in Rademacher composition and linear prediction,

    S. Kakade and A. Tewari, “Lecture notes in Rademacher composition and linear prediction,” 2008. SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 12

    2008

  60. [68]

    Ledoux and M

    M. Ledoux and M. Talagrand,Probability in Banach Spaces: Isoperime- try and Processes. Springer Berlin Heidelberg, 2013

    2013

  61. [69]

    Rademacher and gaussian complexi- ties: Risk bounds and structural results,

    P. L. Bartlett and S. Mendelson, “Rademacher and gaussian complexi- ties: Risk bounds and structural results,”JMLR, 2002

    2002

  62. [70]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” inICLR, 2019

    2019

  63. [71]

    ImageNet large scale visual recognition challenge,

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet large scale visual recognition challenge,”IJCV, 2015

    2015

  64. [72]

    TorchVision: PyTorch’s com- puter vision library,

    TorchVision maintainers and contributors, “TorchVision: PyTorch’s com- puter vision library,”GitHub repository, 2016

    2016

  65. [73]

    Loss- less compression on the GPU,

    A. Subramaniam, B. Karsin, D. LaSalle, G. Thomas-Collignon, M. Nicely, M. Milakov, M. Fan, N. Sakharnykh, and O. Lapicque, “Loss- less compression on the GPU,” https://developer.nvidia.com/nvcomp, 2021

    2021

  66. [74]

    torch ans,

    worldlife123, “torch ans,” https://github.com/worldlife123/torch ans, 2026. Anderson de Andrade(S’22) received his M.Sc. in Applied Computing from the University of Toronto in 2015 and obtained a B.Eng. degree in Networks and Communications in 2007 from Universidad Tec- nol´og...

    2026

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.