Intuitive dissection of the Gaussian information bottleneck method with an application to optimal prediction

Age Tjalma; Pieter Rein ten Wolde; Vahe Galstyan

Intuitive dissection of the Gaussian information bottleneck method with an application to optimal prediction

T0 review · 0 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that the Gaussian information bottleneck's discrete transitions in representation dimensionality are controlled by a single equalization rule: a new encoding component switches on when the relevant information still…

desk verdict A clean interpretive restatement of Gaussian IB with genuinely useful new formulas for a prediction problem; worth refereeing, modest novelty. read the letter →

arxiv 2507.05183 v1 pith:C5SKCGFB submitted 2025-07-07 q-bio.MN cond-mat.stat-mechcs.ITmath.ITphysics.bio-ph

Vahe Galstyan , Age Tjalma , Pieter Rein ten Wolde This is my paper · ORCID

classification q-bio.MNcond-mat.stat-mechcs.ITmath.ITphysics.bio-ph MSC 94A1762B10

keywords informationbottleneckGaussianoptimalpredictionmutualstochasticharmonicoscillatorcanonicalcorrelationanalysissignalencodinggeometrictheory

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to explain why the optimal Gaussian signal representation is scalar at low encoding capacity and gains one additional component at discrete thresholds, rather than using all components from the start. It claims that every threshold is governed by the same rule: a new component is introduced exactly when the amount of relevant information that each existing component can still store becomes equal to the maximum relevant information available to the new, unused component. Geometrically, after standardizing the signal and relevance marginals to spherical Gaussians, this moment is when the decoding ellipse of the representation and the fixed mapping ellipse between relevance and signal have equal aspect ratios. The authors further show that past each threshold all active components are traversed identically as extra capacity is added, so the whole information curve is a piecewise staircase built by one repeated equalization step. Applying the perspective to prediction of a stochastically driven oscillator, they derive closed-form weights for encoding the current position and velocity and identify how the predictive roles of the two components alternate with the forecast interval.

What carries the argument

The argument runs on the geometry of the conditional-Gaussian ellipses after equal-variance standardization of $P(s)$ and $P(y)$. The relevant object is the pair formed by the decoding ellipse $P(y|z)$, whose semi-axis lengths $\Lambda_i$ shrink as each encoding component's noise is lowered, and the fixed mapping ellipse $P(y|s)$, whose semi-axis lengths are the square roots of the eigenvalues of $\Sigma_{y|s}$. Equality of the aspect ratios of these two ellipses is the transition condition, and it is equivalent to the information-capacity equalization rule. The noise strengths that realize optimal navigation are given by a closed expression $\sigma_i^2(\gamma) = r^2 \gamma \tilde{\lambda}_i^2 / ((1-\gamma) - \tilde{\lambda}_i^2)$ in terms of the Lagrange multiplier $\gamma$ and the eigenvalue $\tilde{\lambda}_i^2$ of the conditional covariance matrix; the first time $\gamma$ crosses $1-\tilde{\lambda}_i^2$, component $i$ turns on. Encoding directions themselves are the eigenvectors of $\Sigma_{s|y}$, the same basis as canonical correlation analysis.

What would settle it

Run a numerical information bottleneck calculation on a three-dimensional jointly Gaussian problem with a generic conditional covariance matrix: compute the capacity at which the optimal representation changes from one to two components, then check whether $I_{\max}(z_1;y) - I^\dagger(z_1;y) = I_{\max}(z_2;y)$ and whether the decoding ellipse and mapping ellipse have equal aspect ratios at that capacity. A mismatch would disprove the equalization rule; an equality failure at the second transition point would disprove the iteration of the rule to higher dimensions.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the Gaussian information bottleneck has a ratio-symmetric structure: the discrete transitions in the dimension of the optimal encoding happen when the residual relevant-information capacities of the active components match the maximum capacity of the next inactive component. In symbols, at the first transition $I_{\max}(z_1;y) - I^\dagger(z_1;y) = I_{\max}(z_2;y)$, and in higher dimensions the rule iterates, for example $I_b - I^\ddagger(z_2;y) = I_c$ for the second transition. The equivalent geometric condition is $\Lambda_1^\dagger/r = a/b$: the decoding ellipse of the representation and the relevance-to-signal mapping ellipse become similar. After a transition the aspect ratios stay constant, $\Lambda_1 : \Lambda_2 : \cdots = a : b : \cdots$, and the extra encoded information is divided equally among the active components. In the prediction application, the standardized variables make the optimal encoding angles explicit functions of the forecast interval and damping, showing that the leading component always adds the current derivative with the same sign as the current position, and that in underdamped dynamics the two components exchange priority as the forecast interval grows.

Load-bearing premise

The signal and the relevance variable must be described by a joint Gaussian distribution with a full-rank covariance matrix, so that the optimal encoder is linear, all uncertainty sets are ellipses or ellipsoids, and the standardization step loses nothing; for the prediction application, stationarity of the oscillator and the Markov property of the joint position-velocity process are also assumed.

Editorial extensions

If this is right

At each transition point, a new encoding component is introduced exactly when the relevant-information capacities of active components and the new component equalize; this fixes the location of every dimensionality change on the information curve.
After any transition, additional encoding capacity is distributed equally across active components, so the marginal gain in relevant information is the same from each component.
The aspect ratios of the decoding ellipse and the signal-to-relevance mapping ellipse are locked after each transition; in the infinite-capacity limit the two ellipsoids coincide and $I(z;y)$ reaches $I(s;y)$.
For harmonic-oscillator prediction, the optimal encoding weights follow from a compact arctangent formula; the leading component weights current position and derivative with the same sign, and in the underdamped regime the component carrying more predictive information alternates with forecast interval.
The relative predictive capacities of the two prediction components converge as damping weakens, so accurate prediction at long forecast intervals requires a second encoding direction in the underdamped regime.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial: the equalization rule suggests a general resource-allocation principle, “fill the most informative component first, then top up all active components equally,” which could be tested for non-Gaussian encoders or for generalized information measures such as Rényi or Jeffreys divergences; the paper explicitly leaves that generalization open.
Editorial: the closed-form encoding angles for the damped oscillator provide a direct benchmark for cellular prediction networks; one could compare experimentally inferred readout weights against the predicted cosine and sine weights and use any systematic mismatch to diagnose non-Gaussian statistics or hidden constraints.
Editorial: a concrete extension would be to measure the first transition capacity $I^\dagger(z;s)$ in a finite-capacity channel carrying the harmonic-oscillator signal; if the observed transition deviates from the formula derived from the aspect-ratio condition, the equalization rule would need a correction term.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

I've read the Galstyan/Tjalma/ten Wolde manuscript. The headline: the central 'equalization criterion' is a correct but mostly restated result from Chechik et al., not a new theorem; what's genuinely new is the set of analytic formulas for the prediction problem. A serious referee should be assigned.

The paper does something useful: it recasts Gaussian IB in standardized coordinates so the P(y|z) and P(y|s) ellipses can be compared directly. The transition condition—introduce a new encoding component when the remaining relevant capacity of active components matches the maximum capacity of the unused one—is derived cleanly and matches the γ-threshold solution. The geometric aspect-ratio picture is a nice way to see why the transitions occur. I checked the derivations in Secs. IV-V and appendices C-D; they are internally consistent. The Gaussian assumption is stated up front, and the appendices give enough detail to reproduce. No fitted parameters; no circularity.

The prediction application is where the paper earns novelty. The optimal encoding angle (Eq. 45) and the asymptotic relative predictive capacities (Eq. 47) for the damped oscillator are new analytic results. The point that a purely x0-based strategy is suboptimal and that the derivative matters is demonstrated concretely. This part is a real extension of Sachdeva et al.

Soft spots, in proportion. First, the equalization rule is not a new mathematical result; it's a restatement of Chechik's threshold structure. The paper acknowledges the prior work but the title says 'intuitive dissection,' so that's acceptable—it's a didactic contribution. Second, the thermodynamic analogy with boxes is heuristic, not load-bearing; it's fine as an aside but shouldn't be treated as an independent derivation. Third, the non-Gaussian scope is limited; the authors note this in Sec. VII. Fourth, the paper is long and somewhat repetitious; the component-wise information plots could be condensed.

Overall: sound, useful for people in cellular prediction and Gaussian IB who want intuition. It deserves peer review. I'd accept after minor revision, mainly asking for tighter prose and a clearer statement that the equalization criterion is a reinterpretation, not a new result. I would cite it for the prediction formulas.

Referee Report

0 major / 5 minor

Summary. This paper provides a geometric and information-theoretic dissection of the Gaussian information bottleneck (GIB) method. After standardizing the marginal distributions of the signal and relevance variables, the authors derive the optimal encoding directions as the eigenvectors of the conditional covariance matrix Σ_{s|y} and characterize the optimal allocation of encoding noise. Their central contribution is an equalization rule: a new encoding dimension is introduced when the amount of relevant information that each active component can still store becomes equal to the maximum relevant information available to the next unused component. Equivalently, in the geometric picture, this occurs when the aspect ratios of the P(y|z) and P(y|s) ellipsoids become equal. The rule is shown to reproduce the known piecewise structure of the GIB solution, and it is applied to a signal prediction problem for a stochastically driven damped harmonic oscillator, yielding concise formulas for the optimal encoding angles and predictive capacities as functions of the forecast interval.

Significance. If the results hold, the paper offers a genuinely intuitive way to understand the discrete transitions in the dimensionality of optimal representations in the Gaussian information bottleneck. The derivations are self-contained, and the equalization rule is a new formulation that may be useful for interpreting numerical solutions and for teaching purposes. The prediction application illustrates the power of the approach and provides quantitative predictions for a canonical model. The strengths include the explicit analytical treatment of the standardization procedure, the discussion of the degenerate space of optimal solutions, and the clear geometric visualization of information-theoretic quantities. The paper is a valuable contribution to the conceptual understanding of the GIB method.

minor comments (5)

[Section V, Eq. (34)] The expression for σ_i²(γ) is valid only for active components (γ < γ_c^i); the text should state this explicitly, since for inactive components one has σ_i = ∞.
[Appendix A3] The symbol 'Inz' used in the standardization proof is undefined; it should be written as I_{n_z} with a brief definition.
[Section V, 'Comment on the degenerate space'] The discussion of energy-cost implications following Eq. (40) is speculative; the authors should clearly label it as a hypothesis about potential functional consequences, not as a proven result.
[Figure 1b caption] The caption's statement that 'the three curves terminate at the same value of γ' is slightly misleading; the curves are drawn for a fixed γ at their right endpoints, which is not obvious without additional context.
[Section VI, Eq. (45)] The definitions of κ and ω appear after the equation; moving them just before Eq. (45) would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the equalization criterion is re-derived from the Gaussian IB optimization, not fitted or self-citational.

full rationale

We inspected the derivation chain. The paper's central claim is the equalization criterion for the onset of new encoding components (Eqs. 23 and 32). It is obtained by maximizing I(z;y) for fixed I(z;s): Appendix C2 solves the constrained optimization for the noise strengths θ*1 and θ*2 and identifies the critical encoding information Ωc, and Eqs. 21-23 are exact algebraic rearrangements of the resulting transition condition Λ1†/r = a/b. This is not a fitted parameter renamed as a prediction. The general high-dimensional formulas, Eqs. 34-39, are re-derived from the Lagrangian L = I(z;y) - γI(z;s) in Appendix D1; they agree with Chechik et al. but are derived in the text rather than imported by self-citation. Self-citations [8] and [16] are used only for the oscillator prediction setup and for nonlinear-cost trade-off caveats; they do not carry the derivation. The prediction application in Sec. VI contains no data fitting: the encoding angles (Eq. 45) and information ratios follow analytically from the stationary covariance of the oscillator. Overall, the derivation chain is self-contained, and no circular step was found.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted: the scale r is a normalization constant, η and τ are model inputs, and the encoding noises σ_i are optimization variables, not fitted constants. The axioms are the standard Gaussian/linear assumptions of the Gaussian IB and the physical model assumptions of the prediction application.

assumptions (4)

domain assumption s and y are jointly Gaussian with full-rank covariance matrices.
This is the defining assumption of the Gaussian information bottleneck, stated in Sec. II; it makes the optimal encoding linear and all information measures quadratic.
standard math The optimal encoder can be restricted to the noisy linear form z = W^T s + ξ with diagonal noise and orthonormal encoding directions.
Proven in Appendix A3 by a change of variables equivalent to Chechik et al.; not an ad hoc restriction.
standard math Standardization of the marginal distributions P(s) and P(y) does not change mutual information.
Proven in Appendix A1 using the Schur complement formula; all subsequent geometry relies on this.
domain assumption In the prediction application, the signal is stationary and the pair (x,v) is Markovian, so y=(xτ,vτ) fully specifies future statistics.
Assumed in Sec. VI and Appendix E; it is true for the linear damped oscillator, and is the bridge from trajectory prediction to the two-dimensional bottleneck.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Intuitive dissection of the Gaussian information bottleneck method with an application to optimal prediction." pith.science (2026). https://pith.science/paper/C5SKCGFB

@misc{pith2026250705183,
  author       = {Pith},
  title        = {Pith review of: Intuitive dissection of the Gaussian information bottleneck method with an application to optimal prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C5SKCGFB}},
  note         = {Machine review of arXiv:2507.05183}
}

read the original abstract

Efficient signal representation is essential for the functioning of living and artificial systems operating under resource constraints. A widely recognized framework for deriving such representations is the information bottleneck method, which yields the optimal strategy for encoding a random variable, such as the signal, in a way that preserves maximal information about a functionally relevant variable, subject to an explicit constraint on the amount of information encoded. While in its general formulation the method is numerical, it admits an analytical solution in an important special case where the variables involved are jointly Gaussian. In this setting, the solution predicts discrete transitions in the dimensionality of the optimal representation as the encoding capacity is increased. Although these signature transitions, along with other features of the optimal strategy, can be derived from a constrained optimization problem, a clear and intuitive understanding of their emergence is still lacking. In our work, we advance our understanding of the Gaussian information bottleneck method through multiple mutually enriching perspectives, including geometric and information-theoretic ones. These perspectives offer novel intuition about the set of optimal encoding directions, the nature of the critical points where the optimal number of encoding components changes, and about the way the optimal strategy navigates between these critical points. We then apply our treatment of the method to a previously studied signal prediction problem, obtaining new insights on how different features of the signal are encoded across multiple components to enable optimal prediction of future signals. Altogether, our work deepens the foundational understanding of the information bottleneck method in the Gaussian setting, motivating the exploration of analogous perspectives in broader, non-Gaussian contexts.

Figures

Figures reproduced from arXiv: 2507.05183 by the authors.

**Figure 2.** FIG. 2. Visualization of the key probability distributions and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** FIG. 3. Graphical perspective on optimality in the one-dimensional encoding setting. (a) Formation of the conditional [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (10 more)

**Figure 4.** Figure 4: FIG. 4. Relevant vs. encoded information in the scalar [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: FIG. 5. Alternative perspective on optimality in the scalar encoding scenario. (a) Decoding of the signal [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: a illustrates the formation of the decoding distribution P(s|z). As can be seen, the ellipse corresponding to P(s|z) is compressed along both of its axes, which have half-lengths ℓi = rσi/ p r 2 + σ 2 i . These lengths are the semi-minor axis lengths of the scalar enc…

**Figure 7.** Figure 7: FIG. 7. Distributions [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

**Figure 8.** Figure 8: FIG. 8. Geometric illustration of the increasing representation complexity under the optimal strategy. Ellipses corresponding [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

**Figure 9.** Figure 9: FIG. 9. Increase in encoding complexity and behavior past [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]

**Figure 10.** Figure 10: FIG. 10. Geometric perspective on the transitions in representation complexity with growing encoding capacity in a three [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]

**Figure 11.** Figure 11: FIG. 11. Increasing representation complexity illustrated on the information plane in the case where signal and relevance [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]

**Figure 12.** Figure 12: FIG. 12. Statistical structure of an example underdamped signal with [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]

**Figure 13.** Figure 13: FIG. 13. Features of the optimal encoding strategy for pre [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Theoretical Analysis of Resource-Induced Phase Transitions in Estimation Strategies
physics.bio-ph 2025-11 conditional novelty 6.0 of 10

For a minimal linear-Gaussian estimation model, optimal memory use is governed by dimensionless discriminants ΘQ and ΘM, yielding sudden, nonmonotonic, scale-invariant phase transitions.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages · cited by 1 Pith paper

[1]

Optimal encoding direction In the main text, we considered the scalar encoding of a two-dimensional signal in the form z = ˆw · s + ξ, where the unit vector ˆw is the encoding direction. The information encoded in z about the signal s is given by I(z; s) = 1 2 log σ2 z σ2 z|s ! = 1 2 log 1 + r2 σ2 , (B1) where r2 is the variance of each signal component, ...

work page
[2]

I(z; s) curve Considering Eq

Slope of the I(z; y) vs. I(z; s) curve Considering Eq. B1 for the encoded information, we can also write the relevant information as I opt(z; y) = I(z; s) − 1 2 log 1 + a2 σ2 . (B11) This form makes it clear that I opt(z; y) < I(z; s), and that the slope ∂I opt(z; y)/∂I (z; s) of the I opt(z; y) vs. I(z; s) curve is always less than 1 (see Fig. 4 of the m...

work page
[3]

Decoding

“Decoding” perspective Next, we derive the results presented in Sec. III of the main text concerning the alternative “decoding” perspective on the problem. The two key distributions in this perspective are P (s|z) and P (y|z). We will start off by deriving expressions for the means and variances of these two distributions. To derive the means, we will use...

work page
[4]

decoding

Optimality in the “decoding” perspective Now, we discuss what the optimal encoding strategy means in the decoding perspective. Our starting point will be the optimality condition derived earlier (Eq. B8), which states that the optimal encoding direction ˆw∗ 1 is the eigenvector of the conditional covariance matrix Σs|y with the lower corresponding eigenva...

work page
[5]

A13), we can rewrite the above condition as r2(Is − ˜Σsy ˜Σys) ˆw∗ 1 = a2 ˆw∗

(B36) Considering the expanded form Σs|y = r2(Is − ˜Σsy ˜Σys) (see Eq. A13), we can rewrite the above condition as r2(Is − ˜Σsy ˜Σys) ˆw∗ 1 = a2 ˆw∗

work page
[6]

(B38) Identifying familiar quantities (Eq

(B37) 9 Then, we multiply both sides by ˜Σys and after some rearrangements find r2 Iy − ˜Σys ˜Σsy Σy|s ˜Σys ˆw∗ 1 v∗ 1 = a2 ˜Σys ˆw∗ 1 v∗ 1 . (B38) Identifying familiar quantities (Eq. A14 for Σy|s and Eq. B26 for v1), we rewrite Eq. B38 in its final form as Σy|sv∗ 1 = a2v∗

work page
[7]

5c in the main text)

(B39) This form shows that v∗ 1 is the eigenvector of Σy|s with the lowest eigenvalue a2, and hence implies that, under the optimal encoding strategy, the direction v∗ 1 along which the encoding variable z retains information about the relevance variable y matches the direction along which the s → y stochastic mapping has the least uncertainty (see also F...

work page
[8]

(B42) The corresponding v2 vector is similarly an eigenvector of Σy|s with the larger corresponding eigenvalue: Σy|sv∗ 2 = b2v∗

work page

Show all 26 references

[9]

(B44) Substituting ||v∗ 2||2 into Eq

(B43) It is perpendicular to v∗ 1 and has a squared magnitude given by ||v∗ 2||2 = 1 − ˜b2. (B44) Substituting ||v∗ 2||2 into Eq. B35, we obtain the semi-minor axis length of the P (y|z) distribution under the least optimal strategy to be Λ2 = r r 1 − r2 r2 + σ2 ||v∗ 2||2 (B45...
[10]

B16 and Eq

Statistics of P (s|z) and P (y|z) distributions To compute the means and covariance matrices of the two conditional distributions P (s|z) and P (y|z), we will use the same general formulae introduced in the scalar encoding case (Eq. B16 and Eq. B15). Starting with the mean of ...
[11]

As we will show, the optimal assignment will indicate a transition from scalar to two-dimensional encoding when the amount of encoded information becomes sufficiently large

T ransition from scalar to two-dimensional optimal encoding We now proceed to discuss the question of optimally assigning the encoding noises σ1 and σ2 for different amounts of fixed encoded information. As we will show, the optimal assignment will indicate a transition from s...
[12]

(C24) The fixed Ω constraint in Eq

(C23b) For convenience, we now introduce θi = r2/σ2 i . (C24) The fixed Ω constraint in Eq. C21 can be rewritten as Ω2 = (1 + θ1) (1 +θ2) . (C25) Maximizing I(z; y) for given Ω (see Eq. C23b) then becomes equivalent to minimizing h(θ1, θ2) = 1 + ˜a2θ1 1 + ˜b2θ2 , (C26) where ˜...
[13]

Starting with the dimensions ℓ∗ 1 and ℓ∗ 2 of the P (s|z) ellipse (see Eq

Behavior beyond the transition point We now discuss the implications of optimal noise allocation beyond the transition point where Ω > Ωc or, equiv- alently, I(z; s) > I†(z; s). Starting with the dimensions ℓ∗ 1 and ℓ∗ 2 of the P (s|z) ellipse (see Eq. C14), we write ℓ∗ i = rσ...
[14]

Solution to the Gaussian information bottleneck problem The problem of maximizing the relevant information I(z; y) for a given amount of encoded information I(z; s) can be formulated generally as a constrained optimization problem, where the functional L[p(z|s)] = I(z; y) − γI...
[15]

+ σ2 1 + (σ2 2 − σ2 1) = (λ2 1 + σ2 1)2 + (λ2 1 + σ2 1) (λ2 2 − λ2
[16]

+ (σ2 2 − σ2 1) , (D24a) case 2: λ2 2 + σ2 1 λ2 1 + σ2 2 = λ2 1 + (λ2 2 − λ2
[17]

+ σ2 1 λ2 1 + σ2 1 + (σ2 2 − σ2 1) = λ2 1 + σ2 1 2 + λ2 1 + σ2 1 (λ2 2 − λ2
[18]

(D24b) As can be seen, the product in the second case contains an extra non-negative term because, by our chosen convention, λ2 ≥ λ1 and σ2 ≥ σ1

+ (σ2 2 − σ2 1) + (λ2 2 − λ2 1) ≥0 (σ2 2 − σ2 1) ≥0 . (D24b) As can be seen, the product in the second case contains an extra non-negative term because, by our chosen convention, λ2 ≥ λ1 and σ2 ≥ σ1. What this means is that in the case of optimal two-dimensional encoding, the ...
[19]

These results are analogous to those in the original work of Chechik et al

And in the limit γ → 0, which corresponds to the limit of infinite encoded information, all components are used. These results are analogous to those in the original work of Chechik et al. [10]. At the end, to demonstrate the generality of our geometric interpretation of criti...
[20]

Degenerate space of optimal solutions The general solution to the Gaussian information bottleneck problem presented in the first part of this Appendix section consisted of an orthogonal set of encoding directions { ˆwi} and noise strengths {σ2 i (γ)} associated with these dire...
[21]

The problem of finding the mixing matrix M from Eq

and the least informative ( ˆw∗ n) principal encoding directions. The problem of finding the mixing matrix M from Eq. D42 that makes such an alternative strategy possible is generally underspecified for n ≥ 3 dimensions and may therefore allow for multiple solutions. In two di...
[22]

V of the main text is valid

Thermodynamic analogy for the optimal allocation of encoding capacity Here, we explain why the thermodynamic analogy presented in Sec. V of the main text is valid. We consider a lattice model for the boxes of varying sizes, with the ith box having Ni sites. The different sizes...
[23]

In it, an object of mass m is connected via a spring to the origin and experiences stochastic kicks from the viscous environment

Stochastically driven damped harmonic oscillator model The physically motivated model that we consider in the prediction problem is that of a stochastically driven damped harmonic oscillator. In it, an object of mass m is connected via a spring to the origin and experiences st...
[24]

cosh(κt) + η 2κ sinh(κt) 1 κ sinh(κt) − 1 κ sinh(κt) cosh( κt) − η 2κ sinh(κt) # e−ηt/2, (E34a) η <2 : eFt =

Solution of the stochastic model We now proceed to solve the system of stochastic differential equations for the dimensionless position and velocity (Eq. E18), skipping the “tilde” symbols for convenience. To that end, we first recast Eq. E18 in a matrix form, namely d dt x v ...
[25]

The key quantity that dictates the properties of the optimal strategy is the covariance matrix Σs0|sτ , where τ stands for the prediction interval

Optimal prediction strategy Having obtained the signal statistics, we now turn to the question of optimal prediction. The key quantity that dictates the properties of the optimal strategy is the covariance matrix Σs0|sτ , where τ stands for the prediction interval. The princip...
[26]

left-tilted

values at φ = φ1 and φ = φ2, respectively. Therefore, to find these angles, we differentiate the left-hand side by φ and set it equal to zero, obtaining − sin(2φ) σ2 x0|sτ − σ2 v0|sτ + 2 cos(2φ) cov(x0, v0|sτ ) = 0. (E40) After some rearrangements, the condition for φ ∈ {φ1, φ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Optimal encoding direction In the main text, we considered the scalar encoding of a two-dimensional signal in the form z = ˆw · s + ξ, where the unit vector ˆw is the encoding direction. The information encoded in z about the signal s is given by I(z; s) = 1 2 log σ2 z σ2 z|s ! = 1 2 log 1 + r2 σ2 , (B1) where r2 is the variance of each signal component, ...

work page

[2] [2]

I(z; s) curve Considering Eq

Slope of the I(z; y) vs. I(z; s) curve Considering Eq. B1 for the encoded information, we can also write the relevant information as I opt(z; y) = I(z; s) − 1 2 log 1 + a2 σ2 . (B11) This form makes it clear that I opt(z; y) < I(z; s), and that the slope ∂I opt(z; y)/∂I (z; s) of the I opt(z; y) vs. I(z; s) curve is always less than 1 (see Fig. 4 of the m...

work page

[3] [3]

Decoding

“Decoding” perspective Next, we derive the results presented in Sec. III of the main text concerning the alternative “decoding” perspective on the problem. The two key distributions in this perspective are P (s|z) and P (y|z). We will start off by deriving expressions for the means and variances of these two distributions. To derive the means, we will use...

work page

[4] [4]

decoding

Optimality in the “decoding” perspective Now, we discuss what the optimal encoding strategy means in the decoding perspective. Our starting point will be the optimality condition derived earlier (Eq. B8), which states that the optimal encoding direction ˆw∗ 1 is the eigenvector of the conditional covariance matrix Σs|y with the lower corresponding eigenva...

work page

[5] [5]

A13), we can rewrite the above condition as r2(Is − ˜Σsy ˜Σys) ˆw∗ 1 = a2 ˆw∗

(B36) Considering the expanded form Σs|y = r2(Is − ˜Σsy ˜Σys) (see Eq. A13), we can rewrite the above condition as r2(Is − ˜Σsy ˜Σys) ˆw∗ 1 = a2 ˆw∗

work page

[6] [6]

(B38) Identifying familiar quantities (Eq

(B37) 9 Then, we multiply both sides by ˜Σys and after some rearrangements find r2 Iy − ˜Σys ˜Σsy Σy|s ˜Σys ˆw∗ 1 v∗ 1 = a2 ˜Σys ˆw∗ 1 v∗ 1 . (B38) Identifying familiar quantities (Eq. A14 for Σy|s and Eq. B26 for v1), we rewrite Eq. B38 in its final form as Σy|sv∗ 1 = a2v∗

work page

[7] [7]

5c in the main text)

(B39) This form shows that v∗ 1 is the eigenvector of Σy|s with the lowest eigenvalue a2, and hence implies that, under the optimal encoding strategy, the direction v∗ 1 along which the encoding variable z retains information about the relevance variable y matches the direction along which the s → y stochastic mapping has the least uncertainty (see also F...

work page

[8] [8]

(B42) The corresponding v2 vector is similarly an eigenvector of Σy|s with the larger corresponding eigenvalue: Σy|sv∗ 2 = b2v∗

work page