REVIEW 4 major objections 6 minor 1 cited by
Exploring Information Processing in Large Language Models: Insights from Information Bottleneck Theory
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims LLMs compress inputs into task-specific spaces, then decompress them at critical generation steps, with compression far stronger than decompression.
desk verdict Worthwhile question, but the evidence is circular and one method's loss is a no-op; reject this version but send it to referees. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the 'task space': a set of direction vectors built by subtracting hidden states from positive and negative prompts for each label, e.g., $h^l_t = h^+_{t,l} - h^-_{t,l}$, then purified with PCA. Mutual information between a sample's projected hidden state and the task space, estimated by a K-nearest-neighbor density approximation $\frac{1}{N}\sum_i \log(\hat{p}(x_i,y_i)/(\hat{p}(x_i)\hat{p}(y_i)))$, supplies the compression and decompression curves. The interventions that add or subtract the task-space vector at specific time steps, and the two training methods that inject weighted task vectors into hidden layers, are the mechanisms that turn the diagnosis into a recipe.
What would settle it
A decisive check is to build the task space from prompt pairs that differ only in the emotion label while keeping the dialogue context word-for-word identical, and then see whether the mutual-information curves still rise in the middle layers; if they flatten, the compression pattern is a wording artifact, not task processing.
Extended reading notes
Core claim
On the paper's own account, an LLM's layer-by-layer hidden states trace a predictable arc: early layers fluctuate as syntax and grammar are processed, then mutual information with the input falls while mutual information with a constructed emotion space rises, indicating that the model has routed the input into a low-dimensional task representation. During generation, mutual information with that task space peaks at the 0th, 4th, and 5th time steps, and perturbing the hidden state along the task-space direction at those steps shifts accuracy asymmetrically—adding the direction helps, subtracting hurts. The paper reads this as the model compressing well but failing to fully decompress, and it treats this weak-decompression diagnosis as the reason interventions that re-inject task-space vectors into hidden layers improve accuracy and speed.
Load-bearing premise
The whole argument rests on two premises: the direction obtained by subtracting hidden states from positive and negative prompts actually isolates the task being solved, and the statistical measure used to compare how much information layers share is reliable enough to tell layers apart; if either premise fails, the compression and decompression conclusions do not follow.
Editorial extensions
If this is right
- Injecting the task-space direction at decompression-critical steps should work as a general generation-time steering recipe; the paper's intervention at steps 4–5 raised accuracy from 35 to 82 on the emotion task.
- In-context learning can be sped up by replacing full retrieved examples with their compressed task-space vectors; the paper reports over 40% faster inference with better accuracy.
- Fine-tuning with a space-guided MSE loss should improve performance on tasks where the bottleneck is decompression, without changing the model architecture or requiring complex objectives.
- The non-training task-space probe gives a layer-by-layer diagnostic for where an LLM has committed to a task interpretation, so misclassifications can be localized to compression or decompression failures.
Reading between the lines
- Beyond the paper, the strong-compression/weak-decompression asymmetry suggests that decoding-time steering methods should target the specific layers and time steps identified by the probe, rather than applying uniform interventions across all layers.
- An unstated corollary is that the same direction-vector construction could be extended to regression or open-ended generation by using contrastive prompts for coarse categories, turning the mutual-information curves into a general-purpose alignment diagnostic.
- A testable extension is to check whether the reported over-40% inference speedup holds when the number of retrieved examples grows, because the compression step still requires retrieval and vector addition and the paper's comparison uses a small fixed set.
- The KNN mutual-information estimator is known to be biased in high dimensions; if the cross-layer comparison were repeated with a parametric or variational estimator, the reported compression trend could be confirmed or revealed as an artifact of neighborhood structure.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a non-gradient 'task space' construction for LLMs by subtracting hidden states from positive/negative prompts for each emotion label and applying PCA. It then measures mutual information between layer-wise hidden-state projections and these task-space directions, interpreting the observed MI trends as evidence that LLMs compress input toward the task space. A perturbation experiment adds or subtracts the gold-label direction vector to hidden states, and the resulting accuracy jump is interpreted as evidence that LLMs struggle to decompress. Based on this mechanism, the authors propose two methods, IC-ICL and TS-FT, and evaluate them on Empathetic Dialogues with Llama-3.1-8B.
Significance. The idea of using direction vectors as interpretable task axes is reasonable, and the IC-ICL result (accuracy improvement with a large inference-speed gain over ICL) is a concrete empirical contribution. However, the central mechanistic claim rests on uncontrolled MI comparisons and on a perturbation experiment that injects the gold-label direction into late hidden states, so the claimed compression/decompression mechanism is not established. The TS-FT loss, as written, is not a coherent training signal. If the mechanism were validated with proper controls, the paper would be of interest to the interpretability and in-context-learning communities, but in its current form the contribution is not demonstrated.
major comments (4)
- [§2.3, Eqs. (9)–(12)] The MI evidence for compression is uncontrolled. The task-space axis for each sample is built from the model's own hidden states for that sample's ground-truth label (Eqs. (3)–(4), (7)–(8)), and hp,l is the projection of the hidden state onto that same label-derived axis (Eq. (9)). This makes I(hp,l; espace) a measure of self-projection rather than an independent test of task-space compression. No null controls are reported, such as random directions, permuted labels, or other-task directions, and the KNN estimator in Eq. (12) is used without reporting k, N, or error bars. Since these MI trends are the only support for the central compression claim, they do not establish it.
- [§2.4.2, Table 2] The key-step experiment is confounded by label leakage. Adding the ground-truth emotion direction to the hidden state at steps 4–5 raises accuracy from 35.35 to 81.92, which is exactly what one expects when injecting the label's direction into late layers regardless of any decompression mechanism. The same procedure with random or other-label directions is needed before the result can be attributed to weak decompression, and the term 'decompression' is never given an operational definition independent of adding the direction vector.
- [§3.2, Eqs. (23)–(25)] The MSE loss in Eq. (24) simplifies to Lmse = (1/d) Σi (h_l,e*)², since h* = h + h_e*. If h_e* is a fixed precomputed vector, this term is constant and supplies no gradient during fine-tuning; if it is recomputed from the current model, the loss drives the injected direction to zero. In neither case does the loss encourage hidden layers to converge toward a better-quality direction, so the TS-FT mechanism does not work as described.
- [§4, Table 3] The evaluation does not support the abstract's claim of validation 'across multiple datasets.' Only Empathetic Dialogues and one base model are used; the TS-FT gains over baseline are 0.95 accuracy and 1.02 F1, which are not shown to be significant, and no ablation or significance testing is reported. The central mechanism therefore lacks both internal and external validation.
minor comments (6)
- [§2.3.2, Figures 5–6] The text says Figures 5 and 6 use dk = 1, but the figure captions report dk = 2 and dk = 5; please correct the mismatch.
- [§2.3.1, Eq. (12)] Equation (12) omits all parameters of the KNN MI estimator; the number of neighbors k, the number of samples N, and the dimensionality of the spaces should be reported, along with error bars or confidence intervals.
- [§3.1, Eqs. (15)–(16)] Equation (16) introduces k1 while Eq. (15) defines k2; the relationship between k1 and k2 is never specified.
- [§3.2, Eq. (24)] After Eq. (24), the text says 'n and L are the dimensionality and number of layers' but Eq. (24) uses d, not n; the notation should be made consistent.
- [§4, Table 3] The 'over 40%' acceleration claim is only relative to ICL (14:17 to 7:37); please state the baseline explicitly and report variance across runs.
- [§3.1] The paper says IC-ICL 'compresses retrieved example information into the task space,' but the method as described simply adds weighted direction vectors to hidden states; the compression interpretation is not formalized.
Circularity Check
Compression/decompression evidence is partially self-constructed: projected representations are measured against the same label-derived axes used to define them, and 'weak decompression' is inferred from injecting the ground-truth direction at test time.
-
self definitional
[Section 2.3.1, Eqs. (7)-(10) and (12); Section 2.3.2]
"For sample si, we define its ground-truth emotion label as esi. ... etop = Topcosine_dk(esi, ej) ... hp,l_j = (h^l_j · e^{sl,top}_{dk})/(|e^{sl,top}_{dk} · e^{sl,top}_{dk}|) e^{sl,top}_{dk} ... Using the method described above, the mutual information I(hp,l_si ; esl,top_dk) between the sample si and the emotion space esl,top_dk at the l-th layer can be obtained."
The quantity used as evidence that LLMs compress into a task space is not an independent measurement: h^{p,l}_{s_i} is defined as the orthogonal projection of the hidden state onto e^{sl,top}_{dk}, the very emotion-space vector selected from the ground-truth label. By construction h^{p,l} is collinear with the target emotion space, so I(h^{p,l}; e) measures the behavior of a projection onto a label-derived axis rather than an independently defined task representation. No random-direction, permuted-label, or other-task controls are reported, so the observed increase of MI with layer may be an artifact of projecting onto the label-dependent construction rather than evidence of compression into a task-specific space.
-
self definitional
[Section 2.4.2, Table 2 and following paragraph]
"At each time step t, we directly add or subtract the ground-truth emotion space to the LLM’s hidden state to examine the effect of each time step. ... Step 4-5 Addition 81.92 ... Furthermore, when the hidden state is added to the emotion space, the LLMs’ performance greatly improves. This suggests that LLMs struggle to decompress high-quality information on their own, and better decompression requires additional strategies or support to effectively extract the desired information."
The claim that LLMs have weak decompression is inferred from the accuracy jump obtained by adding the ground-truth emotion direction to the hidden state at prediction time. That direction is constructed from the correct label via Eqs. (3)-(4), so the intervention is equivalent to supplying the answer to the model; the improvement is expected by construction and cannot measure whether the model's own internal decompression is weak. The same oracle quantity is then used to motivate TS-FT, whose MSE loss pushes hidden states toward that label-derived direction, so the evidence for the mechanism and the design of the method share the same ground-truth input.
full rationale
The paper's central mechanistic claims are partially guaranteed by construction. In Section 2.3, the 'task space' is built from direction vectors computed from the model's own hidden states for ground-truth emotion labels (Eqs. 3-4), and the sample representation whose mutual information is measured is defined as a projection onto that same label-derived space (Eqs. 9-10). Thus the observed compression trend is not independent evidence that the model compresses into an intrinsic task space; it is at least partly an artifact of the measurement construction. Similarly, the compression/decompression asymmetry is supported by adding the ground-truth emotion direction at prediction time and observing improved accuracy, which is an oracle injection rather than a measurement of the model's own decompression ability. The proposed IC-ICL and TS-FT methods are evaluated on standard benchmarks without oracle labels at inference, giving some independent empirical content, and there is no load-bearing self-citation chain (the cited direction-subtraction method is external work by Liu et al., 2024). However, because the evidence for the central 'compress into task space, weakly decompress' claim reduces largely to the way the task space and projections are defined, the circularity score is 6 rather than higher.
Assumptions & free parameters
free parameters (6)
- dk (number of nearest emotion directions) =
1, 2, 5, 32
- k1/k2 (retrieval counts in IC-ICL) =
not reported
- we (fixed direction weight) =
not reported
- wa (attention-guided weight) =
not reported
- wmse (MSE loss weight) =
not reported
- PCA retained dimensions =
not reported
assumptions (4)
- domain assumption KNN-based MI estimation (Eq. 12) yields unbiased layer-wise comparisons.
- ad hoc to paper Positive/negative prompt subtraction isolates emotion direction.
- domain assumption Increasing MI with the constructed space indicates IB-style compression (Sec. 2.3.2).
- ad hoc to paper Adding the emotion direction vector to hidden states simulates decompression (Sec. 2.4.2).
invented entities (1)
-
task space (e.g., emotion space)
Cite this review
Pith. "Pith review of Exploring Information Processing in Large Language Models: Insights from Information Bottleneck Theory." pith.science (2026). https://pith.science/paper/TNDPOF5J
@misc{pith2026250100999,
author = {Pith},
title = {Pith review of: Exploring Information Processing in Large Language Models: Insights from Information Bottleneck Theory},
year = {2026},
howpublished = {\url{https://pith.science/paper/TNDPOF5J}},
note = {Machine review of arXiv:2501.00999}
}
read the original abstract
Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of tasks by understanding input information and predicting corresponding outputs. However, the internal mechanisms by which LLMs comprehend input and make effective predictions remain poorly understood. In this paper, we explore the working mechanism of LLMs in information processing from the perspective of Information Bottleneck Theory. We propose a non-training construction strategy to define a task space and identify the following key findings: (1) LLMs compress input information into specific task spaces (e.g., sentiment space, topic space) to facilitate task understanding; (2) they then extract and utilize relevant information from the task space at critical moments to generate accurate predictions. Based on these insights, we introduce two novel approaches: an Information Compression-based Context Learning (IC-ICL) and a Task-Space-guided Fine-Tuning (TS-FT). IC-ICL enhances reasoning performance and inference efficiency by compressing retrieved example information into the task space. TS-FT employs a space-guided loss to fine-tune LLMs, encouraging the learning of more effective compression and selection mechanisms. Experiments across multiple datasets validate the effectiveness of task space construction. Additionally, IC-ICL not only improves performance but also accelerates inference speed by over 40\%, while TS-FT achieves superior results with a minimal strategy adjustment.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
Information-Theoretic Limits of Reliability and Scaling in Language Models
A theoretical framework derives a reliability ceiling and a max-form Chinchilla-type scaling law for LLMs from task entropy and dependency spectra.
Reference graph
Works this paper leans on
-
[3]
arXiv preprint arXiv:2212.04037
Demystifying prompts in language models via perplexity estimation. arXiv preprint arXiv:2212.04037. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly
-
[5]
Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685. Brian Lester, Rami Al-Rfou, and Noah Constant
-
[6]
arXiv preprint arXiv:2212.06800
Diverse demonstrations improve in-context compositional generalization. arXiv preprint arXiv:2212.06800. Xiang Lisa Li and Percy Liang
-
[9]
Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych
-
[11]
arXiv preprint arXiv:2112.08633
Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.08633. Meina Song, Zecheng Zhan, and E Haihong
-
[12]
Self-consistency improves chain of thought reasoning in language models. ArXiv, abs/2203.11171. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al
-
[13]
Tree of thoughts: Deliberate problem solving with large language models. ArXiv, abs/2305.10601
-
[2019]
Parameter-efficient transfer learning for nlp. ArXiv, abs/1902.00751. J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen
arXiv 1902
Show all 13 references
-
[2020]
ArXiv, abs/2005.00247
Adapterfusion: Non-destructive task composition for transfer learning. ArXiv, abs/2005.00247. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al
2005 arXiv
-
[2021]
ArXiv, abs/2104.08821
Simcse: Simple contrastive learning of sentence em- beddings. ArXiv, abs/2104.08821. Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer
-
[2022]
arXiv preprint arXiv:2212.02437
In- context examples selection for machine translation. arXiv preprint arXiv:2212.02437. Maciej Besta, Nils Blach, Aleš Kubí ˇcek, Robert Ger- stenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadom- ski, Piotr Nyczyk, and Torsten Hoefler
-
[2023]
icl: Demonstration-retrieved in-context learning
Dr. icl: Demonstration-retrieved in-context learning. arXiv preprint arXiv:2305.14128. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luk...
-
[2024]
Ctrla: Adaptive retrieval- augmented generation via inherent control. arXiv. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021a. What makes good in-context examples for gpt-3? In Work- shop on Knowledge Extraction and Integration for De...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.