Pith. sign in

REVIEW 3 major objections 5 minor 30 references

A Comparative Study of Learning Paradigms in Large Language Models via Intrinsic Dimension

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Across four models and eight datasets, in-context learning with five or more demonstrations yields higher intrinsic dimension in hidden representations than supervised fine-tuning or zero-shot prompting.

desk verdict A useful empirical survey of ID in LLM representations, but the headline ICL-vs-SFT comparison is confounded by input length and needs a control before the central claim can be believed. read the letter →

arxiv 2412.06245 v2 pith:G6G37HBX submitted 2024-12-09 cs.CL

Saahith Janapati , Yangfeng Ji This is my paper · ORCID
classification cs.CL
keywords intrinsicdimensionin-contextlearningsupervisedfine-tuninglargelanguagemodelshiddenrepresentationsTwoNNestimatorrepresentationgeometryfew-shot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the two main ways of teaching a large language model a new task—fine-tuning its weights or giving it examples in the prompt—leave different geometric footprints in the hidden representations, and that this difference shows up as a measurable quantity called intrinsic dimension (ID), the number of effective degrees of freedom in the set of hidden states. Across four open-weight models and eight datasets, prompts with five or more demonstrations consistently produce higher ID at every layer than both fine-tuned models and zero-shot prompts. Fine-tuned models, meanwhile, score highest on accuracy while staying closest to the zero-shot baseline in ID. The paper also finds that ID first rises and then plateaus or falls as demonstrations are added, and that the ID peak tends to line up with the accuracy plateau, so ID could be a cheap guide for choosing how many demonstrations to use.

What carries the argument

The load-bearing object is the Intrinsic Dimension Curve: collect the last-token hidden-state vectors from all validation prompts at each transformer layer, estimate the intrinsic dimension of each layer's point cloud, and plot ID against layer index. The estimate comes from the TwoNN estimator, which takes each point's ratio μ = r2/r1 of second-nearest to nearest-neighbor distance and reads the dimension off the slope relating log μ to the empirical survival probability, under the local Poisson assumption. The per-layer curve is then compressed into a Normalized AUC (trapezoidal average across layers), which lets models of different depths be compared on a single number. This curve and its AUC are the metric that carries the whole argument.

What would settle it

Run the same protocol many times with fresh random seeds and subsamples of the validation set, then form bootstrap confidence intervals on the Normalized AUC difference between ICL-5 and SFT; if the gap is within noise, the ordering claim fails. A second check is to pad fine-tuning prompts with neutral filler text to match ICL input lengths: if the higher ID follows longer prompts rather than the learning paradigm itself, the central comparison collapses.

Watch

Extended reading notes

Core claim

On its own terms, the central claim is that in-context learning induces a higher intrinsic dimension in LLM hidden representations than supervised fine-tuning does. In every (model, dataset) pair tested, ICL prompts with k ≥ 5 demonstrations yield ID curves above both the zero-shot baseline and the LoRA fine-tuned model at every layer, which the authors read as evidence that ICL representations live on higher-dimensional manifolds in embedding space. A second claim is that ID responds non-monotonically to demonstration count: it grows through small k, then plateaus or declines for larger k, and in most experiments the k with peak ID also reaches peak or near-peak accuracy. A third claim is that the ID of fine-tuned models tracks the zero-shot baseline closely even while accuracy rises, so ID is not a proxy for task performance. The paper contrasts its SFT-versus-ICL ordering with a concurrent study reporting the opposite ordering.

Load-bearing premise

The entire comparison rests on the assumption that the intrinsic-dimension estimator is unbiased for these hidden-state sets and that the ICL-versus-SFT gap is larger than the estimator's own variability, yet the paper reports no repeated runs, error bars, or confidence intervals to show that.

Editorial extensions

If this is right

  • With five or more demonstrations, ICL can be expected to show higher intrinsic dimension than SFT across models of different families and sizes, not just the four models tested here.
  • The k value at which ID peaks can serve as a practical stopping point for choosing the number of demonstrations, since accuracy has usually plateaued by that point.
  • Intrinsic dimension is not a proxy for accuracy: fine-tuned models have the lowest ID and the highest accuracy, so a low ID should not be read as poor task performance.
  • Fine-tuned representations remain geometrically close to the zero-shot baseline, suggesting that LoRA fine-tuning on 1,000 examples reshapes the representation space less than adding demonstrations does.
  • The absence of a universal hunchback ID curve in LLMs, unlike CNNs and image transformers, points to a more diverse layer-wise organization of language representations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the higher ID of ICL could partly be a prompt-length effect, since longer demonstration contexts change the distribution of last-token embeddings; a controlled comparison that pads fine-tuning prompts to the same length would show whether the paradigm or the sequence length drives the gap.
  • Editorial inference: if the geometry reflects mechanism, then parameter-efficient methods that interpolate between ICL and SFT, such as prefix tuning or attention-only adaptation, should produce ID values between the two paradigms, which is testable.
  • Editorial inference: the paper's own control experiments only rule out demonstration reuse as the cause of the ID plateau; an equally cheap check would be fixing k while shuffling or corrupting demonstrations, which should lower ID if the task signal, not the context length, is what expands the manifold.
  • Editorial inference: because the TwoNN estimator is sensitive to sampling density, the 5,000-prompt validation set size may itself set a ceiling on detectable ID differences; re-running with larger pools could push the reported gap up or down.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents an empirical study of the intrinsic dimension (ID) of LLM hidden representations under supervised fine-tuning (SFT) and in-context learning (ICL). Using the TwoNN estimator on last-token hidden states from four open-weight LLMs (7B–13B) across eight datasets, the authors report three main findings: (i) ID generally increases over the course of LoRA fine-tuning, with occasional initial decreases; (ii) ID initially rises with the number of ICL demonstrations k and then plateaus or decreases; and (iii) ICL with k≥5 consistently yields higher ID than SFT or zero-shot prompts, which they interpret as evidence that ICL representations inhabit higher-dimensional manifolds. They also propose the k value at peak ID as a practical heuristic for choosing demonstration count.

Significance. If the central comparison were robust, this would be a useful geometric characterization of two major adaptation paradigms, and the demonstration-count heuristic could save computational resources. The paper has genuine strengths: it is systematic, covering 8 datasets and 4 models; it uses an established estimator with a sanity check against MLE (Appendix F); it reports code availability; and it includes a check against demonstration-reuse artifacts (Appendix D). However, the claim that ICL induces higher ID than SFT is currently undermined by an uncontrolled input-length confound and by the absence of any uncertainty quantification, both of which are load-bearing. The significance is therefore conditional on these issues being resolved.

major comments (3)
  1. [Section 7.2 (and Section 4.1)] The central comparison confounds learning paradigm with input length. In Section 4.1, ID is measured on the last-token hidden state of each input sequence, so the estimate reflects the entire preceding context. ICL prompts with k≥5 are systematically longer than SFT prompts (which contain no demonstrations) and zero-shot prompts. The paper reports no control in which SFT or zero-shot models are evaluated on long prompts, e.g., padded with task-irrelevant text to match ICL-5 length. The k-vs-AUC results in Section 6.2, where ID rises from k=0 to k≈5 and then plateaus or decreases, are precisely the pattern one would expect from a saturating context-length effect. Without such a control, the abstract's conclusion that 'representations generated during ICL reside in higher dimensional manifolds' is not warranted by the data. This is the load-bearing point of the paper and needs an additional experiment.
  2. [All experimental figures (e.g., Figures 4 and 5)] No measure of estimator variability is reported. ID estimates are point values with no error bars, confidence intervals, or repeated runs. The consistency claim 'across all datasets and models' in Section 7.2 depends on differences between paradigms that may be within estimator noise. The only reliability check, the TwoNN-versus-MLE correlation in Appendix F (r=0.70), is moderate. The authors should provide bootstrap confidence intervals for the ID estimates or repeated independent runs, and verify that the ordering ICL-k≥5 > SFT and ICL-k≥5 > ICL-0 holds after accounting for this variability.
  3. [Section 6.2] The proposed heuristic that the peak of the k-vs-AUC curve indicates the optimal number of demonstrations is tested on the same six experiments from which it is derived, and the match is reported as 5 out of 6. This is an in-sample evaluation with a small sample; no held-out validation or out-of-sample test is provided. The heuristic therefore should be framed as a preliminary observation, not as a practical indicator, unless the authors add a validation experiment or cross-validation.
minor comments (5)
  1. [Section 2.2] The displayed formula for TwoNN suggests a closed-form per-point estimate, but the estimator actually computes the slope of a linear fit to the empirical CDF; please clarify the estimation procedure.
  2. [Section 7.2] The first sentence contains a subject-verb agreement error: 'ICL prompts with k ≥ 5 consistently induces higher intrinsic dimensions' should be 'ICL prompts with k ≥ 5 consistently induce higher intrinsic dimensions'.
  3. [Section 7.3.1 and Appendix C] Figure 36 is cited in Section 7.3.1 but is located in Appendix C; please add explicit cross-references to appendices for all cited figures.
  4. [Table 1] Table 1 reports accuracy without any variance; because demonstrations are sampled randomly and fine-tuning has stochasticity, standard errors or a note on single-run reporting should be included.
  5. [Appendix G] Appendix G appears to contain dataset prompts but the text refers to 'details of dataset creation' in Appendix G; add a section heading and describe the split generation procedure used for CommonsenseQA and MMLU (which are drawn from train/test splits rather than a validation split).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports direct measurements using an external ID estimator and does not fit parameters or rely on a self-citation chain for its central claim.

full rationale

The paper's central claim is an empirical measurement: intrinsic dimension is computed from hidden states with the TwoNN estimator of Facco et al. (2017), an external method with no parameters fitted to force the reported ICL-versus-SFT ordering. The ICL-k curves and the SFT checkpoints are generated by standard forward passes and LoRA fine-tuning, and the comparisons in Section 7.2 are descriptive statistics of those measurements. The claim that peak ID indicates a good number of demonstrations is presented as a post-hoc observation from the same data, not as a fitted parameter later relabeled as a prediction, so it does not meet the threshold for a circular step. The paper contains no load-bearing self-citations: the authors do not cite themselves, and the cited prior work (Facco, Levina and Bickel, Doimo et al.) is independent external evidence. Concerns about input-length confounding or estimator variability are threats to empirical validity, not circularity, and the Limitations section does not assert any circular dependency. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The ID estimation itself introduces no fitted parameters. The claims depend on manual hyperparameter choices for fine-tuning and on the specific k values tested; these are listed as free parameters because they are chosen by hand and could affect the comparison. No new entities are introduced.

free parameters (7)
  • LoRA rank r = 64
    Chosen manually; standard for LoRA, affects the expressiveness of fine-tuning and may influence ID.
  • LoRA alpha = 16
    Chosen manually; scales the LoRA update.
  • LoRA learning rate = 1e-4
    Chosen manually; affects fine-tuning dynamics and final representations.
  • Number of fine-tuning epochs = 15 (Section 5), 4 (Section 7)
    Four epochs chosen based on observed overfitting; the comparison may depend on this choice.
  • Training set size for SFT = 1000
    Chosen for computational constraints; SFT ID may differ with more data.
  • Validation set size for ID estimation = 5000
    Chosen to stabilize the TwoNN estimator; ID estimates may be sensitive to sample size.
  • ICL demonstration values in comparison = k in {0,1,2,5,10}
    Popular choices; the claim that k >= 5 yields higher ID than SFT is limited to this range.
assumptions (4)
  • domain assumption Manifold hypothesis: real-world data and learned representations lie on low-dimensional manifolds.
    Invoked in Section 2.2 to justify measuring intrinsic dimension as a meaningful summary of representation geometry.
  • standard math The TwoNN estimator yields unbiased intrinsic dimension estimates for point sets sampled from a manifold.
    The entire analysis rests on this estimator (Section 2.2); its consistency for neural network representations is assumed.
  • domain assumption The representation of the last input token captures task-relevant state.
    All ID measurements use the last-token hidden state (Section 2.1); this is a common but unproven simplification.
  • domain assumption The four open-source models and eight datasets are representative of LLM behavior.
    Claims are phrased as general across the tested set, but generality beyond these models and tasks is asserted rather than established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Comparative Study of Learning Paradigms in Large Language Models via Intrinsic Dimension." pith.science (2026). https://pith.science/paper/G6G37HBX

@misc{pith2026241206245,
  author       = {Pith},
  title        = {Pith review of: A Comparative Study of Learning Paradigms in Large Language Models via Intrinsic Dimension},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G6G37HBX}},
  note         = {Machine review of arXiv:2412.06245}
}
read the original abstract

The performance of Large Language Models (LLMs) on natural language tasks can be improved through both supervised fine-tuning (SFT) and in-context learning (ICL), which operate via distinct mechanisms. Supervised fine-tuning updates the model's weights by minimizing loss on training data, whereas in-context learning leverages task demonstrations embedded in the prompt, without changing the model's parameters. This study investigates the effects of these learning paradigms on the hidden representations of LLMs using Intrinsic Dimension (ID). We use ID to estimate the number of degrees of freedom between representations extracted from LLMs as they perform specific natural language tasks. We first explore how the ID of LLM representations evolves during SFT and how it varies due to the number of demonstrations in ICL. We then compare the IDs induced by SFT and ICL and find that ICL consistently induces a higher ID compared to SFT, suggesting that representations generated during ICL reside in higher dimensional manifolds in the embedding space.

Figures

Figures reproduced from arXiv: 2412.06245 by the authors.

Figure 1
Figure 1. Accuracy, intrinsic dimension, and normalized AUC for the Llama-3-8B model on the MMLU dataset. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Fine-tuning results for Meta-Llama-3-8B on [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. (ICL) results for Meta-Llama-3-8B model on MMLU dataset. (a) Accuracy increases, then plateaus as [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (39 more)
Figure 5
Figure 5. Figure 5: Boxplot displaying the distribution of normal [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Boxplot displaying the distribution of nor [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: ICL Experiment Results for Meta-Llama-3-8B on MMLU [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: ICL Experiment Results for Meta-Llama-3-8B on CommonsenseQA [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: ICL Experiment Results for Meta-Llama-3-8B on QNLI [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: ICL Experiment Results for Llama-2-13b on MMLU [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: ICL Experiment Results for Llama-2-13b on CommonsenseQA [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: ICL Experiment Results for Llama-2-13b on QNLI [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 13
Figure 13. Figure 13: Supervised Fine-Tuning Results for Llama-3-8B on Commonsense QA [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]
Figure 14
Figure 14. Figure 14: Supervised Fine-Tuning Results for Llama-3-8B on MMLU [PITH_FULL_IMAGE:figures/full_fig_p013_14.png]
Figure 15
Figure 15. Figure 15: Supervised Fine-Tuning Results for Llama-3-8B on MNLI [PITH_FULL_IMAGE:figures/full_fig_p013_15.png]
Figure 16
Figure 16. Figure 16: Supervised Fine-Tuning Results for Llama-3-8B on QNLI [PITH_FULL_IMAGE:figures/full_fig_p013_16.png]
Figure 17
Figure 17. Figure 17: Supervised Fine-Tuning Results for Llama-3-8B on QQP [PITH_FULL_IMAGE:figures/full_fig_p013_17.png]
Figure 18
Figure 18. Figure 18: Supervised Fine-Tuning Results for Llama-3-8B on SST-2 [PITH_FULL_IMAGE:figures/full_fig_p014_18.png]
Figure 19
Figure 19. Figure 19: Supervised Fine-Tuning Results for Llama-3-8B on CoLA [PITH_FULL_IMAGE:figures/full_fig_p014_19.png]
Figure 20
Figure 20. Figure 20: Supervised Fine-Tuning Results for Llama-3-8B on AG News [PITH_FULL_IMAGE:figures/full_fig_p014_20.png]
Figure 21
Figure 21. Figure 21: Supervised Fine-Tuning Results for Llama-2-13B on Commonsense QA [PITH_FULL_IMAGE:figures/full_fig_p014_21.png]
Figure 22
Figure 22. Figure 22: Supervised Fine-Tuning Results for Llama-2-13B on MMLU [PITH_FULL_IMAGE:figures/full_fig_p014_22.png]
Figure 23
Figure 23. Figure 23: Supervised Fine-Tuning Results for Llama-2-13B on MNLI [PITH_FULL_IMAGE:figures/full_fig_p015_23.png]
Figure 24
Figure 24. Figure 24: Supervised Fine-Tuning Results for Llama-2-13B on QNLI [PITH_FULL_IMAGE:figures/full_fig_p015_24.png]
Figure 25
Figure 25. Figure 25: Supervised Fine-Tuning Results for Llama-2-13B on QQP [PITH_FULL_IMAGE:figures/full_fig_p015_25.png]
Figure 26
Figure 26. Figure 26: Supervised Fine-Tuning Results for Llama-2-13B on SST-2 [PITH_FULL_IMAGE:figures/full_fig_p015_26.png]
Figure 27
Figure 27. Figure 27: Supervised Fine-Tuning Results for Llama-2-13B on CoLA [PITH_FULL_IMAGE:figures/full_fig_p015_27.png]
Figure 28
Figure 28. Figure 28: Supervised Fine-Tuning Results for Llama-2-13B on AG News [PITH_FULL_IMAGE:figures/full_fig_p016_28.png]
Figure 29
Figure 29. Figure 29: Comparison of Experimental Results for Commonsense QA [PITH_FULL_IMAGE:figures/full_fig_p017_29.png]
Figure 30
Figure 30. Figure 30: Comparison of Experimental Results for MMLU [PITH_FULL_IMAGE:figures/full_fig_p018_30.png]
Figure 31
Figure 31. Figure 31: Comparison of Experimental Results for MNLI [PITH_FULL_IMAGE:figures/full_fig_p019_31.png]
Figure 32
Figure 32. Figure 32: Comparison of Experimental Results for QNLI [PITH_FULL_IMAGE:figures/full_fig_p020_32.png]
Figure 33
Figure 33. Figure 33: Comparison of Experimental Results for QQP [PITH_FULL_IMAGE:figures/full_fig_p021_33.png]
Figure 34
Figure 34. Figure 34: Comparison of Experimental Results for SST-2 [PITH_FULL_IMAGE:figures/full_fig_p022_34.png]
Figure 35
Figure 35. Figure 35: Comparison of Experimental Results for CoLA [PITH_FULL_IMAGE:figures/full_fig_p023_35.png]
Figure 36
Figure 36. Figure 36: Comparison of Experimental Results for AG News [PITH_FULL_IMAGE:figures/full_fig_p024_36.png]
Figure 37
Figure 37. Figure 37: ICL Experiment Results with Unique Demonstrations on AGNews Dataset [PITH_FULL_IMAGE:figures/full_fig_p025_37.png]
Figure 38
Figure 38. Figure 38: ICL Experiment Results with Unique Demonstrations on QNLI Dataset [PITH_FULL_IMAGE:figures/full_fig_p025_38.png]
Figure 39
Figure 39. Figure 39: ICL Experiment Results with Unique Demonstrations on QQP Dataset [PITH_FULL_IMAGE:figures/full_fig_p025_39.png]
Figure 40
Figure 40. Figure 40: Normalized AUC by Model boxplot for ICL [PITH_FULL_IMAGE:figures/full_fig_p026_40.png]
Figure 41
Figure 41. Figure 41: Normalized AUC by Model boxplot for ICL [PITH_FULL_IMAGE:figures/full_fig_p026_41.png]
Figure 45
Figure 45. Figure 45: Normalized AUC by Model boxplot for SFT experiments [PITH_FULL_IMAGE:figures/full_fig_p026_45.png]
Figure 46
Figure 46. Figure 46: Scatterplot plotting ID estimation results for [PITH_FULL_IMAGE:figures/full_fig_p027_46.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 3 canonical work pages

  1. [1]

    Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, et al. 2024. Many-shot in-context learning. arXiv preprint arXiv:2404.11018

    arXiv 2024

  2. [2]

    Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. 2020. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255

    arXiv 2020

  3. [3]

    Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. 2019. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Systems, 32

    2019

  4. [4]

    Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, ...

    doi:10.18653/v1/2022.acl-demo.9 2022

  5. [5]

    Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    arXiv 2020

  6. [6]

    Emily Cheng, Corentin Kervadec, and Marco Baroni. 2023. Bridging information-theoretic and geometric compression in language models. arXiv preprint arXiv:2310.13620

    arXiv 2023

  7. [7]

    Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. 2022. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. arXiv preprint arXiv:2212.10559

    arXiv 2022

  8. [8]

    Diego Doimo, Alessandro Serra, Alessio Ansuini, and Alberto Cazzaniga. 2024. The representation landscape of few-shot learning and fine-tuning in large language models. arXiv preprint arXiv:2409.03662

    work page Pith review arXiv 2024

Show all 30 references
  1. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  2. [10]

    Elena Facco, Maria d’Errico, Alex Rodriguez, and Alessandro Laio. 2017. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific reports, 7(1):12140

    2017

  3. [11]

    Ian Goodfellow. 2016. Deep learning

    2016

  4. [12]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  5. [13]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  6. [14]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  7. [15]

    Jin Hwa Lee, Thomas Jiralerspong, Lei Yu, Yoshua Bengio, and Emily Cheng. 2024. Geometric signatures of compositionality across a language model's lifetime. arXiv preprint arXiv:2410.01444

    2024 arXiv

  8. [16]

    Elizaveta Levina and Peter Bickel. 2004. Maximum likelihood estimation of intrinsic dimension. Advances in neural information processing systems, 17

    2004

  9. [17]

    Utkarsh Sharma and Jared Kaplan. 2022. Scaling laws from the data manifold dimension. Journal of Machine Learning Research, 23(9):1--34

    2022

  10. [18]

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language...

    2013

  11. [19]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937

    2018 arXiv

  12. [20]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  13. [21]

    Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Alberto Cazzaniga. 2024. The geometry of hidden representations of large transformer models. Advances in Neural Information Processing Systems, 36

    2024

  14. [22]

    Alex Wang. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461

    2018 arXiv

  15. [23]

    Zhiguo Wang, Wael Hamza, and Radu Florian. 2017. Bilateral multi-perspective matching for natural language sentences. arXiv preprint arXiv:1702.03814

    2017 arXiv

  16. [24]

    Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471

    2018 arXiv

  17. [25]

    Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426

    2017 arXiv

  18. [26]

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080

    2021 arXiv

  19. [27]

    Fan Yin, Jayanth Srinivasa, and Kai-Wei Chang. 2024. Characterizing truthfulness in large language model generations with local intrinsic dimension. arXiv preprint arXiv:2402.18048

    2024 arXiv

  20. [28]

    Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28

    2015

  21. [29]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  22. [30]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.