Pith. sign in

REVIEW 3 major objections 5 minor 54 references

InnerThoughts: Disentangling Representations and Predictions in Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A small predictor network over all hidden layers improves frozen-LLM multiple-choice accuracy, approaching fine-tuning results at far lower cost.

desk verdict A cheap and well-ablated supervised head over all-layer hidden states buys real multiple-choice QA gains; the evaluation split for two of six datasets is non-standard, but the flagged validation-leak worry does not survive a careful read. read the letter →

arxiv 2501.17994 v1 pith:Z34VOHSD submitted 2025-01-29 cs.CL cs.LG

Didier Chételat , Joseph Cotnareanu , Rylee Thompson , Yingxue Zhang , Mark Coates This is my paper · ORCID
classification cs.CLcs.LG
keywords InnerThoughtshiddenstatesmultiple-choicequestionansweringLLMcalibrationfrozenlanguagemodelspredictornetworkparameter-efficientadaptationLlama370B
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a frozen large language model leaves useful answer information in the hidden states of every layer, and that a small trained network can extract it. The proposed method, InnerThoughts, feeds all layers' hidden states at the final token position into a lightweight predictor, replaces the model's own softmax readout, and improves multiple-choice accuracy on several hard benchmarks. The authors report gains such as 47.24% versus 36.47% on AQuA and 87.24% versus 82.20% on HellaSwag, while requiring only one forward pass through the LLM during training. If the claim holds, per-task adaptation of large frozen models becomes much cheaper than parameter-efficient fine-tuning.

What carries the argument

The central object is the predictor module, an MLPMixer-style network that takes an $(L,d)$ tensor of hidden states from the last token position, applies normalization, linear, and activation blocks to reduce both dimensions, flattens the result, and ends in a softmax over answer choices. Because it sits after the transformer blocks, it can be trained with the LLM frozen and with no backward passes through the LLM. The module is strictly more expressive than the original readout, since specific parameter choices recover the original normalization and embedding projection.

What would settle it

Run the trained InnerThoughts predictor on the official held-out test partitions of AQuA, MedMCQA, CosmosQA, and HellaSwag without retraining on those splits; if the accuracy advantage over direct prompting shrinks to near zero there, the central claim would fail in the conditions the benchmarks are meant to measure.

Watch

Extended reading notes

Core claim

The central claim is that a small predictor network trained on all layers' last-token hidden states can outperform both direct prompting and existing logit-calibration methods on multiple-choice QA, and in some cases approach supervised fine-tuning. The paper reports statistically significant gains over direct prompting on AQuA, CosmosQA, MedMCQA, LogiQA, and HellaSwag, with the largest improvements on datasets where the base LLM's confidence margins are smallest. Training the predictor requires no backward pass through the frozen LLM, so the dominant cost is a single forward pass to extract hidden states. The authors interpret this as disentangling the LLM's representational abilities from its predictive abilities.

Load-bearing premise

The reported gains assume that random 70/15/15 and 80/20 splits of each training set are representative of the official benchmark test sets, so that the measured improvements would also appear in a standard public evaluation.

Editorial extensions

If this is right

  • On AQuA, InnerThoughts reaches 47.24% accuracy versus 36.47% for direct prompting, nearly matching QLoRA's 48.50% at about one-fifth of the training cost.
  • Across the six benchmarks it captures, on average, roughly 35% of QLoRA's accuracy gain at about 16% of the compute cost.
  • The largest improvements occur on benchmarks where the base LLM's confidence margin between top and second answers is smallest, and applying InnerThoughts also increases those margins.
  • Because the predictor input is the same set of last-token hidden states, one extraction pass can support multiple predictor heads or downstream tasks without re-running the LLM.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline numbers come from custom random splits of training sets, not the official test sets; evaluating on official held-out splits would show how much of the gain transfers to real benchmark conditions.
  • InnerThoughts may be learning dataset-specific answer or option-order biases rather than purely better reasoning; shuffling answer order at test time would separate bias correction from genuine task improvement.
  • The method could plausibly stack with fine-tuning: training the same predictor on a QLoRA-adapted model, or jointly, may produce gains beyond either alone, though the paper only raises this as a possibility.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes InnerThoughts, a supervised adapter that predicts multiple-choice answers from the hidden states of all transformer layers at the last token position of a prompt, while keeping the language model frozen. The predictor is a small MLP-Mixer-like network, so training requires only a single forward pass through the LLM. Experiments on six MCQ benchmarks with Llama3 70B compare InnerThoughts with direct prompting, calibration methods, logistic-regression and neural variants, and QLoRA fine-tuning; the authors report substantial accuracy gains (e.g., AQuA 36.47 to 47.24, HellaSwag 82.20 to 87.24) at a fraction of QLoRA's training cost, and they analyze confidence margins and per-layer contributions.

Significance. If the reported gains hold under a clean held-out protocol, the idea is a simple and useful addition to the MCQ toolkit: it converts a frozen LLM into a better MCQ predictor using one forward pass per training example and a small network, with low hardware requirements. The paper's controlled comparisons, bootstrapped confidence intervals, and paired significance tests are strengths, and the cost/gain comparison against QLoRA is informative. However, the evaluation protocol has an unresolved ambiguity for two of the six benchmarks, and the QLoRA baseline is quite weak; both issues need to be addressed before the central claims can be accepted.

major comments (3)
  1. [Section 4.1, Table 2, Table 3] Section 4.1 states that for CosmosQA and HellaSwag "we use a 80%-20% train-validation split of the training set" and does not describe a test split, yet Table 2 lists test sizes (2,985 for CosmosQA and 10,042 for HellaSwag) and Table 3 reports test accuracies for these datasets. Since Section 4.1 also says predictor networks are trained with early stopping using the validation set, if the Table 3 numbers were computed on the same 20% split used for early stopping, the reported HellaSwag improvement (+5.04) and CosmosQA improvement (+1.84) would not be unbiased held-out estimates. Please clarify the exact evaluation split, and if the reported accuracies were computed on the early-stopping validation split, re-run the evaluation on an untouched test split.
  2. [Section 4.1, Table 4] The QLoRA baseline is trained with adapter rank r=1 for three epochs and is described as an upper bound, but Table 4 uses its accuracy to compute gain ratios and the claim that InnerThoughts obtains "roughly 35% of the gains of QLoRA at 16% of the computational costs." A rank-1, three-epoch QLoRA is unlikely to represent the full fine-tuning upper bound, so the efficiency comparison and the statement that InnerThoughts is "approaching parameter-efficient fine-tuning" need either a stronger baseline or explicit caveats that the comparison is against a deliberately weak fine-tuning configuration.
  3. [Section 4.1, Table 2] The paper replaces official test sets with custom random splits for AQuA, MedMCQA, CosmosQA, and HellaSwag, and it does not report accuracy on the official test sets. Because the abstract and introduction claim gains on "hard benchmarks" without qualification, and because random splits can differ in difficulty from the public test sets, the authors should either use the official test sets or provide evidence (for example, by comparing the Direct baseline on both the custom split and the official set) that the custom splits are representative of the intended benchmarks.
minor comments (5)
  1. [Figure 3] The legend labels such as "AQuA (Δ=+0.11)" are described as gains in accuracy, but they do not match the accuracy gains in Table 4 (+10.75 for AQuA, +5.04 for HellaSwag, etc.); please correct the labels or clarify that they are logit-margin means.
  2. [Section 3] The claim that the proposed architecture is strictly more expressive than the original LLM predictor assumes that the normalization and activation blocks can be bypassed or set to identity, but the architecture as drawn includes them as fixed components; please clarify whether these components can be removed or configured as identity mappings.
  3. [Section 4.2] The phrase "neural network on the last 8192 hidden state" should be "neural network on the last hidden state."
  4. [Section 4.3] The description of the significance test is incomplete: please state explicitly that the one-sided Wilcoxon signed-rank test is applied to paired per-example correctness indicators, and indicate whether any multiple-comparison correction was applied across methods and datasets.
  5. [Reproducibility] No code or data-processing scripts are provided; releasing the feature-extraction and split-generation code would substantially improve reproducibility, especially because the evaluation uses custom splits.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: the core result is an empirical comparison of a supervised probe on held-out splits, not a quantity defined by its own inputs.

full rationale

The claimed derivation chain is an empirical supervised-learning pipeline: hidden states from all layers at the last token position are extracted with a frozen LLM, a small predictor is trained on labeled training questions, and accuracy is reported on separate test or validation splits in Table 3. No equation in Sections 3-4 defines the reported accuracy in terms of the training labels or fitted parameters; the fitted predictor is evaluated on data not used for its parameter updates. The expressiveness argument in Section 3, showing that the predictor architecture could in principle recover the original LLM predictor with specific hyperparameter choices, is a construction rather than a circular reduction. The layer-influence analyses in Section 5 and Appendix D are post-hoc and do not feed back into the training objective. There are no load-bearing self-citations; the paper's comparisons are to external baselines (Abbas et al., Zhao et al., QLoRA). The only concern is a protocol ambiguity for CosmosQA and HellaSwag: Section 4.1 describes only an 80%-20% train-validation split while Table 2 also lists test sets, and early stopping uses the validation set. If the Table 3 numbers for those datasets were computed on the early-stopping validation set, the estimate would be optimistic; however, the paper's text is not explicit enough to exhibit that reduction, and this would be an evaluation-validity issue rather than a circular derivation. Under the required evidentiary standard, the central claim is not equivalent to its inputs by construction.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on supervised training of a predictor on hidden states. It does not introduce new physical entities or require heavy mathematical axioms. The main free choices are architecture hyperparameters, training schedule, split proportions, and the comparison settings for QLoRA.

free parameters (7)
  • Predictor width n1 = 32
    Chosen by a small-scale hyperparameter search; affects predictor capacity and results.
  • Predictor width n2 = 8
    Chosen by a small-scale hyperparameter search; affects the flattened feature dimension.
  • Predictor learning rate = 1e-5
    Chosen by hand; used for all InnerThoughts training runs.
  • Predictor training epochs = 50 with early stopping
    Chosen by hand; early stopping uses the validation split.
  • QLoRA adapter rank = 1
    Chosen for the fine-tuning baseline; a low rank may understate fine-tuning performance.
  • QLoRA training epochs = 3
    Chosen for computational feasibility; the paper notes more epochs were unreasonably time-consuming.
  • Random split proportions = 70/15/15 or 80/20 of training set
    Chosen by hand to replace official test sets with larger custom splits.
assumptions (4)
  • domain assumption Hidden states at the last token position from all layers are meaningful inputs for a separate classifier.
    Central premise of the probing approach; Section 3 uses h^l_K for all layers.
  • domain assumption Custom random splits of training data are representative of the intended benchmarks.
    Section 4.1 replaces official test sets with random splits; if these are easier or distributionally different, the reported gains may not transfer.
  • domain assumption Supervised training on a large labeled set is a fair comparison against a zero-shot direct baseline.
    The method uses labels while the direct baseline does not; this asymmetry is acknowledged but is central to interpreting the gains.
  • domain assumption QLoRA with rank 1 and three epochs is a representative parameter-efficient fine-tuning upper bound.
    Table 4 treats QLoRA as an upper bound; a stronger or tuned fine-tuning baseline might reduce the apparent advantage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of InnerThoughts: Disentangling Representations and Predictions in Large Language Models." pith.science (2026). https://pith.science/paper/Z34VOHSD

@misc{pith2026250117994,
  author       = {Pith},
  title        = {Pith review of: InnerThoughts: Disentangling Representations and Predictions in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z34VOHSD}},
  note         = {Machine review of arXiv:2501.17994}
}
read the original abstract

Large language models (LLMs) contain substantial factual knowledge which is commonly elicited by multiple-choice question-answering prompts. Internally, such models process the prompt through multiple transformer layers, building varying representations of the problem within its hidden states. Ultimately, however, only the hidden state corresponding to the final layer and token position are used to predict the answer label. In this work, we propose instead to learn a small separate neural network predictor module on a collection of training questions, that take the hidden states from all the layers at the last temporal position as input and outputs predictions. In effect, such a framework disentangles the representational abilities of LLMs from their predictive abilities. On a collection of hard benchmarks, our method achieves considerable improvements in performance, sometimes comparable to supervised fine-tuning procedures, but at a fraction of the computational cost.

Figures

Figures reproduced from arXiv: 2501.17994 by the authors.

Figure 1
Figure 1. A comparison of the proposed method with how LLMs currently answer multiple-choice questions. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. A diagram of our suggested predictor module architecture. The network takes inputs of the shape [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Histograms of logit-transformed confidence [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Histograms of the logit-transformed confidence margins, logit( [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Final accuracy when using the hidden-state [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Influence score P8192 k=1 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 44 canonical work pages

  1. [1]

    Abbas, M., Zhou, Y., Ram, P., Baracaldo, N., Samulowitz, H., Salonidis, T., and Chen, T. (2024). Enhancing in-context learning via linear probe calibration. In Proceedings of the Twenty-Seventh International Conference on Artificial Intelligence and Statistics , volume 238 of PMLR , pages 307--315

    work page 2024

  2. [3]

    and Mitchell, T

    Azaria, A. and Mitchell, T. (2023). The internal state of an LLM knows when it's lying. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 967--976

    work page 2023

  3. [4]

    Belrose, N., Furman, Z., Smith, L., Halawi, D., Ostrovsky, I., McKinney, L., Biderman, S., and Steinhardt, J. (2023). Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112

    arXiv 2023

  4. [5]

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A.,...

    2020

  5. [6]

    Carlini, N., Tram \`e r, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, \'U ., Oprea, A., and Raffel, C. (2021). Extracting training data from large language models. In Proceedings of the Thirtieth USENIX Security Symposium , pages 2633--2650

    work page 2021

  6. [7]

    Chen, N., Wu, N., Liang, S., Gong, M., Shou, L., Zhang, D., and Li, J. (2023). Beyond surface: Probing LLaMA across scales and layers. arXiv preprint arXiv:2312.04333

    arXiv 2023

  7. [8]

    R., and He, P

    Chuang, Y.-S., Xie, Y., Luo, H., Kim, Y., Glass, J. R., and He, P. (2024). DoLa : Decoding by contrasting layers improves factuality in large language models. In Proceedings of the Twelfth International Conference on Learning Representations

    work page 2024

  8. [9]

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

Show all 54 references
  1. [10]

    Deng, M., Wang, J., Hsieh, C.-P., Wang, Y., Guo, H., Shu, T., Song, M., Xing, E., and Hu, Z. (2022). RLP rompt: Optimizing discrete text prompts with reinforcement learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 3369--3391

    2022

  2. [11]

    Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). QLoRA : Efficient finetuning of quantized LLMs . In Advances in Neural Information Processing Systems , volume 36, pages 10088--10115

    2023

  3. [12]

    Y., Karidi, T., Choshen, L., and Geva, M

    Din, A. Y., Karidi, T., Choshen, L., and Geva, M. (2024). Jump to conclusions: Short-cutting transformers with linear transformations. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation , pages 9615--9625

    2024

  4. [13]

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. (2024). The Llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  5. [14]

    Fei, Y., Hou, Y., Chen, Z., and Bosselut, A. (2023). Mitigating label biases for in-context learning. In Proceedings of the Sixty-First Annual Meeting of the Association for Computational Linguistics (Volume One: Long Papers) , pages 14014--14031

    2023

  6. [15]

    Geiger, A., Lu, H., Icard, T., and Potts, C. (2021). Causal abstractions of neural networks. In Advances in Neural Information Processing Systems , volume 34, pages 9574--9586

    2021

  7. [16]

    Han, Z., Hao, Y., Dong, L., Sun, Y., and Wei, F. (2023). Prototypical calibration for few-shot learning of language models. In Proceedings of the Eleventh International Conference on Learning Representations

    2023

  8. [17]

    He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., and Neubig, G. (2022). Towards a unified view of parameter-efficient transfer learning. In Proceedings of the Tenth International Conference on Learning Representations

    2022

  9. [18]

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021). Measuring massive multitask language understanding. In Proceedings of the Ninth International Conference on Learning Representations

    2021

  10. [19]

    Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019). Parameter-efficient transfer learning for NLP . In Proceedings of the Thirty-Sixth International Conference on Machine Learning , volume 97 of PMLR , ...

    2019

  11. [20]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022). Lo RA : Low-rank adaptation of large language models. In Proceedings of the Tenth International Conference on Learning Representations

    2022

  12. [21]

    Huang, L., Le Bras, R., Bhagavatula, C., and Choi, Y. (2019). Cosmos QA : Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the Ninth International Joint Conference...

    2019

  13. [22]

    J., Madotto, A., and Fung, P

    Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., and Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys , 55:1--38

    2023

  14. [23]

    Jiang, Z., Araki, J., Ding, H., and Neubig, G. (2021). How can we know when language models know? On the calibration of language models for question answering. Transactions of the Association for Computational Linguistics , 9:962--977

    2021

  15. [24]

    Karimi Mahabadi, R., Henderson, J., and Ruder, S. (2021). Compacter: Efficient low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems , volume 34, pages 1022--1035

    2021

  16. [25]

    Ko, M., Lee, J., Kim, H., Kim, G., and Kang, J. (2020). Look at the first sentence: Position bias in question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages 1109--1121

    2020

  17. [26]

    Lester, B., Al-Rfou, R., and Constant, N. (2021). The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 3045--3059

    2021

  18. [27]

    Li, X. L. and Liang, P. (2021). Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the Fifty-Ninth Annual Meeting of the Association for Computational Linguistics and the Eleventh International Joint Conference on Natural Language Processing (Volume...

    2021

  19. [28]

    Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. (2017). Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the Fifty-Fifth Annual Meeting of the Association for Computational Linguistics (Volume One: Long Paper...

    2017

  20. [29]

    Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. A. (2022). Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems , volume 35, pages 1950--1965

    2022

  21. [30]

    Liu, J., Cui, L., Liu, H., Huang, D., Wang, Y., and Zhang, Y. (2020). LogiQA : A challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence , pages 3622--3628

    2020

  22. [31]

    Mao, A., Raman, N., Shu, M., Li, E., Yang, F., and Boyd-Graber, J. (2021). Eliciting bias in question answering models through ambiguity. In Proceedings of the Third Workshop on Machine Reading for Question Answering , pages 92--99

    2021

  23. [32]

    Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., and Gao, J. (2024). Large language models: A survey. arXiv preprint arXiv:2402.06196

    2024 arXiv

  24. [33]

    Interpreting GPT : The logit lens

    nostalgebraist (2020). Interpreting GPT : The logit lens. Less Wrong blog

    2020

  25. [34]

    K., and Sankarasubbu, M

    Pal, A., Umapathi, L. K., and Sankarasubbu, M. (2022). MedMCQA : A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning , volume 174 of PMLR , pages 248--260

    2022

  26. [35]

    Petroni, F., Rockt \"a schel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., and Miller, A. (2019). Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the Ninth International Joint Conference on Na...

    2019

  27. [36]

    and Hruschka, E

    Pezeshkpour, P. and Hruschka, E. (2024). Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024 , pages 2006--2017

    2024

  28. [37]

    Pfeiffer, J., R \"u ckl \'e , A., Poth, C., Kamath, A., Vuli \'c , I., Ruder, S., Cho, K., and Gurevych, I. (2020a). AdapterHub : A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrat...

    2020

  29. [38]

    Pfeiffer, J., Vuli \'c , I., Gurevych, I., and Ruder, S. (2020b). MAD-X : An adapter-based framework for multi-task cross-lingual transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages 7654--7673

    2020

  30. [39]

    and Eisner, J

    Qin, G. and Eisner, J. (2021). Learning how to ask: Querying LM s with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 5203--5212

    2021

  31. [40]

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog

    2019

  32. [41]

    and Wingate, D

    Robinson, J. and Wingate, D. (2023). Leveraging large language models for multiple choice question answering. In Proceedings of the Eleventh International Conference on Learning Representations

    2023

  33. [42]

    Schuster, T., Fisch, A., Gupta, J., Dehghani, M., Bahri, D., Tran, V., Tay, Y., and Metzler, D. (2022). Confident adaptive language modeling. In Advances in Neural Information Processing Systems , volume 35, pages 17456--17472

    2022

  34. [43]

    Talmor, A., Elazar, Y., Goldberg, Y., and Berant, J. (2020). o LM pics -- On what language model pre-training captures. Transactions of the Association for Computational Linguistics , 8:743--758

    2020

  35. [44]

    S., Mueller, A., Wallace, B

    Todd, E., Li, M., Sharma, A. S., Mueller, A., Wallace, B. C., and Bau, D. (2024). Function vectors in large language models. In Proceedings of the Twelfth International Conference on Learning Representations

    2024

  36. [45]

    O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., Lucic, M., and Dosovitskiy, A

    Tolstikhin, I. O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., Lucic, M., and Dosovitskiy, A. (2021). MLP-Mixer : An all- MLP architecture for vision. In Advances in Neural Information Processing Systems...

    2021

  37. [46]

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  38. [47]

    V., and Zhou, D

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., ichter, b., Xia, F., Chi, E., Le, Q. V., and Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems , volume 35, pages 24824--24837

    2022

  39. [48]

    D., and Potts, C

    Wu, Z., Arora, A., Wang, Z., Geiger, A., Jurafsky, D., Manning, C. D., and Potts, C. (2024). Re FT : Representation finetuning for language models. In Advances in Neural Information Processing Systems , volume 38

    2024

  40. [49]

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. (2019). H ella S wag: Can a machine really finish your sentence? In Proceedings of the Fifty-Seventh Annual Meeting of the Association for Computational Linguistics , pages 4791--4800

    2019

  41. [50]

    Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. (2021). Calibrate before use: Improving few-shot performance of language models. In Proceedings of the Thirty-Eighth International Conference on Machine Learning , volume 139 of PMLR , pages 12697--12706

    2021

  42. [51]

    Zheng, C., Zhou, H., Meng, F., Zhou, J., and Huang, M. (2024). Large language models are not robust multiple choice selectors. In Proceedings of the Twelfth International Conference on Learning Representations

    2024

  43. [52]

    A., and Roy, S

    Zhou, H., Wan, X., Proleev, L., Mincu, D., Chen, J., Heller, K. A., and Roy, S. (2024). Batch calibration: Rethinking calibration for in-context learning and prompt engineering. In Proceedings of the Twelfth International Conference on Learning Representations

    2024

  44. [53]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  45. [54]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  46. [55]

    Mitigating Label Biases for In-context Learning

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2021 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.