Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

ICL CIPHERS: Quantifying "Learning" in In-Context Learning via Substitution Ciphers

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLMs perform better on in-context tasks scrambled with a reversible word cipher than with random word noise, and this accuracy gap is proposed as a practical measure of task learning.

desk verdict The bijective-vs-non-bijective cipher gap is a clever but currently confounded measure of ICL task learning; it needs a fixed non-injective control before the 'quantify learning' claim holds. read the letter →

arxiv 2504.19395 v2 pith:JXWKEJN3 submitted 2025-04-28 cs.CL

Zhouxiang Fang , Aayush Mishra , Muhan Gao , Anqi Liu , Daniel Khashabi This is my paper · ORCID
classification cs.CL
keywords in-contextlearningtaskretrievalsubstitutioncipherlargelanguagemodelsfew-shotpromptreformulationlogitlensprobing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to separate two things that are tangled inside in-context learning: the model retrieving a task it recognizes from pretraining, and the model actually learning a new input-label pattern from the demonstrations on the fly. To isolate the learning component, the authors route standard benchmarks through token-level substitution ciphers, swapping a fraction of the words in each prompt for unrelated words according to a fixed one-to-one key. They compare this against a control where each substituted word is replaced by a randomly sampled word, so no consistent pattern exists. The result is a small but consistent accuracy advantage for the reversible cipher across four datasets and six models, which the paper offers as a practical measure of task learning. A probing analysis adds internal evidence that the model increasingly tracks the ciphered tokens as it sees more demonstrations.

What carries the argument

The load-bearing object is a token-level substitution cipher $c: \mathcal{V} \to \mathcal{V}$ applied to the input side of in-context demonstrations, with a shuffle rate $r$ that fixes how many vocabulary tokens are remapped. The argument rests on the contrast between two mappings over the same token set: a bijective cipher, where every occurrence of a token maps consistently to one paired token and the map is in principle invertible, and a non-bijective cipher, where each occurrence is replaced by an independently sampled token so no consistent mapping exists. Two design choices support the comparison: Zipfian shuffling, which only swaps tokens of similar corpus frequency to reduce frequency confounds, and priority demonstration sampling, which ensures the demonstrations expose the model to the specific substitutions that appear in the test input. The accuracy gap between the two conditions is the paper's proposed measure of task learning.

What would settle it

Run a third control in which each token is always replaced by the same random token but the mapping is many-to-one (non-injective): if accuracy on this condition matches the bijective cipher, the gap is explained by consistency alone and would not measure learnability; if it matches the non-bijective condition, the paper's interpretation is supported.

Watch

Extended reading notes

Core claim

The central claim is that LLMs solve in-context tasks whose inputs have been scrambled by a bijective substitution cipher better than they solve the same tasks scrambled by an irreversible random substitution, and that this accuracy gap quantifies task learning. In the bijective condition every occurrence of a chosen token is replaced by the same paired token, so in principle the mapping can be deciphered from enough demonstrations; in the non-bijective condition each occurrence is replaced independently, so there is no pattern to learn. Because the two conditions share the same token set, the same prompts, and the same pipeline, the paper attributes the gap to the model's inference-time ability to learn and apply the latent mapping. The gap is small, often a few accuracy points, but it is consistent across SST-2, Amazon, HellaSwag, and WinoGrande with six models, and it grows with the number of demonstrations before plateauing. Logit Lens probing—decoding internal representations into token probabilities—shows that under the bijective cipher the model's representation increasingly favors the substituted token over the original one as occurrences accumulate, while no such preference develops under random substitution.

Load-bearing premise

The bijective and non-bijective conditions are assumed to differ only in whether the token mapping is consistent and reversible, so the accuracy gap is interpreted as task learning rather than as a difference in how hard the two kinds of scrambled text are for the language model to process.

Editorial extensions

If this is right

  • The bijective-versus-random accuracy gap gives a task-agnostic diagnostic for when an LLM is learning from demonstrations rather than recognizing a memorized task.
  • The gap's growth with demonstration count provides the expected scaling signature of a learning signal, so the measure can be used to compare models or training stages.
  • Since the effect persists in an instruction-tuned model and in a 70B model, the paper's evidence suggests task learning is a general property of current LLMs, not a quirk of one checkpoint.
  • Ciphering inputs rather than labels sidesteps the need to search pretraining corpora for contamination, making it easier to construct 'novel' task variants from existing benchmarks.
  • The internal probing results imply that successful models need not fully decode the cipher; capturing task-relevant attributes of ciphered tokens can suffice, which bears on how ICL mechanisms should be modeled.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A control condition with a fixed many-to-one substitution (consistent but non-reversible) would test whether the gap is driven by mapping consistency rather than reversibility; if that control matches the bijective condition, the metric would need reinterpretation.
  • The same cipher-gap protocol could be applied to non-classification tasks such as arithmetic or structured prediction, where the task-learning component is thought to be larger; whether the gap scales with task complexity is a testable extension.
  • If the gap truly measures task learning, it should predict transfer: a model showing a large gap on one cipher should adapt faster to an independently generated second cipher, a prediction the paper does not test.
  • Restricting shuffling to tokens of a single part of speech (as the paper does for nouns) could be extended to control word length, concreteness, or co-occurrence statistics, which would clarify which properties the model exploits when deciphering.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces ICL CIPHERS, a token-level substitution cipher applied to in-context learning inputs, with two conditions: a BIJECTIVE cipher using a fixed one-to-one token mapping, and a NON-BIJECTIVE baseline in which each occurrence of a ciphered token is replaced by an independently sampled token. The authors report that LLMs achieve higher accuracy on BIJECTIVE-ciphered prompts than on NON-BIJECTIVE-ciphered prompts across six models and four datasets, and they interpret this accuracy gap as a quantitative measure of task learning (TL) rather than task retrieval (TR). They also analyze the effect of shuffle rate, number of demonstrations, alignment, model size, and grammatical roles, and they present a Logit Lens probing analysis suggesting that representations shift toward substituted tokens under the BIJECTIVE condition. The central claim is that the BIJECTIVE-minus-NON-BIJECTIVE gap isolates and quantifies inference-time task learning.

Significance. The paper targets a central open question in ICL research and proposes a novel, general reformulation framework that is easy to apply to arbitrary tasks. The empirical scope is broad (six models, four datasets, multiple shot counts) and the authors release code and report paired significance tests, which is a strength. If the gap truly isolated task learning, the method would be an important tool for probing ICL. However, the core interpretive claim rests on the assumption that the two cipher conditions differ only in reversibility, and that assumption is not satisfied by the current design, so the empirical gap may at least partly reflect low-level text processing difficulty rather than task learning. The probing analysis is suggestive but, as the authors acknowledge, weak. The paper would be substantially stronger with a control condition that keeps per-occurrence consistency while removing injectivity, plus a more careful treatment of null results and rate-selection choices.

major comments (3)
  1. [§2.4 and §2.5] The central measurement assumption is confounded. Section 2.4 defines the NON-BIJECTIVE cipher so that each occurrence of a token is replaced by a uniformly random token from S, whereas the BIJECTIVE cipher uses a fixed mapping so that repeated occurrences are replaced consistently. Section 2.5 then claims that "the only difference between the two is their token mapping functions." That claim is false: the two conditions also differ in per-occurrence consistency, repetition statistics, and overall n-gram coherence. The NON-BIJECTIVE ciphered text is therefore more chaotic and harder for an LM to process for reasons unrelated to whether the mapping is learnable. The reported accuracy gap in Tables 1 and 2 could be driven by this processing difficulty rather than by task learning. This is load-bearing for the paper's central claim. I request a control condition that preserves per-token consistency but removes reversibility (e.g., a fixed many-to-one substitution), and ideally an unciphered baseline, to support the interpretation that the gap measures learning.
  2. [Table 1 and §4.1] The claim of a consistent gap across models and datasets is overstated. Of the sixteen comparisons in Table 1, three are not positive improvements: OLMo on WinoGrande (-0.3), Gemma2 on HellaSwag (-0.5), and Gemma2 on WinoGrande (+0.0). In addition, Qwen2.5 on WinoGrande (+1.2) is not statistically significant (p=0.084 in Appendix Table 7). The text acknowledges only Olmo on WinoGrande and Gemma2 on HellaSwag as exceptions, and does not discuss the null WinoGrande results. Furthermore, the shuffle rate is set to a different value for each dataset (SST-2 r=0.5, Amazon r=0.6, HellaSwag r=0.3, WinoGrande r=0.1) without a stated selection rule. This introduces researcher degrees of freedom and makes the cross-dataset consistency claim harder to evaluate. Please report results for a shared set of shuffle rates (or a principled rule) and discuss the negative/null cases explicitly.
  3. [§3.1, §4.2, and Appendix D] The shuffle-rate analysis is reported mainly for Llama-3.1-8B on Amazon (and SST-2 in the appendix), and the choice of the interval where the gap is largest is used to justify the per-dataset rates. Since the central evidence is the gap, the reader needs to know whether the pattern holds across models and datasets at a common set of rates, or whether the chosen rates were selected because they show the effect. Additionally, the priority-sampling comparison in Appendix D covers only two datasets and is summarized qualitatively as "similar trends but more unstable"; quantitative results for all datasets would help rule out that the gap is an artifact of the sampling procedure interacting with the NON-BIJECTIVE condition.
minor comments (5)
  1. [Abstract and §1] Typo: "disentangling these the two modes" should read "disentangling these two modes."
  2. [§2.4] Typo: "multiple occurences of a token are conisistently replaced" should read "consistently replaced."
  3. [Appendix D and G] Typos: "Peformance" appears in the captions of Figures 7 and 8; "regared" appears in the captions of Tables 7–10.
  4. [Tables 1–3] The tables would be easier to interpret with confidence intervals or standard deviations for the reported accuracies, since each value is an average over three runs. The McNemar tests are reported without correction for the multiple comparisons across models and datasets.
  5. [§4.7 and Appendix H] The probing analysis is described as evidence of deciphering, but the reported rank-difference trends are on a small set of 30 token pairs and the effect sizes are not quantified beyond the figures. A small clarification of how many occurrences and layers are averaged in Figure 5 would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the BIJECTIVE-vs-NON-BIJECTIVE gap is a direct empirical comparison on external benchmarks, and the 'task learning' interpretation is an explicit operationalization rather than a quantity derived from itself.

full rationale

The paper's central measurement is a direct empirical contrast: accuracy on BIJECTIVE versus NON-BIJECTIVE ciphered prompts, computed on held-out test sets of four external benchmarks and six pretrained models. No parameter is fitted to a subset of these results and then fed back into the claim; the reported gap is the raw performance difference, not the output of a model fitted from the same data. Section 2.5 states that 'the gap between the performance on BIJECTIVE and NON-BIJECTIVE ciphered text can be a practical measure of TL'; this is an operational definition, not a derived theorem, so observing a positive gap and calling it evidence of task learning is an interpretive application of that definition rather than a circular reduction. The paper itself concedes in Section 6 that 'Achieving a perfect distinction between "learning" and "retrieval" may be unattainable,' which further shows the claim is a proposed measurement with acknowledged limitations, not a self-referential proof. The strongest skeptical concern, namely that NON-BIJECTIVE differs from BIJECTIVE not only in reversibility but also in per-occurrence stochasticity and resulting text statistics, is a genuine construct-validity threat that deserves additional control conditions (e.g., a fixed many-to-one mapping), but it is not circular: the empirical gap would remain a real observation even if the label 'task learning' were removed. Self-citations (e.g., Shen et al. 2024; Mallen et al. 2023) are used for background or supporting frequency effects and are not load-bearing for the cipher-gap conclusion. Overall, no step of the derivation is equivalent to its own inputs by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The measurement depends on hand-set protocol choices (shuffle rate, frequency bins, priority sampling) and on the domain assumption that the NON-BIJECTIVE baseline controls everything except learnability. These assumptions are plausible but not independently validated; no new entity is postulated.

free parameters (3)
  • shuffle rate r per dataset = SST-2: 0.5; Amazon: 0.6; HellaSwag: 0.3; WinoGrande: 0.1
    Set per dataset as 'a reasonable value' (Sec 3.1); Fig.2 shows the gap is largest around r in (0.4,0.6), so the chosen rates appear to sit in the favorable range.
  • Zipfian frequency bin count k = 10
    Number of frequency groups for restricting substitutable token pairs (Sec 3.1); this is a hand-set design choice that shapes which mappings are possible.
  • priority sampling on/off = True (default); False in Appendix D
    Demos are selected to contain the test input's substituted tokens (Sec 3.1); Appendix D shows non-priority sampling gives more unstable results, so this choice is load-bearing for the reported gap.
assumptions (4)
  • domain assumption The BIJECTIVE and NON-BIJECTIVE conditions differ only in mapping consistency, so the accuracy gap isolates task learning (Sec 2.5).
    The paper states 'The only difference between the two is their token mapping functions', but the NON-BIJECTIVE condition also produces more chaotic, lower-repetition text that may affect LM processing difficulty independently of learnability.
  • domain assumption Ciphered inputs are unlikely to have appeared in pretraining, so task retrieval is largely suppressed (Sec 1, Sec 2.3).
    The paper relies on this to argue that BIJECTIVE gains are due to learning rather than memorization; no contamination check (e.g., fuzzy matching against pretraining corpora) is performed.
  • domain assumption Zipfian shuffling based on Wikipedia token frequencies approximates pretraining token frequencies, controlling frequency confounds (Sec 3.1).
    Token frequency is computed from Wikipedia, not from the actual pretraining corpora of Llama, Qwen, OLMo, and Gemma, so the frequency-matching assumption is approximate.
  • domain assumption Logit Lens ranks at position p-1 reflect the model's token-preference changes and can serve as evidence of cipher decoding (Sec 4.7).
    The paper uses Logit Lens to compare ranks of original versus substituted tokens and interprets the trend as evidence of internal decoding; this relies on the interpretability assumption that the LM head at intermediate layers reflects token preferences, which the authors themselves note is the only method that worked.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ICL CIPHERS: Quantifying "Learning" in In-Context Learning via Substitution Ciphers." pith.science (2026). https://pith.science/paper/JXWKEJN3

@misc{pith2026250419395,
  author       = {Pith},
  title        = {Pith review of: ICL CIPHERS: Quantifying "Learning" in In-Context Learning via Substitution Ciphers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JXWKEJN3}},
  note         = {Machine review of arXiv:2504.19395}
}
read the original abstract

Recent works have suggested that In-Context Learning (ICL) operates in dual modes, i.e. task retrieval (remember learned patterns from pre-training) and task learning (inference-time ''learning'' from demonstrations). However, disentangling these the two modes remains a challenging goal. We introduce ICL CIPHERS, a class of task reformulations based on substitution ciphers borrowed from classic cryptography. In this approach, a subset of tokens in the in-context inputs are substituted with other (irrelevant) tokens, rendering English sentences less comprehensible to human eye. However, by design, there is a latent, fixed pattern to this substitution, making it reversible. This bijective (reversible) cipher ensures that the task remains a well-defined task in some abstract sense, despite the transformations. It is a curious question if LLMs can solve tasks reformulated by ICL CIPHERS with a BIJECTIVE mapping, which requires ''deciphering'' the latent cipher. We show that LLMs are better at solving tasks reformulated by ICL CIPHERS with BIJECTIVE mappings than the NON-BIJECTIVE (irreversible) baseline, providing a novel approach to quantify ''learning'' in ICL. While this gap is small, it is consistent across the board on four datasets and six models. Finally, we examine LLMs' internal representations and identify evidence in their ability to decode the ciphered inputs.

Figures

Figures reproduced from arXiv: 2504.19395 by the authors.

Figure 1
Figure 1. An example of ICL CIPHERS, a cryptographic task reformulation where a subset of tokens are ciphered (replaced with other tokens in the lexicon) via a BI￾JECTIVE mapping (e.g., each instance of “school” is replaced with “apple”.) Since this cipher is a bijection, one can recover the original format of the ICL instance, ensuring the well-defined task upon the transformations. ature’s understanding of the functional as… view at source ↗
Figure 2
Figure 2. Llama 3.1 8B performance on Amazon dataset. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Accuracy comparison of Llama-3.1-8B and Llama-3.1-8B-Instruct on four datasets under [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Accuracy comparison of Llama-3.1-8B and Llama-3.1-70B on four datasets under [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: x-axis indicates the i-th occurrence of ciphered tokens in the Llama 3.1 context. y-axis indicates the rank difference (Eq 1). Positive values (red) indicate the model’s preference for substituted tokens over original ones. In the BIJECTIVE cipher (left), we see a pref…
Figure 6
Figure 6. Figure 6: Llama 3.1 8B performance on SST-2 dataset, which shows similar trends with Fig. [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Peformance of Llama 3.1 8B on SST-2 dataset with non-priority sampling, comparing with Fig. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Peformance of Llama 3.1 8B on Amazon dataset with non-priority sampling, comparing with Fig. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Complete heatmap of original token rank minus substitution token rank on Amazon for Fig. [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Average rank differences (original token rank - substitution token rank) in SST-2 (left) and Amazon [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Controlling the Risk of Corrupted Contexts for Language Models via Early-Exiting

    cs.AI 2025-10 conditional novelty 6.0 of 10

    An early-exit rule with a zero-shot fallback, calibrated by Learn-then-Test risk control, keeps the average loss from corrupted in-context demonstrations under a preset bound.

Reference graph

Works this paper leans on

71 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Ekin Aky \"u rek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2022. https://arxiv.org/abs/2211.15661 What learning algorithm is in-context learning? investigations with linear models . In International Conference on Learning Representations ICLR

    arXiv 2022

  4. [4]

    Nada Aldarrab and Jonathan May. 2020. https://aclanthology.org/2021.acl-long.561.pdf Can sequence-to-sequence models crack substitution ciphers? Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 7226--7235

    work page 2020

  5. [5]

    Suraj Anand, Michael A Lepori, Jack Merullo, and Ellie Pavlick. 2024. https://arxiv.org/pdf/2406.00053 Dual process learning: Controlling use of in-context vs. in-weights strategies with weight forgetting . arXiv preprint arXiv:2406.00053

    arXiv 2024

  6. [6]

    Taylor Berg-Kirkpatrick, Greg Durrett, and Dan Klein. 2013. https://aclanthology.org/P13-1021/ Unsupervised transcription of historical documents . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 207--217

    work page 2013

  7. [7]

    Taylor Berg-Kirkpatrick and Dan Klein. 2011. https://aclanthology.org/D11-1029/ Simple effective decipherment via combinatorial optimization . In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 313--321

    work page 2011

  8. [8]

    O'Reilly Media, Inc

    Steven Bird, Ewan Klein, and Edward Loper. 2009. https://www.nltk.org/book/ Natural language processing with Python: analyzing text with the natural language toolkit . " O'Reilly Media, Inc."

    work page 2009

Show all 71 references
  1. [9]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. https://arxiv.org/abs/2005.14165 Language models are few-shot learners . Advances in Neural Information Proces...

    2020 arXiv

  2. [10]

    Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. 2022. https://arxiv.org/abs/2205.05055 Data distributional properties drive emergent in-context learning in transformers . Advances in Neural Informatio...

    2022 arXiv

  3. [11]

    Eric Corlett and Gerald Penn. 2010. https://aclanthology.org/P10-1106/ An exact a* method for deciphering letter-substitution ciphers . In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1040--1047

    2010

  4. [12]

    Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. 2023. https://doi.org/10.18653/v1/2023.findings-acl.247 Why can GPT learn in-context? language models secretly perform gradient descent as meta-optimizers . In Findings of the Association for Compu...

    2023 doi

  5. [13]

    Verna Dankers and Ivan Titov. 2024. https://arxiv.org/pdf/2408.04965 Generalisation first, memorisation second? memorisation localisation for natural language classification tasks . arXiv preprint arXiv:2408.04965

    2024 arXiv

  6. [14]

    Qing Dou and Kevin Knight. 2012. https://aclanthology.org/D12-1025/ Large scale decipherment for out-of-domain machine translation . In Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning, pag...

    2012

  7. [15]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  8. [16]

    https://dumps.wikimedia.org Wikimedia downloads

    Wikimedia Foundation. https://dumps.wikimedia.org Wikimedia downloads

  9. [17]

    Yao Fu, Hao Peng, and Tushar Khot. 2022. https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1 How does gpt obtain its ability? tracing emergent abilities of language models to ...

    2022

  10. [18]

    Ravi Ganesan and Alan T Sherman. 1993. https://www.tandfonline.com/doi/abs/10.1080/0161-119391867980 Statistical techniques for language recognition: An introduction and guide for cryptanalysts . Cryptologia, 17(4):321--366

    1993 doi

  11. [19]

    Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. 2022. https://arxiv.org/abs/2208.01066 What can transformers learn in-context? a case study of simple function classes . Advances in Neural Information Processing Systems NeurIPS , 35:30583--30598

    2022 arXiv

  12. [20]

    Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. 2024. Patchscopes: A unifying framework for inspecting hidden representations of language models. In Forty-first International Conference on Machine Learning

    2024

  13. [21]

    Shahriar Golchin, Mihai Surdeanu, Steven Bethard, Eduardo Blanco, and Ellen Riloff. 2024. https://arxiv.org/pdf/2408.11546 Memorization in in-context learning . arXiv preprint arXiv:2408.11546

    2024 arXiv

  14. [22]

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, et al. 2024. https://arxiv.org/pdf/2402.00838 Olmo: Accelerating the science of language models . Preprint

    2024 arXiv

  15. [23]

    Michael Hahn and Navin Goyal. 2023. https://arxiv.org/abs/2303.07971 A theory of emergent in-context learning as implicit structure induction . arXiv preprint arXiv:2303.07971

    2023 arXiv

  16. [24]

    Xiaochuang Han, Daniel Simig, Todor Mihaylov, Yulia Tsvetkov, Asli Celikyilmaz, and Tianlu Wang. 2023. https://arxiv.org/pdf/2306.15091 Understanding in-context learning via supportive pretraining data . In Annual Meeting of the Association for Computational Linguistics ACL , ...

    2023 arXiv

  17. [25]

    Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley. 2024. https://arxiv.org/pdf/2403.03952 Bridging language and items for retrieval and recommendation . arXiv preprint arXiv:2403.03952

    2024 arXiv

  18. [26]

    Juno Kim and Taiji Suzuki. 2024. https://arxiv.org/pdf/2402.01258 Transformers learn nonlinear features in context: Nonconvex mean-field dynamics on the attention landscape . International Conference on Machine Learning ICML

    2024 arXiv

  19. [27]

    Kevin Knight, Anish Nair, Nishit Rathod, and Kenji Yamada. 2006. https://mt-archive.net/Coling-ACL-2006-Knight.pdf Unsupervised analysis for decipherment problems . In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 499--506

    2006

  20. [28]

    Shuai Li, Zhao Song, Yu Xia, Tong Yu, and Tianyi Zhou. 2023. https://arxiv.org/abs/2304.13276 The closeness of in-context learning and weight shifting for softmax regression . arXiv preprint arXiv:2304.13276

    2023 arXiv

  21. [29]

    Ziqian Lin and Kangwook Lee. 2024. https://arxiv.org/pdf/2402.18819 Dual operating modes of in-context learning . In International Conference on Machine Learning ICML

    2024 arXiv

  22. [30]

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. https://arxiv.org/pdf/2104.08786.pdf Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity . In Annual Meeting of the Association for Computational L...

    2022 arXiv

  23. [31]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. https://arxiv.org/abs/2212.10511 When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories . In Annual Meeting of...

    2023 arXiv

  24. [32]

    Quinn McNemar. 1947. https://doi.org/10.1007/BF02295996 Note on the sampling error of the difference between correlated proportions or percentages . Psychometrika, 12(2):153–157

    1947 doi

  25. [33]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. https://arxiv.org/abs/2202.12837 Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? arXiv preprint arXiv:2202.12837

    2022 arXiv

  26. [34]

    Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. 2022. https://arxiv.org/abs/2109.07830 Reframing instructional prompts to gptk's language . In Annual Meeting of the Association for Computational Linguistics ACL - Findings

    2022 arXiv

  27. [35]

    Aaron Mueller, Albert Webson, Jackson Petty, and Tal Linzen. 2024. https://arxiv.org/pdf/2311.07811 In-context learning generalizes, but not always robustly: The case of syntax . In Conference of the North American Chapter of the Association for Computational Linguistics NAACL...

    2024 arXiv

  28. [36]

    Aliakbar Nafar, Kristen Brent Venable, and Parisa Kordjamshidi. 2024. https://www.arxiv.org/pdf/2409.04318 Learning vs retrieval: The role of in-context examples in regression with llms . arXiv preprint arXiv:2409.04318

    2024 arXiv

  29. [37]

    nostalgebraist. 2020. https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens Interpreting gpt: The logit lens

    2020

  30. [38]

    Malte Nuhn, Julian Schamper, and Hermann Ney. 2013. https://aclanthology.org/P13-1154.pdf Beam search for solving substitution ciphers . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1568--1576

    2013

  31. [39]

    Edwin Olson. 2007. https://www.apprendre-en-ligne.net/crypto/bibliotheque/PDF/olson2007crypt.pdf Robust dictionary attack of short simple substitution ciphers . Cryptologia, 31(4):332--342

    2007

  32. [40]

    Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. 2023. https://aclanthology.org/2023.findings-acl.527 What in-context learning `` learns '' in-context: Disentangling task recognition and task learning . In Findings of the Association for Computational Linguistics: ACL 2023

    2023

  33. [41]

    Shmuel Peleg and Azriel Rosenfeld. 1979. https://dl.acm.org/doi/10.1145/359168.359174 Breaking substitution ciphers using a relaxation algorithm . Communications of the ACM, 22(11):598--605

    1979

  34. [42]

    Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. https://proceedings.neurips.cc/paper/2021/file/5c04925674920eb58467fb52ce4ef728-Paper.pdf True few-shot learning with language models . In Advances in Neural Information Processing Systems NeurIPS

    2021

  35. [43]

    Steven T Piantadosi. 2014. https://pubmed.ncbi.nlm.nih.gov/24664880/ Zipf’s word frequency law in natural language: A critical review and future directions . Psychonomic bulletin & review, 21:1112--1130

    2014

  36. [44]

    Nima Pourdamghani and Kevin Knight. 2017. https://aclanthology.org/D17-1266/ Deciphering related languages . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2513--2518

    2017

  37. [45]

    Sujith Ravi and Kevin Knight. 2008. https://aclanthology.org/D08-1085/ Attacking decipherment problems optimally with low-order n-gram models . In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 812--819

    2008

  38. [46]

    Sujith Ravi and Kevin Knight. 2011. https://aclanthology.org/P11-1025.pdf Bayesian inference for zodiac and other homophonic ciphers . In Annual Meeting of the Association for Computational Linguistics ACL , pages 239--247

    2011

  39. [47]

    Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. https://arxiv.org/abs/2202.07206 Impact of pretraining term frequencies on few-shot reasoning . In Conference on Empirical Methods in Natural Language Processing EMNLP - Findings

    2022 arXiv

  40. [48]

    Gautam Reddy. 2023. https://arxiv.org/pdf/2312.03002 The mechanistic basis of data dependence and abrupt learning in an in-context classification task . In International Conference on Learning Representations ICLR

    2023 arXiv

  41. [49]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. https://arxiv.org/abs/1907.10641 WINOGRANDE: an adversarial winograd schema challenge at scale . In Conference on Artificial Intelligence AAAI

    2020 arXiv

  42. [50]

    Lingfeng Shen, Aayush Mishra, and Daniel Khashabi. 2024. https://arxiv.org/abs/2310.08540 Do pretrained transformers learn in-context by gradient descent? In International Conference on Machine Learning ICML

    2024 arXiv

  43. [51]

    Seongjin Shin, Sang Woo Lee, Hwijeen Ahn, Sungdong Kim, Hyoung Seok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung Woo Ha, et al. 2022. https://arxiv.org/abs/2204.13509 On the effect of pretraining corpora on in-context learning by a large-scale language mod...

    2022 arXiv

  44. [52]

    Suzanna Sia, David Mueller, and Kevin Duh. 2024. https://arxiv.org/pdf/2403.04510 Where does in-context translation happen in large language models . arXiv preprint arXiv:2403.04510

    2024 arXiv

  45. [53]

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. https://aclanthology.org/D13-1170.pdf Recursive deep models for semantic compositionality over a sentiment treebank . In Conference on Empirical Methods in N...

    2013

  46. [54]

    Jiajun Song, Zhuoyan Xu, and Yiqiao Zhong. 2024. https://www.arxiv.org/pdf/2408.09503 Out-of-distribution generalization via composition: a lens through induction heads in transformers . arXiv preprint arXiv:2408.09503

    2024 arXiv

  47. [55]

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, et al. 2023. https://arxiv.org/abs/2206.04615 Beyond the imitation game: Quantifying and extrapolating the capabilities of language models . Transactions on Machine Learning Research TMLR

    2023 arXiv

  48. [56]

    Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2023. https://arxiv.org/pdf/2210.01296 Recitation-augmented language models . International Conference on Learning Representations ICLR

    2023 arXiv

  49. [57]

    Gemma Team. 2024 a . https://doi.org/10.34740/KAGGLE/M/3301 Gemma

    2024 doi

  50. [58]

    Qwen Team. 2024 b . https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  51. [59]

    Robert Vacareanu, Vlad-Andrei Negru, Vasile Suciu, and Mihai Surdeanu. 2024. https://arxiv.org/pdf/2404.07544 From words to numbers: Your large language model is secretly a capable regressor when given in-context examples . In Conference on Language Modeling COLM

    2024 arXiv

  52. [60]

    Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Jo \ a o Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. 2023. https://arxiv.org/abs/2212.07677 Transformers learn in-context by gradient descent . In International Conference on Learning Represen...

    2023 arXiv

  53. [61]

    Xiaolei Wang, Xinyu Tang, Wayne Xin Zhao, and Ji-Rong Wen. 2024. https://arxiv.org/pdf/2406.14022 Investigating the pre-training dynamics of in-context learning: Task recognition vs. task learning . arXiv preprint arXiv:2406.14022

    2024 arXiv

  54. [62]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. https://arxiv.org/abs/2212.10560 Self-Instruct: Aligning Language Model with Self Generated Instructions . In Annual Meeting of the Association for Computatio...

    2023 arXiv

  55. [63]

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. https://arxiv.org/abs/2111.02080 An explanation of in-context learning as implicit bayesian inference . In International Conference on Learning Representations

    2021 arXiv

  56. [64]

    Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2023. https://arxiv.org/pdf/2308.06463 Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher . arXiv preprint arXiv:2308.06463

    2023 arXiv

  57. [65]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. https://doi.org/10.18653/v1/P19-1472 H ella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791--4...

    2019 doi

  58. [66]

    Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. 2023. https://arxiv.org/abs/2306.09927 Trained transformers learn linear models in-context . arXiv preprint arXiv:2306.09927

    2023 arXiv

  59. [67]

    Jiachen Zhao. 2023. https://arxiv.org/pdf/2311.03498 In-context exemplars as clues to retrieving from large associative memory . arXiv preprint arXiv:2311.03498

    2023 arXiv

  60. [68]

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. http://proceedings.mlr.press/v139/zhao21c/zhao21c.pdf Calibrate before use: Improving few-shot performance of language models . In International Conference on Machine Learning ICML , pages 12697--12706

    2021

  61. [69]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  62. [70]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  63. [71]

    International Conference on Learning Representations ICLR

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2019

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.