Pith. sign in

REVIEW 1 major objections 2 cited by

Many-Shot CoT-ICL: Making In-Context Learning Truly Learn

T0 review · 1 major / 0 minor · reviewed 2026-06-30 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Ordering demonstrations for smooth conceptual progression turns many-shot CoT-ICL into effective test-time learning on reasoning tasks.

desk verdict CDS gives a modest gain on one math task but the abstract leaves the ordering effect unisolated from demo selection. read the letter →

arxiv 2605.13511 v3 pith:A2CQIYS2 submitted 2026-05-13 cs.CL cs.AI

Tsz Ting Chung , Lemao Liu , Mo Yu , Dit-Yan Yeung This is my paper · ORCID
classification cs.CLcs.AI
keywords many-shotin-contextlearningchain-of-thoughtdemonstrationorderingreasoningtaskstest-timecurvilinearselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper examines many-shot chain-of-thought in-context learning on reasoning tasks and identifies properties distinct from non-reasoning cases. It interprets these as evidence that many-shot CoT-ICL functions as in-context test-time learning, leading to two principles: demonstrations should be easy for the model to understand and should be ordered to support smooth conceptual progression. Guided by these principles the authors introduce Curvilinear Demonstration Selection, an ordering method that produces gains of up to 5.42 percentage points on a math task with 64 demonstrations. The work reframes long context windows as structured curricula rather than retrieval buffers.

What carries the argument

Curvilinear Demonstration Selection (CDS), an ordering method that arranges demonstrations to support smooth conceptual progression from easier to more complex examples.

What would settle it

An experiment on the same math task with 64 demonstrations that uses random ordering or a non-curvilinear selection method and measures whether the 5.42-point gain disappears.

Watch

Extended reading notes

Core claim

Viewing many-shot CoT-ICL as in-context test-time learning rather than scaled pattern matching yields the principles that demonstrations should be easy to understand and ordered for smooth conceptual progression; Curvilinear Demonstration Selection based on these principles produces up to a 5.42 percentage-point gain on a math task with 64 demonstrations.

Load-bearing premise

The performance gains come from the proposed ordering principles rather than from other aspects of how the demonstrations were chosen or from task-specific factors.

Editorial extensions

If this is right

  • Many-shot CoT-ICL on reasoning tasks benefits when the context window is treated as a learning curriculum.
  • Performance on math reasoning improves when demonstrations follow a curvilinear ordering that matches the model's understanding.
  • Long context windows can serve as structured test-time curricula beyond simple pattern retrieval.
  • Distinctive scaling behaviors appear for both non-reasoning and reasoning-oriented models under many-shot CoT-ICL.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The curvilinear ordering idea might extend to other reasoning domains if an automatic way to measure conceptual progression can be defined.
  • Models that already receive curriculum-style training during pretraining could show larger or smaller gains from CDS at test time.
  • Whether CDS helps on non-reasoning tasks depends on whether a meaningful progression dimension exists for those tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 0 minor

Summary. The paper studies many-shot chain-of-thought in-context learning (CoT-ICL) on reasoning tasks, identifying distinctive scaling properties compared to non-reasoning tasks and models. It reframes many-shot CoT-ICL as in-context test-time learning rather than pattern matching, proposes two principles (easy-to-understand demonstrations and smooth conceptual progression), and introduces Curvilinear Demonstration Selection (CDS) as an ordering method that yields up to 5.42 percentage-point gains on a math task with 64 demonstrations.

Significance. If the reported gains are shown to arise specifically from the ordering principles rather than demonstration selection, the work would provide a substantive reframing of long-context ICL as structured test-time learning, with implications for curriculum design in prompting. The empirical result on math tasks with many shots is a concrete contribution, though its attribution requires verification.

major comments (1)
  1. [Abstract / CDS description and experimental results] The abstract and method description attribute the 5.42 pp gain on the math task with 64 demonstrations to CDS ordering that enforces smooth conceptual progression (principle ii) after selecting easy demonstrations (principle i). No mention is made of controls that hold the demonstration set fixed while varying only the order (e.g., random permutation of the identical 64 examples versus CDS order). Without such isolation, the performance improvement cannot be unambiguously credited to the curvilinear progression rather than the choice of demonstrations, weakening the claim that many-shot CoT-ICL functions as structured in-context test-time learning.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for this constructive comment on isolating the contribution of ordering. We agree that the current presentation does not fully separate the effects of demonstration selection from ordering, and we will add the requested control experiments in the revision.

read point-by-point responses

  1. Referee: [Abstract / CDS description and experimental results] The abstract and method description attribute the 5.42 pp gain on the math task with 64 demonstrations to CDS ordering that enforces smooth conceptual progression (principle ii) after selecting easy demonstrations (principle i). No mention is made of controls that hold the demonstration set fixed while varying only the order (e.g., random permutation of the identical 64 examples versus CDS order). Without such isolation, the performance improvement cannot be unambiguously credited to the curvilinear progression rather than the choice of demonstrations, weakening the claim that many-shot CoT-ICL functions as structured in-context test-time learning.

    Authors: We agree that an explicit ablation holding the demonstration set fixed while varying only the order is necessary to attribute gains specifically to the curvilinear progression. CDS currently combines selection of easy-to-understand examples (principle i) with curvilinear ordering (principle ii). In the revised manuscript we will add results comparing (a) the CDS-selected set in random order versus (b) the same set in CDS order, on the same math task with 64 demonstrations. These controls will be reported alongside the existing results, with updated discussion clarifying the separate contributions of selection and ordering. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical performance claim from proposed method

full rationale

The paper reports an empirical study identifying properties of many-shot CoT-ICL, proposes CDS ordering guided by two principles, and measures a performance gain on a math task. No equations or derivations are presented that reduce to inputs by construction. No self-citations are invoked as load-bearing for uniqueness theorems or ansatzes. The central result is an observed experimental outcome rather than a fitted parameter renamed as prediction or a self-definitional claim. The analysis remains self-contained via direct reporting of task results.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

No free parameters, axioms, or invented entities are described in the abstract.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Many-Shot CoT-ICL: Making In-Context Learning Truly Learn." pith.science (2026). https://pith.science/paper/A2CQIYS2

@misc{pith2026260513511,
  author       = {Pith},
  title        = {Pith review of: Many-Shot CoT-ICL: Making In-Context Learning Truly Learn},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A2CQIYS2}},
  note         = {Machine review of arXiv:2605.13511}
}
read the original abstract

While many-shot ICL achieves remarkable performance, prior studies of its scaling behavior have mainly focused on non-reasoning tasks. In this work, we study many-shot ICL on reasoning tasks, with a particular focus on many-shot chain-of-thought in-context learning (CoT-ICL). Analyzing across non-reasoning and reasoning tasks and across non-reasoning and reasoning-oriented LLMs, we identify several distinctive properties of many-shot CoT-ICL. We further interpret these findings by viewing many-shot CoT-ICL as in-context test-time learning rather than scaled pattern matching, and suggest two principles: (i) demonstrations should be easy for the target model to understand, and (ii) they should be ordered to support a smooth conceptual progression. Guided by the principle, we propose Curvilinear Demonstration Selection (CDS), a simple ordering method that yields up to a 5.42 percentage-point gain on a math task with 64 demonstrations. Overall, our results reframe the long context window from a retrieval buffer into a structured curriculum for in-context test-time learning.

Figures

Figures reproduced from arXiv: 2605.13511 by the authors.

Figure 1
Figure 1. Reframing of CoT-ICL as in-context test-time learning. 1. Introduction In-context learning (ICL) enables large language models (LLMs) to perform tasks by conditioning on a sequence of input-output demonstrations without updating their parame￾ters (Min et al., 2022; Von Oswald et al., 2023). Research has focused on improving ICL through strategies like se￾lecting effective demonstrations (Sorensen et al., 2022; Liu e… view at source ↗
Figure 3
Figure 3. Scaling disparity between model types on math reasoning tasks. Left: Llama 3.3 (non-reasoning LLM) shows negative gains. Right: QwQ (32B) and R1 (685 B) (reasoning LLM) shows clear positive scaling. Model geometry number theory DetectiveQA Qwen3-14B (en) 73.07 91.30 72.73 Qwen3-14B (dis) 65.76 88.15 72.73 Qwen3-8B (en) 67.01 84.63 69.48 Qwen3-8B (dis) 62.63 79.81 66.88 [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 2
Figure 2. Scaling disparity between task types. Performance (normalized accuracy) of non-reasoning LLMs on classification tasks (warm colors) versus reasoning tasks (cool colors). The x￾axis represents normalized accuracy (i.e., x−x¯ σx for accuracy x), while the y-axis indicates the number of in-context demonstrations. 4. Properties of CoT-ICL 4.1. Scaling with Reasoning Tasks Prior work reports that many-shot ICL yields rel… view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Positive scaling of reasoning LLMs. The Qwen3 family (reasoning LLMs) demonstrates consistent performance improve￾ments with more demonstrations on math reasoning tasks. Left: Qwen3 (8B) Right: Qwen3 (14B) CoT-ICL for reasoning is unstable for non-reasoning LLMs and im…
Figure 6
Figure 6. Figure 6: Standard deviation of performance across five random demonstration orders on classification tasks (warm colors) ver￾sus reasoning tasks (cool colors), where nt corresponds to num￾ber theory. Results shown for Qwen2.5 (14B) (non-reasoning) and Qwen3 (14B) (reasoning). e…
Figure 7
Figure 7. Figure 7: Performance of two sets of self-generated in-context CoT, including the set filtered with only correct answer(cr) and the set filtered with only wrong answer(wr). crqwen14 is prompting the LLaMA model with the in-context CoT generated by Qwen 2.5 (14B). Left: Llama 3.1…
Figure 8
Figure 8. Figure 8: Performance of the first set of self-generated in-context CoT. firstqwen3(14b) is prompting the Qwen 3 (8B) model with the in-context CoT generated by Qwen 3 (14B). Left: Qwen 3 (8B) Right: Qwen 3 (14B) should depend on procedural content. This representation is design…
Figure 9
Figure 9. Figure 9: Performance with original (ori), similarity(sim) and dissimilar(dis) sets averaged across three non-reasoning LLMs. The area between the two sets is filled with colors, indicating the relative performance at each point. 20 60 100 20 45 70 BANKING77 Sim>Ori/Dis Original…
Figure 9
Figure 9. Figure 9: Performance with original (ori), similarity(sim) and dissimilar(dis) sets averaged across three non-reasoning LLMs. The area between the two sets is filled with colors, indicating the relative performance at each point. 20 60 100 20 45 70 BANKING77 Sim>Ori/Dis Original…
Figure 10
Figure 10. Figure 10: Performance with original (ori), similarity(sim) and dissimilar(dis) sets averaged across two reasoning LLMs. The area between the two sets is filled with colors, indicating the relative performance at each point. B. Statistical Robustness on a New ICL Subset We compu…
Figure 10
Figure 10. Figure 10: Performance with original (ori), similarity(sim) and dissimilar(dis) sets averaged across two reasoning LLMs. The area between the two sets is filled with colors, indicating the relative performance at each point. B. Statistical Robustness on a New ICL Subset We compu…
Figure 11
Figure 11. Figure 11: Prompt for WSC task D. CDS: Details and Implementation D.1. Model Studies We design our experiments to isolate the effect of demon￾stration ordering in many-shot CoT-ICL. To this end, we focus on reasoning-oriented LLMs that exhibit a positive scaling trend with more …
Figure 11
Figure 11. Figure 11: Prompt for WSC task We also control for confounding factors unrelated to order￾ing quality by using dataset-provided CoT rationales and answers, avoiding performance degradation due to incorrect or low-quality generated rationales (i.e., self-generated CoT in section …
Figure 14
Figure 14. Figure 14: Prompt for BANKING77 task Given a question, predict the label of the question. You can only make predictions from the following categories: { LIST_OF_CATEGORIES} Please predict the intent category of the FINAL utterance with the provided demonstration example queries …
Figure 12
Figure 12. Figure 12: Prompt for COPA task 16 [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 15
Figure 15. Figure 15: Prompt for NLU task The prompt for inference is presented in [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 15
Figure 15. Figure 15: Prompt for NLU task In the end of the response, add a summary ‘The answer is [answer ].’ Q: {question_1} A: {CoT_1} {answer_1} ... Q: {question_n} A: {CoT_n} {answer_n} ### Q: {question_t} ### A: Let’s think step by step [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 16
Figure 16. Figure 16: Prompt for GSM8K task Write a response that appropriately completes the request and wrap the final answer inside \boxed{}. Problem: {question_1} Solution: {CoT_with_answer_1} ... Problem: {question_n} Solution: {CoT_with_answer_n} ### Problem: {question_t} ### Solutio…
Figure 17
Figure 17. Figure 17: Unified prompt for MATH task Below is an instruction that describes a task.\n Select the correct option from A/B/C/D. Answer with ’The answer is {A/ B/C/D}.’ in the end of your response.\n\n" Question: {question_1} Context: {context_1} Options: A. {option_1_1} B. {opt…
Figure 18
Figure 18. Figure 18: Prompt for DetectiveQA task 18 [PITH_FULL_IMAGE:figures/full_fig_p018_18.png]
Figure 18
Figure 18. Figure 18: Prompt for DetectiveQA task 17 [PITH_FULL_IMAGE:figures/full_fig_p017_18.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A New Role for Relevance: Guiding Corpus Interaction in Agentic Search

    cs.CL 2026-07 conditional novelty 6.0 of 10

    RARG turns relevance scores into an execution order for grep-based corpus search, improving accuracy per tool call over retrieval and direct-corpus-interaction agents.

  2. Onnes: A Physics-Grounded Multi-Agent LLM Simulator for Cryogenic Fault Diagnosis in Quantum Computing Infrastructure

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A physics-grounded dilution-fridge simulator with LLM agents achieves supervised-ML parity on cryogenic fault classification using six demonstrations and no training, validated on simulated telemetry plus a real-hardw...

Reference graph

Works this paper leans on

47 extracted references · 47 canonical work pages · cited by 2 Pith papers

  1. [1]

    In Increasing Naturalness and Flexibility in Spoken Dialogue Interaction, Lecture Notes in Electrical Engineering, pp.\ 165--183

    Benchmarking natural language understanding services for building conversational agents. In Increasing Naturalness and Flexibility in Spoken Dialogue Interaction, Lecture Notes in Electrical Engineering, pp.\ 165--183. Springer, 2021. ISBN 9789811593222. doi:10.1007/978-981-15-9323-9_15. URL https://iwsds2019.unikore.it/

    work page doi:10.1007/978-981-15-9323-9_15 2021

  2. [2]

    Many-shot in-context learning

    Agarwal, R., Singh, A., Zhang, L., Bohnet, B., Rosias, L., Chan, S., Zhang, B., Anand, A., Abbas, Z., Nova, A., et al. Many-shot in-context learning. volume 37, pp.\ 76930--76966, 2024

    work page 2024

  3. [3]

    J., Gupta, P., Oh, G., Dalmia, S., and Kolhar, P

    Baek, J., Lee, S. J., Gupta, P., Oh, G., Dalmia, S., and Kolhar, P. Revisiting in-context learning with long context language models. In Findings of the Association for Computational Linguistics: ACL 2025, pp.\ 26950--26966, 2025. URL https://aclanthology.org/2025.findings-acl.1382/

    work page 2025

  4. [4]

    Benson, J. B. Encyclopedia of infant and early childhood development. Elsevier, 2020

    work page 2020

  5. [5]

    R., and Neubig, G

    Bertsch, A., Ivgi, M., Xiao, E., Alon, U., Berant, J., Gormley, M. R., and Neubig, G. In-context learning with long-context models: An in-depth exploration. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp.\ 12119--12149, 2025

    work page 2025

  6. [6]

    Efficient intent detection with dual sentence encoders

    Casanueva, I., Tem c inas, T., Gerz, D., Henderson, M., and Vuli \'c , I. Efficient intent detection with dual sentence encoders. In Wen, T.-H., Celikyilmaz, A., Yu, Z., Papangelis, A., Eric, M., Kumar, A., Casanueva, I., and Shah, R. (eds.), Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pp.\ 38--45, Online, 2020. A...

    work page doi:10.18653/v1/2020.nlp4convai-1.5 2020

  7. [7]

    Chen, W., Ma, X., Wang, X., and Cohen, W. W. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=YfZ4ZPt8zd

    work page 2023

  8. [8]

    T., Cui, L., Liu, L., Huang, X., Shi, S., and Yeung, D.-Y

    Chung, T. T., Cui, L., Liu, L., Huang, X., Shi, S., and Yeung, D.-Y. Selection-p: Self-supervised task-agnostic prompt compression for faithfulness and transferability. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 11057--11070, Miami, Florida, USA, November 2024. Associa...

    work page doi:10.18653/v1/2024.findings-emnlp.646 2024

Show all 47 references
  1. [9]

    T., Liu, L., Yu, M., and Yeung, D.-Y

    Chung, T. T., Liu, L., Yu, M., and Yeung, D.-Y. D iv L ogic E val: A framework for benchmarking logical reasoning evaluation in large language models. In Christodoulopoulos, C., Chakraborty, T., Rose, C., and Peng, V. (eds.), Findings of the Association for Computational Lingu...

    2025 doi

  2. [10]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. ArXiv preprint, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168

    2021 arXiv

  3. [11]

    Croes, G. A. A method for solving traveling-salesman problems. Operations research, 6 0 (6): 0 791--812, 1958

    1958

  4. [12]

    and Shutova, E

    Crosbie, J. and Shutova, E. Induction heads as an essential mechanism for pattern matching in in-context learning. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp.\ 5049--5111, Albuquerque, New Mexico,...

    2025 doi

  5. [13]

    Deepseek-r1

    DeepSeek-AI. Deepseek-r1. Nature, 645: 0 633--638, 2025. doi:10.1038/s41586-025-09422-z

    2025 doi

  6. [14]

    L., Liu, Y., Shang, N., Sun, Y., Zhu, Y., Yang, F., and Yang, M

    Guan, X., Zhang, L. L., Liu, Y., Shang, N., Sun, Y., Zhu, Y., Yang, F., and Yang, M. rstar-math: Small llms can master math reasoning with self-evolved deep thinking. In Proceedings of the 42nd International Conference on Machine Learning, 2025. URL https://openreview.net/foru...

    2025

  7. [15]

    LM -infinite: Zero-shot extreme length generalization for large language models

    Han, C., Wang, Q., Peng, H., Xiong, W., Chen, Y., Ji, H., and Wang, S. LM -infinite: Zero-shot extreme length generalization for large language models. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Associ...

    2024

  8. [16]

    Measuring mathematical problem solving with the math dataset

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021

    2021

  9. [17]

    Toward semantics-based answer pinpointing

    Hovy, E., Gerber, L., Hermjakob, U., Lin, C.-Y., and Ravichandran, D. Toward semantics-based answer pinpointing. In Proceedings of the first international conference on Human language technology research, 2001

    2001

  10. [18]

    Exploring the role of diversity in example selection for in-context learning

    Kapuriya, J., Kaushik, M., Ganguly, D., and Bhatia, S. Exploring the role of diversity in example selection for in-context learning. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.\ 2962--2966, 2025. URL h...

    2025 doi

  11. [19]

    S., Reid, M., Matsuo, Y., and Iwasawa, Y

    Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. volume 35, pp.\ 22199--22213, 2022

    2022

  12. [20]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  13. [21]

    u ttler, H., Lewis, M., Yih, W.-t., Rockt \

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K \"u ttler, H., Lewis, M., Yih, W.-t., Rockt \"a schel, T., et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. volume 33, pp.\ 9459--9474, 2020

    2020

  14. [22]

    D., Yue, X., and Chen, W

    Li, T., Zhang, G., Do, Q. D., Yue, X., and Chen, W. Long-context llms struggle with long in-context learning. Transactions on Machine Learning Research, 2025 a . URL https://openreview.net/forum?id=Cw2xlg0e46

    2025

  15. [23]

    Test-time preference optimization: On-the-fly alignment via iterative textual feedback

    Li, Y., Hu, X., Qu, X., Li, L., and Cheng, Y. Test-time preference optimization: On-the-fly alignment via iterative textual feedback. In Proceedings of the 42nd International Conference on Machine Learning, 2025 b . URL https://openreview.net/forum?id=ArifAHrEVD

    2025

  16. [24]

    Y., Ravichander, A., Lu, X., Dziri, N., Sclar, M., Chandu, K

    Lin, B. Y., Ravichander, A., Lu, X., Dziri, N., Sclar, M., Chandu, K. R., Bhagavatula, C., and Choi, Y. The unlocking spell on base llms: Rethinking alignment via in-context learning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austr...

    2024

  17. [25]

    What makes good in-context examples for GPT -3? In Agirre, E., Apidianaki, M., and Vuli \'c , I

    Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. What makes good in-context examples for GPT -3? In Agirre, E., Apidianaki, M., and Vuli \'c , I. (eds.), Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integra...

    2022 doi

  18. [26]

    Luo, M., Xu, X., Dai, Z., Pasupat, P., Kazemi, M., Baral, C., Imbrasaite, V., and Zhao, V. Y. Dr.icl: Demonstration-retrieved in-context learning, 2023. URL https://arxiv.org/abs/2305.14128

    2023

  19. [27]

    Introducing meta llama 3: The most capable openly available llm to date, 2024

    MetaAI. Introducing meta llama 3: The most capable openly available llm to date, 2024. URL https://ai.meta.com/blog/meta-llama-3/

    2024

  20. [28]

    Rethinking the role of demonstrations: What makes in-context learning work? In Goldberg, Y., Kozareva, Z., and Zhang, Y

    Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods ...

    2022 doi

  21. [29]

    In-context learning and induction heads, 2022

    Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Cla...

    2022 arXiv

  22. [30]

    Yarn: Efficient context window extension of large language models

    Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview....

    2024

  23. [31]

    Qwen2.5 technical report, 2024

    Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin,...

    2024 arXiv

  24. [32]

    Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning

    Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025 a . URL https://openreview.net/forum?id=4FWAwZtd2n

    2025

  25. [33]

    V., Lee, J., Xu, K., and Kumar, A

    Snell, C. V., Lee, J., Xu, K., and Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025 b . URL https://openreview.net/forum?id=4FWAwZtd2n

    2025

  26. [34]

    An information-theoretic approach to prompt engineering without ground truth labels

    Sorensen, T., Robinson, J., Rytting, C., Shaw, A., Rogers, K., Delorey, A., Khalil, M., Fulda, N., and Wingate, D. An information-theoretic approach to prompt engineering without ground truth labels. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 6...

    2022 doi

  27. [35]

    Transformers learn in-context by gradient descent

    Von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pp.\ 35151--35174. PMLR, 2023

    2023

  28. [36]

    Wan, X., Zhou, H., Sun, R., Nakhost, H., Jiang, K., S erban, C ., and Ar k, S. \"O . From few to many: Self-improving many-shot reasoners through iterative optimization and generation. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview...

    2025

  29. [37]

    Superglue: A stickier benchmark for general-purpose language understanding systems

    Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. Superglue: A stickier benchmark for general-purpose language understanding systems. volume 32, 2019

    2019

  30. [38]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  31. [39]

    Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering

    Wu, Z., Wang, Y., Ye, J., and Kong, L. Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for...

    2023 doi

  32. [40]

    Detectiveqa: Evaluating long-context reasoning on detective novels, 2024

    Xu, Z., Ye, J., Liu, X., Liu, X., Sun, T., Liu, Z., Guo, Q., Li, L., Liu, Q., Huang, X., and Qiu, X. Detectiveqa: Evaluating long-context reasoning on detective novels, 2024. URL https://arxiv.org/abs/2409.02465

    2024

  33. [41]

    Qwen3 technical report, 2025

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., ...

    2025 arXiv

  34. [42]

    Tree of thoughts: Deliberate problem solving with large language models

    Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36: 0 11809--11822, 2023

    2023

  35. [43]

    T., Zhou, C., Li, T., Lu, R., Li, J., Xu, L., Lu, H., Zhang, N., Li, J., and Zhou, J

    Yu, M., Chung, T. T., Zhou, C., Li, T., Lu, R., Li, J., Xu, L., Lu, H., Zhang, N., Li, J., and Zhou, J. Prelude: A benchmark designed to require global comprehension and reasoning over long contexts, 2025 a . URL https://arxiv.org/abs/2508.09848

    2025

  36. [44]

    T., Zhang, S., Li, J., Yeung, D.-Y., and Zhou, J

    Yu, M., Liu, L., Wu, J., Chung, T. T., Zhang, S., Li, J., Yeung, D.-Y., and Zhou, J. The stochastic parrot on LLM ' s shoulder: A summative assessment of physical concept understanding. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.), Proceedings of the 2025 Conference of the...

    2025 doi

  37. [45]

    Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025

    Zhang, Y., Li, M., Long, D., Zhang, X., Lin, H., Yang, B., Xie, P., Yang, A., Liu, D., Lin, J., Huang, F., and Zhou, J. Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025. URL https://arxiv.org/abs/2506.05176

    2025 arXiv

  38. [46]

    Automatic chain of thought prompting in large language models

    Zhang, Z., Zhang, A., Li, M., and Smola, A. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/pdf?id=5...

    2023

  39. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed June 30, 2026 · model on record in the stance chip above.