Pith. sign in

REVIEW 1 major objections 1 cited by

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

Ordering demonstrations for smooth conceptual progression turns many-shot CoT-ICL into effective test-time learning on reasoning tasks.

2026-06-30 21:31 UTC pith:A2CQIYS2

load-bearing objection CDS gives a modest gain on one math task but the abstract leaves the ordering effect unisolated from demo selection. the 1 major comments →

arxiv 2605.13511 v3 pith:A2CQIYS2 submitted 2026-05-13 cs.CL cs.AI

Many-Shot CoT-ICL: Making In-Context Learning Truly Learn

Tsz Ting Chung , Lemao Liu , Mo Yu , Dit-Yan Yeung This is my paper
classification cs.CL cs.AI
keywords many-shot in-context learningchain-of-thoughtdemonstration orderingreasoning taskstest-time learningcurvilinear selection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines many-shot chain-of-thought in-context learning on reasoning tasks and identifies properties distinct from non-reasoning cases. It interprets these as evidence that many-shot CoT-ICL functions as in-context test-time learning, leading to two principles: demonstrations should be easy for the model to understand and should be ordered to support smooth conceptual progression. Guided by these principles the authors introduce Curvilinear Demonstration Selection, an ordering method that produces gains of up to 5.42 percentage points on a math task with 64 demonstrations. The work reframes long context windows as structured curricula rather than retrieval buffers.

Core claim

Viewing many-shot CoT-ICL as in-context test-time learning rather than scaled pattern matching yields the principles that demonstrations should be easy to understand and ordered for smooth conceptual progression; Curvilinear Demonstration Selection based on these principles produces up to a 5.42 percentage-point gain on a math task with 64 demonstrations.

What carries the argument

Curvilinear Demonstration Selection (CDS), an ordering method that arranges demonstrations to support smooth conceptual progression from easier to more complex examples.

Load-bearing premise

The performance gains come from the proposed ordering principles rather than from other aspects of how the demonstrations were chosen or from task-specific factors.

What would settle it

An experiment on the same math task with 64 demonstrations that uses random ordering or a non-curvilinear selection method and measures whether the 5.42-point gain disappears.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Many-shot CoT-ICL on reasoning tasks benefits when the context window is treated as a learning curriculum.
  • Performance on math reasoning improves when demonstrations follow a curvilinear ordering that matches the model's understanding.
  • Long context windows can serve as structured test-time curricula beyond simple pattern retrieval.
  • Distinctive scaling behaviors appear for both non-reasoning and reasoning-oriented models under many-shot CoT-ICL.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The curvilinear ordering idea might extend to other reasoning domains if an automatic way to measure conceptual progression can be defined.
  • Models that already receive curriculum-style training during pretraining could show larger or smaller gains from CDS at test time.
  • Whether CDS helps on non-reasoning tasks depends on whether a meaningful progression dimension exists for those tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 0 minor

Summary. The paper studies many-shot chain-of-thought in-context learning (CoT-ICL) on reasoning tasks, identifying distinctive scaling properties compared to non-reasoning tasks and models. It reframes many-shot CoT-ICL as in-context test-time learning rather than pattern matching, proposes two principles (easy-to-understand demonstrations and smooth conceptual progression), and introduces Curvilinear Demonstration Selection (CDS) as an ordering method that yields up to 5.42 percentage-point gains on a math task with 64 demonstrations.

Significance. If the reported gains are shown to arise specifically from the ordering principles rather than demonstration selection, the work would provide a substantive reframing of long-context ICL as structured test-time learning, with implications for curriculum design in prompting. The empirical result on math tasks with many shots is a concrete contribution, though its attribution requires verification.

major comments (1)
  1. [Abstract / CDS description and experimental results] The abstract and method description attribute the 5.42 pp gain on the math task with 64 demonstrations to CDS ordering that enforces smooth conceptual progression (principle ii) after selecting easy demonstrations (principle i). No mention is made of controls that hold the demonstration set fixed while varying only the order (e.g., random permutation of the identical 64 examples versus CDS order). Without such isolation, the performance improvement cannot be unambiguously credited to the curvilinear progression rather than the choice of demonstrations, weakening the claim that many-shot CoT-ICL functions as structured in-context test-time learning.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for this constructive comment on isolating the contribution of ordering. We agree that the current presentation does not fully separate the effects of demonstration selection from ordering, and we will add the requested control experiments in the revision.

read point-by-point responses

  1. Referee: [Abstract / CDS description and experimental results] The abstract and method description attribute the 5.42 pp gain on the math task with 64 demonstrations to CDS ordering that enforces smooth conceptual progression (principle ii) after selecting easy demonstrations (principle i). No mention is made of controls that hold the demonstration set fixed while varying only the order (e.g., random permutation of the identical 64 examples versus CDS order). Without such isolation, the performance improvement cannot be unambiguously credited to the curvilinear progression rather than the choice of demonstrations, weakening the claim that many-shot CoT-ICL functions as structured in-context test-time learning.

    Authors: We agree that an explicit ablation holding the demonstration set fixed while varying only the order is necessary to attribute gains specifically to the curvilinear progression. CDS currently combines selection of easy-to-understand examples (principle i) with curvilinear ordering (principle ii). In the revised manuscript we will add results comparing (a) the CDS-selected set in random order versus (b) the same set in CDS order, on the same math task with 64 demonstrations. These controls will be reported alongside the existing results, with updated discussion clarifying the separate contributions of selection and ordering. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical performance claim from proposed method

full rationale

The paper reports an empirical study identifying properties of many-shot CoT-ICL, proposes CDS ordering guided by two principles, and measures a performance gain on a math task. No equations or derivations are presented that reduce to inputs by construction. No self-citations are invoked as load-bearing for uniqueness theorems or ansatzes. The central result is an observed experimental outcome rather than a fitted parameter renamed as prediction or a self-definitional claim. The analysis remains self-contained via direct reporting of task results.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, axioms, or invented entities are described in the abstract.

pith-pipeline@v0.9.1-grok · 5732 in / 934 out tokens · 24888 ms · 2026-06-30T21:31:49.136672+00:00 · methodology

0 comments
read the original abstract

While many-shot ICL achieves remarkable performance, prior studies of its scaling behavior have mainly focused on non-reasoning tasks. In this work, we study many-shot ICL on reasoning tasks, with a particular focus on many-shot chain-of-thought in-context learning (CoT-ICL). Analyzing across non-reasoning and reasoning tasks and across non-reasoning and reasoning-oriented LLMs, we identify several distinctive properties of many-shot CoT-ICL. We further interpret these findings by viewing many-shot CoT-ICL as in-context test-time learning rather than scaled pattern matching, and suggest two principles: (i) demonstrations should be easy for the target model to understand, and (ii) they should be ordered to support a smooth conceptual progression. Guided by the principle, we propose Curvilinear Demonstration Selection (CDS), a simple ordering method that yields up to a 5.42 percentage-point gain on a math task with 64 demonstrations. Overall, our results reframe the long context window from a retrieval buffer into a structured curriculum for in-context test-time learning.

Figures

Figures reproduced from arXiv: 2605.13511 by Dit-Yan Yeung, Lemao Liu, Mo Yu, Tsz Ting Chung.

Figure 1
Figure 1. Figure 1: Reframing of CoT-ICL as in-context test-time learning. 1. Introduction In-context learning (ICL) enables large language models (LLMs) to perform tasks by conditioning on a sequence of input-output demonstrations without updating their parame￾ters (Min et al., 2022; Von Oswald et al., 2023). Research has focused on improving ICL through strategies like se￾lecting effective demonstrations (Sorensen et al., 2… view at source ↗
Figure 3
Figure 3. Figure 3: Scaling disparity between model types on math reasoning tasks. Left: Llama 3.3 (non-reasoning LLM) shows negative gains. Right: QwQ (32B) and R1 (685 B) (reasoning LLM) shows clear positive scaling. Model geometry number theory DetectiveQA Qwen3-14B (en) 73.07 91.30 72.73 Qwen3-14B (dis) 65.76 88.15 72.73 Qwen3-8B (en) 67.01 84.63 69.48 Qwen3-8B (dis) 62.63 79.81 66.88 [PITH_FULL_IMAGE:figures/full_fig_p0… view at source ↗
Figure 2
Figure 2. Figure 2: Scaling disparity between task types. Performance (normalized accuracy) of non-reasoning LLMs on classification tasks (warm colors) versus reasoning tasks (cool colors). The x￾axis represents normalized accuracy (i.e., x−x¯ σx for accuracy x), while the y-axis indicates the number of in-context demonstrations. 4. Properties of CoT-ICL 4.1. Scaling with Reasoning Tasks Prior work reports that many-shot ICL … view at source ↗
Figure 4
Figure 4. Figure 4: Positive scaling of reasoning LLMs. The Qwen3 family (reasoning LLMs) demonstrates consistent performance improve￾ments with more demonstrations on math reasoning tasks. Left: Qwen3 (8B) Right: Qwen3 (14B) CoT-ICL for reasoning is unstable for non-reasoning LLMs and improves mainly for reasoning-optimized LLMs. For positive scaling effect, a common explanation for why many-shot ICL works is the retrieval h… view at source ↗
Figure 6
Figure 6. Figure 6: Standard deviation of performance across five random demonstration orders on classification tasks (warm colors) ver￾sus reasoning tasks (cool colors), where nt corresponds to num￾ber theory. Results shown for Qwen2.5 (14B) (non-reasoning) and Qwen3 (14B) (reasoning). et al., 2024). We quantify order sensitivity by sampling five random per￾mutations of the same demonstration set and measuring the standard d… view at source ↗
Figure 7
Figure 7. Figure 7: Performance of two sets of self-generated in-context CoT, including the set filtered with only correct answer(cr) and the set filtered with only wrong answer(wr). crqwen14 is prompting the LLaMA model with the in-context CoT generated by Qwen 2.5 (14B). Left: Llama 3.1 Right: Qwen 2.5 (14B) rameter sizes under the provision of dataset-provided CoT￾ICL. A plausible explanation is that the reasoning within t… view at source ↗
Figure 8
Figure 8. Figure 8: Performance of the first set of self-generated in-context CoT. firstqwen3(14b) is prompting the Qwen 3 (8B) model with the in-context CoT generated by Qwen 3 (14B). Left: Qwen 3 (8B) Right: Qwen 3 (14B) should depend on procedural content. This representation is designed to capture not only topical similarity but also the logical structures and operations expressed in the CoT rationale. For efficient and s… view at source ↗
Figure 9
Figure 9. Figure 9: Performance with original (ori), similarity(sim) and dissimilar(dis) sets averaged across three non-reasoning LLMs. The area between the two sets is filled with colors, indicating the relative performance at each point. 20 60 100 20 45 70 BANKING77 Sim>Ori/Dis Original Most Similar Most Dissimilar 20 60 100 67 69 71 DetectiveQA Ori/Dis>Sim Original Most Similar Most Dissimilar 20 60 100 36 45 56 geometry O… view at source ↗
Figure 9
Figure 9. Figure 9: Performance with original (ori), similarity(sim) and dissimilar(dis) sets averaged across three non-reasoning LLMs. The area between the two sets is filled with colors, indicating the relative performance at each point. 20 60 100 20 45 70 BANKING77 Sim>Ori/Dis Original Most Similar Most Dissimilar 20 60 100 67 69 71 DetectiveQA Ori/Dis>Sim Original Most Similar Most Dissimilar 20 60 100 36 45 56 geometry O… view at source ↗
Figure 10
Figure 10. Figure 10: Performance with original (ori), similarity(sim) and dissimilar(dis) sets averaged across two reasoning LLMs. The area between the two sets is filled with colors, indicating the relative performance at each point. B. Statistical Robustness on a New ICL Subset We compute the mean and standard deviation across five random demonstration-ordering seeds, and repeat the analy￾sis on a newly sampled ICL subset. … view at source ↗
Figure 10
Figure 10. Figure 10: Performance with original (ori), similarity(sim) and dissimilar(dis) sets averaged across two reasoning LLMs. The area between the two sets is filled with colors, indicating the relative performance at each point. B. Statistical Robustness on a New ICL Subset We compute the mean and standard deviation across five random demonstration-ordering seeds, repeating the analy￾sis on a newly sampled ICL subset. T… view at source ↗
Figure 11
Figure 11. Figure 11: Prompt for WSC task D. CDS: Details and Implementation D.1. Model Studies We design our experiments to isolate the effect of demon￾stration ordering in many-shot CoT-ICL. To this end, we focus on reasoning-oriented LLMs that exhibit a positive scaling trend with more demonstrations, since such models demonstrate in-context learning capacity and should benefit from improved ordering. We also control for co… view at source ↗
Figure 11
Figure 11. Figure 11: Prompt for WSC task We also control for confounding factors unrelated to order￾ing quality by using dataset-provided CoT rationales and answers, avoiding performance degradation due to incorrect or low-quality generated rationales (i.e., self-generated CoT in section 5.1). Under this setup, we study models that can interpret and leverage the provided CoT. Our primary evaluation uses Qwen3 (8B and 14B) acr… view at source ↗
Figure 14
Figure 14. Figure 14: Prompt for BANKING77 task Given a question, predict the label of the question. You can only make predictions from the following categories: { LIST_OF_CATEGORIES} Please predict the intent category of the FINAL utterance with the provided demonstration example queries as follows: utterance: {question_1} intent category: {label_1} ... utterance: {question_n} intent category: {label_n} Now predict the intent… view at source ↗
Figure 12
Figure 12. Figure 12: Prompt for COPA task 16 [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗
Figure 15
Figure 15. Figure 15: Prompt for NLU task The prompt for inference is presented in [PITH_FULL_IMAGE:figures/full_fig_p016_15.png] view at source ↗
Figure 15
Figure 15. Figure 15: Prompt for NLU task In the end of the response, add a summary ‘The answer is [answer ].’ Q: {question_1} A: {CoT_1} {answer_1} ... Q: {question_n} A: {CoT_n} {answer_n} ### Q: {question_t} ### A: Let’s think step by step [PITH_FULL_IMAGE:figures/full_fig_p017_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Prompt for GSM8K task Write a response that appropriately completes the request and wrap the final answer inside \boxed{}. Problem: {question_1} Solution: {CoT_with_answer_1} ... Problem: {question_n} Solution: {CoT_with_answer_n} ### Problem: {question_t} ### Solution: Let’s think step by step [PITH_FULL_IMAGE:figures/full_fig_p017_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Unified prompt for MATH task Below is an instruction that describes a task.\n Select the correct option from A/B/C/D. Answer with ’The answer is {A/ B/C/D}.’ in the end of your response.\n\n" Question: {question_1} Context: {context_1} Options: A. {option_1_1} B. {option_1_2} C. {option_1_3} D. {option_1_4} Answer: {derivation_1} The answer is {answer_1}. ... Question: {question_n} Context: {context_n} Op… view at source ↗
Figure 18
Figure 18. Figure 18: Prompt for DetectiveQA task 18 [PITH_FULL_IMAGE:figures/full_fig_p018_18.png] view at source ↗
Figure 18
Figure 18. Figure 18: Prompt for DetectiveQA task 17 [PITH_FULL_IMAGE:figures/full_fig_p017_18.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Onnes: A Physics-Grounded Multi-Agent LLM Simulator for Cryogenic Fault Diagnosis in Quantum Computing Infrastructure

    cs.AI 2026-07 conditional novelty 6.0

    A physics-grounded dilution-fridge simulator with LLM agents achieves supervised-ML parity on cryogenic fault classification using six demonstrations and no training, validated on simulated telemetry plus a real-hardw...

Reference graph

Works this paper leans on

47 extracted references · 47 canonical work pages · cited by 1 Pith paper · 5 internal anchors

  1. [1]

    In Increasing Naturalness and Flexibility in Spoken Dialogue Interaction, Lecture Notes in Electrical Engineering, pp.\ 165--183

    Benchmarking natural language understanding services for building conversational agents. In Increasing Naturalness and Flexibility in Spoken Dialogue Interaction, Lecture Notes in Electrical Engineering, pp.\ 165--183. Springer, 2021. ISBN 9789811593222. doi:10.1007/978-981-15-9323-9_15. URL https://iwsds2019.unikore.it/

    work page doi:10.1007/978-981-15-9323-9_15 2021

  2. [2]

    Many-shot in-context learning

    Agarwal, R., Singh, A., Zhang, L., Bohnet, B., Rosias, L., Chan, S., Zhang, B., Anand, A., Abbas, Z., Nova, A., et al. Many-shot in-context learning. volume 37, pp.\ 76930--76966, 2024

    work page 2024

  3. [3]

    J., Gupta, P., Oh, G., Dalmia, S., and Kolhar, P

    Baek, J., Lee, S. J., Gupta, P., Oh, G., Dalmia, S., and Kolhar, P. Revisiting in-context learning with long context language models. In Findings of the Association for Computational Linguistics: ACL 2025, pp.\ 26950--26966, 2025. URL https://aclanthology.org/2025.findings-acl.1382/

    work page 2025

  4. [4]

    Benson, J. B. Encyclopedia of infant and early childhood development. Elsevier, 2020

    work page 2020

  5. [5]

    R., and Neubig, G

    Bertsch, A., Ivgi, M., Xiao, E., Alon, U., Berant, J., Gormley, M. R., and Neubig, G. In-context learning with long-context models: An in-depth exploration. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp.\ 12119--12149, 2025

    work page 2025

  6. [6]

    Efficient intent detection with dual sentence encoders

    Casanueva, I., Tem c inas, T., Gerz, D., Henderson, M., and Vuli \'c , I. Efficient intent detection with dual sentence encoders. In Wen, T.-H., Celikyilmaz, A., Yu, Z., Papangelis, A., Eric, M., Kumar, A., Casanueva, I., and Shah, R. (eds.), Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pp.\ 38--45, Online, 2020. A...

    work page doi:10.18653/v1/2020.nlp4convai-1.5 2020

  7. [7]

    Chen, W., Ma, X., Wang, X., and Cohen, W. W. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=YfZ4ZPt8zd

    work page 2023

  8. [8]

    T., Cui, L., Liu, L., Huang, X., Shi, S., and Yeung, D.-Y

    Chung, T. T., Cui, L., Liu, L., Huang, X., Shi, S., and Yeung, D.-Y. Selection-p: Self-supervised task-agnostic prompt compression for faithfulness and transferability. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 11057--11070, Miami, Florida, USA, November 2024. Associa...

    work page doi:10.18653/v1/2024.findings-emnlp.646 2024

  9. [9]

    T., Liu, L., Yu, M., and Yeung, D.-Y

    Chung, T. T., Liu, L., Yu, M., and Yeung, D.-Y. D iv L ogic E val: A framework for benchmarking logical reasoning evaluation in large language models. In Christodoulopoulos, C., Chakraborty, T., Rose, C., and Peng, V. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, pp.\ 901--915, Suzhou, China, November 2025. Association for...

    work page doi:10.18653/v1/2025.findings-emnlp.47 2025

  10. [10]

    Training Verifiers to Solve Math Word Problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. ArXiv preprint, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  11. [11]

    Croes, G. A. A method for solving traveling-salesman problems. Operations research, 6 0 (6): 0 791--812, 1958

    work page 1958

  12. [12]

    and Shutova, E

    Crosbie, J. and Shutova, E. Induction heads as an essential mechanism for pattern matching in in-context learning. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp.\ 5049--5111, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-195-7. ...

    work page doi:10.18653/v1/2025.findings-naacl.283 2025

  13. [13]

    Deepseek-r1 incentivizes reasoning in llms through reinforcement learning

    DeepSeek-AI. Deepseek-r1. Nature, 645: 0 633--638, 2025. doi:10.1038/s41586-025-09422-z

    work page doi:10.1038/s41586-025-09422-z 2025

  14. [14]

    L., Liu, Y., Shang, N., Sun, Y., Zhu, Y., Yang, F., and Yang, M

    Guan, X., Zhang, L. L., Liu, Y., Shang, N., Sun, Y., Zhu, Y., Yang, F., and Yang, M. rstar-math: Small llms can master math reasoning with self-evolved deep thinking. In Proceedings of the 42nd International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=5zwF1GizFa

    work page 2025

  15. [15]

    LM -infinite: Zero-shot extreme length generalization for large language models

    Han, C., Wang, Q., Peng, H., Xiong, W., Chen, Y., Ji, H., and Wang, S. LM -infinite: Zero-shot extreme length generalization for large language models. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long...

    work page 2024

  16. [16]

    Measuring mathematical problem solving with the math dataset

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021

    work page 2021

  17. [17]

    Toward semantics-based answer pinpointing

    Hovy, E., Gerber, L., Hermjakob, U., Lin, C.-Y., and Ravichandran, D. Toward semantics-based answer pinpointing. In Proceedings of the first international conference on Human language technology research, 2001

    work page 2001

  18. [18]

    Exploring the role of diversity in example selection for in-context learning

    Kapuriya, J., Kaushik, M., Ganguly, D., and Bhatia, S. Exploring the role of diversity in example selection for in-context learning. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.\ 2962--2966, 2025. URL https://doi.org/10.1145/3726302.3730194

    work page doi:10.1145/3726302.3730194 2025

  19. [19]

    S., Reid, M., Matsuo, Y., and Iwasawa, Y

    Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. volume 35, pp.\ 22199--22213, 2022

    work page 2022

  20. [20]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    work page 2023

  21. [21]

    u ttler, H., Lewis, M., Yih, W.-t., Rockt \

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K \"u ttler, H., Lewis, M., Yih, W.-t., Rockt \"a schel, T., et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. volume 33, pp.\ 9459--9474, 2020

    work page 2020

  22. [22]

    D., Yue, X., and Chen, W

    Li, T., Zhang, G., Do, Q. D., Yue, X., and Chen, W. Long-context llms struggle with long in-context learning. Transactions on Machine Learning Research, 2025 a . URL https://openreview.net/forum?id=Cw2xlg0e46

    work page 2025

  23. [23]

    Test-time preference optimization: On-the-fly alignment via iterative textual feedback

    Li, Y., Hu, X., Qu, X., Li, L., and Cheng, Y. Test-time preference optimization: On-the-fly alignment via iterative textual feedback. In Proceedings of the 42nd International Conference on Machine Learning, 2025 b . URL https://openreview.net/forum?id=ArifAHrEVD

    work page 2025

  24. [24]

    Y., Ravichander, A., Lu, X., Dziri, N., Sclar, M., Chandu, K

    Lin, B. Y., Ravichander, A., Lu, X., Dziri, N., Sclar, M., Chandu, K. R., Bhagavatula, C., and Choi, Y. The unlocking spell on base llms: Rethinking alignment via in-context learning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=w...

    work page 2024

  25. [25]

    What Makes Good In-Context Examples for

    Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. What makes good in-context examples for GPT -3? In Agirre, E., Apidianaki, M., and Vuli \'c , I. (eds.), Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp.\ 100--114, Dublin, Ireland and Online,...

    work page doi:10.18653/v1/2022.deelio-1.10 2022

  26. [26]

    Luo, M., Xu, X., Dai, Z., Pasupat, P., Kazemi, M., Baral, C., Imbrasaite, V., and Zhao, V. Y. Dr.icl: Demonstration-retrieved in-context learning, 2023. URL https://arxiv.org/abs/2305.14128

    work page arXiv 2023

  27. [27]

    Introducing meta llama 3: The most capable openly available llm to date, 2024

    MetaAI. Introducing meta llama 3: The most capable openly available llm to date, 2024. URL https://ai.meta.com/blog/meta-llama-3/

    work page 2024

  28. [28]

    Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

    Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.\ 11048--11064, Abu Dhabi, United Arab Emirat...

    work page doi:10.18653/v1/2022.emnlp-main.759 2022

  29. [29]

    In-context Learning and Induction Heads

    Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. In-context learning and inducti...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  30. [30]

    Yarn: Efficient context window extension of large language models

    Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=wHBfxhZu1u

    work page 2024

  31. [31]

    Qwen2.5 Technical Report

    Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wa...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  32. [32]

    Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning

    Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025 a . URL https://openreview.net/forum?id=4FWAwZtd2n

    work page 2025

  33. [33]

    V., Lee, J., Xu, K., and Kumar, A

    Snell, C. V., Lee, J., Xu, K., and Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025 b . URL https://openreview.net/forum?id=4FWAwZtd2n

    work page 2025

  34. [34]

    An information-theoretic approach to prompt engineering without ground truth labels

    Sorensen, T., Robinson, J., Rytting, C., Shaw, A., Rogers, K., Delorey, A., Khalil, M., Fulda, N., and Wingate, D. An information-theoretic approach to prompt engineering without ground truth labels. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: L...

    work page doi:10.18653/v1/2022.acl-long.60 2022

  35. [35]

    Transformers learn in-context by gradient descent

    Von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pp.\ 35151--35174. PMLR, 2023

    work page 2023

  36. [36]

    Wan, X., Zhou, H., Sun, R., Nakhost, H., Jiang, K., S erban, C ., and Ar k, S. \"O . From few to many: Self-improving many-shot reasoners through iterative optimization and generation. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=JBXO05r4AV

    work page 2025

  37. [37]

    Superglue: A stickier benchmark for general-purpose language understanding systems

    Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. Superglue: A stickier benchmark for general-purpose language understanding systems. volume 32, 2019

    work page 2019

  38. [38]

    V., Zhou, D., et al

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    work page 2022

  39. [39]

    Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering , url =

    Wu, Z., Wang, Y., Ye, J., and Kong, L. Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1423--1436, Toronto, Ca...

    work page doi:10.18653/v1/2023.acl-long.79 2023

  40. [40]

    arXiv preprint arXiv:2409.02465 , year=

    Xu, Z., Ye, J., Liu, X., Liu, X., Sun, T., Liu, Z., Guo, Q., Li, L., Liu, Q., Huang, X., and Qiu, X. Detectiveqa: Evaluating long-context reasoning on detective novels, 2024. URL https://arxiv.org/abs/2409.02465

    work page arXiv 2024

  41. [41]

    Qwen3 Technical Report

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., Yang, K., Yu, L., Deng, L., Li, M., Xue, M., Li, M., Zhang, P., Wang, P., Zhu, Q...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  42. [42]

    Tree of thoughts: Deliberate problem solving with large language models

    Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36: 0 11809--11822, 2023

    work page 2023

  43. [43]

    T., Zhou, C., Li, T., Lu, R., Li, J., Xu, L., Lu, H., Zhang, N., Li, J., and Zhou, J

    Yu, M., Chung, T. T., Zhou, C., Li, T., Lu, R., Li, J., Xu, L., Lu, H., Zhang, N., Li, J., and Zhou, J. Prelude: A benchmark designed to require global comprehension and reasoning over long contexts, 2025 a . URL https://arxiv.org/abs/2508.09848

    work page arXiv 2025

  44. [44]

    T., Zhang, S., Li, J., Yeung, D.-Y., and Zhou, J

    Yu, M., Liu, L., Wu, J., Chung, T. T., Zhang, S., Li, J., Yeung, D.-Y., and Zhou, J. The stochastic parrot on LLM ' s shoulder: A summative assessment of physical concept understanding. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistic...

    work page doi:10.18653/v1/2025.naacl-long.569 2025

  45. [45]

    Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models

    Zhang, Y., Li, M., Long, D., Zhang, X., Lin, H., Yang, B., Xie, P., Yang, A., Liu, D., Lin, J., Huang, F., and Zhou, J. Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025. URL https://arxiv.org/abs/2506.05176

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  46. [46]

    Automatic chain of thought prompting in large language models

    Zhang, Z., Zhang, A., Li, M., and Smola, A. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/pdf?id=5NTt8GFjUHkr

    work page 2023

  47. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

    work page