Pith. sign in

REVIEW 3 major objections 7 minor 67 references

PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from Related Example Banks

T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PromptRefine's central claim is that few-shot generation on low-resource Indic languages improves when demonstrations are drawn from merged example banks of related higher-resource languages and selected for relevance plus diversity.

desk verdict Plausible new retriever recipe for low-resource Indic languages, but the 'significant' gains are not supported because there is no held-out test split or variance—conditional accept. read the letter →

arxiv 2412.05710 v2 pith:FAIX6JRY submitted 2024-12-07 cs.CL cs.AIcs.IR

Soumya Suvra Ghosal , Soumyabrata Pal , Koyel Mukherjee , Dinesh Manocha This is my paper · ORCID
classification cs.CLcs.AIcs.IR
keywords in-contextlearningfew-shotexampleselectionlow-resourceIndiclanguagescross-lingualretrievalretrieverfine-tuningdeterminantalpointprocessesalternatingminimizationmultilingualLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models perform much worse on low-resource Indic languages because their few-shot demonstrations must be built from very small labeled pools. PromptRefine is a trainable retriever that addresses this by borrowing example banks from closely related higher-resource Indic languages, merging language-specific retrievers through alternating rounds of fine-tuning and parameter averaging, and then fine-tuning the merged retriever to choose a relevant and diverse set of demonstrations. The paper evaluates this on four text generation tasks with several 7B-8B open LLMs and with GPT-3.5/GPT-4, and reports consistent gains over zero-shot prompting and over existing retrievers such as EPR and CEIL. The headline results include a +16.07 Token-F1 gain over zero-shot and +8.26 over CEIL on cross-lingual QA, and a +4.85 chrF1 gain over CEIL on Santali-to-English translation. If the method is right, low-resource languages can be served by transferring demonstrations from related languages at prompt time, without updating the LLM.

What carries the argument

The machinery is an alternating-minimization loop. At each round the algorithm starts from shared parameters rho, fine-tunes a separate relevance-based retriever on the target-language bank and on each selected auxiliary bank (the Specialize step), then replaces rho by the average of those retriever parameters (the Merge step). The paper treats the averaged parameter vector as a shared representation space that can retrieve relevant demonstrations across languages; the iteration with best target-language validation accuracy becomes rho, which is then fine-tuned with a determinantal point process (DPP) loss. In the DPP, the probability of selecting a subset is proportional to the determinant of a similarity kernel combining each example's relevance to the test input with pairwise repulsion, so greedy MAP inference returns a diverse but relevant set. The load-bearing operation is the parameter-average merge: it is the only mechanism that transfers knowledge from the auxiliary languages into the target retriever.

What would settle it

Train a control retriever on the concatenated target and auxiliary example banks with the same relevance loss and the same DPP loss, without any alternating specialize-merge loop, and compare downstream Token-F1 and chrF1 on the same four tasks. If the control matches or beats PromptRefine, the parameter-merging step is not the active ingredient. A second, direct test would be to measure cross-lingual nearest-neighbor agreement in the merged embedding space: if retrieved target-language examples for a given query do not overlap with related auxiliary-language examples, the shared-space assumption is violated.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that in-context example selection is the bottleneck for low-resource Indic generation, and that the bottleneck can be broken by a three-part retriever: select auxiliary high-resource languages by mean-embedding similarity above a threshold; build a shared cross-lingual representation by alternating between per-language relevance fine-tuning and parameter averaging; and add a determinantal point process loss so the retrieved subset balances relevance against redundancy. Across cross-lingual QA, multilingual QA, machine translation, and cross-lingual summarization, PromptRefine outperforms Random, BM25, Top-K, Diverse, EPR, and CEIL, with absolute gains reported up to +10.21 Token-F1 over CEIL in Manipuri cross-lingual QA and up to 2.09x relative improvement over the strongest baseline. Including auxiliary data does not help EPR or CEIL much, which the paper reads as evidence that the shared representation produced by alternating minimization is what lets auxiliary examples become useful. The paper also reports that diversity fine-tuning is necessary and that random or unrelated auxiliary languages give little gain.

Load-bearing premise

The load-bearing premise is that averaging the parameters of separately fine-tuned language-specific retrievers yields one shared embedding space that can retrieve useful examples across languages; the paper supports this mostly with a single validation-accuracy curve, and if the averaged space is not aligned, the auxiliary example banks would not contribute and the reported gains would shrink.

Editorial extensions

If this is right

  • PromptRefine's gains come without changing the LLM: only the demonstration subset in the prompt changes, so the approach is a drop-in addition to existing few-shot pipelines.
  • Related auxiliary example banks can substitute for missing target-language data; ablations show that choosing unrelated or random auxiliary languages removes most of the benefit, implying language relatedness is doing real work.
  • Diversity selection is not optional decoration; removing the DPP fine-tuning step lowers generation quality.
  • The same three-step recipe is effective across QA, translation, and summarization, so the method is task-generic within the covered generation settings.
  • Proprietary LLMs also improve on translation, indicating the example selection transfers beyond the open models used to train the retrievers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: If parameter averaging is the true source of cross-lingual alignment, the same alternate-specialize-average loop could transfer to other low-resource language families with a multilingual BERT-style init and related higher-resource banks; the paper only demonstrates Indic languages.
  • Inference: The method's reliance on mean BERT embedding similarity to pick auxiliary languages suggests script and transliteration could strongly affect which languages are deemed related, and the paper does not separate script effects from linguistic relatedness.
  • Inference: A direct test of the merge assumption would be to compare PromptRefine against simply training one retriever on the concatenated banks with the same relevance and DPP losses; the paper reports no such control, so the unique role of alternating minimization remains an open question.
  • Inference: The reported gains with 16 demonstrations suggest the selected examples partly function as cross-lingual task specification; one could test whether the same selected bank works when translated into the target language, which would isolate relevance from language identity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes PromptRefine, a three-stage method for selecting in-context learning (ICL) examples for low-resource Indic languages. The method first selects related high-resource Indic languages as auxiliary example banks using a cosine-similarity threshold, then trains language-specific retrievers through an alternating minimization procedure that interleaves relevance fine-tuning on each language's example bank with parameter averaging to form a shared retriever, and finally fine-tunes the merged retriever with a DPP-based diversity objective. The authors evaluate PromptRefine on four text generation tasks—cross-lingual QA, multilingual QA, machine translation, and cross-lingual summarization—using several open-weight LLMs (LLaMA-2-7B, LLaMA-3.1-8B, Qwen-2-7B, Qwen-2.5-7B) and two proprietary models (GPT-3.5, GPT-4). They report consistent gains over baselines, including EPR and CEIL, with improvements such as +8.26 Token-F1 over CEIL on Bodo cross-lingual QA and +4.85 chrF1 over CEIL on Santali-to-English translation. The paper includes ablations on the auxiliary-language selection threshold, the number of in-context examples K, the importance of the related-language choice, the diversity fine-tuning component, and the alternating-minimization merge step.

Significance. If the reported gains hold, PromptRefine would be a practically useful approach for improving few-shot generation in low-resource languages by leveraging related high-resource language data, an important direction for multilingual NLP. The paper has several strengths: it evaluates across four diverse generation tasks and multiple LLMs, includes code release, and provides ablations for key design decisions. In particular, Figure 3 gives some direct evidence that the parameter-averaging merge improves validation accuracy over iterations, and Figures 1-2 support the importance of related-language selection and diversity training. However, the central empirical claim rests on a missing piece: the paper never states whether the main tables report held-out test results or validation results, and no variance or significance estimates are reported anywhere. Because Algorithm 1 selects the final retriever by maximizing validation accuracy, the headline gains could in principle be artifacts of selection on the evaluation set. This must be resolved before the empirical contribution can be fully assessed.

major comments (3)
  1. [Section 5.1 and Algorithm 1 (lines 11-14)] The paper does not state whether the results in Tables 1-5 are computed on a held-out test split or on the validation set D_T^val. Algorithm 1 (lines 11-14) selects ρ* by arg max over Acc(ρ, D_T^val), and the threshold δ (Fig. 4), the number of examples K (Fig. 5), and the 120-epoch schedule (Section 5) are all tuned on the same benchmark. If the tables report validation numbers, the reported gains (e.g., +8.26 Token-F1 in Table 1, +4.85 chrF1 in Table 2) may reflect favorable selection among many configurations rather than an algorithmic property. Please clarify the split and report results on a separate held-out test set.
  2. [Section 5.1, Tables 1-5] The abstract and Section 5.1 claim that PromptRefine 'significantly outperforms' baselines, but no error bars, confidence intervals, or significance tests are reported anywhere in the paper. Since ICL is known to be sensitive to the choice and order of examples, the reported single-run differences (e.g., +3.21 Token-F1 on Maithili in Table 1, +0.88 chrF1 on Manipuri in Table 3) need variance estimates across seeds or example subsets to be interpretable. This is load-bearing for the central empirical claim.
  3. [Section 4.1.2 and Figure 3] Algorithm 1 (line 10) assumes that parameter averaging of language-specific retrievers yields a shared representation space that enables effective cross-lingual retrieval. Figure 3 provides one validation-accuracy curve for a single task (cross-lingual QA on Manipuri), but there is no analysis of the geometry of the averaged embeddings and no demonstration on other tasks or languages. Please provide additional empirical evidence (e.g., retrieval accuracy or cross-lingual transfer curves for multiple tasks) or a theoretical justification that the average does not degrade language-specific knowledge.
minor comments (7)
  1. [Section 3.2] The description of BM25 as 'simply retrieves K examples with closest vector embedding to ϕ(x)' is inaccurate because BM25 is a lexical retrieval method and does not use the dense embedding ϕ; please correct this sentence.
  2. [Appendix F and Table 6] The CrossSum-In task is described in Appendix F as summarizing a non-English article into the same language, whereas Table 6 and Section 5.1 describe summarizing an English article into a low-resource language; please reconcile these descriptions.
  3. [Algorithm 1, step 7] The notation 'ϕ_i ← min_ρ L_rel(D_i; ρ)' is ambiguous because it suggests minimizing the loss with respect to the already-fixed shared weights ρ rather than fine-tuning a retriever initialized from ρ; please rewrite to clarify the update.
  4. [Section 4.1.2] The text states the alternating procedure runs 'until convergence,' but Algorithm 1 fixes the number of iterations I; please specify the stopping criterion or remove the convergence claim.
  5. [Section 8 (Limitations)] The Limitations section states that several alternative configurations were empirically tested and the proposed approach performed best, but no details or results for these alternatives are provided; please report them or remove the claim.
  6. [Figure 4 caption and surrounding text] The Figure 4 caption says the δ ablation is for cross-lingual QA on Bodo, but the text says the figure shows translation performance from three low-resource languages to English; please align the caption with the text.
  7. [Throughout] Please fix typos: 'Divsersity-induced finetuning' in Section 4.1.3, 'Conlusion' in Section 7, and the broken spacing in '2)merges' in Section 4.1.2.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PromptRefine's reported gains are empirical results against external benchmarks, and the validation-based model selection is an evaluation concern, not a definitional reduction.

full rationale

PromptRefine is an empirical retrieval pipeline rather than a mathematical derivation, so its claims do not reduce to their inputs by construction. The relevance retriever follows the external EPR framework of Rubin et al. (2021), the DPP-based diversity training follows CEIL (Ye et al., 2023), the auxiliary-language selection in Algorithm 2 is an explicit cosine-similarity threshold, and the alternating-minimization loop in Algorithm 1 is stated as parameter averaging followed by selection of the highest-validation-accuracy iterate. None of these equations defines the reported Token-F1 or chrF1 gains in terms of the fitted parameters: the gains are measured on the external IndicGenBench tasks and compared against external baselines including EPR, CEIL, BM25, and random selection. The method is evaluated with multiple LLMs and multiple tasks, and the baselines are prior published systems rather than constructions of this paper. The only potentially concerning practice is that the final retriever is chosen by validation accuracy (Algorithm 1 line 14: rho* <- arg max_rho alpha_iter, where alpha_iter is Acc(rho, D_T^val)) and that hyperparameters such as delta and K are tuned on validation curves. That is a legitimate overfitting or evaluation-protocol weakness, but it is not circularity under the stated criteria: the reported performance is not identical to the training objective by definition, and no fitted quantity is renamed as a prediction. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The central empirical claim stands on externally measured results, so the appropriate finding is no significant circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new entities. The main load-bearing assumptions are the availability of related high-resource languages and that parameter averaging yields a useful shared representation space. Hyperparameters delta and K are fitted to the validation set, and the iteration count I is selected by validation accuracy.

free parameters (6)
  • threshold delta (auxiliary language selection) = 95th percentile of cosine similarities
    Chosen via ablation on the validation set (Appendix C, Figure 4); controls which high-resource languages are included as auxiliary banks.
  • number of in-context examples K = 16
    Chosen via ablation on the validation set (Appendix C, Figure 5); the maximum tested due to compute constraints.
  • number of alternating iterations I = 10
    Set in implementation details; the best iteration is selected by validation accuracy, making I effectively tuned on validation.
  • epochs for relevance fine-tuning = 120 per iteration
    Implementation detail; not tuned across values, but a substantial compute commitment.
  • epochs for DPP fine-tuning = 10
    Implementation detail; not tuned across values.
  • learning rate = 1e-4
    Standard Adam setting; not tuned in the paper.
assumptions (5)
  • domain assumption Example banks from related high-resource Indic languages are available and accessible.
    Stated in the Limitations section; the method depends on the existence of such data.
  • domain assumption Mean BERT embedding cosine similarity reflects language relatedness and selecting languages above a percentile threshold improves ICL.
    Algorithm 2; no theoretical justification, only empirical support via Figure 1.
  • domain assumption Parameter averaging of language-specific retrievers produces a shared representation space that supports cross-language retrieval.
    Algorithm 1, Step 10; the central mechanism of the method, with no theoretical or geometric analysis beyond an empirical validation curve.
  • standard math Greedy MAP inference for DPP is a good approximation to exact MAP.
    Chen et al. 2018; standard approximation because exact MAP is NP-hard.
  • domain assumption Using the same LLM as scorer and inference model gives reliable relevance rankings for example selection.
    Implementation details; the scorer LLM determines the training signal for all retrievers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from Related Example Banks." pith.science (2026). https://pith.science/paper/FAIX6JRY

@misc{pith2026241205710,
  author       = {Pith},
  title        = {Pith review of: PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from Related Example Banks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FAIX6JRY}},
  note         = {Machine review of arXiv:2412.05710}
}
read the original abstract

Large Language Models (LLMs) have recently demonstrated impressive few-shot learning capabilities through in-context learning (ICL). However, ICL performance is highly dependent on the choice of few-shot demonstrations, making the selection of the most optimal examples a persistent research challenge. This issue is further amplified in low-resource Indic languages, where the scarcity of ground-truth data complicates the selection process. In this work, we propose PromptRefine, a novel Alternating Minimization approach for example selection that improves ICL performance on low-resource Indic languages. PromptRefine leverages auxiliary example banks from related high-resource Indic languages and employs multi-task learning techniques to align language-specific retrievers, enabling effective cross-language retrieval. Additionally, we incorporate diversity in the selected examples to enhance generalization and reduce bias. Through comprehensive evaluations on four text generation tasks -- Cross-Lingual Question Answering, Multilingual Question Answering, Machine Translation, and Cross-Lingual Summarization using state-of-the-art LLMs such as LLAMA-3.1-8B, LLAMA-2-7B, Qwen-2-7B, and Qwen-2.5-7B, we demonstrate that PromptRefine significantly outperforms existing frameworks for retrieving examples.

Figures

Figures reproduced from arXiv: 2412.05710 by the authors.

Figure 1
Figure 1. Token-F1 evaluation on the cross-lingual QA task in Manipuri, with retrievers trained using different auxiliary high-resource example banks: (1) Closely related language, (2) Random language, and (3) Unrelated language. XX), Multilingual Question Answering (XQuAD￾IN), Machine Translation (Flores-In-XX-En), and Cross-Lingual Summarization (CrossSum-IN). We refer the reader to Appendix F for a detailed de￾scription of… view at source ↗
Figure 2
Figure 2. To highlight the importance of DPP training, we compare the quality of responses generated with and without diversity-induced fine-tuning on the task of translating from Santali to English. FINE significantly improves generation quality, as measured by chrF1, compared to other retrieval methods. 6 Discussion Note, ablation studies on threshold parameter δ for selecting related languages and number of ICL examples ar… view at source ↗
Figure 4
Figure 4. We vary the threshold parameter δ to evaluate its impact on model performance. Our results show that setting δ to the 95th percentile of cosine similarity values between tar￾get language embeddings and high-resource languages yields optimal performance. The evaluation task for this experiment is cross-lingual QA on Bodo [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: We plot the chrF-1 score for different values of K ∈ {1, 2, 4, 8, 16}, where K represents the number of in￾context examples. The task for this experiment is translation from Rajasthani to English and LLM is LLAMA-3.1-8B. random sampling, with no repeating examples in e…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 6 canonical work pages

  1. [1]

    Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977

    arXiv 2020

  2. [2]

    Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Maxamed Axmed, Kalika Bali, et al. 2023. Megaverse: Benchmarking large language models across languages, modalities, models and tasks. arXiv preprint arXiv:2311.07463

    arXiv 2023

  3. [3]

    Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, et al. 2022. Building machine translation systems for the next thousand languages. arXiv preprint arXiv:2205.03983

    arXiv 2022

  4. [4]

    Christine Benard and Odile Macchi. 1973. Detection and``emission''processes of quantum particles in a``chaotic state''. Journal of mathematical physics, 14(2):155--167

    work page 1973

  5. [5]

    Alexei Borodin and Grigori Olshanski. 2000. Distributions on partitions, point processes, and the hypergeometric kernel. Communications in Mathematical Physics, 211:335--358

    work page 2000

  6. [6]

    Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    arXiv 2020

  7. [7]

    Samuel Cahyawijaya, Holy Lovenia, and Pascale Fung. 2024. Llms are few-shot in-context low-resource language learners. arXiv preprint arXiv:2403.16512

    arXiv 2024

  8. [8]

    Laming Chen, Guoxin Zhang, and Hanning Zhou. 2018. https://arxiv.org/abs/1709.05135 Fast greedy map inference for determinantal point process to improve recommendation diversity . Preprint, arXiv:1709.05135

    work page Pith review arXiv 2018

Show all 67 references
  1. [9]

    Mingda Chen, Jingfei Du, Ramakanth Pasunuru, Todor Mihaylov, Srini Iyer, Veselin Stoyanov, and Zornitsa Kozareva. 2022. Improving in-context few-shot learning via self-supervised training. arXiv preprint arXiv:2205.01703

    2022 arXiv

  2. [10]

    Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. 2022. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. arXiv preprint arXiv:2212.10559

    2022 arXiv

  3. [11]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  4. [12]

    Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. 2023. Do multilingual language models think better in english? arXiv preprint arXiv:2308.01223

    2023 arXiv

  5. [13]

    Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833

    2018 arXiv

  6. [14]

    Markus Freitag and Yaser Al-Onaizan. 2017. Beam search strategies for neural machine translation. arXiv preprint arXiv:1702.01806

    2017 arXiv

  7. [15]

    Chitale, Raghavan AK, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar, Janki Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M

    Jay Gala, Pranjal A. Chitale, Raghavan AK, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar, Janki Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M. Khapra, Raj Dabre, and Anoop Kunchukuttan. 2023. https://arxiv.org/abs/2305.16307 Indictrans2...

    2023 arXiv

  8. [16]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751

    2019 arXiv

  9. [17]

    Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting. arXiv preprint arXiv:2305.07004

    2023 arXiv

  10. [18]

    Chun-Wa Ko, Jon Lee, and Maurice Queyranne. 1995. An exact algorithm for maximum entropy sampling. Operations Research, 43(4):684--691

    1995

  11. [19]

    Alex Kulesza, Ben Taskar, et al. 2012. Determinantal point processes for machine learning. Foundations and Trends in Machine Learning , 5(2--3):123--286

    2012

  12. [20]

    Itay Levy, Ben Bogin, and Jonathan Berant. 2022. Diverse demonstrations improve in-context compositional generalization. arXiv preprint arXiv:2212.06800

    2022 arXiv

  13. [21]

    Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. 2023 a . Unified demonstration retriever for in-context learning. arXiv preprint arXiv:2305.04320

    2023 arXiv

  14. [22]

    Xiaonan Li and Xipeng Qiu. 2023. Finding support examples for in-context learning. arXiv preprint arXiv:2302.13539

    2023 arXiv

  15. [23]

    Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. 2023 b . Transformers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learning, pages 19565--19594. PMLR

    2023

  16. [24]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt- 3 ? arXiv preprint arXiv:2101.06804

    2021 arXiv

  17. [25]

    Yinpeng Liu, Jiawei Liu, Xiang Shi, Qikai Cheng, and Wei Lu. 2024. Let's learn step by step: Enhancing in-context learning ability with curriculum learning. arXiv preprint arXiv:2402.10738

    2024 arXiv

  18. [26]

    Yuli Liu, Christian Walder, and Lexing Xie. 2022. https://arxiv.org/abs/2204.11562 Determinantal point process likelihoods for sequential recommendation . Preprint, arXiv:2204.11562

    2022 arXiv

  19. [27]

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786

    2021 arXiv

  20. [28]

    Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite, and Vincent Y Zhao. 2023. Dr. icl: Demonstration-retrieved in-context learning. arXiv preprint arXiv:2305.14128

    2023 arXiv

  21. [29]

    Odile Macchi. 1975. The coincidence approach to stochastic point processes. Advances in Applied Probability, 7(1):83--122

    1975

  22. [30]

    Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2021. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943

    2021 arXiv

  23. [31]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837

    2022 arXiv

  24. [32]

    Tai Nguyen and Eric Wong. 2023. In-context example selection with influences. arXiv preprint arXiv:2302.11042

    2023 arXiv

  25. [33]

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. 2022. In-context learning and induction heads. arXiv preprint arXiv:2209.11895

    2022 arXiv

  26. [34]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  27. [35]

    Jane Pan. 2023. What in-context learning “learns” in-context: Disentangling task recognition and task learning. Master's thesis, Princeton University

    2023

  28. [36]

    Maja Popovi \'c . 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the tenth workshop on statistical machine translation, pages 392--395

    2015

  29. [37]

    Chengwei Qin, Aston Zhang, Anirudh Dagar, and Wenming Ye. 2023. In-context learning with iterative demonstration selection. arXiv preprint arXiv:2310.09881

    2023 arXiv

  30. [38]

    Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3(4):333--389

    2009

  31. [39]

    Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021. Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.08633

    2021 arXiv

  32. [40]

    Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. 2020. A mathematical exploration of why language models help solve downstream tasks. arXiv preprint arXiv:2010.03648

    2020 arXiv

  33. [41]

    Alexander Scarlatos and Andrew Lan. 2023. Reticl: Sequential retrieval of in-context examples with reinforcement learning. arXiv preprint arXiv:2305.14502

    2023 arXiv

  34. [42]

    Seongjin Shin, Sang-Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung-Woo Ha, et al. 2022. On the effect of pretraining corpora on in-context learning by a large-scale language model. arXiv preprint arXiv:2204.13509

    2022 arXiv

  35. [43]

    Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari, and Partha Talukdar. 2024. Indicgenbench: A multilingual benchmark to evaluate generation capabilities of llms on indic languages. arXiv preprint arXiv:2404.16816

    2024 arXiv

  36. [44]

    Eshaan Tanwar, Subhabrata Dutta, Manish Borthakur, and Tanmoy Chakraborty. 2023. Multilingual llms are better cross-lingual in-context learners with alignment. arXiv preprint arXiv:2305.05940

    2023 arXiv

  37. [45]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  38. [46]

    Kiran Koshy Thekumparampil, Prateek Jain, Praneeth Netrapalli, and Sewoong Oh. 2021. Sample efficient linear meta-learning by alternating minimization. arXiv preprint arXiv:2105.08306

    2021 arXiv

  39. [47]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  40. [48]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32

    2019

  41. [49]

    Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. 2024. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. Advances in Neural Information Processing Systems, 36

    2024

  42. [50]

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560

    2022 arXiv

  43. [51]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022 a . Emergent abilities of large language models. arXiv preprint arXiv:2206.07682

    2022 arXiv

  44. [52]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022 b . Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  45. [53]

    Jerry Wei, Le Hou, Andrew Lampinen, Xiangning Chen, Da Huang, Yi Tay, Xinyun Chen, Yifeng Lu, Denny Zhou, Tengyu Ma, et al. 2023 a . Symbol tuning improves in-context learning in language models. arXiv preprint arXiv:2305.08298

    2023 arXiv

  46. [54]

    Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. 2023 b . Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846

    2023 arXiv

  47. [55]

    Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, and Pascale Fung. 2021. Language models are few-shot multilingual learners. arXiv preprint arXiv:2109.07684

    2021 arXiv

  48. [56]

    Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. 2022. Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering. arXiv preprint arXiv:2212.10375

    2022 arXiv

  49. [57]

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080

    2021 arXiv

  50. [58]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  51. [59]

    Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. In International Conference on Machine Learning, pages 39818--39833. PMLR

    2023

  52. [60]

    Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang-goo Lee, and Taeuk Kim. 2022. Ground-truth labels matter: A deeper look into input-label demonstrations. arXiv preprint arXiv:2205.12685

    2022 arXiv

  53. [61]

    Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. 2021. Differentiable prompt makes pre-trained language models better few-shot learners. arXiv preprint arXiv:2108.13161

    2021 arXiv

  54. [62]

    Yiming Zhang, Shi Feng, and Chenhao Tan. 2022 a . Active example selection for in-context learning. arXiv preprint arXiv:2211.04486

    2022 arXiv

  55. [63]

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022 b . Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493

    2022 arXiv

  56. [64]

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pages 12697--12706. PMLR

    2021

  57. [65]

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910

    2022 arXiv

  58. [66]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  59. [67]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.